Can trackers tool do tracking without running model inference on every frame ?

def callback(frame, _):
    result = model.infer(frame)[0] # <---- this takes 99% of time
    detections = sv.Detections.from_inference(result)
    detections = tracker.update(detections)

my goal is to track objects on a video close to real time (20-30 fps) while being able to filter what to track using vision model