$ query-agent · 2026
Natural-Language Query Agent for Multi-Object Tracking
I built an AI agent that answers plain-English questions about my tracking system's results. To stop it making up numbers, every number in its answer has to come from a real calculation, or the answer is thrown away and it tries again.
View the code on GitHubaskHow many distinct tracks cross the vertical_center line in MOT17-04?
ruleThe model has no tracking data. To give a number it must call a tool.
toolcount_zone_crossings(
sequence="MOT17-04",
line_preset="vertical_center"
)
result11
checknumber in answer, tool call in trace: accepted
answer11
$ guardrail
Stopping it from making up numbers
The biggest problem with using a language model on data is that it will confidently give you a number it made up. So in my design the model never sees the tracking data at all. It can only ask one of my tools, which are normal, tested Python code that do the counting and the geometry.
- The model is told it doesn't have the data and can only answer from tool results.
- A check runs after it answers. If the answer has a number in it but no tool was called, it is rejected and the agent tries once more.
- If it still can't back the answer up, it says so honestly instead of guessing.
askWhat colour jacket is track 3 wearing?
ruleNo tool looks at clothing. The tools only know positions and frames.
toolnone available
answerI can't answer that with the available tools.
An example of the tested behaviour. The reply is the agent's exact wording.
$ tools --list
The tools
The agent has seven tools to choose from, each with a strict, typed input. I wrote the loop that picks and runs them myself, without LangChain or LlamaIndex, so I understand and can test every step.
get_track_duration()
How long one track is present
"For how many frames is track 2 present?"
get_track_trajectory()
The path a track took and how far it moved
"How far did track 5 move?"
count_unique_tracks()
Distinct people in a range of frames
"How many people appear between frames 100 and 200?"
count_zone_crossings()
Distinct tracks crossing a line
"How many tracks crossed the centre line going left to right?"
flag_loitering()
Tracks that stay longer than a set time
"Which tracks loiter for at least 5 seconds in the centre zone?"
get_sequence_summary()
Frame count, track count, mean duration
"Give me a summary of this sequence."
compare_trackers()
The same question asked of BoT-SORT and ByteTrack, with where they agree
"Do the two trackers agree on the number of unique tracks?"
$ run-eval
Testing the agent
Because the tools always give the same answer, my test harness works out the correct answer to each question by itself and marks the agent automatically, so I am not checking answers by eye.
When I ran it live on Gemini, the free tier hit its daily limit after 10 questions. My first version counted the 19 blocked requests as wrong answers, which made it look like 10 out of 29. A request that never reached the model says nothing about the model, so I fixed the metric to report them separately.
Live, Gemini 2.5 Flash10 / 10 answered correctly
- 10 correct
- 0 wrong
- 19 blocked by the rate limit
Offline, run by CI on every push44 / 44
This run uses a stand-in for the model, so it tests the tools and the routing, not a live model. Ten live answers is a small sample, and I would want to run many more before quoting an accuracy.
$ pytest
How it is built
- Tests
- 58
- Coverage
- 77%
- Python
- LLM tool-calling (Gemini API)
- Docker
- pytest
A natural-language query agent built over the MOT17 tracking pipeline, exposing tracking output to an LLM as typed, callable tools rather than raw data dumps.
- Hand-rolled tool-calling agent loop, no LangChain or LlamaIndex, across 7 deterministic tools including a differential comparator that reports where BoT-SORT and ByteTrack agree or disagree
- Hard anti-hallucination guardrail: rejects and retries any numeric answer produced without a corresponding tool call in the trace, and defers honestly when no tool can answer
- Automated eval harness with programmatically generated ground truth and exact-match grading; found and fixed a metric flaw that conflated provider/rate-limit errors with wrong answers
- 58 tests, 77% coverage, fully typed, CI-backed, Dockerised, ready for Hugging Face Spaces