--cif7Nv6BI_chunk_014_scene_018.mp4 · complete · back to vlm_gemini3.8 · all runs
The clip at its own frame rate, with the boxes drawn over it. Switching stage repaints the overlay — it is the same video, so nothing reloads. Boxes were measured 49 times and each is held until the next measurement, so a fast object drifts within its box between them; that drift is this overlay's, not the tracker's.
system instruction · 3_108 chars
You are annotating video clips to build training data for referring multi-object tracking: given a video and a natural-language description, a model must output the boxes of every object matching that description, over time.
Your job is to write the descriptions. You do NOT output coordinates, boxes, pixel positions, or any numbers about where things are. A separate detector finds the objects.
For every clip you produce a list of items. Each item has two texts that must NOT be the same:
EXPRESSION — what a person would say to single out the object or objects. It may describe motion, temporal order, interaction with other objects, and spatial relations. It should read like something a person would actually say.
Good: "the black car that's turning left"
Good: "the cyclist who overtakes the bus"
Good: "the two people who walk out of frame on the right"
NOUN_PHRASE — the same object as a plain, timeless visual category with its visible attributes, and NOTHING else. The detector is prompted with this text and looks at ONE frame at a time, so anything about time, order, or action is useless to it and makes it fail.
Allowed: category nouns, colours, materials, clothing, size, type
Forbidden: verbs and participles of action ("turning", "walks", "is moving"), temporal words ("then", "starts", "after", "first", "last"), relative clauses ("that ...", "which ...", "who ..."), ordinals and rank words ("first", "leftmost", "nearest"), and anything over six words.
"the black car that's turning left" -> "black car"
"the cyclist who overtakes the bus" -> "cyclist"
"the two people who walk out of frame on the right" -> "person"
KIND — what makes the expression hard: "motion", "appearance", "spatial", or "interaction". Aim for about 35% "motion" items: motion is the part a single-frame model cannot already solve, so it is the most valuable, but a list of only motion items is not usable training data either.
EXPECTED_COUNT — how many distinct objects in the clip match the EXPRESSION. Count carefully; this is checked against the detector's answer, and a wrong count discards the item.
IS_NEGATIVE — at most ONE item may be a negative, and only when you also write at least one non-negative item. A negative is an expression that recombines attributes that ARE present in the clip into a whole that is NOT. If the only black car is moving, "the parked black car" is a good negative. Its NOUN_PHRASE must still be the category that IS present ("black car"), because that is what makes the absence checkable rather than merely unverified. A negative's EXPECTED_COUNT is 0.
Rules that discard an item, so follow them exactly:
- Never write coordinates, box values, pixel positions, or percentages of the frame.
- Never let a NOUN_PHRASE contain a verb of action, a temporal word, a relative clause, or an ordinal.
- Never write an EXPRESSION whose object is not visible in the clip, unless it is marked IS_NEGATIVE.
- Prefer objects that are clearly visible and identifiable over background clutter.
- Do not describe the camera or the shot; describe objects in the scene.
user turn · 183 chars
Watch this clip and write between 1 and 3 items following the rules. Cover different objects rather than restating one object several ways, and include any negative in the same list.
raw JSON response · 610 chars
{
"items": [
{
"expression": "the woman who enters the frame on the left",
"noun_phrase": "woman",
"kind": "motion",
"expected_count": 1,
"is_negative": false
},
{
"expression": "the postcard display stand outside the cafe entrance",
"noun_phrase": "display stand",
"kind": "spatial",
"expected_count": 1,
"is_negative": false
},
{
"expression": "a person sitting on the stone bench outside the cafe",
"noun_phrase": "person",
"kind": "spatial",
"expected_count": 0,
"is_negative": true
}
]
}
3 items came back; 3 passed the gate. The gate is what stops a temporal noun phrase reaching the detector, which grounds one frame at a time and cannot use "starts moving".
| kind | expression → noun phrase | expects |
|---|---|---|
| motion | the woman who enters the frame on the left detector is prompted with woman | 1 |
| spatial | the postcard display stand outside the cafe entrance detector is prompted with display stand | 1 |
| spatial negative | a person sitting on the stone bench outside the cafe detector is prompted with person | 0 |
One row per (text prompt, detector object id). Several rows sharing a phrase means the detector believed it found several distinct objects, not that one track was split.
| id | prompt | frames | span | score |
|---|---|---|---|---|
| obj 0 | person | 14 | 8.84–12.48 s | 0.93 |
| obj 0 | woman | 14 | 8.84–12.48 s | 0.94 |
| id | prompt | frames | score | mean area | outcome | why |
|---|---|---|---|---|---|---|
| obj 0 | person | 14 | 0.93 | 0.1973 | shown as 1 | survived every gate |
| obj 0 | woman | 14 | 0.94 | 0.1971 | merged away | same pixels as person obj 0 — IoU 1.00 over the frames they share |
This video and the sentences are the entire input to step 4b. Each frame is one numbered choice with its number burned into the top left, so if a number is unreadable here the model could not read it either. Frame times: 0.0, 1.0, 2.1, 3.1, 4.2, 5.2, 6.2, 7.3, 8.3, 9.4 … s.
| # | expression |
|---|---|
| 1 | the woman who enters the frame on the left |
| 2 | the postcard display stand outside the cafe entrance |
| 3 | a person sitting on the stone bench outside the cafe |
| matched ids | for | frames | confidence |
|---|---|---|---|
| 1 | the woman who enters the frame on the left | 10–13 | 1.00 |
| none — a rejection | the postcard display stand outside the cafe entrance | 1–13 | 1.00 |
| none — a rejection | a person sitting on the stone bench outside the cafe | 1–13 | 1.00 |
No box changes at this step: the model chooses among the numbers above and names a frame range. What that leaves is drawn in the last step.
You are labelling video data for referring multi-object tracking. You will see a numbered sequence of frames from one clip, in time order. Each frame has its own number and timestamp printed in the top-left corner. Candidate objects have been detected for you and drawn on the frames as coloured boxes, each labelled with a number. The same object keeps the same number and the same colour on every frame. Below each frame is a strip of enlarged crops of those same numbered boxes, so you can identify a small or distant object. For each description you are given, answer two questions: 1. MATCHING_IDS — which of the numbered objects the description refers to. List every number that matches, or an EMPTY LIST if none of them do. An empty list is a correct and expected answer: some descriptions are deliberately about something that is not in this clip. Do not pick a number just to have picked one, and do not pick a number for an object that merely resembles the description. 2. START_FRAME and END_FRAME — the frame numbers between which the described state actually holds. If the description is "the car that's turning left", the interval is the frames in which that car is turning, NOT every frame in which the car is visible. Use the frame numbers printed on the frames. START_FRAME must not be greater than END_FRAME. When MATCHING_IDS is empty, give the full range (the first and last frame numbers); it is ignored. 3. CONFIDENCE — how sure you are of the MATCHING_IDS answer, from 0.0 to 1.0. Be honest: a low number costs the example nothing, and an overconfident wrong answer becomes a wrong label. Rules: - Only use numbers that are actually drawn on the frames. - Never output coordinates, box values, or pixel positions. - Judge motion by comparing the object's position across frames, not by guessing from one frame.
The video has 13 frames, numbered 1 to 13. Each frame is one moment of the clip and has its number burned into the top left corner. Answer for each of the following descriptions, in the order given: 1. the woman who enters the frame on the left 2. the postcard display stand outside the cafe entrance 3. a person sitting on the stone bench outside the cafe
{
"answers": [
{
"description_index": 1,
"matching_ids": [
1
],
"start_frame": 10,
"end_frame": 13,
"confidence": 1.0
},
{
"description_index": 2,
"matching_ids": [],
"start_frame": 1,
"end_frame": 13,
"confidence": 1.0
},
{
"description_index": 3,
"matching_ids": [],
"start_frame": 1,
"end_frame": 13,
"confidence": 1.0
}
]
}| kind | expression | tracks | boxes |
|---|---|---|---|
| motion | the woman who enters the frame on the left | 1 | 12 |
| spatial negative | a person sitting on the stone bench outside the cafe | 0 | 0 |
| gate | expression |
|---|---|
| count_mismatch | the postcard display stand outside the cafe entrance |
{
"storage_key": "s3://mv-ytun-external1/silver/test_vehicles/deliveries/videos/delivery1/scene_cutting/v1.0/payload/scenes/--cif7Nv6BI_chunk_014_scene_018.mp4",
"scene_key": "--cif7Nv6BI_chunk_014_scene_018",
"category": "test_vehicles",
"height": 1080,
"width": 1920,
"fps": 50.0,
"num_frames": 628,
"duration": 12.56,
"rmot_expressions": [
{
"expression": "the woman who enters the frame on the left",
"kind": "motion",
"is_negative": false,
"tracks": [
{
"timestamps": [
9.36,
9.620000000000001,
9.88,
10.14,
10.4,
10.66,
10.92,
11.18,
11.44,
11.96,
12.22,
12.48
],
"boxes": [
[
0.0,
0.06666666269302368,
0.3812500238418579,
0.9944444298744202
],
[
0.0,
0.05833333358168602,
0.40416669845581055,
0.9814814887940884
],
[
0.0,
0.029629629105329514,
0.3218750059604645,
0.9833333306014538
],
[
0.0,
0.028703702613711357,
0.2718750238418579,
0.9759259354323149
],
[
0.0,
0.026851851493120193,
0.2916666865348816,
0.9768518395721912
],
[
0.0,
0.03055555559694767,
0.2666666805744171,
0.9777777884155512
],
[
0.0,
0.028703702613711357,
0.2526041865348816,
0.9759259354323149
],
[
0.0,
0.025925925001502037,
0.17500001192092896,
0.9787036683410406
],
[
0.0,
0.17407406866550446,
0.1015625074505806,
0.9759258776903152
],
[
0.0,
0.8305555582046509,
0.06927083432674408,
0.9768518507480621
],
[
0.0,
0.9018518328666687,
0.01770833507180214,
0.9749999791383743
],
[
0.0,
0.654629647731781,
0.0937500074505806,
0.9814814925193787
]
]
}
]
},
{
"expression": "a person sitting on the stone bench outside the cafe",
"kind": "spatial",
"is_negative": true,
"tracks": []
}
],
"rmot_expression_count": 2,
"rmot_negative_count": 1,
"rmot_motion_count": 1
}