Reading a basketball game from the TV camera.
A computer vision pipeline that takes broadcast footage of ORLEN Basket Liga games, Poland's top league, and puts every player on a flat 28 × 15 m court with his team, track and jersey number. The aim is positional data a coaching staff can use to prepare, from footage they already have.
- 0.648
- HOTA on the held-out game
- 0.91
- Person detection AP50
- 752 → 190
- Track fragments
- 87%
- Re-ID across camera cuts
Baseline with public models and no PLK training, scored on gold v0
On 100 box-gold frames, dev and held-out alike
ByteTrack against SAM 2.1 for about 156 people on 12 shots
Rank-1 for a numbered player found in another shot; chance is 27%
Scores are on gold v0, which our AI agent labelled, starting from the same public models the baseline uses. That biases absolute numbers upwards, so we use them to compare approaches. Gold v1 adds human review of the held-out game.
Positional data, without installing a single camera.
Optical tracking installed in an arena is expensive, and few European clubs have it. Every PLK game is already filmed for TV, though. If the broadcast feed can be turned into reliable positions, a club gets tracking data for its own games and for every opponent it scouts.
The camera never stops
Broadcast cameras pan, zoom and cut. Replays, close-ups and graphics are mixed into the feed, so the court must be found again on every frame of every shot.
Polish halls are not NBA arenas
Floor paint, sponsor logos and lighting differ from hall to hall. The public court model maps one PLK arena well and fails in the others.
Numbers are tiny
On 1080p highlights a far-side jersey number is a few pixels tall, often half turned, with a sponsor logo next to it.
Identity must survive contact
Ten players in matching kits cross, screen and stack up in the paint. A tracker that swaps two teammates turns every statistic downstream into fiction.
The court is the coordinate system.
A keypoint model finds court landmarks in each frame, and a homography maps the floor plane from image pixels to centimetres on a FIBA court. A player's foot point goes through the same mapping, so a box in the picture becomes a position on the floor.
Illustration drawn from one frame of a simulated possession. Orange rings are court landmarks; brackets are player detections labelled with team and number.
Nine layers, each one swappable.
Every stage writes its output to disk under a hash of its config. A new detector or tracker goes in as a backend plus a config, and is compared with the old one by a single evaluation command, on the same gold set.
- L0Done
Ingest and shots
Cut detection and camera tags keep only the central wide camera.
- L1Fine-tuning now
Court homography
33 court landmarks per frame, RANSAC fit to FIBA geometry, smoothed within a shot and never across a cut.
- L2Baseline measured
Detection
Players, referees, ball, rim and number boxes; foot points outside the court are dropped.
- L3Baseline measured
Team
SigLIP embeddings of torso crops, UMAP and k-means into two kits, fitted per clip.
- L3bProbed
Appearance
A 512-number re-ID embedding per box, the evidence that keeps an identity across occlusion and cuts.
- L4Baseline measured
Tracking and linking
Short tracklets, then an offline linking pass under the rule of five players per team on the floor.
- L5Baseline measured
Number and identity
Jersey numbers read every 5th frame, confirmed by agreement and restricted to the team's roster.
- L6Baseline measured
2D positions
Ground points projected to metres, then path cleaning. The output is the tracking2d format the replay below reads.
- L7Planned
Ball and possession
Ball in flight, holder, shots made and missed. Built last.
The starting point is Piotr Skalski's published Roboflow basketball pipeline. We follow it, and deviate only where his posts list a limitation or where Polish basketball differs from the NBA.
Six decisions that shaped the build.
Build the gold set first
Four games and twelve central shots: tracks, teams, jersey numbers where legible, and 100 box-gold frames. One game is held out and scored once per milestone. An earlier prototype had identity heuristics that were never measured. This time the baseline was scored before anyone tuned anything.
A better tracker for $3.60
ByteTrack on the public detector's boxes broke about 156 people into 752 tracklets. SAM 2.1, run on a rented L40S, brought that down to 190, and made labelling feasible. The whole prefill cost $3.60 of GPU time, logged job by job.
Draw the answer back onto the picture
The public court model maps Warsaw's hall well. In Sosnowiec it is shifted by a consistent amount, which RANSAC cannot detect, and in Sopot it mostly fails. We found this by drawing the FIBA lines back onto the frames, not from a residual. The fix is a per-hall fine-tune, with a person correcting only the flagged points on one keyframe per shot, in a single session of about 20 minutes.
Constraints are cheap accuracy
A jersey number that is not on that team's roster for that season is no longer shown. That one rule removed 60 wrongly labelled boxes, taking wrong number boxes in the demo from 195 to 135. The remaining errors, mostly 2 read as 8, are for the number-reader fine-tune.
Pick the encoder by measuring
Six published re-ID encoders were probed on the gold set. The one trained on sports broadcasts found the right teammate 86% of the time one second later and 87% of the time across camera cuts. Pedestrian encoders scored 49 to 61% across cuts, below a plain pixel-distance reference at short range.
Size the work to the human
The budget for a person's labelling was 90 minutes in total. A timed dry run took 30 minutes for 9 of 16 tracks in one shot, so the plan changed: an AI agent does every first pass, keeps an auditable log of every operation, and the person's review goes to the held-out game.
Where the public models start.
The first metrics rows: the public models used as published, with no training on Polish footage. Every fine-tune from here is measured against this.
| Split | HOTA | IDF1 | MOTA | AP50 | Number accuracy |
|---|---|---|---|---|---|
| Dev, 9 shots | 0.602 | 0.664 | 0.719 | 0.910 | 0.478 |
| Held-out, 3 shots | 0.648 | 0.710 | 0.768 | 0.911 | 0.577 |
TrackEval over players and referees on every 5th frame; AP50 is class-agnostic on 100 box-gold frames; number accuracy is the share of gold tracks with a legible number whose predicted track has that number confirmed. Gold v0, 2 November 2026.
What a coach gets once tracking is accurate.
The pipeline's 2D output takes the same shape as these eight possessions, so analysis tools can be built before the camera side is finished. Pick a play, scrub through it, and watch the gaps between attackers and defenders open and close.
The replay could not load its data.
- Most open attacker
- –
- Mean gap to nearest defender
- –
- Offence spacing (hull area)
- –
These possessions are simulated, not measured. The offence is choreographed from common set plays, and the defence follows man-to-man or zone rules with reaction delays and speed limits. Player names are left out.
From baseline to fine-tuned, layer by layer.
Each milestone closes only with a metrics row on the gold set. If a milestone does not beat the baseline, we record that and stop adding heuristics to it.
- M3
Court model per PLK hall
Fine-tuned keypoints, with reprojection error reported per arena.
- M4
Detector fine-tune
A bake-off between RF-DETR Large, YOLO26 and DEIM on one split, then ONNX export to run on the Mac.
- M6
Tracking with appearance
BoT-SORT with camera-motion compensation and re-ID, plus the linking pass.
- M7
Numbers and game-level identity
A number reader trained on PLK crops, with votes pooled across every shot of a game.
- M8–9
Positions, ball and possession
Ankle ground points, the export contract, the ball and its holder.