This is the DOTA chapter I promised in Introducing oriented-det. The September retrains are on the official Task 1 server. v0.2.0 already shipped the four-family zoo; this post is what I would actually keep from the work — not the training diary.
I built oriented-det as a lightweight PyTorch stack next to MMRotate: Apache 2.0, JSON configs, no MMCV runtime. The goal was parity, not a new architecture. DOTA v1.0 is the honest benchmark. Four lessons survived.
Quote official Task 1 Link to heading
Published DOTA numbers in this series are evaluation-server VOC Task 1 (hidden test). Recipes train on trainval. Local make eval-val is a monitor. It is not the zoo headline.
| Model | Oriented-det 1× | MMRotate 1× | Oriented-det 3× |
|---|---|---|---|
| Oriented R-CNN | 76.73% | 75.69% | 74.88% |
| Rotated Faster R-CNN | 74.42% | 73.40% | 74.48% |
| Rotated FCOS | 73.07% | 71.28% | 72.91% |
| Rotated RetinaNet (circum-HBB) | 67.87% | 64.55% (HBB) | 70.70% |
MMRotate is the research reference I matched against. Oriented-det 1× sits in the same band on the matching recipes. The inference stitch is not identical — live sliding-window merge versus MMRotate’s on-disk pre-tile merge — so I do not treat a point of AP as a ranking. RetinaNet is circum-HBB on both sides of that row; it is not an OBB comparison.
Advertise and finetune from 1× for Oriented R-CNN, Faster R-CNN, and FCOS. 3× Task 1 AP50 is a drop or a wash; the 3× gain is box tightness (AP75). RetinaNet 3× is the AP50 exception. Full table: v0.2.0 zoo.
The leaky-eval trap Link to heading
DOTA zoo recipes union train and val tiles. make eval-val then scores the same val tiles that were in training. That number looks like a held-out mAP. It is not.
On 1× the leak is small (Oriented R-CNN 77.66% eval-val vs 76.73% Task 1). On 3× it is the whole story: Faster R-CNN eval-val is 83.46% while official Task 1 is 74.48%. The extra 24 epochs memorise trainval tiles. They do not buy hidden-test AP50. If you only read the training log, 3× looks like a win.
This is the obvious trap, and it is DOTA-specific. HRSC eval-val is held-out ImageSets test. Do not import the word “leaky” onto that dataset.
Two smaller cousins of the same mistake: tiled-val mAP is not full-scene mAP, and mAP@0.1 on patches is not mAP@0.5 on the server. Write down the protocol before you subtract two numbers.
The last mile was the loss Link to heading
Architecture parity — same tiling (1024 / overlap 200), same trainval merge, same horizontal RPN on Rotated Faster R-CNN — got boxes that looked right. Extended GT metrics told a different story: high classification score, mean best rIoU stuck around 0.65 on elongated classes. Ships, harbors, large vehicles, bridges. A few pixels or a few degrees off kills AP@0.5 on a thin box.
MMRotate trains that geometry with CUDA rotated IoU in the regression loss. Oriented-det stays in pure PyTorch. The recipes that closed the gap were ProbIoU on the Faster R-CNN ROI head and decoded polygon rIoU on FCOS — surrogates aligned with overlap, not another backbone. Smooth L1 on encoded le90 params is stable and leaves alignment slack. Details: ProbIoU / Faster R-CNN and the FCOS v0.2 note.
Visual quality on a harbor tile does not imply IoU ≥ 0.5. If the canary classes are ships and harbors, look at GT-alignment stats before you add an anchor scale.
Takeaway Link to heading
Quote official Task 1. Treat leaky eval-val as a convergence plot, not a paper number. Use MMRotate as a reference in the same band, not as a leaderboard to beat. When compact classes are already fine and ships still sit a few degrees off, change the box loss, not the detector name.
Hub slugs: oriented_rcnn_dota_le90_1x, rotated_faster_rcnn_dota_le90_1x, rotated_fcos_dota_le90_1x. Reports: docs/eval-reports/.
- Previous: HRSC2016 · Oriented-Det v0.2.0
- Next: Oriented-Det v0.3.0 (21 Sep)