Car Counting - Cross-Domain Evaluation
Model: V2 Density Map (trained on Project 24) | Test: CVAT Project 8 car_flow (508 images, car only)
Cross-Domain Note:
Model was trained on TSP6K + COCO images. Project 8 contains different sources (Vietnam traffic, company footage). Performance gap is expected and reveals domain adaptation needs.
Car MAE
4.61
vs same-domain: 2.90
Car RMSE
9.12
vs same-domain: 5.26
Median Error
2.0
Half the images err ≤ 2 cars
Test Images
508
GT range: 0-58 cars
Mean GT / Pred
7.4 → 3.0
Systematic under-counting
Error Distribution
Error Buckets
Error Percentiles
| Median (p50) | 2.0 | acceptable |
| p75 | 5.2 | moderate |
| p90 | 10.9 | large |
| p95 | 14.9 | very large |
| p99 | 44.9 | extreme |
| Max | 53.7 |
Same-Domain vs Cross-Domain Comparison
Project 24 validation (same-domain) vs Project 8 test (cross-domain)
| Metric | Same-Domain (P24 Val) | Cross-Domain (P8 Test) | Degradation |
|---|---|---|---|
| Car MAE | 2.90 | 4.61 | +59% |
| Car RMSE | 5.26 | 9.12 | +73% |
| Median Error | 1.66 | 2.00 | +20% |
| p95 Error | 9.79 | 14.93 | +52% |
GT vs Pred Scatter Plot
Each dot = 1 test image. Points below diagonal = under-counting (model's main failure mode).
Key Finding: Systematic Under-Counting
- Mean GT = 7.37, Mean Pred = 2.97 — model predicts 60% fewer cars than actual
- Scatter plot shows almost all points below y=x line
- Likely cause: domain gap — Project 8 images have different camera angles, lighting, vehicle appearances than training data (TSP6K + COCO)
- For GT=0 images (40% exact), model correctly predicts ~0 — false positive rate is low
- Remedy: Fine-tune V2 on Project 8 Train set (1,573 images) to close the domain gap
Visual Comparison Gallery
Top-left: Original + GT points (blue) | Top-right: Error heatmap (red=missed, blue=over) | Bottom-left: Grid count (5x5) | Bottom-right: Density map
Click image to enlarge
Recommendations
- Fine-tune on Project 8 Train: Use 1,573 labeled training images from car_flow to adapt model to this domain. Expected MAE improvement: 30-50%.
- Increase training resolution: Some Project 8 images have many small distant cars. Try 512x512 or 640x640.
- Adjust sigma: If cars in Project 8 are smaller, reduce density map sigma from 4.0 to 2.0-3.0.
- Augment with similar data: Mix Project 8 Train + BoxCars116k for better generalization.
Generated 2026-04-12 | Model: V2 Density Map (Project 24) | Eval: CVAT Project 8 car_flow Test Set