LIVIA, ILLS, Dept. of Systems Engineering, ÉTS Montréal, Canada
Weakly Supervised Object Localization trains a model to both classify an image and locate the object inside it, using only class labels. The promise is appealing: no expensive bounding boxes. The catch, hidden in plain sight, is that the field has been quietly using boxes anyway, to choose models and to set thresholds. This paper removes that crutch.
AbstractIn one paragraph
Weakly Supervised Object Localization (WSOL) trains models for classification and localization using only image-level class labels. Standard WSOL evaluation relies on manual bounding-box annotations on a validation set for model selection and on the test set for threshold estimation, which is unrealistic because such boxes are unavailable in practice and inflates reported localization accuracy. We propose a realistic WSOL evaluation protocol that needs no manual boxes: pseudo bounding boxes are generated automatically with off-the-shelf region proposal models (Selective Search, CLIP, and RPN) and filtered with a pointing-game criterion, then used for both model selection and threshold estimation. On natural-image benchmarks (CUB, ImageNet) and medical-image datasets (GlaS, Camelyon16), models selected with pseudo boxes match those selected with ground-truth boxes and outperform class-label selection with automatic thresholding, while reducing the threshold search from 1000 values to one.
01 · BackgroundWhat WSOL promises
Weakly Supervised Object Localization (WSOL) lets a deep network learn classification and localization from coarse class labels alone, sidestepping the cost of annotating datasets with bounding boxes. Class Activation Mapping (CAM) is the dominant approach: it produces a spatial heatmap that highlights the object, which is then thresholded into a box. The appeal is obvious, since labelling a class is far cheaper than drawing a box on every image.
02 · The problemThe evaluation protocol is not realistic
A WSOL method should need class labels only. Yet in practice two steps quietly reintroduce box supervision. First, model selection: choosing when to stop training and which hyper-parameters to keep needs a way to measure localization on a validation set, and the common protocol uses manually annotated ground-truth boxes for that. Second, threshold estimation: turning a heatmap into a box needs a threshold, and many works tune it directly on the test set. Both steps assume boxes that, by definition, do not exist in a weakly supervised setting.
The consequence is inflated numbers. When models are selected with ground-truth boxes and thresholds are read off the test set, reported localization accuracy is much higher than what the same method delivers once those boxes are taken away. The figure below makes the gap concrete across eight well-known methods: the realistic setting (blue, classification-based selection with automatic Otsu thresholding on validation) sits far below the box-assisted settings (red and green).
Having access to manual localization supervision during training goes against the WSOL setting, and it overestimates model performance.
03 · The insightCoarse boxes are good enough to choose a model
Two observations drive the solution. First, classification and localization converge at different times: strong localization often appears early and then degrades, while the classifier keeps improving, so picking the model with the best classifier is not the same as picking the model that localizes best. Localization cues are needed for selection. Second, those cues do not have to be perfect. The authors perturbed ground-truth boxes across ten noise levels and ran hundreds of experiments, and model selection stayed stable: the epochs chosen with noisy boxes closely matched those chosen with clean boxes. Coarse localization signal is enough to select a good model.
04 · The protocolPseudo boxes instead of manual ones
If noisy boxes work, they can be produced automatically. The protocol introduces a pseudo-annotator that labels the validation set with boxes using only class labels and an off-the-shelf region proposal model. Three sources are studied, spanning different levels of supervision:
- Selective Search (SS): a fully unsupervised method that needs no training.
- CLIP: a vision-language model prompted with class names, whose activation maps are converted into boxes.
- RPN: a class-agnostic region proposal network trained on generic detection data.
Each source returns several candidate boxes per image. To pick the right one, a pointing-game analysis harvests the CAM of a pretrained classifier, finds its peak activation, and keeps the proposal that best overlaps that peak, ranking candidates by classifier response. The CAM pointing accuracy is high (about 98.7 percent on the CUB validation split and 88 percent on ImageNet), so the filter reliably finds discriminative boxes.
The pseudo boxes are noisy, but usefully so. Their overlap with ground truth on the validation set is moderate (around 71 IoU for RPN and 69 for CLIP on CUB, lower for Selective Search), yet, as the noise study predicted, that is enough to guide selection. The examples below show how CLIP, RPN and Selective Search boxes compare against the ground truth.
05 · ThresholdOne threshold, estimated honestly
The same validation pseudo boxes solve the second problem. Instead of reading a threshold off the test set, the threshold that converts heatmaps into boxes is estimated on the validation set using the pseudo boxes, then applied unchanged to the test images. This keeps the test set untouched, and it collapses a brute-force search over 1,000 candidate thresholds down to a single estimated value.
06 · ResultsAs good as boxes, without the boxes
Across prominent WSOL methods on CUB and ImageNet, models selected and thresholded with pseudo boxes reach localization accuracy on par with models selected using ground-truth boxes, and they clearly beat the realistic class-label baseline that relies on automatic Otsu thresholding. The result holds when the analysis is extended to the GlaS and Camelyon16 medical imaging datasets, which suggests the protocol generalizes beyond natural images. The paper also recommends reporting non-thresholded IoU measures, which avoid the sensitivity of fixed thresholds such as IoU at 30, 50 or 70.
07 · TakeawaysWhy it matters
The currently used protocol pushed WSOL forward, but its reliance on manual boxes for selection and on the test set for thresholding quietly overstated progress. Replacing both with pseudo boxes from off-the-shelf proposal models gives a more honest measure of what a method would actually achieve when no boxes are available. It is a practical, drop-in alternative to a fully annotated validation set. Selecting WSOL models with no localization annotation at all remains an open problem, and this work narrows the gap toward it.
08 · CiteCitation
@inproceedings{murtaza25realistic,
title = {A Realistic Protocol for Evaluation of Weakly
Supervised Object Localization},
author = {Murtaza, S. and Belharbi, S. and Pedersoli, M.
and Granger, E.},
booktitle= {WACV},
year = {2025}
}Authored at LIVIA, ÉTS Montréal by Shakeeb Murtaza, Soufiane Belharbi, Marco Pedersoli and Eric Granger. Code and generated pseudo boxes: github.com/shakeebmurtaza/wsol_model_selection.