NeurIPS 2026 · Track on Evaluations and Datasets

ShadowTransferA Geographic Transfer Benchmark for Overhead Shadow Detection

Shagun Mittal  ·  Raymond A. Yeh  ·  Satish Ukkusuri
Purdue University

Lay summary. Maps of where shadows fall in aerial photos help cities measure urban heat, estimate sunlight on surfaces, and outline buildings. The programs that draw these maps are often trained on photos of one city and then used in another, but no one had a standard way to measure the accuracy lost in the move. We built ShadowTransfer, a collection of 4,500 aerial images of Chicago, Miami, and Phoenix in which people traced and checked every shadow. We train on two cities and test on the third. Every program we tested lost accuracy in the new city, the loss persisted with four times more training data, and most existing fixes did worse than standard training. Looking inside, we found the programs grow unsure about shadows while staying confident about everything else. Our public data and tools let anyone check whether a shadow map will hold up in a new city.

Paired patches from Chicago, Miami and Phoenix at 0.3 m and 0.6 m with shadow-type overlays, and a chart of within-city versus held-out-city mIoU for four architectures.
Figure 1. Holding out the test city lowers mIoU for all four architectures. Left: paired test patches from the three cities at 0.3 m/px and 0.6 m/px; the white box on each 0.6 m tile marks the paired 0.3 m footprint, and colours mark six shadow types. Right: two-class boundary-tolerant mIoU under vanilla training on the high-resolution cells, averaged over the three rotations. Open marker: within-city upper bound; filled marker: held-out city; Δ: mean gap; error bars: 95% CIs.

Shadow detection under geographic shift

Several overhead-imagery pipelines take shadow masks as input. Solar irradiance estimation uses them to separate shaded from sunlit surfaces, urban heat mapping uses them to correct surface temperature readings, and building extraction uses them to resolve occlusion boundaries. These pipelines are routinely trained on imagery from one region and deployed in another.

For overhead shadow detection, no benchmark measured what that move costs. AISD, the largest human-verified overhead shadow dataset, contains 514 images and defines no train-test split across cities. S-EO is larger but relies on automatically generated masks and reports no geographic holdout.

DatasetDomainScaleHuman QCCitiesMulti-resTransfer protocol
ISTDGround-level1,870✓–××
AISDAerial 0.3 m514✓5××
Airborne-ShadowAerialn/r✓DE only××
S-EOSatellite 0.3 m~19,000× (auto)3 (US)××
ShadowTransferAerial 0.3/0.6 m4,500✓3 (US)✓✓ LOCO

Table 1 of the paper. ISTD is ground-level and included for context only. n/r: not reported.

The dataset

ShadowTransfer contains 4,500 RGB patches of 384×384 pixels from National Agriculture Imagery Program (NAIP) orthoimagery of Chicago, Miami and Phoenix. Each city contributes 1,500 patches, split evenly across two native NAIP resolutions, 0.3 m and 0.6 m. We keep both native resolutions because NAIP releases them as separate acquisitions with different sensor characteristics. For each city and resolution, the 750 patches are split into 450 training, 150 validation and 150 test images, and 150 coordinates per city are imaged at both resolutions.

or click any patch to switch between photo and mask
Building / canyonUnder-structureTree-canopy dappleTopography-castVehicle-castThin linear

Test-split patches from the Hugging Face release. Each 0.6 m patch shares its centre coordinates with the 0.3 m patch above it; the white box marks the 0.3 m footprint. The benchmark evaluates binary masks (shadow vs. background); the six-way typing is released for downstream analyses.

How the masks were made

Seven trained annotators produced the masks in a three-phase pipeline. Phase 1 labels images from scratch with SAM-assisted tools. Phase 2 trains an interim shadow detector on Phase 1 labels and uses its predictions to seed later batches, which annotators then edit. Phase 3 is a retroactive quality-control pass that compares every segment with a final-model prediction and returns disagreements for adjudication. Every 100-image batch includes 5 gold images with frozen reference masks, and Cohen's κ on these images tracks annotator drift (phase-3 mean κ = 0.89).

Three-phase annotation pipeline: SAM-assisted manual labeling, model-in-the-loop acceleration, retroactive quality control.
Masks are produced in three phases, and each phase's output seeds the next. Agreement statistics and adjudication rules are in Appendix B of the paper.
Dataset statistics: sample patches, shadow coverage per city, median blob size by category, shadow-type distribution, and intensity distributions.
The three cities differ in shadow coverage, blob size, type mix and intensity. (A) sample patches with overlays; (B) shadow coverage per image; (C) median shadow blob size by category; (D) shadow-type mix; (E) mean grayscale intensity. Appendix C of the paper.

The leave-one-city-out benchmark

For each held-out city, we train on the other two cities (225 images each) and test on the held-out city's 150 test images. The within-city upper bound trains on the held-out city's own 450 training images and tests on the same 150 images. Because both models see 450 training images, the gap between them isolates the change of city from training-set size. We score with boundary-tolerant mIoU: the mean of shadow and background IoU, with a ±2-pixel don't-care band because shadow boundaries are imprecise to annotate.

Diagram of the three leave-one-city-out rotations and the within-city upper bound.

Under vanilla training, holding out the test city costs 3.5 to 7.8 mIoU across four architectures (MAMNet, OGLANet, DINOv3 and the remote-sensing foundation model CrossEarth). Across all 158 test cells, 157 degrade significantly. Six of eight adaptation methods degrade more than vanilla training on average, and the cost of a method depends on the architecture. ISW costs DINOv3 −18.5 mIoU against −11.1 on OGLANet, while MRFP+ costs OGLANet −14.0 against −4.9 on DINOv3. Without access to the target city, a practitioner has no basis for choosing among them.

Alternative explanations for the gap

We tested four alternative explanations for the gap: training-set scale, appearance statistics, encoder fine-tuning, and the pretraining and fine-tuning regime.

Training-set scale

Quadrupling the LOCO training set from 225 to 900 images narrows the gap by 0.80 mIoU (95% CI [0.46, 1.14]), from 6.48 to 5.50. The upper bound stays significantly ahead in every 900-image cell.

Appearance statistics

Matching each test image's pixel histogram to the training cities improves none of the 9 vanilla cells and lowers mean mIoU by 0.45. Retraining with strong appearance-only augmentation leaves six of nine gaps within twice their seed-to-seed spread.

Encoder fine-tuning

With the DINOv3 encoder frozen and only the decoder trained, the upper bound still exceeds LOCO by 7.10, 2.67 and 2.75 mIoU on Chicago, Miami and Phoenix.

Pretraining and learning rate

We trained MAMNet with a ResNet-50 encoder under supervised or self-supervised pretraining, each at a high or low learning rate. Encoder retraining recovers all four versions (median R 0.80 to 0.87).

Limits of these tests: three cities cannot separate a Phoenix-specific appearance effect from Phoenix's larger seed variance, and pretraining scale remains confounded between DINOv3 and the self-supervised ResNet-50. Full protocols and caveats are in Appendix H.

Diagnosing the transfer failure

Three controlled probes on the LOCO models locate the failure. They are ordered by what a practitioner would try first.

Input, ground truth and predicted shadow probability under upper bound, LOCO and SIB for three architectures.
Under LOCO, predicted shadow probability decreases over true shadow regions while the background stays dark. Columns: input, ground truth (white = shadow), and predicted shadow probability under the within-city upper bound, LOCO and SIB. p̄S: mean predicted shadow probability inside true shadow.

1. Histogram alignment lowers mIoU

Matching each test image's intensity histogram to the training cities lowers mean mIoU by 0.45 across the 9 vanilla LOCO cells, and no cell improves. At the median, matched images lie 2.5 to 3.7 times as far from the upper bound as the unmatched LOCO predictions. A pixel-level shortcut would predict partial recovery, so this result indicates that an effective intervention must act on learned features.

2. The selective-prediction gap concentrates on shadow pixels

Deployed systems use confidence-thresholded predictions, so we measured how threshold-conditioned error rises under transfer. On shadow pixels the gap averages +0.136 across 63 cells; on background pixels it averages +0.006. True-positive logits drop by 0.85 nats on average, while false-positive logits change by −0.15 nats. The same foreground-concentrated pattern appears when we repeat the protocol on INRIA building footprints, a task whose foreground class is defined by building shape.

3. The recoverable component depends on the architecture

Retraining only the decoder on 25% of the held-out city's training data recovers DINOv3 (R = 0.69) and MAMNet (R = 0.67) and leaves OGLANet near its LOCO level (R = 0.12). Retraining only the encoder lowers DINOv3 below LOCO (R = −0.13) and recovers both CNNs (R = 0.88 MAMNet, R = 0.86 OGLANet). This architecture dependence is consistent with the adaptation results above, where ISW and MRFP+ each act on a fixed component and their cost differs across architectures.

Risk-coverage curves, per-method foreground versus background AURC gap, and recovery ratios under decoder and encoder retraining.
Figure 3. The selective-prediction gap concentrates on shadow pixels, and the recoverable component depends on the architecture. (a) Risk-coverage curves on shadow pixels (MAMNet, Phoenix held out); inset: the foreground gap replicates on INRIA. (b) Per-method AURC gap: concentrated on shadow pixels, with a pooled background gap 24× smaller. (c) Recovery ratio R under decoder (Exp A) and encoder (Exp B) retraining; 1 means full recovery.

We repeated the benchmark and the three probes on S-EO (WorldView-3 imagery of Omaha, Jacksonville and San Diego). The upper bound exceeds LOCO in all 12 (architecture, held-out city) pairs, and histogram matching lowers mIoU in all 36 vanilla cells. On S-EO the background probability also shifts in 10 of 12 pairs, so the confidence change there extends beyond the shadow class.

Case study: SIB

To check whether the diagnosis gives usable design guidance, we built the Spectral Information Bottleneck (SIB), a plug-in module with one component per finding. A Haar-wavelet bottleneck on the encoder output compresses low-frequency, city-specific content strongly and boundary subbands gently. Skip gates and Fourier Domain Adaptation preprocessing apply to CNN backbones only, since ViTs carry their recoverable component in the decoder. At deployment, a coverage threshold fitted on source-city validation data uses no held-out labels.

Over the 9 cells, SIB has the smallest mean gap to the within-city upper bound of the ten methods we evaluate, 6.42 mIoU (95% CI [5.01, 7.96]), level with FDA at 6.47. Its worst-case gap, 10.41 mIoU (OGLANet, Miami), is second to FDA's 9.10 and cannot be separated from it. SIB still leaves a significant gap in all 9 cells. We designed SIB on the same cells we evaluate it on, so the result shows that the diagnosis yields a usable design and does not test the diagnosis independently.

SIB module diagram and per-method gap to the within-city upper bound.
Figure 4. SIB has the smallest mean gap to the upper bound of the ten methods, level with FDA. (a) The SIB module, with components grouped by the finding they target. (b) Gap to the within-city upper bound across 9 LOCO cells; open markers: mean with 95% CI; filled: worst cell; dashed line: vanilla mean gap.

Use the benchmark

The dataset is on Hugging Face in two layouts holding the same images and masks: data_cities/ (per city, resolution and split) and data_loco/ (the three leave-one-city-out folds). Download one or both:

from huggingface_hub import snapshot_download

path = snapshot_download(
    repo_id="shagunmittal/ShadowTransfer",
    repo_type="dataset",
    allow_patterns=["data_loco/*"],   # or "data_cities/*"
)
# data_loco/fold_0_holdout_phoenix/highres/{train,val,test}/{images,masks}/

To compare a new method with the paper:

Intended and unsupported uses

Beyond benchmarking geographic transfer, the masks and per-patch metadata support building footprint and façade extraction (as occlusion priors), shadow-removal research, domain generalization for dense prediction, and urban-morphology or solar-geometry analysis. The data does not support claims about generalization beyond grid-pattern North American cities, beyond NAIP RGB sensors, or across seasons, and the 0.3 m and 0.6 m subsets should not be synthesized from each other.

Limitations

The benchmark covers three U.S. cities, RGB-only NAIP imagery, a single fall season and four architectures. We characterize where the failure occurs, how it distorts predictions and which component recovers it, but we do not isolate a causal mechanism connecting these. DINOv3's decoder locus cannot be attributed to its ViT architecture alone, since no CNN in our tests matches its pretraining scale.

Cite

Paper:

@inproceedings{mittal2026shadowtransfer,
  title     = {ShadowTransfer: A Geographic Transfer Benchmark for Overhead Shadow Detection},
  author    = {Mittal, Shagun and Yeh, Raymond A. and Ukkusuri, Satish},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS), Track on Evaluations and Datasets},
  year      = {2026}
}

Dataset release (DOI):

@misc{mittal2026shadowtransferdata,
  author    = {Mittal, Shagun and Yeh, Raymond A. and Ukkusuri, Satish},
  title     = {{ShadowTransfer} (Revision 5bef90e)},
  year      = {2026},
  publisher = {Hugging Face},
  doi       = {10.57967/hf/11483},
  url       = {https://huggingface.co/datasets/shagunmittal/ShadowTransfer}
}

Please also credit the USDA Farm Service Agency Aerial Photography Field Office for the NAIP imagery.