Caribbean AI Association
Back to the archive
Dominica / Working paper

Cloud cover and the evaluation of hurricane damage classifiers in Dominica

A parish-level sensitivity analysis of the 2017 Hurricane Maria assessment

Lead author · Adrian Dunkley   |   2026-11-20   |   v1.0

In plain English

After Hurricane Maria, analysts could see most mapped buildings on Dominica. Clouds hid 1,166 of them. More than half of those buildings were in Saint Patrick. [1]

Imagine software that calls every visible building damaged. It would match three quarters of the map labels while missing every building without a damage flag. To judge software fairly, count both kinds of mistakes and record where clouds hide the answer.

Share of mapped structures hidden by cloud

% of mapped structures
St Andrew0.74
St David1.44
St George0.35
St John0.26
St Joseph1.85
St Luke0.49
St Mark8.25
St Patrick13.34
St Paul4.75
St Peter0.63
Each parish uses its own mapped building count as the denominator. The same numerical value gives the width of the possible damage-share interval in percentage points. Source [1].

Analysis steps

  1. Count mapped buildings
  2. Separate visible footprints
  3. Bound the hidden labels
  4. Compare parishes
Read the full paper

A satellite damage map can hide a local gap in the evidence. Using ten parish totals from the October 2017 UNOSAT and Copernicus assessment, we measured the effect of cloud obstruction on a reference set for computer vision. The images covered 37,368 of 38,534 mapped structures. Of the 1,166 outside the visible area, 636 were in Saint Patrick. Holding visible labels fixed, the island-wide share flagged for possible damage could range from 73.46% to 76.49%. In Saint Patrick the corresponding interval spanned 13.34 percentage points. These are bounds on the map labels, conditional on the building inventory. Field inspections would be needed to measure physical damage. [1]

Research question

How much could unobserved structures change the mapped share with possible damage, and where would those gaps matter most in a test of a computer-vision classifier?

Data and methods

We transcribed the building inventory, the visible footprints and the potentially affected structures for each parish from the 18 October 2017 map. UNOSAT combined its interpretation with the Copernicus analysis using imagery acquired across several dates. A footprint represents a mapped structure. The map gives each visible structure an implicit binary label for this analysis. It flags possible damage or it does not. A cloud-obscured footprint has an unknown label. [1]

For inventory N, visible count V and flagged count A, we calculated the visible positive share A/V. Its counterparts over all mapped footprints range from A/N to (A + N − V)/N. The first endpoint assigns zero flags to hidden structures. The second assigns one flag to each. We summed the counts before calculating the island-wide endpoints. No assumption about the location of damage under clouds enters these bounds.

Results

The source lists 28,309 possible positives among 37,368 visible structures. A classifier that assigned the positive label to every visible footprint would agree with the map 75.76% of the time. It would identify none of the 9,059 visible footprints without a positive label. This calculation establishes a reference score without training an image model. [1]

Saint Patrick accounted for 54.55% of the island’s hidden footprints. Its visible positive share was 72.99%, while the full-inventory endpoints were 63.26% and 76.59%. Saint Mark had the second widest interval at 8.25 points. Saint John had the narrowest at 0.26 points. Rounding each endpoint separately can change the displayed width by 0.01 point. [1]

Discussion

A team testing image classifiers against these labels should give each parish a coverage count and show errors separately for visible positive and negative labels. A model could score highly by predicting damage everywhere. Cloud-obscured structures require another image or a field visit before a reviewer can assign a reference label. The xBD study also distinguishes what satellite imagery can reveal from what a field inspection can establish. [2]

Limitations

The bounds hold the visible classifications and the 2017 footprint inventory fixed. They exclude missing buildings, map errors, differences in image dates and damage hidden beneath a roof. The map described its assessment as preliminary and lacked field validation at publication. Parish totals contain no paired predictions from an AI model, so they cannot establish a model’s sensitivity or an individual building’s condition. [1]

Reproduce the analysis

Extract the three counts for each parish from the table in [1].

Subtract visible structures from mapped structures to obtain the hidden count. Divide by mapped structures and multiply by 100 for the chart.

For each row, divide the flagged count by mapped structures for the lower bound. Add the hidden count to the numerator for the upper bound. Repeat with the column totals.

Data table
ParishMapped structuresVisible structuresPossible damage flags
Saint Andrew621061645505
Saint David396739103357
Saint George878887575691
Saint John339833892246
Saint Joseph401039363269
Saint Luke820816590
Saint Mark1043957597
Saint Patrick476841323016
Saint Paul457043533218
Saint Peter960954820

References

  1. UNITAR–UNOSAT and Copernicus (2017). Buildings Damage Assessment & Related Density in Dominica: Analysis by Parish, 18 October, version 1.0. Parish table.
  2. Gupta et al. (2019). Creating xBD: A Dataset for Assessing Building Damage from Satellite Imagery. CVPR Workshops.

Data credits · UNITAR–UNOSAT / Copernicus, 18 October 2017. Parish totals; CAIA calculations. Source map: CC BY-NC 3.0.

Download the data and analysis code (ZIP)

Working paper · Internal review · How we check the research