EuroCrops

EuroCrops compiles all publicly available self-declared crop reporting datasets from European Union member states into a single collection. The dataset provides crop type classifications at the field/parcel scale across the EU, distributed in multiple cloud-native geospatial formats (GeoParquet, FlatGeobuf, PMTiles) with harmonized crop attributes.

agriculture crop-classification field-boundaries land-parcels

Maja Schneider, Marco Körner, Amelie Broszeit

Dataset files (33 x .parquet)

Source Cooperative product page

## README.md
# EuroCrops (Cloud-Native Geo distribution)

This dataset is a copy of the [EuroCrops](https://github.com/maja601/EuroCrops) dataset, offering the data in [cloud-native geospatial](https://cloudnativegeo.org) formats. The overview of the original dataset is:

> EuroCrops is a dataset collection combining all publicly available self-declared crop reporting datasets from countries of the European Union. The project is funded by the German Space Agency at DLR on behalf of the Federal Ministry for Economic Affairs and Climate Action (BMWK). This work is licensed under a [Creative Commons Attribution 4.0 International License](https://creativecommons.org/licenses/by/4.0/).

> Right now EuroCrops only includes vector data, but stay tuned for a version that includes satellite imagery!

> For any questions, please refer to our [FAQs](https://github.com/maja601/EuroCrops/wiki/FAQs) or use the Discussions/Issues to reach out to us.

You can read more details on the dataset itself on the main EuroCrops sites at https://github.com/maja601/EuroCrops and https://www.eurocrops.tum.de/

## About the data formatting and structure

The original EuroCrops dataset is distributed as Shapefiles, one per country, and takes the original data from the country providers as is, enhancing it with 3-4 attributes that are consistent across all the country files. These data come in a variety of projections, since the source data was not consistent. The folder named [unprojected](https://beta.source.coop/cholmes/eurocrops/browse/unprojected) contains the original data structure in different formats.

Note that a few France boundaries have points that are way outside of France, and the Romania boundaries seem to overlap each other. We'll work with the source provider to try to clean those up there, and will redistribute here when ready.

### Modifications

There are several different data types in this repository. The first set have no changes to the original files, but puts them in some alternate formats ([GeoParquet](https://geoparquet.org) and [Flatgeobuf](https://flatgeobuf.org))

The second set projects all geometries into long / lat to make the data easy to work as a single dataset (about half the source files were already in that projection, the rest were country specific).

Then third set of datasets also are all in long lat and additionally remove all the attributes except those that were made common across datasets by the Eurocrops project:

| Attribute Name | Explanation                                                 |
| -------------- | ----------------------------------------------------------- |
| EC_trans_n     | The original crop name translated into English              |
| EC_hcat_n      | The machine-readable HCAT name of the crop                  |
| EC_hcat_c      | The 10-digit HCAT code indicating the hierarchy of the crop |

The NUTS3 attribute was not included as it was not consistently in the datasets. A future iteration of this dataset may try to add that or other country information in the data itself to be able to do parquet partitioning against it.

## Access the data

The easiest way to visualize the data is with this [Felt map](https://felt.com/map/Eurocrops-DRrR1lxoQEqvmZBz6e9APPC), where you can see (almost) all the data visualized and styled. You can also directly access [this pmtiles](https://s3.us-west-2.amazonaws.com/us-west-2.opendata.source.coop/cholmes/eurocrops/eurocrops-all.pmtiles), and see it [shown with the PMTiles Viewer](https://protomaps.github.io/PMTiles/?url=https%3A%2F%2Fs3.us-west-2.amazonaws.com%2Fus-west-2.opendata.source.coop%2Fcholmes%2Feurocrops%2Feurocrops-all.pmtiles#map=2.36/41.89/31.42). These just contain the harmonized attributes, and are projected into the web mercator projection.

The original shapefiles are in the [unprojected/shapefiles](https://beta.source.coop/cholmes/eurocrops/browse/unprojected/shapefiles) directory, and you can find alternate formats in [unprojected/flatgeobuf](https://beta.source.coop/cholmes/eurocrops/browse/unprojected/flatgeobuf) and [unprojected/geoparquet](https://beta.source.coop/cholmes/eurocrops/browse/unprojected/geoparquet).

The [geoparquet-projected/](https://beta.source.coop/cholmes/eurocrops/browse/geoparquet-projected) folder has all the data projected to long / lat, but retains all the original fields.

And then you can get the data as a [single Flatgeobuf file](https://data.source.coop/cholmes/eurocrops/eurocrops-harmonized-only.fgb) (10.8 gb) that is projected and only has the final harmonized fields. We're also making available a [DuckDB](https://duckdb.org) database to experiment with distributing it directly. The data likely will just show up as a blob that you'll have to parse in with the [spatial extension](https://duckdb.org/docs/extensions/spatial).

## Dataset License

The data is licensed under [Creative Commons Attribution 4.0 International License](https://creativecommons.org/licenses/by/4.0/).

## Authors

* Maja Schneider
* Amelie Broszeit
* Marco Körner

### Additional Processing

* Chris Holmes

## Citation & DOI

From https://github.com/maja601/EuroCrops#reference

**Disclaimer**: The official reference will follow soon. Please also reference the countries' dependent source in case you're using their data.

```
@Misc{schneider2022eurocrops21,
 author     = {Schneider, Maja and K{\"o}rner, Marco},
 title      = {EuroCrops},
 DOI        = {10.5281/zenodo.6866846},
 type       = {Dataset},
 publisher  = {Zenodo},
 year       = {2022}
}
```

Additional references:

```
@InProceedings{Schneider2022Challenges,
  title     = {Challenges and Opportunities of Large Transnational Datasets: A Case Study on European Administrative Crop Data},
  author    = {Schneider, Maja and Marchington, Christian and K{\"o}rner, Marco},
  booktitle = {Workshop on Broadening Research Collaborations in ML (NeurIPS 2022)},
  year      = {2022}
}
```

```
@InProceedings{Schneider2022Harnessing,
  title         = {Harnessing Administrative Data Inventories to Create a Reliable Transnational Reference Database for Crop Type Monitoring},
  author        = {Schneider, Maja and K{\"o}rner, Marco},
  booktitle     = {IGARSS 2022-2022 IEEE International Geoscience and Remote Sensing Symposium},
  pages         = {5385--5388},
  year          = {2022},
  organization  = {IEEE}
}
```

```
@InProceedings{Schneider2021EPE,
  author        = {Schneider, Maja and Broszeit, Amelie and K{\"o}rner, Marco},
  booktitle     = {Proceedings of the Conference on Big Data from Space (BiDS)},
  title         = {{EuroCrops}: A Pan-European Dataset for Time Series Crop Type Classification},
  editor        = {Soille, Pierre and Loekken, Sveinung and Albani, Sergio},
  publisher     = {Publications Office of the European Union},
  date          = {2021-05-18},
  doi           = {10.2760/125905},
  eprint        = {2106.08151},
  eprintclass   = {eess.IV,cs.CV,cs.LG},
  eprinttype    = {arxiv}
}
```

```
@Misc{Schneider2021TEC,
  author       = {Schneider, Maja and K{\"o}rner, Marco},
  date         = {2021-06-15},
  title        = {{TinyEuroCrops}},
  doi          = {10.14459/2021MP1615987},
  organization = {Technical University of Munich (TUM)},
  type         = {Dataset},
  url          = {https://mediatum.ub.tum.de/1615987}
}
```


Experiments with fiboa

Place to try out some larger fiboa files, for remote querying, etc.


geoparquet-io test data

Test data for geoparquet-io performance testing


National Hydrography Dataset (United States)

The National Hydrography Dataset (NHD) represents the water drainage network of the United States with features such as rivers, streams, canals, lakes, ponds, coastline, dams, and stream gages.

More information at https://www.usgs.gov/national-hydrography/national-hydrography-dataset

The initial data in this is just NHDFlowlines, and the 'M' coordinate has been removed (as GeoParquet does not yet support M).


Taxi Zones from TLC Trip Record Data

Geospatial boundaries of New York City taxi zones from the TLC Trip Record Data, available in cloud-native formats (GeoParquet and PMTiles) for efficient spatial analysis and visualization.

New York City Taxi and Limousine Commission

Dataset files (2 x .parquet)

Source Cooperative product page

## README.md
# New York Taxi Zones (Cloud-Native Distribution)

This repository contains [Cloud-Native Geospatial](https://cloudnativegeo.org) versions of the Taxi Zones of the [TLC Trip Record Data](https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page). This doesn't contain all the core Parquet data - it's just to make it a bit easier for non-geospatial users to get the actual boundaries. The current web page with all the links says 'Taxi Zone Shapefile (PARQUET)', but the Shapefile is not Parquet. Hopefully the NYC site will update to actually include GeoParquet, but for now it's accessible here. There are 3 files in this repository:

* [taxi_zones.parquet](https://data.source.coop/cholmes/nyc-taxi-zones/taxi_zones.parquet) - A [GeoParquet](https://geoparquet.org) file translated directly from the original Shapefile (using `ogr2ogr`). The shapefile .prj file was translated to [EPSG:2263](https://epsg.io/2263).
* [taxi_zones_4326.parquet](https://data.source.coop/cholmes/nyc-taxi-zones/taxi_zones_4326.parquet) - The original Shapefile transformed into GeoParquet and projected into Long, Lat (EPSG:4326).
* [taxi_zones.pmtiles](https://data.source.coop/cholmes/nyc-taxi-zones/taxi_zones.pmtiles) - The core data as [PMTiles](https://github.com/protomaps/PMTiles) for easy visualization.

***Note: There is also a taxi zone lookup table CSV that seems to have a 'service zone' field. If anyone is interested I could join that to this Parquet so two files aren't needed. Leaving off that work for now to just ship this.***


Demo of publishing OpenET Data

Data from OpenET, converted to fiboa and pmtiles, as a demo of whats possible.


Ordnance Survey Open Data as Cloud Native Geo

This repo is a set of experiments reformatting Ordnance Survey data in Cloud Native Geospatial formats


Overture Maps Data

Global building footprints and points-of-interest (places/POIs) from Overture Maps, redistributed in cloud-native GeoParquet and PMTiles formats partitioned by country and quadkey. The building dataset covers ~2.8 million items across the globe; the places dataset covers worldwide POIs. Originally an experimental mid-2023 redistribution; Overture now releases data natively in these formats.

building-footprints points-of-interest geoparquet pmtiles quadkey

AD.parquet

bbox

STRUCT(maxx DOUBLE, maxy DOUBLE, minx DOUBLE, miny DOUBLE)

[geometry] Bounding box struct (minx, miny, maxx, maxy) for the feature

geometry

[geometry] Building footprint polygon geometry in EPSG:4326

quadkey

[spatial_index] Quadkey tile index used for spatial partitioning

[admin_code] ISO 3166-1 alpha-2 country code

AE.parquet

AF.parquet

AG.parquet

AI.parquet

YE.parquet

categories

STRUCT(alternate STRUCT(list VARCHAR[]), main VARCHAR)

brand

STRUCT("names" STRUCT(key_value STRUCT("key" VARCHAR, "value" STRUCT(list STRUCT(key_value STRUCT("key" VARCHAR, "value" VARCHAR)[])[]))[]), wikidata INTEGER)

geometry

[geometry] Place/POI point geometry in EPSG:4326

YT.parquet

ZA.parquet

ZM.parquet

ZW.parquet

Source Cooperative product page

## README.md
# Overture Buildings & Places (Cloud-Native Geo experiments)

This data is currently just a set of experiments using the new [overture maps](https://overturemaps.org/). See their
[github repo](https://github.com/OvertureMaps/data) for more info. My hope is that this can try out some better ways
to distribute the data, and to build an ecosystem of tools and data distributions on top of the core.

There are two main folders that I recommend starting with:

* [geoparquet-country-quad/](https://beta.source.coop/cholmes/overture/geoparquet-country-quad) is the Overture Building dataset, repartitioned by country, split by quadkey when larger than ~2 gigs, distributed as GeoParquet with row size group of 20,000.
* [places-geoparquet-country/](https://beta.source.coop/cholmes/overture/places-geoparquet-country) is the Overture Places (POI's) dataset, repartitioned by country. There was no need to split by quadkey as the whole dataset is only ~3 gigabytes.

There's also a PMTiles of the buildings dataset in the [root directory](https://beta.source.coop/cholmes/overture). You can check it out at https://observablehq.com/d/2dd90b64498c3864. I do not have a PMTiles up of the places dataset, but you can get one from https://r2-public.protomaps.com/protomaps-sample-datasets/overture-pois.pmtiles, and view that PMTiles at https://bdon.github.io/overture-tiles/places.html. I

If you just want to use Overture data those are the two to focus on. The other folders are a number of experiments and interim processing steps.

* [geoparquet-refined/](https://beta.source.coop/cholmes/overture/browse/geoparquet-refined) is the same files as the original but turned into Geoparquet, with columns added for iso_country and quadkey, and data ordered by quadkey.
* [duckdb/](https://beta.source.coop/cholmes/overture/duckdb) contains a single DuckDB database. It's pretty easy to make this yourself, it is just all the data in geoparquet refined, loaded up into a DuckDB. I'm not sure if distributing DuckDB databases directly is useful. But the file is definitely useful to partition data in different ways.
* [geoparquet-country-quad-2](https://beta.source.coop/cholmes/overture/geoparquet-country-quad-2) I recommend ignoring. It's some alternate partitioning, and the dataset is incomplete. But until I get a set of different repartitioned datasets I'll keep it.

license: AI returned 'unknown' (README does not state a license). Verify with publisher before flipping to _state: reviewed.

providers: role [processor] for 'Chris Holmes' is an AI guess — Identified as the redistribution author (account 'cholmes') who repartitioned and reformatted the data into GeoParquet/PMTiles.. Verify before flipping to _state: reviewed.

extent.spatial.bbox: omitted; spatial reference is categorical via [AD.geometry (geometry), AD.quadkey (spatial_index), AD.country_iso (admin_code), AD.bbox (geometry), YE.geometry (geometry), YE.quadkey (spatial_index), YE.country_iso (admin_code), YE.bbox (geometry)]. Add bbox once a placename → bbox gazetteer is available.


Dutch Portolan Demo

A collection of datasets from various Dutch institutions to demonstrate a Spatial Data Infrastructure built on Portolan.


Sentinel 2 Grid Vector data

Repository for Sentinel 2 grids (MGRS) in various formats for use online


STAC-GeoParquet Public Missions

(Work In Progress)

This is a collection of public STAC archives, in the STAC-GeoParquet format, for major satellite missions.  To start it's just sentinel 2 and Landsat.

They are copied from Microsoft Planetary computer, updating them to the latest version of Geoparquet and aiming to ensure that they follow all the best practices. Currently there are no 'updates', just a bulk transfer, under the idea that it's pretty easy to get the latest from a STAC API. But in the future we can try to add automated or periodic manual updates.

The current versions aren't yet perfect. They are also quite heavy, so may explore versions that don't include all the assets information.


STAC-GeoParquet of Sentinel 2 level 2a

No Description Provided


Multi-Temporal Cloud Gap Imputation With HLS Imagery Across CONUS

A machine learning benchmark dataset containing 7,852 multi-temporal Harmonized Landsat and Sentinel-2 (HLS) image chips and 21,642 binary cloud masks covering diverse land covers across the Contiguous United States (CONUS) for the year 2022. Each chip is a 224×224 pixel GeoTIFF at 30m resolution with 18 spectral bands (6 bands × 3 time steps). The dataset was created to support training and evaluation of cloud gap imputation models.

remote-sensing land-use cloud-masking multitemporal time-series harmonized-landsat-sentinel landsat sentinel-2 machine-learning imputation

Dataset files (29519 x .tif)

Source Cooperative product page

## README.md
# Multi-Temporal Cloud Gap Imputation With HLS Data Across CONUS

## Dataset Description

### Dataset Summary

This dataset contains temporal Harmonized Landsat and Sentinel-2 (HLS) imagery of diverse land covers across the Contiguous United States (CONUS) for the year 2022 along with binary cloud masks for the same area and year.

### Dataset Structure
```
hls-multi-temporal-cloud-gap-imputation/
├── chip_catalog.csv
├── cloud_catalog.csv
├── train/
│ ├── hls_chips/
│ ├── cloud_masks/
├── test/
│ ├── hls_chips/
│ ├── hls_chips_masked/
```
## Data Catalogs
Data catalogs containing metadata for both HLS scenes and cloud masks are provided. The catalog for HLS scenes is provided in `chip_catalog.csv` and the catalog for cloud masks is provided in `cloud_catalog.csv`.

### Cloud Catalog

`cloud_catalog.csv` contains the following columns:
| Column |  Contains  |
|---------|--------|
| cloud_mask_id       |  'chip_XXX_YYY' corresponding to the file name  |
| cloud_pct       |  ratio of cloudy pixels within the chip, ranges from 0.0 to 1.0 |
| usage       |  'train' or 'validate' |
| bin       |  which of 10 bins the cloud mask fall within, e.g. 0.1-0.2  |

### HLS Chip Catalog

`chip_catalog.csv` contains the following columns:
| Column |  Contains  |
|---------|--------|
| chip_id       |  'chip_XXX_YYY' corresponding to the file name  |
| chip_x       |  x coordinate of bounding box centroid |
| chip_y       |  y coordinate of bounding box centroid   |
| tile       |  HLS tile ID   |
| valid_first       |  count of valid pixels in the first time step  |
| valid_second       |  count of valid pixels in the second time step  |
| valid_third       |  count of valid pixels in the thrid time step  |
| bad_pct_first       |  percent of invalid pixels in the first time step |
| bad_pct_second       |  percent of invalid pixels in the second time step   |
| bad_pct_third       |  percent of invalid pixels in the third time step   |
| first_image_date       |  date of first time step in `YYYY/mm/dd` |
| second_image_date       |  date of second time step in `YYYY/mm/dd`  |
| third_image_date       |  date of third time step in `YYYY/mm/dd`  |
| bad_pct_max       |  maximum of invalid pixels in all time steps  |
| na_count       |  count of pixels in all time steps with no data |
| usage | 'train' or 'validate' |

Invalid pixels are those which intersect with any QA mask.

## Ground Truth HLS Scenes
The ground truth HLS scenes are stored in GeoTIFF format under `train/hls_chips/` and `test/hls_chips/`. Each GeoTIFF file covers a 224 x 224 pixel area at 30m spatial resolution. Each file contains 18 bands consisting of 6 spectral bands in 3 steps stacked together. The file name structure is `chip_XXX_YYY.tif` where `XXX` and `YYY` refer to row and column of a tile grid imposed on the Continental US. Since the dataset is sampled from this country-wide grid not all `XXX`s and `YYY`s are present in the dataset.

## Masked HLS Scenes for Testing
Testing scenes are pre-masked to ensure that all models are evaluated using the same test set. These scenes are stored in GeoTIFF format under `test/hls_chips`. The file name structure for masked scenes is `chip_XXX_YYY_masked.tif`. Each masked scene corresponds to the ground truth scene with the same value of `XXX_YYY` in the file name. For example, the file `test/hls_chips/chip_373_294.tif` is the ground truth for `test/hls_masked/chip_373_294_masked.tif`, with the latter having values of 0 at cloud-masked locations. Cloud masks are present in all possible combinations of time steps in equal proportion. Possible combinations given time steps t1, t2, and t3 are:

- t1
- t2
- t3
- t1, t2
- t1, t3,
- t2, t3,
- t1, t2, t3

So, for example, 1/7th of test scenes are masked at ONLY t2, and 1/7th at t2 AND t3, etc. for each of the possible combinations.

Cloud masks for the test scenes range from 0.01% coverage to 100% coverage, and are equally sampled from 10 equally sized bins between 0-100%.

## Cloud Masks
The training cloud masks are stored in GeoTIFF format under `train/cloud_masks/`. The file name structure for cloud masks is `chip_XXX_YYY_T_cmask.tif` where `XXX` and `YYY` refer to row and column of a tile grid imposed on the Continental US. `T` refers to the time step of each cloud mask and is meant only to distinguish cloud masks derived from the same location from each other. The intent for training is that these cloud masks are randomly paired with training HLS scenes in all time steps. The distribution of cloud mask coverage for the training set does not correspond to the distribution of cloud mask coverage for the validation set, as the distribution of the latter has been equalized. This may lead to higher validation accuracy if the user chooses not to equalize the training dataset - it is left to the user's discretion.

## Band Order
In each HLS GeoTIFF the following bands are repeated for each of three observations throughout the year:
| Channel |  Name  |  HLS   S30 Band number |
|---------|--------|------------------------|
| 1       |  Blue  |  B02                   |
| 2       |  Green |  B03                   |
| 3       |  Red   |  B04                   |
| 4       |  NIR   |  B8A                   |
| 5       |  SWIR 1  |  B11                   |
| 6       |  SWIR 2  |  B12                   |

Masks are a stored as a single-band binary image where 1 denotes the presence of the cloud mask and 0 denotes the absence of the cloud mask.

## Dataset Creation

Code used to generate HLS scenes and cloud masks is available [here](https://github.com/ClarkCGA/cloud-gap-filling-td). Code used to generate masked test scenes is available [here](https://github.com/ClarkCGA/gfm-gap-filling-baseline/blob/main/gfm-gap-filling-baseline/gap-filling-baseline/datasets/gapfill.py). `usage='validate'` was used along with default parameters when initializing the dataset using the `gapfill.py` code. Refer to [Seeing Through the Clouds: Cloud Gap Imputation with Prithvi Foundation Model](https://arxiv.org/abs/2404.19609) for further information about the creation and initial use of this dataset.

### Chip Generation and Partitioning
Three HLS scenes were selected between Mar and Sep 2022 with time difference between scenes varying between 1 and 200 days. After filtering for missing values and cloudy pixels, a total of 7,852 cloud-free chips evenly distributed across the CONUS were generated. This set was randomly partitioned into training (80%) and validation (20%) sets, resulting in 6,231 training chips and 1,621 validation chips.

### Cloud Generation and Partitioning
Cloud masks were generated from the same region of CONUS using HLS cloud mask quality flag and exported as a binary layer of cloudy and non-cloudy pixels. This yielded 21,642 cloud masks, of which 1,600 were randomly selected and reserved for validation, resulting in 20,042 training cloud masks

### License and Citation
This dataset is published under a CC-BY-4.0 license. If you find this dataset useful for your application, you can cite it as following:

```
@misc{hls-multi-temporal-cloud-gap-imputation,
    author = {Godwin, Denys and Li, Hanxi (Steve) and Alemohammad, Hamed},
    doi    = {https://doi.org/10.5281/zenodo.11281740},
    title  = {{Multi-Temporal Cloud Gap Imputation With HLS Data Across CONUS}},
    version = {1.0},
    year   = {2024}
}
```
### Contact
For any questions about the dataset, you can contact [Dr. Hamed Alemohammad](mailto:halemohammad@clarku.edu).

### Funding
This dataset is generated with funding from a grant awarded to [Clark University Center for Geospatial Analytics (CGA)](https://www.clarku.edu/cga/) by NASA.

providers: role [licensor] for 'NASA' is an AI guess — NASA provided funding via grant; not stated as licensor but listed as funder. Verify before flipping to _state: reviewed.


Multi-temporal Harmonized Landsat-Sentinel (HLS) imagery dataset covering 3,854 chips across the Contiguous United States for 2022, each chip containing 18 spectral bands (6 bands × 3 seasonal time steps) at 30m resolution over a 224×224 pixel area. Pixel-level labels are derived from USDA's Crop Data Layer (CDL) across 13 land cover and crop type classes. Intended for training geospatial image segmentation machine learning models.

agriculture crop-classification land-use remote-sensing multi-temporal machine-learning harmonized-landsat-sentinel landsat sentinel-2

Dataset files (7708 x .tif)

Source Cooperative product page

## README.md
# Multi-Temporal Crop Classification with HLS Imagery across CONUS

## Dataset Description

### Dataset Summary

This dataset contains temporal Harmonized Landsat-Sentinel imagery of diverse land cover and crop type classes across the Contiguous United States for the year 2022. The target labels are derived from USDA's Crop Data Layer (CDL). It's primary purpose is for training segmentation geospatial machine learning models.

### Dataset Structure

## HLS Scenes
Each GeoTIFF file located under `hls/` covers a 224 x 224 pixel area at 30m spatial resolution. Each input satellite file contains 18 bands including 6 spectral bands for three time steps stacked together. The file name structure is `chip_XXX_YYY_merged.tif` where `XXX` and `YYY` refer to row and column of a tile grid we imposed on the CDL data layer. Since the dataset is sampled from this country-wide grid not all `XXX`s and `YYY`s are present in the dataset. So you need to load the `train_data.txt` and `validation_data.txt` to get the list of all chips in the dataset.

## CDL Labels
Each GeoTIFF file located under `masks/` covers the same area as the corresponding HLS chips and contains one band with the target classes for each pixel. The file name structure is `chip_XXX_YYY.mask.tif`.

## Band Order
In each HLS input GeoTIFF the following bands are repeated three times for three observations throughout the growing season:
| Channel |  Name  |  HLS   S30 Band number |
|---------|--------|------------------------|
| 1       |  Blue  |  B02                   |
| 2       |  Green |  B03                   |
| 3       |  Red   |  B04                   |
| 4       |  NIR   |  B8A                   |
| 5       |  SW 1  |  B11                   |
| 6       |  SW 2  |  B12                   |

Masks are a single band with the following values:
| Value    | Class Name            |
|----------|-----------------------|
| 0        |  No Data              |
| 1        |  Natural Vegetation   |
| 2        |  Forest               |
| 3        |  Corn                 |
| 4        |  Soybeans             |
| 5        |  Wetlands             |
| 6        |  Developed/Barren     |
| 7        |  Open Water           |
| 8        |  Winter Wheat         |
| 9        |  Alfalfa              |
| 10       |  Fallow/Idle Cropland |
| 11       |  Cotton               |
| 12       |  Sorghum              |
| 13       |  Other                |

## Data Splits
The 3,854 chips have been randomly split into training (80%) and validation (20%) with corresponding ids recorded in cvs files `train_data.txt` and `validation_data.txt`.

## Dataset Creation
### Query and Scene Selection
First, a set of 5,000 chips was defined based on samples from the USDA CDL to ensure a representative sampling across the CONUS. Next, for each chip, the corresponding HLS S30 scenes between March and September 2022 were queried, and scenes with low cloud cover were retrieved. Then, three scenes are selected among the low cloudy scenes to ensure a scene from early in the season, one in the middle, and one toward the end. The three final scenes were then reprojected to CDL's projection grid (`EPSG:5070`) using bilinear interpolation.

### Chip Generation
In the final step, the three scenes for each chip were clipped to the bounding box of the chip, and 18 spectral bands were stacked together. In addition, a quality control was applied to each chip using the `Fmask` layer of the HLS dataset. Any chip containing clouds, cloud shadow, adjacent to cloud or missing values were discarded. This resulted in 3,854 chips.

### License and Citation
This dataset is published under a CC-BY-4.0 license. If you find this dataset useful for your application, you can cite it as following:

```
@misc{hls-multi-temporal-crop-classification,
    author = {Cecil, Michael and Kordi, Fatemeh and Li, Hanxi (Steve) and Khallaghi, Sam and Alemohammad, Hamed},
    doi    = {10.57967/hf/0955},
    month  = aug,
    title  = {{HLS Multi Temporal Crop Classification}},
    year   = {2023}
}
```
### Contact
For any questions about the dataset, you can contact [Dr. Hamed Alemohammad](mailto:halemohammad@clarku.edu).

### Funding
This dataset is generated with funding from a grant awarded to [Clark University Center for Geospatial Analytics (CGA)](https://github.com/ClarkCGA) by NASA.

providers: role [producer] for 'USDA National Agricultural Statistics Service (NASS)' is an AI guess — Produces the Crop Data Layer (CDL) used as pixel-level label source; not stated as co-producer of this derived dataset. Verify before flipping to _state: reviewed.


Clay v1 Embeddings for California NAIP

No Description Provided


Clay Model v0 Embeddings

Semantic embeddings generated by the Clay Foundation Model v0 for all Earth Observation imagery used in its training corpus. Each record pairs a dense float vector embedding with a geographic geometry and acquisition date, covering global satellite tiles. Clay is a Vision Transformer trained via Self-Supervised Learning (Masked Autoencoder) on geospatial and temporal EO data.

remote-sensing earth-observation embeddings machine-learning foundation-model satellite-imagery

01WCN_20190518_20231021_v001.gpq

[geometry] OGC:CRS84 polygon footprint of the EO tile

01WCP_20170701_20210603_v001.gpq

01WFP_20200430_20200430_v001.gpq

03VXL_20210424_20210424_v001.gpq

03WVN_20220610_20220610_v001.gpq

60WVV_20170325_20210327_v001.gpq

60WWB_20170420_20210330_v001.gpq

60WWT_20200421_20230208_v001.gpq

60WWU_20190711_20230518_v001.gpq

60WWV_20190424_20210419_v001.gpq

Source Cooperative product page

## README.md
# Clay Foundation Model

## An open source AI model for Earth

Clay is a foundational model of Earth. It uses a Vision Transformer architecture adapted
to understand geospatial and temporal relations on Earth Observation data. The model is
trained via Self-supervised learning (SSL) using a Masked Autoencoder (MAE) method.

The Clay model can be used in three main ways:
- Generate semantic embeddings for any location and time.
- Fine-tune the model for downstream tasks such as classification, regression, and generative tasks.
- Use the model as a backbone for other models.

## Where is what

- Our **website** is [madewithclay.org](https://madewithclay.org).
- The Clay model **code** lives on [Github](https://github.com/Clay-foundation/model).
  License: [Apache-w.0](https://github.com/Clay-foundation/model/blob/main/LICENSE).
  The latest release is [v0.0.1](https://github.com/Clay-foundation/model/releases/tag/v0.0.1)
- The Clay model **weights**  on [Hugging Face](https://huggingface.co/made-with-clay/Clay/).
  License: [OpenRAIL-M](https://github.com/Clay-foundation/model/blob/main/LICENSE-MODEL.md).
- The Clay **documentation** [lives on this site](https://clay-foundation.github.io/model/index.html).
  License: [CC-BY](http://creativecommons.org/licenses/by/4.0/).
- We maintain a set of **embeddings** on [Source Cooperative](https://beta.source.coop/repositories/brunosan/clay-model-v0-embeddings/description/).
  License: [ODC-BY](https://opendatacommons.org/licenses/by/).

CLAY is a fiscal sponsored project of the 501c3 non-profit
[Radiant Earth Foundation](https://www.radiant.earth).

providers: role [licensor] for 'Radiant Earth Foundation' is an AI guess — Fiscal sponsor and 501c3 host of Clay; inferred as licensor for the embeddings dataset. Verify before flipping to _state: reviewed.

extent.spatial.bbox: omitted; spatial reference is categorical via [01WCN_20190518_20231021_v001.geometry (geometry), 01WCP_20170701_20210603_v001.geometry (geometry), 01WFP_20200430_20200430_v001.geometry (geometry), 03VXL_20210424_20210424_v001.geometry (geometry), 03WVN_20220610_20220610_v001.geometry (geometry), 60WVV_20170325_20210327_v001.geometry (geometry), 60WWB_20170420_20210330_v001.geometry (geometry), 60WWT_20200421_20230208_v001.geometry (geometry), 60WWU_20190711_20230518_v001.geometry (geometry), 60WWV_20190424_20210419_v001.geometry (geometry)]. Add bbox once a placename → bbox gazetteer is available.


NAIP data embedded with Clay v1.5 (rev2)

Clay v1.5 vector embeddings for the complete NAIP (National Agriculture Imagery Program) archive, covering US aerial imagery from 2010 through 2021. Each 256×256-pixel NAIP tile is represented as a 1024-dimensional class embedding generated by the Clay v1.5 geospatial foundation model from 4-band (RGB+NIR) GeoTIFF inputs. Data is stored in Parquet files mirroring the NAIP folder structure, with one geometry and one embedding array column per tile.

remote-sensing aerial-imagery embeddings machine-learning foundation-model naip

m_3008501_ne_16_1_20110815.parquet

[geometry] Tile footprint polygon for the 256x256 NAIP tile

m_3008501_nw_16_1_20110815.parquet

m_3008502_ne_16_1_20110815.parquet

m_3008502_nw_16_1_20110815.parquet

m_3008503_ne_16_1_20110815.parquet

m_4511062_sw_12_060_20190805.parquet

m_4511063_se_12_060_20190821.parquet

m_4511064_se_12_060_20190821.parquet

m_4511064_sw_12_060_20190821.parquet

m_4511164_se_12_060_20190828.parquet

Source Cooperative product page

## README.md
This repository contains all of [NAIP on AWS](https://registry.opendata.aws/naip/) data embedded with [Clay v1.5](https://huggingface.co/made-with-clay/Clay).

Source: This data has been created by [Development Seed](https://developmentseed.org/) under contract with [Clay](https://madewithclay.org/) and comptute support by [AWS](https://aws.amazon.com/).

Contact: For feedback and questions, please file a ticket on the [model repo of Clay](https://github.com/Clay-foundation/model/issues) or email us at contact@madewithclay.org

Data Source: [NAIP on AWS `naip-analytic`](https://registry.opendata.aws/naip/):
* Includes NAIP imagery from 2010 through 2021.
* The data in this bucket is Original Imagery in Geotiff format 4-band (RGB + NIR)

Model source: Embeddings generated from inference with Clay v1.5:
* Embeddings have `1024` dimensions and correspond to the "class" embedding that is used alongside the patch embeddings at the end of the encoder.
* Each tile is split into tiles of size 256x256, and the attention patch size inside Clay v1.5 is 8x8 px.
* Inference run was done on November '24 on AWS using g4 and g6 EC2, for roughly 60k GPU hours, and costing around $65k.

Embeddings License: Clay [CC-By](https://creativecommons.org/licenses/by/4.0/)

Format:
* Folder structure follows the same as NAIP folder structure.
* File format is parquet, with two columns: `geometry` and `embeddings`

Usage example

```python
import duckdb
path = "https://data.source.coop/clay/clay-v1-5-naip-2/az/2015/100cm/rgbir_cog/32110/m_3211002_se_12_1_20150530.parquet"
d = duckdb.read_parquet(path)
df = d.to_df()
df.head()
```

providers: role [producer] for 'USDA Farm Service Agency' is an AI guess — Original NAIP imagery source; inferred from NAIP program ownership, not named explicitly in README. Verify before flipping to _state: reviewed.

extent.spatial.bbox: omitted; spatial reference is categorical via [m_3008501_ne_16_1_20110815.geometry (geometry), m_3008501_nw_16_1_20110815.geometry (geometry), m_3008502_ne_16_1_20110815.geometry (geometry), m_3008502_nw_16_1_20110815.geometry (geometry), m_3008503_ne_16_1_20110815.geometry (geometry), m_4511062_sw_12_060_20190805.geometry (geometry), m_4511063_se_12_060_20190821.geometry (geometry), m_4511064_se_12_060_20190821.geometry (geometry), m_4511064_sw_12_060_20190821.geometry (geometry), m_4511164_se_12_060_20190828.geometry (geometry)]. Add bbox once a placename → bbox gazetteer is available.


NAIP data embedded with Clay v1.5

This repository has the embeddings using Clay vq.5 for the entire NAIP archive. See details on README.md  License: Clay [CC-By](https://creativecommons.org/licenses/by/4.0/)


Sentinel-2 data embedded with Clay v1.5

This product is deprecated, please use https://source.coop/clay/lgnd-clay-v1-5-sentinel-2-l2a instead.


LGND Clay Embeddings - Sentinel2

A global dataset of Clay v1.5 1024-dimensional float32 embeddings derived from Sentinel-2 L2A imagery, created by lgnd.ai. It covers approximately 99.9% of the globe across two time steps (June 2024 and June 2025), organized into scene-level and aggregated monthly products keyed to 2560-meter MajorTOM grid cells. Data are stored as hive-partitioned GeoParquet files totaling ~467 GB across ~127 million embeddings.

remote-sensing sentinel-2 embeddings machine-learning time-series earth-observation geoparquet

[spatial_index] Level-2 geohash hive-partition key for the aggregated product

bbox

STRUCT(xmin DOUBLE, ymin DOUBLE, xmax DOUBLE, ymax DOUBLE)

[geometry] Bounding box struct (xmin, ymin, xmax, ymax) for spatial lookup optimization

geometry

[geometry] Polygon footprint of the chip in WGS84 (EPSG:4326)

gzd

[spatial_index] MGRS Grid Zone Designator hive-partition key for the scene product

Source Cooperative product page

## README.md
# LGND Clay Embeddings - Sentinel2

This repository contains a global dataset of [Clay v1.5](https://madewithclay.org/) embeddings for Sentinel2, created by [lgnd.ai](https://lgnd.ai/).  Each embedding is a 1024-dimensional float32 vector associated with a 2560-meter MajorTOM grid cell.  All imagery was sourced from the [`sentinel-cogs`](https://registry.opendata.aws/sentinel-2-l2a-cogs/) bucket on AWS.

The dataset currently has near complete (~99.9%) global coverage across two time steps - June 2024 and June 2025, and is broken into two distinct products:
- `scene` - embeddings for individual Sentinel2 scenes.
- `aggregated` - an aggregation of scene-level embeddings into a normalized time series of embeddings.

## Scene-Level Data

The scene-level product contains embeddings for each individual Sentinel-2 scene.  For each MGRS tile, we embed the minimum number of scenes required to provide maximum spatial coverage while minimizing cloud cover.  Some MGRS tiles only require one scene, while others require many more.  The embeddings are available in the `scene` directory partitioned by Grid Zone Designator (the first 3 digits of the MGRS tile), year, and month.  For example:

```shell
❯ aws s3 ls s3://us-west-2.opendata.source.coop/clay/lgnd-clay-v1-5-sentinel-2-l2a/scene/gzd=11U/year=2024/month=06/ --summarize --human-readable

2026-01-25 21:44:58  183.1 MiB data_01c1fab1-0004-5b20-0009-b72e01d10bc6_215_4_0.parquet
2026-01-25 21:46:57  141.8 MiB data_01c1fab1-0004-5b20-0009-b72e01d10bc6_215_4_1.parquet

Total Objects: 2
   Total Size: 324.9 MiB
```

The scene-level product is best when analyzing individual Sentinel2 scenes, or when you need the highest possible temporal resolution.  This product contains ~77.9 million embeddings and takes up ~285GB on disk.

## Aggregated Data

The aggregated product provides a normalized time series of embeddings per chip by selecting one representative embedding per month for each MajorTOM cell.  This helps remove redundancy from overlapping scene coverage at the border of MGRS tiles, normalizes observation frequency across regions with varying revisit rates, and simplifies time series analysis by providing consistent monthly snapshots.  The embeddings are available in the `aggregated` directory partitioned by geohash, year, and month.  For example:

```shell
~ ❯ aws s3 ls s3://us-west-2.opendata.source.coop/clay/lgnd-clay-v1-5-sentinel-2-l2a/aggregated/geohash_l2=9y/year=2025/month=06/ --summarize --human-readable

2026-01-26 00:50:31  359.5 MiB data_01c1fab1-0004-5b1c-0009-b72e01d3104e_222_0_0.parquet

Total Objects: 1
   Total Size: 359.5 MiB
```

The aggregated product is easier to use than the scene-level product across large spatial extents, and is best for time series analysis or when constent temporal sampling is required.  This product contains  ~49.7 million embeddings and takes up ~182GB on disk.

## Data Model

Both data products follow the same data model, described below:

| Column | Type | Description |
|--------|------|-------------|
| `chips_id` | string | Spatio-temporally unique identifier for each embedding |
| `cell_id` | string | Spatially unique identier for the MajorTOM cell |
| `rasters_id` | string | Unique, internal LGND identifier for the source raster |
| `stac_item_id` | string | STAC item ID of the source Sentinel-2 scene, from [Earth Search](https://element84.com/earth-search/) |
| `collection` | string | STAC collection name (e.g., `sentinel-2-l2a`), also from Earth Search |
| `datetime` | timestamp | Acquisition timestamp of the source imagery |
| `embedding` | list[float32] | 1024-dimensional Clay v1.5 embedding vector |
| `bbox` | struct | Bounding box with `xmin`, `ymin`, `xmax`, `ymax`, for geoparquet reader optimizations |
| `geometry` | polygon (wkb) | Polygon geometry of the chip footprint |

## Data Format

Both data products are provided as hive-partitioned geoparquet files, following [geoparquet best practices](https://geoparquet.io/concepts/best-practices/):
- Geoparquet format version 1.1.
- ZSTD compression (level 15).
- ~128 MB row groups where possible.
- Spatial-temporal partitioning.
- Spatial ordering with a hilbert curve.
- Bbox column with covering metadata for faster spatial lookups.

## License

This product is licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).

## Known Issues
There are several known data quality issues:
- There are a few small gaps in coverage due to missing remote sensing data or edge cases in our scene selection algorithm, we estimate this dataset is about ~99.7% complete.

## Future work
In the coming months, the LGND team plans to fix these data gaps and push embeddings for more time steps beyond the two originally provided.

providers: role [processor] for 'Made With Clay' is an AI guess — Clay v1.5 is the foundation model used to generate the embeddings; named as model source, not explicitly as a dataset provider. Verify before flipping to _state: reviewed.

extent.spatial.bbox: omitted; spatial reference is categorical via [data_01c1fab1-0004-5b1c-0009-b72e01d3104e_222_0_0.geometry (geometry), data_01c1fab1-0004-5b1c-0009-b72e01d3104e_222_0_0.bbox (geometry), data_01c1fab1-0004-5b1c-0009-b72e01d3104e_222_0_0.geohash_l2 (spatial_index), data_01c1fab1-0004-5b20-0009-b72e01d10bc6_315_2_0.geometry (geometry), data_01c1fab1-0004-5b20-0009-b72e01d10bc6_315_2_0.bbox (geometry), data_01c1fab1-0004-5b20-0009-b72e01d10bc6_315_2_0.gzd (spatial_index)]. Add bbox once a placename → bbox gazetteer is available.


LGND Embeddings

This repository contains Clay embedding datasets created and maintained by [lgnd.ai](lgnd.ai). Licensed under CC BY 4.0.


Google Satellite Embeddings v1

Google Satellite Embeddings v1 (https://developers.google.com/earth-engine/datasets/catalog/GOOGLE_SATELLITE_EMBEDDING_V1_ANNUAL#description).

Aggregated to 250m x 250m tiles for ease of search.

Includes parquet files by MGRS ID, and fully built duckdb tables and FAISS indexes.


3Dclouds

This repository hosts the AI-ready datasets for the paper "Global 3D Reconstruction of Clouds & Tropical Cyclones." It contains paired 2D geostationary imagery (GOES, MSG, Himawari) and 3D vertical profiles (CloudSat/CALIPSO) used to train ML models for 3D cloud reconstruction, with a dedicated dataset for tropical cyclones.


methaneset

methaneset


Model-based results from 2,760 simulated Direct Ocean Removal (DOR) experiments using the Community Earth System Model (CESM), covering 690 global ocean regions and four intervention seasons (January, April, July, October). Each 15-year experiment captures the temporal evolution of dissolved inorganic carbon (DIC) removal on the ocean biogeochemical carbon system, stored as monthly-average netCDF files alongside counterfactual ocean states. Also includes globally-integrated DOR efficiency maps derived from all simulations.

oceanography climate time-series ocean-carbon biogeochemistry climate-modeling

Dataset files (496802 x .nc)

Source Cooperative product page

## README.md
# Ocean Carbon Dioxide Removal Atlas: Global Direct Ocean Removal (DOR) Map

## Description

The files in this repository contain the model-based results from a large number of simulated Direct Ocean Removal (DOR) experiments using the Community Earth System Model ([CESM](https://www.cesm.ucar.edu/models)).  These experiments are intended to explore the temporal and spatial impact of regionally confined carbon removal on the biogeochemical carbon system in the global ocean.

To produce these model-based experiments, the CESM model was run in what is called a “forced ocean sea ice” configuration, wherein an independent dataset of historical atmospheric conditions (including the observed trend in atmospheric CO2) is used to drive the circulation and state of the CESM global ocean model. The details of these simulations identical to those of the [Global Ocean Alkalinity Enhancement Efficiency Map](https://source.coop/repositories/cworthy/oae-efficiency-atlas) — with the sole exception that DOR forcing was applied instead of alkalinity addition.

This repository contains:

1. A series of "DOR perturbation experiments," which span a 15 period beginning in the model simulated year 1999; and the “counterfactual” ocean state for these experiments, where there was no DOR perturbation to the biogeochemical system

2. Globally-integrated DOR efficiencies for each point on the model’s horizontal grid, defined as the cumulative integral in time of CO2 uptake from the atmosphere divided by the amount of alkalinity added to the ocean.

The following elements are **not included** in this repository because they are shared with the OAE dataset and are not duplicated here. Interested users can access them via the OAE dataset, where they were originally published:

1. A nearly 200-year “control” integration used for historical and physical context, including the response to the observed anthropogenic CO₂ trend. Data available at [`data/control/*.nc`](https://source.coop/cworthy/oae-efficiency-atlas/data/control)

2. Region masks identifying the 690 global ocean regions where carbon removal was applied. Available [here](https://source.coop/cworthy/oae-efficiency-atlas/data/polygon_masks.nc)

## Structure

This repository contains:

1. 496,800 netCDF files output from 2760 CESM simulations. Each simulated DOR perturbation experiment represents the time-evolving outcome of removing dissolved inorganic carbon (DIC) from a particular region of the ocean (the "polygon") over a particular time period (the "intervention month"), and integrating an ocean physical-biogeochemical model forwards for 15 years. These results are stored in the `data/experiments/` directory, following the pattern `{polygon_id}/{intervention_month}/smyle.cdr-atlas-v0.glb-dor.{polygon_id}-{intervention_year}-{intervention_month}.pop.h.{year}-{month}.nc`. These files contain the globally complete set of biogeochemical variables that are impacted by the DOR perturbation. The data is saved as monthly averages, with one file per month, for a total of 180 files per 15 year simulation. The meaning of the identifiers are:
   - `{polygon_id}` - the unique integer ID of each polygon, ranging from 000 to 690.
   - `{intervention_year}` - indicates the calendar year during which the DOR intervention occurred. Presently all the data are from simulations beginning in 1999.
   - `{intervention_month}` - the calendar month during which the DIC removal occurred. One of 01, 04, 07, 10.
   - `{year}` - the model year for which data is contained in this particular file. Model year 0347 corresponds to the calendar year 1999.
   - `{month}` - the simulation calendar month for which data is contained in this particular file. Ranges from 01 to 12. Note that the time variable in these files contains values that correspond to the timestamp at the end of each month.
These netCDF files also contain variables representing the counterfactual condition for each of the DOR perturbation experiments. The counterfactual simulation is defined as the time-evolving biogeochemical state of the ocean in the absence of the DIC removal. These variables are stored using the same naming convention as above, but with a suffix `_ALT_CO2` (e.g. `’DIC’` contains the experimental values and `’DIC_ALT_CO2’` contains the counterfactual values).

2. Integrated efficiency curves in the file `data/DOR_efficiency_maps_alltime.nc`. These contain the globally-integrated DOR efficiencies for each point on the model’s horizontal grid (rather than being defined for each spatial polygon). The efficiency curves are defined as the cumulative integral in time of CO2 uptake from the atmosphere divided by the amount of DIC removed from the ocean. This file includes a variable `DOR_efficiency` defined over the dimensions `season`, `N_month`, `nlat`, and `nlon`. `season` is the month of the DIC removal: ['January', 'April', 'July', 'October']; `N_month` is the number of months elapsed since the DIC removal.

## Future Plans

We eventually aim to make this entire dataset accessible as a single Zarr Store via virtual chunk reference files.

## How to access

### BROWSE raw netCDF files

You can find any particular netCDF file of interest by navigating through this repository after clicking `BROWSE` on the left. You can then click `DOWNLOAD` to download the file directly (this does not require AWS credentials).

### ACCESS DATA and open specific netCDF files with xarray

After signing in to Source Cooperative, you can click ACCESS DATA to generate temporary AWS credentials that will allow you to download the data via URL.
You can download the entire dataset (Not recommended due to size) following the “Usage Examples” provided.
You can also access specific files from a URL pointing to the bucket using [xarray](https://xarray.dev/) and [s3fs](https://s3fs.readthedocs.io/en/latest/), which you will need to have installed (e.g. from conda-forge). You will also need to have [h5netcdf](https://github.com/h5netcdf/h5netcdf) installed. You then need to run the following commands from an interactive python shell (e.g. in a [jupyter notebook](https://jupyter.org/)).

```python
import xarray as xr
import s3fs

# set up anonymous S3 access
fs = s3fs.S3FileSystem(anon=True)
data_dir = 's3://us-west-2.opendata.source.coop/cworthy/dor-efficiency-atlas/data'

polygon_id = "000"
intervention_month = "01"
intervention_year = "1999"
year = "0347"
month = "01"

data_file = (
	f"experiments/{polygon_id}/{intervention_month}/"
	f"smyle.cdr-atlas-v0.glb-dor.{polygon_id}-{intervention_year}-{intervention_month}.pop.h.{year}-{month}.nc"
)
data_url = f"{data_dir}/{data_file}"

ds_data = xr.open_dataset(fs.open(data_url), engine='h5netcdf')

```

## References

A paper documenting the generation of these data and analysis of the results is in preparation.

The related OAE Efficiency Map data are documented in the following publication:
M. Zhou, M. Tyka, D. Ho, E. Yankovsky, S. Bachman, T. Nicholas, A. Karspeck, M. Long: Mapping the global variation in the efficiency of ocean alkalinity enhancement for carbon dioxide removal, 2024, Nature Climate Change, in press, doi: 10.1038/s41558-024-02179-9. [see preprint](https://www.researchsquare.com/article/rs-4124909/v1)

## Contact Us

We are interested in how you are using this data! Please reach out to [data@cworthy.org](data@cworthy.org) with any questions. We’d also appreciate information on your analysis goals and whether the data format and access patterns adequately support your use cases.

## License

[Open Data Commons Attribution License (ODC-By) v1.0](https://opendatacommons.org/licenses/by/1-0/) , packaged with the data in the `LICENSE.txt` file.


Model-based results from 2,760 simulated Ocean Alkalinity Enhancement (OAE) experiments using the Community Earth System Model (CESM), exploring the temporal and spatial impact of regional alkalinity addition on the global ocean biogeochemical carbon system across 690 ocean regions. The dataset includes ~497,000 monthly netCDF output files from perturbation and control simulations, globally-integrated OAE efficiency maps, and spatial polygon masks covering the entire global ocean.

ocean-alkalinity-enhancement ocean-biogeochemistry oceanography biodiversity climate-modeling climate marine-carbon-cycle cesm cdr

Dataset files (497250 x .nc)

Source Cooperative product page

## README.md
# Ocean Carbon Dioxide Removal Atlas: Global Ocean Alkalinity Enhancement Efficiency Map

## Description

The files in this repository contain the model-based results from a large number of simulated Ocean Alkalinity Enhancement (OAE) experiments using the Community Earth System Model ([CESM](https://www.cesm.ucar.edu/models)).  These experiments are intended to explore the temporal and spatial impact of regionally confined alkalinity addition on the biogeochemical carbon system in the global ocean. To produce these model-based experiments, the CESM model was run in what is called a “forced ocean sea ice” configuration, wherein an independent dataset of historical atmospheric conditions (including the observed trend in atmospheric CO2) is used to drive the circulation and state of the CESM global ocean model. The details of these simulations are available in Zhou et al. (2024). This repository contains 1) a nearly 200 year “control” integration that can be used for historical and physical context for the experiment including the impact of the observed anthropogenic trend in atmospheric CO2 on the ocean; and 2) A series of “OAE perturbation experiments,” which span a 15 period beginning in the model simulated year 1999; and the “counterfactual” ocean state for these experiments, where there was no OAE perturbation to the biogeochemical system; 3) Globally-integrated OAE efficiencies for each point on the model’s horizontal grid, defined as the cumulative integral in time of CO2 uptake from the atmosphere divided by the amount of alkalinity added to the ocean; 4) region-masks that indicate the location of the 690 regions of the global ocean from which alkalinity releases were simulated.

## Structure

This repository contains:
1. 496,800 netCDF files output from 2760 CESM simulations. Each simulated  OAE perturbation experiment represents the time-evolving outcome of injecting alkalinity into a particular region of the ocean (the "polygon") over a particular time period (the "injection month"), and integrating an ocean physical-biogeochemical model forwards for 15 years. These results are stored in the `data/experiments/` directory, following the pattern `{polygon_id}/{injection_month}/alk-forcing.{polygon_id}-{injection_year}-{injection_month}.pop.h.{year}-{month}.nc`. These files contain the globally complete set of biogeochemical variables that are impacted by the applied alkalinity perturbation. The data is saved as monthly averages, with one file per month, for a total of 180 files per 15 year simulation. The meaning of the identifiers are:
    - `{polygon_id}` - the unique integer ID of each polygon, ranging from 000 to 690.
    - `{injection_year}` - indicates the calendar year during which the alkalinity injection occurred. Presently all the data are from simulations beginning in 1999.
    - `{injection_month}` - the calendar month during which the alkalinity injection occurred. One of 01, 04, 07, 10.
    - `{year}` - the model year for which data is contained in this particular file. Model year 0347 corresponds to the calendar year 1999.
    - `{month}` - the simulation calendar month for which data is contained in this particular file. Ranges from 01 to 12. Note that the time variable in these files contains values that correspond to the timestamp at the end of each month.
These netCDF files also contain variables representing the counterfactual condition for each of the OAE perturbation experiments. The counterfactual simulation is defined as the time-evolving biogeochemical state of the ocean in the absence of the alkalinity injection. These variables are stored using the same naming convention as above, but with a suffix `_ALT_CO2` (e.g. `’ALK’` contains the experimental values and `’ALK_ALT_CO2’` contains the counterfactual values).
2. NetCDF files for the dataset from the control simulation. The control simulation is a so-called forced-ocean-sea-ice CESM integration from 1850 to 2021 and each of the OAE perturbation experiments was branched off this simulation in the year 1999. Since there is no feedback between alkalinity addition and the physical dynamics in the model, data from the control integration can be used to provide access to physical variables (e.g., ocean velocity, mixed layer depth, etc.) that were omitted from the experiments to save data volume. These are stored in various files under `data/control/*.nc`, with one file per data variable.
3. Integrated efficiency curves in the file `data/OAE_efficiency_maps_alltime.nc`. These contain the globally-integrated OAE efficiencies for each point on the model’s horizontal grid (rather than being defined for each spatial polygon). The efficiency curves are defined as the cumulative integral in time of CO2 uptake from the atmosphere divided by the amount of alkalinity added to the ocean. This file includes a variable `OAE_efficiency` defined over the dimensions `season`, `N_month`, `nlat`, and `nlon`. `season` is the month of the alkalinity release: ['January', 'April', 'July', 'October']; `N_month` is the number of months elapsed since the alkalinity release.
4. Integrated efficiency curves in the file `data/oae_eff_curves_global.nc`. These contain the globally-integrated OAE efficiencies for each spatial polygon.
5. Masks for the spatial locations of each polygon_id, stored in the single netCDF4 file `data/polygon_masks.nc`. All 690 polygons together tile the entire global ocean.

## Future Plans
We eventually aim to make this entire dataset accessible as a single Zarr Store via virtual chunk reference files.

## How to access

### BROWSE raw netCDF files

You can find any particular netCDF file of interest by navigating through this repository after clicking `BROWSE` on the left. You can then click `DOWNLOAD` to download the file directly (this does not require AWS credentials).

### ACCESS DATA and open specific netCDF files with xarray

After signing in to Source Cooperative, you can click ACCESS DATA to generate temporary AWS credentials that will allow you to download the data via URL.
You can download the entire dataset (Not recommended due to size) following the “Usage Examples” provided.
You can also access specific files from a URL pointing to the bucket using [xarray](https://xarray.dev/) and [s3fs](https://s3fs.readthedocs.io/en/latest/), which you will need to have installed (e.g. from conda-forge). You will also need to have [h5netcdf](https://github.com/h5netcdf/h5netcdf) installed. You then need to run the following commands from an interactive python shell (e.g. in a [jupyter notebook](https://jupyter.org/)).

```python
import xarray as xr
import s3fs

!<copy/pasted temporary credentials>

fs = s3fs.S3FileSystem(anon=True)
data_dir = 's3://us-west-2.opendata.source.coop/cworthy/oae-efficiency-atlas/data/'
data_file = experiments/{polygon_id}/{injection_month}/alk-forcing.{polygon_id}-{injection_year}-{injection_month}.pop.h.{year}-{month}.nc
data_url = data_dir + data_file
ds_data = xr.open_dataset(fs.open(data_url), engine='h5netcdf')
```

### Open entire dataset as Zarr with xarray
_Note: Access as Zarr is coming soon!_

## References
The generation of this data and analysis of the results is documented in the following publication; to cite this dataset please cite this paper:
M. Zhou, M. Tyka, D. Ho, E. Yankovsky, S. Bachman, T. Nicholas, A. Karspeck, M. Long: Mapping the global variation in the efficiency of ocean alkalinity enhancement for carbon dioxide removal, 2024, Nature Climate Change, in press, doi: 10.1038/s41558-024-02179-9. [see preprint](https://www.researchsquare.com/article/rs-4124909/v1)

## Contact Us
We are interested in how you are using this data! Please reach out to [data@cworthy.org](data@cworthy.org) with any questions. We’d also appreciate information on your analysis goals and whether the data format and access patterns adequately support your use cases.

## License
[Open Data Commons Attribution License (ODC-By) v1.0](https://opendatacommons.org/licenses/by/1-0/) , packaged with the data in the `LICENSE.txt` file.

providers: role [producer, licensor] for 'C-Worthy' is an AI guess — Identified as producer from S3 path prefix 'cworthy/' and contact email data@cworthy.org; licensor inferred from README authorship and license packaging. Verify before flipping to _state: reviewed.


Data for Canada - Foundation Process Laboratory

This repository hosts systems-ready foundational data for Canada, curated to support modern software and analytical pipelines. By transforming fragmented open data into standardized, high-performance formats, it provides developers and researchers with a reliable baseline for geospatial applications without the need for extensive preprocessing.

Part of the Data for Canada project, this collection focuses on the 'Foundation' theme, essential core layers required for building robust Canadian data infrastructure. For detailed documentation on the processing methodologies and schema, please visit https://www.dataforcanada.org/docs/processes/foundation/.


Data for Canada - Orthoimagery Process Laboratory

This repository provides systems-ready orthoimagery data for Canada, optimized for direct use in cloud-native geospatial applications and analysis pipelines. By standardizing diverse aerial and satellite imagery sources, it offers developers a consistent, high-performance visual baseline without the need for complex pre-processing.

As part of the Data for Canada project, this collection focuses on the 'Orthoimagery' theme, serving as a laboratory for open imagery processing and interoperability. For full documentation on the methodologies used, please visit https://www.dataforcanada.org/docs/processes/orthoimagery/ .


Data for Canada - Elevation Data Packages

A collection of elevation data packages for Canada, comprising over 2,800 GeoTIFF raster files covering parts of eastern Canada (Quebec/Maritime region). The dataset includes ancillary files such as a GeoPackage, CSV, PDF, and JSON, totaling approximately 526 GB.

Dataset files (2858 x .tif)

Source Cooperative product page

providers: role [producer, host] for 'Data for Canada' is an AI guess — Assumed producer and host based on API title; no README to confirm. Verify before flipping to _state: reviewed.


Data for Canada - Environment, Climate, and Health Data Packages

Time-series measurements from a network of low-cost IoT Enviro+ environmental monitoring stations distributed across Canada. Each station records particulate matter (PM1, PM2.5, PM10 and particle size distributions), gas concentrations (oxidised, reducing, NH3), temperature, pressure, humidity, and light intensity. A companion asset captures per-station device telemetry (CPU load, memory, WiFi signal, battery) to support quality-control filtering.

air-quality environmental-monitoring meteorology time-series particulate-matter sensor-network

station

[placename] Identifier naming the monitoring station location

23 files sampled for this table

archive/opensensor.space/enviroplus

8 files sampled for this table

Source Cooperative product page

extent.spatial.bbox: omitted; spatial reference is categorical via [archive/opensensor.space/enviroplus.station (placename), archive/opensensor.space/enviroplus-health.station (placename)]. Add bbox once the placename → bbox gazetteer covers it.


Data for Canada - Field Imagery Data Packages


Data for Canada - Foundation Data Packages

Standardized foundation data packages for Canada sourced from Statistics Canada, comprising three temporal snapshots of the National Address Register (June 2024, December 2024, July 2025) and the 2021 Road Network File. Each address record is enriched with a full hierarchy of StatCan geographic identifiers spanning country, province, census division, CSD, CMA, dissemination area, and dissemination block. Assets are distributed in Parquet, FlatGeobuf, and PMTiles formats.

administrative-boundaries census pmtiles national-address-register road-network statistics-canada

v0.1.0-beta

[placename] Province or territory name (English)

[admin_code] ISO province/territory code

[placename] Census subdivision name

[admin_code] StatCan dissemination area unique geographic identifier (DGUID)

[zipcode] Canadian postal code for the address

geom

[geometry] Point geometry in OGC:CRS84 (WGS84)

STRUCT(xmin FLOAT, ymin FLOAT, xmax FLOAT, ymax FLOAT)

4 files sampled for this table

Source Cooperative product page

extent.temporal: None (probe sample) widened to ['2021-01-01T00:00:00Z', '2025-07-01T00:00:00Z'] using README-stated coverage ['2021', '2025-07'].

extent.spatial.bbox: omitted; spatial reference is categorical via [v0.1.0-beta.geom (geometry), v0.1.0-beta.mail_postal_code (zipcode), v0.1.0-beta.pr_iso_code (admin_code), v0.1.0-beta.pr_en_name (placename), v0.1.0-beta.csd_name (placename), v0.1.0-beta.da_dguid (admin_code)]. Add bbox once the placename → bbox gazetteer covers it.


Data for Canada - Orthoimagery Data Packages

A collection of high-resolution orthoimagery tile packages covering Canadian cities and provinces, processed by Data for Canada from proprietary formats (ECW, MrSID, JP2) into cloud-native PMTiles. Source imagery is acquired from municipal governments and federal agencies at resolutions ranging from 7.5 cm to 30 m. Coverage includes cities such as Winnipeg, Edmonton, Vancouver, Burnaby, Red Deer, and Whitehorse, as well as provincial (Ontario) and national extents.

aerial-imagery remote-sensing pmtiles earth-observation cloud-optimized-geotiff time-series

Dataset files (61528 x .tif)

Source Cooperative product page

## README.md
# d4c-datapkg-orthoimagery

Look through our [d4c-datapkg-orthoimagery](https://github.com/dataforcanada/d4c-datapkg-orthoimagery) repo for the datasets that are currently being processed and datasets being acquired.

## Development Environment

Although we prioritize open-source tools, we currently use [MapTiler Engine Pro](https://www.maptiler.com/engine/pricing) because it outperforms available open-source alternatives for this specific workflow.

## Specifications of Tile Packages

The specifications for the tile packages are defined in this code.

```bash
#!/bin/bash
PROJECT_DIR="~/Documents/Personal/Projects/dataforcanada/d4c-datapkg-orthoimagery"
DATASET_ID="ca-mb_winnipeg-2024A00054611040_orthoimagery_2024_075mm"
DATA_DIR="${PROJECT_DIR}/data"
DATA_INPUT_DIR="${DATA_DIR}/input/${DATASET_ID}"
DATA_OUTPUT_DIR="${DATA_DIR}/output/${DATASET_ID}"

MBTILES_OUTPUT_FILE="${DATA_OUTPUT_DIR}/${DATASET_ID}.mbtiles"
PMTILES_OUTPUT_FILE="${DATA_OUTPUT_DIR}/${DATASET_ID}.pmtiles"

# Define arguments in an array
ARGS=(
  -progress
  -name "City of Winnipeg Orthoimagery for 2024 / Ortho-imagerie de la Ville de Winnipeg de 2024"
  -description "Orthoimagery 7.5cm resolution. / Ortho-imagerie à résolution de 7,5 cm."
  -attribution "Source: data.winnipeg.ca / Source: data.winnipeg.ca"
  -srs_epsg
  -mbtiles_compatible
  -wo "NUM_THREADS=ALL_CPUS"
  -wo "USE_OPENCL=TRUE"
  -sparse
  -scale 2.000000
  -work_dir ~/tmp/maptiler_engine
  -f webp32
  -webp_quality 85
  -webp_lossy
  -webp_preset photo
  -resampling cubic
  -overviews_resampling average
  -o "${MBTILES_OUTPUT_FILE}"
  $DATA_INPUT_DIR/*.ecw
)

# Run the command with the array
maptiler-engine "${ARGS[@]}"

pmtiles convert --tmpdir=~/tmp/pmtiles ${MBTILES_OUTPUT_FILE} ${PMTILES_OUTPUT_FILE}
```

## Download and Preview

Here is a table of some of the datasets created from the current process.

| Place | ISO | Year | Provider | Dataset ID & Preview | PMTiles |
| --- | --- | --- | --- | --- | --- |
| Canada | CA | 2025 | Versatiles | [ca_versatiles-2021A000011124_d4c-datapkg-orthoimagery_2025-08-10](https://source.coop/dataforcanada/d4c-datapkg-orthoimagery/processed/ca_versatiles-2021A000011124_d4c-datapkg-orthoimagery_2025-08-10.pmtiles) | [Download](https://source.coop/dataforcanada/d4c-datapkg-orthoimagery/processed/ca_versatiles-2021A000011124_d4c-datapkg-orthoimagery_2025-08-10.pmtiles) |
| Canada | CA | 2020 | NRCan | [ca_nrcan-2021A000011124_d4c-datapkg-orthoimagery_2020_30m_v0.1.0-beta](https://source.coop/dataforcanada/d4c-datapkg-orthoimagery/processed/ca_nrcan-2021A000011124_d4c-datapkg-orthoimagery_2020_30m_v0.1.0-beta.pmtiles) | [Download](https://source.coop/dataforcanada/d4c-datapkg-orthoimagery/processed/ca_nrcan-2021A000011124_d4c-datapkg-orthoimagery_2020_30m_v0.1.0-beta.pmtiles) |
| Edmonton | CA-AB | 2023 | Edmonton | [ca-ab_edmonton-2023A00054811061_d4c-datapkg-orthoimagery_2023_075mm_v0.1.0-beta](https://source.coop/dataforcanada/d4c-datapkg-orthoimagery/processed/ca-ab_edmonton-2023A00054811061_d4c-datapkg-orthoimagery_2023_075mm_v0.1.0-beta.pmtiles) | [Download](https://source.coop/dataforcanada/d4c-datapkg-orthoimagery/processed/ca-ab_edmonton-2023A00054811061_d4c-datapkg-orthoimagery_2023_075mm_v0.1.0-beta.pmtiles) |
| Red Deer | CA-AB | 2024 | Red Deer | [ca-ab_red-deer-2024A00054808011_d4c-datapkg-orthoimagery_2024_075mm_v0.1.0-beta](https://source.coop/dataforcanada/d4c-datapkg-orthoimagery/processed/ca-ab_red-deer-2024A00054808011_d4c-datapkg-orthoimagery_2024_075mm_v0.1.0-beta.pmtiles) | [Download](https://source.coop/dataforcanada/d4c-datapkg-orthoimagery/processed/ca-ab_red-deer-2024A00054808011_d4c-datapkg-orthoimagery_2024_075mm_v0.1.0-beta.pmtiles) |
| Red Deer | CA-AB | 2025 | Red Deer | [ca-ab_red-deer-2025A00054808011_d4c-datapkg-orthoimagery_2025_075mm_v0.1.0-beta](https://source.coop/dataforcanada/d4c-datapkg-orthoimagery/processed/ca-ab_red-deer-2025A00054808011_d4c-datapkg-orthoimagery_2025_075mm_v0.1.0-beta.pmtiles) | [Download](https://source.coop/dataforcanada/d4c-datapkg-orthoimagery/processed/ca-ab_red-deer-2025A00054808011_d4c-datapkg-orthoimagery_2025_075mm_v0.1.0-beta.pmtiles) |
| Burnaby | CA-BC | 2020 | Burnaby | [ca-bc_burnaby-2020A00055915025_d4c-datapkg-orthoimagery_2020_075mm_v0.1.0-beta](https://source.coop/dataforcanada/d4c-datapkg-orthoimagery/processed/ca-bc_burnaby-2020A00055915025_d4c-datapkg-orthoimagery_2020_075mm_v0.1.0-beta.pmtiles) | [Download](https://source.coop/dataforcanada/d4c-datapkg-orthoimagery/processed/ca-bc_burnaby-2020A00055915025_d4c-datapkg-orthoimagery_2020_075mm_v0.1.0-beta.pmtiles) |
| Vancouver | CA-BC | 2022 | Vancouver | [ca-bc_vancouver-2022A00055915022_d4c-datapkg-orthoimagery_2022_075mm_v0.1.0-beta](https://source.coop/dataforcanada/d4c-datapkg-orthoimagery/processed/ca-bc_vancouver-2022A00055915022_d4c-datapkg-orthoimagery_2022_075mm_v0.1.0-beta.pmtiles) | [Download](https://source.coop/dataforcanada/d4c-datapkg-orthoimagery/processed/ca-bc_vancouver-2022A00055915022_d4c-datapkg-orthoimagery_2022_075mm_v0.1.0-beta.pmtiles) |
| Winnipeg | CA-MB | 2024 | Winnipeg | [ca-mb_winnipeg-2024A00054611040_d4c-datapkg-orthoimagery_2024_075mm_v0.1.0-beta](https://source.coop/dataforcanada/d4c-datapkg-orthoimagery/processed/ca-mb_winnipeg-2024A00054611040_d4c-datapkg-orthoimagery_2024_075mm_v0.1.0-beta.pmtiles) | [Download](https://source.coop/dataforcanada/d4c-datapkg-orthoimagery/processed/ca-mb_winnipeg-2024A00054611040_d4c-datapkg-orthoimagery_2024_075mm_v0.1.0-beta.pmtiles) |
| Whitehorse | CA-YT | 2019 | Whitehorse | [ca-yt_whitehorse-2019A000556001009_d4c-datapkg-orthoimagery_2019_200mm_v0.1.0-beta](https://source.coop/dataforcanada/d4c-datapkg-orthoimagery/processed/ca-yt_whitehorse-2019A000556001009_d4c-datapkg-orthoimagery_2019_200mm_v0.1.0-beta.pmtiles) | [Download](https://source.coop/dataforcanada/d4c-datapkg-orthoimagery/processed/ca-yt_whitehorse-2019A000556001009_d4c-datapkg-orthoimagery_2019_200mm_v0.1.0-beta.pmtiles) |
| Ontario | CA-ON | 2024 | Province of Ontario | [ca-on_province_of_ontario-2024A000235_drape_eastern_ontario_orthoimagery_2024_16cm_v0.1.0-beta](https://source.coop/dataforcanada/d4c-datapkg-orthoimagery/processed/ca-on_province_of_ontario-2024A000235_drape_eastern_ontario_orthoimagery_2024_16cm_v0.1.0-beta.pmtiles) | [Download](https://source.coop/dataforcanada/d4c-datapkg-orthoimagery/processed/ca-on_province_of_ontario-2024A000235_drape_eastern_ontario_orthoimagery_2024_16cm_v0.1.0-beta.pmtiles) |

## The Plan

The objective is to standardize the source datasets used by Data for Canada processes. For instance, Vancouver’s orthoimagery is currently distributed in proprietary formats (MrSID and ECW) that require specialized drivers. We are currently evaluating open alternatives, such as [Cloud Optimized GeoTIFFs](https://cogeo.org/), while ensuring that the conversion process preserves full visual fidelity. Future iterations will also explore the integration of multispectral data.


Data for Canada - Statistical Data Packages

Statistical data packages derived from Statistics Canada's 2021 Census of Agriculture and 2021 Population Census, covering all of Canada. Includes agricultural census variables on crop production, livestock counts, farm finances, land use, and farm operator demographics, as well as health workforce demographic tables from the population census. Data are aggregated to agricultural regions, census divisions, consolidated subdivisions, and aggregate dissemination areas, delivered in cloud-native formats (Parquet, PMTiles, FlatGeobuf).

agriculture crop-classification land-use arable-land census demographics administrative-boundaries pmtiles

processed

[placename] English name of the province or territory

[admin_code] ISO province/territory code (e.g. AB, BC, ON)

[geometry] OGC CRS84 geometry for the census geographic unit (agricultural region, census division, etc.)

5 files sampled for this table

processed/tables

GEO

[placename] Geographic area name for the tabular census record

DGUID

[admin_code] Statistics Canada Dissemination Geography Unique Identifier for the geographic unit

Coordinate

[latlon] Coordinate field associated with the census geographic record

Immigrant status (4):Total - Immigrant status[1]

Immigrant status (4):Non-permanent residents[4]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):Total - Major field of Study - Classification of Instructional Programs (CIP) 2021[1]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):Not applicable: less than 15 years of age[2]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):No postsecondary certificate, diploma or degree[3]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):01. Agricultural and veterinary sciences/services/operations and related fields[4]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):03. Natural resources and conservation[5]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):04. Architecture and related services[6]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):05. Area, ethnic, cultural, gender, and group studies[7]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):09. Communication, journalism and related programs[8]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):10. Communications technologies/technicians and support services[9]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):11. Computer and information sciences and support services[10]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):12. Culinary, entertainment, and personal services[11]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):13. Education[12]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):14. Engineering[13]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):15. Engineering/engineering-related technologies/technicians[14]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):16. Indigenous and foreign languages, literatures, and linguistics[15]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):19. Family and consumer sciences/human sciences[16]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):22. Legal professions and studies[17]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):23. English language and literature/letters[18]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):24. Liberal arts and sciences, general studies and humanities[19]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):25. Library science[20]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):26. Biological and biomedical sciences[21]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):27. Mathematics and statistics[22]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):28. Military science, leadership and operational art[23]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):29. Military technologies and applied sciences[24]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):30. Multidisciplinary/interdisciplinary studies[25]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):31. Parks, recreation, leisure, fitness, and kinesiology[26]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):38. Philosophy and religious studies[27]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):39. Theology and religious vocations[28]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):40. Physical sciences[29]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):41. Science technologies/technicians[30]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):42. Psychology[31]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):43. Security and protective services[32]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):44. Public administration and social service professions[33]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):45. Social sciences[34]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):46. Construction trades[35]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):47. Mechanic and repair technologies/technicians[36]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):48. Precision production[37]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):49. Transportation and materials moving[38]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):50. Visual and performing arts[39]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):51. Health professions and related programs[40]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):52. Business, management, marketing and related support services[41]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):54. History[42]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):55. French language and literature/lettersCAN[43]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):60. Health professions residency/fellowship programs[44]

Major field of Study - Classification of Instructional Programs (CIP) 2021 (45):61. Medical residency/fellowship programs[45]

English used at work (6):Total - English used at work[1]

5 files sampled for this table

Source Cooperative product page

extent.spatial.bbox: omitted; spatial reference is categorical via [processed.geom (geometry), processed.pr_iso_code (admin_code), processed.pr_en_name (placename), processed/tables.GEO (placename), processed/tables.DGUID (admin_code), processed/tables.Coordinate (latlon)]. Add bbox once the placename → bbox gazetteer covers it.


Data for Canada - Web Corpus Data Packages


GeoData-Test

No Description Provided

geo spatial
self-supervised
eatch monitoring


Kenol Fields and Buildings Dataset

This dataset contains satellite imagery collected from drones in Kenol section, Murang'a Kenya.
Dataset is intended to be labelled with field boundary and building footprint.

field boundary
building footprint
sentinel-2


Global Airport Weather GeoParquet (1940-present)

Historical and near-real-time surface weather observations from global ASOS and AWOS airport stations spanning 1940 to present, comprising approximately 1.8 billion decoded METAR reports. Each record captures temperature, dewpoint, wind speed and direction, precipitation, visibility, and atmospheric pressure in both imperial and metric units, with station point geometry embedded per record. Files are partitioned by year and stored as GeoParquet.

weather meteorology time-series geoparquet atmospheric-data climate aviation-weather

Iowa Environmental Mesonet (Iowa State University)

data

longitude

[latlon] Station longitude in decimal degrees (EPSG:4326)

latitude

[latlon] Station latitude in decimal degrees (EPSG:4326)

state

[admin_code] State or sub-national administrative code for the station location

geometry

[geometry] Station point geometry encoded as WKB (EPSG:4326)

country

[placename] Country name or code of the station

county

[placename] County or local administrative area name of the station

88 files sampled for this table

Source Cooperative product page

extent.temporal: None (probe sample) widened to ['1940-01-01T00:00:00Z', '2026-01-01T00:00:00Z'] using README-stated coverage ['1940', '2026'].

extent.spatial.bbox: omitted; spatial reference is categorical via [data.longitude (latlon), data.latitude (latlon), data.geometry (geometry), data.state (admin_code), data.country (placename), data.county (placename)]. Add bbox once the placename → bbox gazetteer covers it.


DWD ICON-EU forecast, 5 day

High-resolution weather forecasts for Europe from the ICON-EU model operated by Deutscher Wetterdienst (DWD), transformed into Icechunk Zarr format by dynamical.org.


DWD ICON Weather Model Archive

Archives of Deutscher Wetterdienst (DWD, the German Meteorological Service) ICON weather model output in grib2 format. DWD maintains only a short rolling window of files on their servers, some of which we archive here.


ECMWF AIFS Single forecast

Global weather forecast dataset from ECMWF's Artificial Intelligence Forecasting System (AIFS) Single model, providing atmospheric variables at 0.25-degree global resolution. Forecasts originate from ECMWF and have been reprocessed into Zarr format by dynamical.org for efficient cloud-native access. Coverage begins April 2024 and is updated on an ongoing basis.

weather-forecasting weather numerical-weather-prediction machine-learning atmospheric-data meteorology global-coverage zarr

Dataset files (1 x .zarr)

Source Cooperative product page

## README.md
# ECMWF AIFS Single forecast - dynamical.org Zarr

See the latest dataset documentation and examples here: https://dynamical.org/catalog/ecmwf-aifs-single-forecast/

license: AI returned 'unknown' (README does not state a license). Verify with publisher before flipping to _state: reviewed.


Ensemble weather forecasts from ECMWF's Integrated Forecasting System (IFS), providing 51-member probabilistic 15-day global forecasts at 0.25-degree spatial resolution with full global coverage. Forecast initialization dates are stored in cloud-optimized Zarr format, processed and openly distributed by dynamical.org.

numerical-weather-prediction ensemble-forecast weather-forecasting weather meteorology atmospheric-data global-coverage zarr

ECMWF (European Centre for Medium-Range Weather Forecasts)

Dataset files (10 x .zarr)

Source Cooperative product page

## README.md
# ECMWF IFS ENS Forecast, 15 day, 0.25 degree - dynamical.org Zarr

See the latest documentation and examples here: https://dynamical.org/catalog/ecmwf-ifs-ens-forecast-15-day-0-25-degree/


ECMWF IFS ENS Weather Model Historical Archive

Archive of ECMWF IFS ENS (Integrated Forecasting System Ensemble) historical weather forecast data in GRIB Edition 1 format at 0.25-degree global resolution. Covers approximately 2,946 forecast initialization dates from 2016-03-08 through 2024-04-01, bridging the gap between ECMWF's tape-backed MARS archive and the public S3 open-data bucket. Each date provides 51 ensemble members (1 control + 50 perturbed) with surface meteorological variables and 500 hPa geopotential at forecast steps out to 360 hours.

numerical-weather-prediction ensemble-forecast meteorology weather-forecasting weather atmospheric-data global-coverage time-series

European Centre for Medium-Range Weather Forecasts (ECMWF)

Dataset files (4211 x .grib)

Source Cooperative product page

## README.md
## ECMWF IFS ENS Forecast Archive

Archives of ensemble weather forecast data from the [European Centre for Medium-Range Weather
Forecasts](https://www.ecmwf.int/) (ECMWF).

ECMWF's public open data bucket (`ecmwf-forecasts`) only provides data from April 2024 onward.
Earlier data is stored in ECMWF's [MARS](https://www.ecmwf.int/en/forecasts/access-forecasts/access-archive-datasets)
(Meteorological Archival and Retrieval System), a tape-backed archive that requires careful request
optimization and authenticated API access.

This tool retrieves IFS ENS (Ensemble) forecast data from MARS and stages it as GRIB files in S3,
covering the period from 2016-03-08 (IFS cycle 41r2, when ENS resolution increased to 18km)
through 2024-04-01 (when S3 open data begins). The staged GRIBs can then be processed by the
[reformatters](https://github.com/dynamical-org/reformatters) into cloud-optimized Icechunk Zarr
format, available at [dynamical.org](https://dynamical.org).

The files are in GRIB Edition 1 format at 0.25 degree resolution. Each date produces four GRIB
files:

| File | Description | Approx Size |
|------|-------------|-------------|
| `pf_sfc_a.grib` | Perturbed forecast, surface, members 1-25, all steps | ~53 GB |
| `pf_sfc_b.grib` | Perturbed forecast, surface, members 26-50, all steps | ~53 GB |
| `cf_sfc.grib` | Control forecast, surface, all steps | ~2.1 GB |
| `all_pl.grib` | All members, geopotential at 500 hPa, all steps | ~9 GB |

Surface variables: `sp`, `2t`, `10u`, `10v`, `100u`, `100v`, `tp`, `strd`, `ssrd`, `msl`, `2d`,
`ptype`. Pressure level variable: `z` (geopotential) at 500 hPa. Forecast steps: 3-hourly 0-144h
and 6-hourly 150-360h (85 steps total). Ensemble members: 0 (control) + 1-50 (perturbed).

Each date totals approximately 103 GB of GRIB data. The full archive (2,946 dates) is approximately
304 TB.

The directory structure in S3 is:

`{s3_prefix}/{YYYY-MM-DD}/{request_type}.grib`

For example: `dynamical/ecmwf-ifs-grib/ecmwf-ifs-ens/2023-07-01/pf_sfc_a.grib`

The staging tool is fully resumable, tracking state in a local SQLite database. It uses ECMWF's
asynchronous MARS API, submitting requests and polling for completion. MARS requests are optimized
to maximize data per tape read: all parameters and forecast steps are requested together (same tape
per date), splitting only by ensemble member count (to stay under MARS's 75 GB retrieval limit) and
by level type (`sfc` vs `pl`).

The code is [available on GitHub](https://github.com/dynamical-org/ecmwf-ifs-backfill). The compute
to retrieve the files is provided by [dynamical.org](https://dynamical.org/).

*Data license*: ECMWF data is available under the [Creative Commons BY 4.0](https://creativecommons.org/licenses/by/4.0/)
license, provided the source is acknowledged.


NOAA GEFS analysis

An archive of the NOAA Global Ensemble Forecast System (GEFS) model's best estimate of past weather conditions, covering the globe at 0.25-degree (~20 km) resolution with 3-hourly timesteps from 2000 to present. Variables include temperature, precipitation, wind components, humidity, radiation flux, cloud cover, and precipitation type stored in Zarr format for cloud-native access. The archive is constructed by concatenating early forecast hours from three GEFS source archives: reforecasts (2000–2019), lower-resolution operational data (Jan–Sep 2020), and full-resolution operational forecasts (Sep 2020–present).

noaa meteorology weather numerical-weather-prediction ensemble-forecast global-coverage zarr atmospheric-data

NOAA National Centers for Environmental Prediction (NCEP)

Dataset files (1 x .zarr)

Source Cooperative product page

## README.md
# NOAA GEFS analysis

_View the latest documentation at https://dynamical.org/catalog/noaa-gefs-analysis/ ._

| | |
|---|---|
| Spatial domain | Global |
| Spatial resolution | 0.25 degrees (~20km) |
| Time domain | 2000-01-01 00:00:00 UTC to Present |
| Time resolution | 3.0 hours |

### Data Access
```
https://data.dynamical.org/noaa/gefs/analysis/latest.zarr?email=optional@email.com
```

*\* Email optional. Providing your email as a query param helps us understand usage and impact to keep dynamical.org supported for the long-term. For catalog updates follow [here](https://dynamical.org/updates).*

The Global Ensemble Forecast System (GEFS) is a National Oceanic and Atmospheric Administration (NOAA) National Centers for Environmental Prediction (NCEP) weather forecast model.

This analysis dataset is an archive of the model's best estimate of past weather. It is created by concatenating the first few hours of each historical forecast to provide a dataset with dimensions time, latitude, and longitude.

This dataset is designed to be used in conjunction with the [GEFS forecast 35 day](https://dynamical.org/catalog/noaa-gefs-forecast-35-day) dataset.

Storage for this dataset is generously provided by [Source Cooperative](https://source.coop/), a [Radiant Earth](https://radiant.earth/) initiative.

## Dimensions

| | min | max | units |
|---|---|---|---|
| **latitude** | -90 | 90 | degrees_north |
| **longitude** | -180 | 179.75 | degrees_east |
| **time** | 2000-01-01T00:00:00 | Present | seconds since 1970-01-01 |

## Variables

| | units | dimensions |
|---|---|---|
| **categorical_freezing_rain_surface**
*Categorical freezing rain* | 0=no; 1=yes | time × latitude × longitude |
| **categorical_ice_pellets_surface**
*Categorical ice pellets* | 0=no; 1=yes | time × latitude × longitude |
| **categorical_rain_surface**
*Categorical rain* | 0=no; 1=yes | time × latitude × longitude |
| **categorical_snow_surface**
*Categorical snow* | 0=no; 1=yes | time × latitude × longitude |
| **downward_long_wave_radiation_flux_surface**
*Surface downward long-wave radiation flux*
*Average value in the last 6 hour period (00, 06, 12, 18 UTC) or 3 hour period (03, 09, 15, 21 UTC).* | W/(m^2) | time × latitude × longitude |
| **downward_short_wave_radiation_flux_surface**
*Surface downward short-wave radiation flux*
*Average value in the last 6 hour period (00, 06, 12, 18 UTC) or 3 hour period (03, 09, 15, 21 UTC).* | W/(m^2) | time × latitude × longitude |
| **geopotential_height_cloud_ceiling**
*Geopotential height* | gpm | time × latitude × longitude |
| **maximum_temperature_2m**
*Maximum temperature* | C | time × latitude × longitude |
| **minimum_temperature_2m**
*Minimum temperature* | C | time × latitude × longitude |
| **percent_frozen_precipitation_surface**
*Percent frozen precipitation*
*Contains the value -50 when there is no precipitation.* | % | time × latitude × longitude |
| **precipitable_water_atmosphere**
*Precipitable water* | kg/(m^2) | time × latitude × longitude |
| **precipitation_surface**
*Total Precipitation*
*Average precipitation rate since the previous forecast step.* | mm/s | time × latitude × longitude |
| **pressure_reduced_to_mean_sea_level**
*Pressure reduced to MSL* | Pa | time × latitude × longitude |
| **pressure_surface**
*Surface pressure* | Pa | time × latitude × longitude |
| **relative_humidity_2m**
*2 metre relative humidity* | % | time × latitude × longitude |
| **temperature_2m**
*2 metre temperature* | C | time × latitude × longitude |
| **total_cloud_cover_atmosphere**
*Total Cloud Cover*
*Average value in the last 6 hour period (00, 06, 12, 18 UTC) or 3 hour period (03, 09, 15, 21 UTC).* | % | time × latitude × longitude |
| **wind_u_100m**
*100 metre U wind component* | m/s | time × latitude × longitude |
| **wind_u_10m**
*10 metre U wind component* | m/s | time × latitude × longitude |
| **wind_v_100m**
*100 metre V wind component* | m/s | time × latitude × longitude |
| **wind_v_10m**
*10 metre V wind component* | m/s | time × latitude × longitude |

*Don't see what you're looking for? Let us know at [feedback@dynamical.org](mailto:feedback@dynamical.org).*

## Examples

* [Open notebook in github](https://github.com/dynamical-org/notebooks/blob/main/noaa-gefs-analysis.ipynb)
* [Open notebook in colab](https://colab.research.google.com/github/dynamical-org/notebooks/blob/main/noaa-gefs-analysis.ipynb)

```python
import xarray as xr  # xarray>=2025.1.2 and zarr>=3.0.4 for zarr v3 support

ds = xr.open_zarr("https://data.dynamical.org/noaa/gefs/analysis/latest.zarr?email=optional@email.com")
ds['temperature_2m'].sel(time="2025-01-01T00", latitude=0, longitude=0).compute()
```

## Details

### Sources
To provide the longest possible historical record, this dataset in constructed from three distinct GEFS forecast archives.

- From 2000-01-01 to 2019-12-31 we use the [GEFS reforecast](https://registry.opendata.aws/noaa-gefs-reforecast/).
- From 2020-01-01 to 2020-09-23 we use [GEFS forecast archive](https://registry.opendata.aws/noaa-gefs/) data which has a lower spatial and temporal resolution.
- From 2020-09-23 to Present we use [GEFS operational forecast archives](https://registry.opendata.aws/noaa-gefs/).

### Variable availability
Data is available for all variables at all times with the following exceptions.

- Unavailable before 2020-01-01:
  `relative_humidity_2m`,
  `percent_frozen_precipitation_surface`,
  `categorical_freezing_rain_surface`,
  `categorical_ice_pellets_surface`,
  `categorical_rain_surface`,
  `categorical_snow_surface`
- Unavailable 2020-01-01T00 to 2020-09-22T21:
  `geopotential_height_cloud_ceiling`

### Construction
To create a single time dimension we concatenate the first few hours of each forecast. From 2000-01-01 to 2019-12-31 reforecasts are available once per day and this dataset uses the first 21 or 24 hours of each forecast. From 2020-01-01 to present forecasts are available every 6 hours and this dataset uses the first 3 or 6 hours of each forecast. Variables with an instantaneous `step_type` use the shortest possible lead times (e.g. 0 and 3 hours) while accumulated variables must use one additional forecast step (e.g. 3 and 6 hours) because they do not have an hour 0 forecast value.

### Interpolation
For most of the time range of the archive the source data is available at 0.25-degree resolution and a 3 hourly time step and we perform no interpolation. There are two exceptions to this. 1) From 2020-01-01 to 2020-09-23 the source data has a 1.0-degree spatial resolution and a 6 hourly time step. 2) From 2020-09-23 to present the 100m wind components have a 0.5-degree spatial resolution in the source data. To provide a consistent archive in the above two cases we first perform bilinear interpolation in space to 0.25-degree resolution followed by linear interpolation in time to a 3-hourly timestep if necessary. The original, uninterpolated data can be obtained by selecting latitudes and longitudes evenly divisible by 1 and, in case 1), time steps whose hour is divisible by 6.

### Compression
The data values in this dataset have been rounded in their binary floating point representation to improve compression. See [Klöwer et al. 2021](https://www.nature.com/articles/s43588-021-00156-2) for more information on this approach. The exact number of rounded bits can be found in our [reformatting code](https://github.com/dynamical-org/reformatters/blob/main/src/reformatters/noaa/gefs/common_gefs_template_config.py).