Taxonomic names

A consolidated dataset of taxonomic naming authorities drawn from multiple biodiversity databases, including Catalogue of Life (COL), GBIF, IUCN, ITIS, NCBI, and Open Tree of Life (OTT). Each record links scientific names, vernacular (common) names, taxonomic rank, and taxonomic status across the full Linnaean hierarchy (kingdom through infraspecific epithet). The dataset contains over 27 million records distributed across 9 Parquet files in both Darwin Core and simplified common-name formats.

taxonomy biodiversity species-names nomenclature taxonomic-classification vernacular-names

National Center for Biotechnology Information (NCBI)

common_col_part_0.parquet

common_gbif_part_0.parquet

country

[placename] Country name associated with a vernacular name usage in GBIF

countryCode

[admin_code] ISO 3166-1 alpha-2 country code associated with a vernacular name usage in GBIF

common_itis_part_0.parquet

common_ncbi_part_0.parquet

dwc_col_part_0.parquet

dwc_gbif_part_0.parquet

dwc_itis_part_0.parquet

dwc_ncbi_part_0.parquet

dwc_ott_part_0.parquet

Source Cooperative product page

providers: role [producer] for 'International Union for Conservation of Nature (IUCN)' is an AI guess — Listed in API description as a source but no IUCN-specific asset is visible in the file listing.. Verify before flipping to _state: reviewed.

providers: role [producer] for 'Integrated Taxonomic Information System (ITIS)' is an AI guess — Inferred as producer from asset names (common_itis, dwc_itis); not explicitly named in the API description.. Verify before flipping to _state: reviewed.

providers: role [producer] for 'Open Tree of Life Taxonomy (OTT)' is an AI guess — Inferred as producer from asset name (dwc_ott); not explicitly named in the API description.. Verify before flipping to _state: reviewed.

extent.spatial.bbox: omitted; spatial reference is categorical via [common_gbif_part_0.country (placename), common_gbif_part_0.countryCode (admin_code)]. Add bbox once a placename → bbox gazetteer is available.


US Federal Trails Network

A harmonized composite of 127,619 U.S. federal trail segments from the Forest Service (NFST), National Park Service, and Bureau of Land Management, unified into a single schema with recomputed lengths and a derived National Trail System designation column flagging National Scenic Trails, National Historic Trails, and National Recreation Trails. Published as GeoParquet (per-segment), PMTiles for map rendering, and H3 hex-indexed parquet at resolution 8 for spatial aggregation.

geoparquet pmtiles h3 trails federal-lands national-scenic-trails recreation

federal-trails-2026.parquet

geometry

[geometry] Trail segment linestring geometry in OGC:CRS84 (EPSG:4326), reprojected from source agency data

federal-trails-2026/hex

[spatial_index] H3 hex cell index at resolution 8 covering the line-buffered trail segment footprint

[spatial_index] H3 hex cell index at resolution 0, used as Hive partition key for the hex asset

13 files sampled for this table

Source Cooperative product page

## federal-trails-2026__README.md
# Federal Trails Network 2026

A composite of U.S. federal-agency-managed trails in a single harmonized table:

- **U.S. Forest Service** — National Forest System Trails (NFST), 82,687 segments
- **National Park Service** — Public Trails (filtered to publicly-displayable rows), 31,386 segments
- **Bureau of Land Management** — National GTLF Public Managed Trails, 19,104 segments

**Total: 127,619 segments.** One row per published trail segment, harmonized schema, recomputed `length_miles`, and a `nts_designation` column flagging the 11 National Scenic Trails, 19 National Historic Trails, and National Recreation Trails where applicable.

## Data assets

| Asset | URL |
|---|---|
| GeoParquet (per-segment) | `https://s3-west.nrp-nautilus.io/public-trails/federal-trails-2026.parquet` |
| PMTiles (rendering)       | `https://s3-west.nrp-nautilus.io/public-trails/federal-trails-2026.pmtiles` |
| H3 hex (native res 8)     | `https://s3-west.nrp-nautilus.io/public-trails/federal-trails-2026/hex/h0=*/data_0.parquet` |
| STAC                      | `https://s3-west.nrp-nautilus.io/public-trails/federal-trails-2026/stac-collection.json` |

## Query examples

### How many miles of PCT are within a polygon? (use GeoParquet, not hex)

```sql
LOAD spatial;
SELECT ROUND(SUM(length_miles), 1) AS pct_miles
FROM read_parquet('https://s3-west.nrp-nautilus.io/public-trails/federal-trails-2026.parquet')
WHERE nts_designation = 'PCT'
  AND ST_Intersects(geometry, ST_GeomFromText('POLYGON((...))'));
```

### Which hex cells does the AT pass through?

```sql
SELECT DISTINCT h8
FROM read_parquet(
  'https://s3-west.nrp-nautilus.io/public-trails/federal-trails-2026/hex/h0=*/data_0.parquet',
  hive_partitioning = true
)
WHERE nts_designation = 'AT';
```

> **Do not** `SUM(length_miles)` on the hex asset. Each segment's `length_miles` is repeated on every hex cell it touches after circumradius buffering, so SUM overcounts by the cell-per-segment factor.

### MapLibre

```js
map.addSource('trails', {
  type: 'vector',
  url: 'pmtiles://https://s3-west.nrp-nautilus.io/public-trails/federal-trails-2026.pmtiles'
});
map.addLayer({
  id: 'trails-line',
  source: 'trails',
  'source-layer': 'federal-trails-2026',   // = last segment of cng-datasets --dataset
  type: 'line',
  paint: { 'line-color': '#7a4d24', 'line-width': 1 }
});
```

Highlight the National Scenic Trails:

```js
map.addLayer({
  id: 'nst-line',
  source: 'trails',
  'source-layer': 'federal-trails-2026',
  type: 'line',
  filter: ['in', ['get', 'nts_designation'], ['literal', ['PCT', 'AT', 'CDT', 'PNT', 'ANT', 'NCT', 'NET', 'FNST', 'IANST', 'NTNST', 'PHT']]],
  paint: { 'line-color': '#b9302a', 'line-width': 2 }
});
```

## Sources & licensing

- **USFS NFST** — https://data.fs.usda.gov/geodata/edw/datasets.php (S_USA.TrailNFS_Publish). U.S. federal-government work; public domain.
- **NPS Public Trails** — https://public-nps.opendata.arcgis.com/, FeatureServer at `mapservices.nps.gov/arcgis/rest/services/NationalDatasets/NPS_Public_Trails_Geographic/FeatureServer/0`. U.S. federal-government work; public domain. **Filtered to `DATAACCESS='Unrestricted' AND PUBLICDISPLAY='Public Map Display'`** to respect agency visibility flags.
- **BLM GTLF Public Managed Trails** — https://gbp-blm-egis.hub.arcgis.com/datasets/BLM-EGIS::blm-natl-gtlf-public-managed-trails, FeatureServer at `services1.arcgis.com/KbxwQRRfWyEYLgp4/arcgis/rest/services/BLM_Natl_GTLF_Public_Managed_Trails/FeatureServer/2`. U.S. federal-government work; public domain (with BLM "as is" disclaimer in the dataset license).

Pipeline source: https://github.com/boettiger-lab/data-workflows/tree/main/catalog/trails

## Methods

1. USFS GDB downloaded; NPS and BLM FeatureServers paged (2000 rows / page) into single GeoJSONs. Raw cached to `s3://public-trails/raw/`.
2. All three layers reprojected to EPSG:4326 via `ogr2ogr -t_srs EPSG:4326 -lco GEOMETRY_NAME=geometry`.
3. Schema harmonized in DuckDB; common columns aliased; `admin_agency`, `nts_designation`, `source_dataset` added.
4. **NTS-designation derivation**: USFS's `NATIONAL_TRAIL_DESIGNATION` integer code is mapped (1 = not designated, 2 = NRT, 3 = NST/NHT). For all three sources, trail-name patterns (`%PACIFIC CREST%`, `%CONTINENTAL DIVIDE%`, `%APPALACHIAN%`, etc.) identify the specific named NST/NHT. BLM uses `ROUTE_SCNDRY_SPCL_DSGNTN_NM` for the canonical designation name, which carries the same patterns.
5. `length_miles` recomputed by reprojecting to EPSG:5070 (USA Contiguous Albers Equal Area, meters) and dividing `ST_Length` by 1609.344. CONUS-accurate; AK/HI/PR/VI lengths are approximate.
6. Final GeoParquet written; cng-datasets workflow produced PMTiles + line-buffered H3 hex at native resolution 8.

## Caveats

- **Line-buffered hex semantics**: each line segment is buffered by the H3 res-8 circumradius before polyfill. A hex row means "a trail passes near this cell" not "within". Good for presence/access queries; not for distance/miles queries.
- **NPS public-display filter** removes segments the agency has marked as not for public release (sensitive resources, trail closures, etc.). Counts will be lower than the raw FeatureServer response.
- **NTS designation** is derived from trail-name patterns and the USFS `NATIONAL_TRAIL_DESIGNATION` numeric code. The 11 National Scenic Trails are matched by name; non-PCT/AT-class connectors that share part of an NST name may be tagged with that NST's code (rare).
- **No BLM**: see scope note above.

## Roadmap

- v2: incorporate BLM trails when a unified source is identified
- v2: split into per-NST sibling collections if downstream apps want them separately

license: AI returned 'unknown' (README does not state a license). Verify with publisher before flipping to _state: reviewed.

providers: role [processor] for 'boettiger-lab' is an AI guess — Inferred as processor from the pipeline GitHub URL (github.com/boettiger-lab/data-workflows); not explicitly named in the README as a provider.. Verify before flipping to _state: reviewed.

extent.spatial.bbox: omitted; spatial reference is categorical via [..geometry (geometry), federal-trails-2026/hex.h8 (spatial_index), federal-trails-2026/hex.h0 (spatial_index)]. Add bbox once the placename → bbox gazetteer covers it.


US Political Boundaries, Including Domestic Sovereign Nations (Federally-Recognized Tribes)

US political and administrative boundaries at state and county level (2020), plus federally-recognized tribal land areas. Provided in geoparquet and pmtiles formats for efficient geographic analysis and visualization.

administrative-boundaries tribal-lands state-boundaries county-boundaries

US Department of Agriculture - Natural Resources Conservation Service

Dataset files (4 x .parquet)

Source Cooperative product page

## README.md
# US Political Boundaries

Includes [geoparquet](https://geoparquet.org/) and [pmtiles](https://docs.protomaps.com/pmtiles/) formatted versions of the following data:

- State Boundaries (2020) [original record](https://www.conservation.gov/datasets/e0021066318948eca81e233b3773e4d6_0/about)
- County Boundaries (2020) [original record](https://www.conservation.gov/datasets/17b89622df5643cda5339ae6649247a6_0/about)

## US Domestic Sovereign Nations: Land Areas of Federally-Recognized Tribes (BIA)

[original record](https://www.conservation.gov/datasets/245ffcb63a0b44cb9ed467bbd5f9d7ea_0/about)

See also:

- Cloud-optimized rendering of census-tract (2020) level data can be found in:
  * The [Social Vulnerability Index](https://beta.source.coop/repositories/cboettig/social-vulnerability)
  * The [Climate and Economic Justice Screening Tool](https://beta.source.coop/repositories/cboettig/justice40)


USGS National Hydrography Dataset in cloud-optimized formats

Experimental cloud-optimized vector tiles and geoparquet for USGS NHD dataset (2024) https://www.usgs.gov/national-hydrography/access-national-hydrography-products


Cloud-optimized publication of U.S. Fish and Wildlife Service Endangered Species Act critical habitat designations for terrestrial and freshwater species under USFWS jurisdiction. Includes two polygon layers: Final (728 features, 462 species) and Proposed (49 features, 42 species) critical habitat designations. Excludes marine species and salmonid stream-line habitat under NOAA NMFS jurisdiction.

conservation biodiversity endangered-species geoparquet pmtiles h3

Dataset files (2 x .pmtiles)

Source Cooperative product page

## README.md
# U.S. Fish and Wildlife Service (USFWS) Datasets

Cloud-optimized publication of geospatial datasets administered by the U.S. Fish and Wildlife Service, hosted on the NRP Nautilus S3 bucket `public-usfws`.

Public URL: <https://s3-west.nrp-nautilus.io/public-usfws/>

## Collections

- **[USFWS Critical Habitat (Final + Proposed)](critical-habitat/stac-collection.json)** — ESA-designated critical habitat for species under USFWS jurisdiction.

> The USFWS National Wetlands Inventory (NWI) is published in the separate `public-wetlands` bucket alongside Ramsar and GLWD; it is not duplicated here.

## USFWS Critical Habitat

Critical habitat under the Endangered Species Act for species under USFWS jurisdiction (terrestrial and freshwater). Two layers:

| Layer | Features | Distinct species | Notes |
|---|---|---|---|
| Final | 728 | 462 | Legally designated, in effect |
| Proposed | 49 | 42 | In ESA rulemaking |

**Source:** FWS HQ ES Critical Habitat (ArcGIS Feature Service, item `794de45b9d774d21aed3bf9b5313ee24`). Polygon features only.

> **Marine species and stream-line critical habitat** for salmonids and other species under NOAA NMFS jurisdiction are NOT included here — those are tracked separately under [data-workflows#144](https://github.com/boettiger-lab/data-workflows/issues/144).

> **Spatial accuracy:** every feature carries the note "Please check current species specific shapefile" — the aggregated layer is generalized for visualization. For the legal extent of any single critical habitat designation, consult the species-specific shapefile via [USFWS ECOS](https://ecos.fws.gov/ecp/report/table/critical-habitat.html).

### Assets

| Layer | Type | URL |
|---|---|---|
| Final | GeoParquet | <https://s3-west.nrp-nautilus.io/public-usfws/critical-habitat/final.parquet> |
| Final | PMTiles | <https://s3-west.nrp-nautilus.io/public-usfws/critical-habitat/final.pmtiles> |
| Final | H3 hex (res 8) | `https://s3-west.nrp-nautilus.io/public-usfws/critical-habitat/final/hex/h0=*/data_0.parquet` |
| Proposed | GeoParquet | <https://s3-west.nrp-nautilus.io/public-usfws/critical-habitat/proposed.parquet> |
| Proposed | PMTiles | <https://s3-west.nrp-nautilus.io/public-usfws/critical-habitat/proposed.pmtiles> |
| Proposed | H3 hex (res 8) | `https://s3-west.nrp-nautilus.io/public-usfws/critical-habitat/proposed/hex/h0=*/data_0.parquet` |

### DuckDB example

```sql
INSTALL spatial; LOAD spatial;
INSTALL httpfs; LOAD httpfs;

-- Top species by number of critical-habitat polygons (Final designations)
SELECT comname, sciname, listing_st, COUNT(*) AS n_polys
FROM read_parquet('https://s3-west.nrp-nautilus.io/public-usfws/critical-habitat/final.parquet')
WHERE comname != 'No common name'
GROUP BY comname, sciname, listing_st
ORDER BY n_polys DESC
LIMIT 10;

-- Pre-compute the H3 cell area at resolution 8 once (square metres):
--   approx 7.37e+05 m² per cell; see h3-guide for exact constants.
-- Approximate critical-habitat area covered by a species, from hex count:
SELECT comname,
       COUNT(DISTINCT h8) * 737327.598 AS approx_area_m2
FROM read_parquet('https://s3-west.nrp-nautilus.io/public-usfws/critical-habitat/final/hex/h0=*/data_0.parquet')
WHERE comname = 'Polar bear'
GROUP BY comname;
```

### MapLibre GL JS example

PMTiles `source-layer` is **`final`** for the Final layer and **`proposed`** for the Proposed layer (the last segment of `--dataset` in the cng-datasets workflow).

```js
map.addSource('ch-final', {
  type: 'vector',
  url: 'pmtiles://https://s3-west.nrp-nautilus.io/public-usfws/critical-habitat/final.pmtiles'
});
map.addLayer({
  id: 'ch-final-fill',
  source: 'ch-final',
  'source-layer': 'final',
  type: 'fill',
  paint: {
    'fill-color': ['match', ['get', 'listing_st'],
      'Endangered', '#dc2626',
      'Threatened', '#f59e0b',
      /* default */ '#9ca3af'
    ],
    'fill-opacity': 0.55,
    'fill-outline-color': '#1f2937'
  }
});
```

## Source

- **Producer:** U.S. Fish and Wildlife Service, Endangered Species Program
- **Service:** <https://services.arcgis.com/QVENGdaPbd4LUkLV/arcgis/rest/services/USFWS_Critical_Habitat/FeatureServer>
- **OpenData portal:** <https://gis-fws.opendata.arcgis.com/maps/794de45b9d774d21aed3bf9b5313ee24>
- **Per-species shapefiles & metadata:** <https://ecos.fws.gov/ecp/report/table/critical-habitat.html>
- **License:** Public domain

## Citation

> U.S. Fish and Wildlife Service. *Threatened & Endangered Species Active Critical Habitat Report.* USFWS Environmental Conservation Online System (ECOS). <https://ecos.fws.gov/ecp/report/table/critical-habitat.html>

license: README states 'Public domain'; this is a U.S. federal government work not subject to copyright. Not formally licensed under a SPDX identifier.


Wetlands


Forest Carbon Project Boundaries

A global database of 575 nature-based carbon offset project boundaries across 55 countries, including avoided deforestation, afforestation/reforestation, and improved forest management projects. Boundaries were aggregated from carbon project registries and manual georeferencing with high accuracy validation (IoU score 0.98±0.015).

forest-carbon carbon-offsets nature-based-solutions climate-change-mitigation avoided-deforestation reforestation forest-management project-boundaries

# A Global Database of Nature-Based Carbon Offset Project Boundaries

Provides 575 global forest carbon project boundaries. The files in this repository have been converted to a fiboa GeoParquet file from a database originally published by Akshata Karnik, Jack B. Kilbride, Tristan R.H. Goodbody, Rachael Ross, Elias Ayrey.

## Dataset Information
- **Converted by:** Alex Logan
- **Original Authors:** Akshata Karnik, Jack B. Kilbride, Tristan R.H. Goodbody, Rachael Ross, Elias Ayrey (Corresponding Author)
- **Date:** June 5, 2024
- **License:** CC-BY 4.0 See https://zenodo.org/records/11459391

## Overview

Nature-based climate solutions (NBS) have become an important component of strategies aiming to reduce atmospheric CO2 and mitigate climate change impacts. Carbon offsets have emerged as one of the most widely implemented NBS strategies. However, these projects have also been criticized for exaggerating offsets. Verifying the efficacy of NBS-derived carbon offset is complicated by a lack of readily available geospatial boundary data.

The original database provided the Geospatial boundaries of 575 NBS projects distributed across 55 countries. Geospatial boundaries were aggregated using a combination of scraping data from carbon project registries (n=433, 75.3%) as well as manual georeferencing and digitization (n=127, 22.1%). Database entries include three varieties of carbon projects:
- Avoided deforestation
- Afforestation, reforestation and re-vegetation
- Improved forest management

An accuracy assessment of the georeferencing and digitizing process indicated a high degree of accuracy (intersection over union score of 0.98 ± 0.015).

## Important Notes for Users

- **Area Definitions:**
  - **Project Area:** The geographical area where project participants implement activities to reduce deforestation
  - **Project Accounting Area:** The geographical area used to calculate carbon credit issuance

- This database does not represent a census of all nature-based carbon projects and does not contain all varieties of such projects

- Users should verify that any georeferencing inaccuracies will not significantly impact their analyses

- The boundaries included reflect data available in the registries at the time of access, with some projects regularly updating their information

- We were unable to assess the accuracy of boundaries constructed from linear features or from developer-provided protocols

## Database Schema

Each record in the GeoPackages contains the following attributes:

| Attribute | Description |
|-----------|-------------|
| **Project Name** | Name of the carbon project as given in the documentation |
| **ProjectID** | Combination of registry abbreviation and project number |
| **Registry Name** | Registry hosting project information (American Carbon Registry, BioCarbon Registry, Climate Action Reserve, EcoRegistry, Gold Standard, Verra) |
| **Methodology** | Name of the methodology used for the project's implementation |
| **Project Type** | Type of forestry carbon offset program (ARR, AD, IFM) |
| **Country** | Country where the project is located |
| **Project Developer Name** | Entity or individual organizing the carbon offset project (first name only if multiple) |
| **Project Start Date** | Start date of the crediting period (mm/dd/yyyy) |
| **Project End Date** | End date of the crediting period (mm/dd/yyyy) |
| **Date of Entry** | Date when project information was added to the database (mm/dd/yyyy) |
| **Processing Approach** | Method used to obtain boundary data (Official, Georeferenced, Linear, or Method) |
| **PD Declined to Provide** | Whether the project developer declined to provide geometry information (Yes, No, N/A) |
| **Geometry Type** | Type of geometry in the database (Point or Polygon) |
| **Project Area** | WKT representation of the project implementation area |
| **Project Accounting Area** | WKT representation of the area used to calculate carbon credits |
| **Project Reference Region** | WKT representation of the area used for historical and current deforestation trends |
| **Comment** | Notes about manual referencing or other relevant information |

## Methodology: Manual Georeferencing & Digitization

Project boundaries were manually georeferenced using QGIS (v3.32.2) through the following process:

1. The map with the clearest boundary within the PDD was identified and overlaid on OpenStreetMap and Bing Satellite Imagery basemaps
2. Each map was georeferenced with the basemap layers using at least 4 ground control points
3. Geometries depicted on the georeferenced map were manually digitized
4. Digitized geometries were consolidated into a single geospatial layer

**Special Cases:**

- **Linear Features (9 projects):**
  - Linear geometries were buffered by 30m
  - Holes were eliminated using the "Delete Holes" tool in QGIS
  - An inverse buffer of 29m was applied to obtain the final polygon

- **Protocol-Based (6 projects):**
  - Project boundaries were recreated following steps in the PDD
  - Used resources like ASTER Global Digital Elevation Model (GDEM) and Copernicus Global Land Cover (CGLC) dataset

- **Point Geometries (42 projects):**
  - For projects with insufficient boundary information
  - General location and project type were identified
  - Entered into database as point geometry

Source Cooperative product page

README

Dataset files (6 × .parquet)

Karnik, A., Kilbride, J. B., Goodbody, T. R. H., Ross, R., & Ayrey, E. (2024). A Global Database of Nature-Based Carbon Offset Project Boundaries. Zenodo. https://zenodo.org/records/11459391


Open Administrative Boundaries

Country-level administrative boundaries from Overture Maps Foundation. Contains 219 country records in GeoParquet format covering global coverage.

administrative-boundaries countries geoparquet overture divisions

Dataset files (1 x .parquet)

Source Cooperative product page


Some random aois for testing

No Description Provided


EuroCrops

EuroCrops compiles all publicly available self-declared crop reporting datasets from European Union member states into a single collection. The dataset provides crop type classifications at the field/parcel scale across the EU, distributed in multiple cloud-native geospatial formats (GeoParquet, FlatGeobuf, PMTiles) with harmonized crop attributes.

agriculture crop-classification field-boundaries land-parcels

Maja Schneider, Marco Körner, Amelie Broszeit

Dataset files (33 x .parquet)

Source Cooperative product page

## README.md
# EuroCrops (Cloud-Native Geo distribution)

This dataset is a copy of the [EuroCrops](https://github.com/maja601/EuroCrops) dataset, offering the data in [cloud-native geospatial](https://cloudnativegeo.org) formats. The overview of the original dataset is:

> EuroCrops is a dataset collection combining all publicly available self-declared crop reporting datasets from countries of the European Union. The project is funded by the German Space Agency at DLR on behalf of the Federal Ministry for Economic Affairs and Climate Action (BMWK). This work is licensed under a [Creative Commons Attribution 4.0 International License](https://creativecommons.org/licenses/by/4.0/).

> Right now EuroCrops only includes vector data, but stay tuned for a version that includes satellite imagery!

> For any questions, please refer to our [FAQs](https://github.com/maja601/EuroCrops/wiki/FAQs) or use the Discussions/Issues to reach out to us.

You can read more details on the dataset itself on the main EuroCrops sites at https://github.com/maja601/EuroCrops and https://www.eurocrops.tum.de/

## About the data formatting and structure

The original EuroCrops dataset is distributed as Shapefiles, one per country, and takes the original data from the country providers as is, enhancing it with 3-4 attributes that are consistent across all the country files. These data come in a variety of projections, since the source data was not consistent. The folder named [unprojected](https://beta.source.coop/cholmes/eurocrops/browse/unprojected) contains the original data structure in different formats.

Note that a few France boundaries have points that are way outside of France, and the Romania boundaries seem to overlap each other. We'll work with the source provider to try to clean those up there, and will redistribute here when ready.

### Modifications

There are several different data types in this repository. The first set have no changes to the original files, but puts them in some alternate formats ([GeoParquet](https://geoparquet.org) and [Flatgeobuf](https://flatgeobuf.org))

The second set projects all geometries into long / lat to make the data easy to work as a single dataset (about half the source files were already in that projection, the rest were country specific).

Then third set of datasets also are all in long lat and additionally remove all the attributes except those that were made common across datasets by the Eurocrops project:

| Attribute Name | Explanation                                                 |
| -------------- | ----------------------------------------------------------- |
| EC_trans_n     | The original crop name translated into English              |
| EC_hcat_n      | The machine-readable HCAT name of the crop                  |
| EC_hcat_c      | The 10-digit HCAT code indicating the hierarchy of the crop |

The NUTS3 attribute was not included as it was not consistently in the datasets. A future iteration of this dataset may try to add that or other country information in the data itself to be able to do parquet partitioning against it.

## Access the data

The easiest way to visualize the data is with this [Felt map](https://felt.com/map/Eurocrops-DRrR1lxoQEqvmZBz6e9APPC), where you can see (almost) all the data visualized and styled. You can also directly access [this pmtiles](https://s3.us-west-2.amazonaws.com/us-west-2.opendata.source.coop/cholmes/eurocrops/eurocrops-all.pmtiles), and see it [shown with the PMTiles Viewer](https://protomaps.github.io/PMTiles/?url=https%3A%2F%2Fs3.us-west-2.amazonaws.com%2Fus-west-2.opendata.source.coop%2Fcholmes%2Feurocrops%2Feurocrops-all.pmtiles#map=2.36/41.89/31.42). These just contain the harmonized attributes, and are projected into the web mercator projection.

The original shapefiles are in the [unprojected/shapefiles](https://beta.source.coop/cholmes/eurocrops/browse/unprojected/shapefiles) directory, and you can find alternate formats in [unprojected/flatgeobuf](https://beta.source.coop/cholmes/eurocrops/browse/unprojected/flatgeobuf) and [unprojected/geoparquet](https://beta.source.coop/cholmes/eurocrops/browse/unprojected/geoparquet).

The [geoparquet-projected/](https://beta.source.coop/cholmes/eurocrops/browse/geoparquet-projected) folder has all the data projected to long / lat, but retains all the original fields.

And then you can get the data as a [single Flatgeobuf file](https://data.source.coop/cholmes/eurocrops/eurocrops-harmonized-only.fgb) (10.8 gb) that is projected and only has the final harmonized fields. We're also making available a [DuckDB](https://duckdb.org) database to experiment with distributing it directly. The data likely will just show up as a blob that you'll have to parse in with the [spatial extension](https://duckdb.org/docs/extensions/spatial).

## Dataset License

The data is licensed under [Creative Commons Attribution 4.0 International License](https://creativecommons.org/licenses/by/4.0/).

## Authors

* Maja Schneider
* Amelie Broszeit
* Marco Körner

### Additional Processing

* Chris Holmes

## Citation & DOI

From https://github.com/maja601/EuroCrops#reference

**Disclaimer**: The official reference will follow soon. Please also reference the countries' dependent source in case you're using their data.

```
@Misc{schneider2022eurocrops21,
 author     = {Schneider, Maja and K{\"o}rner, Marco},
 title      = {EuroCrops},
 DOI        = {10.5281/zenodo.6866846},
 type       = {Dataset},
 publisher  = {Zenodo},
 year       = {2022}
}
```

Additional references:

```
@InProceedings{Schneider2022Challenges,
  title     = {Challenges and Opportunities of Large Transnational Datasets: A Case Study on European Administrative Crop Data},
  author    = {Schneider, Maja and Marchington, Christian and K{\"o}rner, Marco},
  booktitle = {Workshop on Broadening Research Collaborations in ML (NeurIPS 2022)},
  year      = {2022}
}
```

```
@InProceedings{Schneider2022Harnessing,
  title         = {Harnessing Administrative Data Inventories to Create a Reliable Transnational Reference Database for Crop Type Monitoring},
  author        = {Schneider, Maja and K{\"o}rner, Marco},
  booktitle     = {IGARSS 2022-2022 IEEE International Geoscience and Remote Sensing Symposium},
  pages         = {5385--5388},
  year          = {2022},
  organization  = {IEEE}
}
```

```
@InProceedings{Schneider2021EPE,
  author        = {Schneider, Maja and Broszeit, Amelie and K{\"o}rner, Marco},
  booktitle     = {Proceedings of the Conference on Big Data from Space (BiDS)},
  title         = {{EuroCrops}: A Pan-European Dataset for Time Series Crop Type Classification},
  editor        = {Soille, Pierre and Loekken, Sveinung and Albani, Sergio},
  publisher     = {Publications Office of the European Union},
  date          = {2021-05-18},
  doi           = {10.2760/125905},
  eprint        = {2106.08151},
  eprintclass   = {eess.IV,cs.CV,cs.LG},
  eprinttype    = {arxiv}
}
```

```
@Misc{Schneider2021TEC,
  author       = {Schneider, Maja and K{\"o}rner, Marco},
  date         = {2021-06-15},
  title        = {{TinyEuroCrops}},
  doi          = {10.14459/2021MP1615987},
  organization = {Technical University of Munich (TUM)},
  type         = {Dataset},
  url          = {https://mediatum.ub.tum.de/1615987}
}
```


Experiments with fiboa

Place to try out some larger fiboa files, for remote querying, etc.


geoparquet-io test data

Test data for geoparquet-io performance testing


National Hydrography Dataset (United States)

The National Hydrography Dataset (NHD) represents the water drainage network of the United States with features such as rivers, streams, canals, lakes, ponds, coastline, dams, and stream gages.

More information at https://www.usgs.gov/national-hydrography/national-hydrography-dataset

The initial data in this is just NHDFlowlines, and the 'M' coordinate has been removed (as GeoParquet does not yet support M).


Taxi Zones from TLC Trip Record Data

Geospatial boundaries of New York City taxi zones from the TLC Trip Record Data, available in cloud-native formats (GeoParquet and PMTiles) for efficient spatial analysis and visualization.

New York City Taxi and Limousine Commission

Dataset files (2 x .parquet)

Source Cooperative product page

## README.md
# New York Taxi Zones (Cloud-Native Distribution)

This repository contains [Cloud-Native Geospatial](https://cloudnativegeo.org) versions of the Taxi Zones of the [TLC Trip Record Data](https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page). This doesn't contain all the core Parquet data - it's just to make it a bit easier for non-geospatial users to get the actual boundaries. The current web page with all the links says 'Taxi Zone Shapefile (PARQUET)', but the Shapefile is not Parquet. Hopefully the NYC site will update to actually include GeoParquet, but for now it's accessible here. There are 3 files in this repository:

* [taxi_zones.parquet](https://data.source.coop/cholmes/nyc-taxi-zones/taxi_zones.parquet) - A [GeoParquet](https://geoparquet.org) file translated directly from the original Shapefile (using `ogr2ogr`). The shapefile .prj file was translated to [EPSG:2263](https://epsg.io/2263).
* [taxi_zones_4326.parquet](https://data.source.coop/cholmes/nyc-taxi-zones/taxi_zones_4326.parquet) - The original Shapefile transformed into GeoParquet and projected into Long, Lat (EPSG:4326).
* [taxi_zones.pmtiles](https://data.source.coop/cholmes/nyc-taxi-zones/taxi_zones.pmtiles) - The core data as [PMTiles](https://github.com/protomaps/PMTiles) for easy visualization.

***Note: There is also a taxi zone lookup table CSV that seems to have a 'service zone' field. If anyone is interested I could join that to this Parquet so two files aren't needed. Leaving off that work for now to just ship this.***


Demo of publishing OpenET Data

Data from OpenET, converted to fiboa and pmtiles, as a demo of whats possible.


Ordnance Survey Open Data as Cloud Native Geo

This repo is a set of experiments reformatting Ordnance Survey data in Cloud Native Geospatial formats


Overture Maps Data

Global building footprints and points-of-interest (places/POIs) from Overture Maps, redistributed in cloud-native GeoParquet and PMTiles formats partitioned by country and quadkey. The building dataset covers ~2.8 million items across the globe; the places dataset covers worldwide POIs. Originally an experimental mid-2023 redistribution; Overture now releases data natively in these formats.

building-footprints points-of-interest geoparquet pmtiles quadkey

AD.parquet

bbox

STRUCT(maxx DOUBLE, maxy DOUBLE, minx DOUBLE, miny DOUBLE)

[geometry] Bounding box struct (minx, miny, maxx, maxy) for the feature

geometry

[geometry] Building footprint polygon geometry in EPSG:4326

quadkey

[spatial_index] Quadkey tile index used for spatial partitioning

[admin_code] ISO 3166-1 alpha-2 country code

AE.parquet

AF.parquet

AG.parquet

AI.parquet

YE.parquet

categories

STRUCT(alternate STRUCT(list VARCHAR[]), main VARCHAR)

brand

STRUCT("names" STRUCT(key_value STRUCT("key" VARCHAR, "value" STRUCT(list STRUCT(key_value STRUCT("key" VARCHAR, "value" VARCHAR)[])[]))[]), wikidata INTEGER)

geometry

[geometry] Place/POI point geometry in EPSG:4326

YT.parquet

ZA.parquet

ZM.parquet

ZW.parquet

Source Cooperative product page

## README.md
# Overture Buildings & Places (Cloud-Native Geo experiments)

This data is currently just a set of experiments using the new [overture maps](https://overturemaps.org/). See their
[github repo](https://github.com/OvertureMaps/data) for more info. My hope is that this can try out some better ways
to distribute the data, and to build an ecosystem of tools and data distributions on top of the core.

There are two main folders that I recommend starting with:

* [geoparquet-country-quad/](https://beta.source.coop/cholmes/overture/geoparquet-country-quad) is the Overture Building dataset, repartitioned by country, split by quadkey when larger than ~2 gigs, distributed as GeoParquet with row size group of 20,000.
* [places-geoparquet-country/](https://beta.source.coop/cholmes/overture/places-geoparquet-country) is the Overture Places (POI's) dataset, repartitioned by country. There was no need to split by quadkey as the whole dataset is only ~3 gigabytes.

There's also a PMTiles of the buildings dataset in the [root directory](https://beta.source.coop/cholmes/overture). You can check it out at https://observablehq.com/d/2dd90b64498c3864. I do not have a PMTiles up of the places dataset, but you can get one from https://r2-public.protomaps.com/protomaps-sample-datasets/overture-pois.pmtiles, and view that PMTiles at https://bdon.github.io/overture-tiles/places.html. I

If you just want to use Overture data those are the two to focus on. The other folders are a number of experiments and interim processing steps.

* [geoparquet-refined/](https://beta.source.coop/cholmes/overture/browse/geoparquet-refined) is the same files as the original but turned into Geoparquet, with columns added for iso_country and quadkey, and data ordered by quadkey.
* [duckdb/](https://beta.source.coop/cholmes/overture/duckdb) contains a single DuckDB database. It's pretty easy to make this yourself, it is just all the data in geoparquet refined, loaded up into a DuckDB. I'm not sure if distributing DuckDB databases directly is useful. But the file is definitely useful to partition data in different ways.
* [geoparquet-country-quad-2](https://beta.source.coop/cholmes/overture/geoparquet-country-quad-2) I recommend ignoring. It's some alternate partitioning, and the dataset is incomplete. But until I get a set of different repartitioned datasets I'll keep it.

providers: role [processor] for 'Chris Holmes' is an AI guess — Identified as the redistribution author (account 'cholmes') who repartitioned and reformatted the data into GeoParquet/PMTiles.. Verify before flipping to _state: reviewed.

extent.spatial.bbox: omitted; spatial reference is categorical via [AD.geometry (geometry), AD.quadkey (spatial_index), AD.country_iso (admin_code), AD.bbox (geometry), YE.geometry (geometry), YE.quadkey (spatial_index), YE.country_iso (admin_code), YE.bbox (geometry)]. Add bbox once a placename → bbox gazetteer is available.


Dutch Portolan Demo

A collection of datasets from various Dutch institutions to demonstrate a Spatial Data Infrastructure built on Portolan.


Sentinel 2 Grid Vector data

Repository for Sentinel 2 grids (MGRS) in various formats for use online


STAC-GeoParquet Public Missions

(Work In Progress)

This is a collection of public STAC archives, in the STAC-GeoParquet format, for major satellite missions.  To start it's just sentinel 2 and Landsat.

They are copied from Microsoft Planetary computer, updating them to the latest version of Geoparquet and aiming to ensure that they follow all the best practices. Currently there are no 'updates', just a bulk transfer, under the idea that it's pretty easy to get the latest from a STAC API. But in the future we can try to add automated or periodic manual updates.

The current versions aren't yet perfect. They are also quite heavy, so may explore versions that don't include all the assets information.


STAC-GeoParquet of Sentinel 2 level 2a

No Description Provided


Multi-Temporal Cloud Gap Imputation With HLS Imagery Across CONUS

A machine learning benchmark dataset containing 7,852 multi-temporal Harmonized Landsat and Sentinel-2 (HLS) image chips and 21,642 binary cloud masks covering diverse land covers across the Contiguous United States (CONUS) for the year 2022. Each chip is a 224×224 pixel GeoTIFF at 30m resolution with 18 spectral bands (6 bands × 3 time steps). The dataset was created to support training and evaluation of cloud gap imputation models.

remote-sensing land-use cloud-masking multitemporal time-series harmonized-landsat-sentinel landsat sentinel-2 machine-learning imputation

Dataset files (29519 x .tif)

Source Cooperative product page

## README.md
# Multi-Temporal Cloud Gap Imputation With HLS Data Across CONUS

## Dataset Description

### Dataset Summary

This dataset contains temporal Harmonized Landsat and Sentinel-2 (HLS) imagery of diverse land covers across the Contiguous United States (CONUS) for the year 2022 along with binary cloud masks for the same area and year.

### Dataset Structure
```
hls-multi-temporal-cloud-gap-imputation/
├── chip_catalog.csv
├── cloud_catalog.csv
├── train/
│ ├── hls_chips/
│ ├── cloud_masks/
├── test/
│ ├── hls_chips/
│ ├── hls_chips_masked/
```
## Data Catalogs
Data catalogs containing metadata for both HLS scenes and cloud masks are provided. The catalog for HLS scenes is provided in `chip_catalog.csv` and the catalog for cloud masks is provided in `cloud_catalog.csv`.

### Cloud Catalog

`cloud_catalog.csv` contains the following columns:
| Column |  Contains  |
|---------|--------|
| cloud_mask_id       |  'chip_XXX_YYY' corresponding to the file name  |
| cloud_pct       |  ratio of cloudy pixels within the chip, ranges from 0.0 to 1.0 |
| usage       |  'train' or 'validate' |
| bin       |  which of 10 bins the cloud mask fall within, e.g. 0.1-0.2  |

### HLS Chip Catalog

`chip_catalog.csv` contains the following columns:
| Column |  Contains  |
|---------|--------|
| chip_id       |  'chip_XXX_YYY' corresponding to the file name  |
| chip_x       |  x coordinate of bounding box centroid |
| chip_y       |  y coordinate of bounding box centroid   |
| tile       |  HLS tile ID   |
| valid_first       |  count of valid pixels in the first time step  |
| valid_second       |  count of valid pixels in the second time step  |
| valid_third       |  count of valid pixels in the thrid time step  |
| bad_pct_first       |  percent of invalid pixels in the first time step |
| bad_pct_second       |  percent of invalid pixels in the second time step   |
| bad_pct_third       |  percent of invalid pixels in the third time step   |
| first_image_date       |  date of first time step in `YYYY/mm/dd` |
| second_image_date       |  date of second time step in `YYYY/mm/dd`  |
| third_image_date       |  date of third time step in `YYYY/mm/dd`  |
| bad_pct_max       |  maximum of invalid pixels in all time steps  |
| na_count       |  count of pixels in all time steps with no data |
| usage | 'train' or 'validate' |

Invalid pixels are those which intersect with any QA mask.

## Ground Truth HLS Scenes
The ground truth HLS scenes are stored in GeoTIFF format under `train/hls_chips/` and `test/hls_chips/`. Each GeoTIFF file covers a 224 x 224 pixel area at 30m spatial resolution. Each file contains 18 bands consisting of 6 spectral bands in 3 steps stacked together. The file name structure is `chip_XXX_YYY.tif` where `XXX` and `YYY` refer to row and column of a tile grid imposed on the Continental US. Since the dataset is sampled from this country-wide grid not all `XXX`s and `YYY`s are present in the dataset.

## Masked HLS Scenes for Testing
Testing scenes are pre-masked to ensure that all models are evaluated using the same test set. These scenes are stored in GeoTIFF format under `test/hls_chips`. The file name structure for masked scenes is `chip_XXX_YYY_masked.tif`. Each masked scene corresponds to the ground truth scene with the same value of `XXX_YYY` in the file name. For example, the file `test/hls_chips/chip_373_294.tif` is the ground truth for `test/hls_masked/chip_373_294_masked.tif`, with the latter having values of 0 at cloud-masked locations. Cloud masks are present in all possible combinations of time steps in equal proportion. Possible combinations given time steps t1, t2, and t3 are:

- t1
- t2
- t3
- t1, t2
- t1, t3,
- t2, t3,
- t1, t2, t3

So, for example, 1/7th of test scenes are masked at ONLY t2, and 1/7th at t2 AND t3, etc. for each of the possible combinations.

Cloud masks for the test scenes range from 0.01% coverage to 100% coverage, and are equally sampled from 10 equally sized bins between 0-100%.

## Cloud Masks
The training cloud masks are stored in GeoTIFF format under `train/cloud_masks/`. The file name structure for cloud masks is `chip_XXX_YYY_T_cmask.tif` where `XXX` and `YYY` refer to row and column of a tile grid imposed on the Continental US. `T` refers to the time step of each cloud mask and is meant only to distinguish cloud masks derived from the same location from each other. The intent for training is that these cloud masks are randomly paired with training HLS scenes in all time steps. The distribution of cloud mask coverage for the training set does not correspond to the distribution of cloud mask coverage for the validation set, as the distribution of the latter has been equalized. This may lead to higher validation accuracy if the user chooses not to equalize the training dataset - it is left to the user's discretion.

## Band Order
In each HLS GeoTIFF the following bands are repeated for each of three observations throughout the year:
| Channel |  Name  |  HLS   S30 Band number |
|---------|--------|------------------------|
| 1       |  Blue  |  B02                   |
| 2       |  Green |  B03                   |
| 3       |  Red   |  B04                   |
| 4       |  NIR   |  B8A                   |
| 5       |  SWIR 1  |  B11                   |
| 6       |  SWIR 2  |  B12                   |

Masks are a stored as a single-band binary image where 1 denotes the presence of the cloud mask and 0 denotes the absence of the cloud mask.

## Dataset Creation

Code used to generate HLS scenes and cloud masks is available [here](https://github.com/ClarkCGA/cloud-gap-filling-td). Code used to generate masked test scenes is available [here](https://github.com/ClarkCGA/gfm-gap-filling-baseline/blob/main/gfm-gap-filling-baseline/gap-filling-baseline/datasets/gapfill.py). `usage='validate'` was used along with default parameters when initializing the dataset using the `gapfill.py` code. Refer to [Seeing Through the Clouds: Cloud Gap Imputation with Prithvi Foundation Model](https://arxiv.org/abs/2404.19609) for further information about the creation and initial use of this dataset.

### Chip Generation and Partitioning
Three HLS scenes were selected between Mar and Sep 2022 with time difference between scenes varying between 1 and 200 days. After filtering for missing values and cloudy pixels, a total of 7,852 cloud-free chips evenly distributed across the CONUS were generated. This set was randomly partitioned into training (80%) and validation (20%) sets, resulting in 6,231 training chips and 1,621 validation chips.

### Cloud Generation and Partitioning
Cloud masks were generated from the same region of CONUS using HLS cloud mask quality flag and exported as a binary layer of cloudy and non-cloudy pixels. This yielded 21,642 cloud masks, of which 1,600 were randomly selected and reserved for validation, resulting in 20,042 training cloud masks

### License and Citation
This dataset is published under a CC-BY-4.0 license. If you find this dataset useful for your application, you can cite it as following:

```
@misc{hls-multi-temporal-cloud-gap-imputation,
    author = {Godwin, Denys and Li, Hanxi (Steve) and Alemohammad, Hamed},
    doi    = {https://doi.org/10.5281/zenodo.11281740},
    title  = {{Multi-Temporal Cloud Gap Imputation With HLS Data Across CONUS}},
    version = {1.0},
    year   = {2024}
}
```
### Contact
For any questions about the dataset, you can contact [Dr. Hamed Alemohammad](mailto:halemohammad@clarku.edu).

### Funding
This dataset is generated with funding from a grant awarded to [Clark University Center for Geospatial Analytics (CGA)](https://www.clarku.edu/cga/) by NASA.

providers: role [licensor] for 'NASA' is an AI guess — NASA provided funding via grant; not stated as licensor but listed as funder. Verify before flipping to _state: reviewed.


Multi-temporal Harmonized Landsat-Sentinel (HLS) imagery dataset covering 3,854 chips across the Contiguous United States for 2022, each chip containing 18 spectral bands (6 bands × 3 seasonal time steps) at 30m resolution over a 224×224 pixel area. Pixel-level labels are derived from USDA's Crop Data Layer (CDL) across 13 land cover and crop type classes. Intended for training geospatial image segmentation machine learning models.

agriculture crop-classification land-use remote-sensing multi-temporal machine-learning harmonized-landsat-sentinel landsat sentinel-2

Dataset files (7708 x .tif)

Source Cooperative product page

## README.md
# Multi-Temporal Crop Classification with HLS Imagery across CONUS

## Dataset Description

### Dataset Summary

This dataset contains temporal Harmonized Landsat-Sentinel imagery of diverse land cover and crop type classes across the Contiguous United States for the year 2022. The target labels are derived from USDA's Crop Data Layer (CDL). It's primary purpose is for training segmentation geospatial machine learning models.

### Dataset Structure

## HLS Scenes
Each GeoTIFF file located under `hls/` covers a 224 x 224 pixel area at 30m spatial resolution. Each input satellite file contains 18 bands including 6 spectral bands for three time steps stacked together. The file name structure is `chip_XXX_YYY_merged.tif` where `XXX` and `YYY` refer to row and column of a tile grid we imposed on the CDL data layer. Since the dataset is sampled from this country-wide grid not all `XXX`s and `YYY`s are present in the dataset. So you need to load the `train_data.txt` and `validation_data.txt` to get the list of all chips in the dataset.

## CDL Labels
Each GeoTIFF file located under `masks/` covers the same area as the corresponding HLS chips and contains one band with the target classes for each pixel. The file name structure is `chip_XXX_YYY.mask.tif`.

## Band Order
In each HLS input GeoTIFF the following bands are repeated three times for three observations throughout the growing season:
| Channel |  Name  |  HLS   S30 Band number |
|---------|--------|------------------------|
| 1       |  Blue  |  B02                   |
| 2       |  Green |  B03                   |
| 3       |  Red   |  B04                   |
| 4       |  NIR   |  B8A                   |
| 5       |  SW 1  |  B11                   |
| 6       |  SW 2  |  B12                   |

Masks are a single band with the following values:
| Value    | Class Name            |
|----------|-----------------------|
| 0        |  No Data              |
| 1        |  Natural Vegetation   |
| 2        |  Forest               |
| 3        |  Corn                 |
| 4        |  Soybeans             |
| 5        |  Wetlands             |
| 6        |  Developed/Barren     |
| 7        |  Open Water           |
| 8        |  Winter Wheat         |
| 9        |  Alfalfa              |
| 10       |  Fallow/Idle Cropland |
| 11       |  Cotton               |
| 12       |  Sorghum              |
| 13       |  Other                |

## Data Splits
The 3,854 chips have been randomly split into training (80%) and validation (20%) with corresponding ids recorded in cvs files `train_data.txt` and `validation_data.txt`.

## Dataset Creation
### Query and Scene Selection
First, a set of 5,000 chips was defined based on samples from the USDA CDL to ensure a representative sampling across the CONUS. Next, for each chip, the corresponding HLS S30 scenes between March and September 2022 were queried, and scenes with low cloud cover were retrieved. Then, three scenes are selected among the low cloudy scenes to ensure a scene from early in the season, one in the middle, and one toward the end. The three final scenes were then reprojected to CDL's projection grid (`EPSG:5070`) using bilinear interpolation.

### Chip Generation
In the final step, the three scenes for each chip were clipped to the bounding box of the chip, and 18 spectral bands were stacked together. In addition, a quality control was applied to each chip using the `Fmask` layer of the HLS dataset. Any chip containing clouds, cloud shadow, adjacent to cloud or missing values were discarded. This resulted in 3,854 chips.

### License and Citation
This dataset is published under a CC-BY-4.0 license. If you find this dataset useful for your application, you can cite it as following:

```
@misc{hls-multi-temporal-crop-classification,
    author = {Cecil, Michael and Kordi, Fatemeh and Li, Hanxi (Steve) and Khallaghi, Sam and Alemohammad, Hamed},
    doi    = {10.57967/hf/0955},
    month  = aug,
    title  = {{HLS Multi Temporal Crop Classification}},
    year   = {2023}
}
```
### Contact
For any questions about the dataset, you can contact [Dr. Hamed Alemohammad](mailto:halemohammad@clarku.edu).

### Funding
This dataset is generated with funding from a grant awarded to [Clark University Center for Geospatial Analytics (CGA)](https://github.com/ClarkCGA) by NASA.

providers: role [producer] for 'USDA National Agricultural Statistics Service (NASS)' is an AI guess — Produces the Crop Data Layer (CDL) used as pixel-level label source; not stated as co-producer of this derived dataset. Verify before flipping to _state: reviewed.


Clay v1 Embeddings for California NAIP

No Description Provided


Clay Model v0 Embeddings

Semantic embeddings generated by the Clay Foundation Model v0 for all Earth Observation imagery used in its training corpus. Each record pairs a dense float vector embedding with a geographic geometry and acquisition date, covering global satellite tiles. Clay is a Vision Transformer trained via Self-Supervised Learning (Masked Autoencoder) on geospatial and temporal EO data.

remote-sensing earth-observation embeddings machine-learning foundation-model satellite-imagery

01WCN_20190518_20231021_v001.gpq

[geometry] OGC:CRS84 polygon footprint of the EO tile

01WCP_20170701_20210603_v001.gpq

01WFP_20200430_20200430_v001.gpq

03VXL_20210424_20210424_v001.gpq

03WVN_20220610_20220610_v001.gpq

60WVV_20170325_20210327_v001.gpq

60WWB_20170420_20210330_v001.gpq

60WWT_20200421_20230208_v001.gpq

60WWU_20190711_20230518_v001.gpq

60WWV_20190424_20210419_v001.gpq

Source Cooperative product page

## README.md
# Clay Foundation Model

## An open source AI model for Earth

Clay is a foundational model of Earth. It uses a Vision Transformer architecture adapted
to understand geospatial and temporal relations on Earth Observation data. The model is
trained via Self-supervised learning (SSL) using a Masked Autoencoder (MAE) method.

The Clay model can be used in three main ways:
- Generate semantic embeddings for any location and time.
- Fine-tune the model for downstream tasks such as classification, regression, and generative tasks.
- Use the model as a backbone for other models.

## Where is what

- Our **website** is [madewithclay.org](https://madewithclay.org).
- The Clay model **code** lives on [Github](https://github.com/Clay-foundation/model).
  License: [Apache-w.0](https://github.com/Clay-foundation/model/blob/main/LICENSE).
  The latest release is [v0.0.1](https://github.com/Clay-foundation/model/releases/tag/v0.0.1)
- The Clay model **weights**  on [Hugging Face](https://huggingface.co/made-with-clay/Clay/).
  License: [OpenRAIL-M](https://github.com/Clay-foundation/model/blob/main/LICENSE-MODEL.md).
- The Clay **documentation** [lives on this site](https://clay-foundation.github.io/model/index.html).
  License: [CC-BY](http://creativecommons.org/licenses/by/4.0/).
- We maintain a set of **embeddings** on [Source Cooperative](https://beta.source.coop/repositories/brunosan/clay-model-v0-embeddings/description/).
  License: [ODC-BY](https://opendatacommons.org/licenses/by/).

CLAY is a fiscal sponsored project of the 501c3 non-profit
[Radiant Earth Foundation](https://www.radiant.earth).

providers: role [licensor] for 'Radiant Earth Foundation' is an AI guess — Fiscal sponsor and 501c3 host of Clay; inferred as licensor for the embeddings dataset. Verify before flipping to _state: reviewed.

extent.spatial.bbox: omitted; spatial reference is categorical via [01WCN_20190518_20231021_v001.geometry (geometry), 01WCP_20170701_20210603_v001.geometry (geometry), 01WFP_20200430_20200430_v001.geometry (geometry), 03VXL_20210424_20210424_v001.geometry (geometry), 03WVN_20220610_20220610_v001.geometry (geometry), 60WVV_20170325_20210327_v001.geometry (geometry), 60WWB_20170420_20210330_v001.geometry (geometry), 60WWT_20200421_20230208_v001.geometry (geometry), 60WWU_20190711_20230518_v001.geometry (geometry), 60WWV_20190424_20210419_v001.geometry (geometry)]. Add bbox once a placename → bbox gazetteer is available.


NAIP data embedded with Clay v1.5 (rev2)

Clay v1.5 vector embeddings for the complete NAIP (National Agriculture Imagery Program) archive, covering US aerial imagery from 2010 through 2021. Each 256×256-pixel NAIP tile is represented as a 1024-dimensional class embedding generated by the Clay v1.5 geospatial foundation model from 4-band (RGB+NIR) GeoTIFF inputs. Data is stored in Parquet files mirroring the NAIP folder structure, with one geometry and one embedding array column per tile.

remote-sensing aerial-imagery embeddings machine-learning foundation-model naip

m_3008501_ne_16_1_20110815.parquet

[geometry] Tile footprint polygon for the 256x256 NAIP tile

m_3008501_nw_16_1_20110815.parquet

m_3008502_ne_16_1_20110815.parquet

m_3008502_nw_16_1_20110815.parquet

m_3008503_ne_16_1_20110815.parquet

m_4511062_sw_12_060_20190805.parquet

m_4511063_se_12_060_20190821.parquet

m_4511064_se_12_060_20190821.parquet

m_4511064_sw_12_060_20190821.parquet

m_4511164_se_12_060_20190828.parquet

Source Cooperative product page

## README.md
This repository contains all of [NAIP on AWS](https://registry.opendata.aws/naip/) data embedded with [Clay v1.5](https://huggingface.co/made-with-clay/Clay).

Source: This data has been created by [Development Seed](https://developmentseed.org/) under contract with [Clay](https://madewithclay.org/) and comptute support by [AWS](https://aws.amazon.com/).

Contact: For feedback and questions, please file a ticket on the [model repo of Clay](https://github.com/Clay-foundation/model/issues) or email us at contact@madewithclay.org

Data Source: [NAIP on AWS `naip-analytic`](https://registry.opendata.aws/naip/):
* Includes NAIP imagery from 2010 through 2021.
* The data in this bucket is Original Imagery in Geotiff format 4-band (RGB + NIR)

Model source: Embeddings generated from inference with Clay v1.5:
* Embeddings have `1024` dimensions and correspond to the "class" embedding that is used alongside the patch embeddings at the end of the encoder.
* Each tile is split into tiles of size 256x256, and the attention patch size inside Clay v1.5 is 8x8 px.
* Inference run was done on November '24 on AWS using g4 and g6 EC2, for roughly 60k GPU hours, and costing around $65k.

Embeddings License: Clay [CC-By](https://creativecommons.org/licenses/by/4.0/)

Format:
* Folder structure follows the same as NAIP folder structure.
* File format is parquet, with two columns: `geometry` and `embeddings`

Usage example

```python
import duckdb
path = "https://data.source.coop/clay/clay-v1-5-naip-2/az/2015/100cm/rgbir_cog/32110/m_3211002_se_12_1_20150530.parquet"
d = duckdb.read_parquet(path)
df = d.to_df()
df.head()
```

providers: role [producer] for 'USDA Farm Service Agency' is an AI guess — Original NAIP imagery source; inferred from NAIP program ownership, not named explicitly in README. Verify before flipping to _state: reviewed.

extent.spatial.bbox: omitted; spatial reference is categorical via [m_3008501_ne_16_1_20110815.geometry (geometry), m_3008501_nw_16_1_20110815.geometry (geometry), m_3008502_ne_16_1_20110815.geometry (geometry), m_3008502_nw_16_1_20110815.geometry (geometry), m_3008503_ne_16_1_20110815.geometry (geometry), m_4511062_sw_12_060_20190805.geometry (geometry), m_4511063_se_12_060_20190821.geometry (geometry), m_4511064_se_12_060_20190821.geometry (geometry), m_4511064_sw_12_060_20190821.geometry (geometry), m_4511164_se_12_060_20190828.geometry (geometry)]. Add bbox once a placename → bbox gazetteer is available.


NAIP data embedded with Clay v1.5

This repository has the embeddings using Clay vq.5 for the entire NAIP archive. See details on README.md  License: Clay [CC-By](https://creativecommons.org/licenses/by/4.0/)


Sentinel-2 data embedded with Clay v1.5

This product is deprecated, please use https://source.coop/clay/lgnd-clay-v1-5-sentinel-2-l2a instead.


LGND Clay Embeddings - Sentinel2

A global dataset of Clay v1.5 1024-dimensional float32 embeddings derived from Sentinel-2 L2A imagery, created by lgnd.ai. It covers approximately 99.9% of the globe across two time steps (June 2024 and June 2025), organized into scene-level and aggregated monthly products keyed to 2560-meter MajorTOM grid cells. Data are stored as hive-partitioned GeoParquet files totaling ~467 GB across ~127 million embeddings.

remote-sensing sentinel-2 embeddings machine-learning time-series earth-observation geoparquet

[spatial_index] Level-2 geohash hive-partition key for the aggregated product

bbox

STRUCT(xmin DOUBLE, ymin DOUBLE, xmax DOUBLE, ymax DOUBLE)

[geometry] Bounding box struct (xmin, ymin, xmax, ymax) for spatial lookup optimization

geometry

[geometry] Polygon footprint of the chip in WGS84 (EPSG:4326)

gzd

[spatial_index] MGRS Grid Zone Designator hive-partition key for the scene product

Source Cooperative product page

## README.md
# LGND Clay Embeddings - Sentinel2

This repository contains a global dataset of [Clay v1.5](https://madewithclay.org/) embeddings for Sentinel2, created by [lgnd.ai](https://lgnd.ai/).  Each embedding is a 1024-dimensional float32 vector associated with a 2560-meter MajorTOM grid cell.  All imagery was sourced from the [`sentinel-cogs`](https://registry.opendata.aws/sentinel-2-l2a-cogs/) bucket on AWS.

The dataset currently has near complete (~99.9%) global coverage across two time steps - June 2024 and June 2025, and is broken into two distinct products:
- `scene` - embeddings for individual Sentinel2 scenes.
- `aggregated` - an aggregation of scene-level embeddings into a normalized time series of embeddings.

## Scene-Level Data

The scene-level product contains embeddings for each individual Sentinel-2 scene.  For each MGRS tile, we embed the minimum number of scenes required to provide maximum spatial coverage while minimizing cloud cover.  Some MGRS tiles only require one scene, while others require many more.  The embeddings are available in the `scene` directory partitioned by Grid Zone Designator (the first 3 digits of the MGRS tile), year, and month.  For example:

```shell
❯ aws s3 ls s3://us-west-2.opendata.source.coop/clay/lgnd-clay-v1-5-sentinel-2-l2a/scene/gzd=11U/year=2024/month=06/ --summarize --human-readable

2026-01-25 21:44:58  183.1 MiB data_01c1fab1-0004-5b20-0009-b72e01d10bc6_215_4_0.parquet
2026-01-25 21:46:57  141.8 MiB data_01c1fab1-0004-5b20-0009-b72e01d10bc6_215_4_1.parquet

Total Objects: 2
   Total Size: 324.9 MiB
```

The scene-level product is best when analyzing individual Sentinel2 scenes, or when you need the highest possible temporal resolution.  This product contains ~77.9 million embeddings and takes up ~285GB on disk.

## Aggregated Data

The aggregated product provides a normalized time series of embeddings per chip by selecting one representative embedding per month for each MajorTOM cell.  This helps remove redundancy from overlapping scene coverage at the border of MGRS tiles, normalizes observation frequency across regions with varying revisit rates, and simplifies time series analysis by providing consistent monthly snapshots.  The embeddings are available in the `aggregated` directory partitioned by geohash, year, and month.  For example:

```shell
~ ❯ aws s3 ls s3://us-west-2.opendata.source.coop/clay/lgnd-clay-v1-5-sentinel-2-l2a/aggregated/geohash_l2=9y/year=2025/month=06/ --summarize --human-readable

2026-01-26 00:50:31  359.5 MiB data_01c1fab1-0004-5b1c-0009-b72e01d3104e_222_0_0.parquet

Total Objects: 1
   Total Size: 359.5 MiB
```

The aggregated product is easier to use than the scene-level product across large spatial extents, and is best for time series analysis or when constent temporal sampling is required.  This product contains  ~49.7 million embeddings and takes up ~182GB on disk.

## Data Model

Both data products follow the same data model, described below:

| Column | Type | Description |
|--------|------|-------------|
| `chips_id` | string | Spatio-temporally unique identifier for each embedding |
| `cell_id` | string | Spatially unique identier for the MajorTOM cell |
| `rasters_id` | string | Unique, internal LGND identifier for the source raster |
| `stac_item_id` | string | STAC item ID of the source Sentinel-2 scene, from [Earth Search](https://element84.com/earth-search/) |
| `collection` | string | STAC collection name (e.g., `sentinel-2-l2a`), also from Earth Search |
| `datetime` | timestamp | Acquisition timestamp of the source imagery |
| `embedding` | list[float32] | 1024-dimensional Clay v1.5 embedding vector |
| `bbox` | struct | Bounding box with `xmin`, `ymin`, `xmax`, `ymax`, for geoparquet reader optimizations |
| `geometry` | polygon (wkb) | Polygon geometry of the chip footprint |

## Data Format

Both data products are provided as hive-partitioned geoparquet files, following [geoparquet best practices](https://geoparquet.io/concepts/best-practices/):
- Geoparquet format version 1.1.
- ZSTD compression (level 15).
- ~128 MB row groups where possible.
- Spatial-temporal partitioning.
- Spatial ordering with a hilbert curve.
- Bbox column with covering metadata for faster spatial lookups.

## License

This product is licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).

## Known Issues
There are several known data quality issues:
- There are a few small gaps in coverage due to missing remote sensing data or edge cases in our scene selection algorithm, we estimate this dataset is about ~99.7% complete.

## Future work
In the coming months, the LGND team plans to fix these data gaps and push embeddings for more time steps beyond the two originally provided.

providers: role [processor] for 'Made With Clay' is an AI guess — Clay v1.5 is the foundation model used to generate the embeddings; named as model source, not explicitly as a dataset provider. Verify before flipping to _state: reviewed.

extent.spatial.bbox: omitted; spatial reference is categorical via [data_01c1fab1-0004-5b1c-0009-b72e01d3104e_222_0_0.geometry (geometry), data_01c1fab1-0004-5b1c-0009-b72e01d3104e_222_0_0.bbox (geometry), data_01c1fab1-0004-5b1c-0009-b72e01d3104e_222_0_0.geohash_l2 (spatial_index), data_01c1fab1-0004-5b20-0009-b72e01d10bc6_315_2_0.geometry (geometry), data_01c1fab1-0004-5b20-0009-b72e01d10bc6_315_2_0.bbox (geometry), data_01c1fab1-0004-5b20-0009-b72e01d10bc6_315_2_0.gzd (spatial_index)]. Add bbox once a placename → bbox gazetteer is available.


LGND Embeddings

Clay foundation model embeddings of global Sentinel-2 L2A satellite imagery, created and maintained by lgnd.ai. Each embedding is a 256- or 1024-dimensional vector representation of a fixed-size Major TOM grid cell chip, covering approximately 15.2 billion embeddings globally from January 2017 to April 2026. Intended for similarity search, change detection, and downstream geospatial machine learning tasks.

embeddings machine-learning foundation-model sentinel-2 satellite-imagery earth-observation remote-sensing geoparquet

monthly-aggregated

bbox

[geometry] Bounding box struct (xmin, ymin, xmax, ymax) of the chip grid cell

geometry

[geometry] OGC:CRS84 polygon geometry of the chip grid cell

geohash

[spatial_index] Geohash spatial index encoding the chip location

12 files sampled for this table

stac.parquet

bbox

[geometry] Bounding box struct (xmin, ymin, xmax, ymax) of the Sentinel-2 scene footprint

assets

STRUCT(blue STRUCT(href VARCHAR), green STRUCT(href VARCHAR), nir STRUCT(href VARCHAR), nir08 STRUCT(href VARCHAR), red STRUCT(href VARCHAR), rededge1 STRUCT(href VARCHAR), rededge2 STRUCT(href VARCHAR), rededge3 STRUCT(href VARCHAR), swir16 STRUCT(href VARCHAR), swir22 STRUCT(href VARCHAR), aot STRUCT(href VARCHAR), coastal STRUCT(href VARCHAR), granule_metadata STRUCT(href VARCHAR), nir09 STRUCT(href VARCHAR), scl STRUCT(href VARCHAR), thumbnail STRUCT(href VARCHAR), tileinfo_metadata STRUCT(href VARCHAR), visual STRUCT(href VARCHAR), wvp STRUCT(href VARCHAR), "aot-jp2" STRUCT(href VARCHAR), "blue-jp2" STRUCT(href VARCHAR), "coastal-jp2" STRUCT(href VARCHAR), "green-jp2" STRUCT(href VARCHAR), "nir-jp2" STRUCT(href VARCHAR), "nir08-jp2" STRUCT(href VARCHAR), "nir09-jp2" STRUCT(href VARCHAR), "red-jp2" STRUCT(href VARCHAR), "rededge1-jp2" STRUCT(href VARCHAR), "rededge2-jp2" STRUCT(href VARCHAR), "rededge3-jp2" STRUCT(href VARCHAR), "scl-jp2" STRUCT(href VARCHAR), "swir16-jp2" STRUCT(href VARCHAR), "swir22-jp2" STRUCT(href VARCHAR), "visual-jp2" STRUCT(href VARCHAR), "wvp-jp2" STRUCT(href VARCHAR))

geometry

[geometry] OGC:CRS84 polygon geometry of the Sentinel-2 scene footprint

Source Cooperative product page

## README.md

This repository contains [Clay](https://clay.earth/) foundation model embeddings created and maintained by [lgnd.ai](https://lgnd.ai/). Each embedding is a multi-dimensional vector representation of a satellite image chip (a fixed-size [Major TOM](https://huggingface.co/Major-TOM) grid cell), useful for similarity search, change detection, and downstream geospatial ML tasks.

The data is broken into several top-level datasets, click into each for more details:
- **scene-level (COMING SOON)** - one embedding per chip per scene acquisition, the full-resolution dataset.
- **[monthly-aggregated](https://source.coop/clay/lgnd-embeddings/monthly-aggregated)** - deduplicated to one embedding per chip per month, derived from the scene-level dataset.
- **[rasters](https://source.coop/clay/lgnd-embeddings/rasters)** - STAC GeoParquet catalog of the source imagery: scene footprints, acquisition times, and COG asset URLs.

## Coverage

| Collection | Coverage | Temporal Range | Scenes | Embeddings | Source Imagery |
|---|---|---|---|---|---|
| `sentinel-2-l2a` | Global | Jan 2017 - April 2026 | ~2.8M | ~15.2B | ~2.2 PB |

## Available Configurations

| Model Version | Collection | Chip Size | Embedding Dims |
|---|---|---|---|
| `v1.5` | `sentinel-2-l2a` | `1280m` | `256` |
| `v1.5` | `sentinel-2-l2a` | `2560m` | `1024` |

## License

All products are licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).

providers: role [processor] for 'Clay' is an AI guess — Clay provides the foundation model used to generate the embeddings; listed as processor because they supply the model architecture rather than directly producing or hosting the dataset.. Verify before flipping to _state: reviewed.

extent.spatial.bbox: omitted; spatial reference is categorical via [monthly-aggregated.geometry (geometry), monthly-aggregated.bbox (geometry), monthly-aggregated.geohash (spatial_index), rasters.geometry (geometry), rasters.bbox (geometry)]. Add bbox once the placename → bbox gazetteer covers it.


Google Satellite Embeddings v1

Google Satellite Embeddings v1 (https://developers.google.com/earth-engine/datasets/catalog/GOOGLE_SATELLITE_EMBEDDING_V1_ANNUAL#description).

Aggregated to 250m x 250m tiles for ease of search.

Includes parquet files by MGRS ID, and fully built duckdb tables and FAISS indexes.


3Dclouds

This repository hosts the AI-ready datasets for the paper "Global 3D Reconstruction of Clouds & Tropical Cyclones." It contains paired 2D geostationary imagery (GOES, MSG, Himawari) and 3D vertical profiles (CloudSat/CALIPSO) used to train ML models for 3D cloud reconstruction, with a dedicated dataset for tropical cyclones.


methaneset

methaneset


Model-based results from 2,760 simulated Direct Ocean Removal (DOR) experiments using the Community Earth System Model (CESM), covering 690 global ocean regions and four intervention seasons (January, April, July, October). Each 15-year experiment captures the temporal evolution of dissolved inorganic carbon (DIC) removal on the ocean biogeochemical carbon system, stored as monthly-average netCDF files alongside counterfactual ocean states. Also includes globally-integrated DOR efficiency maps derived from all simulations.

oceanography climate time-series ocean-carbon biogeochemistry climate-modeling

Dataset files (496802 x .nc)

Source Cooperative product page

## README.md
# Ocean Carbon Dioxide Removal Atlas: Global Direct Ocean Removal (DOR) Map

## Description

The files in this repository contain the model-based results from a large number of simulated Direct Ocean Removal (DOR) experiments using the Community Earth System Model ([CESM](https://www.cesm.ucar.edu/models)).  These experiments are intended to explore the temporal and spatial impact of regionally confined carbon removal on the biogeochemical carbon system in the global ocean.

To produce these model-based experiments, the CESM model was run in what is called a “forced ocean sea ice” configuration, wherein an independent dataset of historical atmospheric conditions (including the observed trend in atmospheric CO2) is used to drive the circulation and state of the CESM global ocean model. The details of these simulations identical to those of the [Global Ocean Alkalinity Enhancement Efficiency Map](https://source.coop/repositories/cworthy/oae-efficiency-atlas) — with the sole exception that DOR forcing was applied instead of alkalinity addition.

This repository contains:

1. A series of "DOR perturbation experiments," which span a 15 period beginning in the model simulated year 1999; and the “counterfactual” ocean state for these experiments, where there was no DOR perturbation to the biogeochemical system

2. Globally-integrated DOR efficiencies for each point on the model’s horizontal grid, defined as the cumulative integral in time of CO2 uptake from the atmosphere divided by the amount of alkalinity added to the ocean.

The following elements are **not included** in this repository because they are shared with the OAE dataset and are not duplicated here. Interested users can access them via the OAE dataset, where they were originally published:

1. A nearly 200-year “control” integration used for historical and physical context, including the response to the observed anthropogenic CO₂ trend. Data available at [`data/control/*.nc`](https://source.coop/cworthy/oae-efficiency-atlas/data/control)

2. Region masks identifying the 690 global ocean regions where carbon removal was applied. Available [here](https://source.coop/cworthy/oae-efficiency-atlas/data/polygon_masks.nc)

## Structure

This repository contains:

1. 496,800 netCDF files output from 2760 CESM simulations. Each simulated DOR perturbation experiment represents the time-evolving outcome of removing dissolved inorganic carbon (DIC) from a particular region of the ocean (the "polygon") over a particular time period (the "intervention month"), and integrating an ocean physical-biogeochemical model forwards for 15 years. These results are stored in the `data/experiments/` directory, following the pattern `{polygon_id}/{intervention_month}/smyle.cdr-atlas-v0.glb-dor.{polygon_id}-{intervention_year}-{intervention_month}.pop.h.{year}-{month}.nc`. These files contain the globally complete set of biogeochemical variables that are impacted by the DOR perturbation. The data is saved as monthly averages, with one file per month, for a total of 180 files per 15 year simulation. The meaning of the identifiers are:
   - `{polygon_id}` - the unique integer ID of each polygon, ranging from 000 to 690.
   - `{intervention_year}` - indicates the calendar year during which the DOR intervention occurred. Presently all the data are from simulations beginning in 1999.
   - `{intervention_month}` - the calendar month during which the DIC removal occurred. One of 01, 04, 07, 10.
   - `{year}` - the model year for which data is contained in this particular file. Model year 0347 corresponds to the calendar year 1999.
   - `{month}` - the simulation calendar month for which data is contained in this particular file. Ranges from 01 to 12. Note that the time variable in these files contains values that correspond to the timestamp at the end of each month.
These netCDF files also contain variables representing the counterfactual condition for each of the DOR perturbation experiments. The counterfactual simulation is defined as the time-evolving biogeochemical state of the ocean in the absence of the DIC removal. These variables are stored using the same naming convention as above, but with a suffix `_ALT_CO2` (e.g. `’DIC’` contains the experimental values and `’DIC_ALT_CO2’` contains the counterfactual values).

2. Integrated efficiency curves in the file `data/DOR_efficiency_maps_alltime.nc`. These contain the globally-integrated DOR efficiencies for each point on the model’s horizontal grid (rather than being defined for each spatial polygon). The efficiency curves are defined as the cumulative integral in time of CO2 uptake from the atmosphere divided by the amount of DIC removed from the ocean. This file includes a variable `DOR_efficiency` defined over the dimensions `season`, `N_month`, `nlat`, and `nlon`. `season` is the month of the DIC removal: ['January', 'April', 'July', 'October']; `N_month` is the number of months elapsed since the DIC removal.

## Future Plans

We eventually aim to make this entire dataset accessible as a single Zarr Store via virtual chunk reference files.

## How to access

### BROWSE raw netCDF files

You can find any particular netCDF file of interest by navigating through this repository after clicking `BROWSE` on the left. You can then click `DOWNLOAD` to download the file directly (this does not require AWS credentials).

### ACCESS DATA and open specific netCDF files with xarray

After signing in to Source Cooperative, you can click ACCESS DATA to generate temporary AWS credentials that will allow you to download the data via URL.
You can download the entire dataset (Not recommended due to size) following the “Usage Examples” provided.
You can also access specific files from a URL pointing to the bucket using [xarray](https://xarray.dev/) and [s3fs](https://s3fs.readthedocs.io/en/latest/), which you will need to have installed (e.g. from conda-forge). You will also need to have [h5netcdf](https://github.com/h5netcdf/h5netcdf) installed. You then need to run the following commands from an interactive python shell (e.g. in a [jupyter notebook](https://jupyter.org/)).

```python
import xarray as xr
import s3fs

# set up anonymous S3 access
fs = s3fs.S3FileSystem(anon=True)
data_dir = 's3://us-west-2.opendata.source.coop/cworthy/dor-efficiency-atlas/data'

polygon_id = "000"
intervention_month = "01"
intervention_year = "1999"
year = "0347"
month = "01"

data_file = (
	f"experiments/{polygon_id}/{intervention_month}/"
	f"smyle.cdr-atlas-v0.glb-dor.{polygon_id}-{intervention_year}-{intervention_month}.pop.h.{year}-{month}.nc"
)
data_url = f"{data_dir}/{data_file}"

ds_data = xr.open_dataset(fs.open(data_url), engine='h5netcdf')

```

## References

A paper documenting the generation of these data and analysis of the results is in preparation.

The related OAE Efficiency Map data are documented in the following publication:
M. Zhou, M. Tyka, D. Ho, E. Yankovsky, S. Bachman, T. Nicholas, A. Karspeck, M. Long: Mapping the global variation in the efficiency of ocean alkalinity enhancement for carbon dioxide removal, 2024, Nature Climate Change, in press, doi: 10.1038/s41558-024-02179-9. [see preprint](https://www.researchsquare.com/article/rs-4124909/v1)

## Contact Us

We are interested in how you are using this data! Please reach out to [data@cworthy.org](data@cworthy.org) with any questions. We’d also appreciate information on your analysis goals and whether the data format and access patterns adequately support your use cases.

## License

[Open Data Commons Attribution License (ODC-By) v1.0](https://opendatacommons.org/licenses/by/1-0/) , packaged with the data in the `LICENSE.txt` file.


Model-based results from 2,760 simulated Ocean Alkalinity Enhancement (OAE) experiments using the Community Earth System Model (CESM), exploring the temporal and spatial impact of regional alkalinity addition on the global ocean biogeochemical carbon system across 690 ocean regions. The dataset includes ~497,000 monthly netCDF output files from perturbation and control simulations, globally-integrated OAE efficiency maps, and spatial polygon masks covering the entire global ocean.

ocean-alkalinity-enhancement ocean-biogeochemistry oceanography biodiversity climate-modeling climate marine-carbon-cycle cesm cdr

Dataset files (497250 x .nc)

Source Cooperative product page

## README.md
# Ocean Carbon Dioxide Removal Atlas: Global Ocean Alkalinity Enhancement Efficiency Map

## Description

The files in this repository contain the model-based results from a large number of simulated Ocean Alkalinity Enhancement (OAE) experiments using the Community Earth System Model ([CESM](https://www.cesm.ucar.edu/models)).  These experiments are intended to explore the temporal and spatial impact of regionally confined alkalinity addition on the biogeochemical carbon system in the global ocean. To produce these model-based experiments, the CESM model was run in what is called a “forced ocean sea ice” configuration, wherein an independent dataset of historical atmospheric conditions (including the observed trend in atmospheric CO2) is used to drive the circulation and state of the CESM global ocean model. The details of these simulations are available in Zhou et al. (2024). This repository contains 1) a nearly 200 year “control” integration that can be used for historical and physical context for the experiment including the impact of the observed anthropogenic trend in atmospheric CO2 on the ocean; and 2) A series of “OAE perturbation experiments,” which span a 15 period beginning in the model simulated year 1999; and the “counterfactual” ocean state for these experiments, where there was no OAE perturbation to the biogeochemical system; 3) Globally-integrated OAE efficiencies for each point on the model’s horizontal grid, defined as the cumulative integral in time of CO2 uptake from the atmosphere divided by the amount of alkalinity added to the ocean; 4) region-masks that indicate the location of the 690 regions of the global ocean from which alkalinity releases were simulated.

## Structure

This repository contains:
1. 496,800 netCDF files output from 2760 CESM simulations. Each simulated  OAE perturbation experiment represents the time-evolving outcome of injecting alkalinity into a particular region of the ocean (the "polygon") over a particular time period (the "injection month"), and integrating an ocean physical-biogeochemical model forwards for 15 years. These results are stored in the `data/experiments/` directory, following the pattern `{polygon_id}/{injection_month}/alk-forcing.{polygon_id}-{injection_year}-{injection_month}.pop.h.{year}-{month}.nc`. These files contain the globally complete set of biogeochemical variables that are impacted by the applied alkalinity perturbation. The data is saved as monthly averages, with one file per month, for a total of 180 files per 15 year simulation. The meaning of the identifiers are:
    - `{polygon_id}` - the unique integer ID of each polygon, ranging from 000 to 690.
    - `{injection_year}` - indicates the calendar year during which the alkalinity injection occurred. Presently all the data are from simulations beginning in 1999.
    - `{injection_month}` - the calendar month during which the alkalinity injection occurred. One of 01, 04, 07, 10.
    - `{year}` - the model year for which data is contained in this particular file. Model year 0347 corresponds to the calendar year 1999.
    - `{month}` - the simulation calendar month for which data is contained in this particular file. Ranges from 01 to 12. Note that the time variable in these files contains values that correspond to the timestamp at the end of each month.
These netCDF files also contain variables representing the counterfactual condition for each of the OAE perturbation experiments. The counterfactual simulation is defined as the time-evolving biogeochemical state of the ocean in the absence of the alkalinity injection. These variables are stored using the same naming convention as above, but with a suffix `_ALT_CO2` (e.g. `’ALK’` contains the experimental values and `’ALK_ALT_CO2’` contains the counterfactual values).
2. NetCDF files for the dataset from the control simulation. The control simulation is a so-called forced-ocean-sea-ice CESM integration from 1850 to 2021 and each of the OAE perturbation experiments was branched off this simulation in the year 1999. Since there is no feedback between alkalinity addition and the physical dynamics in the model, data from the control integration can be used to provide access to physical variables (e.g., ocean velocity, mixed layer depth, etc.) that were omitted from the experiments to save data volume. These are stored in various files under `data/control/*.nc`, with one file per data variable.
3. Integrated efficiency curves in the file `data/OAE_efficiency_maps_alltime.nc`. These contain the globally-integrated OAE efficiencies for each point on the model’s horizontal grid (rather than being defined for each spatial polygon). The efficiency curves are defined as the cumulative integral in time of CO2 uptake from the atmosphere divided by the amount of alkalinity added to the ocean. This file includes a variable `OAE_efficiency` defined over the dimensions `season`, `N_month`, `nlat`, and `nlon`. `season` is the month of the alkalinity release: ['January', 'April', 'July', 'October']; `N_month` is the number of months elapsed since the alkalinity release.
4. Integrated efficiency curves in the file `data/oae_eff_curves_global.nc`. These contain the globally-integrated OAE efficiencies for each spatial polygon.
5. Masks for the spatial locations of each polygon_id, stored in the single netCDF4 file `data/polygon_masks.nc`. All 690 polygons together tile the entire global ocean.

## Future Plans
We eventually aim to make this entire dataset accessible as a single Zarr Store via virtual chunk reference files.

## How to access

### BROWSE raw netCDF files

You can find any particular netCDF file of interest by navigating through this repository after clicking `BROWSE` on the left. You can then click `DOWNLOAD` to download the file directly (this does not require AWS credentials).

### ACCESS DATA and open specific netCDF files with xarray

After signing in to Source Cooperative, you can click ACCESS DATA to generate temporary AWS credentials that will allow you to download the data via URL.
You can download the entire dataset (Not recommended due to size) following the “Usage Examples” provided.
You can also access specific files from a URL pointing to the bucket using [xarray](https://xarray.dev/) and [s3fs](https://s3fs.readthedocs.io/en/latest/), which you will need to have installed (e.g. from conda-forge). You will also need to have [h5netcdf](https://github.com/h5netcdf/h5netcdf) installed. You then need to run the following commands from an interactive python shell (e.g. in a [jupyter notebook](https://jupyter.org/)).

```python
import xarray as xr
import s3fs

!<copy/pasted temporary credentials>

fs = s3fs.S3FileSystem(anon=True)
data_dir = 's3://us-west-2.opendata.source.coop/cworthy/oae-efficiency-atlas/data/'
data_file = experiments/{polygon_id}/{injection_month}/alk-forcing.{polygon_id}-{injection_year}-{injection_month}.pop.h.{year}-{month}.nc
data_url = data_dir + data_file
ds_data = xr.open_dataset(fs.open(data_url), engine='h5netcdf')
```

### Open entire dataset as Zarr with xarray
_Note: Access as Zarr is coming soon!_

## References
The generation of this data and analysis of the results is documented in the following publication; to cite this dataset please cite this paper:
M. Zhou, M. Tyka, D. Ho, E. Yankovsky, S. Bachman, T. Nicholas, A. Karspeck, M. Long: Mapping the global variation in the efficiency of ocean alkalinity enhancement for carbon dioxide removal, 2024, Nature Climate Change, in press, doi: 10.1038/s41558-024-02179-9. [see preprint](https://www.researchsquare.com/article/rs-4124909/v1)

## Contact Us
We are interested in how you are using this data! Please reach out to [data@cworthy.org](data@cworthy.org) with any questions. We’d also appreciate information on your analysis goals and whether the data format and access patterns adequately support your use cases.

## License
[Open Data Commons Attribution License (ODC-By) v1.0](https://opendatacommons.org/licenses/by/1-0/) , packaged with the data in the `LICENSE.txt` file.

providers: role [producer, licensor] for 'C-Worthy' is an AI guess — Identified as producer from S3 path prefix 'cworthy/' and contact email data@cworthy.org; licensor inferred from README authorship and license packaging. Verify before flipping to _state: reviewed.


Data for Canada - Foundation Process Laboratory

This repository hosts systems-ready foundational data for Canada, curated to support modern software and analytical pipelines. By transforming fragmented open data into standardized, high-performance formats, it provides developers and researchers with a reliable baseline for geospatial applications without the need for extensive preprocessing.

Part of the Data for Canada project, this collection focuses on the 'Foundation' theme, essential core layers required for building robust Canadian data infrastructure. For detailed documentation on the processing methodologies and schema, please visit https://www.dataforcanada.org/docs/processes/foundation/.


Data for Canada - Orthoimagery Process Laboratory

This repository provides systems-ready orthoimagery data for Canada, optimized for direct use in cloud-native geospatial applications and analysis pipelines. By standardizing diverse aerial and satellite imagery sources, it offers developers a consistent, high-performance visual baseline without the need for complex pre-processing.

As part of the Data for Canada project, this collection focuses on the 'Orthoimagery' theme, serving as a laboratory for open imagery processing and interoperability. For full documentation on the methodologies used, please visit https://www.dataforcanada.org/docs/processes/orthoimagery/ .


Data for Canada - Elevation Data Packages

A collection of elevation data packages for Canada, comprising over 2,800 GeoTIFF raster files covering parts of eastern Canada (Quebec/Maritime region). The dataset includes ancillary files such as a GeoPackage, CSV, PDF, and JSON, totaling approximately 526 GB.

Dataset files (2858 x .tif)

Source Cooperative product page

license: AI returned 'unknown' (README does not state a license). Verify with publisher before flipping to _state: reviewed.

providers: role [producer, host] for 'Data for Canada' is an AI guess — Assumed producer and host based on API title; no README to confirm. Verify before flipping to _state: reviewed.


Data for Canada - Environment, Climate, and Health Data Packages

Time-series measurements from a network of low-cost IoT Enviro+ environmental monitoring stations distributed across Canada. Each station records particulate matter (PM1, PM2.5, PM10 and particle size distributions), gas concentrations (oxidised, reducing, NH3), temperature, pressure, humidity, and light intensity. A companion asset captures per-station device telemetry (CPU load, memory, WiFi signal, battery) to support quality-control filtering.

air-quality environmental-monitoring meteorology time-series particulate-matter sensor-network

station

[placename] Identifier naming the monitoring station location

23 files sampled for this table

archive/opensensor.space/enviroplus

8 files sampled for this table

Source Cooperative product page

extent.spatial.bbox: omitted; spatial reference is categorical via [archive/opensensor.space/enviroplus.station (placename), archive/opensensor.space/enviroplus-health.station (placename)]. Add bbox once the placename → bbox gazetteer covers it.


Data for Canada - Field Imagery Data Packages


Data for Canada - Foundation Data Packages

Standardized foundation data packages for Canada sourced from Statistics Canada, comprising three temporal snapshots of the National Address Register (June 2024, December 2024, July 2025) and the 2021 Road Network File. Each address record is enriched with a full hierarchy of StatCan geographic identifiers spanning country, province, census division, CSD, CMA, dissemination area, and dissemination block. Assets are distributed in Parquet, FlatGeobuf, and PMTiles formats.

administrative-boundaries census pmtiles national-address-register road-network statistics-canada

v0.1.0-beta

[placename] Province or territory name (English)

[admin_code] ISO province/territory code

[placename] Census subdivision name

[admin_code] StatCan dissemination area unique geographic identifier (DGUID)

[zipcode] Canadian postal code for the address

geom

[geometry] Point geometry in OGC:CRS84 (WGS84)

STRUCT(xmin FLOAT, ymin FLOAT, xmax FLOAT, ymax FLOAT)

4 files sampled for this table

Source Cooperative product page

extent.temporal: None (probe sample) widened to ['2021-01-01T00:00:00Z', '2025-07-01T00:00:00Z'] using README-stated coverage ['2021', '2025-07'].

extent.spatial.bbox: omitted; spatial reference is categorical via [v0.1.0-beta.geom (geometry), v0.1.0-beta.mail_postal_code (zipcode), v0.1.0-beta.pr_iso_code (admin_code), v0.1.0-beta.pr_en_name (placename), v0.1.0-beta.csd_name (placename), v0.1.0-beta.da_dguid (admin_code)]. Add bbox once the placename → bbox gazetteer covers it.