AEF-v1 embeddings Rasteret Collection

A read-ready Rasteret collection derived from the AlphaEarth Foundations (AEF) annual satellite embedding dataset produced by Google and Google DeepMind. It provides a partitioned GeoParquet index and per-tile Cloud Optimized GeoTIFF header metadata for approximately 700,000 annual tiles covering near-global land area, enabling remote spatial and temporal filtering before fetching 64-band embedding vectors from upstream COGs. The dataset stores no pixel data — only Parquet metadata structured for use with Rasteret, DuckDB, Polars, and PyArrow.

remote-sensing embeddings machine-learning earth-observation foundation-model geoparquet global-coverage satellite-imagery

data

[geometry] WGS84 tile geometry (OGC:CRS84) for each Rasteret collection record

assets

STRUCT(A00 STRUCT(band_index BIGINT, href VARCHAR), A01 STRUCT(band_index BIGINT, href VARCHAR), A02 STRUCT(band_index BIGINT, href VARCHAR), A03 STRUCT(band_index BIGINT, href VARCHAR), A04 STRUCT(band_index BIGINT, href VARCHAR), A05 STRUCT(band_index BIGINT, href VARCHAR), A06 STRUCT(band_index BIGINT, href VARCHAR), A07 STRUCT(band_index BIGINT, href VARCHAR), A08 STRUCT(band_index BIGINT, href VARCHAR), A09 STRUCT(band_index BIGINT, href VARCHAR), A10 STRUCT(band_index BIGINT, href VARCHAR), A11 STRUCT(band_index BIGINT, href VARCHAR), A12 STRUCT(band_index BIGINT, href VARCHAR), A13 STRUCT(band_index BIGINT, href VARCHAR), A14 STRUCT(band_index BIGINT, href VARCHAR), A15 STRUCT(band_index BIGINT, href VARCHAR), A16 STRUCT(band_index BIGINT, href VARCHAR), A17 STRUCT(band_index BIGINT, href VARCHAR), A18 STRUCT(band_index BIGINT, href VARCHAR), A19 STRUCT(band_index BIGINT, href VARCHAR), A20 STRUCT(band_index BIGINT, href VARCHAR), A21 STRUCT(band_index BIGINT, href VARCHAR), A22 STRUCT(band_index BIGINT, href VARCHAR), A23 STRUCT(band_index BIGINT, href VARCHAR), A24 STRUCT(band_index BIGINT, href VARCHAR), A25 STRUCT(band_index BIGINT, href VARCHAR), A26 STRUCT(band_index BIGINT, href VARCHAR), A27 STRUCT(band_index BIGINT, href VARCHAR), A28 STRUCT(band_index BIGINT, href VARCHAR), A29 STRUCT(band_index BIGINT, href VARCHAR), A30 STRUCT(band_index BIGINT, href VARCHAR), A31 STRUCT(band_index BIGINT, href VARCHAR), A32 STRUCT(band_index BIGINT, href VARCHAR), A33 STRUCT(band_index BIGINT, href VARCHAR), A34 STRUCT(band_index BIGINT, href VARCHAR), A35 STRUCT(band_index BIGINT, href VARCHAR), A36 STRUCT(band_index BIGINT, href VARCHAR), A37 STRUCT(band_index BIGINT, href VARCHAR), A38 STRUCT(band_index BIGINT, href VARCHAR), A39 STRUCT(band_index BIGINT, href VARCHAR), A40 STRUCT(band_index BIGINT, href VARCHAR), A41 STRUCT(band_index BIGINT, href VARCHAR), A42 STRUCT(band_index BIGINT, href VARCHAR), A43 STRUCT(band_index BIGINT, href VARCHAR), A44 STRUCT(band_index BIGINT, href VARCHAR), A45 STRUCT(band_index BIGINT, href VARCHAR), A46 STRUCT(band_index BIGINT, href VARCHAR), A47 STRUCT(band_index BIGINT, href VARCHAR), A48 STRUCT(band_index BIGINT, href VARCHAR), A49 STRUCT(band_index BIGINT, href VARCHAR), A50 STRUCT(band_index BIGINT, href VARCHAR), A51 STRUCT(band_index BIGINT, href VARCHAR), A52 STRUCT(band_index BIGINT, href VARCHAR), A53 STRUCT(band_index BIGINT, href VARCHAR), A54 STRUCT(band_index BIGINT, href VARCHAR), A55 STRUCT(band_index BIGINT, href VARCHAR), A56 STRUCT(band_index BIGINT, href VARCHAR), A57 STRUCT(band_index BIGINT, href VARCHAR), A58 STRUCT(band_index BIGINT, href VARCHAR), A59 STRUCT(band_index BIGINT, href VARCHAR), A60 STRUCT(band_index BIGINT, href VARCHAR), A61 STRUCT(band_index BIGINT, href VARCHAR), A62 STRUCT(band_index BIGINT, href VARCHAR), A63 STRUCT(band_index BIGINT, href VARCHAR))

bbox

[geometry] WGS84 bounding box struct (xmin/ymin/xmax/ymax) for each tile

STRUCT(image_width INTEGER, image_height INTEGER, tile_width INTEGER, tile_height INTEGER, dtype VARCHAR, "transform" DOUBLE[], predictor INTEGER, "compression" INTEGER, tile_offsets BIGINT[], tile_byte_counts BIGINT[], pixel_scale DOUBLE[], tiepoint DOUBLE[], nodata DOUBLE, samples_per_pixel INTEGER, planar_configuration INTEGER, photometric INTEGER, extra_samples INTEGER[])

8 files sampled for this table

[geometry] UTM-projected bounding box for each tile

[geometry] WGS84 bounding box struct for each index tile

geom

[geometry] WGS84 tile geometry (OGC:CRS84) in the narrow GeoParquet index

location

[placename] Location identifier or name associated with each tile

2 files sampled for this table

Source Cooperative product page

## README.md
# AEF v1 Annual Rasteret Collection

This dataset is a read-ready [Rasteret](https://github.com/terrafloww/rasteret) collection derived from the public AlphaEarth Foundations (AEF) annual index and enriched with Rasteret's per-band Cloud Optimized GeoTIFF header metadata.

It is intended for Rasteret users who want to skip the build/enrich step and start directly from a filterable Parquet index using Rasteret, DuckDB, Polars, or PyArrow.

## What this dataset contains

Each row corresponds to one AEF annual tile and includes:

- `index.parquet`: a narrow GeoParquet 1.1 index with `fid`, `crs`, `path`, `year`,
  `utm_zone`, `bbox_utm`, `bbox`, `geom`, and `location`
- `data/part-xxxxx.parquet`: read-ready Rasteret collection shards with canonical
  runtime columns `id`, `datetime`, `geometry`, `bbox`, `assets`, `proj:epsg`,
  `year`, `utm_zone`, plus `A00_metadata` through `A63_metadata`

This dataset does **not** store embedding pixel arrays inside Parquet. It stores metadata that lets Rasteret fetch pixels directly from the original COGs.

## Why this exists

The source AEF STAC-geoparquet is already public, but Rasteret normally needs one more step: COG header enrichment. This collection publishes that work once so users can:

- filter remotely before reading any pixels
- join the collection to their own labels, manifests, splits, or experiment metadata
- reuse the same read-ready index across notebooks, jobs, and teams

## License and attribution

This derived metadata index is licensed under **CC-BY-SA-4.0**. It contains no pixel data — only Parquet metadata and COG headers of the imagery.

This dataset was built from the public AlphaEarth Foundations Satellite Embedding Dataset published on Source Cooperative by Taylor Geospatial Engine Labs:

- Source Cooperative product page: https://source.coop/tge-labs/aef
- Source Cooperative AEF index used as metadata input: https://source.coop/tge-labs/aef/v1/annual/aef_index.parquet

The underlying AlphaEarth Foundations Satellite Embedding data is produced by Google and Google DeepMind and is licensed under **CC-BY-4.0**.
This Terrafloww dataset is derived from the Source Cooperative AEF index and the Source Cooperative-hosted AEF COGs.

When using this derived index or the underlying embeddings, include the required upstream attribution text:

> "The AlphaEarth Foundations Satellite Embedding dataset is produced by Google and Google DeepMind."

Reference links:

- Google Earth Engine dataset page: https://developers.google.com/earth-engine/datasets/catalog/GOOGLE_SATELLITE_EMBEDDING_V1_ANNUAL
- Google AEF GCS README: https://developers.google.com/earth-engine/guides/aef_on_gcs_readme
- Taylor Geospatial Engine article on the Source Cooperative release: https://tgengine.org/building-frictionless-geospatial-ai-making-alphaearth-foundations-embeddings-accessible/
- This index license (CC-BY-SA-4.0): https://creativecommons.org/licenses/by-sa/4.0/
- Underlying data license (CC-BY-4.0): https://creativecommons.org/licenses/by/4.0/

## Citation

Use this citation for the derived Terrafloww dataset:

```bibtex
@dataset{terrafloww_aef_v1_annual_rasteret_2026,
  author    = {Terrafloww Labs, Inc.},
  title     = {AEF v1 Annual Rasteret Collection},
  year      = {2026},
  publisher = {Terrafloww Labs, Inc.},
  url       = {https://source.coop/terrafloww/aef-v1-annual-rasteret},
  note      = {Derived Rasteret metadata index built from the Source Cooperative AEF index and Source Cooperative-hosted AlphaEarth Foundations COGs published by Taylor Geospatial Engine Labs}
}
```

Also acknowledge the upstream data source in accompanying text:

- Source Cooperative AEF product published by Taylor Geospatial Engine Labs: https://source.coop/tge-labs/aef
- AlphaEarth Foundations data producer: Google and Google DeepMind
- Required attribution: "The AlphaEarth Foundations Satellite Embedding dataset is produced by Google and Google DeepMind."

## Recommended access patterns

Source Cooperative data is stored on S3. Use `s3://` URIs for glob support and predicate pushdown.

### Rasteret

```python
import rasteret

collection = rasteret.load("s3://us-west-2.opendata.source.coop/terrafloww/aef-v1-annual-rasteret/data")

# Filter to your area and time range
sub = collection.subset(
    bbox=(-95.5, 38.4, -95.0, 38.8),
    date_range=("2024-01-01", "2024-12-31"),
)

# Read pixels — choose your output format
ds = sub.get_xarray(geometries=(-95.4, 38.5, -95.1, 38.7), bands=["A00", "A01"], geometry_crs=4326)
```

## Data layout

- `index.parquet`: narrow GeoParquet 1.1 metadata index for remote filtering
- `data/part-xxxxx.parquet`: read-ready Rasteret collection shards
- rows are clustered by `year`, `utm_zone`, and WGS84 spatial order
- Parquet layout is tuned for repeated filtered reads rather than maximum full-scan throughput

## Intended use

This dataset is for metadata filtering, selection, and read-time handoff into Rasteret. It is especially useful when you want to combine AEF tiles with your own Arrow-native metadata workflows before reading embeddings.

## Also available on

- Hugging Face: [terrafloww/aef-v1-annual-rasteret](https://huggingface.co/datasets/terrafloww/aef-v1-annual-rasteret)

## Limitations

- It depends on the upstream AEF COG URLs remaining available.
- It is optimized for Rasteret and Arrow-native tooling, not for storing embedding payloads in Parquet.
- It is a derived metadata index, not a replacement for the upstream source dataset.

extent.spatial.bbox: omitted; spatial reference is categorical via [data.geometry (geometry), data.bbox (geometry), ..geom (geometry), ..bbox (geometry), ..bbox_utm (geometry), ..location (placename)]. Add bbox once the placename → bbox gazetteer covers it.


Cloud Native Admin Boundaries

Global administrative boundaries covering all jurisdictional levels, distributed in Cloud Native Geo parquet format.

Dataset files (11 x .parquet)

Source Cooperative product page


AlphaEarth Foundations GeoZarr Mosaic

A global mosaic of Google AlphaEarth Foundations Satellite Embeddings stored as a Zarr v3 array, covering 2017–2025 at approximately 10m resolution at the equator. Each pixel encodes a 64-dimensional learned embedding derived from satellite imagery, projected in EPSG:4326 and spanning ±83° latitude. Annual time steps (9 years) enable temporal change detection and similarity search across the full globe.

remote-sensing satellite-embeddings embeddings machine-learning foundation-model global-coverage time-series

Dataset files (1 x .zarr)

Source Cooperative product page

## README.md
# AEF Mosaic

A global mosaic of [Google AlphaEarth Foundations Satellite Embeddings](https://developers.google.com/earth-engine/datasets/catalog/projects_sat-io_open-datasets_AEF_AlphaEarthFoundationsSatelliteEmbedding), covering 2017-2025 at ~0.00009° resolution. The dataset is projected in `EPSG:4326` with a fixed pixel size in degrees. This means that pixels cover ~10m on the ground at the equator, but less area at higher latitudes. For area-based analysis, consider reprojecting to an equal-area projection, [creating your own mosaic](https://github.com/geospatial-jeff/aef-mosaic) in a custom CRS, or using the [underlying COGs](https://source.coop/tge-labs/aef) which are already projected to UTM. This copy is not officially supported by Google.

This dataset implements the following Zarr conventions.  These Zarr conventions are not mature and may change.

| Convention | Description |
|------------|-------------|
| [proj:](https://github.com/zarr-experimental/geo-proj/blob/v1/README.md) | Coordinate reference system information |
| [spatial:](https://github.com/zarr-conventions/spatial/blob/v1/README.md) | Spatial coordinate and transform metadata |
| [geoemb:](https://github.com/geo-embeddings/embeddings-zarr-convention/blob/add-convention/README.md) | Metadata for geo-embeddings |

## License

This dataset is licensed under CC-BY 4.0 and requires the following attribution text: "The AlphaEarth Foundations Satellite Embedding dataset is produced by Google and Google DeepMind."

## Visualizations

### RGB Composite
PCA visualization of the embedding space projected to RGB:

### RGB PCA (Zoomed)
Detailed view showing embedding structure at higher resolution:

### Cosine Similarity
Similarity search using a query embedding:

### Change Detection
Temporal embedding differences highlighting landscape changes:

## Quick Start

```python
import zarr
import numpy as np

root = zarr.open_group(
  "s3://us-west-2.opendata.source.coop/tge-labs/aef-mosaic/",
  mode="r",
  storage_options={"anon": True}
)
embeddings = root["embeddings"]

# Access a spatial subset (year 0, all bands, y slice, x slice)
data = embeddings[0, :, 516000:516100, 832000:832100]

# De-quantize to float32
data_float = ((data / 127.5) ** 2) * np.sign(data)
```

## Xarray

```python
import xarray as xr

def fetch_point(ds, lat, lon, time=0):
    """Fetch AEF embedding for a single lat/lon point. Returns shape (64,)."""
    return ds.isel(time=time).sel(x=lon, y=lat, method="nearest")

def fetch_bbox(ds, bbox, time=0):
    """Fetch AEF embeddings for a bounding box. Returns shape (64, y, x)."""
    minx, miny, maxx, maxy = bbox
    return ds.isel(time=time).sel(x=slice(minx, maxx), y=slice(maxy, miny))

def main():
    ds = xr.open_zarr(
        "s3://us-west-2.opendata.source.coop/tge-labs/aef-mosaic/",
        storage_options={"anon": True}
    )

    # Fetch embedding for a single point
    lat, lon = 34.0522, -118.2437  # Los Angeles
    point_data = fetch_point(ds, lat, lon).compute()
    print(f"Point shape: {point_data.embeddings.shape}")

    # Fetch embeddings for a bounding box
    bbox = [-118.279668, 34.036242, -118.20862, 34.089643]
    bbox_data = fetch_bbox(ds, bbox).compute()
    print(f"Bbox shape: {bbox_data.embeddings.shape}")

if __name__ == "__main__":
    main()
```

## Zarr-python

```python
import asyncio
from affine import Affine
from obstore.store import S3Store
from zarr.storage import ObjectStore
from zarr.api.asynchronous import open_group

store = S3Store(
    "us-west-2.opendata.source.coop",
    prefix="tge-labs/aef-mosaic/",
    region="us-west-2",
    skip_signature=True,
)
zarr_store = ObjectStore(store, read_only=True)

def latlon_to_rowcol(lat, lon, transform):
    """Convert lat/lon to row/col using the inverse transform."""
    inv = ~transform
    col, row = inv * (lon, lat)
    return int(row), int(col)

def bbox_to_slices(bbox, transform):
    """Convert a bounding box to row/col slices."""
    minx, miny, maxx, maxy = bbox
    inv = ~transform
    col1, row1 = inv * (minx, maxy)
    col2, row2 = inv * (maxx, miny)
    row_start, row_stop = int(min(row1, row2)), int(max(row1, row2)) + 1
    col_start, col_stop = int(min(col1, col2)), int(max(col1, col2)) + 1
    return slice(row_start, row_stop), slice(col_start, col_stop)

async def fetch_point(embeddings, transform, lat, lon, year=0):
    """Fetch AEF embedding for a single lat/lon point. Returns shape (64,)."""
    row, col = latlon_to_rowcol(lat, lon, transform)
    return await embeddings.getitem((year, slice(None), row, col))

async def fetch_bbox(embeddings, transform, bbox, year=0):
    """Fetch AEF embeddings for a bounding box. Returns shape (64, rows, cols)."""
    row_slice, col_slice = bbox_to_slices(bbox, transform)
    return await embeddings.getitem((year, slice(None), row_slice, col_slice))

async def main():
    root = await open_group(store=zarr_store, mode="r")
    embeddings = await root.get("embeddings")
    transform = Affine(*root.metadata.attributes["spatial:transform"])

    # Fetch embedding for a single point
    lat, lon = 34.0522, -118.2437  # Los Angeles
    point_data = await fetch_point(embeddings, transform, lat, lon)
    print(f"Point shape: {point_data.shape}")

    # Fetch embeddings for a bounding box
    bbox = [-118.279668, 34.036242, -118.20862, 34.089643]
    bbox_data = await fetch_bbox(embeddings, transform, bbox)
    print(f"Bbox shape: {bbox_data.shape}")

if __name__ == "__main__":
    asyncio.run(main())
```

## Dataset Details

| Property | Value |
|----------|-------|
| **Location** | `s3://us-west-2.opendata.source.coop/tge-labs/aef-mosaic/` |
| **Format** | Zarr v3 with sharding |
| **Shape** | `(9, 64, 1859584, 4009984)` |
| **Dimensions** | `(time, band, y, x)` |
| **Years** | 2017-2025 (indices 0-8) |
| **Bands** | 64 embedding dimensions |
| **CRS** | EPSG:4326 |
| **Bbox** | `[-180.0, -83.36, 180.22, 83.69]` |
| **Resolution** | ~0.00009° (~10m at equator) |
| **Data Type** | int8 (quantized), nodata = -128 |
| **Shard Size** | `(1, 64, 4096, 4096)` |
| **Inner Chunk Size** | `(1, 64, 256, 256)` |

## Source Code

The processing pipeline used to create this mosaic is available at:
https://github.com/geospatial-jeff/aef-mosaic

## Contact

Please send any comments, questions, or points of feedback to the [`#opening-aef`](https://cloudnativegeo.slack.com/archives/C09A8SV3WN5) CNG slack channel.


AlphaEarth Foundations Satellite Embedding Dataset

Annual satellite embeddings derived from Google's Earth Engine Satellite Embedding dataset (V1), covering 2017–2025. Each year's data is organized by UTM zone as 8192×8192-pixel Cloud-Optimized GeoTIFFs with 64 embedding channels per pixel, encoding compact learned representations of the Earth's surface from satellite imagery. This copy was produced by tge-labs and is not officially supported by Google.

remote-sensing satellite-embeddings embeddings machine-learning google-earth-engine annual-time-series cloud-optimized-geotiff utm

Dataset files (302466 x .tiff)

Source Cooperative product page

## README.md
# AlphaEarth Foundations GCS data

This product contains COG (Cloud-Optimized GeoTIFF) files that together contain the AlphaEarth Foundations annual [Satellite Embedding dataset](https://developers.google.com/earth-engine/datasets/catalog/GOOGLE_SATELLITE_EMBEDDING_V1_ANNUAL). It contains the annual embeddings for the years from 2017 to 2025.  This copy is not officially supported by Google.

#### **Warning**
The COGs in this dataset are "bottom-up" where the origin of the image is the bottom-left corner, the y-resolution is positive, and image blocks are ordered from bottom-left to top-right in the file.  A standard COG is "top-down" where the origin is the top-left corner, the y-resolution is negative, and image blocks are ordered from top-left to bottom-right. While some viewers like QGIS are smart enough to flip the images, accessing the data directly with most software (like GDAL) may cause errors.  We've provided a corresponding `.vrt` along with each `.tiff` file in the bucket which is responsible for correcting this mistake on-the-fly.  Please use the provided VRTs to access the data for any use cases that go beyond simple visualization.

## License

This dataset is licensed under CC-BY 4.0 and requires the following attribution text: "The AlphaEarth Foundations Satellite Embedding dataset is produced by Google and Google DeepMind."

## Directory structure

They are divided into directories by year; each year's directory is divided into 120 subdirectories, one per UTM zone, whose names reflect the zone number and hemisphere (`N` or `S`).

Within each directory are a number of COG files. These files contain all the pixel data for that UTM zone.

## File structure

Each file is 8192x8192 pixels, with 64 channels. The magnitude of each pixel, after the de-quantization mapping has been applied (see below), has been normalized so that it has a Euclidean length of 1.

The files contain overview layers at 4096x4096 pixels, 2048x2048 pixels, and so on down to a 1x1 top-level overview layer. These overview layers are constructed so that each overview pixel is the mean of the highest-resolution pixels under that overview pixel, where the mean's magnitude has been normalized to have length 1.

The channels correspond, in order, to the `A00` through `A63` axes of the Satellite Embedding dataset. The COGs also contain this naming for the channels.

Each pixel's value for each channel is a signed 8-bit integer. The way in which these values are mapped to the native values (in [-1, 1]) of the embeddings is described below.

The value -128 corresponds to a masked pixel. If it is present in one channel, it will be present in all channels. The COGs reflect this (i.e., they have the `NoData` value set to -128).

The name of each file also carries some information. For, example, consider the file named
[https://source.coop/tge-labs/aef/v1/annual/2019/1S/x8qqwcsisbgygl2ry-0000008192-0000000000.tiff](https://source.coop/tge-labs/aef/v1/annual/2019/1S/x8qqwcsisbgygl2ry-0000008192-0000000000.tiff).

As described above, this file is part of the 2019 annual embedding, and is in UTM zone 1S (zone 1, southern hemisphere). The base filename, `x8qqwcsisbgygl2ry-0000008192-0000000000`, serves to link this file to the corresponding Earth Engine Satellite Embedding Image name. In this example, this file corresponds to a portion of the Earth Engine image `GOOGLE/SATELLITE_EMBEDDING/V1/ANNUAL/x8qqwcsisbgygl2ry`. The two decimal parts of the filename specify where this COG's values are relative to that Earth Engine Image, as an offset in Y followed by an offset in X. In this case, the COG's pixel origin is at (0, 8192) relative to the Earth Engine Image's origin.

This is because it was necessary to subdivide each Earth Engine Image (which are 16384x16384 pixels) so that the resulting COGs would not be too unwieldy.

## De-quantization

To transform the raw signed 8-bit value (which will be between -127 and 127 inclusive - -128 is reserved as the "no data" value) in each channel of each pixel to the analysis-ready floating-point value (which will be between -1 and 1), the mapping to perform is

- divide by 127.5
- square
- multiply by the sign of the original value

This would be expressed in NumPy as

```python
  # values is a NumPy array of raw pixel values
  de_quantized_values = ((values / 127.5) ** 2) * np.sign(values)
```

In Earth Engine, the corresponding operation would be

```javascript
  var de_quantized_values = values.divide(127.5).pow(2).multiply(values.signum());
```

## Manifest and index

A list of the files in this dataset can be found in
[https://source.coop/tge-labs/aef/v1/annual/manifest.txt](https://source.coop/tge-labs/aef/v1/annual/manifest.txt).

As it is not possible to determine from the file names what area of the world they cover, an index has also been provided, in three forms (GeoParquet, GeoPackage, and CSV) in the files
https://source.coop/tge-labs/aef/v1/annual/aef_index.parquet,
https://source.coop/tge-labs/aef/v1/annual/aef_index.gpkg, and
https://source.coop/tge-labs/aef/v1/annual/aef_index.csv. This
index contains one entry for each file in the dataset. The information provided for each file is:

- the geometry of the file, as a WGS84 (i.e., EPSG:4326) polygon. In the CSV form, this is in the column `WKT`. See below for details on how this geometry is computed.
- `crs`: The CRS of the UTM zone this image belongs to, as an EPSG code, like `EPSG:32610`.
- `year`: The year that the image covers.
- `utm_zone`: The UTM zone of the image, like `10N`.
- `utm_west`, `utm_south`, `utm_east`, `utm_north`: The UTM bounds of the raw pixel array. This does not reflect any geometry processing, and includes all pixels whether or not they are valid.
- `wgs84_west`, `wgs84_south`, `wgs84_east`, `wgs84_north`: The min/max longitude and latitude of the WGS84 geometry.

### Geometry processing

The pixel array is natively in some UTM zone, so in that UTM zone the bounding box of the pixel array is a simple rectangle. That bounding box is transformed into a polygon in WGS84. This polygon includes a number of extra points so that its edges closely follow the curved lines in WGS84 that the straight lines in UTM transform into. This polygon does not take into account the validity/invalidity of pixels in the image, just the bounds of the image's pixel array.

The polygon is then clipped to the minimum and maximum longitude of the image's UTM zone. In practice, this may cause it to not include a few valid pixels that hang over the edge of the UTM zone. Omitting these pixels from the index shouldn't cause any problems: some image from the neighboring UTM zone should cover that area.

Note that clipping to the min/max longitude of the UTM zone means that no polygon crosses the antimeridian, which should make processing this file a little simpler.

providers: role [processor, host] for 'tge-labs' is an AI guess — README states this is an unofficial copy; tge-labs assumed to be the processor who converted to COG format and the host.. Verify before flipping to _state: reviewed.


GlobalBuildingAtlas LoD1

Global dataset of approximately 2.75 billion building footprints in Level of Detail 1 (LoD1) format, combining data from Google Open Buildings, OpenStreetMap, Microsoft Building Footprints, and new detections from deep learning analysis of Planet Labs satellite imagery. Includes building geometry, height information, and geographic coordinates.

remote-sensing building-footprints building-heights urban-mapping 3d-models

Dataset files (922 x .parquet)

Source Cooperative product page

## README.md
This repository contains the [GlobalBuildingAtlas](https://mediatum.ub.tum.de/1782307) (GBA) Level of Detail 1 (LoD1) building footprints dataset, converted from the original 922 GeoJSON files into Parquet format for improved performance and accessibility.

## Dataset Overview

The GlobalBuildingAtlas LoD1 dataset contains building footprints for approximately 2.75 billion buildings worldwide. This represents the most comprehensive global building dataset available, covering nearly all inhabited areas on Earth.

The dataset combines multiple existing building footprint datasets with new footprints generated using open source deep learning models applied to Planet Labs satellite imagery.

This Parquet version was converted from the original 922 GeoJSON files (1.1 TB) to improve performance and accessibility. For detailed information about the conversion process and technical analysis, see [Mark Litwintschik's comprehensive blog post](https://tech.marksblogg.com/building-footprints-gba.html).

## Data Format

The dataset is organized as Parquet files, each containing building footprint data with the following key attributes:

- **Building footprints**: Vector polygons representing building outlines
- **Building heights**: Height information for each building
- **Geographic coordinates**: Precise location data
- **Metadata**: Additional building characteristics

### Parquet Structure

Each Parquet file contains:

- Building geometry (polygons)
- Height attributes
- Spatial indexing for efficient querying
- Optimized columnar storage for fast analytics

## Data Sources and Methodology

The dataset combines multiple building footprint sources:

1. **Google Open Buildings (2023)**: 1.62 billion buildings
2. **OpenStreetMap (2025)**: 490 million buildings
3. **Microsoft Building Footprints (2024)**: 432 million buildings
4. **TUM Deep Learning (ours2)**: 135 million buildings from Planet Labs satellite imagery
5. **3D Global Footprints**: 68 million buildings from previous research

The TUM deep learning component uses open source computer vision models applied to Planet Labs satellite imagery. Planet Labs operates several constellations totaling hundreds of satellites in low Earth orbit, capturing images of the entire Earth's landmasses daily.

## Data Limitations

- **Temporal Accuracy**: No timestamps for AI-generated footprints
- **Data Conflicts**: Some footprints may conflict with newer OpenStreetMap data
- **Construction Areas**: Recent construction may not be captured
- **Validation**: Manual verification recommended for critical applications

## Usage Recommendations

- **Map Making**: Use as base layer with selective integration of newer OSM data
- **Construction Areas**: Prioritize latest OSM data for areas with recent development
- **Research**: Suitable for large-scale analysis and modeling
- **Applications**: Urban planning, disaster response, infrastructure analysis

## Technical Specifications

- **Coordinate System**: WGS84
- **File Format**: Parquet
- **Compression**: Optimized for fast querying
- **Spatial Indexing**: H3 and other spatial indices included
- **Total Size**: 210 GB (compressed from 1.1 TB original)

## Data Quality Notes

The dataset combines multiple data sources as detailed above. The TUM deep learning component (ours2) represents new building footprints generated from Planet Labs satellite imagery, while the other sources provide existing building data from various organizations.

For areas with significant recent construction, consider using the latest OpenStreetMap data or Overture's building dataset, which prioritizes recent OSM updates.

## Citation

If you use this Parquet version of the dataset hosted on Source Cooperative, please cite both the original dataset and this hosted version:

**Original Dataset:**

```
@misc{1782307,
  author = {Zhu, Xiao Xiang and Chen, Sining and Zhang, Fahong and Shi, Yilei and Wang, Yuanyuan},
  title = {{GlobalBuildingAtlas: An Open Global and Complete Dataset of Building Polygons, Heights and LoD1 3D Models}},
  type = {Dataset},
  year = {2025},
  type = {Forschungsdaten},
  abstract = {GlobalBuildingAtlas is a dataset providing global and complete coverage of building polygons (GBA.Polygon), heights (GBA.Height) and Level of Detail 1 (LoD1) 3D building models (GBA.LoD1). It is the first open dataset to offer high quality, consistent, and complete building data in 2D and 3D at the individual building level on a global scale. The dataset is delivered in tiles of 5 degree by 5 degree, with GBA.Polygon and GBA.LoD1 in GeoJSON format, and GBA.Height in GeoTiff format. Details see https://github.com/zhu-xlab/GlobalBuildingAtlas.},
  keywords = {Building Footprint; Building Height; 3D Building Models; Remote Sensing; Earth Observation; Deep Learning},
  doi = {10.14459/2025mp1782307}
}
```

**Source Cooperative Hosted Version:**

```
@misc{globalbuildingatlas_lod1_parquet,
  title = {{GlobalBuildingAtlas LoD1 (Parquet Format)}},
  author = {Zhu, Xiao Xiang and Chen, Sining and Zhang, Fahong and Shi, Yilei and Wang, Yuanyuan and {Source Cooperative}},
  year = {2025},
  url = {https://source.coop/tge-labs/globalbuildingatlas-lod1},
  note = {Parquet format conversion of GlobalBuildingAtlas LoD1 dataset, hosted by Source Cooperative}
}
```


Meta CHM v2 (DINOv3 global, ml3) — cloud-native companion

A cloud-native repackaging of Meta's DINOv3 Global Canopy Height Map v2 (ml3), providing a global ~1.19 m resolution canopy height raster spanning 213,109 Web-Mercator tiles in EPSG:3857. Pixel values encode canopy height in meters (uint8, 0–254) derived from satellite imagery and deep learning. The dataset is distributed as cloud-optimized GeoTIFFs, a multiscale GeoZarr/Icechunk store, GeoParquet tile indexes, and a static STAC catalog.

canopy-height forest lidar global-coverage dinov3 meta remote-sensing cloud-optimized-geotiff zarr icechunk geoparquet

Multiscale GeoZarr (Icechunk, virtual)

Self-contained Icechunk virtual GeoZarr; groups 1x (native 1.19 m) .. 64x. Open anonymously with icechunk + xarray, or via the Arraylake repo 'taylor-geospatial/meta-chm-v2'.

Source Cooperative product page

Tolan et al., 2024

Brandt et al., 2026 (CHMv2)

## README.md
# Meta CHM v2 (ml3) — Cloud-Native

A cloud-native repackaging of Meta's **DINOv3 Global Canopy Height Map v2 (ml3)**:
a global ~1.19 m canopy-height raster (213,109 Web-Mercator tiles).

Everything lives in this one bucket with CORS + HTTP range support, so it streams directly
to browsers and analysis tools — no credentials, no second host:

```
s3://us-west-2.opendata.source.coop/tge-labs/meta-chm-v2/   (https://data.source.coop/tge-labs/meta-chm-v2/)
├── chm/                     # 213,109 COGs — uint8 metres, EPSG:3857, ~23.8 TB (CORS mirror of Meta's)
├── overview/
│   └── chm_overview_z8.tif  # single global overview COG (z0–9) for fast world / low-zoom views
├── tiles.parquet            # GeoParquet 1.1 tile index (213,109 rows, bbox-indexed)
├── acq_dates.parquet        # per-tile source acquisition window (start / end / count)
├── stac/
│   ├── collection.json      # STAC Collection (CC-BY-4.0)
│   ├── items.parquet        # stac-geoparquet — one Item per tile, assets → this bucket's COGs
│   └── items_sample/        # ~200 sample STAC Item JSONs (spec inspection)
├── zarr/
│   └── chm.zarr.icechunk    # multiscale GeoZarr (Icechunk, virtual), groups 1x (native 1.19 m) .. 64x;
│                            #   zero-copy byte-range refs into this bucket's COGs
├── README.md
└── LICENSE
```

## What each artifact is for

| Artifact                       | Use it for                                                                                |
| ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `chm/`                         | The pixels — native ~1.19 m COGs, CORS + range-readable. Read directly via `/vsicurl`, rioxarray, rio-tiler, or an in-browser COG reader (geotiff.js).                        |
| `overview/chm_overview_z8.tif` | One global COG covering z0–9 — fast continental/world views without touching 213k tiles.                                                                                |
| `tiles.parquet`                | Fast spatial lookup of which tile covers an AOI; drives maplibre/deck.gl tile loaders and DuckDB spatial queries. Replaces the original 56 MB `tiles.geojson`.                |
| `stac/`                        | Discovery + access via STAC tooling (pystac, stac-fastapi, TiTiler, odc-stac). Asset hrefs point at this bucket's CORS COGs.                                                  |
| `zarr/chm.zarr.icechunk`       | `xarray`/`dask` analytics + multiscale visualization (Arraylake). A 7-level pyramid, `1x` (native 1.19 m) … `64x` (~76 m); reads are byte-range GETs into this bucket's COGs. |

## CRS & grid

- **EPSG:3857** (Web Mercator) — matches the source; zero reprojection for web maps.
- 10-character Bing/Microsoft quadkey tile grid (zoom 10). Native pixel ≈ 1.19 m at the equator.
- Pixel values are **canopy height in meters** (uint8, 0–254). `0` is both true-zero and
    no-data (the source sets no explicit nodata mask).

## Quickstart

### Tile index (DuckDB)

```sql
INSTALL spatial; LOAD spatial;
SELECT quadkey, cog_url, tile_size_bytes
FROM 'https://data.source.coop/tge-labs/meta-chm-v2/tiles.parquet'
WHERE bbox_3857.minx < 1000000 AND bbox_3857.maxx > 0;  -- AOI filter
```

### STAC — serverless, cloud-native search

This is a **static** STAC catalog: a `collection.json` plus a stac-geoparquet
`items.parquet` (213,109 Items). **No STAC API server is required.** The parquet is
written in 54 quadkey-sorted (Z-order) row groups with per-group bbox statistics, so a
spatial query reads only the matching byte ranges over HTTP — e.g. a city-scale bbox
touches ~3 of 54 row groups, skipping ~94% of the file. Point any of these tools straight
at the public URL:

```
https://data.source.coop/tge-labs/meta-chm-v2/stac/items.parquet
```

**Collection metadata (`pystac`):**

```python
import pystac
c = pystac.Collection.from_file(
    "https://data.source.coop/tge-labs/meta-chm-v2/stac/collection.json"
)
```

**Search the Items — pick your tool, all stream + prune row groups:**

```python
# rustac — a pystac-client-like search, in-process, no server
from rustac import DuckdbClient
items = DuckdbClient().search(ITEMS_URL, bbox=[13.0, 52.0, 13.4, 52.3], max_items=100)
```

```sql
-- DuckDB (httpfs range reads + row-group pruning)
INSTALL spatial; LOAD spatial;
SELECT id, assets FROM read_parquet('https://data.source.coop/tge-labs/meta-chm-v2/stac/items.parquet')
WHERE bbox.xmin <= 13.4 AND bbox.xmax >= 13.0 AND bbox.ymin <= 52.3 AND bbox.ymax >= 52.0;
```

```python
# pyarrow.dataset — predicate pushdown over the remote file
import pyarrow.dataset as ds, pyarrow.compute as pc
t = ds.dataset(ITEMS_URL, format="parquet").to_table(
    filter=(pc.field("bbox", "xmin") <= 13.4) & (pc.field("bbox", "xmax") >= 13.0)
)
```

Also works: **geopandas** `read_parquet(..., bbox=...)`, **Polars** `scan_parquet`, and the
**stac-geoparquet** library. For teams that specifically need the `pystac-client` API
(`Client.open(...).search(...)`), serve `items.parquet` behind
[`stac-fastapi-geoparquet`](https://github.com/stac-utils/stac-fastapi-geoparquet) — no
database required.

**Search → xarray (`odc.stac`):** turn found Items into a lazily-loaded, dask-backed
`xarray` cube reading the COGs directly (EPSG:3857, no reprojection):

```python
import os, pystac, odc.stac
from rustac import DuckdbClient

os.environ.update(AWS_NO_SIGN_REQUEST="YES", AWS_REGION="us-west-2")  # this bucket's COGs are anonymous + CORS
bbox = [13.30, 52.45, 13.45, 52.55]
items = [pystac.Item.from_dict(d) for d in DuckdbClient().search(ITEMS_URL, bbox=bbox)]
chm = odc.stac.load(items, bands=["chm"], bbox=bbox, resolution=10, chunks={"x": 2048, "y": 2048})
heights = chm["chm"].isel(time=0).compute()  # canopy height in meters
```

See `examples/search_and_read.py` and `examples/odc_stac_load.py` in the
[chm-zarr](https://github.com/taylor-geospatial/meta-chm-v2) repo.

### Multiscale GeoZarr (xarray + Icechunk)

The GeoZarr is virtual — its chunks are byte-range references into the COGs in this same
bucket, so it carries the full pyramid (`1x` native … `64x`) with no duplicated pixels.

```python
import icechunk, xarray as xr
prefix = "s3://us-west-2.opendata.source.coop/"
cfg = icechunk.RepositoryConfig.default()
cfg.set_virtual_chunk_container(
    icechunk.VirtualChunkContainer(prefix, icechunk.s3_store(region="us-west-2", anonymous=True))
)
repo = icechunk.Repository.open(
    icechunk.s3_storage(
        bucket="us-west-2.opendata.source.coop",
        prefix="tge-labs/meta-chm-v2/zarr/chm.zarr.icechunk",
        region="us-west-2", anonymous=True,
    ),
    config=cfg,
    authorize_virtual_chunk_access=icechunk.containers_credentials(
        {prefix: icechunk.s3_anonymous_credentials()}
    ),
)
dt = xr.open_datatree(repo.readonly_session("main").store, engine="zarr", consolidated=False)
chm_native = dt["1x"]["chm"]   # native ~1.19 m  (use 8x ≈ 9.6 m, 64x ≈ 76 m for overviews)
```

Also available as a managed Arraylake repo: `taylor-geospatial/meta-chm-v2`.

## Provenance & license

CC-BY-4.0. Underlying data © Meta / Data for Good; repackaging by Taylor
Geospatial Engine Labs. See `LICENSE`. Cite Tolan et al. (2024),
https://arxiv.org/abs/2304.07213, and Brandt et al. (2026, CHMv2),
https://arxiv.org/abs/2603.06382.

The GeoZarr store holds only chunk references (offsets/lengths) into the COGs in **this**
bucket (`chm/`), so it is self-contained — no dependency on Meta's bucket.

providers: role [producer] for 'World Resources Institute (WRI)' is an AI guess — Listed as co-producer in the API description alongside Meta; role not explicitly described in the README body. Verify before flipping to _state: reviewed.


Cloud Native MGRS

A complete Military Grid Reference System (MGRS) 1-kilometer reference grid derived from National Geospatial-Intelligence Agency (NGA) source data, converted to cloud-native GeoParquet format. Contains over 305 million polygon grid cells spanning global extent from -80° to 84° latitude, each encoded with MGRS codes, Grid Zone Designators, and 10-kilometer tile identifiers. Data is available as a single file and as ~700 GZD-partitioned files for efficient spatial querying.

geoparquet global-coverage remote-sensing mgrs coordinate-system utm-grid grid-reference

full

GZD

[admin_code] MGRS Grid Zone Designator (6° × 8° zone code used as partition key)

MGRS

[spatial_index] Full MGRS grid reference code identifying the 1 km cell

[spatial_index] 10-kilometer MGRS tile identifier containing this cell

geometry

[geometry] Polygon boundary of each 1 km MGRS grid cell in EPSG:4326

2 files sampled for this table

gzd_partition

GZD

[admin_code] MGRS Grid Zone Designator; one partition file per GZD value

geometry

[geometry] Polygon boundary of each 1 km MGRS grid cell in OGC:CRS84

bbox

STRUCT(xmin DOUBLE, ymin DOUBLE, xmax DOUBLE, ymax DOUBLE)

3 files sampled for this table

partition/by_kdtree

GZD

[admin_code] MGRS Grid Zone Designator

5 files sampled for this table

Source Cooperative product page

## README.md
## About

This project crawled the map download links at https://earth-info.nga.mil/index.php?dir=coordsys&action=utm-1km-polyline-dloads
and put them into a GeoParquet files that follow the [Best Practices for Distributing GeoParquet](https://github.com/opengeospatial/geoparquet/blob/main/format-specs/distributing-geoparquet.md). The `full` directory contains the complete dataset as a single GeoParquet file. For now it just contains `mgrs_clean.parquet` which is all the
download links that worked, but removing the ShapeLen, ShapeArea and area columns, as they weren't all that consistent.

## 📦 Dataset Overview

### **mgrs_clean.parquet**
**Size:** 5.87 GB
**Compression:** ZSTD
**Rows:** 305,799,745
**Row Groups:** 2,483
**GeoParquet Version:** 1.0.0
**CRS:** EPSG:4326
**Bounding Box:** `[-180.000000, -80.000000, 180.000000, 84.000193]`

#### Columns
| Name        | Type   |
|-------------|--------|
| EASTING     | string |
| NORTHING    | string |
| kmSQ_ID     | string |
| GZD         | string |
| MGRS        | string |
| MGRS_10km   | string |
| geometry 🌍 | binary |

---

### Partitioning Notes

The `partitioned/` directory currently includes **`gzd_partition/`**, containing ~700 partition files (one per GZD).
These partitions:

- originate from `mgrs_clean.parquet`
- include **GeoParquet 1.1 bounding boxes** for faster spatial filtering
- are intended to support future **VecOREL** extensions:
  - administrative division references
  - geometry metrics

---

## 📦 Sample Subset

### **mgrs_10X.parquet**
**Size:** 9.02 MB
**Compression:** ZSTD
**Rows:** 187,646
**Row Groups:** 1
**GeoParquet Version:** 1.1.0
**CRS:** EPSG:4326
**Bounding Box:** `[-180.000000, -80.000000, 180.000000, 84.000193]`

#### Columns
| Name        | Type |
|------------|------|
| EASTING     | string |
| NORTHING    | string |
| kmSQ_ID     | string |
| GZD         | string |
| MGRS        | string |
| MGRS_10km   | string |
| geometry 🌍 | binary |
| bbox | struct `<xmin: double, ymin: double, xmax: double, ymax: double>` |

---

## Usage

The best way to use these is to treat the partition as a full parquet dataset and query just what you need out of it. With DuckDB
you can do:

```
CREATE TABLE mgrs_grid AS (
  SELECT * FROM read_parquet('s3://us-west-2.opendata.source.coop/tge-labs/mgrs/gzd_partition/*/*.parquet')
           WHERE "bbox".xmin BETWEEN 11.14686 AND 14.853739
           AND "bbox".ymin BETWEEN 50.49916 AND 54.37598);
```

And then you'll have the table locally. You can go straight to a parquet file with

```
COPY (
  SELECT * FROM read_parquet('s3://us-west-2.opendata.source.coop/tge-labs/mgrs/gzd_partition/*/*.parquet')
           WHERE "bbox".xmin BETWEEN 11.14686 AND 14.853739
           AND "bbox".ymin BETWEEN 50.49916 AND 54.37598)
    TO 'my_mgrs_grid.parquet' (FORMAT PARQUET, COMPRESSION zstd, COMPRESSION_LEVEL 15);
```

extent.spatial.bbox: omitted; spatial reference is categorical via [full.geometry (geometry), full.MGRS (spatial_index), full.MGRS_10km (spatial_index), full.GZD (admin_code), gzd_partition.geometry (geometry), gzd_partition.MGRS (spatial_index), gzd_partition.MGRS_10km (spatial_index), gzd_partition.GZD (admin_code), partition/by_kdtree.geometry (geometry), partition/by_kdtree.MGRS (spatial_index), partition/by_kdtree.MGRS_10km (spatial_index), partition/by_kdtree.GZD (admin_code)]. Add bbox once the placename → bbox gazetteer covers it.


OpenBuildingMap

Global dataset of 2.7 billion building footprints with associated metadata including height, usage classification, and floorspace. Covers all terrestrial areas worldwide.

building-footprints building-heights building-usage floor-area

Dataset files (1270 x .parquet)

Source Cooperative product page


TerraBit


Global Risk/Resilience Index - Supporting Data

A collection of global, open datasets relating to climate hazards and infrastructure systems, processed into analysis-ready formats. Hazards include: coastal and river flooding, extreme heat and drought, and tropical cyclone wind speeds. Exposure layers include: population, built-up area, roads, railways, power plants and transmission lines. Contextual information includes: elevation, land use and land cover, and administrative boundaries.


Trazo Fields

Agricultural field boundaries produced by Trazo models for key geographies in South America for the 2023-2024 planting season. Post-processed.


Trazo Training Data

# South America Field Boundary Annotations

A training dataset of **46,908** high-quality, diverse field boundary annotations with representation from 17 ecoregions and a test set of **7,453** field boundary annotations.


HURREVAC Storm Advisory Archive

This dataset provides a point-in-time archive and persistent mirror of the tropical cyclone advisory data from HURREVAC, the decision support tool used by the National Hurricane Program (a partnership between FEMA, USACE, and NOAA).

The data was programmatically captured from the HURREVAC API (/v1/advisories/storms) to ensure long-term accessibility for the emergency management and research communities, specifically to mitigate the risk of data loss due to potential funding shifts or service changes.

The archive consists of JSON-formatted records containing:

Current Storm Positions: Historical and active tropical cyclone coordinates.

Forecast Tracks: Official forecast data as issued during active storm events.

Metadata: Unique identifiers and structural data used by the Sea Island Software HVX platform.

Data Source
Provider: National Hurricane Program via HURREVAC API.
Original URL: http://hvx.hurrevac.com/hvx-api/v1/advisories/storms
Format: JSON


Global Administrative Area Boundaries

Administrative boundaries for several levels (GADM Level 0 - 5)
Source https://gadm.org
License: https://gadm.org/license.html


maxar-opendata-v1.1

Testing out conversion of the Maxar Open Data Program data to geoparquet v1.1


Test Files

A variety of files for testing Source Cooperative.


Wall tiles

Uploading TIFF files


Test Title

Test description


USDA Cropland Data Layer

A raster crop-specific land cover data layer of the continental United States, published annually by the USDA National Agricultural Statistics Service.


testproduct

this is a test


USASpending Data

Bulk data from USASpending.gov


A merged global dataset combining Google's V3 Open Buildings (1.8 billion detections) and Microsoft's Global Building Footprints (1.24 billion detections) into 2.58 billion building footprints covering Africa, South Asia, South-East Asia, Latin America, and more. Each footprint carries its source label, confidence score, area in square meters, and country ISO code. Available in GeoParquet, FlatGeobuf, and PMTiles formats, partitioned by country and S2 grid.

building-footprints remote-sensing urban-mapping global-coverage geoparquet

AFG.fgb

AGO.fgb

ALB.fgb

AND.fgb

ARE.fgb

2211267417038913536.fgb

[spatial_index] S2 grid cell ID used for sub-country partitioning (S2-partitioned assets only)

[admin_code] 3-letter ISO country code for the footprint's country (S2-partitioned assets only)

geohash

[spatial_index] Geohash at precision 8 used for spatial ordering (S2-partitioned assets only)

2220274616293654528.fgb

2229281815548395520.fgb

2017612633061982208.fgb

Source Cooperative product page

## README.md
# Google-Microsoft Open Buildings - combined by VIDA
## Overview
This dataset merges Google's V3 Open Buildings and Microsoft's latest Building Footprints. It contains 2,579,035,323 footprints and is divided into 185 partitions. Each footprint is labelled with its respective source, either Google or Microsoft. It can be accessed in cloud-native geospatial formats such as GeoParquet, FlatGeobuf and PMTiles.

## See it in action
You can [Observable](https://observablehq.com/d/d84e078c16baa3ae) to get a quick overview of the dataset or go to [VIDA](https://map.vida.place/explore) to see it in action.

## Original datasets

### Google
The original Google V3 open buildings is downloadable from [this link](https://sites.research.google/open-buildings/#download) as gzipped CSV files. Here are some key details about the original dataset:

> The dataset contains 1.8 billion building detections, across an inference area of 58M km2 within Africa, South Asia, South-East Asia, Latin America and the Caribbean.
>
> Each building in the dataset has a polygon defining its footprint on the ground, a confidence score indicating how certain we are that this is a building, and a Plus Code corresponding to the centre of the building. There is no information about the type of building, its street address, or any details other than its geometry.

For more comprehensive information, please visit the [description page](https://sites.research.google/open-buildings/). You can also check out the [FAQ](https://sites.research.google/open-buildings/#faq) section for additional information.

### Microsoft
The latest version of Microsoft's building footprints can be downloaded from [Microsoft Planetary Computer](https://planetarycomputer.microsoft.com/dataset/ms-buildings) as gzipped partitioned files.

> The Microsoft Global Open Buildings dataset was generated through Bing Maps, which detected a total of 1.24 billion buildings. These buildings were identified using imagery from Bing Maps, encompassing data collected between 2014 and 2023, including images from Maxar, Airbus, and IGN France.
>

For more detailed information please visit the [github page](https://github.com/microsoft/GlobalMLBuildingFootprints/)

## Data Formats
The data is available in the following formats:
- GeoParquet 1.1.0
  - By country - single file
  - By country - S2 partitioned
- FlatGeobuf
  - By country - single file
  - By country - S2 partitioned
- PMTiles
  - Global - single layer
  - Global - layer per country based on the 3-letter ISO code
  - By country

## Partitioning
This extensive dataset is organized into 185 root partitions. Each partition typically corresponds to a country's administrative boundary, as defined by the Comprehensive Global Administrative Zones (CGAZ) at the ADM0 level, which can be accessed [here](https://www.geoboundaries.org/downloadCGAZ.html). There is also a sub-partition available, based on the S2 grid.

### By country
Both FlatGeobuf and GeoParquet are categorized by country boundaries, in accordance with the ADM0 level of the CGAZ geoboundary definition. This means that building footprints are separated by countries within each format. For naming conventions, we utilize the country's ISO CODE.

`/geoparquet/by_country/country_iso={ISO}/{ISO}.parquet`

> **Note:** There is a partition labeled `country_iso=None`, which represents a MULTIPOLYGON containing geoboundaries (POLYGONS) that have not been explicitly defined or named by CGAZ. These geoboundaries are still captured by CGAZ at the ADM0 level, but they lack specific names and therefore labelled `null`. As a result, building footprints located within these geoboundaries are included in this partition labeled `country_iso=None`. For instance, the area between Sudan and South Sudan includes a piece of land known as "Abyei" which remains unclaimed due to recurring conflicts, and therefore, it lacks an assigned name.

### By country + S2 grid
To enhance performance, particularly with GeoParquet files, we've introduced an S2 sub-partitioning strategy. Each ISO partition is further divided using an S2 grid ID, ensuring a cap of 20 million building footprints per grid ID. This S2 grid partitioning is exclusive to GeoParquet files.

`/geoparquet/by_country_s2/country_iso={ISO}/{S2_GRID_ID}.parquet`

## Schema
Each row in the dataset provides information on a specific building footprint with associated information on individual columns:
- **boundary_id (INTEGER):** A unique ID linking the CGAZ level 0 boundary ISO to an integer, created for partitioning the datasets within BigQuery.
- **confidence (FLOAT):** A metric denoting the model's confidence about the accuracy of the building footprint. Microsoft-sourced footprints set this column to null since the original dataset doesn't feature this attribute.
- **bf_source (STRING):** Indicates the footprint's origin - Google or Microsoft.
- **area_in_meters (FLOAT):** Represents the polygon's area in square meters.
- **s2_id (INT):** Exclusive to the S2 partitioning scheme, it represents the S2 grid ID.
- **country_iso (STRING):** 3-letter ISO code of the country the footprint belongs within.
- **geohash (STRING):** Geohash for the geometry at a precision level of 8.
- **bbox (STRUCT):** Struct containing `xmin, ymin, xmax, ymax` values for the bounding box of the geometry.

## Data Processing
We invite you to read our [blog post](https://cloudnativegeo.org/blog/2023/09/introducing-the-ultimate-cloud-native-building-footprints-dataset/) for more detailed information on our dataset merging approach, which includes insights into the optimization techniques we investigated and the query performance on BigQuery. In this section, we provide a high-level summary of the merging process, highlighting its crucial aspects.
\
We imported both datasets into BigQuery for further processing. From the Google dataset, we excluded columns like `full_plus_code`, `latitude`, and `longitude`. For the Microsoft dataset we did not drop any columns.
We then matched each building footprint with a boundary ID, determined by the intersection of its centroid with the country geoboundaries in the CGAZ ADM0 dataset. Footprints whose centroids didn't overlap with any country geoboundary were mapped to the nearest geoboundary based on their centroid's position.

## Contact details
If you'd like more information about the dataset or the processing steps, feel free to write an email to [support@vida.place](mailto:support@vida.place).

## Changelog
Current version: 2.0

### Version 2.0 - 2024-09-04
- Add 32,784,238 building footprints for various regions by updating to the latest Microsoft [GlobalMLBuildingFootprints](https://github.com/microsoft/GlobalMLBuildingFootprints) as of 2024-05-28.
- Update to GeoParquet schema version 1.1.0.
  - Includes bbox struct for easy filtering.
- Introduce spatial ordering by geohash for FlatGeobuf and GeoParquet files.
- Add PMTiles files per country.
- Add PMTiles file with a layer per country ISO code.

### Version 1.1 - 2023-10-02
- Added 11,631,283 building footprints for Morroco from the [Google Earthquake dataset](https://data.humdata.org/dataset/openbuildings_morocco_earthquake_footprint)
- Added 24,532 building footprints for Libya from the [Google Derna Flooding dataset](https://data.humdata.org/dataset/buildings-footprint-floods-derna-libya)
- Building footprints are added to the GeoParquet, FlatGeobuf and PMTiles archives.
- Fixed the missing GeoParquet version bug.
- Refactored S2 grid naming strategy from unsigned 64bit integers to signed 64bit integers.

### Version 1 - 2023-08-29

## Dataset Licenses
The data is shared under the [Open Data Commons Open Database License (ODbL) v1.0](https://opendatacommons.org/licenses/odbl/1-0/) license.

providers: role [processor, licensor] for 'VIDA' is an AI guess — VIDA performed the merge and publishes the combined dataset; licensor role inferred from their ODbL release — not explicitly stated. Verify before flipping to _state: reviewed.

extent.spatial.bbox: omitted; spatial reference is categorical via [2211267417038913536.s2_id (spatial_index), 2211267417038913536.country_iso (admin_code), 2211267417038913536.geohash (spatial_index)]. Add bbox once a placename → bbox gazetteer is available.


A global dataset of 2.7 billion building footprints created by merging Google's V3 Open Buildings, Microsoft's GlobalMLFootprints, and OpenStreetMap building data. Each record includes the footprint polygon, source attribution (Google, Microsoft, or OSM), area in square meters, confidence score, and country ISO code. Data is partitioned by country and S2 grid, available in GeoParquet, FlatGeobuf, and PMTiles formats.

building-footprints remote-sensing openstreetmap administrative-boundaries global-coverage geoparquet

AFG.fgb

[spatial_index] S2 grid cell ID used for sub-country partitioning (cap of 20M footprints per cell)

[admin_code] ISO 3-letter country code assigned from CGAZ ADM0 boundaries

[spatial_index] Geohash spatial index for the building footprint centroid

AGO.fgb

ALB.fgb

AND.fgb

ARE.fgb

2211267417038913536.fgb

2220274616293654528.fgb

2229281815548395520.fgb

2017612633061982208.fgb

Source Cooperative product page

## README.md
# Google-Microsoft-OSM Open Buildings - combined by VIDA
## Overview
This dataset merges Google's V3 Open Buildings, Microsoft's GlobalMLFootprints, and OpenStreetMap building footprints. It contains 2,705,459,584 footprints and is divided into 200 partitions. Each footprint is labelled with its respective source, either Google, Microsoft, or OpenStreetMap. It can be accessed in cloud-native geospatial formats such as GeoParquet, FlatGeobuf and PMTiles.

## See it in action
You can [Observable](https://observablehq.com/d/d84e078c16baa3ae) to get a quick overview of the dataset or go to [VIDA](https://map.vida.place/explore) to see it in action.

## Original datasets

### Google
The original Google V3 open buildings is downloadable from [this link](https://sites.research.google/open-buildings/#download) as gzipped CSV files. Here are some key details about the original dataset:

> The dataset contains 1.8 billion building detections, across an inference area of 58M km2 within Africa, South Asia, South-East Asia, Latin America and the Caribbean.
>
> Each building in the dataset has a polygon defining its footprint on the ground, a confidence score indicating how certain we are that this is a building, and a Plus Code corresponding to the centre of the building. There is no information about the type of building, its street address, or any details other than its geometry.

For more comprehensive information, please visit the [description page](https://sites.research.google/open-buildings/). You can also check out the [FAQ](https://sites.research.google/open-buildings/#faq) section for additional information.

### Microsoft
The latest version of Microsoft's building footprints can be downloaded from [Microsoft Planetary Computer](https://planetarycomputer.microsoft.com/dataset/ms-buildings) as gzipped partitioned files.

> The Microsoft Global Open Buildings dataset was generated through Bing Maps, which detected a total of 1.24 billion buildings. These buildings were identified using imagery from Bing Maps, encompassing data collected between 2014 and 2023, including images from Maxar, Airbus, and IGN France.

For more detailed information please visit the [github page](https://github.com/microsoft/GlobalMLBuildingFootprints/)

### OpenStreetMap
The OpenStreetMap building footprints are sourced from [OpenStreetMap](https://www.openstreetmap.org/). The dataset is updated regularly and contains building footprints from all over the world.

> OpenStreetMap is a collaborative project that creates a free editable map of the world. The data is collected by volunteers and can be used for various purposes, including building footprints.
> There is more than 615 million building footprints in the OpenStreetMap dataset at the time of writing.

For more detailed information please visit the [OpenStreetMap Wiki](https://wiki.openstreetmap.org/wiki/Buildings)

## Data Formats
The data is available in the following formats:
- GeoParquet 1.1.0
  - By country - single file
  - By country - S2 partitioned
- FlatGeobuf
  - By country - single file
  - By country - S2 partitioned
- PMTiles
  - Global - single layer
  - Global - layer per country based on the 3-letter ISO code
  - By country

## Partitioning
This extensive dataset is organized into 182 root partitions. Each partition typically corresponds to a country's administrative boundary, as defined by the Comprehensive Global Administrative Zones (CGAZ) at the ADM0 level, which can be accessed [here](https://www.geoboundaries.org/downloadCGAZ.html). There is also a sub-partition available, based on the S2 grid.

### By country
Both FlatGeobuf and GeoParquet are categorized by country boundaries, in accordance with the ADM0 level of the CGAZ geoboundary definition. This means that building footprints are separated by countries within each format. For naming conventions, we utilize the country's ISO CODE.

`/geoparquet/by_country/country_iso={ISO}/{ISO}.parquet`

> **Note:** There is a partition labeled `country_iso=None`, which represents a MULTIPOLYGON containing geoboundaries (POLYGONS) that have not been explicitly defined or named by CGAZ. These geoboundaries are still captured by CGAZ at the ADM0 level, but they lack specific names and therefore labelled `null`. As a result, building footprints located within these geoboundaries are included in this partition labeled `country_iso=None`. For instance, the area between Sudan and South Sudan includes a piece of land known as "Abyei" which remains unclaimed due to recurring conflicts, and therefore, it lacks an assigned name.

### By country + S2 grid
To enhance performance, particularly with GeoParquet files, we've introduced an S2 sub-partitioning strategy. Each ISO partition is further divided using an S2 grid ID, ensuring a cap of 20 million building footprints per grid ID. This S2 grid partitioning is exclusive to GeoParquet files.

`/geoparquet/by_country_s2/country_iso={ISO}/{S2_GRID_ID}.parquet`

## Schema
Each row in the dataset provides information on a specific building footprint with associated information on individual columns:
- **boundary_id (INTEGER):** A unique ID linking the CGAZ level 0 boundary ISO to an integer, created for partitioning the datasets within BigQuery.
- **confidence (FLOAT):** A metric denoting the model's confidence about the accuracy of the building footprint. Microsoft-sourced footprints set this column to null since the original dataset doesn't feature this attribute.
- **bf_source (STRING):** Indicates the footprint's origin - Google or Microsoft.
- **area_in_meters (FLOAT):** Represents the polygon's area in square meters.
- **s2_id (INT):** Exclusive to the S2 partitioning scheme, it represents the S2 grid ID.

## Data Processing
We invite you to read our [blog post](https://medium.com/vida-engineering/updating-the-ultimate-cloud-native-building-footprints-dataset-6d4384cb93c4) for more detailed information on our dataset merging approach, which includes insights into the optimization techniques we investigated and the query performance on BigQuery. In this section, we provide a high-level summary of the merging process, highlighting its crucial aspects.
\
We imported both datasets into BigQuery for further processing. From the Google dataset, we excluded columns like `full_plus_code`, `latitude`, and `longitude`. For the Microsoft dataset we did not drop any columns.
We then matched each building footprint with a boundary ID, determined by the intersection of its centroid with the country geoboundaries in the CGAZ ADM0 dataset. Footprints whose centroids didn't overlap with any country geoboundary were mapped to the nearest geoboundary based on their centroid's position.

## Contact details
If you'd like more information about the dataset or the processing steps, feel free to write an email to [support@vida.place](mailto:support@vida.place).

## Changelog

Current version: 2.0

### Version 2.0 - 2024-09-04
- Initial release!
  - 2,705,459,584 building footprints for various regions by merging Google's V3 Open Buildings, Microsoft's GlobalMLFootprints, and OpenStreetMap building footprints.
  - Using GeoParquet schema version 1.1.0.
  - Data is partitioned by country and S2 grid.
  - Data is available in GeoParquet, FlatGeobuf, and PMTiles formats.
  - Google Open Buildings last updated 2023-10-02
  - Microsoft GlobalMLFootprints last updated 2024-05-28
  - OpenStreetMap last updated 2024-05-28

## Dataset Licenses
The data is shared under the [Open Data Commons Open Database License (ODbL) v1.0](https://opendatacommons.org/licenses/odbl/1-0/) license.

providers: role [processor, licensor] for 'VIDA' is an AI guess — Processor role is stated; licensor inferred as VIDA publishes the combined dataset under ODbL. Verify before flipping to _state: reviewed.

extent.spatial.bbox: omitted; spatial reference is categorical via [AFG.country_iso (admin_code), AFG.geohash (spatial_index), AFG.s2_id (spatial_index)]. Add bbox once a placename → bbox gazetteer is available.


Merged Google-Microsoft Buildings

This dataset is a combination of Google's and Microsoft's Open Buildings. It is an openly accessible collection that provides the geometric information of buildings across a large area in Africa, South Asia, and South-East Asia. This specific version of the dataset is organized based on admin 0 boundaries and is accessible in cloud-native geospatial formats like PMTiles, FlatGeobuf, and GeoParquet.

geoparquet
building footprints
pmtiles
flatgeobuf


Global 100m resolution raster maps of the Terrestrial Human Footprint (HFP-100) for 2017–2021, quantifying cumulative anthropogenic pressure on land surfaces. Pressure layers include built environments, croplands, pasture, population density, nighttime lights, roads, railways, and navigable waterways, combined into a standardized 0–50 score. Produced annually to monitor changing human impacts on terrestrial ecosystems worldwide.

land-use remote-sensing human-footprint biodiversity anthropogenic-pressure population-density land-cover

Dataset files (5 x .tif)

Source Cooperative product page

## README.md
# Global 100m Terrestrial Human Footprint (HFP-100) v1.2

This repository contains the global results for the Human Footprint 100m product for 2017, 2018, 2019, 2020 and 2021. The Human Footprint (HFP) is used to monitor changing anthropogenic pressures on a yearly cadence based on land cover change, population density, nighttime lights, roads, railways, and navigable rivers.

This version 1.2 includes the years 2017-2021 based on land cover from the Esri/Impact Observatory 10m Annual Land Use Land Cover (9-class) dataset. Version 1 of this dataset covering the years 2015-2020 can be found at https://datadryad.org/stash/dataset/doi:10.5061/dryad.ttdz08m1f.

This dataset was created by combining data on human pressures across the period 2017-2021 to map: 1) Land cover change (built environments, crop lands, and pasture lands), 2) population density, 3) electric infrastructure, 4) roadways, 5) railways, and 6) navigable waterways. Each pressure layer is assigned a score relative to its level of human pressure, then computed into a standardized scale of 0-50 as the sum of all pressure layers. Pressures are not mutually exclusive, rather the co-occurrence of pressures is intended to identify the greatest levels of human impact. The majority of layers cover the complete time period of 2017-2021, however pressures from pasture, roads, and railways are treated as static due to limitations in the input datasets.

Overall methodology is based on the following:
 - B. A. Williams, O. Venter, J. R. Allan, S. C. Atkinson, J. A. Rehbein, M. Ward, M. Di Marco, H. S. Grantham, J. Ervin, S. J. Goetz, A. J. Hansen, P. Jantz, R. Pillay, S. Rodríguez-Buriticá, C. Supples, A. L. S. Virnig, J. E. M. Watson, Change in Terrestrial Human Footprint Drives Continued Loss of Intact Ecosystems. One Earth. 3, 371–382 (2020).
 - E. W. Sanderson, M. Jaiteh, M. A. Levy, K. H. Redford, A. V. Wannebo, G. Woolmer, The Human Footprint and the Last of the Wild: The human footprint is a global map of human influence on the land surface, which suggests that human beings are stewards of nature, whether we like it or not. BioScience. 52, 891–904 (2002).
 - O. Venter, E. W. Sanderson, A. Magrach, J. R. Allan, J. Beher, K. R. Jones, H. P. Possingham, W. F. Laurance, P. Wood, B. M. Fekete, M. A. Levy, J. E. M. Watson, Global terrestrial Human Footprint maps for 1993 and 2009. Sci. Data. 3, 160067 (2016).

Please see the following publication for more information: Gassert F., Venter O., Watson J.E.M., Brumby S.P., Mazzariello J.C., Atkinson S.C. and Hyde S., An Operational Approach to Near Real Time Global High Resolution Mapping of the Terrestrial Human Footprint. Front. Remote Sens. 4:1130896 doi: 10.3389/frsen.2023.1130896 (2023). https://www.frontiersin.org/articles/10.3389/frsen.2023.1130896/full

## Data information

HFP scores range from 0 to 50, with 0 representing virtually no human pressure and 50 representing severe human pressure on the environment. The data available here is multiplied by 1000 and converted to uint16 data type to reduce file sizes such that a value of 50000 corresponds to a score of 50. No data regions are coded as 65535.

## Code/Software

Data was produced using open source Python scripts and processed in the Microsoft Planetary Computer infrastructure. Scripts used to produce this data are available at https://gitlab.com/impactobservatory/dwi-humanfootprint.

## Suggested Citation

F. Gassert, O. Venter, J. E.M. Watson, S.P. Brumby, J.C. Mazzariello, S.C. Atkinson, S. Hyde. 2023: "Global 100m Terrestrial Human Footprint (HFP-100) v1.2"

## License

Creative Commons CC-BY-4.0

## Disclosure

Funding provided by Microsoft.

providers: role [producer, processor] for 'Impact Observatory' is an AI guess — Named as source of input land cover data and code repository host; assumed processor role based on GitLab repo under impactobservatory. Verify before flipping to _state: reviewed.

providers: role [producer] for 'Gassert et al. / study authors' is an AI guess — Corresponding authors of the methodology publication; assumed producers of the HFP-100 dataset. Verify before flipping to _state: reviewed.

providers: role [processor] for 'Microsoft' is an AI guess — Provided Planetary Computer infrastructure for processing; funding disclosed but no explicit role label given. Verify before flipping to _state: reviewed.


Global gridded raster data for estimating supply chain impacts related to deforestation, greenhouse gas emissions, natural ecosystem conversion, and biodiversity loss. Originally prepared for the LandGriffon service following international standards for reporting and target setting in the agriculture, forestry, and land use sector. Includes 100m-resolution impact layers and 1km-resolution annualized impact-per-hectare layers for five indicators across two time windows.

agriculture land-use deforestation conservation carbon-emissions biodiversity

Dataset files (11 x .tif)

Source Cooperative product page

## README.md
# Land, carbon and biodiversity data for supply chain impact calculation

This repository contains global gridded data for estimating supply chain impacts relating to deforestation, related greenhouse gas emissions, conversion of natural ecosystems, and loss of intact or high integrity ecosystems. These data were originally prepared for the LandGriffon service (https://landgriffon.com) following international standards and guidelines for reporting and target setting in the agriculture, forestry, and land use sector. In particular, they employ a spatial adaptation of the Statistical Land Use Change (sLUC) by land occupation accounting method in which responsibility for impacts are allocated across a 50km radius instead of across a jurisdiction.

Methodology and usage notes: https://www.biorxiv.org/content/10.1101/2023.11.01.565036v1

Code: https://github.com/Vizzuality/lg-landscape-indicators-processing

Five indicators are included, with two layers per indicator: the impact, and annualized impact per hectare of human land use.

**Impact layers (100m)**

| file | indicator | units |
| --- | --- | --- |
| deforest_100m_cog.tif | Deforestation (2002-2022) | portion of cell |
| deforest_carbon_100m_cog.tif | Deforestation carbon (2002-2022) | tC/ha |
| natcrop_expansion_100m_cog.tif | Cropland expansion in natural ecosystems (2020-2022) | portion of cell |
| natcrop_reduction_100m_cog.tif | Cropland reduction in natural ecosystems (2020-2022) | portion of cell |
| natcrop_bii_100m_cog.tif | Biodiversity Intactness score of converted areas | score |
| natcrop_fii_100m_cog.tif | Forest Landscape Integrity Index of converted areas | score |

**Annualized impact per hectare of human land use (1000m)**

| file | indicator | units |
| --- | --- | --- |
| deforest_by_human_lu_50km_1000m_cog.tif | Deforestation | ha/yr / haLU |
| deforest_carbon_by_human_lu_50km_1000m_cog.tif | Deforestation carbon emissions | tCO2/yr / haLU |
| natural_crop_net_conversion_by_human_lu_50km_1000m_cog.tif | Net cropland expansion in natural lands | ha/yr / haLU |
| natural_crop_bii_loss_by_human_lu_50km_1000m_cog.tif | Loss of forest integrity to cropland expansion | score / haLU |
| natural_crop_flii_loss_by_human_lu_50km_1000m_cog.tif | Loss of biodiversity intactness to cropland expansion | score / haLU |

Of note, these datasets are not constrained to areas of current human land use, e.g. such that land use in remote wilderness may be assigned a maximum impact score.

See https://landgriffon.com/methodology for techniques for combining these results with land footprint data and production data to calculate the impacts of raw materials sourced from for local, regional, or national supply-sheds. For example, to calculate the deforestation footprint of sourcing 100 tons of soy from Argentina, a user could first calculate the average hectares of deforestation per hectare of land use for all soy producing areas in Argentina, and then multiply that value by the land footprint of 100 tons of soy production.

### License

(c) Vizzuality 2023 CC-BY. This data is made available under a Creative Commons Attribution 4.0 International [license](https://creativecommons.org/licenses/by/4.0/).

### Suggested citation

Francis Gassert, Biel Stela, Elena Palao Martinez, and Michael Harfoot. 2023. “Land, Carbon and Biodiversity Data for Supply Chain Impact Calculations.” Preprint. BioRxiv, 2023.11.01.565036. https://doi.org/10.1101/2023.11.01.565036.


Wadhwani AI Pest Management Open Data for Pink Bollworms and American Bollworms

This repository acts as a gateway to the Wadhwani Institute for Artificial Intelligence (Wadhwani AI) pest data set. The data set is primarily intended for training pest recognition models. Such models are used by Wadhwani AI to help cotton farmers make better pest management decisions.
Please view our github repository (https://github.com/WadhwaniAI/pest-management-opendata) which provides detailed documentation about the data and also contains scripts to format the data for consumption by popular modelling frameworks.

computer vision
agriculture
cotton
pink bollworms
american bollworms
wadhwaniai


DEM-Terrain

H3-indexed terrain derivatives computed from the GEDTM-30m global 30m resolution digital elevation model. Includes elevation, slope, aspect, terrain ruggedness index, and topographic position index across ten H3 resolutions with ~10.5 billion cells. Available in Apache Parquet format (v2: BIGINT H3 index; v1: legacy VARCHAR with geometry columns).

remote-sensing digital-elevation-model terrain-derivatives elevation h3-hexagons

Dataset files (20 x .parquet)

Source Cooperative product page

## README.md
# Global DEM Terrain Derivatives

H3-indexed terrain data from [GEDTM-30m](https://stac.openlandmap.org/gedtm-30m/collection.json) in Apache Parquet format. Ten H3 resolutions (1–10), one file per resolution, sorted by `h3_index`. Available in two versions: **v2 (recommended)** with BIGINT `h3_index` and no geometry columns, and **v1 (legacy)** with VARCHAR `h3_index` and [native Parquet 2.11+ GEOMETRY](https://github.com/apache/parquet-format/blob/master/Geospatial.md).

| | |
|---|---|
| **Source** | GEDTM-30m (OpenGeoHub), 30m resolution, global land coverage |
| **Format** | Apache Parquet — v2: BIGINT h3_index, no geometry; v1: VARCHAR h3_index, native GEOMETRY (DuckDB 1.5) |
| **CRS** | EPSG:4326 (WGS 84) |
| **License** | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) by [walkthru.earth](https://walkthru.earth/links) |
| **Code** | [walkthru-earth/dem-terrain](https://github.com/walkthru-earth/dem-terrain) |

## Quick Start

### v2 (recommended)

```sql
-- DuckDB — no spatial extension needed for v2
INSTALL h3 FROM community; LOAD h3;
INSTALL httpfs; LOAD httpfs;
SET s3_region = 'us-west-2';

SELECT h3_index,
       h3_cell_to_lat(h3_index) AS lat,
       h3_cell_to_lng(h3_index) AS lng,
       elev, slope, aspect
FROM read_parquet('s3://us-west-2.opendata.source.coop/walkthru-earth/dem-terrain/v2/h3/h3_res=5/data.parquet')
WHERE h3_index BETWEEN ? AND ?  -- row group statistics prune efficiently
ORDER BY elev DESC
LIMIT 20;
```

For deck.gl H3HexagonLayer (needs hex string):

```sql
SELECT h3_h3_to_string(h3_index) AS h3_hex, elev
FROM read_parquet('s3://us-west-2.opendata.source.coop/walkthru-earth/dem-terrain/v2/h3/h3_res=5/data.parquet')
```

```python
# Python (v2)
import duckdb

con = duckdb.connect()
con.install_extension("h3", repository="community"); con.load_extension("h3")
for ext in ("httpfs",):
    con.install_extension(ext); con.load_extension(ext)
con.sql("SET s3_region = 'us-west-2'")

df = con.sql("""
    SELECT h3_index,
           h3_cell_to_lat(h3_index) AS lat,
           h3_cell_to_lng(h3_index) AS lng,
           elev, slope, aspect, tri, tpi
    FROM read_parquet(
        's3://us-west-2.opendata.source.coop/walkthru-earth/dem-terrain/v2/h3/h3_res=5/data.parquet'
    )
    ORDER BY elev DESC
    LIMIT 100
""").fetchdf()
```

### v1 (legacy — requires `INSTALL spatial`)

```sql
-- DuckDB
INSTALL spatial; LOAD spatial;
INSTALL httpfs;  LOAD httpfs;
SET s3_region = 'us-west-2';

SELECT h3_index, elev, slope, aspect, tri, tpi
FROM read_parquet('s3://us-west-2.opendata.source.coop/walkthru-earth/dem-terrain/v1/h3/h3_res=5/data.parquet')
WHERE lat BETWEEN 35 AND 45
  AND lon BETWEEN -10 AND 5
ORDER BY elev DESC
LIMIT 20;
```

## Files

### v2 (recommended)

```
walkthru-earth/dem-terrain/
  v2/h3/
    h3_res=1/data.parquet       12 KB           223 cells
    h3_res=2/data.parquet       57 KB         1,546 cells
    h3_res=3/data.parquet      373 KB        10,851 cells
    h3_res=4/data.parquet      2.5 MB        76,135 cells
    h3_res=5/data.parquet       17 MB       533,062 cells
    h3_res=6/data.parquet      115 MB     3,730,922 cells
    h3_res=7/data.parquet      783 MB    26,115,785 cells
    h3_res=8/data.parquet      5.3 GB   182,814,924 cells
    h3_res=9/data.parquet     36.1 GB 1,279,700,961 cells
    h3_res=10/data.parquet   244.7 GB 8,957,910,337 cells
```

### v1 (legacy)

```
walkthru-earth/dem-terrain/
  v1/h3/
    h3_res=1/data.parquet  ...  h3_res=10/data.parquet
```

Same cell counts and resolutions as v2.

**Total: 10,450,894,746 cells (~10.5 billion) in ~287 GB.** Compression: ZSTD level 3. Row groups: 1,000,000 rows.

## Schema

### v2 (recommended)

| Column | Type | Unit | Description |
|--------|------|------|-------------|
| `h3_index` | BIGINT | — | H3 cell ID (int64). Sorted for delta encoding compression and range-based queries. |
| `elev` | FLOAT | meters | Elevation above sea level |
| `slope` | FLOAT | degrees | Terrain slope (0 = flat, 90 = cliff) |
| `aspect` | FLOAT | degrees | Slope direction (0/360 = N, 90 = E, 180 = S, 270 = W) |
| `tri` | FLOAT | meters | Terrain Ruggedness Index — mean abs. elevation diff to 8 neighbors |
| `tpi` | FLOAT | meters | Topographic Position Index — elevation minus mean of 8 neighbors (+ridge, -valley) |

`geometry`, `lat`, and `lon` columns are removed in v2 — derive them via the DuckDB h3 extension: `h3_cell_to_lat(h3_index)`, `h3_cell_to_lng(h3_index)`, `h3_cell_to_boundary_wkt(h3_index)`.

### v1 (legacy)

| Column | Type | Unit | Description |
|--------|------|------|-------------|
| `h3_index` | VARCHAR | — | H3 cell ID (hex string) |
| `geometry` | GEOMETRY | — | Cell center point (native Parquet 2.11+ GEOMETRY, EPSG:4326) |
| `lat` | FLOAT | degrees | Cell center latitude |
| `lon` | FLOAT | degrees | Cell center longitude |
| `elev` | FLOAT | meters | Elevation above sea level |
| `slope` | FLOAT | degrees | Terrain slope (0 = flat, 90 = cliff) |
| `aspect` | FLOAT | degrees | Slope direction (0/360 = N, 90 = E, 180 = S, 270 = W) |
| `tri` | FLOAT | meters | Terrain Ruggedness Index — mean abs. elevation diff to 8 neighbors |
| `tpi` | FLOAT | meters | Topographic Position Index — elevation minus mean of 8 neighbors (+ridge, -valley) |

**Sample values** (res 5, Everest region):

| h3_index | elev | slope | aspect | tri | tpi |
|----------|------|-------|--------|-----|-----|
| 853c0317fffffff | 6,879 m | 40.8° | 134° | 327 m | 52 m |
| 853c03bbfffffff | 6,568 m | 23.0° | 221° | 223 m | 31 m |

## How It Works

Terrain derivatives are computed from the GEDTM-30m elevation raster:

| Derivative | Method | Reference |
|------------|--------|-----------|
| **Slope** | `arctan(√(dz/dx² + dz/dy²))` via central differences, pixel sizes converted to meters using latitude-dependent scale | Horn (1981) |
| **Aspect** | `atan2(-dz/dx, dz/dy)`, converted to 0–360° compass bearing | Horn (1981) |
| **TRI** | Mean absolute elevation difference across 3×3 neighborhood | Riley et al. (1999) |
| **TPI** | Center pixel minus mean of 3×3 neighborhood | Weiss (2001) |

Values at H3 cell centers are obtained by **bilinear interpolation** from the raster grid (scipy `RegularGridInterpolator`).

DEM is read at different resolutions depending on H3 level:

| H3 Res | DEM Sampling | Rationale |
|--------|-------------|-----------|
| 1–5 | ~500 m | Cells are 8–418 km; coarse DEM sufficient |
| 6–7 | ~110 m | Cells are 1.2–3.2 km; finer detail needed |
| 8–10 | ~30 m (native) | Cells are 66–461 m; full resolution |

## More Examples

### v2 queries

```sql
-- Range query on sorted BIGINT h3_index — row group min/max statistics prune efficiently
INSTALL h3 FROM community; LOAD h3;
INSTALL httpfs; LOAD httpfs;
SET s3_region = 'us-west-2';

SELECT h3_index,
       h3_cell_to_lat(h3_index) AS lat,
       h3_cell_to_lng(h3_index) AS lng,
       elev, slope
FROM read_parquet('s3://us-west-2.opendata.source.coop/walkthru-earth/dem-terrain/v2/h3/h3_res=7/data.parquet')
WHERE h3_index BETWEEN ? AND ?
ORDER BY elev DESC
LIMIT 100;

-- Query across all resolutions (Hive partitioning)
SELECT h3_res, h3_index, elev
FROM read_parquet(
    's3://us-west-2.opendata.source.coop/walkthru-earth/dem-terrain/v2/h3/h3_res=*/data.parquet',
    hive_partitioning = true
)
WHERE h3_res = 5
ORDER BY elev DESC
LIMIT 20;

-- DuckDB-WASM (browser) — use HTTPS URL
SELECT h3_index,
       h3_cell_to_lat(h3_index) AS lat,
       h3_cell_to_lng(h3_index) AS lng,
       elev, slope
FROM read_parquet(
    'https://data.source.coop/walkthru-earth/dem-terrain/v2/h3/h3_res=5/data.parquet'
)
ORDER BY elev DESC
LIMIT 100;
```

### v1 legacy queries (requires `INSTALL spatial`)

```sql
-- Spatial pushdown using native geometry bbox stats (no lat/lon filter needed)
SELECT h3_index, elev, slope
FROM read_parquet('s3://us-west-2.opendata.source.coop/walkthru-earth/dem-terrain/v1/h3/h3_res=7/data.parquet')
WHERE geometry && ST_MakeEnvelope(86.5, 27.5, 87.5, 28.5);

-- lat/lon filtering (v1 only — these columns exist in v1)
SELECT *
FROM read_parquet(
    's3://us-west-2.opendata.source.coop/walkthru-earth/dem-terrain/v1/h3/h3_res=*/data.parquet',
    hive_partitioning = true
)
WHERE h3_res = 5
  AND lat BETWEEN 27 AND 29 AND lon BETWEEN 86 AND 88;
```

## Geometry Format

**v2** drops the `geometry` column entirely. Coordinates are derivable from the BIGINT `h3_index` via the DuckDB h3 extension (`h3_cell_to_lat`, `h3_cell_to_lng`, `h3_cell_to_boundary_wkt`). This eliminates the need for `INSTALL spatial` and reduces file size.

**v1** retains the `geometry` column, written with `GEOPARQUET_VERSION 'BOTH'`, providing dual compatibility:

- **Native Parquet 2.11+ GEOMETRY logical type** — per-row-group bounding box statistics for spatial filter pushdown. Supported by DuckDB 1.5+, Apache Arrow (Rust), Apache Iceberg, GDAL 3.12+.
- **GeoParquet 1.0 `geo` file-level metadata** — backwards compatibility with older tools (QGIS, pyarrow, GeoPandas).

## Source

[GEDTM-30m](https://stac.openlandmap.org/gedtm-30m/collection.json) — Global Ensemble Digital Terrain Model at 30m resolution. Cloud Optimized GeoTIFF, 1,440,010 × 600,010 px, Int32 (decimeters, scale 0.1), EPSG:4326, global land coverage 65°S–85°N.

> Ho, Y., Grohmann, C. H., Lindsay, J., Reuter, H. I., Parente, L., Witjes, M., & Hengl, T. (2025). GEDTM30: global ensemble digital terrain model at 30 m and derived multiscale terrain variables. *PeerJ*, 13, e19673. [doi:10.7717/peerj.19673](https://doi.org/10.7717/peerj.19673)
>
> Dataset: [doi:10.5281/zenodo.14900181](https://doi.org/10.5281/zenodo.14900181)

## License

This dataset is licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) by [walkthru.earth](https://walkthru.earth/links). The source [GEDTM-30m](https://doi.org/10.5281/zenodo.14900181) is licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) by [OpenGeoHub](https://opengeohub.org/).


Walkthru-Indices

A suite of four interoperable global planetary indices — terrain morphology (elevation, slope, aspect, TRI, TPI), building density and volume, SSP2 population projections (2025–2100), and near-real-time weather forecasts — derived from scientific source datasets and aggregated onto a unified H3 hexagonal grid at multiple resolutions. All indices are stored as Apache Parquet files partitioned by H3 resolution and share a common h3_index key, enabling cross-dataset spatial joins in a single SQL statement. Source data includes GEDTM-30m elevation, Global Building Atlas (2.75 billion building polygons), WorldPop SSP rasters, and NOAA AI-NWP forecasts.

h3 global-coverage building-footprints digital-elevation-model numerical-weather-prediction demographics topography elevation

city_index.parquet

country

[placename] Country name for the address record.

region

[placename] Sub-national region or administrative area name.

city

[placename] City name for the address record.

addresses-index/v1/city_index

region

[placename] Sub-national region name for the city index record.

city

[placename] City name for the city index record.

[latlon] Western bounding longitude of city extent (integer degrees × 10⁶).

[latlon] Eastern bounding longitude of city extent (integer degrees × 10⁶).

[latlon] Southern bounding latitude of city extent (integer degrees × 10⁶).

[latlon] Northern bounding latitude of city extent (integer degrees × 10⁶).

country

[placename] Country name for the city index record.

4 files sampled for this table

weather

[spatial_index] H3 cell identifier (BIGINT int64) encoding the geographic location of each weather record.

5 files sampled for this table

Source Cooperative product page

## README.md
# walkthru-earth / indices

Open planetary indices that turn raw scientific data into a single, H3-hexagonal grid anyone can query with one SQL statement. Every index shares the same cell ID (`h3_index`), so joining terrain + buildings + population + weather is a one-line `JOIN ON h3_index`.

**Why?** Cities shape how people feel — but the data to prove it is scattered across dozens of agencies, formats, and projections. We unify it into one queryable, open layer so researchers, planners, and communities can finally see the full picture. Read more: [walkthru.earth/hormones-cities](https://walkthru.earth/hormones-cities)

```mermaid
flowchart LR
    subgraph Raw["Raw sources"]
        DEM_RAW["GEDTM-30m\n(30 m GeoTIFF)"]
        GBA_RAW["Global Building Atlas\n(2.75 B polygons)"]
        WP_RAW["WorldPop SSP\n(1 km rasters)"]
        NOAA_RAW["NOAA AI-NWP\n(0.25° NetCDF)"]
    end

    subgraph H3["H3 conversion"]
        direction TB
        DT["dem-terrain\nelev · slope · aspect · TRI · TPI"]
        BI["walkthru-building-index\ndensity · height · volume"]
        PI["walkthru-pop-index\n2025 → 2100 projections"]
        WI["walkthru-weather-index\ntemp · wind · precip · pressure"]
    end

    DEM_RAW --> DT
    GBA_RAW --> BI
    WP_RAW --> PI
    NOAA_RAW --> WI
    DT -. topo corrections .-> WI

    subgraph S3["s3://…/walkthru-earth/"]
        T[dem-terrain/v2/h3/]
        B[indices/building/v2/h3/]
        P[indices/population/v2/]
        W[indices/weather/]
    end

    DT --> T
    BI --> B
    PI --> P
    WI --> W

    T -. JOIN h3_index .-> Q((Query))
    B -. JOIN .-> Q
    P -. JOIN .-> Q
    W -. JOIN .-> Q
```

## Indices

| Index | What | H3 Res | Update | Size |
|-------|------|--------|--------|------|
| **[dem-terrain](https://source.coop/walkthru-earth/dem-terrain)** | Elevation, slope, aspect, TRI, TPI from GEDTM-30m | 1–10 | One-time | ~287 GB, 10.5 B cells |
| **[walkthru-building-index](https://source.coop/walkthru-earth/indices)** | Density, height, footprint, volume from 2.75 B buildings | 3–8 | One-time | ~2.6 GB, 53 M cells |
| **[walkthru-pop-index](https://source.coop/walkthru-earth/indices)** | SSP2 projections 2025–2100 (16 timesteps) from WorldPop | 1–8 | One-time | ~4.5 GB, 44 M cells |
| **[walkthru-weather-index](https://source.coop/walkthru-earth/indices)** | Temperature, wind, precip, pressure + topo corrections | 0–5 | Every 12 h | ~3.6 GB / forecast |

All datasets: [Apache Parquet](https://parquet.apache.org/), ZSTD compression, sorted by `h3_index`.
- **v2 (recommended):** `h3_index` is BIGINT (int64) — no `geometry`, `lat`, `lon`, or `area_km2` columns (all derivable from the H3 index via DuckDB's h3 extension). Sorted by int64 for better compression and fast range-based spatial queries.
- **v1 (legacy):** `h3_index` is VARCHAR (hex string) with native Parquet 2.11+ GEOMETRY and `lat`/`lon`/`area_km2` columns.

## Quick start

```sql
-- DuckDB: join buildings + population in one query (v2 — BIGINT h3_index, no geometry)
INSTALL httpfs;  LOAD httpfs;
INSTALL h3 FROM community; LOAD h3;
SET s3_region = 'us-west-2';

SELECT b.h3_index,
       h3_h3_to_string(b.h3_index) AS h3_hex,          -- hex string for deck.gl
       h3_cell_to_lat(b.h3_index)  AS lat,              -- derive lat on the fly
       h3_cell_to_lng(b.h3_index)  AS lon,              -- derive lon on the fly
       b.building_count, b.building_density,
       p.pop_2025, p.pop_2050,
       (b.building_count::FLOAT / NULLIF(p.pop_2025, 0))::DECIMAL(6,3) AS bldg_per_person
FROM read_parquet('s3://us-west-2.opendata.source.coop/walkthru-earth/indices/building/v2/h3/h3_res=5/data.parquet') b
JOIN read_parquet('s3://us-west-2.opendata.source.coop/walkthru-earth/indices/population/v2/scenario=SSP2/h3_res=5/data.parquet') p
  ON b.h3_index = p.h3_index                            -- BIGINT join — fast
WHERE p.pop_2025 > 10000
ORDER BY bldg_per_person DESC
LIMIT 10;
```

## S3 layout

```
s3://us-west-2.opendata.source.coop/walkthru-earth/
├── dem-terrain/
│   ├── v1/h3/h3_res={1-10}/data.parquet          ← legacy (VARCHAR h3, has geometry/lat/lon)
│   └── v2/h3/h3_res={1-10}/data.parquet          ← recommended (BIGINT h3, lean schema)
└── indices/
    ├── building/
    │   ├── v1/h3/h3_res={3-8}/data.parquet        ← legacy
    │   ├── v2/h3/h3_res={3-8}/data.parquet        ← recommended
    │   └── globalbuildingatlas/zoomlevel={2-14}/data*.parquet
    ├── population/
    │   ├── v1/scenario=SSP2/h3_res={1-8}/data.parquet  ← legacy
    │   └── v2/scenario=SSP2/h3_res={1-8}/data.parquet  ← recommended
    └── weather/
        └── model=GraphCast_GFS/date=YYYY-MM-DD/hour={0,12}/h3_res={0-5}/data.parquet
```

## License

[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) by [walkthru.earth](https://walkthru.earth/links)

extent.spatial.bbox: omitted; spatial reference is categorical via [weather.h3_index (spatial_index), addresses-index/v1.country (placename), addresses-index/v1.region (placename), addresses-index/v1.city (placename), addresses-index/v1/city_index.country (placename), addresses-index/v1/city_index.region (placename), addresses-index/v1/city_index.city (placename), addresses-index/v1/city_index.bbox_min_lon_e6 (latlon), addresses-index/v1/city_index.bbox_max_lon_e6 (latlon), addresses-index/v1/city_index.bbox_min_lat_e6 (latlon), addresses-index/v1/city_index.bbox_max_lat_e6 (latlon)]. Add bbox once the placename → bbox gazetteer covers it.


OpenSensor.Space

Continuous time-series environmental and weather readings from an IoT sensor network in the Cairo/Nile Delta region of Egypt. Data is collected via Raspberry Pi devices with Enviro+ sensors, capturing temperature, pressure, humidity, gas concentrations (oxidised, reducing, NH3), ambient light, and particulate matter (PM1, PM2.5, PM10) at multiple particle size fractions, partitioned by station and date. A companion asset provides geospatial suitability analysis for air quality monitoring station placement.

air-quality environmental-monitoring time-series weather particulate-matter sensor-network egypt

enviroplus

40 files sampled for this table

suitability_analysis_of_aq.parquet

geom

[geometry] OGC CRS84 geometry for candidate air quality monitoring station suitability locations

Source Cooperative product page

extent.spatial.bbox: omitted; spatial reference is categorical via [share.geom (geometry)]. Add bbox once the placename → bbox gazetteer covers it.