Benthic Data Provenance (BDP)

Every dataset published here is accompanied by a signed manifest: a JSON document that states who built the dataset, which code built it, what shape the data is in, and where it can be reached. The manifest is signed with an Ed25519 key. Anyone can verify it without asking benthic.io for permission, without an API key, and without trusting this server.

The point is auditability. If a dataset claims to be the output of a public ETL pipeline, you should be able to check that claim yourself — read the pipeline source, read the manifest, confirm the signature, and decide for yourself whether the two match.

BDP is transport-agnostic. A manifest describes endpoints, and an endpoint may be a live query API or a static file. Today benthic.io publishes a single live PostgREST endpoint per dataset. Parquet and GeoParquet transports are defined in the schema and will be added without any change to the protocol.

Discovery

Start at the index and follow the chain. Every step is a plain JSON file over HTTPS.

# 1. the discovery root: what collections exist here
curl https://benthic.io/bdp/index.json

# 2. a collection: its members, their pinned hashes, and how they join
curl https://benthic.io/bdp/ngopen/collection.json

# 3. a dataset manifest: full schema, provenance, endpoints, signature
curl https://benthic.io/bdp/ngopen/usaspending/manifest.json

Manifests carry a complete column-level schema, so some are large. For cheap discovery each one has a summary alongside it, which lists relations and column counts and pins the hash of the full manifest:

curl https://benthic.io/bdp/ngopen/samer/manifest.summary.json

Read the summary, decide whether the dataset is interesting, then fetch and verify the full manifest. The summary is deliberately unsigned — it makes no claim of its own, it only points at a document that does.

Published documents

Dataset manifests, all signed:

Verifying a signature

A manifest is signed over the SHA-256 digest of its own canonical form, with the signature block removed. Canonicalization is RFC 8785 (JCS). The digest is recorded in the manifest as payload_hash, which makes it a standalone content address you can log, publish, and compare across mirrors.

import base64, hashlib, json, urllib.request
import rfc8785
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PublicKey

url = "https://benthic.io/bdp/ngopen/samer/manifest.json"
doc = json.load(urllib.request.urlopen(url))

sig = doc.pop("cryptographic_signature")          # detach, do not null it
digest = hashlib.sha256(rfc8785.dumps(doc)).digest()

assert digest.hex() == sig["payload_hash"]         # content matches its hash

Ed25519PublicKey.from_public_bytes(
    base64.b64decode(doc["author_pubkey"])
).verify(base64.b64decode(sig["signature_base64"]), digest)

print("verified:", doc["dataset_name"])

Two things trip people up:

• author_pubkey is a raw 32-byte Ed25519 public key in standard Base64 — not ssh-ed25519 wire format, not PEM SPKI.
• The signature covers the raw 32 digest bytes, never the hex string.

Or use the reference tool:

pip install git+https://github.com/benthic-io/bdp
curl -s https://benthic.io/bdp/ngopen/samer/manifest.json | bdp verify -

Provenance is graded

Being honest about where data comes from matters more than making every relation look equally authoritative. Each relation in a manifest declares one of three provenance values:

Relations also declare queryable. A large upstream source table may appear in a manifest as the provenance root of a derived view while being marked unreachable through the API — visible as lineage, honestly marked as not directly queryable.

Manifests additionally carry etl_provenance.migration_status. All five NGOpen datasets are currently pending: the pipelines are published and runnable, but each is being migrated and re-verified one at a time. As each migration completes, that manifest is re-signed and the status advances. The git history of these manifests is the audit trail.

Source

• Provenance specification and reference tooling — github.com/benthic-io/bdp
• The pipelines that build these datasets — github.com/benthic-io/ngopen-bdp-pipelines

Questions, corrections, and criticism are all welcome at brian@benthic.io.