Incremental sync

If you're ingesting Pulse data into your own systems, you don't want to re-crawl every dossier on every run. The company and mining-asset searches expose two watermark columns that let you sweep "what changed since my checkpoint" — and a second, slower loop detects what left the universe, which watermarks can never show you. A reliable mirror runs both.

The two watermarks

Both POST /v1/companies/search/ and POST /v1/mining-assets/search/ expose these as ordinary filterable, sortable columns:

• updated_at — the entity row itself changed (name, status, profile fields). It advances on entity-level changes only — not when related data like production summaries or market snapshots change.
• data_refreshed_at — the entity's derived data was recomputed. It advances when production, reserves, economics, drill summaries, or market-data snapshots change. Refreshes are batched, so this can lag the underlying change by up to about an hour — bounded staleness, never a missed change. It's null for an entity whose derived data has never been computed; the updated_at arm of the sweep covers those.

Neither timestamp alone is a complete change signal; together they are. Sweep on both, combined with "logic": "any".

The change sweep

Filter both watermarks against your checkpoint, page through the results with an explicit sort, and re-fetch the dossier (and any sub-resources you mirror) of every entity returned:

{
  "filters": [
    { "field": "updated_at", "operator": "gte", "value": "2026-08-01" },
    { "field": "data_refreshed_at", "operator": "gte", "value": "2026-08-01" }
  ],
  "logic": "any",
  "columns": "name,updated_at,data_refreshed_at",
  "sort": "updated_at",
  "limit": 100,
  "offset": 0
}

Advance offset by limit while truncated is true (see Pagination and enumeration). Always pass an explicit sort when paging a sweep: without one, rows fall back to a default ordering that isn't stable while rows are being updated underneath you — sorting on the watermark you're checkpointing is the natural choice.

When the sweep completes, checkpoint on the maximum value seen across both columns, and start the next sweep from there. Using gte (not gt) means a boundary row can appear in two consecutive sweeps — re-fetching an entity twice is harmless, missing one is not.

Both columns are indexed, so watermark sweeps are cheap to serve even over the full universe. They're returned in the default column set like any other column; passing an explicit narrow columns (as above) keeps sweep rows small, which also keeps you well inside the daily row quota.

Detecting departures

The watermarks answer "what changed", never "what left". An entity that is deactivated — acquired, dissolved, delisted — drops out of the searchable universe at the moment it changes, so no sweep window ever returns it again. A mirror built from watermark sweeps alone would show a departed entity as live forever.

If your mirror is scoped by an industry filter, a company that changes industry leaves your universe the same way, even though it stays in the endpoint's.

Departures are detected by enumeration instead. On a slower cadence than the change sweep (weekly is usually enough), re-enumerate the universe — "filters": [], paged with an explicit sort — and retire every local id the new enumeration no longer contains:

departed = previous_enumeration_ids − current_enumeration_ids

This diff is the only reliable departure detector. Re-fetching a departed entity's dossier is not a substitute, and the two resources don't even behave alike once an entity has departed:

• A departed mining asset's dossier stops resolving and returns the standard error envelope.
• A deactivated company's dossier keeps returning 200 — its name carries an (INACTIVE) suffix as the in-band signal, not an error status.

The sweep universe

A sweep only ever returns entities inside the search endpoint's base population — active companies and active, verified mining assets, the same scope bare enumeration returns (see Enumerating the universe). Industry is not part of that scope. A mining-only mirror must carry an explicit industry filter on both loops — the enumeration and the change sweep — or it will take in companies from other industries.

Putting it together

1. Bootstrap: enumerate the universe ("filters": []), record all ids, fetch each entity's dossier and sub-resources, and set your checkpoint to the max watermark seen.
2. Change loop (as often as you like): run the two-watermark sweep from your checkpoint; re-fetch what it returns; advance the checkpoint.
3. Departure loop (slower cadence): re-enumerate, diff ids against your mirror, retire what's gone.

Go deeper

• Pagination and enumeration — paging mechanics and the enumeration scope both loops rely on.
• Dataset sub-resources — the per-entity datasets worth re-fetching when a sweep flags an entity.
• Rate limits and errors — budget your sweep cadence against your organization's quota.