Every store feeding the engine — what it is, how it's populated, and whether it's current.
The serving index — one fused vector doc per scored developer, queried by /v1/search and /v1/developer.
How it works —Written by the pipeline index step after signals compute; each doc carries output_power, gem_score, and the full signal vector.
Per-repository vectors for repo-level search and matching.
How it works —Written by the repo index step; empty until repository indexing runs.
The upstream hourly GitHub event firehose — every push/PR/issue/star, for every developer, free.
How it works —Public hourly archives (gharchive.org). A cron ingests them into the events_ledger lake table — not yet running, so this is the intended feed, not a live consume.
Public GitHub-events dataset (3.2B events) — the free population breadth + benchmark source.
How it works —Queried anonymously (play user) to compute the benchmark grids over a 10% actor sample. ~10-day lag; no ingest, we read it live.
Our copy of the GH Archive firehose in the R2 Data Catalog — the population-scale activity ledger behind liveness signals.
How it works —Appended by the GH Archive cron into the R2 Data Catalog. Empty until firehose ingest runs — benchmarks read ClickHouse directly today.
LinkedIn-native enrichment — Swarm profiles joined to GitHub logins (name, company, work email, education).
How it works —Ingested from the Swarm R2 delta files, filtered to the github-handle slice (~12.7% of Swarm carries a handle).
Per-developer commit provenance — author email + repo per commit, the native reachability + employer signal.
How it works —Extracted during the deep GitHub crawl; one row per (login, repo, commit date).
The scored signal vectors — ~78 signals per developer, cohort-percentiled into gem_score.
How it works —Computed by the signals engine per crawl; feeds gem_score and the serving index.
The crawl queue — logins pending deep extraction, tiered by priority.
How it works —Seeded from discovery + graph expansion, drained as the crawler works it. Empty = idle (nothing queued).
The GitHub API token pool — rotated credentials powering the crawler.
How it works —Managed in KV; each token is probed for live / rate-limited status (see the token pool panel).