Upgrades the quarterly workflow from a checklist to an operating blueprint. The skeleton is TOGAF (business → data → application → technology → operations); the mechanics come from the modern-data-architecture playbook — an ingestion layer, an immutable batch store, change-data-capture deltas instead of full re-scans, a Lambda speed layer for mid-quarter events, serving-layer regeneration, and backpressure rules. Every mapped concept points at a real file, source, or command in this repo.
The mission is unchanged — measure whether import substitution is working, gate by gate — but verdicts decay: both FY25 greens reversed within one year. The quarterly cadence exists to re-measure six KPIs before anyone cites a stale one:
| Quarterly KPI | Baseline (2026-07) | Gate |
|---|---|---|
| Trade deficit, level + growth | −$333.2bn, widened 17.5%/yr | G7 |
| PLI disbursal % of outlay | ₹28,748cr ≈ 15%; ACC still 0 | G5 |
| IEM implementation ratio | 82% (2024) → 109.9% (2025) | G4 |
| EC grant flow + EC-IEM correlation | 198 → 734 → 1,102/yr; r = 0.96 | G3 |
| Sector verdict distribution | 1 green / 6 amber / 3 red (FY26) | G7 |
| FDI equity inflow | $58.85bn (+18%) | G6 |
114k EC proposals in one call; 2.1M trade rows in the companion warehouse; small enough for files, big enough to need aggregates.
JSON APIs, HTML releases, proxied PDFs, JS single-page apps, XLS/CSV — each source keeps its documented access method.
Nightly (AHIDF) → monthly (Exim, trade) → quarterly (FDI newsletter) → session (Parliament) → annual (ministry reports). Refresh follows native cadence.
Registered-office FDI, dead IEM series, anonymous beneficiaries, bot-blocked portals — defects are first-class data, restated wherever used.
The playbook's CDC principle, applied to documents: capture what changed since the last snapshot, and obey the dedup mandate — PIB re-announces the same cumulative figures across releases, so the latest cumulative supersedes, is never summed, and the trajectory is logged.
| Gate | Source | Quarterly delta method | Validation gate |
|---|---|---|---|
| G1Policy | PIB ministry pages | New PRIDs since last refresh; cumulative supersedes | PRID on every figure |
| G2IEM A | DPIIT PDFs / Parliament | New filings PDF, else sansad answers (monthly series is dead) | Grand-Total cross-check |
| G3Land+EC | PARIVESH API | Full re-pull, diff proposal IDs: new grants, ToR→EC moves | row monotonicity + 5 flagship spot-checks |
| G4IEM B | DPIIT / Parliament | New year appended; ratio + correlation recomputed | same-year ratio basis |
| G5Disbursal | PIB + Parliament answers | Per-scheme totals; Parliament is the laggard-scheme channel | three-numbers rule |
| G6FDI | DPIIT quarterly newsletter | The one natively quarterly source; sector+state tables re-extracted | registered-office defect stated |
| G7Trade | TRADESTAT + Exim | New monthly cumulative; provisional→final in Apr–May | cross-validation ≤ 0.3% |
| # | Step | What happens | Tooling |
|---|---|---|---|
| 1 | Delta ingest | CDC table top-to-bottom; dated snapshots written; empty deltas logged as findings | curl · browser pane · PARIVESH API · agents |
| 2 | Verify | Cross-checks; schema sanity on new JSONs; failures block the quarter | snippet checks |
| 3 | Recompute | Verdicts, correlation, ratio, open grade revisions | python (in-bulletin methods) |
| 4 | Bulletins | Data arrays + retrieved dates advanced; new findings sections | charts/*.html (data-JSON-first) |
| 5 | Master report | Master JSON + corrections ledger + manifest → docx/pdf regenerated | report_synthesizer.py --manifest |
| 6 | Deck + compendium | Slides re-captured; compendium re-paginated | build_deck.sh · build_compendium.py (Homebrew python — SIP blocks system Python's pango) |
| 7 | Publish | Commit, push, tag the quarter; Pages serves automatically | git · quarterly-dashboard-refresh.yml |
Refresh months July · October · January · April — trailing the DPIIT quarterly newsletter and Parliament sessions, with full-FY trade figures landing in the April cycle. The cycle maps onto SAFe for Government's cadence model: each refresh is a Program Increment (plan the deltas → execute → demo the regenerated compendium as the "working system"); the backlog below is the long-lived prioritized backlog replacing project-based thinking; quality/compliance run inside the flow as the verification gates, not end-stage inspection; and the corrections ledger plays the governance-for-accountability role stage-gate documents would.
Run the CDC table; write snapshots; a source with no delta is logged as a finding, not skipped.
Cross-checks, KPI recomputation, draft revisions with old → new shown.
Bulletins → master report → deck → compendium; visual pagination check.
Push, tag, corrections ledger, next-quarter watch-list (beyond-E20 decision due 31.10.2026; flat-10% tariff expiry).
| Success metric, checked every refresh |
|---|
| All 7 gates re-measured with a dated snapshot each |
| Zero silent changes — every revision displayed old → new |
| TRADESTAT-vs-Exim within 0.3% before verdicts publish |
| Compendium + deck rebuilt; page/slide counts recorded |
| Every bulletin's retrieved date advanced to the quarter |
| HIGH | parivesh_diff.py | Scripted API re-pull + proposal-ID diff — turns the largest manual delta into one command |
| SHIPPED | pib_prid_sweep | Shipped 2026-07-19 as scripts/pib_index.py — day-wise SQLite index (data/pib_index.sqlite) of releases by ministry + factsheets + cited-PRID seeding; quarterly delta = --update --factsheets --seed-cited |
| MED | JSON schema check in CI | Required keys (purpose, retrieved, sources) validated at the ingestion boundary |
| MED | FDI newsletter fetcher | The one natively quarterly source shouldn't need a browser session |
| LOW | Freshness dashboard | Per-gate last-snapshot date + delta size, visible to readers |