pg_stream — Project Roadmap

Last updated: 2026-02-26 Current version: 0.1.1

For a concise description of what pg_stream is and why it exists, read ESSENCE.md — it explains the core problem (full REFRESH MATERIALIZED VIEW recomputation), how the differential dataflow approach solves it, the hybrid trigger→WAL CDC architecture, and the broad SQL coverage, all in plain language.

Overview

pg_stream is a PostgreSQL 18 extension that implements streaming tables with incremental view maintenance (IVM) via differential dataflow. All 13 design phases are complete. This roadmap tracks the path from pre-release to 1.0 and beyond.

 We are here
     │
     ▼
 ┌─────────┐   ┌─────────┐   ┌──────────┐   ┌──────────┐   ┌──────────┐   ┌──────────┐
 │  0.1.0  │──▶│  0.2.0  │──▶│  0.3.0   │──▶│  0.4.0   │──▶│  1.0.0   │──▶│  1.x+    │
 │ Pre-    │   │ Correct-│   │ Prod-    │   │ Observ-  │   │ Stable   │   │ Scale &  │
 │ release │   │ ness    │   │ ready    │   │ ability  │   │ Release  │   │ Ecosystem│
 └─────────┘   └─────────┘   └──────────┘   └──────────┘   └──────────┘   └──────────┘

v0.1.0 — Released (2026-02-26)

Status: Released — all 13 design phases implemented.

Core engine, DVM with 21 OpTree operators, trigger-based CDC, DAG-aware scheduling, monitoring, dbt macro package, and 1,300+ tests.

See CHANGELOG.md for the full feature list.

Late additions (pre-March 1st)

Low-risk, high-value items pulled forward from v0.2.0:

ItemDescriptionEffortRef
F4WAL decoder: pgoutput message parsing edge cases2–3hSQL_GAPS_7.md G3.3
F7Document JOIN key column change limitations1–2hSQL_GAPS_7.md G4.2
F11Keyless table duplicate-rows: document known behavior1hSQL_GAPS_7.md G7.1
F14CUBE explosion guard (reject oversized CUBE grouping sets)1hSQL_GAPS_7.md G5.2

v0.2.0 — Correctness & Stability

Goal: Close all critical and high-priority gaps to reach a provably correct baseline. No new features — only fixes, verification, and test coverage.

Tier 0 — Critical (must-fix)

ItemDescriptionEffortRef
F1DELETE+INSERT merge strategy double-evaluation guard3–4hSQL_GAPS_7.md G1.1 (P0)
F2WAL decoder: keyless-table pk_hash computation4–6hSQL_GAPS_7.md G3.1 · PLAN_HYBRID_CDC.md
F3WAL decoder: old_* column population for UPDATEs4–6hSQL_GAPS_7.md G3.2 · PLAN_HYBRID_CDC.md
F5JOIN key column change detection in delta SQL3–4hSQL_GAPS_7.md G4.1 (P1)
F6ALTER TYPE / ALTER POLICY DDL tracking3–5hSQL_GAPS_7.md G9.1 (P1)

Subtotal: 17–25 hours (F4 and F7 moved to v0.1.0)

Tier 1 — Verification

ItemDescriptionEffortRef
F8–F10, F12Window partition key E2E, recursive CTE monotonicity audit, ALTER DOMAIN tracking, PgBouncer compatibility docs16–23hSQL_GAPS_7.md G5–G9

F11 (keyless table duplicate-rows documentation) moved to v0.1.0.

Tier 2 — Robustness

ItemDescriptionEffortRef
F13, F15–F16LIMIT-in-subquery warning, RANGE_AGG recognition, read replica detection, SPI error classification6–8hSQL_GAPS_7.md G2, G5, G6, G8

F14 (CUBE explosion guard) moved to v0.1.0.

Tier 3 — Test coverage

ItemDescriptionEffortRef
F17–F2621 aggregate differential E2E, FULL JOIN E2E, INTERSECT/EXCEPT pairs, GUC variation tests, CI combined coverage29–38hSQL_GAPS_7.md G7 · STATUS_TESTING.md

v0.2.0 total: ~68–94 hours (F4, F7, F11, F14 moved to v0.1.0)

Exit criteria:

  • Zero P0 gaps
  • All P1 gaps resolved or documented as known limitations
  • E2E test count ≥ 400 with 0 pre-existing failures
  • Combined coverage ≥ 75%

v0.3.0 — Production Readiness

Goal: Operational polish, parallel refresh, and production-grade WAL-based CDC. The extension is suitable for production use after this milestone.

Performance & Parallelism

ItemDescriptionEffortRef
P1Verify PostgreSQL parallel query for delta SQL0hREPORT_PARALLELIZATION.md §E
P2DAG level extraction (topological_levels())2–4hREPORT_PARALLELIZATION.md §B
P3Dynamic background worker dispatch per level12–16hREPORT_PARALLELIZATION.md §A+B

Operational

ItemDescriptionEffortRef
O1Extension upgrade migrations (ALTER EXTENSION UPDATE)4–6hSQL_GAPS_7.md G8.2 · PLAN_UPGRADE_MIGRATIONS.md
O2Prepared statement cleanup on cache invalidation3–4hSQL_GAPS_7.md G8.3
O3Adaptive fallback threshold exposure via monitoring2–3hSQL_GAPS_7.md G8.4
O4SPI SQLSTATE error classification for retry3–4hSQL_GAPS_7.md G8.6
O5Slot lag alerting thresholds (configurable)2–3hSQL_GAPS_7.md G10

WAL CDC Hardening

ItemDescriptionEffortRef
W1WAL decoder fixes (F2–F4 prerequisite from v0.2.0)Done in v0.2.0PLAN_HYBRID_CDC.md
W2WAL mode E2E test suite (parallel to trigger suite)8–12hPLAN_HYBRID_CDC.md
W3WAL→trigger automatic fallback hardening4–6hPLAN_HYBRID_CDC.md
W4Promote pg_stream.cdc_mode = 'auto' to recommendedDocumentationPLAN_HYBRID_CDC.md

v0.3.0 total: ~40–58 hours (excluding v0.2.0 prerequisites)

Exit criteria:

  • max_concurrent_refreshes drives real parallel refresh
  • WAL CDC mode passes full E2E suite
  • Extension upgrade path tested (0.1.0 → 0.3.0)
  • Zero P0/P1 gaps remaining

v0.4.0 — Observability & Integration

Goal: Prometheus/Grafana observability, dbt-pgstream formal release, complete documentation review, and validated upgrade path. After this milestone the product is externally visible and monitored.

Observability

ItemDescriptionEffortRef
M1Prometheus exporter configuration guide4–6hPLAN_ECO_SYSTEM.md §1
M2Grafana dashboard (refresh latency, staleness, CDC lag)4–6hPLAN_ECO_SYSTEM.md §1

Integration & Release prep

ItemDescriptionEffortRef
R5dbt-pgstream 0.1.0 formal release (PyPI)2–3hdbt-pgstream/ · PLAN_DBT_MACRO.md
R6Complete documentation review & polish4–6hdocs/
O1Extension upgrade migrations (ALTER EXTENSION UPDATE)4–6hSQL_GAPS_7.md G8.2 · PLAN_UPGRADE_MIGRATIONS.md

v0.4.0 total: ~18–27 hours

Exit criteria:

  • Grafana dashboard published
  • dbt-pgstream 0.1.0 on PyPI
  • ALTER EXTENSION pg_stream UPDATE tested (0.3.0 → 0.4.0)
  • All public documentation current and reviewed

v1.0.0 — Stable Release

Goal: First officially supported release. Semantic versioning locks in. API, catalog schema, and GUC names are considered stable. Focus is distribution — getting pg_stream onto package registries.

Release engineering

ItemDescriptionEffortRef
R1Semantic versioning policy + compatibility guarantees2–3hPLAN_VERSIONING.md
R2PGXN / apt / rpm packaging8–12hPLAN_PACKAGING.md
R3Docker Hub official image → CNPG extension image✅ DonePLAN_CLOUDNATIVEPG.md
R4CNPG operator hardening (K8s 1.33+ native ImageVolume)4–6hPLAN_CLOUDNATIVEPG.md

v1.0.0 total: ~18–27 hours

Exit criteria:

  • Published on PGXN and Docker Hub
  • CNPG extension image published to GHCR (pg_stream-ext)
  • CNPG cluster-example.yaml validated (Image Volume approach)
  • Upgrade path from v0.4.0 tested
  • Semantic versioning policy in effect

Post-1.0 — Scale & Ecosystem

These are not gated on 1.0 but represent the longer-term horizon.

Ecosystem expansion

ItemDescriptionEffortRef
E1dbt full adapter (dbt-pgstream extending dbt-postgres)20–30hPLAN_DBT_ADAPTER.md
E2Airflow provider (apache-airflow-providers-pgstream)16–20hPLAN_ECO_SYSTEM.md §4
E3CLI tool (pgstream) for management outside SQL16–20hPLAN_ECO_SYSTEM.md §4
E4Flyway / Liquibase migration support8–12hPLAN_ECO_SYSTEM.md §5
E5ORM integrations guide (SQLAlchemy, Django, etc.)8–12hPLAN_ECO_SYSTEM.md §5

Scale

ItemDescriptionEffortRef
S1External orchestrator sidecar for 100+ STs20–40hREPORT_PARALLELIZATION.md §D
S2Citus / distributed PostgreSQL compatibility~6 monthsplans/infra/CITUS.md
S3Multi-database support (beyond postgres DB)TBDPLAN_MULTI_DATABASE.md

Advanced SQL

ItemDescriptionEffortRef
A1Circular dependency support (SCC fixpoint iteration)~40hCIRCULAR_REFERENCES.md
A2Streaming aggregation (sub-second latency path)TBDPLAN_STREAMING_AGGREGATION.md
A3PostgreSQL 19 forward-compatibilityTBDPLAN_PG19_COMPAT.md

Effort Summary

MilestoneEffort estimateCumulative
v0.2.0 — Correctness68–94h68–94h
v0.3.0 — Production ready40–58h108–152h
v0.4.0 — Observability & Integration18–27h126–179h
v1.0.0 — Stable release18–27h144–206h
Post-1.0 (ecosystem)88–134h232–340h
Post-1.0 (scale)6+ months—

References

DocumentPurpose
CHANGELOG.mdWhat's been built
plans/PLAN.mdOriginal 13-phase design plan
plans/sql/SQL_GAPS_7.md53 known gaps, prioritized
plans/performance/REPORT_PARALLELIZATION.mdParallelization options analysis
plans/performance/STATUS_PERFORMANCE.mdBenchmark results
plans/ecosystem/PLAN_ECO_SYSTEM.mdEcosystem project catalog
plans/dbt/PLAN_DBT_ADAPTER.mdFull dbt adapter plan
plans/infra/CITUS.mdCitus compatibility plan
plans/infra/PLAN_VERSIONING.mdVersioning & compatibility policy
plans/infra/PLAN_PACKAGING.mdPGXN / deb / rpm packaging
plans/infra/PLAN_DOCKER_IMAGE.mdOfficial Docker image (superseded by CNPG extension image)
plans/ecosystem/PLAN_CLOUDNATIVEPG.mdCNPG Image Volume extension image
plans/infra/PLAN_MULTI_DATABASE.mdMulti-database support
plans/infra/PLAN_PG19_COMPAT.mdPostgreSQL 19 forward-compatibility
plans/sql/PLAN_UPGRADE_MIGRATIONS.mdExtension upgrade migrations
plans/sql/PLAN_STREAMING_AGGREGATION.mdSub-second streaming aggregation
plans/adrs/PLAN_ADRS.mdArchitectural decisions
docs/ARCHITECTURE.mdSystem architecture