📝 Documenation

docs/exec-plans/active/001-implementation-roadmap.md

@@ -0,0 +1,189 @@
+# ExecPlan 001 — Implementation Roadmap: AquaCom Integration Gateway
+
+**Document type:** ExecPlan (umbrella roadmap)
+**Status:** Active
+**Created:** 2026-10-06
+**Scope:** Phases 0–14 (implementation specification §60) through the initial production release
+**Authoritative references:** `ARCHITECTURE.md` · `docs/product/software-specification.md` · `docs/product/codex-implementation-specification.md` · `docs/configuration/configuration-reference.md` · `docs/protocol/` (gap registry, operations registry, test vectors)
+
+> This is the **umbrella plan**: it sequences the whole implementation and records cross-phase decisions. Substantial individual features may spawn focused child ExecPlans in this directory (implementation specification §59); this roadmap remains the index. Per `ARCHITECTURE.md` §1, role-authoritative documents win over any restatement here.
+
+---
+
+## 1. Purpose
+
+Provide a single, gated execution path from the current documentation-complete, evidence-prepared repository state to the initial production release defined by `software-specification.md` §47 / implementation specification §74, such that:
+
+- every implemented AquaCom behavior traces to verified evidence (`AGENTS.md`, `known-operations.md`);
+- each phase ends in a working, testable repository state (implementation specification §79);
+- the one open blocking decision (technology selection) is resolved explicitly and recorded before code begins;
+- protocol gaps are consumed at the phase that needs them, with documented workarounds where possible.
+
+## 2. Current state (2026-10-06)
+
+**Exists and verified:**
+
+- Complete, consistent documentation set: short-form `AGENTS.md` entry point; `ARCHITECTURE.md` with document precedence (§1) and invariants INV-001…INV-016; `software-specification.md` with requirement IDs AQG-001…AQG-030 and RFC-2119 keywords; agent-neutral implementation specification; `configuration-reference.md`.
+- Protocol evidence: authoritative source (*AquaCom – RTU level, Rev 1.6, ITT Flygt AB, 2004*) as PDF + markdown conversion in `docs/protocol/`.
+- `docs/protocol/protocol-gaps.md` — 10 entries (1 `VERIFY`, 9 `OPEN`), each with working hypotheses where applicable.
+- `docs/protocol/known-operations.md` — OPS-001…OPS-012 registered with evidence sections, scope status, and an evidence-complete implementation order.
+- `docs/protocol/test-vectors.md` — **35 source checksum vectors verified by recomputation** against the §2.4 algorithm (including the §3.6.4 Wait-byte pair), framing/session vectors, 4 synthetic wraparound vectors, WGS/WGH/WGD decode scenarios, simulator scenario requirements.
+
+**Does not exist yet:** `src/`, `tests/`, `simulator/`, `schemas/`, `config/`, CI, `README.md`, `CHANGELOG.md`, any implementation.
+
+**Open:** technology selection (decision D-001 below) — the only blocking item.
+
+## 3. Desired state
+
+The initial production scope (implementation specification §74): verified live and historical AquaCom acquisition for multiple stations; canonical data model; live cache; historical pipeline with deduplication and durable spool; ≥1 queryable history repository; ≥2 independently configurable storage destinations; OPC UA Data Access + Historical Access with stable NodeIds and security on by default; diagnostics and structured logging; simulator; automated tests including restart recovery; configuration validation. Explicitly excluded: process control, setpoint writes, OPC UA HistoryUpdate, HA clustering (implementation specification §75).
+
+## 4. Constraints
+
+Hard rules that bind every phase (traceability in parentheses):
+
+1. Never invent AquaCom protocol behavior; evidence classes only (`AGENTS.md`; implementation specification §3.1).
+2. AquaCom logic only in `src/aquacom`; OPC UA and storage never parse telegrams (INV-001/002/003).
+3. Historical samples never update current state (INV-004); SourceTimestamp preserved (INV-005).
+4. OPC UA HistoryRead queries a repository, never the RTU (INV-006).
+5. Storage failure never stops live acquisition (INV-007).
+6. Stable StationId/PointId/NodeIds across restart, upgrade, rename, transport change (INV-008/009).
+7. Watermark advances on durable spool commit (INV-014); dedup correct across restart (INV-015).
+8. Out-of-order historical arrival supported (INV-013); missing-timestamp records quarantined (INV-016).
+9. Live acquisition priority over backfill (AQG-028); every queue bounded (ARCHITECTURE §51).
+10. Secure defaults; no secrets in the repo (`AGENTS.md` Security).
+
+## 5. Open decisions
+
+| ID | Decision | Blocks | Recommendation / resolution path |
+|---|---|---|---|
+| **D-001** | Implementation language + OPC UA server stack | **Phase 1 (all code)** | The hard constraint is **server-side Historical Access** (Part 11) — far less commonly supported than Data Access. Run a HistoryRead smoke test per candidate before committing. Candidates in priority order: C#/.NET (OPC Foundation .NET Standard stack — reference HA implementation), Java/Milo (HA supported), Python/asyncua (fast development; **verify HA server completeness first**). |
+| D-002 | Configuration format | Phase 0 (schema skeleton) | YAML — all examples in `configuration-reference.md` are already YAML. Low risk. |
+| D-003 | First history repository technology | Phase 8 | Behind the `HistoryRepository` interface (ARCHITECTURE §25), so replaceable. Embedded (e.g. SQLite) minimizes external dependencies for the first release; external (PostgreSQL/TimescaleDB) remains a valid Phase 12+ sink. Decide at Phase 8 entry. |
+| D-004 | Second storage sink technology | Phase 12 | NDJSON file archive is the lowest-risk second destination (config examples already use it). Decide at Phase 12 entry. |
+| D-005 | CI platform | Phase 0 | Whatever the repository host provides; must run format/lint/static/unit/integration + schema validation without network access (implementation specification §57). |
+| D-006 | Configuration schema technology | Phase 0 | JSON Schema if D-002 = YAML (standard YAML-schema validation path). |
+
+**Pre-implementation checklist (project owner, in order):**
+
+1. **Resolve D-001.** I can build and run the HA smoke test for each candidate stack on request — a ~50-line HistoryRead server+client per candidate settles it empirically.
+2. **PDF spot-checks** (~30 min, recommended before Phase 3/6; non-blocking for Phases 0–2): confirm AQ-GAP-001(a) MCC type table, (b) WGS answer `mp1–mp128` byte ranges, (e) whether §3.7 truly ends mid-telegram; AQ-GAP-002 whether the PDF defines `alp` more explicitly; AQ-GAP-006 whether the MAT request really prints 557.
+3. **Confirm deployment target**: OS/host, and whether field contact is TCP via terminal server (serial-over-IP) or direct serial — determines which Phase 2 transports to implement first.
+
+**What does *not* block starting:** real RTU hardware or captures (the simulator drives Phases 0–13; every gap that would need a live capture has a documented workaround), `aquacom.md` consolidation, operations documentation (Phase 14).
+
+## 6. Implementation sequence
+
+### 6.1 Phase/gate overview
+
+| Phase | Scope (§60) | Gate — exit criteria | Key gap dependency |
+|---|---|---|---|
+| **0** | Repository skeleton, CI, README/CHANGELOG, config schema skeleton | **G0:** CI green on placeholder test; schema validates the §72 minimal config; tree matches implementation specification §5 | — |
+| **1** | Canonical core: StationId, PointId, CurrentValue, HistoricalSample, DataQuality, AcquisitionSource, config domain | **G1:** domain unit tests + serialization round-trips pass; identifier-stability rule enforced (rename ≠ new PointId); `core` has zero aquacom/opcua/storage imports | — |
+| **2** | Transport abstraction + mock transport; TCP first (serial-over-IP per deployment answer) | **G2:** connect/send/receive/timeout/reconnect tests pass; supports shared-medium arbitration (software-specification §5.1) | — |
+| **3** | Codec framework: framing, checksum, Wait-byte stripping, validation, fixture framework | **G3:** **100 % of approved vectors green** (AQ-CK-001…035, S1–S4, framing vectors AQ-FR-001…005); invalid telegrams rejected with diagnostics | AQ-GAP-001 (`VERIFY` items against PDF) |
+| **4** | AquaCom simulator | **G4:** scripted WGS success/timeout/malformed/checksum-error/configurable values; **no shared codec code** (self-validation ban, §49) | — |
+| **5** | Station/session manager | **G5:** multiple simulated stations; one failing station does not affect others; state machine per §18; configurable thresholds | — |
+| **6** | Live acquisition: WGS decode → point mapping → scaling → CurrentValue | **G6:** simulator → CurrentValue with correct value, quality, timestamps (Source vs Received distinguished); scaling formula applied; alarm/point split via `Nal` | AQ-GAP-002 (workaround: decode via `Nal`, `alp` diagnostic-only) |
+| **7** | OPC UA live interface: DA, subscriptions, stable NodeIds, security | **G7:** OPC UA test client browses/reads/subscribes; NodeIds stable across restart; security `None` disabled | — |
+| **8** | Local historical repository (first queryable store) | **G8:** store / query interval / bounds / restart persistence behind the §25 interface | — |
+| **9** | Historical pipeline: validation → dedup → durable spool → distribution | **G9:** duplicate submissions produce no duplicate records (idempotent across restart, INV-015); spool survives restart; out-of-order writes accepted (INV-013) | — |
+| **10** | AquaCom historical acquisition: WGH retrieval, watermark, backfill state machine, throttling | **G10:** simulator's missing interval recovered after reconnect; watermark resumes from persistent state; backfill throttled so live polling stays responsive | AQ-GAP-003 (hypothesis: block value count from station CFG — validate against simulator) |
+| **11** | OPC UA Historical Access: HistoryRead raw values, continuation | **G11:** HA client reads both originally stored and recovered values; continuation works; **INV-006 test: HistoryRead never triggers AquaCom traffic** | — |
+| **12** | Second storage destination (sink abstraction) | **G12:** one HistoricalSample → repository + sink B; failure of B does not stop A | — |
+| **13** | Store-and-forward resilience | **G13:** destination offline → history accumulates → delivered on restore; restart during backlog loses no acknowledged data (§53) | — |
+| **14** | Production hardening: security, resource limits, performance validation, ops documentation | **G14:** §77 DoD reviewed item-by-item; §69 performance validation with capacity targets recorded | — |
+| **O-1** (optional, after 10) | WGD periodical reports (`AQUACOM_REPORT`) | report day retrieved, scaled per point, `NDA` handled as missing-not-failed | AQ-GAP-009 (documented defaults) |
+
+### 6.2 Task mapping (implementation specification §79)
+
+T001→P0 · T002→P1 · T003→P0/P1 (config model + schema validation) · T004→P2 · T005+T006→P3 · T007→P4 · T008→P6 (WGS, first verified live operation) · T009→P5/P6 · T010→P6 (current-state service) · T011→P7 · T012→P8 · T013→P9 · T014→P9/P13 (durable spool) · T015+T016→P10 (WGH + backfill state machine) · T017→P11 · T018→P12 · T019→P13 (full reconnect/backfill acceptance) · T020→P14.
+
+Each task ends in a working repository state. Recommended task granularity: one §79-sized task per working session, each with the Goal / Files / Behavior / Constraints / Tests / Acceptance structure (§61).
+
+### 6.3 Gap consumption schedule
+
+| Gap | Consumed at | Handling |
+|---|---|---|
+| AQ-GAP-001 | before Phase 3 table implementations | `VERIFY` against PDF; 35 approved vectors already cover framing/checksum |
+| AQ-GAP-002 | Phase 6 | workaround documented (decode via `Nal`); hypothesis strengthened by all four examples |
+| AQ-GAP-003 | Phase 10 | hypothesis wired to station CFG; simulator validation; close on first real capture |
+| AQ-GAP-004 | text points (none in initial scope) | deferred |
+| AQ-GAP-005 | any time PDF text is available | transcribe long examples → recompute → approve as wraparound fixtures (synthetics cover the branch meanwhile) |
+| AQ-GAP-006 | Phase 3 fixture import | use 540 once PDF-confirmed; verified answer vector approvable independently |
+| AQ-GAP-007 | SCE handling (Phase 3/6) | payload treated as optional diagnostic, never parse-critical |
+| AQ-GAP-008/009/010 | dialled lines / WGD / alarms | extension scope; not in initial release |
+
+## 7. Files/components affected
+
+`src/{core,aquacom,history,storage,opcua,diagnostics,host}/` per `ARCHITECTURE.md` §6 and dependency direction §7 · `tests/{unit,integration,protocol,opcua,history,acceptance}/` · `simulator/` · `fixtures/` (imported from `docs/protocol/test-vectors.md`) · `schemas/` · `config/examples/` · `README.md`, `CHANGELOG.md` · CI definition · documentation updates in `docs/` as behavior lands (never after).
+
+## 8. Data model changes
+
+None yet — the canonical contracts are frozen in `ARCHITECTURE.md` §13–§14 and implementation specification §10–§13. Any contract change requires: update the role-authoritative document first, update dependents, update tests, record the decision here. The core module must compile without AquaCom/OPC UA/storage imports (enforced in CI from Phase 1).
+
+## 9. Failure handling
+
+The failure matrix of `software-specification.md` §35 / `ARCHITECTURE.md` §55 is normative: station failure isolates; storage failure isolates and buffers; OPC UA failure never stops acquisition; malformed telegrams are rejected with diagnostics; restart loses no acknowledged history. Each phase's tests must cover the failure modes it introduces (implementation specification §56 fault injection); every new failure mode gets a diagnostic (AGENTS.md Task completion).
+
+## 10. Testing
+
+- Pyramid per §50: deterministic unit/component tests dominate; integration via simulator; acceptance scenario at the top.
+- Fixtures: imported **only** from `docs/protocol/test-vectors.md`; approved fixtures immutable; new vectors require evidence and registry update (never delete).
+- Protocol tests run without external network (§57); simulator must not share codec code (§49).
+- Restart-recovery tests at every phase that adds persistent state (§53).
+- Bug fixes carry regression tests (AGENTS.md).
+
+## 11. Migration/compatibility
+
+Storage schemas versioned with explicit migrations; startup never silently destroys incompatible data; destructive migration requires administrative action (implementation specification §65). `configVersion` gating with explicit migration tooling (§66). OPC UA NodeId stability across releases (§68); configuration schema changes never force namespace changes (`configuration-reference.md` §71).
+
+## 12. Progress
+
+| Date | Entry |
+|---|---|
+| 2026-10-05 | Documentation set completed and made consistent; protocol evidence acquired (Rev 1.6 source); gap registry (10 entries), operations registry (OPS-001…012), and test-vector catalogue (35 verified vectors) created. Phase 0 documentation deliverables done. |
+| 2026-10-06 | This roadmap created. Awaiting D-001 (technology selection) before Phase 0 code tasks. |
+| 2026-10-06 | `README.md` and `CHANGELOG.md` created — the §5 repository-structure documentation items for Phase 0 are complete. Remaining Phase 0 deliverables: technology selection (D-001), code skeleton, CI, configuration schema. |
+
+_Update this log at every phase gate; spawn child ExecPlans for §59-triggering work and link them here._
+
+## 13. Decisions log
+
+| ID | Date | Decision | Reference |
+|---|---|---|---|
+| DR-001 | 2026-10-05 | Documentation unescaped, de-duplicated; single-source-of-truth precedence defined | `ARCHITECTURE.md` §1 |
+| DR-002 | 2026-10-05 | `AGENTS.md` reduced to short entry-point form per its own spec §4/§6 | `AGENTS.md` |
+| DR-003 | 2026-10-05 | Watermark advances on **durable spool commit**, not final delivery | `ARCHITECTURE.md` §21, INV-014 |
+| DR-004 | 2026-10-05 | Duplicate-conflict default **REJECT_CONFLICT** | software-spec §13 |
+| DR-005 | 2026-10-05 | Deduplication correct across restart (persisted state or idempotent storage) | INV-015 |
+| DR-006 | 2026-10-05 | Pipeline accepts out-of-order historical arrival; queries order by SourceTimestamp | INV-013 |
+| DR-007 | 2026-10-05 | Records without usable SourceTimestamp quarantined with diagnostics | INV-016 |
+| DR-008 | 2026-10-05 | Control/setpoint writes OUT-OF-SCOPE for initial release; read-only integration first | known-operations OPS-005/006/009/010 |
+| DR-009 | 2026-10-05 | WGS decode via `Nal`; `alp` diagnostic-only until AQ-GAP-002 closes | AQ-GAP-002 hypothesis |
+| DR-010 | 2026-10-05 | WGH block value count derived from station CFG | AQ-GAP-003 hypothesis |
+| DR-011 | 2026-10-06 | Umbrella roadmap established; D-001 recorded as the blocking decision | this document |
+
+## 14. Unexpected findings
+
+- Source document contains checksum/typo defects: MAT fetch request prints 557 (recomputes 540; 557 = SLT fetch checksum — copy-paste error), narrative destination mismatches (§2.2.3.1 "plant 89" vs telegram dest 86; §3.4.10.2 fixed-line dest given as plant number) — recorded in AQ-GAP-005/006.
+- The PDF→markdown conversion is lossy at specific tables and truncates §3.7; byte-exact recomputation of long examples from the conversion is impossible (AQ-GAP-001/005). The PDF is authoritative.
+- The §2.4 checksum algorithm verified exactly against all 35 single-line examples — including the subtle rules that spaces around `#` are summed and Wait bytes are excluded.
+
+## 15. Acceptance evidence
+
+Production evaluation requires all of (implementation specification §77), evidenced by:
+
+1. **Mandatory end-to-end scenario** (§54 T0–T8; `software-specification.md` §46): recorded as an automated acceptance test (T019); passing run attached to the Phase 14 gate.
+2. **Protocol traceability audit:** every implemented operation maps to an `known-operations.md` entry with evidence; remaining gaps explicitly listed (§77 items 2, 15) — the gap registry *is* the evidence.
+3. **Vector suite green:** all approved vectors passing in CI (§77 item 12).
+4. **Restart/data-integrity tests** (§53): no acknowledged history lost across restart, outage, backlog (items 7–9).
+5. **Security tests** (§55): untrusted certs rejected, None disabled in production config, no secrets logged.
+6. **DoD checklist review:** the 15 items of §77 walked item-by-item at G14, each with its artifact.
+7. Items 13 (validated against real equipment/traffic) and operational documentation remain **post-simulator** milestones dependent on field access — explicitly tracked, not silently dropped.
+
+## 16. Plan maintenance
+
+- Update §12 (Progress) at every phase gate; record new decisions in §13 immediately.
+- Spawn a child ExecPlan (same structure, §59) for: any new AquaCom operation, storage/persistence architecture, OPC UA namespace redesign, security changes, cross-module refactors, data migration. Link children from §12.
+- Move this plan to `docs/exec-plans/completed/` when G14 and §15 are satisfied.
+- This plan never overrides the role-authoritative documents (`ARCHITECTURE.md` §1).