189 lines
19 KiB
Markdown
189 lines
19 KiB
Markdown
# ExecPlan 001 — Implementation Roadmap: AquaCom Integration Gateway
|
||
|
||
**Document type:** ExecPlan (umbrella roadmap)
|
||
**Status:** Active
|
||
**Created:** 2026-10-06
|
||
**Scope:** Phases 0–14 (implementation specification §60) through the initial production release
|
||
**Authoritative references:** `ARCHITECTURE.md` · `docs/product/software-specification.md` · `docs/product/codex-implementation-specification.md` · `docs/configuration/configuration-reference.md` · `docs/protocol/` (gap registry, operations registry, test vectors)
|
||
|
||
> This is the **umbrella plan**: it sequences the whole implementation and records cross-phase decisions. Substantial individual features may spawn focused child ExecPlans in this directory (implementation specification §59); this roadmap remains the index. Per `ARCHITECTURE.md` §1, role-authoritative documents win over any restatement here.
|
||
|
||
---
|
||
|
||
## 1. Purpose
|
||
|
||
Provide a single, gated execution path from the current documentation-complete, evidence-prepared repository state to the initial production release defined by `software-specification.md` §47 / implementation specification §74, such that:
|
||
|
||
- every implemented AquaCom behavior traces to verified evidence (`AGENTS.md`, `known-operations.md`);
|
||
- each phase ends in a working, testable repository state (implementation specification §79);
|
||
- the one open blocking decision (technology selection) is resolved explicitly and recorded before code begins;
|
||
- protocol gaps are consumed at the phase that needs them, with documented workarounds where possible.
|
||
|
||
## 2. Current state (2026-10-06)
|
||
|
||
**Exists and verified:**
|
||
|
||
- Complete, consistent documentation set: short-form `AGENTS.md` entry point; `ARCHITECTURE.md` with document precedence (§1) and invariants INV-001…INV-016; `software-specification.md` with requirement IDs AQG-001…AQG-030 and RFC-2119 keywords; agent-neutral implementation specification; `configuration-reference.md`.
|
||
- Protocol evidence: authoritative source (*AquaCom – RTU level, Rev 1.6, ITT Flygt AB, 2004*) as PDF + markdown conversion in `docs/protocol/`.
|
||
- `docs/protocol/protocol-gaps.md` — 10 entries (1 `VERIFY`, 9 `OPEN`), each with working hypotheses where applicable.
|
||
- `docs/protocol/known-operations.md` — OPS-001…OPS-012 registered with evidence sections, scope status, and an evidence-complete implementation order.
|
||
- `docs/protocol/test-vectors.md` — **35 source checksum vectors verified by recomputation** against the §2.4 algorithm (including the §3.6.4 Wait-byte pair), framing/session vectors, 4 synthetic wraparound vectors, WGS/WGH/WGD decode scenarios, simulator scenario requirements.
|
||
|
||
**Does not exist yet:** `src/`, `tests/`, `simulator/`, `schemas/`, `config/`, CI, `README.md`, `CHANGELOG.md`, any implementation.
|
||
|
||
**Open:** technology selection (decision D-001 below) — the only blocking item.
|
||
|
||
## 3. Desired state
|
||
|
||
The initial production scope (implementation specification §74): verified live and historical AquaCom acquisition for multiple stations; canonical data model; live cache; historical pipeline with deduplication and durable spool; ≥1 queryable history repository; ≥2 independently configurable storage destinations; OPC UA Data Access + Historical Access with stable NodeIds and security on by default; diagnostics and structured logging; simulator; automated tests including restart recovery; configuration validation. Explicitly excluded: process control, setpoint writes, OPC UA HistoryUpdate, HA clustering (implementation specification §75).
|
||
|
||
## 4. Constraints
|
||
|
||
Hard rules that bind every phase (traceability in parentheses):
|
||
|
||
1. Never invent AquaCom protocol behavior; evidence classes only (`AGENTS.md`; implementation specification §3.1).
|
||
2. AquaCom logic only in `src/aquacom`; OPC UA and storage never parse telegrams (INV-001/002/003).
|
||
3. Historical samples never update current state (INV-004); SourceTimestamp preserved (INV-005).
|
||
4. OPC UA HistoryRead queries a repository, never the RTU (INV-006).
|
||
5. Storage failure never stops live acquisition (INV-007).
|
||
6. Stable StationId/PointId/NodeIds across restart, upgrade, rename, transport change (INV-008/009).
|
||
7. Watermark advances on durable spool commit (INV-014); dedup correct across restart (INV-015).
|
||
8. Out-of-order historical arrival supported (INV-013); missing-timestamp records quarantined (INV-016).
|
||
9. Live acquisition priority over backfill (AQG-028); every queue bounded (ARCHITECTURE §51).
|
||
10. Secure defaults; no secrets in the repo (`AGENTS.md` Security).
|
||
|
||
## 5. Open decisions
|
||
|
||
| ID | Decision | Blocks | Recommendation / resolution path |
|
||
|---|---|---|---|
|
||
| **D-001** | Implementation language + OPC UA server stack | **Phase 1 (all code)** | The hard constraint is **server-side Historical Access** (Part 11) — far less commonly supported than Data Access. Run a HistoryRead smoke test per candidate before committing. Candidates in priority order: C#/.NET (OPC Foundation .NET Standard stack — reference HA implementation), Java/Milo (HA supported), Python/asyncua (fast development; **verify HA server completeness first**). |
|
||
| D-002 | Configuration format | Phase 0 (schema skeleton) | YAML — all examples in `configuration-reference.md` are already YAML. Low risk. |
|
||
| D-003 | First history repository technology | Phase 8 | Behind the `HistoryRepository` interface (ARCHITECTURE §25), so replaceable. Embedded (e.g. SQLite) minimizes external dependencies for the first release; external (PostgreSQL/TimescaleDB) remains a valid Phase 12+ sink. Decide at Phase 8 entry. |
|
||
| D-004 | Second storage sink technology | Phase 12 | NDJSON file archive is the lowest-risk second destination (config examples already use it). Decide at Phase 12 entry. |
|
||
| D-005 | CI platform | Phase 0 | Whatever the repository host provides; must run format/lint/static/unit/integration + schema validation without network access (implementation specification §57). |
|
||
| D-006 | Configuration schema technology | Phase 0 | JSON Schema if D-002 = YAML (standard YAML-schema validation path). |
|
||
|
||
**Pre-implementation checklist (project owner, in order):**
|
||
|
||
1. **Resolve D-001.** I can build and run the HA smoke test for each candidate stack on request — a ~50-line HistoryRead server+client per candidate settles it empirically.
|
||
2. **PDF spot-checks** (~30 min, recommended before Phase 3/6; non-blocking for Phases 0–2): confirm AQ-GAP-001(a) MCC type table, (b) WGS answer `mp1–mp128` byte ranges, (e) whether §3.7 truly ends mid-telegram; AQ-GAP-002 whether the PDF defines `alp` more explicitly; AQ-GAP-006 whether the MAT request really prints 557.
|
||
3. **Confirm deployment target**: OS/host, and whether field contact is TCP via terminal server (serial-over-IP) or direct serial — determines which Phase 2 transports to implement first.
|
||
|
||
**What does *not* block starting:** real RTU hardware or captures (the simulator drives Phases 0–13; every gap that would need a live capture has a documented workaround), `aquacom.md` consolidation, operations documentation (Phase 14).
|
||
|
||
## 6. Implementation sequence
|
||
|
||
### 6.1 Phase/gate overview
|
||
|
||
| Phase | Scope (§60) | Gate — exit criteria | Key gap dependency |
|
||
|---|---|---|---|
|
||
| **0** | Repository skeleton, CI, README/CHANGELOG, config schema skeleton | **G0:** CI green on placeholder test; schema validates the §72 minimal config; tree matches implementation specification §5 | — |
|
||
| **1** | Canonical core: StationId, PointId, CurrentValue, HistoricalSample, DataQuality, AcquisitionSource, config domain | **G1:** domain unit tests + serialization round-trips pass; identifier-stability rule enforced (rename ≠ new PointId); `core` has zero aquacom/opcua/storage imports | — |
|
||
| **2** | Transport abstraction + mock transport; TCP first (serial-over-IP per deployment answer) | **G2:** connect/send/receive/timeout/reconnect tests pass; supports shared-medium arbitration (software-specification §5.1) | — |
|
||
| **3** | Codec framework: framing, checksum, Wait-byte stripping, validation, fixture framework | **G3:** **100 % of approved vectors green** (AQ-CK-001…035, S1–S4, framing vectors AQ-FR-001…005); invalid telegrams rejected with diagnostics | AQ-GAP-001 (`VERIFY` items against PDF) |
|
||
| **4** | AquaCom simulator | **G4:** scripted WGS success/timeout/malformed/checksum-error/configurable values; **no shared codec code** (self-validation ban, §49) | — |
|
||
| **5** | Station/session manager | **G5:** multiple simulated stations; one failing station does not affect others; state machine per §18; configurable thresholds | — |
|
||
| **6** | Live acquisition: WGS decode → point mapping → scaling → CurrentValue | **G6:** simulator → CurrentValue with correct value, quality, timestamps (Source vs Received distinguished); scaling formula applied; alarm/point split via `Nal` | AQ-GAP-002 (workaround: decode via `Nal`, `alp` diagnostic-only) |
|
||
| **7** | OPC UA live interface: DA, subscriptions, stable NodeIds, security | **G7:** OPC UA test client browses/reads/subscribes; NodeIds stable across restart; security `None` disabled | — |
|
||
| **8** | Local historical repository (first queryable store) | **G8:** store / query interval / bounds / restart persistence behind the §25 interface | — |
|
||
| **9** | Historical pipeline: validation → dedup → durable spool → distribution | **G9:** duplicate submissions produce no duplicate records (idempotent across restart, INV-015); spool survives restart; out-of-order writes accepted (INV-013) | — |
|
||
| **10** | AquaCom historical acquisition: WGH retrieval, watermark, backfill state machine, throttling | **G10:** simulator's missing interval recovered after reconnect; watermark resumes from persistent state; backfill throttled so live polling stays responsive | AQ-GAP-003 (hypothesis: block value count from station CFG — validate against simulator) |
|
||
| **11** | OPC UA Historical Access: HistoryRead raw values, continuation | **G11:** HA client reads both originally stored and recovered values; continuation works; **INV-006 test: HistoryRead never triggers AquaCom traffic** | — |
|
||
| **12** | Second storage destination (sink abstraction) | **G12:** one HistoricalSample → repository + sink B; failure of B does not stop A | — |
|
||
| **13** | Store-and-forward resilience | **G13:** destination offline → history accumulates → delivered on restore; restart during backlog loses no acknowledged data (§53) | — |
|
||
| **14** | Production hardening: security, resource limits, performance validation, ops documentation | **G14:** §77 DoD reviewed item-by-item; §69 performance validation with capacity targets recorded | — |
|
||
| **O-1** (optional, after 10) | WGD periodical reports (`AQUACOM_REPORT`) | report day retrieved, scaled per point, `NDA` handled as missing-not-failed | AQ-GAP-009 (documented defaults) |
|
||
|
||
### 6.2 Task mapping (implementation specification §79)
|
||
|
||
T001→P0 · T002→P1 · T003→P0/P1 (config model + schema validation) · T004→P2 · T005+T006→P3 · T007→P4 · T008→P6 (WGS, first verified live operation) · T009→P5/P6 · T010→P6 (current-state service) · T011→P7 · T012→P8 · T013→P9 · T014→P9/P13 (durable spool) · T015+T016→P10 (WGH + backfill state machine) · T017→P11 · T018→P12 · T019→P13 (full reconnect/backfill acceptance) · T020→P14.
|
||
|
||
Each task ends in a working repository state. Recommended task granularity: one §79-sized task per working session, each with the Goal / Files / Behavior / Constraints / Tests / Acceptance structure (§61).
|
||
|
||
### 6.3 Gap consumption schedule
|
||
|
||
| Gap | Consumed at | Handling |
|
||
|---|---|---|
|
||
| AQ-GAP-001 | before Phase 3 table implementations | `VERIFY` against PDF; 35 approved vectors already cover framing/checksum |
|
||
| AQ-GAP-002 | Phase 6 | workaround documented (decode via `Nal`); hypothesis strengthened by all four examples |
|
||
| AQ-GAP-003 | Phase 10 | hypothesis wired to station CFG; simulator validation; close on first real capture |
|
||
| AQ-GAP-004 | text points (none in initial scope) | deferred |
|
||
| AQ-GAP-005 | any time PDF text is available | transcribe long examples → recompute → approve as wraparound fixtures (synthetics cover the branch meanwhile) |
|
||
| AQ-GAP-006 | Phase 3 fixture import | use 540 once PDF-confirmed; verified answer vector approvable independently |
|
||
| AQ-GAP-007 | SCE handling (Phase 3/6) | payload treated as optional diagnostic, never parse-critical |
|
||
| AQ-GAP-008/009/010 | dialled lines / WGD / alarms | extension scope; not in initial release |
|
||
|
||
## 7. Files/components affected
|
||
|
||
`src/{core,aquacom,history,storage,opcua,diagnostics,host}/` per `ARCHITECTURE.md` §6 and dependency direction §7 · `tests/{unit,integration,protocol,opcua,history,acceptance}/` · `simulator/` · `fixtures/` (imported from `docs/protocol/test-vectors.md`) · `schemas/` · `config/examples/` · `README.md`, `CHANGELOG.md` · CI definition · documentation updates in `docs/` as behavior lands (never after).
|
||
|
||
## 8. Data model changes
|
||
|
||
None yet — the canonical contracts are frozen in `ARCHITECTURE.md` §13–§14 and implementation specification §10–§13. Any contract change requires: update the role-authoritative document first, update dependents, update tests, record the decision here. The core module must compile without AquaCom/OPC UA/storage imports (enforced in CI from Phase 1).
|
||
|
||
## 9. Failure handling
|
||
|
||
The failure matrix of `software-specification.md` §35 / `ARCHITECTURE.md` §55 is normative: station failure isolates; storage failure isolates and buffers; OPC UA failure never stops acquisition; malformed telegrams are rejected with diagnostics; restart loses no acknowledged history. Each phase's tests must cover the failure modes it introduces (implementation specification §56 fault injection); every new failure mode gets a diagnostic (AGENTS.md Task completion).
|
||
|
||
## 10. Testing
|
||
|
||
- Pyramid per §50: deterministic unit/component tests dominate; integration via simulator; acceptance scenario at the top.
|
||
- Fixtures: imported **only** from `docs/protocol/test-vectors.md`; approved fixtures immutable; new vectors require evidence and registry update (never delete).
|
||
- Protocol tests run without external network (§57); simulator must not share codec code (§49).
|
||
- Restart-recovery tests at every phase that adds persistent state (§53).
|
||
- Bug fixes carry regression tests (AGENTS.md).
|
||
|
||
## 11. Migration/compatibility
|
||
|
||
Storage schemas versioned with explicit migrations; startup never silently destroys incompatible data; destructive migration requires administrative action (implementation specification §65). `configVersion` gating with explicit migration tooling (§66). OPC UA NodeId stability across releases (§68); configuration schema changes never force namespace changes (`configuration-reference.md` §71).
|
||
|
||
## 12. Progress
|
||
|
||
| Date | Entry |
|
||
|---|---|
|
||
| 2026-10-05 | Documentation set completed and made consistent; protocol evidence acquired (Rev 1.6 source); gap registry (10 entries), operations registry (OPS-001…012), and test-vector catalogue (35 verified vectors) created. Phase 0 documentation deliverables done. |
|
||
| 2026-10-06 | This roadmap created. Awaiting D-001 (technology selection) before Phase 0 code tasks. |
|
||
| 2026-10-06 | `README.md` and `CHANGELOG.md` created — the §5 repository-structure documentation items for Phase 0 are complete. Remaining Phase 0 deliverables: technology selection (D-001), code skeleton, CI, configuration schema. |
|
||
|
||
_Update this log at every phase gate; spawn child ExecPlans for §59-triggering work and link them here._
|
||
|
||
## 13. Decisions log
|
||
|
||
| ID | Date | Decision | Reference |
|
||
|---|---|---|---|
|
||
| DR-001 | 2026-10-05 | Documentation unescaped, de-duplicated; single-source-of-truth precedence defined | `ARCHITECTURE.md` §1 |
|
||
| DR-002 | 2026-10-05 | `AGENTS.md` reduced to short entry-point form per its own spec §4/§6 | `AGENTS.md` |
|
||
| DR-003 | 2026-10-05 | Watermark advances on **durable spool commit**, not final delivery | `ARCHITECTURE.md` §21, INV-014 |
|
||
| DR-004 | 2026-10-05 | Duplicate-conflict default **REJECT_CONFLICT** | software-spec §13 |
|
||
| DR-005 | 2026-10-05 | Deduplication correct across restart (persisted state or idempotent storage) | INV-015 |
|
||
| DR-006 | 2026-10-05 | Pipeline accepts out-of-order historical arrival; queries order by SourceTimestamp | INV-013 |
|
||
| DR-007 | 2026-10-05 | Records without usable SourceTimestamp quarantined with diagnostics | INV-016 |
|
||
| DR-008 | 2026-10-05 | Control/setpoint writes OUT-OF-SCOPE for initial release; read-only integration first | known-operations OPS-005/006/009/010 |
|
||
| DR-009 | 2026-10-05 | WGS decode via `Nal`; `alp` diagnostic-only until AQ-GAP-002 closes | AQ-GAP-002 hypothesis |
|
||
| DR-010 | 2026-10-05 | WGH block value count derived from station CFG | AQ-GAP-003 hypothesis |
|
||
| DR-011 | 2026-10-06 | Umbrella roadmap established; D-001 recorded as the blocking decision | this document |
|
||
|
||
## 14. Unexpected findings
|
||
|
||
- Source document contains checksum/typo defects: MAT fetch request prints 557 (recomputes 540; 557 = SLT fetch checksum — copy-paste error), narrative destination mismatches (§2.2.3.1 "plant 89" vs telegram dest 86; §3.4.10.2 fixed-line dest given as plant number) — recorded in AQ-GAP-005/006.
|
||
- The PDF→markdown conversion is lossy at specific tables and truncates §3.7; byte-exact recomputation of long examples from the conversion is impossible (AQ-GAP-001/005). The PDF is authoritative.
|
||
- The §2.4 checksum algorithm verified exactly against all 35 single-line examples — including the subtle rules that spaces around `#` are summed and Wait bytes are excluded.
|
||
|
||
## 15. Acceptance evidence
|
||
|
||
Production evaluation requires all of (implementation specification §77), evidenced by:
|
||
|
||
1. **Mandatory end-to-end scenario** (§54 T0–T8; `software-specification.md` §46): recorded as an automated acceptance test (T019); passing run attached to the Phase 14 gate.
|
||
2. **Protocol traceability audit:** every implemented operation maps to an `known-operations.md` entry with evidence; remaining gaps explicitly listed (§77 items 2, 15) — the gap registry *is* the evidence.
|
||
3. **Vector suite green:** all approved vectors passing in CI (§77 item 12).
|
||
4. **Restart/data-integrity tests** (§53): no acknowledged history lost across restart, outage, backlog (items 7–9).
|
||
5. **Security tests** (§55): untrusted certs rejected, None disabled in production config, no secrets logged.
|
||
6. **DoD checklist review:** the 15 items of §77 walked item-by-item at G14, each with its artifact.
|
||
7. Items 13 (validated against real equipment/traffic) and operational documentation remain **post-simulator** milestones dependent on field access — explicitly tracked, not silently dropped.
|
||
|
||
## 16. Plan maintenance
|
||
|
||
- Update §12 (Progress) at every phase gate; record new decisions in §13 immediately.
|
||
- Spawn a child ExecPlan (same structure, §59) for: any new AquaCom operation, storage/persistence architecture, OPC UA namespace redesign, security changes, cross-module refactors, data migration. Link children from §12.
|
||
- Move this plan to `docs/exec-plans/completed/` when G14 and §15 are satisfied.
|
||
- This plan never overrides the role-authoritative documents (`ARCHITECTURE.md` §1). |