# OPC DA Diagnozer — Implementation Plan

Companion to [`docs/SPECIFICATION.md`](SPECIFICATION.md). Targets **v1.0** as the first
production-deployable release.

---

## 1. Milestones

| Milestone | Content | Exit condition |
|-----------|---------|----------------|
| **v0.1 — Foreground proof** | CLI reads configured tags once per second in a console, prints events | Successfully reads from a real (or simulation) OPC DA server |
| **v0.2 — Detection engine** | All detectors (D1–D15), event model, JSONL logging | Unit tests green; chaos scenarios produce correct events |
| **v0.5 — Windows service** | pywin32 service, watchdog/hang recovery, Event Log, CLI install/run | Runs unattended through reboot + server-kill test |
| **v1.0 — Production** | Packaging, runbook, 24 h soak, optional webhook | Acceptance criteria AC1–AC10 (spec §13) met |

## 2. Project layout

```
opcda-diagnozer/
├── README.md
├── pyproject.toml
├── config.example.yaml
├── docs/
│   ├── SPECIFICATION.md
│   └── IMPLEMENTATION_PLAN.md
├── src/opcda_diagnozer/
│   ├── __init__.py
│   ├── __main__.py            # python -m opcda_diagnozer
│   ├── main.py                # CLI: run / install / uninstall / check-config / probe / list-devices / list-tags
│   ├── config.py              # pydantic-free typed config (dataclasses) + YAML load + validation
│   ├── events.py              # DiagEvent, Severity, EventKind
│   ├── quality.py             # OPC DA quality decode tables
│   ├── opc/
│   │   ├── base.py            # OpcClient + Browser protocols, ReadResult, OpcError(hresult)
│   │   ├── discovery.py       # list-servers: Automation GetOPCServers / registry scan
│   │   ├── browse.py          # list-devices/list-tags: address-space walk (branches/leaves)
│   │   ├── automation.py      # OPC DA Automation Wrapper adapter (OPCAutomation.dll)
│   │   └── fake.py            # in-process fake client + fake address space (tests, demo)
│   ├── monitor/
│   │   ├── state.py           # connection state machine + backoff
│   │   ├── poller.py          # COM thread: connect → poll loop → feed detectors
│   │   └── detectors.py       # D1–D15 rule implementations
│   ├── report/
│   │   ├── base.py            # Reporter protocol
│   │   ├── jsonl.py           # rotating JSONL file
│   │   ├── eventlog.py        # Windows Event Log (WARN+)
│   │   └── http.py            # optional webhook
│   └── service.py             # pywin32 ServiceFramework wrapper
└── tests/
    ├── test_quality.py
    ├── test_detectors.py
    ├── test_state.py
    ├── test_config.py
    └── test_poller_fake.py    # end-to-end against opc.fake, no COM needed
```

## 3. Phase 0 — Feasibility & environment (≈0.5–1 day)

**Goal:** validate the COM access strategy against reality before building anything.

Tasks:
1. Identify target OPC DA server: ProgID, host, DA version, bitness (32/64-bit EXE?).
2. Choose wrapper backend:
   - **A. OPC DA Automation Wrapper** (`OPCAutomation.dll`) — ubiquitous in industry,
     ships with many SCADA installs; likely default.
   - **B. OpenOPC / Graybox wrapper** (`GBDaCtl.dll`) — fallback.
   - Note: both are typically **32-bit in-proc DLLs** → client Python must be 32-bit
     (see spec §11). Decide interpreter bitness here.
3. Stand up a **test server** (Matrikon OPC Simulation Server, KepServerEX demo, or the
     actual control-system server in a maintenance window) and successfully read one tag
     from Python (`win32com.client.Dispatch`).
4. Verify DCOM configuration for the service account (`dcomcnfg`: Launch/Activation →
   Remote Activation; Access → Local+Remote). Record required firewall ports
   (TCP 135 + RPC dynamic range).
5. Record baseline behavior: normal read RTT, quality values, what error the wrapper
   surfaces when the server is stopped (feeds D1 implementation).

**Exit:** a throwaway script reads ≥ 3 tags from the remote server with measured RTT.

## 4. Phase 1 — Core engine (≈2–3 days)

1. `config.py` — YAML config (see `config.example.yaml` sketch in README), dataclass
   model, strict validation, `check-config` messages that explain exactly what is wrong.
2. `quality.py` — decode tables (spec §7), `decode_quality(raw: int) -> QualityInfo`.
3. `opc/base.py` — client abstraction:
   ```python
   class OpcClient(Protocol):
       def connect(self) -> None: ...
       def read(self, tags: Sequence[str]) -> list[ReadResult]: ...
       def close(self) -> None: ...
   ```
   `ReadResult`: `tag, value, quality_raw, device_time, error` — plus per-call RTT
   measured by the poller. COM errors are normalized to `OpcError` carrying the raw
   `HRESULT` + decoded name.
4. `opc/automation.py` — Automation Wrapper adapter (single group, static items,
   `SyncRead`); every COM exception mapped to `OpcError` with `win32com` scode.
5. `monitor/state.py` — state machine (spec §10.2), backoff, reconnect accounting.
6. `monitor/detectors.py` — D1–D15 as small pure functions/classes over
   `PollOutcome` (list[ReadResult] + rtt_ms + error), using an injected clock for
   testability. Rolling window (deque, size `window_polls`) for p95/streak logic.
7. `monitor/poller.py` — the single COM thread; watchdog logic for hang detection
   (spec §10.4): a second thread timestamps poll starts/ends; on `hard_hang_s` the COM
   thread is abandoned and rebuilt.
8. `events.py` — event dataclass + JSON serialization.
9. `opc/fake.py` — scriptable fake client (sequences of outcomes: good reads, errors,
   hangs, quality flips) → powers the entire unit-test suite without COM/hardware.

**Exit:** `opcda-diagnozer run` (foreground console) produces correct events against the
fake client; unit tests for all detectors.

## 5. Phase 2 — Reporters (≈1 day)

1. `report/jsonl.py` — rotation (size + backups), atomic-ish append, header line with
   config digest for provenance.
2. `report/eventlog.py` — pywin32 `win32evtlog` usage; map severities; source
   registration via `install` step.
3. `report/http.py` — optional webhook with timeout + simple retry/drop on failure
   (never blocks the poll loop; queue with bound).
4. Heartbeat assembly (D15): uptime, reconnects, RTT percentiles, tags-bad counts.

**Exit:** events visible in file + Event Log; heartbeat cadence verified.

## 6. Phase 3 — Windows service (≈1 day)

1. `service.py` — `win32serviceutil.ServiceFramework`; service host thread spawns the
   COM worker; clean `stop` handling (`SetServiceStatus STOP_PENDING`, join with
   timeout, abandon hung COM thread on shutdown).
2. CLI verbs: `install /uninstall /start /stop` (wraps `sc.exe` semantics),
   `run` (foreground), `check-config`, `probe` (connect once, print one read),
   `list-items` (browse validation tool).
3. SCM recovery config (`sc failure ... restart`), auto-start, service account
   documentation.
4. Config/log default dir: `%ProgramData%\OPCDA-Diagnozer\`.

**Exit:** survives reboot; survives OPC-server kill (produces D1/D2 events, reconnects);
`probe` and `check-config` give usable diagnostics to an ops engineer.

## 7. Phase 4 — Integration & chaos testing (≈1–2 days)

Scenario matrix against a simulation server (and, when available, the real server):

| Scenario | How to inject | Expected events |
|----------|---------------|-----------------|
| Server process killed | `taskkill` | D1 (HRESULT), D2 with outage duration |
| Server service stopped/restarted | `net stop/start` | D1, D2 |
| Firewall drop (silent) | block outbound RPC ports / pull NIC | D7 hang, then D1 |
| Latency injection | Clumsy / `netsh` delay | D4 spikes, D5 streak, D6 trend |
| Device comm failure | simulation server "communication disturbance" items | D8 with `BAD_COMM_FAILURE`, D10 persistence |
| Quality oscillation | toggling sim items | D11 flapping |
| Stale/frozen tag | simulation server static item marked `expect_change` | D12 / D13 |
| Mass bad quality | all sim items to bad | D14 |
| Config errors | bad ProgID, unknown tag | fast fail + actionable message; `OPC_E_INVALIDITEMID` surfaced at startup |
| Soak | 24 h healthy run | no leaks, heartbeats every 5 min, zero false positives |

**Exit:** each row verified manually against the event log; AC2/AC3/AC4/AC5 confirmed.

## 8. Phase 5 — Packaging & docs (≈0.5–1 day)

- `pyproject.toml` wheel installable with pip into the chosen (32-bit) Python;
  document the on-host install procedure (PyInstaller packaging for pywin32 services is
  fragile — deferred).
- `README.md` finalized (install, config, DCOM checklist, HRESULT table).
- Ops runbook: how to read events, what each kind means (link to spec §12 matrix),
  how to reconfigure thresholds, where evidence lives.
- Version 1.0.0 tag.

## 9. Testing strategy

- **Unit** (no COM): detectors against scripted `PollOutcome` sequences with a fake
  clock; quality decode; config validation; backoff/state transitions; JSONL rotation.
- **Integration:** `test_poller_fake.py` runs the full poller+reporters against
  `opc.fake` in-process — the whole event pipeline without Windows COM.
- **Manual/chaos:** Phase 4 matrix (necessarily on Windows against a real/simulated
  OPC DA server — cannot be automated in CI without the DCOM environment).
- CI: unit + fake-integration tests run on any OS (COM adapters are import-guarded);
  chaos tests are a documented manual procedure.

## 10. Risk register

| Risk | Impact | Mitigation |
|------|--------|------------|
| Wrapper DLL bitness mismatch (32-bit only) | Cannot load wrapper | Phase 0 decision; use 32-bit Python; keep client abstraction so backend can be swapped |
| Automation Wrapper not present on diag host | No COM access | Phase 0 check; ship/distribute wrapper install in runbook (vendor redistribution rules apply) |
| DCOM hardening (KB5004442) blocks activation | Connect fails with `E_ACCESSDENIED`/activation errors | Runbook covers `dcomcnfg` steps; events surface raw HRESULT |
| COM calls hang indefinitely | Service appears alive but blind | Explicit watchdog + thread recycling (spec §10.4) |
| COM apartment violations (cross-thread calls) | Random `RPC_E_WRONG_THREAD` crashes | Single dedicated COM thread; all COM handles stay on it |
| Our client perturbs the system | Skewed diagnosis | One group, static small tag list, 1 s default poll, read-only |
| pywin32 service + COM quirks (GIL, STA init) | Service instability | Phase 3 spike; foreground `run` mode always available as fallback |
| Tag list too large / wrong item IDs | Noise, misleading RTT | `probe` + `check-config` validate up-front; per-poll error accounting |
| SCADA/host clock skew | Misleading staleness events | Local receive time always recorded; `max_age_s` configurable |

## 11. Effort estimate

| Phase | Estimate |
|-------|----------|
| 0 Feasibility | 0.5–1 d |
| 1 Core engine | 2–3 d |
| 2 Reporters | 1 d |
| 3 Service | 1 d |
| 4 Chaos testing | 1–2 d |
| 5 Packaging/docs | 0.5–1 d |
| **Total v1.0** | **≈ 6–9 days** |

## 12. Definition of done (v1.0)

- All Phase 4 scenarios pass; AC1–AC10 (spec §13) verified.
- 24 h soak clean.
- README + runbook complete enough for an ops engineer to deploy unaided.
- Code reviewed; no write path to OPC anywhere in the codebase.
- Config example covers every configurable threshold.