OPC DA Diagnozer — Implementation Plan

Companion to docs/SPECIFICATION.md. Targets v1.0 as the first production-deployable release.

1. Milestones

Milestone	Content	Exit condition
v0.1 — Foreground proof	CLI reads configured tags once per second in a console, prints events	Successfully reads from a real (or simulation) OPC DA server
v0.2 — Detection engine	All detectors (D1–D15), event model, JSONL logging	Unit tests green; chaos scenarios produce correct events
v0.5 — Windows service	pywin32 service, watchdog/hang recovery, Event Log, CLI install/run	Runs unattended through reboot + server-kill test
v1.0 — Production	Packaging, runbook, 24 h soak, optional webhook	Acceptance criteria AC1–AC10 (spec §13) met

2. Project layout

opcda-diagnozer/
├── README.md
├── pyproject.toml
├── config.example.yaml
├── docs/
│   ├── SPECIFICATION.md
│   └── IMPLEMENTATION_PLAN.md
├── src/opcda_diagnozer/
│   ├── __init__.py
│   ├── __main__.py            # python -m opcda_diagnozer
│   ├── main.py                # CLI: run / install / uninstall / check-config / probe / read / list-items
│   ├── config.py              # pydantic-free typed config (dataclasses) + YAML load + validation
│   ├── events.py              # DiagEvent, Severity, EventKind
│   ├── quality.py             # OPC DA quality decode tables
│   ├── opc/
│   │   ├── base.py            # OpcClient + Browser protocols, ReadResult, OpcError(hresult)
│   │   ├── discovery.py       # list-servers: Automation GetOPCServers / registry scan
│   │   ├── browse.py          # list-devices/list-tags: address-space walk (branches/leaves)
│   │   ├── automation.py      # OPC DA Automation Wrapper adapter (OPCAutomation.dll)
│   │   └── fake.py            # in-process fake client + fake address space (tests, demo)
│   ├── monitor/
│   │   ├── state.py           # connection state machine + backoff
│   │   ├── poller.py          # COM thread: connect → poll loop → feed detectors
│   │   └── detectors.py       # D1–D15 rule implementations
│   ├── report/
│   │   ├── base.py            # Reporter protocol
│   │   ├── jsonl.py           # rotating JSONL file
│   │   ├── textlog.py         # plain-text anomaly log (human-readable, §9.1)
│   │   ├── eventlog.py        # Windows Event Log (WARN+)
│   │   └── http.py            # optional webhook
│   └── service.py             # pywin32 ServiceFramework wrapper
└── tests/
    ├── test_quality.py
    ├── test_detectors.py
    ├── test_state.py
    ├── test_config.py
    └── test_poller_fake.py    # end-to-end against opc.fake, no COM needed

3. Phase 0 — Feasibility & environment (≈0.5–1 day)

Goal: validate the COM access strategy against reality before building anything.

Tasks:

Identify target OPC DA server: ProgID, host, DA version, bitness (32/64-bit EXE?).
Choose wrapper backend:
- A. OPC DA Automation Wrapper (OPCAutomation.dll) — ubiquitous in industry, ships with many SCADA installs; likely default.
- B. OpenOPC / Graybox wrapper (GBDaCtl.dll) — fallback.
- Note: both are typically 32-bit in-proc DLLs → client Python must be 32-bit (see spec §11). Decide interpreter bitness here.
Stand up a test server (Matrikon OPC Simulation Server, KepServerEX demo, or the actual control-system server in a maintenance window) and successfully read one tag from Python (win32com.client.Dispatch).
Verify DCOM configuration for the service account (dcomcnfg: Launch/Activation → Remote Activation; Access → Local+Remote). Record required firewall ports (TCP 135 + RPC dynamic range).
Record baseline behavior: normal read RTT, quality values, what error the wrapper surfaces when the server is stopped (feeds D1 implementation).

Exit: a throwaway script reads ≥ 3 tags from the remote server with measured RTT.

4. Phase 1 — Core engine (≈2–3 days)

config.py — YAML config (see config.example.yaml sketch in README), dataclass model, strict validation, check-config messages that explain exactly what is wrong. Includes the device-link indicators (spec §6.1 / D16): devices: — each entry a human-readable device name plus its connection-indicator tag ItemID. Indicators are added to the polled tag set automatically; check-config warns when an indicator tag is also listed in tags: (it should not be) and fails on an empty device name.
quality.py — decode tables (spec §7), decode_quality(raw: int) -> QualityInfo, plus the clear-text status helper: status_text(raw) -> "NAME (0xNN)" used by every reporter (spec §7 closing rule — e.g. BAD / communication failure (0x18)).
opc/base.py — client abstraction:
```
class OpcClient(Protocol):
    def connect(self) -> None: ...
    def read(self, tags: Sequence[str]) -> list[ReadResult]: ...
    def close(self) -> None: ...
```
ReadResult: tag, value, quality_raw, device_time, error — plus per-call RTT measured by the poller. COM errors are normalized to OpcError carrying the raw HRESULT + decoded name.
opc/automation.py — Automation Wrapper adapter (single group, static items, SyncRead); every COM exception mapped to OpcError with win32com scode.
monitor/state.py — state machine (spec §10.2), backoff, reconnect accounting.
monitor/detectors.py — D1–D15 as small pure functions/classes over PollOutcome (list[ReadResult] + rtt_ms + error), using an injected clock for testability. Rolling window (deque, size window_polls) for p95/streak logic. D16 device-link detector (spec §6.1): per configured indicator — quality not GOOD, a read error, or a GOOD read whose value is not in the configurable ok_values list (default [true]; string-equality compare of the rendered value, so inverted fault flags [false, 0] and string states ["RUN"] both work) ⇒ DEVICE_LINK_DOWN (CRITICAL) carrying the device name, indicator tag, clear-text status and value; recovery (DEVICE_LINK_UP) when it reads GOOD with an ok value again. Edge-level per device, so flapping links don't spam (one event per transition).
monitor/poller.py — the single COM thread; watchdog logic for hang detection (spec §10.4): a second thread timestamps poll starts/ends; on hard_hang_s the COM thread is abandoned and rebuilt.
events.py — event dataclass + JSON serialization.
opc/fake.py — scriptable fake client (sequences of outcomes: good reads, errors, hangs, quality flips) → powers the entire unit-test suite without COM/hardware.

Exit: opcda-diagnozer run (foreground console) produces correct events against the fake client; unit tests for all detectors — including D16 edge semantics (false / non-GOOD / read-error / misconfigured value / recovery) against scripted indicator reads.

5. Phase 2 — Reporters (≈1 day)

report/jsonl.py — rotation (size + backups), atomic-ish append, header line with config digest for provenance.
report/textlog.py — plain-text anomaly log (spec §9.1): anomalies-YYYYMMDD.log in the same directory as the JSONL files, one human-readable line per detected anomaly (timestamp SEVERITY KIND message) with clear-text status NAME (0xNN). Anomalies only — heartbeats/CONNECTED are filtered out (a small is_anomaly(event) predicate on EventKind); recoveries (D2, DEVICE_LINK_UP, quality back to GOOD) are logged, so the file reads as an incident narrative. Daily + size rotation with backups kept, same scheme as the JSONL reporter. Every line corresponds to a JSONL record (text = convenience view, JSONL = source of truth).
report/eventlog.py — pywin32 win32evtlog usage; map severities; source registration via install step.
report/http.py — optional webhook with timeout + simple retry/drop on failure (never blocks the poll loop; queue with bound).
Heartbeat assembly (D15): uptime, reconnects, RTT percentiles, tags-bad counts.

Exit: events visible in file + Event Log; heartbeat cadence verified.

6. Phase 3 — Windows service (≈1 day)

service.py — win32serviceutil.ServiceFramework; service host thread spawns the COM worker; clean stop handling (SetServiceStatus STOP_PENDING, join with timeout, abandon hung COM thread on shutdown).
CLI verbs: install /uninstall /start /stop (wraps sc.exe semantics), run (foreground), check-config, probe (connect once, print one read), list-items (browse validation tool).
SCM recovery config (sc failure ... restart), auto-start, service account documentation.
Config/log default dir: %ProgramData%\OPCDA-Diagnozer\.

Exit: survives reboot; survives OPC-server kill (produces D1/D2 events, reconnects); probe and check-config give usable diagnostics to an ops engineer.

7. Phase 4 — Integration & chaos testing (≈1–2 days)

Scenario matrix against a simulation server (and, when available, the real server):

Scenario	How to inject	Expected events
Server process killed	`taskkill`	D1 (HRESULT), D2 with outage duration
Server service stopped/restarted	`net stop/start`	D1, D2
Firewall drop (silent)	block outbound RPC ports / pull NIC	D7 hang, then D1
Latency injection	Clumsy / `netsh` delay	D4 spikes, D5 streak, D6 trend
Device comm failure	simulation server "communication disturbance" items	D8 with `BAD_COMM_FAILURE`, D10 persistence
Quality oscillation	toggling sim items	D11 flapping
Stale/frozen tag	simulation server static item marked `expect_change`	D12 / D13
Mass bad quality	all sim items to bad	D14
Device link down	indicator tag flipped to `false` / set BAD (alias a bool sim item per device, e.g. `P5001.Connected` on Matrikon)	D16 `DEVICE_LINK_DOWN`, recovery line when restored
Config errors	bad ProgID, unknown tag	fast fail + actionable message; `OPC_E_INVALIDITEMID` surfaced at startup
Soak	24 h healthy run	no leaks, heartbeats every 5 min, zero false positives

Exit: each row verified manually against the event log; AC2/AC3/AC4/AC5 confirmed.

8. Phase 5 — Packaging & docs (≈0.5–1 day)

pyproject.toml wheel installable with pip into the chosen (32-bit) Python; document the on-host install procedure (PyInstaller packaging for pywin32 services is fragile — deferred).
README.md finalized (install, config, DCOM checklist, HRESULT table).
Ops runbook: how to read events, what each kind means (link to spec §12 matrix), how to reconfigure thresholds, where evidence lives.
Version 1.0.0 tag.

9. Testing strategy

Unit (no COM): detectors against scripted PollOutcome sequences with a fake clock; quality decode; config validation; backoff/state transitions; JSONL rotation.
Integration: test_poller_fake.py runs the full poller+reporters against opc.fake in-process — the whole event pipeline without Windows COM.
Manual/chaos: Phase 4 matrix (necessarily on Windows against a real/simulated OPC DA server — cannot be automated in CI without the DCOM environment).
CI: unit + fake-integration tests run on any OS (COM adapters are import-guarded); chaos tests are a documented manual procedure.

10. Risk register

Risk	Impact	Mitigation
Wrapper DLL bitness mismatch (32-bit only)	Cannot load wrapper	Phase 0 decision; use 32-bit Python; keep client abstraction so backend can be swapped
Automation Wrapper not present on diag host	No COM access	Phase 0 check; ship/distribute wrapper install in runbook (vendor redistribution rules apply)
DCOM hardening (KB5004442) blocks activation	Connect fails with `E_ACCESSDENIED`/activation errors	Runbook covers `dcomcnfg` steps; events surface raw HRESULT
COM calls hang indefinitely	Service appears alive but blind	Explicit watchdog + thread recycling (spec §10.4)
COM apartment violations (cross-thread calls)	Random `RPC_E_WRONG_THREAD` crashes	Single dedicated COM thread; all COM handles stay on it
Our client perturbs the system	Skewed diagnosis	One group, static small tag list, 1 s default poll, read-only
pywin32 service + COM quirks (GIL, STA init)	Service instability	Phase 3 spike; foreground `run` mode always available as fallback
Tag list too large / wrong item IDs	Noise, misleading RTT	`probe` + `check-config` validate up-front; per-poll error accounting
SCADA/host clock skew	Misleading staleness events	Local receive time always recorded; `max_age_s` configurable

11. Effort estimate

Phase	Estimate
0 Feasibility	0.5–1 d
1 Core engine	2–3 d
2 Reporters	1 d
3 Service	1 d
4 Chaos testing	1–2 d
5 Packaging/docs	0.5–1 d
Total v1.0	≈ 6–9 days

12. Definition of done (v1.0)

All Phase 4 scenarios pass; AC1–AC10 (spec §13) verified.
24 h soak clean.
README + runbook complete enough for an ops engineer to deploy unaided.
Code reviewed; no write path to OPC anywhere in the codebase.
Config example covers every configurable threshold.

14 KiB Raw Blame History Unescape Escape