236 lines
14 KiB
Markdown
236 lines
14 KiB
Markdown
# OPC DA Diagnozer — Implementation Plan
|
||
|
||
Companion to [`docs/SPECIFICATION.md`](SPECIFICATION.md). Targets **v1.0** as the first
|
||
production-deployable release.
|
||
|
||
---
|
||
|
||
## 1. Milestones
|
||
|
||
| Milestone | Content | Exit condition |
|
||
|-----------|---------|----------------|
|
||
| **v0.1 — Foreground proof** | CLI reads configured tags once per second in a console, prints events | Successfully reads from a real (or simulation) OPC DA server |
|
||
| **v0.2 — Detection engine** | All detectors (D1–D15), event model, JSONL logging | Unit tests green; chaos scenarios produce correct events |
|
||
| **v0.5 — Windows service** | pywin32 service, watchdog/hang recovery, Event Log, CLI install/run | Runs unattended through reboot + server-kill test |
|
||
| **v1.0 — Production** | Packaging, runbook, 24 h soak, optional webhook | Acceptance criteria AC1–AC10 (spec §13) met |
|
||
|
||
## 2. Project layout
|
||
|
||
```
|
||
opcda-diagnozer/
|
||
├── README.md
|
||
├── pyproject.toml
|
||
├── config.example.yaml
|
||
├── docs/
|
||
│ ├── SPECIFICATION.md
|
||
│ └── IMPLEMENTATION_PLAN.md
|
||
├── src/opcda_diagnozer/
|
||
│ ├── __init__.py
|
||
│ ├── __main__.py # python -m opcda_diagnozer
|
||
│ ├── main.py # CLI: run / install / uninstall / check-config / probe / read / list-items
|
||
│ ├── config.py # pydantic-free typed config (dataclasses) + YAML load + validation
|
||
│ ├── events.py # DiagEvent, Severity, EventKind
|
||
│ ├── quality.py # OPC DA quality decode tables
|
||
│ ├── opc/
|
||
│ │ ├── base.py # OpcClient + Browser protocols, ReadResult, OpcError(hresult)
|
||
│ │ ├── discovery.py # list-servers: Automation GetOPCServers / registry scan
|
||
│ │ ├── browse.py # list-devices/list-tags: address-space walk (branches/leaves)
|
||
│ │ ├── automation.py # OPC DA Automation Wrapper adapter (OPCAutomation.dll)
|
||
│ │ └── fake.py # in-process fake client + fake address space (tests, demo)
|
||
│ ├── monitor/
|
||
│ │ ├── state.py # connection state machine + backoff
|
||
│ │ ├── poller.py # COM thread: connect → poll loop → feed detectors
|
||
│ │ └── detectors.py # D1–D15 rule implementations
|
||
│ ├── report/
|
||
│ │ ├── base.py # Reporter protocol
|
||
│ │ ├── jsonl.py # rotating JSONL file
|
||
│ │ ├── textlog.py # plain-text anomaly log (human-readable, §9.1)
|
||
│ │ ├── eventlog.py # Windows Event Log (WARN+)
|
||
│ │ └── http.py # optional webhook
|
||
│ └── service.py # pywin32 ServiceFramework wrapper
|
||
└── tests/
|
||
├── test_quality.py
|
||
├── test_detectors.py
|
||
├── test_state.py
|
||
├── test_config.py
|
||
└── test_poller_fake.py # end-to-end against opc.fake, no COM needed
|
||
```
|
||
|
||
## 3. Phase 0 — Feasibility & environment (≈0.5–1 day)
|
||
|
||
**Goal:** validate the COM access strategy against reality before building anything.
|
||
|
||
Tasks:
|
||
1. Identify target OPC DA server: ProgID, host, DA version, bitness (32/64-bit EXE?).
|
||
2. Choose wrapper backend:
|
||
- **A. OPC DA Automation Wrapper** (`OPCAutomation.dll`) — ubiquitous in industry,
|
||
ships with many SCADA installs; likely default.
|
||
- **B. OpenOPC / Graybox wrapper** (`GBDaCtl.dll`) — fallback.
|
||
- Note: both are typically **32-bit in-proc DLLs** → client Python must be 32-bit
|
||
(see spec §11). Decide interpreter bitness here.
|
||
3. Stand up a **test server** (Matrikon OPC Simulation Server, KepServerEX demo, or the
|
||
actual control-system server in a maintenance window) and successfully read one tag
|
||
from Python (`win32com.client.Dispatch`).
|
||
4. Verify DCOM configuration for the service account (`dcomcnfg`: Launch/Activation →
|
||
Remote Activation; Access → Local+Remote). Record required firewall ports
|
||
(TCP 135 + RPC dynamic range).
|
||
5. Record baseline behavior: normal read RTT, quality values, what error the wrapper
|
||
surfaces when the server is stopped (feeds D1 implementation).
|
||
|
||
**Exit:** a throwaway script reads ≥ 3 tags from the remote server with measured RTT.
|
||
|
||
## 4. Phase 1 — Core engine (≈2–3 days)
|
||
|
||
1. `config.py` — YAML config (see `config.example.yaml` sketch in README), dataclass
|
||
model, strict validation, `check-config` messages that explain exactly what is wrong.
|
||
Includes the **device-link indicators** (spec §6.1 / D16): `devices:` — each entry a
|
||
human-readable device name plus its connection-indicator tag ItemID. Indicators are
|
||
added to the polled tag set automatically; `check-config` warns when an indicator tag
|
||
is also listed in `tags:` (it should not be) and fails on an empty device name.
|
||
2. `quality.py` — decode tables (spec §7), `decode_quality(raw: int) -> QualityInfo`,
|
||
plus the **clear-text status helper**: `status_text(raw) -> "NAME (0xNN)"` used by
|
||
every reporter (spec §7 closing rule — e.g. `BAD / communication failure (0x18)`).
|
||
3. `opc/base.py` — client abstraction:
|
||
```python
|
||
class OpcClient(Protocol):
|
||
def connect(self) -> None: ...
|
||
def read(self, tags: Sequence[str]) -> list[ReadResult]: ...
|
||
def close(self) -> None: ...
|
||
```
|
||
`ReadResult`: `tag, value, quality_raw, device_time, error` — plus per-call RTT
|
||
measured by the poller. COM errors are normalized to `OpcError` carrying the raw
|
||
`HRESULT` + decoded name.
|
||
4. `opc/automation.py` — Automation Wrapper adapter (single group, static items,
|
||
`SyncRead`); every COM exception mapped to `OpcError` with `win32com` scode.
|
||
5. `monitor/state.py` — state machine (spec §10.2), backoff, reconnect accounting.
|
||
6. `monitor/detectors.py` — D1–D15 as small pure functions/classes over
|
||
`PollOutcome` (list[ReadResult] + rtt_ms + error), using an injected clock for
|
||
testability. Rolling window (deque, size `window_polls`) for p95/streak logic.
|
||
**D16 device-link detector** (spec §6.1): per configured indicator — quality
|
||
not GOOD, a read error, or a GOOD read whose value is **not in the
|
||
configurable `ok_values` list** (default `[true]`; string-equality compare
|
||
of the rendered value, so inverted fault flags `[false, 0]` and string states
|
||
`["RUN"]` both work) ⇒ `DEVICE_LINK_DOWN` (CRITICAL) carrying the device name,
|
||
indicator tag, clear-text status and value; recovery (`DEVICE_LINK_UP`) when
|
||
it reads GOOD with an ok value again. Edge-level per device, so flapping
|
||
links don't spam (one event per transition).
|
||
7. `monitor/poller.py` — the single COM thread; watchdog logic for hang detection
|
||
(spec §10.4): a second thread timestamps poll starts/ends; on `hard_hang_s` the COM
|
||
thread is abandoned and rebuilt.
|
||
8. `events.py` — event dataclass + JSON serialization.
|
||
9. `opc/fake.py` — scriptable fake client (sequences of outcomes: good reads, errors,
|
||
hangs, quality flips) → powers the entire unit-test suite without COM/hardware.
|
||
|
||
**Exit:** `opcda-diagnozer run` (foreground console) produces correct events against the
|
||
fake client; unit tests for all detectors — including D16 edge semantics (false /
|
||
non-GOOD / read-error / misconfigured value / recovery) against scripted indicator reads.
|
||
|
||
## 5. Phase 2 — Reporters (≈1 day)
|
||
|
||
1. `report/jsonl.py` — rotation (size + backups), atomic-ish append, header line with
|
||
config digest for provenance.
|
||
2. `report/textlog.py` — **plain-text anomaly log** (spec §9.1): `anomalies-YYYYMMDD.log`
|
||
in the same directory as the JSONL files, one human-readable line per detected
|
||
anomaly (`timestamp SEVERITY KIND message`) with clear-text status `NAME (0xNN)`.
|
||
Anomalies only — heartbeats/CONNECTED are filtered out (a small `is_anomaly(event)`
|
||
predicate on `EventKind`); recoveries (D2, `DEVICE_LINK_UP`, quality back to GOOD)
|
||
**are** logged, so the file reads as an incident narrative. Daily + size rotation
|
||
with `backups` kept, same scheme as the JSONL reporter. Every line corresponds to
|
||
a JSONL record (text = convenience view, JSONL = source of truth).
|
||
3. `report/eventlog.py` — pywin32 `win32evtlog` usage; map severities; source
|
||
registration via `install` step.
|
||
4. `report/http.py` — optional webhook with timeout + simple retry/drop on failure
|
||
(never blocks the poll loop; queue with bound).
|
||
5. Heartbeat assembly (D15): uptime, reconnects, RTT percentiles, tags-bad counts.
|
||
|
||
**Exit:** events visible in file + Event Log; heartbeat cadence verified.
|
||
|
||
## 6. Phase 3 — Windows service (≈1 day)
|
||
|
||
1. `service.py` — `win32serviceutil.ServiceFramework`; service host thread spawns the
|
||
COM worker; clean `stop` handling (`SetServiceStatus STOP_PENDING`, join with
|
||
timeout, abandon hung COM thread on shutdown).
|
||
2. CLI verbs: `install /uninstall /start /stop` (wraps `sc.exe` semantics),
|
||
`run` (foreground), `check-config`, `probe` (connect once, print one read),
|
||
`list-items` (browse validation tool).
|
||
3. SCM recovery config (`sc failure ... restart`), auto-start, service account
|
||
documentation.
|
||
4. Config/log default dir: `%ProgramData%\OPCDA-Diagnozer\`.
|
||
|
||
**Exit:** survives reboot; survives OPC-server kill (produces D1/D2 events, reconnects);
|
||
`probe` and `check-config` give usable diagnostics to an ops engineer.
|
||
|
||
## 7. Phase 4 — Integration & chaos testing (≈1–2 days)
|
||
|
||
Scenario matrix against a simulation server (and, when available, the real server):
|
||
|
||
| Scenario | How to inject | Expected events |
|
||
|----------|---------------|-----------------|
|
||
| Server process killed | `taskkill` | D1 (HRESULT), D2 with outage duration |
|
||
| Server service stopped/restarted | `net stop/start` | D1, D2 |
|
||
| Firewall drop (silent) | block outbound RPC ports / pull NIC | D7 hang, then D1 |
|
||
| Latency injection | Clumsy / `netsh` delay | D4 spikes, D5 streak, D6 trend |
|
||
| Device comm failure | simulation server "communication disturbance" items | D8 with `BAD_COMM_FAILURE`, D10 persistence |
|
||
| Quality oscillation | toggling sim items | D11 flapping |
|
||
| Stale/frozen tag | simulation server static item marked `expect_change` | D12 / D13 |
|
||
| Mass bad quality | all sim items to bad | D14 |
|
||
| **Device link down** | indicator tag flipped to `false` / set BAD (alias a bool sim item per device, e.g. `P5001.Connected` on Matrikon) | D16 `DEVICE_LINK_DOWN`, recovery line when restored |
|
||
| Config errors | bad ProgID, unknown tag | fast fail + actionable message; `OPC_E_INVALIDITEMID` surfaced at startup |
|
||
| Soak | 24 h healthy run | no leaks, heartbeats every 5 min, zero false positives |
|
||
|
||
**Exit:** each row verified manually against the event log; AC2/AC3/AC4/AC5 confirmed.
|
||
|
||
## 8. Phase 5 — Packaging & docs (≈0.5–1 day)
|
||
|
||
- `pyproject.toml` wheel installable with pip into the chosen (32-bit) Python;
|
||
document the on-host install procedure (PyInstaller packaging for pywin32 services is
|
||
fragile — deferred).
|
||
- `README.md` finalized (install, config, DCOM checklist, HRESULT table).
|
||
- Ops runbook: how to read events, what each kind means (link to spec §12 matrix),
|
||
how to reconfigure thresholds, where evidence lives.
|
||
- Version 1.0.0 tag.
|
||
|
||
## 9. Testing strategy
|
||
|
||
- **Unit** (no COM): detectors against scripted `PollOutcome` sequences with a fake
|
||
clock; quality decode; config validation; backoff/state transitions; JSONL rotation.
|
||
- **Integration:** `test_poller_fake.py` runs the full poller+reporters against
|
||
`opc.fake` in-process — the whole event pipeline without Windows COM.
|
||
- **Manual/chaos:** Phase 4 matrix (necessarily on Windows against a real/simulated
|
||
OPC DA server — cannot be automated in CI without the DCOM environment).
|
||
- CI: unit + fake-integration tests run on any OS (COM adapters are import-guarded);
|
||
chaos tests are a documented manual procedure.
|
||
|
||
## 10. Risk register
|
||
|
||
| Risk | Impact | Mitigation |
|
||
|------|--------|------------|
|
||
| Wrapper DLL bitness mismatch (32-bit only) | Cannot load wrapper | Phase 0 decision; use 32-bit Python; keep client abstraction so backend can be swapped |
|
||
| Automation Wrapper not present on diag host | No COM access | Phase 0 check; ship/distribute wrapper install in runbook (vendor redistribution rules apply) |
|
||
| DCOM hardening (KB5004442) blocks activation | Connect fails with `E_ACCESSDENIED`/activation errors | Runbook covers `dcomcnfg` steps; events surface raw HRESULT |
|
||
| COM calls hang indefinitely | Service appears alive but blind | Explicit watchdog + thread recycling (spec §10.4) |
|
||
| COM apartment violations (cross-thread calls) | Random `RPC_E_WRONG_THREAD` crashes | Single dedicated COM thread; all COM handles stay on it |
|
||
| Our client perturbs the system | Skewed diagnosis | One group, static small tag list, 1 s default poll, read-only |
|
||
| pywin32 service + COM quirks (GIL, STA init) | Service instability | Phase 3 spike; foreground `run` mode always available as fallback |
|
||
| Tag list too large / wrong item IDs | Noise, misleading RTT | `probe` + `check-config` validate up-front; per-poll error accounting |
|
||
| SCADA/host clock skew | Misleading staleness events | Local receive time always recorded; `max_age_s` configurable |
|
||
|
||
## 11. Effort estimate
|
||
|
||
| Phase | Estimate |
|
||
|-------|----------|
|
||
| 0 Feasibility | 0.5–1 d |
|
||
| 1 Core engine | 2–3 d |
|
||
| 2 Reporters | 1 d |
|
||
| 3 Service | 1 d |
|
||
| 4 Chaos testing | 1–2 d |
|
||
| 5 Packaging/docs | 0.5–1 d |
|
||
| **Total v1.0** | **≈ 6–9 days** |
|
||
|
||
## 12. Definition of done (v1.0)
|
||
|
||
- All Phase 4 scenarios pass; AC1–AC10 (spec §13) verified.
|
||
- 24 h soak clean.
|
||
- README + runbook complete enough for an ops engineer to deploy unaided.
|
||
- Code reviewed; no write path to OPC anywhere in the codebase.
|
||
- Config example covers every configurable threshold. |