🎉 Initial commit

2026-10-07 10:38:21 +02:00
commit 1c64aeb071
38 changed files with 3391 additions and 0 deletions
@@ -0,0 +1,263 @@
+# OPC DA Diagnozer — Project Specification
+
+**Version:** 0.1 (draft)
+**Status:** Planning
+**Target platform:** Windows only (OPC DA is a COM/DCOM technology)
+
+---
+
+## 1. Problem statement
+
+We experience intermittent **disconnections** and/or **high communication latency** between a
+SCADA system and a control system that communicate over **OPC DA** (Data Access, the
+COM/DCOM-based predecessor of OPC UA). Today we have no objective evidence to answer the
+key triage question:
+
+> *Is the problem in the SCADA host, the network/DCOM layer, the OPC DA server, or the
+> link between the OPC server and the control devices?*
+
+OPC DA failure modes are notoriously hard to diagnose because failures surface as generic
+`HRESULT` errors, hung calls with **no client-side timeout**, or silently degraded item
+quality — long before the SCADA operator sees anything.
+
+## 2. Approach
+
+Deploy an **independent, read-only OPC DA reference client** that continuously exercises
+the same path the SCADA uses:
+
+- Connect to the OPC DA server (local or remote via DCOM) from a Windows machine.
+- Poll a small, pre-defined list of tags at a fixed interval.
+- Measure **read round-trip time (RTT)**, track **item quality** (GOOD / UNCERTAIN / BAD
+  and subcodes), **device timestamps**, and **connection state**.
+- Emit structured, timestamped **diagnostic events** whenever anything deviates from
+  normal — disconnects, slow reads, hung calls, quality degradation, frozen/stale values.
+
+Because the client is independent of the SCADA, its observations provide a **reference
+baseline** against which the SCADA's complaints can be triangulated (see §12).
+
+## 3. Deployment architecture
+
+```mermaid
+flowchart LR
+    subgraph SCADAzone["SCADA zone (or dedicated diag host)"]
+        DIAG["OPC DA Diagnozer<br/>(Windows service)"]
+    end
+    subgraph CTL["Control system / OPC DA server host"]
+        OPC["OPC DA server<br/>(COM/DCOM out-of-proc)"]
+        DEV["Control devices / PLC link"]
+    end
+    DIAG -- "DCOM reads (RPC 135 + dynamic ports)" --> OPC
+    OPC -.-> DEV
+    DIAG -- "Events (JSONL / Event Log / HTTP)" --> OPS["Engineers & monitoring"]
+```
+
+Placement note: run the tool **as close to the SCADA machine as possible** (same host or
+same network segment) so its measured latencies are comparable to what the SCADA
+experiences. Optionally run a second instance next to the OPC server host for differential
+diagnosis.
+
+## 4. Goals
+
+1. Run unattended as a **Windows service** (auto-start, SCM crash recovery).
+2. Connect to a configured OPC DA server (local or remote DCOM) and read a configured tag list.
+3. Detect and report, with precise timestamps:
+   - disconnections and reconnects (including DCOM error codes),
+   - hung / timed-out RPC calls,
+   - read latencies exceeding thresholds (single spikes, streaks, and trends),
+   - tags whose quality is BAD or UNCERTAIN (with decoded subcode),
+   - persistent-bad and flapping (oscillating) quality per tag,
+   - stale device timestamps / frozen values while quality reports GOOD,
+   - mass quality degradation (many tags BAD at once → control-link problem).
+4. Produce durable, machine-readable evidence (structured logs + Windows Event Log +
+   optional HTTP webhook) suitable for incident review and correlation.
+5. Be **strictly read-only** and lightweight — must not perturb the system it measures.
+
+## 5. Non-goals (v1)
+
+- No writes to the OPC server (no `Write` code path at all).
+- No OPC UA, OPC HDA, or Alarms & Events support.
+- No data historian / storage of every sample (only events + periodic summaries).
+- No UI/dashboard in v1 (JSONL logs + webhook are enough for Loki/Grafana ingestion).
+- No automatic remediation (diagnose only).
+
+## 6. Detection rules
+
+| ID | Condition | Default rule | Severity |
+|----|-----------|--------------|----------|
+| D1 | Connection lost | Any read/connect COM failure | CRITICAL (event incl. decoded HRESULT) |
+| D2 | Reconnect succeeded | After D1, reconnect works | INFO (event with outage duration) |
+| D3 | Reconnect flapping | > 3 disconnects in 10 min | CRITICAL |
+| D4 | Slow poll (spike) | Single read RTT > `warn_ms` (default 1000) | WARN |
+| D5 | Sustained slow polls | RTT > `warn_ms` for `slow_poll_streak` (5) consecutive polls | CRITICAL |
+| D6 | Latency trend | p95 RTT over last `window_polls` (60) > `warn_ms` | WARN |
+| D7 | Poll hang | Read call does not return within `timeout_ms` (5000) — see §10.4 | CRITICAL (immediate), see recovery behavior |
+| D8 | Tag quality → BAD | Any transition into BAD, decoded subcode | WARN (per tag) |
+| D9 | Tag quality → UNCERTAIN | Any transition into UNCERTAIN | INFO |
+| D10 | Persistent bad tag | Single tag BAD longer than `persistent_bad_s` (30) | WARN (escalated from D8) |
+| D11 | Quality flapping | > 5 quality transitions for one tag within 10 min | WARN |
+| D12 | Stale timestamp | Device timestamp older than `max_age_s` (120) while quality GOOD | WARN |
+| D13 | Frozen value | `expect_change: true` tag with unchanged value+timestamp for `max_age_s` | WARN |
+| D14 | Mass quality degradation | ≥ 50% of monitored tags BAD simultaneously while connection OK | CRITICAL (`CONTROL_LINK_DEGRADED` — points at OPC-server↔device link, not DCOM) |
+| D15 | Heartbeat / summary | Periodic stats snapshot every `heartbeat_s` (300) | INFO |
+
+All thresholds are configurable; severities are INFO / WARN / CRITICAL.
+
+## 7. OPC DA quality codes (reference)
+
+OPC DA quality is a 16-bit field. Bits 7–6 of the low byte give the class:
+`0xC0` GOOD, `0x40` UNCERTAIN, `0x00` BAD. Bits 5–0 are the substatus. Values the tool
+decodes and names:
+
+| Code | Meaning |
+|------|---------|
+| `0xC0` | GOOD |
+| `0xD8` | GOOD / local override |
+| `0x44` | UNCERTAIN / last usable value |
+| `0x40` | UNCERTAIN (unspecified) |
+| `0x00` | BAD (unspecified) |
+| `0x04` | BAD / configuration error |
+| `0x08` | BAD / not connected |
+| `0x0C` | BAD / device failure |
+| `0x10` | BAD / sensor failure |
+| `0x14` | BAD / last known value |
+| `0x18` | BAD / communication failure |
+| `0x1C` | BAD / out of service |
+
+Remaining UNCERTAIN subcodes (DA 2.05a) are decoded by the same table in `quality.py`.
+Every quality event carries the raw hex code **and** the decoded name.
+
+## 8. Event model
+
+Every event is a structured record:
+
+```json
+{
+  "time_utc": "2026-10-07T12:34:56.789Z",
+  "kind": "QUALITY_BAD",
+  "severity": "WARN",
+  "detail": {
+    "tag": "Channel1.Device1.Counter",
+    "quality_raw": "0x18",
+    "quality": "BAD_COMM_FAILURE",
+    "previous_quality": "GOOD",
+    "value": 1234.0,
+    "device_time_utc": "2026-10-07T12:34:55.100Z",
+    "rtt_ms": 42.1
+  },
+  "context": {
+    "uptime_s": 3600.5,
+    "reconnects_total": 2,
+    "rtt_p95_ms": 55.2,
+    "tags_bad": 3,
+    "tags_total": 25
+  }
+}
+```
+
+`time_utc` is always the **local client receive time** (wall clock), so events remain
+comparable even if the remote host's clock is wrong. Device timestamps are reported
+alongside and clearly labeled.
+
+## 9. Outputs (reporters)
+
+| Reporter | Purpose | Notes |
+|----------|---------|-------|
+| JSONL rotating file | Primary evidence trail | One JSON event per line; rotation by size; default dir `%ProgramData%\OPCDA-Diagnozer\logs` |
+| Windows Event Log | Ops visibility | WARN+ only; source `OPCDA-Diagnozer` |
+| Console | Foreground/debug runs | Human-readable |
+| HTTP webhook (optional) | Alert integration | POST one JSON event per request, optional bearer token |
+
+Reporters are pluggable (`Reporter.emit(event)`); a heartbeat event every 5 minutes
+proves liveness and enables baseline graphs even when everything is healthy.
+
+## 10. Operational behavior
+
+### 10.1 Run as a Windows service
+- Registered via pywin32 (`opcda-diagnozer install`), auto-start, runs under a
+  **dedicated service account** (required for DCOM authentication, see §11).
+- SCM recovery: restart on failure (first/second failure), plus internal watchdog.
+
+### 10.2 Connection state machine
+```mermaid
+stateDiagram-v2
+    [*] --> Connecting
+    Connecting --> Connected : connect OK
+    Connecting --> ReconnectWait : COM error
+    Connected --> ReconnectWait : read error / hang
+    ReconnectWait --> Connecting : backoff elapsed
+```
+Reconnect backoff: exponential 1 s → 2 s → … capped at 60 s. Every disconnect/reconnect
+pair is an event; outage duration is computed and reported.
+
+### 10.3 COM apartment handling
+All OPC/COM work runs on one dedicated STA thread inside the service process
+(`pythoncom.CoInitialize()` on that thread only). Detectors run on the same thread (pure
+Python, cheap); event reporting happens off-thread via a queue so slow I/O never blocks
+polling.
+
+### 10.4 The DCOM hang problem (important)
+DCOM/RPC calls have **no client-side timeout**. If the network path breaks silently
+(e.g., firewall drops), a read call can block for tens of seconds to minutes without
+erroring. The tool handles this explicitly:
+
+1. A watchdog flags a **poll hang** (D7) as soon as a call exceeds `timeout_ms`.
+2. If the hang exceeds `hard_hang_s` (default 120 s), the service abandons the stuck
+   thread and creates a fresh connection on a new thread (stale `RPC` objects are
+   released by COM).
+3. Hang events are distinguished from clean disconnects — this is often the smoking gun
+   for silent network drops.
+
+### 10.5 Safety constraints
+- Read-only by construction: no write API is called; the code has no write path.
+- Single OPC group, static item list, modest poll rate (default 1 s) to keep added
+  server load negligible.
+- Configuration errors fail fast at startup with actionable messages
+  (`opcda-diagnozer check-config` validates without connecting… and `--probe` connects
+  once and reports).
+
+## 11. Environment & constraints
+
+| Constraint | Consequence |
+|------------|-------------|
+| OPC DA is Windows/COM only | Service runs on Windows only (Server 2012 R2 … 2022, Win 10/11) |
+| OPC DA wrappers are often **32-bit** DLLs (OPC DA Automation Wrapper `OPCAutomation.dll`, Graybox `GBDaCtl.dll`) | Python interpreter bitness must match the wrapper DLL — likely **32-bit Python** for v1; decision in Phase 0 |
+| DCOM security | The service account needs Launch+Activation and Access permissions on the OPC server (via `dcomcnfg`); account must authenticate across the network (not `LocalSystem`) |
+| Post-2022 Windows DCOM hardening (KB5004442 family) | DCOM activation now requires explicit authentication; legacy OPC setups may fail activation — errors must be reported clearly |
+| RPC over TCP | Firewall: TCP 135 + RPC dynamic port range on the server host; misconfig shows up as `RPC_S_SERVER_UNAVAILABLE` |
+| Clock skew | Device timestamps may be misleading; every event also carries local receive time; recommend NTP on both hosts |
+
+## 12. Diagnostic interpretation matrix (why this tool exists)
+
+| Observation from the Diagnozer | Most likely problem location |
+|--------------------------------|------------------------------|
+| Clean reads, low RTT, all GOOD — while SCADA still complains | SCADA host itself: per-machine DCOM permissions, CPU/memory saturation, its own group/subscription load |
+| Diagnozer also sees disconnects / RPC errors | Shared path: network, firewall, DCOM layer, or OPC server host |
+| Connection stable, but `BAD_COMM_FAILURE` on many device tags | Link between the OPC server and the control devices (downstream of OPC) — **not** a DCOM/network problem |
+| Slow RTT spikes at consistent times of day | Server load pattern; correlate with heartbeat summaries |
+| Hangs without errors returned (D7) | Silent network drop (firewall/MTU/NIC) — classic DCOM pitfall |
+| One instance (SCADA zone) slow, other (server zone) fast | Network segment between the zones |
+
+## 13. Acceptance criteria (v1.0)
+
+- **AC1** Runs as a service, survives reboot, auto-starts, recovers from a killed process.
+- **AC2** Detects OPC server process stop within one poll interval (or one hang window for
+  silent drops).
+- **AC3** Detects injected latency above thresholds within 1–2 polls; event contains
+  measured RTT.
+- **AC4** Quality transitions produce events with raw code and decoded name.
+- **AC5** Staleness/freeze detection within tolerance of `max_age_s`.
+- **AC6** No OPC writes are ever issued (code review + no write path exists).
+- **AC7** Events land in JSONL and Event Log; heartbeats appear even when all is healthy.
+- **AC8** Invalid config fails fast with actionable messages.
+- **AC9** Footprint ≤ ~50 MB RAM, ~0% CPU while idle-polling on a healthy link.
+- **AC10** 24 h soak against a simulation server: no leaks, correct reconnect accounting.
+
+## 14. Future extensions (post-v1)
+
+- Async subscription monitoring (`IOPCAsyncIO2` DataChange) to compare push vs. poll
+  latency — validates the SCADA's actual callback path.
+- Per-item latency bisection (read items individually to isolate a slow tag).
+- SQLite time-series of samples for post-incident graphing.
+- Prometheus exporter / Grafana dashboard.
+- Email / Teams / webhook alerting with deduplication.
+- Multi-server monitoring in one service.