opcda-diagnozer/docs/SPECIFICATION.md

# OPC DA Diagnozer — Project Specification

**Version:** 0.1 (draft)
**Status:** Planning
**Target platform:** Windows only (OPC DA is a COM/DCOM technology)

---

## 1. Problem statement

We experience intermittent **disconnections** and/or **high communication latency** between a
SCADA system and a control system that communicate over **OPC DA** (Data Access, the
COM/DCOM-based predecessor of OPC UA). Today we have no objective evidence to answer the
key triage question:

> *Is the problem in the SCADA host, the network/DCOM layer, the OPC DA server, or the
> link between the OPC server and the control devices?*

OPC DA failure modes are notoriously hard to diagnose because failures surface as generic
`HRESULT` errors, hung calls with **no client-side timeout**, or silently degraded item
quality — long before the SCADA operator sees anything.

## 2. Approach

Deploy an **independent, read-only OPC DA reference client** that continuously exercises
the same path the SCADA uses:

- Connect to the OPC DA server (local or remote via DCOM) from a Windows machine.
- Poll a small, pre-defined list of tags at a fixed interval.
- Measure **read round-trip time (RTT)**, track **item quality** (GOOD / UNCERTAIN / BAD
  and subcodes), **device timestamps**, and **connection state**.
- Emit structured, timestamped **diagnostic events** whenever anything deviates from
  normal — disconnects, slow reads, hung calls, quality degradation, frozen/stale values.

Because the client is independent of the SCADA, its observations provide a **reference
baseline** against which the SCADA's complaints can be triangulated (see §12).

## 3. Deployment architecture

```mermaid
flowchart LR
    subgraph SCADAzone["SCADA zone (or dedicated diag host)"]
        DIAG["OPC DA Diagnozer<br/>(Windows service)"]
    end
    subgraph CTL["Control system / OPC DA server host"]
        OPC["OPC DA server<br/>(COM/DCOM out-of-proc)"]
        DEV["Control devices / PLC link"]
    end
    DIAG -- "DCOM reads (RPC 135 + dynamic ports)" --> OPC
    OPC -.-> DEV
    DIAG -- "Events (JSONL / Event Log / HTTP)" --> OPS["Engineers & monitoring"]
```

Placement note: run the tool **as close to the SCADA machine as possible** (same host or
same network segment) so its measured latencies are comparable to what the SCADA
experiences. Optionally run a second instance next to the OPC server host for differential
diagnosis.

## 4. Goals

1. Run unattended as a **Windows service** (auto-start, SCM crash recovery).
2. Connect to a configured OPC DA server (local or remote DCOM) and read a configured tag list.
3. Detect and report, with precise timestamps:
   - disconnections and reconnects (including DCOM error codes),
   - hung / timed-out RPC calls,
   - read latencies exceeding thresholds (single spikes, streaks, and trends),
   - tags whose quality is BAD or UNCERTAIN (with decoded subcode),
   - persistent-bad and flapping (oscillating) quality per tag,
   - stale device timestamps / frozen values while quality reports GOOD,
   - mass quality degradation (many tags BAD at once → control-link problem).
4. Produce durable, machine-readable evidence (structured logs + Windows Event Log +
   optional HTTP webhook) suitable for incident review and correlation.
5. Be **strictly read-only** and lightweight — must not perturb the system it measures.

## 5. Non-goals (v1)

- No writes to the OPC server (no `Write` code path at all).
- No OPC UA, OPC HDA, or Alarms & Events support.
- No data historian / storage of every sample (only events + periodic summaries).
- No UI/dashboard in v1 (JSONL logs + webhook are enough for Loki/Grafana ingestion).
- No automatic remediation (diagnose only).

## 6. Detection rules

| ID | Condition | Default rule | Severity |
|----|-----------|--------------|----------|
| D1 | Connection lost | Any read/connect COM failure | CRITICAL (event incl. decoded HRESULT) |
| D2 | Reconnect succeeded | After D1, reconnect works | INFO (event with outage duration) |
| D3 | Reconnect flapping | > 3 disconnects in 10 min | CRITICAL |
| D4 | Slow poll (spike) | Single read RTT > `warn_ms` (default 1000) | WARN |
| D5 | Sustained slow polls | RTT > `warn_ms` for `slow_poll_streak` (5) consecutive polls | CRITICAL |
| D6 | Latency trend | p95 RTT over last `window_polls` (60) > `warn_ms` | WARN |
| D7 | Poll hang | Read call does not return within `timeout_ms` (5000) — see §10.4 | CRITICAL (immediate), see recovery behavior |
| D8 | Tag quality → BAD | Any transition into BAD, decoded subcode | WARN (per tag) |
| D9 | Tag quality → UNCERTAIN | Any transition into UNCERTAIN | INFO |
| D10 | Persistent bad tag | Single tag BAD longer than `persistent_bad_s` (30) | WARN (escalated from D8) |
| D11 | Quality flapping | > 5 quality transitions for one tag within 10 min | WARN |
| D12 | Stale timestamp | Device timestamp older than `max_age_s` (120) while quality GOOD | WARN |
| D13 | Frozen value | `expect_change: true` tag with unchanged value+timestamp for `max_age_s` | WARN |
| D14 | Mass quality degradation | ≥ 50% of monitored tags BAD simultaneously while connection OK | CRITICAL (`CONTROL_LINK_DEGRADED` — points at OPC-server↔device link, not DCOM) |
| D15 | Heartbeat / summary | Periodic stats snapshot every `heartbeat_s` (300) | INFO |
| D16 | Device link down | A configured device-indicator tag reads `false`, or any quality other than GOOD, or a read error | CRITICAL per device (`DEVICE_LINK_DOWN`); D1/D2 do **not** imply device-link state |

All thresholds are configurable; severities are INFO / WARN / CRITICAL.

### 6.1 Device connection-indicator tags (D16)

The config names one or more tags as **device-link indicators**: values the
control system maintains that say whether the OPC server can reach the device.
The **normal (connected) reading is configurable per indicator** — conventions
differ per control system, so it is never hardcoded:

```yaml
devices:
  - { name: "Conveyor 3", tag: "...Connected" }              # any truthy: true/1
  - { name: "Pump A",     tag: "...PumpAFault", ok_values: [false, 0] }   # inverted fault flag
  - { name: "Boiler 2",   tag: "...State",   ok_values: ["RUN"] }     # enum/string state
```

Default `ok_values: [true]` (boolean connected-flags are the common case).
Value comparison is by **string equality** on the read value rendered as text
(`"True"`, `"1"`, `"RUN"`), so type quirks across systems cannot cause false
positives; quality must also be GOOD. Semantics (all DOWN cases are logged as
`DEVICE_LINK_DOWN`):

| Indicator reading | Interpretation |
|-------------------|------------------|
| GOOD + value ∈ `ok_values` | Device connected — normal |
| GOOD + value ∉ `ok_values` | **Device link down / fault** — CRITICAL (works for inverted fault-flags: a fault `1` is not in `[false, 0]`) |
| Any quality other than GOOD (BAD/UNCERTAIN) | **Something is wrong with the device link** — CRITICAL |
| Read error (tag unreadable) | **Something is wrong with the device link** — CRITICAL |

Device-link state is per indicator tag: the config maps each indicator to a
human-readable device name for the log lines (see `config.example.yaml`).
Recovery (`DEVICE_LINK_UP`) is logged when the same indicator reads GOOD with
an ok value again. A value that is neither in `ok_values` nor a recognized
fault shape still logs DOWN — the normal state is explicit, and everything
else is "not normal" by definition.

**Section optionality:** `tags:` and `devices:` are each optional in the
config — a devices-only setup monitors just the links, a tags-only setup
runs classic quality/latency surveillance. Both empty (or absent) is a
validation error: the tool would read nothing and every heartbeat would
report `tags_total: 0`.

## 7. OPC DA quality codes (reference)

OPC DA quality is a 16-bit field. Bits 7–6 of the low byte give the class:
`0xC0` GOOD, `0x40` UNCERTAIN, `0x00` BAD. Bits 5–0 are the substatus. Values the tool
decodes and names:

| Code | Meaning |
|------|---------|
| `0xC0` | GOOD |
| `0xD8` | GOOD / local override |
| `0x44` | UNCERTAIN / last usable value |
| `0x40` | UNCERTAIN (unspecified) |
| `0x00` | BAD (unspecified) |
| `0x04` | BAD / configuration error |
| `0x08` | BAD / not connected |
| `0x0C` | BAD / device failure |
| `0x10` | BAD / sensor failure |
| `0x14` | BAD / last known value |
| `0x18` | BAD / communication failure |
| `0x1C` | BAD / out of service |

Remaining UNCERTAIN subcodes (DA 2.05a) are decoded by the same table in `quality.py`.
Every quality event carries the raw hex code **and** the decoded name.

**Clear-text status rule:** wherever a status or quality is reported — JSONL
fields, the text log, Event Log, console — it is written in clear text with
the raw code in parentheses: `BAD / communication failure (0x18)`,
`GOOD (0xC0)`. Nobody should have to decode hex codes by hand, but the raw
code stays for unambiguous correlation with vendor documentation and the
HRESULT table.

## 8. Event model

Every event is a structured record:

```json
{
  "time_utc": "2026-10-07T12:34:56.789Z",
  "kind": "QUALITY_BAD",
  "severity": "WARN",
  "detail": {
    "tag": "Channel1.Device1.Counter",
    "quality_raw": "0x18",
    "quality": "BAD_COMM_FAILURE",
    "previous_quality": "GOOD",
    "value": 1234.0,
    "device_time_utc": "2026-10-07T12:34:55.100Z",
    "rtt_ms": 42.1
  },
  "context": {
    "uptime_s": 3600.5,
    "reconnects_total": 2,
    "rtt_p95_ms": 55.2,
    "tags_bad": 3,
    "tags_total": 25
  }
}
```

`time_utc` is always the **local client receive time** (wall clock), so events remain
comparable even if the remote host's clock is wrong. Device timestamps are reported
alongside and clearly labeled.

## 9. Outputs (reporters)

| Reporter | Purpose | Notes |
|----------|---------|-------|
| JSONL rotating file | Primary evidence trail | One JSON event per line; rotation by size; default dir `%ProgramData%\OPCDA-Diagnozer\logs` |
| **Plain-text anomaly log** | Human-readable evidence | One line per **detected anomaly** (not heartbeats): timestamp, kind, tag/device, message with clear-text status `NAME (0xNN)`; same rotation scheme as JSONL. See §9.1 for the line format |
| Windows Event Log | Ops visibility | WARN+ only; source `OPCDA-Diagnozer` |
| Console | Foreground/debug runs | Human-readable |
| HTTP webhook (optional) | Alert integration | POST one JSON event per request, optional bearer token |

Reporters are pluggable (`Reporter.emit(event)`); a heartbeat event every 5 minutes
proves liveness and enables baseline graphs even when everything is healthy.
The JSONL file stays the machine-readable source of truth; the text log is a
human-first convenience view of the same events — every text line corresponds
to a JSONL record.

### 9.1 Plain-text anomaly log format

File `anomalies-YYYYMMDD.log` (daily rotation, size-capped) in the same log
directory. One line per detected anomaly — anomalies only; heartbeats and
reconnect-OK events do not appear here. Clear text first, codes in parentheses:

```
2026-10-07 14:23:01.123 UTC  WARN      QUALITY_BAD       tag 'Channel1.Device1.Counter' changed GOOD (0xC0) -> BAD / communication failure (0x18)
2026-10-07 14:24:31.500 UTC  CRITICAL  DEVICE_LINK_DOWN  device 'Conveyor 3': link indicator 'Channel1.Device3.Connected' read false (GOOD)
2026-10-07 14:26:02.780 UTC  CRITICAL  DEVICE_LINK_DOWN  device 'Conveyor 3': link indicator 'Channel1.Device3.Connected' quality BAD / not connected (0x08)
2026-10-07 14:27:10.001 UTC  CRITICAL  POLL_HANG        read call hung > 5000 ms; COM thread abandoned and rebuilt
2026-10-07 14:30:00.000 UTC  CRITICAL  CONTROL_LINK_DEGRADED  12 of 14 monitored tags BAD while connection OK
```

Fields (pipe-free, fixed order): `timestamp  SEVERITY  KIND  message`.
Timestamps are the local receive time in UTC (§8). The message carries the
clear-text status with the code between parentheses per §7. A matching
recovery (link back up, quality back to GOOD) is also logged as an anomaly
line, so the log reads as a continuous incident narrative.

## 10. Operational behavior

### 10.1 Run as a Windows service
- Registered via pywin32 (`opcda-diagnozer install`), auto-start, runs under a
  **dedicated service account** (required for DCOM authentication, see §11).
- SCM recovery: restart on failure (first/second failure), plus internal watchdog.

### 10.2 Connection state machine
```mermaid
stateDiagram-v2
    [*] --> Connecting
    Connecting --> Connected : connect OK
    Connecting --> ReconnectWait : COM error
    Connected --> ReconnectWait : read error / hang
    ReconnectWait --> Connecting : backoff elapsed
```
Reconnect backoff: exponential 1 s → 2 s → … capped at 60 s. Every disconnect/reconnect
pair is an event; outage duration is computed and reported.

### 10.3 COM apartment handling
All OPC/COM work runs on one dedicated STA thread inside the service process
(`pythoncom.CoInitialize()` on that thread only). Detectors run on the same thread (pure
Python, cheap); event reporting happens off-thread via a queue so slow I/O never blocks
polling.

### 10.4 The DCOM hang problem (important)
DCOM/RPC calls have **no client-side timeout**. If the network path breaks silently
(e.g., firewall drops), a read call can block for tens of seconds to minutes without
erroring. The tool handles this explicitly:

1. A watchdog flags a **poll hang** (D7) as soon as a call exceeds `timeout_ms`.
2. If the hang exceeds `hard_hang_s` (default 120 s), the service abandons the stuck
   thread and creates a fresh connection on a new thread (stale `RPC` objects are
   released by COM).
3. Hang events are distinguished from clean disconnects — this is often the smoking gun
   for silent network drops.

### 10.5 Safety constraints
- Read-only by construction: no write API is called; the code has no write path.
- Single OPC group, static item list, modest poll rate (default 1 s) to keep added
  server load negligible.
- Configuration errors fail fast at startup with actionable messages
  (`opcda-diagnozer check-config` validates without connecting… and `--probe` connects
  once and reports).

## 11. Environment & constraints

| Constraint | Consequence |
|------------|-------------|
| OPC DA is Windows/COM only | Service runs on Windows only (Server 2012 R2 … 2022, Win 10/11) |
| OPC DA wrappers are often **32-bit** DLLs (OPC DA Automation Wrapper `OPCAutomation.dll`, Graybox `GBDaCtl.dll`) | Python interpreter bitness must match the wrapper DLL — likely **32-bit Python** for v1; decision in Phase 0 |
| DCOM security | The service account needs Launch+Activation and Access permissions on the OPC server (via `dcomcnfg`); account must authenticate across the network (not `LocalSystem`) |
| Post-2022 Windows DCOM hardening (KB5004442 family) | DCOM activation now requires explicit authentication; legacy OPC setups may fail activation — errors must be reported clearly |
| RPC over TCP | Firewall: TCP 135 + RPC dynamic port range on the server host; misconfig shows up as `RPC_S_SERVER_UNAVAILABLE` |
| Clock skew | Device timestamps may be misleading; every event also carries local receive time; recommend NTP on both hosts |

## 12. Diagnostic interpretation matrix (why this tool exists)

| Observation from the Diagnozer | Most likely problem location |
|--------------------------------|------------------------------|
| Clean reads, low RTT, all GOOD — while SCADA still complains | SCADA host itself: per-machine DCOM permissions, CPU/memory saturation, its own group/subscription load |
| Diagnozer also sees disconnects / RPC errors | Shared path: network, firewall, DCOM layer, or OPC server host |
| Connection stable, but `BAD_COMM_FAILURE` on many device tags | Link between the OPC server and the control devices (downstream of OPC) — **not** a DCOM/network problem |
| Slow RTT spikes at consistent times of day | Server load pattern; correlate with heartbeat summaries |
| Hangs without errors returned (D7) | Silent network drop (firewall/MTU/NIC) — classic DCOM pitfall |
| One instance (SCADA zone) slow, other (server zone) fast | Network segment between the zones |

## 13. Acceptance criteria (v1.0)

- **AC1** Runs as a service, survives reboot, auto-starts, recovers from a killed process.
- **AC2** Detects OPC server process stop within one poll interval (or one hang window for
  silent drops).
- **AC3** Detects injected latency above thresholds within 1–2 polls; event contains
  measured RTT.
- **AC4** Quality transitions produce events with raw code and decoded name.
- **AC5** Staleness/freeze detection within tolerance of `max_age_s`.
- **AC6** No OPC writes are ever issued (code review + no write path exists).
- **AC7** Events land in JSONL and Event Log; heartbeats appear even when all is healthy.
- **AC8** Invalid config fails fast with actionable messages.
- **AC9** Footprint ≤ ~50 MB RAM, ~0% CPU while idle-polling on a healthy link.
- **AC10** 24 h soak against a simulation server: no leaks, correct reconnect accounting.

## 14. Future extensions (post-v1)

- Async subscription monitoring (`IOPCAsyncIO2` DataChange) to compare push vs. poll
  latency — validates the SCADA's actual callback path.
- Per-item latency bisection (read items individually to isolate a slow tag).
- SQLite time-series of samples for post-incident graphing.
- Prometheus exporter / Grafana dashboard.
- Email / Teams / webhook alerting with deduplication.
- Multi-server monitoring in one service.