opcda-diagnozer/docs/SPECIFICATION.md

OPC DA Diagnozer — Project Specification

Version: 0.1 (draft) Status: Planning Target platform: Windows only (OPC DA is a COM/DCOM technology)

---

1. Problem statement

We experience intermittent disconnections and/or high communication latency between a SCADA system and a control system that communicate over OPC DA (Data Access, the COM/DCOM-based predecessor of OPC UA). Today we have no objective evidence to answer the key triage question:

Is the problem in the SCADA host, the network/DCOM layer, the OPC DA server, or the link between the OPC server and the control devices?

OPC DA failure modes are notoriously hard to diagnose because failures surface as generic HRESULT errors, hung calls with no client-side timeout, or silently degraded item quality — long before the SCADA operator sees anything.

2. Approach

Deploy an independent, read-only OPC DA reference client that continuously exercises the same path the SCADA uses:

• Connect to the OPC DA server (local or remote via DCOM) from a Windows machine.
• Poll a small, pre-defined list of tags at a fixed interval.
• Measure read round-trip time (RTT), track item quality (GOOD / UNCERTAIN / BAD and subcodes), device timestamps, and connection state.
• Emit structured, timestamped diagnostic events whenever anything deviates from normal — disconnects, slow reads, hung calls, quality degradation, frozen/stale values.

Because the client is independent of the SCADA, its observations provide a reference baseline against which the SCADA's complaints can be triangulated (see §12).

3. Deployment architecture

flowchart LR
    subgraph SCADAzone["SCADA zone (or dedicated diag host)"]
        DIAG["OPC DA Diagnozer<br/>(Windows service)"]
    end
    subgraph CTL["Control system / OPC DA server host"]
        OPC["OPC DA server<br/>(COM/DCOM out-of-proc)"]
        DEV["Control devices / PLC link"]
    end
    DIAG -- "DCOM reads (RPC 135 + dynamic ports)" --> OPC
    OPC -.-> DEV
    DIAG -- "Events (JSONL / Event Log / HTTP)" --> OPS["Engineers & monitoring"]

Placement note: run the tool as close to the SCADA machine as possible (same host or same network segment) so its measured latencies are comparable to what the SCADA experiences. Optionally run a second instance next to the OPC server host for differential diagnosis.

4. Goals

1. Run unattended as a Windows service (auto-start, SCM crash recovery).
2. Connect to a configured OPC DA server (local or remote DCOM) and read a configured tag list.
3. Detect and report, with precise timestamps:
   • disconnections and reconnects (including DCOM error codes),
   • hung / timed-out RPC calls,
   • read latencies exceeding thresholds (single spikes, streaks, and trends),
   • tags whose quality is BAD or UNCERTAIN (with decoded subcode),
   • persistent-bad and flapping (oscillating) quality per tag,
   • stale device timestamps / frozen values while quality reports GOOD,
   • mass quality degradation (many tags BAD at once → control-link problem).
4. Produce durable, machine-readable evidence (structured logs + Windows Event Log + optional HTTP webhook) suitable for incident review and correlation.
5. Be strictly read-only and lightweight — must not perturb the system it measures.

5. Non-goals (v1)

• No writes to the OPC server (no Write code path at all).
• No OPC UA, OPC HDA, or Alarms & Events support.
• No data historian / storage of every sample (only events + periodic summaries).
• No UI/dashboard in v1 (JSONL logs + webhook are enough for Loki/Grafana ingestion).
• No automatic remediation (diagnose only).

6. Detection rules

ID	Condition	Default rule	Severity
D1	Connection lost	Any read/connect COM failure	CRITICAL (event incl. decoded HRESULT)
D2	Reconnect succeeded	After D1, reconnect works	INFO (event with outage duration)
D3	Reconnect flapping	> 3 disconnects in 10 min	CRITICAL
D4	Slow poll (spike)	Single read RTT > warn_ms (default 1000)	WARN
D5	Sustained slow polls	RTT > warn_ms for slow_poll_streak (5) consecutive polls	CRITICAL
D6	Latency trend	p95 RTT over last window_polls (60) > warn_ms	WARN
D7	Poll hang	Read call does not return within timeout_ms (5000) — see §10.4	CRITICAL (immediate), see recovery behavior
D8	Tag quality → BAD	Any transition into BAD, decoded subcode	WARN (per tag)
D9	Tag quality → UNCERTAIN	Any transition into UNCERTAIN	INFO
D10	Persistent bad tag	Single tag BAD longer than persistent_bad_s (30)	WARN (escalated from D8)
D11	Quality flapping	> 5 quality transitions for one tag within 10 min	WARN
D12	Stale timestamp	Device timestamp older than max_age_s (120) while quality GOOD	WARN
D13	Frozen value	expect_change: true tag with unchanged value+timestamp for max_age_s	WARN
D14	Mass quality degradation	≥ 50% of monitored tags BAD simultaneously while connection OK	CRITICAL (CONTROL_LINK_DEGRADED — points at OPC-server↔device link, not DCOM)
D15	Heartbeat / summary	Periodic stats snapshot every heartbeat_s (300)	INFO
D16	Device link down	A configured device-indicator tag reads false, or any quality other than GOOD, or a read error	CRITICAL per device (DEVICE_LINK_DOWN); D1/D2 do not imply device-link state

All thresholds are configurable; severities are INFO / WARN / CRITICAL.

6.1 Device connection-indicator tags (D16)

The config names one or more tags as device-link indicators: values the control system maintains that say whether the OPC server can reach the device. The normal (connected) reading is configurable per indicator — conventions differ per control system, so it is never hardcoded:

devices:
  - { name: "Conveyor 3", tag: "...Connected" }              # any truthy: true/1
  - { name: "Pump A",     tag: "...PumpAFault", ok_values: [false, 0] }   # inverted fault flag
  - { name: "Boiler 2",   tag: "...State",   ok_values: ["RUN"] }     # enum/string state

Default ok_values: [true] (boolean connected-flags are the common case). Value comparison is by string equality on the read value rendered as text ("True", "1", "RUN"), so type quirks across systems cannot cause false positives; quality must also be GOOD. Semantics (all DOWN cases are logged as DEVICE_LINK_DOWN):

Indicator reading	Interpretation
GOOD + value ∈ ok_values	Device connected — normal
GOOD + value ∉ ok_values	Device link down / fault — CRITICAL (works for inverted fault-flags: a fault 1 is not in [false, 0])
Any quality other than GOOD (BAD/UNCERTAIN)	Something is wrong with the device link — CRITICAL
Read error (tag unreadable)	Something is wrong with the device link — CRITICAL

Device-link state is per indicator tag: the config maps each indicator to a human-readable device name for the log lines (see config.example.yaml). Recovery (DEVICE_LINK_UP) is logged when the same indicator reads GOOD with an ok value again. A value that is neither in ok_values nor a recognized fault shape still logs DOWN — the normal state is explicit, and everything else is "not normal" by definition.

Section optionality: tags: and devices: are each optional in the config — a devices-only setup monitors just the links, a tags-only setup runs classic quality/latency surveillance. Both empty (or absent) is a validation error: the tool would read nothing and every heartbeat would report tags_total: 0.

7. OPC DA quality codes (reference)

OPC DA quality is a 16-bit field. Bits 7–6 of the low byte give the class: 0xC0 GOOD, 0x40 UNCERTAIN, 0x00 BAD. Bits 5–0 are the substatus. Values the tool decodes and names:

Code	Meaning
0xC0	GOOD
0xD8	GOOD / local override
0x44	UNCERTAIN / last usable value
0x40	UNCERTAIN (unspecified)
0x00	BAD (unspecified)
0x04	BAD / configuration error
0x08	BAD / not connected
0x0C	BAD / device failure
0x10	BAD / sensor failure
0x14	BAD / last known value
0x18	BAD / communication failure
0x1C	BAD / out of service

Remaining UNCERTAIN subcodes (DA 2.05a) are decoded by the same table in quality.py. Every quality event carries the raw hex code and the decoded name.

Clear-text status rule: wherever a status or quality is reported — JSONL fields, the text log, Event Log, console — it is written in clear text with the raw code in parentheses: BAD / communication failure (0x18), GOOD (0xC0). Nobody should have to decode hex codes by hand, but the raw code stays for unambiguous correlation with vendor documentation and the HRESULT table.

8. Event model

Every event is a structured record:

{
  "time_utc": "2026-10-07T12:34:56.789Z",
  "kind": "QUALITY_BAD",
  "severity": "WARN",
  "detail": {
    "tag": "Channel1.Device1.Counter",
    "quality_raw": "0x18",
    "quality": "BAD_COMM_FAILURE",
    "previous_quality": "GOOD",
    "value": 1234.0,
    "device_time_utc": "2026-10-07T12:34:55.100Z",
    "rtt_ms": 42.1
  },
  "context": {
    "uptime_s": 3600.5,
    "reconnects_total": 2,
    "rtt_p95_ms": 55.2,
    "tags_bad": 3,
    "tags_total": 25
  }
}

time_utc is always the local client receive time (wall clock), so events remain comparable even if the remote host's clock is wrong. Device timestamps are reported alongside and clearly labeled.

9. Outputs (reporters)

Reporter	Purpose	Notes
JSONL rotating file	Primary evidence trail	One JSON event per line; rotation by size; default dir %ProgramData%\OPCDA-Diagnozer\logs
Plain-text anomaly log	Human-readable evidence	One line per detected anomaly (not heartbeats): timestamp, kind, tag/device, message with clear-text status NAME (0xNN); same rotation scheme as JSONL. See §9.1 for the line format
Windows Event Log	Ops visibility	WARN+ only; source OPCDA-Diagnozer
Console	Foreground/debug runs	Human-readable
HTTP webhook (optional)	Alert integration	POST one JSON event per request, optional bearer token

Reporters are pluggable (Reporter.emit(event)); a heartbeat event every 5 minutes proves liveness and enables baseline graphs even when everything is healthy. The JSONL file stays the machine-readable source of truth; the text log is a human-first convenience view of the same events — every text line corresponds to a JSONL record.

9.1 Plain-text anomaly log format

File anomalies-YYYYMMDD.log (daily rotation, size-capped) in the same log directory. One line per detected anomaly — anomalies only; heartbeats and reconnect-OK events do not appear here. Clear text first, codes in parentheses:

2026-10-07 14:23:01.123 UTC  WARN      QUALITY_BAD       tag 'Channel1.Device1.Counter' changed GOOD (0xC0) -> BAD / communication failure (0x18)
2026-10-07 14:24:31.500 UTC  CRITICAL  DEVICE_LINK_DOWN  device 'Conveyor 3': link indicator 'Channel1.Device3.Connected' read false (GOOD)
2026-10-07 14:26:02.780 UTC  CRITICAL  DEVICE_LINK_DOWN  device 'Conveyor 3': link indicator 'Channel1.Device3.Connected' quality BAD / not connected (0x08)
2026-10-07 14:27:10.001 UTC  CRITICAL  POLL_HANG        read call hung > 5000 ms; COM thread abandoned and rebuilt
2026-10-07 14:30:00.000 UTC  CRITICAL  CONTROL_LINK_DEGRADED  12 of 14 monitored tags BAD while connection OK

Fields (pipe-free, fixed order): timestamp SEVERITY KIND message. Timestamps are the local receive time in UTC (§8). The message carries the clear-text status with the code between parentheses per §7. A matching recovery (link back up, quality back to GOOD) is also logged as an anomaly line, so the log reads as a continuous incident narrative.

10. Operational behavior

10.1 Run as a Windows service

• Registered via pywin32 (opcda-diagnozer install), auto-start, runs under a dedicated service account (required for DCOM authentication, see §11).
• SCM recovery: restart on failure (first/second failure), plus internal watchdog.

10.2 Connection state machine

stateDiagram-v2
    [*] --> Connecting
    Connecting --> Connected : connect OK
    Connecting --> ReconnectWait : COM error
    Connected --> ReconnectWait : read error / hang
    ReconnectWait --> Connecting : backoff elapsed

Reconnect backoff: exponential 1 s → 2 s → … capped at 60 s. Every disconnect/reconnect pair is an event; outage duration is computed and reported.

10.3 COM apartment handling

All OPC/COM work runs on one dedicated STA thread inside the service process (pythoncom.CoInitialize() on that thread only). Detectors run on the same thread (pure Python, cheap); event reporting happens off-thread via a queue so slow I/O never blocks polling.

10.4 The DCOM hang problem (important)

DCOM/RPC calls have no client-side timeout. If the network path breaks silently (e.g., firewall drops), a read call can block for tens of seconds to minutes without erroring. The tool handles this explicitly:

1. A watchdog flags a poll hang (D7) as soon as a call exceeds timeout_ms.
2. If the hang exceeds hard_hang_s (default 120 s), the service abandons the stuck thread and creates a fresh connection on a new thread (stale RPC objects are released by COM).
3. Hang events are distinguished from clean disconnects — this is often the smoking gun for silent network drops.

10.5 Safety constraints

• Read-only by construction: no write API is called; the code has no write path.
• Single OPC group, static item list, modest poll rate (default 1 s) to keep added server load negligible.
• Configuration errors fail fast at startup with actionable messages (opcda-diagnozer check-config validates without connecting… and --probe connects once and reports).

11. Environment & constraints

Constraint	Consequence
OPC DA is Windows/COM only	Service runs on Windows only (Server 2012 R2 … 2022, Win 10/11)
OPC DA wrappers are often 32-bit DLLs (OPC DA Automation Wrapper OPCAutomation.dll, Graybox GBDaCtl.dll)	Python interpreter bitness must match the wrapper DLL — likely 32-bit Python for v1; decision in Phase 0
DCOM security	The service account needs Launch+Activation and Access permissions on the OPC server (via dcomcnfg); account must authenticate across the network (not LocalSystem)
Post-2022 Windows DCOM hardening (KB5004442 family)	DCOM activation now requires explicit authentication; legacy OPC setups may fail activation — errors must be reported clearly
RPC over TCP	Firewall: TCP 135 + RPC dynamic port range on the server host; misconfig shows up as RPC_S_SERVER_UNAVAILABLE
Clock skew	Device timestamps may be misleading; every event also carries local receive time; recommend NTP on both hosts

12. Diagnostic interpretation matrix (why this tool exists)

Observation from the Diagnozer	Most likely problem location
Clean reads, low RTT, all GOOD — while SCADA still complains	SCADA host itself: per-machine DCOM permissions, CPU/memory saturation, its own group/subscription load
Diagnozer also sees disconnects / RPC errors	Shared path: network, firewall, DCOM layer, or OPC server host
Connection stable, but BAD_COMM_FAILURE on many device tags	Link between the OPC server and the control devices (downstream of OPC) — not a DCOM/network problem
Slow RTT spikes at consistent times of day	Server load pattern; correlate with heartbeat summaries
Hangs without errors returned (D7)	Silent network drop (firewall/MTU/NIC) — classic DCOM pitfall
One instance (SCADA zone) slow, other (server zone) fast	Network segment between the zones

13. Acceptance criteria (v1.0)

• AC1 Runs as a service, survives reboot, auto-starts, recovers from a killed process.
• AC2 Detects OPC server process stop within one poll interval (or one hang window for silent drops).
• AC3 Detects injected latency above thresholds within 1–2 polls; event contains measured RTT.
• AC4 Quality transitions produce events with raw code and decoded name.
• AC5 Staleness/freeze detection within tolerance of max_age_s.
• AC6 No OPC writes are ever issued (code review + no write path exists).
• AC7 Events land in JSONL and Event Log; heartbeats appear even when all is healthy.
• AC8 Invalid config fails fast with actionable messages.
• AC9 Footprint ≤ ~50 MB RAM, ~0% CPU while idle-polling on a healthy link.
• AC10 24 h soak against a simulation server: no leaks, correct reconnect accounting.

14. Future extensions (post-v1)

• Async subscription monitoring (IOPCAsyncIO2 DataChange) to compare push vs. poll latency — validates the SCADA's actual callback path.
• Per-item latency bisection (read items individually to isolate a slow tag).
• SQLite time-series of samples for post-incident graphing.
• Prometheus exporter / Grafana dashboard.
• Email / Teams / webhook alerting with deduplication.
• Multi-server monitoring in one service.