263 lines
13 KiB
Markdown
263 lines
13 KiB
Markdown
# OPC DA Diagnozer — Project Specification
|
||
|
||
**Version:** 0.1 (draft)
|
||
**Status:** Planning
|
||
**Target platform:** Windows only (OPC DA is a COM/DCOM technology)
|
||
|
||
---
|
||
|
||
## 1. Problem statement
|
||
|
||
We experience intermittent **disconnections** and/or **high communication latency** between a
|
||
SCADA system and a control system that communicate over **OPC DA** (Data Access, the
|
||
COM/DCOM-based predecessor of OPC UA). Today we have no objective evidence to answer the
|
||
key triage question:
|
||
|
||
> *Is the problem in the SCADA host, the network/DCOM layer, the OPC DA server, or the
|
||
> link between the OPC server and the control devices?*
|
||
|
||
OPC DA failure modes are notoriously hard to diagnose because failures surface as generic
|
||
`HRESULT` errors, hung calls with **no client-side timeout**, or silently degraded item
|
||
quality — long before the SCADA operator sees anything.
|
||
|
||
## 2. Approach
|
||
|
||
Deploy an **independent, read-only OPC DA reference client** that continuously exercises
|
||
the same path the SCADA uses:
|
||
|
||
- Connect to the OPC DA server (local or remote via DCOM) from a Windows machine.
|
||
- Poll a small, pre-defined list of tags at a fixed interval.
|
||
- Measure **read round-trip time (RTT)**, track **item quality** (GOOD / UNCERTAIN / BAD
|
||
and subcodes), **device timestamps**, and **connection state**.
|
||
- Emit structured, timestamped **diagnostic events** whenever anything deviates from
|
||
normal — disconnects, slow reads, hung calls, quality degradation, frozen/stale values.
|
||
|
||
Because the client is independent of the SCADA, its observations provide a **reference
|
||
baseline** against which the SCADA's complaints can be triangulated (see §12).
|
||
|
||
## 3. Deployment architecture
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
subgraph SCADAzone["SCADA zone (or dedicated diag host)"]
|
||
DIAG["OPC DA Diagnozer<br/>(Windows service)"]
|
||
end
|
||
subgraph CTL["Control system / OPC DA server host"]
|
||
OPC["OPC DA server<br/>(COM/DCOM out-of-proc)"]
|
||
DEV["Control devices / PLC link"]
|
||
end
|
||
DIAG -- "DCOM reads (RPC 135 + dynamic ports)" --> OPC
|
||
OPC -.-> DEV
|
||
DIAG -- "Events (JSONL / Event Log / HTTP)" --> OPS["Engineers & monitoring"]
|
||
```
|
||
|
||
Placement note: run the tool **as close to the SCADA machine as possible** (same host or
|
||
same network segment) so its measured latencies are comparable to what the SCADA
|
||
experiences. Optionally run a second instance next to the OPC server host for differential
|
||
diagnosis.
|
||
|
||
## 4. Goals
|
||
|
||
1. Run unattended as a **Windows service** (auto-start, SCM crash recovery).
|
||
2. Connect to a configured OPC DA server (local or remote DCOM) and read a configured tag list.
|
||
3. Detect and report, with precise timestamps:
|
||
- disconnections and reconnects (including DCOM error codes),
|
||
- hung / timed-out RPC calls,
|
||
- read latencies exceeding thresholds (single spikes, streaks, and trends),
|
||
- tags whose quality is BAD or UNCERTAIN (with decoded subcode),
|
||
- persistent-bad and flapping (oscillating) quality per tag,
|
||
- stale device timestamps / frozen values while quality reports GOOD,
|
||
- mass quality degradation (many tags BAD at once → control-link problem).
|
||
4. Produce durable, machine-readable evidence (structured logs + Windows Event Log +
|
||
optional HTTP webhook) suitable for incident review and correlation.
|
||
5. Be **strictly read-only** and lightweight — must not perturb the system it measures.
|
||
|
||
## 5. Non-goals (v1)
|
||
|
||
- No writes to the OPC server (no `Write` code path at all).
|
||
- No OPC UA, OPC HDA, or Alarms & Events support.
|
||
- No data historian / storage of every sample (only events + periodic summaries).
|
||
- No UI/dashboard in v1 (JSONL logs + webhook are enough for Loki/Grafana ingestion).
|
||
- No automatic remediation (diagnose only).
|
||
|
||
## 6. Detection rules
|
||
|
||
| ID | Condition | Default rule | Severity |
|
||
|----|-----------|--------------|----------|
|
||
| D1 | Connection lost | Any read/connect COM failure | CRITICAL (event incl. decoded HRESULT) |
|
||
| D2 | Reconnect succeeded | After D1, reconnect works | INFO (event with outage duration) |
|
||
| D3 | Reconnect flapping | > 3 disconnects in 10 min | CRITICAL |
|
||
| D4 | Slow poll (spike) | Single read RTT > `warn_ms` (default 1000) | WARN |
|
||
| D5 | Sustained slow polls | RTT > `warn_ms` for `slow_poll_streak` (5) consecutive polls | CRITICAL |
|
||
| D6 | Latency trend | p95 RTT over last `window_polls` (60) > `warn_ms` | WARN |
|
||
| D7 | Poll hang | Read call does not return within `timeout_ms` (5000) — see §10.4 | CRITICAL (immediate), see recovery behavior |
|
||
| D8 | Tag quality → BAD | Any transition into BAD, decoded subcode | WARN (per tag) |
|
||
| D9 | Tag quality → UNCERTAIN | Any transition into UNCERTAIN | INFO |
|
||
| D10 | Persistent bad tag | Single tag BAD longer than `persistent_bad_s` (30) | WARN (escalated from D8) |
|
||
| D11 | Quality flapping | > 5 quality transitions for one tag within 10 min | WARN |
|
||
| D12 | Stale timestamp | Device timestamp older than `max_age_s` (120) while quality GOOD | WARN |
|
||
| D13 | Frozen value | `expect_change: true` tag with unchanged value+timestamp for `max_age_s` | WARN |
|
||
| D14 | Mass quality degradation | ≥ 50% of monitored tags BAD simultaneously while connection OK | CRITICAL (`CONTROL_LINK_DEGRADED` — points at OPC-server↔device link, not DCOM) |
|
||
| D15 | Heartbeat / summary | Periodic stats snapshot every `heartbeat_s` (300) | INFO |
|
||
|
||
All thresholds are configurable; severities are INFO / WARN / CRITICAL.
|
||
|
||
## 7. OPC DA quality codes (reference)
|
||
|
||
OPC DA quality is a 16-bit field. Bits 7–6 of the low byte give the class:
|
||
`0xC0` GOOD, `0x40` UNCERTAIN, `0x00` BAD. Bits 5–0 are the substatus. Values the tool
|
||
decodes and names:
|
||
|
||
| Code | Meaning |
|
||
|------|---------|
|
||
| `0xC0` | GOOD |
|
||
| `0xD8` | GOOD / local override |
|
||
| `0x44` | UNCERTAIN / last usable value |
|
||
| `0x40` | UNCERTAIN (unspecified) |
|
||
| `0x00` | BAD (unspecified) |
|
||
| `0x04` | BAD / configuration error |
|
||
| `0x08` | BAD / not connected |
|
||
| `0x0C` | BAD / device failure |
|
||
| `0x10` | BAD / sensor failure |
|
||
| `0x14` | BAD / last known value |
|
||
| `0x18` | BAD / communication failure |
|
||
| `0x1C` | BAD / out of service |
|
||
|
||
Remaining UNCERTAIN subcodes (DA 2.05a) are decoded by the same table in `quality.py`.
|
||
Every quality event carries the raw hex code **and** the decoded name.
|
||
|
||
## 8. Event model
|
||
|
||
Every event is a structured record:
|
||
|
||
```json
|
||
{
|
||
"time_utc": "2026-10-07T12:34:56.789Z",
|
||
"kind": "QUALITY_BAD",
|
||
"severity": "WARN",
|
||
"detail": {
|
||
"tag": "Channel1.Device1.Counter",
|
||
"quality_raw": "0x18",
|
||
"quality": "BAD_COMM_FAILURE",
|
||
"previous_quality": "GOOD",
|
||
"value": 1234.0,
|
||
"device_time_utc": "2026-10-07T12:34:55.100Z",
|
||
"rtt_ms": 42.1
|
||
},
|
||
"context": {
|
||
"uptime_s": 3600.5,
|
||
"reconnects_total": 2,
|
||
"rtt_p95_ms": 55.2,
|
||
"tags_bad": 3,
|
||
"tags_total": 25
|
||
}
|
||
}
|
||
```
|
||
|
||
`time_utc` is always the **local client receive time** (wall clock), so events remain
|
||
comparable even if the remote host's clock is wrong. Device timestamps are reported
|
||
alongside and clearly labeled.
|
||
|
||
## 9. Outputs (reporters)
|
||
|
||
| Reporter | Purpose | Notes |
|
||
|----------|---------|-------|
|
||
| JSONL rotating file | Primary evidence trail | One JSON event per line; rotation by size; default dir `%ProgramData%\OPCDA-Diagnozer\logs` |
|
||
| Windows Event Log | Ops visibility | WARN+ only; source `OPCDA-Diagnozer` |
|
||
| Console | Foreground/debug runs | Human-readable |
|
||
| HTTP webhook (optional) | Alert integration | POST one JSON event per request, optional bearer token |
|
||
|
||
Reporters are pluggable (`Reporter.emit(event)`); a heartbeat event every 5 minutes
|
||
proves liveness and enables baseline graphs even when everything is healthy.
|
||
|
||
## 10. Operational behavior
|
||
|
||
### 10.1 Run as a Windows service
|
||
- Registered via pywin32 (`opcda-diagnozer install`), auto-start, runs under a
|
||
**dedicated service account** (required for DCOM authentication, see §11).
|
||
- SCM recovery: restart on failure (first/second failure), plus internal watchdog.
|
||
|
||
### 10.2 Connection state machine
|
||
```mermaid
|
||
stateDiagram-v2
|
||
[*] --> Connecting
|
||
Connecting --> Connected : connect OK
|
||
Connecting --> ReconnectWait : COM error
|
||
Connected --> ReconnectWait : read error / hang
|
||
ReconnectWait --> Connecting : backoff elapsed
|
||
```
|
||
Reconnect backoff: exponential 1 s → 2 s → … capped at 60 s. Every disconnect/reconnect
|
||
pair is an event; outage duration is computed and reported.
|
||
|
||
### 10.3 COM apartment handling
|
||
All OPC/COM work runs on one dedicated STA thread inside the service process
|
||
(`pythoncom.CoInitialize()` on that thread only). Detectors run on the same thread (pure
|
||
Python, cheap); event reporting happens off-thread via a queue so slow I/O never blocks
|
||
polling.
|
||
|
||
### 10.4 The DCOM hang problem (important)
|
||
DCOM/RPC calls have **no client-side timeout**. If the network path breaks silently
|
||
(e.g., firewall drops), a read call can block for tens of seconds to minutes without
|
||
erroring. The tool handles this explicitly:
|
||
|
||
1. A watchdog flags a **poll hang** (D7) as soon as a call exceeds `timeout_ms`.
|
||
2. If the hang exceeds `hard_hang_s` (default 120 s), the service abandons the stuck
|
||
thread and creates a fresh connection on a new thread (stale `RPC` objects are
|
||
released by COM).
|
||
3. Hang events are distinguished from clean disconnects — this is often the smoking gun
|
||
for silent network drops.
|
||
|
||
### 10.5 Safety constraints
|
||
- Read-only by construction: no write API is called; the code has no write path.
|
||
- Single OPC group, static item list, modest poll rate (default 1 s) to keep added
|
||
server load negligible.
|
||
- Configuration errors fail fast at startup with actionable messages
|
||
(`opcda-diagnozer check-config` validates without connecting… and `--probe` connects
|
||
once and reports).
|
||
|
||
## 11. Environment & constraints
|
||
|
||
| Constraint | Consequence |
|
||
|------------|-------------|
|
||
| OPC DA is Windows/COM only | Service runs on Windows only (Server 2012 R2 … 2022, Win 10/11) |
|
||
| OPC DA wrappers are often **32-bit** DLLs (OPC DA Automation Wrapper `OPCAutomation.dll`, Graybox `GBDaCtl.dll`) | Python interpreter bitness must match the wrapper DLL — likely **32-bit Python** for v1; decision in Phase 0 |
|
||
| DCOM security | The service account needs Launch+Activation and Access permissions on the OPC server (via `dcomcnfg`); account must authenticate across the network (not `LocalSystem`) |
|
||
| Post-2022 Windows DCOM hardening (KB5004442 family) | DCOM activation now requires explicit authentication; legacy OPC setups may fail activation — errors must be reported clearly |
|
||
| RPC over TCP | Firewall: TCP 135 + RPC dynamic port range on the server host; misconfig shows up as `RPC_S_SERVER_UNAVAILABLE` |
|
||
| Clock skew | Device timestamps may be misleading; every event also carries local receive time; recommend NTP on both hosts |
|
||
|
||
## 12. Diagnostic interpretation matrix (why this tool exists)
|
||
|
||
| Observation from the Diagnozer | Most likely problem location |
|
||
|--------------------------------|------------------------------|
|
||
| Clean reads, low RTT, all GOOD — while SCADA still complains | SCADA host itself: per-machine DCOM permissions, CPU/memory saturation, its own group/subscription load |
|
||
| Diagnozer also sees disconnects / RPC errors | Shared path: network, firewall, DCOM layer, or OPC server host |
|
||
| Connection stable, but `BAD_COMM_FAILURE` on many device tags | Link between the OPC server and the control devices (downstream of OPC) — **not** a DCOM/network problem |
|
||
| Slow RTT spikes at consistent times of day | Server load pattern; correlate with heartbeat summaries |
|
||
| Hangs without errors returned (D7) | Silent network drop (firewall/MTU/NIC) — classic DCOM pitfall |
|
||
| One instance (SCADA zone) slow, other (server zone) fast | Network segment between the zones |
|
||
|
||
## 13. Acceptance criteria (v1.0)
|
||
|
||
- **AC1** Runs as a service, survives reboot, auto-starts, recovers from a killed process.
|
||
- **AC2** Detects OPC server process stop within one poll interval (or one hang window for
|
||
silent drops).
|
||
- **AC3** Detects injected latency above thresholds within 1–2 polls; event contains
|
||
measured RTT.
|
||
- **AC4** Quality transitions produce events with raw code and decoded name.
|
||
- **AC5** Staleness/freeze detection within tolerance of `max_age_s`.
|
||
- **AC6** No OPC writes are ever issued (code review + no write path exists).
|
||
- **AC7** Events land in JSONL and Event Log; heartbeats appear even when all is healthy.
|
||
- **AC8** Invalid config fails fast with actionable messages.
|
||
- **AC9** Footprint ≤ ~50 MB RAM, ~0% CPU while idle-polling on a healthy link.
|
||
- **AC10** 24 h soak against a simulation server: no leaks, correct reconnect accounting.
|
||
|
||
## 14. Future extensions (post-v1)
|
||
|
||
- Async subscription monitoring (`IOPCAsyncIO2` DataChange) to compare push vs. poll
|
||
latency — validates the SCADA's actual callback path.
|
||
- Per-item latency bisection (read items individually to isolate a slow tag).
|
||
- SQLite time-series of samples for post-incident graphing.
|
||
- Prometheus exporter / Grafana dashboard.
|
||
- Email / Teams / webhook alerting with deduplication.
|
||
- Multi-server monitoring in one service. |