198 lines
8.9 KiB
Markdown
198 lines
8.9 KiB
Markdown
# OPC DA Diagnozer
|
|
|
|
A read-only Windows service that acts as an independent OPC DA reference client to
|
|
diagnose **disconnections**, **communication latency**, and **data-quality problems**
|
|
between a SCADA system and a control system communicating over OPC DA.
|
|
|
|
It continuously connects to the OPC DA server, polls a pre-defined list of tags, and
|
|
reports every anomaly with precise timestamps — unnatural read latencies, DCOM
|
|
disconnects, hung RPC calls, tags marked BAD/UNCERTAIN, stale timestamps, frozen values,
|
|
and flapping quality — as structured events (JSONL logs, Windows Event Log, optional
|
|
HTTP webhook).
|
|
|
|
> **Status:** planning. Read the
|
|
> [specification](docs/SPECIFICATION.md) and the
|
|
> [implementation plan](docs/IMPLEMENTATION_PLAN.md) first.
|
|
|
|
## Why
|
|
|
|
OPC DA failures are notoriously hard to pin down because they hide behind generic COM
|
|
errors, calls that hang with no timeout, or silently degraded item quality. This tool
|
|
answers the triage question *where is the problem?* with evidence:
|
|
|
|
| Observation from the Diagnozer | Most likely problem location |
|
|
|--------------------------------|------------------------------|
|
|
| Clean reads while SCADA complains | SCADA host itself (DCOM perms, CPU, its own load) |
|
|
| Diagnozer also sees disconnects/RPC errors | Shared path: network, firewall, DCOM, OPC server host |
|
|
| Connection stable but many tags `BAD_COMM_FAILURE` | Link between OPC server and the control devices |
|
|
| Hangs without errors | Silent network drop (firewall/MTU) — classic DCOM pitfall |
|
|
|
|
See the full interpretation matrix in the [specification](docs/SPECIFICATION.md).
|
|
|
|
## Planned features
|
|
|
|
- Windows service (auto-start, SCM crash recovery) with a foreground `run` mode for testing
|
|
- Local or remote (DCOM) connections to any OPC DA server
|
|
- Server browsing: `list-servers` (discovery), `list-devices` and `list-tags`
|
|
(address-space walk) — live now
|
|
- Latency monitoring: spikes, streaks, and p95 trends against configurable thresholds
|
|
- Hang watchdog: flags RPC calls that never return — the silent killer
|
|
- Quality monitoring: BAD/UNCERTAIN transitions with decoded subcodes, persistent-bad
|
|
and flapping detection, mass-degradation alarm
|
|
- Staleness & frozen-value detection while quality still reports GOOD
|
|
- Structured events: rotating JSONL, Windows Event Log, optional HTTP webhook
|
|
- Strictly read-only; single group, modest poll rate — does not perturb the measured system
|
|
|
|
## Quick start
|
|
|
|
```
|
|
# list the OPC DA servers available locally (or on a DCOM host with --host)
|
|
opcda-diagnozer list-servers
|
|
opcda-diagnozer list-servers --host CONTROL-HOST01 # remote, via DCOM
|
|
opcda-diagnozer list-servers --source registry --json # no-COM local scan
|
|
|
|
# list the devices (address-space branches) of the configured server
|
|
opcda-diagnozer list-devices
|
|
opcda-diagnozer list-devices --depth 2 # include sub-devices
|
|
opcda-diagnozer list-devices --json # script-friendly output
|
|
|
|
# list the tags of one device (the item_id values go into config `tags:`)
|
|
opcda-diagnozer list-tags "Simulation Items.Bucket Brigade"
|
|
opcda-diagnozer list-tags "Simulation Items" "Bucket Brigade" # equivalent
|
|
opcda-diagnozer list-tags --recursive "Simulation Items" # whole subtree
|
|
opcda-diagnozer list-tags # every tag on the server
|
|
|
|
# validate config
|
|
opcda-diagnozer check-config
|
|
|
|
# connect once and read every configured tag, print results
|
|
opcda-diagnozer probe
|
|
|
|
# foreground run (console) — best for first deployment validation
|
|
opcda-diagnozer run
|
|
|
|
# install as Windows service (dedicated account required for DCOM auth)
|
|
opcda-diagnozer install --account DOMAIN\svc-opcdiag
|
|
```
|
|
`list-servers` uses two discovery mechanisms (`--source`): the OPC DA Automation
|
|
Wrapper's `GetOPCServers` over COM (works remotely via DCOM; needs Launch+Activation
|
|
permissions on the target host), and a local COM-registration scan that reads the OPC
|
|
DA component categories directly from the registry — no COM, useful on locked-down
|
|
hosts or when the wrapper DLL is not registered. `auto` tries the first, falls back to
|
|
the second. Note the bitness rule still applies: a 32-bit process enumerates the
|
|
32-bit registry view — exactly the servers this tool can load.
|
|
|
|
`list-devices` / `list-tags` browse the **connected server's** address space through
|
|
the Automation Wrapper's browser (`CreateBrowser`): devices are *branches*, tags are
|
|
*leaves*. They work locally and over DCOM (same Launch+Activation requirements). A
|
|
device path is either one dotted string (`"Simulation Items.Bucket Brigade"`) or
|
|
several quoted branch names — use the latter when a branch name itself contains a
|
|
dot. On flat-namespace servers (`organization: flat`) every tag hangs directly off
|
|
the root; `list-tags` without a device lists them all.
|
|
### Configuration example (`config.example.yaml`)
|
|
|
|
```yaml
|
|
opc:
|
|
server: "Matrikon.OPC.Simulation.1" # server ProgID
|
|
host: "CONTROL-HOST01" # DCOM host; omit for local
|
|
connect_timeout_s: 10
|
|
|
|
poll:
|
|
interval_s: 1.0
|
|
timeout_ms: 5000 # beyond this the call is flagged as a hang
|
|
|
|
tags:
|
|
- { name: "Channel1.Device1.Counter", expect_change: true }
|
|
- { name: "Channel1.Device1.Level" }
|
|
|
|
latency:
|
|
warn_ms: 1000
|
|
critical_ms: 5000
|
|
slow_poll_streak: 5
|
|
window_polls: 60
|
|
|
|
quality:
|
|
persistent_bad_s: 30
|
|
flapping_transitions_per_window: 5
|
|
|
|
stale:
|
|
enabled: true
|
|
max_age_s: 120
|
|
|
|
report:
|
|
heartbeat_s: 300
|
|
jsonl: { dir: "C:/ProgramData/OPCDA-Diagnozer/logs", max_mb: 50, backups: 10 }
|
|
eventlog: { source: "OPCDA-Diagnozer", min_severity: WARN }
|
|
```
|
|
|
|
## DCOM troubleshooting quick reference
|
|
|
|
**Setup checklist for the service account (both hosts):**
|
|
1. `dcomcnfg` → Component Services → DCOM Config → the OPC server → Properties:
|
|
- Security → Launch & Activation: allow **Remote Activation** for the service account.
|
|
- Security → Access Permissions: allow Local + Remote access.
|
|
- Identity: prefer a dedicated account over "Launching user".
|
|
2. Firewall on the OPC server host: TCP 135 (RPC endpoint mapper) + the RPC dynamic
|
|
port range (optionally restricted via `netsh int ipv4 set dynamicport tcp ...`).
|
|
3. NTP time sync on both hosts — OPC DA timestamps and staleness detection depend on it.
|
|
4. Be aware of post-2022 Windows **DCOM hardening** (KB5004442 family): DCOM activation
|
|
requires explicit authentication and may break legacy OPC setups with
|
|
`E_ACCESSDENIED`-type failures.
|
|
|
|
**Common HRESULTs the tool decodes and reports:**
|
|
|
|
| HRESULT | Meaning |
|
|
|---------|---------|
|
|
| `0x800706BA` | RPC server unavailable — server process down, firewall, or DCOM activation refused |
|
|
| `0x800706BE` | RPC call failed mid-flight — server crashed or connection reset |
|
|
| `0x80010108` | Object disconnected from clients — OPC server dropped us (crash/restart) |
|
|
| `0x80070005` | Access denied — DCOM security misconfiguration |
|
|
| `0x80040154` | Class not registered — wrong ProgID, or wrapper/server registered under a different bitness |
|
|
| `0xC0040007` | Invalid item ID — tag name wrong in config (surfaced by `probe`) |
|
|
|
|
**Bitness warning:** the OPC DA Automation Wrapper (`OPCAutomation.dll`) and the Graybox
|
|
wrapper are typically **32-bit in-proc DLLs** — the Python interpreter must match
|
|
(32-bit). This is decided and verified in Phase 0 of the implementation plan.
|
|
|
|
## Safety
|
|
|
|
- The tool never writes to the OPC server — there is no write path in the code.
|
|
- It uses one OPC group and a static, small tag list at a modest poll rate (default 1 s)
|
|
so it does not load the system it is measuring.
|
|
|
|
## Development setup
|
|
|
|
The OPC DA wrappers (`OPCAutomation.dll`, `GBDaCtl.dll`) are **32-bit in-proc COM DLLs**,
|
|
so the Python interpreter **must be 32-bit** (SPECIFICATION §11 / Phase 0 decision).
|
|
|
|
```powershell
|
|
# create the 32-bit venv (requires the 32-bit Python installed via the py launcher)
|
|
& "C:\Users\<user>\AppData\Local\Programs\Python\Python314-32\python.exe" -m venv .venv
|
|
& ".\.venv\Scripts\python.exe" -m pip install -e ".[dev]"
|
|
|
|
# verify bitness — must print 32
|
|
& ".\.venv\Scripts\python.exe" -c "import struct; print(struct.calcsize('P') * 8)"
|
|
|
|
# run the (COM-free) unit + fake-integration tests
|
|
& ".\.venv\Scripts\python.exe" -m pytest
|
|
```
|
|
|
|
Notes:
|
|
- Dependencies are chosen to be installable without a C compiler: **ruamel.yaml**
|
|
(pure Python) is used instead of PyYAML, which publishes no 32-bit CPython 3.14
|
|
wheels and whose sdist requires MSVC.
|
|
- COM-dependent modules (Automation Wrapper, Event Log, service) are import-guarded;
|
|
the unit test suite runs on any OS via the in-process fake client
|
|
(`opcda_diagnozer.opc.fake`).
|
|
- 32-bit CPython can be installed with the Python Install Manager
|
|
(`py install 3.14 -a x86`) or the installer from python.org ("Download Windows
|
|
installer (32-bit)").
|
|
|
|
## Documentation
|
|
|
|
- [Specification](docs/SPECIFICATION.md) — detection rules, event model, operational behavior, acceptance criteria
|
|
- [Implementation plan](docs/IMPLEMENTATION_PLAN.md) — phases, milestones, testing, risks
|
|
|
|
## License
|
|
|
|
TBD |