OPC DA Diagnozer

A read-only Windows service that acts as an independent OPC DA reference client to diagnose disconnections, communication latency, and data-quality problems between a SCADA system and a control system communicating over OPC DA.

It continuously connects to the OPC DA server, polls a pre-defined list of tags, and reports every anomaly with precise timestamps — unnatural read latencies, DCOM disconnects, hung RPC calls, tags marked BAD/UNCERTAIN, stale timestamps, frozen values, and flapping quality — as structured events (JSONL logs, Windows Event Log, optional HTTP webhook).

Status: in active development — discovery, browsing (list-items), one-shot reads (read/probe) and the exe build are live and field-verified against a real server; the continuous monitoring loop, reporters and Windows service host are next. See the TODO for current status and the implementation plan for the phases; read the specification for detection rules and behavior.

Why

OPC DA failures are notoriously hard to pin down because they hide behind generic COM errors, calls that hang with no timeout, or silently degraded item quality. This tool answers the triage question where is the problem? with evidence:

Observation from the Diagnozer	Most likely problem location
Clean reads while SCADA complains	SCADA host itself (DCOM perms, CPU, its own load)
Diagnozer also sees disconnects/RPC errors	Shared path: network, firewall, DCOM, OPC server host
Connection stable but many tags BAD_COMM_FAILURE	Link between OPC server and the control devices
Hangs without errors	Silent network drop (firewall/MTU) — classic DCOM pitfall

See the full interpretation matrix in the specification.

Planned features

• Windows service (auto-start, SCM crash recovery) with a foreground run mode for testing
• Local or remote (DCOM) connections to any OPC DA server
• Server browsing: list-servers (discovery) and list-items (address-space browse: full tree / branches / tags) — live now
• Latency monitoring: spikes, streaks, and p95 trends against configurable thresholds
• Hang watchdog: flags RPC calls that never return — the silent killer
• Quality monitoring: BAD/UNCERTAIN transitions with decoded subcodes, persistent-bad and flapping detection, mass-degradation alarm
• Staleness & frozen-value detection while quality still reports GOOD
• Structured events: rotating JSONL, Windows Event Log, optional HTTP webhook
• Strictly read-only; single group, modest poll rate — does not perturb the measured system

Quick start

# list the OPC DA servers available locally (or on a DCOM host with --host)
opcda-diagnozer list-servers
opcda-diagnozer list-servers --host CONTROL-HOST01        # remote, via DCOM
opcda-diagnozer list-servers --source registry --json     # no-COM local scan

# browse the server's item space (list-items all|branches|tags)
#   all      = the full structure tree: every branch AND every tag in place
#   branches = the structure nodes (channels/devices/folders — all "branches" to OPC DA)
#   tags     = the flat ItemID list that goes into config `tags:`
opcda-diagnozer list-items                       # top level of the tree
opcda-diagnozer list-items --depth 0              # the whole tree (browse.max_depth guards)
opcda-diagnozer list-items branches --depth 2     # structure, two levels
opcda-diagnozer list-items tags --branch "Simulation Items"   # one subtree, flat ItemIDs
opcda-diagnozer list-items --json                 # script-friendly output

# read tags right now (validation tool) — no config needed with --server
opcda-diagnozer read --server Matrikon.OPC.Simulation.1 "Bucket Brigade.Int1"
opcda-diagnozer read --json "Bucket Brigade.Int1" "Bucket Brigade.Real8"
opcda-diagnozer read                                     # the config's tags
opcda-diagnozer read --source cache "Bucket Brigade.Int1" # server cache, no device trip

# validate config
opcda-diagnozer check-config

# connect once and read every configured tag, print results
opcda-diagnozer probe

# foreground run (console) — best for first deployment validation
opcda-diagnozer run

# install as Windows service (dedicated account required for DCOM auth)
opcda-diagnozer install --account DOMAIN\svc-opcdiag

list-servers uses two discovery mechanisms (--source): the OPC DA Automation Wrapper's GetOPCServers over COM (works remotely via DCOM; needs Launch+Activation permissions on the target host), and a local COM-registration scan that reads the OPC DA component categories directly from the registry — no COM, useful on locked-down hosts or when the wrapper DLL is not registered. auto tries the first, falls back to the second. Note the bitness rule still applies: a 32-bit process enumerates the 32-bit registry view — exactly the servers this tool can load.

list-items browses the connected server's address space through the Automation Wrapper's browser (CreateBrowser). OPC DA defines exactly two element kinds — branches (the structure: what SCADA vendors call channels, devices, groups or folders; the protocol cannot tell them apart) and leaves (the tags). The all view renders the full tree — including leaves that hang directly at the server root and belong to no branch (server-internal items like @ClientCount). It works locally and over DCOM (same Launch+Activation requirements). A --branch path is either one dotted string ("Simulation Items.Bucket Brigade") or repeated flag segments — use the latter when a branch name itself contains a dot. --depth 1 (default) lists one level; --depth 0 walks the whole subtree, guarded by browse.max_depth from the config (default 32) against server-side cycles.

Every connecting verb (read, probe, list-items) accepts --server PROGID and --host HOST to override the config's opc section — point a one-shot check at any server without touching the deployed config.yaml. With --server given, no config file is needed at all; without it, the config must exist (and probe/read read the config's tag list unless tags are passed).

read prints one row per tag with the decoded quality and the device timestamp; a non-GOOD or failing tag makes the exit code non-zero, so it doubles as a scriptable health check. --source cache reads the server cache instead of forcing a device round-trip.

Configuration example (config.example.yaml)

opc:
  server: "Matrikon.OPC.Simulation.1"   # server ProgID
  host: "CONTROL-HOST01"                # DCOM host; omit for local
  connect_timeout_s: 10

poll:
  interval_s: 1.0
  timeout_ms: 5000                      # beyond this the call is flagged as a hang

tags:
  - { name: "Channel1.Device1.Counter", expect_change: true }
  - { name: "Channel1.Device1.Level" }

# D16 device-link indicators: tags saying whether the OPC server can reach
# the device. DOWN (DEVICE_LINK_DOWN, CRITICAL) when the tag reads non-GOOD,
# errors, or its GOOD value is NOT in ok_values. ok_values defaults to [true]
# (boolean connected flags); inverted fault flags -> [false, 0]; string
# states -> ["RUN"]. Do NOT also list these under tags:.
devices:
  - { name: "Conveyor 3", tag: "Channel1.Device3.Connected" }
  - { name: "Pump A", tag: "Channel1.PumpA.Fault", ok_values: [false, 0] }

latency:
  warn_ms: 1000
  critical_ms: 5000
  slow_poll_streak: 5
  window_polls: 60

quality:
  persistent_bad_s: 30
  flapping_transitions_per_window: 5

stale:
  enabled: true
  max_age_s: 120

browse:
  max_depth: 32                         # address-space walk guard (list-items --depth 0)

report:
  heartbeat_s: 300
  jsonl:  { dir: "C:/ProgramData/OPCDA-Diagnozer/logs", max_mb: 50, backups: 10 }
  eventlog: { source: "OPCDA-Diagnozer", min_severity: WARN }

DCOM troubleshooting quick reference

Setup checklist for the service account (both hosts):

1. dcomcnfg → Component Services → DCOM Config → the OPC server → Properties:
   • Security → Launch & Activation: allow Remote Activation for the service account.
   • Security → Access Permissions: allow Local + Remote access.
   • Identity: prefer a dedicated account over "Launching user".
2. Firewall on the OPC server host: TCP 135 (RPC endpoint mapper) + the RPC dynamic port range (optionally restricted via netsh int ipv4 set dynamicport tcp ...).
3. NTP time sync on both hosts — OPC DA timestamps and staleness detection depend on it.
4. Be aware of post-2022 Windows DCOM hardening (KB5004442 family): DCOM activation requires explicit authentication and may break legacy OPC setups with E_ACCESSDENIED-type failures.

Common HRESULTs the tool decodes and reports:

HRESULT	Meaning
0x800706BA	RPC server unavailable — server process down, firewall, or DCOM activation refused
0x800706BE	RPC call failed mid-flight — server crashed or connection reset
0x80010108	Object disconnected from clients — OPC server dropped us (crash/restart)
0x80070005	Access denied — DCOM security misconfiguration
0x80040154	Class not registered — wrong ProgID, or wrapper/server registered under a different bitness
0xC0040007	Invalid item ID — tag name wrong in config (surfaced by probe)

Bitness warning: the OPC DA Automation Wrapper (OPCAutomation.dll) and the Graybox wrapper are typically 32-bit in-proc DLLs — the Python interpreter must match (32-bit). This is decided and verified in Phase 0 of the implementation plan.

Safety

• The tool never writes to the OPC server — there is no write path in the code.
• It uses one OPC group and a static, small tag list at a modest poll rate (default 1 s) so it does not load the system it is measuring.

Development setup

The OPC DA wrappers (OPCAutomation.dll, GBDaCtl.dll) are 32-bit in-proc COM DLLs, so the Python interpreter must be 32-bit (SPECIFICATION §11 / Phase 0 decision).

# create the 32-bit venv (requires the 32-bit Python installed via the py launcher)
& "C:\Users\<user>\AppData\Local\Programs\Python\Python314-32\python.exe" -m venv .venv
& ".\.venv\Scripts\python.exe" -m pip install -e ".[dev]"

# verify bitness — must print 32
& ".\.venv\Scripts\python.exe" -c "import struct; print(struct.calcsize('P') * 8)"

# run the (COM-free) unit + fake-integration tests
& ".\.venv\Scripts\python.exe" -m pytest

Notes:

• Dependencies are chosen to be installable without a C compiler: ruamel.yaml (pure Python) is used instead of PyYAML, which publishes no 32-bit CPython 3.14 wheels and whose sdist requires MSVC.
• COM-dependent modules (Automation Wrapper, Event Log, service) are import-guarded; the unit test suite runs on any OS via the in-process fake client (opcda_diagnozer.opc.fake).
• 32-bit CPython can be installed with the Python Install Manager (py install 3.14 -a x86) or the installer from python.org ("Download Windows installer (32-bit)").

Building the Windows executable

scripts/build_exe.ps1 packages the CLI as a standalone 32-bit console exe with PyInstaller — no Python installation needed on the target host:

& ".\.venv\Scripts\python.exe" -m pip install pyinstaller   # once
.\scripts\build_exe.ps1                                     # dist\opcda-diagnozer\
.\scripts\build_exe.ps1 -Onefile                            # single exe alternative
.\scripts\build_exe.ps1 -Clean                              # fresh rebuild

The script refuses to run from a 64-bit interpreter (the exe must load the 32-bit OPC DA wrapper DLLs, SPECIFICATION §11), re-verifies the produced PE really is 32-bit, embeds the project version in the exe properties, declares the hidden imports the code loads lazily (pywin32 COM, browse modules, the Phase 3 service host), and smoke-tests the artifact (--version, check-config, list-servers --source registry) after building. It also installs PyInstaller into the venv automatically if it is missing.

If your policy blocks the script, run it with powershell -NoProfile -ExecutionPolicy Bypass -File scripts\build_exe.ps1.

Deployment notes:

• The default config path of a frozen exe is config.yaml next to the exe — not the working directory. A Windows service starts with CWD = C:\Windows\System32, so the relative default would silently point there; the exe-dir default keeps the deployment layout simple (C:\Program Files\OPCDA-Diagnozer\{opcda-diagnozer.exe, config.yaml}).
• The exe is a console application that hosts every CLI verb, including the future run / probe / service verbs. The Phase 3 Windows service (install verb) will point sc.exe at this same exe with the service-host entry point.
• The Automation Wrapper DLL (OPCAutomation.dll) is not bundled — it is a machine-installed COM component (registered in the registry), exactly like the OPC DA servers the tool connects to. Only Python code and its dependencies are frozen.
• onedir is the default (faster startup, AV-friendlier); --onefile trades startup time for a single file. --clean removes build/ + dist/ first.

Documentation

• Specification — detection rules, event model, operational behavior, acceptance criteria
• Implementation plan — phases, milestones, testing, risks
• TODO — implementation status: what's done, what's next
• Changelog — notable changes per release

License

TBD