opcda-diagnozer/README.md

# OPC DA Diagnozer

A read-only Windows service that acts as an independent OPC DA reference client to
diagnose **disconnections**, **communication latency**, and **data-quality problems**
between a SCADA system and a control system communicating over OPC DA.

It continuously connects to the OPC DA server, polls a pre-defined list of tags, and
reports every anomaly with precise timestamps — unnatural read latencies, DCOM
disconnects, hung RPC calls, tags marked BAD/UNCERTAIN, stale timestamps, frozen values,
and flapping quality — as structured events (JSONL logs, Windows Event Log, optional
HTTP webhook).

> **Status:** planning. Read the
> [specification](docs/SPECIFICATION.md) and the
> [implementation plan](docs/IMPLEMENTATION_PLAN.md) first.

## Why

OPC DA failures are notoriously hard to pin down because they hide behind generic COM
errors, calls that hang with no timeout, or silently degraded item quality. This tool
answers the triage question *where is the problem?* with evidence:

| Observation from the Diagnozer | Most likely problem location |
|--------------------------------|------------------------------|
| Clean reads while SCADA complains | SCADA host itself (DCOM perms, CPU, its own load) |
| Diagnozer also sees disconnects/RPC errors | Shared path: network, firewall, DCOM, OPC server host |
| Connection stable but many tags `BAD_COMM_FAILURE` | Link between OPC server and the control devices |
| Hangs without errors | Silent network drop (firewall/MTU) — classic DCOM pitfall |

See the full interpretation matrix in the [specification](docs/SPECIFICATION.md).

## Planned features

- Windows service (auto-start, SCM crash recovery) with a foreground `run` mode for testing
- Local or remote (DCOM) connections to any OPC DA server
- Server browsing: `list-servers` (discovery), `list-devices` and `list-tags`
  (address-space walk) — live now
- Latency monitoring: spikes, streaks, and p95 trends against configurable thresholds
- Hang watchdog: flags RPC calls that never return — the silent killer
- Quality monitoring: BAD/UNCERTAIN transitions with decoded subcodes, persistent-bad
  and flapping detection, mass-degradation alarm
- Staleness & frozen-value detection while quality still reports GOOD
- Structured events: rotating JSONL, Windows Event Log, optional HTTP webhook
- Strictly read-only; single group, modest poll rate — does not perturb the measured system

## Quick start

```
# list the OPC DA servers available locally (or on a DCOM host with --host)
opcda-diagnozer list-servers
opcda-diagnozer list-servers --host CONTROL-HOST01        # remote, via DCOM
opcda-diagnozer list-servers --source registry --json     # no-COM local scan

# list the devices (address-space branches) of the configured server
opcda-diagnozer list-devices
opcda-diagnozer list-devices --depth 2                    # include sub-devices
opcda-diagnozer list-devices --json                       # script-friendly output

# list the tags of one device (the item_id values go into config `tags:`)
opcda-diagnozer list-tags "Simulation Items.Bucket Brigade"
opcda-diagnozer list-tags "Simulation Items" "Bucket Brigade"   # equivalent
opcda-diagnozer list-tags --recursive "Simulation Items"  # whole subtree
opcda-diagnozer list-tags                                 # every tag on the server

# read tags right now (validation tool) — no config needed with --server
opcda-diagnozer read --server Matrikon.OPC.Simulation.1 "Bucket Brigade.Int1"
opcda-diagnozer read --json "Bucket Brigade.Int1" "Bucket Brigade.Real8"
opcda-diagnozer read                                     # the config's tags
opcda-diagnozer read --source cache "Bucket Brigade.Int1" # server cache, no device trip

# validate config
opcda-diagnozer check-config

# connect once and read every configured tag, print results
opcda-diagnozer probe

# foreground run (console) — best for first deployment validation
opcda-diagnozer run

# install as Windows service (dedicated account required for DCOM auth)
opcda-diagnozer install --account DOMAIN\svc-opcdiag
```
`list-servers` uses two discovery mechanisms (`--source`): the OPC DA Automation
Wrapper's `GetOPCServers` over COM (works remotely via DCOM; needs Launch+Activation
permissions on the target host), and a local COM-registration scan that reads the OPC
DA component categories directly from the registry — no COM, useful on locked-down
hosts or when the wrapper DLL is not registered. `auto` tries the first, falls back to
the second. Note the bitness rule still applies: a 32-bit process enumerates the
32-bit registry view — exactly the servers this tool can load.

`list-devices` / `list-tags` browse the **connected server's** address space through
the Automation Wrapper's browser (`CreateBrowser`): devices are *branches*, tags are
*leaves*. They work locally and over DCOM (same Launch+Activation requirements). A
device path is either one dotted string (`"Simulation Items.Bucket Brigade"`) or
several quoted branch names — use the latter when a branch name itself contains a
dot. On flat-namespace servers (`organization: flat`) every tag hangs directly off
the root; `list-tags` without a device lists them all.

Every connecting verb (`read`, `probe`, `list-devices`, `list-tags`) accepts
`--server PROGID` and `--host HOST` to **override the config's `opc` section** —
point a one-shot check at any server without touching the deployed config.yaml.
With `--server` given, no config file is needed at all; without it, the config must
exist (and `probe`/`read` read the config's tag list unless tags are passed).

`read` prints one row per tag with the decoded quality and the device timestamp;
a non-GOOD or failing tag makes the exit code non-zero, so it doubles as a
scriptable health check. `--source cache` reads the server cache instead of
forcing a device round-trip.
### Configuration example (`config.example.yaml`)

```yaml
opc:
  server: "Matrikon.OPC.Simulation.1"   # server ProgID
  host: "CONTROL-HOST01"                # DCOM host; omit for local
  connect_timeout_s: 10

poll:
  interval_s: 1.0
  timeout_ms: 5000                      # beyond this the call is flagged as a hang

tags:
  - { name: "Channel1.Device1.Counter", expect_change: true }
  - { name: "Channel1.Device1.Level" }

latency:
  warn_ms: 1000
  critical_ms: 5000
  slow_poll_streak: 5
  window_polls: 60

quality:
  persistent_bad_s: 30
  flapping_transitions_per_window: 5

stale:
  enabled: true
  max_age_s: 120

report:
  heartbeat_s: 300
  jsonl:  { dir: "C:/ProgramData/OPCDA-Diagnozer/logs", max_mb: 50, backups: 10 }
  eventlog: { source: "OPCDA-Diagnozer", min_severity: WARN }
```

## DCOM troubleshooting quick reference

**Setup checklist for the service account (both hosts):**
1. `dcomcnfg` → Component Services → DCOM Config → the OPC server → Properties:
   - Security → Launch & Activation: allow **Remote Activation** for the service account.
   - Security → Access Permissions: allow Local + Remote access.
   - Identity: prefer a dedicated account over "Launching user".
2. Firewall on the OPC server host: TCP 135 (RPC endpoint mapper) + the RPC dynamic
   port range (optionally restricted via `netsh int ipv4 set dynamicport tcp ...`).
3. NTP time sync on both hosts — OPC DA timestamps and staleness detection depend on it.
4. Be aware of post-2022 Windows **DCOM hardening** (KB5004442 family): DCOM activation
   requires explicit authentication and may break legacy OPC setups with
   `E_ACCESSDENIED`-type failures.

**Common HRESULTs the tool decodes and reports:**

| HRESULT | Meaning |
|---------|---------|
| `0x800706BA` | RPC server unavailable — server process down, firewall, or DCOM activation refused |
| `0x800706BE` | RPC call failed mid-flight — server crashed or connection reset |
| `0x80010108` | Object disconnected from clients — OPC server dropped us (crash/restart) |
| `0x80070005` | Access denied — DCOM security misconfiguration |
| `0x80040154` | Class not registered — wrong ProgID, or wrapper/server registered under a different bitness |
| `0xC0040007` | Invalid item ID — tag name wrong in config (surfaced by `probe`) |

**Bitness warning:** the OPC DA Automation Wrapper (`OPCAutomation.dll`) and the Graybox
wrapper are typically **32-bit in-proc DLLs** — the Python interpreter must match
(32-bit). This is decided and verified in Phase 0 of the implementation plan.

## Safety

- The tool never writes to the OPC server — there is no write path in the code.
- It uses one OPC group and a static, small tag list at a modest poll rate (default 1 s)
  so it does not load the system it is measuring.

## Development setup

The OPC DA wrappers (`OPCAutomation.dll`, `GBDaCtl.dll`) are **32-bit in-proc COM DLLs**,
so the Python interpreter **must be 32-bit** (SPECIFICATION §11 / Phase 0 decision).

```powershell
# create the 32-bit venv (requires the 32-bit Python installed via the py launcher)
& "C:\Users\<user>\AppData\Local\Programs\Python\Python314-32\python.exe" -m venv .venv
& ".\.venv\Scripts\python.exe" -m pip install -e ".[dev]"

# verify bitness — must print 32
& ".\.venv\Scripts\python.exe" -c "import struct; print(struct.calcsize('P') * 8)"

# run the (COM-free) unit + fake-integration tests
& ".\.venv\Scripts\python.exe" -m pytest
```

Notes:
- Dependencies are chosen to be installable without a C compiler: **ruamel.yaml**
  (pure Python) is used instead of PyYAML, which publishes no 32-bit CPython 3.14
  wheels and whose sdist requires MSVC.
- COM-dependent modules (Automation Wrapper, Event Log, service) are import-guarded;
  the unit test suite runs on any OS via the in-process fake client
  (`opcda_diagnozer.opc.fake`).
- 32-bit CPython can be installed with the Python Install Manager
  (`py install 3.14 -a x86`) or the installer from python.org ("Download Windows
  installer (32-bit)").

## Building the Windows executable

`scripts/build_exe.ps1` packages the CLI as a standalone **32-bit** console exe
with [PyInstaller](https://pyinstaller.org) — no Python installation needed on
the target host:

```powershell
& ".\.venv\Scripts\python.exe" -m pip install pyinstaller   # once
.\scripts\build_exe.ps1                                     # dist\opcda-diagnozer\
.\scripts\build_exe.ps1 -Onefile                            # single exe alternative
.\scripts\build_exe.ps1 -Clean                              # fresh rebuild
```

The script refuses to run from a 64-bit interpreter (the exe must load the
32-bit OPC DA wrapper DLLs, SPECIFICATION §11), re-verifies the produced PE
really is 32-bit, embeds the project version in the exe properties, declares
the hidden imports the code loads lazily (pywin32 COM, browse modules, the
Phase 3 service host), and smoke-tests the artifact (`--version`,
`check-config`, `list-servers --source registry`) after building. It also
installs PyInstaller into the venv automatically if it is missing.

If your policy blocks the script, run it with
`powershell -NoProfile -ExecutionPolicy Bypass -File scripts\build_exe.ps1`.

Deployment notes:
- The **default config path** of a frozen exe is `config.yaml` **next to the
  exe** — not the working directory. A Windows service starts with
  `CWD = C:\Windows\System32`, so the relative default would silently point
  there; the exe-dir default keeps the deployment layout simple
  (`C:\Program Files\OPCDA-Diagnozer\{opcda-diagnozer.exe, config.yaml}`).
- The exe is a console application that hosts every CLI verb, including the
  future `run` / `probe` / service verbs. The Phase 3 Windows service
  (`install` verb) will point `sc.exe` at this same exe with the service-host
  entry point.
- The Automation Wrapper DLL (`OPCAutomation.dll`) is **not** bundled — it is
  a machine-installed COM component (registered in the registry), exactly like
  the OPC DA servers the tool connects to. Only Python code and its
  dependencies are frozen.
- `onedir` is the default (faster startup, AV-friendlier); `--onefile` trades
  startup time for a single file. `--clean` removes `build/` + `dist/` first.

## Documentation

- [Specification](docs/SPECIFICATION.md) — detection rules, event model, operational behavior, acceptance criteria
- [Implementation plan](docs/IMPLEMENTATION_PLAN.md) — phases, milestones, testing, risks

## License

TBD