opcda-diagnozer/README.md

# OPC DA Diagnozer

A read-only Windows service that acts as an independent OPC DA reference client to
diagnose **disconnections**, **communication latency**, and **data-quality problems**
between a SCADA system and a control system communicating over OPC DA.

It continuously connects to the OPC DA server, polls a pre-defined list of tags, and
reports every anomaly with precise timestamps — unnatural read latencies, DCOM
disconnects, hung RPC calls, tags marked BAD/UNCERTAIN, stale timestamps, frozen values,
and flapping quality — as structured events (JSONL logs, Windows Event Log, optional
HTTP webhook).

> **Status:** in active development — discovery, browsing (`list-items`), one-shot
> reads (`read`/`probe`), the continuous monitoring loop (`run`), all reporters,
> the exe build and the Windows service host are **live and field-verified**
> (service: in-process debug host verified; the elevated SCM pass is the remaining
> live check). See the [TODO](TODO.md) for current status and the
> [implementation plan](docs/IMPLEMENTATION_PLAN.md) for the phases; read the
> [specification](docs/SPECIFICATION.md) for detection rules and behavior.

## Why

OPC DA failures are notoriously hard to pin down because they hide behind generic COM
errors, calls that hang with no timeout, or silently degraded item quality. This tool
answers the triage question *where is the problem?* with evidence:

| Observation from the Diagnozer | Most likely problem location |
|--------------------------------|------------------------------|
| Clean reads while SCADA complains | SCADA host itself (DCOM perms, CPU, its own load) |
| Diagnozer also sees disconnects/RPC errors | Shared path: network, firewall, DCOM, OPC server host |
| Connection stable but many tags `BAD_COMM_FAILURE` | Link between OPC server and the control devices |
| Hangs without errors | Silent network drop (firewall/MTU) — classic DCOM pitfall |

See the full interpretation matrix in the [specification](docs/SPECIFICATION.md).

## Features

- Windows service (auto-start, SCM crash recovery) with a foreground `run` mode for testing — live
- Local or remote (DCOM) connections to any OPC DA server
- Server browsing: `list-servers` (discovery) and `list-items`
  (address-space browse: full tree / branches / tags) — live now
- **Continuous monitoring** (`run`): DCOM disconnect/reconnect accounting,
  latency spike/streak/trend, hang watchdog (soft flag + thread rebuild),
  BAD/UNCERTAIN transitions with persistence & flapping, stale/frozen tags,
  mass-degradation alarm, per-device link indicators (D16) — live now
- Latency monitoring: spikes, streaks, and p95 trends against configurable thresholds
- Hang watchdog: flags RPC calls that never return — the silent killer
- Quality monitoring: BAD/UNCERTAIN transitions with decoded subcodes, persistent-bad
  and flapping detection, mass-degradation alarm
- Staleness & frozen-value detection while quality still reports GOOD
- Structured events: rotating JSONL, Windows Event Log, optional HTTP webhook
- Strictly read-only; single group, modest poll rate — does not perturb the measured system

## Quick start

```
# list the OPC DA servers available locally (or on a DCOM host with --host)
opcda-diagnozer list-servers
opcda-diagnozer list-servers --host CONTROL-HOST01        # remote, via DCOM
opcda-diagnozer list-servers --source registry --json     # no-COM local scan

# browse the server's item space (list-items all|branches|tags)
#   all      = the full structure tree: every branch AND every tag in place
#   branches = the structure nodes (channels/devices/folders — all "branches" to OPC DA)
#   tags     = the flat ItemID list that goes into config `tags:`
opcda-diagnozer list-items                       # top level of the tree
opcda-diagnozer list-items --depth 0              # the whole tree (browse.max_depth guards)
opcda-diagnozer list-items branches --depth 2     # structure, two levels
opcda-diagnozer list-items tags --branch "Simulation Items"   # one subtree, flat ItemIDs
opcda-diagnozer list-items --json                 # script-friendly output

# read tags right now (validation tool) — no config needed with --server
opcda-diagnozer read --server Matrikon.OPC.Simulation.1 "Bucket Brigade.Int1"
opcda-diagnozer read --json "Bucket Brigade.Int1" "Bucket Brigade.Real8"
opcda-diagnozer read                                     # the config's tags
opcda-diagnozer read --source cache "Bucket Brigade.Int1" # server cache, no device trip

# validate config
opcda-diagnozer check-config

# connect once and read every configured tag, print results
opcda-diagnozer probe

# foreground run (console) — continuous monitoring; Ctrl+C stops
opcda-diagnozer run

# install as a Windows service (auto-start + SCM crash recovery; admin shell)
# LocalSystem default; a dedicated account is required for remote DCOM auth
opcda-diagnozer install
opcda-diagnozer install --account DOMAIN\svc-opcdiag --password ****

# service lifecycle + state
opcda-diagnozer start
opcda-diagnozer stop
opcda-diagnozer status
opcda-diagnozer uninstall
```
`list-servers` uses two discovery mechanisms (`--source`): the OPC DA Automation
Wrapper's `GetOPCServers` over COM (works remotely via DCOM; needs Launch+Activation
permissions on the target host), and a local COM-registration scan that reads the OPC
DA component categories directly from the registry — no COM, useful on locked-down
hosts or when the wrapper DLL is not registered. `auto` tries the first, falls back to
the second. Note the bitness rule still applies: a 32-bit process enumerates the
32-bit registry view — exactly the servers this tool can load.

`list-items` browses the **connected server's** address space through the
Automation Wrapper's browser (`CreateBrowser`). OPC DA defines exactly two
element kinds — **branches** (the structure: what SCADA vendors call channels,
devices, groups or folders; the protocol cannot tell them apart) and **leaves**
(the tags). The `all` view renders the full tree — including leaves that hang
directly at the server root and belong to no branch (server-internal items
like `@ClientCount`). It works locally and over DCOM (same Launch+Activation
requirements). A `--branch` path is either one dotted string
(`"Simulation Items.Bucket Brigade"`) or repeated flag segments — use the
latter when a branch name itself contains a dot. `--depth 1` (default) lists
one level; `--depth 0` walks the whole subtree, guarded by `browse.max_depth`
from the config (default 32) against server-side cycles.

Every connecting verb (`read`, `probe`, `list-items`) accepts
`--server PROGID` and `--host HOST` to **override the config's `opc` section** —
point a one-shot check at any server without touching the deployed config.yaml.
With `--server` given, no config file is needed at all; without it, the config must
exist (and `probe`/`read` read the config's tag list unless tags are passed).

`read` prints one row per tag with the decoded quality and the device timestamp;
a non-GOOD or failing tag makes the exit code non-zero, so it doubles as a
scriptable health check. `--source cache` reads the server cache instead of
forcing a device round-trip.

## Windows service

The monitor runs unattended as a **Windows service** (auto-start, SCM crash
recovery) — `opcda-diagnozer install` registers the exe itself as the service
binary (no `pythonservice.exe`, no Python install on the diag host):

- **Auto-start** at boot and **restart on failure**: the SCM restarts the
  service process 5 s after the first and second failure (the failure counter
  resets after 24 h of clean running) — `sc qfailure OPCDA-Diagnozer` shows it.
- **Config path** is recorded at install time (`install --config ...`); the
  service never depends on its working directory (the SCM starts services
  with CWD = System32). The default is `config.yaml` next to the exe.
- **Clean stop**: a stop request parks the service in `STOP_PENDING` and
  gives the poll loop 30 s to finish; a COM poll stuck in a dead RPC longer
  than that is abandoned (the thread is never joined — spec §10.4).
- **Event Log** records service start/stop and every configured reporter
  event; `install` also registers the event source so messages render
  correctly (needs the admin shell anyway).
- **Status** works without admin: `opcda-diagnozer status` prints the SCM
  state (`STOPPED` / `RUNNING` / …).

Service **account** (spec §11): `LocalSystem` (the default) works for OPC
DA servers on the same machine. Remote DCOM connections need a **dedicated
domain account** that has Launch+Activation and Access permissions on the
OPC server host (`dcomcnfg` — see the checklist above) and can authenticate
across the network; install with `--account DOMAIN\USER --password ****`
(the password is only used once, to register the account with the SCM).

**Deployment layout** (frozen exe):

```
C:\Program Files\OPCDA-Diagnozer\
    opcda-diagnozer.exe     # the console exe: every CLI verb AND the service host
    config.yaml             # the deployed config (install records its path)
C:\ProgramData\OPCDA-Diagnozer\
    logs\                   # events.jsonl + anomalies-YYYYMMDD.log (report.dir)
```

Log directories are created on demand; `%ProgramData%` is the default because
the service account must be able to write the evidence logs.

### Configuration example (`config.example.yaml`)

```yaml
opc:
  server: "Matrikon.OPC.Simulation.1"   # server ProgID
  host: "CONTROL-HOST01"                # DCOM host; omit for local
  connect_timeout_s: 10

poll:
  interval_s: 1.0
  timeout_ms: 5000                      # beyond this the call is flagged as a hang

tags:
  - { name: "Channel1.Device1.Counter", expect_change: true }
  - { name: "Channel1.Device1.Level" }

# D16 device-link indicators: tags saying whether the OPC server can reach
# the device. DOWN (DEVICE_LINK_DOWN, CRITICAL) when the tag reads non-GOOD,
# errors, or its GOOD value is NOT in ok_values. ok_values defaults to [true]
# (boolean connected flags); inverted fault flags -> [false, 0]; string
# states -> ["RUN"]. Do NOT also list these under tags:.
devices:
  - { name: "Conveyor 3", tag: "Channel1.Device3.Connected" }
  - { name: "Pump A", tag: "Channel1.PumpA.Fault", ok_values: [false, 0] }

latency:
  warn_ms: 1000
  critical_ms: 5000
  slow_poll_streak: 5
  window_polls: 60

quality:
  persistent_bad_s: 30
  flapping_transitions_per_window: 5

stale:
  enabled: true
  max_age_s: 120

browse:
  max_depth: 32                         # address-space walk guard (list-items --depth 0)

report:
  heartbeat_s: 300
  jsonl:  { dir: "C:/ProgramData/OPCDA-Diagnozer/logs", max_mb: 50, backups: 10 }
  textlog: { dir: "C:/ProgramData/OPCDA-Diagnozer/logs", max_mb: 20, backups: 10 }
  eventlog: { source: "OPCDA-Diagnozer", min_severity: WARN }
  #http: { url: "https://alerts.example.com/api", bearer_token: "...", timeout_s: 5 }
  eventlog: { source: "OPCDA-Diagnozer", min_severity: WARN }
```

## DCOM troubleshooting quick reference

**Setup checklist for the service account (both hosts):**
1. `dcomcnfg` → Component Services → DCOM Config → the OPC server → Properties:
   - Security → Launch & Activation: allow **Remote Activation** for the service account.
   - Security → Access Permissions: allow Local + Remote access.
   - Identity: prefer a dedicated account over "Launching user".
2. Firewall on the OPC server host: TCP 135 (RPC endpoint mapper) + the RPC dynamic
   port range (optionally restricted via `netsh int ipv4 set dynamicport tcp ...`).
3. NTP time sync on both hosts — OPC DA timestamps and staleness detection depend on it.
4. Be aware of post-2022 Windows **DCOM hardening** (KB5004442 family): DCOM activation
   requires explicit authentication and may break legacy OPC setups with
   `E_ACCESSDENIED`-type failures.

**Common HRESULTs the tool decodes and reports:**

| HRESULT | Meaning |
|---------|---------|
| `0x800706BA` | RPC server unavailable — server process down, firewall, or DCOM activation refused |
| `0x800706BE` | RPC call failed mid-flight — server crashed or connection reset |
| `0x80010108` | Object disconnected from clients — OPC server dropped us (crash/restart) |
| `0x80070005` | Access denied — DCOM security misconfiguration |
| `0x80040154` | Class not registered — wrong ProgID, or wrapper/server registered under a different bitness |
| `0xC0040007` | Invalid item ID — tag name wrong in config (surfaced by `probe`) |

**Bitness warning:** the OPC DA Automation Wrapper (`OPCAutomation.dll`) and the Graybox
wrapper are typically **32-bit in-proc DLLs** — the Python interpreter must match
(32-bit). This is decided and verified in Phase 0 of the implementation plan.

## Safety

- The tool never writes to the OPC server — there is no write path in the code.
- It uses one OPC group and a static, small tag list at a modest poll rate (default 1 s)
  so it does not load the system it is measuring.

## Development setup

The OPC DA wrappers (`OPCAutomation.dll`, `GBDaCtl.dll`) are **32-bit in-proc COM DLLs**,
so the Python interpreter **must be 32-bit** (SPECIFICATION §11 / Phase 0 decision).

```powershell
# create the 32-bit venv (requires the 32-bit Python installed via the py launcher)
& "C:\Users\<user>\AppData\Local\Programs\Python\Python314-32\python.exe" -m venv .venv
& ".\.venv\Scripts\python.exe" -m pip install -e . -r requirements.dev.txt

# verify bitness — must print 32
& ".\.venv\Scripts\python.exe" -c "import struct; print(struct.calcsize('P') * 8)"

# run the (COM-free) unit + fake-integration tests
& ".\.venv\Scripts\python.exe" -m pytest
```

Dependency files: `requirements.txt` (production) and
`requirements.dev.txt` (adds pytest + pyinstaller, chains production via
`-r requirements.txt`). `pip install -e ".[dev]"` works too —
`pyproject.toml` stays the canonical package source.

Notes:
- Dependencies are chosen to be installable without a C compiler: **ruamel.yaml**
  (pure Python) is used instead of PyYAML, which publishes no 32-bit CPython 3.14
  wheels and whose sdist requires MSVC.
- COM-dependent modules (Automation Wrapper, Event Log, service) are import-guarded;
  the unit test suite runs on any OS via the in-process fake client
  (`opcda_diagnozer.opc.fake`).
- 32-bit CPython can be installed with the Python Install Manager
  (`py install 3.14 -a x86`) or the installer from python.org ("Download Windows
  installer (32-bit)").

## Building the Windows executable

`scripts/build_exe.ps1` packages the CLI as a standalone **32-bit** console exe
with [PyInstaller](https://pyinstaller.org) — no Python installation needed on
the target host:

```powershell
& ".\.venv\Scripts\python.exe" -m pip install pyinstaller   # once
.\scripts\build_exe.ps1                                     # dist\opcda-diagnozer\
.\scripts\build_exe.ps1 -Onefile                            # single exe alternative
.\scripts\build_exe.ps1 -Clean                              # fresh rebuild
```

The script refuses to run from a 64-bit interpreter (the exe must load the
32-bit OPC DA wrapper DLLs, SPECIFICATION §11), re-verifies the produced PE
really is 32-bit, embeds the project version in the exe properties, declares
the hidden imports the code loads lazily (pywin32 COM, browse modules, the
service host + its SCM modules), and smoke-tests the artifact (`--version`,
`check-config`, `list-servers --source registry`) after building. It also
installs PyInstaller into the venv automatically if it is missing.

If your policy blocks the script, run it with
`powershell -NoProfile -ExecutionPolicy Bypass -File scripts\build_exe.ps1`.

Deployment notes:
- The **default config path** of a frozen exe is `config.yaml` **next to the
  exe** — not the working directory. A Windows service starts with
  `CWD = C:\Windows\System32`, so the relative default would silently point
  there; the exe-dir default keeps the deployment layout simple
  (`C:\Program Files\OPCDA-Diagnozer\{opcda-diagnozer.exe, config.yaml}`).
- The exe is a console application that hosts every CLI verb **and the
  Windows service**: `install` registers this same exe with the SCM, and
  when the SCM starts it the process recognizes the fixed service-start
  argv shape (`exe service`, CWD = System32) and dispatches to the service
  host. No `pythonservice.exe`, no Python install on the diag host.
- The Automation Wrapper DLL (`OPCAutomation.dll`) is **not** bundled — it is
  a machine-installed COM component (registered in the registry), exactly like
  the OPC DA servers the tool connects to. Only Python code and its
  dependencies are frozen.
- `onedir` is the default (faster startup, AV-friendlier); `--onefile` trades
  startup time for a single file. `--clean` removes `build/` + `dist/` first.

## Documentation

- [Specification](docs/SPECIFICATION.md) — detection rules, event model, operational behavior, acceptance criteria
- [Implementation plan](docs/IMPLEMENTATION_PLAN.md) — phases, milestones, testing, risks
- [TODO](TODO.md) — implementation status: what's done, what's next
- [Changelog](CHANGELOG.md) — notable changes per release

## License

TBD