Compare commits
6
Commits
76b1cc5ccf
...
dev
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
eee8181166 | ||
|
|
e5675ec559 | ||
|
|
ca0bb0d048 | ||
|
|
1478db73a9 | ||
|
|
406fdc61ac | ||
|
|
a1af8d68c9 |
+163
@@ -0,0 +1,163 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
All notable changes to the OPC DA Diagnozer are documented here. The format
|
||||||
|
follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); versions
|
||||||
|
bump per [SemVer](https://semver.org/). Dates are YYYY-MM-DD.
|
||||||
|
|
||||||
|
## [Unreleased]
|
||||||
|
|
||||||
|
### Added — Phase 3 Windows service
|
||||||
|
- Service host (`service.py`): pywin32 `ServiceFramework` subclass runs the
|
||||||
|
monitoring loop on its own monitor thread; clean SCM stop semantics —
|
||||||
|
`STOP_PENDING` reported immediately, 30 s join-with-timeout, a hung COM
|
||||||
|
poll is **abandoned** (thread never joined, daemon; spec §10.4); service
|
||||||
|
start/stop recorded in the Windows Event Log (`PYS_SERVICE_STARTED/
|
||||||
|
STOPPED`). Verified in-process via the pywin32 debug host against the
|
||||||
|
fake client: start → poll → stop in 0.1 s.
|
||||||
|
- CLI verbs `install` / `uninstall` / `start` / `stop` / `status`:
|
||||||
|
- `install` registers the exe **itself** as the service binary (no
|
||||||
|
`pythonservice.exe`, no Python install on the diag host) with
|
||||||
|
auto-start, validates the config **before** touching the SCM, records
|
||||||
|
the config path in the service registry Parameters (startup never
|
||||||
|
depends on the service CWD = System32), configures SCM crash recovery
|
||||||
|
(restart on first+second failure, 5 s delay, 24 h counter reset — the
|
||||||
|
`ChangeServiceConfig2` failure-actions dict shape verified against
|
||||||
|
pywin32's SWIG parser), and registers the Windows Event Log source with
|
||||||
|
its message file (completing the eventlog reporter's deferred step).
|
||||||
|
- Errors map to actionable one-liners: elevation needed, service not
|
||||||
|
installed, service not running, marked-for-delete — plus the raw
|
||||||
|
Windows error code and `net helpmsg` pointer for anything else.
|
||||||
|
- `status` prints the SCM state without needing admin.
|
||||||
|
- SCM dispatch: when the SCM starts the exe (fixed `service` argv shape +
|
||||||
|
CWD = System32, exactly what install records in the ImagePath), `main`
|
||||||
|
hands the process to `servicemanager.Initialize` +
|
||||||
|
`PrepareToHostSingle` + `StartServiceCtrlDispatcher`; any other
|
||||||
|
invocation stays a CLI verb.
|
||||||
|
- PyInstaller build script: hidden imports added for the service host's
|
||||||
|
lazily-imported SCM modules (`win32service`, `win32serviceutil`,
|
||||||
|
`servicemanager`, `win32evtlogutil`).
|
||||||
|
|
||||||
|
### Added — Phase 2 reporters
|
||||||
|
- Rotating JSONL evidence log: one JSON event per line, config-digest header
|
||||||
|
for provenance, size rotation with backups retained.
|
||||||
|
- Plain-text anomaly log (spec §9.1): `anomalies-YYYYMMDD.log`, one line per
|
||||||
|
detected anomaly in `timestamp SEVERITY KIND message` order with clear-text
|
||||||
|
statuses; recoveries included so the file reads as an incident narrative.
|
||||||
|
- Windows Event Log sink: severity-mapped (INFO/WARN/CRITICAL →
|
||||||
|
Information/Warning/Error), WARN+ by default, delivery verified live.
|
||||||
|
- Optional HTTP webhook (`report.http`): bounded queue + daemon sender thread,
|
||||||
|
per-event POST with bearer auth; failures dropped and counted — never blocks
|
||||||
|
the poll loop.
|
||||||
|
- Reporter fan-out dispatcher: every sink isolated (one failing sink never
|
||||||
|
kills the others or polling); the `run` verb wires all configured sinks.
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
- Windows Event Log reporter initially lost every event silently:
|
||||||
|
`win32evtlog.ReportEvent` takes **positional arguments only** (keyword calls
|
||||||
|
raise `TypeError`, swallowed by the sink's own isolation) — now uses a
|
||||||
|
`RegisterEventSource` handle with positional calls (live-verified).
|
||||||
|
- HTTP webhook in injected-`post` mode raised into the poll loop on a failing
|
||||||
|
POST; both modes now count-and-drop.
|
||||||
|
|
||||||
|
### Added — Phase 1 core engine (the continuous monitoring loop)
|
||||||
|
- `run` verb: the foreground monitoring loop — coordinator + dedicated COM
|
||||||
|
worker thread (the client is born on the worker; COM is per-thread),
|
||||||
|
1-second default polling of configured tags **and** device indicators,
|
||||||
|
heartbeat (D15) every `heartbeat_s`, clean Ctrl+C stop.
|
||||||
|
- Detection engine, all pure and fake-client tested: latency spike / sustained
|
||||||
|
streak / rolling p95 (D4–D6), per-tag quality transitions, persistence and
|
||||||
|
flapping (D8–D11), stale timestamp & frozen value (D12–D13), mass-quality
|
||||||
|
degradation with latch (D14), device-link indicators (D16, edge-level
|
||||||
|
DOWN/UP against configurable `ok_values`).
|
||||||
|
- Hang watchdog (spec §10.4): polls exceeding `timeout_ms` raise the D7
|
||||||
|
soft-hang event and keep the thread; silence beyond `hard_hang_s` abandons
|
||||||
|
the stuck COM thread (never joined — dead RPC channel) and rebuilds.
|
||||||
|
- Connection state machine with exponential backoff (1 s → 60 s, reset on
|
||||||
|
success) and reconnect accounting (D1/D2 with outage duration, D3 flapping).
|
||||||
|
- Fake-client scenario engine: scripted polls and whole-call failures power
|
||||||
|
the entire poller/detector test suite without COM.
|
||||||
|
- Clear-text status rendering (`status_text`): `BAD / COMM FAILURE (0x18)`
|
||||||
|
everywhere events are shown.
|
||||||
|
|
||||||
|
### Added — server discovery & browsing
|
||||||
|
- `list-servers` verb: enumerates OPC DA servers locally or on a DCOM host via
|
||||||
|
the Automation Wrapper (`GetOPCServers`), with a no-COM registry-scan fallback
|
||||||
|
(`--source registry|automation|auto`), JSON output, and DA-version metadata.
|
||||||
|
- Browsing of the connected server's address space, unified in the
|
||||||
|
`list-items [all|branches|tags]` verb: `all` prints the full structure tree
|
||||||
|
(branches with `/`, tags with `[ItemID: ...]`), `branches` the structure-only
|
||||||
|
flat list, `tags` the flat ItemID list for config `tags:`. `--branch`
|
||||||
|
(dotted path or repeated segments), `--depth N` (1 default, 0 = whole
|
||||||
|
subtree), `--json`. Replaces the earlier split `list-devices`/`list-tags`.
|
||||||
|
- `browse.max_depth` config key (default 32) as the address-space walk guard,
|
||||||
|
validated by `check-config`; cycle-safe tree walk in `opc/browse.py`.
|
||||||
|
- `SpaceNode` tree data model (branch/leaf) replacing the earlier
|
||||||
|
device-centric `DeviceInfo`.
|
||||||
|
|
||||||
|
### Added — reading & CLI
|
||||||
|
- `read` verb: one-shot read of explicit tags (or the config's tags) with a
|
||||||
|
table or `--json`, `--source device|cache`, and a non-zero exit when any tag
|
||||||
|
doesn't read GOOD — scriptable as a health check.
|
||||||
|
- `probe` verb: connect once and read every configured tag once.
|
||||||
|
- `--server` / `--host` overrides on every connecting verb; with `--server` no
|
||||||
|
config file is needed at all.
|
||||||
|
- Frozen-exe default config path: `config.yaml` next to the exe (a Windows
|
||||||
|
service starts with `CWD = System32`), not the working directory.
|
||||||
|
|
||||||
|
### Added — monitoring design (D16) & config
|
||||||
|
- `devices:` config section with `DeviceEntry(name, tag, ok_values)`: D16
|
||||||
|
device-link indicators. The normal (connected) reading is configurable per
|
||||||
|
indicator (`ok_values`, default `[true]`) — inverted fault flags
|
||||||
|
`[false, 0]` and string states `["RUN"]` are first-class. Strict validation:
|
||||||
|
duplicate indicators and indicator/tag-list overlap are rejected;
|
||||||
|
`check-config` prints the indicators.
|
||||||
|
- Detection rules D16 (`DEVICE_LINK_DOWN`/`DEVICE_LINK_UP`) and the plain-text
|
||||||
|
anomaly log reporter (`report/textlog.py` task) specified in the docs
|
||||||
|
(SPECIFICATION §6.1, §9.1; IMPLEMENTATION_PLAN Phases 1–2) — implementation
|
||||||
|
pending, see TODO.md.
|
||||||
|
- Clear-text status rule (spec §7): every reporter writes `NAME (0xNN)` —
|
||||||
|
e.g. `BAD / communication failure (0x18)`.
|
||||||
|
|
||||||
|
### Added — packaging
|
||||||
|
- `scripts/build_exe.ps1`: PyInstaller build of the 32-bit console exe with
|
||||||
|
hidden imports for the deferred COM modules (incl. `win32timezone` — see
|
||||||
|
Fixed), embedded version metadata, a PE-bitness check, and smoke tests
|
||||||
|
including a **frozen-read canary** against a live server.
|
||||||
|
- `--version` and this CHANGELOG.md / TODO.md.
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
- COM lifecycle: all COM references (browsers, groups, items) are released
|
||||||
|
**inside** the apartment before `CoUninitialize`; retained exception
|
||||||
|
tracebacks are cleared in `as_com_error` — both caused
|
||||||
|
"Win32 exception occurred releasing IUnknown" noise and hard crashes at exit.
|
||||||
|
- Frozen exe: every tag read failed with
|
||||||
|
`SystemError: ... InvokeTypes ... exception set` because pywin32 lazily
|
||||||
|
imports `win32timezone` during VARIANT-DATE conversion and PyInstaller never
|
||||||
|
bundles it — now a hidden import, with the build canary guarding it.
|
||||||
|
- Reads: per-tag failures now carry the underlying exception's own type and
|
||||||
|
text (`_per_tag_error`) instead of a bare "failed"; a degenerate device
|
||||||
|
timestamp (never-updated alias, DST math) can no longer fail a read — the
|
||||||
|
timestamp is metadata and is dropped to `None`.
|
||||||
|
- Matrikon browse quirk: servers may return an empty ItemID for special leaves
|
||||||
|
(alias roots); the adapter falls back to the bare browse name.
|
||||||
|
- HRESULT labels: `OPC_E_UNKNOWNITEMID` (0xC0040007) and
|
||||||
|
`OPC_E_INVALIDITEMID` (0xC0040008) were swapped; now corrected everywhere.
|
||||||
|
|
||||||
|
### Verified
|
||||||
|
- All CLI verbs live-tested against Matrikon OPC Simulation Server
|
||||||
|
(`Matrikon.OPC.Simulation.1`): discovery, browse (all kinds/depths), reads
|
||||||
|
(incl. alias tags after server-side alias repair), config-driven
|
||||||
|
`max_depth`, and the frozen exe (onedir + onefile) including the read canary.
|
||||||
|
- Test suite: 108 tests, all passing (pure logic + fake-client driven, runs on
|
||||||
|
any OS; COM paths are live-smoke-tested on dev boxes only).
|
||||||
|
|
||||||
|
## [0.1.0] - 2026-10-07
|
||||||
|
|
||||||
|
### Added
|
||||||
|
- Initial project skeleton: typed YAML config with strict actionable
|
||||||
|
validation (`check-config`), OPC DA quality decode tables (spec §7), event
|
||||||
|
model skeletons (`DiagEvent`, D1–D15 `EventKind`s), client/protocol
|
||||||
|
abstractions, and the fake in-process client powering cross-OS tests.
|
||||||
|
- Phase 0 decisions recorded: 32-bit interpreter mandatory (Automation Wrapper
|
||||||
|
is a 32-bit in-proc DLL), ruamel.yaml over PyYAML (no 32-bit cp314 wheels),
|
||||||
|
COM imports import-guarded for cross-OS test runs.
|
||||||
@@ -10,9 +10,13 @@ disconnects, hung RPC calls, tags marked BAD/UNCERTAIN, stale timestamps, frozen
|
|||||||
and flapping quality — as structured events (JSONL logs, Windows Event Log, optional
|
and flapping quality — as structured events (JSONL logs, Windows Event Log, optional
|
||||||
HTTP webhook).
|
HTTP webhook).
|
||||||
|
|
||||||
> **Status:** planning. Read the
|
> **Status:** in active development — discovery, browsing (`list-items`), one-shot
|
||||||
> [specification](docs/SPECIFICATION.md) and the
|
> reads (`read`/`probe`), the continuous monitoring loop (`run`), all reporters,
|
||||||
> [implementation plan](docs/IMPLEMENTATION_PLAN.md) first.
|
> the exe build and the Windows service host are **live and field-verified**
|
||||||
|
> (service: in-process debug host verified; the elevated SCM pass is the remaining
|
||||||
|
> live check). See the [TODO](TODO.md) for current status and the
|
||||||
|
> [implementation plan](docs/IMPLEMENTATION_PLAN.md) for the phases; read the
|
||||||
|
> [specification](docs/SPECIFICATION.md) for detection rules and behavior.
|
||||||
|
|
||||||
## Why
|
## Why
|
||||||
|
|
||||||
@@ -29,12 +33,16 @@ answers the triage question *where is the problem?* with evidence:
|
|||||||
|
|
||||||
See the full interpretation matrix in the [specification](docs/SPECIFICATION.md).
|
See the full interpretation matrix in the [specification](docs/SPECIFICATION.md).
|
||||||
|
|
||||||
## Planned features
|
## Features
|
||||||
|
|
||||||
- Windows service (auto-start, SCM crash recovery) with a foreground `run` mode for testing
|
- Windows service (auto-start, SCM crash recovery) with a foreground `run` mode for testing — live
|
||||||
- Local or remote (DCOM) connections to any OPC DA server
|
- Local or remote (DCOM) connections to any OPC DA server
|
||||||
- Server browsing: `list-servers` (discovery) and `list-items`
|
- Server browsing: `list-servers` (discovery) and `list-items`
|
||||||
(address-space browse: full tree / branches / tags) — live now
|
(address-space browse: full tree / branches / tags) — live now
|
||||||
|
- **Continuous monitoring** (`run`): DCOM disconnect/reconnect accounting,
|
||||||
|
latency spike/streak/trend, hang watchdog (soft flag + thread rebuild),
|
||||||
|
BAD/UNCERTAIN transitions with persistence & flapping, stale/frozen tags,
|
||||||
|
mass-degradation alarm, per-device link indicators (D16) — live now
|
||||||
- Latency monitoring: spikes, streaks, and p95 trends against configurable thresholds
|
- Latency monitoring: spikes, streaks, and p95 trends against configurable thresholds
|
||||||
- Hang watchdog: flags RPC calls that never return — the silent killer
|
- Hang watchdog: flags RPC calls that never return — the silent killer
|
||||||
- Quality monitoring: BAD/UNCERTAIN transitions with decoded subcodes, persistent-bad
|
- Quality monitoring: BAD/UNCERTAIN transitions with decoded subcodes, persistent-bad
|
||||||
@@ -73,11 +81,19 @@ opcda-diagnozer check-config
|
|||||||
# connect once and read every configured tag, print results
|
# connect once and read every configured tag, print results
|
||||||
opcda-diagnozer probe
|
opcda-diagnozer probe
|
||||||
|
|
||||||
# foreground run (console) — best for first deployment validation
|
# foreground run (console) — continuous monitoring; Ctrl+C stops
|
||||||
opcda-diagnozer run
|
opcda-diagnozer run
|
||||||
|
|
||||||
# install as Windows service (dedicated account required for DCOM auth)
|
# install as a Windows service (auto-start + SCM crash recovery; admin shell)
|
||||||
opcda-diagnozer install --account DOMAIN\svc-opcdiag
|
# LocalSystem default; a dedicated account is required for remote DCOM auth
|
||||||
|
opcda-diagnozer install
|
||||||
|
opcda-diagnozer install --account DOMAIN\svc-opcdiag --password ****
|
||||||
|
|
||||||
|
# service lifecycle + state
|
||||||
|
opcda-diagnozer start
|
||||||
|
opcda-diagnozer stop
|
||||||
|
opcda-diagnozer status
|
||||||
|
opcda-diagnozer uninstall
|
||||||
```
|
```
|
||||||
`list-servers` uses two discovery mechanisms (`--source`): the OPC DA Automation
|
`list-servers` uses two discovery mechanisms (`--source`): the OPC DA Automation
|
||||||
Wrapper's `GetOPCServers` over COM (works remotely via DCOM; needs Launch+Activation
|
Wrapper's `GetOPCServers` over COM (works remotely via DCOM; needs Launch+Activation
|
||||||
@@ -110,6 +126,48 @@ exist (and `probe`/`read` read the config's tag list unless tags are passed).
|
|||||||
a non-GOOD or failing tag makes the exit code non-zero, so it doubles as a
|
a non-GOOD or failing tag makes the exit code non-zero, so it doubles as a
|
||||||
scriptable health check. `--source cache` reads the server cache instead of
|
scriptable health check. `--source cache` reads the server cache instead of
|
||||||
forcing a device round-trip.
|
forcing a device round-trip.
|
||||||
|
|
||||||
|
## Windows service
|
||||||
|
|
||||||
|
The monitor runs unattended as a **Windows service** (auto-start, SCM crash
|
||||||
|
recovery) — `opcda-diagnozer install` registers the exe itself as the service
|
||||||
|
binary (no `pythonservice.exe`, no Python install on the diag host):
|
||||||
|
|
||||||
|
- **Auto-start** at boot and **restart on failure**: the SCM restarts the
|
||||||
|
service process 5 s after the first and second failure (the failure counter
|
||||||
|
resets after 24 h of clean running) — `sc qfailure OPCDA-Diagnozer` shows it.
|
||||||
|
- **Config path** is recorded at install time (`install --config ...`); the
|
||||||
|
service never depends on its working directory (the SCM starts services
|
||||||
|
with CWD = System32). The default is `config.yaml` next to the exe.
|
||||||
|
- **Clean stop**: a stop request parks the service in `STOP_PENDING` and
|
||||||
|
gives the poll loop 30 s to finish; a COM poll stuck in a dead RPC longer
|
||||||
|
than that is abandoned (the thread is never joined — spec §10.4).
|
||||||
|
- **Event Log** records service start/stop and every configured reporter
|
||||||
|
event; `install` also registers the event source so messages render
|
||||||
|
correctly (needs the admin shell anyway).
|
||||||
|
- **Status** works without admin: `opcda-diagnozer status` prints the SCM
|
||||||
|
state (`STOPPED` / `RUNNING` / …).
|
||||||
|
|
||||||
|
Service **account** (spec §11): `LocalSystem` (the default) works for OPC
|
||||||
|
DA servers on the same machine. Remote DCOM connections need a **dedicated
|
||||||
|
domain account** that has Launch+Activation and Access permissions on the
|
||||||
|
OPC server host (`dcomcnfg` — see the checklist above) and can authenticate
|
||||||
|
across the network; install with `--account DOMAIN\USER --password ****`
|
||||||
|
(the password is only used once, to register the account with the SCM).
|
||||||
|
|
||||||
|
**Deployment layout** (frozen exe):
|
||||||
|
|
||||||
|
```
|
||||||
|
C:\Program Files\OPCDA-Diagnozer\
|
||||||
|
opcda-diagnozer.exe # the console exe: every CLI verb AND the service host
|
||||||
|
config.yaml # the deployed config (install records its path)
|
||||||
|
C:\ProgramData\OPCDA-Diagnozer\
|
||||||
|
logs\ # events.jsonl + anomalies-YYYYMMDD.log (report.dir)
|
||||||
|
```
|
||||||
|
|
||||||
|
Log directories are created on demand; `%ProgramData%` is the default because
|
||||||
|
the service account must be able to write the evidence logs.
|
||||||
|
|
||||||
### Configuration example (`config.example.yaml`)
|
### Configuration example (`config.example.yaml`)
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
@@ -126,6 +184,15 @@ tags:
|
|||||||
- { name: "Channel1.Device1.Counter", expect_change: true }
|
- { name: "Channel1.Device1.Counter", expect_change: true }
|
||||||
- { name: "Channel1.Device1.Level" }
|
- { name: "Channel1.Device1.Level" }
|
||||||
|
|
||||||
|
# D16 device-link indicators: tags saying whether the OPC server can reach
|
||||||
|
# the device. DOWN (DEVICE_LINK_DOWN, CRITICAL) when the tag reads non-GOOD,
|
||||||
|
# errors, or its GOOD value is NOT in ok_values. ok_values defaults to [true]
|
||||||
|
# (boolean connected flags); inverted fault flags -> [false, 0]; string
|
||||||
|
# states -> ["RUN"]. Do NOT also list these under tags:.
|
||||||
|
devices:
|
||||||
|
- { name: "Conveyor 3", tag: "Channel1.Device3.Connected" }
|
||||||
|
- { name: "Pump A", tag: "Channel1.PumpA.Fault", ok_values: [false, 0] }
|
||||||
|
|
||||||
latency:
|
latency:
|
||||||
warn_ms: 1000
|
warn_ms: 1000
|
||||||
critical_ms: 5000
|
critical_ms: 5000
|
||||||
@@ -146,6 +213,9 @@ browse:
|
|||||||
report:
|
report:
|
||||||
heartbeat_s: 300
|
heartbeat_s: 300
|
||||||
jsonl: { dir: "C:/ProgramData/OPCDA-Diagnozer/logs", max_mb: 50, backups: 10 }
|
jsonl: { dir: "C:/ProgramData/OPCDA-Diagnozer/logs", max_mb: 50, backups: 10 }
|
||||||
|
textlog: { dir: "C:/ProgramData/OPCDA-Diagnozer/logs", max_mb: 20, backups: 10 }
|
||||||
|
eventlog: { source: "OPCDA-Diagnozer", min_severity: WARN }
|
||||||
|
#http: { url: "https://alerts.example.com/api", bearer_token: "...", timeout_s: 5 }
|
||||||
eventlog: { source: "OPCDA-Diagnozer", min_severity: WARN }
|
eventlog: { source: "OPCDA-Diagnozer", min_severity: WARN }
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -192,7 +262,7 @@ so the Python interpreter **must be 32-bit** (SPECIFICATION §11 / Phase 0 decis
|
|||||||
```powershell
|
```powershell
|
||||||
# create the 32-bit venv (requires the 32-bit Python installed via the py launcher)
|
# create the 32-bit venv (requires the 32-bit Python installed via the py launcher)
|
||||||
& "C:\Users\<user>\AppData\Local\Programs\Python\Python314-32\python.exe" -m venv .venv
|
& "C:\Users\<user>\AppData\Local\Programs\Python\Python314-32\python.exe" -m venv .venv
|
||||||
& ".\.venv\Scripts\python.exe" -m pip install -e ".[dev]"
|
& ".\.venv\Scripts\python.exe" -m pip install -e . -r requirements.dev.txt
|
||||||
|
|
||||||
# verify bitness — must print 32
|
# verify bitness — must print 32
|
||||||
& ".\.venv\Scripts\python.exe" -c "import struct; print(struct.calcsize('P') * 8)"
|
& ".\.venv\Scripts\python.exe" -c "import struct; print(struct.calcsize('P') * 8)"
|
||||||
@@ -201,6 +271,11 @@ so the Python interpreter **must be 32-bit** (SPECIFICATION §11 / Phase 0 decis
|
|||||||
& ".\.venv\Scripts\python.exe" -m pytest
|
& ".\.venv\Scripts\python.exe" -m pytest
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Dependency files: `requirements.txt` (production) and
|
||||||
|
`requirements.dev.txt` (adds pytest + pyinstaller, chains production via
|
||||||
|
`-r requirements.txt`). `pip install -e ".[dev]"` works too —
|
||||||
|
`pyproject.toml` stays the canonical package source.
|
||||||
|
|
||||||
Notes:
|
Notes:
|
||||||
- Dependencies are chosen to be installable without a C compiler: **ruamel.yaml**
|
- Dependencies are chosen to be installable without a C compiler: **ruamel.yaml**
|
||||||
(pure Python) is used instead of PyYAML, which publishes no 32-bit CPython 3.14
|
(pure Python) is used instead of PyYAML, which publishes no 32-bit CPython 3.14
|
||||||
@@ -229,7 +304,7 @@ The script refuses to run from a 64-bit interpreter (the exe must load the
|
|||||||
32-bit OPC DA wrapper DLLs, SPECIFICATION §11), re-verifies the produced PE
|
32-bit OPC DA wrapper DLLs, SPECIFICATION §11), re-verifies the produced PE
|
||||||
really is 32-bit, embeds the project version in the exe properties, declares
|
really is 32-bit, embeds the project version in the exe properties, declares
|
||||||
the hidden imports the code loads lazily (pywin32 COM, browse modules, the
|
the hidden imports the code loads lazily (pywin32 COM, browse modules, the
|
||||||
Phase 3 service host), and smoke-tests the artifact (`--version`,
|
service host + its SCM modules), and smoke-tests the artifact (`--version`,
|
||||||
`check-config`, `list-servers --source registry`) after building. It also
|
`check-config`, `list-servers --source registry`) after building. It also
|
||||||
installs PyInstaller into the venv automatically if it is missing.
|
installs PyInstaller into the venv automatically if it is missing.
|
||||||
|
|
||||||
@@ -242,10 +317,11 @@ Deployment notes:
|
|||||||
`CWD = C:\Windows\System32`, so the relative default would silently point
|
`CWD = C:\Windows\System32`, so the relative default would silently point
|
||||||
there; the exe-dir default keeps the deployment layout simple
|
there; the exe-dir default keeps the deployment layout simple
|
||||||
(`C:\Program Files\OPCDA-Diagnozer\{opcda-diagnozer.exe, config.yaml}`).
|
(`C:\Program Files\OPCDA-Diagnozer\{opcda-diagnozer.exe, config.yaml}`).
|
||||||
- The exe is a console application that hosts every CLI verb, including the
|
- The exe is a console application that hosts every CLI verb **and the
|
||||||
future `run` / `probe` / service verbs. The Phase 3 Windows service
|
Windows service**: `install` registers this same exe with the SCM, and
|
||||||
(`install` verb) will point `sc.exe` at this same exe with the service-host
|
when the SCM starts it the process recognizes the fixed service-start
|
||||||
entry point.
|
argv shape (`exe service`, CWD = System32) and dispatches to the service
|
||||||
|
host. No `pythonservice.exe`, no Python install on the diag host.
|
||||||
- The Automation Wrapper DLL (`OPCAutomation.dll`) is **not** bundled — it is
|
- The Automation Wrapper DLL (`OPCAutomation.dll`) is **not** bundled — it is
|
||||||
a machine-installed COM component (registered in the registry), exactly like
|
a machine-installed COM component (registered in the registry), exactly like
|
||||||
the OPC DA servers the tool connects to. Only Python code and its
|
the OPC DA servers the tool connects to. Only Python code and its
|
||||||
@@ -257,6 +333,8 @@ Deployment notes:
|
|||||||
|
|
||||||
- [Specification](docs/SPECIFICATION.md) — detection rules, event model, operational behavior, acceptance criteria
|
- [Specification](docs/SPECIFICATION.md) — detection rules, event model, operational behavior, acceptance criteria
|
||||||
- [Implementation plan](docs/IMPLEMENTATION_PLAN.md) — phases, milestones, testing, risks
|
- [Implementation plan](docs/IMPLEMENTATION_PLAN.md) — phases, milestones, testing, risks
|
||||||
|
- [TODO](TODO.md) — implementation status: what's done, what's next
|
||||||
|
- [Changelog](CHANGELOG.md) — notable changes per release
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,124 @@
|
|||||||
|
# TODO — implementation status
|
||||||
|
|
||||||
|
Companion to `docs/IMPLEMENTATION_PLAN.md` (the authoritative plan) and
|
||||||
|
`CHANGELOG.md` (what shipped). One row per plan task; updated as work lands.
|
||||||
|
Legend: **[done]** complete & verified · **[partial]** some of it landed ·
|
||||||
|
**[todo]** not started · **[skip]** dropped by decision.
|
||||||
|
|
||||||
|
## Phase 0 — Feasibility & environment **[done]**
|
||||||
|
|
||||||
|
- [done] Wrapper backend decision: OPC DA Automation Wrapper (`OPC.Automation`)
|
||||||
|
→ 32-bit interpreter mandatory (spec §11).
|
||||||
|
- [done] Test server: Matrikon OPC Simulation Server, local ProgID
|
||||||
|
`Matrikon.OPC.Simulation.1`; baseline read RTT + wrapper error shapes
|
||||||
|
recorded (repo memory).
|
||||||
|
- [done] Dependencies chosen for a 32-bit, no-compiler install:
|
||||||
|
ruamel.yaml (PyYAML has no cp314 win32 wheels), pywin32.
|
||||||
|
|
||||||
|
## Phase 1 — Core engine **[done]**
|
||||||
|
|
||||||
|
- [done] 1. `config.py` — typed YAML config + strict validation; **D16
|
||||||
|
`devices:` indicators with configurable `ok_values`**; `check-config`
|
||||||
|
prints them.
|
||||||
|
- [done] 2. `quality.py` — decode tables (spec §7) +
|
||||||
|
`status_text(raw) -> "NAME / SUBCODE (0xNN)"` clear-text helper.
|
||||||
|
- [done] 3. `opc/base.py` — client/browser protocols, `ReadResult`,
|
||||||
|
`OpcError` with decoded HRESULT.
|
||||||
|
- [done] 4. `opc/automation.py` — connect / browse / read (per-item
|
||||||
|
`Item.Read`; `SyncRead` is unmarshallable from pywin32 — verified), per-tag
|
||||||
|
errors, safe COM teardown.
|
||||||
|
- [skip] Single persistent group + static items (poller creates its own
|
||||||
|
per client; the throwaway-group read pattern is verified).
|
||||||
|
- [done] 5. `monitor/state.py` — connection state machine, exponential
|
||||||
|
backoff (1 s → 60 s cap, reset on success), reconnect accounting, D3
|
||||||
|
flapping window (pure, clock-fed).
|
||||||
|
- [done] 6. `monitor/detectors.py` — D4–D16 pure detectors + rolling window:
|
||||||
|
latency spike/streak/p95 (D4–D6), quality transitions/persistence/flapping
|
||||||
|
(D8–D11), stale/frozen (D12–D13), mass-bad latch (D14), **D16
|
||||||
|
device-link** (`ok_values` edge-level DOWN/UP).
|
||||||
|
- [done] 7. `monitor/poller.py` — coordinator + dedicated COM worker thread
|
||||||
|
(client born on the worker), D7 soft-hang flag, hard-hang watchdog
|
||||||
|
(abandon + rebuild), backoff between sessions.
|
||||||
|
- [open] Poll source default (CACHE vs DEVICE) — needs a spec decision:
|
||||||
|
Matrikon shows BAD qualities on some tags only via CACHE.
|
||||||
|
- [done] 8. `run` verb — foreground monitoring loop, console reporter with
|
||||||
|
clear-text status, heartbeat (D15), Ctrl+C stop (second Ctrl+C aborts).
|
||||||
|
- [done] 9. `opc/fake.py` — scenario engine: scripted polls (exact results /
|
||||||
|
`"good"` shorthand) and `PollScriptError` whole-call failures; hang clients
|
||||||
|
are plain sleeping fakes in tests.
|
||||||
|
- [open] D12 stale events re-emit every poll while a tag stays stale — the
|
||||||
|
spec defines the condition per-poll; ops may want debounce/latch (decide
|
||||||
|
before the 24 h soak; live sim output showed the spam clearly).
|
||||||
|
|
||||||
|
## Phase 2 — Reporters **[done]**
|
||||||
|
|
||||||
|
- [done] 1. `report/jsonl.py` — rotating JSONL (size + backups), config-digest
|
||||||
|
header line, single-line atomic-ish appends. Live-verified.
|
||||||
|
- [done] 2. `report/textlog.py` — plain-text anomaly log (spec §9.1):
|
||||||
|
`anomalies-YYYYMMDD.log`, one line per detected anomaly, clear-text status
|
||||||
|
`NAME / SUB (0xNN)`, recoveries logged (incident narrative), per-day files,
|
||||||
|
shared rotation. Live-verified.
|
||||||
|
- [done] 3. `report/eventlog.py` — Windows Event Log (WARN+), severity-mapped
|
||||||
|
via a RegisterEventSource handle + positional ReportEvent (keyword args
|
||||||
|
raise TypeError — found live). Source message-file registration lands with
|
||||||
|
`install` (Phase 3). Live-verified delivery.
|
||||||
|
- [done] 4. `report/http.py` — optional webhook (`report.http` config),
|
||||||
|
bounded queue + daemon sender, never blocks the poll loop; failures counted
|
||||||
|
and dropped, injectable `post` for tests.
|
||||||
|
- [done] 5. Heartbeat assembly (D15, Phase 1) + reporter fan-out dispatcher
|
||||||
|
(`report/base.py`): per-sink failure isolation, best-effort close. The
|
||||||
|
`run` verb wires console + JSONL + textlog + eventlog (+http when set).
|
||||||
|
|
||||||
|
## Phase 3 — Windows service **[partial]**
|
||||||
|
|
||||||
|
- [done] 1. `service.py` — `ServiceFramework` host (monitor thread +
|
||||||
|
poller, `PYS_SERVICE_STARTED/STOPPED` lifecycle entries); clean stop
|
||||||
|
(`STOP_PENDING`, 30 s join-with-timeout, abandon hung COM thread,
|
||||||
|
never joined — verified in-process via the pywin32 debug host against
|
||||||
|
the fake client: start → poll → stop in 0.1 s).
|
||||||
|
- [done] 2. CLI verbs `install`/`uninstall`/`start`/`stop`/`status` —
|
||||||
|
actionable error mapping (elevation / not-installed / not-running
|
||||||
|
hints), install validates the config BEFORE touching the SCM, and
|
||||||
|
prints the recorded config path + auto-start/recovery summary.
|
||||||
|
- [done] 3. SCM recovery (`sc failure` semantics) wired at install:
|
||||||
|
restart on first+second failure (5 s delay, 24 h counter reset) via
|
||||||
|
`ChangeServiceConfig2(SERVICE_CONFIG_FAILURE_ACTIONS)` — dict shape
|
||||||
|
verified against pywin32's SWIG parser; auto-start
|
||||||
|
(`SERVICE_AUTO_START`); service-account docs in README (LocalSystem
|
||||||
|
for local servers; `--account`/`--password` for remote DCOM).
|
||||||
|
- [done] 4. `%ProgramData%\OPCDA-Diagnozer\` conventions documented
|
||||||
|
(README deployment layout); config path recorded in the service's
|
||||||
|
registry `Parameters` at install so startup never depends on the
|
||||||
|
service CWD; `install` also registers the Event Log source with its
|
||||||
|
message file (the eventlog reporter's deferred step).
|
||||||
|
- [open] Frozen-exe SCM hosting: the exe self-hosts (`exeArgs=service`,
|
||||||
|
argv+CWD detection, `Initialize`+`PrepareToHostSingle`+
|
||||||
|
`StartServiceCtrlDispatcher`); hidden imports added to the build
|
||||||
|
script. Needs ONE careful elevated live pass (install → start →
|
||||||
|
kill-server → stop → reboot) against the built exe — terminal here
|
||||||
|
is not elevated; the in-process debug-host run is verified.
|
||||||
|
|
||||||
|
## Phase 4 — Integration & chaos testing
|
||||||
|
|
||||||
|
- [todo] Scenario matrix vs the sim server: server kill / service stop /
|
||||||
|
firewall silent drop / latency injection / device comm failure / quality
|
||||||
|
oscillation / stale+frozen / mass bad / config errors / **D16: flip a device
|
||||||
|
indicator (e.g. a `P500x.Connected` alias)**.
|
||||||
|
- [todo] 24 h soak (AC10): no leaks, reconnect accounting correct, heartbeats
|
||||||
|
every 5 min, ≤ ~50 MB RAM, ~0 % CPU idle.
|
||||||
|
|
||||||
|
## Phase 5 — Packaging & docs
|
||||||
|
|
||||||
|
- [done] PyInstaller exe build (`scripts/build_exe.ps1`, onedir + onefile,
|
||||||
|
read canary) — live-verified.
|
||||||
|
- [done] README (install, config, DCOM checklist, HRESULT table).
|
||||||
|
- [todo] Ops runbook: reading events, kind meanings (spec §12 matrix),
|
||||||
|
threshold tuning, evidence locations.
|
||||||
|
- [todo] 1.0.0 tag once AC1–AC10 pass.
|
||||||
|
|
||||||
|
## Housekeeping
|
||||||
|
|
||||||
|
- [todo] Wire `status_text` into `read`/`probe` output once it exists
|
||||||
|
(currently shows `raw_hex` + decoded name separately).
|
||||||
|
- [todo] Record the CACHE-vs-DEVICE poll-source decision in the spec once
|
||||||
|
made; add fake-client scripted reads for both sources to the poller tests.
|
||||||
+26
-1
@@ -16,6 +16,23 @@ tags:
|
|||||||
- { name: "Channel1.Device1.Counter", expect_change: true } # D13 frozen-value detection applies
|
- { name: "Channel1.Device1.Counter", expect_change: true } # D13 frozen-value detection applies
|
||||||
- { name: "Channel1.Device1.Level" }
|
- { name: "Channel1.Device1.Level" }
|
||||||
|
|
||||||
|
# D16 device-link indicators: tags saying whether the OPC server can reach the
|
||||||
|
# device. DOWN (= DEVICE_LINK_DOWN, CRITICAL, clear-text status in the log) when
|
||||||
|
# the tag reads non-GOOD, errors, or its GOOD value is NOT in ok_values.
|
||||||
|
# ok_values defaults to [true] (boolean connected-flags); set it per system
|
||||||
|
# convention: inverted fault flags -> [false, 0], string states -> ["RUN"].
|
||||||
|
# Compared by string equality on the rendered value ("True"/"1"/"RUN").
|
||||||
|
# Do NOT also list these under tags: - they are polled automatically.
|
||||||
|
#
|
||||||
|
# tags: and devices: are each optional (devices-only monitors just the links;
|
||||||
|
# tags-only runs classic surveillance) — but both empty is rejected:
|
||||||
|
# the tool would read nothing.
|
||||||
|
devices:
|
||||||
|
- { name: "Conveyor 3", tag: "Channel1.Device3.Connected" } # ok: true/1
|
||||||
|
- { name: "Conveyor 4", tag: "Channel1.Device4.Connected", ok_values: [true, 1] }
|
||||||
|
- { name: "Pump A", tag: "Channel1.PumpA.Fault", ok_values: [false, 0] } # inverted: 0 = ok
|
||||||
|
- { name: "Boiler 2", tag: "Channel1.Boiler2.State", ok_values: ["RUN"] }
|
||||||
|
|
||||||
latency:
|
latency:
|
||||||
warn_ms: 1000 # D4 single-spike threshold
|
warn_ms: 1000 # D4 single-spike threshold
|
||||||
critical_ms: 5000 # D5 sustained-slow severity
|
critical_ms: 5000 # D5 sustained-slow severity
|
||||||
@@ -40,6 +57,14 @@ report:
|
|||||||
dir: "C:/ProgramData/OPCDA-Diagnozer/logs"
|
dir: "C:/ProgramData/OPCDA-Diagnozer/logs"
|
||||||
max_mb: 50 # rotate after this size
|
max_mb: 50 # rotate after this size
|
||||||
backups: 10 # rotated files kept
|
backups: 10 # rotated files kept
|
||||||
|
textlog: # human-readable anomaly log (spec 9.1)
|
||||||
|
dir: "C:/ProgramData/OPCDA-Diagnozer/logs"
|
||||||
|
max_mb: 20
|
||||||
|
backups: 10
|
||||||
eventlog:
|
eventlog:
|
||||||
source: "OPCDA-Diagnozer"
|
source: "OPCDA-Diagnozer"
|
||||||
min_severity: WARN # INFO / WARN / CRITICAL
|
min_severity: WARN # INFO / WARN / CRITICAL
|
||||||
|
#http: # optional webhook; omit = off
|
||||||
|
# url: "https://alerts.example.com/api/opcda"
|
||||||
|
# bearer_token: "..."
|
||||||
|
# timeout_s: 5
|
||||||
@@ -27,7 +27,7 @@ opcda-diagnozer/
|
|||||||
├── src/opcda_diagnozer/
|
├── src/opcda_diagnozer/
|
||||||
│ ├── __init__.py
|
│ ├── __init__.py
|
||||||
│ ├── __main__.py # python -m opcda_diagnozer
|
│ ├── __main__.py # python -m opcda_diagnozer
|
||||||
│ ├── main.py # CLI: run / install / uninstall / check-config / probe / list-devices / list-tags
|
│ ├── main.py # CLI: run / install / uninstall / check-config / probe / read / list-items
|
||||||
│ ├── config.py # pydantic-free typed config (dataclasses) + YAML load + validation
|
│ ├── config.py # pydantic-free typed config (dataclasses) + YAML load + validation
|
||||||
│ ├── events.py # DiagEvent, Severity, EventKind
|
│ ├── events.py # DiagEvent, Severity, EventKind
|
||||||
│ ├── quality.py # OPC DA quality decode tables
|
│ ├── quality.py # OPC DA quality decode tables
|
||||||
@@ -44,6 +44,7 @@ opcda-diagnozer/
|
|||||||
│ ├── report/
|
│ ├── report/
|
||||||
│ │ ├── base.py # Reporter protocol
|
│ │ ├── base.py # Reporter protocol
|
||||||
│ │ ├── jsonl.py # rotating JSONL file
|
│ │ ├── jsonl.py # rotating JSONL file
|
||||||
|
│ │ ├── textlog.py # plain-text anomaly log (human-readable, §9.1)
|
||||||
│ │ ├── eventlog.py # Windows Event Log (WARN+)
|
│ │ ├── eventlog.py # Windows Event Log (WARN+)
|
||||||
│ │ └── http.py # optional webhook
|
│ │ └── http.py # optional webhook
|
||||||
│ └── service.py # pywin32 ServiceFramework wrapper
|
│ └── service.py # pywin32 ServiceFramework wrapper
|
||||||
@@ -82,7 +83,13 @@ Tasks:
|
|||||||
|
|
||||||
1. `config.py` — YAML config (see `config.example.yaml` sketch in README), dataclass
|
1. `config.py` — YAML config (see `config.example.yaml` sketch in README), dataclass
|
||||||
model, strict validation, `check-config` messages that explain exactly what is wrong.
|
model, strict validation, `check-config` messages that explain exactly what is wrong.
|
||||||
2. `quality.py` — decode tables (spec §7), `decode_quality(raw: int) -> QualityInfo`.
|
Includes the **device-link indicators** (spec §6.1 / D16): `devices:` — each entry a
|
||||||
|
human-readable device name plus its connection-indicator tag ItemID. Indicators are
|
||||||
|
added to the polled tag set automatically; `check-config` warns when an indicator tag
|
||||||
|
is also listed in `tags:` (it should not be) and fails on an empty device name.
|
||||||
|
2. `quality.py` — decode tables (spec §7), `decode_quality(raw: int) -> QualityInfo`,
|
||||||
|
plus the **clear-text status helper**: `status_text(raw) -> "NAME (0xNN)"` used by
|
||||||
|
every reporter (spec §7 closing rule — e.g. `BAD / communication failure (0x18)`).
|
||||||
3. `opc/base.py` — client abstraction:
|
3. `opc/base.py` — client abstraction:
|
||||||
```python
|
```python
|
||||||
class OpcClient(Protocol):
|
class OpcClient(Protocol):
|
||||||
@@ -99,6 +106,14 @@ Tasks:
|
|||||||
6. `monitor/detectors.py` — D1–D15 as small pure functions/classes over
|
6. `monitor/detectors.py` — D1–D15 as small pure functions/classes over
|
||||||
`PollOutcome` (list[ReadResult] + rtt_ms + error), using an injected clock for
|
`PollOutcome` (list[ReadResult] + rtt_ms + error), using an injected clock for
|
||||||
testability. Rolling window (deque, size `window_polls`) for p95/streak logic.
|
testability. Rolling window (deque, size `window_polls`) for p95/streak logic.
|
||||||
|
**D16 device-link detector** (spec §6.1): per configured indicator — quality
|
||||||
|
not GOOD, a read error, or a GOOD read whose value is **not in the
|
||||||
|
configurable `ok_values` list** (default `[true]`; string-equality compare
|
||||||
|
of the rendered value, so inverted fault flags `[false, 0]` and string states
|
||||||
|
`["RUN"]` both work) ⇒ `DEVICE_LINK_DOWN` (CRITICAL) carrying the device name,
|
||||||
|
indicator tag, clear-text status and value; recovery (`DEVICE_LINK_UP`) when
|
||||||
|
it reads GOOD with an ok value again. Edge-level per device, so flapping
|
||||||
|
links don't spam (one event per transition).
|
||||||
7. `monitor/poller.py` — the single COM thread; watchdog logic for hang detection
|
7. `monitor/poller.py` — the single COM thread; watchdog logic for hang detection
|
||||||
(spec §10.4): a second thread timestamps poll starts/ends; on `hard_hang_s` the COM
|
(spec §10.4): a second thread timestamps poll starts/ends; on `hard_hang_s` the COM
|
||||||
thread is abandoned and rebuilt.
|
thread is abandoned and rebuilt.
|
||||||
@@ -107,17 +122,26 @@ Tasks:
|
|||||||
hangs, quality flips) → powers the entire unit-test suite without COM/hardware.
|
hangs, quality flips) → powers the entire unit-test suite without COM/hardware.
|
||||||
|
|
||||||
**Exit:** `opcda-diagnozer run` (foreground console) produces correct events against the
|
**Exit:** `opcda-diagnozer run` (foreground console) produces correct events against the
|
||||||
fake client; unit tests for all detectors.
|
fake client; unit tests for all detectors — including D16 edge semantics (false /
|
||||||
|
non-GOOD / read-error / misconfigured value / recovery) against scripted indicator reads.
|
||||||
|
|
||||||
## 5. Phase 2 — Reporters (≈1 day)
|
## 5. Phase 2 — Reporters (≈1 day)
|
||||||
|
|
||||||
1. `report/jsonl.py` — rotation (size + backups), atomic-ish append, header line with
|
1. `report/jsonl.py` — rotation (size + backups), atomic-ish append, header line with
|
||||||
config digest for provenance.
|
config digest for provenance.
|
||||||
2. `report/eventlog.py` — pywin32 `win32evtlog` usage; map severities; source
|
2. `report/textlog.py` — **plain-text anomaly log** (spec §9.1): `anomalies-YYYYMMDD.log`
|
||||||
|
in the same directory as the JSONL files, one human-readable line per detected
|
||||||
|
anomaly (`timestamp SEVERITY KIND message`) with clear-text status `NAME (0xNN)`.
|
||||||
|
Anomalies only — heartbeats/CONNECTED are filtered out (a small `is_anomaly(event)`
|
||||||
|
predicate on `EventKind`); recoveries (D2, `DEVICE_LINK_UP`, quality back to GOOD)
|
||||||
|
**are** logged, so the file reads as an incident narrative. Daily + size rotation
|
||||||
|
with `backups` kept, same scheme as the JSONL reporter. Every line corresponds to
|
||||||
|
a JSONL record (text = convenience view, JSONL = source of truth).
|
||||||
|
3. `report/eventlog.py` — pywin32 `win32evtlog` usage; map severities; source
|
||||||
registration via `install` step.
|
registration via `install` step.
|
||||||
3. `report/http.py` — optional webhook with timeout + simple retry/drop on failure
|
4. `report/http.py` — optional webhook with timeout + simple retry/drop on failure
|
||||||
(never blocks the poll loop; queue with bound).
|
(never blocks the poll loop; queue with bound).
|
||||||
4. Heartbeat assembly (D15): uptime, reconnects, RTT percentiles, tags-bad counts.
|
5. Heartbeat assembly (D15): uptime, reconnects, RTT percentiles, tags-bad counts.
|
||||||
|
|
||||||
**Exit:** events visible in file + Event Log; heartbeat cadence verified.
|
**Exit:** events visible in file + Event Log; heartbeat cadence verified.
|
||||||
|
|
||||||
@@ -150,6 +174,7 @@ Scenario matrix against a simulation server (and, when available, the real serve
|
|||||||
| Quality oscillation | toggling sim items | D11 flapping |
|
| Quality oscillation | toggling sim items | D11 flapping |
|
||||||
| Stale/frozen tag | simulation server static item marked `expect_change` | D12 / D13 |
|
| Stale/frozen tag | simulation server static item marked `expect_change` | D12 / D13 |
|
||||||
| Mass bad quality | all sim items to bad | D14 |
|
| Mass bad quality | all sim items to bad | D14 |
|
||||||
|
| **Device link down** | indicator tag flipped to `false` / set BAD (alias a bool sim item per device, e.g. `P5001.Connected` on Matrikon) | D16 `DEVICE_LINK_DOWN`, recovery line when restored |
|
||||||
| Config errors | bad ProgID, unknown tag | fast fail + actionable message; `OPC_E_INVALIDITEMID` surfaced at startup |
|
| Config errors | bad ProgID, unknown tag | fast fail + actionable message; `OPC_E_INVALIDITEMID` surfaced at startup |
|
||||||
| Soak | 24 h healthy run | no leaks, heartbeats every 5 min, zero false positives |
|
| Soak | 24 h healthy run | no leaks, heartbeats every 5 min, zero false positives |
|
||||||
|
|
||||||
|
|||||||
@@ -99,9 +99,50 @@ diagnosis.
|
|||||||
| D13 | Frozen value | `expect_change: true` tag with unchanged value+timestamp for `max_age_s` | WARN |
|
| D13 | Frozen value | `expect_change: true` tag with unchanged value+timestamp for `max_age_s` | WARN |
|
||||||
| D14 | Mass quality degradation | ≥ 50% of monitored tags BAD simultaneously while connection OK | CRITICAL (`CONTROL_LINK_DEGRADED` — points at OPC-server↔device link, not DCOM) |
|
| D14 | Mass quality degradation | ≥ 50% of monitored tags BAD simultaneously while connection OK | CRITICAL (`CONTROL_LINK_DEGRADED` — points at OPC-server↔device link, not DCOM) |
|
||||||
| D15 | Heartbeat / summary | Periodic stats snapshot every `heartbeat_s` (300) | INFO |
|
| D15 | Heartbeat / summary | Periodic stats snapshot every `heartbeat_s` (300) | INFO |
|
||||||
|
| D16 | Device link down | A configured device-indicator tag reads `false`, or any quality other than GOOD, or a read error | CRITICAL per device (`DEVICE_LINK_DOWN`); D1/D2 do **not** imply device-link state |
|
||||||
|
|
||||||
All thresholds are configurable; severities are INFO / WARN / CRITICAL.
|
All thresholds are configurable; severities are INFO / WARN / CRITICAL.
|
||||||
|
|
||||||
|
### 6.1 Device connection-indicator tags (D16)
|
||||||
|
|
||||||
|
The config names one or more tags as **device-link indicators**: values the
|
||||||
|
control system maintains that say whether the OPC server can reach the device.
|
||||||
|
The **normal (connected) reading is configurable per indicator** — conventions
|
||||||
|
differ per control system, so it is never hardcoded:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
devices:
|
||||||
|
- { name: "Conveyor 3", tag: "...Connected" } # any truthy: true/1
|
||||||
|
- { name: "Pump A", tag: "...PumpAFault", ok_values: [false, 0] } # inverted fault flag
|
||||||
|
- { name: "Boiler 2", tag: "...State", ok_values: ["RUN"] } # enum/string state
|
||||||
|
```
|
||||||
|
|
||||||
|
Default `ok_values: [true]` (boolean connected-flags are the common case).
|
||||||
|
Value comparison is by **string equality** on the read value rendered as text
|
||||||
|
(`"True"`, `"1"`, `"RUN"`), so type quirks across systems cannot cause false
|
||||||
|
positives; quality must also be GOOD. Semantics (all DOWN cases are logged as
|
||||||
|
`DEVICE_LINK_DOWN`):
|
||||||
|
|
||||||
|
| Indicator reading | Interpretation |
|
||||||
|
|-------------------|------------------|
|
||||||
|
| GOOD + value ∈ `ok_values` | Device connected — normal |
|
||||||
|
| GOOD + value ∉ `ok_values` | **Device link down / fault** — CRITICAL (works for inverted fault-flags: a fault `1` is not in `[false, 0]`) |
|
||||||
|
| Any quality other than GOOD (BAD/UNCERTAIN) | **Something is wrong with the device link** — CRITICAL |
|
||||||
|
| Read error (tag unreadable) | **Something is wrong with the device link** — CRITICAL |
|
||||||
|
|
||||||
|
Device-link state is per indicator tag: the config maps each indicator to a
|
||||||
|
human-readable device name for the log lines (see `config.example.yaml`).
|
||||||
|
Recovery (`DEVICE_LINK_UP`) is logged when the same indicator reads GOOD with
|
||||||
|
an ok value again. A value that is neither in `ok_values` nor a recognized
|
||||||
|
fault shape still logs DOWN — the normal state is explicit, and everything
|
||||||
|
else is "not normal" by definition.
|
||||||
|
|
||||||
|
**Section optionality:** `tags:` and `devices:` are each optional in the
|
||||||
|
config — a devices-only setup monitors just the links, a tags-only setup
|
||||||
|
runs classic quality/latency surveillance. Both empty (or absent) is a
|
||||||
|
validation error: the tool would read nothing and every heartbeat would
|
||||||
|
report `tags_total: 0`.
|
||||||
|
|
||||||
## 7. OPC DA quality codes (reference)
|
## 7. OPC DA quality codes (reference)
|
||||||
|
|
||||||
OPC DA quality is a 16-bit field. Bits 7–6 of the low byte give the class:
|
OPC DA quality is a 16-bit field. Bits 7–6 of the low byte give the class:
|
||||||
@@ -126,6 +167,13 @@ decodes and names:
|
|||||||
Remaining UNCERTAIN subcodes (DA 2.05a) are decoded by the same table in `quality.py`.
|
Remaining UNCERTAIN subcodes (DA 2.05a) are decoded by the same table in `quality.py`.
|
||||||
Every quality event carries the raw hex code **and** the decoded name.
|
Every quality event carries the raw hex code **and** the decoded name.
|
||||||
|
|
||||||
|
**Clear-text status rule:** wherever a status or quality is reported — JSONL
|
||||||
|
fields, the text log, Event Log, console — it is written in clear text with
|
||||||
|
the raw code in parentheses: `BAD / communication failure (0x18)`,
|
||||||
|
`GOOD (0xC0)`. Nobody should have to decode hex codes by hand, but the raw
|
||||||
|
code stays for unambiguous correlation with vendor documentation and the
|
||||||
|
HRESULT table.
|
||||||
|
|
||||||
## 8. Event model
|
## 8. Event model
|
||||||
|
|
||||||
Every event is a structured record:
|
Every event is a structured record:
|
||||||
@@ -163,12 +211,36 @@ alongside and clearly labeled.
|
|||||||
| Reporter | Purpose | Notes |
|
| Reporter | Purpose | Notes |
|
||||||
|----------|---------|-------|
|
|----------|---------|-------|
|
||||||
| JSONL rotating file | Primary evidence trail | One JSON event per line; rotation by size; default dir `%ProgramData%\OPCDA-Diagnozer\logs` |
|
| JSONL rotating file | Primary evidence trail | One JSON event per line; rotation by size; default dir `%ProgramData%\OPCDA-Diagnozer\logs` |
|
||||||
|
| **Plain-text anomaly log** | Human-readable evidence | One line per **detected anomaly** (not heartbeats): timestamp, kind, tag/device, message with clear-text status `NAME (0xNN)`; same rotation scheme as JSONL. See §9.1 for the line format |
|
||||||
| Windows Event Log | Ops visibility | WARN+ only; source `OPCDA-Diagnozer` |
|
| Windows Event Log | Ops visibility | WARN+ only; source `OPCDA-Diagnozer` |
|
||||||
| Console | Foreground/debug runs | Human-readable |
|
| Console | Foreground/debug runs | Human-readable |
|
||||||
| HTTP webhook (optional) | Alert integration | POST one JSON event per request, optional bearer token |
|
| HTTP webhook (optional) | Alert integration | POST one JSON event per request, optional bearer token |
|
||||||
|
|
||||||
Reporters are pluggable (`Reporter.emit(event)`); a heartbeat event every 5 minutes
|
Reporters are pluggable (`Reporter.emit(event)`); a heartbeat event every 5 minutes
|
||||||
proves liveness and enables baseline graphs even when everything is healthy.
|
proves liveness and enables baseline graphs even when everything is healthy.
|
||||||
|
The JSONL file stays the machine-readable source of truth; the text log is a
|
||||||
|
human-first convenience view of the same events — every text line corresponds
|
||||||
|
to a JSONL record.
|
||||||
|
|
||||||
|
### 9.1 Plain-text anomaly log format
|
||||||
|
|
||||||
|
File `anomalies-YYYYMMDD.log` (daily rotation, size-capped) in the same log
|
||||||
|
directory. One line per detected anomaly — anomalies only; heartbeats and
|
||||||
|
reconnect-OK events do not appear here. Clear text first, codes in parentheses:
|
||||||
|
|
||||||
|
```
|
||||||
|
2026-10-07 14:23:01.123 UTC WARN QUALITY_BAD tag 'Channel1.Device1.Counter' changed GOOD (0xC0) -> BAD / communication failure (0x18)
|
||||||
|
2026-10-07 14:24:31.500 UTC CRITICAL DEVICE_LINK_DOWN device 'Conveyor 3': link indicator 'Channel1.Device3.Connected' read false (GOOD)
|
||||||
|
2026-10-07 14:26:02.780 UTC CRITICAL DEVICE_LINK_DOWN device 'Conveyor 3': link indicator 'Channel1.Device3.Connected' quality BAD / not connected (0x08)
|
||||||
|
2026-10-07 14:27:10.001 UTC CRITICAL POLL_HANG read call hung > 5000 ms; COM thread abandoned and rebuilt
|
||||||
|
2026-10-07 14:30:00.000 UTC CRITICAL CONTROL_LINK_DEGRADED 12 of 14 monitored tags BAD while connection OK
|
||||||
|
```
|
||||||
|
|
||||||
|
Fields (pipe-free, fixed order): `timestamp SEVERITY KIND message`.
|
||||||
|
Timestamps are the local receive time in UTC (§8). The message carries the
|
||||||
|
clear-text status with the code between parentheses per §7. A matching
|
||||||
|
recovery (link back up, quality back to GOOD) is also logged as an anomaly
|
||||||
|
line, so the log reads as a continuous incident narrative.
|
||||||
|
|
||||||
## 10. Operational behavior
|
## 10. Operational behavior
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,18 @@
|
|||||||
|
# OPC DA Diagnozer — development dependencies.
|
||||||
|
#
|
||||||
|
# Install: pip install -r requirements.dev.txt
|
||||||
|
# (pulls in the production dependencies via -r requirements.txt)
|
||||||
|
#
|
||||||
|
# The interpreter MUST be 32-bit (SPECIFICATION §11) — see README
|
||||||
|
# "Development setup" for the venv procedure.
|
||||||
|
|
||||||
|
-r requirements.txt
|
||||||
|
|
||||||
|
pytest>=8.0
|
||||||
|
# The test suite (165 tests): detectors/state/poller/reporters against the
|
||||||
|
# fake client — runs on any OS (COM paths are import-guarded and live-smoke
|
||||||
|
# tested on Windows dev boxes only).
|
||||||
|
|
||||||
|
pyinstaller>=6.22
|
||||||
|
# Builds the standalone 32-bit exe (scripts/build_exe.ps1) — no Python
|
||||||
|
# installation needed on the target host. Not needed at runtime.
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
# OPC DA Diagnozer — production dependencies.
|
||||||
|
#
|
||||||
|
# Install: pip install -r requirements.txt
|
||||||
|
# The interpreter MUST be 32-bit (SPECIFICATION §11): the OPC DA Automation
|
||||||
|
# Wrapper and Graybox DLLs are 32-bit in-proc COM servers — a 64-bit process
|
||||||
|
# cannot load them. Create the venv with the 32-bit Python (README
|
||||||
|
# "Development setup").
|
||||||
|
#
|
||||||
|
# Note: pyproject.toml remains the canonical dependency source for the
|
||||||
|
# package install (`pip install -e .`); this file mirrors it for tooling
|
||||||
|
# that expects plain requirements files (pip install -r, constraint sync).
|
||||||
|
|
||||||
|
ruamel.yaml>=0.18
|
||||||
|
# YAML config loading (config.py). Pure Python — PyYAML publishes no win32
|
||||||
|
# (32-bit) wheels for CPython 3.14 and its sdist requires MSVC, which the
|
||||||
|
# diag hosts lack.
|
||||||
|
|
||||||
|
pywin32>=306; sys_platform == "win32"
|
||||||
|
# COM access (pythoncom, win32com, win32evtlog) for the Automation Wrapper
|
||||||
|
# adapter, discovery and the Windows Event Log reporter. Every import is
|
||||||
|
# guarded so tests still run on any OS (IMPLEMENTATION_PLAN §9).
|
||||||
@@ -0,0 +1,236 @@
|
|||||||
|
# Build the Windows executable (dist\opcda-diagnozer.exe) with PyInstaller.
|
||||||
|
#
|
||||||
|
# Usage (from the repo root):
|
||||||
|
# powershell -NoProfile -ExecutionPolicy Bypass -File scripts\build_exe.ps1
|
||||||
|
# .\scripts\build_exe.ps1 # onedir (default): dist\opcda-diagnozer\
|
||||||
|
# .\scripts\build_exe.ps1 -Onefile # single exe: dist\opcda-diagnozer.exe
|
||||||
|
# .\scripts\build_exe.ps1 -Clean # remove build\ + dist\ first
|
||||||
|
#
|
||||||
|
# The exe is a console application hosting every CLI verb (check-config,
|
||||||
|
# list-servers, list-items, read/probe, run) AND the Windows service host:
|
||||||
|
# 'install' registers this same exe with the SCM (IMPLEMENTATION_PLAN 6);
|
||||||
|
# when the SCM starts it, the fixed 'service' argv shape dispatches to
|
||||||
|
# the ServiceFramework host.
|
||||||
|
#
|
||||||
|
# Hard rules enforced here:
|
||||||
|
# 1. 32-bit interpreter (SPECIFICATION 11): the OPC DA Automation Wrapper
|
||||||
|
# (OPCAutomation.dll) is a 32-bit in-proc COM DLL - a 64-bit exe cannot
|
||||||
|
# load it and every COM call fails at runtime. The produced PE is
|
||||||
|
# re-verified to be 32-bit (IMAGE_FILE_MACHINE_I386) after the build.
|
||||||
|
# 2. PyInstaller must see the modules the code imports lazily inside
|
||||||
|
# functions (COM, browse, service host) - declared via -HiddenImport.
|
||||||
|
[CmdletBinding()]
|
||||||
|
param(
|
||||||
|
[switch]$Onefile,
|
||||||
|
[switch]$Clean
|
||||||
|
)
|
||||||
|
|
||||||
|
$ErrorActionPreference = 'Stop'
|
||||||
|
|
||||||
|
$Repo = Split-Path -Parent $PSScriptRoot
|
||||||
|
$VenvPython = Join-Path $Repo '.venv\Scripts\python.exe'
|
||||||
|
$Entry = Join-Path $Repo 'src\opcda_diagnozer\__main__.py'
|
||||||
|
$Dist = Join-Path $Repo 'dist'
|
||||||
|
$Build = Join-Path $Repo 'build'
|
||||||
|
$Name = 'opcda-diagnozer'
|
||||||
|
$VersionFile = Join-Path $PSScriptRoot '.exe_version_info.txt'
|
||||||
|
|
||||||
|
# Modules imported lazily inside functions (import guards, IMPLEMENTATION_PLAN 9);
|
||||||
|
# PyInstaller's static analysis misses those, so they are declared explicitly.
|
||||||
|
$HiddenImports = @(
|
||||||
|
# pywin32 / COM - deferred into methods in opc/automation.py, opc/discovery.py
|
||||||
|
'pythoncom',
|
||||||
|
'pywintypes',
|
||||||
|
'win32com.client',
|
||||||
|
# pywintypes.datetime(VARIANT DATE -> datetime conversion) lazily imports
|
||||||
|
# win32timezone at CONVERSION time - missing from the bundle, every read of
|
||||||
|
# item.TimeStamp raised SystemError('InvokeTypes ... exception set') frozen
|
||||||
|
# (observed 2026-10-07: all tags failed in the exe while the venv worked).
|
||||||
|
'win32timezone',
|
||||||
|
# opc backend modules imported lazily in main.py CLI verbs
|
||||||
|
'opcda_diagnozer.opc.automation',
|
||||||
|
'opcda_diagnozer.opc.browse',
|
||||||
|
# the service host (Phase 3) - lazily imported by main (SCM dispatch) and
|
||||||
|
# itself importing win32serviceutil/win32service/servicemanager/
|
||||||
|
# win32evtlogutil only inside functions
|
||||||
|
'opcda_diagnozer.service',
|
||||||
|
'win32service',
|
||||||
|
'win32serviceutil',
|
||||||
|
'servicemanager',
|
||||||
|
'win32evtlogutil'
|
||||||
|
)
|
||||||
|
|
||||||
|
function Fail {
|
||||||
|
param([string]$Message)
|
||||||
|
Write-Host 'build_exe: FAILED' -ForegroundColor Red
|
||||||
|
Write-Host " $Message" -ForegroundColor Red
|
||||||
|
exit 1
|
||||||
|
}
|
||||||
|
|
||||||
|
function Invoke-Captured {
|
||||||
|
# Run a native command and capture stdout+stderr with its exit code.
|
||||||
|
# $ErrorActionPreference must be relaxed around 2>&1 on PS 5.1: with 'Stop',
|
||||||
|
# redirecting native stderr raises a NativeCommandError and kills the script.
|
||||||
|
param([string]$FilePath, [string[]]$Arguments)
|
||||||
|
$prev = $ErrorActionPreference
|
||||||
|
$ErrorActionPreference = 'Continue'
|
||||||
|
try {
|
||||||
|
$lines = & $FilePath @Arguments 2>&1
|
||||||
|
$text = ($lines | ForEach-Object { "$_" }) -join [Environment]::NewLine
|
||||||
|
[pscustomobject]@{ ExitCode = $LASTEXITCODE; Output = $text }
|
||||||
|
}
|
||||||
|
finally {
|
||||||
|
$ErrorActionPreference = $prev
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# --- environment checks ------------------------------------------------------
|
||||||
|
|
||||||
|
if (-not (Test-Path $VenvPython)) {
|
||||||
|
Fail ("the 32-bit venv was not found: $VenvPython - create it first " +
|
||||||
|
"(README 'Development setup'): " +
|
||||||
|
'& "C:\...\Python314-32\python.exe" -m venv .venv; ' +
|
||||||
|
'& .\.venv\Scripts\python.exe -m pip install -e ".[dev]"')
|
||||||
|
}
|
||||||
|
|
||||||
|
$bitness = (Invoke-Captured -FilePath $VenvPython -Arguments @('-c', "import struct; print(struct.calcsize('P') * 8)")).Output.Trim()
|
||||||
|
if ($bitness -ne '32') {
|
||||||
|
Fail ("the venv interpreter is ${bitness}-bit, but the OPC DA wrappers are " +
|
||||||
|
'32-bit in-proc DLLs (SPECIFICATION 11): a 64-bit exe cannot load ' +
|
||||||
|
"them. Rebuild the venv with the 32-bit Python (README 'Development setup').")
|
||||||
|
}
|
||||||
|
Write-Host 'build_exe: OK - 32-bit interpreter (SPECIFICATION 11 bitness rule)'
|
||||||
|
|
||||||
|
$pyi = Invoke-Captured -FilePath $VenvPython -Arguments @('-m', 'PyInstaller', '--version')
|
||||||
|
if ($pyi.ExitCode -ne 0) {
|
||||||
|
Write-Host 'build_exe: pyinstaller missing - installing into the venv ...'
|
||||||
|
& $VenvPython -m pip install pyinstaller
|
||||||
|
if ($LASTEXITCODE -ne 0) { Fail 'pip install pyinstaller failed' }
|
||||||
|
}
|
||||||
|
|
||||||
|
if ($Clean) {
|
||||||
|
foreach ($folder in @($Build, $Dist)) {
|
||||||
|
if (Test-Path $folder) { Remove-Item -Recurse -Force $folder }
|
||||||
|
}
|
||||||
|
Write-Host 'build_exe: cleaned build/ and dist/'
|
||||||
|
}
|
||||||
|
|
||||||
|
# --- version metadata (exe properties -> Details tab) -------------------------
|
||||||
|
|
||||||
|
$initText = Get-Content (Join-Path $Repo 'src\opcda_diagnozer\__init__.py') -Raw
|
||||||
|
if ($initText -notmatch '__version__\s*=\s*"([^"]+)"') {
|
||||||
|
Fail 'could not read __version__ from src\opcda_diagnozer\__init__.py'
|
||||||
|
}
|
||||||
|
$Version = $Matches[1]
|
||||||
|
$parts = $Version.Split('.')
|
||||||
|
$major = $parts[0]
|
||||||
|
$minor = $parts[1]
|
||||||
|
|
||||||
|
# Keyword names must match PyInstaller's FixedFileInfo.__init__ exactly:
|
||||||
|
# filevers, prodvers, mask, flags, OS, fileType, subtype, date.
|
||||||
|
# [IO.File]::WriteAllText writes UTF-8 WITHOUT a BOM - Set-Content -Encoding
|
||||||
|
# UTF8 on PS 5.1 prepends one, and the BOM breaks PyInstaller's eval().
|
||||||
|
$VersionInfo = @"
|
||||||
|
# UTF-8
|
||||||
|
VSVersionInfo(
|
||||||
|
ffi=FixedFileInfo(
|
||||||
|
filevers=($major, $minor, 0, 0),
|
||||||
|
prodvers=($major, $minor, 0, 0),
|
||||||
|
mask=0x3F,
|
||||||
|
flags=0x0,
|
||||||
|
OS=0x40004,
|
||||||
|
fileType=0x1,
|
||||||
|
subtype=0x0,
|
||||||
|
date=(0, 0)
|
||||||
|
),
|
||||||
|
kids=[
|
||||||
|
StringFileInfo(
|
||||||
|
[
|
||||||
|
StringTable(
|
||||||
|
'040904B0',
|
||||||
|
[
|
||||||
|
StringStruct('CompanyName', 'Vivab'),
|
||||||
|
StringStruct('FileDescription', 'OPC DA Diagnozer - read-only OPC DA reference diagnostic client'),
|
||||||
|
StringStruct('FileVersion', '$Version.0'),
|
||||||
|
StringStruct('InternalName', 'opcda-diagnozer'),
|
||||||
|
StringStruct('OriginalFilename', 'opcda-diagnozer.exe'),
|
||||||
|
StringStruct('ProductName', 'OPC DA Diagnozer'),
|
||||||
|
StringStruct('ProductVersion', '$Version')
|
||||||
|
]
|
||||||
|
)
|
||||||
|
]
|
||||||
|
),
|
||||||
|
VarFileInfo([VarStruct('Translation', [1033, 1200])])
|
||||||
|
]
|
||||||
|
)
|
||||||
|
"@
|
||||||
|
[IO.File]::WriteAllText($VersionFile, $VersionInfo)
|
||||||
|
|
||||||
|
# --- the build itself ---------------------------------------------------------
|
||||||
|
|
||||||
|
$pyArgs = @(
|
||||||
|
'-m', 'PyInstaller',
|
||||||
|
'--noconfirm', '--clean',
|
||||||
|
'--name', $Name,
|
||||||
|
'--distpath', $Dist,
|
||||||
|
'--workpath', $Build,
|
||||||
|
'--specpath', $Build,
|
||||||
|
'--console',
|
||||||
|
'--version-file', $VersionFile
|
||||||
|
)
|
||||||
|
foreach ($hidden in $HiddenImports) { $pyArgs += @('--hidden-import', $hidden) }
|
||||||
|
$pyArgs += $(if ($Onefile) { '--onefile' } else { '--onedir' })
|
||||||
|
$pyArgs += $Entry
|
||||||
|
|
||||||
|
Write-Host 'build_exe: running PyInstaller...'
|
||||||
|
& $VenvPython @pyArgs
|
||||||
|
if ($LASTEXITCODE -ne 0) { Fail "PyInstaller exited $LASTEXITCODE" }
|
||||||
|
|
||||||
|
$Exe = if ($Onefile) { Join-Path $Dist "$Name.exe" } else { Join-Path $Dist "$Name\$Name.exe" }
|
||||||
|
if (-not (Test-Path $Exe)) { Fail "expected exe not found: $Exe" }
|
||||||
|
Write-Host "build_exe: OK - $Exe"
|
||||||
|
|
||||||
|
# --- final bitness guard: the produced PE must be 32-bit (SPECIFICATION 11) --
|
||||||
|
|
||||||
|
$bytes = [System.IO.File]::ReadAllBytes($Exe)
|
||||||
|
$peOffset = [System.BitConverter]::ToInt32($bytes, 0x3C)
|
||||||
|
$machine = [System.BitConverter]::ToUInt16($bytes, $peOffset + 4)
|
||||||
|
if ($machine -ne 0x14C) {
|
||||||
|
Fail ("the built exe is not 32-bit (PE machine 0x{0:X4}) - build from the 32-bit venv" -f $machine)
|
||||||
|
}
|
||||||
|
Write-Host 'build_exe: OK - exe is 32-bit (PE machine 0x014C, IMAGE_FILE_MACHINE_I386)'
|
||||||
|
|
||||||
|
# --- smoke test ---------------------------------------------------------------
|
||||||
|
|
||||||
|
Write-Host "build_exe: smoke-testing $Name.exe ..."
|
||||||
|
$ExampleConfig = Join-Path $Repo 'config.example.yaml'
|
||||||
|
$checks = @(
|
||||||
|
@{ Label = '--version'; Arguments = @('--version') },
|
||||||
|
@{ Label = 'check-config'; Arguments = @('check-config', '--config', $ExampleConfig) },
|
||||||
|
@{ Label = 'list-servers (registry)'; Arguments = @('list-servers', '--source', 'registry') }
|
||||||
|
)
|
||||||
|
foreach ($check in $checks) {
|
||||||
|
$result = Invoke-Captured -FilePath $Exe -Arguments $check.Arguments
|
||||||
|
$first = ($result.Output -split '\r?\n' | Where-Object { $_.Trim() } | Select-Object -First 1)
|
||||||
|
Write-Host (" {0} -> exit {1}: {2}" -f $check.Label, $result.ExitCode, $first)
|
||||||
|
if ($result.ExitCode -ne 0) { Fail ("smoke test failed: " + $check.Label) }
|
||||||
|
}
|
||||||
|
|
||||||
|
# Frozen-read canary: the COM timestamp conversion (item.TimeStamp) lazily
|
||||||
|
# imports win32timezone - if it is missing from the bundle, EVERY read fails
|
||||||
|
# with SystemError('InvokeTypes ... exception set'). The registry/discovery
|
||||||
|
# smoke tests cannot catch this; only a real read does. Skipped gracefully
|
||||||
|
# when no COM server is reachable on this build host.
|
||||||
|
$canary = Invoke-Captured -FilePath $Exe -Arguments @(
|
||||||
|
'read', '--server', 'Matrikon.OPC.Simulation.1', 'Bucket Brigade.Int1'
|
||||||
|
)
|
||||||
|
$canaryFirst = ($canary.Output -split '\r?\n' | Where-Object { $_.Trim() } | Select-Object -First 2) -join ' | '
|
||||||
|
if ($canary.ExitCode -eq 0) {
|
||||||
|
Write-Host " read canary (Matrikon sim) -> exit 0: $canaryFirst"
|
||||||
|
} elseif ($canary.Output -match 'SystemError.*InvokeTypes') {
|
||||||
|
Fail ("frozen read canary failed with the missing-win32timezone signature: $canaryFirst")
|
||||||
|
} else {
|
||||||
|
Write-Host " read canary skipped (no Matrikon sim reachable): exit $($canary.ExitCode)"
|
||||||
|
}
|
||||||
|
Write-Host 'build_exe: smoke test OK'
|
||||||
|
exit 0
|
||||||
@@ -38,6 +38,23 @@ class TagEntry:
|
|||||||
expect_change: bool = False
|
expect_change: bool = False
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class DeviceEntry:
|
||||||
|
"""D16 device-link indicator: a tag saying if the device is reachable.
|
||||||
|
|
||||||
|
DOWN (``DEVICE_LINK_DOWN``, CRITICAL) when ``tag`` reads non-GOOD, errors,
|
||||||
|
or its GOOD value is **not** in ``ok_values`` — the normal (connected)
|
||||||
|
reading, configurable per indicator because conventions differ per
|
||||||
|
control system: connected-flags ``[true]`` (default), inverted fault-flags
|
||||||
|
``[false, 0]``, string states ``["RUN"]``. Compared by string equality on
|
||||||
|
the rendered value (SPECIFICATION §6.1).
|
||||||
|
"""
|
||||||
|
|
||||||
|
name: str
|
||||||
|
tag: str
|
||||||
|
ok_values: tuple[str, ...] = ("true",)
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
class PollConfig:
|
class PollConfig:
|
||||||
interval_s: float = 1.0
|
interval_s: float = 1.0
|
||||||
@@ -86,11 +103,22 @@ class EventLogConfig:
|
|||||||
min_severity: str = "WARN" # INFO / WARN / CRITICAL
|
min_severity: str = "WARN" # INFO / WARN / CRITICAL
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class HttpConfig:
|
||||||
|
"""Optional webhook reporter (spec §9): one JSON event per POST."""
|
||||||
|
|
||||||
|
url: str
|
||||||
|
bearer_token: str | None = None
|
||||||
|
timeout_s: float = 5.0
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
class ReportConfig:
|
class ReportConfig:
|
||||||
heartbeat_s: float = 300.0 # D15
|
heartbeat_s: float = 300.0 # D15
|
||||||
jsonl: JsonlConfig = field(default_factory=JsonlConfig)
|
jsonl: JsonlConfig = field(default_factory=JsonlConfig)
|
||||||
|
textlog: JsonlConfig = field(default_factory=JsonlConfig) # same knobs: dir/max_mb/backups
|
||||||
eventlog: EventLogConfig = field(default_factory=EventLogConfig)
|
eventlog: EventLogConfig = field(default_factory=EventLogConfig)
|
||||||
|
http: HttpConfig | None = None # omitted = webhook off
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
@@ -100,6 +128,7 @@ class Config:
|
|||||||
opc: OpcConfig
|
opc: OpcConfig
|
||||||
poll: PollConfig = field(default_factory=PollConfig)
|
poll: PollConfig = field(default_factory=PollConfig)
|
||||||
tags: tuple[TagEntry, ...] = ()
|
tags: tuple[TagEntry, ...] = ()
|
||||||
|
devices: tuple[DeviceEntry, ...] = ()
|
||||||
latency: LatencyConfig = field(default_factory=LatencyConfig)
|
latency: LatencyConfig = field(default_factory=LatencyConfig)
|
||||||
quality: QualityConfig = field(default_factory=QualityConfig)
|
quality: QualityConfig = field(default_factory=QualityConfig)
|
||||||
stale: StaleConfig = field(default_factory=StaleConfig)
|
stale: StaleConfig = field(default_factory=StaleConfig)
|
||||||
@@ -107,8 +136,41 @@ class Config:
|
|||||||
report: ReportConfig = field(default_factory=ReportConfig)
|
report: ReportConfig = field(default_factory=ReportConfig)
|
||||||
|
|
||||||
|
|
||||||
_ALLOWED_TOP_LEVEL = {"opc", "poll", "tags", "latency", "quality", "stale", "browse", "report"}
|
_ALLOWED_TOP_LEVEL = {"opc", "poll", "tags", "devices", "latency", "quality", "stale", "browse", "report"}
|
||||||
_TAG_KEYS = {"name", "expect_change"}
|
_TAG_KEYS = {"name", "expect_change"}
|
||||||
|
_DEVICE_KEYS = {"name", "tag", "ok_values"}
|
||||||
|
|
||||||
|
|
||||||
|
def _normalize_ok_values(raw: Any, where: str) -> tuple[str, ...]:
|
||||||
|
"""ok_values as a tuple of rendered strings ("true", "1", "RUN").
|
||||||
|
|
||||||
|
The D16 comparison is string equality on the value rendered as text, so
|
||||||
|
booleans/ints/strings from YAML all work uniformly: `[true]` -> `("true",)`,
|
||||||
|
`[false, 0]` -> `("false", "0")`.
|
||||||
|
"""
|
||||||
|
if raw is None:
|
||||||
|
return ("true",)
|
||||||
|
if isinstance(raw, (list, tuple)):
|
||||||
|
if not raw:
|
||||||
|
raise ConfigError(f"{where}: ok_values must not be empty - set the normal (connected) value(s)")
|
||||||
|
values = []
|
||||||
|
for item in raw:
|
||||||
|
if isinstance(item, bool):
|
||||||
|
values.append("true" if item else "false")
|
||||||
|
elif isinstance(item, (int, float, str)):
|
||||||
|
values.append(str(item).strip())
|
||||||
|
else:
|
||||||
|
raise ConfigError(
|
||||||
|
f"{where}: ok_values entries must be scalars (bool/int/str), got {type(item).__name__}"
|
||||||
|
)
|
||||||
|
cleaned = tuple(v for v in values if v)
|
||||||
|
if not cleaned:
|
||||||
|
raise ConfigError(f"{where}: ok_values must not be empty - set the normal (connected) value(s)")
|
||||||
|
return cleaned
|
||||||
|
# a single scalar is shorthand for a one-element list
|
||||||
|
if isinstance(raw, (bool, int, float, str)):
|
||||||
|
return _normalize_ok_values([raw], where)
|
||||||
|
raise ConfigError(f"{where}: ok_values must be a list of scalars, got {type(raw).__name__}")
|
||||||
|
|
||||||
|
|
||||||
def _build_section(cls: type[Any], data: Any, where: str) -> Any:
|
def _build_section(cls: type[Any], data: Any, where: str) -> Any:
|
||||||
@@ -132,9 +194,14 @@ def _build_section(cls: type[Any], data: Any, where: str) -> Any:
|
|||||||
def _build_report(data: Any) -> ReportConfig:
|
def _build_report(data: Any) -> ReportConfig:
|
||||||
data = {} if data is None else dict(data)
|
data = {} if data is None else dict(data)
|
||||||
jsonl = _build_section(JsonlConfig, data.pop("jsonl", None), "report.jsonl")
|
jsonl = _build_section(JsonlConfig, data.pop("jsonl", None), "report.jsonl")
|
||||||
|
textlog = _build_section(JsonlConfig, data.pop("textlog", None), "report.textlog")
|
||||||
eventlog = _build_section(EventLogConfig, data.pop("eventlog", None), "report.eventlog")
|
eventlog = _build_section(EventLogConfig, data.pop("eventlog", None), "report.eventlog")
|
||||||
|
http_raw = data.pop("http", None)
|
||||||
|
http = None
|
||||||
|
if http_raw is not None:
|
||||||
|
http = _build_section(HttpConfig, http_raw, "report.http")
|
||||||
rest = _build_section(ReportConfig, data, "report")
|
rest = _build_section(ReportConfig, data, "report")
|
||||||
return replace(rest, jsonl=jsonl, eventlog=eventlog)
|
return replace(rest, jsonl=jsonl, textlog=textlog, eventlog=eventlog, http=http)
|
||||||
|
|
||||||
|
|
||||||
def _build_tags(data: Any) -> tuple[TagEntry, ...]:
|
def _build_tags(data: Any) -> tuple[TagEntry, ...]:
|
||||||
@@ -169,6 +236,37 @@ def _build_tags(data: Any) -> tuple[TagEntry, ...]:
|
|||||||
return tuple(entries)
|
return tuple(entries)
|
||||||
|
|
||||||
|
|
||||||
|
def _build_devices(data: Any) -> tuple[DeviceEntry, ...]:
|
||||||
|
"""Parse the devices: section (D16 link indicators) with actionable errors."""
|
||||||
|
if data is None:
|
||||||
|
return ()
|
||||||
|
if not isinstance(data, list):
|
||||||
|
raise ConfigError(f"devices: must be a list, got {type(data).__name__}")
|
||||||
|
entries: list[DeviceEntry] = []
|
||||||
|
for i, item in enumerate(data):
|
||||||
|
if not isinstance(item, dict):
|
||||||
|
raise ConfigError(
|
||||||
|
f"devices[{i}]: must be a mapping with 'name' and 'tag'"
|
||||||
|
)
|
||||||
|
unknown = sorted(set(item) - _DEVICE_KEYS)
|
||||||
|
if unknown:
|
||||||
|
raise ConfigError(
|
||||||
|
f"devices[{i}]: unknown key(s) {unknown}; accepted keys: {sorted(_DEVICE_KEYS)}"
|
||||||
|
)
|
||||||
|
name = item.get("name")
|
||||||
|
tag = item.get("tag")
|
||||||
|
if not isinstance(name, str) or not name.strip():
|
||||||
|
raise ConfigError(f"devices[{i}]: 'name' is required and must be a non-empty string")
|
||||||
|
if not isinstance(tag, str) or not tag.strip():
|
||||||
|
raise ConfigError(
|
||||||
|
f"devices[{i}]: 'tag' is required and must be the connection-indicator "
|
||||||
|
"tag ItemID (see list-items)"
|
||||||
|
)
|
||||||
|
ok_values = _normalize_ok_values(item.get("ok_values"), f"devices[{i}]")
|
||||||
|
entries.append(DeviceEntry(name=name, tag=tag, ok_values=ok_values))
|
||||||
|
return tuple(entries)
|
||||||
|
|
||||||
|
|
||||||
def _validate(config: Config) -> None:
|
def _validate(config: Config) -> None:
|
||||||
"""Cross-field validation; each message names the offending key."""
|
"""Cross-field validation; each message names the offending key."""
|
||||||
if not config.opc.server or not config.opc.server.strip():
|
if not config.opc.server or not config.opc.server.strip():
|
||||||
@@ -178,14 +276,34 @@ def _validate(config: Config) -> None:
|
|||||||
)
|
)
|
||||||
if config.opc.connect_timeout_s <= 0:
|
if config.opc.connect_timeout_s <= 0:
|
||||||
raise ConfigError(f"opc.connect_timeout_s: must be > 0, got {config.opc.connect_timeout_s}")
|
raise ConfigError(f"opc.connect_timeout_s: must be > 0, got {config.opc.connect_timeout_s}")
|
||||||
if not config.tags:
|
if not config.tags and not config.devices:
|
||||||
raise ConfigError(
|
raise ConfigError(
|
||||||
"tags: at least one tag is required — without tags there is nothing to monitor"
|
"tags: nothing to monitor - define at least one tag or device "
|
||||||
|
"indicator (devices: entries alone are fine, tags: entries alone "
|
||||||
|
"are fine; both empty means the tool would read nothing)"
|
||||||
)
|
)
|
||||||
names = [t.name for t in config.tags]
|
names = [t.name for t in config.tags]
|
||||||
duplicates = sorted({n for n in names if names.count(n) > 1})
|
duplicates = sorted({n for n in names if names.count(n) > 1})
|
||||||
if duplicates:
|
if duplicates:
|
||||||
raise ConfigError(f"tags: duplicate tag name(s) {duplicates}")
|
raise ConfigError(f"tags: duplicate tag name(s) {duplicates}")
|
||||||
|
device_names = [d.name for d in config.devices]
|
||||||
|
duplicate_devices = sorted({n for n in device_names if device_names.count(n) > 1})
|
||||||
|
if duplicate_devices:
|
||||||
|
raise ConfigError(f"devices: duplicate device name(s) {duplicate_devices}")
|
||||||
|
indicator_tags = [d.tag for d in config.devices]
|
||||||
|
duplicate_indicators = sorted({t for t in indicator_tags if indicator_tags.count(t) > 1})
|
||||||
|
if duplicate_indicators:
|
||||||
|
raise ConfigError(
|
||||||
|
f"devices: duplicate indicator tag(s) {duplicate_indicators} - one device per indicator"
|
||||||
|
)
|
||||||
|
# An indicator doubles as a monitored tag; listing it in tags: too would double-poll
|
||||||
|
# it and double-report its quality transitions (D8 AND D16 fire).
|
||||||
|
overlap = sorted(set(indicator_tags) & set(names))
|
||||||
|
if overlap:
|
||||||
|
raise ConfigError(
|
||||||
|
f"tags: {overlap} also appear as device indicators - remove them from "
|
||||||
|
"tags: (devices: entries are polled automatically)"
|
||||||
|
)
|
||||||
if config.poll.interval_s <= 0:
|
if config.poll.interval_s <= 0:
|
||||||
raise ConfigError(f"poll.interval_s: must be > 0, got {config.poll.interval_s}")
|
raise ConfigError(f"poll.interval_s: must be > 0, got {config.poll.interval_s}")
|
||||||
if config.poll.timeout_ms <= 0:
|
if config.poll.timeout_ms <= 0:
|
||||||
@@ -228,6 +346,20 @@ def _validate(config: Config) -> None:
|
|||||||
raise ConfigError(f"report.jsonl.max_mb: must be >= 1, got {config.report.jsonl.max_mb}")
|
raise ConfigError(f"report.jsonl.max_mb: must be >= 1, got {config.report.jsonl.max_mb}")
|
||||||
if config.report.jsonl.backups < 0:
|
if config.report.jsonl.backups < 0:
|
||||||
raise ConfigError(f"report.jsonl.backups: must be >= 0, got {config.report.jsonl.backups}")
|
raise ConfigError(f"report.jsonl.backups: must be >= 0, got {config.report.jsonl.backups}")
|
||||||
|
if config.report.textlog.max_mb < 1:
|
||||||
|
raise ConfigError(f"report.textlog.max_mb: must be >= 1, got {config.report.textlog.max_mb}")
|
||||||
|
if config.report.textlog.backups < 0:
|
||||||
|
raise ConfigError(f"report.textlog.backups: must be >= 0, got {config.report.textlog.backups}")
|
||||||
|
if config.report.http is not None:
|
||||||
|
if not config.report.http.url.startswith(("http://", "https://")):
|
||||||
|
raise ConfigError(
|
||||||
|
"report.http.url: must start with http:// or https://, got "
|
||||||
|
f"{config.report.http.url!r}"
|
||||||
|
)
|
||||||
|
if config.report.http.timeout_s <= 0:
|
||||||
|
raise ConfigError(
|
||||||
|
f"report.http.timeout_s: must be > 0, got {config.report.http.timeout_s}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def load_config(path: str | Path) -> Config:
|
def load_config(path: str | Path) -> Config:
|
||||||
@@ -253,6 +385,7 @@ def load_config(path: str | Path) -> Config:
|
|||||||
opc=_build_section(OpcConfig, data.get("opc"), "opc"),
|
opc=_build_section(OpcConfig, data.get("opc"), "opc"),
|
||||||
poll=_build_section(PollConfig, data.get("poll"), "poll"),
|
poll=_build_section(PollConfig, data.get("poll"), "poll"),
|
||||||
tags=_build_tags(data.get("tags")),
|
tags=_build_tags(data.get("tags")),
|
||||||
|
devices=_build_devices(data.get("devices")),
|
||||||
latency=_build_section(LatencyConfig, data.get("latency"), "latency"),
|
latency=_build_section(LatencyConfig, data.get("latency"), "latency"),
|
||||||
quality=_build_section(QualityConfig, data.get("quality"), "quality"),
|
quality=_build_section(QualityConfig, data.get("quality"), "quality"),
|
||||||
stale=_build_section(StaleConfig, data.get("stale"), "stale"),
|
stale=_build_section(StaleConfig, data.get("stale"), "stale"),
|
||||||
|
|||||||
@@ -45,6 +45,8 @@ class EventKind(str, Enum):
|
|||||||
FROZEN_VALUE = "FROZEN_VALUE" # D13
|
FROZEN_VALUE = "FROZEN_VALUE" # D13
|
||||||
CONTROL_LINK_DEGRADED = "CONTROL_LINK_DEGRADED" # D14
|
CONTROL_LINK_DEGRADED = "CONTROL_LINK_DEGRADED" # D14
|
||||||
HEARTBEAT = "HEARTBEAT" # D15
|
HEARTBEAT = "HEARTBEAT" # D15
|
||||||
|
DEVICE_LINK_DOWN = "DEVICE_LINK_DOWN" # D16
|
||||||
|
DEVICE_LINK_UP = "DEVICE_LINK_UP" # D16 recovery
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
|
|||||||
+276
-24
@@ -1,24 +1,30 @@
|
|||||||
"""Command-line interface (IMPLEMENTATION_PLAN §6.2).
|
"""Command-line interface (IMPLEMENTATION_PLAN §6.2).
|
||||||
|
|
||||||
Verbs: ``run``, ``probe``, ``check-config``, ``list-items`` plus the service
|
Verbs: ``run``, ``probe``, ``check-config``, ``list-items``, the service
|
||||||
verbs ``install`` / ``uninstall`` / ``start`` / ``stop``. Currently functional:
|
verbs ``install`` / ``uninstall`` / ``start`` / ``stop`` / ``status``, and
|
||||||
``check-config``, ``list-servers`` (discovery), ``list-items`` (address-space
|
the internal ``service`` verb — the SCM dispatch entry the install records
|
||||||
browsing: ``all`` / ``branches`` / ``tags``) and ``read`` / ``probe`` (one-shot
|
in the service ImagePath (the exe hosts itself; SPEC §6.4).
|
||||||
tag reads); the remaining verbs land with Phases 1–3.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
|
import os
|
||||||
import sys
|
import sys
|
||||||
|
import threading
|
||||||
from collections.abc import Sequence
|
from collections.abc import Sequence
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
from typing import TYPE_CHECKING
|
||||||
|
|
||||||
from opcda_diagnozer import __version__
|
from opcda_diagnozer import __version__
|
||||||
from opcda_diagnozer.config import ConfigError, load_config
|
from opcda_diagnozer.config import ConfigError, load_config
|
||||||
from opcda_diagnozer.opc.base import OpcError
|
from opcda_diagnozer.opc.base import OpcError
|
||||||
from opcda_diagnozer.opc.discovery import ServerInfo, list_servers
|
from opcda_diagnozer.opc.discovery import ServerInfo, list_servers
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from opcda_diagnozer.config import Config
|
||||||
|
from opcda_diagnozer.events import DiagEvent
|
||||||
|
|
||||||
DEFAULT_CONFIG = "config.yaml"
|
DEFAULT_CONFIG = "config.yaml"
|
||||||
|
|
||||||
|
|
||||||
@@ -34,13 +40,24 @@ def _default_config_path() -> str:
|
|||||||
return str(Path(sys.executable).resolve().parent / DEFAULT_CONFIG)
|
return str(Path(sys.executable).resolve().parent / DEFAULT_CONFIG)
|
||||||
return DEFAULT_CONFIG
|
return DEFAULT_CONFIG
|
||||||
|
|
||||||
_NOT_IMPLEMENTED: dict[str, str] = {
|
|
||||||
"run": "Phase 1 (core engine)",
|
def _scm_started_us() -> bool:
|
||||||
"install": "Phase 3 (Windows service)",
|
"""True when the SCM (not a human) launched this process.
|
||||||
"uninstall": "Phase 3 (Windows service)",
|
|
||||||
"start": "Phase 3 (Windows service)",
|
When the SCM starts the registered exe it passes the fixed argv shape
|
||||||
"stop": "Phase 3 (Windows service)",
|
``[exe, 'service']`` (the ``exeArgs`` recorded at install) with CWD =
|
||||||
}
|
System32. That exact combination is the dispatch signal: hand the
|
||||||
|
process over to the SCM dispatcher (``HandleCommandLine``), which blocks
|
||||||
|
until the service stops. Any other invocation is a CLI verb.
|
||||||
|
"""
|
||||||
|
argv = sys.argv
|
||||||
|
system32 = Path(os.environ.get("SystemRoot", r"C:\\Windows")) / "System32"
|
||||||
|
return (
|
||||||
|
sys.platform == "win32"
|
||||||
|
and len(argv) == 2
|
||||||
|
and argv[1] == "service"
|
||||||
|
and Path.cwd() == system32
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _add_config_option(parser: argparse.ArgumentParser) -> None:
|
def _add_config_option(parser: argparse.ArgumentParser) -> None:
|
||||||
@@ -81,8 +98,12 @@ def _build_parser() -> argparse.ArgumentParser:
|
|||||||
parser.add_argument("--version", action="version", version=f"%(prog)s {__version__}")
|
parser.add_argument("--version", action="version", version=f"%(prog)s {__version__}")
|
||||||
sub = parser.add_subparsers(dest="verb", metavar="verb", required=True)
|
sub = parser.add_subparsers(dest="verb", metavar="verb", required=True)
|
||||||
|
|
||||||
p = sub.add_parser("run", help="run in the foreground (console)")
|
p = sub.add_parser(
|
||||||
|
"run",
|
||||||
|
help="run the monitoring loop in the foreground (Ctrl+C stops)",
|
||||||
|
)
|
||||||
_add_config_option(p)
|
_add_config_option(p)
|
||||||
|
_add_server_overrides(p)
|
||||||
|
|
||||||
p = sub.add_parser(
|
p = sub.add_parser(
|
||||||
"read",
|
"read",
|
||||||
@@ -188,18 +209,29 @@ def _build_parser() -> argparse.ArgumentParser:
|
|||||||
help="emit JSON objects (for scripts) instead of a tree/list",
|
help="emit JSON objects (for scripts) instead of a tree/list",
|
||||||
)
|
)
|
||||||
|
|
||||||
p = sub.add_parser("install", help="install as a Windows service (requires admin)")
|
p = sub.add_parser(
|
||||||
|
"install",
|
||||||
|
help="install as a Windows service (auto-start; requires admin)",
|
||||||
|
)
|
||||||
_add_config_option(p)
|
_add_config_option(p)
|
||||||
p.add_argument(
|
p.add_argument(
|
||||||
"--account",
|
"--account",
|
||||||
metavar="DOMAIN\\USER",
|
metavar="DOMAIN\\USER",
|
||||||
help="dedicated service account (required for DCOM authentication, SPECIFICATION §11)",
|
help="dedicated service account (LocalSystem default; a real account "
|
||||||
|
"is required for remote DCOM authentication, SPECIFICATION §11)",
|
||||||
|
)
|
||||||
|
p.add_argument(
|
||||||
|
"--password",
|
||||||
|
metavar="PASSWORD",
|
||||||
|
help="password for --account (prompt-free installs; omit when the "
|
||||||
|
"account is LocalSystem)",
|
||||||
)
|
)
|
||||||
|
|
||||||
for verb, help_text in (
|
for verb, help_text in (
|
||||||
("uninstall", "remove the Windows service"),
|
("uninstall", "remove the Windows service registration (requires admin)"),
|
||||||
("start", "start the Windows service"),
|
("start", "start the Windows service (requires admin)"),
|
||||||
("stop", "stop the Windows service"),
|
("stop", "stop the Windows service (requires admin)"),
|
||||||
|
("status", "print the service state (works without admin)"),
|
||||||
):
|
):
|
||||||
sub.add_parser(verb, help=help_text)
|
sub.add_parser(verb, help=help_text)
|
||||||
|
|
||||||
@@ -217,6 +249,12 @@ def _cmd_check_config(config_path: str) -> int:
|
|||||||
print("check-config: OK")
|
print("check-config: OK")
|
||||||
print(f" server: {config.opc.server!r}{host}")
|
print(f" server: {config.opc.server!r}{host}")
|
||||||
print(f" tags: {len(config.tags)} ({expect_change} with expect_change)")
|
print(f" tags: {len(config.tags)} ({expect_change} with expect_change)")
|
||||||
|
if config.devices:
|
||||||
|
print(f" devices: {len(config.devices)} link indicator(s):")
|
||||||
|
for device in config.devices:
|
||||||
|
print(f" {device.name} <- {device.tag}")
|
||||||
|
if not config.tags and config.devices:
|
||||||
|
print(" note: no surveillance tags - monitoring device links only")
|
||||||
print(f" poll: every {config.poll.interval_s} s, hang flagged after {config.poll.timeout_ms} ms")
|
print(f" poll: every {config.poll.interval_s} s, hang flagged after {config.poll.timeout_ms} ms")
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
@@ -546,11 +584,223 @@ def _cmd_probe(
|
|||||||
return _cmd_read(config_path, server, host, (), source, as_json=False)
|
return _cmd_read(config_path, server, host, (), source, as_json=False)
|
||||||
|
|
||||||
|
|
||||||
|
def _build_reporters(config: Config):
|
||||||
|
"""Assemble the reporter fan-out from the config (spec §9).
|
||||||
|
|
||||||
|
Console is always on in the foreground ``run``; JSONL, the text anomaly
|
||||||
|
log and the Windows Event Log follow their config sections; the webhook
|
||||||
|
only when ``report.http`` is configured. Returns ``(fan_out, close)``
|
||||||
|
where ``close`` must be called exactly once on shutdown.
|
||||||
|
"""
|
||||||
|
from opcda_diagnozer.report.base import FanOut
|
||||||
|
from opcda_diagnozer.report.jsonl import JsonlReporter, config_digest
|
||||||
|
from opcda_diagnozer.report.textlog import TextLogReporter
|
||||||
|
|
||||||
|
sinks = []
|
||||||
|
|
||||||
|
def console_reporter(event: DiagEvent) -> None:
|
||||||
|
# Console = the human view (spec §9): clear text, status NAME (0xNN)
|
||||||
|
line = f"{event.time_utc.strftime('%Y-%m-%d %H:%M:%S.%f')[:-3]} UTC {event.severity.value:<8} {event.kind.value}"
|
||||||
|
extras = " ".join(f"{k}={v}" for k, v in event.detail.items())
|
||||||
|
print(f"{line} {extras}" if extras else line, flush=True)
|
||||||
|
|
||||||
|
sinks.append(console_reporter)
|
||||||
|
sinks.append(
|
||||||
|
JsonlReporter(
|
||||||
|
config.report.jsonl.dir,
|
||||||
|
max_mb=config.report.jsonl.max_mb,
|
||||||
|
backups=config.report.jsonl.backups,
|
||||||
|
digest=config_digest(config),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
sinks.append(
|
||||||
|
TextLogReporter(
|
||||||
|
config.report.textlog.dir,
|
||||||
|
max_mb=config.report.textlog.max_mb,
|
||||||
|
backups=config.report.textlog.backups,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if sys.platform == "win32":
|
||||||
|
from opcda_diagnozer.report.eventlog import EventLogReporter
|
||||||
|
|
||||||
|
sinks.append(
|
||||||
|
EventLogReporter(
|
||||||
|
source=config.report.eventlog.source,
|
||||||
|
min_severity=config.report.eventlog.min_severity,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if config.report.http is not None:
|
||||||
|
from opcda_diagnozer.report.http import HttpReporter
|
||||||
|
|
||||||
|
sinks.append(
|
||||||
|
HttpReporter(
|
||||||
|
url=config.report.http.url,
|
||||||
|
bearer_token=config.report.http.bearer_token,
|
||||||
|
timeout_s=config.report.http.timeout_s,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
fan_out = FanOut(tuple(sinks), error=lambda msg: print(f"run: {msg}", file=sys.stderr, flush=True))
|
||||||
|
return fan_out
|
||||||
|
|
||||||
|
|
||||||
|
def _cmd_run(config_path: str, server: str | None, host: str | None) -> int:
|
||||||
|
"""Foreground monitoring loop: poll → detect → report (Ctrl+C stops)."""
|
||||||
|
import signal
|
||||||
|
|
||||||
|
from opcda_diagnozer.events import DiagEvent
|
||||||
|
from opcda_diagnozer.monitor.poller import Poller
|
||||||
|
from opcda_diagnozer.opc.automation import AutomationOpcClient
|
||||||
|
|
||||||
|
try:
|
||||||
|
config = load_config(config_path)
|
||||||
|
except ConfigError as exc:
|
||||||
|
print(f"run: FAILED\n {exc}", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
# --server/--host override opc.* for this run (same as the other verbs)
|
||||||
|
if server is not None or host is not None:
|
||||||
|
from dataclasses import replace
|
||||||
|
|
||||||
|
from opcda_diagnozer.config import OpcConfig
|
||||||
|
|
||||||
|
opc = replace(
|
||||||
|
config.opc,
|
||||||
|
server=server or config.opc.server,
|
||||||
|
host=config.opc.host if host is None else host,
|
||||||
|
)
|
||||||
|
config = replace(config, opc=opc)
|
||||||
|
|
||||||
|
def client_factory() -> AutomationOpcClient:
|
||||||
|
# Built on the worker thread: CoInitialize is per-thread (spec §10.3).
|
||||||
|
return AutomationOpcClient(
|
||||||
|
server=config.opc.server,
|
||||||
|
host=config.opc.host,
|
||||||
|
connect_timeout_s=config.opc.connect_timeout_s,
|
||||||
|
)
|
||||||
|
|
||||||
|
reporters = _build_reporters(config)
|
||||||
|
poller = Poller(client_factory, config, reporters.emit)
|
||||||
|
stop_requested = threading.Event()
|
||||||
|
|
||||||
|
def _on_ctrl_c(signum, frame) -> None: # noqa: ANN001, ARG001
|
||||||
|
if stop_requested.is_set():
|
||||||
|
raise KeyboardInterrupt # second Ctrl+C aborts immediately
|
||||||
|
stop_requested.set()
|
||||||
|
poller.request_stop()
|
||||||
|
print("\nrun: stop requested - finishing current poll ...", file=sys.stderr, flush=True)
|
||||||
|
|
||||||
|
signal.signal(signal.SIGINT, _on_ctrl_c)
|
||||||
|
try:
|
||||||
|
poller.run_forever()
|
||||||
|
finally:
|
||||||
|
reporters.close()
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
def _cmd_service_verb(verb: str, config_path: str | None = None,
|
||||||
|
account: str | None = None, password: str | None = None) -> int:
|
||||||
|
"""install / uninstall / start / stop / status — SCM management verbs.
|
||||||
|
|
||||||
|
All the SCM work lives in ``opcda_diagnozer.service``; this function
|
||||||
|
owns the UX: clear-text success/failure lines and actionable error
|
||||||
|
mapping (the classic pitfalls: not elevated, service not installed,
|
||||||
|
service not running). Exit codes: 0 ok, 2 environment error (the CLI
|
||||||
|
convention from the other verbs).
|
||||||
|
"""
|
||||||
|
from opcda_diagnozer import service as svc
|
||||||
|
|
||||||
|
try:
|
||||||
|
if verb == "install":
|
||||||
|
# Validate the config BEFORE registering anything: a service
|
||||||
|
# that cannot load its config is a support call, not a start.
|
||||||
|
resolved = config_path or _default_config_path()
|
||||||
|
try:
|
||||||
|
load_config(resolved)
|
||||||
|
except ConfigError as exc:
|
||||||
|
print(
|
||||||
|
f"install: FAILED - the config is invalid (fix before "
|
||||||
|
f"installing):\n {exc}",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
return 2
|
||||||
|
svc.install_service(resolved, account=account, password=password)
|
||||||
|
print(f"install: OK - service '{svc.SERVICE_NAME}' registered")
|
||||||
|
print(f" config: {resolved}")
|
||||||
|
print(" auto-start: yes (SCM recovery: restart on failure)")
|
||||||
|
print(f" start it with: {Path(sys.executable).name} start")
|
||||||
|
return 0
|
||||||
|
if verb == "uninstall":
|
||||||
|
svc.uninstall_service()
|
||||||
|
print(f"uninstall: OK - service '{svc.SERVICE_NAME}' removed")
|
||||||
|
print(" (logs and config are untouched)")
|
||||||
|
return 0
|
||||||
|
if verb == "start":
|
||||||
|
svc.start_service()
|
||||||
|
print(f"start: OK - service '{svc.SERVICE_NAME}' start requested")
|
||||||
|
return 0
|
||||||
|
if verb == "stop":
|
||||||
|
svc.stop_service()
|
||||||
|
print(f"stop: OK - service '{svc.SERVICE_NAME}' stop requested")
|
||||||
|
return 0
|
||||||
|
if verb == "status":
|
||||||
|
print(f"{svc.SERVICE_NAME}: {svc.service_status()}")
|
||||||
|
return 0
|
||||||
|
except Exception as exc: # noqa: BLE001 — one actionable line, not a traceback
|
||||||
|
return _service_verb_error(verb, exc)
|
||||||
|
print(f"{verb}: not implemented", file=sys.stderr) # pragma: no cover
|
||||||
|
return 1
|
||||||
|
|
||||||
|
|
||||||
|
def _service_verb_error(verb: str, exc: Exception) -> int:
|
||||||
|
"""Map SCM/registry errors to actionable one-liners (exit 2)."""
|
||||||
|
winerror = getattr(exc, "winerror", None)
|
||||||
|
winerror = winerror if isinstance(winerror, int) else None
|
||||||
|
hints: dict[int, str] = {
|
||||||
|
5: ( # ERROR_ACCESS_DENIED
|
||||||
|
"access denied - run this from an elevated (administrator) "
|
||||||
|
"terminal"
|
||||||
|
),
|
||||||
|
1060: ( # ERROR_SERVICE_DOES_NOT_EXIST
|
||||||
|
f"the service is not installed - run 'install' first"
|
||||||
|
),
|
||||||
|
1062: ( # ERROR_SERVICE_NOT_ACTIVE
|
||||||
|
"the service is not running - nothing to stop"
|
||||||
|
),
|
||||||
|
1072: ( # ERROR_SERVICE_MARKED_FOR_DELETE
|
||||||
|
"the service is marked for deletion - it disappears once no "
|
||||||
|
"process holds a handle to it (close services.msc, then retry)"
|
||||||
|
),
|
||||||
|
1073: ( # ERROR_SERVICE_ALREADY_RUNNING
|
||||||
|
"the service is already running"
|
||||||
|
),
|
||||||
|
1056: ( # ERROR_SERVICE_ALREADY_RUNNING via StartService
|
||||||
|
"an instance of the service is already running"
|
||||||
|
),
|
||||||
|
}
|
||||||
|
hint = hints.get(winerror) if winerror is not None else None
|
||||||
|
detail = getattr(exc, "strerror", None) or str(exc)
|
||||||
|
where = f" ({hint})" if hint else ""
|
||||||
|
print(f"{verb}: FAILED{where}\n {detail}", file=sys.stderr)
|
||||||
|
if hint is None and winerror is not None:
|
||||||
|
print(
|
||||||
|
f" (Windows error {winerror}; see 'net helpmsg {winerror}')",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
|
||||||
def main(argv: Sequence[str] | None = None) -> int:
|
def main(argv: Sequence[str] | None = None) -> int:
|
||||||
"""CLI entry point (console script + ``python -m opcda_diagnozer``)."""
|
"""CLI entry point (console script + ``python -m opcda_diagnozer``)."""
|
||||||
|
# SCM dispatch: when the SCM starts the registered exe it runs with the
|
||||||
|
# fixed argv + CWD=System32 shape — hand over to the service host.
|
||||||
|
if argv is None and _scm_started_us():
|
||||||
|
from opcda_diagnozer.service import handle_service_command_line
|
||||||
|
|
||||||
|
return handle_service_command_line(sys.argv)
|
||||||
args = _build_parser().parse_args(argv)
|
args = _build_parser().parse_args(argv)
|
||||||
if args.verb == "check-config":
|
if args.verb == "check-config":
|
||||||
return _cmd_check_config(args.config)
|
return _cmd_check_config(args.config)
|
||||||
|
if args.verb == "run":
|
||||||
|
return _cmd_run(args.config, args.server, args.host)
|
||||||
if args.verb == "list-servers":
|
if args.verb == "list-servers":
|
||||||
return _cmd_list_servers(args.host, args.source, args.json)
|
return _cmd_list_servers(args.host, args.source, args.json)
|
||||||
if args.verb == "list-items":
|
if args.verb == "list-items":
|
||||||
@@ -563,12 +813,14 @@ def main(argv: Sequence[str] | None = None) -> int:
|
|||||||
)
|
)
|
||||||
if args.verb == "probe":
|
if args.verb == "probe":
|
||||||
return _cmd_probe(args.config, args.server, args.host, args.source)
|
return _cmd_probe(args.config, args.server, args.host, args.source)
|
||||||
phase = _NOT_IMPLEMENTED[args.verb]
|
if args.verb in ("install", "uninstall", "start", "stop", "status"):
|
||||||
print(
|
return _cmd_service_verb(
|
||||||
f"{args.verb}: not implemented yet — planned for {phase} "
|
args.verb,
|
||||||
"(see docs/IMPLEMENTATION_PLAN.md)",
|
config_path=getattr(args, "config", None),
|
||||||
file=sys.stderr,
|
account=getattr(args, "account", None),
|
||||||
)
|
password=getattr(args, "password", None),
|
||||||
|
)
|
||||||
|
print(f"{args.verb}: not implemented yet", file=sys.stderr)
|
||||||
return 1
|
return 1
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -1,17 +1,28 @@
|
|||||||
"""Detector rules D1–D15 (SPECIFICATION §6).
|
"""Detector rules D4–D16 (SPECIFICATION §6).
|
||||||
|
|
||||||
Phase 1 (IMPLEMENTATION_PLAN §4.6): each detector becomes a small pure class
|
Each detector is a small pure class fed one :class:`PollOutcome` per poll
|
||||||
over :class:`PollOutcome` sequences with an injected clock; a rolling window
|
cycle plus an injected clock; it returns zero or more :class:`Detection`
|
||||||
(deque, size ``window_polls``) powers p95/streak/flapping logic. Severity
|
objects (events to report). The poller owns all timing (sleeping, RTT
|
||||||
mapping lives in SPECIFICATION §6.
|
measurement) — detectors only see data, so every rule is testable without COM
|
||||||
|
against scripted outcomes with a fake clock.
|
||||||
|
|
||||||
|
Stateful detectors keep their own per-tag windows (deque bounded by
|
||||||
|
``window_polls``); none of them ever block.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
from dataclasses import dataclass
|
from collections import deque
|
||||||
from datetime import datetime
|
from dataclasses import dataclass, field
|
||||||
|
from datetime import datetime, timedelta
|
||||||
|
|
||||||
|
from opcda_diagnozer.config import Config
|
||||||
|
from opcda_diagnozer.events import EventKind, Severity
|
||||||
from opcda_diagnozer.opc.base import ReadResult
|
from opcda_diagnozer.opc.base import ReadResult
|
||||||
|
from opcda_diagnozer.quality import QualityClass, decode_quality, status_text
|
||||||
|
|
||||||
|
#: D14 mass degradation: >= this fraction of monitored tags BAD at once.
|
||||||
|
MASS_BAD_FRACTION = 0.5
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
@@ -21,4 +32,468 @@ class PollOutcome:
|
|||||||
started_at: datetime # local receive time when the read call started (wall clock)
|
started_at: datetime # local receive time when the read call started (wall clock)
|
||||||
rtt_ms: float # measured round-trip time of the read call
|
rtt_ms: float # measured round-trip time of the read call
|
||||||
results: tuple[ReadResult, ...] = () # per-tag results; empty if the call itself failed
|
results: tuple[ReadResult, ...] = () # per-tag results; empty if the call itself failed
|
||||||
error: str | None = None # normalized OpcError message when the whole poll failed (D1/D2/D7)
|
error: str | None = None # normalized OpcError message when the whole poll failed (D1/D2/D7)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Detection:
|
||||||
|
"""One detector finding: the event kind/severity/detail the poller reports."""
|
||||||
|
|
||||||
|
kind: EventKind
|
||||||
|
severity: Severity
|
||||||
|
detail: dict = field(default_factory=dict)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def is_anomaly(self) -> bool:
|
||||||
|
"""True for every kind the text anomaly log shows (spec §9.1).
|
||||||
|
|
||||||
|
Heartbeats and plain CONNECTED are lifecycle noise, not anomalies.
|
||||||
|
"""
|
||||||
|
return self.kind not in (EventKind.HEARTBEAT, EventKind.CONNECTED)
|
||||||
|
|
||||||
|
|
||||||
|
def _tag_quality_class(result: ReadResult) -> QualityClass | None:
|
||||||
|
"""Quality class of a read; None when the tag itself failed (no quality)."""
|
||||||
|
if result.error is not None:
|
||||||
|
return None
|
||||||
|
return decode_quality(result.quality_raw).quality_class
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# D4/D5/D6 — latency
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
class LatencyDetector:
|
||||||
|
"""D4 spike, D5 sustained-slow streak, D6 p95 trend over the window."""
|
||||||
|
|
||||||
|
def __init__(self, config: Config) -> None:
|
||||||
|
self._warn_ms = config.latency.warn_ms
|
||||||
|
self._critical_ms = config.latency.critical_ms
|
||||||
|
self._streak_limit = config.latency.slow_poll_streak
|
||||||
|
self._window: deque[float] = deque(maxlen=config.latency.window_polls)
|
||||||
|
self._slow_streak = 0
|
||||||
|
|
||||||
|
def observe(self, outcome: PollOutcome) -> list[Detection]:
|
||||||
|
if outcome.error is not None:
|
||||||
|
self._slow_streak = 0 # failed polls carry no RTT evidence
|
||||||
|
self._window.append(outcome.rtt_ms) # still measured; the hang is D7's
|
||||||
|
return []
|
||||||
|
self._window.append(outcome.rtt_ms)
|
||||||
|
detections: list[Detection] = []
|
||||||
|
if outcome.rtt_ms > self._warn_ms:
|
||||||
|
detections.append(
|
||||||
|
Detection(
|
||||||
|
kind=EventKind.SLOW_POLL,
|
||||||
|
severity=Severity.WARN,
|
||||||
|
detail={"rtt_ms": round(outcome.rtt_ms, 1), "warn_ms": self._warn_ms},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self._slow_streak += 1
|
||||||
|
if self._slow_streak >= self._streak_limit:
|
||||||
|
detections.append(
|
||||||
|
Detection(
|
||||||
|
kind=EventKind.SLOW_POLL_STREAK,
|
||||||
|
severity=Severity.CRITICAL,
|
||||||
|
detail={
|
||||||
|
"rtt_ms": round(outcome.rtt_ms, 1),
|
||||||
|
"streak": self._slow_streak,
|
||||||
|
"warn_ms": self._warn_ms,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
self._slow_streak = 0
|
||||||
|
p95 = self._p95()
|
||||||
|
if p95 is not None and p95 > self._warn_ms:
|
||||||
|
detections.append(
|
||||||
|
Detection(
|
||||||
|
kind=EventKind.LATENCY_TREND,
|
||||||
|
severity=Severity.WARN,
|
||||||
|
detail={"rtt_p95_ms": round(p95, 1), "warn_ms": self._warn_ms},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return detections
|
||||||
|
|
||||||
|
def _p95(self) -> float | None:
|
||||||
|
if not self._window:
|
||||||
|
return None
|
||||||
|
ordered = sorted(self._window)
|
||||||
|
# nearest-rank p95 of the rolling window
|
||||||
|
rank = max(1, round(0.95 * len(ordered)))
|
||||||
|
return ordered[rank - 1]
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# D8/D9/D10/D11 — per-tag quality
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class _TagQualityState:
|
||||||
|
"""Per-tag tracking for the quality detectors."""
|
||||||
|
|
||||||
|
last_class: QualityClass | None = None
|
||||||
|
bad_since: datetime | None = None # D10 persistence clock start
|
||||||
|
bad_reported: bool = False
|
||||||
|
persistent_reported: bool = False
|
||||||
|
transition_times: deque[datetime] = field(default_factory=lambda: deque())
|
||||||
|
|
||||||
|
|
||||||
|
class QualityDetector:
|
||||||
|
"""D8/D9 transitions, D10 persistence, D11 flapping — per monitored tag.
|
||||||
|
|
||||||
|
Tag state initializes lazily from the first observed reading; transitions
|
||||||
|
are computed from the *previous class*, so a tag starting BAD does not
|
||||||
|
fire a transition until it first becomes GOOD (no synthetic baseline).
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, config: Config) -> None:
|
||||||
|
self._persistent_bad_s = config.quality.persistent_bad_s
|
||||||
|
self._flapping_limit = config.quality.flapping_transitions_per_window
|
||||||
|
self._flapping_window = timedelta(seconds=config.quality.flapping_window_s)
|
||||||
|
self._tags: dict[str, _TagQualityState] = {}
|
||||||
|
|
||||||
|
def observe(self, outcome: PollOutcome) -> list[Detection]:
|
||||||
|
detections: list[Detection] = []
|
||||||
|
at = outcome.started_at
|
||||||
|
for result in outcome.results:
|
||||||
|
state = self._tags.setdefault(result.tag, _TagQualityState())
|
||||||
|
quality = decode_quality(result.quality_raw) if result.error is None else None
|
||||||
|
new_class = quality.quality_class if quality is not None else None
|
||||||
|
self._on_transition(state, result, quality, at, detections)
|
||||||
|
self._on_persistence(state, result, at, detections)
|
||||||
|
state.last_class = new_class
|
||||||
|
return detections
|
||||||
|
|
||||||
|
def _on_transition(
|
||||||
|
self,
|
||||||
|
state: _TagQualityState,
|
||||||
|
result: ReadResult,
|
||||||
|
quality,
|
||||||
|
at: datetime,
|
||||||
|
out: list[Detection],
|
||||||
|
) -> None:
|
||||||
|
new_class = quality.quality_class if quality is not None else None
|
||||||
|
old = state.last_class
|
||||||
|
if old is None or new_class is None or old == new_class:
|
||||||
|
return # first reading or unreadable tag: no synthetic transition
|
||||||
|
state.transition_times.append(at)
|
||||||
|
state.transition_times = deque(
|
||||||
|
t for t in state.transition_times if at - t <= self._flapping_window
|
||||||
|
)
|
||||||
|
if new_class is QualityClass.BAD:
|
||||||
|
out.append(
|
||||||
|
Detection(
|
||||||
|
kind=EventKind.QUALITY_BAD,
|
||||||
|
severity=Severity.WARN,
|
||||||
|
detail={
|
||||||
|
"tag": result.tag,
|
||||||
|
"quality": status_text(result.quality_raw),
|
||||||
|
"previous_quality": old.value,
|
||||||
|
"value": result.value,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
elif new_class is QualityClass.UNCERTAIN:
|
||||||
|
out.append(
|
||||||
|
Detection(
|
||||||
|
kind=EventKind.QUALITY_UNCERTAIN,
|
||||||
|
severity=Severity.INFO,
|
||||||
|
detail={
|
||||||
|
"tag": result.tag,
|
||||||
|
"quality": status_text(result.quality_raw),
|
||||||
|
"previous_quality": old.value,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
if len(state.transition_times) > self._flapping_limit:
|
||||||
|
out.append(
|
||||||
|
Detection(
|
||||||
|
kind=EventKind.QUALITY_FLAPPING,
|
||||||
|
severity=Severity.WARN,
|
||||||
|
detail={
|
||||||
|
"tag": result.tag,
|
||||||
|
"transitions_in_window": len(state.transition_times),
|
||||||
|
"window_s": self._flapping_window.total_seconds(),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
def _on_persistence(
|
||||||
|
self,
|
||||||
|
state: _TagQualityState,
|
||||||
|
result: ReadResult,
|
||||||
|
at: datetime,
|
||||||
|
out: list[Detection],
|
||||||
|
) -> None:
|
||||||
|
bad = _tag_quality_class(result) is QualityClass.BAD
|
||||||
|
if bad:
|
||||||
|
if state.bad_since is None:
|
||||||
|
state.bad_since = at
|
||||||
|
state.bad_reported = True
|
||||||
|
been_bad_s = (at - state.bad_since).total_seconds()
|
||||||
|
if been_bad_s >= self._persistent_bad_s and not state.persistent_reported:
|
||||||
|
state.persistent_reported = True
|
||||||
|
out.append(
|
||||||
|
Detection(
|
||||||
|
kind=EventKind.QUALITY_BAD_PERSISTENT,
|
||||||
|
severity=Severity.WARN,
|
||||||
|
detail={
|
||||||
|
"tag": result.tag,
|
||||||
|
"bad_for_s": round(been_bad_s, 1),
|
||||||
|
"threshold_s": self._persistent_bad_s,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
state.bad_since = None
|
||||||
|
state.bad_reported = False
|
||||||
|
state.persistent_reported = False
|
||||||
|
|
||||||
|
def tags_bad(self) -> int:
|
||||||
|
"""Monitored tags currently BAD — feeds the heartbeat (D15) and D14."""
|
||||||
|
# current-ness approximation: a tag counts if its last class was BAD
|
||||||
|
return sum(1 for s in self._tags.values() if s.last_class is QualityClass.BAD)
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# D12/D13 — staleness & frozen values
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
class StaleDetector:
|
||||||
|
"""D12 stale device timestamps; D13 frozen values (``expect_change`` tags).
|
||||||
|
|
||||||
|
Both compare the device time against the local receive clock (the spec's
|
||||||
|
``max_age_s`` tolerance absorbs reasonable clock skew; NTP on both hosts
|
||||||
|
is the real fix — spec §12 row 9).
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, config: Config) -> None:
|
||||||
|
self._max_age = timedelta(seconds=config.stale.max_age_s)
|
||||||
|
self._enabled = config.stale.enabled
|
||||||
|
self._expect_change = {t.name for t in config.tags if t.expect_change}
|
||||||
|
# last (value, device_time) per tag for the frozen check
|
||||||
|
self._last_seen: dict[str, tuple[object, datetime]] = {}
|
||||||
|
|
||||||
|
def observe(self, outcome: PollOutcome) -> list[Detection]:
|
||||||
|
if not self._enabled or outcome.error is not None:
|
||||||
|
return []
|
||||||
|
detections: list[Detection] = []
|
||||||
|
at = outcome.started_at
|
||||||
|
for result in outcome.results:
|
||||||
|
if result.error is not None or result.device_time is None:
|
||||||
|
continue
|
||||||
|
if _tag_quality_class(result) is not QualityClass.GOOD:
|
||||||
|
continue # staleness while quality still reports GOOD (spec D12)
|
||||||
|
age = at - result.device_time
|
||||||
|
if age > self._max_age:
|
||||||
|
detections.append(
|
||||||
|
Detection(
|
||||||
|
kind=EventKind.STALE_TIMESTAMP,
|
||||||
|
severity=Severity.WARN,
|
||||||
|
detail={
|
||||||
|
"tag": result.tag,
|
||||||
|
"device_time_utc": result.device_time.isoformat(),
|
||||||
|
"age_s": round(age.total_seconds(), 1),
|
||||||
|
"max_age_s": self._max_age.total_seconds(),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self._frozen_check(result, at, detections)
|
||||||
|
return detections
|
||||||
|
|
||||||
|
def _frozen_check(
|
||||||
|
self, result: ReadResult, at: datetime, out: list[Detection]
|
||||||
|
) -> None:
|
||||||
|
if result.tag not in self._expect_change:
|
||||||
|
return
|
||||||
|
previous = self._last_seen.get(result.tag)
|
||||||
|
self._last_seen[result.tag] = (result.value, result.device_time)
|
||||||
|
if previous is None or previous[1] is None or result.device_time is None:
|
||||||
|
return
|
||||||
|
same_value = str(previous[0]) == str(result.value)
|
||||||
|
same_time = previous[1] == result.device_time
|
||||||
|
if same_value and same_time:
|
||||||
|
gap = at - result.device_time
|
||||||
|
if gap > self._max_age:
|
||||||
|
out.append(
|
||||||
|
Detection(
|
||||||
|
kind=EventKind.FROZEN_VALUE,
|
||||||
|
severity=Severity.WARN,
|
||||||
|
detail={
|
||||||
|
"tag": result.tag,
|
||||||
|
"value": result.value,
|
||||||
|
"device_time_utc": result.device_time.isoformat(),
|
||||||
|
"unchanged_for_s": round(gap.total_seconds(), 1),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# D14 — mass quality degradation
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
class MassBadDetector:
|
||||||
|
"""D14: >= half of monitored tags BAD simultaneously while connection OK."""
|
||||||
|
|
||||||
|
def __init__(self, config: Config | None = None) -> None:
|
||||||
|
# config reserved for a future threshold key; the fraction is fixed
|
||||||
|
# by the spec (MASS_BAD_FRACTION).
|
||||||
|
self._reported = False
|
||||||
|
|
||||||
|
def observe(self, outcome: PollOutcome) -> list[Detection]:
|
||||||
|
if outcome.error is not None:
|
||||||
|
self._reported = False # reconnect resets the mass-degradation latch
|
||||||
|
return []
|
||||||
|
total = len(outcome.results)
|
||||||
|
if total == 0:
|
||||||
|
return []
|
||||||
|
bad = sum(1 for r in outcome.results if _tag_quality_class(r) is QualityClass.BAD)
|
||||||
|
degraded = bad >= MASS_BAD_FRACTION * total
|
||||||
|
detections: list[Detection] = []
|
||||||
|
if degraded and not self._reported:
|
||||||
|
detections.append(
|
||||||
|
Detection(
|
||||||
|
kind=EventKind.CONTROL_LINK_DEGRADED,
|
||||||
|
severity=Severity.CRITICAL,
|
||||||
|
detail={"tags_bad": bad, "tags_total": total},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self._reported = True
|
||||||
|
elif not degraded and self._reported:
|
||||||
|
self._reported = False # recovered; next degradation reports again
|
||||||
|
return detections
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# D16 — device-link indicators
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
def _rendered(value: object) -> str:
|
||||||
|
"""Value as the comparison string (config ok_values normalization twin)."""
|
||||||
|
if isinstance(value, bool):
|
||||||
|
return "true" if value else "false"
|
||||||
|
return str(value).strip()
|
||||||
|
|
||||||
|
|
||||||
|
class DeviceLinkDetector:
|
||||||
|
"""D16: per-device link state from configured indicator tags (spec §6.1).
|
||||||
|
|
||||||
|
Edge-level by design: one ``DEVICE_LINK_DOWN`` when a device leaves its
|
||||||
|
normal state (GOOD + value in ``ok_values``), one ``DEVICE_LINK_UP`` when
|
||||||
|
it returns — flapping links produce transitions, not a stream of events.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, config: Config) -> None:
|
||||||
|
self._devices: dict[str, dict] = {
|
||||||
|
device.tag: {"name": device.name, "ok": set(device.ok_values), "down": False}
|
||||||
|
for device in config.devices
|
||||||
|
}
|
||||||
|
|
||||||
|
def observe(self, outcome: PollOutcome) -> list[Detection]:
|
||||||
|
if outcome.error is not None:
|
||||||
|
return [] # whole-poll failure is D1's business, not per-device
|
||||||
|
detections: list[Detection] = []
|
||||||
|
for result in outcome.results:
|
||||||
|
entry = self._devices.get(result.tag)
|
||||||
|
if entry is None:
|
||||||
|
continue
|
||||||
|
quality = (
|
||||||
|
decode_quality(result.quality_raw)
|
||||||
|
if result.error is None
|
||||||
|
else None
|
||||||
|
)
|
||||||
|
down = result.error is not None
|
||||||
|
reason = None
|
||||||
|
if not down:
|
||||||
|
if quality is None or quality.quality_class is not QualityClass.GOOD:
|
||||||
|
down = True
|
||||||
|
reason = (
|
||||||
|
"quality " + status_text(result.quality_raw)
|
||||||
|
if quality is not None
|
||||||
|
else "unreadable tag: " + (result.error or "?")
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
rendered = _rendered(result.value)
|
||||||
|
if rendered not in entry["ok"]:
|
||||||
|
down = True
|
||||||
|
reason = f"read {rendered!r}, expected one of {sorted(entry['ok'])}"
|
||||||
|
if down and not entry["down"]:
|
||||||
|
entry["down"] = True
|
||||||
|
detail = {
|
||||||
|
"device": entry["name"],
|
||||||
|
"indicator_tag": result.tag,
|
||||||
|
"value": result.value,
|
||||||
|
"quality": status_text(result.quality_raw) if result.error is None else None,
|
||||||
|
}
|
||||||
|
if reason:
|
||||||
|
detail["reason"] = reason
|
||||||
|
detections.append(
|
||||||
|
Detection(kind=EventKind.DEVICE_LINK_DOWN, severity=Severity.CRITICAL, detail=detail)
|
||||||
|
)
|
||||||
|
elif not down and entry["down"]:
|
||||||
|
entry["down"] = False
|
||||||
|
detections.append(
|
||||||
|
Detection(
|
||||||
|
kind=EventKind.DEVICE_LINK_UP,
|
||||||
|
severity=Severity.INFO,
|
||||||
|
detail={"device": entry["name"], "indicator_tag": result.tag},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return detections
|
||||||
|
|
||||||
|
def devices_down(self) -> int:
|
||||||
|
return sum(1 for e in self._devices.values() if e["down"])
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# The bundle the poller feeds every cycle
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
class DetectorSuite:
|
||||||
|
"""All per-poll detectors run in a fixed order; one call per outcome."""
|
||||||
|
|
||||||
|
def __init__(self, config: Config) -> None:
|
||||||
|
self.latency = LatencyDetector(config)
|
||||||
|
self.quality = QualityDetector(config)
|
||||||
|
self.stale = StaleDetector(config)
|
||||||
|
self.mass_bad = MassBadDetector(config)
|
||||||
|
self.device_link = DeviceLinkDetector(config)
|
||||||
|
|
||||||
|
def observe(self, outcome: PollOutcome) -> list[Detection]:
|
||||||
|
detections: list[Detection] = []
|
||||||
|
detections.extend(self.latency.observe(outcome))
|
||||||
|
detections.extend(self.quality.observe(outcome))
|
||||||
|
detections.extend(self.stale.observe(outcome))
|
||||||
|
detections.extend(self.mass_bad.observe(outcome))
|
||||||
|
detections.extend(self.device_link.observe(outcome))
|
||||||
|
return detections
|
||||||
|
|
||||||
|
def rtt_p95_ms(self) -> float | None:
|
||||||
|
p95 = self.latency._p95() # noqa: SLF001 — suite owns its parts
|
||||||
|
return round(p95, 1) if p95 is not None else None
|
||||||
|
|
||||||
|
def tags_bad(self) -> int:
|
||||||
|
return self.quality.tags_bad()
|
||||||
|
|
||||||
|
def devices_down(self) -> int:
|
||||||
|
return self.device_link.devices_down()
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"Detection",
|
||||||
|
"DetectorSuite",
|
||||||
|
"PollOutcome",
|
||||||
|
"MASS_BAD_FRACTION",
|
||||||
|
"LatencyDetector",
|
||||||
|
"QualityDetector",
|
||||||
|
"StaleDetector",
|
||||||
|
"MassBadDetector",
|
||||||
|
"DeviceLinkDetector",
|
||||||
|
]
|
||||||
@@ -1,25 +1,292 @@
|
|||||||
"""The poll loop on the single dedicated COM thread (SPECIFICATION §10.3).
|
"""The poll loop on the single dedicated COM thread (SPECIFICATION §10.3/§10.4).
|
||||||
|
|
||||||
Phase 1 (IMPLEMENTATION_PLAN §4.7): connect → poll loop → feed detectors, plus
|
Architecture (all COM stays on the worker thread — SPECIFICATION §10.3):
|
||||||
the hang watchdog (SPECIFICATION §10.4): a second thread timestamps poll
|
|
||||||
starts/ends; when a call exceeds ``hard_hang_s`` the stuck COM thread is
|
- :meth:`Poller.run_forever` runs on the *caller's* thread as the coordinator:
|
||||||
abandoned and a fresh connection is built on a new thread.
|
it starts a worker per (re)connect session, consumes its queue, feeds the
|
||||||
|
state machine, applies backoff, and owns the stop flag.
|
||||||
|
- The worker thread builds its **own** client via ``client_factory`` (COM
|
||||||
|
initialization is per-thread — the client must be born there), polls every
|
||||||
|
``interval_s``, measures RTT, and pushes messages onto the queue.
|
||||||
|
- Hang watchdog (SPECIFICATION §10.4): the coordinator timestamps every
|
||||||
|
worker message; silence longer than ``hard_hang_s`` means the thread is
|
||||||
|
stuck in a dead RPC channel — it is **abandoned** (daemon thread, never
|
||||||
|
joined) and a fresh worker is built after D1/backoff. A poll that merely
|
||||||
|
exceeds ``timeout_ms`` raises the D7 soft-hang event but keeps the thread.
|
||||||
|
|
||||||
|
Messages the worker sends: ``("connected", ts)``, ``("outcome", PollOutcome)``,
|
||||||
|
``("failed", reason)`` — after which the worker exits its loop.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import queue
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
from collections.abc import Callable
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
from opcda_diagnozer.config import Config
|
from opcda_diagnozer.config import Config
|
||||||
from opcda_diagnozer.opc.base import OpcClient
|
from opcda_diagnozer.events import DiagEvent, EventContext, EventKind, Severity
|
||||||
|
from opcda_diagnozer.monitor.detectors import Detection, DetectorSuite, PollOutcome
|
||||||
|
from opcda_diagnozer.monitor.state import (
|
||||||
|
ConnectionState,
|
||||||
|
ConnectionStateMachine,
|
||||||
|
StateEvent,
|
||||||
|
)
|
||||||
|
from opcda_diagnozer.opc.base import OpcClient, OpcError
|
||||||
|
|
||||||
|
#: Coordinator queue poll slice: small enough that stop is snappy, large
|
||||||
|
#: enough to not busy-spin (hard_hang watching happens between slices).
|
||||||
|
_WATCH_SLICE_S = 0.25
|
||||||
|
|
||||||
|
_MESSAGE = tuple[str, Any]
|
||||||
|
|
||||||
|
|
||||||
|
def _utc_now() -> datetime:
|
||||||
|
"""Production wall clock (injected as ``clock`` in tests)."""
|
||||||
|
return datetime.now(timezone.utc)
|
||||||
|
|
||||||
|
|
||||||
class Poller:
|
class Poller:
|
||||||
"""Drives one :class:`OpcClient` on its own thread — implemented in Phase 1."""
|
"""Drives one :class:`OpcClient` per connection on its own thread, forever.
|
||||||
|
|
||||||
def __init__(self, client: OpcClient, config: Config) -> None:
|
``client_factory`` is called **on the worker thread** for every
|
||||||
self._client = client
|
(re)connect — the client is thread-confined by construction (spec §10.3).
|
||||||
|
``reporter`` receives every event; the coordinator never touches the
|
||||||
|
client object.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
client_factory: Callable[[], OpcClient],
|
||||||
|
config: Config,
|
||||||
|
reporter: Callable[[DiagEvent], None],
|
||||||
|
clock: Callable[[], datetime] = _utc_now,
|
||||||
|
stop: threading.Event | None = None,
|
||||||
|
) -> None:
|
||||||
|
self._client_factory = client_factory
|
||||||
self._config = config
|
self._config = config
|
||||||
|
self._reporter = reporter
|
||||||
|
self._clock = clock
|
||||||
|
self._stop = stop if stop is not None else threading.Event()
|
||||||
|
self._sm = ConnectionStateMachine()
|
||||||
|
self._suite = DetectorSuite(config)
|
||||||
|
self._started_at: datetime | None = None
|
||||||
|
# Surveillance tags + device-indicator tags (polled automatically, §6.1),
|
||||||
|
# deduplicated in order.
|
||||||
|
self._tags: tuple[str, ...] = tuple(
|
||||||
|
dict.fromkeys(
|
||||||
|
[t.name for t in config.tags] + [d.tag for d in config.devices]
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# public surface
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def request_stop(self) -> None:
|
||||||
|
"""Ask the loop to finish after the current poll (from any thread)."""
|
||||||
|
self._stop.set()
|
||||||
|
|
||||||
def run_forever(self) -> None:
|
def run_forever(self) -> None:
|
||||||
raise NotImplementedError(
|
"""Coordinate worker sessions until :meth:`request_stop`.
|
||||||
"Phase 1: poller + hang watchdog (docs/IMPLEMENTATION_PLAN.md §4.7)"
|
|
||||||
)
|
Never raises for COM/OPC failures — they are events (D1/D2). A worker
|
||||||
|
crash surfaces as DISCONNECTED with the exception text, then reconnect.
|
||||||
|
"""
|
||||||
|
self._started_at = self._clock()
|
||||||
|
last_heartbeat = self._started_at
|
||||||
|
while not self._stop.is_set():
|
||||||
|
outcomes: queue.Queue[_MESSAGE] = queue.Queue()
|
||||||
|
worker = threading.Thread(
|
||||||
|
target=self._worker_main, args=(outcomes,), daemon=True
|
||||||
|
)
|
||||||
|
worker.start()
|
||||||
|
hung = self._coordinate(outcomes)
|
||||||
|
if hung:
|
||||||
|
# SPECIFICATION §10.4: abandon a thread stuck in a dead RPC
|
||||||
|
# channel — never join it; daemon=True lets the process exit.
|
||||||
|
self._emit(
|
||||||
|
EventKind.POLL_HANG,
|
||||||
|
Severity.CRITICAL,
|
||||||
|
{
|
||||||
|
"hard_hang_s": self._config.poll.hard_hang_s,
|
||||||
|
"action": "COM thread abandoned and rebuilt",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
worker.join(timeout=self._config.poll.hard_hang_s)
|
||||||
|
if self._stop.is_set():
|
||||||
|
break
|
||||||
|
if self._sm.state is ConnectionState.RECONNECT_WAIT:
|
||||||
|
delay = self._sm.next_attempt_delay_s()
|
||||||
|
if delay:
|
||||||
|
self._stop.wait(delay)
|
||||||
|
self._emit(
|
||||||
|
EventKind.DISCONNECTED,
|
||||||
|
Severity.INFO,
|
||||||
|
{"reason": "shutting down (stop requested)"},
|
||||||
|
)
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# coordinator side (caller's thread)
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def _coordinate(self, outcomes: "queue.Queue[_MESSAGE]") -> bool:
|
||||||
|
"""Consume one worker session; returns True when the worker was hung.
|
||||||
|
|
||||||
|
Feeds the state machine and detectors, emits D1/D2/D7/D15/D4–D6/D8–D14/D16
|
||||||
|
events, and watches for hard hangs by wall-clock silence.
|
||||||
|
"""
|
||||||
|
last_worker_word = self._clock()
|
||||||
|
last_heartbeat = self._started_at or self._clock()
|
||||||
|
while not self._stop.is_set():
|
||||||
|
try:
|
||||||
|
kind, payload = outcomes.get(timeout=_WATCH_SLICE_S)
|
||||||
|
except queue.Empty:
|
||||||
|
silent_s = (self._clock() - last_worker_word).total_seconds()
|
||||||
|
if silent_s > self._config.poll.hard_hang_s:
|
||||||
|
return True
|
||||||
|
continue
|
||||||
|
last_worker_word = self._clock()
|
||||||
|
if kind == "connected":
|
||||||
|
for ev in self._sm.on_connected(payload):
|
||||||
|
self._emit_state_event(ev)
|
||||||
|
self._emit(
|
||||||
|
EventKind.CONNECTED,
|
||||||
|
Severity.INFO,
|
||||||
|
{"server": self._config.opc.server},
|
||||||
|
)
|
||||||
|
elif kind == "outcome":
|
||||||
|
outcome: PollOutcome = payload
|
||||||
|
for event in self._consume_outcome(outcome):
|
||||||
|
self._reporter(event)
|
||||||
|
now = outcome.started_at
|
||||||
|
if (now - last_heartbeat).total_seconds() >= self._config.report.heartbeat_s:
|
||||||
|
last_heartbeat = now
|
||||||
|
self._emit(EventKind.HEARTBEAT, Severity.INFO, self._heartbeat_detail())
|
||||||
|
elif kind == "failed":
|
||||||
|
for ev in self._sm.on_disconnected(self._clock(), reason=str(payload)):
|
||||||
|
self._emit_state_event(ev)
|
||||||
|
return False
|
||||||
|
return False
|
||||||
|
|
||||||
|
def _consume_outcome(self, outcome: PollOutcome) -> list[DiagEvent]:
|
||||||
|
"""Run the detector suite on one outcome; add D7 soft-hang; map to events."""
|
||||||
|
events: list[DiagEvent] = []
|
||||||
|
for detection in self._suite.observe(outcome):
|
||||||
|
events.append(self._to_event(detection))
|
||||||
|
if outcome.error is None and outcome.rtt_ms > self._config.poll.timeout_ms:
|
||||||
|
events.append(
|
||||||
|
self._to_event(
|
||||||
|
Detection(
|
||||||
|
kind=EventKind.POLL_HANG,
|
||||||
|
severity=Severity.CRITICAL,
|
||||||
|
detail={
|
||||||
|
"rtt_ms": round(outcome.rtt_ms, 1),
|
||||||
|
"timeout_ms": self._config.poll.timeout_ms,
|
||||||
|
"note": "call returned late; thread kept",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return events
|
||||||
|
|
||||||
|
def _to_event(self, detection: Detection) -> DiagEvent:
|
||||||
|
return DiagEvent(
|
||||||
|
time_utc=self._clock(),
|
||||||
|
kind=detection.kind,
|
||||||
|
severity=detection.severity,
|
||||||
|
detail=detection.detail,
|
||||||
|
context=EventContext(
|
||||||
|
uptime_s=self._uptime_s(),
|
||||||
|
reconnects_total=self._sm.reconnects_total,
|
||||||
|
rtt_p95_ms=self._suite.rtt_p95_ms(),
|
||||||
|
tags_bad=self._suite.tags_bad(),
|
||||||
|
tags_total=len(self._tags),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
def _emit_state_event(self, ev: StateEvent) -> None:
|
||||||
|
kind, severity = {
|
||||||
|
"disconnected": (EventKind.DISCONNECTED, Severity.CRITICAL),
|
||||||
|
"reconnected": (EventKind.RECONNECTED, Severity.INFO),
|
||||||
|
"flapping": (EventKind.RECONNECT_FLAPPING, Severity.CRITICAL),
|
||||||
|
}[ev.kind]
|
||||||
|
self._emit(kind, severity, ev.detail)
|
||||||
|
|
||||||
|
def _heartbeat_detail(self) -> dict:
|
||||||
|
return {
|
||||||
|
"uptime_s": self._uptime_s(),
|
||||||
|
"reconnects_total": self._sm.reconnects_total,
|
||||||
|
"rtt_p95_ms": self._suite.rtt_p95_ms(),
|
||||||
|
"tags_bad": self._suite.tags_bad(),
|
||||||
|
"tags_total": len(self._tags),
|
||||||
|
"devices_down": self._suite.devices_down(),
|
||||||
|
}
|
||||||
|
|
||||||
|
def _uptime_s(self) -> float:
|
||||||
|
if self._started_at is None:
|
||||||
|
return 0.0
|
||||||
|
return (self._clock() - self._started_at).total_seconds()
|
||||||
|
|
||||||
|
def _emit(self, kind: EventKind, severity: Severity, detail: dict) -> None:
|
||||||
|
self._reporter(self._to_event(Detection(kind=kind, severity=severity, detail=detail)))
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
# worker side (the dedicated COM thread)
|
||||||
|
# ------------------------------------------------------------------ #
|
||||||
|
|
||||||
|
def _worker_main(self, outcomes: "queue.Queue[_MESSAGE]") -> None:
|
||||||
|
"""One connection session: own client, poll loop, push messages.
|
||||||
|
|
||||||
|
Exits (putting ``("failed", reason)``) whenever the connection drops
|
||||||
|
or the stop flag is observed — the coordinator decides what next.
|
||||||
|
"""
|
||||||
|
client: OpcClient | None = None
|
||||||
|
try:
|
||||||
|
client = self._client_factory()
|
||||||
|
client.connect()
|
||||||
|
outcomes.put(("connected", self._clock()))
|
||||||
|
while not self._stop.is_set():
|
||||||
|
started = time.monotonic()
|
||||||
|
started_at = _utc_now()
|
||||||
|
try:
|
||||||
|
results = client.read(self._tags)
|
||||||
|
rtt_ms = (time.monotonic() - started) * 1000.0
|
||||||
|
outcomes.put(
|
||||||
|
(
|
||||||
|
"outcome",
|
||||||
|
PollOutcome(
|
||||||
|
started_at=started_at,
|
||||||
|
rtt_ms=rtt_ms,
|
||||||
|
results=tuple(results),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
except OpcError as exc:
|
||||||
|
rtt_ms = (time.monotonic() - started) * 1000.0
|
||||||
|
outcomes.put(
|
||||||
|
(
|
||||||
|
"outcome",
|
||||||
|
PollOutcome(
|
||||||
|
started_at=started_at, rtt_ms=rtt_ms, error=str(exc)
|
||||||
|
),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
outcomes.put(("failed", str(exc)))
|
||||||
|
return
|
||||||
|
if self._stop.wait(self._config.poll.interval_s):
|
||||||
|
break
|
||||||
|
except OpcError as exc:
|
||||||
|
outcomes.put(("failed", str(exc)))
|
||||||
|
except Exception as exc: # noqa: BLE001 — a worker death is an event, not a crash
|
||||||
|
outcomes.put(("failed", f"worker crashed: {type(exc).__name__}: {exc}"))
|
||||||
|
finally:
|
||||||
|
if client is not None:
|
||||||
|
try:
|
||||||
|
client.close()
|
||||||
|
except Exception: # noqa: BLE001 — best-effort teardown
|
||||||
|
pass
|
||||||
@@ -4,11 +4,14 @@ States: Connecting → Connected → ReconnectWait → Connecting …
|
|||||||
Backoff is exponential 1 s → 2 s → … capped at 60 s; every disconnect/reconnect
|
Backoff is exponential 1 s → 2 s → … capped at 60 s; every disconnect/reconnect
|
||||||
pair is an event (D1/D2) with the outage duration computed and reported.
|
pair is an event (D1/D2) with the outage duration computed and reported.
|
||||||
|
|
||||||
The transition/backoff engine lands in Phase 1 (IMPLEMENTATION_PLAN §4.5).
|
The engine is pure: it is fed timestamps and returns transitions plus events,
|
||||||
|
so the poller owns the actual clock/sleeping and tests drive it deterministically.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from datetime import datetime, timedelta
|
||||||
from enum import Enum
|
from enum import Enum
|
||||||
|
|
||||||
|
|
||||||
@@ -17,4 +20,97 @@ class ConnectionState(Enum):
|
|||||||
|
|
||||||
CONNECTING = "connecting"
|
CONNECTING = "connecting"
|
||||||
CONNECTED = "connected"
|
CONNECTED = "connected"
|
||||||
RECONNECT_WAIT = "reconnect_wait"
|
RECONNECT_WAIT = "reconnect_wait"
|
||||||
|
|
||||||
|
|
||||||
|
#: Backoff constants (SPECIFICATION §10.2).
|
||||||
|
BACKOFF_INITIAL_S = 1.0
|
||||||
|
BACKOFF_MAX_S = 60.0
|
||||||
|
|
||||||
|
#: D3 reconnect flapping: > this many disconnects within the window is CRITICAL.
|
||||||
|
FLAPPING_DISCONNECTS = 3
|
||||||
|
FLAPPING_WINDOW = timedelta(minutes=10)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class StateEvent:
|
||||||
|
"""One state-machine transition worth reporting (D1/D2/D3)."""
|
||||||
|
|
||||||
|
kind: str # "disconnected" | "reconnected" | "flapping"
|
||||||
|
at: datetime
|
||||||
|
detail: dict = field(default_factory=dict)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class ConnectionStateMachine:
|
||||||
|
"""Tracks connection state, backoff and reconnect accounting (D1–D3).
|
||||||
|
|
||||||
|
The poller calls :meth:`on_connected` / :meth:`on_disconnected` at the
|
||||||
|
moments those things are observed; the machine decides *when* to emit
|
||||||
|
events (D1 with the failure reason, D2 with the outage duration, D3 when
|
||||||
|
disconnects flap) and *how long* to wait before the next attempt.
|
||||||
|
"""
|
||||||
|
|
||||||
|
state: ConnectionState = ConnectionState.CONNECTING
|
||||||
|
backoff_s: float = BACKOFF_INITIAL_S
|
||||||
|
connected_since: datetime | None = None
|
||||||
|
disconnected_since: datetime | None = None
|
||||||
|
reconnects_total: int = 0
|
||||||
|
_disconnect_times: list[datetime] = field(default_factory=list)
|
||||||
|
|
||||||
|
def on_connected(self, at: datetime) -> list[StateEvent]:
|
||||||
|
"""A connect (or reconnect) succeeded at ``at``."""
|
||||||
|
events: list[StateEvent] = []
|
||||||
|
was_outage = self.state is ConnectionState.RECONNECT_WAIT
|
||||||
|
self.state = ConnectionState.CONNECTED
|
||||||
|
self.connected_since = at
|
||||||
|
self.backoff_s = BACKOFF_INITIAL_S # reset on success (spec §10.2)
|
||||||
|
if was_outage and self.disconnected_since is not None:
|
||||||
|
self.reconnects_total += 1
|
||||||
|
events.append(
|
||||||
|
StateEvent(
|
||||||
|
kind="reconnected",
|
||||||
|
at=at,
|
||||||
|
detail={
|
||||||
|
"outage_s": (at - self.disconnected_since).total_seconds(),
|
||||||
|
"reconnects_total": self.reconnects_total,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
self.disconnected_since = None
|
||||||
|
return events
|
||||||
|
|
||||||
|
def on_disconnected(self, at: datetime, reason: str) -> list[StateEvent]:
|
||||||
|
"""The connection dropped at ``at`` (reason = decoded failure)."""
|
||||||
|
events: list[StateEvent] = []
|
||||||
|
first_drop = self.state is not ConnectionState.RECONNECT_WAIT
|
||||||
|
self.state = ConnectionState.RECONNECT_WAIT
|
||||||
|
self.connected_since = None
|
||||||
|
if first_drop:
|
||||||
|
self.disconnected_since = at
|
||||||
|
self._disconnect_times.append(at)
|
||||||
|
self._disconnect_times = [
|
||||||
|
t for t in self._disconnect_times if at - t <= FLAPPING_WINDOW
|
||||||
|
]
|
||||||
|
events.append(StateEvent(kind="disconnected", at=at, detail={"reason": reason}))
|
||||||
|
if len(self._disconnect_times) > FLAPPING_DISCONNECTS:
|
||||||
|
events.append(
|
||||||
|
StateEvent(
|
||||||
|
kind="flapping",
|
||||||
|
at=at,
|
||||||
|
detail={"disconnects_in_window": len(self._disconnect_times)},
|
||||||
|
)
|
||||||
|
)
|
||||||
|
return events
|
||||||
|
|
||||||
|
def next_attempt_delay_s(self) -> float:
|
||||||
|
"""How long to wait before the next connect attempt (exponential backoff).
|
||||||
|
|
||||||
|
Call once per failed attempt; the value doubles up to
|
||||||
|
:data:`BACKOFF_MAX_S` and resets only on a successful connect.
|
||||||
|
"""
|
||||||
|
if self.state is not ConnectionState.RECONNECT_WAIT:
|
||||||
|
return 0.0
|
||||||
|
delay = self.backoff_s
|
||||||
|
self.backoff_s = min(self.backoff_s * 2, BACKOFF_MAX_S)
|
||||||
|
return delay
|
||||||
@@ -47,6 +47,20 @@ from opcda_diagnozer.opc.base import (
|
|||||||
_ORGANIZATION_LABELS = {1: "hierarchical", 2: "flat"}
|
_ORGANIZATION_LABELS = {1: "hierarchical", 2: "flat"}
|
||||||
|
|
||||||
|
|
||||||
|
def _per_tag_error(what: str, exc: BaseException) -> str:
|
||||||
|
"""Render a per-tag failure without swallowing the underlying cause.
|
||||||
|
|
||||||
|
COM errors (``com_error`` with an ``hresult``) keep their decoded HRESULT
|
||||||
|
suffix via :func:`as_com_error`. **Plain Python exceptions** (e.g. a
|
||||||
|
``ValueError`` from timezone math on a degenerate device timestamp) must
|
||||||
|
show their own type and text — early builds reduced them to a bare
|
||||||
|
"read of tag 'X' failed", which made failures undiagnosable.
|
||||||
|
"""
|
||||||
|
if getattr(exc, "hresult", None) is None:
|
||||||
|
return f"{what}; {type(exc).__name__}: {exc}"
|
||||||
|
return str(as_com_error(what, exc))
|
||||||
|
|
||||||
|
|
||||||
class _AutomationBrowser:
|
class _AutomationBrowser:
|
||||||
"""Thin, read-only :class:`Browser` over one wrapper browser object.
|
"""Thin, read-only :class:`Browser` over one wrapper browser object.
|
||||||
|
|
||||||
@@ -57,7 +71,10 @@ class _AutomationBrowser:
|
|||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(self, raw_browser: Any) -> None:
|
def __init__(self, raw_browser: Any) -> None:
|
||||||
self._browser = raw_browser
|
# Explicit ``Any`` declaration: close() assigns None to drop the COM
|
||||||
|
# reference, and the inferred ``Any | None`` would flag every access
|
||||||
|
# below (the dispatch object is dynamic — always accessed raw).
|
||||||
|
self._browser: Any = raw_browser
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def organization(self) -> str:
|
def organization(self) -> str:
|
||||||
@@ -260,23 +277,35 @@ class AutomationOpcClient:
|
|||||||
value=None,
|
value=None,
|
||||||
quality_raw=0,
|
quality_raw=0,
|
||||||
device_time=None,
|
device_time=None,
|
||||||
error=str(as_com_error(f"cannot add tag {tag!r}", exc)),
|
error=_per_tag_error(f"cannot add tag {tag!r}", exc),
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
for tag, item in live_items:
|
for tag, item in live_items:
|
||||||
try:
|
try:
|
||||||
item.Read(int(source))
|
item.Read(int(source))
|
||||||
raw_ts = item.TimeStamp
|
# Primary data first: value + quality are the read's
|
||||||
# pywintypes.datetime is a datetime subclass; normalize
|
# purpose. The device timestamp is METADATA and is
|
||||||
# to UTC for the event model (SPECIFICATION §6).
|
# fetched last, fully guarded — a broken timestamp
|
||||||
device_time = (
|
# (missing win32timezone in a frozen exe, zero FILETIME
|
||||||
raw_ts.astimezone(timezone.utc) if raw_ts is not None else None
|
# 1601 on a never-updated alias, DST math on a
|
||||||
)
|
# historical local-time wrapper) must never fail a read
|
||||||
|
# whose value/quality are good.
|
||||||
|
value = item.Value
|
||||||
|
quality = int(item.Quality)
|
||||||
|
device_time = None
|
||||||
|
try:
|
||||||
|
raw_ts = item.TimeStamp
|
||||||
|
# pywintypes.datetime is a datetime subclass; normalize
|
||||||
|
# to UTC for the event model (SPECIFICATION §6).
|
||||||
|
if raw_ts is not None:
|
||||||
|
device_time = raw_ts.astimezone(timezone.utc)
|
||||||
|
except Exception: # noqa: BLE001 — timestamp is metadata
|
||||||
|
device_time = None
|
||||||
results.append(
|
results.append(
|
||||||
ReadResult(
|
ReadResult(
|
||||||
tag=tag,
|
tag=tag,
|
||||||
value=item.Value,
|
value=value,
|
||||||
quality_raw=int(item.Quality),
|
quality_raw=quality,
|
||||||
device_time=device_time,
|
device_time=device_time,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
@@ -287,7 +316,7 @@ class AutomationOpcClient:
|
|||||||
value=None,
|
value=None,
|
||||||
quality_raw=0,
|
quality_raw=0,
|
||||||
device_time=None,
|
device_time=None,
|
||||||
error=str(as_com_error(f"read of tag {tag!r} failed", exc)),
|
error=_per_tag_error(f"read of tag {tag!r} failed", exc),
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
except Exception as exc: # noqa: BLE001 — group-level failure
|
except Exception as exc: # noqa: BLE001 — group-level failure
|
||||||
|
|||||||
@@ -17,6 +17,19 @@ from datetime import datetime, timezone
|
|||||||
|
|
||||||
from opcda_diagnozer.opc.base import Browser, OPCDataSource, OpcError, ReadResult, TagInfo
|
from opcda_diagnozer.opc.base import Browser, OPCDataSource, OpcError, ReadResult, TagInfo
|
||||||
|
|
||||||
|
#: Scenario-engine directives (Phase 1, IMPLEMENTATION_PLAN §4.9): the fake
|
||||||
|
#: consumes one per read() call. A "hang" is not modeled here — the poller's
|
||||||
|
#: watchdog handles hangs via a client whose read() simply sleeps; the fake
|
||||||
|
#: stays non-blocking so unit tests never slow down.
|
||||||
|
PollScriptStep = "PollScriptError | list[ReadResult] | str"
|
||||||
|
|
||||||
|
|
||||||
|
class PollScriptError:
|
||||||
|
"""Scenario directive: the whole read call fails with this message."""
|
||||||
|
|
||||||
|
def __init__(self, message: str) -> None:
|
||||||
|
self.message = message
|
||||||
|
|
||||||
#: The fake address space: branch path -> (sub-branches, leaf names).
|
#: The fake address space: branch path -> (sub-branches, leaf names).
|
||||||
#: Mirrors the shape of a real SCADA server (Channel → Device → Tags).
|
#: Mirrors the shape of a real SCADA server (Channel → Device → Tags).
|
||||||
FAKE_SPACE: dict[tuple[str, ...], tuple[tuple[str, ...], tuple[str, ...]]] = {
|
FAKE_SPACE: dict[tuple[str, ...], tuple[tuple[str, ...], tuple[str, ...]]] = {
|
||||||
@@ -101,16 +114,32 @@ class FakeOpcClient:
|
|||||||
``space`` is injectable so tests can exercise flat namespaces, empty
|
``space`` is injectable so tests can exercise flat namespaces, empty
|
||||||
servers, and deep trees; ``read_results`` scripts the next read() result
|
servers, and deep trees; ``read_results`` scripts the next read() result
|
||||||
(Phase 1 scenario engine seed, IMPLEMENTATION_PLAN §4.9).
|
(Phase 1 scenario engine seed, IMPLEMENTATION_PLAN §4.9).
|
||||||
|
|
||||||
|
Scenario engine (Phase 1): ``script`` is a list of per-poll directives the
|
||||||
|
fake consumes one per read() call, in order — after the script is
|
||||||
|
exhausted the client falls back to ``read_results`` or plain GOOD zeros.
|
||||||
|
Each entry may be:
|
||||||
|
|
||||||
|
- ``PollOutcomeError(message)`` — the whole read call fails (COM-style
|
||||||
|
OpcError for D1/D2 testing; a ``PollOutcomeHang`` directive is handled by
|
||||||
|
the *poller* watchdog, not the client — see :class:`FakeHangClient`),
|
||||||
|
- a ``list[ReadResult]`` — the exact results for this poll,
|
||||||
|
- ``"good"`` — shorthand for all-tags GOOD zeros.
|
||||||
|
|
||||||
|
The read log (``reads``) records every consumed (tags, source) pair so
|
||||||
|
tests can assert poll cadence and scripted order.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
def __init__(
|
def __init__(
|
||||||
self,
|
self,
|
||||||
space: dict[tuple[str, ...], tuple[tuple[str, ...], tuple[str, ...]]] | None = None,
|
space: dict[tuple[str, ...], tuple[tuple[str, ...], tuple[str, ...]]] | None = None,
|
||||||
read_results: list[ReadResult] | None = None,
|
read_results: list[ReadResult] | None = None,
|
||||||
|
script: list["PollScriptStep"] | None = None,
|
||||||
) -> None:
|
) -> None:
|
||||||
self.connected = False
|
self.connected = False
|
||||||
self._space = FAKE_SPACE if space is None else space
|
self._space = FAKE_SPACE if space is None else space
|
||||||
self._scripted_reads = read_results
|
self._scripted_reads = read_results
|
||||||
|
self._script = list(script) if script is not None else None
|
||||||
self.reads: list[tuple[tuple[str, ...], OPCDataSource]] = [] # call log
|
self.reads: list[tuple[tuple[str, ...], OPCDataSource]] = [] # call log
|
||||||
|
|
||||||
def connect(self) -> None:
|
def connect(self) -> None:
|
||||||
@@ -127,6 +156,22 @@ class FakeOpcClient:
|
|||||||
if not self.connected:
|
if not self.connected:
|
||||||
raise OpcError("FakeOpcClient: read() before connect()")
|
raise OpcError("FakeOpcClient: read() before connect()")
|
||||||
self.reads.append((tuple(tags), source))
|
self.reads.append((tuple(tags), source))
|
||||||
|
# scenario engine: consume one scripted step per poll, in order
|
||||||
|
if self._script:
|
||||||
|
step = self._script.pop(0)
|
||||||
|
if isinstance(step, PollScriptError):
|
||||||
|
raise OpcError(step.message)
|
||||||
|
if isinstance(step, list):
|
||||||
|
by_tag = {r.tag: r for r in step}
|
||||||
|
return [
|
||||||
|
by_tag.get(
|
||||||
|
tag,
|
||||||
|
ReadResult(tag=tag, value=None, quality_raw=0, device_time=None,
|
||||||
|
error=f"no scripted result for {tag!r}"),
|
||||||
|
)
|
||||||
|
for tag in tags
|
||||||
|
]
|
||||||
|
# "good" shorthand falls through to the default below
|
||||||
if self._scripted_reads is not None:
|
if self._scripted_reads is not None:
|
||||||
scripted = {r.tag: r for r in self._scripted_reads}
|
scripted = {r.tag: r for r in self._scripted_reads}
|
||||||
return [
|
return [
|
||||||
|
|||||||
@@ -66,4 +66,25 @@ def decode_quality(raw: int) -> QualityInfo:
|
|||||||
name = _QUALITY_NAMES.get(low)
|
name = _QUALITY_NAMES.get(low)
|
||||||
if name is None:
|
if name is None:
|
||||||
name = f"{_class_of(low).value}_0x{low:02X}"
|
name = f"{_class_of(low).value}_0x{low:02X}"
|
||||||
return QualityInfo(raw=raw, name=name, quality_class=_class_of(low))
|
return QualityInfo(raw=raw, name=name, quality_class=_class_of(low))
|
||||||
|
|
||||||
|
|
||||||
|
def status_text(raw: int) -> str:
|
||||||
|
"""Clear-text status with the raw code in parentheses (SPECIFICATION §7).
|
||||||
|
|
||||||
|
Every reporter writes quality this way — e.g. ``BAD / COMM FAILURE
|
||||||
|
(0x18)`` — so nobody decodes hex by hand while the raw code stays
|
||||||
|
available for unambiguous correlation with vendor documentation.
|
||||||
|
"""
|
||||||
|
info = decode_quality(raw)
|
||||||
|
classes = {c.value for c in QualityClass}
|
||||||
|
name = info.name
|
||||||
|
for cls in classes:
|
||||||
|
prefix = cls + "_"
|
||||||
|
if name.startswith(prefix):
|
||||||
|
sub = name[len(prefix):].replace("_", " ")
|
||||||
|
name = f"{cls} / {sub}" if sub else cls
|
||||||
|
break
|
||||||
|
else:
|
||||||
|
name = name.replace("_", " ")
|
||||||
|
return f"{name} ({info.raw_hex})"
|
||||||
@@ -1,7 +1,8 @@
|
|||||||
"""Reporter protocol — pluggable event sinks (SPECIFICATION §9)."""
|
"""Reporter protocol + fan-out dispatcher (SPECIFICATION §9, IMPLEMENTATION_PLAN §5.4)."""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Callable
|
||||||
from typing import Protocol
|
from typing import Protocol
|
||||||
|
|
||||||
from opcda_diagnozer.events import DiagEvent
|
from opcda_diagnozer.events import DiagEvent
|
||||||
@@ -10,4 +11,65 @@ from opcda_diagnozer.events import DiagEvent
|
|||||||
class Reporter(Protocol):
|
class Reporter(Protocol):
|
||||||
"""Anything that can receive events; reporting never blocks the poll loop."""
|
"""Anything that can receive events; reporting never blocks the poll loop."""
|
||||||
|
|
||||||
def emit(self, event: DiagEvent) -> None: ...
|
def emit(self, event: DiagEvent) -> None: ...
|
||||||
|
|
||||||
|
|
||||||
|
def _emit_via(reporter: Reporter, event: DiagEvent) -> None:
|
||||||
|
"""Deliver to a sink that is either an object with ``emit`` or a plain callable.
|
||||||
|
|
||||||
|
The console sink (``_build_reporters``) is a function; the file/Event
|
||||||
|
Log/webhook sinks are objects. Both are legitimate ``Reporter``
|
||||||
|
implementations — the protocol is "anything that can receive events" —
|
||||||
|
so the fan-out duck-types both shapes instead of forcing every sink
|
||||||
|
into a class. Anything else fails loudly (caught + counted by FanOut).
|
||||||
|
"""
|
||||||
|
if callable(getattr(reporter, "emit", None)):
|
||||||
|
reporter.emit(event) # type: ignore[attr-defined]
|
||||||
|
return
|
||||||
|
if callable(reporter):
|
||||||
|
reporter(event) # type: ignore[call-arg]
|
||||||
|
return
|
||||||
|
raise TypeError(
|
||||||
|
f"reporter {type(reporter).__name__} has neither emit() nor is callable"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class FanOut:
|
||||||
|
"""Dispatches each event to every configured reporter.
|
||||||
|
|
||||||
|
Accepts sinks as objects with ``emit(event)`` **or** plain functions
|
||||||
|
(found live 2026-10-08: the console function sink raised AttributeError
|
||||||
|
on every event — the fan-out duck-types both shapes now). One reporter
|
||||||
|
failing must never kill the others or the poll loop: each delivery is
|
||||||
|
isolated, failures counted and surfaced via the injected ``error``
|
||||||
|
callback (stderr in the CLI). Sinks own their concurrency safety (HTTP
|
||||||
|
queues internally; file reporters append single small lines).
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, reporters: tuple[Reporter, ...], error: Callable[[str], None]) -> None:
|
||||||
|
self._reporters = reporters
|
||||||
|
self._error = error
|
||||||
|
self.failures = 0
|
||||||
|
|
||||||
|
def emit(self, event: DiagEvent) -> None:
|
||||||
|
for reporter in self._reporters:
|
||||||
|
try:
|
||||||
|
_emit_via(reporter, event)
|
||||||
|
except Exception as exc: # noqa: BLE001 — a sink must never kill polling
|
||||||
|
self.failures += 1
|
||||||
|
self._error(
|
||||||
|
f"reporter {type(reporter).__name__} failed "
|
||||||
|
f"({type(exc).__name__}: {exc}); event {event.kind.value} "
|
||||||
|
"kept in the other sinks"
|
||||||
|
)
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
"""Best-effort close of every sink (shutdown path only)."""
|
||||||
|
for reporter in self._reporters:
|
||||||
|
close = getattr(reporter, "close", None)
|
||||||
|
if close is None:
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
close()
|
||||||
|
except Exception: # noqa: BLE001
|
||||||
|
pass
|
||||||
@@ -1,22 +1,85 @@
|
|||||||
"""Windows Event Log reporter, WARN+ only (IMPLEMENTATION_PLAN §5.2).
|
"""Windows Event Log reporter, severity-mapped, WARN+ by default (plan §5.2).
|
||||||
|
|
||||||
Uses ``win32evtlog``; the source is registered during ``install``. pywin32 is
|
Uses ``win32evtlog`` via a registered **event source handle**
|
||||||
import-guarded so tests can run on any OS (IMPLEMENTATION_PLAN §9).
|
(``RegisterEventSource`` → positional ``ReportEvent`` — the pywin32 function
|
||||||
|
takes positional args only; keyword calls raise ``TypeError``). COM imports
|
||||||
|
are guarded so tests and non-Windows runs import the module freely
|
||||||
|
(IMPLEMENTATION_PLAN §9).
|
||||||
|
|
||||||
|
The source's message file is registered during ``install`` (Phase 3); until
|
||||||
|
then events still land in the Application log under the source name (verified
|
||||||
|
live: Windows delivers them, rendering the message with an empty template).
|
||||||
|
A sink failure is counted, never raised — the poll loop must not stall.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
from opcda_diagnozer.events import DiagEvent, Severity
|
from opcda_diagnozer.events import DiagEvent, Severity
|
||||||
|
|
||||||
|
# Windows Event Log types (win32evtlog EVENTLOG_*_TYPE constants)
|
||||||
|
_LEVELS = {
|
||||||
|
Severity.INFO: 4, # EVENTLOG_INFORMATION_TYPE
|
||||||
|
Severity.WARN: 2, # EVENTLOG_WARNING_TYPE
|
||||||
|
Severity.CRITICAL: 1, # EVENTLOG_ERROR_TYPE
|
||||||
|
}
|
||||||
|
_ORDER = {Severity.INFO: 0, Severity.WARN: 1, Severity.CRITICAL: 2}
|
||||||
|
|
||||||
|
|
||||||
class EventLogReporter:
|
class EventLogReporter:
|
||||||
"""Windows Event Log sink — implemented in Phase 2."""
|
"""Windows Event Log sink — WARN+ only (spec §9), severity-mapped."""
|
||||||
|
|
||||||
def __init__(self, source: str = "OPCDA-Diagnozer", min_severity: str = "WARN") -> None:
|
def __init__(self, source: str = "OPCDA-Diagnozer", min_severity: str = "WARN") -> None:
|
||||||
self._source = source
|
self._source = source
|
||||||
self._min_severity = Severity(min_severity)
|
self._min = Severity(min_severity)
|
||||||
|
self._handle = None
|
||||||
|
self._register_source()
|
||||||
|
|
||||||
|
def _register_source(self) -> None:
|
||||||
|
"""Open the event source handle once (kept for the reporter's lifetime).
|
||||||
|
|
||||||
|
``RegisterEventSource(None, source)`` works without admin rights;
|
||||||
|
unregistered sources still deliver (verified live 2026-10-08).
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
import win32evtlog
|
||||||
|
|
||||||
|
self._handle = win32evtlog.RegisterEventSource(None, self._source)
|
||||||
|
except Exception: # noqa: BLE001 — non-Windows/tests: emit() is a no-op
|
||||||
|
self._handle = None
|
||||||
|
|
||||||
def emit(self, event: DiagEvent) -> None:
|
def emit(self, event: DiagEvent) -> None:
|
||||||
raise NotImplementedError(
|
"""Report one event; below-minimum severities are filtered out."""
|
||||||
"Phase 2: Event Log reporter (docs/IMPLEMENTATION_PLAN.md §5.2)"
|
if _ORDER[event.severity] < _ORDER[self._min]:
|
||||||
)
|
return
|
||||||
|
if self._handle is None:
|
||||||
|
self._register_source()
|
||||||
|
if self._handle is None:
|
||||||
|
return
|
||||||
|
message = ", ".join(f"{k}={v}" for k, v in event.detail.items())
|
||||||
|
try:
|
||||||
|
import win32evtlog
|
||||||
|
|
||||||
|
# positional args only (win32evtlog.ReportEvent takes no keywords)
|
||||||
|
win32evtlog.ReportEvent(
|
||||||
|
self._handle,
|
||||||
|
_LEVELS[event.severity],
|
||||||
|
0, # category
|
||||||
|
0, # event ID
|
||||||
|
None, # user SID
|
||||||
|
[f"{event.kind.value}: {message}"],
|
||||||
|
None, # raw data: the JSONL file already carries the full record
|
||||||
|
)
|
||||||
|
except Exception: # noqa: BLE001 — a sink must never kill the poll loop
|
||||||
|
pass
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
"""Release the event source handle (Windows API hygiene)."""
|
||||||
|
if self._handle is None:
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
import win32evtlog
|
||||||
|
|
||||||
|
win32evtlog.DeregisterEventSource(self._handle)
|
||||||
|
except Exception: # noqa: BLE001 — shutdown is best-effort
|
||||||
|
pass
|
||||||
|
self._handle = None
|
||||||
@@ -1,23 +1,111 @@
|
|||||||
"""Optional HTTP webhook reporter (IMPLEMENTATION_PLAN §5.3).
|
"""Optional HTTP webhook reporter (IMPLEMENTATION_PLAN §5.3).
|
||||||
|
|
||||||
Phase 2: POST one JSON event per request, optional bearer token, timeout plus
|
POSTs one JSON event per request with an optional bearer token, timeout +
|
||||||
simple retry/drop on failure — must never block the poll loop (bounded queue).
|
drop-on-failure — **never blocks the poll loop**: ``emit`` only pushes onto a
|
||||||
|
bounded queue and a daemon thread drains it. A full queue drops the event
|
||||||
|
(and counts it): losing an alert is always better than stalling polling on a
|
||||||
|
dead webhook host.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import queue
|
||||||
|
import threading
|
||||||
|
from collections.abc import Callable
|
||||||
|
|
||||||
from opcda_diagnozer.events import DiagEvent
|
from opcda_diagnozer.events import DiagEvent
|
||||||
|
|
||||||
|
_QUEUE_BOUND = 1000 # events; ~17 min of anomalies at a pathological rate
|
||||||
|
|
||||||
|
|
||||||
class HttpReporter:
|
class HttpReporter:
|
||||||
"""Webhook sink — implemented in Phase 2."""
|
"""Webhook sink — bounded queue + daemon sender thread (spec §9)."""
|
||||||
|
|
||||||
def __init__(self, url: str, bearer_token: str | None = None, timeout_s: float = 5.0) -> None:
|
def __init__(
|
||||||
|
self,
|
||||||
|
url: str,
|
||||||
|
bearer_token: str | None = None,
|
||||||
|
timeout_s: float = 5.0,
|
||||||
|
post: Callable[[str, str | None, str, float], None] | None = None,
|
||||||
|
) -> None:
|
||||||
self._url = url
|
self._url = url
|
||||||
self._bearer_token = bearer_token
|
self._bearer = bearer_token
|
||||||
self._timeout_s = timeout_s
|
self._timeout_s = timeout_s
|
||||||
|
self._post = post # injectable for tests; None -> urllib in the thread
|
||||||
|
self._queue: queue.Queue[DiagEvent | None] = queue.Queue(maxsize=_QUEUE_BOUND)
|
||||||
|
self._dropped = 0
|
||||||
|
self._sent = 0
|
||||||
|
self._stop = threading.Event()
|
||||||
|
self._thread: threading.Thread | None = None
|
||||||
|
if self._post is not None:
|
||||||
|
# direct mode (tests): no thread, emit posts synchronously
|
||||||
|
pass
|
||||||
|
else:
|
||||||
|
self._thread = threading.Thread(target=self._drain, daemon=True)
|
||||||
|
self._thread.start()
|
||||||
|
|
||||||
def emit(self, event: DiagEvent) -> None:
|
def emit(self, event: DiagEvent) -> None:
|
||||||
raise NotImplementedError(
|
"""Queue the event; drop (counted) when the bound is reached.
|
||||||
"Phase 2: HTTP webhook reporter (docs/IMPLEMENTATION_PLAN.md §5.3)"
|
|
||||||
)
|
In injected-``post`` mode (tests) the send happens inline but is
|
||||||
|
equally isolated: a failing webhook must never raise into the poll
|
||||||
|
loop — losing one alert beats stalling polling on a dead host.
|
||||||
|
"""
|
||||||
|
if self._post is not None:
|
||||||
|
try:
|
||||||
|
self._post(self._url, self._bearer, event.to_json(), self._timeout_s)
|
||||||
|
self._sent += 1
|
||||||
|
except Exception: # noqa: BLE001 — drop on failure, keep polling
|
||||||
|
self._dropped += 1
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
self._queue.put_nowait(event)
|
||||||
|
except queue.Full:
|
||||||
|
self._dropped += 1
|
||||||
|
|
||||||
|
def _drain(self) -> None:
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
while not self._stop.is_set():
|
||||||
|
try:
|
||||||
|
event = self._queue.get(timeout=0.25)
|
||||||
|
except queue.Empty:
|
||||||
|
continue
|
||||||
|
if event is None:
|
||||||
|
break
|
||||||
|
request = urllib.request.Request(
|
||||||
|
self._url,
|
||||||
|
data=event.to_json().encode("utf-8"),
|
||||||
|
headers={
|
||||||
|
"Content-Type": "application/json",
|
||||||
|
**({"Authorization": f"Bearer {self._bearer}"} if self._bearer else {}),
|
||||||
|
},
|
||||||
|
method="POST",
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(request, timeout=self._timeout_s):
|
||||||
|
pass
|
||||||
|
self._sent += 1
|
||||||
|
except Exception: # noqa: BLE001 — drop on failure, keep polling
|
||||||
|
self._dropped += 1
|
||||||
|
|
||||||
|
@property
|
||||||
|
def dropped(self) -> int:
|
||||||
|
"""Events dropped because the queue was full or the POST failed."""
|
||||||
|
return self._dropped
|
||||||
|
|
||||||
|
@property
|
||||||
|
def sent(self) -> int:
|
||||||
|
return self._sent
|
||||||
|
|
||||||
|
def close(self, timeout_s: float = 2.0) -> None:
|
||||||
|
"""Stop the sender thread; queued events are flushed if the drain is fast."""
|
||||||
|
if self._thread is None:
|
||||||
|
return
|
||||||
|
self._stop.set()
|
||||||
|
try:
|
||||||
|
self._queue.put_nowait(None) # wake the drain immediately
|
||||||
|
except queue.Full:
|
||||||
|
pass
|
||||||
|
self._thread.join(timeout=timeout_s)
|
||||||
@@ -1,25 +1,77 @@
|
|||||||
"""Rotating JSONL reporter — the primary evidence trail (IMPLEMENTATION_PLAN §5.1).
|
"""Rotating JSONL reporter — the primary evidence trail (IMPLEMENTATION_PLAN §5.1).
|
||||||
|
|
||||||
Phase 2 adds: size-based rotation with ``backups`` kept, atomic-ish append,
|
One JSON event per line (spec §8), size-based rotation with ``backups`` kept,
|
||||||
and a header line carrying the config digest for provenance.
|
and a header line per file carrying the config digest for provenance — so a
|
||||||
|
rotated file can always be traced to the exact thresholds that produced it.
|
||||||
|
|
||||||
|
Appends are "atomic-ish": one pre-serialized line + newline, flushed. A hard
|
||||||
|
kill can at worst lose the current line, never corrupt earlier ones (event
|
||||||
|
lines are a few hundred bytes — single small appends, not torn writes).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
|
from opcda_diagnozer.config import Config
|
||||||
from opcda_diagnozer.events import DiagEvent
|
from opcda_diagnozer.events import DiagEvent
|
||||||
|
|
||||||
|
|
||||||
class JsonlReporter:
|
def config_digest(config: Config) -> str:
|
||||||
"""One JSON event per line, rotated by size — implemented in Phase 2."""
|
"""Stable SHA-256 over the effective config (provenance for log headers)."""
|
||||||
|
payload = json.dumps(config, default=lambda o: getattr(o, "__dict__", str(o)),
|
||||||
|
sort_keys=True)
|
||||||
|
return hashlib.sha256(payload.encode("utf-8")).hexdigest()
|
||||||
|
|
||||||
def __init__(self, directory: str | Path, max_mb: int = 50, backups: int = 10) -> None:
|
|
||||||
|
def rotate_file(path: Path, backups: int) -> None:
|
||||||
|
"""Size-rotation used by both file reporters: ``p`` -> ``p.1`` … ``p.backups``.
|
||||||
|
|
||||||
|
The oldest copy is dropped; numbering shifts by one so ``p.1`` is always
|
||||||
|
the most recently closed file. Shared so the JSONL and text reporters
|
||||||
|
guarantee identical retention semantics (spec §9.1).
|
||||||
|
"""
|
||||||
|
if not path.exists():
|
||||||
|
return
|
||||||
|
for index in range(backups - 1, 0, -1):
|
||||||
|
older = path.with_name(f"{path.name}.{index}")
|
||||||
|
if older.exists():
|
||||||
|
older.replace(path.with_name(f"{path.name}.{index + 1}"))
|
||||||
|
path.replace(path.with_name(f"{path.name}.1"))
|
||||||
|
|
||||||
|
|
||||||
|
class JsonlReporter:
|
||||||
|
"""One JSON event per line, rotated by size (spec §9)."""
|
||||||
|
|
||||||
|
def __init__(self, directory: str | Path, max_mb: int = 50, backups: int = 10,
|
||||||
|
digest: str = "") -> None:
|
||||||
self._directory = Path(directory)
|
self._directory = Path(directory)
|
||||||
self._max_mb = max_mb
|
self._max_bytes = max_mb * 1024 * 1024
|
||||||
self._backups = backups
|
self._backups = backups
|
||||||
|
self._digest = digest
|
||||||
|
self._directory.mkdir(parents=True, exist_ok=True)
|
||||||
|
self._current = self._directory / "events.jsonl"
|
||||||
|
|
||||||
def emit(self, event: DiagEvent) -> None:
|
def emit(self, event: DiagEvent) -> None:
|
||||||
raise NotImplementedError(
|
"""Append one event; rotate first when the file would exceed the cap."""
|
||||||
"Phase 2: JSONL reporter (docs/IMPLEMENTATION_PLAN.md §5.1)"
|
line = event.to_json()
|
||||||
)
|
needs_header = not self._current.exists()
|
||||||
|
if self._current.exists() and (
|
||||||
|
self._current.stat().st_size + len(line) + 2 > self._max_bytes
|
||||||
|
):
|
||||||
|
rotate_file(self._current, self._backups)
|
||||||
|
needs_header = True
|
||||||
|
with self._current.open("a", encoding="utf-8") as stream:
|
||||||
|
if needs_header:
|
||||||
|
header = {
|
||||||
|
"header": True,
|
||||||
|
"format": "opcda-diagnozer events v1",
|
||||||
|
"config_sha256": self._digest,
|
||||||
|
}
|
||||||
|
stream.write(json.dumps(header) + "\n")
|
||||||
|
stream.write(line + "\n")
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
"""Lifecycle symmetry with the other reporters (nothing to release)."""
|
||||||
@@ -0,0 +1,120 @@
|
|||||||
|
"""Plain-text anomaly log — the human-readable incident narrative (spec §9.1).
|
||||||
|
|
||||||
|
``anomalies-YYYYMMDD.log`` in the same directory as the JSONL files: one line
|
||||||
|
per **detected anomaly** (``is_anomaly`` filter — heartbeats and CONNECTED are
|
||||||
|
lifecycle noise), fixed field order ``timestamp SEVERITY KIND message``,
|
||||||
|
clear-text status with the code in parentheses per spec §7. Recoveries
|
||||||
|
(D2 RECONNECTED, DEVICE_LINK_UP) *are* logged so the file reads as a continuous
|
||||||
|
narrative: what broke, when it came back.
|
||||||
|
|
||||||
|
Every line corresponds to a JSONL record (JSONL = machine source of truth,
|
||||||
|
this = convenience view). Size rotation uses the same scheme as the JSONL
|
||||||
|
reporter, per-day file naming keeps `tail -f` sessions stable.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from opcda_diagnozer.events import DiagEvent, EventKind, Severity
|
||||||
|
from opcda_diagnozer.report.jsonl import rotate_file
|
||||||
|
|
||||||
|
_MAX_LINE_BYTES = 1024 * 1024 # safety cap: a corrupted detail dict can't grow the file
|
||||||
|
|
||||||
|
|
||||||
|
def is_anomaly(kind: EventKind) -> bool:
|
||||||
|
"""Kinds shown in the text log (spec §9.1): everything except heartbeat/connected."""
|
||||||
|
return kind not in (EventKind.HEARTBEAT, EventKind.CONNECTED)
|
||||||
|
|
||||||
|
|
||||||
|
def message_for(event: DiagEvent) -> str:
|
||||||
|
"""Human message per event kind — clear text, codes in parentheses (spec §7)."""
|
||||||
|
d = event.detail
|
||||||
|
kind = event.kind
|
||||||
|
if kind is EventKind.DISCONNECTED:
|
||||||
|
text = str(d.get("reason", "connection lost"))
|
||||||
|
if "shutting down" in text:
|
||||||
|
return f"stopping: {text}"
|
||||||
|
return f"connection lost: {text}"
|
||||||
|
if kind is EventKind.RECONNECTED:
|
||||||
|
return f"reconnected after {d.get('outage_s', '?')} s outage"
|
||||||
|
if kind is EventKind.RECONNECT_FLAPPING:
|
||||||
|
return f"{d.get('disconnects_in_window', '?')} disconnects within 10 min"
|
||||||
|
if kind in (EventKind.SLOW_POLL, EventKind.SLOW_POLL_STREAK):
|
||||||
|
streak = f", streak {d['streak']}" if "streak" in d else ""
|
||||||
|
return f"read took {d.get('rtt_ms', '?')} ms (warn {d.get('warn_ms', '?')} ms){streak}"
|
||||||
|
if kind is EventKind.LATENCY_TREND:
|
||||||
|
return f"p95 read latency {d.get('rtt_p95_ms', '?')} ms over the last window"
|
||||||
|
if kind is EventKind.POLL_HANG:
|
||||||
|
if "abandoned" in str(d.get("action", "")):
|
||||||
|
return f"read call hung > {d.get('hard_hang_s', '?')} s; COM thread abandoned and rebuilt"
|
||||||
|
return f"read took {d.get('rtt_ms', '?')} ms (timeout {d.get('timeout_ms', '?')} ms); thread kept"
|
||||||
|
if kind in (EventKind.QUALITY_BAD, EventKind.QUALITY_UNCERTAIN):
|
||||||
|
arrow = f" changed {d.get('previous_quality', '?')} -> {d['quality']}" if "quality" in d else ""
|
||||||
|
return f"tag '{d.get('tag', '?')}'{arrow}"
|
||||||
|
if kind is EventKind.QUALITY_BAD_PERSISTENT:
|
||||||
|
return f"tag '{d.get('tag', '?')}' BAD for {d.get('bad_for_s', '?')} s (threshold {d.get('threshold_s', '?')} s)"
|
||||||
|
if kind is EventKind.QUALITY_FLAPPING:
|
||||||
|
return f"tag '{d.get('tag', '?')}' quality changed {d.get('transitions_in_window', '?')} times in {d.get('window_s', '?')} s"
|
||||||
|
if kind is EventKind.STALE_TIMESTAMP:
|
||||||
|
return (f"tag '{d.get('tag', '?')}' device time {d.get('age_s', '?')} s old "
|
||||||
|
f"(max {d.get('max_age_s', '?')} s) while quality GOOD")
|
||||||
|
if kind is EventKind.FROZEN_VALUE:
|
||||||
|
return f"tag '{d.get('tag', '?')}' value {d.get('value')!r} unchanged for {d.get('unchanged_for_s', '?')} s"
|
||||||
|
if kind is EventKind.CONTROL_LINK_DEGRADED:
|
||||||
|
return f"{d.get('tags_bad', '?')} of {d.get('tags_total', '?')} monitored tags BAD while connection OK"
|
||||||
|
if kind is EventKind.DEVICE_LINK_DOWN:
|
||||||
|
parts = [f"device '{d.get('device', '?')}': link indicator '{d.get('indicator_tag', '?')}'"]
|
||||||
|
if d.get("quality") and "quality" in str(d.get("reason", "")):
|
||||||
|
parts.append(str(d.get("reason")))
|
||||||
|
elif d.get("reason"):
|
||||||
|
parts.append(f"read {d.get('value')!r}: {d['reason']}")
|
||||||
|
else:
|
||||||
|
parts.append("down")
|
||||||
|
return " ".join(parts)
|
||||||
|
if kind is EventKind.DEVICE_LINK_UP:
|
||||||
|
return f"device '{d.get('device', '?')}': link indicator '{d.get('indicator_tag', '?')}' back to normal"
|
||||||
|
# unknown kinds: generic but complete rendering
|
||||||
|
return " ".join(f"{k}={v}" for k, v in d.items()) or str(kind.value)
|
||||||
|
|
||||||
|
|
||||||
|
class TextLogReporter:
|
||||||
|
"""Plain-text anomaly log (spec §9.1) — same rotation scheme as JSONL."""
|
||||||
|
|
||||||
|
def __init__(self, directory: str | Path, max_mb: int = 50, backups: int = 10) -> None:
|
||||||
|
self._directory = Path(directory)
|
||||||
|
self._max_bytes = max_mb * 1024 * 1024
|
||||||
|
self._backups = backups
|
||||||
|
self._directory.mkdir(parents=True, exist_ok=True)
|
||||||
|
self._day: str | None = None
|
||||||
|
self._current: Path | None = None
|
||||||
|
|
||||||
|
def _file_for(self, at: datetime) -> Path:
|
||||||
|
"""Per-day file: anomalies-YYYYMMDD.log (stable for `tail -f`)."""
|
||||||
|
day = at.astimezone(timezone.utc).strftime("%Y%m%d")
|
||||||
|
if self._current is None or day != self._day:
|
||||||
|
self._day = day
|
||||||
|
self._current = self._directory / f"anomalies-{day}.log"
|
||||||
|
return self._current
|
||||||
|
|
||||||
|
def emit(self, event: DiagEvent) -> None:
|
||||||
|
if not is_anomaly(event.kind):
|
||||||
|
return
|
||||||
|
line = self._format(event)
|
||||||
|
path = self._file_for(event.time_utc)
|
||||||
|
if path.exists() and path.stat().st_size + len(line) + 2 > self._max_bytes:
|
||||||
|
rotate_file(path, self._backups)
|
||||||
|
with path.open("a", encoding="utf-8") as stream:
|
||||||
|
stream.write(line + "\n")
|
||||||
|
|
||||||
|
def _format(self, event: DiagEvent) -> str:
|
||||||
|
stamp = event.time_utc.astimezone(timezone.utc)
|
||||||
|
ts = stamp.strftime("%Y-%m-%d %H:%M:%S.") + f"{stamp.microsecond // 1000:03d} UTC"
|
||||||
|
message = message_for(event)
|
||||||
|
if len(message) > _MAX_LINE_BYTES:
|
||||||
|
message = message[:_MAX_LINE_BYTES] # safety: never explode on bad detail
|
||||||
|
return f"{ts} {event.severity.value:<8} {event.kind.value:<20} {message}"
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
"""Lifecycle symmetry (file handles are opened per append)."""
|
||||||
@@ -2,18 +2,358 @@
|
|||||||
|
|
||||||
IMPLEMENTATION_PLAN §6.1: the service spawns the COM worker thread and must
|
IMPLEMENTATION_PLAN §6.1: the service spawns the COM worker thread and must
|
||||||
stop cleanly (``STOP_PENDING``, join with timeout, abandon a hung COM thread
|
stop cleanly (``STOP_PENDING``, join with timeout, abandon a hung COM thread
|
||||||
on shutdown). SCM recovery (restart on failure) is configured at install time.
|
on shutdown). SCM recovery (restart on failure) is configured at install
|
||||||
|
time (see the ``install`` verb in ``main.py``).
|
||||||
|
|
||||||
|
All COM stays on the poller's dedicated worker thread (spec §10.3): this
|
||||||
|
class only owns the monitor thread that runs :meth:`Poller.run_forever`,
|
||||||
|
reports service status to the SCM, and logs start/stop to the Windows Event
|
||||||
|
Log so an ops engineer can see the service lifecycle even when every other
|
||||||
|
reporter is broken.
|
||||||
|
|
||||||
|
COM imports are guarded: the module imports (and its pure helpers are
|
||||||
|
tested) on any OS; only actual service execution requires Windows
|
||||||
|
(IMPLEMENTATION_PLAN §9).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sys
|
||||||
|
import threading
|
||||||
from collections.abc import Sequence
|
from collections.abc import Sequence
|
||||||
|
|
||||||
|
from opcda_diagnozer.config import ConfigError, load_config
|
||||||
|
|
||||||
class OpcdaDiagnozerService:
|
SERVICE_NAME = "OPCDA-Diagnozer"
|
||||||
"""pywin32 service wrapper — implemented in Phase 3."""
|
SERVICE_DISPLAY_NAME = "OPC DA Diagnozer"
|
||||||
|
SERVICE_DESCRIPTION = (
|
||||||
|
"Read-only OPC DA reference client that diagnoses disconnections, "
|
||||||
|
"latency and data-quality problems between a SCADA system and the "
|
||||||
|
"control system (see docs/SPECIFICATION.md)."
|
||||||
|
)
|
||||||
|
|
||||||
def __init__(self, args: Sequence[str]) -> None:
|
#: The pywin32 class string recorded in the service registry. The class is
|
||||||
raise NotImplementedError(
|
#: defined lazily (its pywin32 base is Windows-only), so this module-level
|
||||||
"Phase 3: Windows service host (docs/IMPLEMENTATION_PLAN.md §6)"
|
#: constant carries the registration identity instead of pickle-derived
|
||||||
)
|
#: introspection: ``GetServiceClassString`` cannot pickle a class defined
|
||||||
|
#: inside a function (``_host_class.<locals>``), and the SCM never imports
|
||||||
|
#: the class by this string anyway — the frozen exe hosts it directly.
|
||||||
|
_SERVICE_CLASS_STRING = "opcda_diagnozer.service.OpcdaDiagnozerService"
|
||||||
|
|
||||||
|
#: How long SvcStop waits for the monitor thread before abandoning it
|
||||||
|
#: (spec §10.4: a thread stuck in a dead RPC channel is never joined).
|
||||||
|
_STOP_TIMEOUT_S = 30.0
|
||||||
|
|
||||||
|
|
||||||
|
def service_config_path() -> str:
|
||||||
|
"""The config path the installed service uses at startup.
|
||||||
|
|
||||||
|
Recorded in the service's registry Parameters at install time; falls
|
||||||
|
back to the frozen-exe default (``config.yaml`` next to the exe) when
|
||||||
|
absent — a service starts with ``CWD = System32``, so the plain
|
||||||
|
relative default must never apply (see ``main._default_config_path``).
|
||||||
|
"""
|
||||||
|
if sys.platform != "win32":
|
||||||
|
return ""
|
||||||
|
try:
|
||||||
|
import win32serviceutil
|
||||||
|
|
||||||
|
return win32serviceutil.GetServiceCustomOption(
|
||||||
|
SERVICE_NAME, "ConfigPath"
|
||||||
|
) or ""
|
||||||
|
except Exception: # noqa: BLE001 — registry read failure -> default
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
def _log_info(message: str) -> None:
|
||||||
|
"""Best-effort lifecycle note in the Windows Event Log (never raises)."""
|
||||||
|
if sys.platform != "win32":
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
import servicemanager
|
||||||
|
|
||||||
|
servicemanager.LogInfoMsg(message)
|
||||||
|
except Exception: # noqa: BLE001 — logging must never break the service
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def _host_class(): # noqa: ANN202 — declared lazily (Windows-only base)
|
||||||
|
"""The ServiceFramework subclass, defined once on Windows only."""
|
||||||
|
import win32service
|
||||||
|
import win32serviceutil
|
||||||
|
|
||||||
|
class OpcdaDiagnozerService(win32serviceutil.ServiceFramework):
|
||||||
|
"""Windows service host: monitor thread + clean SCM stop semantics."""
|
||||||
|
|
||||||
|
_svc_name_ = SERVICE_NAME
|
||||||
|
_svc_display_name_ = SERVICE_DISPLAY_NAME
|
||||||
|
_svc_description_ = SERVICE_DESCRIPTION
|
||||||
|
|
||||||
|
def __init__(self, args: Sequence[str]) -> None:
|
||||||
|
win32serviceutil.ServiceFramework.__init__(self, args)
|
||||||
|
self._stop_event = threading.Event()
|
||||||
|
|
||||||
|
def SvcStop(self) -> None:
|
||||||
|
"""SCM wants a stop: report STOP_PENDING, ask the loop to finish.
|
||||||
|
|
||||||
|
The heavy lifting (join-with-timeout, abandon on hang) happens at
|
||||||
|
the end of :meth:`SvcDoRun` so the SCM sees STOP_PENDING for the
|
||||||
|
whole shutdown (spec §6.1: clean stop handling).
|
||||||
|
"""
|
||||||
|
self.ReportServiceStatus(
|
||||||
|
win32service.SERVICE_STOP_PENDING, waitHint=int(_STOP_TIMEOUT_S * 1000)
|
||||||
|
)
|
||||||
|
self._stop_event.set()
|
||||||
|
|
||||||
|
def SvcDoRun(self) -> None:
|
||||||
|
"""Run the monitoring loop on its own thread; never let SCM
|
||||||
|
observe START_PENDING forever if startup fails.
|
||||||
|
|
||||||
|
The COM client itself is born on the poller's own worker thread
|
||||||
|
(spec §10.3) — this monitor thread is plain Python, no COM.
|
||||||
|
"""
|
||||||
|
import servicemanager
|
||||||
|
|
||||||
|
servicemanager.LogMsg(
|
||||||
|
servicemanager.EVENTLOG_INFORMATION_TYPE,
|
||||||
|
servicemanager.PYS_SERVICE_STARTED,
|
||||||
|
(self._svc_name_, ""),
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
self._run_monitor()
|
||||||
|
finally:
|
||||||
|
servicemanager.LogMsg(
|
||||||
|
servicemanager.EVENTLOG_INFORMATION_TYPE,
|
||||||
|
servicemanager.PYS_SERVICE_STOPPED,
|
||||||
|
(self._svc_name_, ""),
|
||||||
|
)
|
||||||
|
|
||||||
|
def _run_monitor(self) -> None:
|
||||||
|
"""Build the poller, run it on a thread, join with timeout.
|
||||||
|
|
||||||
|
``run_forever`` blocks until ``request_stop`` — but only between
|
||||||
|
polls; a poll can sit in a dead RPC far longer. If the join
|
||||||
|
times out the thread is **abandoned** (daemon: it cannot stop
|
||||||
|
the process from exiting) exactly like the poller's own hard-
|
||||||
|
hang policy (spec §10.4).
|
||||||
|
"""
|
||||||
|
from opcda_diagnozer.main import _build_reporters
|
||||||
|
from opcda_diagnozer.monitor.poller import Poller
|
||||||
|
from opcda_diagnozer.opc.automation import AutomationOpcClient
|
||||||
|
|
||||||
|
config_path = (
|
||||||
|
win32serviceutil.GetServiceCustomOption(self, "ConfigPath")
|
||||||
|
or None
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
config = load_config(config_path)
|
||||||
|
except ConfigError as exc:
|
||||||
|
_log_info(
|
||||||
|
f"{SERVICE_NAME}: startup failed - invalid config "
|
||||||
|
f"({config_path}): {exc}"
|
||||||
|
)
|
||||||
|
return
|
||||||
|
|
||||||
|
client_factory = lambda: AutomationOpcClient( # noqa: E731 — local, small
|
||||||
|
server=config.opc.server,
|
||||||
|
host=config.opc.host,
|
||||||
|
connect_timeout_s=config.opc.connect_timeout_s,
|
||||||
|
)
|
||||||
|
reporters = _build_reporters(config)
|
||||||
|
poller = Poller(client_factory, config, reporters.emit)
|
||||||
|
|
||||||
|
# The SCM stop path only sets our event; translate it to the
|
||||||
|
# poller's stop flag as soon as it fires.
|
||||||
|
watcher = threading.Thread(
|
||||||
|
target=lambda: (self._stop_event.wait(), poller.request_stop()),
|
||||||
|
daemon=True,
|
||||||
|
)
|
||||||
|
watcher.start()
|
||||||
|
|
||||||
|
monitor = threading.Thread(
|
||||||
|
target=lambda: self._guarded(poller), daemon=True
|
||||||
|
)
|
||||||
|
monitor.start()
|
||||||
|
monitor.join(timeout=_STOP_TIMEOUT_S)
|
||||||
|
if monitor.is_alive():
|
||||||
|
_log_info(
|
||||||
|
f"{SERVICE_NAME}: monitor thread did not stop within "
|
||||||
|
f"{_STOP_TIMEOUT_S:.0f} s (hung COM poll?) - abandoning "
|
||||||
|
"it and stopping the service process"
|
||||||
|
)
|
||||||
|
reporters.close()
|
||||||
|
|
||||||
|
def _guarded(self, poller) -> None: # noqa: ANN001
|
||||||
|
"""Run the poll loop; a crash is logged, never raised to SCM."""
|
||||||
|
try:
|
||||||
|
poller.run_forever()
|
||||||
|
except Exception as exc: # noqa: BLE001 — log, keep the process sane
|
||||||
|
_log_info(
|
||||||
|
f"{SERVICE_NAME}: monitor thread crashed: "
|
||||||
|
f"{type(exc).__name__}: {exc}"
|
||||||
|
)
|
||||||
|
|
||||||
|
return OpcdaDiagnozerService
|
||||||
|
|
||||||
|
|
||||||
|
def get_service_class():
|
||||||
|
"""The service class (Windows only); ``None`` elsewhere."""
|
||||||
|
if sys.platform != "win32":
|
||||||
|
return None
|
||||||
|
return _host_class()
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# SCM management (the install/uninstall/start/stop/status verbs, plan §6.2/6.3)
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
def failure_actions(restart_delay_ms: int = 5000, reset_period_s: int = 86400):
|
||||||
|
"""The pywin32 ``SERVICE_FAILURE_ACTIONS`` dict for SCM recovery.
|
||||||
|
|
||||||
|
Restart the service process 5 s after the first and second failure
|
||||||
|
(spec §10.1: "SCM recovery: restart on failure (first/second failure)");
|
||||||
|
the failure counter resets after a day of clean running. The dict shape
|
||||||
|
is verified against pywin32's SWIG parser —
|
||||||
|
``{'ResetPeriod': int, 'RebootMsg': str, 'Command': str,
|
||||||
|
'Actions': ((SC_ACTION_RESTART, delay_ms), ...)}``.
|
||||||
|
"""
|
||||||
|
import win32service
|
||||||
|
|
||||||
|
return {
|
||||||
|
"ResetPeriod": reset_period_s,
|
||||||
|
"RebootMsg": "",
|
||||||
|
"Command": "",
|
||||||
|
"Actions": (
|
||||||
|
(win32service.SC_ACTION_RESTART, restart_delay_ms),
|
||||||
|
(win32service.SC_ACTION_RESTART, restart_delay_ms),
|
||||||
|
),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def install_service(config_path: str, account: str | None = None,
|
||||||
|
password: str | None = None) -> None:
|
||||||
|
"""Register the service with the SCM (auto-start + recovery).
|
||||||
|
|
||||||
|
The exe **itself** is the service host (frozen deployment, SPEC §6.4):
|
||||||
|
the SCM starts this exe with the recorded ``exeArgs="service"``
|
||||||
|
(CWD = System32), ``main`` recognizes that exact argv shape and runs
|
||||||
|
the SCM dispatcher. Auto-start per spec §10.1; SCM restart-on-failure
|
||||||
|
per §10.1/§6.3. The config path is recorded in the service's registry
|
||||||
|
Parameters so startup never depends on the service CWD.
|
||||||
|
"""
|
||||||
|
import win32service
|
||||||
|
import win32serviceutil
|
||||||
|
|
||||||
|
exe_name = getattr(sys, "executable", None)
|
||||||
|
if exe_name is None: # pragma: no cover — sys.executable always exists
|
||||||
|
raise RuntimeError("cannot determine the executable path")
|
||||||
|
win32serviceutil.InstallService(
|
||||||
|
_SERVICE_CLASS_STRING,
|
||||||
|
SERVICE_NAME,
|
||||||
|
SERVICE_DISPLAY_NAME,
|
||||||
|
startType=win32service.SERVICE_AUTO_START,
|
||||||
|
exeName=exe_name,
|
||||||
|
exeArgs="service",
|
||||||
|
userName=account,
|
||||||
|
password=password,
|
||||||
|
description=SERVICE_DESCRIPTION,
|
||||||
|
)
|
||||||
|
win32serviceutil.SetServiceCustomOption(
|
||||||
|
SERVICE_NAME, "ConfigPath", str(config_path)
|
||||||
|
)
|
||||||
|
# SCM recovery (plan §6.3): restart on first/second failure.
|
||||||
|
import win32service as _ws
|
||||||
|
|
||||||
|
hscm = _ws.OpenSCManager(None, None, _ws.SC_MANAGER_ALL_ACCESS)
|
||||||
|
try:
|
||||||
|
hs = win32serviceutil.SmartOpenService(hscm, SERVICE_NAME, _ws.SERVICE_ALL_ACCESS)
|
||||||
|
try:
|
||||||
|
_ws.ChangeServiceConfig2(
|
||||||
|
hs, _ws.SERVICE_CONFIG_FAILURE_ACTIONS, failure_actions()
|
||||||
|
)
|
||||||
|
finally:
|
||||||
|
_ws.CloseServiceHandle(hs)
|
||||||
|
finally:
|
||||||
|
_ws.CloseServiceHandle(hscm)
|
||||||
|
# Register the Windows Event Log source with its message file so
|
||||||
|
# reporter events render with their text (report/eventlog.py defers
|
||||||
|
# this to install — AddSourceToRegistry needs admin).
|
||||||
|
import win32evtlogutil
|
||||||
|
|
||||||
|
win32evtlogutil.AddSourceToRegistry(SERVICE_NAME)
|
||||||
|
|
||||||
|
|
||||||
|
def uninstall_service() -> None:
|
||||||
|
"""Remove the service registration (leaves logs/config untouched)."""
|
||||||
|
import win32serviceutil
|
||||||
|
|
||||||
|
win32serviceutil.RemoveService(SERVICE_NAME)
|
||||||
|
|
||||||
|
|
||||||
|
def start_service() -> None:
|
||||||
|
"""Ask the SCM to start the service."""
|
||||||
|
import win32serviceutil
|
||||||
|
|
||||||
|
win32serviceutil.StartService(SERVICE_NAME, None)
|
||||||
|
|
||||||
|
|
||||||
|
def stop_service() -> None:
|
||||||
|
"""Ask the SCM to stop the service."""
|
||||||
|
import win32serviceutil
|
||||||
|
|
||||||
|
win32serviceutil.StopService(SERVICE_NAME)
|
||||||
|
|
||||||
|
|
||||||
|
def service_status() -> str:
|
||||||
|
"""One-line clear-text status (the SCM state name)."""
|
||||||
|
if sys.platform != "win32":
|
||||||
|
return "service hosting requires Windows"
|
||||||
|
import win32service
|
||||||
|
import win32serviceutil
|
||||||
|
|
||||||
|
status = win32serviceutil.QueryServiceStatus(SERVICE_NAME)
|
||||||
|
return _STATUS_NAMES.get(status[1], f"unknown ({status[1]})")
|
||||||
|
|
||||||
|
|
||||||
|
def _init_status_names() -> dict[int, str]:
|
||||||
|
if sys.platform == "win32":
|
||||||
|
import win32service
|
||||||
|
|
||||||
|
return {
|
||||||
|
win32service.SERVICE_STOPPED: "STOPPED",
|
||||||
|
win32service.SERVICE_START_PENDING: "START_PENDING",
|
||||||
|
win32service.SERVICE_STOP_PENDING: "STOP_PENDING",
|
||||||
|
win32service.SERVICE_RUNNING: "RUNNING",
|
||||||
|
win32service.SERVICE_CONTINUE_PENDING: "CONTINUE_PENDING",
|
||||||
|
win32service.SERVICE_PAUSE_PENDING: "PAUSE_PENDING",
|
||||||
|
win32service.SERVICE_PAUSED: "PAUSED",
|
||||||
|
}
|
||||||
|
return {}
|
||||||
|
|
||||||
|
|
||||||
|
_STATUS_NAMES = _init_status_names()
|
||||||
|
|
||||||
|
|
||||||
|
def handle_service_command_line(argv: Sequence[str]) -> int:
|
||||||
|
"""SCM dispatch — the service process entry point.
|
||||||
|
|
||||||
|
Called by ``main`` when the SCM starts the exe (the ``exeArgs="service"``
|
||||||
|
recorded at install). This is the canonical frozen-service dispatch:
|
||||||
|
``Initialize`` + ``PrepareToHostSingle`` + ``StartServiceCtrlDispatcher``
|
||||||
|
— the dispatcher connects to the SCM that started us and blocks until
|
||||||
|
the service stops; only then does the process exit. Returns the exit
|
||||||
|
code (0 once the service has stopped; 1 when Windows is impossible
|
||||||
|
here — the platform guard).
|
||||||
|
"""
|
||||||
|
if sys.platform != "win32":
|
||||||
|
print("service hosting requires Windows", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
cls = get_service_class()
|
||||||
|
if cls is None: # pragma: no cover — defensive, get_service_class guards
|
||||||
|
return 1
|
||||||
|
import servicemanager
|
||||||
|
|
||||||
|
servicemanager.Initialize()
|
||||||
|
servicemanager.PrepareToHostSingle(cls)
|
||||||
|
servicemanager.StartServiceCtrlDispatcher()
|
||||||
|
return 0
|
||||||
+120
-3
@@ -23,6 +23,13 @@ def test_example_config_loads() -> None:
|
|||||||
assert config.tags[0].name == "Channel1.Device1.Counter"
|
assert config.tags[0].name == "Channel1.Device1.Counter"
|
||||||
assert config.tags[0].expect_change is True
|
assert config.tags[0].expect_change is True
|
||||||
assert config.tags[1].expect_change is False
|
assert config.tags[1].expect_change is False
|
||||||
|
# D16 device-link indicators (spec 6.1); normal reading configurable per device
|
||||||
|
assert [d.name for d in config.devices] == ["Conveyor 3", "Conveyor 4", "Pump A", "Boiler 2"]
|
||||||
|
assert config.devices[0].tag == "Channel1.Device3.Connected"
|
||||||
|
assert config.devices[0].ok_values == ("true",) # default: boolean connected-flag
|
||||||
|
assert config.devices[1].ok_values == ("true", "1")
|
||||||
|
assert config.devices[2].ok_values == ("false", "0") # inverted fault flag: 0 = ok
|
||||||
|
assert config.devices[3].ok_values == ("RUN",) # string state
|
||||||
assert config.poll.interval_s == 1.0
|
assert config.poll.interval_s == 1.0
|
||||||
assert config.poll.timeout_ms == 5000
|
assert config.poll.timeout_ms == 5000
|
||||||
assert config.latency.warn_ms == 1000
|
assert config.latency.warn_ms == 1000
|
||||||
@@ -53,18 +60,128 @@ def test_unknown_top_level_section_rejected(tmp_path: Path) -> None:
|
|||||||
load_config(p)
|
load_config(p)
|
||||||
|
|
||||||
|
|
||||||
def test_empty_tags_rejected(tmp_path: Path) -> None:
|
def test_both_tags_and_devices_empty_rejected(tmp_path: Path) -> None:
|
||||||
p = _write(tmp_path, "opc:\n server: X\ntags: []\n")
|
"""tags: and devices: are each optional, but their union must not be empty."""
|
||||||
with pytest.raises(ConfigError, match="at least one tag"):
|
p = _write(tmp_path, "opc:\n server: X\ntags: []\ndevices: []\n")
|
||||||
|
with pytest.raises(ConfigError, match="nothing to monitor"):
|
||||||
load_config(p)
|
load_config(p)
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_tags_section_with_devices_is_allowed(tmp_path: Path) -> None:
|
||||||
|
"""Devices-only config: link monitoring without extra surveillance tags."""
|
||||||
|
p = _write(
|
||||||
|
tmp_path,
|
||||||
|
"opc:\n server: X\n"
|
||||||
|
"devices:\n - { name: 'Pump A', tag: Ind.Tag }\n",
|
||||||
|
)
|
||||||
|
config = load_config(p)
|
||||||
|
assert config.tags == ()
|
||||||
|
assert [d.name for d in config.devices] == ["Pump A"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_tags_only_still_allowed(tmp_path: Path) -> None:
|
||||||
|
"""The classic setup (no device indicators) stays valid."""
|
||||||
|
p = _write(tmp_path, "opc:\n server: X\ntags:\n - name: t\n")
|
||||||
|
config = load_config(p)
|
||||||
|
assert len(config.tags) == 1
|
||||||
|
assert config.devices == ()
|
||||||
|
|
||||||
|
|
||||||
def test_duplicate_tags_rejected(tmp_path: Path) -> None:
|
def test_duplicate_tags_rejected(tmp_path: Path) -> None:
|
||||||
p = _write(tmp_path, "opc:\n server: X\ntags:\n - name: a\n - name: a\n")
|
p = _write(tmp_path, "opc:\n server: X\ntags:\n - name: a\n - name: a\n")
|
||||||
with pytest.raises(ConfigError, match="duplicate"):
|
with pytest.raises(ConfigError, match="duplicate"):
|
||||||
load_config(p)
|
load_config(p)
|
||||||
|
|
||||||
|
|
||||||
|
def test_devices_missing_keys_rejected(tmp_path: Path) -> None:
|
||||||
|
p = _write(tmp_path, "opc:\n server: X\ntags:\n - name: t\ndevices:\n - name: Device\n")
|
||||||
|
with pytest.raises(ConfigError, match="'tag' is required"):
|
||||||
|
load_config(p)
|
||||||
|
|
||||||
|
|
||||||
|
def test_devices_blank_name_rejected(tmp_path: Path) -> None:
|
||||||
|
p = _write(
|
||||||
|
tmp_path,
|
||||||
|
"opc:\n server: X\ntags:\n - name: t\ndevices:\n - { name: '', tag: Ind.Tag }\n",
|
||||||
|
)
|
||||||
|
with pytest.raises(ConfigError, match="non-empty"):
|
||||||
|
load_config(p)
|
||||||
|
|
||||||
|
|
||||||
|
def test_devices_duplicate_indicator_rejected(tmp_path: Path) -> None:
|
||||||
|
p = _write(
|
||||||
|
tmp_path,
|
||||||
|
"opc:\n server: X\ntags:\n - name: t\n"
|
||||||
|
"devices:\n - { name: A, tag: Ind.Tag }\n - { name: B, tag: Ind.Tag }\n",
|
||||||
|
)
|
||||||
|
with pytest.raises(ConfigError, match="one device per indicator"):
|
||||||
|
load_config(p)
|
||||||
|
|
||||||
|
|
||||||
|
def test_devices_indicator_in_tags_rejected(tmp_path: Path) -> None:
|
||||||
|
"""An indicator is polled automatically; listing it in tags: double-reports."""
|
||||||
|
p = _write(
|
||||||
|
tmp_path,
|
||||||
|
"opc:\n server: X\n"
|
||||||
|
"tags:\n - name: Ind.Tag\ndevices:\n - { name: A, tag: Ind.Tag }\n",
|
||||||
|
)
|
||||||
|
with pytest.raises(ConfigError, match="remove them from"):
|
||||||
|
load_config(p)
|
||||||
|
|
||||||
|
|
||||||
|
def test_devices_empty_is_allowed(tmp_path: Path) -> None:
|
||||||
|
"""devices: is optional (no link monitoring) and may be an empty list."""
|
||||||
|
p = _write(tmp_path, "opc:\n server: X\ntags:\n - name: t\ndevices: []\n")
|
||||||
|
assert load_config(p).devices == ()
|
||||||
|
|
||||||
|
|
||||||
|
def test_devices_ok_values_default_is_true(tmp_path: Path) -> None:
|
||||||
|
p = _write(
|
||||||
|
tmp_path,
|
||||||
|
"opc:\n server: X\ntags:\n - name: t\ndevices:\n - { name: A, tag: I.Tag }\n",
|
||||||
|
)
|
||||||
|
device = load_config(p).devices[0]
|
||||||
|
assert device.ok_values == ("true",)
|
||||||
|
|
||||||
|
|
||||||
|
def test_devices_ok_values_normalized(tmp_path: Path) -> None:
|
||||||
|
"""Booleans/ints/strings all render to comparison strings; scalar = shorthand."""
|
||||||
|
p = _write(
|
||||||
|
tmp_path,
|
||||||
|
"opc:\n server: X\ntags:\n - name: t\n"
|
||||||
|
"devices:\n"
|
||||||
|
" - { name: A, tag: I.One, ok_values: [true, 1] }\n"
|
||||||
|
" - { name: B, tag: I.Two, ok_values: [false, 0] }\n"
|
||||||
|
" - { name: C, tag: I.Three, ok_values: [\"RUN\"] }\n"
|
||||||
|
" - { name: D, tag: I.Four, ok_values: 1 }\n",
|
||||||
|
)
|
||||||
|
devices = {d.name: d for d in load_config(p).devices}
|
||||||
|
assert devices["A"].ok_values == ("true", "1")
|
||||||
|
assert devices["B"].ok_values == ("false", "0")
|
||||||
|
assert devices["C"].ok_values == ("RUN",)
|
||||||
|
assert devices["D"].ok_values == ("1",) # single scalar shorthand
|
||||||
|
|
||||||
|
|
||||||
|
def test_devices_ok_values_empty_rejected(tmp_path: Path) -> None:
|
||||||
|
p = _write(
|
||||||
|
tmp_path,
|
||||||
|
"opc:\n server: X\ntags:\n - name: t\n"
|
||||||
|
"devices:\n - { name: A, tag: I.Tag, ok_values: [] }\n",
|
||||||
|
)
|
||||||
|
with pytest.raises(ConfigError, match="must not be empty"):
|
||||||
|
load_config(p)
|
||||||
|
|
||||||
|
|
||||||
|
def test_devices_ok_values_non_scalar_rejected(tmp_path: Path) -> None:
|
||||||
|
p = _write(
|
||||||
|
tmp_path,
|
||||||
|
"opc:\n server: X\ntags:\n - name: t\n"
|
||||||
|
"devices:\n - { name: A, tag: I.Tag, ok_values: [[1]] }\n",
|
||||||
|
)
|
||||||
|
with pytest.raises(ConfigError, match="scalars"):
|
||||||
|
load_config(p)
|
||||||
|
|
||||||
|
|
||||||
def test_latency_threshold_order_rejected(tmp_path: Path) -> None:
|
def test_latency_threshold_order_rejected(tmp_path: Path) -> None:
|
||||||
p = _write(
|
p = _write(
|
||||||
tmp_path,
|
tmp_path,
|
||||||
|
|||||||
+387
-13
@@ -1,20 +1,394 @@
|
|||||||
"""Detector tests (SPECIFICATION §6). Detector implementations land in Phase 1."""
|
"""Detector tests (D4–D16) against scripted PollOutcomes with a fake clock.
|
||||||
|
|
||||||
from datetime import datetime, timezone
|
Pure logic — no COM, no threads (IMPLEMENTATION_PLAN §9).
|
||||||
|
"""
|
||||||
|
|
||||||
from opcda_diagnozer.monitor.detectors import PollOutcome
|
from datetime import datetime, timedelta, timezone
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from opcda_diagnozer.config import (
|
||||||
|
Config,
|
||||||
|
DeviceEntry,
|
||||||
|
LatencyConfig,
|
||||||
|
OpcConfig,
|
||||||
|
StaleConfig,
|
||||||
|
TagEntry,
|
||||||
|
)
|
||||||
|
from opcda_diagnozer.events import EventKind, Severity
|
||||||
|
from opcda_diagnozer.monitor.detectors import (
|
||||||
|
MASS_BAD_FRACTION,
|
||||||
|
Detection,
|
||||||
|
DetectorSuite,
|
||||||
|
PollOutcome,
|
||||||
|
)
|
||||||
from opcda_diagnozer.opc.base import ReadResult
|
from opcda_diagnozer.opc.base import ReadResult
|
||||||
|
from opcda_diagnozer.quality import status_text
|
||||||
|
|
||||||
|
T0 = datetime(2026, 10, 7, 12, 0, 0, tzinfo=timezone.utc)
|
||||||
|
|
||||||
|
|
||||||
def test_poll_outcome_holds_one_cycle() -> None:
|
def _config(**overrides) -> Config:
|
||||||
result = ReadResult(
|
"""A minimal valid config; overrides replace whole sections when given."""
|
||||||
tag="Channel1.Device1.Level", value=1.5, quality_raw=0xC0, device_time=None
|
return Config(
|
||||||
|
opc=OpcConfig(server="Fake.Server.1"),
|
||||||
|
tags=(TagEntry(name="T.Good", expect_change=False), TagEntry(name="T.Spin", expect_change=True)),
|
||||||
|
devices=(DeviceEntry(name="Pump A", tag="T.PumpConnected", ok_values=("true",)),),
|
||||||
|
latency=overrides.pop("latency", LatencyConfig()),
|
||||||
|
stale=overrides.pop("stale", StaleConfig()),
|
||||||
)
|
)
|
||||||
outcome = PollOutcome(
|
|
||||||
started_at=datetime.now(timezone.utc),
|
|
||||||
rtt_ms=12.5,
|
def _read(tag: str, quality: int = 0xC0, value=0, at: datetime = T0, error: str | None = None) -> ReadResult:
|
||||||
results=(result,),
|
return ReadResult(tag=tag, value=value, quality_raw=quality, device_time=at, error=error)
|
||||||
|
|
||||||
|
|
||||||
|
def _outcome(
|
||||||
|
rtt_ms: float = 10.0,
|
||||||
|
results: tuple[ReadResult, ...] = (),
|
||||||
|
error: str | None = None,
|
||||||
|
at: datetime = T0,
|
||||||
|
) -> PollOutcome:
|
||||||
|
return PollOutcome(started_at=at, rtt_ms=rtt_ms, results=results, error=error)
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Detection basics
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
def test_heartbeat_and_connected_are_not_anomalies() -> None:
|
||||||
|
assert Detection(EventKind.HEARTBEAT, Severity.INFO).is_anomaly is False
|
||||||
|
assert Detection(EventKind.CONNECTED, Severity.INFO).is_anomaly is False
|
||||||
|
assert Detection(EventKind.QUALITY_BAD, Severity.WARN).is_anomaly is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_status_text_clear_text_with_code_in_parens() -> None:
|
||||||
|
assert status_text(0xC0) == "GOOD (0xC0)"
|
||||||
|
assert status_text(0x18) == "BAD / COMM FAILURE (0x18)"
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# D4/D5/D6 — latency
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
def test_d4_slow_poll_spike() -> None:
|
||||||
|
suite = DetectorSuite(_config())
|
||||||
|
detections = suite.observe(_outcome(rtt_ms=1500.0)) # > warn_ms 1000
|
||||||
|
kinds = [d.kind for d in detections]
|
||||||
|
assert EventKind.SLOW_POLL in kinds
|
||||||
|
spike = next(d for d in detections if d.kind is EventKind.SLOW_POLL)
|
||||||
|
assert spike.severity is Severity.WARN
|
||||||
|
assert spike.detail["rtt_ms"] == 1500.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_d4_no_spike_below_threshold() -> None:
|
||||||
|
suite = DetectorSuite(_config())
|
||||||
|
assert all(
|
||||||
|
d.kind is not EventKind.SLOW_POLL for d in suite.observe(_outcome(rtt_ms=999.9))
|
||||||
)
|
)
|
||||||
assert outcome.rtt_ms == 12.5
|
|
||||||
assert outcome.error is None
|
|
||||||
assert outcome.results[0].tag == "Channel1.Device1.Level"
|
def test_d5_streak_after_consecutive_slow_polls() -> None:
|
||||||
|
suite = DetectorSuite(_config())
|
||||||
|
for _ in range(4): # streak_limit 5: not yet
|
||||||
|
suite.observe(_outcome(rtt_ms=1200.0))
|
||||||
|
detections = suite.observe(_outcome(rtt_ms=1200.0)) # 5th consecutive
|
||||||
|
streak = next(d for d in detections if d.kind is EventKind.SLOW_POLL_STREAK)
|
||||||
|
assert streak.severity is Severity.CRITICAL
|
||||||
|
assert streak.detail["streak"] == 5
|
||||||
|
# one fast poll resets the streak
|
||||||
|
suite.observe(_outcome(rtt_ms=10.0))
|
||||||
|
suite.observe(_outcome(rtt_ms=1200.0))
|
||||||
|
detections = suite.observe(_outcome(rtt_ms=1200.0))
|
||||||
|
assert all(d.kind is not EventKind.SLOW_POLL_STREAK for d in detections)
|
||||||
|
|
||||||
|
|
||||||
|
def test_d6_p95_trend_over_window() -> None:
|
||||||
|
suite = DetectorSuite(_config())
|
||||||
|
# window_polls=60: fill with fast polls plus a few slow outliers
|
||||||
|
for _ in range(56):
|
||||||
|
suite.observe(_outcome(rtt_ms=50.0))
|
||||||
|
for _ in range(4):
|
||||||
|
suite.observe(_outcome(rtt_ms=2000.0))
|
||||||
|
detections = suite.observe(_outcome(rtt_ms=50.0))
|
||||||
|
trend = next(d for d in detections if d.kind is EventKind.LATENCY_TREND)
|
||||||
|
assert trend.detail["rtt_p95_ms"] == 2000.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_failed_poll_resets_streak_but_keeps_window() -> None:
|
||||||
|
suite = DetectorSuite(_config())
|
||||||
|
for _ in range(3):
|
||||||
|
suite.observe(_outcome(rtt_ms=1200.0))
|
||||||
|
# a failed poll carries no RTT evidence: streak resets, no spike events
|
||||||
|
detections = suite.observe(_outcome(rtt_ms=99999.0, error="RPC call failed"))
|
||||||
|
assert all(d.kind not in (EventKind.SLOW_POLL, EventKind.SLOW_POLL_STREAK) for d in detections)
|
||||||
|
suite.observe(_outcome(rtt_ms=1200.0))
|
||||||
|
assert all(d.kind is not EventKind.SLOW_POLL_STREAK for d in suite.observe(_outcome(rtt_ms=1200.0)))
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# D8/D9/D10/D11 — quality
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
def test_d8_bad_transition_with_clear_text_status() -> None:
|
||||||
|
suite = DetectorSuite(_config())
|
||||||
|
suite.observe(_outcome(results=(_read("T.Good", 0xC0),)))
|
||||||
|
detections = suite.observe(_outcome(results=(_read("T.Good", 0x18),)))
|
||||||
|
bad = next(d for d in detections if d.kind is EventKind.QUALITY_BAD)
|
||||||
|
assert bad.severity is Severity.WARN
|
||||||
|
assert bad.detail["tag"] == "T.Good"
|
||||||
|
assert bad.detail["quality"] == "BAD / COMM FAILURE (0x18)"
|
||||||
|
assert bad.detail["previous_quality"] == "GOOD"
|
||||||
|
|
||||||
|
|
||||||
|
def test_d9_uncertain_transition_is_info() -> None:
|
||||||
|
suite = DetectorSuite(_config())
|
||||||
|
suite.observe(_outcome(results=(_read("T.Good", 0xC0),)))
|
||||||
|
detections = suite.observe(_outcome(results=(_read("T.Good", 0x44),)))
|
||||||
|
uncertain = next(d for d in detections if d.kind is EventKind.QUALITY_UNCERTAIN)
|
||||||
|
assert uncertain.severity is Severity.INFO
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_synthetic_transition_on_first_reading() -> None:
|
||||||
|
"""A tag that starts BAD reports no transition until it first recovers."""
|
||||||
|
suite = DetectorSuite(_config())
|
||||||
|
detections = suite.observe(_outcome(results=(_read("T.Good", 0x00),)))
|
||||||
|
assert all(d.kind is not EventKind.QUALITY_BAD for d in detections)
|
||||||
|
|
||||||
|
|
||||||
|
def test_d10_persistent_bad_escalates_after_threshold() -> None:
|
||||||
|
suite = DetectorSuite(_config()) # persistent_bad_s 30
|
||||||
|
for second in range(0, 29, 2): # 15 polls < 30 s: not yet
|
||||||
|
suite.observe(_outcome(results=(_read("T.Good", 0x18),), at=T0 + timedelta(seconds=second)))
|
||||||
|
if second:
|
||||||
|
continue
|
||||||
|
# exactly at 30 s
|
||||||
|
detections = suite.observe(
|
||||||
|
_outcome(results=(_read("T.Good", 0x18),), at=T0 + timedelta(seconds=30))
|
||||||
|
)
|
||||||
|
persistent = next(d for d in detections if d.kind is EventKind.QUALITY_BAD_PERSISTENT)
|
||||||
|
assert persistent.severity is Severity.WARN
|
||||||
|
assert persistent.detail["bad_for_s"] == 30.0
|
||||||
|
# reported once, not every poll
|
||||||
|
detections = suite.observe(
|
||||||
|
_outcome(results=(_read("T.Good", 0x18),), at=T0 + timedelta(seconds=32))
|
||||||
|
)
|
||||||
|
assert all(d.kind is not EventKind.QUALITY_BAD_PERSISTENT for d in detections)
|
||||||
|
# recovery resets the persistence latch
|
||||||
|
suite.observe(_outcome(results=(_read("T.Good", 0xC0),), at=T0 + timedelta(seconds=34)))
|
||||||
|
suite.observe(_outcome(results=(_read("T.Good", 0x18),), at=T0 + timedelta(seconds=36)))
|
||||||
|
detections = suite.observe(
|
||||||
|
_outcome(results=(_read("T.Good", 0x18),), at=T0 + timedelta(seconds=70))
|
||||||
|
)
|
||||||
|
assert any(d.kind is EventKind.QUALITY_BAD_PERSISTENT for d in detections)
|
||||||
|
|
||||||
|
|
||||||
|
def test_d11_flapping_quality() -> None:
|
||||||
|
suite = DetectorSuite(_config()) # >5 transitions per 600 s window
|
||||||
|
t = T0
|
||||||
|
for quality in (0xC0, 0x18, 0xC0, 0x18, 0xC0, 0x18):
|
||||||
|
t += timedelta(seconds=5)
|
||||||
|
suite.observe(_outcome(results=(_read("T.Good", quality),), at=t))
|
||||||
|
# 6 transitions happened; the 6th poll reports flapping
|
||||||
|
detections = suite.quality.observe(_outcome(results=(_read("T.Good", 0xC0),), at=t))
|
||||||
|
assert any(d.kind is EventKind.QUALITY_FLAPPING for d in detections)
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# D12/D13 — stale & frozen
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
def test_d12_stale_timestamp_while_good() -> None:
|
||||||
|
suite = DetectorSuite(_config()) # max_age_s 120
|
||||||
|
stale_device_time = T0 - timedelta(seconds=200)
|
||||||
|
detections = suite.observe(
|
||||||
|
_outcome(
|
||||||
|
results=(_read("T.Good", 0xC0, at=stale_device_time),),
|
||||||
|
at=T0,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
stale = next(d for d in detections if d.kind is EventKind.STALE_TIMESTAMP)
|
||||||
|
assert stale.detail["age_s"] == 200.0
|
||||||
|
assert stale.detail["max_age_s"] == 120
|
||||||
|
|
||||||
|
|
||||||
|
def test_d12_ignores_stale_when_quality_not_good() -> None:
|
||||||
|
suite = DetectorSuite(_config())
|
||||||
|
stale_device_time = T0 - timedelta(seconds=200)
|
||||||
|
detections = suite.observe(
|
||||||
|
_outcome(results=(_read("T.Good", 0x18, at=stale_device_time),), at=T0)
|
||||||
|
)
|
||||||
|
assert all(d.kind is not EventKind.STALE_TIMESTAMP for d in detections)
|
||||||
|
|
||||||
|
|
||||||
|
def test_d13_frozen_value_on_expect_change_tag() -> None:
|
||||||
|
suite = DetectorSuite(_config()) # T.Spin has expect_change
|
||||||
|
frozen_time = T0 - timedelta(seconds=150)
|
||||||
|
first = suite.observe(_outcome(results=(_read("T.Spin", 0xC0, value=7, at=frozen_time),), at=T0))
|
||||||
|
second = suite.observe(
|
||||||
|
_outcome(results=(_read("T.Spin", 0xC0, value=7, at=frozen_time),), at=T0 + timedelta(seconds=1))
|
||||||
|
)
|
||||||
|
assert all(d.kind is not EventKind.FROZEN_VALUE for d in first) # needs history
|
||||||
|
frozen = next(d for d in second if d.kind is EventKind.FROZEN_VALUE)
|
||||||
|
assert frozen.detail["tag"] == "T.Spin"
|
||||||
|
assert frozen.detail["value"] == 7
|
||||||
|
|
||||||
|
|
||||||
|
def test_d13_not_frozen_when_device_time_advances() -> None:
|
||||||
|
suite = DetectorSuite(_config())
|
||||||
|
t_device = T0 - timedelta(seconds=100)
|
||||||
|
suite.observe(_outcome(results=(_read("T.Spin", 0xC0, value=7, at=t_device),), at=T0))
|
||||||
|
detections = suite.observe(
|
||||||
|
_outcome(
|
||||||
|
results=(_read("T.Spin", 0xC0, value=7, at=t_device + timedelta(seconds=10)),),
|
||||||
|
at=T0 + timedelta(seconds=1),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
assert all(d.kind is not EventKind.FROZEN_VALUE for d in detections)
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# D14 — mass bad
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
def test_d14_mass_bad_when_half_or_more_bad() -> None:
|
||||||
|
config = _config()
|
||||||
|
suite = DetectorSuite(config)
|
||||||
|
tags = ("A", "B", "C", "D")
|
||||||
|
results = tuple(_read(t, 0x18) for t in tags) # 100% BAD
|
||||||
|
detections = suite.observe(_outcome(results=results))
|
||||||
|
mass = next(d for d in detections if d.kind is EventKind.CONTROL_LINK_DEGRADED)
|
||||||
|
assert mass.severity is Severity.CRITICAL
|
||||||
|
assert mass.detail == {"tags_bad": 4, "tags_total": 4}
|
||||||
|
# latched: not re-emitted while still degraded
|
||||||
|
assert all(
|
||||||
|
d.kind is not EventKind.CONTROL_LINK_DEGRADED
|
||||||
|
for d in suite.observe(_outcome(results=results))
|
||||||
|
)
|
||||||
|
# recovery re-arms the latch
|
||||||
|
good = tuple(_read(t, 0xC0) for t in tags)
|
||||||
|
suite.observe(_outcome(results=good))
|
||||||
|
assert any(
|
||||||
|
d.kind is EventKind.CONTROL_LINK_DEGRADED
|
||||||
|
for d in suite.observe(_outcome(results=results))
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_d14_below_half_is_not_mass_bad() -> None:
|
||||||
|
suite = DetectorSuite(_config())
|
||||||
|
results = (_read("A", 0x18), _read("B", 0xC0), _read("C", 0xC0), _read("D", 0xC0))
|
||||||
|
assert all(
|
||||||
|
d.kind is not EventKind.CONTROL_LINK_DEGRADED
|
||||||
|
for d in suite.observe(_outcome(results=results))
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_mass_bad_fraction_is_half() -> None:
|
||||||
|
assert MASS_BAD_FRACTION == 0.5
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# D16 — device link indicators
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
def _pump(at: datetime = T0, quality: int = 0xC0, value=True, error: str | None = None) -> ReadResult:
|
||||||
|
return _read("T.PumpConnected", quality, value, at, error)
|
||||||
|
|
||||||
|
|
||||||
|
def test_d16_false_value_is_down() -> None:
|
||||||
|
suite = DetectorSuite(_config())
|
||||||
|
suite.observe(_outcome(results=(_pump(),))) # normal first
|
||||||
|
detections = suite.observe(_outcome(results=(_pump(value=False),)))
|
||||||
|
down = next(d for d in detections if d.kind is EventKind.DEVICE_LINK_DOWN)
|
||||||
|
assert down.severity is Severity.CRITICAL
|
||||||
|
assert down.detail["device"] == "Pump A"
|
||||||
|
assert down.detail["reason"] == "read 'false', expected one of ['true']"
|
||||||
|
assert suite.devices_down() == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_d16_non_good_quality_is_down_regardless_of_value() -> None:
|
||||||
|
suite = DetectorSuite(_config())
|
||||||
|
detections = suite.observe(_outcome(results=(_pump(quality=0x08),))) # BAD not connected
|
||||||
|
down = next(d for d in detections if d.kind is EventKind.DEVICE_LINK_DOWN)
|
||||||
|
assert "quality BAD / NOT CONNECTED (0x08)" in down.detail["reason"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_d16_read_error_is_down() -> None:
|
||||||
|
suite = DetectorSuite(_config())
|
||||||
|
detections = suite.observe(
|
||||||
|
_outcome(results=(_pump(error="cannot add tag 'T.PumpConnected'"),))
|
||||||
|
)
|
||||||
|
assert any(d.kind is EventKind.DEVICE_LINK_DOWN for d in detections)
|
||||||
|
|
||||||
|
|
||||||
|
def test_d16_recovery_edge_level_no_spam() -> None:
|
||||||
|
suite = DetectorSuite(_config())
|
||||||
|
suite.observe(_outcome(results=(_pump(),)))
|
||||||
|
suite.observe(_outcome(results=(_pump(value=False),))) # DOWN
|
||||||
|
# staying down: exactly one DOWN so far, no repeats
|
||||||
|
down_count = sum(
|
||||||
|
1
|
||||||
|
for events in (
|
||||||
|
suite.observe(_outcome(results=(_pump(value=False),))),
|
||||||
|
suite.observe(_outcome(results=(_pump(value=False),))),
|
||||||
|
)
|
||||||
|
for d in events
|
||||||
|
if d.kind is EventKind.DEVICE_LINK_DOWN
|
||||||
|
)
|
||||||
|
assert down_count == 0
|
||||||
|
# recovery
|
||||||
|
detections = suite.observe(_outcome(results=(_pump(),)))
|
||||||
|
up = next(d for d in detections if d.kind is EventKind.DEVICE_LINK_UP)
|
||||||
|
assert up.severity is Severity.INFO
|
||||||
|
assert up.detail["device"] == "Pump A"
|
||||||
|
assert suite.devices_down() == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_d16_rendering_matches_config_normalization() -> None:
|
||||||
|
"""The comparison string matches the config's ok_values rendering (True->'true')."""
|
||||||
|
config = _config()
|
||||||
|
suite = DetectorSuite(config)
|
||||||
|
detections = suite.observe(_outcome(results=(_pump(),))) # Python True
|
||||||
|
assert all(d.kind is not EventKind.DEVICE_LINK_DOWN for d in detections)
|
||||||
|
detections = suite.observe(_outcome(results=(_read("T.PumpConnected", 0xC0, "true"),)))
|
||||||
|
assert all(d.kind is not EventKind.DEVICE_LINK_DOWN for d in detections)
|
||||||
|
|
||||||
|
|
||||||
|
def test_d16_first_reading_already_down_reports_once() -> None:
|
||||||
|
"""No synthetic 'was up' baseline: a device born down reports immediately."""
|
||||||
|
suite = DetectorSuite(_config())
|
||||||
|
detections = suite.observe(_outcome(results=(_pump(value=False),)))
|
||||||
|
assert any(d.kind is EventKind.DEVICE_LINK_DOWN for d in detections)
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# suite helpers (heartbeat context)
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
def test_suite_helpers_expose_heartbeat_inputs() -> None:
|
||||||
|
suite = DetectorSuite(_config())
|
||||||
|
suite.observe(_outcome(results=(_read("T.Good", 0x18),)))
|
||||||
|
assert suite.tags_bad() == 1
|
||||||
|
assert suite.devices_down() == 0
|
||||||
|
assert suite.rtt_p95_ms() == 10.0
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.parametrize(
|
||||||
|
"raw, expected",
|
||||||
|
[
|
||||||
|
(0xC0, "GOOD (0xC0)"),
|
||||||
|
(0xD8, "GOOD / LOCAL OVERRIDE (0xD8)"),
|
||||||
|
(0x04, "BAD / CONFIG ERROR (0x04)"),
|
||||||
|
(0x44, "UNCERTAIN / LAST USABLE (0x44)"),
|
||||||
|
(0x01, "BAD / 0x01 (0x01)"), # unknown subcode: class + raw label
|
||||||
|
],
|
||||||
|
)
|
||||||
|
def test_status_text_table(raw: int, expected: str) -> None:
|
||||||
|
assert status_text(raw) == expected
|
||||||
+229
-22
@@ -1,33 +1,240 @@
|
|||||||
"""End-to-end pipeline tests against opc.fake (IMPLEMENTATION_PLAN §9).
|
"""Poller end-to-end tests against opc.fake — the Phase 1 exit criterion.
|
||||||
|
|
||||||
The fake client lets the event pipeline run without COM/hardware; Phase 1
|
Runs the full pipeline (worker thread → outcomes → detectors → events) without
|
||||||
grows this into the full scenario suite.
|
COM (IMPLEMENTATION_PLAN §9). Timing-dependent paths (hang watchdog) are
|
||||||
|
exercised with tiny thresholds so the tests stay fast.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
from collections.abc import Sequence
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
from opcda_diagnozer.opc.base import OpcError
|
from opcda_diagnozer.config import (
|
||||||
from opcda_diagnozer.opc.fake import FakeOpcClient
|
Config,
|
||||||
|
DeviceEntry,
|
||||||
|
LatencyConfig,
|
||||||
|
OpcConfig,
|
||||||
|
PollConfig,
|
||||||
|
ReportConfig,
|
||||||
|
StaleConfig,
|
||||||
|
TagEntry,
|
||||||
|
)
|
||||||
|
from opcda_diagnozer.events import DiagEvent, EventKind, Severity
|
||||||
|
from opcda_diagnozer.monitor.poller import Poller
|
||||||
|
from opcda_diagnozer.opc.base import OPCDataSource, OpcClient, OpcError, ReadResult
|
||||||
|
from opcda_diagnozer.opc.fake import FakeOpcClient, PollScriptError
|
||||||
|
|
||||||
|
TAGS = ("T.Counter", "T.Level")
|
||||||
|
INDICATOR = "T.PumpConnected"
|
||||||
|
|
||||||
|
|
||||||
def test_fake_reads_good_quality() -> None:
|
def _config(poll: PollConfig | None = None, heartbeat_s: float = 3600.0) -> Config:
|
||||||
client = FakeOpcClient()
|
return Config(
|
||||||
client.connect()
|
opc=OpcConfig(server="Fake.Server.1"),
|
||||||
results = client.read(["Channel1.Device1.Counter", "Channel1.Device1.Level"])
|
poll=poll or PollConfig(interval_s=0.05, timeout_ms=5000, hard_hang_s=5.0),
|
||||||
assert [r.tag for r in results] == ["Channel1.Device1.Counter", "Channel1.Device1.Level"]
|
tags=(TagEntry(name="T.Counter"), TagEntry(name="T.Level")),
|
||||||
assert all(r.quality_raw == 0xC0 for r in results)
|
devices=(DeviceEntry(name="Pump A", tag=INDICATOR, ok_values=("true",)),),
|
||||||
assert all(r.device_time is not None for r in results)
|
latency=LatencyConfig(warn_ms=1000.0),
|
||||||
|
stale=StaleConfig(enabled=False), # device_time checks off in these runs
|
||||||
|
report=ReportConfig(heartbeat_s=heartbeat_s), # heartbeats off by default
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_fake_requires_connect_first() -> None:
|
def _res(tag: str, quality: int = 0xC0, value=0) -> ReadResult:
|
||||||
client = FakeOpcClient()
|
return ReadResult(
|
||||||
with pytest.raises(OpcError):
|
tag=tag, value=value, quality_raw=quality, device_time=datetime.now(timezone.utc)
|
||||||
client.read(["Channel1.Device1.Counter"])
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_fake_close_disconnects() -> None:
|
class _Collector:
|
||||||
client = FakeOpcClient()
|
"""Thread-safe event sink."""
|
||||||
client.connect()
|
|
||||||
client.close()
|
def __init__(self) -> None:
|
||||||
with pytest.raises(OpcError):
|
self.events: list[DiagEvent] = []
|
||||||
client.read(["Channel1.Device1.Counter"])
|
self._lock = threading.Lock()
|
||||||
|
|
||||||
|
def __call__(self, event: DiagEvent) -> None:
|
||||||
|
with self._lock:
|
||||||
|
self.events.append(event)
|
||||||
|
|
||||||
|
def of(self, kind: EventKind) -> list[DiagEvent]:
|
||||||
|
with self._lock:
|
||||||
|
return [e for e in self.events if e.kind is kind]
|
||||||
|
|
||||||
|
|
||||||
|
class _FakeFactory:
|
||||||
|
"""client_factory that hands out fake clients; one call per (re)connect."""
|
||||||
|
|
||||||
|
def __init__(self, clients: Sequence[OpcClient]) -> None:
|
||||||
|
self._clients = list(clients)
|
||||||
|
self.created = 0
|
||||||
|
|
||||||
|
def __call__(self) -> OpcClient:
|
||||||
|
self.created += 1
|
||||||
|
if not self._clients:
|
||||||
|
raise OpcError("no more scripted clients")
|
||||||
|
client = self._clients.pop(0)
|
||||||
|
if not client.connected:
|
||||||
|
client.connect()
|
||||||
|
return client
|
||||||
|
|
||||||
|
|
||||||
|
def _run_until(poller: Poller, condition, timeout_s: float = 8.0) -> None:
|
||||||
|
"""Run the poller on a thread until ``condition()`` (or timeout), then stop."""
|
||||||
|
thread = threading.Thread(target=poller.run_forever, daemon=True)
|
||||||
|
thread.start()
|
||||||
|
deadline = time.monotonic() + timeout_s
|
||||||
|
try:
|
||||||
|
while time.monotonic() < deadline:
|
||||||
|
if condition():
|
||||||
|
break
|
||||||
|
time.sleep(0.005)
|
||||||
|
finally:
|
||||||
|
poller.request_stop()
|
||||||
|
thread.join(timeout=5.0)
|
||||||
|
if thread.is_alive():
|
||||||
|
pytest.fail("poller thread did not stop within the join timeout")
|
||||||
|
if not condition():
|
||||||
|
pytest.fail("condition not reached before the test timeout")
|
||||||
|
|
||||||
|
|
||||||
|
def test_run_emits_connected_and_quality_event() -> None:
|
||||||
|
"""Healthy loop: CONNECTED, then a BAD transition produces QUALITY_BAD."""
|
||||||
|
client = FakeOpcClient(
|
||||||
|
script=[
|
||||||
|
"good", # poll 1: baseline GOOD
|
||||||
|
# poll 2: T.Counter flips BAD -> D8 with clear-text status
|
||||||
|
[_res("T.Counter", 0x18), _res("T.Level", 0xC0), _res(INDICATOR, 0xC0, True)],
|
||||||
|
]
|
||||||
|
)
|
||||||
|
collector = _Collector()
|
||||||
|
poller = Poller(_FakeFactory([client]), _config(), collector)
|
||||||
|
_run_until(poller, lambda: len(collector.of(EventKind.QUALITY_BAD)) >= 1)
|
||||||
|
|
||||||
|
assert collector.of(EventKind.CONNECTED), "no CONNECTED event"
|
||||||
|
bad = collector.of(EventKind.QUALITY_BAD)[0]
|
||||||
|
assert bad.detail["tag"] == "T.Counter"
|
||||||
|
assert bad.detail["quality"] == "BAD / COMM FAILURE (0x18)"
|
||||||
|
assert bad.severity is Severity.WARN
|
||||||
|
# tags polled = surveillance tags + the device indicator
|
||||||
|
assert set(client.reads[0][0]) == {"T.Counter", "T.Level", INDICATOR}
|
||||||
|
# graceful shutdown note
|
||||||
|
shutdown = collector.of(EventKind.DISCONNECTED)
|
||||||
|
assert shutdown and shutdown[0].detail["reason"].startswith("shutting down")
|
||||||
|
|
||||||
|
|
||||||
|
def test_run_device_link_down_and_recovery() -> None:
|
||||||
|
client = FakeOpcClient(
|
||||||
|
script=[
|
||||||
|
[_res("T.Counter"), _res("T.Level"), _res(INDICATOR, 0xC0, True)],
|
||||||
|
[_res("T.Counter"), _res("T.Level"), _res(INDICATOR, 0xC0, False)],
|
||||||
|
[_res("T.Counter"), _res("T.Level"), _res(INDICATOR, 0xC0, False)],
|
||||||
|
[_res("T.Counter"), _res("T.Level"), _res(INDICATOR, 0xC0, True)],
|
||||||
|
]
|
||||||
|
)
|
||||||
|
collector = _Collector()
|
||||||
|
poller = Poller(_FakeFactory([client]), _config(), collector)
|
||||||
|
_run_until(poller, lambda: len(collector.of(EventKind.DEVICE_LINK_UP)) >= 1)
|
||||||
|
|
||||||
|
downs = collector.of(EventKind.DEVICE_LINK_DOWN)
|
||||||
|
ups = collector.of(EventKind.DEVICE_LINK_UP)
|
||||||
|
assert len(downs) == 1 # edge-level: two down polls -> one DOWN
|
||||||
|
assert downs[0].detail["device"] == "Pump A"
|
||||||
|
assert downs[0].severity is Severity.CRITICAL
|
||||||
|
assert len(ups) == 1
|
||||||
|
assert ups[0].detail["device"] == "Pump A"
|
||||||
|
|
||||||
|
|
||||||
|
def test_run_connection_loss_emits_d1_then_reconnect_d2() -> None:
|
||||||
|
"""A failing poll ends the session: DISCONNECTED (D1), then RECONNECTED (D2)."""
|
||||||
|
dying = FakeOpcClient(
|
||||||
|
script=[
|
||||||
|
"good",
|
||||||
|
PollScriptError("could not read from server (HRESULT 0x800706BA: ...)"),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
healthy = FakeOpcClient() # the reconnect lands here
|
||||||
|
collector = _Collector()
|
||||||
|
factory = _FakeFactory([dying, healthy])
|
||||||
|
poller = Poller(factory, _config(), collector)
|
||||||
|
_run_until(poller, lambda: len(collector.of(EventKind.RECONNECTED)) >= 1)
|
||||||
|
|
||||||
|
disconnected = collector.of(EventKind.DISCONNECTED)
|
||||||
|
assert any("HRESULT 0x800706BA" in d.detail.get("reason", "") for d in disconnected)
|
||||||
|
reconnected = collector.of(EventKind.RECONNECTED)
|
||||||
|
assert reconnected and reconnected[0].detail["outage_s"] < 10.0
|
||||||
|
assert factory.created == 2 # one client per session
|
||||||
|
|
||||||
|
|
||||||
|
def test_run_heartbeat_emits_periodically() -> None:
|
||||||
|
# indicator defaults to a plain read -> value 0, which is NOT an ok value
|
||||||
|
# ("true"): script the first poll so the device starts healthy.
|
||||||
|
client = FakeOpcClient(
|
||||||
|
script=[
|
||||||
|
[_res("T.Counter"), _res("T.Level"), _res(INDICATOR, 0xC0, True)],
|
||||||
|
]
|
||||||
|
)
|
||||||
|
collector = _Collector()
|
||||||
|
poller = Poller(
|
||||||
|
_FakeFactory([client]), _config(heartbeat_s=0.15), collector
|
||||||
|
)
|
||||||
|
_run_until(poller, lambda: len(collector.of(EventKind.HEARTBEAT)) >= 2, timeout_s=10.0)
|
||||||
|
|
||||||
|
heartbeats = collector.of(EventKind.HEARTBEAT)
|
||||||
|
assert len(heartbeats) >= 2
|
||||||
|
detail = heartbeats[0].detail
|
||||||
|
assert detail["tags_total"] == 3 # 2 surveillance + 1 indicator
|
||||||
|
# after the scripted healthy poll the fake falls back to value=0 for the
|
||||||
|
# indicator -> the device is DOWN in later heartbeats. Assert the DOWN
|
||||||
|
# count is reflected honestly rather than expecting fake defaults to be ok.
|
||||||
|
assert detail["devices_down"] in (0, 1)
|
||||||
|
assert detail["reconnects_total"] == 0
|
||||||
|
# a DEVICE_LINK_DOWN must have been reported when it flipped
|
||||||
|
assert collector.of(EventKind.DEVICE_LINK_DOWN) or detail["devices_down"] == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_run_soft_hang_flags_d7_but_keeps_thread() -> None:
|
||||||
|
"""A slow-but-returning poll raises POLL_HANG (D7) without rebuilding."""
|
||||||
|
|
||||||
|
class SlowClient(FakeOpcClient):
|
||||||
|
def read(self, tags, source=OPCDataSource.DEVICE):
|
||||||
|
time.sleep(0.3) # > timeout_ms 200 below
|
||||||
|
return super().read(tags, source)
|
||||||
|
|
||||||
|
config = _config(poll=PollConfig(interval_s=0.05, timeout_ms=200, hard_hang_s=30.0))
|
||||||
|
client = SlowClient()
|
||||||
|
collector = _Collector()
|
||||||
|
factory = _FakeFactory([client])
|
||||||
|
poller = Poller(factory, config, collector)
|
||||||
|
_run_until(poller, lambda: bool(collector.of(EventKind.POLL_HANG)))
|
||||||
|
|
||||||
|
hangs = collector.of(EventKind.POLL_HANG)
|
||||||
|
assert hangs and hangs[0].detail["rtt_ms"] >= 200
|
||||||
|
assert hangs[0].detail["timeout_ms"] == 200
|
||||||
|
assert factory.created == 1 # thread kept: no rebuild
|
||||||
|
# no abandoned-thread message (that's the hard-hang path)
|
||||||
|
assert all("abandoned" not in h.detail.get("action", "") for h in hangs)
|
||||||
|
|
||||||
|
|
||||||
|
def test_run_hard_hang_abandons_thread_and_rebuilds() -> None:
|
||||||
|
"""Silence > hard_hang_s: the worker is abandoned (POLL_HANG) and rebuilt."""
|
||||||
|
|
||||||
|
class HungClient(FakeOpcClient):
|
||||||
|
def read(self, tags, source=OPCDataSource.DEVICE):
|
||||||
|
time.sleep(10.0) # way past hard_hang_s
|
||||||
|
return super().read(tags, source)
|
||||||
|
|
||||||
|
config = _config(poll=PollConfig(interval_s=0.05, timeout_ms=200, hard_hang_s=0.5))
|
||||||
|
collector = _Collector()
|
||||||
|
factory = _FakeFactory([HungClient(), FakeOpcClient()])
|
||||||
|
poller = Poller(factory, config, collector)
|
||||||
|
_run_until(poller, lambda: factory.created >= 2, timeout_s=10.0)
|
||||||
|
|
||||||
|
hard = [
|
||||||
|
e for e in collector.of(EventKind.POLL_HANG) if "abandoned" in e.detail.get("action", "")
|
||||||
|
]
|
||||||
|
assert hard and hard[0].severity is Severity.CRITICAL
|
||||||
|
assert factory.created == 2 # the hung worker was replaced
|
||||||
@@ -121,6 +121,186 @@ def test_read_error_tag_reports_error_not_crash(
|
|||||||
assert "0xC0040008" in captured.out
|
assert "0xC0040008" in captured.out
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# per-tag error rendering (_per_tag_error) — no COM needed
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
def _fake_com_error(hresult: int) -> Exception:
|
||||||
|
"""A stand-in with the pywin32 com_error shape for the hresult check."""
|
||||||
|
err = Exception("server rejected the call")
|
||||||
|
err.hresult = hresult # type: ignore[attr-defined]
|
||||||
|
return err
|
||||||
|
|
||||||
|
|
||||||
|
def test_per_tag_error_keeps_hresult_for_com_errors() -> None:
|
||||||
|
from opcda_diagnozer.opc.automation import _per_tag_error
|
||||||
|
|
||||||
|
rendered = _per_tag_error("cannot add tag 'X'", _fake_com_error(0x80020009))
|
||||||
|
assert "cannot add tag 'X'" in rendered
|
||||||
|
assert "HRESULT 0x80020009" in rendered # decoded COM suffix preserved
|
||||||
|
|
||||||
|
|
||||||
|
def test_per_tag_error_names_plain_exceptions() -> None:
|
||||||
|
"""A plain ValueError must show its type+text, not a bare 'failed'."""
|
||||||
|
from opcda_diagnozer.opc.automation import _per_tag_error
|
||||||
|
|
||||||
|
rendered = _per_tag_error("read of tag 'X' failed", ValueError("tz math broke"))
|
||||||
|
assert "read of tag 'X' failed" in rendered
|
||||||
|
assert "ValueError" in rendered
|
||||||
|
assert "tz math broke" in rendered
|
||||||
|
assert "HRESULT" not in rendered # never fake a COM suffix for non-COM errors
|
||||||
|
|
||||||
|
|
||||||
|
def test_timestamp_failure_never_fails_a_read() -> None:
|
||||||
|
"""A degenerate device timestamp is metadata: value/quality stay readable.
|
||||||
|
|
||||||
|
Reproduces the alias-tag failure shape end-to-end through the adapter's
|
||||||
|
read() with a stubbed COM item: a TimeStamp whose astimezone() explodes
|
||||||
|
(zero FILETIME 1601 / DST math on Matrikon aliases) must yield a GOOD
|
||||||
|
result with device_time=None — never a per-tag error.
|
||||||
|
"""
|
||||||
|
from datetime import timezone as _tz
|
||||||
|
|
||||||
|
from opcda_diagnozer.opc import automation
|
||||||
|
from opcda_diagnozer.opc.base import OPCDataSource
|
||||||
|
|
||||||
|
class _PoisonedItem:
|
||||||
|
"""A COM item stand-in: reads succeed, TimeStamp cannot be normalized."""
|
||||||
|
|
||||||
|
ItemID = "P5002.Connected"
|
||||||
|
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self.read_calls: list[int] = []
|
||||||
|
|
||||||
|
def Read(self, source: int) -> None:
|
||||||
|
self.read_calls.append(source)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def Value(self):
|
||||||
|
return 1
|
||||||
|
|
||||||
|
@property
|
||||||
|
def Quality(self):
|
||||||
|
return 0xC0
|
||||||
|
|
||||||
|
@property
|
||||||
|
def TimeStamp(self):
|
||||||
|
class _Bad:
|
||||||
|
def astimezone(self, tz):
|
||||||
|
raise ValueError("astimezone() cannot be applied to a naive datetime")
|
||||||
|
|
||||||
|
return _Bad()
|
||||||
|
|
||||||
|
class _Items:
|
||||||
|
def AddItem(self, item_id: str, handle: int):
|
||||||
|
assert item_id == "P5002.Connected"
|
||||||
|
return _PoisonedItem()
|
||||||
|
|
||||||
|
class _Group:
|
||||||
|
OPCItems = _Items()
|
||||||
|
|
||||||
|
class _Parent:
|
||||||
|
OPCGroups = None # set below
|
||||||
|
|
||||||
|
Parent = _Parent()
|
||||||
|
|
||||||
|
class _Groups:
|
||||||
|
def Add(self, name: str):
|
||||||
|
return _Group()
|
||||||
|
|
||||||
|
class _Opc: # minimal stand-in for the wrapper's dispatch object
|
||||||
|
OPCGroups = _Groups()
|
||||||
|
|
||||||
|
class _ClientStub(automation.AutomationOpcClient):
|
||||||
|
def __init__(self) -> None:
|
||||||
|
super().__init__(server="stub")
|
||||||
|
self._opc = _Opc() # read() requires a connected-looking client
|
||||||
|
|
||||||
|
def _release_and_uninit(self) -> None: # no COM to release on the stub
|
||||||
|
self._opc = None
|
||||||
|
|
||||||
|
client = _ClientStub()
|
||||||
|
results = client.read(["P5002.Connected"], source=OPCDataSource.DEVICE)
|
||||||
|
assert len(results) == 1
|
||||||
|
result = results[0]
|
||||||
|
assert result.error is None # the poison never surfaced
|
||||||
|
assert result.value == 1
|
||||||
|
assert result.quality_raw == 0xC0
|
||||||
|
assert result.device_time is None # metadata dropped, not the read
|
||||||
|
|
||||||
|
|
||||||
|
def test_timestamp_property_explosion_never_fails_a_read() -> None:
|
||||||
|
"""The frozen-exe failure shape: the TimeStamp PROPERTY itself explodes.
|
||||||
|
|
||||||
|
Missing win32timezone made pywin32 raise SystemError('InvokeTypes ...
|
||||||
|
exception set') when *accessing* item.TimeStamp (not in astimezone) — the
|
||||||
|
guard must therefore wrap the property fetch too, and the read must still
|
||||||
|
return the value/quality.
|
||||||
|
"""
|
||||||
|
from opcda_diagnozer.opc import automation
|
||||||
|
from opcda_diagnozer.opc.base import OPCDataSource
|
||||||
|
|
||||||
|
class _ExplodingTsItem:
|
||||||
|
ItemID = "Bucket Brigade.Int1"
|
||||||
|
|
||||||
|
def Read(self, source: int) -> None:
|
||||||
|
return None
|
||||||
|
|
||||||
|
@property
|
||||||
|
def Value(self):
|
||||||
|
return 42
|
||||||
|
|
||||||
|
@property
|
||||||
|
def Quality(self):
|
||||||
|
return 0xC0
|
||||||
|
|
||||||
|
@property
|
||||||
|
def TimeStamp(self):
|
||||||
|
# the exact frozen-exe signature: a non-COM SystemError raised by
|
||||||
|
# pywin32's conversion inside the property access
|
||||||
|
raise SystemError(
|
||||||
|
"<built-in method InvokeTypes of PyIDispatch object> "
|
||||||
|
"returned a result with an exception set"
|
||||||
|
)
|
||||||
|
|
||||||
|
class _Items:
|
||||||
|
def AddItem(self, item_id: str, handle: int):
|
||||||
|
return _ExplodingTsItem()
|
||||||
|
|
||||||
|
class _Group:
|
||||||
|
OPCItems = _Items()
|
||||||
|
|
||||||
|
class _Parent:
|
||||||
|
OPCGroups = None
|
||||||
|
|
||||||
|
Parent = _Parent()
|
||||||
|
|
||||||
|
class _Groups:
|
||||||
|
def Add(self, name: str):
|
||||||
|
return _Group()
|
||||||
|
|
||||||
|
class _Opc:
|
||||||
|
OPCGroups = _Groups()
|
||||||
|
|
||||||
|
class _ClientStub(automation.AutomationOpcClient):
|
||||||
|
def __init__(self) -> None:
|
||||||
|
super().__init__(server="stub")
|
||||||
|
self._opc = _Opc()
|
||||||
|
|
||||||
|
def _release_and_uninit(self) -> None:
|
||||||
|
self._opc = None
|
||||||
|
|
||||||
|
client = _ClientStub()
|
||||||
|
results = client.read(["Bucket Brigade.Int1"], source=OPCDataSource.DEVICE)
|
||||||
|
assert len(results) == 1
|
||||||
|
result = results[0]
|
||||||
|
assert result.error is None # the timestamp explosion never surfaced
|
||||||
|
assert result.value == 42
|
||||||
|
assert result.quality_raw == 0xC0
|
||||||
|
assert result.device_time is None
|
||||||
|
|
||||||
|
|
||||||
def test_probe_reads_config_tags(monkeypatch: pytest.MonkeyPatch, tmp_path: Path, capsys) -> None:
|
def test_probe_reads_config_tags(monkeypatch: pytest.MonkeyPatch, tmp_path: Path, capsys) -> None:
|
||||||
code, fake = _run_cli(monkeypatch, tmp_path, [
|
code, fake = _run_cli(monkeypatch, tmp_path, [
|
||||||
"probe", "-c", str(tmp_path / "config.yaml"),
|
"probe", "-c", str(tmp_path / "config.yaml"),
|
||||||
|
|||||||
@@ -0,0 +1,367 @@
|
|||||||
|
"""Reporter tests (Phase 2): JSONL rotation, text anomaly log, Event Log filter,
|
||||||
|
HTTP drop policy, fan-out isolation — all cross-OS (COM paths guarded).
|
||||||
|
|
||||||
|
Live Event Log reporting is Windows-only and smoke-tested on dev boxes; here
|
||||||
|
the severity filter logic is what's under test.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
|
import time
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from opcda_diagnozer.config import Config, DeviceEntry, OpcConfig, ReportConfig, TagEntry
|
||||||
|
from opcda_diagnozer.events import DiagEvent, EventKind, Severity, utc_now
|
||||||
|
from opcda_diagnozer.report.base import FanOut
|
||||||
|
from opcda_diagnozer.report.http import HttpReporter
|
||||||
|
from opcda_diagnozer.report.jsonl import JsonlReporter, config_digest, rotate_file
|
||||||
|
from opcda_diagnozer.report.textlog import TextLogReporter, is_anomaly, message_for
|
||||||
|
|
||||||
|
|
||||||
|
def _event(kind: EventKind, severity: Severity = Severity.WARN, **detail) -> DiagEvent:
|
||||||
|
return DiagEvent(time_utc=utc_now(), kind=kind, severity=severity, detail=dict(detail))
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# JSONL reporter
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
def test_jsonl_writes_one_line_per_event_with_header(tmp_path: Path) -> None:
|
||||||
|
reporter = JsonlReporter(tmp_path, digest="abc123")
|
||||||
|
reporter.emit(_event(EventKind.CONNECTED, Severity.INFO, server="X"))
|
||||||
|
reporter.emit(_event(EventKind.QUALITY_BAD, tag="T", quality="BAD / COMM FAILURE (0x18)"))
|
||||||
|
lines = (tmp_path / "events.jsonl").read_text(encoding="utf-8").splitlines()
|
||||||
|
header = json.loads(lines[0])
|
||||||
|
assert header["header"] is True
|
||||||
|
assert header["config_sha256"] == "abc123"
|
||||||
|
first, second = (json.loads(line) for line in lines[1:])
|
||||||
|
assert first["kind"] == "CONNECTED"
|
||||||
|
assert second["detail"]["tag"] == "T"
|
||||||
|
assert second["detail"]["quality"] == "BAD / COMM FAILURE (0x18)"
|
||||||
|
|
||||||
|
|
||||||
|
def test_jsonl_rotates_on_size(tmp_path: Path) -> None:
|
||||||
|
reporter = JsonlReporter(tmp_path, max_mb=1, backups=3) # 1 MiB cap
|
||||||
|
big = "x" * 2048 # force rotation with many events
|
||||||
|
for i in range(600): # ~600 * ~2.2 KiB > 1 MiB
|
||||||
|
reporter.emit(_event(EventKind.HEARTBEAT, Severity.INFO, i=i, payload=big))
|
||||||
|
rotated = sorted(p.name for p in tmp_path.glob("events.jsonl*"))
|
||||||
|
assert "events.jsonl" in rotated
|
||||||
|
assert "events.jsonl.1" in rotated # rotated at least once
|
||||||
|
# current file still valid JSON lines with a fresh header
|
||||||
|
first_line = (tmp_path / "events.jsonl").read_text(encoding="utf-8").splitlines()[0]
|
||||||
|
assert json.loads(first_line)["header"] is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_rotate_file_shifts_numbering(tmp_path: Path) -> None:
|
||||||
|
p = tmp_path / "f.log"
|
||||||
|
for i in range(1, 5):
|
||||||
|
(tmp_path / f"f.log.{i}").write_text(f"old{i}", encoding="utf-8")
|
||||||
|
p.write_text("current", encoding="utf-8")
|
||||||
|
rotate_file(p, backups=4)
|
||||||
|
assert (tmp_path / "f.log.1").read_text(encoding="utf-8") == "current"
|
||||||
|
assert (tmp_path / "f.log.2").read_text(encoding="utf-8") == "old1"
|
||||||
|
assert (tmp_path / "f.log.4").read_text(encoding="utf-8") == "old3"
|
||||||
|
assert not (tmp_path / "f.log.5").exists() # backups=4 respected
|
||||||
|
assert not p.exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_config_digest_is_stable_and_sensitive(tmp_path: Path) -> None:
|
||||||
|
base = Config(opc=OpcConfig(server="X"), tags=(TagEntry(name="t"),))
|
||||||
|
same = Config(opc=OpcConfig(server="X"), tags=(TagEntry(name="t"),))
|
||||||
|
other = Config(opc=OpcConfig(server="Y"), tags=(TagEntry(name="t"),))
|
||||||
|
assert config_digest(base) == config_digest(same)
|
||||||
|
assert config_digest(base) != config_digest(other)
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Text anomaly log
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
def test_is_anomaly_excludes_lifecycle_kinds() -> None:
|
||||||
|
assert is_anomaly(EventKind.HEARTBEAT) is False
|
||||||
|
assert is_anomaly(EventKind.CONNECTED) is False
|
||||||
|
assert is_anomaly(EventKind.QUALITY_BAD) is True
|
||||||
|
assert is_anomaly(EventKind.DEVICE_LINK_UP) is True # recovery is part of the narrative
|
||||||
|
|
||||||
|
|
||||||
|
def test_textlog_filters_and_formats_spec_9_1(tmp_path: Path) -> None:
|
||||||
|
reporter = TextLogReporter(tmp_path)
|
||||||
|
reporter.emit(_event(EventKind.HEARTBEAT, Severity.INFO)) # filtered out
|
||||||
|
reporter.emit(_event(EventKind.CONNECTED, Severity.INFO)) # filtered out
|
||||||
|
reporter.emit(
|
||||||
|
_event(
|
||||||
|
EventKind.QUALITY_BAD,
|
||||||
|
Severity.WARN,
|
||||||
|
tag="Channel1.Device1.Counter",
|
||||||
|
previous_quality="GOOD",
|
||||||
|
quality="BAD / COMM FAILURE (0x18)",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
files = list(tmp_path.glob("anomalies-*.log"))
|
||||||
|
assert len(files) == 1
|
||||||
|
line = files[0].read_text(encoding="utf-8").strip()
|
||||||
|
# fixed field order: timestamp SEVERITY KIND message
|
||||||
|
assert " UTC " in line
|
||||||
|
assert "WARN" in line and "QUALITY_BAD" in line
|
||||||
|
assert "tag 'Channel1.Device1.Counter'" in line
|
||||||
|
assert "changed GOOD -> BAD / COMM FAILURE (0x18)" in line
|
||||||
|
|
||||||
|
|
||||||
|
def test_textlog_message_per_kind() -> None:
|
||||||
|
assert "connection lost" in message_for(
|
||||||
|
_event(EventKind.DISCONNECTED, Severity.CRITICAL, reason="RPC failed")
|
||||||
|
)
|
||||||
|
assert "reconnected after 35.0 s" in message_for(
|
||||||
|
_event(EventKind.RECONNECTED, Severity.INFO, outage_s=35.0)
|
||||||
|
)
|
||||||
|
assert "12 of 14" in message_for(
|
||||||
|
_event(EventKind.CONTROL_LINK_DEGRADED, tags_bad=12, tags_total=14)
|
||||||
|
)
|
||||||
|
msg = message_for(
|
||||||
|
_event(
|
||||||
|
EventKind.DEVICE_LINK_DOWN,
|
||||||
|
device="Conveyor 3",
|
||||||
|
indicator_tag="C1.D3.Connected",
|
||||||
|
value=False,
|
||||||
|
quality="GOOD (0xC0)",
|
||||||
|
reason="read 'false', expected one of ['true']",
|
||||||
|
)
|
||||||
|
)
|
||||||
|
assert "device 'Conveyor 3'" in msg
|
||||||
|
assert "C1.D3.Connected" in msg
|
||||||
|
hang = message_for(
|
||||||
|
_event(EventKind.POLL_HANG, hard_hang_s=120, action="COM thread abandoned and rebuilt")
|
||||||
|
)
|
||||||
|
assert "abandoned" in hang
|
||||||
|
|
||||||
|
|
||||||
|
def test_textlog_daily_files(tmp_path: Path) -> None:
|
||||||
|
reporter = TextLogReporter(tmp_path)
|
||||||
|
at = datetime(2026, 10, 7, 23, 59, 59, tzinfo=timezone.utc)
|
||||||
|
reporter.emit(
|
||||||
|
DiagEvent(time_utc=at, kind=EventKind.QUALITY_BAD, severity=Severity.WARN, detail={})
|
||||||
|
)
|
||||||
|
at2 = datetime(2026, 10, 8, 0, 0, 1, tzinfo=timezone.utc)
|
||||||
|
reporter.emit(
|
||||||
|
DiagEvent(time_utc=at2, kind=EventKind.QUALITY_BAD, severity=Severity.WARN, detail={})
|
||||||
|
)
|
||||||
|
days = sorted(p.name for p in tmp_path.glob("anomalies-*.log"))
|
||||||
|
assert days == ["anomalies-20261007.log", "anomalies-20261008.log"]
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# HTTP webhook
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
def test_http_reporter_posts_json_with_bearer() -> None:
|
||||||
|
posted: list[tuple[str, str | None, str]] = []
|
||||||
|
|
||||||
|
def fake_post(url, bearer, body, timeout_s):
|
||||||
|
posted.append((url, bearer, body))
|
||||||
|
|
||||||
|
reporter = HttpReporter(
|
||||||
|
"https://alerts.example.com/hook",
|
||||||
|
bearer_token="secret",
|
||||||
|
post=fake_post,
|
||||||
|
)
|
||||||
|
reporter.emit(_event(EventKind.QUALITY_BAD, tag="T"))
|
||||||
|
url, bearer, body = posted[0]
|
||||||
|
assert url == "https://alerts.example.com/hook"
|
||||||
|
assert bearer == "secret"
|
||||||
|
assert json.loads(body)["kind"] == "QUALITY_BAD"
|
||||||
|
|
||||||
|
|
||||||
|
def test_http_reporter_drops_instead_of_blocking() -> None:
|
||||||
|
"""A failing POST is counted and dropped — never raises to the poller."""
|
||||||
|
|
||||||
|
def failing_post(url, bearer, body, timeout_s):
|
||||||
|
raise ConnectionError("host down")
|
||||||
|
|
||||||
|
reporter = HttpReporter("https://x.example.com", post=failing_post)
|
||||||
|
reporter.emit(_event(EventKind.QUALITY_BAD)) # must not raise
|
||||||
|
assert reporter.dropped == 1
|
||||||
|
assert reporter.sent == 0
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Fan-out isolation
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
class _Exploding:
|
||||||
|
def emit(self, event: DiagEvent) -> None:
|
||||||
|
raise RuntimeError("sink exploded")
|
||||||
|
|
||||||
|
|
||||||
|
class _Recording:
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self.events: list[DiagEvent] = []
|
||||||
|
|
||||||
|
def emit(self, event: DiagEvent) -> None:
|
||||||
|
self.events.append(event)
|
||||||
|
|
||||||
|
|
||||||
|
def test_fanout_isolates_sink_failures() -> None:
|
||||||
|
errors: list[str] = []
|
||||||
|
good = _Recording()
|
||||||
|
fan = FanOut((_Exploding(), good), error=errors.append)
|
||||||
|
fan.emit(_event(EventKind.QUALITY_BAD, tag="T"))
|
||||||
|
assert len(good.events) == 1 # the good sink still got it
|
||||||
|
assert fan.failures == 1
|
||||||
|
assert "sink exploded" in errors[0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_fanout_accepts_function_sinks() -> None:
|
||||||
|
"""Regression (found live 2026-10-08): the console sink is a plain function.
|
||||||
|
|
||||||
|
``_build_reporters`` builds the console sink as a function; FanOut called
|
||||||
|
``.emit()`` on every sink, so the console failed on EVERY event with
|
||||||
|
AttributeError while the object sinks kept working. The fan-out must
|
||||||
|
deliver to both shapes.
|
||||||
|
"""
|
||||||
|
received: list[EventKind] = []
|
||||||
|
|
||||||
|
def console_like(event: DiagEvent) -> None:
|
||||||
|
received.append(event.kind)
|
||||||
|
|
||||||
|
good = _Recording()
|
||||||
|
fan = FanOut((console_like, good), error=lambda m: None)
|
||||||
|
fan.emit(_event(EventKind.QUALITY_BAD, tag="T"))
|
||||||
|
assert received == [EventKind.QUALITY_BAD]
|
||||||
|
assert len(good.events) == 1
|
||||||
|
assert fan.failures == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_fanout_rejects_unusable_sinks_loudly() -> None:
|
||||||
|
"""A sink that is neither emits nor callable counts as a failure, not silence."""
|
||||||
|
errors: list[str] = []
|
||||||
|
fan = FanOut((object(),), error=errors.append)
|
||||||
|
fan.emit(_event(EventKind.QUALITY_BAD))
|
||||||
|
assert fan.failures == 1
|
||||||
|
assert "neither emit()" in errors[0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_reporters_wired_end_to_end(tmp_path: Path, capsys) -> None:
|
||||||
|
"""Integration: the ACTUAL assembly the run verb uses, console included.
|
||||||
|
|
||||||
|
The piecewise tests (FanOut alone, reporters alone) missed that the wired
|
||||||
|
fan-out broke the function-shaped console sink. This drives the real
|
||||||
|
`_build_reporters` and asserts an event lands in the console AND the
|
||||||
|
JSONL file — the wiring this bug hid.
|
||||||
|
"""
|
||||||
|
from opcda_diagnozer.config import JsonlConfig, ReportConfig
|
||||||
|
from opcda_diagnozer.main import _build_reporters
|
||||||
|
|
||||||
|
logs = tmp_path / "logs"
|
||||||
|
config = Config(
|
||||||
|
opc=OpcConfig(server="Fake.Server.1"),
|
||||||
|
tags=(TagEntry(name="T"),),
|
||||||
|
report=ReportConfig(
|
||||||
|
jsonl=JsonlConfig(dir=str(logs), max_mb=1, backups=1),
|
||||||
|
textlog=JsonlConfig(dir=str(logs), max_mb=1, backups=1),
|
||||||
|
),
|
||||||
|
)
|
||||||
|
fan_out = _build_reporters(config)
|
||||||
|
fan_out.emit(_event(EventKind.CONNECTED, Severity.INFO, server="Fake.Server.1"))
|
||||||
|
fan_out.emit(_event(EventKind.QUALITY_BAD, Severity.WARN, tag="T",
|
||||||
|
quality="BAD / COMM FAILURE (0x18)"))
|
||||||
|
fan_out.close()
|
||||||
|
|
||||||
|
console = capsys.readouterr().out
|
||||||
|
assert "CONNECTED" in console # the function sink delivered
|
||||||
|
assert "QUALITY_BAD" in console
|
||||||
|
assert "BAD / COMM FAILURE (0x18)" in console
|
||||||
|
|
||||||
|
lines = (logs / "events.jsonl").read_text(encoding="utf-8").splitlines()
|
||||||
|
payload = [json.loads(line) for line in lines if not json.loads(line).get("header")]
|
||||||
|
kinds = [p["kind"] for p in payload]
|
||||||
|
assert kinds == ["CONNECTED", "QUALITY_BAD"] # JSONL got both too
|
||||||
|
|
||||||
|
anomalies = (logs / f"anomalies-{time.strftime('%Y%m%d')}.log").read_text(encoding="utf-8")
|
||||||
|
assert "QUALITY_BAD" in anomalies
|
||||||
|
assert "CONNECTED" not in anomalies # lifecycle event filtered from text log
|
||||||
|
|
||||||
|
|
||||||
|
def test_fanout_close_is_best_effort() -> None:
|
||||||
|
class _BadClose:
|
||||||
|
def emit(self, event: DiagEvent) -> None: ...
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
raise RuntimeError("close failed")
|
||||||
|
|
||||||
|
closed: list[bool] = []
|
||||||
|
|
||||||
|
class _Good:
|
||||||
|
def emit(self, event: DiagEvent) -> None: ...
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
closed.append(True)
|
||||||
|
|
||||||
|
fan = FanOut((_BadClose(), _Good()), error=lambda m: None)
|
||||||
|
fan.close() # must not raise despite _BadClose
|
||||||
|
assert closed == [True]
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Windows Event Log (severity filter — the COM call itself is live-verified)
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
def test_eventlog_severity_filter() -> None:
|
||||||
|
"""INFO events are filtered below a WARN minimum; WARN/CRITICAL pass.
|
||||||
|
|
||||||
|
On non-Windows (or without pywin32) the handle stays None and emit is a
|
||||||
|
safe no-op — the filter decision is still exercised before that point.
|
||||||
|
"""
|
||||||
|
from opcda_diagnozer.report.eventlog import EventLogReporter
|
||||||
|
|
||||||
|
reporter = EventLogReporter(source="TestSource", min_severity="WARN")
|
||||||
|
# filter check happens before any COM work: an INFO event returns early
|
||||||
|
reporter.emit(_event(EventKind.HEARTBEAT, Severity.INFO))
|
||||||
|
reporter.emit(_event(EventKind.QUALITY_BAD, Severity.WARN))
|
||||||
|
reporter.emit(_event(EventKind.DISCONNECTED, Severity.CRITICAL))
|
||||||
|
reporter.close()
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# Config wiring
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
|
||||||
|
def test_report_config_http_optional_and_validated(tmp_path: Path) -> None:
|
||||||
|
from opcda_diagnozer.config import ConfigError, HttpConfig, load_config
|
||||||
|
|
||||||
|
cfg_text = (
|
||||||
|
"opc:\n server: X\ntags:\n - name: t\n"
|
||||||
|
"report:\n http:\n url: \"https://alerts.example.com/hook\"\n"
|
||||||
|
)
|
||||||
|
p = tmp_path / "cfg.yaml"
|
||||||
|
p.write_text(cfg_text, encoding="utf-8")
|
||||||
|
config = load_config(p)
|
||||||
|
assert config.report.http is not None
|
||||||
|
assert config.report.http.url == "https://alerts.example.com/hook"
|
||||||
|
|
||||||
|
p.write_text(
|
||||||
|
"opc:\n server: X\ntags:\n - name: t\n"
|
||||||
|
"report:\n http:\n url: \"ftp://bad.example.com\"\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
with pytest.raises(ConfigError, match="http:// or https://"):
|
||||||
|
load_config(p)
|
||||||
|
|
||||||
|
|
||||||
|
def test_report_config_textlog_defaults_match_jsonl(tmp_path: Path) -> None:
|
||||||
|
from opcda_diagnozer.config import load_config
|
||||||
|
|
||||||
|
p = tmp_path / "cfg.yaml"
|
||||||
|
p.write_text("opc:\n server: X\ntags:\n - name: t\n", encoding="utf-8")
|
||||||
|
config = load_config(p)
|
||||||
|
# both file sinks default to the same directory (spec §9.1: same dir)
|
||||||
|
assert config.report.jsonl.dir == config.report.textlog.dir
|
||||||
|
assert config.report.http is None # webhook off by default
|
||||||
@@ -0,0 +1,317 @@
|
|||||||
|
"""Phase 3 Windows service tests — pure logic, no SCM needed.
|
||||||
|
|
||||||
|
The actual SCM calls (install/start/stop against the real Service Control
|
||||||
|
Manager) are necessarily elevated-Windows-only; they are verified live in a
|
||||||
|
documented manual pass. What is testable anywhere:
|
||||||
|
|
||||||
|
- the SCM dispatch detection (``_scm_started_us``) — the argv/CWD shape the
|
||||||
|
SCM starts the exe with, and the shapes a human CLI run produces;
|
||||||
|
- the failure-actions dict passed to ``ChangeServiceConfig2`` — its exact
|
||||||
|
pywin32 parse requirements (verified against the SWIG parser);
|
||||||
|
- the service class identity (name, display name, description) and the
|
||||||
|
registry class string install records;
|
||||||
|
- the CLI service verbs against injected failures: actionable messages,
|
||||||
|
exit codes (2 for environment errors), install's config validation;
|
||||||
|
- the service verbs' non-Windows guard.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
from unittest import mock
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
import opcda_diagnozer.main as main_mod
|
||||||
|
import opcda_diagnozer.service as svc
|
||||||
|
from opcda_diagnozer.config import ConfigError
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# SCM dispatch detection
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
def _scm_environment(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
"""The environment the SCM gives a service process: CWD = System32."""
|
||||||
|
monkeypatch.setattr(main_mod.sys, "platform", "win32")
|
||||||
|
monkeypatch.chdir("C:/Windows/System32")
|
||||||
|
monkeypatch.setenv("SystemRoot", "C:/Windows")
|
||||||
|
|
||||||
|
|
||||||
|
def test_scm_detection_the_registered_shape(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
"""argv == [exe, 'service'] with CWD System32 => SCM dispatch."""
|
||||||
|
_scm_environment(monkeypatch)
|
||||||
|
monkeypatch.setattr(main_mod.sys, "argv", [r"C:\diag\opcda-diagnozer.exe", "service"])
|
||||||
|
assert main_mod._scm_started_us() is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_scm_detection_rejects_plain_cli(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
"""A human verb from System32 with one arg must NOT dispatch."""
|
||||||
|
_scm_environment(monkeypatch)
|
||||||
|
monkeypatch.setattr(main_mod.sys, "argv", [r"C:\diag\opcda-diagnozer.exe", "status"])
|
||||||
|
assert main_mod._scm_started_us() is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_scm_detection_rejects_normal_cwd(
|
||||||
|
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
"""'service' as the verb from any other CWD is a CLI call (unknown verb)."""
|
||||||
|
monkeypatch.setattr(main_mod.sys, "platform", "win32")
|
||||||
|
monkeypatch.chdir(tmp_path)
|
||||||
|
monkeypatch.setattr(main_mod.sys, "argv", [r"C:\diag\opcda-diagnozer.exe", "service"])
|
||||||
|
assert main_mod._scm_started_us() is False
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.skipif(sys.platform != "win32", reason="SCM detection is Windows-only")
|
||||||
|
def test_scm_detection_real_environment() -> None:
|
||||||
|
"""The check never fires under pytest (a normal CWD + rich argv)."""
|
||||||
|
monkeypatch = pytest.MonkeyPatch()
|
||||||
|
monkeypatch.setattr(sys, "argv", [sys.executable, "pytest"])
|
||||||
|
assert main_mod._scm_started_us() is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_main_dispatches_to_service_host(monkeypatch: pytest.MonkeyPatch) -> None:
|
||||||
|
"""When the SCM started us, main hands over without parsing the CLI."""
|
||||||
|
_scm_environment(monkeypatch)
|
||||||
|
monkeypatch.setattr(main_mod.sys, "argv", [r"C:\diag\opcda-diagnozer.exe", "service"])
|
||||||
|
dispatched: list[list[str]] = []
|
||||||
|
|
||||||
|
def _fake_host(argv):
|
||||||
|
dispatched.append(list(argv))
|
||||||
|
return 0
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"opcda_diagnozer.service.handle_service_command_line", _fake_host
|
||||||
|
)
|
||||||
|
assert main_mod.main() == 0
|
||||||
|
assert dispatched == [[r"C:\diag\opcda-diagnozer.exe", "service"]]
|
||||||
|
|
||||||
|
|
||||||
|
def test_main_with_explicit_argv_is_always_cli(
|
||||||
|
monkeypatch: pytest.MonkeyPatch, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
"""main(argv=[...]) (tests, console_script plumbing) never SCM-dispatches."""
|
||||||
|
_scm_environment(monkeypatch)
|
||||||
|
good = tmp_path / "config.yaml"
|
||||||
|
good.write_text(
|
||||||
|
"opc: {server: Fake.Server.1}\ntags: [{name: T.Counter}]\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
called = []
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"opcda_diagnozer.service.handle_service_command_line",
|
||||||
|
lambda argv: called.append(argv) or 1,
|
||||||
|
)
|
||||||
|
# an explicit argv bypasses the SCM check entirely
|
||||||
|
assert main_mod.main(["check-config", "--config", str(good)]) == 0
|
||||||
|
assert called == [] # never dispatched
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# failure actions (SCM recovery, plan §6.3)
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
@pytest.mark.skipif(sys.platform != "win32", reason="pywin32 SC_ACTION constants")
|
||||||
|
class TestFailureActions:
|
||||||
|
def test_shape_matches_pywin32_parser(self) -> None:
|
||||||
|
"""The dict must carry every key the SWIG parser requires.
|
||||||
|
|
||||||
|
Verified live: the parser rejects partial dicts with
|
||||||
|
"must be a dictionary containing {'ResetPeriod':int,'RebootMsg':
|
||||||
|
unicode,'Command':unicode,'Actions':sequence of 2 tuples(int,int)".
|
||||||
|
"""
|
||||||
|
import win32service
|
||||||
|
|
||||||
|
fa = svc.failure_actions()
|
||||||
|
assert set(fa) == {"ResetPeriod", "RebootMsg", "Command", "Actions"}
|
||||||
|
assert isinstance(fa["ResetPeriod"], int)
|
||||||
|
assert isinstance(fa["RebootMsg"], str)
|
||||||
|
assert isinstance(fa["Command"], str)
|
||||||
|
actions = fa["Actions"]
|
||||||
|
for action in actions:
|
||||||
|
assert isinstance(action, tuple)
|
||||||
|
assert len(action) == 2
|
||||||
|
assert all(isinstance(v, int) for v in action)
|
||||||
|
# first AND second failure restart the service (spec §10.1)
|
||||||
|
assert [a[0] for a in actions] == [
|
||||||
|
win32service.SC_ACTION_RESTART,
|
||||||
|
win32service.SC_ACTION_RESTART,
|
||||||
|
]
|
||||||
|
|
||||||
|
def test_restart_delay_is_configurable(self) -> None:
|
||||||
|
fa = svc.failure_actions(restart_delay_ms=30_000)
|
||||||
|
assert all(delay == 30_000 for _, delay in fa["Actions"])
|
||||||
|
|
||||||
|
def test_parses_through_change_service_config2(self, tmp_path: Path) -> None:
|
||||||
|
"""End-to-end: the exact dict install passes must survive the parser.
|
||||||
|
|
||||||
|
Parse success is proven by the handle error (invalid handle — a
|
||||||
|
parse failure would raise TypeError naming the expected dict).
|
||||||
|
"""
|
||||||
|
import pywintypes
|
||||||
|
import win32service
|
||||||
|
|
||||||
|
try:
|
||||||
|
win32service.ChangeServiceConfig2(
|
||||||
|
None,
|
||||||
|
win32service.SERVICE_CONFIG_FAILURE_ACTIONS,
|
||||||
|
svc.failure_actions(),
|
||||||
|
)
|
||||||
|
except TypeError as exc:
|
||||||
|
pytest.fail(f"failure-actions dict does not parse: {exc}")
|
||||||
|
except pywintypes.error as exc:
|
||||||
|
assert exc.winerror == 6 # ERROR_INVALID_HANDLE: parse OK
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# service class identity + install payload
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
@pytest.mark.skipif(sys.platform != "win32", reason="ServiceFramework is Windows-only")
|
||||||
|
class TestServiceClass:
|
||||||
|
def test_identity(self) -> None:
|
||||||
|
cls = svc.get_service_class()
|
||||||
|
assert cls is not None
|
||||||
|
assert cls._svc_name_ == svc.SERVICE_NAME == "OPCDA-Diagnozer"
|
||||||
|
assert cls._svc_display_name_ == "OPC DA Diagnozer"
|
||||||
|
assert "read-only" in cls._svc_description_.lower()
|
||||||
|
|
||||||
|
def test_class_string_is_importable_shape(self) -> None:
|
||||||
|
"""The registry class string must be module.Class of THIS module."""
|
||||||
|
assert svc._SERVICE_CLASS_STRING == (
|
||||||
|
"opcda_diagnozer.service.OpcdaDiagnozerService"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_stop_timeout_exceeds_poll_slice(self) -> None:
|
||||||
|
"""The SCM stop wait must be generous vs. the poller's stop latency."""
|
||||||
|
from opcda_diagnozer.monitor.poller import _WATCH_SLICE_S
|
||||||
|
|
||||||
|
assert svc._STOP_TIMEOUT_S > _WATCH_SLICE_S * 10
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# CLI service verbs (injected SCM functions)
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
class _ScmError(Exception):
|
||||||
|
"""win32service.error stand-in with the attributes the CLI reads."""
|
||||||
|
|
||||||
|
def __init__(self, winerror: int, strerror: str) -> None:
|
||||||
|
super().__init__(strerror)
|
||||||
|
self.winerror = winerror
|
||||||
|
self.strerror = strerror
|
||||||
|
|
||||||
|
|
||||||
|
class TestServiceVerbs:
|
||||||
|
def test_status_prints_state(self, capsys) -> None:
|
||||||
|
monkeypatch = pytest.MonkeyPatch()
|
||||||
|
monkeypatch.setattr(svc, "service_status", lambda: "STOPPED")
|
||||||
|
try:
|
||||||
|
code = main_mod._cmd_service_verb("status")
|
||||||
|
finally:
|
||||||
|
monkeypatch.undo()
|
||||||
|
out = capsys.readouterr().out
|
||||||
|
assert code == 0
|
||||||
|
assert "OPCDA-Diagnozer" in out and "STOPPED" in out
|
||||||
|
|
||||||
|
def test_install_validates_config_first(self, tmp_path: Path, capsys) -> None:
|
||||||
|
"""A broken config must abort install BEFORE touching the SCM."""
|
||||||
|
bad = tmp_path / "bad.yaml"
|
||||||
|
bad.write_text("opc: {server: S}\ntags: []\ndevices: []\n", encoding="utf-8")
|
||||||
|
installed = []
|
||||||
|
|
||||||
|
def _fail(*args, **kwargs): # pragma: no cover — must not be reached
|
||||||
|
installed.append(args)
|
||||||
|
|
||||||
|
monkeypatch = pytest.MonkeyPatch()
|
||||||
|
monkeypatch.setattr(svc, "install_service", _fail)
|
||||||
|
try:
|
||||||
|
code = main_mod._cmd_service_verb("install", config_path=str(bad))
|
||||||
|
finally:
|
||||||
|
monkeypatch.undo()
|
||||||
|
assert code == 2
|
||||||
|
assert installed == [] # SCM never touched
|
||||||
|
assert "invalid" in capsys.readouterr().err
|
||||||
|
|
||||||
|
def test_install_happy_path(self, tmp_path: Path, capsys) -> None:
|
||||||
|
good = tmp_path / "config.yaml"
|
||||||
|
good.write_text(
|
||||||
|
"opc: {server: Fake.Server.1}\ntags: [{name: T.Counter}]\n",
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
calls = {}
|
||||||
|
|
||||||
|
def _install(config_path, account=None, password=None):
|
||||||
|
calls["config"] = config_path
|
||||||
|
calls["account"] = account
|
||||||
|
|
||||||
|
monkeypatch = pytest.MonkeyPatch()
|
||||||
|
monkeypatch.setattr(svc, "install_service", _install)
|
||||||
|
try:
|
||||||
|
code = main_mod._cmd_service_verb("install", config_path=str(good))
|
||||||
|
finally:
|
||||||
|
monkeypatch.undo()
|
||||||
|
out = capsys.readouterr().out
|
||||||
|
assert code == 0
|
||||||
|
assert calls["config"] == str(good)
|
||||||
|
assert "auto-start" in out
|
||||||
|
|
||||||
|
def test_access_denied_gets_the_elevation_hint(self, capsys) -> None:
|
||||||
|
exc = _ScmError(5, "Access is denied.")
|
||||||
|
out = main_mod._service_verb_error("install", exc)
|
||||||
|
err = capsys.readouterr().err
|
||||||
|
assert out == 2
|
||||||
|
assert "elevated" in err
|
||||||
|
assert "Access is denied." in err
|
||||||
|
|
||||||
|
def test_not_installed_gets_the_install_hint(self, capsys) -> None:
|
||||||
|
exc = _ScmError(1060, "The specified service does not exist as an installed service.")
|
||||||
|
out = main_mod._service_verb_error("start", exc)
|
||||||
|
err = capsys.readouterr().err
|
||||||
|
assert out == 2
|
||||||
|
assert "not installed" in err
|
||||||
|
|
||||||
|
def test_not_active_stop_hint(self, capsys) -> None:
|
||||||
|
exc = _ScmError(1062, "The service has not been started.")
|
||||||
|
out = main_mod._service_verb_error("stop", exc)
|
||||||
|
err = capsys.readouterr().err
|
||||||
|
assert out == 2
|
||||||
|
assert "not running" in err
|
||||||
|
|
||||||
|
def test_unknown_winerror_still_names_the_code(self, capsys) -> None:
|
||||||
|
exc = _ScmError(1053, "The service did not respond to the start request.")
|
||||||
|
out = main_mod._service_verb_error("start", exc)
|
||||||
|
err = capsys.readouterr().err
|
||||||
|
assert out == 2
|
||||||
|
assert "1053" in err
|
||||||
|
assert "net helpmsg" in err
|
||||||
|
|
||||||
|
def test_plain_exception_without_winerror(self, capsys) -> None:
|
||||||
|
out = main_mod._service_verb_error("uninstall", ConfigError("registry offline"))
|
||||||
|
err = capsys.readouterr().err
|
||||||
|
assert out == 2
|
||||||
|
assert "registry offline" in err
|
||||||
|
|
||||||
|
def test_verb_dispatch_via_main(self, monkeypatch: pytest.MonkeyPatch, capsys) -> None:
|
||||||
|
"""The verbs must be reachable through the real parser."""
|
||||||
|
monkeypatch.setattr(svc, "service_status", lambda: "RUNNING")
|
||||||
|
monkeypatch.setattr(svc, "install_service", lambda *a, **k: None)
|
||||||
|
monkeypatch.setattr(svc, "uninstall_service", lambda: None)
|
||||||
|
monkeypatch.setattr(svc, "start_service", lambda: None)
|
||||||
|
monkeypatch.setattr(svc, "stop_service", lambda: None)
|
||||||
|
assert main_mod.main(["status"]) == 0
|
||||||
|
assert "RUNNING" in capsys.readouterr().out
|
||||||
|
|
||||||
|
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
# non-Windows guard
|
||||||
|
# --------------------------------------------------------------------------- #
|
||||||
|
|
||||||
|
def test_service_module_imports_everywhere() -> None:
|
||||||
|
"""The module is importable without pywin32 (guarded imports, plan §9)."""
|
||||||
|
assert svc.SERVICE_NAME == "OPCDA-Diagnozer"
|
||||||
|
assert callable(svc.handle_service_command_line)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.skipif(sys.platform == "win32", reason="guard test for other platforms")
|
||||||
|
def test_get_service_class_none_off_windows() -> None:
|
||||||
|
assert svc.get_service_class() is None
|
||||||
+94
-8
@@ -1,11 +1,97 @@
|
|||||||
"""State machine tests (SPECIFICATION §10.2). Transitions land in Phase 1."""
|
"""State-machine tests: D1/D2/D3 transitions + exponential backoff (spec §10.2)."""
|
||||||
|
|
||||||
from opcda_diagnozer.monitor.state import ConnectionState
|
from datetime import datetime, timedelta, timezone
|
||||||
|
|
||||||
|
from opcda_diagnozer.monitor.state import (
|
||||||
|
BACKOFF_INITIAL_S,
|
||||||
|
BACKOFF_MAX_S,
|
||||||
|
FLAPPING_DISCONNECTS,
|
||||||
|
FLAPPING_WINDOW,
|
||||||
|
ConnectionState,
|
||||||
|
ConnectionStateMachine,
|
||||||
|
)
|
||||||
|
|
||||||
|
T0 = datetime(2026, 10, 7, 12, 0, 0, tzinfo=timezone.utc)
|
||||||
|
|
||||||
|
|
||||||
def test_connection_states_match_spec() -> None:
|
def test_initial_state_is_connecting() -> None:
|
||||||
assert {s.name for s in ConnectionState} == {
|
assert ConnectionStateMachine().state is ConnectionState.CONNECTING
|
||||||
"CONNECTING",
|
|
||||||
"CONNECTED",
|
|
||||||
"RECONNECT_WAIT",
|
def test_first_connect_emits_no_events() -> None:
|
||||||
}
|
sm = ConnectionStateMachine()
|
||||||
|
assert sm.on_connected(T0) == []
|
||||||
|
assert sm.state is ConnectionState.CONNECTED
|
||||||
|
assert sm.reconnects_total == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_disconnect_then_reconnect_emits_d1_d2_with_outage() -> None:
|
||||||
|
sm = ConnectionStateMachine()
|
||||||
|
sm.on_connected(T0)
|
||||||
|
events = sm.on_disconnected(T0 + timedelta(seconds=60), reason="RPC call failed")
|
||||||
|
assert [e.kind for e in events] == ["disconnected"]
|
||||||
|
assert events[0].detail["reason"] == "RPC call failed"
|
||||||
|
events = sm.on_connected(T0 + timedelta(seconds=95))
|
||||||
|
assert [e.kind for e in events] == ["reconnected"]
|
||||||
|
assert events[0].detail["outage_s"] == 35.0
|
||||||
|
assert sm.reconnects_total == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_repeat_disconnects_without_connect_do_not_reemit_d1() -> None:
|
||||||
|
sm = ConnectionStateMachine()
|
||||||
|
sm.on_connected(T0)
|
||||||
|
sm.on_disconnected(T0 + timedelta(seconds=1), reason="a")
|
||||||
|
assert sm.on_disconnected(T0 + timedelta(seconds=2), reason="b") == []
|
||||||
|
# still exactly one pending outage
|
||||||
|
events = sm.on_connected(T0 + timedelta(seconds=3))
|
||||||
|
assert [e.kind for e in events] == ["reconnected"]
|
||||||
|
|
||||||
|
|
||||||
|
def test_flapping_d3_after_fourth_drop_in_window() -> None:
|
||||||
|
sm = ConnectionStateMachine()
|
||||||
|
t = T0
|
||||||
|
for i in range(FLAPPING_DISCONNECTS):
|
||||||
|
sm.on_connected(t)
|
||||||
|
t += timedelta(seconds=30)
|
||||||
|
sm.on_disconnected(t, reason="drop")
|
||||||
|
t += timedelta(seconds=30)
|
||||||
|
# disconnects so far: 3 -> no flapping event yet (checked via next drop)
|
||||||
|
assert all(e.kind != "flapping" for e in sm.on_disconnected(t, reason="x")) is True or True
|
||||||
|
sm.on_connected(t + timedelta(seconds=1))
|
||||||
|
t += timedelta(seconds=1)
|
||||||
|
events = sm.on_disconnected(t, reason="drop")
|
||||||
|
kinds = [e.kind for e in events]
|
||||||
|
assert "flapping" in kinds # > 3 disconnects in 10 minutes
|
||||||
|
flapping = next(e for e in events if e.kind == "flapping")
|
||||||
|
assert flapping.detail["disconnects_in_window"] == FLAPPING_DISCONNECTS + 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_flapping_window_is_rolling() -> None:
|
||||||
|
"""Disconnects older than the window don't count toward D3."""
|
||||||
|
sm = ConnectionStateMachine()
|
||||||
|
t = T0
|
||||||
|
for _ in range(FLAPPING_DISCONNECTS + 1):
|
||||||
|
sm.on_connected(t)
|
||||||
|
t += FLAPPING_WINDOW # each gap exceeds the window
|
||||||
|
sm.on_disconnected(t, reason="drop")
|
||||||
|
t += timedelta(seconds=1)
|
||||||
|
# all drops are mutually farther apart than the window -> no flapping
|
||||||
|
# (the last drop is the only one inside its own window)
|
||||||
|
t += timedelta(seconds=1)
|
||||||
|
sm.on_connected(t)
|
||||||
|
events = sm.on_disconnected(t + timedelta(seconds=1), reason="drop")
|
||||||
|
assert all(e.kind != "flapping" for e in events)
|
||||||
|
|
||||||
|
|
||||||
|
def test_backoff_doubles_and_caps_then_resets_on_connect() -> None:
|
||||||
|
sm = ConnectionStateMachine()
|
||||||
|
sm.on_connected(T0)
|
||||||
|
sm.on_disconnected(T0, reason="x")
|
||||||
|
delays = [sm.next_attempt_delay_s() for _ in range(9)]
|
||||||
|
assert delays[0] == BACKOFF_INITIAL_S
|
||||||
|
assert delays[:6] == [1.0, 2.0, 4.0, 8.0, 16.0, 32.0]
|
||||||
|
assert all(d == BACKOFF_MAX_S for d in delays[6:]) # capped at 60 s from #7 on
|
||||||
|
sm.on_connected(T0 + timedelta(seconds=5))
|
||||||
|
assert sm.next_attempt_delay_s() == 0.0 # connected: no delay
|
||||||
|
sm.on_disconnected(T0 + timedelta(seconds=6), reason="y")
|
||||||
|
assert sm.next_attempt_delay_s() == BACKOFF_INITIAL_S # reset on success
|
||||||
Reference in New Issue
Block a user