174 lines
8.8 KiB
Markdown
174 lines
8.8 KiB
Markdown
# git-auto-tracker
|
||
|
||
Unattended change tracker that records modifications to SCADA project and
|
||
configuration files in a Git repository.
|
||
|
||
The tracker periodically inspects a configured repository, detects
|
||
added / modified / deleted files, resolves the last modifying user for each
|
||
change (where a metadata source is available), and creates one Git commit per
|
||
collection cycle with a structured, auditable commit message.
|
||
|
||
See `functional specification.md` for the full specification.
|
||
|
||
## Features
|
||
|
||
- Run-once execution, designed for Windows Task Scheduler / cron / systemd timers
|
||
- Change detection via `git status --porcelain -z` (NUL-safe, UTF-8)
|
||
- Configurable include/exclude file filters (globs), with every excluded file and its matching rule logged at DEBUG level
|
||
- Security validation with forbidden patterns (abort by default)
|
||
- Pluggable author resolvers with priority chain and confidence levels
|
||
- Per-file attribution recorded in the commit message
|
||
- One-line-per-file commit entries in `git diff --name-status` style (letter, path, author, last-modified time), with a configurable separator (`commit.file_separator`, default `;`)
|
||
- Derived artifacts: SQLite content extraction into JSON Lines (one sub-folder per database, one file per table), byte-stable, read-only source access (Section 45)
|
||
- Structured commit trailers (`Tracker-Run`, `Tracker-Host`, ...)
|
||
- Repository-level lock with stale-lock detection (stored outside the repo)
|
||
- Rotating file logs plus console logging
|
||
- Dry-run mode, exit codes suitable for external monitoring
|
||
- `export` command: the collected history as analysis-ready CSV/JSONL (one row per file change, with author, timestamps and Tracker-Run identifiers)
|
||
- Cross-platform: Windows / Linux / macOS
|
||
|
||
## Installation
|
||
|
||
```bash
|
||
python -m venv .venv
|
||
.venv/Scripts/activate # Windows
|
||
source .venv/bin/activate # Linux/macOS
|
||
pip install -e .
|
||
```
|
||
|
||
## Usage
|
||
|
||
```bash
|
||
git-auto-tracker run [--dry-run] # collect changes and commit
|
||
git-auto-tracker status # show detected changes without committing
|
||
git-auto-tracker validate # validate configuration / repository / resolvers
|
||
git-auto-tracker config # print effective configuration
|
||
git-auto-tracker export -o out.csv # export the commit history for analysis
|
||
```
|
||
|
||
Global options:
|
||
|
||
```text
|
||
--config PATH Path to configuration file (default: git-auto-tracker.yaml)
|
||
--log-level LEVEL Override the configured log level
|
||
--version Print version and exit
|
||
```
|
||
|
||
### Analyzing the audit trail
|
||
|
||
`export` turns the tracker commits of the configured repository into a
|
||
flat table — one row per file change, with the collection time, run
|
||
identifier, host, change letter, path, author and last-modified time:
|
||
|
||
```bash
|
||
git-auto-tracker export -o history.csv # CSV (default format)
|
||
git-auto-tracker export -o history.jsonl # JSON Lines
|
||
git-auto-tracker export -o history.csv --since 2026-09-01
|
||
```
|
||
|
||
The rows load directly into pandas/Excel for analysis (who changes most,
|
||
which subtrees churn, change volume per day, everything ever deleted).
|
||
Commits written with a different `commit.file_separator` than the current
|
||
configuration are skipped with a warning, since their fields cannot be
|
||
split reliably. Non-tracker commits (e.g. the repository's initial
|
||
import) are ignored automatically.
|
||
|
||
## Configuration
|
||
|
||
See [`config/example.yaml`](config/example.yaml). Configuration is YAML;
|
||
deployment-specific values can be overridden through environment variables
|
||
using the `GIT_AUTO_TRACKER_<SECTION>__<KEY>` convention.
|
||
|
||
## Specification status
|
||
|
||
Progress against `functional specification.md` (FR-001 – FR-034). The v1
|
||
scope is defined by Section 44, *Definition of Done – Initial Release*;
|
||
FR-030 – FR-034 (Section 45, derived artifacts) are prioritized for the
|
||
next release.
|
||
|
||
### Implemented
|
||
|
||
- [x] Inspect an existing Git repository (FR-001)
|
||
- [x] Detect added, modified and deleted files (FR-002 – FR-004)
|
||
- [x] Rename detection via the add+delete fallback (FR-005, Section 6.1)
|
||
- [x] Configurable author resolver chain with confidence levels and `UNKNOWN` fallback; sidecar metadata resolver implemented (FR-006, FR-007, Sections 9 – 11)
|
||
- [x] Per-file attribution in the commit message (FR-008, FR-012)
|
||
- [x] Automatic commit generation (FR-009)
|
||
- [x] Structured commit message: summary counts, one-line-per-file entries with change letter / path / author / local-time mtime, `Tracker-*` trailers (FR-010, FR-011, Sections 15/16/36)
|
||
- [x] No empty commits (FR-014)
|
||
- [x] Rotating application logs, console + file (FR-015, Sections 22 – 23)
|
||
- [x] Cross-platform code base: `pathlib`, argument arrays, UTF-8/NUL-safe Git parsing (FR-016, FR-029)
|
||
- [x] Dry-run mode (FR-017, Section 31)
|
||
- [x] YAML configuration with environment overrides (FR-018, Section 26)
|
||
- [x] Include/exclude file filters with gitignore-style `**` globs (FR-019, Section 7)
|
||
- [x] Repository lock with stale-lock recovery, stored outside the repository (FR-020, FR-028, Section 21)
|
||
- [x] Author-resolution errors never block an otherwise valid commit (FR-021)
|
||
- [x] Git / repository integrity errors prevent the commit; index restored on failure (FR-022, Section 14)
|
||
- [x] One atomic commit per collection cycle (FR-023)
|
||
- [x] Forbidden-file abort with dedicated security exit code (FR-024, FR-027, Section 33)
|
||
- [x] Index-state validation: pre-existing staged changes / unfinished merge abort (FR-026, Section 5.1)
|
||
- [x] Exit codes 0 – 7 for external monitoring (Section 25)
|
||
- [x] CLI: `run` / `status` / `validate` / `config` with secret redaction (Section 30)
|
||
- [x] ULID run identifiers for log/commit traceability (Section 35)
|
||
- [x] Commit author is the tracker service identity, engineers recorded per file (Section 17/18/43)
|
||
- [x] Derived artifacts: SQLite content extraction – `extraction:` config block, deterministic JSON Lines output, stale artifact removal, extraction failure aborts (exit 6), read-only snapshot source access, skipped in dry-run (FR-030 – FR-034, Section 45)
|
||
|
||
### Left to do
|
||
|
||
Required by the specification (v1 gaps):
|
||
|
||
- [ ] Concurrent-modification detection (FR-025, Section 20) – **deferred, risk accepted**: engineering-only repository, once-daily midnight cadence; staged-set verification already prevents broken commits. Revisit if run frequency increases (Section 4.2).
|
||
- [ ] Log execution duration per collection (Section 22)
|
||
- [ ] `validate`: also verify file access (Section 30)
|
||
- [ ] `status`: apply include/exclude filters and preview author resolution (Section 30)
|
||
- [ ] Configurable alternative policy for pre-existing staged changes: commit with `UNKNOWN` attribution (Section 5.1)
|
||
- [ ] Configurable alternative to skip (instead of abort on) forbidden files (Section 33)
|
||
- [ ] Native rename detection instead of the add+delete fallback (FR-005 "should", Section 6.1)
|
||
- [ ] Missing tests (Section 41): failed commit, unavailable Git executable, inaccessible file, file removed during collection, macOS in the CI matrix. (The "file modified during collection" case is deferred together with FR-025.)
|
||
|
||
Blocked on SCADA platform knowledge:
|
||
|
||
- [ ] SCADA metadata resolver – embedded project metadata (Section 10)
|
||
- [ ] SCADA audit-log resolver (Section 10)
|
||
- [ ] Operating-system audit resolver (Section 10, optional)
|
||
|
||
Optional / future extensions (Section 42, "may be added later"):
|
||
|
||
- [ ] Scheduled service mode (Section 4.2)
|
||
- [ ] Remote push with `COMMIT_SUCCESS` / `PUSH_FAILURE` distinction (Section 34)
|
||
- [ ] Signed Git commits
|
||
- [ ] SCADA semantic / XML-aware / JSON-aware comparison
|
||
- [ ] Proprietary SCADA project parsers
|
||
- [ ] Automatic reports, REST API, web interface, Grafana integration
|
||
- [ ] Email/Teams notifications
|
||
- [ ] Change approval workflow, correlation with maintenance/change requests
|
||
- [ ] Active Directory integration
|
||
- [ ] Cryptographic file hashes
|
||
- [ ] IEC/ISO 81346 asset mapping
|
||
- [ ] Detection of changes outside approved maintenance windows
|
||
- [ ] Honor `application.timezone` in log and commit-message timestamps (Sections 22, 26); add the timezone test from Section 41
|
||
## Development
|
||
|
||
```bash
|
||
pip install -e . --group dev
|
||
pytest
|
||
ruff check src tests
|
||
```
|
||
|
||
### Building an executable (local dev option)
|
||
|
||
`scripts/build_exe.py` builds a self-contained executable with
|
||
PyInstaller and smoke-tests it. It creates a dedicated clean build
|
||
environment (`.venv-build`) and installs the package non-editable there —
|
||
the reliable input for PyInstaller analysis. Git is not bundled; it
|
||
remains an external prerequisite on the target machine (Section 28).
|
||
|
||
```bash
|
||
python scripts/build_exe.py # build + smoke test (onedir)
|
||
python scripts/build_exe.py --onefile # single-file binary
|
||
python scripts/build_exe.py --skip-smoke # build only
|
||
```
|
||
|
||
Output: `dist/git-auto-tracker/` (onedir — deploy the whole folder) or a
|
||
single `dist/git-auto-tracker.exe` with `--onefile` (slower startup, more
|
||
prone to antivirus false positives — prefer onedir for SCADA hosts). |