🚧 Add sqlite extraction

This commit is contained in:
Jonas Bengtsson
2026-09-12 23:32:19 +02:00
parent e773703ed3
commit 2e24683347
11 changed files with 988 additions and 19 deletions
+27 -5
View File
@@ -19,6 +19,7 @@ See `functional specification.md` for the full specification.
- Pluggable author resolvers with priority chain and confidence levels
- Per-file attribution recorded in the commit message
- One-line-per-file commit entries in `git diff --name-status` style (letter, path, author, last-modified time), with a configurable separator (`commit.file_separator`, default `;`)
- Derived artifacts: SQLite content extraction into JSON Lines (one sub-folder per database, one file per table), byte-stable, read-only source access (Section 45)
- Structured commit trailers (`Tracker-Run`, `Tracker-Host`, ...)
- Repository-level lock with stale-lock detection (stored outside the repo)
- Rotating file logs plus console logging
@@ -59,8 +60,10 @@ using the `GIT_AUTO_TRACKER_<SECTION>__<KEY>` convention.
## Specification status
Progress against `functional specification.md` (FR-001 – FR-029). The v1
scope is defined by Section 44, *Definition of Done – Initial Release*.
Progress against `functional specification.md` (FR-001 – FR-034). The v1
scope is defined by Section 44, *Definition of Done – Initial Release*;
FR-030 – FR-034 (Section 45, derived artifacts) are prioritized for the
next release.
### Implemented
@@ -87,19 +90,20 @@ scope is defined by Section 44, *Definition of Done – Initial Release*.
- [x] CLI: `run` / `status` / `validate` / `config` with secret redaction (Section 30)
- [x] ULID run identifiers for log/commit traceability (Section 35)
- [x] Commit author is the tracker service identity, engineers recorded per file (Section 17/18/43)
- [x] Derived artifacts: SQLite content extraction – `extraction:` config block, deterministic JSON Lines output, stale artifact removal, extraction failure aborts (exit 6), read-only snapshot source access, skipped in dry-run (FR-030 – FR-034, Section 45)
### Left to do
Required by the specification (v1 gaps):
- [ ] Concurrent-modification detection (FR-025, Section 20) – snapshot helpers exist in `git/status.py` but are not wired into collection; add before/after attribute checks (size / mtime / optional hash) with abort-on-change policy
- [ ] Concurrent-modification detection (FR-025, Section 20) – **deferred, risk accepted**: engineering-only repository, once-daily midnight cadence; staged-set verification already prevents broken commits. Revisit if run frequency increases (Section 4.2).
- [ ] Log execution duration per collection (Section 22)
- [ ] `validate`: also verify file access (Section 30)
- [ ] `status`: apply include/exclude filters and preview author resolution (Section 30)
- [ ] Configurable alternative policy for pre-existing staged changes: commit with `UNKNOWN` attribution (Section 5.1)
- [ ] Configurable alternative to skip (instead of abort on) forbidden files (Section 33)
- [ ] Native rename detection instead of the add+delete fallback (FR-005 "should", Section 6.1)
- [ ] Missing tests (Section 41): failed commit, unavailable Git executable, inaccessible file, file removed during collection, macOS in the CI matrix
- [ ] Missing tests (Section 41): failed commit, unavailable Git executable, inaccessible file, file removed during collection, macOS in the CI matrix. (The "file modified during collection" case is deferred together with FR-025.)
Blocked on SCADA platform knowledge:
@@ -128,4 +132,22 @@ Optional / future extensions (Section 42, "may be added later"):
pip install -e . --group dev
pytest
ruff check src tests
```
```
### Building an executable (local dev option)
`scripts/build_exe.py` builds a self-contained executable with
PyInstaller and smoke-tests it. It creates a dedicated clean build
environment (`.venv-build`) and installs the package non-editable there —
the reliable input for PyInstaller analysis. Git is not bundled; it
remains an external prerequisite on the target machine (Section 28).
```bash
python scripts/build_exe.py # build + smoke test (onedir)
python scripts/build_exe.py --onefile # single-file binary
python scripts/build_exe.py --skip-smoke # build only
```
Output: `dist/git-auto-tracker/` (onedir — deploy the whole folder) or a
single `dist/git-auto-tracker.exe` with `--onefile` (slower startup, more
prone to antivirus false positives — prefer onedir for SCADA hosts).