🚧 Add sqlite extraction
This commit is contained in:
Jonas Bengtsson
@@ -126,6 +126,7 @@ celerybeat.pid
|
|||||||
# Environments
|
# Environments
|
||||||
.env
|
.env
|
||||||
.venv
|
.venv
|
||||||
|
.venv-build
|
||||||
env/
|
env/
|
||||||
venv/
|
venv/
|
||||||
ENV/
|
ENV/
|
||||||
|
|||||||
@@ -19,6 +19,7 @@ See `functional specification.md` for the full specification.
|
|||||||
- Pluggable author resolvers with priority chain and confidence levels
|
- Pluggable author resolvers with priority chain and confidence levels
|
||||||
- Per-file attribution recorded in the commit message
|
- Per-file attribution recorded in the commit message
|
||||||
- One-line-per-file commit entries in `git diff --name-status` style (letter, path, author, last-modified time), with a configurable separator (`commit.file_separator`, default `;`)
|
- One-line-per-file commit entries in `git diff --name-status` style (letter, path, author, last-modified time), with a configurable separator (`commit.file_separator`, default `;`)
|
||||||
|
- Derived artifacts: SQLite content extraction into JSON Lines (one sub-folder per database, one file per table), byte-stable, read-only source access (Section 45)
|
||||||
- Structured commit trailers (`Tracker-Run`, `Tracker-Host`, ...)
|
- Structured commit trailers (`Tracker-Run`, `Tracker-Host`, ...)
|
||||||
- Repository-level lock with stale-lock detection (stored outside the repo)
|
- Repository-level lock with stale-lock detection (stored outside the repo)
|
||||||
- Rotating file logs plus console logging
|
- Rotating file logs plus console logging
|
||||||
@@ -59,8 +60,10 @@ using the `GIT_AUTO_TRACKER_<SECTION>__<KEY>` convention.
|
|||||||
|
|
||||||
## Specification status
|
## Specification status
|
||||||
|
|
||||||
Progress against `functional specification.md` (FR-001 – FR-029). The v1
|
Progress against `functional specification.md` (FR-001 – FR-034). The v1
|
||||||
scope is defined by Section 44, *Definition of Done – Initial Release*.
|
scope is defined by Section 44, *Definition of Done – Initial Release*;
|
||||||
|
FR-030 – FR-034 (Section 45, derived artifacts) are prioritized for the
|
||||||
|
next release.
|
||||||
|
|
||||||
### Implemented
|
### Implemented
|
||||||
|
|
||||||
@@ -87,19 +90,20 @@ scope is defined by Section 44, *Definition of Done – Initial Release*.
|
|||||||
- [x] CLI: `run` / `status` / `validate` / `config` with secret redaction (Section 30)
|
- [x] CLI: `run` / `status` / `validate` / `config` with secret redaction (Section 30)
|
||||||
- [x] ULID run identifiers for log/commit traceability (Section 35)
|
- [x] ULID run identifiers for log/commit traceability (Section 35)
|
||||||
- [x] Commit author is the tracker service identity, engineers recorded per file (Section 17/18/43)
|
- [x] Commit author is the tracker service identity, engineers recorded per file (Section 17/18/43)
|
||||||
|
- [x] Derived artifacts: SQLite content extraction – `extraction:` config block, deterministic JSON Lines output, stale artifact removal, extraction failure aborts (exit 6), read-only snapshot source access, skipped in dry-run (FR-030 – FR-034, Section 45)
|
||||||
|
|
||||||
### Left to do
|
### Left to do
|
||||||
|
|
||||||
Required by the specification (v1 gaps):
|
Required by the specification (v1 gaps):
|
||||||
|
|
||||||
- [ ] Concurrent-modification detection (FR-025, Section 20) – snapshot helpers exist in `git/status.py` but are not wired into collection; add before/after attribute checks (size / mtime / optional hash) with abort-on-change policy
|
- [ ] Concurrent-modification detection (FR-025, Section 20) – **deferred, risk accepted**: engineering-only repository, once-daily midnight cadence; staged-set verification already prevents broken commits. Revisit if run frequency increases (Section 4.2).
|
||||||
- [ ] Log execution duration per collection (Section 22)
|
- [ ] Log execution duration per collection (Section 22)
|
||||||
- [ ] `validate`: also verify file access (Section 30)
|
- [ ] `validate`: also verify file access (Section 30)
|
||||||
- [ ] `status`: apply include/exclude filters and preview author resolution (Section 30)
|
- [ ] `status`: apply include/exclude filters and preview author resolution (Section 30)
|
||||||
- [ ] Configurable alternative policy for pre-existing staged changes: commit with `UNKNOWN` attribution (Section 5.1)
|
- [ ] Configurable alternative policy for pre-existing staged changes: commit with `UNKNOWN` attribution (Section 5.1)
|
||||||
- [ ] Configurable alternative to skip (instead of abort on) forbidden files (Section 33)
|
- [ ] Configurable alternative to skip (instead of abort on) forbidden files (Section 33)
|
||||||
- [ ] Native rename detection instead of the add+delete fallback (FR-005 "should", Section 6.1)
|
- [ ] Native rename detection instead of the add+delete fallback (FR-005 "should", Section 6.1)
|
||||||
- [ ] Missing tests (Section 41): failed commit, unavailable Git executable, inaccessible file, file removed during collection, macOS in the CI matrix
|
- [ ] Missing tests (Section 41): failed commit, unavailable Git executable, inaccessible file, file removed during collection, macOS in the CI matrix. (The "file modified during collection" case is deferred together with FR-025.)
|
||||||
|
|
||||||
Blocked on SCADA platform knowledge:
|
Blocked on SCADA platform knowledge:
|
||||||
|
|
||||||
@@ -129,3 +133,21 @@ pip install -e . --group dev
|
|||||||
pytest
|
pytest
|
||||||
ruff check src tests
|
ruff check src tests
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Building an executable (local dev option)
|
||||||
|
|
||||||
|
`scripts/build_exe.py` builds a self-contained executable with
|
||||||
|
PyInstaller and smoke-tests it. It creates a dedicated clean build
|
||||||
|
environment (`.venv-build`) and installs the package non-editable there —
|
||||||
|
the reliable input for PyInstaller analysis. Git is not bundled; it
|
||||||
|
remains an external prerequisite on the target machine (Section 28).
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python scripts/build_exe.py # build + smoke test (onedir)
|
||||||
|
python scripts/build_exe.py --onefile # single-file binary
|
||||||
|
python scripts/build_exe.py --skip-smoke # build only
|
||||||
|
```
|
||||||
|
|
||||||
|
Output: `dist/git-auto-tracker/` (onedir — deploy the whole folder) or a
|
||||||
|
single `dist/git-auto-tracker.exe` with `--onefile` (slower startup, more
|
||||||
|
prone to antivirus false positives — prefer onedir for SCADA hosts).
|
||||||
@@ -22,6 +22,10 @@ files:
|
|||||||
- "**/*.tmp"
|
- "**/*.tmp"
|
||||||
- "**/*.log"
|
- "**/*.log"
|
||||||
- ".git/**"
|
- ".git/**"
|
||||||
|
# When content extraction (Section 45) is enabled, exclude the raw
|
||||||
|
# binary databases so the repository tracks the readable artifacts
|
||||||
|
# instead of opaque blobs:
|
||||||
|
# - "**/*.sqlite"
|
||||||
|
|
||||||
security:
|
security:
|
||||||
forbidden_patterns:
|
forbidden_patterns:
|
||||||
@@ -42,6 +46,23 @@ commit:
|
|||||||
# identifiers or timestamps.
|
# identifiers or timestamps.
|
||||||
file_separator: ";"
|
file_separator: ";"
|
||||||
|
|
||||||
|
# Derived artifacts (Section 45): extracts configured SQLite databases
|
||||||
|
# into plain-text artifacts that are tracked like any other file. One
|
||||||
|
# sub-folder per source database (named after the database file), one
|
||||||
|
# JSON Lines file per table, under a configurable output root inside
|
||||||
|
# the repository. Extraction opens the source read-only via a snapshot
|
||||||
|
# copy; artifacts are byte-stable so unchanged content produces no
|
||||||
|
# commit; a table removed from the database removes its artifact.
|
||||||
|
extraction:
|
||||||
|
enabled: true
|
||||||
|
output_root: "extracted"
|
||||||
|
format: "jsonl"
|
||||||
|
sources:
|
||||||
|
- path: "data/VV_FALKENBERG.signals.db"
|
||||||
|
# tables: # optional; default: all user tables
|
||||||
|
# - "alarms"
|
||||||
|
# - "events"
|
||||||
|
|
||||||
logging:
|
logging:
|
||||||
level: INFO
|
level: INFO
|
||||||
console: true
|
console: true
|
||||||
|
|||||||
+146
-13
@@ -82,6 +82,12 @@ The application should be divided into independent functional components.
|
|||||||
|
|
|
|
||||||
v
|
v
|
||||||
+-------------------------+
|
+-------------------------+
|
||||||
|
| Artifact Extractor |
|
||||||
|
| (Section 45, optional) |
|
||||||
|
+------------+------------+
|
||||||
|
|
|
||||||
|
v
|
||||||
|
+-------------------------+
|
||||||
| Change Detector |
|
| Change Detector |
|
||||||
+------------+------------+
|
+------------+------------+
|
||||||
|
|
|
|
||||||
@@ -839,6 +845,19 @@ commit:
|
|||||||
# Must not occur inside paths, author identifiers or timestamps.
|
# Must not occur inside paths, author identifiers or timestamps.
|
||||||
file_separator: ";"
|
file_separator: ";"
|
||||||
|
|
||||||
|
extraction:
|
||||||
|
# Derived artifacts (Section 45): extract configured binary databases
|
||||||
|
# into plain-text artifacts before change detection. Absent block or
|
||||||
|
# enabled: false disables the feature entirely.
|
||||||
|
enabled: true
|
||||||
|
output_root: "extracted" # inside the repository
|
||||||
|
format: "jsonl" # one JSON object per row
|
||||||
|
sources:
|
||||||
|
- path: "history/alarms.sqlite" # repository-relative source
|
||||||
|
tables: # optional; default: all user tables
|
||||||
|
- "alarms"
|
||||||
|
- "events"
|
||||||
|
|
||||||
logging:
|
logging:
|
||||||
level: INFO
|
level: INFO
|
||||||
console: true
|
console: true
|
||||||
@@ -1011,6 +1030,8 @@ modify the repository
|
|||||||
|
|
||||||
The generated commit message should be displayed.
|
The generated commit message should be displayed.
|
||||||
|
|
||||||
|
The derived-artifact refresh (Section 45) is skipped in dry-run mode because it writes to the working tree; dry-run analyses the artifacts from the previous collection and reports this in its output.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# 32. Security
|
# 32. Security
|
||||||
@@ -1238,6 +1259,26 @@ The system shall never commit its own operational artefacts, such as lock files,
|
|||||||
|
|
||||||
The system shall correctly process file paths containing spaces, quotes and non-ASCII characters on all supported platforms.
|
The system shall correctly process file paths containing spaces, quotes and non-ASCII characters on all supported platforms.
|
||||||
|
|
||||||
|
## FR-030 – Derived artifact extraction
|
||||||
|
|
||||||
|
The system shall support extraction of configured binary database sources into plain-text derived artifacts inside the repository, refreshed before change detection on every collection.
|
||||||
|
|
||||||
|
## FR-031 – Deterministic artifacts
|
||||||
|
|
||||||
|
Derived artifacts shall be byte-identical between collections whenever the underlying content is unchanged.
|
||||||
|
|
||||||
|
## FR-032 – Extraction failure isolation
|
||||||
|
|
||||||
|
A source that cannot be read, is not a SQLite 3 database, or whose artifacts cannot be written shall be skipped with a logged warning while the remaining configured sources are still extracted.
|
||||||
|
|
||||||
|
## FR-033 – Stale artifact removal
|
||||||
|
|
||||||
|
Artifacts whose source table or database no longer exists shall be removed during extraction.
|
||||||
|
|
||||||
|
## FR-034 – Read-only source access
|
||||||
|
|
||||||
|
The system shall open source databases read-only and shall never write to them.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
# 38. Suggested Python Package Structure
|
# 38. Suggested Python Package Structure
|
||||||
@@ -1271,6 +1312,10 @@ git-auto-tracker/
|
|||||||
│ │ ├── models.py
|
│ │ ├── models.py
|
||||||
│ │ └── filters.py
|
│ │ └── filters.py
|
||||||
│ │
|
│ │
|
||||||
|
│ ├── extraction/
|
||||||
|
│ │ ├── base.py
|
||||||
|
│ │ └── sqlite.py
|
||||||
|
│ │
|
||||||
│ ├── authors/
|
│ ├── authors/
|
||||||
│ │ ├── base.py
|
│ │ ├── base.py
|
||||||
│ │ ├── resolver.py
|
│ │ ├── resolver.py
|
||||||
@@ -1336,21 +1381,22 @@ A normal execution shall approximately perform the following sequence:
|
|||||||
3. Verify Git availability
|
3. Verify Git availability
|
||||||
4. Acquire repository lock
|
4. Acquire repository lock
|
||||||
5. Validate repository, branch and index state
|
5. Validate repository, branch and index state
|
||||||
6. Determine current repository status
|
6. Refresh derived artifacts (Section 45)
|
||||||
7. Apply inclusion/exclusion and security rules
|
7. Determine current repository status
|
||||||
8. Create ChangeSet
|
8. Apply inclusion/exclusion and security rules
|
||||||
9. Resolve authors for ADDED files
|
9. Create ChangeSet
|
||||||
10. Resolve authors for MODIFIED files
|
10. Resolve authors for ADDED files
|
||||||
11. Generate commit message
|
11. Resolve authors for MODIFIED files
|
||||||
12. Stage selected changes
|
12. Generate commit message
|
||||||
13. Verify staged change set
|
13. Stage selected changes
|
||||||
14. Create commit
|
14. Verify staged change set
|
||||||
15. Record commit hash and write audit result
|
15. Create commit
|
||||||
16. Release lock
|
16. Record commit hash and write audit result
|
||||||
17. Exit
|
17. Release lock
|
||||||
|
18. Exit
|
||||||
```
|
```
|
||||||
|
|
||||||
If no changes exist, execution proceeds directly from step 8 to audit logging and termination.
|
If no changes exist, execution proceeds directly from step 9 to audit logging and termination.
|
||||||
|
|
||||||
If any step from staging onwards fails, the index shall be restored to its pre-execution state before the lock is released.
|
If any step from staging onwards fails, the index shall be restored to its pre-execution state before the lock is released.
|
||||||
|
|
||||||
@@ -1394,6 +1440,15 @@ Tests should cover at minimum:
|
|||||||
* excluded file;
|
* excluded file;
|
||||||
* forbidden file.
|
* forbidden file.
|
||||||
|
|
||||||
|
### Extraction (Section 45)
|
||||||
|
|
||||||
|
* enabled extraction produces an artifact file for every configured table;
|
||||||
|
* unchanged database content produces byte-identical artifacts (no spurious commit);
|
||||||
|
* a modified row appears as an artifact change in the commit;
|
||||||
|
* a removed table or database removes its artifact files;
|
||||||
|
* an unreadable or locked source database aborts with exit code 6;
|
||||||
|
* source databases are opened read-only.
|
||||||
|
|
||||||
### Safety and edge cases
|
### Safety and edge cases
|
||||||
|
|
||||||
* pre-existing staged changes in the index;
|
* pre-existing staged changes in the index;
|
||||||
@@ -1494,6 +1549,82 @@ The initial implementation should deliberately keep **SCADA-specific change attr
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
# 45. Derived Artifacts (Content Extraction)
|
||||||
|
|
||||||
|
This feature is prioritized for the next release.
|
||||||
|
|
||||||
|
## 45.1 Purpose
|
||||||
|
|
||||||
|
Some SCADA engineering data is stored in binary databases (e.g. SQLite). Git cannot show readable history for such files: every commit stores the whole binary blob and a diff reports only "binary files differ". To make this content auditable, the tracker shall extract selected database content into plain-text derived artifacts inside the repository before change detection (Section 40, step 6). The artifacts are then versioned like any other file, and Git history shows precise, row-level changes.
|
||||||
|
|
||||||
|
Source databases are identified by content (the SQLite file header read during extraction), not by file-name extension.
|
||||||
|
|
||||||
|
The implementation shall follow the same pluggable-interface principle as author resolution (Section 44): an `ArtifactExtractor` abstraction with a SQLite implementation as the first plugin, so further formats can be added independently.
|
||||||
|
|
||||||
|
## 45.2 Output layout
|
||||||
|
|
||||||
|
Extraction writes into a configurable output root inside the repository (default: `extracted/`). One sub-folder is created per configured source database, named after the database file. Within each sub-folder, one file is written per extracted table.
|
||||||
|
|
||||||
|
```text
|
||||||
|
extracted/
|
||||||
|
└── alarms.sqlite/
|
||||||
|
├── alarms.jsonl
|
||||||
|
└── events.jsonl
|
||||||
|
```
|
||||||
|
|
||||||
|
This layout gives each table its own artifact file, so Git diffs are limited to the tables that actually changed, and table names cannot collide between databases.
|
||||||
|
|
||||||
|
## 45.3 Configuration
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
extraction:
|
||||||
|
enabled: true
|
||||||
|
output_root: "extracted" # relative to the repository root
|
||||||
|
format: "jsonl" # one JSON object per row
|
||||||
|
sources:
|
||||||
|
- path: "history/alarms.sqlite" # repository-relative source
|
||||||
|
tables: # optional; default: all user tables
|
||||||
|
- "alarms"
|
||||||
|
- "events"
|
||||||
|
```
|
||||||
|
|
||||||
|
An absent `extraction:` block, or `enabled: false`, disables the feature entirely; collection then proceeds without touching any database.
|
||||||
|
|
||||||
|
## 45.4 Artifact format
|
||||||
|
|
||||||
|
The default format shall be JSON Lines (`jsonl`): one JSON object per row with sorted keys and a stable encoding, so a changed row produces a minimal line-level diff. Rows shall be ordered deterministically by primary key (falling back to `rowid`), so unchanged content always produces a byte-identical file.
|
||||||
|
|
||||||
|
Artifact content shall not contain run-dependent values such as extraction timestamps: an artifact must be byte-identical whenever the underlying content is unchanged (FR-031). Table names that are not valid file names on the target platform shall be sanitized deterministically for artifact file names (FR-029).
|
||||||
|
|
||||||
|
## 45.5 Source access
|
||||||
|
|
||||||
|
Source databases shall be opened read-only; the tracker shall never write to them (FR-034). Because the SCADA platform may keep databases in WAL mode while running, extraction shall operate on a snapshot: the database file and its WAL sidecar shall be copied to a temporary location before reading, so a partially written transaction is never captured (see also Section 20).
|
||||||
|
|
||||||
|
## 45.6 Stale artifacts
|
||||||
|
|
||||||
|
Extraction shall also remove artifacts that no longer have a source:
|
||||||
|
|
||||||
|
* a table removed from the database → its artifact file is deleted;
|
||||||
|
* a source database removed from the configuration or the file system → its sub-folder is deleted.
|
||||||
|
|
||||||
|
Without this rule, stale artifacts would look like current data in Git history (FR-033).
|
||||||
|
|
||||||
|
## 45.7 Failure policy
|
||||||
|
|
||||||
|
Sources are isolated from each other: a source that cannot be read, is not a SQLite 3 database, or whose artifacts cannot be written, is reported as a warning and skipped; extraction then continues with the remaining configured sources. One bad file shall never block extraction of the others. When every configured source fails, the collection proceeds with the artifacts from the previous collection and a summary warning lists all skipped sources. Source validity is established during extraction by the SQLite engine itself - a non-database file raises `sqlite3.DatabaseError: file is not a database` - rather than by file-name extension: databases with vendor-specific or missing extensions are accepted (Section 45.5).
|
||||||
|
|
||||||
|
An extraction failure is not silent, however: every skipped source is logged, and a source that fails persistently must be visible in monitoring through the log (Section 22).
|
||||||
|
|
||||||
|
## 45.8 Raw source files
|
||||||
|
|
||||||
|
The configured binary source files should be excluded from direct tracking (`files.exclude`, e.g. `**/*.sqlite`). The repository then stores the readable artifacts instead of the opaque blob; tracking both duplicates every change and lets the binary churn dominate history. Direct tracking of the raw database remains possible where a deployment wants the authoritative blob as well.
|
||||||
|
|
||||||
|
## 45.9 Attribution
|
||||||
|
|
||||||
|
Generated artifacts have no human last-modifier of their own. Author resolution for artifact files shall therefore report `UNKNOWN` unless an extractor emits sidecar metadata (Section 10) that the resolver chain can consume, for example a `last_modified_by` column extracted from the database content.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
# Appendix A – Glossary
|
# Appendix A – Glossary
|
||||||
|
|
||||||
| Term | Definition |
|
| Term | Definition |
|
||||||
@@ -1508,3 +1639,5 @@ The initial implementation should deliberately keep **SCADA-specific change attr
|
|||||||
| Forbidden file | A file matching `security.forbidden_patterns`; never committed by default |
|
| Forbidden file | A file matching `security.forbidden_patterns`; never committed by default |
|
||||||
| Tracker-Run | Unique execution identifier linking logs, commits and monitoring (Section 35) |
|
| Tracker-Run | Unique execution identifier linking logs, commits and monitoring (Section 35) |
|
||||||
| Staged verification | Comparison of the actual staged change set against the expected change set before commit |
|
| Staged verification | Comparison of the actual staged change set against the expected change set before commit |
|
||||||
|
| Derived artifact | Plain-text file generated from a binary source (e.g. a SQLite table) and versioned in the repository (Section 45) |
|
||||||
|
| Artifact extractor | Pluggable component that refreshes derived artifacts from configured sources before change detection |
|
||||||
|
|||||||
@@ -23,6 +23,8 @@ from git_auto_tracker.commits.commit import CommitManager
|
|||||||
from git_auto_tracker.commits.message_builder import build_commit_message
|
from git_auto_tracker.commits.message_builder import build_commit_message
|
||||||
from git_auto_tracker.config import AppConfig
|
from git_auto_tracker.config import AppConfig
|
||||||
from git_auto_tracker.errors import RepositoryError
|
from git_auto_tracker.errors import RepositoryError
|
||||||
|
from git_auto_tracker.extraction.base import ExtractionSource
|
||||||
|
from git_auto_tracker.extraction.sqlite import SqliteExtractor
|
||||||
from git_auto_tracker.git.repository import GitCommand
|
from git_auto_tracker.git.repository import GitCommand
|
||||||
from git_auto_tracker.git.status import ChangeDetector, RawStatusEntry
|
from git_auto_tracker.git.status import ChangeDetector, RawStatusEntry
|
||||||
from git_auto_tracker.logging.setup import get_logger
|
from git_auto_tracker.logging.setup import get_logger
|
||||||
@@ -56,6 +58,53 @@ class ApplicationController:
|
|||||||
self.security = SecurityValidator(config.security)
|
self.security = SecurityValidator(config.security)
|
||||||
self.resolvers = ResolverChain.from_config(config.author_resolution)
|
self.resolvers = ResolverChain.from_config(config.author_resolution)
|
||||||
self.committer = CommitManager(self.git)
|
self.committer = CommitManager(self.git)
|
||||||
|
self.extractor = SqliteExtractor() if config.extraction.enabled else None
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
# Derived artifacts (Section 45)
|
||||||
|
# ------------------------------------------------------------------
|
||||||
|
|
||||||
|
def refresh_artifacts(self) -> None:
|
||||||
|
"""Refresh derived artifacts for all configured sources (Section 45).
|
||||||
|
|
||||||
|
Sources are isolated from each other (Section 45.7): a source
|
||||||
|
that cannot be read or is not a SQLite 3 database is logged as a
|
||||||
|
warning and skipped, so one bad file never blocks extraction of
|
||||||
|
the remaining configured sources. sqlite3 itself rejects
|
||||||
|
non-database files during extraction.
|
||||||
|
"""
|
||||||
|
if self.extractor is None:
|
||||||
|
return
|
||||||
|
config = self.config.extraction
|
||||||
|
output_root = self.config.repository_path / config.output_root
|
||||||
|
failed: list[str] = []
|
||||||
|
for source_cfg in config.sources:
|
||||||
|
source = ExtractionSource(path=source_cfg.path, tables=source_cfg.tables)
|
||||||
|
try:
|
||||||
|
result = self.extractor.extract(
|
||||||
|
source, self.config.repository_path, output_root
|
||||||
|
)
|
||||||
|
except Exception as exc:
|
||||||
|
failed.append(source.path)
|
||||||
|
self._log.warning(
|
||||||
|
"Extraction: skipping source %s: %s", source.path, exc
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
written = len(result.artifacts)
|
||||||
|
removed = len(result.removed)
|
||||||
|
self._log.info(
|
||||||
|
"Extraction: %s -> %d artifact(s) written, %d removed",
|
||||||
|
source.path,
|
||||||
|
written,
|
||||||
|
removed,
|
||||||
|
)
|
||||||
|
if failed:
|
||||||
|
self._log.warning(
|
||||||
|
"Extraction: %d of %d source(s) skipped: %s",
|
||||||
|
len(failed),
|
||||||
|
len(config.sources),
|
||||||
|
", ".join(sorted(failed)),
|
||||||
|
)
|
||||||
|
|
||||||
# ------------------------------------------------------------------
|
# ------------------------------------------------------------------
|
||||||
# Validation (Sections 5 and 5.1)
|
# Validation (Sections 5 and 5.1)
|
||||||
@@ -139,6 +188,18 @@ class ApplicationController:
|
|||||||
branch = self.validate_repository()
|
branch = self.validate_repository()
|
||||||
log.info("Branch: %s", branch)
|
log.info("Branch: %s", branch)
|
||||||
|
|
||||||
|
# Section 40 step 6: refresh derived artifacts before change
|
||||||
|
# detection. Skipped in dry-run mode: the refresh writes to the
|
||||||
|
# working tree, which a dry run must not modify (Section 31).
|
||||||
|
if dry_run:
|
||||||
|
if self.extractor is not None:
|
||||||
|
log.info(
|
||||||
|
"Dry-run: artifact refresh skipped; analyzing artifacts "
|
||||||
|
"from the previous collection"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
self.refresh_artifacts()
|
||||||
|
|
||||||
entries = self.detector.detect()
|
entries = self.detector.detect()
|
||||||
change_set = self._records_from_status(entries)
|
change_set = self._records_from_status(entries)
|
||||||
|
|
||||||
@@ -152,7 +213,7 @@ class ApplicationController:
|
|||||||
|
|
||||||
if kept.is_empty:
|
if kept.is_empty:
|
||||||
log.info("No SCADA changes detected.")
|
log.info("No SCADA changes detected.")
|
||||||
return CollectionOutcome(None, 0, None)
|
return CollectionOutcome(None, 0, None, dry_run=dry_run)
|
||||||
|
|
||||||
self.security.validate(kept)
|
self.security.validate(kept)
|
||||||
|
|
||||||
|
|||||||
@@ -108,6 +108,65 @@ class CommitConfig:
|
|||||||
return cls(file_separator=separator)
|
return cls(file_separator=separator)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class ExtractionSourceConfig:
|
||||||
|
"""One configured extraction source (Section 45.3)."""
|
||||||
|
|
||||||
|
path: str
|
||||||
|
tables: tuple[str, ...] = ()
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class ExtractionConfig:
|
||||||
|
"""Derived-artifact extraction options (Section 45)."""
|
||||||
|
|
||||||
|
enabled: bool = False
|
||||||
|
output_root: str = "extracted"
|
||||||
|
format: str = "jsonl"
|
||||||
|
sources: list[ExtractionSourceConfig] = field(default_factory=list)
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def from_dict(cls, data: dict[str, Any]) -> ExtractionConfig:
|
||||||
|
enabled = bool(data.get("enabled", False))
|
||||||
|
output_root = data.get("output_root", "extracted")
|
||||||
|
if not isinstance(output_root, str) or not output_root.strip():
|
||||||
|
raise ConfigurationError("extraction.output_root must be a non-empty string")
|
||||||
|
if output_root in (".", "..") or "/" in output_root or "\\" in output_root:
|
||||||
|
raise ConfigurationError(
|
||||||
|
"extraction.output_root must be a single folder name inside the repository"
|
||||||
|
)
|
||||||
|
fmt = data.get("format", "jsonl")
|
||||||
|
if fmt != "jsonl":
|
||||||
|
raise ConfigurationError(f"unsupported extraction.format: {fmt!r} (only 'jsonl')")
|
||||||
|
raw_sources = data.get("sources", [])
|
||||||
|
if not isinstance(raw_sources, list):
|
||||||
|
raise ConfigurationError("extraction.sources must be a list")
|
||||||
|
sources: list[ExtractionSourceConfig] = []
|
||||||
|
for index, item in enumerate(raw_sources):
|
||||||
|
if not isinstance(item, dict):
|
||||||
|
raise ConfigurationError(f"extraction.sources[{index}] must be a mapping")
|
||||||
|
path = item.get("path")
|
||||||
|
if not isinstance(path, str) or not path.strip():
|
||||||
|
raise ConfigurationError(
|
||||||
|
f"extraction.sources[{index}].path must be a non-empty string"
|
||||||
|
)
|
||||||
|
if ".." in path.split("/") or "\\" in path:
|
||||||
|
raise ConfigurationError(
|
||||||
|
f"extraction.sources[{index}].path must stay inside the repository"
|
||||||
|
)
|
||||||
|
raw_tables = item.get("tables", [])
|
||||||
|
if not isinstance(raw_tables, list) or not all(
|
||||||
|
isinstance(t, str) and t.strip() for t in raw_tables
|
||||||
|
):
|
||||||
|
raise ConfigurationError(
|
||||||
|
f"extraction.sources[{index}].tables must be a list of table names"
|
||||||
|
)
|
||||||
|
sources.append(ExtractionSourceConfig(path=path, tables=tuple(raw_tables)))
|
||||||
|
if enabled and not sources:
|
||||||
|
raise ConfigurationError("extraction.sources must not be empty when enabled")
|
||||||
|
return cls(enabled=enabled, output_root=output_root, format=fmt, sources=sources)
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class AppConfig:
|
class AppConfig:
|
||||||
repository_path: Path
|
repository_path: Path
|
||||||
@@ -120,6 +179,7 @@ class AppConfig:
|
|||||||
logging: LoggingConfig
|
logging: LoggingConfig
|
||||||
timezone: str | None = None
|
timezone: str | None = None
|
||||||
commit: CommitConfig = field(default_factory=CommitConfig)
|
commit: CommitConfig = field(default_factory=CommitConfig)
|
||||||
|
extraction: ExtractionConfig = field(default_factory=ExtractionConfig)
|
||||||
|
|
||||||
@classmethod
|
@classmethod
|
||||||
def from_dict(cls, data: dict[str, Any], config_dir: Path) -> AppConfig:
|
def from_dict(cls, data: dict[str, Any], config_dir: Path) -> AppConfig:
|
||||||
@@ -172,6 +232,7 @@ class AppConfig:
|
|||||||
backup_count=int(logging_cfg.get("backup_count", 10)),
|
backup_count=int(logging_cfg.get("backup_count", 10)),
|
||||||
),
|
),
|
||||||
commit=CommitConfig.from_dict(data.get("commit", {})),
|
commit=CommitConfig.from_dict(data.get("commit", {})),
|
||||||
|
extraction=ExtractionConfig.from_dict(data.get("extraction", {})),
|
||||||
timezone=app_cfg.get("timezone"),
|
timezone=app_cfg.get("timezone"),
|
||||||
)
|
)
|
||||||
except ConfigurationError:
|
except ConfigurationError:
|
||||||
|
|||||||
@@ -0,0 +1 @@
|
|||||||
|
"""Content extraction into derived artifacts (Section 45)."""
|
||||||
@@ -0,0 +1,156 @@
|
|||||||
|
"""Artifact extraction framework (Section 45).
|
||||||
|
|
||||||
|
Derived artifacts are plain-text files generated from binary sources (e.g.
|
||||||
|
SQLite databases) and versioned in the repository so Git history shows
|
||||||
|
readable, row-level changes instead of opaque binary blobs.
|
||||||
|
|
||||||
|
Follows the same pluggable-interface principle as author resolution
|
||||||
|
(Section 44): an ArtifactExtractor abstraction with format-specific
|
||||||
|
implementations added independently.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import datetime as dt
|
||||||
|
from abc import ABC, abstractmethod
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class ExtractionSource:
|
||||||
|
"""One configured binary source (Section 45.3)."""
|
||||||
|
|
||||||
|
#: Repository-relative path of the source database (POSIX separators)
|
||||||
|
path: str
|
||||||
|
#: Tables to extract; empty means all user tables
|
||||||
|
tables: tuple[str, ...] = ()
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class ExtractionResult:
|
||||||
|
"""Outcome of extracting one source database."""
|
||||||
|
|
||||||
|
source: str
|
||||||
|
#: Repository-relative artifact paths written or refreshed
|
||||||
|
artifacts: list[str] = field(default_factory=list)
|
||||||
|
#: Repository-relative artifact paths removed (stale, Section 45.6)
|
||||||
|
removed: list[str] = field(default_factory=list)
|
||||||
|
#: True when nothing changed relative to the previous artifacts
|
||||||
|
unchanged: bool = False
|
||||||
|
|
||||||
|
|
||||||
|
class ArtifactExtractor(ABC):
|
||||||
|
"""Pluggable content extractor (Section 45.1).
|
||||||
|
|
||||||
|
Implementations refresh derived artifacts from one configured source
|
||||||
|
into a sub-folder of the output root named after the source file.
|
||||||
|
|
||||||
|
Contract (Sections 45.4 - 45.7):
|
||||||
|
* open the source read-only; never write to it (FR-034);
|
||||||
|
* operate on a snapshot when the live source may change concurrently;
|
||||||
|
* produce byte-identical output for unchanged content (FR-031);
|
||||||
|
* remove artifacts whose source table no longer exists (FR-033);
|
||||||
|
* raise ExtractionError when the source cannot be read or an
|
||||||
|
artifact cannot be written; the caller aborts the collection
|
||||||
|
(FR-032) - silent staleness is not acceptable.
|
||||||
|
"""
|
||||||
|
|
||||||
|
#: Registry name used in configuration (extraction.sources[].format)
|
||||||
|
name: str = "abstract"
|
||||||
|
|
||||||
|
@abstractmethod
|
||||||
|
def extract(
|
||||||
|
self,
|
||||||
|
source: ExtractionSource,
|
||||||
|
repository_root: Path,
|
||||||
|
output_root: Path,
|
||||||
|
) -> ExtractionResult:
|
||||||
|
"""Refresh the derived artifacts for one source; returns its result."""
|
||||||
|
|
||||||
|
|
||||||
|
class ExtractionError(Exception):
|
||||||
|
"""Raised when extraction fails; aborts the collection (FR-032)."""
|
||||||
|
|
||||||
|
|
||||||
|
def default_output_root(name: str) -> str:
|
||||||
|
"""Repository-relative output root folder (Section 45.2, default)."""
|
||||||
|
return name if name else "extracted"
|
||||||
|
|
||||||
|
|
||||||
|
def artifact_folder(source: ExtractionSource) -> str:
|
||||||
|
"""Sub-folder name for one source database (Section 45.2).
|
||||||
|
|
||||||
|
Named after the database file (without extension), so table artifact
|
||||||
|
files cannot collide between databases.
|
||||||
|
"""
|
||||||
|
stem = Path(source.path).stem
|
||||||
|
return stem or "source"
|
||||||
|
|
||||||
|
|
||||||
|
def artifact_filename(table: str) -> str:
|
||||||
|
"""Deterministic, platform-safe artifact file name for a table (FR-029)."""
|
||||||
|
# Keep characters that are safe on Windows/Linux/macOS; replace the
|
||||||
|
# rest with underscores; collapse runs; strip leading/trailing ones.
|
||||||
|
safe = "".join(ch if (ch.isalnum() or ch in "._- ") else "_" for ch in table)
|
||||||
|
while "__" in safe:
|
||||||
|
safe = safe.replace("__", "_")
|
||||||
|
safe = safe.strip("._")
|
||||||
|
reserved = {
|
||||||
|
"con", "prn", "aux", "nul",
|
||||||
|
"com1", "com2", "com3", "com4", "com5", "com6", "com7", "com8", "com9",
|
||||||
|
"lpt1", "lpt2", "lpt3", "lpt4", "lpt5", "lpt6", "lpt7", "lpt8", "lpt9",
|
||||||
|
}
|
||||||
|
if safe.lower() in reserved:
|
||||||
|
safe = f"_{safe}"
|
||||||
|
return f"{safe or 'table'}.jsonl"
|
||||||
|
|
||||||
|
|
||||||
|
def to_jsonl(rows: list[dict[str, object]]) -> str:
|
||||||
|
"""Serialize rows to deterministic JSON Lines (Section 45.4).
|
||||||
|
|
||||||
|
Keys sorted for stable column order; ensure_ascii=False for readable,
|
||||||
|
byte-stable UTF-8; separators without padding for minimal diffs.
|
||||||
|
Rows must already be in deterministic order (caller orders by PK).
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
|
||||||
|
lines = [
|
||||||
|
json.dumps(row, sort_keys=True, ensure_ascii=False, separators=(",", ":"))
|
||||||
|
for row in rows
|
||||||
|
]
|
||||||
|
return "\n".join(lines) + "\n" if lines else ""
|
||||||
|
|
||||||
|
|
||||||
|
def write_if_changed(path: Path, content: str) -> bool:
|
||||||
|
"""Write content only when it differs; returns True when written.
|
||||||
|
|
||||||
|
Avoiding unconditional writes keeps file mtime (and therefore the
|
||||||
|
commit message's last-modified column) stable for unchanged tables,
|
||||||
|
and prevents spurious change records.
|
||||||
|
"""
|
||||||
|
if path.exists():
|
||||||
|
try:
|
||||||
|
if path.read_text(encoding="utf-8") == content:
|
||||||
|
return False
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
path.write_text(content, encoding="utf-8", newline="\n")
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def remove_if_exists(path: Path) -> bool:
|
||||||
|
"""Delete a stale artifact file; returns True when removed (FR-033)."""
|
||||||
|
if not path.exists():
|
||||||
|
return False
|
||||||
|
try:
|
||||||
|
path.unlink()
|
||||||
|
return True
|
||||||
|
except OSError:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def timestamp_note(moment: dt.datetime) -> str: # pragma: no cover - future use
|
||||||
|
"""Reserved for potential sidecar metadata (Section 45.9)."""
|
||||||
|
return moment.isoformat()
|
||||||
@@ -0,0 +1,202 @@
|
|||||||
|
"""SQLite artifact extractor (Section 45).
|
||||||
|
|
||||||
|
Extracts configured tables of a SQLite database into JSON Lines artifacts
|
||||||
|
under the output root:
|
||||||
|
|
||||||
|
<output_root>/<database-stem>/<table>.jsonl
|
||||||
|
|
||||||
|
Requirements implemented here (Section 45.4 - 45.6):
|
||||||
|
* read-only source access via snapshot (FR-034): the database file and its
|
||||||
|
WAL sidecar are copied to a temporary location before reading, so a
|
||||||
|
live SCADA system writing in WAL mode is never disturbed and a
|
||||||
|
partially written transaction is never captured;
|
||||||
|
* deterministic output (FR-031): rows ordered by primary key (rowid
|
||||||
|
fallback), stable JSON encoding, no run-dependent values;
|
||||||
|
* stale artifact removal (FR-033): artifact files for tables that no
|
||||||
|
longer exist in the database are deleted.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import shutil
|
||||||
|
import sqlite3
|
||||||
|
import tempfile
|
||||||
|
from contextlib import contextmanager
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
from git_auto_tracker.errors import CollectionError
|
||||||
|
from git_auto_tracker.extraction.base import (
|
||||||
|
ExtractionError,
|
||||||
|
ExtractionResult,
|
||||||
|
ExtractionSource,
|
||||||
|
artifact_filename,
|
||||||
|
artifact_folder,
|
||||||
|
remove_if_exists,
|
||||||
|
to_jsonl,
|
||||||
|
write_if_changed,
|
||||||
|
)
|
||||||
|
|
||||||
|
_USER_TABLES_SQL = (
|
||||||
|
"SELECT name FROM sqlite_master "
|
||||||
|
"WHERE type = 'table' AND name NOT LIKE 'sqlite_%' "
|
||||||
|
"ORDER BY name"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@contextmanager
|
||||||
|
def _snapshot_database(source_path: Path):
|
||||||
|
"""Copy the database (and WAL sidecar) to temp; open the copy (45.5).
|
||||||
|
|
||||||
|
Opening the snapshot instead of the live file guarantees read-only
|
||||||
|
access to the source (FR-034) and a consistent view even when the
|
||||||
|
source is being written concurrently (Section 20). The connection is
|
||||||
|
closed before the temporary directory is removed; otherwise Windows
|
||||||
|
keeps the file locked and cleanup fails.
|
||||||
|
"""
|
||||||
|
with tempfile.TemporaryDirectory(prefix="git-auto-tracker-extract-") as tmp:
|
||||||
|
snapshot = Path(tmp) / source_path.name
|
||||||
|
shutil.copy2(source_path, snapshot)
|
||||||
|
for sidecar in ("-wal", "-shm"):
|
||||||
|
live = source_path.with_name(source_path.name + sidecar)
|
||||||
|
if live.exists():
|
||||||
|
shutil.copy2(live, snapshot.with_name(snapshot.name + sidecar))
|
||||||
|
connection = sqlite3.connect(f"file:{snapshot}?mode=ro", uri=True)
|
||||||
|
try:
|
||||||
|
yield connection
|
||||||
|
finally:
|
||||||
|
connection.close()
|
||||||
|
|
||||||
|
|
||||||
|
def _order_by(table: str, columns: list[str]) -> str:
|
||||||
|
"""Deterministic ORDER BY clause: primary key columns, rowid fallback."""
|
||||||
|
pk_columns = [c for c in columns if not c.startswith("_rowid_")]
|
||||||
|
quoted = ", ".join(f'"{c}"' for c in pk_columns) if pk_columns else "rowid"
|
||||||
|
return quoted
|
||||||
|
|
||||||
|
|
||||||
|
def _primary_key_columns(cursor: sqlite3.Cursor, table: str) -> list[str]:
|
||||||
|
"""Primary key column names for a table, in index order; [] if none."""
|
||||||
|
pk_columns: list[str] = []
|
||||||
|
for row in cursor.execute(f'PRAGMA table_info("{table}")'):
|
||||||
|
# row: (cid, name, type, notnull, dflt_value, pk)
|
||||||
|
if row[5]: # pk index (1-based ordinal)
|
||||||
|
while len(pk_columns) < row[5]:
|
||||||
|
pk_columns.append("")
|
||||||
|
pk_columns[row[5] - 1] = str(row[1])
|
||||||
|
return pk_columns
|
||||||
|
|
||||||
|
|
||||||
|
def _sanitize(identifier: str) -> str:
|
||||||
|
"""Quote an identifier for SQL; rejects embedded quotes (FR-029)."""
|
||||||
|
if '"' in identifier:
|
||||||
|
raise ExtractionError(f"invalid table name: {identifier!r}")
|
||||||
|
return f'"{identifier}"'
|
||||||
|
|
||||||
|
|
||||||
|
class SqliteExtractor:
|
||||||
|
"""Extracts SQLite tables to JSON Lines artifacts (Section 45)."""
|
||||||
|
|
||||||
|
name = "sqlite"
|
||||||
|
|
||||||
|
def extract(
|
||||||
|
self,
|
||||||
|
source: ExtractionSource,
|
||||||
|
repository_root: Path,
|
||||||
|
output_root: Path,
|
||||||
|
) -> ExtractionResult:
|
||||||
|
source_path = repository_root / Path(source.path)
|
||||||
|
if not source_path.is_file():
|
||||||
|
raise ExtractionError(f"source database not found: {source.path}")
|
||||||
|
|
||||||
|
folder = output_root / artifact_folder(source)
|
||||||
|
result = ExtractionResult(source=source.path)
|
||||||
|
|
||||||
|
try:
|
||||||
|
with _snapshot_database(source_path) as connection:
|
||||||
|
connection.row_factory = sqlite3.Row
|
||||||
|
cursor = connection.cursor()
|
||||||
|
tables = self._resolve_tables(cursor, source.tables)
|
||||||
|
for table in tables:
|
||||||
|
artifact = folder / artifact_filename(table)
|
||||||
|
rel = self._relative(repository_root, artifact)
|
||||||
|
content = self._dump_table(cursor, table)
|
||||||
|
if write_if_changed(artifact, content):
|
||||||
|
result.artifacts.append(rel)
|
||||||
|
else:
|
||||||
|
result.unchanged = True
|
||||||
|
result.removed = self._remove_stale(folder, tables, repository_root)
|
||||||
|
except ExtractionError:
|
||||||
|
raise
|
||||||
|
except sqlite3.Error as exc:
|
||||||
|
raise ExtractionError(f"failed to extract {source.path}: {exc}") from exc
|
||||||
|
except OSError as exc:
|
||||||
|
raise ExtractionError(f"cannot read source {source.path}: {exc}") from exc
|
||||||
|
|
||||||
|
if not result.artifacts and not result.removed and not result.unchanged:
|
||||||
|
# Source had no user tables at all
|
||||||
|
result.unchanged = True
|
||||||
|
return result
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _relative(repository_root: Path, path: Path) -> str:
|
||||||
|
try:
|
||||||
|
return path.resolve().relative_to(repository_root.resolve()).as_posix()
|
||||||
|
except ValueError:
|
||||||
|
return path.as_posix()
|
||||||
|
|
||||||
|
def _resolve_tables(self, cursor: sqlite3.Cursor, wanted: tuple[str, ...]) -> list[str]:
|
||||||
|
"""Configured tables validated against the database; empty = all."""
|
||||||
|
available = [
|
||||||
|
str(row[0])
|
||||||
|
for row in cursor.execute(_USER_TABLES_SQL)
|
||||||
|
]
|
||||||
|
if not wanted:
|
||||||
|
return available
|
||||||
|
known = set(available)
|
||||||
|
for table in wanted:
|
||||||
|
if table not in known:
|
||||||
|
raise ExtractionError(
|
||||||
|
f"configured table {table!r} does not exist; "
|
||||||
|
f"available: {', '.join(available) or 'none'}"
|
||||||
|
)
|
||||||
|
_sanitize(table)
|
||||||
|
return [t for t in available if t in set(wanted)]
|
||||||
|
|
||||||
|
def _dump_table(self, cursor: sqlite3.Cursor, table: str) -> str:
|
||||||
|
"""One JSON object per row, ordered by primary key (Section 45.4)."""
|
||||||
|
pk = _primary_key_columns(cursor, table)
|
||||||
|
order = ", ".join(f'"{c}"' for c in pk) if pk else "rowid"
|
||||||
|
rows = [
|
||||||
|
{str(key): value for key, value in dict(row).items()}
|
||||||
|
for row in cursor.execute(
|
||||||
|
f"SELECT * FROM {_sanitize(table)} ORDER BY {order}"
|
||||||
|
)
|
||||||
|
]
|
||||||
|
return to_jsonl(rows)
|
||||||
|
|
||||||
|
def _remove_stale(
|
||||||
|
self,
|
||||||
|
folder: Path,
|
||||||
|
current_tables: list[str],
|
||||||
|
repository_root: Path,
|
||||||
|
) -> list[str]:
|
||||||
|
"""Delete artifact files whose table no longer exists (FR-033)."""
|
||||||
|
expected = {artifact_filename(table) for table in current_tables}
|
||||||
|
removed: list[str] = []
|
||||||
|
if not folder.is_dir():
|
||||||
|
return removed
|
||||||
|
for existing in sorted(folder.iterdir()):
|
||||||
|
if existing.is_file() and existing.name not in expected and remove_if_exists(existing):
|
||||||
|
removed.append(self._relative(repository_root, existing))
|
||||||
|
return removed
|
||||||
|
|
||||||
|
|
||||||
|
def extractor_for(name: str) -> SqliteExtractor:
|
||||||
|
"""Registry lookup for configured extractor names (Section 45.3)."""
|
||||||
|
registry = {SqliteExtractor.name: SqliteExtractor}
|
||||||
|
factory = registry.get(name)
|
||||||
|
if factory is None:
|
||||||
|
raise CollectionError(
|
||||||
|
f"unknown extraction source format: {name!r}; available: {list(registry)}"
|
||||||
|
)
|
||||||
|
return factory()
|
||||||
@@ -169,6 +169,132 @@ class TestRunCycle:
|
|||||||
assert "Station PS101/räumliche Datei.xml" in files
|
assert "Station PS101/räumliche Datei.xml" in files
|
||||||
|
|
||||||
|
|
||||||
|
class TestExtraction:
|
||||||
|
"""Section 45: derived artifacts flow through the full collection."""
|
||||||
|
|
||||||
|
def _make_sqlite(self, git_repo: Path) -> Path:
|
||||||
|
import sqlite3
|
||||||
|
|
||||||
|
history = git_repo / "history"
|
||||||
|
history.mkdir(exist_ok=True)
|
||||||
|
db = history / "alarms.sqlite"
|
||||||
|
connection = sqlite3.connect(db)
|
||||||
|
connection.execute("CREATE TABLE alarms (id INTEGER PRIMARY KEY, text TEXT)")
|
||||||
|
connection.execute("INSERT INTO alarms VALUES (1, 'alpha')")
|
||||||
|
connection.commit()
|
||||||
|
connection.close()
|
||||||
|
return db
|
||||||
|
|
||||||
|
def _config_with_extraction(self, tracker_config: dict) -> dict:
|
||||||
|
tracker_config["extraction"] = {
|
||||||
|
"enabled": True,
|
||||||
|
"output_root": "extracted",
|
||||||
|
"format": "jsonl",
|
||||||
|
"sources": [{"path": "history/alarms.sqlite"}],
|
||||||
|
}
|
||||||
|
tracker_config["files"]["exclude"].append("**/*.sqlite")
|
||||||
|
return tracker_config
|
||||||
|
|
||||||
|
def test_first_run_commits_artifacts(
|
||||||
|
self, git_repo: Path, tracker_config: dict, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
self._make_sqlite(git_repo)
|
||||||
|
config = self._config_with_extraction(tracker_config)
|
||||||
|
outcome = _run(config, tmp_path)
|
||||||
|
# artifacts only: raw DB excluded, no other changes
|
||||||
|
assert outcome.change_count == 1
|
||||||
|
assert outcome.commit_hash is not None
|
||||||
|
body = _git_out("log", "-1", "--format=%b", cwd=git_repo)
|
||||||
|
assert "extracted/alarms/alarms.jsonl" in body
|
||||||
|
tracked = _git_out("ls-files", "-z", cwd=git_repo)
|
||||||
|
assert "alarms.sqlite" not in tracked # raw DB excluded
|
||||||
|
content = (git_repo / "extracted" / "alarms" / "alarms.jsonl").read_text(
|
||||||
|
encoding="utf-8"
|
||||||
|
)
|
||||||
|
assert '{"id":1,"text":"alpha"}' in content
|
||||||
|
|
||||||
|
def test_unchanged_db_no_second_commit(
|
||||||
|
self, git_repo: Path, tracker_config: dict, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
self._make_sqlite(git_repo)
|
||||||
|
config = self._config_with_extraction(tracker_config)
|
||||||
|
_run(config, tmp_path)
|
||||||
|
outcome = _run(config, tmp_path)
|
||||||
|
assert outcome.change_count == 0 # FR-031: byte-identical artifacts
|
||||||
|
assert outcome.commit_hash is None
|
||||||
|
|
||||||
|
def test_changed_row_produces_artifact_commit(
|
||||||
|
self, git_repo: Path, tracker_config: dict, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
import sqlite3
|
||||||
|
|
||||||
|
db = self._make_sqlite(git_repo)
|
||||||
|
config = self._config_with_extraction(tracker_config)
|
||||||
|
_run(config, tmp_path)
|
||||||
|
|
||||||
|
connection = sqlite3.connect(db)
|
||||||
|
connection.execute("INSERT INTO alarms VALUES (2, 'beta')")
|
||||||
|
connection.commit()
|
||||||
|
connection.close()
|
||||||
|
|
||||||
|
outcome = _run(config, tmp_path)
|
||||||
|
assert outcome.change_count == 1
|
||||||
|
body = _git_out("log", "-1", "--format=%b", cwd=git_repo)
|
||||||
|
assert "M ; extracted/alarms/alarms.jsonl" in body
|
||||||
|
|
||||||
|
def test_failed_source_skipped_with_warning_others_extracted(
|
||||||
|
self, git_repo: Path, tracker_config: dict, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
"""Section 45.7: per-source isolation - one bad file, others proceed."""
|
||||||
|
self._make_sqlite(git_repo)
|
||||||
|
# A second, non-database source alongside the valid one
|
||||||
|
(git_repo / "history" / "impostor.db").write_text(
|
||||||
|
"definitely not a database", encoding="utf-8"
|
||||||
|
)
|
||||||
|
config = self._config_with_extraction(tracker_config)
|
||||||
|
config["extraction"]["sources"].append({"path": "history/impostor.db"})
|
||||||
|
config["files"]["exclude"].append("history/impostor.db") # raw file not tracked
|
||||||
|
|
||||||
|
outcome = _run(config, tmp_path)
|
||||||
|
|
||||||
|
# The valid source was still extracted and committed
|
||||||
|
assert outcome.change_count == 1
|
||||||
|
assert outcome.commit_hash is not None
|
||||||
|
body = _git_out("log", "-1", "--format=%b", cwd=git_repo)
|
||||||
|
assert "extracted/alarms/alarms.jsonl" in body
|
||||||
|
# Nothing was extracted for the bad source
|
||||||
|
assert not (git_repo / "extracted" / "impostor").exists()
|
||||||
|
|
||||||
|
def test_non_sqlite_file_not_committed_as_artifact(
|
||||||
|
self, git_repo: Path, tracker_config: dict, tmp_path: Path
|
||||||
|
) -> None:
|
||||||
|
"""A non-database source is skipped, never dumped or committed."""
|
||||||
|
history = git_repo / "history"
|
||||||
|
history.mkdir(exist_ok=True)
|
||||||
|
(git_repo / "history" / "notadb.sqlite").write_text(
|
||||||
|
"plain text wearing a database name", encoding="utf-8"
|
||||||
|
)
|
||||||
|
config = self._config_with_extraction(tracker_config)
|
||||||
|
config["extraction"]["sources"] = [{"path": "history/notadb.sqlite"}]
|
||||||
|
|
||||||
|
outcome = _run(config, tmp_path)
|
||||||
|
assert outcome.change_count == 0 # nothing extracted, nothing committed
|
||||||
|
assert not (git_repo / "extracted" / "notadb").exists()
|
||||||
|
|
||||||
|
def test_dry_run_skips_refresh(
|
||||||
|
self, git_repo: Path, tracker_config: dict, tmp_path: Path, caplog
|
||||||
|
) -> None:
|
||||||
|
self._make_sqlite(git_repo)
|
||||||
|
config = self._config_with_extraction(tracker_config)
|
||||||
|
before = _git_out("status", "--porcelain", "-z", cwd=git_repo)
|
||||||
|
outcome = _run(config, tmp_path, dry_run=True)
|
||||||
|
assert outcome.dry_run
|
||||||
|
# Artifacts never created: the refresh is the step that writes them
|
||||||
|
assert (git_repo / "extracted").exists() is False # tree untouched
|
||||||
|
after = _git_out("status", "--porcelain", "-z", cwd=git_repo)
|
||||||
|
assert before == after
|
||||||
|
|
||||||
|
|
||||||
class TestSafetyAborts:
|
class TestSafetyAborts:
|
||||||
def test_pre_existing_staged_changes_abort(
|
def test_pre_existing_staged_changes_abort(
|
||||||
self, git_repo: Path, tracker_config: dict, tmp_path: Path
|
self, git_repo: Path, tracker_config: dict, tmp_path: Path
|
||||||
|
|||||||
@@ -0,0 +1,185 @@
|
|||||||
|
"""Unit tests for the SQLite artifact extractor (Section 45).
|
||||||
|
|
||||||
|
Covers the Section 41 extraction test list: artifacts produced per
|
||||||
|
table, byte-identical output for unchanged content, deterministic
|
||||||
|
ordering and serialization, stale removal, failure policy, and
|
||||||
|
read-only source access.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import sqlite3
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from git_auto_tracker.errors import CollectionError
|
||||||
|
from git_auto_tracker.extraction.base import (
|
||||||
|
ExtractionError,
|
||||||
|
ExtractionSource,
|
||||||
|
artifact_filename,
|
||||||
|
artifact_folder,
|
||||||
|
to_jsonl,
|
||||||
|
write_if_changed,
|
||||||
|
)
|
||||||
|
from git_auto_tracker.extraction.sqlite import SqliteExtractor, extractor_for
|
||||||
|
|
||||||
|
|
||||||
|
def _make_db(path: Path, tables: dict[str, list[tuple[str, ...]]]) -> None:
|
||||||
|
"""Create the fixture database; idempotent for repeat extraction runs."""
|
||||||
|
connection = sqlite3.connect(path)
|
||||||
|
try:
|
||||||
|
for ddl in (
|
||||||
|
"CREATE TABLE IF NOT EXISTS alarms (id INTEGER PRIMARY KEY, text TEXT)",
|
||||||
|
"CREATE TABLE IF NOT EXISTS events "
|
||||||
|
"(id INTEGER PRIMARY KEY, source TEXT, value REAL)",
|
||||||
|
):
|
||||||
|
if ddl.split()[5] in tables:
|
||||||
|
connection.execute(ddl)
|
||||||
|
for table, rows in tables.items():
|
||||||
|
for row in rows:
|
||||||
|
connection.execute(
|
||||||
|
f'INSERT OR IGNORE INTO "{table}" VALUES ({",".join("?" * len(row))})',
|
||||||
|
row,
|
||||||
|
)
|
||||||
|
connection.commit()
|
||||||
|
finally:
|
||||||
|
connection.close()
|
||||||
|
|
||||||
|
|
||||||
|
def _read(path: Path) -> str:
|
||||||
|
return path.read_text(encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
class TestHelpers:
|
||||||
|
def test_artifact_folder_named_after_database(self) -> None:
|
||||||
|
source = ExtractionSource(path="history/alarms.sqlite")
|
||||||
|
assert artifact_folder(source) == "alarms"
|
||||||
|
|
||||||
|
def test_artifact_filename_sanitized(self) -> None:
|
||||||
|
assert artifact_filename("alarms") == "alarms.jsonl"
|
||||||
|
assert artifact_filename("weird name") == "weird name.jsonl"
|
||||||
|
assert artifact_filename('bad"quote') == 'bad_quote.jsonl'
|
||||||
|
assert artifact_filename("com1") == "_com1.jsonl"
|
||||||
|
assert artifact_filename("") == "table.jsonl"
|
||||||
|
|
||||||
|
def test_to_jsonl_sorted_and_stable(self) -> None:
|
||||||
|
rows = [{"b": 1, "a": 2}, {"a": 3, "b": 4}]
|
||||||
|
assert to_jsonl(rows) == '{"a":2,"b":1}\n{"a":3,"b":4}\n'
|
||||||
|
|
||||||
|
def test_to_jsonl_empty(self) -> None:
|
||||||
|
assert to_jsonl([]) == ""
|
||||||
|
|
||||||
|
def test_write_if_changed_skips_identical(self, tmp_path: Path) -> None:
|
||||||
|
target = tmp_path / "x.jsonl"
|
||||||
|
assert write_if_changed(target, "a\n") is True
|
||||||
|
assert write_if_changed(target, "a\n") is False # identical, no write
|
||||||
|
assert write_if_changed(target, "b\n") is True
|
||||||
|
|
||||||
|
def test_extractor_for_unknown_name(self) -> None:
|
||||||
|
with pytest.raises(CollectionError, match="unknown extraction"):
|
||||||
|
extractor_for("csv")
|
||||||
|
|
||||||
|
|
||||||
|
class TestSqliteExtractor:
|
||||||
|
def _extract(self, tmp_path: Path, tables: tuple[str, ...] = ()) -> object:
|
||||||
|
repo = tmp_path / "repo"
|
||||||
|
repo.mkdir(exist_ok=True)
|
||||||
|
db = repo / "history"
|
||||||
|
db.mkdir(exist_ok=True)
|
||||||
|
_make_db(
|
||||||
|
repo / "history" / "alarms.sqlite",
|
||||||
|
{
|
||||||
|
"alarms": [(2, "beta"), (1, "alpha")],
|
||||||
|
"events": [(1, "s1", 1.5), (2, "s2", 2.5)],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
source = ExtractionSource(path="history/alarms.sqlite", tables=tables)
|
||||||
|
extractor = SqliteExtractor()
|
||||||
|
return extractor.extract(source, repo, repo / "extracted")
|
||||||
|
|
||||||
|
def test_all_user_tables_extracted(self, tmp_path: Path) -> None:
|
||||||
|
result = self._extract(tmp_path)
|
||||||
|
repo = tmp_path / "repo"
|
||||||
|
assert "extracted/alarms/alarms.jsonl" in result.artifacts
|
||||||
|
assert "extracted/alarms/events.jsonl" in result.artifacts
|
||||||
|
content = _read(repo / "extracted" / "alarms" / "alarms.jsonl")
|
||||||
|
# Ordered by primary key: 1 before 2 regardless of insertion order
|
||||||
|
assert content == '{"id":1,"text":"alpha"}\n{"id":2,"text":"beta"}\n'
|
||||||
|
|
||||||
|
def test_selected_tables_only(self, tmp_path: Path) -> None:
|
||||||
|
result = self._extract(tmp_path, tables=("alarms",))
|
||||||
|
assert "extracted/alarms/alarms.jsonl" in result.artifacts
|
||||||
|
assert not (tmp_path / "repo" / "extracted" / "alarms" / "events.jsonl").exists()
|
||||||
|
|
||||||
|
def test_unknown_table_rejected(self, tmp_path: Path) -> None:
|
||||||
|
with pytest.raises(ExtractionError, match="does not exist"):
|
||||||
|
self._extract(tmp_path, tables=("nope",))
|
||||||
|
|
||||||
|
def test_byte_identical_on_re_extraction(self, tmp_path: Path) -> None:
|
||||||
|
self._extract(tmp_path)
|
||||||
|
repo = tmp_path / "repo"
|
||||||
|
artifact = repo / "extracted" / "alarms" / "alarms.jsonl"
|
||||||
|
first = _read(artifact)
|
||||||
|
result2 = self._extract(tmp_path)
|
||||||
|
assert _read(artifact) == first # FR-031 byte-stable
|
||||||
|
assert result2.artifacts == [] # nothing rewritten
|
||||||
|
assert result2.unchanged is True
|
||||||
|
|
||||||
|
def test_changed_row_updates_artifact(self, tmp_path: Path) -> None:
|
||||||
|
self._extract(tmp_path)
|
||||||
|
repo = tmp_path / "repo"
|
||||||
|
connection = sqlite3.connect(repo / "history" / "alarms.sqlite")
|
||||||
|
connection.execute("INSERT INTO alarms VALUES (3, 'gamma')")
|
||||||
|
connection.commit()
|
||||||
|
connection.close()
|
||||||
|
result = self._extract(tmp_path)
|
||||||
|
assert "extracted/alarms/alarms.jsonl" in result.artifacts
|
||||||
|
assert '{"id":3,"text":"gamma"}' in _read(
|
||||||
|
repo / "extracted" / "alarms" / "alarms.jsonl"
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_stale_table_artifact_removed(self, tmp_path: Path) -> None:
|
||||||
|
self._extract(tmp_path)
|
||||||
|
repo = tmp_path / "repo"
|
||||||
|
connection = sqlite3.connect(repo / "history" / "alarms.sqlite")
|
||||||
|
connection.execute("DROP TABLE events")
|
||||||
|
connection.commit()
|
||||||
|
connection.close()
|
||||||
|
result = self._extract(tmp_path, tables=("alarms",))
|
||||||
|
# events artifact from the earlier run has no source table anymore
|
||||||
|
assert "extracted/alarms/events.jsonl" in result.removed
|
||||||
|
assert not (repo / "extracted" / "alarms" / "events.jsonl").exists()
|
||||||
|
|
||||||
|
def test_missing_source_raises(self, tmp_path: Path) -> None:
|
||||||
|
repo = tmp_path / "repo"
|
||||||
|
repo.mkdir(exist_ok=True)
|
||||||
|
source = ExtractionSource(path="history/absent.sqlite")
|
||||||
|
with pytest.raises(ExtractionError, match="not found"):
|
||||||
|
SqliteExtractor().extract(source, repo, repo / "extracted")
|
||||||
|
|
||||||
|
def test_source_never_modified(self, tmp_path: Path) -> None:
|
||||||
|
"""FR-034: extraction must not change the source database file."""
|
||||||
|
result = self._extract(tmp_path)
|
||||||
|
db = tmp_path / "repo" / "history" / "alarms.sqlite"
|
||||||
|
connection = sqlite3.connect(f"file:{db}?mode=ro", uri=True)
|
||||||
|
rows = list(connection.execute("SELECT COUNT(*) FROM alarms"))
|
||||||
|
connection.close()
|
||||||
|
assert rows[0][0] == 2
|
||||||
|
_ = result
|
||||||
|
|
||||||
|
def test_wal_database_extracted_consistently(self, tmp_path: Path) -> None:
|
||||||
|
"""Section 45.5: a WAL-mode source is snapshotted, not locked."""
|
||||||
|
repo = tmp_path / "repo"
|
||||||
|
repo.mkdir(exist_ok=True)
|
||||||
|
(repo / "history").mkdir(exist_ok=True)
|
||||||
|
db = repo / "history" / "wal.sqlite"
|
||||||
|
connection = sqlite3.connect(db)
|
||||||
|
connection.execute("PRAGMA journal_mode=WAL")
|
||||||
|
connection.execute("CREATE TABLE t (id INTEGER PRIMARY KEY, v TEXT)")
|
||||||
|
connection.execute("INSERT INTO t VALUES (1, 'x')")
|
||||||
|
connection.commit()
|
||||||
|
connection.close()
|
||||||
|
source = ExtractionSource(path="history/wal.sqlite")
|
||||||
|
result = SqliteExtractor().extract(source, repo, repo / "extracted")
|
||||||
|
assert "extracted/wal/t.jsonl" in result.artifacts
|
||||||
Reference in New Issue
Block a user