🚧 Add sqlite extraction
This commit is contained in:
Jonas Bengtsson
+146
-13
@@ -82,6 +82,12 @@ The application should be divided into independent functional components.
|
||||
|
|
||||
v
|
||||
+-------------------------+
|
||||
| Artifact Extractor |
|
||||
| (Section 45, optional) |
|
||||
+------------+------------+
|
||||
|
|
||||
v
|
||||
+-------------------------+
|
||||
| Change Detector |
|
||||
+------------+------------+
|
||||
|
|
||||
@@ -839,6 +845,19 @@ commit:
|
||||
# Must not occur inside paths, author identifiers or timestamps.
|
||||
file_separator: ";"
|
||||
|
||||
extraction:
|
||||
# Derived artifacts (Section 45): extract configured binary databases
|
||||
# into plain-text artifacts before change detection. Absent block or
|
||||
# enabled: false disables the feature entirely.
|
||||
enabled: true
|
||||
output_root: "extracted" # inside the repository
|
||||
format: "jsonl" # one JSON object per row
|
||||
sources:
|
||||
- path: "history/alarms.sqlite" # repository-relative source
|
||||
tables: # optional; default: all user tables
|
||||
- "alarms"
|
||||
- "events"
|
||||
|
||||
logging:
|
||||
level: INFO
|
||||
console: true
|
||||
@@ -1011,6 +1030,8 @@ modify the repository
|
||||
|
||||
The generated commit message should be displayed.
|
||||
|
||||
The derived-artifact refresh (Section 45) is skipped in dry-run mode because it writes to the working tree; dry-run analyses the artifacts from the previous collection and reports this in its output.
|
||||
|
||||
---
|
||||
|
||||
# 32. Security
|
||||
@@ -1238,6 +1259,26 @@ The system shall never commit its own operational artefacts, such as lock files,
|
||||
|
||||
The system shall correctly process file paths containing spaces, quotes and non-ASCII characters on all supported platforms.
|
||||
|
||||
## FR-030 – Derived artifact extraction
|
||||
|
||||
The system shall support extraction of configured binary database sources into plain-text derived artifacts inside the repository, refreshed before change detection on every collection.
|
||||
|
||||
## FR-031 – Deterministic artifacts
|
||||
|
||||
Derived artifacts shall be byte-identical between collections whenever the underlying content is unchanged.
|
||||
|
||||
## FR-032 – Extraction failure isolation
|
||||
|
||||
A source that cannot be read, is not a SQLite 3 database, or whose artifacts cannot be written shall be skipped with a logged warning while the remaining configured sources are still extracted.
|
||||
|
||||
## FR-033 – Stale artifact removal
|
||||
|
||||
Artifacts whose source table or database no longer exists shall be removed during extraction.
|
||||
|
||||
## FR-034 – Read-only source access
|
||||
|
||||
The system shall open source databases read-only and shall never write to them.
|
||||
|
||||
---
|
||||
|
||||
# 38. Suggested Python Package Structure
|
||||
@@ -1271,6 +1312,10 @@ git-auto-tracker/
|
||||
│ │ ├── models.py
|
||||
│ │ └── filters.py
|
||||
│ │
|
||||
│ ├── extraction/
|
||||
│ │ ├── base.py
|
||||
│ │ └── sqlite.py
|
||||
│ │
|
||||
│ ├── authors/
|
||||
│ │ ├── base.py
|
||||
│ │ ├── resolver.py
|
||||
@@ -1336,21 +1381,22 @@ A normal execution shall approximately perform the following sequence:
|
||||
3. Verify Git availability
|
||||
4. Acquire repository lock
|
||||
5. Validate repository, branch and index state
|
||||
6. Determine current repository status
|
||||
7. Apply inclusion/exclusion and security rules
|
||||
8. Create ChangeSet
|
||||
9. Resolve authors for ADDED files
|
||||
10. Resolve authors for MODIFIED files
|
||||
11. Generate commit message
|
||||
12. Stage selected changes
|
||||
13. Verify staged change set
|
||||
14. Create commit
|
||||
15. Record commit hash and write audit result
|
||||
16. Release lock
|
||||
17. Exit
|
||||
6. Refresh derived artifacts (Section 45)
|
||||
7. Determine current repository status
|
||||
8. Apply inclusion/exclusion and security rules
|
||||
9. Create ChangeSet
|
||||
10. Resolve authors for ADDED files
|
||||
11. Resolve authors for MODIFIED files
|
||||
12. Generate commit message
|
||||
13. Stage selected changes
|
||||
14. Verify staged change set
|
||||
15. Create commit
|
||||
16. Record commit hash and write audit result
|
||||
17. Release lock
|
||||
18. Exit
|
||||
```
|
||||
|
||||
If no changes exist, execution proceeds directly from step 8 to audit logging and termination.
|
||||
If no changes exist, execution proceeds directly from step 9 to audit logging and termination.
|
||||
|
||||
If any step from staging onwards fails, the index shall be restored to its pre-execution state before the lock is released.
|
||||
|
||||
@@ -1394,6 +1440,15 @@ Tests should cover at minimum:
|
||||
* excluded file;
|
||||
* forbidden file.
|
||||
|
||||
### Extraction (Section 45)
|
||||
|
||||
* enabled extraction produces an artifact file for every configured table;
|
||||
* unchanged database content produces byte-identical artifacts (no spurious commit);
|
||||
* a modified row appears as an artifact change in the commit;
|
||||
* a removed table or database removes its artifact files;
|
||||
* an unreadable or locked source database aborts with exit code 6;
|
||||
* source databases are opened read-only.
|
||||
|
||||
### Safety and edge cases
|
||||
|
||||
* pre-existing staged changes in the index;
|
||||
@@ -1494,6 +1549,82 @@ The initial implementation should deliberately keep **SCADA-specific change attr
|
||||
|
||||
---
|
||||
|
||||
# 45. Derived Artifacts (Content Extraction)
|
||||
|
||||
This feature is prioritized for the next release.
|
||||
|
||||
## 45.1 Purpose
|
||||
|
||||
Some SCADA engineering data is stored in binary databases (e.g. SQLite). Git cannot show readable history for such files: every commit stores the whole binary blob and a diff reports only "binary files differ". To make this content auditable, the tracker shall extract selected database content into plain-text derived artifacts inside the repository before change detection (Section 40, step 6). The artifacts are then versioned like any other file, and Git history shows precise, row-level changes.
|
||||
|
||||
Source databases are identified by content (the SQLite file header read during extraction), not by file-name extension.
|
||||
|
||||
The implementation shall follow the same pluggable-interface principle as author resolution (Section 44): an `ArtifactExtractor` abstraction with a SQLite implementation as the first plugin, so further formats can be added independently.
|
||||
|
||||
## 45.2 Output layout
|
||||
|
||||
Extraction writes into a configurable output root inside the repository (default: `extracted/`). One sub-folder is created per configured source database, named after the database file. Within each sub-folder, one file is written per extracted table.
|
||||
|
||||
```text
|
||||
extracted/
|
||||
└── alarms.sqlite/
|
||||
├── alarms.jsonl
|
||||
└── events.jsonl
|
||||
```
|
||||
|
||||
This layout gives each table its own artifact file, so Git diffs are limited to the tables that actually changed, and table names cannot collide between databases.
|
||||
|
||||
## 45.3 Configuration
|
||||
|
||||
```yaml
|
||||
extraction:
|
||||
enabled: true
|
||||
output_root: "extracted" # relative to the repository root
|
||||
format: "jsonl" # one JSON object per row
|
||||
sources:
|
||||
- path: "history/alarms.sqlite" # repository-relative source
|
||||
tables: # optional; default: all user tables
|
||||
- "alarms"
|
||||
- "events"
|
||||
```
|
||||
|
||||
An absent `extraction:` block, or `enabled: false`, disables the feature entirely; collection then proceeds without touching any database.
|
||||
|
||||
## 45.4 Artifact format
|
||||
|
||||
The default format shall be JSON Lines (`jsonl`): one JSON object per row with sorted keys and a stable encoding, so a changed row produces a minimal line-level diff. Rows shall be ordered deterministically by primary key (falling back to `rowid`), so unchanged content always produces a byte-identical file.
|
||||
|
||||
Artifact content shall not contain run-dependent values such as extraction timestamps: an artifact must be byte-identical whenever the underlying content is unchanged (FR-031). Table names that are not valid file names on the target platform shall be sanitized deterministically for artifact file names (FR-029).
|
||||
|
||||
## 45.5 Source access
|
||||
|
||||
Source databases shall be opened read-only; the tracker shall never write to them (FR-034). Because the SCADA platform may keep databases in WAL mode while running, extraction shall operate on a snapshot: the database file and its WAL sidecar shall be copied to a temporary location before reading, so a partially written transaction is never captured (see also Section 20).
|
||||
|
||||
## 45.6 Stale artifacts
|
||||
|
||||
Extraction shall also remove artifacts that no longer have a source:
|
||||
|
||||
* a table removed from the database → its artifact file is deleted;
|
||||
* a source database removed from the configuration or the file system → its sub-folder is deleted.
|
||||
|
||||
Without this rule, stale artifacts would look like current data in Git history (FR-033).
|
||||
|
||||
## 45.7 Failure policy
|
||||
|
||||
Sources are isolated from each other: a source that cannot be read, is not a SQLite 3 database, or whose artifacts cannot be written, is reported as a warning and skipped; extraction then continues with the remaining configured sources. One bad file shall never block extraction of the others. When every configured source fails, the collection proceeds with the artifacts from the previous collection and a summary warning lists all skipped sources. Source validity is established during extraction by the SQLite engine itself - a non-database file raises `sqlite3.DatabaseError: file is not a database` - rather than by file-name extension: databases with vendor-specific or missing extensions are accepted (Section 45.5).
|
||||
|
||||
An extraction failure is not silent, however: every skipped source is logged, and a source that fails persistently must be visible in monitoring through the log (Section 22).
|
||||
|
||||
## 45.8 Raw source files
|
||||
|
||||
The configured binary source files should be excluded from direct tracking (`files.exclude`, e.g. `**/*.sqlite`). The repository then stores the readable artifacts instead of the opaque blob; tracking both duplicates every change and lets the binary churn dominate history. Direct tracking of the raw database remains possible where a deployment wants the authoritative blob as well.
|
||||
|
||||
## 45.9 Attribution
|
||||
|
||||
Generated artifacts have no human last-modifier of their own. Author resolution for artifact files shall therefore report `UNKNOWN` unless an extractor emits sidecar metadata (Section 10) that the resolver chain can consume, for example a `last_modified_by` column extracted from the database content.
|
||||
|
||||
---
|
||||
|
||||
# Appendix A – Glossary
|
||||
|
||||
| Term | Definition |
|
||||
@@ -1508,3 +1639,5 @@ The initial implementation should deliberately keep **SCADA-specific change attr
|
||||
| Forbidden file | A file matching `security.forbidden_patterns`; never committed by default |
|
||||
| Tracker-Run | Unique execution identifier linking logs, commits and monitoring (Section 35) |
|
||||
| Staged verification | Comparison of the actual staged change set against the expected change set before commit |
|
||||
| Derived artifact | Plain-text file generated from a binary source (e.g. a SQLite table) and versioned in the repository (Section 45) |
|
||||
| Artifact extractor | Pluggable component that refreshes derived artifacts from configured sources before change detection |
|
||||
|
||||
Reference in New Issue
Block a user