jobe/git-automatic-tracker-py

Fork 0

Files

T

Jonas Bengtsson 3bbfd6c161 🚧 Add export feature and refined exclusion info

2026-09-13 12:04:42 +02:00

45 KiB

Raw Permalink Blame History

Functional Specification

SCADA Unattended Change Tracker

1. Purpose

The purpose of the application is to provide automated version tracking of configuration and project files associated with a SCADA system.

The application shall periodically inspect a configured set of files or directories for changes and automatically record detected changes in a Git repository.

The resulting Git history shall provide an auditable record showing:

when changes were detected,
which files were added,
which files were modified,
which files were removed,
and, where applicable, who last modified each added or modified file.

The application is intended to operate without normal user interaction and shall be capable of running on Windows, Linux and macOS.

1.1 Conformance language

In this specification:

shall – the requirement is mandatory;
should – the requirement is recommended, but deviations are acceptable with documented justification;
may – the behaviour is optional.

2. Scope

The application shall:

monitor one or more configured SCADA project directories;
identify file-system changes relative to the current Git repository state;
determine whether files have been:
- added,
- modified,
- deleted,
- optionally renamed;
determine the last modifying user for added and modified files;
generate a structured Git commit message describing the detected changes;
stage the changes;
create a Git commit automatically;
maintain logs of application execution and errors;
support unattended scheduled execution;
operate consistently across supported operating systems.

The initial implementation does not need to:

interpret the semantic meaning of SCADA configuration changes;
restore files;
approve or reject changes;
provide a graphical user interface;
automatically push commits to a remote Git repository.

These functions may be added later.

3. General Architecture

The application should be divided into independent functional components.

+-------------------------+
| Scheduler / Invocation  |
+------------+------------+
             |
             v
+-------------------------+
| Application Controller  |
|  (config, lock, exit)   |
+------------+------------+
             |
             v
+-------------------------+
| Repository Inspector    |
+------------+------------+
             |
             v
+-------------------------+
| Artifact Extractor      |
| (Section 45, optional)  |
+------------+------------+
             |
             v
+-------------------------+
| Change Detector         |
+------------+------------+
             |
             v
+-------------------------+
| Security Validator      |
+------------+------------+
             |
             v
+-------------------------+
| Author Resolver (chain) |
+------------+------------+
             |
             v
+-------------------------+
| Commit Message Builder  |
+------------+------------+
             |
             v
+-------------------------+
| Git Commit Manager      |
+------------+------------+
             |
             v
+-------------------------+
| Logging / Audit         |
+-------------------------+

Logging / Audit is a cross-cutting concern: every component reports to it
directly rather than only through the final stage.

The components should be loosely coupled so that SCADA-specific metadata extraction can be implemented independently of Git handling.

4. Operating Modes

The application shall support at least two operating modes.

4.1 Run-once mode

Example:

git-auto-tracker run

The application performs one complete inspection and exits.

This mode is suitable for execution through:

Windows Task Scheduler,
cron,
systemd timers,
CI/CD systems,
Docker scheduled execution,
other external schedulers.

Run-once mode should be the primary operating mode because it simplifies reliability and cross-platform deployment.

4.2 Scheduled service mode

An optional internal scheduler may allow the application to remain running and execute according to configured times.

Example configuration:

schedule:
  enabled: true
  times:
    - "06:00"
    - "12:00"
    - "18:00"
    - "23:30"

The application should use local system time unless a timezone is explicitly configured.

Example (see also Section 26):

application:
  timezone: Europe/Stockholm

5. Repository Configuration

The application shall operate against an existing Git repository.

Example:

repository:
  path: "/path/to/scada/project"
  branch: "main"

The configured directory shall contain a valid Git repository.

Before attempting any modification, the application shall verify:

the directory exists;
the directory is accessible;
the directory is a Git repository;
the configured branch can be determined;
Git operations can be performed.

If validation fails, the execution shall stop without modifying the repository.

5.1 Repository and index state

Before collecting changes the application shall additionally verify that:

the repository index does not already contain staged changes;
the repository is not in an unfinished merge, rebase or cherry-pick state.

A non-empty index at startup indicates that a human or another process prepared changes outside the tracker. Because such changes carry no tracker attribution and may be incomplete, the default policy shall be to abort with a repository error (exit code 3, see Section 25). A configurable policy may allow such changes to be committed with UNKNOWN attribution. Staged changes left behind by an interrupted tracker execution are not special-cased; recovery requires manual review, which keeps unattended behaviour safe and predictable.

If the current branch differs from the configured branch, the application shall abort by default rather than switch branches, since switching modifies the working tree and is not an unattended-safe operation.

6. Change Detection

6.1 Change comparison

Changes shall be detected relative to the current Git HEAD.

The following change types shall be supported:

Change	Identifier
Added file	`ADDED`
Modified file	`MODIFIED`
Deleted file	`DELETED`
Renamed file	`RENAMED`

Rename detection may initially be treated as one deletion and one addition if native rename detection is not implemented.

Git does not report renames for unstaged working-tree changes; rename detection is only reliable after staging. The implementation should therefore either stage first and inspect the staged diff, or deliberately report renames as one deletion plus one addition.

6.2 Git status

The application should use Git itself as the authoritative mechanism for determining changed files.

The implementation should therefore use an equivalent of:

git status --porcelain -z

rather than independently comparing file timestamps.

This avoids false change detection caused by modification times alone.

The NUL-terminated (-z) output format shall be used for all Git output that contains file paths. Line-based parsing of the default porcelain output is unsafe because Git C-quotes filenames containing spaces, quotes or non-ASCII characters. All Git output shall be decoded as UTF-8, and commit messages shall be written as UTF-8.

7. File Inclusion and Exclusion

The application shall support configurable inclusion and exclusion rules.

Example:

files:
  include:
    - "**/*"

  exclude:
    - "**/*.tmp"
    - "**/*.log"
    - "**/~*"
    - ".git/**"
    - "runtime/**"
    - "cache/**"

Glob-style matching should be supported.

Files excluded by configuration shall neither be committed nor included in commit messages.

The application log shall record every excluded file together with the rule that matched it (e.g. matched exclude 'runtime/**') at DEBUG level. The INFO-level summary shall contain only the number of in-scope and excluded changes, so normal-run logs stay compact while investigations can switch on DEBUG (Section 22).

Files ignored by the repository's own .gitignore shall also be excluded. The tracker shall not override ignore rules unless explicitly configured to do so.

8. Last Modifier Identification

8.1 Requirement

For every:

added file;
modified file;

the application shall attempt to determine the person or account responsible for its most recent modification.

Example:

Modified:
  configuration/plant1.xml
  Last modified by: DOMAIN\jsmith

Deleted files do not require a last-modifier lookup because the original file may no longer be available. Resolvers may nevertheless attribute deletions when possible, for example through the SCADA audit log or the file's previous commit history. Attribution for deletions is permitted but not mandatory.

9. Author Resolution

Determining the last modifying user shall be implemented as a separate abstraction.

Conceptually:

class AuthorResolver:
    def resolve(file_path) -> AuthorResult:
        ...

A result should contain at least:

name
identifier
source
confidence
timestamp

Example:

{
  "name": "John Smith",
  "identifier": "DOMAIN\\jsmith",
  "source": "scada_metadata",
  "confidence": "authoritative",
  "timestamp": "2026-09-11T14:31:22+02:00"
}

The confidence field shall use one of the following levels:

Level	Meaning
`authoritative`	The source records the change actor in an authenticated way
`probable`	The source is reliable but not an authenticated change record
`weak`	Weak correlation, e.g. ownership or workstation inference
`unknown`	No identity could be determined

10. Author Resolver Strategies

The application shall support configurable resolver implementations.

Suggested resolver types are:

SCADA metadata resolver

Reads user information embedded in the SCADA project file.

This should be preferred when the SCADA platform records information such as:

last modified by;
engineering user;
operator;
change timestamp;
workstation;
project revision.

Companion metadata resolver

Reads metadata associated with the file.

For example:

pumpstation.xml
pumpstation.xml.metadata

or from a project database.

SCADA audit-log resolver

Looks up the changed file or object in the SCADA system's audit log.

This is likely to provide the strongest attribution where the SCADA system maintains authenticated engineering change records.

Operating-system resolver

An OS-specific resolver may be implemented if the environment provides authoritative file auditing.

File ownership alone shall not be interpreted as proof of the last modifying user.

For example, on Windows:

DOMAIN\SCADAService

could own the file even though another engineering user performed the modification.

Unknown resolver

If no reliable identity can be determined:

Last modified by: UNKNOWN

The application shall still create the commit unless configured otherwise.

11. Resolver Priority

Multiple author resolvers may be configured.

Example:

author_resolution:
  resolvers:
    - scada_metadata
    - scada_audit_log
    - sidecar_metadata

  minimum_confidence: probable
  fallback: unknown

The resolvers shall execute in configured priority order.

The first result whose confidence meets the configured minimum confidence (default: probable) shall be used. Results below the threshold shall be logged but not used. If no resolver produces a usable result, the configured fallback shall be used.

12. Change Data Model

Each detected change should internally be represented using a structure similar to:

ChangeRecord

path
change_type
previous_path
size
file_timestamp
author
author_source
author_timestamp

For example:

{
  "path": "Stations/PS101/config.xml",
  "change_type": "MODIFIED",
  "previous_path": null,
  "size": 34127,
  "file_timestamp": "2026-09-11T13:44:12+02:00",
  "author": "DOMAIN\\jsmith",
  "author_source": "scada_audit_log",
  "author_timestamp": "2026-09-11T13:43:58+02:00"
}

13. Commit Generation

If no relevant files have changed, the application shall:

record the execution in its log;
not create an empty Git commit;
return a successful execution result.

Example log:

2026-09-11 18:00:02 INFO No SCADA changes detected.

14. Git Staging

When changes are detected the application shall:

collect the complete change set;
resolve author information;
generate the commit message;
stage the relevant files;
verify the staged change set;
create the Git commit.

Files outside the configured scope shall not be staged.

The application should avoid blindly executing:

git add --all

unless the repository is exclusively dedicated to tracked SCADA data.

Staging shall instead use pathspec-limited operations that stage additions, modifications and deletions for the selected paths only.

Step 5 (verification) shall compare the actual staged change set against the expected change set using an operation equivalent to:

git diff --cached --name-status -z

If the staged change set does not match the expected change set, the commit shall be aborted and the index restored to its pre-execution state, so that the next execution starts from a well-defined state.

15. Commit Message Format

Commit messages shall be human-readable while retaining a predictable structure that can later be parsed automatically.

Each changed file shall occupy exactly one line, in the style of git diff --name-status. A single-letter change type is followed by the file path, the last modifying user and the filesystem last-modified time, with the parts joined by a configurable separator (see Section 26, commit.file_separator; default ;).

Line grammar:

A <sep> <path> <sep> <author> <sep> <modified-at>   added
M <sep> <path> <sep> <author> <sep> <modified-at>   modified
R <sep> <old-path> -> <new-path>                     renamed (bare)
D <sep> <path>                                      deleted (bare)

Recommended format:

SCADA automatic change capture - 2026-09-11 18:00:03+02:00

Summary:
  Added:    2
  Modified: 3
  Deleted:  1
  Renamed:  0

Changes:
  A ; Stations/PS101/pump2.xml ; DOMAIN\jsmith ; 2026-09-11 15:41:22+02:00
  A ; Stations/PS104/settings.xml ; DOMAIN\adoe ; 2026-09-11 16:02:14+02:00
  M ; Stations/PS101/project.xml ; DOMAIN\jsmith ; 2026-09-11 15:42:09+02:00
  M ; Stations/PS103/alarm.xml ; DOMAIN\adoe ; 2026-09-11 16:13:54+02:00
  M ; System/global.xml ; UNKNOWN ; 2026-09-11 16:20:01+02:00
  D ; Stations/PS027/old-config.xml

Deleted files are listed without author or time information (FR-013). Renames use git's old -> new arrow notation and stay bare: a pure rename preserves the original modification time, so a timestamp would be misleading. The time column is the filesystem last-modified time (Section 12: file_timestamp) rendered in local time with an explicit UTC offset, matching the log timestamp style (Section 22).

The one-line format keeps the message proportional to the number of changed files, which matters for large collections.

16. Commit Subject

The first commit line should be concise.

Recommended format:

SCADA automatic change capture - YYYY-MM-DD HH:MM:SS

Alternative:

SCADA: 2 added, 3 modified, 1 deleted

Exactly one subject format shall be used consistently; the timestamp format shall be the default. The detailed information shall be placed in the commit body.

17. Git Commit Identity

The Git commit author shall represent the automated tracking system rather than the person who changed an individual file.

Example:

git:
  user_name: "SCADA Change Tracker"
  user_email: "scada-change-tracker@localhost"

The engineers responsible for individual changes are instead recorded within the commit message.

This avoids incorrectly assigning a single engineer as the author of a commit containing changes made by several people.

18. Multiple Authors

One execution may contain changes made by multiple individuals.

Example:

Changes:
  M ; plant1.xml ; DOMAIN\jsmith ; 2026-09-11 15:42:09+02:00
  M ; plant2.xml ; DOMAIN\adoe ; 2026-09-11 16:02:14+02:00

The application shall therefore store attribution per file rather than per commit.

19. Atomic Processing

A commit shall represent one complete collection cycle.

Conceptually:

Start collection
     ↓
Detect all changes
     ↓
Resolve metadata
     ↓
Build complete change record
     ↓
Stage changes
     ↓
Validate staged state
     ↓
Commit
     ↓
Finish collection

The application shall not create multiple commits for one collection unless explicitly configured to do so.

20. Concurrent Changes

Files may be modified while the collection is taking place.

The application should therefore detect this situation.

For each file the application may record attributes before and after processing:

size
modification timestamp
optional content hash

If the file changes while being processed, the application shall either:

restart analysis for that file; or
abort the complete commit and retry during the next execution.

The default behaviour should favour data integrity over forcing a commit.

21. Repository Lock

Only one tracker execution shall be allowed against a repository at the same time.

The application shall implement a repository-level execution lock.

The lock file shall be stored outside the tracked repository, for example as a sibling of the repository directory or in a configured state directory. A lock file placed inside the repository would itself appear as an untracked change and be committed by the tracker.

For example:

<repository-path>.git-auto-tracker.lock

The lock shall include information such as:

process_id
hostname
start_time

Stale locks shall be detected safely. A lock may be considered stale when the recorded process no longer exists (verifiable only on the same host) or when the lock is older than a configurable age threshold, which is required for locks left by other hosts. Removing a stale lock shall be logged as a warning.

22. Logging

Application activity shall be logged independently of Git.

Minimum log information shall include:

execution start
execution completion
repository
number of detected changes
files processed
author resolution result
excluded files and the rule that matched them (DEBUG level, Section 7)
Git operations
warnings
errors
commit hash
execution duration

Example:

2026-09-11 18:00:01 INFO Starting SCADA change collection
2026-09-11 18:00:01 INFO Repository: D:\SCADA\Project
2026-09-11 18:00:02 INFO Detected 6 changes
2026-09-11 18:00:02 INFO Author resolved: Stations/PS101/project.xml -> DOMAIN\jsmith
2026-09-11 18:00:03 INFO Commit created: 93ab4e21
2026-09-11 18:00:03 INFO Collection completed

23. Log Rotation

File-based logging shall support rotation.

Example:

logging:
  file: "logs/git-auto-tracker.log"
  max_size_mb: 20
  backup_count: 10

Console logging shall also be supported.

Relative log paths shall be resolved relative to the configuration file location. The log file shall not be placed inside the tracked repository unless explicitly excluded; otherwise the tracker's own log would be detected as a change and committed.

24. Error Handling

Errors shall be classified.

Recommended categories:

CONFIGURATION_ERROR
REPOSITORY_ERROR
FILE_ACCESS_ERROR
AUTHOR_RESOLUTION_ERROR
GIT_ERROR
LOCK_ERROR
SECURITY_ERROR
INTERNAL_ERROR

An author-resolution error shall normally not prevent a commit.

For example:

Author: UNKNOWN
Reason: Audit log unavailable

A Git or repository integrity error shall prevent the commit.

A security violation (see Section 33) shall prevent the commit and shall be reported through the SECURITY_ERROR category and the dedicated exit code.

25. Exit Codes

Suggested process exit codes:

Code	Meaning
0	Successful execution
1	General application failure
2	Configuration error
3	Repository error
4	Git operation failure
5	Repository locked
6	Change collection failed
7	Security violation

Exit code 0 shall be returned both when changes were committed and when no changes were detected; the execution log distinguishes the two cases.

This makes the application suitable for external monitoring.

26. Configuration

Configuration should primarily be stored in YAML.

Example:

application:
  timezone: Europe/Stockholm

repository:
  path: "D:/SCADA/Project"
  branch: main

git:
  user_name: "SCADA Change Tracker"
  user_email: "git-auto-tracker@localhost"

files:
  include:
    - "**/*"

  exclude:
    - "**/*.tmp"
    - "**/*.log"
    - ".git/**"

author_resolution:
  resolvers:
    - scada_metadata
    - scada_audit_log

  minimum_confidence: probable
  fallback: unknown

commit:
  # Separator between the parts of a one-line file entry (Section 15).
  # Must not occur inside paths, author identifiers or timestamps.
  file_separator: ";"

extraction:
  # Derived artifacts (Section 45): extract configured binary databases
  # into plain-text artifacts before change detection. Absent block or
  # enabled: false disables the feature entirely.
  enabled: true
  output_root: "extracted"     # inside the repository
  format: "jsonl"             # one JSON object per row
  sources:
    - path: "history/alarms.sqlite"   # repository-relative source
      tables:                          # optional; default: all user tables
        - "alarms"
        - "events"

logging:
  level: INFO
  console: true
  file: "logs/git-auto-tracker.log"

Environment variables should be capable of overriding sensitive or deployment-specific settings, using a consistent naming convention such as:

GIT_AUTO_TRACKER_<SECTION>__<KEY>

Example:

GIT_AUTO_TRACKER_REPOSITORY__PATH=D:/SCADA/Project

The prohibition of empty commits (FR-014) is unconditional and therefore deliberately not configurable.

27. Cross-Platform Requirements

The project shall support:

Windows,
Linux,
macOS.

The implementation shall therefore avoid assumptions about:

path separators;
shell syntax;
drive letters;
filesystem ownership models;
username formats;
case sensitivity.

Python's:

pathlib

shall be preferred for filesystem paths.

Shell command strings should not be constructed manually where avoidable.

For subprocess execution, argument arrays should be used.

Conceptually:

subprocess.run(
    ["git", "status", "--porcelain"],
    ...
)

rather than:

os.system("git status --porcelain")

28. Git Integration

The Git integration may use either:

the Git CLI through subprocess; or
a Python Git library.

Using the Git CLI is recommended initially because it closely follows native Git behaviour and makes troubleshooting straightforward.

The Git executable shall therefore be treated as an external prerequisite. A minimum Git version of 2.20 shall be assumed.

At startup the application shall verify:

git --version

and fail gracefully if Git is unavailable.

29. Python Version

Recommended minimum:

Python >= 3.12

The application should avoid platform-specific Python dependencies unless they are isolated behind optional adapters.

30. Command-Line Interface

Suggested commands:

git-auto-tracker run
git-auto-tracker status
git-auto-tracker validate
git-auto-tracker config

Global options shall include:

--config PATH      Path to the configuration file (default: git-auto-tracker.yaml)
--log-level LEVEL  Override the configured log level
--version          Print the application version and exit

`run`

Perform change collection and create a commit if necessary.

`status`

Display currently detected changes without staging or committing them.

`validate`

Validate:

configuration,
Git installation,
repository,
file access,
configured author resolvers.

`config`

Display the effective configuration with secrets redacted.

31. Dry-Run Mode

The application shall support:

git-auto-tracker run --dry-run

Dry-run mode shall perform:

change detection
author resolution
commit message generation
validation

but shall not:

stage files
create commits
push data
modify the repository

The generated commit message should be displayed.

The derived-artifact refresh (Section 45) is skipped in dry-run mode because it writes to the working tree; dry-run analyses the artifacts from the previous collection and reports this in its output.

32. Security

The application shall follow least-privilege principles.

The execution account should require only:

read access to source metadata;
read/write access to the tracked repository;
access required to query SCADA audit information.

SCADA engineering credentials shall not be embedded in source code.

Secrets shall be provided through:

environment variables;
operating-system secret stores;
external secret-management systems.

Secrets must never appear in Git commit messages or application logs.

33. Sensitive File Protection

The application should support rules preventing accidental versioning of sensitive data.

Example:

security:
  forbidden_patterns:
    - "**/*.key"
    - "**/*.pem"
    - "**/secrets.*"
    - "**/credentials.*"

If such a file is detected, the default behaviour shall be to abort the commit, log a security warning, and exit with the dedicated security exit code (see Section 25). A configurable alternative may skip the forbidden file and commit the remaining changes, but skipping shall never be the default.

34. Remote Repository

Remote synchronization shall be optional.

Future configuration could contain:

remote:
  enabled: true
  name: origin
  push_after_commit: true

A failed push shall not invalidate the locally created commit.

The application shall distinguish:

COMMIT_SUCCESS
PUSH_FAILURE

35. Audit Information

Each collection should internally generate an execution identifier. A ULID, or an equivalent lexicographically sortable and timezone-independent identifier, is recommended.

For example:

01J7K8HYV84ZKFH1J4T9R8DZQN

The identifier may be included in the commit:

Tracker-Run: 01J7K8HYV84ZKFH1J4T9R8DZQN

This provides traceability between:

application logs;
Git commits;
external monitoring;
SCADA audit records.

36. Structured Commit Metadata

To simplify future machine processing, the end of the commit message may include structured trailers.

Example:

Tracker-Version: 1.0.0
Tracker-Run: 01J7K8HYV84ZKFH1J4T9R8DZQN
Tracker-Host: SCADA-SRV-01
Tracker-Changes: 6

This enables later processing without parsing the human-readable sections. The trailer block shall follow the Git trailer convention so that standard tooling such as git interpret-trailers can extract it.

37. Functional Requirements

FR-001 – Repository inspection

The system shall inspect a configured Git repository for uncommitted file changes.

FR-002 – Added files

The system shall identify newly added files.

FR-003 – Modified files

The system shall identify modified files.

FR-004 – Deleted files

The system shall identify deleted files.

FR-005 – Rename detection

The system should identify renamed files where practical.

FR-006 – Author identification

The system shall attempt to determine the last modifying identity for every added and modified file.

FR-007 – Unknown author

The system shall support an UNKNOWN author state.

FR-008 – Per-file attribution

Author information shall be associated with individual files rather than the complete commit.

FR-009 – Commit generation

The system shall automatically generate a Git commit for detected changes.

FR-010 – Commit summary

The commit message shall contain counts for each change type.

FR-011 – Commit details

The commit message shall list every affected file.

FR-012 – Attribution

Added and modified files shall contain last-modifier information in the commit message.

FR-013 – Deleted files

Deleted files shall be listed without requiring modifier information.

FR-014 – Empty executions

The system shall not create an empty commit when no changes are detected.

FR-015 – Logging

Every execution shall produce an audit log.

FR-016 – Cross-platform support

The system shall operate on Windows, Linux and macOS.

FR-017 – Dry run

The system shall support change analysis without repository modification.

FR-018 – Configuration

Application behaviour shall be configurable without modifying source code.

FR-019 – File filters

The system shall support file inclusion and exclusion patterns.

FR-020 – Concurrency protection

The system shall prevent simultaneous tracker executions against the same repository.

FR-021 – Error isolation

Failure to identify an author shall not normally prevent an otherwise valid change from being committed.

FR-022 – Repository protection

Git, filesystem or repository integrity errors shall prevent commit creation.

FR-023 – Atomic collection

A normal execution shall produce at most one commit representing the complete detected change set.

FR-024 – Sensitive-data protection

Configured forbidden files shall not automatically be committed.

FR-025 – Concurrent modification protection

The system shall detect files modified during collection and abort or retry according to a configured policy. The default policy shall abort the commit.

FR-026 – Index-state validation

The system shall refuse to execute when the repository index already contains staged changes or the repository is in an unfinished merge/rebase state.

FR-027 – Security abort

The system shall abort the commit and exit with a dedicated exit code when a forbidden file is detected.

FR-028 – Self-exclusion

The system shall never commit its own operational artefacts, such as lock files, logs and state files.

FR-029 – Filename safety

The system shall correctly process file paths containing spaces, quotes and non-ASCII characters on all supported platforms.

FR-030 – Derived artifact extraction

The system shall support extraction of configured binary database sources into plain-text derived artifacts inside the repository, refreshed before change detection on every collection.

FR-031 – Deterministic artifacts

Derived artifacts shall be byte-identical between collections whenever the underlying content is unchanged.

FR-032 – Extraction failure isolation

A source that cannot be read, is not a SQLite 3 database, or whose artifacts cannot be written shall be skipped with a logged warning while the remaining configured sources are still extracted.

FR-033 – Stale artifact removal

Artifacts whose source table or database no longer exists shall be removed during extraction.

FR-034 – Read-only source access

The system shall open source databases read-only and shall never write to them.

38. Suggested Python Package Structure

git-auto-tracker/
│
├── pyproject.toml
├── README.md
├── LICENSE
│
├── config/
│   └── example.yaml
│
├── src/
│   └── git_auto_tracker/
│       │
│       ├── __init__.py
│       ├── __main__.py
│       ├── cli.py
│       ├── config.py
│       ├── application.py
│       │
│       ├── git/
│       │   ├── repository.py
│       │   ├── status.py
│       │   └── commit.py
│       │
│       ├── changes/
│       │   ├── detector.py
│       │   ├── models.py
│       │   └── filters.py
│       │
│       ├── extraction/
│       │   ├── base.py
│       │   └── sqlite.py
│       │
│       ├── authors/
│       │   ├── base.py
│       │   ├── resolver.py
│       │   ├── unknown.py
│       │   └── scada/
│       │       └── ...
│       │
│       ├── commits/
│       │   └── message_builder.py
│       │
│       ├── security/
│       │   └── validator.py
│       │
│       ├── locking/
│       │   └── repository_lock.py
│       │
│       └── logging/
│           └── setup.py
│
└── tests/
    ├── unit/
    ├── integration/
    └── fixtures/

39. Core Domain Objects

The project should initially define the following central objects:

Repository
ChangeSet
ChangeRecord
AuthorIdentity
AuthorResolution
TrackerRun
CommitResult

A ChangeSet conceptually contains:

ChangeSet
 ├── added[]
 ├── modified[]
 ├── deleted[]
 └── renamed[]

Each item is represented by a ChangeRecord.

40. Processing Sequence

A normal execution shall approximately perform the following sequence:

1.  Load configuration
2.  Validate configuration
3.  Verify Git availability
4.  Acquire repository lock
5.  Validate repository, branch and index state
6.  Refresh derived artifacts (Section 45)
7.  Determine current repository status
8.  Apply inclusion/exclusion and security rules
9.  Create ChangeSet
10. Resolve authors for ADDED files
11. Resolve authors for MODIFIED files
12. Generate commit message
13. Stage selected changes
14. Verify staged change set
15. Create commit
16. Record commit hash and write audit result
17. Release lock
18. Exit

If no changes exist, execution proceeds directly from step 9 to audit logging and termination.

If any step from staging onwards fails, the index shall be restored to its pre-execution state before the lock is released.

41. Testing Requirements

Tests should cover at minimum:

Change detection

new file;
modified file;
deleted file;
rename;
multiple simultaneous changes;
Unicode filenames;
filenames containing spaces.

Git

clean repository;
dirty repository;
invalid repository;
failed commit;
unavailable Git executable.

Author resolution

one successful resolver;
multiple resolver fallback;
unknown user;
unavailable audit source;
malformed SCADA metadata.

File handling

inaccessible file;
file removed during collection;
file modified during collection;
excluded file;
forbidden file.

Extraction (Section 45)

enabled extraction produces an artifact file for every configured table;
unchanged database content produces byte-identical artifacts (no spurious commit);
a modified row appears as an artifact change in the commit;
a removed table or database removes its artifact files;
an unreadable or locked source database aborts with exit code 6;
source databases are opened read-only.

Safety and edge cases

pre-existing staged changes in the index;
repository in an unfinished merge/rebase state;
current branch differs from the configured branch;
forbidden-file abort produces the dedicated exit code;
stale lock left by a dead process is recovered;
lock file and log file never appear in git status;
configured timezone is honoured in commit timestamps.

Cross-platform

Automated testing should run against at least:

Windows
Linux

with macOS included where practical.

A CI test matrix is recommended.

42. Future Functional Extensions

The architecture should permit future introduction of:

automatic push to central Git server;
signed Git commits;
commit signing using a machine identity;
SCADA semantic comparison;
XML-aware comparison;
JSON-aware comparison;
proprietary SCADA project parsers;
alarm configuration comparison;
PLC source comparison;
automatic reports;
REST API;
web interface;
Grafana integration;
email/Teams notifications;
change approval workflow;
correlation against maintenance or change requests;
integration with Active Directory;
integration with SCADA audit logs;
cryptographic file hashes;
asset/reference-designation correlation;
automatic mapping of files to IEC/ISO 81346 objects;
detection of changes outside approved maintenance windows.

43. Important Design Principle

The application shall distinguish between three separate concepts:

Git commit author
    ≠
File last modifier
    ≠
SCADA change originator

The Git commit author is the automated service.

The file last modifier is information obtained from the best available metadata source.

The actual SCADA change originator should only be stated when an authoritative source, such as the SCADA audit trail, provides that information.

The application shall not infer a human change originator merely from filesystem ownership.

44. Definition of Done – Initial Release

Version 1.0 can be considered complete when it can:

run on Windows and Linux;
inspect an existing Git repository;
detect added, modified and deleted files;
filter files through configurable include/exclude rules;
invoke a configurable author resolver;
record per-file author information for additions and modifications;
generate the defined structured commit message;
automatically stage and commit changes;
avoid empty commits;
provide dry-run functionality;
produce rotating application logs;
prevent concurrent execution;
return useful process exit codes;
safely handle unresolved authors;
execute unattended through an external scheduler.

The initial implementation should deliberately keep SCADA-specific change attribution behind the AuthorResolver interface. That makes the generic Git/change-tracking engine reusable while allowing a resolver tailored to the actual SCADA platform to be added independently.

45. Derived Artifacts (Content Extraction)

This feature is prioritized for the next release.

45.1 Purpose

Some SCADA engineering data is stored in binary databases (e.g. SQLite). Git cannot show readable history for such files: every commit stores the whole binary blob and a diff reports only "binary files differ". To make this content auditable, the tracker shall extract selected database content into plain-text derived artifacts inside the repository before change detection (Section 40, step 6). The artifacts are then versioned like any other file, and Git history shows precise, row-level changes.

Source databases are identified by content (the SQLite file header read during extraction), not by file-name extension.

The implementation shall follow the same pluggable-interface principle as author resolution (Section 44): an ArtifactExtractor abstraction with a SQLite implementation as the first plugin, so further formats can be added independently.

45.2 Output layout

Extraction writes into a configurable output root inside the repository (default: extracted/). One sub-folder is created per configured source database, named after the database file. Within each sub-folder, one file is written per extracted table.

extracted/
└── alarms.sqlite/
    ├── alarms.jsonl
    └── events.jsonl

This layout gives each table its own artifact file, so Git diffs are limited to the tables that actually changed, and table names cannot collide between databases.

45.3 Configuration

extraction:
  enabled: true
  output_root: "extracted"    # relative to the repository root
  format: "jsonl"             # one JSON object per row
  sources:
    - path: "history/alarms.sqlite"   # repository-relative source
      tables:                          # optional; default: all user tables
        - "alarms"
        - "events"

An absent extraction: block, or enabled: false, disables the feature entirely; collection then proceeds without touching any database.

45.4 Artifact format

The default format shall be JSON Lines (jsonl): one JSON object per row with sorted keys and a stable encoding, so a changed row produces a minimal line-level diff. Rows shall be ordered deterministically by primary key (falling back to rowid), so unchanged content always produces a byte-identical file.

Artifact content shall not contain run-dependent values such as extraction timestamps: an artifact must be byte-identical whenever the underlying content is unchanged (FR-031). Table names that are not valid file names on the target platform shall be sanitized deterministically for artifact file names (FR-029).

45.5 Source access

Source databases shall be opened read-only; the tracker shall never write to them (FR-034). Because the SCADA platform may keep databases in WAL mode while running, extraction shall operate on a snapshot: the database file and its WAL sidecar shall be copied to a temporary location before reading, so a partially written transaction is never captured (see also Section 20).

45.6 Stale artifacts

Extraction shall also remove artifacts that no longer have a source:

a table removed from the database → its artifact file is deleted;
a source database removed from the configuration or the file system → its sub-folder is deleted.

Without this rule, stale artifacts would look like current data in Git history (FR-033).

45.7 Failure policy

Sources are isolated from each other: a source that cannot be read, is not a SQLite 3 database, or whose artifacts cannot be written, is reported as a warning and skipped; extraction then continues with the remaining configured sources. One bad file shall never block extraction of the others. When every configured source fails, the collection proceeds with the artifacts from the previous collection and a summary warning lists all skipped sources. Source validity is established during extraction by the SQLite engine itself - a non-database file raises sqlite3.DatabaseError: file is not a database - rather than by file-name extension: databases with vendor-specific or missing extensions are accepted (Section 45.5).

An extraction failure is not silent, however: every skipped source is logged, and a source that fails persistently must be visible in monitoring through the log (Section 22).

45.8 Raw source files

The configured binary source files should be excluded from direct tracking (files.exclude, e.g. **/*.sqlite). The repository then stores the readable artifacts instead of the opaque blob; tracking both duplicates every change and lets the binary churn dominate history. Direct tracking of the raw database remains possible where a deployment wants the authoritative blob as well.

45.9 Attribution

Generated artifacts have no human last-modifier of their own. Author resolution for artifact files shall therefore report UNKNOWN unless an extractor emits sidecar metadata (Section 10) that the resolver chain can consume, for example a last_modified_by column extracted from the database content.

Appendix A – Glossary

Term	Definition
Collection	One complete detection-and-commit cycle performed by the tracker
ChangeSet	The complete set of changes detected during one collection
ChangeRecord	A single detected change together with its metadata
AuthorResolver	Pluggable component attempting to determine the last modifying identity of a file
AuthorResult	The outcome of one author-resolution attempt, including source and confidence
Confidence	Trustworthiness of an author result: `authoritative`, `probable`, `weak` or `unknown`
Fallback	Result used when no resolver produces a usable result (default: `UNKNOWN`)
Forbidden file	A file matching `security.forbidden_patterns`; never committed by default
Tracker-Run	Unique execution identifier linking logs, commits and monitoring (Section 35)
Staged verification	Comparison of the actual staged change set against the expected change set before commit
Derived artifact	Plain-text file generated from a binary source (e.g. a SQLite table) and versioned in the repository (Section 45)
Artifact extractor	Pluggable component that refreshes derived artifacts from configured sources before change detection

45 KiB Raw Permalink Blame History Unescape Escape

Functional Specification

SCADA Unattended Change Tracker

1. Purpose

1.1 Conformance language

2. Scope

3. General Architecture

4. Operating Modes

4.1 Run-once mode

4.2 Scheduled service mode

5. Repository Configuration

5.1 Repository and index state

6. Change Detection

6.1 Change comparison

6.2 Git status

7. File Inclusion and Exclusion

8. Last Modifier Identification

8.1 Requirement

9. Author Resolution

10. Author Resolver Strategies

SCADA metadata resolver

Companion metadata resolver

SCADA audit-log resolver

Operating-system resolver

Unknown resolver

11. Resolver Priority

12. Change Data Model

13. Commit Generation

14. Git Staging

15. Commit Message Format

16. Commit Subject

17. Git Commit Identity

18. Multiple Authors

19. Atomic Processing

20. Concurrent Changes

21. Repository Lock

22. Logging

23. Log Rotation

24. Error Handling

25. Exit Codes

26. Configuration

27. Cross-Platform Requirements

28. Git Integration

29. Python Version

30. Command-Line Interface

run

status

validate

config

31. Dry-Run Mode

32. Security

33. Sensitive File Protection

34. Remote Repository

35. Audit Information

36. Structured Commit Metadata

37. Functional Requirements

FR-001 – Repository inspection

FR-002 – Added files

FR-003 – Modified files

FR-004 – Deleted files

FR-005 – Rename detection

FR-006 – Author identification

FR-007 – Unknown author

FR-008 – Per-file attribution

FR-009 – Commit generation

FR-010 – Commit summary

FR-011 – Commit details

FR-012 – Attribution

FR-013 – Deleted files

FR-014 – Empty executions

FR-015 – Logging

FR-016 – Cross-platform support

FR-017 – Dry run

FR-018 – Configuration

FR-019 – File filters

FR-020 – Concurrency protection

FR-021 – Error isolation

FR-022 – Repository protection

FR-023 – Atomic collection

FR-024 – Sensitive-data protection

45 KiB

Raw Permalink Blame History

`run`

`status`

`validate`

`config`