jobe/git-automatic-tracker-py

Fork 0

Files

T

Jonas Bengtsson e8d6e70f42 🎉 Initial commit

2026-09-11 23:41:47 +02:00

27 KiB

Raw Blame History

Functional Specification

SCADA Unattended Change Tracker

1. Purpose

The purpose of the application is to provide automated version tracking of configuration and project files associated with a SCADA system.

The application shall periodically inspect a configured set of files or directories for changes and automatically record detected changes in a Git repository.

The resulting Git history shall provide an auditable record showing:

when changes were detected,
which files were added,
which files were modified,
which files were removed,
and, where applicable, who last modified each added or modified file.

The application is intended to operate without normal user interaction and shall be capable of running on Windows, Linux and macOS.

2. Scope

The application shall:

monitor one or more configured SCADA project directories;
identify file-system changes relative to the current Git repository state;
determine whether files have been:
- added,
- modified,
- deleted,
- optionally renamed;
determine the last modifying user for added and modified files;
generate a structured Git commit message describing the detected changes;
stage the changes;
create a Git commit automatically;
maintain logs of application execution and errors;
support unattended scheduled execution;
operate consistently across supported operating systems.

The initial implementation does not need to:

interpret the semantic meaning of SCADA configuration changes;
restore files;
approve or reject changes;
provide a graphical user interface;
automatically push commits to a remote Git repository.

These functions may be added later.

3. General Architecture

The application should be divided into independent functional components.

+-------------------------+
| Scheduler / Invocation  |
+------------+------------+
             |
             v
+-------------------------+
| Application Controller  |
+------------+------------+
             |
             v
+-------------------------+
| Repository Inspector    |
+------------+------------+
             |
             v
+-------------------------+
| Change Detector         |
+------------+------------+
             |
       +-----+------+
       |            |
       v            v
+-------------+ +----------------+
| File Change | | Author Resolver|
| Information | |                |
+------+------+ +-------+--------+
       |                |
       +-------+--------+
               |
               v
+-------------------------+
| Commit Message Builder  |
+------------+------------+
             |
             v
+-------------------------+
| Git Commit Manager      |
+------------+------------+
             |
             v
+-------------------------+
| Logging / Audit         |
+-------------------------+

The components should be loosely coupled so that SCADA-specific metadata extraction can be implemented independently of Git handling.

4. Operating Modes

The application shall support at least two operating modes.

4.1 Run-once mode

Example:

scada-change-tracker run

The application performs one complete inspection and exits.

This mode is suitable for execution through:

Windows Task Scheduler,
cron,
systemd timers,
CI/CD systems,
Docker scheduled execution,
other external schedulers.

Run-once mode should be the primary operating mode because it simplifies reliability and cross-platform deployment.

4.2 Scheduled service mode

An optional internal scheduler may allow the application to remain running and execute according to configured times.

Example configuration:

schedule:
  enabled: true
  times:
    - "06:00"
    - "12:00"
    - "18:00"
    - "23:30"

The application should use local system time unless a timezone is explicitly configured.

Example:

timezone: Europe/Stockholm

5. Repository Configuration

The application shall operate against an existing Git repository.

Example:

repository:
  path: "/path/to/scada/project"
  branch: "main"

The configured directory shall contain a valid Git repository.

Before attempting any modification, the application shall verify:

the directory exists;
the directory is accessible;
the directory is a Git repository;
the configured branch can be determined;
Git operations can be performed.

If validation fails, the execution shall stop without modifying the repository.

6. Change Detection

6.1 Change comparison

Changes shall be detected relative to the current Git HEAD.

The following change types shall be supported:

Change	Identifier
Added file	`ADDED`
Modified file	`MODIFIED`
Deleted file	`DELETED`
Renamed file	`RENAMED`

Rename detection may initially be treated as one deletion and one addition if native rename detection is not implemented.

6.2 Git status

The application should use Git itself as the authoritative mechanism for determining changed files.

The implementation should therefore use an equivalent of:

git status --porcelain

rather than independently comparing file timestamps.

This avoids false change detection caused by modification times alone.

7. File Inclusion and Exclusion

The application shall support configurable inclusion and exclusion rules.

Example:

files:
  include:
    - "**/*"

  exclude:
    - "**/*.tmp"
    - "**/*.log"
    - "**/~*"
    - ".git/**"
    - "runtime/**"
    - "cache/**"

Glob-style matching should be supported.

Files excluded by configuration shall neither be committed nor included in commit messages.

8. Last Modifier Identification

8.1 Requirement

For every:

added file;
modified file;

the application shall attempt to determine the person or account responsible for its most recent modification.

Example:

Modified:
  configuration/plant1.xml
  Last modified by: DOMAIN\jsmith

Deleted files do not require a last-modifier lookup because the original file may no longer be available.

9. Author Resolution

Determining the last modifying user shall be implemented as a separate abstraction.

Conceptually:

class AuthorResolver:
    def resolve(file_path) -> AuthorResult:
        ...

A result should contain at least:

name
identifier
source
confidence
timestamp

Example:

{
  "name": "John Smith",
  "identifier": "DOMAIN\\jsmith",
  "source": "scada_metadata",
  "confidence": "authoritative",
  "timestamp": "2026-09-11T14:31:22+02:00"
}

10. Author Resolver Strategies

The application shall support configurable resolver implementations.

Suggested resolver types are:

SCADA metadata resolver

Reads user information embedded in the SCADA project file.

This should be preferred when the SCADA platform records information such as:

last modified by;
engineering user;
operator;
change timestamp;
workstation;
project revision.

Companion metadata resolver

Reads metadata associated with the file.

For example:

pumpstation.xml
pumpstation.xml.metadata

or from a project database.

SCADA audit-log resolver

Looks up the changed file or object in the SCADA system's audit log.

This is likely to provide the strongest attribution where the SCADA system maintains authenticated engineering change records.

Operating-system resolver

An OS-specific resolver may be implemented if the environment provides authoritative file auditing.

File ownership alone shall not be interpreted as proof of the last modifying user.

For example, on Windows:

DOMAIN\SCADAService

could own the file even though another engineering user performed the modification.

Unknown resolver

If no reliable identity can be determined:

Last modified by: UNKNOWN

The application shall still create the commit unless configured otherwise.

11. Resolver Priority

Multiple author resolvers may be configured.

Example:

author_resolution:
  resolvers:
    - scada_metadata
    - scada_audit_log
    - sidecar_metadata

  fallback: unknown

The resolvers shall execute in configured priority order.

The first authoritative result shall be used.

12. Change Data Model

Each detected change should internally be represented using a structure similar to:

ChangeRecord

path
change_type
previous_path
size
file_timestamp
author
author_source
author_timestamp

For example:

{
  "path": "Stations/PS101/config.xml",
  "change_type": "MODIFIED",
  "previous_path": null,
  "size": 34127,
  "file_timestamp": "2026-09-11T13:44:12+02:00",
  "author": "DOMAIN\\jsmith",
  "author_source": "scada_audit_log",
  "author_timestamp": "2026-09-11T13:43:58+02:00"
}

13. Commit Generation

If no relevant files have changed, the application shall:

record the execution in its log;
not create an empty Git commit;
return a successful execution result.

Example log:

2026-09-11 18:00:02 INFO No SCADA changes detected.

14. Git Staging

When changes are detected the application shall:

collect the complete change set;
resolve author information;
generate the commit message;
stage the relevant files;
verify the staged change set;
create the Git commit.

Files outside the configured scope shall not be staged.

The application should avoid blindly executing:

git add --all

unless the repository is exclusively dedicated to tracked SCADA data.

15. Commit Message Format

Commit messages shall be human-readable while retaining a predictable structure that can later be parsed automatically.

Recommended format:

SCADA automatic change capture - 2026-09-11 18:00:03

Summary:
  Added:    2
  Modified: 3
  Deleted:  1
  Renamed:  0

Added:
  Stations/PS101/pump2.xml
    Last modified by: DOMAIN\jsmith
    Modified at: 2026-09-11 15:41:22

  Stations/PS104/settings.xml
    Last modified by: DOMAIN\adoe
    Modified at: 2026-09-11 16:02:14

Modified:
  Stations/PS101/project.xml
    Last modified by: DOMAIN\jsmith
    Modified at: 2026-09-11 15:42:09

  Stations/PS103/alarm.xml
    Last modified by: DOMAIN\adoe
    Modified at: 2026-09-11 16:13:54

  System/global.xml
    Last modified by: UNKNOWN

Deleted:
  Stations/PS027/old-config.xml

16. Commit Subject

The first commit line should be concise.

Recommended format:

SCADA automatic change capture - YYYY-MM-DD HH:MM:SS

Alternative:

SCADA: 2 added, 3 modified, 1 deleted

The detailed information shall be placed in the commit body.

17. Git Commit Identity

The Git commit author shall represent the automated tracking system rather than the person who changed an individual file.

Example:

git:
  user_name: "SCADA Change Tracker"
  user_email: "scada-change-tracker@localhost"

The engineers responsible for individual changes are instead recorded within the commit message.

This avoids incorrectly assigning a single engineer as the author of a commit containing changes made by several people.

18. Multiple Authors

One execution may contain changes made by multiple individuals.

Example:

Modified:
  plant1.xml
    Last modified by: DOMAIN\jsmith

  plant2.xml
    Last modified by: DOMAIN\adoe

The application shall therefore store attribution per file rather than per commit.

19. Atomic Processing

A commit shall represent one complete collection cycle.

Conceptually:

Start collection
     ↓
Detect all changes
     ↓
Resolve metadata
     ↓
Build complete change record
     ↓
Stage changes
     ↓
Validate staged state
     ↓
Commit
     ↓
Finish collection

The application shall not create multiple commits for one collection unless explicitly configured to do so.

20. Concurrent Changes

Files may be modified while the collection is taking place.

The application should therefore detect this situation.

For each file the application may record attributes before and after processing:

size
modification timestamp
optional content hash

If the file changes while being processed, the application shall either:

restart analysis for that file; or
abort the complete commit and retry during the next execution.

The default behaviour should favour data integrity over forcing a commit.

21. Repository Lock

Only one tracker execution shall be allowed against a repository at the same time.

The application shall implement a repository-level execution lock.

For example:

.scada-change-tracker.lock

The lock shall include information such as:

process_id
hostname
start_time

Stale locks shall be detected safely.

22. Logging

Application activity shall be logged independently of Git.

Minimum log information shall include:

execution start
execution completion
repository
number of detected changes
files processed
author resolution result
Git operations
warnings
errors
commit hash
execution duration

Example:

2026-09-11 18:00:01 INFO Starting SCADA change collection
2026-09-11 18:00:01 INFO Repository: D:\SCADA\Project
2026-09-11 18:00:02 INFO Detected 6 changes
2026-09-11 18:00:02 INFO Author resolved: Stations/PS101/project.xml -> DOMAIN\jsmith
2026-09-11 18:00:03 INFO Commit created: 93ab4e21
2026-09-11 18:00:03 INFO Collection completed

23. Log Rotation

File-based logging shall support rotation.

Example:

logging:
  file: "logs/scada-change-tracker.log"
  max_size_mb: 20
  backup_count: 10

Console logging shall also be supported.

24. Error Handling

Errors shall be classified.

Recommended categories:

CONFIGURATION_ERROR
REPOSITORY_ERROR
FILE_ACCESS_ERROR
AUTHOR_RESOLUTION_ERROR
GIT_ERROR
LOCK_ERROR
INTERNAL_ERROR

An author-resolution error shall normally not prevent a commit.

For example:

Author: UNKNOWN
Reason: Audit log unavailable

A Git or repository integrity error shall prevent the commit.

25. Exit Codes

Suggested process exit codes:

Code	Meaning
0	Successful execution
1	General application failure
2	Configuration error
3	Repository error
4	Git operation failure
5	Repository locked
6	Change collection failed

This makes the application suitable for external monitoring.

26. Configuration

Configuration should primarily be stored in YAML.

Example:

application:
  timezone: Europe/Stockholm

repository:
  path: "D:/SCADA/Project"
  branch: main

git:
  user_name: "SCADA Change Tracker"
  user_email: "scada-change-tracker@localhost"

files:
  include:
    - "**/*"

  exclude:
    - "**/*.tmp"
    - "**/*.log"
    - ".git/**"

author_resolution:
  resolvers:
    - scada_metadata
    - scada_audit_log

  fallback: unknown

commit:
  create_when_no_changes: false

logging:
  level: INFO
  console: true
  file: "logs/scada-change-tracker.log"

Environment variables should be capable of overriding sensitive or deployment-specific settings.

27. Cross-Platform Requirements

The project shall support:

Windows,
Linux,
macOS.

The implementation shall therefore avoid assumptions about:

path separators;
shell syntax;
drive letters;
filesystem ownership models;
username formats;
case sensitivity.

Python's:

pathlib

shall be preferred for filesystem paths.

Shell command strings should not be constructed manually where avoidable.

For subprocess execution, argument arrays should be used.

Conceptually:

subprocess.run(
    ["git", "status", "--porcelain"],
    ...
)

rather than:

os.system("git status --porcelain")

28. Git Integration

The Git integration may use either:

the Git CLI through subprocess; or
a Python Git library.

Using the Git CLI is recommended initially because it closely follows native Git behaviour and makes troubleshooting straightforward.

The Git executable shall therefore be treated as an external prerequisite.

At startup the application shall verify:

git --version

and fail gracefully if Git is unavailable.

29. Python Version

Recommended minimum:

Python >= 3.12

The application should avoid platform-specific Python dependencies unless they are isolated behind optional adapters.

30. Command-Line Interface

Suggested commands:

scada-change-tracker run
scada-change-tracker status
scada-change-tracker validate
scada-change-tracker config

`run`

Perform change collection and create a commit if necessary.

`status`

Display currently detected changes without staging or committing them.

`validate`

Validate:

configuration,
Git installation,
repository,
file access,
configured author resolvers.

`config`

Display the effective configuration with secrets redacted.

31. Dry-Run Mode

The application shall support:

scada-change-tracker run --dry-run

Dry-run mode shall perform:

change detection
author resolution
commit message generation
validation

but shall not:

stage files
create commits
push data
modify the repository

The generated commit message should be displayed.

32. Security

The application shall follow least-privilege principles.

The execution account should require only:

read access to source metadata;
read/write access to the tracked repository;
access required to query SCADA audit information.

SCADA engineering credentials shall not be embedded in source code.

Secrets shall be provided through:

environment variables;
operating-system secret stores;
external secret-management systems.

Secrets must never appear in Git commit messages or application logs.

33. Sensitive File Protection

The application should support rules preventing accidental versioning of sensitive data.

Example:

security:
  forbidden_patterns:
    - "**/*.key"
    - "**/*.pem"
    - "**/secrets.*"
    - "**/credentials.*"

If such a file is detected, the default behaviour should be to abort the commit and log a security warning.

34. Remote Repository

Remote synchronization shall be optional.

Future configuration could contain:

remote:
  enabled: true
  name: origin
  push_after_commit: true

A failed push shall not invalidate the locally created commit.

The application shall distinguish:

COMMIT_SUCCESS
PUSH_FAILURE

35. Audit Information

Each collection should internally generate an execution identifier.

For example:

01J7K8HYV84ZKFH1J4T9R8DZQN

The identifier may be included in the commit:

Tracker-Run: 01J7K8HYV84ZKFH1J4T9R8DZQN

This provides traceability between:

application logs;
Git commits;
external monitoring;
SCADA audit records.

36. Structured Commit Metadata

To simplify future machine processing, the end of the commit message may include structured trailers.

Example:

Tracker-Version: 1.0.0
Tracker-Run: 01J7K8HYV84ZKFH1J4T9R8DZQN
Tracker-Host: SCADA-SRV-01
Tracker-Changes: 6

This enables later processing without parsing the human-readable sections.

37. Functional Requirements

FR-001 – Repository inspection

The system shall inspect a configured Git repository for uncommitted file changes.

FR-002 – Added files

The system shall identify newly added files.

FR-003 – Modified files

The system shall identify modified files.

FR-004 – Deleted files

The system shall identify deleted files.

FR-005 – Rename detection

The system should identify renamed files where practical.

FR-006 – Author identification

The system shall attempt to determine the last modifying identity for every added and modified file.

FR-007 – Unknown author

The system shall support an UNKNOWN author state.

FR-008 – Per-file attribution

Author information shall be associated with individual files rather than the complete commit.

FR-009 – Commit generation

The system shall automatically generate a Git commit for detected changes.

FR-010 – Commit summary

The commit message shall contain counts for each change type.

FR-011 – Commit details

The commit message shall list every affected file.

FR-012 – Attribution

Added and modified files shall contain last-modifier information in the commit message.

FR-013 – Deleted files

Deleted files shall be listed without requiring modifier information.

FR-014 – Empty executions

The system shall not create an empty commit when no changes are detected.

FR-015 – Logging

Every execution shall produce an audit log.

FR-016 – Cross-platform support

The system shall operate on Windows, Linux and macOS.

FR-017 – Dry run

The system shall support change analysis without repository modification.

FR-018 – Configuration

Application behaviour shall be configurable without modifying source code.

FR-019 – File filters

The system shall support file inclusion and exclusion patterns.

FR-020 – Concurrency protection

The system shall prevent simultaneous tracker executions against the same repository.

FR-021 – Error isolation

Failure to identify an author shall not normally prevent an otherwise valid change from being committed.

FR-022 – Repository protection

Git, filesystem or repository integrity errors shall prevent commit creation.

FR-023 – Atomic collection

A normal execution shall produce at most one commit representing the complete detected change set.

FR-024 – Sensitive-data protection

Configured forbidden files shall not automatically be committed.

38. Suggested Python Package Structure

scada-change-tracker/
│
├── pyproject.toml
├── README.md
├── LICENSE
│
├── config/
│   └── example.yaml
│
├── src/
│   └── scada_change_tracker/
│       │
│       ├── __init__.py
│       ├── __main__.py
│       ├── cli.py
│       ├── config.py
│       ├── application.py
│       │
│       ├── git/
│       │   ├── repository.py
│       │   ├── status.py
│       │   └── commit.py
│       │
│       ├── changes/
│       │   ├── detector.py
│       │   ├── models.py
│       │   └── filters.py
│       │
│       ├── authors/
│       │   ├── base.py
│       │   ├── resolver.py
│       │   ├── unknown.py
│       │   └── scada/
│       │       └── ...
│       │
│       ├── commits/
│       │   └── message_builder.py
│       │
│       ├── security/
│       │   └── validator.py
│       │
│       ├── locking/
│       │   └── repository_lock.py
│       │
│       └── logging/
│           └── setup.py
│
└── tests/
    ├── unit/
    ├── integration/
    └── fixtures/

39. Core Domain Objects

The project should initially define the following central objects:

Repository
ChangeSet
ChangeRecord
AuthorIdentity
AuthorResolution
TrackerRun
CommitResult

A ChangeSet conceptually contains:

ChangeSet
 ├── added[]
 ├── modified[]
 ├── deleted[]
 └── renamed[]

Each item is represented by a ChangeRecord.

40. Processing Sequence

A normal execution shall approximately perform the following sequence:

1. Load configuration
2. Validate configuration
3. Acquire repository lock
4. Verify Git availability
5. Validate repository
6. Determine current repository status
7. Apply inclusion/exclusion rules
8. Create ChangeSet
9. Resolve authors for ADDED files
10. Resolve authors for MODIFIED files
11. Validate files against security rules
12. Generate commit message
13. Stage selected changes
14. Verify staged changes
15. Create commit
16. Record commit hash
17. Write audit result
18. Release lock
19. Exit

If no changes exist, execution proceeds directly from step 8 to audit logging and termination.

41. Testing Requirements

Tests should cover at minimum:

Change detection

new file;
modified file;
deleted file;
rename;
multiple simultaneous changes;
Unicode filenames;
filenames containing spaces.

Git

clean repository;
dirty repository;
invalid repository;
failed commit;
unavailable Git executable.

Author resolution

one successful resolver;
multiple resolver fallback;
unknown user;
unavailable audit source;
malformed SCADA metadata.

File handling

inaccessible file;
file removed during collection;
file modified during collection;
excluded file;
forbidden file.

Cross-platform

Automated testing should run against at least:

Windows
Linux

with macOS included where practical.

A CI test matrix is recommended.

42. Future Functional Extensions

The architecture should permit future introduction of:

automatic push to central Git server;
signed Git commits;
commit signing using a machine identity;
SCADA semantic comparison;
XML-aware comparison;
JSON-aware comparison;
proprietary SCADA project parsers;
alarm configuration comparison;
PLC source comparison;
automatic reports;
REST API;
web interface;
Grafana integration;
email/Teams notifications;
change approval workflow;
correlation against maintenance or change requests;
integration with Active Directory;
integration with SCADA audit logs;
cryptographic file hashes;
asset/reference-designation correlation;
automatic mapping of files to IEC/ISO 81346 objects;
detection of changes outside approved maintenance windows.

43. Important Design Principle

The application shall distinguish between three separate concepts:

Git commit author
    ≠
File last modifier
    ≠
SCADA change originator

The Git commit author is the automated service.

The file last modifier is information obtained from the best available metadata source.

The actual SCADA change originator should only be stated when an authoritative source, such as the SCADA audit trail, provides that information.

The application shall not infer a human change originator merely from filesystem ownership.

44. Definition of Done – Initial Release

Version 1.0 can be considered complete when it can:

run on Windows and Linux;
inspect an existing Git repository;
detect added, modified and deleted files;
filter files through configurable include/exclude rules;
invoke a configurable author resolver;
record per-file author information for additions and modifications;
generate the defined structured commit message;
automatically stage and commit changes;
avoid empty commits;
provide dry-run functionality;
produce rotating application logs;
prevent concurrent execution;
return useful process exit codes;
safely handle unresolved authors;
execute unattended through an external scheduler.

The initial implementation should deliberately keep SCADA-specific change attribution behind the AuthorResolver interface. That makes the generic Git/change-tracking engine reusable while allowing a resolver tailored to the actual SCADA platform to be added independently.

27 KiB Raw Blame History Unescape Escape

Functional Specification

SCADA Unattended Change Tracker

1. Purpose

2. Scope

3. General Architecture

4. Operating Modes

4.1 Run-once mode

4.2 Scheduled service mode

5. Repository Configuration

6. Change Detection

6.1 Change comparison

6.2 Git status

7. File Inclusion and Exclusion

8. Last Modifier Identification

8.1 Requirement

9. Author Resolution

10. Author Resolver Strategies

SCADA metadata resolver

Companion metadata resolver

SCADA audit-log resolver

Operating-system resolver

Unknown resolver

11. Resolver Priority

12. Change Data Model

13. Commit Generation

14. Git Staging

15. Commit Message Format

16. Commit Subject

17. Git Commit Identity

18. Multiple Authors

19. Atomic Processing

20. Concurrent Changes

21. Repository Lock

22. Logging

23. Log Rotation

24. Error Handling

25. Exit Codes

26. Configuration

27. Cross-Platform Requirements

28. Git Integration

29. Python Version

30. Command-Line Interface

run

status

validate

config

31. Dry-Run Mode

32. Security

33. Sensitive File Protection

34. Remote Repository

35. Audit Information

36. Structured Commit Metadata

37. Functional Requirements

FR-001 – Repository inspection

FR-002 – Added files

FR-003 – Modified files

FR-004 – Deleted files

FR-005 – Rename detection

FR-006 – Author identification

FR-007 – Unknown author

FR-008 – Per-file attribution

FR-009 – Commit generation

FR-010 – Commit summary

FR-011 – Commit details

FR-012 – Attribution

FR-013 – Deleted files

FR-014 – Empty executions

FR-015 – Logging

FR-016 – Cross-platform support

FR-017 – Dry run

FR-018 – Configuration

FR-019 – File filters

FR-020 – Concurrency protection

FR-021 – Error isolation

FR-022 – Repository protection

FR-023 – Atomic collection

FR-024 – Sensitive-data protection

38. Suggested Python Package Structure

39. Core Domain Objects

27 KiB

Raw Blame History

`run`

`status`

`validate`

`config`