🎉 Initial commit

2026-08-30 09:56:33 +02:00
commit c2ae24818e
14 changed files with 1058 additions and 0 deletions
@@ -0,0 +1,205 @@
+# MQTT Ingester Stack
+
+A Docker-based stack that ingests time-series data from an MQTTS broker into
+TimescaleDB, exposes it through an authenticated HTTPS API, and optionally
+forwards logs to Loki.
+
+## Architecture
+
+```mermaid
+flowchart LR
+    Broker[MQTTS Broker] -->|TLS + client cert| MQTT[MQTT Client]
+    MQTT -->|SQL| TSDB[(TimescaleDB)]
+    MQTT -->|logs| Loki[Loki]
+    TSDB --> API[HTTPS API]
+    Client[API Client] -->|mTLS| API
+```
+
+| Service      | Purpose                                                        |
+| ------------ | -------------------------------------------------------------- |
+| `timescaledb`| Time-series database storing telemetry as hypertables.         |
+| `mqtt-client`| Subscribes to MQTTS topics, writes payloads to the database.   |
+| `api`        | FastAPI service serving telemetry over HTTPS with mTLS.        |
+| `loki`       | Optional log aggregation (enabled via the `telemetry` profile).|
+
+## Prerequisites
+
+- Docker and Docker Compose v2.
+- A CA certificate and signed certificates for the MQTT client, the API
+  server, and any API clients.
+
+## Configuration
+
+Copy the example environment file and edit it:
+
+```bash
+cp .env.example .env
+```
+
+Key variables:
+
+| Variable            | Description                                   |
+| ------------------- | --------------------------------------------- |
+| `POSTGRES_USER`     | TimescaleDB user.                             |
+| `POSTGRES_PASSWORD` | TimescaleDB password.                         |
+| `MQTT_BROKER_HOST`  | MQTTS broker hostname.                        |
+| `MQTT_BROKER_PORT`  | MQTTS broker port (default `8883`).           |
+| `MQTT_TOPIC`        | Topic(s) to subscribe to (default `sensors/#`).|
+| `MQTT_USERNAME`     | MQTT username for broker auth (optional).     |
+| `MQTT_PASSWORD`     | MQTT password for broker auth (optional).     |
+| `MQTT_PSK_IDENTITY` | TLS-PSK identity (optional; enables PSK mode).|
+| `MQTT_PSK_HEX`      | TLS-PSK key as hex (optional).                |
+| `DB_BATCH_SIZE`     | Rows per batched DB insert (default `500`).    |
+| `DB_FLUSH_SECONDS`  | Max seconds a batch waits before flushing (default `1.0`). |
+| `DB_POOL_MIN_SIZE`  | Min API DB pool connections (default `2`).     |
+| `DB_POOL_MAX_SIZE`  | Max API DB pool connections (default `10`).    |
+| `LOKI_URL`          | Loki push URL; leave empty to disable.        |
+| `LOKI_FLUSH_SECONDS`| Seconds between batched Loki pushes (default `5`). |
+| `API_PORT`          | Host port for the HTTPS API (default `8443`). |
+
+## Certificates
+
+Place the following files in the `certs/` directory:
+
+| File         | Used by      | Purpose                          |
+| ------------ | ------------ | -------------------------------- |
+| `ca.crt`     | all          | CA certificate.                  |
+| `client.crt` | mqtt-client  | Client certificate for the broker.|
+| `client.key` | mqtt-client  | Client private key.              |
+| `server.crt` | api          | API server certificate.          |
+| `server.key` | api          | API server private key.          |
+
+Generate a self-signed CA and certificates for testing:
+
+```bash
+mkdir -p certs && cd certs
+
+# CA
+openssl req -x509 -newkey rsa:2048 -days 365 -nodes \
+  -keyout ca.key -out ca.crt -subj "/CN=Test CA"
+
+# MQTT client certificate
+openssl req -newkey rsa:2048 -nodes -keyout client.key -out client.csr \
+  -subj "/CN=mqtt-client"
+openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key \
+  -CAcreateserial -out client.crt -days 365
+
+# API server certificate
+openssl req -newkey rsa:2048 -nodes -keyout server.key -out server.csr \
+  -subj "/CN=localhost"
+openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key \
+  -CAcreateserial -out server.crt -days 365
+```
+
+> The broker must trust `ca.crt` and accept `client.crt` for authentication.
+
+## MQTT Broker Authentication
+
+The client always connects over TLS and picks its authentication mode from
+the environment:
+
+| Mode | Variables | Notes |
+| ---- | --------- | ----- |
+| **TLS-PSK** | `MQTT_PSK_IDENTITY` + `MQTT_PSK_HEX` | Takes precedence when set. No certificates needed; the broker must have a PSK listener (Mosquitto: `psk_hint` + `psk_file`). Requires Python 3.13+ (the mqtt-client image provides this). |
+| **Client certificate (mTLS)** | presence of `certs/client.crt` + `certs/client.key` | Default when the files exist and PSK is not configured. |
+| **Username/password** | `MQTT_USERNAME` (+ `MQTT_PASSWORD`) | Sent in the MQTT CONNECT packet; combinable with either TLS mode. Matches Mosquitto's `password_file`. |
+| **Anonymous** | none of the above | Only works if the broker allows it; a warning is logged. |
+
+Examples:
+
+```bash
+# TLS-PSK
+MQTT_PSK_IDENTITY=ingester MQTT_PSK_HEX=0123456789abcdef
+
+# Username/password over CA-verified TLS (no client cert required)
+MQTT_USERNAME=ingester MQTT_PASSWORD=secret
+```
+
+TLS-PSK in Mosquitto requires a dedicated listener, e.g.:
+
+```
+listener 8884
+psk_hint mqtt-ingester
+psk_file /mosquitto/psk_file   # line: ingester:0123456789abcdef
+```
+
+## Running
+
+Start the core stack:
+
+```bash
+docker compose up -d
+```
+
+Start with Loki enabled:
+
+```bash
+docker compose --profile telemetry up -d
+```
+
+## API
+
+The API is served over HTTPS with mutual TLS. Clients must present a
+certificate signed by the configured CA.
+
+```bash
+curl --cacert certs/ca.crt \
+     --cert certs/client.crt \
+     --key certs/client.key \
+     https://localhost:8443/health
+```
+
+Endpoints:
+
+| Method | Path                 | Description                              |
+| ------ | -------------------- | ---------------------------------------- |
+| GET    | `/health`            | Health check.                            |
+| GET    | `/telemetry`         | List telemetry (filters: `topic`, `device_id`, `limit`). |
+| GET    | `/telemetry/latest`  | Latest record (filters: `topic`, `device_id`). |
+
+Example:
+
+```bash
+curl --cacert certs/ca.crt --cert certs/client.crt --key certs/client.key \
+  "https://localhost:8443/telemetry?topic=sensors/temp&limit=10"
+```
+
+## Logging
+
+The MQTT client logs in JSON format to:
+
+- **Console** (always).
+- **File** (`/var/log/mqtt-client/mqtt-client.log`) when `LOG_TO_FILE=true`.
+- **Loki** when `LOKI_URL` is set. Records are buffered and pushed in
+  batches from a background thread, so a slow or unavailable Loki never
+  blocks message ingestion.
+
+## Performance
+
+- The MQTT client buffers incoming messages in an in-memory queue and
+  inserts them into TimescaleDB in batches (one transaction per batch),
+  keeping database I/O off the MQTT network loop. Tune with
+  `DB_BATCH_SIZE` and `DB_FLUSH_SECONDS`. If the queue fills (broker
+  outpacing the database), the client logs the condition and drops new
+  messages rather than stalling the broker connection.
+- The API serves queries from a pool of reused database connections
+  (`DB_POOL_MIN_SIZE`/`DB_POOL_MAX_SIZE`).
+- Chunks older than 7 days are compressed, and data older than 365 days
+  is dropped automatically (see `db/init/001_schema.sql`).
+
+## Database Schema
+
+Telemetry is stored in the `telemetry` hypertable:
+
+| Column     | Type        | Description                    |
+| ---------- | ----------- | ------------------------------ |
+| `time`     | `TIMESTAMPTZ`| Ingestion timestamp.           |
+| `topic`    | `TEXT`      | MQTT topic.                    |
+| `device_id`| `TEXT`      | Extracted from payload if present.|
+| `payload`  | `JSONB`     | Full message payload.          |
+
+## Security Notes
+
+- Change `POSTGRES_PASSWORD` before deploying.
+- Use a real CA and rotate certificates regularly.
+- Restrict the API port with a firewall or reverse proxy.