# MQTT Ingester Stack

A Docker-based stack that ingests time-series data from an MQTTS broker into
TimescaleDB, exposes it through an authenticated HTTPS API, and optionally
forwards logs to Loki.

## Architecture

```mermaid
flowchart LR
    Broker[MQTTS Broker] -->|TLS + client cert| MQTT[MQTT Client]
    MQTT -->|SQL| TSDB[(TimescaleDB)]
    MQTT -->|logs| Loki[Loki]
    TSDB --> API[HTTPS API]
    Client[API Client] -->|mTLS| API
```

| Service      | Purpose                                                        |
| ------------ | -------------------------------------------------------------- |
| `timescaledb`| Time-series database storing telemetry as hypertables.         |
| `mqtt-client`| Subscribes to MQTTS topics, writes payloads to the database.   |
| `api`        | FastAPI service serving telemetry over HTTPS with mTLS.        |
| `loki`       | Optional log aggregation (enabled via the `telemetry` profile).|

## Prerequisites

- Docker and Docker Compose v2.
- A CA certificate and signed certificates for the MQTT client, the API
  server, and any API clients.

## Configuration

Copy the example environment file and edit it:

```bash
cp .env.example .env
```

Key variables:

| Variable            | Description                                   |
| ------------------- | --------------------------------------------- |
| `POSTGRES_USER`     | TimescaleDB user.                             |
| `POSTGRES_PASSWORD` | TimescaleDB password.                         |
| `MQTT_BROKER_HOST`  | MQTTS broker hostname.                        |
| `MQTT_BROKER_PORT`  | MQTTS broker port (default `8883`).           |
| `MQTT_TOPIC`        | Topic(s) to subscribe to (default `sensors/#`).|
| `MQTT_USERNAME`     | MQTT username for broker auth (optional).     |
| `MQTT_PASSWORD`     | MQTT password for broker auth (optional).     |
| `MQTT_PSK_IDENTITY` | TLS-PSK identity (optional; enables PSK mode).|
| `MQTT_PSK_HEX`      | TLS-PSK key as hex (optional).                |
| `DB_BATCH_SIZE`     | Rows per batched DB insert (default `500`).    |
| `DB_FLUSH_SECONDS`  | Max seconds a batch waits before flushing (default `1.0`). |
| `DB_POOL_MIN_SIZE`  | Min API DB pool connections (default `2`).     |
| `DB_POOL_MAX_SIZE`  | Max API DB pool connections (default `10`).    |
| `LOKI_URL`          | Loki push URL; leave empty to disable.        |
| `LOKI_FLUSH_SECONDS`| Seconds between batched Loki pushes (default `5`). |
| `API_PORT`          | Host port for the HTTPS API (default `8443`). |

## Certificates

Place the following files in the `certs/` directory:

| File         | Used by      | Purpose                          |
| ------------ | ------------ | -------------------------------- |
| `ca.crt`     | all          | CA certificate.                  |
| `client.crt` | mqtt-client  | Client certificate for the broker.|
| `client.key` | mqtt-client  | Client private key.              |
| `server.crt` | api          | API server certificate.          |
| `server.key` | api          | API server private key.          |

Generate a self-signed CA and certificates for testing:

```bash
mkdir -p certs && cd certs

# CA
openssl req -x509 -newkey rsa:2048 -days 365 -nodes \
  -keyout ca.key -out ca.crt -subj "/CN=Test CA"

# MQTT client certificate
openssl req -newkey rsa:2048 -nodes -keyout client.key -out client.csr \
  -subj "/CN=mqtt-client"
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out client.crt -days 365

# API server certificate
openssl req -newkey rsa:2048 -nodes -keyout server.key -out server.csr \
  -subj "/CN=localhost"
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out server.crt -days 365
```

> The broker must trust `ca.crt` and accept `client.crt` for authentication.

## MQTT Broker Authentication

The client always connects over TLS and picks its authentication mode from
the environment:

| Mode | Variables | Notes |
| ---- | --------- | ----- |
| **TLS-PSK** | `MQTT_PSK_IDENTITY` + `MQTT_PSK_HEX` | Takes precedence when set. No certificates needed; the broker must have a PSK listener (Mosquitto: `psk_hint` + `psk_file`). Requires Python 3.13+ (the mqtt-client image provides this). |
| **Client certificate (mTLS)** | presence of `certs/client.crt` + `certs/client.key` | Default when the files exist and PSK is not configured. |
| **Username/password** | `MQTT_USERNAME` (+ `MQTT_PASSWORD`) | Sent in the MQTT CONNECT packet; combinable with either TLS mode. Matches Mosquitto's `password_file`. |
| **Anonymous** | none of the above | Only works if the broker allows it; a warning is logged. |

Examples:

```bash
# TLS-PSK
MQTT_PSK_IDENTITY=ingester MQTT_PSK_HEX=0123456789abcdef

# Username/password over CA-verified TLS (no client cert required)
MQTT_USERNAME=ingester MQTT_PASSWORD=secret
```

TLS-PSK in Mosquitto requires a dedicated listener, e.g.:

```
listener 8884
psk_hint mqtt-ingester
psk_file /mosquitto/psk_file   # line: ingester:0123456789abcdef
```

## Running

Start the core stack:

```bash
docker compose up -d
```

Start with Loki enabled:

```bash
docker compose --profile telemetry up -d
```

## API

The API is served over HTTPS with mutual TLS. Clients must present a
certificate signed by the configured CA.

```bash
curl --cacert certs/ca.crt \
     --cert certs/client.crt \
     --key certs/client.key \
     https://localhost:8443/health
```

Endpoints:

| Method | Path                 | Description                              |
| ------ | -------------------- | ---------------------------------------- |
| GET    | `/health`            | Health check.                            |
| GET    | `/telemetry`         | List telemetry (filters: `topic`, `device_id`, `limit`). |
| GET    | `/telemetry/latest`  | Latest record (filters: `topic`, `device_id`). |

Example:

```bash
curl --cacert certs/ca.crt --cert certs/client.crt --key certs/client.key \
  "https://localhost:8443/telemetry?topic=sensors/temp&limit=10"
```

## Logging

The MQTT client logs in JSON format to:

- **Console** (always).
- **File** (`/var/log/mqtt-client/mqtt-client.log`) when `LOG_TO_FILE=true`.
- **Loki** when `LOKI_URL` is set. Records are buffered and pushed in
  batches from a background thread, so a slow or unavailable Loki never
  blocks message ingestion.

## Performance

- The MQTT client buffers incoming messages in an in-memory queue and
  inserts them into TimescaleDB in batches (one transaction per batch),
  keeping database I/O off the MQTT network loop. Tune with
  `DB_BATCH_SIZE` and `DB_FLUSH_SECONDS`. If the queue fills (broker
  outpacing the database), the client logs the condition and drops new
  messages rather than stalling the broker connection.
- The API serves queries from a pool of reused database connections
  (`DB_POOL_MIN_SIZE`/`DB_POOL_MAX_SIZE`).
- Chunks older than 7 days are compressed, and data older than 365 days
  is dropped automatically (see `db/init/001_schema.sql`).

## Database Schema

Telemetry is stored in the `telemetry` hypertable:

| Column     | Type        | Description                    |
| ---------- | ----------- | ------------------------------ |
| `time`     | `TIMESTAMPTZ`| Ingestion timestamp.           |
| `topic`    | `TEXT`      | MQTT topic.                    |
| `device_id`| `TEXT`      | Extracted from payload if present.|
| `payload`  | `JSONB`     | Full message payload.          |

## Security Notes

- Change `POSTGRES_PASSWORD` before deploying.
- Use a real CA and rotate certificates regularly.
- Restrict the API port with a firewall or reverse proxy.