mqtt-ingester/README.md

MQTT Ingester Stack

A Docker-based stack that ingests time-series data from an MQTTS broker into TimescaleDB, exposes it through an authenticated HTTPS API, and optionally forwards logs to Loki.

Architecture

flowchart LR
    Broker[MQTTS Broker] -->|TLS + client cert| MQTT[MQTT Client]
    MQTT -->|SQL| TSDB[(TimescaleDB)]
    MQTT -->|logs| Loki[Loki]
    TSDB --> API[HTTPS API]
    Client[API Client] -->|mTLS| API

Service	Purpose
timescaledb	Time-series database storing telemetry as hypertables.
mqtt-client	Subscribes to MQTTS topics, writes payloads to the database.
api	FastAPI service serving telemetry over HTTPS with mTLS.
loki	Optional log aggregation (enabled via the telemetry profile).

Prerequisites

• Docker and Docker Compose v2.
• A CA certificate and signed certificates for the MQTT client, the API server, and any API clients.

Configuration

Copy the example environment file and edit it:

cp .env.example .env

Key variables:

Variable	Description
POSTGRES_USER	TimescaleDB user.
POSTGRES_PASSWORD	TimescaleDB password.
MQTT_BROKER_HOST	MQTTS broker hostname.
MQTT_BROKER_PORT	MQTTS broker port (default 8883).
MQTT_TOPIC	Topic(s) to subscribe to (default sensors/#).
MQTT_USERNAME	MQTT username for broker auth (optional).
MQTT_PASSWORD	MQTT password for broker auth (optional).
MQTT_PSK_IDENTITY	TLS-PSK identity (optional; enables PSK mode).
MQTT_PSK_HEX	TLS-PSK key as hex (optional).
DB_BATCH_SIZE	Rows per batched DB insert (default 500).
DB_FLUSH_SECONDS	Max seconds a batch waits before flushing (default 1.0).
DB_POOL_MIN_SIZE	Min API DB pool connections (default 2).
DB_POOL_MAX_SIZE	Max API DB pool connections (default 10).
LOKI_URL	Loki push URL; leave empty to disable.
LOKI_FLUSH_SECONDS	Seconds between batched Loki pushes (default 5).
API_PORT	Host port for the HTTPS API (default 8443).

Certificates

Place the following files in the certs/ directory:

File	Used by	Purpose
ca.crt	all	CA certificate.
client.crt	mqtt-client	Client certificate for the broker.
client.key	mqtt-client	Client private key.
server.crt	api	API server certificate.
server.key	api	API server private key.

Generate a self-signed CA and certificates for testing:

mkdir -p certs && cd certs

# CA
openssl req -x509 -newkey rsa:2048 -days 365 -nodes \
  -keyout ca.key -out ca.crt -subj "/CN=Test CA"

# MQTT client certificate
openssl req -newkey rsa:2048 -nodes -keyout client.key -out client.csr \
  -subj "/CN=mqtt-client"
openssl x509 -req -in client.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out client.crt -days 365

# API server certificate
openssl req -newkey rsa:2048 -nodes -keyout server.key -out server.csr \
  -subj "/CN=localhost"
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out server.crt -days 365

The broker must trust ca.crt and accept client.crt for authentication.

MQTT Broker Authentication

The client always connects over TLS and picks its authentication mode from the environment:

Mode	Variables	Notes
TLS-PSK	MQTT_PSK_IDENTITY + MQTT_PSK_HEX	Takes precedence when set. No certificates needed; the broker must have a PSK listener (Mosquitto: psk_hint + psk_file). Requires Python 3.13+ (the mqtt-client image provides this).
Client certificate (mTLS)	presence of certs/client.crt + certs/client.key	Default when the files exist and PSK is not configured.
Username/password	MQTT_USERNAME (+ MQTT_PASSWORD)	Sent in the MQTT CONNECT packet; combinable with either TLS mode. Matches Mosquitto's password_file.
Anonymous	none of the above	Only works if the broker allows it; a warning is logged.

Examples:

# TLS-PSK
MQTT_PSK_IDENTITY=ingester MQTT_PSK_HEX=0123456789abcdef

# Username/password over CA-verified TLS (no client cert required)
MQTT_USERNAME=ingester MQTT_PASSWORD=secret

TLS-PSK in Mosquitto requires a dedicated listener, e.g.:

listener 8884
psk_hint mqtt-ingester
psk_file /mosquitto/psk_file   # line: ingester:0123456789abcdef

Running

Start the core stack:

docker compose up -d

Start with Loki enabled:

docker compose --profile telemetry up -d

API

The API is served over HTTPS with mutual TLS. Clients must present a certificate signed by the configured CA.

curl --cacert certs/ca.crt \
     --cert certs/client.crt \
     --key certs/client.key \
     https://localhost:8443/health

Endpoints:

Method	Path	Description
GET	/health	Health check.
GET	/telemetry	List telemetry (filters: topic, device_id, limit).
GET	/telemetry/latest	Latest record (filters: topic, device_id).

Example:

curl --cacert certs/ca.crt --cert certs/client.crt --key certs/client.key \
  "https://localhost:8443/telemetry?topic=sensors/temp&limit=10"

Logging

The MQTT client logs in JSON format to:

• Console (always).
• File (/var/log/mqtt-client/mqtt-client.log) when LOG_TO_FILE=true.
• Loki when LOKI_URL is set. Records are buffered and pushed in batches from a background thread, so a slow or unavailable Loki never blocks message ingestion.

Performance

• The MQTT client buffers incoming messages in an in-memory queue and inserts them into TimescaleDB in batches (one transaction per batch), keeping database I/O off the MQTT network loop. Tune with DB_BATCH_SIZE and DB_FLUSH_SECONDS. If the queue fills (broker outpacing the database), the client logs the condition and drops new messages rather than stalling the broker connection.
• The API serves queries from a pool of reused database connections (DB_POOL_MIN_SIZE/DB_POOL_MAX_SIZE).
• Chunks older than 7 days are compressed, and data older than 365 days is dropped automatically (see db/init/001_schema.sql).

Database Schema

Telemetry is stored in the telemetry hypertable:

Column	Type	Description
time	TIMESTAMPTZ	Ingestion timestamp.
topic	TEXT	MQTT topic.
device_id	TEXT	Extracted from payload if present.
payload	JSONB	Full message payload.

Security Notes

• Change POSTGRES_PASSWORD before deploying.
• Use a real CA and rotate certificates regularly.
• Restrict the API port with a firewall or reverse proxy.