jobe/nodeledge-mqtt-listener
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

🚧 Add handling of static metadata

Browse Source
This commit is contained in:
Jonas Bengtsson
2026-09-04 22:56:02 +02:00
parent 46639a837c
commit c1b9006f78
9 changed files with 762 additions and 87 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+29 -13
View File
@@ -79,7 +79,8 @@ PASS: count endpoint returned {'count': N}
- The IMEI is the **last segment** of the topic, e.g. `/vivab/350123456789012` → `350123456789012`.
- It must be exactly **15 digits**; anything else (letters, separators, other lengths) is skipped.
- It looks the IMEI up in the `imei_map` table; on a match the device name is injected, otherwise the IMEI itself is used as the device name. The listener re-reads the table every 5 seconds, so API edits take effect without a restart.
- It looks the IMEI up in the `imei_map` table and injects the device's settings document's `device` value as the device name; on no mapping the IMEI itself is used. The listener re-reads the table every 5 seconds, so API edits take effect without a restart.
- Each device's **settings document** (JSONB) can hold more than the name — e.g. `{"device": "Greenhouse sensor A", "location": "Växthus 1"}`. The listener only uses `device` at ingest; the API merges the whole current document into message payloads at read time (see below).
- The injected keys are `imei` and `device` (configurable via `INJECT_IMEI_KEY` / `INJECT_DEVICE_KEY`). Any existing values in the payload are **overwritten** — the topic is authoritative.
- A **device timestamp** is extracted at ingest time from the first parseable payload key in `PAYLOAD_TIME_KEYS` (default `timestamp,time,ts`; ISO 8601 strings, epoch seconds and epoch milliseconds all accepted) and stored in the `payload_time` column. A missing or unparseable timestamp does **not** skip the message — `payload_time` is simply NULL and `received_at` (server receipt time) remains authoritative.
- The payload can be a **single JSON object or a JSON array of objects**. Each object in an array is injected and stored as its own database row (each with its own `payload_time`); array elements that are not objects are skipped.
@@ -93,9 +94,16 @@ Authentication: pass one of the configured app tokens either as the `X-App-Token
| -------- | ---- | ----------- |
| `GET /health` | none | Liveness probe (public) |
| `GET /messages/count` | token | Total number of stored messages |
| `GET /messages` | token | List messages, newest first. Query parameters: `imei`, `device` (exact-match filters), `from`/`to` (ISO 8601 range on `received_at` — server receipt time; inclusive; naive values treated as UTC), `payload_from`/`payload_to` (same but on `payload_time` — the device timestamp extracted from the JSON; messages without one are excluded; ordering switches to `payload_time DESC`), `limit` (default 100, capped at `API_MAX_LIMIT`), `offset` for paging. Reversed ranges return 422 |
| `GET /configs/{name}` | token | Read a config as JSON. `name`: `app_tokens` (secrets are not recoverable — app names with `null` placeholders) or `imei_map` (the full mapping) |
| `PUT/POST /configs/{name}` | token | Replace a config in the database. Body **must be a JSON object**; per-config schema is validated (imei_map keys must be 15-digit IMEIs; app_tokens must keep the `{"tokens": {name: secret}}` shape). Invalid payloads → 422, unknown names → 404. Writes are transactional (delete + insert). `app_tokens` changes take effect immediately; `imei_map` changes are picked up by the listener within ~5 seconds |
| `GET /messages` | token | List messages, newest first. The device's **current settings document** (from `imei_map`) is merged into each item's `payload`; on duplicate keys the stored payload wins. Query parameters: `imei`, `device` (exact-match filters), `from`/`to` (ISO 8601 range on `received_at` — server receipt time; inclusive; naive values treated as UTC), `payload_from`/`payload_to` (same but on `payload_time` — the device timestamp extracted from the JSON; messages without one are excluded; ordering switches to `payload_time DESC`), `limit` (default 100, capped at `API_MAX_LIMIT`), `offset` for paging. Reversed ranges return 422 |
| `GET /configs/{name}` | token | Read a config as JSON. `name`: `app_tokens` (secrets are not recoverable — app names with `null` placeholders) or `imei_map` (all device settings documents) |
| `PUT/POST /configs/{name}` | token | **Bulk replace** a config in the database (delete + insert, transactional). Body **must be a JSON object**; per-config schema is validated (`imei_map` keys must be 15-digit IMEIs, values are settings documents — flat `"name"` values are promoted to `{"device": "name"}`; `app_tokens` must keep the `{"tokens": {name: secret}}` shape). Invalid payloads → 422, unknown names → 404. `app_tokens` changes take effect immediately; `imei_map` changes are picked up by the listener within ~5 seconds |
| `GET /devices` | token | All devices, each as `{"imei": ..., **settings}` |
| `GET /devices/{imei}` | token | One device's settings document; unknown IMEI → 404, non-15-digit IMEI → 422 |
| `PUT /devices/{imei}` | token | **Upsert one device's settings document** (create or replace). Body is the settings object (must have a non-empty `device` key; extra keys are arbitrary JSON); a body `imei` key is tolerated when it matches the URL. Returns the saved document, so a GET response can be edited and PUT straight back |
| `DELETE /devices/{imei}` | token | Remove one device (its messages stay); unknown IMEI → 404 |
| `GET /tokens` | token | App names with a token (secrets are unrecoverable, shown as `null`) |
| `PUT /tokens/{app}` | token | **Add or rotate ONE app token** (single-row upsert — never touches the other apps' rows). Body: `{"token": "secret"}`. Takes effect immediately |
| `DELETE /tokens/{app}` | token | Remove one app token; unknown name → 404. Deleting the last token locks the API until a new one is added |
| `GET /docs`, `GET /redoc` | none | OpenAPI docs (public) |
When the database is unreachable, authenticated endpoints return **503** (the API has no purpose without the db).
@@ -108,25 +116,32 @@ GET /messages?device=Water%20tank%20sensor%20B&payload_from=2026-08-31T00:00:00Z
### Managing config through the API
Both configs live in the database (`app_tokens`, `imei_map` tables) and can be read and replaced over HTTP (authenticated like all other endpoints). This is how to add a device or rotate a token without direct database access:
Both configs live in the database (`app_tokens`, `imei_map` tables). **A management frontend should use the per-resource endpoints** (`/devices/{imei}`, `/tokens/{app}`) — they save one row at a time, so two users editing different devices (or an admin rotating one token) never clobber each other. The whole-document `PUT /configs/{name}` endpoints remain for bulk imports:
```bash
# read the current IMEI map
# read the current device settings
TOKEN='your-bootstrap-or-app-token'
curl -H "X-App-Token: $TOKEN" http://localhost:18080/configs/imei_map
curl -H "X-App-Token: $TOKEN" http://localhost:18080/devices
# replace it (JSON object only; keys must be 15-digit IMEIs)
# upsert ONE device with its settings document (the frontend way)
curl -X PUT -H "X-App-Token: $TOKEN" -H "Content-Type: application/json" \
-d '{"350123456789012": "Greenhouse sensor A", "490154203237518": "Water tank sensor B"}' \
-d '{"device": "Greenhouse sensor A", "location": "Växthus 1", "battery_warning": 20}' \
http://localhost:18080/devices/350123456789012
# bulk import: replace ALL device settings at once (old flat values still accepted)
curl -X PUT -H "X-App-Token: $TOKEN" -H "Content-Type: application/json" \
-d '{"350123456789012": {"device": "Greenhouse sensor A", "location": "Växthus 1"}}' \
http://localhost:18080/configs/imei_map
```
- `app_tokens`: `{"tokens": {"app-name": "secret"}}` — secrets are stored as SHA-256 digests (never plaintext) and cannot be read back; `GET /configs/app_tokens` shows only the app names. After an update the API re-authenticates with the **new** tokens immediately.
- `imei_map`: `{"<15-digit-IMEI>": "<device name>"}` — the listener polls the table every ~5 seconds; a temporarily unreachable database never breaks ingestion (the last good mapping stays in effect).
- `app_tokens`: `"tokens": {"app-name": "secret"}` — secrets are stored as SHA-256 digests (never plaintext) and cannot be read back; listing endpoints show only the app names. After an update the API re-authenticates with the **new** tokens immediately. For single-token rotation prefer `PUT /tokens/{app}`.
- `imei_map`: one settings document per IMEI; `"device"` is required, other keys are arbitrary JSON merged into message payloads at read time. The listener polls the table every ~5 seconds; a temporarily unreachable database never breaks ingestion (the last good mapping stays in effect).
**Read-time settings merge:** `GET /messages` joins `imei_map` and merges the device's *current* settings document into each item's `payload` (`COALESCE(im.meta, '{}'::jsonb) || m.payload` — the stored payload wins on duplicate keys). Historical messages therefore always reflect the *latest* device settings; the ingest-time `imei`/`device` snapshots in the stored payload and columns are unaffected.
#### Migrating an existing deployment from the JSON files
For a database that already has data (init scripts only run on an empty volume), apply the schema and migrate the JSON configs:
For a database that already has data (init scripts only run on an empty volume), apply the schema and migrate the JSON configs. The schema script is idempotent and **migrates the old `imei_map(device TEXT)` layout in place**: every existing row is converted to `{"device": ...}` and the legacy column is dropped:
```bash
docker compose exec db psql -U nodeledge -d nodeledge -f - < init/01_schema.sql
@@ -157,7 +172,8 @@ Example response:
"temperature": 21.5,
"timestamp": "2026-08-31T09:59:58+00:00",
"imei": "350123456789012",
"device": "Greenhouse sensor A"
"device": "Greenhouse sensor A",
"location": "Växthus 1"
},
"payload_time": "2026-08-31T09:59:58+00:00"
}