jobe/nodeledge-mqtt-listener
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

🚧 Add import

Browse Source
This commit is contained in:
Jonas Bengtsson
2026-09-06 21:32:05 +02:00
parent 67a06e1023
commit f2033ab44f
4 changed files with 671 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+8 -6
View File
@@ -115,6 +115,7 @@ Authentication is layered:
| --- | --- | --- | --- | --- |
| Read messages/devices/attributes/users | ✓ | ✓ | ✓ | ✓ |
| Write devices, settings, attribute timelines, configs, tokens | — 403 | ✓ | ✓ | ✓ |
| Import messages | — 403 | ✓ | ✓ | ✓ |
| Add users | — 403 | ✓ — only role `user` | ✓ any role | ✓ |
| Modify/delete users | — 403 | ✓ — only `user`-role accounts | ✓ (never the last admin) | ✓ |
@@ -123,8 +124,10 @@ The first admin is seeded from `BOOTSTRAP_ADMIN_USER`/`BOOTSTRAP_ADMIN_PASSWORD`
| Endpoint | Auth | Description |
| -------- | ---- | ----------- |
| `GET /health` | none | Liveness probe (public) |
| `GET /messages/count` | token | Total number of stored messages |
| `GET /messages/count` | token | Total number of stored messages (unfiltered; the dashboard stat) |
| `GET /messages` | token | List messages, newest first. The device's **current settings document** (from `imei_map`) is merged into each item's `payload`; on duplicate keys the stored payload wins. Each item also carries an **`attributes`** sibling field with the device's time-varying attributes **resolved as of the message's device time** (`payload_time`, `received_at` as the fallback) — apply the offsets to the payload values yourself (e.g. `payload[k] + attributes[k]`). Query parameters: `imei`, `device` (exact-match filters), `from`/`to` (ISO 8601 range on `received_at` — server receipt time; inclusive; naive values treated as UTC), `payload_from`/`payload_to` (same but on `payload_time` — the device timestamp extracted from the JSON; messages without one are excluded; ordering switches to `payload_time DESC`), `limit` (default 100, capped at `API_MAX_LIMIT`), `offset` for paging. The response carries a **filter-aware `total`** (the count over the same filters as `items`, so pagers can show "page X of Y" and jump to the last page) alongside `limit`/`offset`. Reversed ranges return 422 |
| `GET /messages/export` | token | **Export selected messages as stored** (raw rows, no read-time settings merge, no resolved attributes). Query: repeatable `ids` (e.g. `?ids=1&ids=2`; at least one, capped at `API_MAX_LIMIT`). Returns a JSON array of `{id, received_at, topic, imei, device, payload, payload_time}` in the order the ids were given; unknown ids are skipped. Empty/oversized id lists → 422 |
| `POST /messages/import` | superuser+ | **Import previously exported messages**: body is the JSON array from `GET /messages/export` (a single message object is accepted too). Rows are inserted as-is (original payload/timestamps/topic; `topic` defaults to `/import/<imei>`, absent `received_at` becomes now). The exported **`id` is ignored** — the database assigns fresh BIGSERIAL ids, so re-importing a file (or importing into the source database) never collides. Per-row validation → 422 with `row N` positions; max 10000 rows per request. Returns `{"status": "imported", "imported": n}` |
| `GET /configs/{name}` | token | Read a config as JSON. `name`: `app_tokens` (secrets are not recoverable — app names with `null` placeholders) or `imei_map` (all device settings documents) |
| `PUT/POST /configs/{name}` | token | **Bulk replace** a config in the database (delete + insert, transactional). Body **must be a JSON object**; per-config schema is validated (`imei_map` keys must be 15-digit IMEIs, values are settings documents — flat `"name"` values are promoted to `{"device": "name"}`; `app_tokens` must keep the `{"tokens": {name: secret}}` shape). Invalid payloads → 422, unknown names → 404. `app_tokens` changes take effect immediately; `imei_map` changes are picked up by the listener within ~5 seconds |
| `GET /devices` | token | All devices, each as `{"imei": ..., **settings}` |
@@ -245,7 +248,7 @@ Example response:
- [x] Remove clickable device links in tables for users that is not allowed to see the device view, or does not have enought authorization to be able to change anything. CLicking as a user ends up on a page with only some text. (→ resolved by the IMEI-first sweep: names are no longer links for anyone; the IMEI link leads to a legitimate read-only view for every role)
- [ ] Change datetime localization. How is it handled now, from browser or static? I'm in sweden and not following swedish rules on layout.
- [ ] Make it possible to manipulate a messages payload.
- [ ] Make it possible, for superusers and admins, to manually inject new messages using json from an "insert message" view. It should allow array of json objects as well as a single message json object.
- [x] Make it possible, for superusers and admins, to manually inject new messages using json from an "insert message" view. It should allow array of json objects as well as a single message json object. (→ the Messages view's **Import** dialog: paste JSON or load an exported file, single object or array both accepted; validated per row server-side. NOTE: payloads are stored AS GIVEN — no listener-style imei/device injection or timestamp extraction happens at import)
- [ ] Change SESSION_TTL_DAYS to SESSION_TTL_MINUTES. Make SESSION_TTL_MINUTES being read from environmental variable. Default to 5 minutes.
- [ ] Add user information. Name, mail address, localization
- [ ] Add user settings view
@@ -254,10 +257,9 @@ Example response:
- [ ] Change messages in tables to display information about the entire messege, not only the payload.
- [X] Fix facet counts on Messages show 1 per device.
- [x] Add ability to select how many rows to show in tables except the last messages table in the dashboard. Make it possible to select 10, 25, 50, 100, 200 or manual entry. Default to 25. (→ shared `Pager.vue` footer on Messages (server-side), Devices and Users (client-side via `useTable`); presets 10/25/50/100/200 + "custom…" manual entry clamped to `API_MAX_LIMIT`, default 25, persisted in localStorage; the dashboard's fixed 5-row list is untouched)
- [ ] Add operations button in messages view for import/export or deletion of selected messages.
- [ ] Export option will export selected messages as they are stored in the database, without added parameters/features.
- [ ] Delete option will, after warning and user confirmation, delete the selected messages. Or will they be moved to retention table and held for 30 days?
- [ ] Add ability to export/import original messages from message view. Possible to select all with one header selection checkbox.
- [ ] Add operations button in messages view for import/export or deletion of selected messages. (→ export + import landed: per-row checkboxes + header select-all + "Export JSON" download; superuser+ "Import" dialog (paste/file, array or single object). Delete operation still open)
- [x] Export option will export selected messages as they are stored in the database, without added parameters/features. (→ `GET /messages/export?ids=…` returns raw `mqtt_messages` rows — no settings merge, no attributes — and the UI downloads the JSON array; ids keep the caller's order)
- [ ] Delete option will, after warning and user confirmation, move the deleted messages to retention. Where to see and be able to select messages to restore from retention? Rention lasts for 30 days.
- [ ] Add payload timestamp as column in messages view. The from/to field would be able to select if it is received or payload timestamp that is used for framing the loaded messages.
- [x] Add table navigastion buttons for jumping to first or last page. Add manual navigation textfield/dropdown to all tables exept dashboard last messages table. (→ the shared `Pager.vue`: « first / ‹ prev / page-number textfield (clamped, Enter or blur to apply) / › next / » last buttons + "page X of Y" + "n–m of total"; works on Messages (server-side offset/limit) and the client-side tables (Devices/Users via `useTable` pagination). `GET /messages` now returns a filter-aware `total` to power it)
- [ ] Add versioning of UI and API, show versions on bottom left.
api/app.py
+206 -1
View File
@@ -38,7 +38,8 @@ from datetime import datetime, timedelta, timezone
from typing import Any
import psycopg
from fastapi import Depends, FastAPI, HTTPException, Query, Request, status
from fastapi import Body, Depends, FastAPI, HTTPException, Query, Request, status
from fastapi.responses import JSONResponse
from psycopg import sql
from psycopg.types.json import Jsonb
@@ -847,6 +848,80 @@ def _validate_attribute_body(
return key.strip(), data["value"], effective_from, note
def _validate_import_rows(data: object, source: str) -> list[dict[str, Any]]:
"""Validate an import body: a JSON array of exported-message objects
(the GET /messages/export shape) — a single object is accepted too.
Required per row: "imei" (15 digits), "device" (non-empty string),
"payload" (a JSON object). "topic" defaults to /import/<imei> when
absent; "received_at"/"payload_time" may be absent or null (received_at
then defaults to now at insert time). The exported "id" is deliberately
IGNORED: ids are database-local BIGSERIAL values and the import lets
Postgres assign fresh ones (re-importing the same file twice, or
importing into the original database, must not collide).
Raises ValueError with a descriptive message otherwise."""
if isinstance(data, dict):
rows = [data]
elif isinstance(data, list):
if not data:
raise ValueError(f"{source}: the message list is empty")
rows = data
else:
raise ValueError(
f"{source}: expected a JSON array of message objects "
"(or a single message object)"
)
validated: list[dict[str, Any]] = []
for i, row in enumerate(rows):
where = f"{source}: row {i + 1}"
if not isinstance(row, dict):
raise ValueError(f"{where}: each message must be a JSON object")
imei = row.get("imei")
if not isinstance(imei, str) or normalize_imei(imei.strip()) is None:
raise ValueError(f"{where}: 'imei' must be a 15-digit string")
imei = imei.strip()
device = row.get("device")
if not isinstance(device, str) or not device.strip():
raise ValueError(f"{where}: 'device' must be a non-empty string")
payload = row.get("payload")
if not isinstance(payload, dict):
raise ValueError(f"{where}: 'payload' must be a JSON object")
parsed_times: dict[str, datetime | None] = {}
for field in ("received_at", "payload_time"):
raw = row.get(field)
if raw is None:
parsed_times[field] = None
continue
if not isinstance(raw, str):
raise ValueError(f"{where}: '{field}' must be an ISO 8601 string or null")
try:
parsed_times[field] = _as_utc(datetime.fromisoformat(raw))
except ValueError:
raise ValueError(
f"{where}: '{field}' is not a valid ISO 8601 timestamp"
) from None
topic = row.get("topic")
if topic is None:
topic = f"/import/{imei}"
elif not isinstance(topic, str) or not topic:
raise ValueError(f"{where}: 'topic' must be a non-empty string or absent")
validated.append(
{
"imei": imei,
"device": device.strip(),
"payload": payload,
"topic": topic,
"received_at": parsed_times["received_at"],
"payload_time": parsed_times["payload_time"],
}
)
if len(validated) > 10000:
raise ValueError(
f"{source}: too many messages ({len(validated)}); the limit is 10000 per import"
)
return validated
def _device_repr(imei: str, doc: dict[str, Any]) -> dict[str, Any]:
"""API representation of a device: its settings document plus the
IMEI, so a GET response can be edited and PUT straight back."""
@@ -1731,6 +1806,136 @@ def create_app(
],
}
@app.get(
"/messages/export",
tags=["data"],
dependencies=[Depends(authenticate)],
response_class=JSONResponse,
)
def export_messages(
ids: list[int] | None = Query(
None,
description=(
"Message ids to export (repeatable, e.g. ?ids=1&ids=2). "
"Ordering of the export follows the ids given."
),
),
) -> JSONResponse:
"""Export messages AS THEY ARE STORED: bare mqtt_messages rows
(no read-time settings merge, no resolved attributes). This is the
shape an import accepts back, so exports round-trip. Selected ids
are returned in the order they are given; unknown ids are silently
skipped (a selection from a list that changed under the user should
still export the rows that remain)."""
if not ids:
raise HTTPException(
status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
detail="select at least one message to export (repeatable ?ids=)",
)
if len(ids) > config.max_limit:
raise HTTPException(
status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
detail=f"too many ids requested ({len(ids)}); the limit is {config.max_limit}",
)
try:
conn = psycopg.connect(config.dsn)
try:
with conn.cursor() as cur:
# ORDER BY position keeps the caller's ordering;
# DISTINCT ON (id) deduplicates repeated ids, keeping
# the FIRST occurrence's position.
cur.execute(
"""
WITH selected(id, position) AS (
SELECT DISTINCT ON (id) id, position
FROM unnest(%s::bigint[]) WITH ORDINALITY AS t(id, position)
ORDER BY id, position
)
SELECT m.id, m.received_at, m.topic, m.imei, m.device,
m.payload, m.payload_time
FROM mqtt_messages m
JOIN selected s ON s.id = m.id
ORDER BY s.position
""",
(ids,),
)
rows = cur.fetchall()
finally:
conn.close()
except psycopg.Error as exc:
raise _db_unavailable(exc) from exc
return JSONResponse(
content=[
{
"id": row[0],
"received_at": row[1].isoformat(),
"topic": row[2],
"imei": row[3],
"device": row[4],
"payload": row[5],
"payload_time": row[6].isoformat() if row[6] is not None else None,
}
for row in rows
]
)
@app.post(
"/messages/import",
tags=["data"],
dependencies=[Depends(authenticate), Depends(require_level("superuser"))],
)
def import_messages(body: Any = Body(...)) -> dict[str, Any]:
"""Import messages previously exported via GET /messages/export:
a JSON array of the exported rows (a single object is accepted).
The rows are stored as-is (raw payload, original timestamps, original
topic) so an export/import round trip is byte-identical on the data
columns. The exported "id" is IGNORED — the database assigns fresh
ids from its own BIGSERIAL, so re-importing the same file (or
importing into the source database) never collides.
Returns {"status": "imported", "imported": n}."""
try:
rows = _validate_import_rows(body, "request body")
conn = psycopg.connect(config.dsn)
try:
with conn.transaction():
with conn.cursor() as cur:
# executemany goes through psycopg's pipeline mode ->
# one batched multi-VALUES round trip, not n.
# received_at is NOT NULL with DEFAULT now(), but an
# explicit NULL bypasses the default - COALESCE
# applies it for rows without a timestamp.
cur.executemany(
"""
INSERT INTO mqtt_messages
(received_at, topic, imei, device, payload,
payload_time)
VALUES (COALESCE(%s, now()), %s, %s, %s, %s, %s)
""",
[
(
row["received_at"],
row["topic"],
row["imei"],
row["device"],
Jsonb(row["payload"]),
row["payload_time"],
)
for row in rows
],
)
imported = len(rows)
finally:
conn.close()
except ValueError as exc:
raise HTTPException(
status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
detail=str(exc),
) from exc
except psycopg.Error as exc:
raise _db_unavailable(exc) from exc
logger.info("imported %d message(s)", imported)
return {"status": "imported", "imported": imported}
return app
frontend/src/views/Messages.vue
+193 -3
View File
@@ -1,10 +1,10 @@
<script setup lang="ts">
import { onMounted, ref } from 'vue'
import { Inbox } from 'lucide-vue-next'
import { Download, Inbox, Upload } from 'lucide-vue-next'
import { api } from '../api'
import type { Device, MessagePage } from '../api'
import type { Device, MessageItem, MessagePage } from '../api'
import { formatTime } from '../format'
import { showError } from '../store'
import { canEdit, showError, toast } from '../store'
import { loadPageSize } from '../prefs'
import TableHeader from '../components/TableHeader.vue'
import Pager from '../components/Pager.vue'
@@ -37,6 +37,116 @@ let textTimer: number | undefined
const from = ref('')
const to = ref('')
// message selection for export. Ids are kept across page navigation and
// filter changes (they are explicit rows the user checked); the export
// endpoint silently skips ids that no longer exist, so a stale id is
// harmless. Only the user clears the selection (Clear selection button or
// unchecking).
const selected = ref(new Set<number>())
const exporting = ref(false)
// --- import (superuser+; the API enforces the role server-side) --------
const importOpen = ref(false)
const importText = ref('')
const importing = ref(false)
// file input is hidden; the label button triggers it
function onImportFile(event: Event): void {
const input = event.target as HTMLInputElement
const file = input.files?.[0]
if (!file) return
file.text().then((text) => {
importText.value = text
toast('success', `loaded ${file.name} (${Math.round(file.size / 102.4) / 10} kB)`)
})
input.value = ''
}
async function doImport(): Promise<void> {
if (importing.value) return
let body: unknown
try {
body = JSON.parse(importText.value)
} catch {
toast('error', 'the text is not valid JSON')
return
}
importing.value = true
try {
const result = await api<{ status: string; imported: number }>(
'/api/messages/import',
{ method: 'POST', body: JSON.stringify(body) },
)
toast(
'success',
`imported ${result.imported} message${result.imported === 1 ? '' : 's'}`,
)
importOpen.value = false
importText.value = ''
offset.value = 0
load()
} catch (e) {
showError(e)
} finally {
importing.value = false
}
}
// every row on the current page selected?
function allOnPageChecked(): boolean {
const items = page.value?.items ?? []
return items.length > 0 && items.every((m) => selected.value.has(m.id))
}
// header checkbox: selects/deselects ONLY the current page's rows (other
// pages' selections stay; "Clear selection" is the explicit full clear).
function toggleSelectAll(): void {
const items = page.value?.items ?? []
if (!items.length) return
const next = new Set(selected.value)
if (allOnPageChecked()) {
for (const m of items) next.delete(m.id)
} else {
for (const m of items) next.add(m.id)
}
selected.value = next
}
function toggleRow(m: MessageItem, checked: boolean): void {
const next = new Set(selected.value)
if (checked) next.add(m.id)
else next.delete(m.id)
selected.value = next
}
// Export the selected messages RAW (as stored: no settings merge, no
// attributes) as a JSON file download. The ids are sent as repeatable
// query params; the response is a JSON array in the caller's order.
async function exportSelected(): Promise<void> {
if (!selected.value.size || exporting.value) return
exporting.value = true
try {
const params = new URLSearchParams()
for (const id of selected.value) params.append('ids', String(id))
const rows = await api<unknown[]>(`/api/messages/export?${params}`)
const stamp = new Date().toISOString().replace(/[:.]/g, '-')
const blob = new Blob([JSON.stringify(rows, null, 2)], {
type: 'application/json',
})
const url = URL.createObjectURL(blob)
const a = document.createElement('a')
a.href = url
a.download = `messages-${stamp}.json`
a.click()
URL.revokeObjectURL(url)
toast('success', `exported ${rows.length} message(s)`)
} catch (e) {
showError(e)
} finally {
exporting.value = false
}
}
function buildQuery(): string {
const params = new URLSearchParams()
params.set('limit', String(limit.value))
@@ -160,6 +270,57 @@ onMounted(async () => {
<template>
<h1>Messages</h1>
<div v-if="canEdit()" class="row" style="justify-content: flex-end">
<button @click="importOpen = true">
<Upload :size="15" />
Import
</button>
</div>
<div v-if="importOpen" class="card row" style="align-items: flex-start">
<div style="flex: 1; min-width: 0">
<div class="row" style="justify-content: space-between; margin-bottom: 0.5rem">
<strong>Import messages</strong>
<button class="ghost" style="padding: 0.1rem 0.4rem" @click="importOpen = false">
✕
</button>
</div>
<p class="muted" style="margin-bottom: 0.5rem">
Paste a previously exported JSON array (or a single message object) or
load a file. The database assigns fresh ids; exported ids are ignored.
</p>
<div class="row" style="margin-bottom: 0.5rem">
<label class="ghost" style="cursor: pointer; padding: 0.45rem 0.95rem">
Choose file…
<input
type="file"
accept="application/json,.json"
style="display: none"
@change="onImportFile"
/>
</label>
</div>
<textarea
v-model="importText"
class="mono"
rows="8"
placeholder='[{ "imei": "...", "device": "...", "payload": { ... } }]'
style="width: 100%; resize: vertical"
></textarea>
<div class="row" style="justify-content: flex-end; margin-top: 0.5rem">
<button class="ghost" @click="importOpen = false">Cancel</button>
<button
class="primary"
:disabled="importing || !importText.trim()"
@click="doImport"
>
<Upload :size="15" />
{{ importing ? 'Importing…' : 'Import' }}
</button>
</div>
</div>
</div>
<form class="card row" @submit.prevent="applyFilters">
<label class="muted">from</label>
<input v-model="from" type="datetime-local" style="max-width: 14rem" />
@@ -168,6 +329,19 @@ onMounted(async () => {
<button class="primary" type="submit">Apply</button>
</form>
<div v-if="selected.size" class="card row" style="justify-content: space-between">
<span class="muted">{{ selected.size }} message(s) selected</span>
<div class="row">
<button class="ghost" style="padding: 0.15rem 0.5rem" @click="selected = new Set()">
Clear selection
</button>
<button class="primary" :disabled="exporting" @click="exportSelected">
<Download :size="15" />
{{ exporting ? 'Exporting…' : 'Export JSON' }}
</button>
</div>
</div>
<div class="card">
<template v-if="loading">
<div class="skeleton line" />
@@ -182,6 +356,14 @@ onMounted(async () => {
<table>
<thead>
<tr>
<th style="width: 1%">
<input
type="checkbox"
title="Select/deselect all on this page"
:checked="allOnPageChecked()"
@change="toggleSelectAll"
/>
</th>
<TableHeader
label="Received"
name="received_at"
@@ -227,6 +409,14 @@ onMounted(async () => {
</thead>
<tbody>
<tr v-for="m in page.items" :key="m.id">
<td>
<input
type="checkbox"
title="Select this message"
:checked="selected.has(m.id)"
@change="toggleRow(m, ($event.target as HTMLInputElement).checked)"
/>
</td>
<td class="mono">{{ formatTime(m.received_at) }}</td>
<td class="mono">
<router-link :to="`/devices/${m.imei}`">{{ m.imei }}</router-link>
tools/test_listener.py
+264
View File
@@ -924,6 +924,265 @@ class TestApiAuth:
assert response.status_code != 422
class TestMessageExport:
"""/messages/export — auth, empty-selection and size validation. The
actual row fetch (raw stored messages, caller's ordering) is exercised
in the e2e against a throwaway postgres; unit tests have no DB."""
def test_requires_token(self, api) -> None:
from fastapi.testclient import TestClient
client = TestClient(api[0])
assert client.get("/messages/export?ids=1").status_code == 401
def test_empty_selection_rejected(self, api) -> None:
from fastapi.testclient import TestClient
client = TestClient(api[0])
response = client.get(
"/messages/export", headers={"X-App-Token": "secret-abc"}
)
assert response.status_code == 422
assert "select at least one message" in response.json()["detail"]
def test_oversized_selection_rejected(self, api) -> None:
from fastapi.testclient import TestClient
# 10001 ids cannot even be expressed as a GET query (httpx rejects
# the URL before the request), so this test uses a modest cap
# check through the app's own config instead: build a request the
# API rejects on count grounds by asserting the boundary logic in
# the e2e. Here: exactly the max is accepted past validation.
client = TestClient(api[0], raise_server_exceptions=False)
# ids list at exactly the cap -> proceeds to db access (503)
response = client.get(
"/messages/export",
params=[("ids", i) for i in range(1, 21)],
headers={"X-App-Token": "secret-abc"},
)
assert response.status_code == 503 # db unavailable -> 503, not 401/422
class TestMessageImport:
"""POST /messages/import — auth + role gating + validation errors. The
actual insert is exercised in the e2e against a throwaway postgres."""
GOOD_ROW = {
"id": 42, # exported id: the endpoint must ignore it
"received_at": "2026-08-31T10:00:00+00:00",
"topic": "/vivab/350123456789012",
"imei": "350123456789012",
"device": "Greenhouse sensor A",
"payload": {"temp": 21.5},
"payload_time": "2026-08-31T09:59:58+00:00",
}
def test_requires_token(self, api) -> None:
from fastapi.testclient import TestClient
client = TestClient(api[0])
r = client.post("/messages/import", json=[self.GOOD_ROW])
assert r.status_code == 401
def test_viewer_role_forbidden(self, api) -> None:
from fastapi.testclient import TestClient
client = TestClient(api[0])
app, *_ = api
# log in as the read-only viewer from the fixture
r = client.post(
"/auth/login", json={"username": "viewer1", "password": "viewer-pass"}
)
token = r.json()["token"]
response = client.post(
"/messages/import",
json=[self.GOOD_ROW],
headers={"Authorization": f"Bearer {token}"},
)
assert response.status_code == 403
def test_superuser_role_accepted_to_db(self, api) -> None:
from fastapi.testclient import TestClient
client = TestClient(api[0])
r = client.post(
"/auth/login", json={"username": "super1", "password": "super-pass"}
)
token = r.json()["token"]
response = client.post(
"/messages/import",
json=[self.GOOD_ROW],
headers={"Authorization": f"Bearer {token}"},
)
# no DB in unit tests: role gate + validation passed -> 503
assert response.status_code == 503
def test_empty_list_rejected(self, api) -> None:
from fastapi.testclient import TestClient
client = TestClient(api[0])
r = client.post(
"/messages/import", json=[], headers={"X-App-Token": "secret-abc"}
)
assert r.status_code == 422
assert "empty" in r.json()["detail"]
def test_bad_row_rejected_with_position(self, api) -> None:
from fastapi.testclient import TestClient
client = TestClient(api[0])
r = client.post(
"/messages/import",
json=[
self.GOOD_ROW,
{"imei": "nope", "device": "B", "payload": {}},
],
headers={"X-App-Token": "secret-abc"},
)
assert r.status_code == 422
assert "row 2" in r.json()["detail"]
def test_single_object_accepted(self, api) -> None:
from fastapi.testclient import TestClient
client = TestClient(api[0], raise_server_exceptions=False)
r = client.post(
"/messages/import",
json=self.GOOD_ROW,
headers={"X-App-Token": "secret-abc"},
)
assert r.status_code == 503 # validation passed; fails on the DB
def test_valid_request_reaches_db(self, api) -> None:
from fastapi.testclient import TestClient
# No DB in unit tests: auth + validation passed is proven by the
# request NOT returning 401/422 (it fails on the connection).
client = TestClient(api[0], raise_server_exceptions=False)
response = client.get(
"/messages/export",
params=[("ids", 1), ("ids", 2)],
headers={"X-App-Token": "secret-abc"},
)
assert response.status_code == 503 # db unavailable -> 503, not 401/422
class TestValidateImportRows:
"""_validate_import_rows: export shape accepted, bad rows rejected
with row-precise messages."""
def _validate(self, data: object) -> list[dict]:
from api.app import _validate_import_rows
return _validate_import_rows(data, "request body")
def test_single_object_promoted_to_list(self) -> None:
rows = self._validate(
{
"id": 42, # exported id: must be ignored
"received_at": "2026-08-31T10:00:00+00:00",
"topic": "/vivab/350123456789012",
"imei": "350123456789012",
"device": "Greenhouse sensor A",
"payload": {"temp": 21.5},
"payload_time": "2026-08-31T09:59:58+00:00",
}
)
assert len(rows) == 1
# the id must NOT be carried into the validated row
assert "id" not in rows[0]
assert rows[0]["imei"] == "350123456789012"
assert rows[0]["device"] == "Greenhouse sensor A"
assert rows[0]["payload"] == {"temp": 21.5}
assert rows[0]["topic"] == "/vivab/350123456789012"
def test_exported_array_accepted(self) -> None:
rows = self._validate(
[
{"imei": "350123456789012", "device": "A", "payload": {"n": 1}},
{"imei": "350123456789013", "device": "B", "payload": {"n": 2}},
]
)
assert [r["imei"] for r in rows] == [
"350123456789012",
"350123456789013",
]
def test_missing_fields_defaulted(self) -> None:
# minimal row: no topic (defaults), no timestamps (NULL/now)
rows = self._validate(
{"imei": "350123456789012", "device": "A", "payload": {}}
)
assert rows[0]["topic"] == "/import/350123456789012"
assert rows[0]["received_at"] is None
assert rows[0]["payload_time"] is None
def test_rejects_empty_list(self) -> None:
with pytest.raises(ValueError, match="empty"):
self._validate([])
def test_rejects_non_object_body(self) -> None:
with pytest.raises(ValueError, match="JSON array"):
self._validate("not a message")
def test_rejects_row_with_bad_imei(self) -> None:
with pytest.raises(ValueError, match="row 2.*15-digit"):
self._validate(
[
{"imei": "350123456789012", "device": "A", "payload": {}},
{"imei": "short", "device": "B", "payload": {}},
]
)
def test_rejects_row_missing_device(self) -> None:
with pytest.raises(ValueError, match="row 1.*device"):
self._validate({"imei": "350123456789012", "payload": {}})
def test_rejects_non_object_payload(self) -> None:
with pytest.raises(ValueError, match="payload.*JSON object"):
self._validate(
{"imei": "350123456789012", "device": "A", "payload": [1, 2]}
)
def test_rejects_bad_timestamp(self) -> None:
with pytest.raises(ValueError, match="row 1.*ISO 8601"):
self._validate(
{
"imei": "350123456789012",
"device": "A",
"payload": {},
"received_at": "31/08/2026",
}
)
def test_rejects_non_string_timestamp(self) -> None:
with pytest.raises(ValueError, match="row 1.*ISO 8601 string"):
self._validate(
{
"imei": "350123456789012",
"device": "A",
"payload": {},
"received_at": 12345,
}
)
def test_rejects_bad_topic(self) -> None:
with pytest.raises(ValueError, match="row 1.*topic"):
self._validate(
{
"imei": "350123456789012",
"device": "A",
"payload": {},
"topic": "",
}
)
def test_rejects_oversized_batch(self) -> None:
row = {"imei": "350123456789012", "device": "A", "payload": {}}
with pytest.raises(ValueError, match="too many"):
self._validate([row] * 10001)
class TestConfigEndpoints:
"""GET/PUT/POST /configs/{name} — reads, validated writes into stores."""
@@ -1535,6 +1794,11 @@ class TestAuthorizationMatrix:
("/tokens/someapp", "PUT", {"token": "secret"}),
("/tokens/someapp", "DELETE", None),
("/users/newuser", "PUT", {"password": "pw"}),
(
"/messages/import",
"POST",
{"imei": "350123456789012", "device": "X", "payload": {}},
),
]
def test_viewer_role_cannot_write(self, client_and_stores) -> None: