🐛 Fix container stop issue

README.md

@@ -53,8 +53,9 @@ Bootstrap sequence on first start:
   the MQTT `$CONTROL/dynamic-security/v1` API. No files involved.
 * **Generated mosquitto.conf (rare)** — listeners/global settings are
   generated from the DB model and deployed through `mosquitto-control`,
-  which shares the broker's PID namespace (`pid: "service:mosquitto"`)
-  to signal reloads. No Docker socket is mounted anywhere (spec §29).
+  which OWNS the shared PID namespace (`pid: "service:mosquitto-control"`
+  on the broker); it signals the broker process and can restart it — and
+  survives broker restarts — without Docker sockets (spec §29).
 * Mosquitto 2.1's HTTP API is **not** used as a management channel: it
   has no authentication and (currently) exposes read-only plugin info.
 

broker-control/Dockerfile

@@ -1,17 +1,19 @@
 # broker-control — internal control sidecar (spec §29)
 #
-# Runs in the mosquitto container's PID namespace (compose `pid: "service:mosquitto"`)
-# so it can signal the broker process (PID 1) without docker.sock access.
-# The backend calls its HTTP API over the backend-net network.
+# PID-namespace OWNER: the broker joins THIS container's PID namespace
+# (compose `pid: "service:mosquitto-control"` on the mosquitto service).
+# Owning the namespace means the sidecar survives broker restarts — it
+# was previously the other way around and the kernel killed the sidecar
+# whenever the broker exited. tini runs as PID 1: it forwards signals to
+# python and reaps the broker's zombies (the broker re-parents to tini
+# when its container entrypoint forks).
 #
 # Responsibilities (deliberately minimal, spec §29):
 #   - validate staged configuration (mosquitto -c <staged> --test-config)
 #   - activate a configuration revision (atomic rename into place)
-#   - signal config reload (mosquitto_signal -p 1 config-reload == SIGHUP)
+#   - signal config reload (mosquitto_signal -p <broker-pid> config-reload)
+#   - signal broker restart (SIGTERM; Docker policy revives the broker)
 #   - check broker process state
-#
-# Release 1 ships validate/activate/reload/status only; restarts are a
-# Release 2 concern (reload-vs-restart classification, spec §28).
 
 FROM eclipse-mosquitto:2.1.2-alpine
 
@@ -20,7 +22,8 @@ FROM eclipse-mosquitto:2.1.2-alpine
 
 # The base image ships NO Python. control.py is stdlib-only, so the
 # distro package is sufficient (no pip, no venv needed).
-RUN apk add --no-cache python3
+# tini: minimal init as PID 1 — signal forwarding + zombie reaping.
+RUN apk add --no-cache python3 tini
 
 COPY control.py /app/control.py
 
@@ -36,4 +39,4 @@ CMD []
 
 EXPOSE 8090
 
-ENTRYPOINT ["python3", "/app/control.py"]
+ENTRYPOINT ["/sbin/tini", "--", "python3", "/app/control.py"]

broker-control/control.py

@@ -11,7 +11,10 @@ Security model:
       Docker network; the port is not published to the host.
     * Requires a bearer token (CONTROL_API_TOKEN) shared with the backend.
     * Never mounts docker.sock; signals the broker over the shared PID
-      namespace (broker is PID 1 in this container's view).
+      namespace. OWNERSHIP IS INVERTED: this sidecar owns the namespace
+      (tini is PID 1) and the broker joins it — so the sidecar survives
+      broker restarts and can orchestrate them (spec section 29). The
+      broker's PID is discovered via /proc, not hardcoded.
 
 Implementation notes:
     * Atomic activation writes content to a temp file in the SAME
@@ -98,18 +101,47 @@ def _activate(content: str) -> None:
         raise
 
 
+def _broker_pid() -> int | None:
+    """Find the broker process in this PID namespace.
+
+    The namespace is OWNED by this sidecar (compose inversion: the broker
+    joins us via `pid: "service:mosquitto-control"`), so the broker is
+    NOT PID 1 — discover it by scanning /proc for the mosquitto binary.
+    PID 1 here is tini (the sidecar's init).
+    """
+    try:
+        for entry in os.listdir("/proc"):
+            if not entry.isdigit():
+                continue
+            pid = int(entry)
+            if pid == 1:
+                continue  # tini (this sidecar's init), never the broker
+            try:
+                with open(f"/proc/{pid}/comm", encoding="utf-8") as f:
+                    if f.read().strip() == "mosquitto":
+                        return pid
+            except OSError:
+                continue  # process vanished mid-scan
+    except OSError:
+        pass
+    return None
+
+
 def _signal_reload() -> None:
-    """Signal the broker (PID 1 in the shared PID namespace) to reload."""
+    """Signal the broker to reload its configuration."""
+    pid = _broker_pid()
+    if pid is None:
+        raise _Error(503, "broker process not found in the PID namespace")
     try:
         proc = subprocess.run(
-            ["mosquitto_signal", "-p", "1", "config-reload"],
+            ["mosquitto_signal", "-p", str(pid), "config-reload"],
             capture_output=True,
             text=True,
             timeout=15,
         )
     except FileNotFoundError:
         # Fallback if the helper app is unavailable for any reason.
-        os.kill(1, signal.SIGHUP)
+        os.kill(pid, signal.SIGHUP)
         return
     if proc.returncode != 0:
         raise _Error(500, f"mosquitto_signal failed: {proc.stderr.strip()}")
@@ -118,25 +150,25 @@ def _signal_reload() -> None:
 def _restart_broker() -> None:
     """Gracefully stop the broker; Docker's restart policy revives it.
 
-    The broker is PID 1 in the shared PID namespace. SIGTERM makes it
-    shut down cleanly (persistence saved); `restart: unless-stopped`
-    on the mosquitto service then starts the container again. This
-    sidecar is a separate container and survives, so it can report
-    status while the broker is down (spec section 29: restart broker
-    where explicitly supported — no Docker socket involved).
+    The broker is a normal process in THIS sidecar's PID namespace (we
+    own it; the broker joined us). SIGTERM makes it shut down cleanly
+    (persistence saved); `restart: unless-stopped` on the mosquitto
+    service then starts the container again. Because the sidecar is the
+    namespace init, it SURVIVES the broker's death and keeps reporting
+    status through the outage (spec section 29 — no Docker socket).
     """
-    # The HTTP response MUST be flushed before PID 1 dies, so the kill
-    # happens after the handler returns — orchestrated by the route.
-    os.kill(1, signal.SIGTERM)
+    # The HTTP response MUST be flushed before the broker dies, so the
+    # kill happens after the handler returns — orchestrated by the route.
+    pid = _broker_pid()
+    if pid is None:
+        # Already down (restart in flight): nothing to signal, and the
+        # caller still gets a clean "restarting" response.
+        return
+    os.kill(pid, signal.SIGTERM)
 
 
 def _broker_pid_alive() -> bool:
-    try:
-        # Broker is PID 1 in the shared PID namespace.
-        with open("/proc/1/comm", encoding="utf-8") as f:
-            return f.read().strip() == "mosquitto"
-    except OSError:
-        return False
+    return _broker_pid() is not None
 
 
 def _post_broker_health() -> bool:

compose.yaml

@@ -50,8 +50,18 @@ services:
     build:
       context: ./mosquitto
     restart: unless-stopped
-    # Spec §29: no docker.sock. Control sidecar shares the broker's PID namespace
-    # and the mosquitto_config volume; it signals PID 1 (the broker) directly.
+    # Spec §29: no docker.sock. INVERTED PID ownership: the broker joins
+    # the CONTROL SIDECAR's PID namespace, not the other way around.
+    # The sidecar is the namespace init and survives broker restarts
+    # (previously the sidecar joined the broker's namespace and was
+    # kernel-killed whenever the broker exited — with a race on its
+    # own restart because it could not re-join a still-down broker).
+    # The broker is therefore NOT PID 1: control.py discovers its PID
+    # via /proc instead of hardcoding PID 1.
+    pid: "service:mosquitto-control"
+    depends_on:
+      mosquitto-control:
+        condition: service_started
     environment:
       # Used by the dynsec plugin on first start to seed the 'admin' client
       # (verified: Mosquitto 2.1 dynamic security docs).
@@ -87,12 +97,14 @@ services:
     build:
       context: ./broker-control
     restart: unless-stopped
-    depends_on:
-      mosquitto:
-        condition: service_started
-    # Shares PID namespace with the broker so `mosquitto_signal -p 1 ...`
-    # reaches the broker process without mounting docker.sock.
-    pid: "service:mosquitto"
+    # PID-namespace OWNER: the broker joins us via its `pid:` directive.
+    # Owning the namespace means a broker restart (SIGTERM to the broker
+    # process) only kills the broker — this sidecar keeps running to
+    # report status and orchestrate the revival. Runs tini as PID 1
+    # (signal forwarding + zombie reaping). If THIS container dies, the
+    # kernel tears down the namespace and kills the broker too — the
+    # sidecar is deliberately minimal so that path should never trigger.
+    # No depends_on: the broker depends on us (inverted), not vice versa.
     environment:
       # Minimal default token for the internal control API; in production
       # this MUST be replaced (it only guards a backend-net-only endpoint).