Running Entrosity Sphere

Entrosity Sphere runs next to Entrosity Hub, Axis and Edge on the same host. It is optional per host: the compose profile sphere holds its services, and the deploy scripts start them only when deploy/.env has SPHERE_ENABLED=true (Enabling Sphere on a host).

Part	Where
Web app (entrosity-sphere.frontend)	https://hub.entrosity.com/sphere/, served by Caddy from the web image
API (entrosity-sphere.backend, image ghcr.io/entrosity/sphere-backend)	https://hub.entrosity.com/sphere/api/…; Caddy strips /sphere, so the backend serves /api/v1 (browsers), /api/connector/v1 (connectors) and /api/releases/v1 (connector installers: CI uploads, expiring download links)
Media server (MediaMTX)	Connectors publish to rtsps://media.entrosity.com:8322 (a DNS-only host name of the same server, below); browsers negotiate through https://hub.entrosity.com/sphere/media/… (WHEP) and receive the media on port 8189 of the server's public address
Database	Its own PostgreSQL database (sphere), migrated by the backend
Sign-in	Entrosity Hub, product sphere

Services

deploy/docker-compose.prod.yml of entrosity-infra, profile sphere:

Service	Image	What it does
sphere-db-init	postgres:18-alpine	Creates the sphere database (owner rmm) once.
sphere-migrate	sphere-backend	Runs migrate up as the owner, and enables the sphere_app login with SPHERE_APP_DATABASE_PASSWORD.
sphere	sphere-backend	sphere-server serve, one replica: connectors hold a WebSocket to it and the web app an event stream. Public API on :8080 behind Caddy; the media server's hook on the internal listener :8084 (never routed by Caddy); metrics on :9094. Health check: sphere-server healthcheck. Memory limit SPHERE_MEMORY (512m), GOMEMLIMIT from SPHERE_GOMEMLIMIT (400MiB).
sphere-media	bluenviron/mediamtx, pinned by tag and digest	The media server (below). Publishes 8189/udp and 8189/tcp on the host. Memory limit SPHERE_MEDIA_MEMORY (1g). Starts after sphere is healthy.

The web service (Caddy) serves Sphere too:

Path or port	Goes to
/sphere/	The web app (from the sphere-frontend image, built into the web image)
/sphere/api/*	sphere:8080, with /sphere stripped
/sphere/media/*	sphere-media:8889 (WHEP), with /sphere/media stripped; the media server's session locations are rewritten back under /sphere/media
:8322 (TCP)	A layer-4 route: Caddy terminates TLS with SPHERE_MEDIA_DOMAIN's certificate and forwards plain RTSP to sphere-media:8554
SPHERE_MEDIA_DOMAIN (site)	Exists only so Caddy obtains the media host's certificate (HTTP-01 on port 80); it answers 404. Without Sphere the site is the placeholder http://media.invalid.

The backend binary is sphere-server:

sphere-server serve             run the HTTP API and job workers (default)
sphere-server migrate [cmd]     database migrations: up | down [N] | version | force N
                                (up also sets the sphere_app login password when
                                SPHERE_APP_DATABASE_PASSWORD is set)
sphere-server healthcheck       exit 0 when this host's server is ready (/readyz), for
                                container health checks
sphere-server version           print the version

On the Hub, the product sphere (roles tenant_admin, operator, viewer) is registered by the Hub's migration 0006 disabled and in beta, and enabled by migration 0007 (Rollout). The Hub must know Sphere's token: PLATFORM_PRODUCT_TOKENS includes sphere:<token>, the same value as SPHERE_PLATFORM_TOKEN. In production both come from PLATFORM_PRODUCT_TOKEN_SPHERE in deploy/.env.

Configuration reference

The backend is configured only through SPHERE_* environment variables (entrosity-sphere.backend/internal/config). Invalid values stop it at start with a list of every problem. The RETENTION section accepts one or two underscores: SPHERE_RETENTION_ALARM_DAYS = SPHERE_RETENTION__ALARM_DAYS.

Server

Variable	Default	Meaning
SPHERE_ENV	dev	dev, test or prod. prod refuses the development example secrets and logs JSON.
SPHERE_PUBLIC_URL	http://localhost:8083	Sphere's public address including the /sphere prefix the proxy strips (https://hub.entrosity.com/sphere). Connectors are given <SPHERE_PUBLIC_URL>/api/connector/v1/ws, and the install commands shown with new enrollment tokens use it as SERVER_URL.
SPHERE_HTTP_ADDR	:8083	API listener (production: :8080).
SPHERE_INTERNAL_ADDR	:8084	Internal listener for the media server's authentication hook (POST /media/auth). Never route it publicly.
SPHERE_METRICS_ADDR	:9094	Internal listener for Prometheus /metrics. Never expose it publicly; empty disables it.
SPHERE_LOG_LEVEL	info	debug, info, warn or error.
SPHERE_CORS_ORIGINS	http://localhost:5177	Allowed browser origins, comma separated, no wildcards or paths (production: https://hub.entrosity.com).
SPHERE_TRUSTED_PROXIES	none	CIDRs or IPs allowed to set X-Forwarded-For (production: the compose network 172.30.0.0/24).
SPHERE_API_RATE_PER_SECOND, SPHERE_API_RATE_BURST	20, 60	Web API rate limit per client IP (an office behind one NAT address shares it).
SPHERE_ENROLL_RATE_PER_MINUTE	60	Connector enrollments per client IP.
SPHERE_CONNECTOR_DOWNLOAD_URL	none	Where the connector MSI can be downloaded; shown with new enrollment tokens while no connector release is stored (Connector releases).

Database

Variable	Default	Meaning
SPHERE_DATABASE_URL	none (required)	Connection URL. Production connects as sphere_app.
SPHERE_DATABASE_ROLE	sphere_app	Role the pool switches to after connecting (SET ROLE), so row-level security applies even when connecting as the owner. Empty when the login already is the application role and cannot switch.
SPHERE_DATABASE_MAX_CONNS	20	Pool size.
SPHERE_DATABASE_STATEMENT_TIMEOUT	30s	Upper bound for any statement.
SPHERE_APP_DATABASE_PASSWORD	none	Read by migrate up: enables the sphere_app login with this password.

Secrets

Variable	Meaning
SPHERE_JWT_SECRET	Required. At least 32 bytes, raw or base64. Signs the one-minute live-update stream tokens (POST /auth/sse-token) and the five-minute media tokens browsers play video with. Users' access tokens come from the Hub.
SPHERE_JWT_SECRET_OLD	The previous secret during a rotation (still verifies).
SPHERE_CREDENTIALS_KEY	Required. Exactly 32 bytes, base64: encrypts the NVRs' passwords in the database (AES-256-GCM; the NVR's id is bound to its ciphertext, so a ciphertext copied to another NVR does not decrypt).
SPHERE_CREDENTIALS_KEY_OLD	The previous key during a rotation: it still decrypts; new and changed passwords use the current key.

With SPHERE_ENV=prod, values containing dev-only (the development examples) are refused.

Keep the credentials key

Without SPHERE_CREDENTIALS_KEY (or with a different one) no NVR password can be decrypted: connectors cannot sign in to any NVR until every password has been entered again. Keep a copy of deploy/.env with the backups.

Sign-in on Entrosity Hub

Variable	Default	Meaning
SPHERE_PLATFORM_URL	none (required)	The Hub's public origin (https://hub.entrosity.com): the issuer of product tokens and where browsers sign in.
SPHERE_PLATFORM_INTERNAL_URL	none (required)	The Hub's internal API, reached directly and not through the proxy (compose: http://platform:8081): signing keys and the access snapshot.
SPHERE_PLATFORM_TOKEN	none (required)	Sphere's bearer token on the Hub's internal API, at least 32 characters; sphere:<token> in the Hub's PLATFORM_PRODUCT_TOKENS.
SPHERE_PLATFORM_SYNC_INTERVAL	30s	How often Sphere pulls users, organizations and roles from the Hub (at least 1s).

Media and streams

Variable	Default	Meaning
SPHERE_MEDIA_PUBLISH_URL	rtsp://localhost:8554	Where connectors publish: rtsp(s)://host:port, no path (production: rtsps://media.entrosity.com:8322). Sent to the connector with every stream start.
SPHERE_MEDIA_WHEP_URL	/sphere/media	The base browsers play from, <base>/<path>/whep: a path on Sphere's origin or an absolute URL, without a trailing slash.
SPHERE_MAX_STREAMS_PER_CONNECTOR	64	Streams (live and playback) open at once per connector: the site's upload. 64 fills an 8×8 grid.
SPHERE_MAX_MAIN_STREAMS_PER_NVR	4	Full-resolution live streams per NVR.
SPHERE_MAX_PLAYBACKS_PER_NVR	4	Playbacks per NVR (NVRs serve few at once).
SPHERE_LIVE_IDLE_GRACE	5m	How long a live sub stream nobody watches keeps running, so switching back to the camera skips the start on the NVR. A Go duration, at least 10s (was a fixed 30 seconds).
SPHERE_LIVE_IDLE_GRACE_MAIN	2m	The same for main streams. At least 10s.

Idle streams cost the site's upload bandwidth while they run, and count towards the limits. At a limit (SPHERE_MAX_STREAMS_PER_CONNECTOR or SPHERE_MAX_MAIN_STREAMS_PER_NVR), the idle live stream unwatched the longest is stopped to make room; streams with viewers are never stopped, and playbacks are never idle (they stop when their viewer leaves). Only when no idle stream can make room does a viewer's request answer 409 stream_limit. The connector has its own cap (SPHERE_CONNECTOR_MAX_STREAMS, default 64; Sphere connector).

Connector updates

Variable	Default	Meaning
SPHERE_RELEASE_SIGNING_KEY	none	Signs connector releases: an Ed25519 key, base64 of the 32-byte seed or the 64-byte private key. Its public half is the RELEASE_PUBLIC_KEY connectors are built with. Empty disables uploads and self-updates (uploads answer 503 releases_disabled). An invalid value stops the server.
SPHERE_RELEASE_TOKEN	none	CI's bearer token for uploading connector builds (POST /api/releases/v1/connector), at least 32 characters. Empty: every upload answers 401.
SPHERE_CONNECTOR_UPDATE_CHANNEL	stable	What connectors are updated to (and what Download connector offers): stable (tagged releases only) or dev (development builds and tagged releases). The production compose file sets dev while Sphere is in beta.

See Connector releases.

Retention windows

Variable	Default	Meaning
SPHERE_RETENTION_ALARM_DAYS	365	Alarm log.
SPHERE_RETENTION_JOB_DAYS	90	Finished connector jobs. Tenants may override it (7–730).
SPHERE_RETENTION_AUDIT_DAYS	365	Audit log. Tenants may override it (30–3650).

0 or unset keeps the default.

Host settings

In production the compose file fixes most of the above (SPHERE_PUBLIC_URL=https://${PLATFORM_DOMAIN}/sphere, SPHERE_MEDIA_PUBLISH_URL=rtsps://${SPHERE_MEDIA_DOMAIN}:8322, the listeners, the trusted proxies). deploy/.env (deploy/.env.prod.example lists them) sets the rest:

Variable	Meaning
SPHERE_ENABLED	true runs the sphere profile. Default false.
SPHERE_APP_DATABASE_PASSWORD	The sphere_app login's password. Required when enabled.
SPHERE_JWT_SECRET, SPHERE_JWT_SECRET_OLD	As above. The first is required when enabled.
SPHERE_CREDENTIALS_KEY, SPHERE_CREDENTIALS_KEY_OLD	As above. The first is required when enabled.
PLATFORM_PRODUCT_TOKEN_SPHERE	Sphere's token on the Hub: becomes SPHERE_PLATFORM_TOKEN and the sphere: entry of the Hub's PLATFORM_PRODUCT_TOKENS. Required when enabled.
SPHERE_MEDIA_DOMAIN	The media host name connectors publish video to (media.entrosity.com): an A record pointing straight at this server, DNS only. Used for SPHERE_MEDIA_PUBLISH_URL and given to the web service, which obtains its certificate. Required when enabled.
SPHERE_MEDIA_PUBLIC_IP	The host's public IP address, announced to browsers as the address of the media (webrtcAdditionalHosts). Required when enabled.
SPHERE_MAX_STREAMS_PER_CONNECTOR, SPHERE_MAX_MAIN_STREAMS_PER_NVR, SPHERE_MAX_PLAYBACKS_PER_NVR	Stream limits (64, 4, 4).
SPHERE_LIVE_IDLE_GRACE, SPHERE_LIVE_IDLE_GRACE_MAIN	Idle grace periods (5m, 2m).
SPHERE_RELEASE_SIGNING_KEY, SPHERE_RELEASE_TOKEN	Connector self-updates, as above. Optional: empty turns them off (Enable connector self-updates).
SPHERE_CONNECTOR_UPDATE_CHANNEL	As above; default dev on the host while Sphere is in beta.
SPHERE_CONNECTOR_DOWNLOAD_URL, SPHERE_LOG_LEVEL	As above.
SPHERE_BACKEND_IMAGE	Default ghcr.io/entrosity/sphere-backend.
SPHERE_MEMORY, SPHERE_GOMEMLIMIT, SPHERE_MEDIA_MEMORY	Memory limits: 512m, 400MiB, 1g.

Generating the secrets

openssl rand -base64 32   # SPHERE_JWT_SECRET, SPHERE_CREDENTIALS_KEY (exactly 32 bytes)
openssl rand -hex 32      # SPHERE_APP_DATABASE_PASSWORD, PLATFORM_PRODUCT_TOKEN_SPHERE, SPHERE_RELEASE_TOKEN

The release signing key is a key pair: Enable connector self-updates.

The database password ends up inside a connection URL, so use a value without /, + or = (hex). The product token must be at least 32 characters.

Ports and firewall

Besides 443, Sphere needs these ports open inbound in the host's firewall:

Port	Service	For
80/tcp	web (Caddy)	Must reach the server directly on SPHERE_MEDIA_DOMAIN: Caddy obtains the media host's certificate with the HTTP-01 challenge.
8322/tcp	web (Caddy)	Connectors publish video to SPHERE_MEDIA_DOMAIN: RTSP over TLS, terminated by Caddy with that name's certificate (the connector sends it as SNI).
8189/udp and 8189/tcp	sphere-media	WebRTC media to browsers (UDP preferred, TCP when UDP is blocked).

Browsers negotiate WebRTC through /sphere/media on 443 and then receive the media on 8189 at SPHERE_MEDIA_PUBLIC_IP. There is no TURN server: a viewer whose network blocks outbound 8189 (UDP and TCP) sees no video.

Behind Cloudflare (or another proxy)

hub.entrosity.com is behind Cloudflare's proxy, which forwards HTTP(S) on 443 but not raw TCP on 8322, and not UDP or TCP on 8189. So:

• Connectors publish video to a separate, DNS-only host name, SPHERE_MEDIA_DOMAIN (media.entrosity.com): an A record pointing at the server with the proxy switched off (Cloudflare's grey cloud), like the files host. Caddy obtains its certificate over port 80 and presents it on 8322.
• The connector's control connection (WebSocket, jobs, alarms) stays on hub.entrosity.com:443, and so does the browsers' WebRTC signalling (/sphere/media, WHEP), which the proxy forwards like any HTTPS request.
• The WebRTC media goes straight from the server's SPHERE_MEDIA_PUBLIC_IP on 8189 to the browser, never through the proxy.

The server's public IP address therefore becomes visible (through the media host's DNS record and the WebRTC candidates), as it already is through the files host.

The media server

sphere-media runs MediaMTX with deploy/mediamtx.yml:

• Every action is decided by Sphere: authMethod: http posts each publish and read to http://sphere:8084/media/auth. The hook allows:
  • publish over RTSP with the connector's id as user and its key as password, and only under its own tenant's prefix t/<tenant id>/;
  • read over WebRTC (or HLS) with a media token naming exactly the path (as a bearer token, ?jwt=, or the password of basic credentials);
  • nothing else (MediaMTX's API, metrics and pprof are excluded from the hook and stay inside the compose network or off).
• Paths: only t/<tenant>/live/<camera>/<main|sub> and t/<tenant>/pb/<session> exist.
• RTSP on :8554, TCP only (Caddy proxies bytes, not UDP), no encryption (TLS ends at Caddy), basic authentication. A connector restarting a stream takes over from its previous session (overridePublisher). Nothing is recorded.
• WebRTC on :8889 (WHEP, behind Caddy) with the media on :8189 UDP and TCP; only SPHERE_MEDIA_PUBLIC_IP is announced (webrtcIPsFromInterfaces: false).
• RTMP, SRT, MoQ, HLS, playback server and metrics are off.

Upgrading MediaMTX

The image is pinned by tag and digest (bluenviron/mediamtx:1.21.1@sha256:…) in both deploy/docker-compose.prod.yml and the development deploy/docker-compose.yml. To upgrade:

1. Read the release notes between the two versions.
2. Check every key of deploy/mediamtx.yml against the new version's reference configuration, in particular authMethod, authHTTPAddress, authHTTPExclude, rtspTransports, rtspEncryption, rtspAuthMethods, webrtcAdditionalHosts, webrtcIPsFromInterfaces, webrtcLocalUDPAddress, webrtcLocalTCPAddress, webrtcICEServers2, overridePublisher, and the protocols that must stay off.
3. Change the tag and the digest in both compose files.
4. Test locally with a connector (the simulator will do): a stream publishes through TLS on 8322 of the media host, plays in the browser through /sphere/media, and a read without a token, or with the token of another path, is refused.

Enabling Sphere on a host

1. Create the DNS-only A record of the media host (media.entrosity.com → the server's public IP, proxy off) and wait until it resolves to the server: Caddy needs it to obtain the certificate.
2. Generate the secrets and set them, PLATFORM_PRODUCT_TOKEN_SPHERE, SPHERE_MEDIA_DOMAIN and SPHERE_MEDIA_PUBLIC_IP in deploy/.env.
3. Open 8322/tcp and 8189/udp+tcp in the host's firewall (and 80/tcp to the media host).
4. Set SPHERE_ENABLED=true.
5. Deploy (the next deploy run, or deploy/scripts/upgrade.sh).

With SPHERE_ENABLED=true, deploy/scripts/lib.sh adds --profile sphere to every compose command, and refuses to run while SPHERE_APP_DATABASE_PASSWORD, SPHERE_JWT_SECRET, SPHERE_CREDENTIALS_KEY, PLATFORM_PRODUCT_TOKEN_SPHERE, SPHERE_MEDIA_DOMAIN or SPHERE_MEDIA_PUBLIC_IP is empty. upgrade.sh then pulls the Sphere images and, after the Hub, runs sphere-migrate, restarts sphere (and waits until it is healthy) and then sphere-media. The deploy workflow's final check also waits for /sphere/api/v1/healthz on hosts with SPHERE_ENABLED=true.

Rollout

Sphere reaches users in four steps:

1. Registered, disabled. The Hub's migration 0006 registers the product sphere disabled and in beta: nobody sees it.
2. Deployed. The sphere-backend and sphere-frontend main images exist, the media host's DNS-only record resolves, the host is set up as above with SPHERE_ENABLED=true, and the deploy is done.
3. Enabled, in beta. The Hub's migration 0007 enables the product. It stays in beta: only platform admins see and open it, and they can enable it for organizations.
4. Released. A platform admin chooses Release to organizations (Products in beta).

Order of the deploy

The deploy workflow pulls sphere-backend:main and builds the web image with sphere-frontend:main on every run, whether the host enables Sphere or not. Push the entrosity-infra change that adds Sphere only after both images have been published, or every deploy fails (Continuous deployment). Migration 0007 belongs in a Hub release deployed after step 2: before it, the launcher would link to a Sphere that does not answer.

Connector releases

Sphere keeps the connector's installers and rolls them out: CI uploads every build, Sphere signs it, connectors that can update themselves are offered the newest one, and tenant admins download it from Download connector (Connectors → Updates).

Enable connector self-updates

1. Generate the Ed25519 key pair once (OpenSSL 3):

   openssl genpkey -algorithm ed25519 -out sphere-release.pem
   openssl pkey -in sphere-release.pem -outform DER | tail -c 32 | base64          # SPHERE_RELEASE_SIGNING_KEY (the seed)
   openssl pkey -in sphere-release.pem -pubout -outform DER | tail -c 32 | base64  # RELEASE_PUBLIC_KEY
   openssl rand -hex 32                                                            # SPHERE_RELEASE_TOKEN

   Keep the seed as secret as the other keys, then delete sphere-release.pem.

2. In the host's deploy/.env set SPHERE_RELEASE_SIGNING_KEY (the seed) and SPHERE_RELEASE_TOKEN, and optionally SPHERE_CONNECTOR_UPDATE_CHANNEL (default dev on the host).

3. In the entrosity-sphere-connector repository set the secrets RELEASE_PUBLIC_KEY (the public key, compiled into the connector) and SPHERE_RELEASE_TOKEN (the same token), and the variable RELEASE_PUBLISH_ENABLED=true. SPHERE_RELEASE_API (a variable) changes the target, default https://hub.entrosity.com/sphere/api/releases/v1 (CI workflows).

4. Redeploy (the next deploy run, or deploy/scripts/upgrade.sh). The next connector build is uploaded and offered.

5. Connectors installed before this were built without the public key and cannot update themselves: install the new version on each of them once, from Download connector (Connectors → Updates).

Keep the signing key

Connectors accept only releases signed with the key their build contains. With another key (a lost or replaced seed), every update fails with signature_invalid until each connector has been reinstalled by hand with a build that contains the new public key.

Publishing

CI uploads each MSI with scripts/publish-release.sh of the connector repository:

Build	Version	Channel
Tag vX.Y.Z (release.yml)	X.Y.Z	stable
Tag vX.Y.Z-suffix	X.Y.Z-suffix	dev
Every push to main (dev-release.yml)	0.0.<build>-dev.<commit>	dev

curl -X POST -H "Authorization: Bearer $SPHERE_RELEASE_TOKEN" \
  -H "Content-Type: application/octet-stream" --data-binary @sphere-connector.msi \
  "https://hub.entrosity.com/sphere/api/releases/v1/connector?version=0.2.0&channel=stable&notes=Release%20v0.2.0"

The body is the raw MSI (at most 64 MiB). Sphere signs its manifest (proto.ReleaseManifest, component sphere-connector: version, SHA-256 and size) with SPHERE_RELEASE_SIGNING_KEY and stores it in the connector_releases table, keeping the newest ten releases.

Answer	Meaning
201	Stored: {id, version, channel, sha256, size_bytes, created_at}.
401 unauthorized	Wrong or missing token, or SPHERE_RELEASE_TOKEN is not set.
409 release_exists	This version is already published (the script treats it as done: re-runs of a workflow).
422 validation	Not a semantic version, a channel other than stable or dev, or an empty or larger than 64 MiB body.
503 releases_disabled	SPHERE_RELEASE_SIGNING_KEY is not set.

Offering updates

Sphere offers the newest release of SPHERE_CONNECTOR_UPDATE_CHANNEL (stable: stable releases; dev: dev and stable) when a connector says hello and every 5 minutes (releases.rollout), to online connectors that:

• announce the update capability (builds with RELEASE_PUBLIC_KEY);
• run an older version (a connector whose version is not a semantic version, such as a local dev build, is never offered one);
• have no update_agent job open, and were not offered the same version within the last hour (connectors.update_version, connectors.update_requested_at).

The offer is an update_agent job (30 minutes timeout) with download links valid 2 hours, and the installer of the running version as the rollback when Sphere still stores it. What the connector does: Sphere connector → Self-update.

Download links (GET /api/releases/v1/connector/{releaseID}/{file}?t=…) need no sign-in: t is an expiry and an HMAC keyed from the signing key, and a wrong or expired link answers 404. Download connector and new enrollment tokens get links valid one hour.

Background jobs

The backend runs its jobs in PostgreSQL (River):

Job	When	What
realtime.sweep	Every 30 seconds	Marks connectors silent for 3 minutes offline (their NVRs become unknown, their streams not live); expires overdue connector jobs and resends unacknowledged ones (after 60 seconds).
streams.sweep	Every 10 seconds	Drops viewers without a keepalive for 45 seconds; stops live streams nobody has watched for their idle grace period (SPHERE_LIVE_IDLE_GRACE, 5 minutes, for sub streams; SPHERE_LIVE_IDLE_GRACE_MAIN, 2 minutes, for main streams; a sphere.stream.stop job) and playbacks as soon as their viewer is gone (sphere.playback.stop, no grace period); forgets stopped streams after 10 minutes.
nvr.reconcile	Every 2 minutes	Sends sphere.nvr.apply again for NVRs whose connector is online but does not hold the wanted state (connected but reported unknown or disconnected, or with older credentials; disconnected but still signed in), unchanged for 2 minutes and without an open apply job.
releases.rollout	Every 5 minutes	Offers the newest connector release to online connectors that update themselves (Offering updates). Does nothing without SPHERE_RELEASE_SIGNING_KEY.
alarms.partitions	Daily	Creates the monthly alarm partitions two months ahead.
retention.cleanup	Daily	See Retention.

Besides, every status report of a connector is compared with what viewers want: a stream it publishes that nobody wants (or that was stopped) is stopped again, and a wanted live stream it has not reported for 45 seconds is started again, even one that was live (for example after the connector restarted). A playback it no longer reports has reached its end: it ends, and is not started again.

How quickly a camera starts

Measured in production on 2026-10-09 (sub streams, H.265, Dahua): a camera nobody was watching took 4.1–5.3 seconds to show, one whose stream was still running 1.6–2.8 seconds. The parts:

Step	Time
The connector opens the stream on the NVR and publishes it (only when the stream is not running)	1.9–2.6 s
The viewer waits for the camera's next keyframe	1–2 s with Dahua's default 2-second interval
WebRTC set-up in the browser	about 0.6 s

What shortens them:

• Idle grace periods keep unwatched streams running, so switching back skips the start on the NVR (SPHERE_LIVE_IDLE_GRACE, SPHERE_LIVE_IDLE_GRACE_MAIN, above).
• A stream counts as live as soon as the connector's start job succeeds, not at its next status report. The browser prepares the WebRTC connection (offer, ICE) while the camera starts, and the connector opens its connection to the media server (TCP, TLS) while the NVR answers instead of after it.
• Optimise for live view on an NVR's page shortens the cameras' sub-stream keyframe interval to 1–4 seconds (NVRs).

The stream lifecycle in detail: Sphere API → Watching video, Sphere connector protocol.

Retention

A daily retention.cleanup job deletes old data in batches of 10,000 rows:

Data	Kept
Alarm log	SPHERE_RETENTION_ALARM_DAYS, for every tenant. Alarms are stored in monthly partitions; a month is dropped as a whole once it is entirely older than the window, so alarms are kept between the window and the window plus one month.
Finished connector jobs	The tenant's override (settings.retention.job_days, 7–730), else SPHERE_RETENTION_JOB_DAYS.
Audit log	The tenant's override (settings.retention.audit_days, 30–3650), else SPHERE_RETENTION_AUDIT_DAYS.
Enrollment tokens	30 days after they expired or were revoked.
Hub sessions ended, used step-up tokens	An hour after they ended; once expired.

Tenant admins set the overrides in the tenant's settings (PATCH /tenants/{tenantID}, Sphere API). sphere_retention_rows_deleted_total{category} counts what was removed.

Database and migrations

Sphere has its own database. Migrations are embedded in the binary (entrosity-sphere.backend/db/migrations) and applied by migrate up:

Migration	Adds
0001_init	Sphere's copy of the Hub's tenants, users and roles (tenants, users, tenant_memberships, Hub sync state, ended sessions, used step-up tokens), sites, the append-only audit_log, enrollment_tokens, connectors (with the alarm cursor alarms_last_seq), jobs, and row-level security with the sphere_app role.
0002_video	nvrs (encrypted credentials with a version, wanted state and reported status), cameras, streams and stream_viewers, the monthly-partitioned alarms log with its partition functions, and saved_views.
0003_view_splits	Saved views take every grid split: layout 1, 4, 6, 8, 9, 16, 25, 36 or 64 (was 1, 4, 9 or 16). Down: the new splits become 9 or 16, keeping at most 16 cameras.
0004_connector_releases	connector_releases (the installers with their version, channel, SHA-256, size, signature and notes; global rows, readable by every tenant, written only by global operations), and the connectors' update_version and update_requested_at (the last update offered).

• Tenant isolation: every tenant table has a row-level security policy; the server runs as sphere_app (NOBYPASSRLS). References between tenant tables are composite (tenant_id, id) keys, so a cross-tenant reference is impossible.
• Append-only: sphere_app cannot change or delete audit_log (retention purges through a dedicated function), and may only acknowledge alarms (it can update acked_at and acked_by, nothing else; retention drops partitions).
• NVR passwords are stored only encrypted (credentials_enc).
• Alarms whose time falls outside the monthly partitions land in alarms_default and are moved when their month's partition is created.

Backups and restore

deploy/scripts/backup.sh dumps the sphere database whenever it exists, next to the others: backups/db/sphere-<timestamp>.dump (pg_dump -Fc, checked by reading it back), pruned with the other dumps after the retention days.

deploy/scripts/restore.sh with SPHERE_ENABLED=true stops sphere and sphere-media, restores the sphere-… dump of the same backup run when there is one (it recreates the database and the sphere_app role), runs sphere-migrate, and starts sphere and sphere-media again.

The dump holds the NVR passwords encrypted with SPHERE_CREDENTIALS_KEY: a restore needs the same key. Alarms stored after the backup was taken are not in the restored database, and connectors do not send them again: they drop alarms once the server has acknowledged them.

Monitoring

• GET /healthz (liveness) and GET /readyz (readiness: the database answers; it also reports whether the copy of the Hub's data is current, stale after 5 minutes, without failing) on the API listener. Publicly: https://hub.entrosity.com/sphere/api/v1/healthz.
• Prometheus metrics on SPHERE_METRICS_ADDR (:9094), among them sphere_ws_connections (connected connectors), sphere_jobs_total{type,status}, sphere_job_dispatch_seconds, sphere_http_request_duration_seconds, sphere_platform_syncs_total, sphere_platform_sync_age_seconds, sphere_sse_subscribers, sphere_retention_rows_deleted_total, sphere_river_queue_depth and sphere_db_pool_connections.
• Logs are JSON in prod. MediaMTX logs at warn; a refused publish or read is logged by the backend at debug (media access refused).

Local development

The development stack of entrosity-infra runs Sphere next to the Hub (Development setup):

1. deploy/.env.example has a Sphere block (SPHERE_* on :8083, hook on :8084, metrics on :9094, development secrets, SPHERE_MEDIA_PUBLISH_URL=rtsp://localhost:8554, SPHERE_PUBLIC_URL=http://localhost:5175/sphere), and the Hub's PLATFORM_PRODUCT_TOKENS includes the sphere token.

2. Create the database once: docker compose -f deploy/docker-compose.yml exec postgres createdb -U rmm sphere.

3. Start the media server, the production configuration pointed at the backend on the host:

   docker compose -f deploy/docker-compose.yml --profile sphere up -d sphere-media

   It listens on 8554 (RTSP, plain), 8889 (WHEP) and 8189 (media), announces 127.0.0.1, and asks http://host.docker.internal:8084/media/auth.

4. In entrosity-sphere.backend: make migrate, then make dev (API on :8083).

5. In entrosity-sphere.frontend: its dev server on :5177.

6. Open http://localhost:5175/sphere/. The Hub's dev server proxies /sphere/api to :8083 (prefix stripped), /sphere/media to :8889 (keeping WHEP session locations under /sphere/media, as Caddy does) and /sphere to :5177.

7. For video, run a connector with the simulator: sphere-connector enroll --server http://localhost:5175/sphere --token <token>, then make dev in entrosity-sphere-connector (Trying Sphere with the simulator).