Backend

Feature modules, RuntimeConfig, SaltAPIClient, and schedulers.

The backend is a FastAPI application whose source lives in backend/src/halite/. Every feature is a self-contained Python package; shared infrastructure lives at the top level.

Feature module shape

Each feature follows the same internal layout:

halite/
  minions/
    __init__.py
    routes.py          # FastAPI router — thin, delegates to service
    service.py         # business logic + DB queries
    schemas.py         # Pydantic request/response models
    snapshot_model.py  # SQLAlchemy model for the snapshot table
    ingest.py          # reconciles snapshot table with Salt data
    scheduler.py       # periodic task owned by RuntimeConfig

Features that serve only live Salt calls (like run/ and keys/) omit ingest.py and scheduler.py. Features with additional sub-schemas (like jobs/) may add more files, but the core four — routes.py, service.py, schemas.py, and a model file — are always present.

When you add a new feature, keep it in this shape. The router is included in main.py::create_app; the scheduler (if any) is wired into RuntimeConfig.

Poll-into-DB features

Features that maintain a snapshot follow the poll-into-DB pattern:

1. A scheduler.py runs a background task at a configurable interval (set in AppSettings).
2. On each tick, ingest.py calls Salt-API and reconciles the snapshot table — inserting new rows, updating changed rows, deleting vanished ones.
3. service.py + routes.py serve the snapshot table directly; they do not call Salt live.

This makes reads fast even when Salt-API is slow or temporarily unavailable. The current poll-based features are:

Feature	Scheduler	Snapshot table
minions/	MinionStateScheduler	minion_snapshots
fleet/	FleetIngestScheduler	fleet index tables
inventory/	InventoryScheduler	inventory index
jobs/	JobsIndexScheduler	jobs_index

Event-stream ingestion

The activity/ module is the exception to the poll-into-DB shape. Instead of a scheduler that polls on a timer, it ingests Salt’s event bus live and uses a different set of files:

halite/
  activity/
    routes.py          # /api/activity list + /stream (SSE) + /widget-config
    service.py         # event persistence, list/filter queries, pruning
    api_schemas.py     # Pydantic response models
    models.py          # the activity_events table
    normalize.py       # raw Salt tag/data -> NormalizedEvent (or dropped)
    consumer.py        # EventStreamConsumer — long-lived /events tailer
    hub.py             # EventHub — in-process ring buffer + subscriber queues

The flow: EventStreamConsumer (started by RuntimeConfig when the stream is enabled) calls SaltAPIClient.stream_events(), runs each raw event through normalize_event(), persists the keepers via service.persist_event(), and publishes them to the EventHub. The SSE route subscribes to the hub to push events to browsers. See How Halite Works for the architecture and the single-worker constraint.

SaltAPIClient

SaltAPIClient (salt/client.py) is the single shared async client for salt-api’s rest_cherrypy endpoint. RuntimeConfig holds the one instance; routes never construct their own.

Authentication is lazy: the first real request triggers a login exchange. An asyncio.Lock guards the token refresh so N concurrent requests with an expired token trigger exactly one re-login, not N. The token is also refreshed proactively when fewer than 60 seconds remain on it.

Retry behaviour:

• 5xx responses are retried up to 3 times with exponential backoff (base 0.5 s, capped at 4 s).
• A 401 response triggers one forced re-login and one immediate retry.

Errors:

• SaltAPIError — raised for non-2xx responses from salt-api.
• SaltAPIUnavailable — raised when salt-api is unreachable (network, DNS, TLS failures after all retries are exhausted).

Convenience helpers cover all the Salt client types Halite uses:

Method	Salt client type	Purpose
local_call(target, fun, ...)	local	Execution module calls targeting minions
wheel_call(fun, ...)	wheel	Master-side operations (key management)
runner_call(fun, ...)	runner	Master runner calls (manage, jobs, cache)
list_connected_minions()	runner	Returns {minion_id: ip} via manage.present
stream_events()	—	Async generator yielding (tag, data) from the /events SSE bus (used by the activity consumer)

Wiring routes to the salt client

Routes that need Salt access use the salt_client_or_503 dependency from salt/deps.py:

from halite.salt.deps import salt_client_or_503, wrap_salt_errors
from halite.salt.client import SaltAPIClient
from fastapi import Depends

@router.post("/run")
async def run_command(
    client: SaltAPIClient = Depends(salt_client_or_503),
):
    try:
        result = await client.local_call(...)
    except Exception as exc:
        raise wrap_salt_errors(exc)

salt_client_or_503(request) reads request.app.state.runtime.salt and raises HTTP 503 if it is None (Salt unconfigured). wrap_salt_errors(exc) translates SaltAPIUnavailable into 503 and SaltAPIError into 502.

RBAC guards

Routes are protected by require_perm(verb, resource), a dependency factory in deps.py:

from halite.deps import require_perm

@router.delete("/keys/{minion_id}")
async def delete_key(
    minion_id: str,
    _: None = require_perm("delete", "key:*"),
):
    ...

require_perm checks whether the current user’s permissions_cache contains a (verb, resource_glob) pair that matches (verb, resource) using anchored shell globs with : as the namespace separator. A mismatch raises HTTP 403.

Every authorization decision is written to the audit log automatically.

Conventions

• All backend modules start with from __future__ import annotations.
• Tests live in backend/tests/ as test_<feature>_<layer>.py.
• New migrations go in backend/alembic/versions/ with a YYYYMMDD_NNNN_description filename.
• config.py’s Settings reads all infra config from environment variables at app-factory time — never at import time.

Related pages

• Architecture — the two settings systems and the RuntimeConfig hub
• RBAC — how permissions and roles work
• How Halite Works — poll-into-DB and graceful degradation