This commit is contained in:
2026-11-15 17:43:04 +01:00
parent 39ba6bd7da
commit 6aec4f6082
4 changed files with 310 additions and 0 deletions

68
docs/ARCHITECTURE.md Normal file
View File

@@ -0,0 +1,68 @@
# Controller architecture
## Alert ingestion
Suricata is the client of an AF_UNIX stream socket bound by `AlertSocket` in
`alertsock.py`. The reader frames newline-delimited JSON, accepts alert records,
and places them on a bounded queue. Oversized or malformed records and queue
overflow are counted. A socket-path lock prevents competing controllers from
replacing the listener, even when they use different state databases.
The queue wakes the main loop independently of file polling. The reader does
not run policy or wait for router commits. During graceful shutdown, intake is
stopped and accepted queued alerts are processed before SQLite closes.
The normal EVE file contains stats. `EveTailer` in `eve.py` follows complete
records, preserves partial writes and drains the old inode during rotation.
When socket mode is enabled, file alerts are ignored to avoid counting the same
alert through two outputs.
For file-alert mode, set `alert_socket.path: ""` and configure Suricata to write
alerts into the EVE file. Checkpoints advance per delivered record. File-hit
identities include record content, inode, offset, SID and candidate address to
prevent replay from inflating threshold counts. Processing is at-least-once;
a restart cannot automatically recover unread records in an older rotated file.
## Policy and state
`policy.py` reads `leaf_action` metadata, applies SID restrictions, rejects
protected addresses, and evaluates threshold or immediate mode. `state.py`
stores hit windows, desired blocks, expiry times, applied flags, file checkpoints
and pending router-save status in SQLite WAL mode.
The desired block row is persisted before API submission. Qualifying alerts
refresh expiry. The active-address cap bounds the number of desired blocks.
Source and destination actions share a router group; `kind` records the initial
reason rather than choosing a separate group.
## Router reconciliation
A separate worker in `main.py` serializes router I/O using its own SQLite
connection. Each pass reads actual membership, computes changes, commits a batch,
and reads membership back before marking decisions applied or removals complete.
A timeout may still mean a commit succeeded. The following pass reads state
before retrying. Expiry deletion uses a timestamp condition so that a fresh
alert received during a slow request does not lose its refreshed decision.
Failures retain desired state for retry.
`vyos_client.py` handles multipart API requests and response validation. TLS can
use normal CA verification or a certificate fingerprint checked before sending
credentials. Pinned pool classes are private to each client. Environment proxies
are disabled for the API session.
## Metrics and persistence
Stats records update `metrics.py` counters without entering policy or changing
router state. Reports include capture/decoder deltas and ingress counters.
`decisions.jsonl` records decisions and verified enforcement changes; `capture.log`
contains periodic metric windows. Both logs retain the newest size-bounded archives.
A zero controller queue-drop count covers that queue only. Suricata can lose
alerts before delivery under send-buffer pressure or disconnection. The socket
queue is volatile on crash. Durable decisions can be reconciled after restart,
but alerts that never reached SQLite cannot be reconstructed from the stats file.
Dry-run creates temporary state and logs, disables socket ownership, avoids the
API, and reads the configured file from its beginning. Use an alert fixture for
policy replay; the normal stats-only file contains no alert history.

144
docs/INSTALLATION.md Normal file
View File

@@ -0,0 +1,144 @@
# Installation and operations
## Prerequisites
Use a Linux sensor with systemd, Python 3, `requests`, `PyYAML`, Suricata 7,
OpenSSH and tcpdump. Live tests also require Python 3 and ping on the test hosts.
The supplied service uses `/usr/bin/python3`; install its dependencies for that
interpreter. On Debian-based sensors, use the `python3-requests` and
`python3-yaml` packages. A separate virtual environment can be used for local
tests with `controller/requirements.txt`.
The supplied lab configuration uses sensor capture interface `eth1`, P2P
address `10.0.249.1/31`, VyOS API `https://10.0.249.0:8443`, and address group
`IDS-BLOCK`. Test hosts are `172.19.32.3` and `172.19.32.5`. These are example
site settings; inspect and adapt them for another network.
Provision mirroring, API access and firewall hooks before starting the sensor.
Both source and destination drop hooks must be reachable before accept rules
that would bypass enforcement. The controller verifies hooks but does not
provision routing or mirroring. The file in `vyos/` is an illustrative template,
not a complete router configuration.
## Configuration and credentials
The installer preserves an existing `/etc/leaf-controller/config.yaml`. For a
new installation, it seeds that file from `config/prototype-vyos.yaml`.
`controller/config.yaml` is a generic example with different API/TLS settings;
it is not the installer's lab default.
Review these settings before installation:
- API URL, timeout, retry count and TLS certificate fingerprint or CA bundle.
- Protected networks, allowed SIDs, source actions, threshold, TTL and block cap.
- Capture interface in the Suricata configuration and service unit.
- Socket, EVE, SQLite and audit paths.
The lab API uses a certificate fingerprint because its self-signed certificate
has no matching IP subject alternative name. Verify the fingerprint through a
trusted channel before setting `vyos.tls_cert_pin`. For a CA-issued certificate,
configure the appropriate CA bundle and hostname. Keep TLS verification enabled.
Create `/etc/leaf-controller/env`, owned by root with mode `0600`, containing:
```text
LEAF_VYOS_KEY='replace-with-existing-api-key'
```
Use a root editor or secret-management mechanism. Do not commit the real value
or place it in command arguments. The credential file is read by systemd.
The router address group must be dedicated to this controller. With
`adopt_foreign_members: false`, foreign members are removed during reconciliation.
The optional adoption setting imports eligible foreign members only at startup.
## Log storage and installation
The installer requires `/var/log/suricata-leaf` to be on tmpfs. On a new sensor,
review and run the provisioning script before installing:
```bash
sudo env TMPFS_SIZE=8G bash scripts/provision-ids-tmpfs.sh
sudo bash scripts/install-sensor.sh
```
Allow enough memory and swap headroom for the configured tmpfs cap and Suricata.
The provisioning script writes fstab and runs `mount -a`; review existing mounts
first. Mounting over a nonempty directory hides its previous contents. EVE files
on tmpfs are lost at reboot. SQLite and controller audit remain on persistent disk.
The installer stops the sensor/controller for upgrade, backs up existing files,
units and SQLite under `/var/backups/leaf-proto/<timestamp>/`, installs the code
and helpers, and enables these units:
- `leaf-controller.service`
- `suricata-leaf.service`
- `leaf-eve-rotate.timer`
It restores the previous deployment on installation failure. Backups can contain
credentials and must stay private. The controller runs as `leafctrl`; Suricata
runs as root with group `leafctrl`. The controller reports readiness after its
socket is bound, allowing the sensor to start afterward.
## Status and diagnostics
```bash
sudo systemctl status leaf-controller suricata-leaf leaf-eve-rotate.timer
sudo journalctl -u leaf-controller -f
sudo python3 /opt/leaf-controller/tools/vyos_smoke.py
sudo python3 /opt/leaf-controller/tools/vyos_ctl.py members
```
The smoke check reads API information, members and enforcement hooks.
`vyos_show.py` reads scoped configuration and redacts sensitive fields.
These diagnostic scripts do not write router configuration.
| Resource | Installed path |
|---|---|
| Controller | `/opt/leaf-controller/controller/leaf_controller/` |
| Configuration | `/etc/leaf-controller/config.yaml` |
| Credential | `/etc/leaf-controller/env` |
| SQLite WAL state | `/var/lib/leaf-controller/state.sqlite` |
| Decision audit | `/var/log/leaf-controller/decisions.jsonl` |
| Capture metrics | `/var/log/leaf-controller/capture.log` |
| EVE stats | `/var/log/suricata-leaf/eve.json` |
| Alert socket | `/run/suricata/alerts.sock` |
The retention timer checks each minute after an initial two-minute delay. It
renames EVE hourly or at 64 MiB, asks Suricata to reopen it, and retains six
archives. A startup grace period prevents signalling a newly started engine.
Retention can overshoot between timer runs; tmpfs capacity is the hard limit.
Audit defaults to 16 MiB with three archives; capture metrics default to 10 MiB
with six archives.
`save_config: false` leaves transient membership out of router boot configuration.
SQLite supplies desired state when the router returns. If saving is enabled,
failed saves remain dirty in SQLite and are retried.
## Credential rotation
Identify the controller's existing key ID before rotating:
```bash
sudo python3 /opt/leaf-controller/tools/rotate-vyos-key.py --old-id <controller-key-id>
```
The tool requires the selected ID to match the current credential. It creates
and verifies a candidate, saves both keys, switches the local environment file,
restarts the service, then retires only the selected old ID and saves again.
Other API users' IDs are preserved.
Switch failures restore the local credential. A failure after retirement requires
inspection before retrying; deleting arbitrary IDs is not a recovery procedure.
No credential values are printed by the tool.
## Rollback
Stop Suricata, the controller and the rotation timer. Restore the selected file
archive and its matching SQLite backup, reload systemd, and restart the previous
units. Keep SQLite closed while replacing its files. When reverting a schema
migration, restore the matching database as well as the code.
Do not clear the live database or shared router configuration as an upgrade or
test shortcut. Live tests operate on existing routes and enforcement hooks;
uplink, OSPF and static-route changes are outside their scope.

11
docs/REFERENCES.md Normal file
View File

@@ -0,0 +1,11 @@
# References
The lab uses Suricata 7 and VyOS 2026.03. Configuration and API schemas differ
between releases; use documentation matching the installed software.
- [VyOS HTTP API](https://docs.vyos.io/en/rolling/automation/vyos-api.html)
- [VyOS HTTPS service and API configuration](https://docs.vyos.io/en/rolling/configuration/service/https.html)
- [VyOS Ethernet interfaces and mirroring](https://docs.vyos.io/en/rolling/configuration/interfaces/ethernet.html)
- [VyOS firewall groups](https://docs.vyos.io/en/rolling/configuration/firewall/groups.html)
- [Suricata EVE output](https://docs.suricata.io/en/suricata-7.0.10/output/eve/eve-json-output.html)
- [Suricata EVE JSON format](https://docs.suricata.io/en/suricata-7.0.10/output/eve/eve-json-format.html)

87
docs/TESTING.md Normal file
View File

@@ -0,0 +1,87 @@
# Testing
## Unit and mock integration tests
Run from the repository root:
```bash
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -p 'test_*.py' -v
bash tests/run_e2e.sh
LEAF_E2E_SOCKET=/tmp/leaf-test.sock bash tests/run_e2e.sh
python3 tests/checkpoint_e2e.py
```
The unit suite covers socket ownership/framing, protected-address policy,
thresholds, persistence, replay identities, reconciliation races, TLS pin
isolation, retention, malformed telemetry and credential-switch failure paths.
It uses local fixtures rather than the lab network.
The shell integration starts a localhost mock VyOS API and a controller. The
first invocation uses file alerts; the second uses the Unix socket. They check
thresholding, protected addresses, source blocking, rotation, adoption and TTL
cleanup. They use port 18443 and disposable `tests/.e2e/` files: run them
sequentially. Keep the socket path short enough for AF_UNIX pathname limits.
The checkpoint check verifies that file mode resumes at the next unread record.
It does not assert socket-alert backlog during controller downtime.
## Offline Suricata replay
On a sensor with Suricata installed and its rules in `/etc/suricata-leaf/`:
```bash
sudo python3 tests/replay_fixture.py
```
The script generates a valid 24-SYN PCAP, replays it with isolated outputs and
requires 24 decoded SYNs, two SID 1000030 alerts and zero fragment/checksum errors.
It leaves the live capture configuration and services in place.
`tests/syn_packets.py` is the maintained TEST-NET SYN generator.
`tests/drill_synsensor.py` can write ICMP/SYN PCAP fixtures or send bounded raw
frames on a selected route. Capture visibility depends on the mirror topology;
use the test hosts for the live enforcement checks.
## Live enforcement tests
Verify the sensor identity, SSH account/key, test hosts, routes and dedicated
block group before running. Defaults describe the supplied lab. Override the
connection arguments for another environment:
```bash
python3 tests/live_e2e.py --help
python3 tests/live_negative.py --help
python3 tests/live_e2e.py --output evidence/live-e2e.json
python3 tests/live_negative.py --output evidence/negative.json
```
These tests use SSH to the sensor and LXCs, deploy temporary helpers, transmit
bounded TEST-NET traffic and temporarily select a 180-second TTL. They require
an empty block group and state for counter attribution. Run them sequentially.
The original configuration is restored on exit; failure cleanup expires only
the test's own targets. An interrupted connection can require checking the
retained configuration backup before another run.
The positive test requires fresh capture, audited decisions, durable applied
rows, verified router membership, source/destination drop-counter growth,
natural TTL removal and clean post-expiry probes. It compares hashes of
protected router configuration before and after.
The negative test sends benign traffic from another source while a block is
active. It checks ingress observation, absence of the benign address from
block membership/state, and selective IDS counter changes. Ingress capture
is not a destination receipt or application-connectivity test.
Live results describe the tested rules, topology and traffic. They do not
establish sustained throughput, production detection quality, lossless alert
delivery, or successful full router/sensor reboot recovery.
## Recovery inspection
```bash
VM=<sensor-user>@<sensor-address> LEAF_SSH_KEY=<key-path> bash scripts/reboot-check.sh
```
Despite its filename, this script performs read-only recovery inspection. It
checks service activity/enabling, tmpfs, socket presence and API/state status,
and prints the current boot identity. It does not initiate or prove a reboot.