Docs
This commit is contained in:
68
docs/ARCHITECTURE.md
Normal file
68
docs/ARCHITECTURE.md
Normal file
@@ -0,0 +1,68 @@
|
||||
# Controller architecture
|
||||
|
||||
## Alert ingestion
|
||||
|
||||
Suricata is the client of an AF_UNIX stream socket bound by `AlertSocket` in
|
||||
`alertsock.py`. The reader frames newline-delimited JSON, accepts alert records,
|
||||
and places them on a bounded queue. Oversized or malformed records and queue
|
||||
overflow are counted. A socket-path lock prevents competing controllers from
|
||||
replacing the listener, even when they use different state databases.
|
||||
|
||||
The queue wakes the main loop independently of file polling. The reader does
|
||||
not run policy or wait for router commits. During graceful shutdown, intake is
|
||||
stopped and accepted queued alerts are processed before SQLite closes.
|
||||
|
||||
The normal EVE file contains stats. `EveTailer` in `eve.py` follows complete
|
||||
records, preserves partial writes and drains the old inode during rotation.
|
||||
When socket mode is enabled, file alerts are ignored to avoid counting the same
|
||||
alert through two outputs.
|
||||
|
||||
For file-alert mode, set `alert_socket.path: ""` and configure Suricata to write
|
||||
alerts into the EVE file. Checkpoints advance per delivered record. File-hit
|
||||
identities include record content, inode, offset, SID and candidate address to
|
||||
prevent replay from inflating threshold counts. Processing is at-least-once;
|
||||
a restart cannot automatically recover unread records in an older rotated file.
|
||||
|
||||
## Policy and state
|
||||
|
||||
`policy.py` reads `leaf_action` metadata, applies SID restrictions, rejects
|
||||
protected addresses, and evaluates threshold or immediate mode. `state.py`
|
||||
stores hit windows, desired blocks, expiry times, applied flags, file checkpoints
|
||||
and pending router-save status in SQLite WAL mode.
|
||||
|
||||
The desired block row is persisted before API submission. Qualifying alerts
|
||||
refresh expiry. The active-address cap bounds the number of desired blocks.
|
||||
Source and destination actions share a router group; `kind` records the initial
|
||||
reason rather than choosing a separate group.
|
||||
|
||||
## Router reconciliation
|
||||
|
||||
A separate worker in `main.py` serializes router I/O using its own SQLite
|
||||
connection. Each pass reads actual membership, computes changes, commits a batch,
|
||||
and reads membership back before marking decisions applied or removals complete.
|
||||
|
||||
A timeout may still mean a commit succeeded. The following pass reads state
|
||||
before retrying. Expiry deletion uses a timestamp condition so that a fresh
|
||||
alert received during a slow request does not lose its refreshed decision.
|
||||
Failures retain desired state for retry.
|
||||
|
||||
`vyos_client.py` handles multipart API requests and response validation. TLS can
|
||||
use normal CA verification or a certificate fingerprint checked before sending
|
||||
credentials. Pinned pool classes are private to each client. Environment proxies
|
||||
are disabled for the API session.
|
||||
|
||||
## Metrics and persistence
|
||||
|
||||
Stats records update `metrics.py` counters without entering policy or changing
|
||||
router state. Reports include capture/decoder deltas and ingress counters.
|
||||
`decisions.jsonl` records decisions and verified enforcement changes; `capture.log`
|
||||
contains periodic metric windows. Both logs retain the newest size-bounded archives.
|
||||
|
||||
A zero controller queue-drop count covers that queue only. Suricata can lose
|
||||
alerts before delivery under send-buffer pressure or disconnection. The socket
|
||||
queue is volatile on crash. Durable decisions can be reconciled after restart,
|
||||
but alerts that never reached SQLite cannot be reconstructed from the stats file.
|
||||
|
||||
Dry-run creates temporary state and logs, disables socket ownership, avoids the
|
||||
API, and reads the configured file from its beginning. Use an alert fixture for
|
||||
policy replay; the normal stats-only file contains no alert history.
|
||||
144
docs/INSTALLATION.md
Normal file
144
docs/INSTALLATION.md
Normal file
@@ -0,0 +1,144 @@
|
||||
# Installation and operations
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Use a Linux sensor with systemd, Python 3, `requests`, `PyYAML`, Suricata 7,
|
||||
OpenSSH and tcpdump. Live tests also require Python 3 and ping on the test hosts.
|
||||
The supplied service uses `/usr/bin/python3`; install its dependencies for that
|
||||
interpreter. On Debian-based sensors, use the `python3-requests` and
|
||||
`python3-yaml` packages. A separate virtual environment can be used for local
|
||||
tests with `controller/requirements.txt`.
|
||||
|
||||
The supplied lab configuration uses sensor capture interface `eth1`, P2P
|
||||
address `10.0.249.1/31`, VyOS API `https://10.0.249.0:8443`, and address group
|
||||
`IDS-BLOCK`. Test hosts are `172.19.32.3` and `172.19.32.5`. These are example
|
||||
site settings; inspect and adapt them for another network.
|
||||
|
||||
Provision mirroring, API access and firewall hooks before starting the sensor.
|
||||
Both source and destination drop hooks must be reachable before accept rules
|
||||
that would bypass enforcement. The controller verifies hooks but does not
|
||||
provision routing or mirroring. The file in `vyos/` is an illustrative template,
|
||||
not a complete router configuration.
|
||||
|
||||
## Configuration and credentials
|
||||
|
||||
The installer preserves an existing `/etc/leaf-controller/config.yaml`. For a
|
||||
new installation, it seeds that file from `config/prototype-vyos.yaml`.
|
||||
`controller/config.yaml` is a generic example with different API/TLS settings;
|
||||
it is not the installer's lab default.
|
||||
|
||||
Review these settings before installation:
|
||||
|
||||
- API URL, timeout, retry count and TLS certificate fingerprint or CA bundle.
|
||||
- Protected networks, allowed SIDs, source actions, threshold, TTL and block cap.
|
||||
- Capture interface in the Suricata configuration and service unit.
|
||||
- Socket, EVE, SQLite and audit paths.
|
||||
|
||||
The lab API uses a certificate fingerprint because its self-signed certificate
|
||||
has no matching IP subject alternative name. Verify the fingerprint through a
|
||||
trusted channel before setting `vyos.tls_cert_pin`. For a CA-issued certificate,
|
||||
configure the appropriate CA bundle and hostname. Keep TLS verification enabled.
|
||||
|
||||
Create `/etc/leaf-controller/env`, owned by root with mode `0600`, containing:
|
||||
|
||||
```text
|
||||
LEAF_VYOS_KEY='replace-with-existing-api-key'
|
||||
```
|
||||
|
||||
Use a root editor or secret-management mechanism. Do not commit the real value
|
||||
or place it in command arguments. The credential file is read by systemd.
|
||||
|
||||
The router address group must be dedicated to this controller. With
|
||||
`adopt_foreign_members: false`, foreign members are removed during reconciliation.
|
||||
The optional adoption setting imports eligible foreign members only at startup.
|
||||
|
||||
## Log storage and installation
|
||||
|
||||
The installer requires `/var/log/suricata-leaf` to be on tmpfs. On a new sensor,
|
||||
review and run the provisioning script before installing:
|
||||
|
||||
```bash
|
||||
sudo env TMPFS_SIZE=8G bash scripts/provision-ids-tmpfs.sh
|
||||
sudo bash scripts/install-sensor.sh
|
||||
```
|
||||
|
||||
Allow enough memory and swap headroom for the configured tmpfs cap and Suricata.
|
||||
The provisioning script writes fstab and runs `mount -a`; review existing mounts
|
||||
first. Mounting over a nonempty directory hides its previous contents. EVE files
|
||||
on tmpfs are lost at reboot. SQLite and controller audit remain on persistent disk.
|
||||
|
||||
The installer stops the sensor/controller for upgrade, backs up existing files,
|
||||
units and SQLite under `/var/backups/leaf-proto/<timestamp>/`, installs the code
|
||||
and helpers, and enables these units:
|
||||
|
||||
- `leaf-controller.service`
|
||||
- `suricata-leaf.service`
|
||||
- `leaf-eve-rotate.timer`
|
||||
|
||||
It restores the previous deployment on installation failure. Backups can contain
|
||||
credentials and must stay private. The controller runs as `leafctrl`; Suricata
|
||||
runs as root with group `leafctrl`. The controller reports readiness after its
|
||||
socket is bound, allowing the sensor to start afterward.
|
||||
|
||||
## Status and diagnostics
|
||||
|
||||
```bash
|
||||
sudo systemctl status leaf-controller suricata-leaf leaf-eve-rotate.timer
|
||||
sudo journalctl -u leaf-controller -f
|
||||
sudo python3 /opt/leaf-controller/tools/vyos_smoke.py
|
||||
sudo python3 /opt/leaf-controller/tools/vyos_ctl.py members
|
||||
```
|
||||
|
||||
The smoke check reads API information, members and enforcement hooks.
|
||||
`vyos_show.py` reads scoped configuration and redacts sensitive fields.
|
||||
These diagnostic scripts do not write router configuration.
|
||||
|
||||
| Resource | Installed path |
|
||||
|---|---|
|
||||
| Controller | `/opt/leaf-controller/controller/leaf_controller/` |
|
||||
| Configuration | `/etc/leaf-controller/config.yaml` |
|
||||
| Credential | `/etc/leaf-controller/env` |
|
||||
| SQLite WAL state | `/var/lib/leaf-controller/state.sqlite` |
|
||||
| Decision audit | `/var/log/leaf-controller/decisions.jsonl` |
|
||||
| Capture metrics | `/var/log/leaf-controller/capture.log` |
|
||||
| EVE stats | `/var/log/suricata-leaf/eve.json` |
|
||||
| Alert socket | `/run/suricata/alerts.sock` |
|
||||
|
||||
The retention timer checks each minute after an initial two-minute delay. It
|
||||
renames EVE hourly or at 64 MiB, asks Suricata to reopen it, and retains six
|
||||
archives. A startup grace period prevents signalling a newly started engine.
|
||||
Retention can overshoot between timer runs; tmpfs capacity is the hard limit.
|
||||
Audit defaults to 16 MiB with three archives; capture metrics default to 10 MiB
|
||||
with six archives.
|
||||
|
||||
`save_config: false` leaves transient membership out of router boot configuration.
|
||||
SQLite supplies desired state when the router returns. If saving is enabled,
|
||||
failed saves remain dirty in SQLite and are retried.
|
||||
|
||||
## Credential rotation
|
||||
|
||||
Identify the controller's existing key ID before rotating:
|
||||
|
||||
```bash
|
||||
sudo python3 /opt/leaf-controller/tools/rotate-vyos-key.py --old-id <controller-key-id>
|
||||
```
|
||||
|
||||
The tool requires the selected ID to match the current credential. It creates
|
||||
and verifies a candidate, saves both keys, switches the local environment file,
|
||||
restarts the service, then retires only the selected old ID and saves again.
|
||||
Other API users' IDs are preserved.
|
||||
|
||||
Switch failures restore the local credential. A failure after retirement requires
|
||||
inspection before retrying; deleting arbitrary IDs is not a recovery procedure.
|
||||
No credential values are printed by the tool.
|
||||
|
||||
## Rollback
|
||||
|
||||
Stop Suricata, the controller and the rotation timer. Restore the selected file
|
||||
archive and its matching SQLite backup, reload systemd, and restart the previous
|
||||
units. Keep SQLite closed while replacing its files. When reverting a schema
|
||||
migration, restore the matching database as well as the code.
|
||||
|
||||
Do not clear the live database or shared router configuration as an upgrade or
|
||||
test shortcut. Live tests operate on existing routes and enforcement hooks;
|
||||
uplink, OSPF and static-route changes are outside their scope.
|
||||
11
docs/REFERENCES.md
Normal file
11
docs/REFERENCES.md
Normal file
@@ -0,0 +1,11 @@
|
||||
# References
|
||||
|
||||
The lab uses Suricata 7 and VyOS 2026.03. Configuration and API schemas differ
|
||||
between releases; use documentation matching the installed software.
|
||||
|
||||
- [VyOS HTTP API](https://docs.vyos.io/en/rolling/automation/vyos-api.html)
|
||||
- [VyOS HTTPS service and API configuration](https://docs.vyos.io/en/rolling/configuration/service/https.html)
|
||||
- [VyOS Ethernet interfaces and mirroring](https://docs.vyos.io/en/rolling/configuration/interfaces/ethernet.html)
|
||||
- [VyOS firewall groups](https://docs.vyos.io/en/rolling/configuration/firewall/groups.html)
|
||||
- [Suricata EVE output](https://docs.suricata.io/en/suricata-7.0.10/output/eve/eve-json-output.html)
|
||||
- [Suricata EVE JSON format](https://docs.suricata.io/en/suricata-7.0.10/output/eve/eve-json-format.html)
|
||||
87
docs/TESTING.md
Normal file
87
docs/TESTING.md
Normal file
@@ -0,0 +1,87 @@
|
||||
# Testing
|
||||
|
||||
## Unit and mock integration tests
|
||||
|
||||
Run from the repository root:
|
||||
|
||||
```bash
|
||||
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -p 'test_*.py' -v
|
||||
bash tests/run_e2e.sh
|
||||
LEAF_E2E_SOCKET=/tmp/leaf-test.sock bash tests/run_e2e.sh
|
||||
python3 tests/checkpoint_e2e.py
|
||||
```
|
||||
|
||||
The unit suite covers socket ownership/framing, protected-address policy,
|
||||
thresholds, persistence, replay identities, reconciliation races, TLS pin
|
||||
isolation, retention, malformed telemetry and credential-switch failure paths.
|
||||
It uses local fixtures rather than the lab network.
|
||||
|
||||
The shell integration starts a localhost mock VyOS API and a controller. The
|
||||
first invocation uses file alerts; the second uses the Unix socket. They check
|
||||
thresholding, protected addresses, source blocking, rotation, adoption and TTL
|
||||
cleanup. They use port 18443 and disposable `tests/.e2e/` files: run them
|
||||
sequentially. Keep the socket path short enough for AF_UNIX pathname limits.
|
||||
|
||||
The checkpoint check verifies that file mode resumes at the next unread record.
|
||||
It does not assert socket-alert backlog during controller downtime.
|
||||
|
||||
## Offline Suricata replay
|
||||
|
||||
On a sensor with Suricata installed and its rules in `/etc/suricata-leaf/`:
|
||||
|
||||
```bash
|
||||
sudo python3 tests/replay_fixture.py
|
||||
```
|
||||
|
||||
The script generates a valid 24-SYN PCAP, replays it with isolated outputs and
|
||||
requires 24 decoded SYNs, two SID 1000030 alerts and zero fragment/checksum errors.
|
||||
It leaves the live capture configuration and services in place.
|
||||
|
||||
`tests/syn_packets.py` is the maintained TEST-NET SYN generator.
|
||||
`tests/drill_synsensor.py` can write ICMP/SYN PCAP fixtures or send bounded raw
|
||||
frames on a selected route. Capture visibility depends on the mirror topology;
|
||||
use the test hosts for the live enforcement checks.
|
||||
|
||||
## Live enforcement tests
|
||||
|
||||
Verify the sensor identity, SSH account/key, test hosts, routes and dedicated
|
||||
block group before running. Defaults describe the supplied lab. Override the
|
||||
connection arguments for another environment:
|
||||
|
||||
```bash
|
||||
python3 tests/live_e2e.py --help
|
||||
python3 tests/live_negative.py --help
|
||||
python3 tests/live_e2e.py --output evidence/live-e2e.json
|
||||
python3 tests/live_negative.py --output evidence/negative.json
|
||||
```
|
||||
|
||||
These tests use SSH to the sensor and LXCs, deploy temporary helpers, transmit
|
||||
bounded TEST-NET traffic and temporarily select a 180-second TTL. They require
|
||||
an empty block group and state for counter attribution. Run them sequentially.
|
||||
The original configuration is restored on exit; failure cleanup expires only
|
||||
the test's own targets. An interrupted connection can require checking the
|
||||
retained configuration backup before another run.
|
||||
|
||||
The positive test requires fresh capture, audited decisions, durable applied
|
||||
rows, verified router membership, source/destination drop-counter growth,
|
||||
natural TTL removal and clean post-expiry probes. It compares hashes of
|
||||
protected router configuration before and after.
|
||||
|
||||
The negative test sends benign traffic from another source while a block is
|
||||
active. It checks ingress observation, absence of the benign address from
|
||||
block membership/state, and selective IDS counter changes. Ingress capture
|
||||
is not a destination receipt or application-connectivity test.
|
||||
|
||||
Live results describe the tested rules, topology and traffic. They do not
|
||||
establish sustained throughput, production detection quality, lossless alert
|
||||
delivery, or successful full router/sensor reboot recovery.
|
||||
|
||||
## Recovery inspection
|
||||
|
||||
```bash
|
||||
VM=<sensor-user>@<sensor-address> LEAF_SSH_KEY=<key-path> bash scripts/reboot-check.sh
|
||||
```
|
||||
|
||||
Despite its filename, this script performs read-only recovery inspection. It
|
||||
checks service activity/enabling, tmpfs, socket presence and API/state status,
|
||||
and prints the current boot identity. It does not initiate or prove a reboot.
|
||||
Reference in New Issue
Block a user