From 6aec4f60827e33528c2c3d7b0c3fcc9ef0c937e9 Mon Sep 17 00:00:00 2001 From: YuruC3 Date: Sun, 15 Nov 2026 17:43:04 +0100 Subject: [PATCH] Docs --- docs/ARCHITECTURE.md | 68 ++++++++++++++++++++ docs/INSTALLATION.md | 144 +++++++++++++++++++++++++++++++++++++++++++ docs/REFERENCES.md | 11 ++++ docs/TESTING.md | 87 ++++++++++++++++++++++++++ 4 files changed, 310 insertions(+) create mode 100644 docs/ARCHITECTURE.md create mode 100644 docs/INSTALLATION.md create mode 100644 docs/REFERENCES.md create mode 100644 docs/TESTING.md diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 0000000..100738d --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,68 @@ +# Controller architecture + +## Alert ingestion + +Suricata is the client of an AF_UNIX stream socket bound by `AlertSocket` in +`alertsock.py`. The reader frames newline-delimited JSON, accepts alert records, +and places them on a bounded queue. Oversized or malformed records and queue +overflow are counted. A socket-path lock prevents competing controllers from +replacing the listener, even when they use different state databases. + +The queue wakes the main loop independently of file polling. The reader does +not run policy or wait for router commits. During graceful shutdown, intake is +stopped and accepted queued alerts are processed before SQLite closes. + +The normal EVE file contains stats. `EveTailer` in `eve.py` follows complete +records, preserves partial writes and drains the old inode during rotation. +When socket mode is enabled, file alerts are ignored to avoid counting the same +alert through two outputs. + +For file-alert mode, set `alert_socket.path: ""` and configure Suricata to write +alerts into the EVE file. Checkpoints advance per delivered record. File-hit +identities include record content, inode, offset, SID and candidate address to +prevent replay from inflating threshold counts. Processing is at-least-once; +a restart cannot automatically recover unread records in an older rotated file. + +## Policy and state + +`policy.py` reads `leaf_action` metadata, applies SID restrictions, rejects +protected addresses, and evaluates threshold or immediate mode. `state.py` +stores hit windows, desired blocks, expiry times, applied flags, file checkpoints +and pending router-save status in SQLite WAL mode. + +The desired block row is persisted before API submission. Qualifying alerts +refresh expiry. The active-address cap bounds the number of desired blocks. +Source and destination actions share a router group; `kind` records the initial +reason rather than choosing a separate group. + +## Router reconciliation + +A separate worker in `main.py` serializes router I/O using its own SQLite +connection. Each pass reads actual membership, computes changes, commits a batch, +and reads membership back before marking decisions applied or removals complete. + +A timeout may still mean a commit succeeded. The following pass reads state +before retrying. Expiry deletion uses a timestamp condition so that a fresh +alert received during a slow request does not lose its refreshed decision. +Failures retain desired state for retry. + +`vyos_client.py` handles multipart API requests and response validation. TLS can +use normal CA verification or a certificate fingerprint checked before sending +credentials. Pinned pool classes are private to each client. Environment proxies +are disabled for the API session. + +## Metrics and persistence + +Stats records update `metrics.py` counters without entering policy or changing +router state. Reports include capture/decoder deltas and ingress counters. +`decisions.jsonl` records decisions and verified enforcement changes; `capture.log` +contains periodic metric windows. Both logs retain the newest size-bounded archives. + +A zero controller queue-drop count covers that queue only. Suricata can lose +alerts before delivery under send-buffer pressure or disconnection. The socket +queue is volatile on crash. Durable decisions can be reconciled after restart, +but alerts that never reached SQLite cannot be reconstructed from the stats file. + +Dry-run creates temporary state and logs, disables socket ownership, avoids the +API, and reads the configured file from its beginning. Use an alert fixture for +policy replay; the normal stats-only file contains no alert history. diff --git a/docs/INSTALLATION.md b/docs/INSTALLATION.md new file mode 100644 index 0000000..009a7f5 --- /dev/null +++ b/docs/INSTALLATION.md @@ -0,0 +1,144 @@ +# Installation and operations + +## Prerequisites + +Use a Linux sensor with systemd, Python 3, `requests`, `PyYAML`, Suricata 7, +OpenSSH and tcpdump. Live tests also require Python 3 and ping on the test hosts. +The supplied service uses `/usr/bin/python3`; install its dependencies for that +interpreter. On Debian-based sensors, use the `python3-requests` and +`python3-yaml` packages. A separate virtual environment can be used for local +tests with `controller/requirements.txt`. + +The supplied lab configuration uses sensor capture interface `eth1`, P2P +address `10.0.249.1/31`, VyOS API `https://10.0.249.0:8443`, and address group +`IDS-BLOCK`. Test hosts are `172.19.32.3` and `172.19.32.5`. These are example +site settings; inspect and adapt them for another network. + +Provision mirroring, API access and firewall hooks before starting the sensor. +Both source and destination drop hooks must be reachable before accept rules +that would bypass enforcement. The controller verifies hooks but does not +provision routing or mirroring. The file in `vyos/` is an illustrative template, +not a complete router configuration. + +## Configuration and credentials + +The installer preserves an existing `/etc/leaf-controller/config.yaml`. For a +new installation, it seeds that file from `config/prototype-vyos.yaml`. +`controller/config.yaml` is a generic example with different API/TLS settings; +it is not the installer's lab default. + +Review these settings before installation: + +- API URL, timeout, retry count and TLS certificate fingerprint or CA bundle. +- Protected networks, allowed SIDs, source actions, threshold, TTL and block cap. +- Capture interface in the Suricata configuration and service unit. +- Socket, EVE, SQLite and audit paths. + +The lab API uses a certificate fingerprint because its self-signed certificate +has no matching IP subject alternative name. Verify the fingerprint through a +trusted channel before setting `vyos.tls_cert_pin`. For a CA-issued certificate, +configure the appropriate CA bundle and hostname. Keep TLS verification enabled. + +Create `/etc/leaf-controller/env`, owned by root with mode `0600`, containing: + +```text +LEAF_VYOS_KEY='replace-with-existing-api-key' +``` + +Use a root editor or secret-management mechanism. Do not commit the real value +or place it in command arguments. The credential file is read by systemd. + +The router address group must be dedicated to this controller. With +`adopt_foreign_members: false`, foreign members are removed during reconciliation. +The optional adoption setting imports eligible foreign members only at startup. + +## Log storage and installation + +The installer requires `/var/log/suricata-leaf` to be on tmpfs. On a new sensor, +review and run the provisioning script before installing: + +```bash +sudo env TMPFS_SIZE=8G bash scripts/provision-ids-tmpfs.sh +sudo bash scripts/install-sensor.sh +``` + +Allow enough memory and swap headroom for the configured tmpfs cap and Suricata. +The provisioning script writes fstab and runs `mount -a`; review existing mounts +first. Mounting over a nonempty directory hides its previous contents. EVE files +on tmpfs are lost at reboot. SQLite and controller audit remain on persistent disk. + +The installer stops the sensor/controller for upgrade, backs up existing files, +units and SQLite under `/var/backups/leaf-proto//`, installs the code +and helpers, and enables these units: + +- `leaf-controller.service` +- `suricata-leaf.service` +- `leaf-eve-rotate.timer` + +It restores the previous deployment on installation failure. Backups can contain +credentials and must stay private. The controller runs as `leafctrl`; Suricata +runs as root with group `leafctrl`. The controller reports readiness after its +socket is bound, allowing the sensor to start afterward. + +## Status and diagnostics + +```bash +sudo systemctl status leaf-controller suricata-leaf leaf-eve-rotate.timer +sudo journalctl -u leaf-controller -f +sudo python3 /opt/leaf-controller/tools/vyos_smoke.py +sudo python3 /opt/leaf-controller/tools/vyos_ctl.py members +``` + +The smoke check reads API information, members and enforcement hooks. +`vyos_show.py` reads scoped configuration and redacts sensitive fields. +These diagnostic scripts do not write router configuration. + +| Resource | Installed path | +|---|---| +| Controller | `/opt/leaf-controller/controller/leaf_controller/` | +| Configuration | `/etc/leaf-controller/config.yaml` | +| Credential | `/etc/leaf-controller/env` | +| SQLite WAL state | `/var/lib/leaf-controller/state.sqlite` | +| Decision audit | `/var/log/leaf-controller/decisions.jsonl` | +| Capture metrics | `/var/log/leaf-controller/capture.log` | +| EVE stats | `/var/log/suricata-leaf/eve.json` | +| Alert socket | `/run/suricata/alerts.sock` | + +The retention timer checks each minute after an initial two-minute delay. It +renames EVE hourly or at 64 MiB, asks Suricata to reopen it, and retains six +archives. A startup grace period prevents signalling a newly started engine. +Retention can overshoot between timer runs; tmpfs capacity is the hard limit. +Audit defaults to 16 MiB with three archives; capture metrics default to 10 MiB +with six archives. + +`save_config: false` leaves transient membership out of router boot configuration. +SQLite supplies desired state when the router returns. If saving is enabled, +failed saves remain dirty in SQLite and are retried. + +## Credential rotation + +Identify the controller's existing key ID before rotating: + +```bash +sudo python3 /opt/leaf-controller/tools/rotate-vyos-key.py --old-id +``` + +The tool requires the selected ID to match the current credential. It creates +and verifies a candidate, saves both keys, switches the local environment file, +restarts the service, then retires only the selected old ID and saves again. +Other API users' IDs are preserved. + +Switch failures restore the local credential. A failure after retirement requires +inspection before retrying; deleting arbitrary IDs is not a recovery procedure. +No credential values are printed by the tool. + +## Rollback + +Stop Suricata, the controller and the rotation timer. Restore the selected file +archive and its matching SQLite backup, reload systemd, and restart the previous +units. Keep SQLite closed while replacing its files. When reverting a schema +migration, restore the matching database as well as the code. + +Do not clear the live database or shared router configuration as an upgrade or +test shortcut. Live tests operate on existing routes and enforcement hooks; +uplink, OSPF and static-route changes are outside their scope. diff --git a/docs/REFERENCES.md b/docs/REFERENCES.md new file mode 100644 index 0000000..c6f1609 --- /dev/null +++ b/docs/REFERENCES.md @@ -0,0 +1,11 @@ +# References + +The lab uses Suricata 7 and VyOS 2026.03. Configuration and API schemas differ +between releases; use documentation matching the installed software. + +- [VyOS HTTP API](https://docs.vyos.io/en/rolling/automation/vyos-api.html) +- [VyOS HTTPS service and API configuration](https://docs.vyos.io/en/rolling/configuration/service/https.html) +- [VyOS Ethernet interfaces and mirroring](https://docs.vyos.io/en/rolling/configuration/interfaces/ethernet.html) +- [VyOS firewall groups](https://docs.vyos.io/en/rolling/configuration/firewall/groups.html) +- [Suricata EVE output](https://docs.suricata.io/en/suricata-7.0.10/output/eve/eve-json-output.html) +- [Suricata EVE JSON format](https://docs.suricata.io/en/suricata-7.0.10/output/eve/eve-json-format.html) diff --git a/docs/TESTING.md b/docs/TESTING.md new file mode 100644 index 0000000..dec6a9c --- /dev/null +++ b/docs/TESTING.md @@ -0,0 +1,87 @@ +# Testing + +## Unit and mock integration tests + +Run from the repository root: + +```bash +PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -p 'test_*.py' -v +bash tests/run_e2e.sh +LEAF_E2E_SOCKET=/tmp/leaf-test.sock bash tests/run_e2e.sh +python3 tests/checkpoint_e2e.py +``` + +The unit suite covers socket ownership/framing, protected-address policy, +thresholds, persistence, replay identities, reconciliation races, TLS pin +isolation, retention, malformed telemetry and credential-switch failure paths. +It uses local fixtures rather than the lab network. + +The shell integration starts a localhost mock VyOS API and a controller. The +first invocation uses file alerts; the second uses the Unix socket. They check +thresholding, protected addresses, source blocking, rotation, adoption and TTL +cleanup. They use port 18443 and disposable `tests/.e2e/` files: run them +sequentially. Keep the socket path short enough for AF_UNIX pathname limits. + +The checkpoint check verifies that file mode resumes at the next unread record. +It does not assert socket-alert backlog during controller downtime. + +## Offline Suricata replay + +On a sensor with Suricata installed and its rules in `/etc/suricata-leaf/`: + +```bash +sudo python3 tests/replay_fixture.py +``` + +The script generates a valid 24-SYN PCAP, replays it with isolated outputs and +requires 24 decoded SYNs, two SID 1000030 alerts and zero fragment/checksum errors. +It leaves the live capture configuration and services in place. + +`tests/syn_packets.py` is the maintained TEST-NET SYN generator. +`tests/drill_synsensor.py` can write ICMP/SYN PCAP fixtures or send bounded raw +frames on a selected route. Capture visibility depends on the mirror topology; +use the test hosts for the live enforcement checks. + +## Live enforcement tests + +Verify the sensor identity, SSH account/key, test hosts, routes and dedicated +block group before running. Defaults describe the supplied lab. Override the +connection arguments for another environment: + +```bash +python3 tests/live_e2e.py --help +python3 tests/live_negative.py --help +python3 tests/live_e2e.py --output evidence/live-e2e.json +python3 tests/live_negative.py --output evidence/negative.json +``` + +These tests use SSH to the sensor and LXCs, deploy temporary helpers, transmit +bounded TEST-NET traffic and temporarily select a 180-second TTL. They require +an empty block group and state for counter attribution. Run them sequentially. +The original configuration is restored on exit; failure cleanup expires only +the test's own targets. An interrupted connection can require checking the +retained configuration backup before another run. + +The positive test requires fresh capture, audited decisions, durable applied +rows, verified router membership, source/destination drop-counter growth, +natural TTL removal and clean post-expiry probes. It compares hashes of +protected router configuration before and after. + +The negative test sends benign traffic from another source while a block is +active. It checks ingress observation, absence of the benign address from +block membership/state, and selective IDS counter changes. Ingress capture +is not a destination receipt or application-connectivity test. + +Live results describe the tested rules, topology and traffic. They do not +establish sustained throughput, production detection quality, lossless alert +delivery, or successful full router/sensor reboot recovery. + +## Recovery inspection + +```bash +VM=@ LEAF_SSH_KEY= bash scripts/reboot-check.sh +``` + +Despite its filename, this script performs read-only recovery inspection. It +checks service activity/enabling, tmpfs, socket presence and API/state status, +and prints the current boot identity. It does not initiate or prove a reboot.