Compare commits

...

5 Commits

Author SHA1 Message Date
cb117e1008 Others 2026-11-15 17:43:13 +01:00
6aec4f6082 Docs 2026-11-15 17:43:04 +01:00
39ba6bd7da Helper scripts 2026-11-15 17:42:55 +01:00
b2f1baaac5 Suricata configs and settings 2026-11-15 17:42:15 +01:00
7864d2ed18 Code 2026-11-15 17:41:40 +01:00
48 changed files with 4422 additions and 1 deletions

25
.gitignore vendored Normal file
View File

@@ -0,0 +1,25 @@
evidence/
.local_keys
*.pyc
__pycache__/
tests/.dbg*
tests/.e2e/
# Local credentials and runtime data
.env
.env.*
*.key
*.pem
*.sqlite
*.sqlite-*
*.sock
*.sock.lock
*.pcap
.venv/
.pytest_cache/
# Editor backup files
*.bkp
fig

View File

@@ -1 +1,93 @@
# Hi
# LEAF
LEAF connects a passive Suricata sensor to VyOS firewall enforcement. It reads
alerts, applies a threshold policy, stores block decisions in SQLite, and
updates a dedicated router address group through the VyOS API.
Suricata receives mirrored traffic. Enforcement happens asynchronously on the
router, so packets can pass before a block is committed.
```text
Mirrored traffic -> Suricata -> Unix socket -> policy -> SQLite
|
reconciliation worker
|
VyOS API + readback
|
firewall drop rules
Suricata stats -> EVE file on tmpfs -> metrics
Controller decisions -> persistent audit log
```
## Policy
Rules must carry a `leaf_action` metadata value before they can produce a block:
| Metadata | Target | Requirement |
|---|---|---|
| `leaf_action block` | Destination IPv4 address | Policy permits the SID and address |
| `leaf_action block_src` | Source IPv4 address | Also requires `policy.block_src: true` |
The lab configuration allows SIDs **1000001** and **1000030**, requires two alerts
for the same SID and address within 300 seconds, and assigns a 3600-second TTL.
Qualifying alerts refresh the TTL. The default cap is 50 active addresses.
Private, loopback, link-local, multicast and configured trusted networks are
protected. The included rules are examples for controlled TEST-NET traffic.
One worker serializes router changes and verifies group membership after each
commit. SQLite retains desired blocks during API failures. Expiry and drift
reconciliation run independently of new alerts. Source and destination hooks
share the same address group; the stored `kind` records the initial reason.
## Repository
| Path | Contents |
|---|---|
| `controller/leaf_controller/` | Ingestion, policy, state, metrics and API client |
| `controller/leaf-controller.service` | Controller service |
| `config/prototype-vyos.yaml` | Lab controller configuration |
| `controller/config.yaml` | Generic configuration example |
| `suricata/` | Sensor configuration, rules and retention units |
| `scripts/` | Installation, maintenance and read-only diagnostics |
| `tests/` | Unit tests, mock integration and bounded live tests |
| `vyos/router-leaf.snippet.txt` | Illustrative enforcement hooks |
| `fig/` | Architecture figures |
## Getting started
The sensor requires Python 3, the packages in `controller/requirements.txt`,
Suricata 7, systemd and a capture interface receiving the intended mirror.
Review the interface names, API endpoint, TLS trust, protected networks and
firewall hooks before installing.
```bash
sudo apt install python3 python3-requests python3-yaml
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -p 'test_*.py' -v
bash tests/run_e2e.sh
```
See [Installation and operations](docs/INSTALLATION.md) for deployment,
[Architecture](docs/ARCHITECTURE.md) for the controller design, and
[Testing](docs/TESTING.md) for offline and live checks.
## Limits
LEAF is an IPv4 lab prototype. The sample rules do not establish production
detection quality, and no high-throughput capacity is claimed. Router commits
can take tens of seconds.
Socket alerts are not a durable journal. A process crash, disconnection or
Suricata send-buffer pressure can lose alerts before they reach SQLite. A zero
controller queue-drop count does not establish that Suricata delivered every
alert. Graceful shutdown persists accepted queued alerts; previously stored
block decisions survive restarts.
Live counter tests establish detection and firewall-rule hits. Ingress capture
alone does not prove destination receipt or application connectivity. Sustained
throughput and full sensor/router reboot recovery require separate testing.
![Mary Sue my goat](https://data.shupogaki.org/assets/1/HUHV0zHbEAEdCyM.jpeg)

View File

@@ -0,0 +1,52 @@
# LEAF-PROTO prototype config — run on the LEAF VM (eth1 P2P link to VIPERA).
# Secrets stay in env: export LEAF_VYOS_KEY=... before starting the controller.
vyos:
url: "https://10.0.249.0:8443"
key_env: LEAF_VYOS_KEY
# Router serves a self-signed CN=localhost cert with no IP SAN, so plain
# CA/hostname verification cannot work; trust is anchored on the pinned
# fingerprint below (regenerate with:
# echo | openssl s_client -connect 10.0.249.0:8443 2>/dev/null | openssl x509 -noout -fingerprint -sha256)
tls_cert_pin: "21:3A:73:9A:B5:EE:8A:83:F8:50:E8:F9:4E:0C:DD:28:81:29:75:1F:33:18:F9:C9:0A:9C:AB:F0:91:EF:3C:C4"
timeout: 60
retries: 2
block_group: IDS-BLOCK
ensure_primitives: true
adopt_foreign_members: false # this dedicated group is owned by durable controller state
save_config: false # transient membership; SQLite is the durable desired state
eve:
path: /var/log/suricata-leaf/eve.json
checkpoint: true # resume at the sqlite-persisted offset after restarts
alert_socket:
path: /run/suricata/alerts.sock # AF_UNIX listener owned by this controller;
# suricata unix_stream connects as the client
queue_size: 1000
audit:
max_bytes: 16777216 # rotate decisions.jsonl at 16 MiB...
keep: 3 # ...keeping three rotated files
policy:
mode: threshold
min_hits: 2
window_seconds: 300
ttl_seconds: 3600
max_blocks: 50
sids_only: [1000001, 1000030] # lab-only automatic actions; expand after validating rules
block_src: true # permit source actions for the explicitly allowed lab SIDs
allow_cidrs: # never block these (management/fabric floor)
- 10.0.249.0/31 # VIPERA<->sensor P2P link
- 192.168.88.0/27 # uplink transit
- 192.168.1.0/24 # management LAN (my workstation path)
- 192.168.100.0/24 # LEAF management LAN (sensor itself)
state_db: /var/lib/leaf-controller/state.sqlite
audit_log: /var/log/leaf-controller/decisions.jsonl
poll_interval: 0.2
router_interval: 2
batch_delay: 0.2
metrics: # stats records: counting only, never policy input
reconcile_interval: 30 # = suricata stats period; one health line per window
capture_interval: 3600 # capture.log metrics line cadence (s)
capture_log: /var/log/leaf-controller/capture.log
capture_max_bytes: 10485760 # rotate at 10 MiB...
capture_log_keep: 6 # ...keeping six rotated files
log_level: INFO

32
controller/config.yaml Normal file
View File

@@ -0,0 +1,32 @@
# Generic deployment example. The lab installer seeds config/prototype-vyos.yaml.
vyos:
url: https://10.66.0.1 # VyOS management address
key_env: LEAF_VYOS_KEY # API key read from this env var (never stored here)
ca_bundle: /etc/leaf-controller/vyos-ca.pem # CA that signed router cert (see snippet)
tls_verify: true
timeout: 10
retries: 3
block_group: IDS-BLOCK
ensure_primitives: true
adopt_foreign_members: true
save_config: false # true = persist config on router after each change
eve:
path: /var/log/suricata-leaf/eve.json
policy:
mode: threshold # threshold | immediate
min_hits: 2
window_seconds: 300
ttl_seconds: 3600
max_blocks: 50
allow_cidrs: [] # additional never-block CIDRs
sids_only: [] # non-empty = only act on listed sids
sids_deny: []
block_src: false # honor metadata leaf_action=block_src
state_db: /var/lib/leaf-controller/state.sqlite
audit_log: /var/log/leaf-controller/decisions.jsonl
poll_interval: 0.2
sync_interval: 300.0

View File

@@ -0,0 +1,30 @@
[Unit]
Description=LEAF alert controller and VyOS reconciliation worker
# ordering contract: the controller binds the alert socket BEFORE the
# sensor unit starts (suricata is the unix_stream CLIENT); do not reorder.
After=network-online.target
Wants=network-online.target
RequiresMountsFor=/var/log/suricata-leaf
[Service]
Type=notify
NotifyAccess=main
TimeoutStartSec=30
EnvironmentFile=/etc/leaf-controller/env
Environment=PYTHONDONTWRITEBYTECODE=1
ExecStart=/usr/bin/python3 -m leaf_controller.main --config /etc/leaf-controller/config.yaml
WorkingDirectory=/opt/leaf-controller/controller
Restart=on-failure
RestartSec=5
TimeoutStopSec=180
User=leafctrl
Group=leafctrl
UMask=0027
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/leaf-controller /var/log/leaf-controller /run/suricata
PrivateTmp=yes
[Install]
WantedBy=multi-user.target

View File

@@ -0,0 +1,2 @@
"""leaf_controller — Suricata EVE -> VyOS dynamic block controller (prototype)."""
__version__ = "0.1.0"

View File

@@ -0,0 +1,170 @@
"""Bounded AF_UNIX EVE ingress. Accepted queues are volatile until policy persists them.
Suricata can also lose records before delivery on EAGAIN. Local counters do
not prove end-to-end losslessness; this is a lab control plane, not a journal.
"""
from __future__ import annotations
import fcntl
import json
import logging
import os
import queue
import socket
import stat
import threading
log = logging.getLogger('leaf.sock')
MAX_RECORD = 1 << 20
class AlertSocket:
def __init__(self, path, queue_size=1000, rcvbuf=1 << 20):
if int(queue_size) <= 0:
raise ValueError('alert_socket.queue_size must be positive')
self.path = path
self.q = queue.Queue(maxsize=int(queue_size))
self.counts = dict(connections=0, records=0, parse_errors=0, queue_drops=0, other=0)
self.available = threading.Event()
self._rcvbuf = int(rcvbuf)
self._sock = self._conn = self._thread = self._lock = self._identity = None
self._stop = threading.Event()
def bind(self):
os.makedirs(os.path.dirname(os.path.abspath(self.path)), exist_ok=True)
self._lock = open(self.path + '.lock', 'a')
try:
fcntl.flock(self._lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
if os.path.lexists(self.path):
st = os.lstat(self.path)
if not stat.S_ISSOCK(st.st_mode):
raise OSError('refusing to replace a non-socket path')
with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as probe:
probe.settimeout(0.2)
try:
probe.connect(self.path)
except (ConnectionRefusedError, FileNotFoundError):
os.unlink(self.path)
else:
raise OSError('another listener owns this socket')
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
self._sock = sock
sock.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, self._rcvbuf)
sock.settimeout(0.2)
sock.bind(self.path)
st = os.lstat(self.path)
self._identity = (st.st_dev, st.st_ino)
os.chmod(self.path, 0o660)
sock.listen(1)
except BaseException:
self.close()
raise
def start(self):
self._thread = threading.Thread(target=self._accept_loop, name='alert-sock', daemon=True)
self._thread.start()
def _accept_loop(self):
while not self._stop.is_set():
try:
conn, _ = self._sock.accept()
except socket.timeout:
continue
except OSError:
break
self._conn = conn
self.counts['connections'] += 1
try:
conn.settimeout(0.2)
conn.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, self._rcvbuf)
self._read_conn(conn)
except OSError:
pass
finally:
conn.close()
self._conn = None
def _read_conn(self, conn):
pending = bytearray()
oversized = False
while not self._stop.is_set():
try:
data = conn.recv(65536)
except socket.timeout:
continue
if not data:
if pending or oversized:
self.counts['parse_errors'] += 1
return
for part in data.splitlines(keepends=True):
complete = part.endswith(b'\n')
if not oversized:
pending.extend(part)
if len(pending) > MAX_RECORD:
pending.clear()
oversized = True
if complete:
if oversized:
self.counts['parse_errors'] += 1
else:
self._enqueue(pending)
pending.clear()
oversized = False
def _enqueue(self, line):
if len(line) > MAX_RECORD:
self.counts['parse_errors'] += 1
return
try:
rec = json.loads(line)
except (ValueError, RecursionError, UnicodeError):
self.counts['parse_errors'] += 1
return
if not isinstance(rec, dict) or rec.get('event_type') != 'alert':
self.counts['other'] += 1
return
try:
self.q.put_nowait(rec)
self.counts['records'] += 1
self.available.set()
except queue.Full:
self.counts['queue_drops'] += 1
log.error('alert queue full: dropped %d', self.counts['queue_drops'])
def drain(self, limit=256):
self.available.clear()
out = []
while len(out) < limit:
try:
out.append(self.q.get_nowait())
except queue.Empty:
break
if not self.q.empty():
self.available.set()
return out
def close(self):
self._stop.set()
if self._conn:
try:
self._conn.shutdown(socket.SHUT_RDWR)
except OSError:
pass
if self._sock:
self._sock.close()
if self._thread:
self._thread.join(timeout=2)
if self._thread.is_alive():
raise RuntimeError('socket reader did not stop')
self._thread = None
self._sock = None
if self._identity:
try:
st = os.lstat(self.path)
if (st.st_dev, st.st_ino) == self._identity:
os.unlink(self.path)
except FileNotFoundError:
pass
self._identity = None
if self._lock:
self._lock.close()
self._lock = None

View File

@@ -0,0 +1,155 @@
"""Config loading + defaults. YAML file, secrets via env var only."""
from __future__ import annotations
import ipaddress
import os
from dataclasses import dataclass, field
from typing import Any
import yaml
DEFAULTS: dict[str, Any] = {
"vyos": {
"url": "https://10.66.0.1",
"key_env": "LEAF_VYOS_KEY",
"ca_bundle": None, # path to CA/pinned cert; None -> system CAs
"tls_verify": True,
"tls_cert_pin": "", # sha256 cert fingerprint; pins TLS trust (IP-hosted self-signed router cert)
"timeout": 10.0,
"retries": 3,
},
"block_group": "IDS-BLOCK",
"ensure_primitives": True, # verify group/rule hook exists on startup
"adopt_foreign_members": True, # members found on router but not in state db
"save_config": False, # POST /config-file save after every mutation
"eve": {
"path": "/var/log/suricata-leaf/eve.json",
"checkpoint": True, # resume at persisted offset across restarts
},
"alert_socket": {
# AF_UNIX listener for suricata's unix_stream output: alerts arrive
# through the socket instead of eve.json. empty path disables the
# socket path entirely (offline/mock runs keep pure file tailing).
"path": "/run/suricata/alerts.sock",
"queue_size": 1000,
},
"audit": {
"max_bytes": 16777216, # rotate decisions.jsonl at 16 MiB
"keep": 3, # rotated files retained
},
"metrics": {
# stats records are counting-only: never reach state/audit/API.
"reconcile_interval": 30.0, # one journal line per stats window
"capture_interval": 3600.0, # capture.log line cadence
"capture_log": None, # default: capture.log next to audit_log
"capture_max_bytes": 10485760, # rotate capture.log at 10 MiB...
"capture_log_keep": 6, # ... keeping six rotated files
"log_level": "INFO",
},
"policy": {
"mode": "threshold", # threshold | immediate
"min_hits": 2,
"window_seconds": 300,
"ttl_seconds": 3600,
"max_blocks": 50,
"allow_cidrs": [], # extra never-block networks
"sids_deny": [], # never act on these sids
"sids_only": [], # if non-empty: act ONLY on these sids
},
"always_allow_cidrs": [
# hard floor: never block routers/LAN/mgmt/mcast-ish space no matter what
"10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16",
"0.0.0.0/8", "127.0.0.0/8", "240.0.0.0/4",
"100.64.0.0/10", "169.254.0.0/16", "224.0.0.0/4",
"fc00::/7", "fe80::/10",
],
"state_db": "/var/lib/leaf-controller/state.sqlite",
"audit_log": "/var/log/leaf-controller/decisions.jsonl",
"poll_interval": 0.2,
"sync_interval": 300.0,
}
def _merge(base: dict, override: dict) -> dict:
out = dict(base)
for k, v in override.items():
if isinstance(v, dict) and isinstance(out.get(k), dict):
out[k] = _merge(out[k], v)
else:
out[k] = v
return out
@dataclass
class Config:
raw: dict = field(default_factory=dict)
def get(self, dotted: str, default: Any = None) -> Any:
cur: Any = self.raw
for part in dotted.split("."):
if not isinstance(cur, dict) or part not in cur:
return default
cur = cur[part]
return cur
# --- derived helpers ---
def vyos_url(self) -> str:
return self.get("vyos.url").rstrip("/")
def vyos_key(self) -> str:
env = self.get("vyos.key_env", "LEAF_VYOS_KEY")
key = os.environ.get(env, "")
if not key and not self.get("dry_run"):
raise SystemExit(f"missing API key: set env var {env}")
return key
def allow_networks(self) -> list:
nets = list(DEFAULTS["always_allow_cidrs"]) + list(self.get("always_allow_cidrs", [])) + list(self.get("policy.allow_cidrs", []))
return [ipaddress.ip_network(c, strict=False) for c in nets]
def is_allowed_ip(self, ip: str) -> bool:
try:
addr = ipaddress.ip_address(ip)
except ValueError:
return True # malformed / non-ip (eg:66 etc) -> treat as allowed, never block
return any(addr in net for net in self.allow_networks())
def load_config(path: str) -> Config:
with open(path) as fh:
user = yaml.safe_load(fh) or {}
if not isinstance(user, dict):
raise SystemExit(f"{path}: expected a mapping")
cfg = Config(raw=_merge(DEFAULTS, user))
import math
for name in ('alert_socket.queue_size', 'policy.min_hits', 'policy.max_blocks',
'policy.window_seconds', 'policy.ttl_seconds', 'poll_interval',
'sync_interval', 'router_interval', 'vyos.timeout', 'vyos.retries'):
value = cfg.get(name, 2 if name == 'router_interval' else None)
if isinstance(value, bool) or not isinstance(value, (int, float)) or not math.isfinite(value) or value <= 0:
raise ValueError(name + ' must be a positive finite number')
for name in ('alert_socket.queue_size', 'policy.min_hits', 'policy.max_blocks', 'vyos.retries'):
if int(cfg.get(name)) != cfg.get(name):
raise ValueError(name + ' must be an integer')
for name in ('audit.max_bytes', 'audit.keep', 'metrics.capture_max_bytes', 'metrics.capture_log_keep'):
v = cfg.get(name)
if isinstance(v,bool) or not isinstance(v,int) or v < 0:
raise ValueError(name + ' must be a non-negative integer')
if cfg.get('policy.mode') not in ('threshold', 'immediate'):
raise ValueError('policy.mode must be threshold or immediate')
cfg.allow_networks() # Validate every configured CIDR at startup.
return cfg
def read_key_env(path, name='LEAF_VYOS_KEY'):
"""Read the simple quoted assignment emitted by the credential tool; never eval."""
import shlex
with open(path) as fh:
for line in fh:
key, sep, value = line.strip().partition('=')
if sep and key == name:
parts = shlex.split(value, comments=True)
if len(parts) != 1 or not parts[0]:
raise ValueError('invalid API key assignment')
return parts[0]
raise ValueError('API key assignment missing')

View File

@@ -0,0 +1,106 @@
"""Read complete EVE records, preserving partial writes and handling rotation."""
from __future__ import annotations
import json
import logging
import os
import time
log = logging.getLogger("leaf.eve")
class EveTailer:
def __init__(self, path, poll_interval=0.2, from_start=False, checkpoint=None, idle_wait=None):
self.path, self.poll, self.from_start = path, poll_interval, from_start
# (dev, ino, pos) persisted while the controller was running; if it
# matches the file on disk we resume there instead of skipping ahead.
self.checkpoint = checkpoint
self.idle_wait = idle_wait or time.sleep
self._fh = None
self._ino = None
self._pos = 0
def _open(self):
try:
fh = open(self.path, "rb")
except FileNotFoundError:
# A file created after startup must be read from its beginning.
self.from_start = True
return None
st = os.fstat(fh.fileno())
self._ino = (st.st_dev, st.st_ino)
if not self.from_start:
ck = self.checkpoint
if ck and ck[:2] != self._ino:
log.warning("eve checkpoint inode mismatch on %s; processing "
"current file from start", self.path)
fh.seek(0)
elif ck and ck[2] <= st.st_size:
fh.seek(ck[2])
elif ck:
log.warning("eve checkpoint pos %d beyond size %d; rereading %s from 0",
ck[2], st.st_size, self.path)
fh.seek(0)
else:
# First run ever: never replay pre-existing history.
fh.seek(0, os.SEEK_END)
self.from_start = True
self._fh, self._pos = fh, fh.tell()
log.info("opened eve file %s (pos=%d)", self.path, self._pos)
return fh
def _needs_reopen(self):
try:
st = os.stat(self.path)
except FileNotFoundError:
return True
return (st.st_dev, st.st_ino) != self._ino or st.st_size < (self._fh.tell() if self._fh else self._pos)
def _read_ready(self, fh):
# Never consume a partial record: the writer may append its remainder.
out = []
for _ in range(1000):
pos = fh.tell()
line = fh.readline()
if not line.endswith(b"\n"):
fh.seek(pos)
break
out.append((line.decode("utf-8", errors="replace"), fh.tell()))
return out
@staticmethod
def _parse(line):
if isinstance(line, tuple):
line = line[0]
try:
event = json.loads(line)
return event if isinstance(event, dict) else None
except (ValueError, RecursionError):
log.warning("skipping invalid EVE JSON record")
return None
def events(self):
try:
while True:
fh = self._fh or self._open()
if fh is None:
self.idle_wait(self.poll)
yield None
continue
lines = self._read_ready(fh)
for line, end in lines:
self._pos = end
parsed = self._parse(line)
if parsed is not None:
yield parsed
if self._needs_reopen() and not lines:
fh.close()
self._fh = None
continue
if not lines:
self.idle_wait(self.poll)
yield None
finally:
self.close()
def close(self):
if self._fh:
self._fh.close()
self._fh = None

View File

@@ -0,0 +1,403 @@
"""Durable alert decisions with a separate, serialized VyOS reconciliation worker."""
from __future__ import annotations
import argparse
import fcntl
import glob
import hashlib
import json
import logging
import os
import signal
import socket
import threading
import tempfile
import time
from .alertsock import AlertSocket
from .config import load_config
from .eve import EveTailer
from .metrics import Metrics
from .policy import Policy
from .state import State
from .vyos_client import VyosApiError, VyosClient
log = logging.getLogger('leaf')
class Audit:
def __init__(self, path, max_bytes=0, keep=3):
os.makedirs(os.path.dirname(os.path.abspath(path)), exist_ok=True)
self.path, self.max_bytes, self.keep = path, int(max_bytes), int(keep)
self.fh = open(path, 'a')
self.lock = threading.Lock()
def write(self, obj):
with self.lock:
self.fh.write(json.dumps(dict(ts=time.time(), **obj)) + '\n')
self.fh.flush()
if self.max_bytes and self.fh.tell() >= self.max_bytes:
self._rotate_locked()
def _rotate_locked(self):
"""Rotate at the configured size and retain the newest archives."""
self.fh.close()
stamp = str(time.time_ns()).zfill(20)
dst = self.path + '.' + stamp
n = 2
while os.path.exists(dst): # tolerate a frozen/coarse clock
dst = self.path + '.' + stamp + f'.{n:08d}'
n += 1
os.replace(self.path, dst)
self.fh = open(self.path, 'a')
rotated = sorted(glob.glob(self.path + '.*'), key=lambda f: (os.stat(f).st_mtime_ns, f))
doomed = rotated[:-self.keep] if self.keep else rotated
for old in doomed:
try:
os.remove(old)
except FileNotFoundError:
pass
log.info('rotated audit log, keeping %d rotated file(s)', self.keep or 0)
def close(self):
self.fh.close()
class Controller:
def __init__(self, cfg, dry_run=False):
self.cfg, self.dry_run = cfg, dry_run
self._closed = False
self.group = cfg.get('block_group', 'IDS-BLOCK')
self._scratch = tempfile.TemporaryDirectory(prefix='leaf-dry-run-') if dry_run else None
state_path = os.path.join(self._scratch.name, 'state.sqlite') if self._scratch else cfg.get('state_db')
os.makedirs(os.path.dirname(os.path.abspath(state_path)), exist_ok=True)
self._lock = open(state_path + '.lock', 'a')
try:
fcntl.flock(self._lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
except BlockingIOError:
self._lock.close()
raise SystemExit('another controller owns this state database')
self.state = State(state_path)
self.policy = Policy(cfg, self.state)
self.audit = Audit(os.path.join(self._scratch.name, "audit.jsonl") if dry_run else cfg.get("audit_log"),
int(cfg.get("audit.max_bytes", 16777216)),
int(cfg.get("audit.keep", 3)))
# metrics: counting only; stats records never touch state/audit/API.
# health line once per full stats window, capture.log every
# metrics.capture_interval seconds (same bounded scheme as audit).
self.metrics = Metrics(float(cfg.get("metrics.reconcile_interval", 30.0)))
self._metrics_log = logging.getLogger("leaf.metrics")
self._metrics_log.setLevel(getattr(
logging, str(cfg.get("metrics.log_level", "INFO")).upper(), logging.INFO))
self._capture_path = (cfg.get("metrics.capture_log")
or os.path.join(os.path.dirname(
os.path.abspath(cfg.get("audit_log"))), "capture.log"))
if dry_run:
self._capture_path = os.path.join(self._scratch.name, "capture.jsonl")
self._capture_interval = max(0.0, float(cfg.get("metrics.capture_interval", 3600)))
self._capture_log = None
self._next_capture = 0.0
self.client = None if dry_run else VyosClient(
cfg.vyos_url(), cfg.vyos_key(), ca_bundle=cfg.get('vyos.ca_bundle'),
tls_verify=cfg.get('vyos.tls_verify', True),
cert_pin=cfg.get('vyos.tls_cert_pin') or None,
timeout=float(cfg.get('vyos.timeout', 75)),
retries=int(cfg.get('vyos.retries', 2)))
self.stop = threading.Event()
self.wake = threading.Event()
self._last_sync = 0.0
self._initial_sync = True
self._worker = None
# socket ingestion: suricata unix_stream output connects to our
# listener; empty path disables (offline/mock runs keep file tailing)
sock_path = cfg.get('alert_socket.path')
self._sock = None
if sock_path and not dry_run:
self._sock = AlertSocket(
sock_path, int(cfg.get('alert_socket.queue_size') or 1000))
def router_members(self):
return set(self.client.op_with_retry(self.client.group_members, self.group))
def startup(self):
if self.dry_run:
return
info = self.client.info()
log.info('connected to VyOS API: version=%s', info.get('version', '?'))
if self.cfg.get('ensure_primitives', True):
# Both metadata actions use one group; validate the corresponding hooks.
self.client.verify_enforcement(self.group, self.cfg.get('policy.block_src', False))
self.sync_router_state()
def sync_router_state(self):
"""One batched commit, then read back before acknowledging any change."""
if self.dry_run:
return
now = time.time()
members = self.router_members()
rows = {r.ip: r for r in self.state.all_blocks()}
# Only adopt on startup, never resurrect known expired entries.
foreign = members - rows.keys()
if self._initial_sync and self.cfg.get('adopt_foreign_members', True):
for ip in sorted(foreign):
if self.cfg.is_allowed_ip(ip):
log.warning('leaving protected foreign router member unowned: %s', ip)
continue
if self.state.count_active() >= int(self.cfg.get('policy.max_blocks', 50)):
log.warning('foreign member adoption capped; leaving unowned: %s', ip)
continue
self.state.upsert_hit(ip, 'dest', 0, float(self.cfg.get('policy.ttl_seconds', 3600)), now)
self.state.set_applied(ip)
self.audit.write({'action': 'adopt', 'ip': ip})
log.info('adopted foreign member %s into state', ip)
rows = {r.ip: r for r in self.state.all_blocks()}
self._initial_sync = False
active = {ip for ip, r in rows.items() if r.expires > now and not self.cfg.is_allowed_ip(ip)}
expired = rows.keys() - active
adds, deletes = active - members, set(expired) & members
if not self.cfg.get('adopt_foreign_members', True):
deletes |= foreign
ops = [self.client.member_op('set', self.group, ip) for ip in sorted(adds)]
ops += [self.client.member_op('delete', self.group, ip) for ip in sorted(deletes)]
if ops:
# A timeout may mean the commit succeeded: next pass reads before retrying.
try:
self.state.set_router_dirty(True)
self.client.configure(ops)
except Exception:
self.audit.write({'action': 'reconcile', 'ok': False,
'adds': sorted(adds), 'deletes': sorted(deletes)})
raise
members = self.router_members()
if not adds <= members or deletes & members:
raise VyosApiError('router readback does not match requested membership changes')
if self.cfg.get('save_config') and self.state.router_dirty():
self.client.save_config()
self.state.set_router_dirty(False)
for ip in active & members:
row = rows[ip]
if not row.applied or ip in adds:
self.state.set_applied(ip)
self.audit.write({'action': 'block', 'ip': ip, 'kind': row.kind,
'sigs': row.sigs, 'expires': row.expires,
'ok': True, 'verified': True})
log.info('BLOCKED %s (%s), router membership verified', ip, row.kind)
for ip in expired - members:
row = rows[ip]
# Do not delete a fresh decision made during the slow router request.
removed = self.state.remove(ip, expired_before=now)
if removed:
self.audit.write({'action': 'unblock', 'ip': ip, 'kind': row.kind,
'reason': 'ttl-expired', 'ok': True, 'verified': True})
log.info('UNBLOCKED %s, router absence verified', ip)
elif self.cfg.is_allowed_ip(ip):
# A newly protected address must not remain locally desired.
self.state.remove(ip)
else:
self.state.set_applied(ip, 0)
self.wake.set()
self._last_sync = time.time()
def _route(self, event):
"""Dispatch by event_type; only alerts can act. Stats records are
counting-only: they must never reach policy/state/audit/API."""
etype = event.get('event_type')
if etype == 'alert':
self.metrics.note_alert()
self.handle_event(event)
elif etype == 'stats':
try:
self.metrics.note_stats(event)
except (TypeError, ValueError, AttributeError, OverflowError, RecursionError):
log.warning('ignoring malformed stats record')
self.metrics.note_other()
else:
self.metrics.note_other()
def _sock_desc(self):
if self._sock is None:
return 'off(eve-file)'
c = self._sock.counts
return ('rec:%d conn:%d qdrop:%d perr:%d other:%d'
% (c['records'], c['connections'], c['queue_drops'],
c['parse_errors'], c['other']))
def _emit_metrics(self, now):
"""One health line per full stats window; capture.log line on cadence.
Read-only against controller state (count_active is a query)."""
rep = self.metrics.report(now)
if rep is None:
return
if rep['init']:
self._metrics_log.info('metrics baseline: %s',
json.dumps(rep['stats'], separators=(',', ':')))
return
flag = ' (counters reset: suricata restart)' if rep['restart'] else ''
self._metrics_log.info(
'health: alerts=%d cps=%.1f eps=%.1f dropped+=%d errors+=%d '
'invalid+=%d suppressed+=%d overflow+=%d blocks=%d uptime=%ds sock=%s%s',
rep['alerts'], rep['cps'], rep['eps'], rep['dropped'],
rep['delta']['capture_errors'], rep['delta']['invalid'],
rep['delta']['suppressed'], rep['delta']['queue_overflow'],
self.state.count_active(), rep['stats']['uptime'],
self._sock_desc(), flag)
if now >= self._next_capture:
self._next_capture = now + self._capture_interval
if self._capture_log is None:
self._capture_log = Audit(
self._capture_path,
int(self.cfg.get('metrics.capture_max_bytes', 10485760)),
int(self.cfg.get('metrics.capture_log_keep', 6)))
self._capture_log.write({'window': rep['window'],
'alerts': rep['alerts'],
'cps': rep['cps'], 'eps': rep['eps'],
'dropped': rep['dropped'],
'delta': rep['delta'],
'stats': rep['stats']})
def handle_event(self, event):
try:
dec = self.policy.evaluate(event)
except (TypeError, ValueError, AttributeError, OverflowError, RecursionError):
log.warning('ignoring malformed alert record')
return
if not dec:
return
existing = self.state.get(dec.ip)
if (not existing or existing.expires <= time.time()) and self.state.count_active() >= int(self.cfg.get('policy.max_blocks', 50)):
self.audit.write({'action': 'guard', 'ip': dec.ip, 'sid': dec.sid, 'reason': 'max_blocks'})
return
row = self.state.upsert_hit(dec.ip, dec.kind, dec.sid,
float(self.cfg.get('policy.ttl_seconds', 3600)))
if not row.applied:
self.audit.write({'action': 'decision', 'ip': dec.ip, 'kind': dec.kind,
'sid': dec.sid, 'expires': row.expires, 'dry_run': self.dry_run})
if self.dry_run:
if not row.applied:
self.audit.write({'action': 'block', 'ip': dec.ip, 'kind': dec.kind, 'dry_run': True})
self.state.set_applied(dec.ip)
else:
self.wake.set()
def housekeeping(self):
now = time.time()
self.state.prune_hits(float(self.cfg.get('policy.window_seconds', 300)) * 2, now)
if self.dry_run:
for row in self.state.expired(now):
self.state.remove(row.ip, expired_before=now)
return
if (self.cfg.get('save_config') and self.state.router_dirty()) or self.state.unapplied() or self.state.expired(now) or now - self._last_sync >= float(self.cfg.get('sync_interval', 300)):
self.sync_router_state()
def _router_loop(self):
ready = False
try:
while not self.stop.is_set():
try:
if not ready:
self.startup()
ready = True
else:
self.housekeeping()
except Exception as exc:
log.error('router reconciliation failed; decisions retained for retry: %s', exc)
self.wake.wait(float(self.cfg.get('router_interval', 2)))
self.wake.clear()
# A short batching window combines bursts into one commit.
self.stop.wait(float(self.cfg.get('batch_delay', 0.2)))
finally:
self.state.close()
def _save_checkpoint(self, tailer):
if self.dry_run or tailer._ino is None:
return
self.state.save_checkpoint(tailer._ino[0], tailer._ino[1], tailer._pos)
def run(self):
if self._sock:
# bind BEFORE suricata connects: the sensor unit's ordering
# guarantees it (Wants/After leaf-controller). a bind failure is
# fatal, not soft: a dead listener would mean silent alert loss.
try:
self._sock.bind()
except OSError as exc:
raise SystemExit('cannot bind alert socket %s: %s'
% (self._sock.path, exc))
self._sock.start()
log.info('listening for sensor alerts on %s', self._sock.path)
ckpt = self.state.get_checkpoint() if (not self.dry_run and self.cfg.get('eve.checkpoint', True)) else None
tailer = EveTailer(self.cfg.get('eve.path'), float(self.cfg.get('poll_interval', 0.2)),
from_start=self.dry_run, checkpoint=ckpt,
idle_wait=self._sock.available.wait if self._sock else self.stop.wait)
# Establish the live position before any slow API startup calls.
tailer._open()
self._save_checkpoint(tailer)
notify = os.environ.get('NOTIFY_SOCKET')
if notify:
with socket.socket(socket.AF_UNIX, socket.SOCK_DGRAM) as ready:
ready.connect('\0' + notify[1:] if notify.startswith('@') else notify)
ready.sendall(b'READY=1')
if not self.dry_run:
self._worker = threading.Thread(target=self._router_loop, name='vyos-worker')
self._worker.start()
log.info('watching %s (mode=%s)', self.cfg.get('eve.path'), self.cfg.get('policy.mode'))
try:
for event in tailer.events():
if self.stop.is_set():
break
if self._sock:
for rec in self._sock.drain():
self._route(rec)
if event is not None:
# Socket mode has exactly one alert ingress; the file carries stats.
if not self._sock or event.get('event_type') != 'alert':
identity = hashlib.sha256(json.dumps(event, sort_keys=True).encode()).hexdigest()
self.state._local.event_key = '%s:%s:%s:%s' % (*tailer._ino, tailer._pos, identity)
try:
self._route(event)
finally:
self.state._local.event_key = None
self._emit_metrics(time.time())
# Offset follows the delivered record, never the unread batch.
# File mode is at-least-once across a crash before checkpoint.
self._save_checkpoint(tailer)
elif self.dry_run:
self.housekeeping()
finally:
tailer.close()
self.close()
def close(self):
if self._closed:
return
self._closed = True
if self._sock:
self._sock.close()
# Stop intake first, then persist every accepted queued alert.
for rec in self._sock.drain(limit=self._sock.q.qsize()):
self._route(rec)
self.stop.set()
self.wake.set()
if self._worker:
self._worker.join()
self._worker = None
self.state.close()
self.audit.close()
if self._capture_log:
self._capture_log.close()
self._lock.close()
if self._scratch:
self._scratch.cleanup()
def main(argv=None):
ap = argparse.ArgumentParser()
ap.add_argument('--config', default=os.path.join(os.path.dirname(__file__), '..', 'config.yaml'))
ap.add_argument('--dry-run', action='store_true')
ap.add_argument('-v', '--verbose', action='store_true')
args = ap.parse_args(argv)
logging.basicConfig(level=logging.DEBUG if args.verbose else logging.INFO,
format='%(asctime)s %(levelname)s %(name)s: %(message)s')
ctl = Controller(load_config(args.config), args.dry_run)
def shutdown(*args):
ctl.stop.set()
ctl.wake.set()
signal.signal(signal.SIGTERM, shutdown)
signal.signal(signal.SIGINT, shutdown)
ctl.run()
return 0
if __name__ == '__main__':
raise SystemExit(main())

View File

@@ -0,0 +1,105 @@
"""Counting-only metrics from Suricata stats records.
Stats records update counters without changing policy, SQLite state or
router configuration. The caller controls when reports are logged.
Live eve.json (Suricata 7, threads: no + totals: yes) carries
capture/decoder/detect sub-objects on each stats record; there is no
'ifaces' object at totals level, so capture packets come from
capture.kernel_packets (the real field on the wire).
"""
from __future__ import annotations
class Metrics:
"""Cumulative Suricata counters; one delta report per full window.
The report gate exists because stats records arrive every
metrics.reconcile_interval seconds; a backlog drain can deliver two
stats records back-to-back, and the journal must still see at most one
line per window.
"""
def __init__(self, reconcile_interval: float = 30.0):
self.interval = max(1.0, float(reconcile_interval))
self.alerts = 0
self.non_alert_events = 0
self.stats_seen = 0
self._totals: dict[str, int] = {}
self._baseline: dict[str, int] | None = None
self._last = 0.0
def note_alert(self) -> None:
self.alerts += 1
def note_other(self) -> None:
self.non_alert_events += 1
def note_stats(self, event: dict) -> None:
"""Extract the cumulative counters; nothing else is logged or stored.
Live suricata emits both thread records (flush=False: per-thread
counters, 'engine' key) and a totals record (uptime/memcap_pressure
keys, 'detect' key). Totals are authoritative for rates; thread
records are only accepted if no totals have ever been seen.
"""
stats = event.get("stats") or {}
cap = stats.get("capture") or {}
dec = stats.get("decoder") or {}
det = stats.get("detect") or stats.get("engine") or {}
totals = {
"uptime": int(stats.get("uptime", 0)),
"memcap_pressure": int(stats.get("memcap_pressure", 0)),
"packets": int(cap.get("kernel_packets", 0)),
"dropped": int(cap.get("kernel_drops", 0)),
"capture_errors": int(cap.get("errors", 0)),
"decoded": int(dec.get("pkts", 0)),
"bytes": int(dec.get("bytes", 0)),
"invalid": int(dec.get("invalid", 0)),
"engine_alerts": int(det.get("alert", 0)),
"suppressed": int(det.get("alerts_suppressed", 0)),
"queue_overflow": int(det.get("alert_queue_overflow", 0)),
}
if "uptime" in stats or not self._totals:
self._totals = totals
self.stats_seen += 1
def report(self, now: float) -> dict | None:
"""None unless a full flush window has passed since the last report.
First stats record only adopts the cumulative baseline: deltas are
meaningless before the second stats tick (counters run since process
start), so the first window must not pretend to be a measured one.
Suricata's counters reset on restart: deltas never go negative, the
reset is flagged instead.
"""
if not self._totals:
return None
if self._baseline is None:
self._baseline = dict(self._totals)
self._last = now
return {"init": True, "alerts": self.alerts, "stats": dict(self._totals)}
if now - self._last < self.interval * 0.75: # tolerate flush jitter
return None
window = now - self._last
delta, restart = {}, False
for key, value in self._totals.items():
n = value - self._baseline.get(key, 0)
if n < 0:
restart = True
n = value
delta[key] = n
self._baseline = dict(self._totals)
self._last = now
return {
"init": False,
"restart": restart,
"window": round(window, 1),
"alerts": self.alerts,
"engine_alerts_delta": delta["engine_alerts"],
"cps": round(delta["packets"] / window, 1), # captured pps
"eps": round(delta["decoded"] / window, 1), # decoded pps
"dropped": delta["dropped"],
"delta": delta,
"stats": dict(self._totals),
}

View File

@@ -0,0 +1,22 @@
"""Sensor-side operational helpers sharing the installed controller trust settings."""
import os
from .config import load_config, read_key_env
from .vyos_client import VyosClient
def configured_client():
cfg=load_config('/etc/leaf-controller/config.yaml')
name=cfg.get('vyos.key_env','LEAF_VYOS_KEY')
key=os.environ.get(name) or read_key_env('/etc/leaf-controller/env', name)
client=VyosClient(cfg.vyos_url(),key,cert_pin=cfg.get('vyos.tls_cert_pin') or None,
ca_bundle=cfg.get('vyos.ca_bundle'),tls_verify=cfg.get('vyos.tls_verify',True),
timeout=float(cfg.get('vyos.timeout',75)),retries=int(cfg.get('vyos.retries',2)))
return cfg,client
def redact(node):
if isinstance(node,dict):
return {k: '[REDACTED]' if any(s in k.lower() for s in ('key','password','secret','token','private'))
else redact(v) for k,v in node.items()}
if isinstance(node,list):return [redact(v) for v in node]
return node

View File

@@ -0,0 +1,91 @@
"""Alert -> decision. Rule metadata leaf_action drives intent, policy gates it."""
from __future__ import annotations
import logging
import time
from dataclasses import dataclass, field
from typing import Any
log = logging.getLogger("leaf.policy")
@dataclass
class Decision:
ip: str
kind: str # dest | src
sid: int
reason: str
sig: str = ""
hits: int = 1
class Policy:
def __init__(self, cfg, state):
self.cfg = cfg
self.state = state
@staticmethod
def metadata_actions(event: dict) -> list[str]:
md = event.get("alert", {}).get("metadata", {}) or {}
vals = md.get("leaf_action", []) or []
if isinstance(vals, str):
vals = [vals]
return [str(v).strip().lower() for v in vals]
def evaluate(self, event: dict, now: float | None = None) -> Decision | None:
"""Return Decision to block, or None. `event` = eve alert record."""
now = now or time.time()
if event.get("event_type") != "alert":
return None
alert = event.get("alert", {})
sid = int(alert.get("signature_id", -1))
sig = str(alert.get("signature", ""))
actions = self.metadata_actions(event)
if not actions:
return None
sids_only = [int(s) for s in self.cfg.get("policy.sids_only", [])]
sids_deny = [int(s) for s in self.cfg.get("policy.sids_deny", [])]
if sids_only and sid not in sids_only:
return None
if sid in sids_deny:
return None
mode = self.cfg.get("policy.mode", "threshold")
src_ip = str(event.get("src_ip", ""))
dest_ip = str(event.get("dest_ip", ""))
candidates: list[tuple[str, str]] = []
if "block" in actions:
candidates.append((dest_ip, "dest"))
if "block_src" in actions and self.cfg.get("policy.block_src", False):
candidates.append((src_ip, "src"))
for ip, kind in candidates:
dec = self._gate(ip, kind, sid, sig, now, mode)
if dec:
return dec
return None
def _gate(self, ip: str, kind: str, sid: int, sig: str,
now: float, mode: str) -> Decision | None:
if not ip:
return None
import ipaddress
try:
if ipaddress.ip_address(ip).version != 4:
return None
except ValueError:
return None
if self.cfg.is_allowed_ip(ip):
log.debug("allowlisted, not blocking: %s (sid=%s)", ip, sid)
return None
self.state.record_hit(sid, ip, now)
min_hits = int(self.cfg.get("policy.min_hits", 1))
window = float(self.cfg.get("policy.window_seconds", 300))
hits = self.state.hit_count(sid, ip, window, now)
if mode == "immediate" or hits >= min_hits:
return Decision(ip=ip, kind=kind, sid=sid, sig=sig, hits=hits,
reason=f"{mode}:{hits} hit(s) sid={sid} in {window:.0f}s")
log.info("hit %d/%d for %s (sid=%s) — not yet blocking", hits, min_hits, ip, sid)
return None

View File

@@ -0,0 +1,176 @@
"""sqlite-backed state: block list w/ TTL, hit windowing, applied tracking."""
from __future__ import annotations
import sqlite3
import time
import threading
from dataclasses import dataclass
SCHEMA = """
CREATE TABLE IF NOT EXISTS controller_meta (key TEXT PRIMARY KEY, value TEXT NOT NULL);
CREATE TABLE IF NOT EXISTS blocks (
ip TEXT PRIMARY KEY, -- ip/cidr as pushed to VyOS (plain ip = /32 semantics)
kind TEXT NOT NULL, -- dest|src
sigs TEXT NOT NULL DEFAULT '',-- csv of triggering sids
first_seen REAL NOT NULL,
last_hit REAL NOT NULL,
expires REAL NOT NULL,
applied INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE IF NOT EXISTS hits (
sid INTEGER NOT NULL,
ip TEXT NOT NULL,
ts REAL NOT NULL
);
CREATE INDEX IF NOT EXISTS hits_idx ON hits (sid, ip, ts);
CREATE TABLE IF NOT EXISTS eve_ckpt (
id INTEGER PRIMARY KEY CHECK (id = 1),
dev INTEGER NOT NULL, -- st_dev/st_ino of the EVE file last read
ino INTEGER NOT NULL,
pos INTEGER NOT NULL, -- byte offset of the next unread record
updated_at REAL NOT NULL
);
"""
@dataclass
class BlockRow:
ip: str
kind: str
sigs: str
first_seen: float
last_hit: float
expires: float
applied: int
class State:
def __init__(self, path: str):
self.path = path
self._local = threading.local()
self.con.executescript(SCHEMA)
columns = {r[1] for r in self.con.execute("PRAGMA table_info(hits)")}
if "event_key" not in columns:
self.con.execute("ALTER TABLE hits ADD COLUMN event_key TEXT")
self.con.execute("CREATE UNIQUE INDEX IF NOT EXISTS hits_event ON hits(event_key)")
self.con.commit()
@property
def con(self):
# Alert processing and router I/O have independent connections.
if not hasattr(self._local, "con"):
con = sqlite3.connect(self.path, timeout=10)
con.execute("PRAGMA journal_mode=WAL")
con.execute("PRAGMA busy_timeout=10000")
self._local.con = con
return self._local.con
def close(self):
if hasattr(self._local, "con"):
self._local.con.close()
del self._local.con
# ---- hit windowing (threshold mode) ----
def record_hit(self, sid: int, ip: str, ts: float | None = None) -> None:
event_key = getattr(self._local, "event_key", None)
if event_key is not None:
event_key = f'{event_key}:{sid}:{ip}'
self.con.execute("INSERT OR IGNORE INTO hits (sid, ip, ts, event_key) VALUES (?,?,?,?)",
(sid, ip, ts or time.time(), event_key))
self.con.commit()
def hit_count(self, sid: int, ip: str, window: float, now: float | None = None) -> int:
now = now or time.time()
(n,) = self.con.execute(
"SELECT count(*) FROM hits WHERE sid=? AND ip=? AND ts>=?",
(sid, ip, now - window)).fetchone()
return n
def prune_hits(self, max_age: float, now: float | None = None) -> None:
now = now or time.time()
self.con.execute("DELETE FROM hits WHERE ts < ?", (now - max_age,))
self.con.commit()
# ---- blocks ----
def _row(self, r: sqlite3.Row | tuple) -> BlockRow:
return BlockRow(*r)
def get(self, ip: str) -> BlockRow | None:
r = self.con.execute("SELECT * FROM blocks WHERE ip=?", (ip,)).fetchone()
return self._row(r) if r else None
def upsert_hit(self, ip: str, kind: str, sid: int, ttl: float,
now: float | None = None) -> BlockRow:
"""Create block entry or refresh expiry/last_hit of existing one."""
now = now or time.time()
# Serialize the read/modify/write with expiry cleanup in the worker.
with self.con:
self.con.execute("BEGIN IMMEDIATE")
row = self.get(ip)
if row:
sigs = row.sigs
if str(sid) not in sigs.split(","):
sigs = f"{sigs},{sid}" if sigs else str(sid)
self.con.execute(
"UPDATE blocks SET expires=?, last_hit=?, sigs=? WHERE ip=?",
(now + ttl, now, sigs, ip))
else:
self.con.execute("INSERT INTO blocks VALUES (?,?,?,?,?,?,0)",
(ip, kind, str(sid), now, now, now + ttl))
result = self.get(ip)
return result
def set_applied(self, ip: str, applied: int = 1) -> None:
self.con.execute("UPDATE blocks SET applied=? WHERE ip=?", (applied, ip))
self.con.commit()
def remove(self, ip: str, expired_before: float | None = None) -> bool:
# A new alert may refresh a block while the router commit is in flight.
with self.con:
if expired_before is None:
cur = self.con.execute("DELETE FROM blocks WHERE ip=?", (ip,))
else:
cur = self.con.execute("DELETE FROM blocks WHERE ip=? AND expires<=?",
(ip, expired_before))
if cur.rowcount:
self.con.execute("DELETE FROM hits WHERE ip=?", (ip,))
return bool(cur.rowcount)
def all_blocks(self) -> list[BlockRow]:
return [self._row(r) for r in self.con.execute("SELECT * FROM blocks")]
def expired(self, now: float | None = None) -> list[BlockRow]:
now = now or time.time()
return [self._row(r) for r in
self.con.execute("SELECT * FROM blocks WHERE expires<=?", (now,))]
def unapplied(self) -> list[BlockRow]:
return [self._row(r) for r in
self.con.execute("SELECT * FROM blocks WHERE applied=0")]
def count_active(self, now: float | None = None) -> int:
now = now or time.time()
(n,) = self.con.execute("SELECT count(*) FROM blocks WHERE expires>?", (now,)).fetchone()
return n
# ---- EVE read checkpoint (survives controller restarts) ----
def get_checkpoint(self) -> tuple | None:
r = self.con.execute("SELECT dev, ino, pos FROM eve_ckpt WHERE id=1").fetchone()
return tuple(r) if r else None
def save_checkpoint(self, dev: int, ino: int, pos: int, now: float | None = None) -> None:
with self.con:
self.con.execute(
"INSERT INTO eve_ckpt VALUES (1,?,?,?,?) "
"ON CONFLICT(id) DO UPDATE SET dev=excluded.dev, ino=excluded.ino, "
"pos=excluded.pos, updated_at=excluded.updated_at",
(dev, ino, pos, now or time.time()))
def router_dirty(self):
r = self.con.execute("SELECT value FROM controller_meta WHERE key='router_dirty'").fetchone()
return bool(r and r[0] == '1')
def set_router_dirty(self, dirty):
with self.con:
self.con.execute("INSERT OR REPLACE INTO controller_meta VALUES ('router_dirty', ?)",
('1' if dirty else '0',))

View File

@@ -0,0 +1,236 @@
"""Minimal VyOS HTTP API client (Stream 2026.03 / rolling: service https api rest).
Protocol (docs: automation/vyos-api.html):
POST {base}/configure form fields: data=<json>, key=<key>
POST {base}/retrieve {"op":"returnValues","path":[tokens]} -> {"returnValues":[...]}
POST {base}/show {"op":"show","path":[...]}
POST {base}/config-file {"op":"save"}
GET {base}/info version/health, no auth
Multipart/form-data encoding, exactly what curl -F sends.
"""
from __future__ import annotations
import hashlib
import json
import logging
import socket
import ssl
import time
from typing import Any
import requests
from requests.adapters import HTTPAdapter
from urllib3.connection import HTTPSConnection
from urllib3.connectionpool import HTTPSConnectionPool
log = logging.getLogger("leaf.vyos")
class VyosApiError(RuntimeError):
def __init__(self, msg: str, status: int | None = None, body: Any = None):
super().__init__(msg)
self.status = status
self.body = body
def normalize_pin(pin: str) -> str:
"""Accept openssl-style 'AA:BB:..' or plain hex; return lowercase hex digest."""
p = pin.strip().lower().removeprefix("sha256:").replace(":", "")
if len(p) != 64 or any(c not in "0123456789abcdef" for c in p):
raise ValueError("tls_cert_pin must be a sha256 certificate fingerprint (hex)")
return p
class PinMismatchError(VyosApiError):
pass
class _PinnedHTTPSConnection(HTTPSConnection):
"""TLS to an IP-hosted self-signed endpoint, anchored on a pinned cert.
Hostname verification is impossible for this router's cert (CN=localhost,
no IP SAN), so trust comes from comparing the peer's SHA-256 fingerprint
against the configured pin. CERT_NONE is safe *because* the pin is the
trust anchor: a mismatch aborts the connection before any request.
"""
leaf_pin: str | None = None
def connect(self):
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE
sock = socket.create_connection((self.host, self.port), self.timeout,
self.source_address)
self.sock = ctx.wrap_socket(sock, server_hostname=self.host)
der = self.sock.getpeercert(binary_form=True)
if der is None:
raise PinMismatchError("TLS handshake returned no peer certificate")
digest = hashlib.sha256(der).hexdigest()
if digest != self.leaf_pin:
self.sock.close()
raise PinMismatchError(
f"TLS cert fingerprint mismatch for {self.host}; peer is "
f"{digest}, pinned is {self.leaf_pin} (possible MiTM?)")
# Pin matched: this connection IS authenticated to the pinned anchor.
self.is_verified = True
return None
class VyosClient:
def __init__(self, url: str, key: str, ca_bundle: str | None = None,
tls_verify: bool = True, timeout: float = 10.0, retries: int = 3,
cert_pin: str | None = None):
self.url = url.rstrip("/")
self.key = key
self.timeout = timeout
self.retries = retries
self.sess = requests.Session()
# Do not bypass the pinned adapter through an environment HTTP proxy.
self.sess.trust_env = False
if cert_pin:
pin = normalize_pin(cert_pin)
class PinAdapter(HTTPAdapter):
def init_poolmanager(self, *args, **kwargs):
super().init_poolmanager(*args, **kwargs)
conn_cls = type(
"PinnedConn", (_PinnedHTTPSConnection,),
{"leaf_pin": pin})
self.poolmanager.pool_classes_by_scheme = dict(self.poolmanager.pool_classes_by_scheme)
self.poolmanager.pool_classes_by_scheme["https"] = type(
"PinnedPool", (HTTPSConnectionPool,),
{"ConnectionCls": conn_cls})
return self.poolmanager
self.sess.mount("https://", PinAdapter())
elif ca_bundle:
self.sess.verify = ca_bundle
else:
self.sess.verify = bool(tls_verify)
if self.sess.verify is False:
requests.packages.urllib3.disable_warnings() # explicit opt-in only
# ------------------------------------------------------------------ core
def _post(self, endpoint: str, payload: dict, auth: bool = True) -> dict:
# multipart/form-data exactly like `curl -F "data=..." -F "key=..."`:
# part tuples (None, value) -> no filename -> plain form field
fields = {"data": (None, json.dumps(payload))}
if auth:
fields["key"] = (None, self.key)
resp = self.sess.post(f"{self.url}{endpoint}", files=fields, timeout=self.timeout)
try:
body = resp.json()
except ValueError:
body = resp.text
if resp.status_code >= 400:
raise VyosApiError(f"{endpoint} -> HTTP {resp.status_code}: {body!r}",
status=resp.status_code, body=body)
# VyOS returns success:false + error for semantic failures (often 200/400)
if isinstance(body, dict) and body.get("success") is False:
raise VyosApiError(f"{endpoint} rejected: {body.get('error')!r}",
status=resp.status_code, body=body)
if not isinstance(body, dict) or body.get("success") is not True:
raise VyosApiError(f"{endpoint}: missing success acknowledgement")
return body
# ------------------------------------------------------------- public API
def info(self) -> dict:
resp = self.sess.get(f"{self.url}/info", timeout=self.timeout)
if resp.status_code >= 400:
raise VyosApiError(f"/info -> HTTP {resp.status_code}", status=resp.status_code)
body = resp.json()
if isinstance(body, dict) and isinstance(body.get("data"), dict):
return body["data"]
return body
def configure(self, ops: list[dict] | dict) -> dict:
"""One commit: a single op dict or a list of ops (atomic multi-op)."""
payload = ops if isinstance(ops, list) else ops
return self._post("/configure", payload)
def retrieve(self, op: str, path: list[str]) -> dict:
return self._post("/retrieve", {"op": op, "path": path})
def group_members(self, group: str) -> list[str]:
body = self.retrieve("returnValues",
["firewall", "group", "address-group", group, "address"])
# live 2026.03 returns {"success":true,"data":[...]}; older builds used
# {"returnValues":[...]}. Accept both, never silently hide members.
vals = body.get("data")
if vals is None:
if "returnValues" not in body and "data" not in body:
raise VyosApiError("group member response contains no values field")
vals = body.get("returnValues") or []
if not isinstance(vals, list):
vals = [vals] if vals else []
return [str(v) for v in vals]
@staticmethod
def member_op(op, group, ip):
import ipaddress
ipaddress.IPv4Address(ip) # This prototype manages IPv4 address groups only.
return {"op": op, "path": ["firewall", "group", "address-group", group, "address", ip]}
def set_group_member(self, group: str, ip: str) -> dict:
return self.configure(self.member_op("set", group, ip))
def del_group_member(self, group: str, ip: str) -> dict:
return self.configure(self.member_op("delete", group, ip))
def verify_enforcement(self, group, require_src=False):
fw = self.retrieve("showConfig", ["firewall"]).get("data", {})
if not isinstance(fw, dict):
raise VyosApiError("cannot inspect firewall enforcement hooks")
directions = set()
def walk(obj):
if not isinstance(obj, dict):
return
if obj.get("action") == "drop":
for side in ("source", "destination"):
if obj.get(side, {}).get("group", {}).get("address-group") == group:
directions.add(side)
for value in obj.values():
if isinstance(value, dict):
walk(value)
ipv4 = fw.get("ipv4", {})
walk(ipv4.get("forward", {}))
attached = set()
def attachments(obj):
if not isinstance(obj, dict):
return
hook = obj.get("firewall", {})
if isinstance(hook, dict) and isinstance(hook.get("name"), str):
attached.add(hook["name"])
for value in obj.values():
if isinstance(value, dict):
attachments(value)
attachments(fw.get("zone", {}))
for name in attached:
walk(ipv4.get("name", {}).get(name, {}))
required = {"destination", "source"} if require_src else {"destination"}
if not required <= directions:
raise VyosApiError("missing drop hooks for " + ", ".join(sorted(required - directions)))
def show(self, command: list[str]) -> str:
body = self._post("/show", {"op": "show", "path": command})
out = body.get("output")
if out is None:
out = body.get("data", "")
return str(out)
def save_config(self) -> dict:
return self._post("/config-file", {"op": "save"})
# ----------------------------------------------------- retrying operations
def op_with_retry(self, fn, *args, **kwargs) -> Any:
last: Exception | None = None
for attempt in range(1, self.retries + 1):
try:
return fn(*args, **kwargs)
except (requests.RequestException, VyosApiError) as exc:
last = exc
wait = min(2 ** attempt, 30)
log.warning("vyos op failed (attempt %d/%d): %s — retry in %ss",
attempt, self.retries, exc, wait)
time.sleep(wait)
raise VyosApiError(f"vyos op failed after {self.retries} attempts: {last}")

View File

@@ -0,0 +1,2 @@
requests>=2.31
PyYAML>=6.0

68
docs/ARCHITECTURE.md Normal file
View File

@@ -0,0 +1,68 @@
# Controller architecture
## Alert ingestion
Suricata is the client of an AF_UNIX stream socket bound by `AlertSocket` in
`alertsock.py`. The reader frames newline-delimited JSON, accepts alert records,
and places them on a bounded queue. Oversized or malformed records and queue
overflow are counted. A socket-path lock prevents competing controllers from
replacing the listener, even when they use different state databases.
The queue wakes the main loop independently of file polling. The reader does
not run policy or wait for router commits. During graceful shutdown, intake is
stopped and accepted queued alerts are processed before SQLite closes.
The normal EVE file contains stats. `EveTailer` in `eve.py` follows complete
records, preserves partial writes and drains the old inode during rotation.
When socket mode is enabled, file alerts are ignored to avoid counting the same
alert through two outputs.
For file-alert mode, set `alert_socket.path: ""` and configure Suricata to write
alerts into the EVE file. Checkpoints advance per delivered record. File-hit
identities include record content, inode, offset, SID and candidate address to
prevent replay from inflating threshold counts. Processing is at-least-once;
a restart cannot automatically recover unread records in an older rotated file.
## Policy and state
`policy.py` reads `leaf_action` metadata, applies SID restrictions, rejects
protected addresses, and evaluates threshold or immediate mode. `state.py`
stores hit windows, desired blocks, expiry times, applied flags, file checkpoints
and pending router-save status in SQLite WAL mode.
The desired block row is persisted before API submission. Qualifying alerts
refresh expiry. The active-address cap bounds the number of desired blocks.
Source and destination actions share a router group; `kind` records the initial
reason rather than choosing a separate group.
## Router reconciliation
A separate worker in `main.py` serializes router I/O using its own SQLite
connection. Each pass reads actual membership, computes changes, commits a batch,
and reads membership back before marking decisions applied or removals complete.
A timeout may still mean a commit succeeded. The following pass reads state
before retrying. Expiry deletion uses a timestamp condition so that a fresh
alert received during a slow request does not lose its refreshed decision.
Failures retain desired state for retry.
`vyos_client.py` handles multipart API requests and response validation. TLS can
use normal CA verification or a certificate fingerprint checked before sending
credentials. Pinned pool classes are private to each client. Environment proxies
are disabled for the API session.
## Metrics and persistence
Stats records update `metrics.py` counters without entering policy or changing
router state. Reports include capture/decoder deltas and ingress counters.
`decisions.jsonl` records decisions and verified enforcement changes; `capture.log`
contains periodic metric windows. Both logs retain the newest size-bounded archives.
A zero controller queue-drop count covers that queue only. Suricata can lose
alerts before delivery under send-buffer pressure or disconnection. The socket
queue is volatile on crash. Durable decisions can be reconciled after restart,
but alerts that never reached SQLite cannot be reconstructed from the stats file.
Dry-run creates temporary state and logs, disables socket ownership, avoids the
API, and reads the configured file from its beginning. Use an alert fixture for
policy replay; the normal stats-only file contains no alert history.

144
docs/INSTALLATION.md Normal file
View File

@@ -0,0 +1,144 @@
# Installation and operations
## Prerequisites
Use a Linux sensor with systemd, Python 3, `requests`, `PyYAML`, Suricata 7,
OpenSSH and tcpdump. Live tests also require Python 3 and ping on the test hosts.
The supplied service uses `/usr/bin/python3`; install its dependencies for that
interpreter. On Debian-based sensors, use the `python3-requests` and
`python3-yaml` packages. A separate virtual environment can be used for local
tests with `controller/requirements.txt`.
The supplied lab configuration uses sensor capture interface `eth1`, P2P
address `10.0.249.1/31`, VyOS API `https://10.0.249.0:8443`, and address group
`IDS-BLOCK`. Test hosts are `172.19.32.3` and `172.19.32.5`. These are example
site settings; inspect and adapt them for another network.
Provision mirroring, API access and firewall hooks before starting the sensor.
Both source and destination drop hooks must be reachable before accept rules
that would bypass enforcement. The controller verifies hooks but does not
provision routing or mirroring. The file in `vyos/` is an illustrative template,
not a complete router configuration.
## Configuration and credentials
The installer preserves an existing `/etc/leaf-controller/config.yaml`. For a
new installation, it seeds that file from `config/prototype-vyos.yaml`.
`controller/config.yaml` is a generic example with different API/TLS settings;
it is not the installer's lab default.
Review these settings before installation:
- API URL, timeout, retry count and TLS certificate fingerprint or CA bundle.
- Protected networks, allowed SIDs, source actions, threshold, TTL and block cap.
- Capture interface in the Suricata configuration and service unit.
- Socket, EVE, SQLite and audit paths.
The lab API uses a certificate fingerprint because its self-signed certificate
has no matching IP subject alternative name. Verify the fingerprint through a
trusted channel before setting `vyos.tls_cert_pin`. For a CA-issued certificate,
configure the appropriate CA bundle and hostname. Keep TLS verification enabled.
Create `/etc/leaf-controller/env`, owned by root with mode `0600`, containing:
```text
LEAF_VYOS_KEY='replace-with-existing-api-key'
```
Use a root editor or secret-management mechanism. Do not commit the real value
or place it in command arguments. The credential file is read by systemd.
The router address group must be dedicated to this controller. With
`adopt_foreign_members: false`, foreign members are removed during reconciliation.
The optional adoption setting imports eligible foreign members only at startup.
## Log storage and installation
The installer requires `/var/log/suricata-leaf` to be on tmpfs. On a new sensor,
review and run the provisioning script before installing:
```bash
sudo env TMPFS_SIZE=8G bash scripts/provision-ids-tmpfs.sh
sudo bash scripts/install-sensor.sh
```
Allow enough memory and swap headroom for the configured tmpfs cap and Suricata.
The provisioning script writes fstab and runs `mount -a`; review existing mounts
first. Mounting over a nonempty directory hides its previous contents. EVE files
on tmpfs are lost at reboot. SQLite and controller audit remain on persistent disk.
The installer stops the sensor/controller for upgrade, backs up existing files,
units and SQLite under `/var/backups/leaf-proto/<timestamp>/`, installs the code
and helpers, and enables these units:
- `leaf-controller.service`
- `suricata-leaf.service`
- `leaf-eve-rotate.timer`
It restores the previous deployment on installation failure. Backups can contain
credentials and must stay private. The controller runs as `leafctrl`; Suricata
runs as root with group `leafctrl`. The controller reports readiness after its
socket is bound, allowing the sensor to start afterward.
## Status and diagnostics
```bash
sudo systemctl status leaf-controller suricata-leaf leaf-eve-rotate.timer
sudo journalctl -u leaf-controller -f
sudo python3 /opt/leaf-controller/tools/vyos_smoke.py
sudo python3 /opt/leaf-controller/tools/vyos_ctl.py members
```
The smoke check reads API information, members and enforcement hooks.
`vyos_show.py` reads scoped configuration and redacts sensitive fields.
These diagnostic scripts do not write router configuration.
| Resource | Installed path |
|---|---|
| Controller | `/opt/leaf-controller/controller/leaf_controller/` |
| Configuration | `/etc/leaf-controller/config.yaml` |
| Credential | `/etc/leaf-controller/env` |
| SQLite WAL state | `/var/lib/leaf-controller/state.sqlite` |
| Decision audit | `/var/log/leaf-controller/decisions.jsonl` |
| Capture metrics | `/var/log/leaf-controller/capture.log` |
| EVE stats | `/var/log/suricata-leaf/eve.json` |
| Alert socket | `/run/suricata/alerts.sock` |
The retention timer checks each minute after an initial two-minute delay. It
renames EVE hourly or at 64 MiB, asks Suricata to reopen it, and retains six
archives. A startup grace period prevents signalling a newly started engine.
Retention can overshoot between timer runs; tmpfs capacity is the hard limit.
Audit defaults to 16 MiB with three archives; capture metrics default to 10 MiB
with six archives.
`save_config: false` leaves transient membership out of router boot configuration.
SQLite supplies desired state when the router returns. If saving is enabled,
failed saves remain dirty in SQLite and are retried.
## Credential rotation
Identify the controller's existing key ID before rotating:
```bash
sudo python3 /opt/leaf-controller/tools/rotate-vyos-key.py --old-id <controller-key-id>
```
The tool requires the selected ID to match the current credential. It creates
and verifies a candidate, saves both keys, switches the local environment file,
restarts the service, then retires only the selected old ID and saves again.
Other API users' IDs are preserved.
Switch failures restore the local credential. A failure after retirement requires
inspection before retrying; deleting arbitrary IDs is not a recovery procedure.
No credential values are printed by the tool.
## Rollback
Stop Suricata, the controller and the rotation timer. Restore the selected file
archive and its matching SQLite backup, reload systemd, and restart the previous
units. Keep SQLite closed while replacing its files. When reverting a schema
migration, restore the matching database as well as the code.
Do not clear the live database or shared router configuration as an upgrade or
test shortcut. Live tests operate on existing routes and enforcement hooks;
uplink, OSPF and static-route changes are outside their scope.

11
docs/REFERENCES.md Normal file
View File

@@ -0,0 +1,11 @@
# References
The lab uses Suricata 7 and VyOS 2026.03. Configuration and API schemas differ
between releases; use documentation matching the installed software.
- [VyOS HTTP API](https://docs.vyos.io/en/rolling/automation/vyos-api.html)
- [VyOS HTTPS service and API configuration](https://docs.vyos.io/en/rolling/configuration/service/https.html)
- [VyOS Ethernet interfaces and mirroring](https://docs.vyos.io/en/rolling/configuration/interfaces/ethernet.html)
- [VyOS firewall groups](https://docs.vyos.io/en/rolling/configuration/firewall/groups.html)
- [Suricata EVE output](https://docs.suricata.io/en/suricata-7.0.10/output/eve/eve-json-output.html)
- [Suricata EVE JSON format](https://docs.suricata.io/en/suricata-7.0.10/output/eve/eve-json-format.html)

87
docs/TESTING.md Normal file
View File

@@ -0,0 +1,87 @@
# Testing
## Unit and mock integration tests
Run from the repository root:
```bash
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -p 'test_*.py' -v
bash tests/run_e2e.sh
LEAF_E2E_SOCKET=/tmp/leaf-test.sock bash tests/run_e2e.sh
python3 tests/checkpoint_e2e.py
```
The unit suite covers socket ownership/framing, protected-address policy,
thresholds, persistence, replay identities, reconciliation races, TLS pin
isolation, retention, malformed telemetry and credential-switch failure paths.
It uses local fixtures rather than the lab network.
The shell integration starts a localhost mock VyOS API and a controller. The
first invocation uses file alerts; the second uses the Unix socket. They check
thresholding, protected addresses, source blocking, rotation, adoption and TTL
cleanup. They use port 18443 and disposable `tests/.e2e/` files: run them
sequentially. Keep the socket path short enough for AF_UNIX pathname limits.
The checkpoint check verifies that file mode resumes at the next unread record.
It does not assert socket-alert backlog during controller downtime.
## Offline Suricata replay
On a sensor with Suricata installed and its rules in `/etc/suricata-leaf/`:
```bash
sudo python3 tests/replay_fixture.py
```
The script generates a valid 24-SYN PCAP, replays it with isolated outputs and
requires 24 decoded SYNs, two SID 1000030 alerts and zero fragment/checksum errors.
It leaves the live capture configuration and services in place.
`tests/syn_packets.py` is the maintained TEST-NET SYN generator.
`tests/drill_synsensor.py` can write ICMP/SYN PCAP fixtures or send bounded raw
frames on a selected route. Capture visibility depends on the mirror topology;
use the test hosts for the live enforcement checks.
## Live enforcement tests
Verify the sensor identity, SSH account/key, test hosts, routes and dedicated
block group before running. Defaults describe the supplied lab. Override the
connection arguments for another environment:
```bash
python3 tests/live_e2e.py --help
python3 tests/live_negative.py --help
python3 tests/live_e2e.py --output evidence/live-e2e.json
python3 tests/live_negative.py --output evidence/negative.json
```
These tests use SSH to the sensor and LXCs, deploy temporary helpers, transmit
bounded TEST-NET traffic and temporarily select a 180-second TTL. They require
an empty block group and state for counter attribution. Run them sequentially.
The original configuration is restored on exit; failure cleanup expires only
the test's own targets. An interrupted connection can require checking the
retained configuration backup before another run.
The positive test requires fresh capture, audited decisions, durable applied
rows, verified router membership, source/destination drop-counter growth,
natural TTL removal and clean post-expiry probes. It compares hashes of
protected router configuration before and after.
The negative test sends benign traffic from another source while a block is
active. It checks ingress observation, absence of the benign address from
block membership/state, and selective IDS counter changes. Ingress capture
is not a destination receipt or application-connectivity test.
Live results describe the tested rules, topology and traffic. They do not
establish sustained throughput, production detection quality, lossless alert
delivery, or successful full router/sensor reboot recovery.
## Recovery inspection
```bash
VM=<sensor-user>@<sensor-address> LEAF_SSH_KEY=<key-path> bash scripts/reboot-check.sh
```
Despite its filename, this script performs read-only recovery inspection. It
checks service activity/enabling, tmpfs, socket presence and API/state status,
and prints the current boot identity. It does not initiate or prove a reboot.

87
scripts/install-sensor.sh Normal file
View File

@@ -0,0 +1,87 @@
#!/bin/bash
set -euo pipefail
[ "$(id -u)" -eq 0 ] || { echo 'Run as root on the sensor' >&2; exit 1; }
STAGE=$(cd "$(dirname "$0")/.." && pwd)
python3 -c 'import yaml, requests'
[ -s /etc/leaf-controller/env ] || { echo 'Create root-only /etc/leaf-controller/env with LEAF_VYOS_KEY first' >&2; exit 1; }
[ "$(findmnt -n -o FSTYPE --target /var/log/suricata-leaf)" = tmpfs ] || { echo 'Provision the sensor log tmpfs first' >&2; exit 1; }
if systemctl is-active --quiet suricata.service; then
echo 'The stock Suricata service is active; stop it before installing a second sensor.' >&2
exit 1
fi
systemctl stop leaf-eve-rotate.timer leaf-eve-rotate.service 2>/dev/null || true
systemctl stop suricata-leaf.service leaf-controller.service 2>/dev/null || true
BACKUP=/var/backups/leaf-proto/$(date +%Y%m%d-%H%M%S)
install -d -m 700 "$BACKUP"
items=()
for item in etc/suricata-leaf etc/leaf-controller opt/leaf-controller home/yuru/leaf-proto etc/systemd/system/leaf-controller.service etc/systemd/system/suricata-leaf.service etc/systemd/system/leaf-eve-rotate.service etc/systemd/system/leaf-eve-rotate.timer etc/tmpfiles.d/leaf-suricata.conf; do
if [ -e "/$item" ]; then items+=("$item"); fi
done
tar -czf "$BACKUP/sensor-files.tar.gz" -C / "${items[@]}"
python3 - "$BACKUP" <<'PY'
import sqlite3,sys,pathlib
p=pathlib.Path('/var/lib/leaf-controller/state.sqlite')
if p.exists():
with sqlite3.connect(p) as source, sqlite3.connect(sys.argv[1]+'/state.sqlite') as target: source.backup(target)
PY
rollback() {
rc=$?
trap - ERR
echo "Installation failed; restoring previous files from $BACKUP" >&2
systemctl stop suricata-leaf leaf-controller leaf-eve-rotate.timer 2>/dev/null || true
for unit in leaf-controller.service suricata-leaf.service leaf-eve-rotate.service leaf-eve-rotate.timer; do
if ! tar -tzf "$BACKUP/sensor-files.tar.gz" | grep -qx "etc/systemd/system/$unit"; then
systemctl disable "$unit" 2>/dev/null || true
rm -f "/etc/systemd/system/$unit"
fi
done
tar -xzf "$BACKUP/sensor-files.tar.gz" -C /
if [ -f "$BACKUP/state.sqlite" ]; then
install -o leafctrl -g leafctrl -m 640 "$BACKUP/state.sqlite" /var/lib/leaf-controller/state.sqlite
rm -f /var/lib/leaf-controller/state.sqlite-wal /var/lib/leaf-controller/state.sqlite-shm
fi
systemctl daemon-reload
systemctl start leaf-controller suricata-leaf || true
exit "$rc"
}
trap rollback ERR
getent group leafctrl >/dev/null || groupadd --system leafctrl
id leafctrl >/dev/null 2>&1 || useradd --system --gid leafctrl --home-dir /nonexistent --shell /usr/sbin/nologin leafctrl
install -d -o leafctrl -g leafctrl -m 750 /var/lib/leaf-controller /var/log/leaf-controller
chown -R leafctrl:leafctrl /var/lib/leaf-controller /var/log/leaf-controller
find /var/lib/leaf-controller /var/log/leaf-controller -type f -exec chmod 640 {} +
install -d -m 755 /opt/leaf-controller/controller
cp -a "$STAGE/controller/leaf_controller" /opt/leaf-controller/controller/
chown -R root:root /opt/leaf-controller
chmod -R a+rX /opt/leaf-controller
install -d -o root -g leafctrl -m 750 /etc/leaf-controller
# Preserve site policy and TLS settings on upgrades; seed only a new install.
if [ ! -s /etc/leaf-controller/config.yaml ]; then
install -m 644 "$STAGE/config/prototype-vyos.yaml" /etc/leaf-controller/config.yaml
fi
install -d -m 755 /opt/leaf-controller/tools
install -m 755 "$STAGE/tests/live_helper.py" /opt/leaf-controller/tools/leaf-live-helper.py
install -m 755 "$STAGE/scripts/rotate-eve.py" "$STAGE/scripts/rotate-vyos-key.py" "$STAGE/scripts/vyos_show.py" "$STAGE/scripts/vyos_smoke.py" "$STAGE/scripts/vyos_ctl.py" "$STAGE/scripts/probe-vyos-read.py" /opt/leaf-controller/tools/
install -m 644 "$STAGE/suricata/leaf-eve-rotate.service" "$STAGE/suricata/leaf-eve-rotate.timer" /etc/systemd/system/
install -m 644 "$STAGE/controller/leaf-controller.service" /etc/systemd/system/leaf-controller.service
install -d -m 755 /etc/suricata-leaf/rules
install -m 644 "$STAGE/suricata/suricata-leaf.yaml" /etc/suricata-leaf/suricata-leaf.yaml
install -m 644 "$STAGE/suricata/rules/leaf-local.rules" /etc/suricata-leaf/rules/leaf-local.rules
install -m 644 "$STAGE/suricata/suricata-leaf.service" /etc/systemd/system/suricata-leaf.service
# tmpfs mount stays in place; recreate permissions after reboot via tmpfiles.
printf '%s\n' 'd /var/log/suricata-leaf 0750 root leafctrl -' \
'd /run/suricata 0770 root leafctrl -' > /etc/tmpfiles.d/leaf-suricata.conf
systemd-tmpfiles --create /etc/tmpfiles.d/leaf-suricata.conf
chown root:leafctrl /var/log/suricata-leaf
chmod 750 /var/log/suricata-leaf
find /var/log/suricata-leaf -maxdepth 1 -type f -exec chgrp leafctrl {} +
find /var/log/suricata-leaf -maxdepth 1 -type f -exec chmod 640 {} +
systemctl daemon-reload
systemctl enable suricata-leaf.service leaf-controller.service leaf-eve-rotate.timer
systemctl start suricata-leaf.service leaf-controller.service leaf-eve-rotate.timer
sleep 2
systemctl is-active --quiet suricata-leaf.service
systemctl is-active --quiet leaf-controller.service
trap - ERR
printf 'Backup: %s\n' "$BACKUP"
systemctl --no-pager --full status suricata-leaf.service leaf-controller.service

View File

@@ -0,0 +1,5 @@
#!/usr/bin/env python3
"""Compatibility entry point for the scoped, redacted read-only inspector."""
from pathlib import Path
import runpy
if __name__=='__main__':runpy.run_path(str(Path(__file__).with_name('vyos_show.py')),run_name='__main__')

76
scripts/provision-ids-tmpfs.sh Executable file
View File

@@ -0,0 +1,76 @@
#!/usr/bin/env bash
# LEAF-PROTO sensor provisioning: RAM disk (tmpfs) for the suricata eve dir.
#
# eve.json under /var/log/suricata-leaf is rewritten constantly; on a
# journal/flash disk that burns write cycles and adds latency. This puts the
# directory on a size-capped tmpfs (RAM+swap backed), persisted via fstab.
# Controller state stays on persistent storage in /var/lib/leaf-controller.
# See docs/INSTALLATION.md for memory sizing and log retention.
#
# Usage: TMPFS_SIZE=6G sudo scripts/provision-ids-tmpfs.sh [mount_point]
# (default mount point /var/log/suricata-leaf, default size 8G)
set -euo pipefail
MOUNT_POINT="${1:-${MOUNT_POINT:-/var/log/suricata-leaf}}"
SIZE="${TMPFS_SIZE:-${SIZE:-8G}}"
usage() {
echo "usage: TMPFS_SIZE=8G sudo $0 [mount_point (default: /var/log/suricata-leaf)]"
}
if [[ "${EUID:-$(id -u)}" -ne 0 ]]; then
echo "error: run as root (needs mount/fstab writes)" >&2
usage
exit 1
fi
if findmnt -rn -T "$MOUNT_POINT" | grep -q 'tmpfs'; then
echo "skip: $MOUNT_POINT is already on tmpfs:"
findmnt -rn -T "$MOUNT_POINT"
exit 0
fi
if ! [[ "$SIZE" =~ ^[0-9]+([MGK])?$ ]]; then
echo "error: TMPFS_SIZE must like 6G/8G/512M (got: $SIZE)" >&2
exit 1
fi
echo "warning: tmpfs is RAM-backed — $SIZE cap counts against RAM+swap."
free -h
echo "note: eve.json on this mount is lost on reboot by design;"
echo "note: controller block state lives on disk (/var/lib/leaf-controller), unaffected."
mkdir -p "$MOUNT_POINT"
# A pre-existing non-empty dir on another filesystem would be hidden under
# the tmpfs after mounting — warn, since old eve files become unreachable.
if [[ -n "$(findmnt -rn -T "$MOUNT_POINT" 2>/dev/null)" ]] &&
! findmnt -rn -T "$MOUNT_POINT" | grep -q 'tmpfs' &&
[[ -n "$(ls -A "$MOUNT_POINT" 2>/dev/null)" ]]; then
echo "warning: $MOUNT_POINT already has content on another filesystem;"
echo "warning: it will be hidden underneath the tmpfs — move it first if"
echo "warning: you need the old eve.json:"
ls -A "$MOUNT_POINT" | head -5
fi
FSTAB_LINE="tmpfs ${MOUNT_POINT} tmpfs defaults,size=${SIZE},nosuid,noexec,nodev 0 0"
if grep -qF " ${MOUNT_POINT} " /etc/fstab; then
echo "fstab already has an entry for $MOUNT_POINT — not appending:"
grep -F " ${MOUNT_POINT} " /etc/fstab
else
printf '\n# LEAF-PROTO: suricata eve on RAM disk (provision-ids-tmpfs.sh)\n%s\n' "$FSTAB_LINE" >> /etc/fstab
echo "fstab entry appended."
fi
systemctl daemon-reload
mount -a
if findmnt -rn -T "$MOUNT_POINT" | grep -q 'tmpfs'; then
echo "OK: mounted."
findmnt -rn -T "$MOUNT_POINT"
else
echo "error: mount did not come up — check /etc/fstab" >&2
exit 1
fi
df -h "$MOUNT_POINT"

17
scripts/reboot-check.sh Normal file
View File

@@ -0,0 +1,17 @@
#!/usr/bin/env bash
# Read-only recovery inspection. Never reboots either the sensor or VIPERA.
set -euo pipefail
VM="${VM:-yuru@192.168.100.84}"
KEY="${LEAF_SSH_KEY:-$HOME/.ssh/id_ed25519_alice}"
ssh -F /dev/null -o BatchMode=yes -o ConnectTimeout=8 -i "$KEY" "$VM" '
set -eu
[ "$(hostname)" = LEAF-PROTO ]
systemctl is-active --quiet leaf-controller suricata-leaf leaf-eve-rotate.timer
systemctl is-enabled --quiet leaf-controller suricata-leaf leaf-eve-rotate.timer
[ "$(findmnt -n -o FSTYPE --target /var/log/suricata-leaf)" = tmpfs ]
sudo -n test -S /run/suricata/alerts.sock
printf "boot_id="; cat /proc/sys/kernel/random/boot_id
uptime -s
sudo -n python3 /opt/leaf-controller/tools/leaf-live-helper.py status
printf "RECOVERY INSPECTION PASS (no reboot performed)\n"
'

60
scripts/rotate-eve.py Normal file
View File

@@ -0,0 +1,60 @@
#!/usr/bin/env python3
"""Rename EVE and ask Suricata to reopen it; invoked each minute by a timer.
Normal bound: seven 64-MiB/hour segments plus bytes written between timer
runs. The tmpfs itself is the hard capacity limit. Not copytruncate.
"""
import argparse
import fcntl
import os
from pathlib import Path
import subprocess
import time
def rotate(path, max_bytes=64*1024*1024, max_age=3600, keep=6, reopen=None, now=None):
now = time.time() if now is None else now
try:
st = path.stat()
except FileNotFoundError:
return False
# Both mtime and ctime change on writes. Keep an independent age clock.
clock = path.with_name('.' + path.name + '.rotation-clock')
try:
last = float(clock.read_text())
except (FileNotFoundError, ValueError):
last = now
clock.write_text(str(now))
if st.st_size < max_bytes and now-last < max_age:
return False
dst = path.with_name(path.name + '.leaf-' + str(time.time_ns()).zfill(20))
path.rename(dst)
try:
reopen()
except BaseException:
if not path.exists():
dst.rename(path)
raise
clock.write_text(str(now))
for f in sorted(path.parent.glob(path.name + '.leaf-*'))[:-keep] if keep else path.parent.glob(path.name + '.leaf-*'):
f.unlink()
return True
def main():
ap=argparse.ArgumentParser(description=__doc__)
ap.add_argument('--path', type=Path, default=Path('/var/log/suricata-leaf/eve.json'))
a=ap.parse_args()
# Type=simple is active before Suricata installs signal handlers. Never
# signal a just-started engine, including manual timer/service triggers.
started = subprocess.run(['systemctl','show','--property=ActiveEnterTimestampMonotonic',
'--value','suricata-leaf.service'],
check=True,capture_output=True,text=True).stdout.strip()
if not started or int(started) == 0 or time.monotonic() - int(started)/1e6 < 60:
return
with open('/run/leaf-eve-rotation.lock', 'a') as lock:
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
rotate(a.path, reopen=lambda: subprocess.run(
['systemctl','kill','--kill-whom=main','--signal=HUP','suricata-leaf.service'], check=True))
if __name__ == '__main__': main()

120
scripts/rotate-vyos-key.py Normal file
View File

@@ -0,0 +1,120 @@
#!/usr/bin/env python3
"""Rotate exactly one explicitly selected LEAF key. Never prints API bodies/keys.
Run on sensor: sudo python3 .../rotate-vyos-key.py --old-id <LEAF-owned-id>
Other applications' credentials are preserved. Retrying after retirement/save
failure requires inspecting router key IDs first; no blind mass-delete mode.
"""
import argparse
import os
from pathlib import Path
import re
import secrets
import string
import subprocess
import sys
import time
sys.path.insert(0, '/opt/leaf-controller/controller')
from leaf_controller.config import load_config, read_key_env
from leaf_controller.vyos_client import VyosClient, VyosApiError
ENV_PATH = '/etc/leaf-controller/env'
CFG_PATH = '/etc/leaf-controller/config.yaml'
SERVICE = 'leaf-controller.service'
KEYS_PATH = ['service', 'https', 'api', 'keys', 'id']
def atomic_env(text):
tmp=ENV_PATH+'.new'
fd=os.open(tmp, os.O_WRONLY|os.O_CREAT|os.O_TRUNC, 0o600)
os.fchmod(fd, 0o600)
with os.fdopen(fd, 'w') as f:
f.write(text); f.flush(); os.fsync(f.fileno())
os.replace(tmp, ENV_PATH)
fd=os.open(str(Path(ENV_PATH).parent), os.O_DIRECTORY)
try: os.fsync(fd)
finally: os.close(fd)
def restart_service():
subprocess.run(['systemctl','restart',SERVICE], check=True, capture_output=True)
subprocess.run(['systemctl','is-active','--quiet',SERVICE], check=True, capture_output=True)
def wait_auth(client, group, accepted=True, attempts=24):
for i in range(attempts):
try:
client.group_members(group)
if accepted: return
except VyosApiError as exc:
if not accepted and exc.status in (401,403): return
except Exception:
pass
if i+1<attempts: time.sleep(5)
raise RuntimeError('credential verification timed out')
def rotate(old, new, old_id, new_id, old_key, new_key, env_text, group):
body=old.retrieve('showConfig', KEYS_PATH[:-1])
ids=(body.get('data') or {}).get('id', {})
if old_id not in ids or ids[old_id].get('key') != old_key:
raise RuntimeError('selected key ID does not match the current controller credential')
if new_id in ids:
raise RuntimeError('candidate ID already exists')
swapped=False
installed=False
try:
# A timeout can still commit; rollback attempts candidate deletion either way.
installed=True
old.configure({'op':'set','path':KEYS_PATH+[new_id,'key',new_key]})
wait_auth(new, group)
# Persist both keys before the local switch; old credential remains a recovery path.
new.save_config()
out,n=re.subn(r'(?m)^LEAF_VYOS_KEY=.*$', "LEAF_VYOS_KEY='"+new_key+"'", env_text)
if n!=1: raise RuntimeError('expected exactly one credential assignment')
swapped=True
atomic_env(out)
restart_service()
wait_auth(new, group, attempts=2)
except BaseException:
# Restore the credential before restarting. If recovery fails, retain candidate.
if swapped:
atomic_env(env_text)
restart_service()
if installed:
old.configure({'op':'delete','path':KEYS_PATH+[new_id]})
old.save_config()
raise
# Retirement is a separate phase: rollback is unsafe after this commit.
new.configure({'op':'delete','path':KEYS_PATH+[old_id]})
new.save_config() # Failure is fatal; never claim persistence succeeded.
wait_auth(old, group, accepted=False)
def main():
ap=argparse.ArgumentParser(description=__doc__)
ap.add_argument('--old-id', required=True)
a=ap.parse_args()
if os.geteuid()!=0: ap.error('run as root on the sensor')
cfg=load_config(CFG_PATH)
if cfg.get('vyos.key_env','LEAF_VYOS_KEY')!='LEAF_VYOS_KEY':
ap.error('this tool requires the standard LEAF_VYOS_KEY assignment')
text=Path(ENV_PATH).read_text()
old_key=read_key_env(ENV_PATH)
new_key=''.join(secrets.choice(string.ascii_letters+string.digits) for _ in range(32))
new_id='leaf-'+str(time.time_ns())
def client(key):
return VyosClient(cfg.vyos_url(), key, cert_pin=cfg.get('vyos.tls_cert_pin') or None,
ca_bundle=cfg.get('vyos.ca_bundle'), tls_verify=cfg.get('vyos.tls_verify',True),
timeout=float(cfg.get('vyos.timeout',75)), retries=1)
try:
rotate(client(old_key),client(new_key),a.old_id,new_id,old_key,new_key,text,cfg.get('block_group'))
except BaseException:
# Deliberately omit exception details: router responses can contain secrets.
print('ROTATION FAILED. Inspect service and router key IDs before retrying. Candidate ID: '+new_id,
file=sys.stderr)
return 1
print('ROTATION COMPLETE: selected old ID retired, config persisted, new ID='+new_id)
return 0
if __name__=='__main__': raise SystemExit(main())

19
scripts/vyos_ctl.py Normal file
View File

@@ -0,0 +1,19 @@
#!/usr/bin/env python3
"""Read-only lab client. Membership writes belong to the durable controller.
Legacy add/del/drop-chain commands have been retired; they bypassed ownership.
"""
import argparse
import json
from pathlib import Path
import sys
sys.path[:0]=[str(Path(__file__).resolve().parents[1]/'controller'),'/opt/leaf-controller/controller']
from leaf_controller.ops import configured_client,redact
def main():
p=argparse.ArgumentParser(description=__doc__)
p.add_argument('command',choices=['members','forward-rule'],default='members',nargs='?')
a=p.parse_args();cfg,c=configured_client()
if a.command=='members':print(json.dumps(c.group_members(cfg.get('block_group'))))
else:print(json.dumps(redact(c.retrieve('showConfig',['firewall','ipv4','forward']))))
if __name__=='__main__':main()

13
scripts/vyos_show.py Normal file
View File

@@ -0,0 +1,13 @@
#!/usr/bin/env python3
"""Read-only firewall, mirror and protected-path inspection. No credential dump."""
import json
from pathlib import Path
import sys
sys.path[:0]=[str(Path(__file__).resolve().parents[1]/'controller'),'/opt/leaf-controller/controller']
from leaf_controller.ops import configured_client, redact
def main():
_,c=configured_client()
for path in (['firewall'],['interfaces'],['protocols','ospf'],['protocols','static']):
print(json.dumps({'path':path,'data':redact(c.retrieve('showConfig',path).get('data'))}))
if __name__=='__main__':main()

15
scripts/vyos_smoke.py Normal file
View File

@@ -0,0 +1,15 @@
#!/usr/bin/env python3
"""Read-only pinned API smoke check. Does not add members or save router config."""
import json
from pathlib import Path
import sys
sys.path[:0]=[str(Path(__file__).resolve().parents[1]/'controller'),'/opt/leaf-controller/controller']
from leaf_controller.ops import configured_client
def main():
cfg,c=configured_client()
info=c.info();members=c.group_members(cfg.get('block_group'))
c.verify_enforcement(cfg.get('block_group'),cfg.get('policy.block_src',False))
print(json.dumps({'version':info.get('version'),'members':members,'enforcement_hooks':True}))
print('READ-ONLY SMOKE PASS')
if __name__=='__main__':main()

View File

@@ -0,0 +1,8 @@
[Unit]
Description=Rotate LEAF EVE stats on tmpfs
Requisite=suricata-leaf.service
After=suricata-leaf.service
[Service]
Type=oneshot
ExecStart=/usr/bin/python3 /opt/leaf-controller/tools/rotate-eve.py
UMask=0027

View File

@@ -0,0 +1,8 @@
[Unit]
Description=Check LEAF EVE retention every minute
[Timer]
OnActiveSec=2min
OnUnitActiveSec=1min
AccuracySec=5s
[Install]
WantedBy=timers.target

View File

@@ -0,0 +1,34 @@
# LEAF-PROTO local rules. Only rules with `metadata: leaf_action block;` are
# acted on by leaf-controller (override via config policy). SIDs reserved: 1000000-1099999.
#
# Inbound flows (attacker->victim): default policy blocks nothing (dest is your LAN).
# Outbound flows (victim->external C2 / exfil / scan-out): dest_ip gets blocked on VyOS.
# Suricata 7: HOME_NET/EXTERNAL_NET are defined in suricata-leaf.yaml
# (vars: address-groups). 'var' statements in rule files are not supported.
# --- lab test rule: any outbound ICMP to TEST-NET (trigger demo) ---
alert icmp any any -> 192.0.2.0/24 any (msg:"LEAF TEST ICMP to TEST-NET"; sid:1000001; rev:1; metadata: leaf_action block;)
# --- outbound scan: many TCP SYNs from one host to many hosts in 10s ---
alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:"LEAF SCAN-OUT tcp syn flood"; flags:S; threshold: type threshold, track by_src, count 25, seconds 10; sid:1000002; rev:1; metadata: leaf_action block;)
# NOTE: the threshold is engine-side noisy for lab use; controller also windows by (sid,dest)
# --- obvious C2 patterns (examples; extend with your own) ---
alert tcp any any -> any any (msg:"LEAF possible meterpreter reverse"; flow:to_server; content:"|0a 16 1a 05|"; depth:4; sid:1000010; rev:1; metadata: leaf_action block;)
# --- DNS tunneling heuristic: prefer et open ruleset + thresholding for this ---
# --- inbound (src-side block candidates; block_src must be enabled in controller) ---
# SYN+FIN is an abnormal flag combination, not an SSH authentication failure.
# Encrypted SSH login failures cannot be inferred from TCP flags.
alert tcp $EXTERNAL_NET any -> $HOME_NET 22 (msg:"LEAF abnormal SYN FIN inbound"; flow:to_server; flags:SF; threshold: type threshold, track by_src, count 20, seconds 30; sid:1000020; rev:1; metadata: leaf_action block_src;)
# --- lateral/brute observable on the VEX-mirror posture: SYN-only toward sensor ---
alert tcp 172.19.32.0/24 any -> 10.0.249.1 22 (msg:"LEAF LATERAL-BRUTE ssh sweep from VEX-LAN at sensor"; flags:S; threshold: type threshold, track by_src, count 15, seconds 30; sid:1000021; rev:1; metadata: leaf_action block_src;)
# SRC-MODE drill: mirror-visible transit wave with out-of-floor spoofed src.
# dst 192.0.2.x is transit-only dst reachable through the mirror (locally-dst
# traffic toward 10.0.249.1 never appears on the mirror, so 1000020/1000021 cannot
# fire there). Spoofed src is 203.0.113.0/24 (RFC5737) -> never a real peer.
alert tcp 203.0.113.0/24 any -> 192.0.2.0/24 22 (msg:"LEAF SRC-MODE spoofed-src at TEST-NET dst"; flags:S; threshold: type threshold, track by_src, count 12, seconds 30; sid:1000030; rev:1; metadata: leaf_action block_src;)

View File

@@ -0,0 +1,25 @@
[Unit]
Description=LEAF passive Suricata sensor on eth1
# Start after the controller has bound its alert socket and reported ready.
# Runtime reconnect occurs on a later alert write; alerts during downtime
# are not stored for replay.
After=network-online.target leaf-controller.service
Wants=network-online.target leaf-controller.service
RequiresMountsFor=/var/log/suricata-leaf
[Service]
Type=simple
ExecStartPre=/usr/bin/suricata -T -c /etc/suricata-leaf/suricata-leaf.yaml -l /var/log/suricata-leaf
ExecStart=/usr/bin/suricata --af-packet=eth1 -c /etc/suricata-leaf/suricata-leaf.yaml -l /var/log/suricata-leaf
Restart=on-failure
RestartSec=5
KillSignal=SIGINT
TimeoutStopSec=60
User=root
Group=leafctrl
UMask=0027
ProtectSystem=full
ProtectHome=yes
[Install]
WantedBy=multi-user.target

View File

@@ -0,0 +1,89 @@
%YAML 1.1
---
# LEAF-PROTO Suricata IDS config (IDS VM). Self-contained drop-in.
# Run: suricata -c /etc/suricata/suricata-leaf.yaml -i MIRROR_IFACE
# (Debian: apt install suricata suricata-update; mirror NIC must be promisc/untagged-capable)
vars:
main: /var/lib/suricata-leaf
address-groups:
HOME_NET: "[192.168.100.0/24,10.0.249.0/31,172.19.32.0/24]"
EXTERNAL_NET: "!$HOME_NET"
default-rule-path: /etc/suricata-leaf/rules
rule-files:
- leaf-local.rules
# --- log output location --------------------------------------------------
# eve.json is written below as an absolute path. Provision
# /var/log/suricata-leaf on a 6-8G tmpfs (RAM disk) to keep eve writes off
# the physical disk. leaf-eve-rotate.timer renames the file hourly or at
# 64 MiB, signals HUP to reopen it, and keeps six backups. The controller
# drains the old inode before opening the new one.
logging:
default-log-level: info
outputs:
- console:
enabled: no
- file:
enabled: yes
level: info
filename: suricata.log
# --- capture: mirror NIC -------------------------------------------------
af-packet:
- interface: eth1 # mirror NIC on LEAF-PROTO VM (10.0.249.1/31)
cluster-id: 98
cluster-type: cluster_flow
defrag: yes
use-mmap: yes
tpacket-v3: no
bpf-filter: "not ip host 255.255.255.255 and not udp port 5353"
# --- output split: alerts over the controller socket, stats on file ------
# The controller binds /run/suricata/alerts.sock (AF_UNIX listener) and
# suricata connects as the CLIENT; ordering is contract-critical: the
# leaf-controller unit binds at startup, the sensor unit Wants+After it.
# The controller reports readiness after binding the socket.
# Stats stay in eve.json on tmpfs. Alerts use the socket and can be lost
# during disconnection or if the sender cannot write without blocking.
outputs:
- eve-log:
enabled: yes
filetype: unix_stream
filename: /run/suricata/alerts.sock # the socket path, not a log file
types:
- alert:
metadata: yes
payload: no # keep EVE small; set packet-based extras later if needed
- eve-log:
enabled: yes
filetype: regular
filename: /var/log/suricata-leaf/eve.json
types:
- stats:
totals: yes
threads: no
# --- engine: fine for 1-2 Gbps mirror ------------------------------------
threading:
pfring:
enabled: no
pcap:
buffer-size: 16777216
stream:
memcap: 128mb
flow:
memcap: 64mb
stats:
enabled: yes
interval: 30
host-mode: auto
# reference: ET open ruleset optional via `suricata-update --sources` but do NOT
# tag whole ET for leaf_action; only our local rules carry metadata: leaf_action.

27
tests/checkpoint_e2e.py Normal file
View File

@@ -0,0 +1,27 @@
#!/usr/bin/env python3
"""Offline file-mode checkpoint proof; never stops the live sensor.
Socket mode has no alert backlog while disconnected. File mode checkpoint
behavior is verified here; live socket reconnect is a separate drill.
"""
import json
import tempfile
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'controller'))
from leaf_controller.eve import EveTailer
def main():
with tempfile.TemporaryDirectory() as tmp:
path=Path(tmp)/'eve.json'
path.write_text('')
t=EveTailer(str(path)); t._open(); ck=(*t._ino,t._pos); t.close()
path.write_text('{"event_type":"alert","n":1}\n{"event_type":"alert","n":2}\n')
t=EveTailer(str(path),checkpoint=ck); events=t.events()
assert next(events)['n']==1
ck=(*t._ino,t._pos); events.close()
t=EveTailer(str(path),checkpoint=ck); events=t.events()
assert next(events)['n']==2
events.close()
print('CHECKPOINT PASS: restart resumes at the next unread record')
if __name__=='__main__': main()

189
tests/drill_synsensor.py Normal file
View File

@@ -0,0 +1,189 @@
#!/usr/bin/env python3
"""Sensor-side synthetic injector for the live socket drills.
Uses AF_PACKET raw sockets and requires root. Frames are addressed to
the selected gateway MAC. Traffic is visible to the sensor only when its
path crosses the configured mirror; select the interface and route accordingly.
Bounded by design: source and destination must be TEST-NET
(192.0.2.0/24, 198.51.100.0/24, 203.0.113.0/24) unless --allow-real-src
is passed (then src stays the sensor's real address). count is capped.
--pcap writes a fixture without sending traffic.
Modes:
icmp echo-request -> sid 1000001 (leaf_action block, any src)
syn TCP SYN ->port -> sid 1000030 when src is 203.0.113.0/24
--no-arp: do not ping the gateway first; use the existing ARP entry only.
--src SRC: forge src (TEST-NET enforced); ARP is still only ever
for the gateway MAC, the forged src never touches ARP.
"""
import argparse
import ipaddress
import re
import socket
import struct
import subprocess
import time
from pathlib import Path
TESTNET = ('192.0.2.0/24', '198.51.100.0/24', '203.0.113.0/24')
_NETS = tuple(ipaddress.ip_network(n) for n in TESTNET)
def in_testnet(addr):
ip = ipaddress.ip_address(addr)
return any(ip in net for net in _NETS)
def checksum(data):
if len(data) % 2:
data += b'\x00'
total = sum(struct.unpack('!%dH' % (len(data) // 2), data))
while total >> 16:
total = (total & 0xffff) + (total >> 16)
return (~total) & 0xffff
def syn_packet(src, dst, dport, index, smac, dmac, flags=2):
a, b = socket.inet_aton(src), socket.inet_aton(dst)
tcp = struct.pack('!HHIIBBHHH', 40000 + index % 20000, dport,
7000 + index, 0, 0x50, flags, 1024, 0, 0)
pseudo = a + b + struct.pack('!BBH', 0, 6, len(tcp))
tcp = tcp[:16] + struct.pack('!H', checksum(pseudo + tcp)) + tcp[18:]
ip = struct.pack('!BBHHHBBH4s4s', 0x45, 0, 40, index % 65536, 0, 64, 6,
0, a, b)
ip = ip[:10] + struct.pack('!H', checksum(ip)) + ip[12:]
return dmac + smac + b'\x08\x00' + ip + tcp
def icmp_packet(src, dst, index, smac, dmac):
payload = b'LEAFDRILL' + bytes((index % 100,) * 8)
icmp = struct.pack('!BBHHH', 8, 0, 0, 0x4C42, index % 65536) + payload
ck = checksum(icmp)
icmp = icmp[:2] + struct.pack('!H', ck) + icmp[4:]
total = 20 + len(icmp)
a, b = socket.inet_aton(src), socket.inet_aton(dst)
ip = struct.pack('!BBHHHBBH4s4s', 0x45, 0, total, index % 65536, 0, 64,
1, 0, a, b)
ip = ip[:10] + struct.pack('!H', checksum(ip)) + ip[12:]
return dmac + smac + b'\x08\x00' + ip + icmp
def pcap_write(path, packets):
with path.open('wb') as f:
f.write(struct.pack('<IHHIIII', 0xa1b2c3d4, 2, 4, 0, 0, 65535, 1))
t = time.time()
for i, data in enumerate(packets):
stamp = t + i * 0.001
f.write(struct.pack('<IIII', int(stamp), int(stamp % 1 * 1e6),
len(data), len(data)) + data)
def resolve_gateway_mac(gateway, interface):
text = subprocess.run(['ip', 'neigh', 'show', gateway, 'dev', interface],
capture_output=True, text=True).stdout
match = re.search(r'lladdr ([0-9a-f:]{17})', text)
return bytes.fromhex(match.group(1).replace(':', '')) if match else None
def route_via(dst):
"""Resolve (gw, dev) for a destination via the kernel routing table."""
out = subprocess.run(['ip', 'route', 'get', dst],
capture_output=True, text=True).stdout
m = re.search(r'via (\S+) dev (\S+)', out)
return m.groups() if m else (None, None)
def mac(value):
return bytes.fromhex(value.replace(':', ''))
def main():
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument('mode', choices=['icmp', 'syn'])
ap.add_argument('--dst', required=True,
help='comma-separated TEST-NET destinations (one-shot '
'flows per dst in (a)); single dst for bursts in (b)')
ap.add_argument('--count', type=int, default=1, help='packets per dst')
ap.add_argument('--port', type=int, default=22)
ap.add_argument('--src', default='', help='spoofed TEST-NET source')
ap.add_argument('--allow-real-src', action='store_true',
help='use the interface address as src (no spoofing)')
ap.add_argument('--interface', default='eth0')
ap.add_argument('--gateway', default='', help='override next-hop (default: '
'kernel route for first --dst)')
ap.add_argument('--no-arp', action='store_true',
help='skip gateway ping probe, use cached ARP entry only')
ap.add_argument('--interval', type=float, default=0.25)
ap.add_argument('--pcap', type=Path,
help='write fixture pcap instead of sending (never on live)')
a = ap.parse_args()
nets = list(_NETS)
dsts = [d.strip() for d in a.dst.split(',') if d.strip()]
if not dsts or not all(in_testnet(d) for d in dsts):
ap.error('--dst must be TEST-NET address(es) only')
if a.src and not in_testnet(a.src):
ap.error('--src must be TEST-NET')
if a.src and a.allow_real_src:
ap.error('--src and --allow-real-src are mutually exclusive')
if not 1 <= a.count <= 500 or not 1 <= a.port <= 65535:
ap.error('count must be 1..500 and port 1..65535')
smac = dmac = b'\x00' * 6 # PCAP fixtures need no real Ethernet endpoint
def build(src, dst, i):
if a.mode == 'icmp':
return icmp_packet(src, dst, i, smac, dmac)
return syn_packet(src, dst, a.port, i, smac, dmac)
if a.pcap:
packets = []
for dst in dsts:
for i in range(a.count):
packets.append(build(a.src or '198.51.100.7', dst, i))
pcap_write(a.pcap, packets)
print('wrote %d packets to %s' % (len(packets), a.pcap))
return
gw, dev = route_via(dsts[0])
if a.gateway:
gw = a.gateway
iface = dev or a.interface
if not gw:
ap.error('no via-route for %s on this sensor' % dsts[0])
smac = mac(Path('/sys/class/net', iface, 'address').read_text().strip())
dmac = resolve_gateway_mac(gw, iface)
if dmac is None:
if a.no_arp:
ap.error('gateway MAC not cached and --no-arp set')
subprocess.run(['ping', '-c1', '-W1', gw], capture_output=True)
dmac = resolve_gateway_mac(gw, iface)
if dmac is None:
ap.error('gateway MAC unresolved for %s' % gw)
if not a.src and not a.allow_real_src:
ap.error('src defaults to real interface address: pass --src '
'or --allow-real-src')
if a.allow_real_src:
out = subprocess.run(['ip', '-4', 'addr', 'show', iface],
capture_output=True, text=True).stdout
m = re.search(r'inet (\d+\.\d+\.\d+\.\d+)/', out)
if not m:
ap.error('no inet address on interface')
real = m.group(1)
sent = 0
with socket.socket(socket.AF_PACKET, socket.SOCK_RAW, socket.htons(0x0800)) as s:
s.bind((iface, 0))
for dst in dsts:
src = a.src or real
for i in range(a.count):
s.send(build(src, dst, i))
sent += 1
time.sleep(a.interval)
print('sent %d %s packets -> %s' % (sent, a.mode, ','.join(dsts)))
if __name__ == '__main__':
main()

117
tests/live_e2e.py Normal file
View File

@@ -0,0 +1,117 @@
#!/usr/bin/env python3
"""Bounded lab proof: real packets, fresh alerts, router drop counters, TTL cleanup.
Run on workstation: python3 tests/live_e2e.py --output /path/to/evidence.json
Requires deployed LEAF services. Temporarily uses TTL=180; restores config on exit.
"""
import argparse,json,subprocess,time
from pathlib import Path
def main():
ap=argparse.ArgumentParser(description=__doc__)
ap.add_argument('--sensor',default='yuru@192.168.100.84') # lease drifts; verify by hostname LEAF-PROTO
ap.add_argument('--lxc',default='root@172.19.32.3')
ap.add_argument('--second-lxc',default='root@172.19.32.5')
ap.add_argument('--ssh-key',default=str(Path.home()/'.ssh/id_ed25519_alice'))
ap.add_argument('--output',type=Path,required=True)
a=ap.parse_args(); root=Path(__file__).resolve().parents[1]
ssh=['ssh','-F','/dev/null','-i',a.ssh_key,'-o','BatchMode=yes','-o','ConnectTimeout=8']
scp=['scp','-F','/dev/null','-i',a.ssh_key,'-o','BatchMode=yes']
def run(host,cmd):
return subprocess.run(ssh+[host,cmd],capture_output=True,text=True,check=True,timeout=150).stdout
def status(since=0.0):
return json.loads(run(a.sensor,f'sudo -n python3 /tmp/leaf-live-helper.py status --since {since}'))
def syn(n):
print(run(a.lxc,f'python3 /tmp/leaf-send-syn.py 203.0.113.12 192.0.2.201 22 {n}').strip(),flush=True)
def ping(n):
# Different source hosts guarantee distinct Suricata ICMP flows.
for i in range(n):
host=a.lxc if i%2==0 else a.second_lxc
run(host,'ping -c 1 -W 1 192.0.2.202 >/dev/null 2>&1 || test $? -eq 1')
evidence={'started':time.time(),'passed':False}
changed=False
try:
subprocess.run(scp+[str(root/'tests/live_helper.py'),a.sensor+':/tmp/leaf-live-helper.py'],check=True)
subprocess.run(scp+[str(root/'tests/syn_packets.py'),a.lxc+':/tmp/leaf-send-syn.py'],check=True)
baseline=status(); evidence['baseline']=baseline
targets={'203.0.113.12','192.0.2.202'}
if baseline['members'] or baseline['blocks']: raise RuntimeError('live proof requires an idle empty block group/state for counter attribution')
run(a.sensor,'sudo -n python3 /tmp/leaf-live-helper.py short-ttl')
changed=True
run(a.sensor,'sudo -n systemctl restart leaf-controller')
t0=time.time();evidence['traffic_started']=t0
capture=subprocess.Popen(ssh+[a.sensor,f"sudo -n timeout 15 tcpdump -ni eth1 -c 24 -w /tmp/leaf-live-wave-{int(t0)}.pcap 'tcp and src host 203.0.113.12'"],stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
time.sleep(1);syn(24);ping(2)
out,err=capture.communicate(timeout=20)
evidence['capture']={'exit':capture.returncode,'summary':err.strip()}
if capture.returncode!=0 or '24 packets captured' not in err: raise RuntimeError('capture did not prove 24 packets')
deadline=time.time()+180
while time.time()<deadline:
current=status(t0)
applied={row['ip'] for row in current['blocks'] if row['applied']}
verified={row['ip'] for row in current['audit'] if row.get('action')=='block' and row.get('verified')}
if targets <= set(current['members']) & applied & verified: break
print('waiting for router commits',flush=True);time.sleep(3)
else: raise RuntimeError('both blocks were not verified')
evidence['blocked']=current
# Detection evidence comes from the controller's own audit trail
# (decisions.jsonl, surfaced by the helper as audit rows), NOT from
# /var/log/suricata-leaf/eve.json: since the unix_stream migration,
# the eve FILE only carries stats; alerts flow over the socket and
# land as 'decision' rows with sid. Both sids must be represented.
ids=[r['sid'] for r in current['audit'] if r.get('action')=='decision']
if not {1000030,1000001} <= set(ids):
# 'decision' rows are written the moment the alert lands on the
# socket; the block row lands up to ~60 s later (VyOS commit). If
# the reads disagree, re-poll briefly before declaring failure.
grace=time.time()+30
while not {1000030,1000001} <= set(ids):
if time.time()>=grace: raise RuntimeError('fresh detection evidence missing')
time.sleep(5)
current=status(t0); evidence['blocked']=current
ids=[r['sid'] for r in current['audit'] if r.get('action')=='decision']
print('Both blocks verified; testing real drop counters',flush=True)
before=current['drop_counters'];syn(3);ping(1)
current=status(t0); evidence['enforced']=current
for rule in ('8','9'):
if current['drop_counters'].get(rule,0)<=before.get(rule,0): raise RuntimeError('drop counter did not increment for rule '+rule)
print('Source and destination drop counters increased; waiting for TTL cleanup',flush=True)
deadline=time.time()+300
while time.time()<deadline:
current=status(t0)
tracked={row['ip'] for row in current['blocks']}
if not targets & (set(current['members']) | tracked): break
time.sleep(5)
else: raise RuntimeError('TTL cleanup did not remove test members')
evidence['expired']=current
# With no remaining IDS-BLOCK members, post-expiry probes must not
# increase either IDS drop counter.
before=current['drop_counters'].get('8',0); before9=current['drop_counters'].get('9',0)
syn(3);ping(1)
time.sleep(2)
current=status(t0);evidence['after_expiry_probes']=current
if current['drop_counters'].get('8',0)>before or current['drop_counters'].get('9',0)>before9:
raise RuntimeError('test drop counters still increment after expiry')
unblocked={e['ip'] for e in current['audit'] if e.get('action')=='unblock' and e.get('verified')}
if not targets<=unblocked: raise RuntimeError('verified unblock audit missing')
final=status();evidence['final']=final
if final['protected_hashes']!=baseline['protected_hashes']: raise RuntimeError('protected router configuration changed')
evidence['passed']=True
print('LIVE E2E PASS: capture -> alert -> durable decisions -> verified router blocks -> drop counters -> TTL cleanup',flush=True)
finally:
try:
if changed:
try:
if not evidence['passed']:
run(a.sensor,f'sudo -n python3 /tmp/leaf-live-helper.py cleanup --since {evidence.get("traffic_started",evidence["started"])}')
finally:
print(run(a.sensor,'sudo -n python3 /tmp/leaf-live-helper.py restore && sudo -n systemctl restart leaf-controller').strip(),flush=True)
except Exception as exc:
evidence['restoration_error']=str(exc)
evidence['passed']=False
raise
finally:
evidence['finished']=time.time()
a.output.parent.mkdir(parents=True,exist_ok=True)
a.output.write_text(json.dumps(evidence,indent=2)+'\n')
if __name__=='__main__':main()

60
tests/live_helper.py Normal file
View File

@@ -0,0 +1,60 @@
"""Sensor-side helper for the bounded live proof. Never emits credentials."""
import argparse,json,sys,time,sqlite3,re,hashlib
from pathlib import Path
import yaml
sys.path.insert(0,'/opt/leaf-controller/controller')
from leaf_controller.vyos_client import VyosClient
p=argparse.ArgumentParser();p.add_argument('command',choices=['status','short-ttl','restore','cleanup']);p.add_argument('--since',type=float,default=0)
a=p.parse_args()
config=Path('/etc/leaf-controller/config.yaml'); backup=Path('/run/leaf-e2e-config.original.yaml')
if a.command=='short-ttl':
if backup.exists(): raise SystemExit('Previous live test backup exists; restore it first')
backup.write_bytes(config.read_bytes());backup.chmod(0o600)
cfg=yaml.safe_load(config.read_text());cfg['policy']['ttl_seconds']=180; cfg['batch_delay']=3
config.write_text(yaml.safe_dump(cfg,sort_keys=False));print('temporary TTL=180, batching=3s');raise SystemExit
if a.command=='restore':
if backup.exists(): config.write_bytes(backup.read_bytes());backup.unlink()
print('original controller configuration restored');raise SystemExit
if a.command=='cleanup':
con=sqlite3.connect('/var/lib/leaf-controller/state.sqlite',timeout=10)
with con:
con.execute("UPDATE blocks SET expires=? WHERE ip IN (?,?) AND first_seen>=?",
(time.time(), '203.0.113.12', '192.0.2.202', a.since))
print('Requested expiry for this run only'); raise SystemExit
from leaf_controller.config import load_config, read_key_env
cfg=load_config(str(config))
key=read_key_env('/etc/leaf-controller/env', cfg.get('vyos.key_env', 'LEAF_VYOS_KEY'))
c=VyosClient(cfg.vyos_url(),key,cert_pin=cfg.get('vyos.tls_cert_pin') or None,
ca_bundle=cfg.get('vyos.ca_bundle'),tls_verify=cfg.get('vyos.tls_verify',True),
timeout=75,retries=1)
result={'now':time.time(),'members':c.group_members('IDS-BLOCK')}
text=c._post('/show',{'op':'show','path':['firewall','ipv4','name','VEXLAN-MAOSPF-v4']}).get('data','')
result['drop_counters']={}
for line in text.splitlines():
match=re.match(r'^\s*([89])\s+drop\s+\S+\s+(\d+)\s+(\d+)',line)
if match: result['drop_counters'][match[1]]=int(match[2])
text=c._post('/show',{'op':'show','path':['firewall','ipv4','forward','filter']}).get('data','')
result['forward_destination_drops']=0
for line in text.splitlines():
match=re.match(r'^\s*10\s+drop\s+\S+\s+(\d+)\s+(\d+)',line)
if match: result['forward_destination_drops']=int(match[1])
result['named_drop_counters']=dict(result['drop_counters'])
result['drop_counters']['8']=result['drop_counters'].get('8',0)+result['forward_destination_drops']
con=sqlite3.connect('/var/lib/leaf-controller/state.sqlite')
result['blocks']=[dict(zip(['ip','kind','expires','applied'],r)) for r in con.execute('select ip,kind,expires,applied from blocks')]
result['audit']=[]
for line in Path('/var/log/leaf-controller/decisions.jsonl').read_text().splitlines():
try: row=json.loads(line)
except ValueError: continue
if row.get('ts',0)>=a.since and row.get('ip') in ('203.0.113.12','192.0.2.202'): result['audit'].append(row)
# The live EVE file is stats-only and can temporarily be absent during rename.
# Detection evidence is the audit trail above; do not interpret no file alerts
# as proof of no detections, or depend on stats-file availability for API status.
result['alerts']=None
result['alert_evidence']='controller audit; socket alerts are not journaled to EVE'
result['protected_hashes']={}
if a.since==0:
for path in [['interfaces','ethernet','eth0'],['protocols','ospf'],['protocols','static']]:
data=c.retrieve('showConfig',path).get('data')
result['protected_hashes']['/'.join(path)]=hashlib.sha256(json.dumps(data,sort_keys=True).encode()).hexdigest()
print(json.dumps(result))

144
tests/live_negative.py Normal file
View File

@@ -0,0 +1,144 @@
#!/usr/bin/env python3
"""Negative IDS counter drill: benign traffic does not match the IDS drop while a blocked
src sits in IDS-BLOCK. Run on workstation: python3 tests/live_negative.py \
--output /path/to/evidence.json
Flow: trigger a real block (sid 1000030 injector, spoofed src 203.0.113.12),
then from a DIFFERENT host (second LXC) with a clean TEST-NET src (192.0.2.97)
push benign TCP SYN
flows (count below the 12-packet sid threshold) at the existing transit dst on
a port the drill never touches. Proof while the block is active:
- benign SYN packets are captured on the ingress mirror (does not establish destination receipt);
- the IDS-BLOCK member for the blocked src is still present and verified;
- rule-9 drop counter grows ONLY for the re-probed blocked src, not benign;
- the benign source is absent from IDS-BLOCK and the state database.
Then TTL expiry cleans up like live_e2e does. TTL=180 for the run; config
restored on exit. Never adds listeners/services on the router; uses existing
transit destinations. Health and capture metrics are included in the results.
"""
import argparse,json,subprocess,time,ipaddress
from pathlib import Path
BLOCKED='203.0.113.12'
def main():
ap=argparse.ArgumentParser(description=__doc__)
ap.add_argument('--sensor',default='yuru@192.168.100.84')
ap.add_argument('--lxc',default='root@172.19.32.3')
ap.add_argument('--benign-host',default='root@172.19.32.5')
ap.add_argument('--benign-src',default='192.0.2.97')
ap.add_argument('--ssh-key',default=str(Path.home()/'.ssh/id_ed25519_alice'))
ap.add_argument('--output',type=Path,required=True)
a=ap.parse_args(); root=Path(__file__).resolve().parents[1]
try:
addr=ipaddress.IPv4Address(a.benign_src)
except ValueError:
ap.error('--benign-src must be a TEST-NET IPv4 address')
if not any(addr in ipaddress.ip_network(n) for n in ('192.0.2.0/24','198.51.100.0/24','203.0.113.0/24')) or a.benign_src==BLOCKED:
ap.error('--benign-src must be a distinct TEST-NET address')
ssh=['ssh','-F','/dev/null','-i',a.ssh_key,'-o','BatchMode=yes','-o','ConnectTimeout=8']
scp=['scp','-F','/dev/null','-i',a.ssh_key,'-o','BatchMode=yes']
def run(host,cmd):
return subprocess.run(ssh+[host,cmd],capture_output=True,text=True,check=True,timeout=150).stdout
def status(since=0):
return json.loads(run(a.sensor,f'sudo -n python3 /tmp/leaf-live-helper.py status --since {since}'))
def syn(host,src,dst,port,n):
print(run(host,f'python3 /tmp/leaf-send-syn.py {src} {dst} {port} {n}').strip(),flush=True)
def metrics_tail():
lines=run(a.sensor,'sudo -n journalctl -u leaf-controller --since "-4 min" --no-pager | grep "leaf.metrics" | tail -4')
capture=run(a.sensor,'sudo -n tail -n 2 /var/log/leaf-controller/capture.log 2>/dev/null || true')
return {'health':[l.split('leaf.metrics: ',1)[1] for l in lines.splitlines() if 'leaf.metrics'],
'capture_windows':[json.loads(l) for l in capture.splitlines() if l]}
evidence={'started':time.time(),'passed':False}
changed=False
try:
subprocess.run(scp+[str(root/'tests/live_helper.py'),a.sensor+':/tmp/leaf-live-helper.py'],check=True)
subprocess.run(scp+[str(root/'tests/syn_packets.py'),a.lxc+':/tmp/leaf-send-syn.py'],check=True)
subprocess.run(scp+[str(root/'tests/syn_packets.py'),a.benign_host+':/tmp/leaf-send-syn.py'],check=True)
baseline=status(); evidence['baseline']=baseline
if baseline['members'] or baseline['blocks']: raise RuntimeError('negative proof requires idle empty group/state for counter attribution')
if BLOCKED in set(baseline['members']): raise RuntimeError('drill address already blocked; refusing to alter existing ownership')
if a.benign_src in set(baseline['members']): raise RuntimeError('benign src already blocked; run clean of it first')
# An empty baseline allows rule-9 changes to be attributed to the
# test source. Ingress capture establishes observation, not delivery.
clean_member_state = not baseline['members']
run(a.sensor,'sudo -n python3 /tmp/leaf-live-helper.py short-ttl')
changed=True
run(a.sensor,'sudo -n systemctl restart leaf-controller')
t0=time.time(); evidence['traffic_started']=t0
capture=subprocess.Popen(ssh+[a.sensor,f"sudo -n timeout 15 tcpdump -ni eth1 -c 24 -w /tmp/leaf-neg-block-{int(t0)}.pcap 'tcp and src host 203.0.113.12 and dst port 22'"],stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
time.sleep(1); syn(a.lxc,BLOCKED,'192.0.2.201',22,24)
out,err=capture.communicate(timeout=20)
evidence['block_trigger']={'exit':capture.returncode,'summary':err.strip()}
if capture.returncode!=0 or '24 packets captured' not in err: raise RuntimeError('block-trigger capture did not prove 24 packets')
deadline=time.time()+180
while time.time()<deadline:
current=status(t0)
applied={row['ip'] for row in current['blocks'] if row['applied']}
verified={row['ip'] for row in current['audit'] if row.get('action')=='block' and row.get('verified')}
if BLOCKED in set(current['members']) and BLOCKED in applied and BLOCKED in verified: break
print('waiting for router commit',flush=True);time.sleep(3)
else: raise RuntimeError('block was not verified')
evidence['blocked']=current
print(f'block verified; benign flows from {a.benign_src} under active block',flush=True)
before_neg=current['drop_counters']
neg=subprocess.Popen(ssh+[a.sensor,f"sudo -n timeout 15 tcpdump -ni eth1 -c 12 -w /tmp/leaf-neg-benign-{int(t0)}.pcap 'tcp and src host {a.benign_src} and dst port 80'"],stdout=subprocess.PIPE,stderr=subprocess.PIPE,text=True)
time.sleep(1); syn(a.benign_host,a.benign_src,'192.0.2.201',80,6)
time.sleep(2); syn(a.benign_host,a.benign_src,'192.0.2.201',80,6)
out,err=neg.communicate(timeout=20)
evidence['benign_capture']={'exit':neg.returncode,'summary':err.strip()}
if neg.returncode!=0 or '12 packets captured' not in err: raise RuntimeError('benign flow was not observed on the ingress mirror under active block')
current=status(t0); evidence['during_benign']=current
if BLOCKED not in set(current['members']): raise RuntimeError('blocked member vanished during benign window')
if a.benign_src in set(current['members']): raise RuntimeError('benign src entered IDS-BLOCK')
if a.benign_src in {row['ip'] for row in current['blocks']}: raise RuntimeError('benign src entered state db')
if clean_member_state and current['drop_counters'].get('9',0)!=before_neg.get('9',0):
raise RuntimeError('rule-9 drop counter grew for benign src')
print('benign flows observed without IDS drop-counter growth; re-probing blocked src',flush=True)
before=current['drop_counters']; syn(a.lxc,BLOCKED,'192.0.2.201',22,3)
current=status(t0); evidence['enforced']=current
if current['drop_counters'].get('9',0)<=before.get('9',0): raise RuntimeError('rule-9 drop counter did not grow for blocked src')
if clean_member_state:
again=status(t0); before2=again['drop_counters']; syn(a.benign_host,a.benign_src,'192.0.2.201',80,3)
current=status(t0); evidence['benign_reprobe']=current
if current['drop_counters'].get('9',0)!=before2.get('9',0): raise RuntimeError('rule-9 counter grew for benign re-probe')
# Socket mode has no file alerts. State/member absence above and counters
# are the evidence; ingress capture cannot prove destination delivery.
print('drop counter grew only for blocked src; waiting for TTL cleanup',flush=True)
deadline=time.time()+300
while time.time()<deadline:
current=status(t0)
tracked={row['ip'] for row in current['blocks']}
if BLOCKED not in set(current['members'])|tracked: break
time.sleep(5)
else: raise RuntimeError('TTL cleanup did not remove drill member')
evidence['expired']=current
counters=current['drop_counters']; syn(a.lxc,BLOCKED,'192.0.2.201',22,3)
current=status(t0); evidence['after_expiry_probes']=current
if current['drop_counters']!=counters: raise RuntimeError('drill drop counters still increment after expiry')
unblocked={row['ip'] for row in current['audit'] if row.get('action')=='unblock' and row.get('verified')}
if BLOCKED not in unblocked: raise RuntimeError('verified unblock audit missing')
final=status(); evidence['final']=final
if final['protected_hashes']!=baseline['protected_hashes']: raise RuntimeError('protected router configuration changed')
evidence['metrics']=metrics_tail()
evidence['passed']=True
print('NEGATIVE DRILL PASS: block verified -> benign flows observed on mirror -> '
'rule-9 grows only for blocked src -> benign never entered IDS-BLOCK -> TTL cleanup',flush=True)
finally:
try:
if changed:
try:
if not evidence['passed']:
run(a.sensor,f'sudo -n python3 /tmp/leaf-live-helper.py cleanup --since {evidence.get("traffic_started",evidence["started"])}')
finally:
print(run(a.sensor,'sudo -n python3 /tmp/leaf-live-helper.py restore && sudo -n systemctl restart leaf-controller').strip(),flush=True)
except Exception as exc:
evidence['restoration_error']=str(exc)
evidence['passed']=False
raise
finally:
evidence['finished']=time.time()
a.output.parent.mkdir(parents=True,exist_ok=True)
a.output.write_text(json.dumps(evidence,indent=2)+'\n')
if __name__=='__main__':main()

176
tests/mock_vyos_server.py Normal file
View File

@@ -0,0 +1,176 @@
#!/usr/bin/env python3
"""Mock VyOS HTTP API (2026.03 subset) for offline E2E tests.
Endpoints: GET /info ; POST /configure|/retrieve|/show|/config-file (multipart form: data, key)
State: in-memory address-group; member list mirrored to --state-file on every mutation.
Semantics mirrored from docs: bad key -> 401; semantic errors -> success:false + error.
"""
from __future__ import annotations
import argparse
import email.parser
import json
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
STATE = {"members": set(), "config_comments": {}}
LOCK = threading.Lock()
def dump_state(path: str) -> None:
with open(path, "w") as fh:
json.dump({"members": sorted(STATE["members"])}, fh)
def parse_form(handler) -> dict:
ctype = handler.headers.get("Content-Type", "")
length = int(handler.headers.get("Content-Length", "0"))
body = handler.rfile.read(length)
if "multipart/form-data" in ctype:
import email
import email.policy
msg = email.parser.BytesParser(policy=email.policy.compat32).parsebytes(
b"Content-Type: " + ctype.encode() + b"\r\nMIME-Version: 1.0\r\n\r\n" + body)
out = {}
for part in msg.walk():
if part.get_content_type() == "multipart/form-data":
continue
disp = part.get("Content-Disposition", "")
if "name=" not in disp:
continue
name = disp.split('name="', 1)[1].split('"', 1)[0]
out[name] = part.get_payload(decode=True).decode()
return out
# urlencoded fallback
from urllib.parse import parse_qs
return {k: v[0] for k, v in parse_qs(body.decode()).items()}
class Handler(BaseHTTPRequestHandler):
key_expected = "testkey"
state_file = "mock_state.json"
log_lines = []
def log_message(self, fmt, *args): # quiet
Handler.log_lines.append(fmt % args)
def _send(self, code: int, obj) -> None:
body = json.dumps(obj).encode()
self.send_response(code)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def do_GET(self):
if self.path == "/info":
self._send(200, {"version": "VYOS mock (22.04 mock for 2026.03 semantics)",
"api": {"status": "healthy"}})
else:
self._send(404, {"success": False, "error": "not found"})
def do_POST(self):
form = parse_form(self)
if form.get("key") != self.key_expected:
self._send(401, {"success": False, "error": "bad auth token"})
return
try:
payload = json.loads(form.get("data", "{}"))
except json.JSONDecodeError:
self._send(400, {"success": False, "error": "data is not valid json"})
return
path = self.path.rstrip("/")
try:
if path == "/configure":
self.handle_configure(payload)
elif path == "/retrieve":
self.handle_retrieve(payload)
elif path == "/show":
self._send(200, {"success": True, "data": "\n".join(
f"[{k}] {v}" for k, v in STATE["config_comments"].items())})
elif path == "/config-file":
if payload.get("op") == "save":
self._send(200, {"success": True})
else:
self._send(400, {"success": False, "error": "unknown op"})
else:
self._send(404, {"success": False, "error": "no endpoint"})
except Exception as exc: # mock: keep simple
self._send(500, {"success": False, "error": f"mock internal: {exc}"})
# ------------------------------------------------------------ configure
def _group_path(self, tokens) -> bool:
return (len(tokens) >= 6 and tokens[:4] ==
["firewall", "group", "address-group", "IDS-BLOCK"] and
tokens[4] == "address")
@staticmethod
def _member(tokens) -> str:
return tokens[5]
def handle_configure(self, payload) -> None:
ops = payload if isinstance(payload, list) else [payload]
with LOCK:
for op in ops:
if not isinstance(op, dict):
self._send(400, {"success": False, "error": "op must be object"})
return
kind, tokens = op.get("op"), op.get("path", [])
if kind not in ("set", "delete", "comment") or not self._group_path(tokens):
self._send(400, {"success": False,
"error": f"mock: unhandled op/path {kind} {tokens}"})
return
member = self._member(tokens)
if kind == "comment":
STATE["config_comments"][member] = op.get("comment", "")
continue
if kind == "set":
STATE["members"].add(member)
else:
STATE["members"].discard(member)
dump_state(self.state_file)
self._send(200, {"success": True,
"comments": [STATE["config_comments"].get(self._member(o["path"]), "")
for o in ops
if isinstance(o.get("path"), list) and self._group_path(o["path"])]})
# ------------------------------------------------------------ retrieve
def handle_retrieve(self, payload) -> None:
if payload.get("op") == "showConfig" and payload.get("path") == ["firewall"]:
self._send(200, {"success": True, "data": {"ipv4": {"forward": {"filter": {"rule": {
"8": {"action": "drop", "destination": {"group": {"address-group": "IDS-BLOCK"}}},
"9": {"action": "drop", "source": {"group": {"address-group": "IDS-BLOCK"}}}
}}}}}})
return
if payload.get("op") != "returnValues":
self._send(400, {"success": False, "error": "mock: only returnValues supported"})
return
tokens = payload.get("path", [])
if (len(tokens) == 5 and tokens[:4] ==
["firewall", "group", "address-group", "IDS-BLOCK"] and tokens[4] == "address"):
with LOCK:
self._send(200, {"success": True, "data": sorted(STATE["members"])})
else:
self._send(400, {"success": False, "error": f"mock: no values for {tokens}"})
def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("--port", type=int, default=18443)
ap.add_argument("--state-file", default="mock_state.json")
ap.add_argument("--key", default="testkey")
args = ap.parse_args()
Handler.state_file = args.state_file
Handler.key_expected = args.key
try: # allow pre-seeding foreign members
with open(args.state_file) as fh:
STATE["members"] = set(json.load(fh).get("members", []))
except FileNotFoundError:
pass
srv = ThreadingHTTPServer(("127.0.0.1", args.port), Handler)
print(f"mock vyos api on 127.0.0.1:{args.port} state={args.state_file}", flush=True)
srv.serve_forever()
if __name__ == "__main__":
main()

31
tests/replay_fixture.py Normal file
View File

@@ -0,0 +1,31 @@
#!/usr/bin/env python3
"""Run on the sensor as root: offline packet/rule proof with isolated outputs."""
import json,subprocess,tempfile
from pathlib import Path
import yaml
from syn_packets import packet
import struct,time
def main():
root=Path(tempfile.mkdtemp(prefix='leaf-replay-'))
pcap=root/'fixture.pcap'
with pcap.open('wb') as out:
out.write(struct.pack('<IHHIIII',0xa1b2c3d4,2,4,0,0,65535,1))
for i in range(24):
frame=packet('203.0.113.12','192.0.2.201',22,i)
out.write(struct.pack('<IIII',int(time.time()),i*1000,len(frame),len(frame))+frame)
cfg=yaml.safe_load(Path('/etc/suricata-leaf/suricata-leaf.yaml').read_text())
cfg['outputs']=[{'eve-log': {'enabled': True, 'filetype': 'regular',
'filename': str(root/'eve.json'), 'types': [{'alert': {'metadata': True}}, 'stats']}}]
conf=root/'config.yaml';conf.write_text('%YAML 1.1\n---\n'+yaml.safe_dump(cfg))
subprocess.run(['suricata','-r',str(pcap),'-c',str(conf),'-l',str(root),'--runmode','single'],check=True)
events=[json.loads(line) for line in (root/'eve.json').read_text().splitlines()]
alerts=[e for e in events if e.get('event_type')=='alert']
stats=[e['stats'] for e in events if e.get('event_type')=='stats'][-1]
assert [e['alert']['signature_id'] for e in alerts]==[1000030,1000030],alerts
assert stats['decoder']['pkts']==24 and stats['decoder']['invalid']==0
assert stats['tcp']['syn']==24 and stats['tcp']['invalid_checksum']==0
assert stats['defrag']['ipv4']['fragments']==0
print('OFFLINE REPLAY PASS: 24 valid SYNs, zero fragments/checksum errors, two sid 1000030 alerts')
print('Artifacts:',root)
if __name__=='__main__':main()

166
tests/run_e2e.sh Normal file
View File

@@ -0,0 +1,166 @@
#!/usr/bin/env bash
# E2E prototype test: mock VyOS API + fake Suricata EVE -> controller decisions.
# Run from repo root: bash tests/run_e2e.sh
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
T="$ROOT/tests/.e2e"
PORT=18443
VAULT="$(python3 -c 'import secrets; print(secrets.token_urlsafe(18))')" # ephemeral test credential
rm -rf "$T"; mkdir -p "$T"
trap 'kill $(cat "$T/pids" 2>/dev/null) 2>/dev/null' EXIT
# seed a foreign member directly into mock state (controller should adopt+expire it)
python3 - "$T/mock_state.json" <<'PY'
import json,sys
json.dump({"members": ["198.51.100.7"]}, open(sys.argv[1],"w"))
PY
# --- 1. mock vyos api ---------------------------------------------------
python3 "$ROOT/tests/mock_vyos_server.py" --port $PORT --key "$VAULT" \
--state-file "$T/mock_state.json" > "$T/mock.log" 2>&1 &
echo $! >> "$T/pids"
sleep 1
grep -q "mock vyos api" "$T/mock.log" || { echo "FAIL: mock api not started"; cat "$T/mock.log"; exit 1; }
# --- 2. controller config ------------------------------------------------
cat > "$T/config.yaml" <<YAML
vyos:
url: http://127.0.0.1:$PORT
key_env: LEAF_VAULT
tls_verify: true
block_group: IDS-BLOCK
adopt_foreign_members: true
eve:
path: $T/eve.json
alert_socket:
path: "${LEAF_E2E_SOCKET:-}"
queue_size: 1000
policy:
mode: threshold
min_hits: 2
window_seconds: 300
ttl_seconds: 6
max_blocks: 10
block_src: true
state_db: $T/state.sqlite
audit_log: $T/decisions.jsonl
poll_interval: 0.5
sync_interval: 4.0
YAML
touch "$T/eve.json"
send_event() {
if [ -n "${LEAF_E2E_SOCKET:-}" ]; then
python3 -c 'import socket,sys; s=socket.socket(socket.AF_UNIX); s.connect(sys.argv[1]); s.sendall(sys.stdin.buffer.read()); s.close()' "$LEAF_E2E_SOCKET"
else
cat >> "$T/eve.json"
fi
}
# --- 3. run controller ---------------------------------------------------
( cd "$ROOT/controller" && LEAF_VAULT=$VAULT exec python3 -m leaf_controller.main \
--config "$T/config.yaml" ) > "$T/controller.log" 2>&1 &
echo $! >> "$T/pids"
sleep 2
alert() { # sid dest_ip action
python3 -c '
import json,sys
print(json.dumps({"timestamp":"2026-11-03T12:00:00.000000Z","event_type":"alert",
"src_ip":"203.0.113.7","dest_ip":sys.argv[2],"src_port":41000,"dest_port":80,
"proto":"TCP","alert":{"signature_id":int(sys.argv[1]),"signature":"e2e test",
"severity":2,"action":"allowed",
"metadata":{"leaf_action":[sys.argv[3]]}}}))' "$1" "$2" "$3" | send_event; }
# --- 4. threshold: first hit no-op, second hit must block 192.0.2.5 ------
alert 1000001 192.0.2.5 block; sleep 1.5
alert 1000001 192.0.2.5 block; sleep 2
grep -q "192.0.2.5" "$T/mock_state.json" || { echo "FAIL: expected block of 192.0.2.5"; cat "$T/controller.log"; exit 1; }
# --- 5. allow-list: LAN dest must never be blockable ----------------------
alert 1000001 10.66.0.9 block; alert 1000001 10.66.0.9 block; sleep 2
grep -q "10.66.0.9" "$T/mock_state.json" && { echo "FAIL: allowlisted IP was blocked!"; exit 1; }
# --- 6. log rotation: rotate eve.json, new events must still be seen -----
mv "$T/eve.json" "$T/eve.json.1"; touch "$T/eve.json"
send_event() {
if [ -n "${LEAF_E2E_SOCKET:-}" ]; then
python3 -c 'import socket,sys; s=socket.socket(socket.AF_UNIX); s.connect(sys.argv[1]); s.sendall(sys.stdin.buffer.read()); s.close()' "$LEAF_E2E_SOCKET"
else
cat >> "$T/eve.json"
fi
}
alert 1000002 192.0.2.9 block; sleep 1
alert 1000002 192.0.2.9 block; sleep 2
grep -q "192.0.2.9" "$T/mock_state.json" || { echo "FAIL: rotation: expected block of 192.0.2.9"; cat "$T/controller.log"; exit 1; }
# --- 7. foreign member adoption: 198.51.100.7 was adopted, then expires --
grep -q "adopted foreign member 198.51.100.7" "$T/controller.log" || { echo "FAIL: foreign member not adopted"; grep -i adopt "$T/controller.log"; exit 1; }
# --- 8. TTL expiry: both blocked IPs disappear within ~10s ---------------
end=$((SECONDS+15))
while [ $SECONDS -lt $end ]; do
if ! grep -q "192.0.2.5" "$T/mock_state.json" && ! grep -q "192.0.2.9" "$T/mock_state.json"; then break; fi
sleep 1
done
grep -q "192.0.2.5" "$T/mock_state.json" && { echo "FAIL: TTL expiry didn't unblock 192.0.2.5"; cat "$T/mock_state.json"; exit 1; }
grep -q "192.0.2.9" "$T/mock_state.json" && { echo "FAIL: TTL expiry didn't unblock 192.0.2.9"; cat "$T/mock_state.json"; exit 1; }
# --- 9b. src-mode: block_src metadata + policy.block_src=true ------------
# attacker src from TEST-NET, victim on allowlisted LAN -> src must be blocked,
# victim dest must NOT be blocked, and allowlisted src must be ignored.
# allowlisted (RFC1918-floor) src must never be blocked, even in src mode
cat > "$T/fakeeve_floor.py" <<'PY'
import json,sys
print(json.dumps({"timestamp":"2026-11-03T12:04:00.000000Z","event_type":"alert",
"src_ip":"10.130.0.5","dest_ip":"10.66.0.9","src_port":41000,"dest_port":22,
"proto":"TCP","alert":{"signature_id":1000020,"signature":"e2e floor test","severity":2,
"action":"allowed","metadata":{"leaf_action":["block_src"]}}}))
PY
python3 "$T/fakeeve_floor.py" | send_event; sleep 1
python3 "$T/fakeeve_floor.py" | send_event; sleep 2
grep -q "10.130.0.5" "$T/mock_state.json" && { echo "FAIL: allowlisted src was blocked!"; exit 1; }
cat > "$T/fakeeve_src.py" <<'PY'
import json,sys
# src from TEST-NET (blockable) hitting allowlisted dest
line = json.dumps({"timestamp":"2026-11-03T12:05:00.000000Z","event_type":"alert",
"src_ip":"198.51.100.42","dest_ip":"10.66.0.9","src_port":41001,"dest_port":22,
"proto":"TCP","alert":{"signature_id":1000020,"signature":"e2e src test","severity":2,
"action":"allowed","metadata":{"leaf_action":["block_src"]}}})
print(line)
PY
python3 "$T/fakeeve_src.py" | send_event; sleep 1
python3 "$T/fakeeve_src.py" | send_event; sleep 2
grep -q "198.51.100.42" "$T/mock_state.json" || { echo "FAIL: src-mode block of 198.51.100.42 missing"; cat "$T/controller.log"; exit 1; }
# --- 9c. unix_stream socket replay: fake suricata client -> controller ----
# exercises the real AF_UNIX path end-to-end: the controller listens, a
# client connects and streams newline-delimited JSON; the same threshold
# pair must block 192.0.2.42 through the socket path alone
if [ -n "${LEAF_E2E_SOCKET:-}" ]; then
python3 - "$LEAF_E2E_SOCKET" <<'PY'
import json, socket, sys
s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
s.connect(sys.argv[1])
rec = {"timestamp": "2026-11-03T12:06:00.000000Z", "event_type": "alert",
"src_ip": "203.0.113.77", "dest_ip": "192.0.2.42",
"src_port": 41002, "dest_port": 80, "proto": "TCP",
"alert": {"signature_id": 1000001, "signature": "socket e2e",
"severity": 2, "action": "allowed",
"metadata": {"leaf_action": ["block"]}}}
for _ in range(2):
s.sendall((json.dumps(rec) + "\n").encode())
s.close()
PY
sleep 2
grep -q "192.0.2.42" "$T/mock_state.json" || { echo "FAIL: socket replay: expected block of 192.0.2.42"; cat "$T/controller.log"; exit 1; }
fi
# --- 10. audit log sanity --------------------------------------------------
grep -q '"action": "block"' "$T/decisions.jsonl" || { echo "FAIL: audit missing block entries"; exit 1; }
grep -q '"action": "unblock"' "$T/decisions.jsonl" || { echo "FAIL: audit missing unblock entries"; exit 1; }
echo "E2E PASS"

59
tests/syn_packets.py Normal file
View File

@@ -0,0 +1,59 @@
#!/usr/bin/env python3
"""Bounded TEST-NET SYN generator; --pcap writes a fixture without sending traffic."""
import argparse
import ipaddress
import re
import socket
import struct
import subprocess
import time
from pathlib import Path
def checksum(data):
if len(data)%2: data += b'\0'
total=sum(struct.unpack('!%dH'%(len(data)//2),data))
while total>>16: total=(total&0xffff)+(total>>16)
return (~total)&0xffff
def packet(src,dst,dport,index,smac=b'\x02\0\0\0\0\1',dmac=b'\x02\0\0\0\0\2',flags=2):
a,b=socket.inet_aton(src),socket.inet_aton(dst)
tcp=struct.pack('!HHIIBBHHH',40000+index%20000,dport,7000+index,0,0x50,flags,1024,0,0)
pseudo=a+b+struct.pack('!BBH',0,6,len(tcp))
tcp=tcp[:16]+struct.pack('!H',checksum(pseudo+tcp))+tcp[18:]
ip=struct.pack('!BBHHHBBH4s4s',0x45,0,40,index%65536,0,64,6,0,a,b)
ip=ip[:10]+struct.pack('!H',checksum(ip))+ip[12:]
return dmac+smac+b'\x08\0'+ip+tcp
def main():
ap=argparse.ArgumentParser(description=__doc__)
ap.add_argument('src'); ap.add_argument('dst'); ap.add_argument('port',type=int); ap.add_argument('count',type=int)
ap.add_argument('--pcap',type=Path); ap.add_argument('--interface',default='eth0'); ap.add_argument('--gateway',default='172.19.32.1')
a=ap.parse_args()
nets=[ipaddress.ip_network(n) for n in ('192.0.2.0/24','198.51.100.0/24','203.0.113.0/24')]
if not all(any(ipaddress.ip_address(ip) in n for n in nets) for ip in (a.src,a.dst)):
ap.error('source and destination must be TEST-NET addresses')
if not 1<=a.count<=500 or not 1<=a.port<=65535: ap.error('count must be 1..500 and port 1..65535')
if a.pcap:
with a.pcap.open('wb') as f:
f.write(struct.pack('<IHHIIII',0xa1b2c3d4,2,4,0,0,65535,1))
t=time.time()
for i in range(a.count):
data=packet(a.src,a.dst,a.port,i)
stamp=t+i*.001
f.write(struct.pack('<IIII',int(stamp),int(stamp%1*1e6),len(data),len(data))+data)
print('wrote',a.count,'packets to',a.pcap)
return
def mac(value): return bytes.fromhex(value.replace(':',''))
smac=mac(Path('/sys/class/net',a.interface,'address').read_text().strip())
subprocess.run(['ping','-c1','-W1',a.gateway],capture_output=True)
text=subprocess.run(['ip','neigh','show',a.gateway,'dev',a.interface],capture_output=True,text=True,check=True).stdout
match=re.search(r'lladdr ([0-9a-f:]{17})',text)
if not match: ap.error('gateway MAC unresolved')
with socket.socket(socket.AF_PACKET,socket.SOCK_RAW,socket.htons(0x0800)) as s:
s.bind((a.interface,0))
for i in range(a.count):
s.send(packet(a.src,a.dst,a.port,i,smac,mac(match.group(1))))
time.sleep(.01)
print('sent',a.count,'SYNs',a.src,'->',a.dst+':'+str(a.port))
if __name__=='__main__': main()

135
tests/test_alertsock.py Normal file
View File

@@ -0,0 +1,135 @@
"""Framing tests for the suricata unix_stream listener (suricata is the
client; the controller owns the listening socket). Uses real AF_UNIX
connections, no suricata binary needed."""
import os
import socket
import tempfile
import time
import unittest
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'controller'))
from leaf_controller.alertsock import AlertSocket
class AlertSocketTest(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.path = os.path.join(self.tmp.name, 'alerts.sock')
self.sock = AlertSocket(self.path, 100)
self.sock.bind()
self.sock.start()
def tearDown(self):
self.sock.close()
self.tmp.cleanup()
def _client(self):
s = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
s.connect(self.path)
return s
def _wait(self, n=1, pred=None, timeout=5.0):
"""wait until drain() has >= n records, or pred() is true"""
deadline = time.time() + timeout
recs = []
while time.time() < deadline:
recs = self.sock.drain()
if len(recs) >= n and (pred is None or pred()):
return recs
time.sleep(0.05)
return recs
def _rec(self, n):
return ('{"event_type": "alert", "n": %d}\n' % n).encode()
def test_single_connection_streams_records(self):
s = self._client()
s.sendall(self._rec(1) + self._rec(2))
s.close()
recs = self._wait(2)
self.assertEqual([r['n'] for r in recs], [1, 2])
def test_partial_line_frames_only_on_newline(self):
s = self._client()
s.sendall(b'{"event_type": "alert", "n"')
time.sleep(0.3)
self.assertEqual(self.sock.drain(), [])
s.sendall(b': 7}\n')
recs = self._wait(1)
self.assertEqual([r['n'] for r in recs], [7])
s.close()
def test_non_alert_records_counted_other_never_queued(self):
# ingress gate: the socket output block carries alerts only; a
# stray stats record must never reach policy/state
s = self._client()
s.sendall(b'{"event_type": "stats", "x": 1}\n')
s.close()
self._wait(0, pred=lambda: self.sock.counts['other'] >= 1)
self.assertEqual(self.sock.drain(), [])
self.assertGreaterEqual(self.sock.counts['other'], 1)
def test_reconnect_yields_continuous_stream(self):
self._client().close()
s = self._client()
s.sendall(self._rec(1) + self._rec(2))
s.close()
recs = self._wait(2)
self.assertEqual([r['n'] for r in recs], [1, 2])
self.assertGreaterEqual(self.sock.counts['connections'], 2)
def test_garbage_and_non_object_lines_counted_not_hung_controller(self):
s = self._client()
s.sendall(b'not-json\n[1,2,3]\n' + self._rec(1))
s.close()
# unparseable line -> parse_errors; valid JSON but not an alert
# object -> other gate; the good record must still flow through
recs = self._wait(1, pred=lambda: self.sock.counts['parse_errors'] >= 1
and self.sock.counts['other'] >= 1)
self.assertEqual([r['n'] for r in recs], [1])
self.assertGreaterEqual(self.sock.counts['parse_errors'], 1)
self.assertGreaterEqual(self.sock.counts['other'], 1)
def test_queue_overflow_drops_are_counted(self):
# bounded queue: a stalled consumer forces the eager-drain semantics
# and losses must be visible in counts, never silently dropped
self.sock.close()
self.sock = AlertSocket(self.path, 1)
self.sock.bind()
self.sock.start()
s = self._client()
s.sendall(self._rec(1) + self._rec(2) + self._rec(3))
s.close()
deadline = time.time() + 5.0
while time.time() < deadline and self.sock.counts['queue_drops'] < 1:
time.sleep(0.05)
self.assertGreaterEqual(self.sock.counts['queue_drops'], 1)
recs = self.sock.drain()
self.assertEqual(len(recs), 1) # exactly the survivor of the burst
def test_close_removes_socket_file(self):
self.assertTrue(os.path.exists(self.path))
self.sock.close()
self.assertFalse(os.path.exists(self.path))
def test_bind_replaces_stale_socket_path(self):
# An unowned socket whose listener has exited can be reclaimed.
# Active listeners and separately locked socket paths are protected.
self.sock.close()
stale = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
stale.bind(self.path)
stale.close()
self.sock = AlertSocket(self.path, 10)
self.sock.bind()
self.sock.start()
s = self._client()
s.sendall(self._rec(1))
s.close()
recs = self._wait(1)
self.assertEqual([r['n'] for r in recs], [1])
if __name__ == '__main__':
unittest.main()

448
tests/test_controller.py Normal file
View File

@@ -0,0 +1,448 @@
"""Offline regression tests. No lab access or router writes."""
import copy
import json
import os
from pathlib import Path
import sys
import tempfile
import threading
import time
import unittest
from unittest.mock import patch
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'controller'))
from leaf_controller.config import Config, DEFAULTS
from leaf_controller.eve import EveTailer
from leaf_controller.main import Controller
from leaf_controller.vyos_client import (
PinMismatchError, VyosClient, VyosApiError, normalize_pin)
class Router:
member_op = staticmethod(VyosClient.member_op)
def __init__(self, members=()):
self.members = set(members)
self.commits = []
self.fail = False
self.after_commit = None
def op_with_retry(self, fn, *args): return fn(*args)
def group_members(self, group): return sorted(self.members)
def configure(self, ops):
self.commits.append(ops)
for op in ops:
ip = op['path'][-1]
if op['op'] == 'set': self.members.add(ip)
else: self.members.discard(ip)
if self.after_commit: self.after_commit()
if self.fail: raise TimeoutError('commit acknowledged too late')
return {'success': True}
def save_config(self): pass
class Tests(unittest.TestCase):
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.root = Path(self.tmp.name)
raw = copy.deepcopy(DEFAULTS)
raw.update(state_db=str(self.root/'state.sqlite'), audit_log=str(self.root/'audit.jsonl'))
raw['policy'].update(mode='immediate', block_src=True)
with patch.object(Config, 'vyos_key', return_value='offline-test'):
self.ctl = Controller(Config(raw), dry_run=False)
self.router = self.ctl.client = Router()
def tearDown(self):
self.ctl.close()
self.tmp.cleanup()
def alert(self, ip='192.0.2.77', action='block', src='203.0.113.11'):
return {'event_type':'alert','src_ip':src,'dest_ip':ip,
'alert':{'signature_id':1000001,'metadata':{'leaf_action':[action]}}}
def test_decision_is_durable_without_api(self):
self.ctl.handle_event(self.alert())
self.assertIsNotNone(self.ctl.state.get('192.0.2.77'))
self.assertFalse(self.router.commits)
def test_batch_dedup_readback_and_expiry(self):
for ip in ('192.0.2.77','192.0.2.78','192.0.2.77'):
self.ctl.handle_event(self.alert(ip))
self.ctl.sync_router_state()
self.assertEqual(len(self.router.commits),1)
self.assertEqual(len(self.router.commits[0]),2)
self.assertTrue(self.ctl.state.get('192.0.2.77').applied)
self.ctl.state.con.execute('UPDATE blocks SET expires=?',(time.time()-1,))
self.ctl.state.con.commit()
self.ctl.sync_router_state()
self.assertEqual(self.router.members,set())
self.assertEqual(self.ctl.state.all_blocks(),[])
def test_expired_startup_not_adopted(self):
self.ctl.state.upsert_hit('192.0.2.77','dest',1,1,now=time.time()-10)
self.router.members.add('192.0.2.77')
self.ctl.sync_router_state()
self.assertIsNone(self.ctl.state.get('192.0.2.77'))
self.assertFalse(self.router.members)
def test_state_owned_group_removes_stale_router_members(self):
self.ctl.cfg.raw['adopt_foreign_members']=False
self.router.members.add('192.0.2.77')
self.ctl.sync_router_state()
self.assertFalse(self.router.members)
self.assertEqual(self.ctl.state.all_blocks(),[])
def test_timeout_verified_before_resubmit(self):
self.ctl.handle_event(self.alert())
self.router.fail=True
with self.assertRaises(TimeoutError): self.ctl.sync_router_state()
self.router.fail=False
self.ctl.sync_router_state()
self.assertEqual(len(self.router.commits),1)
self.assertTrue(self.ctl.state.get('192.0.2.77').applied)
def test_refresh_during_delete_is_retained(self):
self.ctl.state.upsert_hit('192.0.2.77','dest',1,1,now=time.time()-10)
self.router.members.add('192.0.2.77')
self.router.after_commit=lambda: self.ctl.handle_event(self.alert())
self.ctl.sync_router_state()
self.assertGreater(self.ctl.state.get('192.0.2.77').expires,time.time())
self.assertFalse(self.ctl.state.get('192.0.2.77').applied)
self.router.after_commit=None
self.ctl.sync_router_state()
self.assertIn('192.0.2.77',self.router.members)
def test_slow_commit_does_not_block_decisions(self):
self.ctl.handle_event(self.alert())
started, release = threading.Event(), threading.Event()
self.router.after_commit=lambda: (started.set(),release.wait(3))
def worker():
try: self.ctl.sync_router_state()
finally: self.ctl.state.close()
t=threading.Thread(target=worker); t.start()
self.assertTrue(started.wait(2))
t0=time.monotonic()
for n in range(10): self.ctl.handle_event(self.alert(f'192.0.2.{n+100}'))
elapsed=time.monotonic()-t0
release.set(); t.join(3)
self.assertLess(elapsed,1)
self.assertEqual(self.ctl.state.count_active(),11)
def test_floor_src_ipv6_and_bad_events(self):
for ip in ('10.0.249.0','172.19.32.3','192.168.100.83','127.0.0.1','0.0.0.0','255.255.255.255','::1','bogus'):
self.ctl.handle_event(self.alert(ip))
for e in ({'event_type':'alert','alert':None}, {'event_type':'alert','alert':{'signature_id':'bad'}}, {'event_type':'tls'}):
self.ctl.handle_event(e)
self.assertEqual(self.ctl.state.count_active(),0)
self.ctl.handle_event(self.alert(action='block_src'))
self.assertIsNotNone(self.ctl.state.get('203.0.113.11'))
def test_dry_run_cannot_seed_live_blocks(self):
dry = Controller(self.ctl.cfg, dry_run=True)
try:
dry.handle_event(self.alert())
self.assertIsNotNone(dry.state.get('192.0.2.77'))
self.assertIsNone(self.ctl.state.get('192.0.2.77'))
finally: dry.close()
def test_cap(self):
self.ctl.cfg.raw['policy']['max_blocks']=1
self.ctl.handle_event(self.alert())
self.ctl.handle_event(self.alert('192.0.2.78'))
self.assertEqual(self.ctl.state.count_active(),1)
def test_single_instance_and_wal(self):
with self.assertRaises(SystemExit): Controller(self.ctl.cfg,dry_run=False)
self.assertEqual(self.ctl.state.con.execute('pragma journal_mode').fetchone()[0],'wal')
def test_drift_repaired(self):
self.ctl.handle_event(self.alert()); self.ctl.sync_router_state()
self.router.members.clear(); self.ctl.sync_router_state()
self.assertIn('192.0.2.77',self.router.members)
def test_partial_write_rotation_truncation(self):
f=self.root/'eve.json'; f.write_bytes(b'')
tail=EveTailer(str(f),from_start=True); fh=tail._open()
f.write_bytes(b'{"event_type":')
self.assertEqual(tail._read_ready(fh),[])
with f.open('ab') as w: w.write(b'"alert"}\n')
lines=tail._read_ready(fh)
self.assertEqual(tail._parse(lines[0]),{'event_type':'alert'})
f.write_bytes(b'')
self.assertTrue(tail._needs_reopen())
f.rename(self.root/'eve.old'); f.write_text('{}\n')
self.assertTrue(tail._needs_reopen())
tail.close()
def test_rotation_drains_more_than_one_batch(self):
f=self.root/'eve.json'; f.write_text('{}\n'*1100)
tail=EveTailer(str(f),poll_interval=.001,from_start=True)
tail._open()
f.rename(self.root/'eve.old'); f.write_text('{"new":true}\n')
events=tail.events()
received=[next(events) for _ in range(1101)]
self.assertEqual(received[-1],{'new':True})
self.assertEqual(received[:-1],[{}]*1100)
events.close()
def test_checkpoint_resumes_alerts_written_during_shutdown(self):
f=self.root/'eve.json'; f.write_text('')
tail=EveTailer(str(f)); tail._open()
self.ctl.state.save_checkpoint(*tail._ino, tail._pos)
tail.close()
# controller is down; suricata keeps appending
with f.open('ab') as w: w.write(b'{"event_type":"alert","missed":true}\n')
ck=self.ctl.state.get_checkpoint()
resumed=EveTailer(str(f), checkpoint=ck); fh=resumed._open()
self.assertEqual([resumed._parse(l) for l in resumed._read_ready(fh)],
[{'event_type':'alert','missed':True}])
resumed.close()
def test_checkpoint_ignored_on_first_run_never_replays_history(self):
f=self.root/'eve.json'; f.write_text('{"ancient":true}\n')
t=EveTailer(str(f)); fh=t._open()
self.assertEqual(t._read_ready(fh),[])
self.assertIsNone(self.ctl.state.get_checkpoint())
t.close()
def test_checkpoint_across_rotation_reads_new_file_from_start(self):
f=self.root/'eve.json'; f.write_text('{"old":true}\n')
t=EveTailer(str(f)); t._open()
self.ctl.state.save_checkpoint(*t._ino, t._pos); t.close()
f.rename(self.root/'eve.old'); f.write_text('{"fresh":true}\n')
r=EveTailer(str(f), checkpoint=self.ctl.state.get_checkpoint()); fh=r._open()
self.assertEqual([r._parse(l) for l in r._read_ready(fh)],[{'fresh':True}])
r.close()
def test_audit_log_rotation_bounded(self):
from leaf_controller.main import Audit
p=str(self.root/'dec.jsonl')
a=Audit(p, max_bytes=50, keep=2)
for i in range(20): a.write({'i':i,'pad':'x'*80})
a.close()
rotated=list(self.root.glob('dec.jsonl.*'))
self.assertEqual(len(rotated),2) # exactly 'keep' retained
for old in rotated:
self.assertTrue(old.stat().st_size>0)
# every rotated line is a valid decision record
recs=[]
for f in rotated:
with open(f) as fh:
recs += [json.loads(l) for l in fh]
self.assertTrue(all(r['pad']=='x'*80 for r in recs))
def test_new_file_after_startup_is_not_skipped(self):
f=self.root/'missing-eve.json'
tail=EveTailer(str(f))
self.assertIsNone(tail._open())
f.write_text('{"event_type":"alert"}\n')
fh=tail._open()
self.assertEqual(len(tail._read_ready(fh)),1)
tail.close()
def test_idle_reader_waits(self):
f=self.root/'eve.json'; f.touch()
tail=EveTailer(str(f),poll_interval=.05)
events=tail.events(); start=time.monotonic()
self.assertIsNone(next(events))
self.assertGreaterEqual(time.monotonic()-start,.04)
events.close()
def test_packet_headers_and_checksums(self):
import pathlib, sys
from syn_packets import packet, checksum
import socket, struct
frame=packet('203.0.113.12','192.0.2.201',22,0)
ip,tcp=frame[14:34],frame[34:]
self.assertEqual(len(frame),54)
self.assertEqual(struct.unpack('!H',ip[6:8])[0],0)
self.assertEqual(ip[8:10],bytes([64,6]))
self.assertEqual(tcp[13],2)
self.assertEqual(checksum(ip),0)
pseudo=socket.inet_aton('203.0.113.12')+socket.inet_aton('192.0.2.201')+struct.pack('!BBH',0,6,20)
self.assertEqual(checksum(pseudo+tcp),0)
def test_unattached_firewall_hook_rejected(self):
client=VyosClient('http://example.invalid','test')
hook={'action':'drop','source':{'group':{'address-group':'IDS-BLOCK'}}}
fw={'ipv4':{'forward':{'filter':{'rule':{'10':{'action':'drop','destination':{'group':{'address-group':'IDS-BLOCK'}}}}}},'name':{'unused':{'rule':{'9':hook}}}}}
with patch.object(client,'retrieve',return_value={'data':fw}):
with self.assertRaises(VyosApiError): client.verify_enforcement('IDS-BLOCK',True)
fw['zone']={'zone':{'MAOSPF':{'from':{'VEXLAN':{'firewall':{'name':'unused'}}}}}}
with patch.object(client,'retrieve',return_value={'data':fw}): client.verify_enforcement('IDS-BLOCK',True)
def test_show_uses_path(self):
client=VyosClient('http://example.invalid','test')
with patch.object(client,'_post',return_value={'success':True,'data':'counters'}) as post:
self.assertEqual(client.show(['firewall']), 'counters')
post.assert_called_once_with('/show',{'op':'show','path':['firewall']})
def test_ambiguous_api_success_rejected(self):
client=VyosClient('http://example.invalid','test')
response=type('Response',(),{'status_code':200,'json':lambda _: {}})()
with patch.object(client.sess,'post',return_value=response):
with self.assertRaises(VyosApiError): client.configure([])
def test_readback_failure_keeps_pending(self):
self.ctl.handle_event(self.alert())
self.router.group_members=lambda group: []
with self.assertRaises(VyosApiError): self.ctl.sync_router_state()
self.assertFalse(self.ctl.state.get('192.0.2.77').applied)
class MetricsTests(unittest.TestCase):
"""Stats records update metrics without entering the policy path."""
def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.root = Path(self.tmp.name)
raw = copy.deepcopy(DEFAULTS)
raw.update(state_db=str(self.root/'state.sqlite'),
audit_log=str(self.root/'audit.jsonl'))
raw['metrics'].update(capture_interval=0, capture_max_bytes=200,
capture_log_keep=2)
with patch.object(Config, 'vyos_key', return_value='offline-test'):
self.ctl = Controller(Config(raw), dry_run=False)
self.router = self.ctl.client = Router()
def tearDown(self):
self.ctl.close()
self.tmp.cleanup()
def stats(self, packets=1000, dropped=0, engine_alerts=3, uptime=120):
"""Mirrors the live eve.json shape: capture/decoder/detect sub-objects,
no 'ifaces' at totals level (threads: no + totals: yes)."""
return {'event_type': 'stats', 'stats': {
'uptime': uptime, 'memcap_pressure': 0,
'capture': {'kernel_packets': packets, 'kernel_drops': dropped,
'errors': 0},
'decoder': {'pkts': packets, 'bytes': packets * 200, 'invalid': 0},
'detect': {'alert': engine_alerts, 'alerts_suppressed': 0,
'alert_queue_overflow': 0}}}
def thread_stats(self, packets=500):
"""Live thread records (flush=False): per-thread counters, no uptime."""
return {'event_type': 'stats', 'stats': {
'capture': {'kernel_packets': packets, 'kernel_drops': 0,
'errors': 0},
'decoder': {'pkts': packets, 'bytes': packets * 100, 'invalid': 0},
'engine': {'alert': 1, 'alerts_suppressed': 0,
'alert_queue_overflow': 0}}}
def test_stats_never_touch_state_router_or_audit(self):
# Verify that stats cannot produce policy decisions:
# 1) routed stats can only ever be counted
for i in range(20):
self.ctl._route(self.stats(packets=100 * i))
self.ctl._route({'event_type': 'dns', 'src_ip': '192.0.2.77'})
self.ctl._route({'event_type': 'http', 'src_ip': '192.0.2.78'})
# 2) even a stats record handed straight to handle_event cannot act
self.ctl.handle_event(self.stats())
self.assertEqual(self.ctl.state.all_blocks(), [])
self.assertEqual(self.ctl.state.count_active(), 0)
self.assertFalse(self.router.commits)
self.assertFalse(self.router.members)
self.assertEqual((self.root/'audit.jsonl').read_text(), '')
self.assertFalse(list(self.root.glob('capture.log*')))
def test_health_line_once_per_full_window(self):
self.ctl._route(self.stats(packets=1000))
self.ctl._emit_metrics(1000.0) # baseline adoption only
self.ctl._route(self.stats(packets=2000, dropped=1))
with self.assertLogs('leaf.metrics', level='INFO') as logs:
self.ctl._emit_metrics(1030.0) # a full window: one line
self.ctl._route(self.stats(packets=2100))
self.ctl._emit_metrics(1040.0) # partial window: stays silent
health = [l for l in logs.output if 'health:' in l]
self.assertEqual(len(health), 1)
self.assertIn('cps=33.3', health[0]) # 1000 pkts / 30 s window
self.assertIn('dropped+=1', health[0])
def test_counter_restart_never_goes_negative(self):
self.ctl._route(self.stats(packets=5000))
self.ctl._emit_metrics(1000.0)
self.ctl._route(self.stats(packets=100, uptime=30)) # suricata restart
rep = self.ctl.metrics.report(1030.0)
self.assertTrue(rep['restart'])
self.assertEqual(rep['delta']['packets'], 100)
self.assertGreaterEqual(rep['cps'], 0)
def test_totals_authoritative_over_thread_records(self):
# live emits alternating thread(flush=False)+totals; mixing the two
# counter sets would produce bogus rates
self.ctl._route(self.thread_stats(packets=500)) # only kind seen: adopted
self.ctl._emit_metrics(1000.0) # baseline
self.ctl._route(self.stats(packets=2000)) # totals authoritative
self.ctl._route(self.thread_stats(packets=100)) # must not override
rep = self.ctl.metrics.report(1030.0)
self.assertFalse(rep['restart'])
self.assertEqual(rep['delta']['packets'], 1500)
def test_capture_log_bounded_by_size(self):
t = 10000.0
for i in range(1, 40):
self.ctl._route(self.stats(packets=1000 * i, dropped=i))
self.ctl._emit_metrics(t)
t += 40
current = self.root/'capture.log'
rotated = sorted(self.root.glob('capture.log.*'))
# bounded growth is the point: current + at most keep rotated files.
# Audit rotates write-then-check, so the newest line may itself sit in
# the most recent rotated file; current can legitimately be empty.
self.assertLessEqual(len(rotated), 2)
self.assertTrue(current.exists())
recs = []
for p in [current] + rotated:
for l in p.read_text().splitlines():
rec = json.loads(l)
self.assertIn('dropped', rec)
self.assertIn('delta', rec)
recs.append(rec)
assert recs
# Each retained line is a complete window snapshot.
# This check covers the size bound; retention order is tested separately.
self.assertTrue(any(r['dropped'] > 0 for r in recs))
# audit rotation is: append, size-check, rotate, prune-to-keep — so an
# over-cap write can land on disk briefly before the prune settles:
# worst settled state is current (~cap) + keep rotated (~cap each)
disk = sum(x.stat().st_size for x in [current] + rotated)
self.assertLess(disk, 200 * 4 + 400) # (keep+2) x cap + slop
class TLSPinTests(unittest.TestCase):
"""Cert-pin trust against a local TLS server with a throwaway cert."""
@classmethod
def setUpClass(cls):
import shutil, ssl, subprocess
from http.server import BaseHTTPRequestHandler, HTTPServer
if shutil.which('openssl') is None:
raise unittest.SkipTest('openssl required for TLS pin tests')
cls.tmp = tempfile.TemporaryDirectory()
key, crt = cls.tmp.name + '/k.pem', cls.tmp.name + '/c.pem'
subprocess.run(['openssl', 'req', '-x509', '-newkey', 'rsa:2048', '-nodes',
'-keyout', key, '-out', crt, '-days', '1',
'-subj', '/CN=localhost'],
check=True, capture_output=True)
out = subprocess.run(['openssl', 'x509', '-in', crt, '-noout',
'-fingerprint', '-sha256'],
check=True, capture_output=True, text=True).stdout
cls.pin = out.split('=')[1].strip()
class H(BaseHTTPRequestHandler):
def do_GET(self):
body = b'{"success": true, "data": {"proto": "pin-test"}}'
self.send_response(200)
self.send_header('Content-Type', 'application/json')
self.send_header('Content-Length', str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, *a):
pass
cls.srv = HTTPServer(('127.0.0.1', 0), H)
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
ctx.load_cert_chain(crt, key)
cls.srv.socket = ctx.wrap_socket(cls.srv.socket, server_side=True)
cls.port = cls.srv.socket.getsockname()[1]
threading.Thread(target=cls.srv.serve_forever, daemon=True).start()
cls.url = f'https://127.0.0.1:{cls.port}'
@classmethod
def tearDownClass(cls):
cls.srv.shutdown(); cls.srv.server_close(); cls.tmp.cleanup()
def test_pin_normalizes_colon_hex(self):
self.assertEqual(normalize_pin(self.pin), self.pin.lower().replace(':', ''))
self.assertEqual(normalize_pin('sha256:' + self.pin.upper()),
self.pin.lower().replace(':', ''))
with self.assertRaises(ValueError):
normalize_pin('not-a-pin')
def test_pinned_client_connects(self):
c = VyosClient(self.url, 'k', cert_pin=self.pin, timeout=10, retries=1)
self.assertEqual(c.info().get('proto'), 'pin-test')
def test_wrong_pin_aborts_before_request(self):
wrong = 'ab' * 32
c = VyosClient(self.url, 'k', cert_pin=wrong, timeout=10, retries=1)
with self.assertRaises(PinMismatchError):
c.info()
if __name__=='__main__': unittest.main()

View File

@@ -0,0 +1,58 @@
#!/usr/bin/env python3
"""Offline unit tests for the drill injector builders (pure local)."""
import importlib.util
import struct
import unittest
from pathlib import Path
spec = importlib.util.spec_from_file_location(
'drill_synsensor', Path(__file__).resolve().parent.parent
/ 'tests' / 'drill_synsensor.py')
ds = importlib.util.module_from_spec(spec)
spec.loader.exec_module(ds)
def onesc(buf):
if len(buf) % 2:
buf += b'\x00'
t = sum(struct.unpack('!%dH' % (len(buf) // 2), buf))
t = (t & 0xFFFF) + (t >> 16)
t = (t & 0xFFFF) + (t >> 16)
return t
class Builders(unittest.TestCase):
def test_icmp_checksums(self):
p = ds.icmp_packet('192.168.100.84', '192.0.2.44', 7,
b'\x11\x22\x33\x44\x55\x66', b'\x66\x55\x44\x33\x22\x11')
self.assertEqual(p[12:14], b'\x08\x00')
ip = p[14:34]
self.assertEqual(onesc(ip) & 0xFFFF, 0xFFFF) # hdr sum incl cksum
icmp = p[34:]
self.assertEqual(icmp[0], 8)
z = icmp[:2] + b'\x00\x00' + icmp[4:]
t = onesc(z)
self.assertEqual((~t) & 0xFFFF, struct.unpack('!H', icmp[2:4])[0])
def test_syn_pseudo_checksum(self):
p = ds.syn_packet('203.0.113.5', '192.0.2.5', 22, 3,
b'\x00' * 6, b'\x00' * 6)
ip = p[14:34]
self.assertEqual(onesc(ip) & 0xFFFF, 0xFFFF)
tcp = p[34:]
self.assertEqual(tcp[13] & 0x02, 0x02) # SYN flag
a, b = ip[12:16], ip[16:20]
ph = a + b + struct.pack('!BBH', 0, 6, len(tcp))
z = tcp[:16] + b'\x00\x00' + tcp[18:]
t = onesc(ph + z)
self.assertEqual((~t) & 0xFFFF, struct.unpack('!H', tcp[16:18])[0])
def test_testnet_guard(self):
self.assertTrue(ds.in_testnet('192.0.2.5'))
self.assertTrue(ds.in_testnet('203.0.113.12'))
self.assertFalse(ds.in_testnet('172.16.0.1'))
self.assertFalse(ds.in_testnet('192.168.100.84'))
if __name__ == '__main__':
unittest.main()

211
tests/test_repairs.py Normal file
View File

@@ -0,0 +1,211 @@
"""Regression tests for ingestion, replay, persistence and credential handling."""
import copy
import importlib.util
import json
import os
from pathlib import Path
import socket
import sys
import tempfile
import threading
import time
import unittest
from unittest.mock import patch, Mock
sys.path.insert(0, str(Path(__file__).resolve().parents[1]/'controller'))
from leaf_controller.alertsock import AlertSocket, MAX_RECORD
from leaf_controller.config import Config, DEFAULTS, load_config, read_key_env
from leaf_controller.eve import EveTailer
from leaf_controller.main import Audit, Controller
from leaf_controller.vyos_client import VyosClient
from test_controller import Router
ROOT=Path(__file__).resolve().parents[1]
def module(name, path):
spec=importlib.util.spec_from_file_location(name, ROOT/path)
m=importlib.util.module_from_spec(spec);spec.loader.exec_module(m);return m
rotation=module('key_rotation', 'scripts/rotate-vyos-key.py')
retention=module('eve_rotation', 'scripts/rotate-eve.py')
injector=module('injector','tests/drill_synsensor.py')
class Repairs(unittest.TestCase):
def setUp(self):
self.tmp=tempfile.TemporaryDirectory();self.root=Path(self.tmp.name)
def tearDown(self): self.tmp.cleanup()
def controller(self):
raw=copy.deepcopy(DEFAULTS)
raw.update(state_db=str(self.root/'state'),audit_log=str(self.root/'audit'))
raw['alert_socket']['path']=str(self.root/'alerts.sock')
raw['policy'].update(mode='immediate',block_src=True)
with patch.object(Config,'vyos_key',return_value='fake'):
c=Controller(Config(raw))
c.client=Router();self.addCleanup(c.close);return c
def event(self):
return dict(event_type='alert',src_ip='203.0.113.12',dest_ip='192.0.2.202',
alert=dict(signature_id=1000001,metadata=dict(leaf_action=['block'])))
def test_dry_run_preserves_live_socket_and_audit(self):
c=self.controller();c._sock.bind()
before=os.stat(c._sock.path).st_ino
dry=Controller(c.cfg,dry_run=True)
try:
dry.handle_event(self.event())
self.assertIsNone(dry._sock)
finally: dry.close()
self.assertEqual(os.stat(c._sock.path).st_ino,before)
self.assertEqual((self.root/'audit').read_text(),'')
def test_distinct_state_cannot_steal_socket(self):
a=AlertSocket(str(self.root/'sock'));a.bind();self.addCleanup(a.close)
b=AlertSocket(a.path)
with self.assertRaises(OSError):b.bind()
self.assertTrue(Path(a.path).exists())
def test_unmanaged_active_listener_preserved(self):
path=str(self.root/'sock')
with socket.socket(socket.AF_UNIX) as s:
s.bind(path);s.listen()
with self.assertRaises(OSError):AlertSocket(path).bind()
self.assertTrue(Path(path).exists())
def test_non_socket_path_preserved(self):
p=self.root/'sock';p.write_text('preserve')
with self.assertRaises(OSError):AlertSocket(str(p)).bind()
self.assertEqual(p.read_text(),'preserve')
def test_shutdown_persists_accepted_queue(self):
c=self.controller();c._sock._enqueue(json.dumps(self.event()).encode())
c.close()
from leaf_controller.state import State
state=State(str(self.root/'state'))
try:self.assertIsNotNone(state.get('192.0.2.202'))
finally:state.close()
def test_oversized_and_recursive_frames_do_not_kill_reader(self):
a=AlertSocket(str(self.root/'sock'));a.bind();a.start();self.addCleanup(a.close)
with socket.socket(socket.AF_UNIX) as s:
s.connect(a.path)
s.sendall(b'x'*(MAX_RECORD+65536)+b'\n'+b'['*2000+b']'*2000+b'\n'+
json.dumps(self.event()).encode()+b'\n')
self.assertTrue(a.available.wait(3))
self.assertGreaterEqual(a.counts['parse_errors'],1)
self.assertEqual(a.counts['parse_errors']+a.counts['other'],2)
self.assertEqual(len(a.drain()),1)
self.assertTrue(a._thread.is_alive())
def test_partial_eof_counted(self):
a=AlertSocket(str(self.root/'sock'));a.bind();a.start();self.addCleanup(a.close)
with socket.socket(socket.AF_UNIX) as s:s.connect(a.path);s.sendall(b'{')
end=time.monotonic()+2
while a.counts['parse_errors']==0 and time.monotonic()<end:time.sleep(.01)
self.assertEqual(a.counts['parse_errors'],1)
def test_checkpoint_does_not_skip_unread_batch(self):
p=self.root/'eve';p.write_text('{"n":1}\n{"n":2}\n{"n":3}\n')
t=EveTailer(str(p),from_start=True);g=t.events()
self.assertEqual(next(g),{'n':1});ck=(*t._ino,t._pos);g.close()
self.assertEqual(ck[2],len('{"n":1}\n'))
t=EveTailer(str(p),checkpoint=ck);g=t.events()
self.assertEqual(next(g),{'n':2});g.close()
def test_replayed_file_hit_does_not_inflate_threshold(self):
c=self.controller();c.cfg.raw['policy'].update(mode='threshold',min_hits=2)
c.state._local.event_key='1:2:10'
c.handle_event(self.event());c.handle_event(self.event())
self.assertIsNone(c.state.get('192.0.2.202'))
c.state._local.event_key='1:2:20';c.handle_event(self.event())
self.assertIsNotNone(c.state.get('192.0.2.202'))
def test_failed_save_retried_after_membership_already_matches(self):
c=self.controller();c.cfg.raw['save_config']=True;c.handle_event(self.event())
c.client.save_config=Mock(side_effect=[RuntimeError('save failed'),None])
with self.assertRaises(RuntimeError):c.sync_router_state()
self.assertTrue(c.state.router_dirty())
c.sync_router_state()
self.assertEqual(c.client.save_config.call_count,2)
self.assertEqual(len(c.client.commits),1)
self.assertFalse(c.state.router_dirty())
def test_audit_keeps_latest_in_rapid_rotation(self):
p=self.root/'log';a=Audit(str(p),max_bytes=1,keep=3)
for i in range(20):a.write({'i':i})
a.close()
rows=[json.loads(f.read_text())['i'] for f in self.root.glob('log.*')]
self.assertEqual(sorted(rows),[17,18,19])
def test_tls_pins_do_not_mutate_global_or_other_clients(self):
from urllib3.poolmanager import pool_classes_by_scheme
before=dict(pool_classes_by_scheme)
a=VyosClient('https://localhost','x',cert_pin='a'*64)
b=VyosClient('https://localhost','x',cert_pin='b'*64)
self.assertEqual(pool_classes_by_scheme,before)
self.assertFalse(a.sess.trust_env)
self.assertEqual(a.sess.adapters['https://'].poolmanager.pool_classes_by_scheme['https'].ConnectionCls.leaf_pin,'a'*64)
self.assertEqual(b.sess.adapters['https://'].poolmanager.pool_classes_by_scheme['https'].ConnectionCls.leaf_pin,'b'*64)
def test_rotation_reopens_and_keeps_newest(self):
p=self.root/'eve';p.write_text('first\n');calls=[]
for i in range(9):
p.write_text(str(i))
retention.rotate(p,max_bytes=1,keep=3,reopen=lambda:(calls.append(1),p.write_text('')))
self.assertEqual(len(calls),9)
self.assertEqual(sorted(int(f.read_text()) for f in self.root.glob('eve.leaf-*')),[6,7,8])
def test_rotation_reopen_failure_restores_file(self):
p=self.root/'eve';p.write_text('original')
with self.assertRaises(RuntimeError):
retention.rotate(p,max_bytes=1,reopen=Mock(side_effect=RuntimeError()))
self.assertEqual(p.read_text(),'original')
def test_env_quotes_are_parsed_without_evaluation(self):
p=self.root/'env';p.write_text("LEAF_VYOS_KEY='abc123'\n")
self.assertEqual(read_key_env(p),'abc123')
def test_invalid_queue_and_ttl_rejected(self):
p=self.root/'config'
for value in ('alert_socket: {queue_size: 0}', 'policy: {ttl_seconds: -1}', 'policy: {min_hits: 1.5}'):
p.write_text(value)
with self.assertRaises(ValueError):load_config(p)
def test_key_rotation_preserves_other_app_ids(self):
old=Mock();new=Mock();old.retrieve.return_value={'data':{'id':{'leaf':{'key':'old'},'other':{'key':'unrelated'}}}}
with patch.object(rotation,'wait_auth'),patch.object(rotation,'atomic_env'),patch.object(rotation,'restart_service'):
rotation.rotate(old,new,'leaf','leaf-new','old','new','LEAF_VYOS_KEY=old\n','IDS-BLOCK')
self.assertEqual(new.configure.call_args.args[0]['path'],rotation.KEYS_PATH+['leaf'])
self.assertEqual(new.configure.call_count,1)
def test_key_rotation_restart_failure_rolls_back(self):
old=Mock();new=Mock();old.retrieve.return_value={'data':{'id':{'leaf':{'key':'old'}}}}
with patch.object(rotation,'wait_auth'),patch.object(rotation,'atomic_env') as write,patch.object(rotation,'restart_service',side_effect=[RuntimeError('failed'),None]):
with self.assertRaises(RuntimeError):rotation.rotate(old,new,'leaf','new-id','old','new','LEAF_VYOS_KEY=old\n','IDS-BLOCK')
self.assertEqual(write.call_args.args[0],'LEAF_VYOS_KEY=old\n')
self.assertEqual(old.configure.call_args.args[0]['path'],rotation.KEYS_PATH+['new-id'])
new.configure.assert_not_called()
def test_key_rotation_refuses_nonmatching_id(self):
old=Mock();old.retrieve.return_value={'data':{'id':{'other':{'key':'different'}}}}
with self.assertRaises(RuntimeError):rotation.rotate(old,Mock(),'other','new','old','new','text','group')
old.configure.assert_not_called()
def test_live_injector_uses_resolved_ethernet_addresses(self):
fake=Mock();fake.__enter__=Mock(return_value=fake);fake.__exit__=Mock(return_value=False)
with patch.object(sys,'argv',['injector','syn','--src','203.0.113.12','--dst','192.0.2.201','--count','1','--interval','0']),patch.object(injector,'route_via',return_value=('10.0.249.0','eth1')),patch.object(Path,'read_text',return_value='11:22:33:44:55:66'),patch.object(injector,'resolve_gateway_mac',return_value=bytes.fromhex('665544332211')),patch.object(injector.socket,'socket',return_value=fake):
injector.main()
self.assertEqual(fake.send.call_args.args[0][:12],bytes.fromhex('665544332211112233445566'))
def test_hourly_rotation_clock_survives_continuous_writes(self):
p=self.root/'eve';p.write_text('one')
self.assertFalse(retention.rotate(p,max_bytes=999999,now=100,reopen=lambda:None))
p.write_text('two')
self.assertTrue(retention.rotate(p,max_bytes=999999,now=3701,reopen=lambda:p.write_text('')))
def test_rotation_skips_engine_startup(self):
with patch.object(sys,'argv',['rotate-eve']),patch.object(retention.subprocess,'run',return_value=Mock(stdout='100000000')),patch.object(retention.time,'monotonic',return_value=101),patch.object(retention,'rotate') as r:
retention.main()
r.assert_not_called()
def test_diagnostics_redact_nested_credentials(self):
from leaf_controller.ops import redact
result=redact({'service':{'api':{'keys':{'id':{'leaf':{'key':'secret'}}}}},
'password':'secret', 'interfaces':{'eth1':{'address':['10.0.249.1/31']}}})
self.assertNotIn('secret',json.dumps(result))
self.assertEqual(result['interfaces']['eth1']['address'],['10.0.249.1/31'])
def test_file_identity_can_count_both_candidate_targets(self):
c=self.controller();c.cfg.raw['policy'].update(mode='threshold',min_hits=2)
c.state._local.event_key='one-file-record'
event=self.event();event['alert']['metadata']['leaf_action']=['block','block_src']
c.handle_event(event)
self.assertEqual(c.state.hit_count(1000001,'192.0.2.202',300),1)
self.assertEqual(c.state.hit_count(1000001,'203.0.113.12',300),1)
def test_malformed_stats_does_not_poison_following_alerts(self):
c=self.controller()
for stats in ({'capture':[1]}, {'uptime':'bad'}, {'uptime':float('inf')}):
c._route({'event_type':'stats','stats':stats})
c._route(self.event())
self.assertIsNotNone(c.state.get('192.0.2.202'))
def test_nonfinite_signature_does_not_stop_processing(self):
c=self.controller();bad=self.event();bad['alert']['signature_id']=float('inf')
c.handle_event(bad);c.handle_event(self.event())
self.assertIsNotNone(c.state.get('192.0.2.202'))

View File

@@ -0,0 +1,15 @@
#### Illustrative new-install template only. NOT the live VIPERA configuration.
#### Do not apply this file to VIPERA. Existing OSPF, uplink, mirror and routes
#### are outside LEAF maintenance scope. Verify live hooks through the client.
#### Router OS variants differ: inspect the target schema before configuring.
# One dedicated group, and source/destination drop hooks ahead of accept rules.
set firewall group address-group IDS-BLOCK
set firewall ipv4 forward filter rule 10 action drop
set firewall ipv4 forward filter rule 10 destination group address-group IDS-BLOCK
set firewall ipv4 forward filter rule 11 action drop
set firewall ipv4 forward filter rule 11 source group address-group IDS-BLOCK
# API service, TLS trust and mirroring are provisioned by the administrator.
# No embedded key, interface changes, broad accept rules or commit/save here.
# Controller save_config=false keeps transient membership out of boot config.