About
Most "AI SOC" demos wire a chatbot summary onto the end of an existing pipeline and call it augmentation. PFA-SOC-IA does something narrower and more honest: it puts a local, open-weight LLM (Gemma2 9B, quantized, served by Ollama, zero cloud dependency) directly into the triage path of a real seven-stage SOC pipeline — Wazuh detects, Shuffle orchestrates, Gemma2 classifies, TheHive opens the case, Cortex enriches the indicator, MISP shares the intelligence, and a real Gmail notification closes the loop.
The whole stack runs on a single 8 vCPU / 10 GB RAM lab VM, which turned out to matter more than expected: RAM pressure across Wazuh, Shuffle, TheHive, Cortex, MISP, and Ollama running side by side produced a genuine TheHive 500 error during validation (JVM thread starvation while Ollama held the CPU) — a real failure condition that the pipeline's failure-guard nodes caught correctly rather than something staged for a demo.
Six real attack scenarios — SSH brute force, a suspicious external download, encoded PowerShell execution, SSH-plus-sudo lateral movement, C2 beaconing, and network probing — were executed live on the lab VM and indexed by Wazuh, each with its own custom or native rule ID and MITRE technique.
Why triage, not decisions
The project's central design decision is what the LLM is not allowed to do. Gemma2's output is non-deterministic and can hallucinate, so nothing security-relevant is allowed to depend on it. The branch between "create a shareable MISP event" and "just tag it quietly" reads Wazuh's own rule.level — present in the raw alert before Gemma2 ever runs — never the model's self-reported criticality field. The LLM's job is narrower and still valuable: turning a bare rule ID into a structured incident type, MITRE ATT&CK tactic and technique, a human-readable summary, and a recommendation, so the analyst opens a case that already reads like triage notes instead of a log line.
That narrow scope is also what makes the evaluation numbers meaningful instead of marketing: the model is being measured on the one job it was actually given.
Architecture
| Stage | Component | Responsibility |
|---|---|---|
| 1 | Wazuh | Continuous monitoring via auditd; fires native and custom correlation rules (e.g. rule 100103 for C2 beaconing) and produces the alert that starts the chain |
| 2 | Shuffle (SOAR) | Receives the alert over a webhook trigger and drives all 13 nodes of the workflow — 6 business nodes plus 6 dedicated failure-guard nodes — with no manual step |
| 3 | Gemma2 9B (local LLM) | Advisory triage only: returns strict JSON — incident type, MITRE tactic/technique, summary, recommendation — never a security decision |
| 4 | TheHive | A service account automatically creates a structured investigation case with Gemma2's raw triage embedded in the description |
| 5 | Cortex | Submits the extracted IOC to the AbuseIPDB analyzer for automated reputation enrichment |
| 6 | MISP | Creates a shareable threat-intelligence event on the high-severity branch, or applies a quiet tag on the low-severity branch — routed by rule.level, not by the LLM |
| 7 | Notification | A local receiver relays the result over real Gmail SMTP to the analyst's inbox — closing the loop, full cycle, no human action required to get there |
Every edge between stages carries an explicit HTTP-status guard: below 300 continues to the next business node, 300 and above branches to a paired failure-guard node. In a healthy run all six guards stay SKIPPED — the correct state, not dead code — and during this project's own validation they proved themselves against real failures rather than synthetic ones.
Evaluation — Gemma2 vs. the SIEM baseline
Wazuh's own MITRE mapping (rule.mitre) covers none of the six custom correlation rules used in this project — they're project-specific and carry no built-in ATT&CK metadata, so without AI triage none of these six alerts would get a technique automatically. To measure whether Gemma2's triage actually closes that gap reliably, the project ran a dedicated evaluation script against a 25-alert, deduplicated holdout set — a real, freshly executed run with live Ollama calls, not cached or reused results.
The 100%/40% figure formally replaces an earlier 94.4% number that had been measured on a contaminated, duplicate-containing dataset and never recalculated after the bug was found — worth stating plainly, since a number quietly superseded without a correction is worse than no number at all. The current figure is deduplicated and freshly run.
Case walkthrough — a real C2 beaconing detection, start to finish
The clearest way to show the pipeline is one alert followed all the way through, with the original detection timestamp preserved end-to-end so each downstream artifact can be cross-checked against it. This is Shuffle execution 63e59cbe-9d4a-4c67-b1f9-8aae54dd3609: status FINISHED, 12/12 node results received, all business nodes SUCCESS, all six failure guards correctly SKIPPED.
Case 1 — Wazuh detects a real C2 beaconing pattern
A genuine curl beaconing pattern to an isolated lab-only destination fires custom rule 100103 ("repeated network fetch commands to the same destination in a short window") at level 10 — MITRE T1071, Command and Control. This rule itself had a real correlation bug during development (matching the wrong auditd field, a1 instead of a3), found and fixed, then re-verified with a live positive test (fires on the third repeated request) and a live negative test (three different destinations, no false positive).
Case 2 — Local LLM triage lands inside the case, not beside it
Gemma2 runs locally (roughly two minutes on the lab's shared CPU) and returns structured JSON that TheHive's service account embeds directly into the case description — the analyst opens the case already reading a proposed incident type, MITRE mapping, and plain-language summary, rather than a raw rule ID they have to look up themselves. The case is created by a service account, never a human, proving the automation runs the full chain unattended.
Case 3 — Severity, not the model, decides whether MISP gets a shareable event
Because Wazuh's own rule.level: 10 puts this alert on the high-severity branch, MISP automatically creates a shareable threat-intelligence event — the low-severity "tag only" branch correctly stays SKIPPED for this run. Swap in a low-level alert and the same workflow quietly tags instead of broadcasting, without Gemma2's opinion ever entering the routing logic.
Case 4 — Failure guards proven against a real outage, not a simulated one
Every business node's outgoing edge carries an explicit HTTP-status condition, branching to a paired failure-guard node on anything ≥300. During final validation, RAM pressure from running the whole stack on one 10 GB VM produced a genuine TheHive 500 (JVM thread starvation while Ollama held the CPU), and the http_case_creation_failed guard caught it correctly rather than the pipeline silently dropping the alert. A separate Shuffle templating race that briefly returned an empty case description was reproduced and confirmed transient, not a payload-encoding bug.
Case 5 — The final delivery is a real inbox, not a mocked webhook sink
The last node relays the pipeline's result through a local notification receiver that forwards over real Gmail SMTP. The subject line references the exact run ID, and the body carries the same Gemma2 triage JSON visible in the TheHive case — the same alert, traceable end-to-end through six independent tools to one inbox.
Security, privacy, and isolation
- All offensive scenarios run against
.invaliddomains,localhost, and the lab's own private subnet exclusively — nothing in this project ever touches a real external target. - The LLM never receives or handles credentials; its input is limited to an alert's technical fields (rule description, log line, agent name).
- No secret is committed — credentials live in a git-ignored file outside the tracked tree, and every commit is swept for API keys and tokens before staging, with automated secret scanning (Gitleaks) on every push.
- A real credential-leak incident happened and was fixed by the project itself: an early Wazuh password was accidentally captured in plaintext by
auditd(which logs full CLI arguments, includingcurl -u user:pass). The affected index documents were purged, the password rotated, and authentication moved to~/.netrcso a credential is never passed as a CLI argument again. - Every screenshot and JSON artifact cited as evidence is SHA-256 hashed against a manifest — a capture is never used as proof without a machine-readable artifact behind it.
Local inference took roughly two minutes per alert on the lab's shared 8 vCPU host — an infrastructure constraint, not an architectural one. The project treats that honestly rather than hiding it: on dedicated GPU infrastructure, the same model returns a triage in a few seconds with no code change required.
Lessons learned
- Advisory AI is more defensible than autonomous AI. Keeping severity routing strictly on
rule.levelmeans a hallucinated triage field can produce an embarrassing case note, never a wrong security action — the failure mode that actually matters stays bounded. - Guard coverage has to include timeouts, not just status codes. An early version of the pipeline caught HTTP error codes but missed application-level timeouts; the fix was a dedicated guard node per business step covering both.
- Shared infrastructure produces the failures worth documenting. Running Wazuh, Shuffle, TheHive, Cortex, MISP, and Ollama on one 10 GB VM created a genuine RAM-pressure outage that a cleanly resourced deployment never would have surfaced — and proved the guards actually work.
- A superseded number should say so. Replacing the earlier, contaminated 94.4% evaluation figure with a corrected, re-run 100%/40% result — and stating clearly why the old number no longer applies — matters more for credibility than the number itself.