LLM Guard on Kubernetes — Governing the Content Layer on OpenShift

The Layer the Cluster Doesn't Cover

The cluster governs what AI processes can reach. That's what the last post covered — namespace isolation, Sealed Secrets for credential handling, SCCs at the API level, a host OS that can't drift between deployments. Those controls answer one question: what can an AI workload access inside this environment?

LLM Guard answers a different question: what passes through the AI workload itself?

What Declared Boundaries Cover — and Where They Reach Their Limit

CLAUDE.md handles a real class of problems. File access restrictions, tool permissions, behavioral constraints — rules that scope what the agent can and can't do within a project context. On simpler workflows where all inputs are controlled and trusted, that coverage holds.

The limit shows up when external data enters the prompt from outside that context. Alertmanager labels and annotations can carry credential-adjacent content if upstream systems aren't clean. API responses, webhook payloads, log data — none of it was part of the repo context when the CLAUDE.md was written, and no declared rule covers data that arrives from a system the rule author didn't anticipate.

CLAUDE.md tells the agent what to do. It doesn't intercept what arrives.

What the Proxy Layer Adds

LLM Guard deploys as a proxy — sitting between the agent and the API endpoint, scanning content in both directions. Callers are directed to it explicitly (ANTHROPIC_BASE_URL for Anthropic API calls, OLLAMA_URL for local Ollama inference); it forwards to the upstream after scanning input, buffers the full response for output scanning, then returns it. Two instances cover two surfaces: Anthropic's API for Claude Code, and local Ollama inference for the alert handler and Jarvis scripts.

Input scanners run before the prompt reaches the model. The Secrets scanner (bc-detect-secrets, regex) catches API keys and tokens in incoming data before they reach the model's context. The PromptInjection scanner (protectai/deberta-v3-base-prompt-injection-v2) detects injection attempts in input content — external data trying to redirect the model's behavior before it even reaches the system prompt.

Output scanners run after the model responds. The Regex scanner redacts token=, api_key=, secret=, and password= patterns from LLM responses before they're dispatched or logged — catching leakage even when the model surfaces something from its context it wasn't supposed to repeat.

Making the API Call Visible

Deploying LLM Guard as a proxy changed how I think about agentic workflows.

Data was leaving the environment — I knew that. Every API call sends something and receives something. What the proxy forced was a more precise mental model of what that something is at each step: what content enters the prompt, what the model produces, and where in the pipeline content decisions are actually being made.

The proxy makes the flow governable because it first makes it visible. I hadn't fully mapped what was entering and leaving each step of the pipeline until LLM Guard was sitting in that path. Instrumenting it produced something close to a complete content-flow diagram of the agentic pipeline — including paths I'd assumed were safe but hadn't explicitly traced.

Secrets Governance — From Per-Repo to Cluster-Level

Running multiple self-hosted projects means managing env variables across all of them — separate files, separate rotation cycles, no unified view of what exists or when it was last changed. It's manageable until it isn't.

The cluster changes that model entirely. With Sealed Secrets handling credential management at the cluster level, secrets are governed programmatically through GitOps rather than maintained per project. A secret sealed against the cluster's public key, committed to version control, and decrypted at runtime by the cluster itself is a secret with an audit trail, a known state, and a single governance point — regardless of how many workloads depend on it.

LLM Guard adds the content layer on top of that. Sealed Secrets governs where credentials live. LLM Guard governs whether they move — catching them in transit if they surface in a prompt or a response.

What the Networking Friction Confirmed

Deploying LLM Guard on OKD surfaced a networking difference that K3s didn't have. The ClusterIP service DNS wasn't reachable from the host-side toolbox — internal service discovery works within the cluster, but the host sits outside that boundary. Reaching LLM Guard from the toolbox required using the Ingress Route, not the internal service address.

That's OKD's isolation model working as designed. The network boundary between the cluster and the host is stricter than what K3s enforced by default. On K3s, the boundary was something you configured. On OKD, it's the starting position.

What the False Positives Proved

The first sign LLM Guard was working was that it blocked things in real time — including benign inputs.

The initial deployment used deberta v1. That model flagged benign inputs as injection attempts — a "say hi" test input scoring 0.99 injection confidence. That's a false positive problem, but it's also confirmation the scanner was active and inferring, not just constructed. A scanner that initializes cleanly but fails at inference looks identical to a working scanner until a real input arrives.

Switching to protectai/deberta-v3-base-prompt-injection-v2 resolved the false positive rate without affecting detection of actual injection attempts. The deployment now includes a startup self-test scan so readiness reflects actual inference capability — not just a successful container init. That detail only matters when you're watching what the scanner does rather than assuming it works because the pod is running.

The Governance Architecture as Layers

The SSP for this cluster maps these controls to NIST AI RMF:

GOVERN 1.2 — CLAUDE.md and the Claude Code permissions model define the behavioral policy: what the agent can do, what it can't, what requires escalation. This is the declared layer.

MEASURE 2.7 — LLM Guard input scanning and code-level gate enforcement address security and resilience: secrets scanner on all incoming content before it reaches the model, PromptInjection detection on external data, output redaction before responses are dispatched. The gates don't ask whether the model complied with the declared policy. They run regardless.

The code-level enforcement operates at the shell layer — Claude Code PreToolUse hooks block kubectl get secret and oc exec -- env/printenv before those commands run; a PostToolUse hook scans file read output for credential patterns before values enter the context window. These gates intercept what the agent does. LLM Guard intercepts what the agent processes — external content arriving through alert payloads, API responses, and webhook data that the hook layer was never in a position to intercept.

LLM Guard doesn't depend on the model respecting the CLAUDE.md instructions. It scans what actually passed through. CLAUDE.md is the declared policy. LLM Guard is the enforcement that operates whether or not the model followed it.