AI Governance at the Infrastructure Layer on OpenShift

Why Kubernetes Was the Right Answer to a Governance Problem

FEDLIN runs its own infrastructure. Client delivery work, AI-assisted operations, security research, and deployment pipelines — all of it on a self-hosted production cluster. It didn't start that way.

It was started on local tools and SaaS. No self-hosted infrastructure, no orchestration. As the deliverable complexity grew — client work, security research, infrastructure code — the case for AI-assisted operations was clear enough to build toward proactively. The governance gap appeared before the tooling did.

The first boundary was web UIs. Claude.ai, ChatGPT — tools that ran in a browser and never touched the local file system. That held until Cursor entered the picture. Useful enough to actually scale delivery capacity, but it introduced a different kind of exposure: Cursor running locally, SSH'd into the git server for project work, operating through a personal session with no isolation between client work, security research, and infrastructure code. The boundary was a login, not an architecture.

Claude Code changed the geometry of the problem. Installed directly on the server to explore governance strategies, it was an AI agent running in the same environment as the cluster infrastructure — git repos, cluster configuration, credential-adjacent files all within reach. No audit trail of what it touched, no credential isolation, no enforced boundary between work types. Claude Code brought some structure through CLAUDE.md — project-level instructions defining what the agent can and can't do — and a permissions model scoping which tools and commands it can run. Those set the declared boundary. They don't enforce it.

That's a finding I'd surface in a client environment. Kubernetes was the right answer to it. Credential isolation, audit logging, network policy, namespace-level separation between work types — the architecture exists to enforce exactly this kind of boundary. Running AI workloads in a governed environment meant running them in containers with declared resource limits, inside a cluster that could audit what they accessed and revoke what they shouldn't have. The question became: which distribution, on what hardware, and starting with what?

K3s Was the Path. OKD Was Always the Destination.

The goal was OpenShift-grade architecture. An earlier attempt at getting there didn't complete. K3s was the decision that followed: production-ready, lighter than OKD, and capable of carrying the full GitOps workflow while the operational model developed. That was the plan.

It wasn't run minimally. The K3s cluster distributed workloads across three nodes by function — a dedicated control plane, a GPU worker handling AI inference, and a storage and security operations node carrying git, monitoring, and host-level tooling. The topology was intentional from the start.

The security stack followed the same logic. Sealed Secrets for credential handling — secrets encrypted with the cluster's public key and committed to version control, decryptable only by the cluster itself. ArgoCD for deployment governance — the cluster's running state reconciles against declared state in the repository, and nothing is applied outside that pipeline. Wazuh for host-level monitoring — agent-side, watching syscalls and process behavior on the node itself, not only what's happening inside containers. Container observability doesn't cover the host. Wazuh does. Headscale for partner access — VPN-gated, least-privilege scoped by role, so the cluster governs who reaches it before any question of what they can do inside it. LLM Guard for AI workload scanning — inputs to AI processes screened for secrets before they reach the model, outputs scanned for sensitive patterns before they're dispatched. The cluster governs what AI processes can do; LLM Guard governs what passes through them.

The pipeline ran security scans on every push — credential detection, container vulnerability scanning, static analysis. Security gates before code ships.

K3s Did Its Job. OpenShift Was the Next Step.

After running K3s through real delivery cycles — secrets management, ArgoCD reconciliation, CI gates on every push — the architecture was well understood. Not in theory: in practice, through the places it held and the places it didn't. That operational familiarity made the next decision straightforward. The cluster had done what it was built to do. It was time to graduate it.

The original target was always OpenShift-grade infrastructure. Red Hat OpenShift would have been ideal — enterprise-standard, the kind of platform that shows up in regulated industries and security-conscious client environments. But open source was a hard constraint from the start. Fedora CoreOS was the initial target — the natural path toward OpenShift-grade infrastructure without the licensing dependency. That attempt didn't complete. The tooling and depth to pull it off weren't there yet, so the decision was to stay on Fedora Server and get operational on K3s while that understanding developed.

OKD was the practical path to that destination: enterprise-grade controls and the same architecture, on terms that fit. OpenShift is the platform the field actually uses, and building toward it wasn't only a security posture decision — it was about developing real operational depth on the same stack a client engagement might require. The viable route to the same place.

K3s doesn't enforce Security Context Constraints natively. Pod Security Standards served as the compensating control — Kubernetes-native admission enforcement at the namespace level, restricting privilege escalation within declared profiles. It worked. But it was a compensating control: something wired in to approximate a guarantee the platform doesn't provide on its own. OKD enforces SCCs at the API level as a gate on every pod that runs — native, fine-grained, and scoped per project. A workload that requests elevated privileges in K3s can get them if the policy isn't explicitly set. In OKD, the default posture is restrictive and any deviation has to be explicitly granted. The audit logging pipeline is built into the control plane rather than wired in separately. For a cluster governing agentic AI workloads — processes that make external API calls, read files, handle credentials — the difference between a compensating control and a native enforcement boundary is the difference between a security posture and a claim about one.

The Migration Had Real Friction. It Was Worth It.

The migration had real friction. Boot issues, DNS resolution problems that took time to trace. The networking model required real attention — the CNI change alone touches how traffic routes between pods, and working through that isn't optional. Understanding what changed at the network layer is part of understanding what the new architecture actually does.

What came out the other side was worth the work — and not only because of what OKD enforces at the workload layer. SCOS changes what the node itself is. On K3s running Fedora Server 43, the host OS was a general-purpose Linux installation: mutable, patchable, hand-maintained. That's why Wazuh watched it — agent-side, monitoring syscalls and process behavior on the node itself, because a host that can drift needs something watching it. The security controls applied on top of it were real, but the surface they sat on could change. SCOS removes that dependency by design — the root filesystem is read-only, there's no package manager at runtime, and all node configuration runs through the MachineConfigOperator as declared state. The same GitOps discipline that governs workloads now governs the OS. That closes a category of risk that workload-layer controls never reach: what happens to the host between changes.

SCCs at the workload layer — already covered. SELinux enforcing by default at the process layer. OVN-Kubernetes at the network layer, governing traffic between pods at the switch rather than through a CNI you selected and configured on top of the cluster. In regulated environments and engagements, that enforcement boundary carries compliance weight — what AI processes can and can't reach is a data boundary question, and NetworkPolicy is how the architecture answers it rather than relying on policy alone. Audit logging built into the control plane, not wired in separately. Each of these was either a compensating control or a gap on K3s. On OKD, they're properties of the platform. The security posture isn't something assembled on top of the cluster — it's what the cluster is.

The cluster is where this particular story lives, but the principles travel. GitOps-driven infrastructure, declared state, governance built into the toolchain from the start — that pattern shows up in cloud infrastructure, landing zone design, how Terraform fits into a deployment model you can actually audit. The cluster is the clearest example of that pattern in practice — and it's not the only one. There's a local inference model running on a worker node right now, triaging security findings against a live security plan in real time. That's a post on its own.

When the Architecture You Recommend Is the Architecture You Operate

Operating your own infrastructure changes what a recommendation means. When you advise, you can recommend a control. When you operate it, you live with the boot failures, the DNS debugging, the decision of whether the friction was worth it — not in theory, but because you're the one maintaining the thing.

The architecture governing FEDLIN's AI workloads is the architecture client work runs on. The infrastructure came first — the recommendations follow from operating it, not the other way around. Agentic governance is an emerging discipline. The cluster is a working answer to it.