Agenda

Everyone is building SRE agents. Most of them are barely junior sysadmins in a trench coat — useful for "how does X work?", useless when PostgreSQL replication is lagging at 3 AM across three datacenters at your specific company with your specific Puppet module.

We started out obsessing over the tech stack, but after months in production, we learned a few hard truths: The agent framework? It doesn't really matter that much. The LLM model or vendor? Doesn't matter nearly as much as you think. Providing misleading docs? A massive problem. Fear of autonomy? Like handcuffing your agent and getting only 10% out of it. Not unit-testing the context? Likely you'll degrade in the future versions and lose people's trust.

We'll walk through what surprised us, what the agent still gets wrong, and why security is the hardest third nobody talks about. No vendor pitch, just a story from the trenches so you can get "more realistic" about this.

Isolation was a solved problem in 2000, just not in the way the industry remembers it.

FreeBSD Jails were not stronger than everything that came after. They were coherent: one kernel-level boundary, designed as a boundary, and auditable as a single thing. Solaris Zones refined the same idea a few years later.

Then Linux containers took a different path.

Docker did not win because it was the purest isolation model. It won because it made containers usable. It gave developers a clean interface, a simple workflow, a good mental model, and a way to package and run software without caring too much about the machinery underneath. That mattered. Good developer experience beat a cleaner theory of isolation, and for the workloads of the time, that was mostly the right trade.

A service was usually predictable. It had a known filesystem shape, a known network posture, a mostly stable syscall surface, and most of its behavior was decided before runtime. You could build an image, ship it, constrain it, observe it, and move on.

Agents break that assumption.

An agent can spawn subprocesses at runtime. It can call endpoints nobody hardcoded. It can write to paths created from context gathered mid-run. It can use credentials, tools, plugins, prompts, generated code, and remote APIs as part of one execution flow. The behavior that matters is no longer fully visible when the container starts.

Agents also do not carry the operational judgment humans quietly rely on. They do not naturally understand ownership, blast radius, production etiquette, or the difference between “technically allowed” and “obviously a bad idea.” They can follow instructions and use tools, but they do not share the human context that usually stops an operator before the damage happens.

That is why the old isolation question comes back.

Not because Jails or Zones should simply return as they were. And not because Docker was wrong. Docker solved usability for a generation of software delivery. But agentic workflows expose a different problem: how do you isolate something whose shape is discovered while it runs?

This talk traces the line from Jails and Zones to Docker and Linux containers, and then asks what should be revived for the agentic era. The answer is not nostalgia. It is old ideas rebuilt with new mechanisms: coherent boundaries, runtime policy, and isolation-first design for workloads that no longer know their own edges in advance.

eBPF LSM is one possible way Linux can get there. It matters because it lets policy live close to the kernel boundary while still being programmable at runtime. But the larger point is simpler: the next runtime should not only be easy to use. It should be built for the strange, dynamic, semi-informed workloads we are now creating.

Telemetry attributes have always been simple: strings, numbers, booleans. However, when the data you actually want to capture isn't simple (like a structured LLM tool call, a nested agent context, a batch database operation), you flatten it, or JSON-encode it into a string and hope your backend can deal with it later.

Agentic workflows made this tension impossible to ignore. That workaround era is ending.

OpenTelemetry now supports complex attribute types on spans — nested maps, arrays of objects, byte arrays, structured values — starting with the wire protocol, with API and SDK support rolling out across languages. Spans join logs in being able to carry data that reflects reality, not a flattened approximation of it.

This matters most right now for agentic AI workloads, where the data is inherently structured: multi-turn message histories, tool definitions, per-step agent contexts. Forcing this into flat attributes was never a good fit - it was just the only option.

But expressive power comes with new responsibilities. This session covers:

Why flat attributes exist and the real tradeoffs they encode

What complex attribute support unlocks at the spec and SDK level.

Applying limits in a complex world

Equality, serialization, and interoperability - what happens on non-OTLP paths and why it matters for processors and exporters

Performance costs across APIs, SDKs, and the Collector

When to use complex attributes and when flat is still the right choice - and why semantic conventions still prefer flat for indexed, aggregatable dimensions

Leave with concrete rules for when to embrace structure, when to flatten, and how to avoid query, storage, and portability surprises.

Utilization-based autoscaling is popular for a reason: it’s simple, measurable, and usually “good enough.” Until it isn’t. In production, you can hit tail-latency spikes while CPU usage still looks comfortably safe and autoscaling is not triggered.

You can try to make VPA react to app-level signals, but turning those into actionable CPU and memory requests is harder than it sounds. It often needs better instrumentation, app-specific knowledge, and runtime-aware tuning (threads, GC, heap sizing) that doesn’t generalize well across teams and workloads.

So we focused on a signal that’s easier to adopt and closer to the real problem: contention. CPU utilization answers “how much did we run?”, but not “how long did we want to run and couldn’t?” Linux Pressure Stall Information (PSI) captures exactly that by measuring time spent stalled due to insufficient CPU, memory, or I/O. With Kubernetes 1.34, PSI becomes available out of the box, making it realistic to use in everyday vertical scaling decisions.

In this talk, I'll share our journey of using CPU pressure to detect hidden starvation that utilization misses on tightly packed nodes. We’ll also cover what failed: noisy-neighbor eviction and workload shuffling turned into “musical chairs,” spreading disruption without reducing contention.

Our working approach is dynamic resource tuning: PSI triggers short-lived CPU request adjustments that decay over time, stabilizing workloads without permanently over-provisioning.

If you want vertical autoscaling that remains reliable under contention, PSI is a signal worth promoting from “nice to have” to “first-class.”

What to learn

• A practical mental model of what PSI is and what it can (and can’t) tell you
• How requests and limits really behave in Kubernetes, how they map to cgroups
• How to access PSI metrics per workload
• Anti-patterns to avoid

The best enterprise use case for agents is not glamorous. It is the work every engineer agrees matters, every team deprioritizes, and every security review eventually drags back into the room.

CVE remediation is the perfect example because the small-scale story is so seductive: bump the dependency, run CI, merge the PR and move on. That story collapses when you own thousands of repositories. Now the hard part is not the patch. It is finding the right artifact, proving the right fix, reconciling CI, containing the agent, and producing enough evidence and observability for people to trust the outcome.

This talk is a production case study of PatchPilot, an agentic remediation platform built in a regulated industry. I will show how we moved from prototype to production with controller loops, remediation agents, CI reconciliation, approval boundaries, egress controls, observability, and metrics that proved whether the system was creating value or just burning tokens.

The hard part was not getting an agent to open a pull request. The hard part was making it constrained enough for infosec and useful enough for engineers. This is the story of what failed, what changed, and why boring work might be the best place for agents to prove they belong.

Kubernetes is everywhere now—and so are attacks against it. Misconfigurations, exposed APIs, and overly permissive RBAC have made clusters a prime target, while most security tools stop at scanning images and YAML instead of showing what attackers do after they get in.

This talk introduces KubeDecoy – a lightweight, open-source Kubernetes honeypot built from familiar components: vcluster, Falco, Falcosidekick, and NGINX. The goal is simple: stand up a convincing fake cluster, expose realistic attack surfaces, and quietly observe how real adversaries interact with them.

We’ll walk through the architecture, a live-style deployment on Minikube, and practical workarounds for vcluster’s limitations using only open-source components. Along the way, we’ll map KubeDecoy’s detections to the Kubernetes Threat Matrix, highlight where it shines (and where it doesn’t), and show how you can apply these ideas in your own clusters—whether you’re blue team, red team, or somewhere in between.

With a pay-as-you-go model, we are charged for what we use. However, newly provisioned compute is not immediately useful. At any moment, traffic can increase, triggering autoscaling and cold starts, putting pressure on startup time.

What does the startup process of a Pod look like? What takes the most time, what are the limitations? What can we do to reduce the time between scheduling and readiness?

In this talk I will go over stages of starting a pod in Kubernetes. I will explain how image formats influence startup time, what cluster changes are required to take advantage of lazy loading, and how eStargz and SOCI compare. How changing compression formats influences build and startup time. What are the tradeoffs between gzip and zstd? I will discuss peer-to-peer image distribution, its effect on cluster behavior, and the operational complexity it introduces. At the end I will show benchmarks of how much each change affects scheduling to readiness latency.

After this talk you will know how to approach container startup time, what solutions can be applied. What complexities and tradeoffs are there.

I'm sure we all agree, Kubernetes is *amazing*. But sometimes, it’s also... confusing. That’s why Marcus and Márk are here to deliver a brutally honest (and thoroughly entertaining) deep dive into the "Kube-Oddities" - those baffling decisions, peculiar behaviors, and downright WTF moments that make this platform so uniquely interesting.

Let’s ditch the sales pitches and feature announcements. We’re diving down the rabbit hole. We'll explore the weirdness surrounding sidecars, the baffling behavior of image tags, and the downright confusing default behavior of Pod DNS.

This isn’t a technical lecture, it's a fun chat among friends, a shared experience of frustration and occasional triumph. Join Marcus and Márk as they unpack the quirks that keep you constantly questioning, and hopefully, leave you with a deeper appreciation (and a slightly more skeptical eye) for the world of Kubernetes. Prepare for laughter, head-scratching, and maybe just a few ‘WTF?’ moments.

Why move a mountain when you can just swap a brick? The first instinct for many MLOps enthusiasts is to deploy a full model for every specific task—but that’s an infrastructure nightmare.

The smarter way? Build your ML like a Lego constructor. In this talk, I’ll show you how to take one powerful foundation like Gemma 3 and "snap on" tiny, specialized LoRA adapters to handle 100 different tasks. You get all the intelligence of a massive LLM without the heavy lifting

What we will explore:

• Training LoRA adapters instead of re-training the whole model.
• Using TGI to hot-swap these specialized skills in milliseconds.
• Orchestrating it all with GCP Inference Gateway to maximize efficiency

Infrastructure teams already use declarative tools for applications, clusters, and cloud resources: Dockerfiles, Kubernetes manifests, Helm charts, GitOps workflows, and CI/CD pipelines. Yet the operating system underneath the platform is often still treated differently: installed interactively, changed over time, and repaired manually when something breaks.

This talk explores the idea of “Operating Systems as Code”: treating the host OS as a versioned, declarative artifact that is provisioned rather than installed. Using Flatcar Container Linux, Ignition, and Butane as concrete open source examples, it shows how first-boot configuration, immutable root filesystems, and automated updates fit into cloud native platform design.

The session focuses on the mindset shift for platform teams: what should be declared up front, what should move into containers or automation, and how this model changes debugging, upgrades, and long-term maintenance of Kubernetes hosts.

When our team needed to automate the creation of multiple app instances, we quickly realized that existing tools weren’t enough. Writing a custom operator turned out to be the most practical way to manage their lifecycle — and a surprisingly effective way to see how Kubernetes works under the hood.

In this talk, I’ll walk you through the lessons learned while building an operator for a real-world multi-service application. We’ll explore designing Custom Resources, implementing reconciliation logic, and handling state and configuration challenges. Along the way, I’ll share the pitfalls I ran into, debugging tricks that helped, and the patterns that made the operator framework click.

Whether you’re curious about what goes on behind the scenes in Kubernetes operators or planning to build one yourself, you’ll leave with practical tips and a clear roadmap to get started confidently.

Yes, we're now running our agents in the cloud. and some of us also do inference and training. In fact 66% of organizations self-hosting AI models have fully integrated their inference workloads into Kubernetes clusters. At this stage - most of AI workloads are just experiments, but we already know they're costly and often unreliable. In this talk I'll show how you can start optimizing your AI workloads so they're not only valuable but also reliable and efficient.

Most Kubernetes platforms and GitOps tools are designed for organizations operating between 10 and 300 clusters.

But a different category begins when organizations operate between 1,000 and 10,000 clusters across edge, retail, telecom, IoT, supermarkets, or franchise environments. At this scale, many GitOps architectures and tools start to break down, even with HA setups and extensive tuning.

Public scaling data at this level is limited because these environments are rare, expensive, and difficult to reproduce realistically.

In this talk, we share how we approached predicting the operation of 15,000+ Kubernetes clusters based on a real-world customer requirement. Using large-scale GitOps load testing in a hub-and-spoke architecture, we explored scaling limits, bottlenecks, infrastructure costs, and operational trade-offs within a budget of roughly €100,000.

The session also covers what our findings may tell us about the future of GitOps at extreme scale with Argo CD and Sveltos.

Many organizations still treat observability as a tooling decision: select a vendor, deploy agents, and expect insight to follow. Yet teams continue to struggle with fragmented signals, rising complexity, and slow incident response. The real issue is rarely the tool itself - it’s the absence of a shared foundation and a coherent product experience.

This talk reframes observability as an internal platform product, designed with clear users, sane defaults, and long-term evolution in mind. Drawing parallels to Kubernetes and platform engineering, we’ll explore why powerful primitives are not enough - and why standardization, correlation, and paved paths matter more than features.

We’ll look at how OpenTelemetry provides a vendor-neutral foundation for structured telemetry, decoupling instrumentation from backends and enabling correlation by default. This structured approach not only reduces cognitive load and improves reliability, it also lays the groundwork for AI-assisted debugging and natural language interaction with production systems.

Observability isn’t just about collecting more data. It’s about designing a platform that makes understanding systems easier - for both humans and machines.

Mature cloud native platforms provide shared capabilities so teams don't have to reinvent them: CI runners, observability pipelines, and secret management. But one property - instantly, safely, and accountably undoing a change - is rarely provided as a service. Teams build their own escape routes - badly, in parallel.

This talk argues that by applying FeatureOps deliberate practice, reversibility will be the single property that unifies progressive delivery, surgical rollback, experimentation, and chaos engineering, and shows what it means to deliver it as platform infrastructure. You'll leave with a concrete reversal-cost ladder for your own stack - and a reason to put one capability on the roadmap.

GPUs are often the most expensive resources in a Kubernetes cluster, and many workloads use only a fraction of their capacity. In this session, I'll demonstrate how to build a multi-tenant GPU platform using NVIDIA GPU Operator and Multi-Instance GPU (MIG). I will walk through GPU Operator deployment, GPU discovery and scheduling in Kubernetes, and how MIG enables a single physical GPU to be securely shared across multiple workloads. Using a combination of hands-on demonstrations, architecture walkthroughs, and real-world examples, I'll explain how MIG is configured and managed in Kubernetes environments. I'll also cover practical operational considerations, common challenges, and lessons learned from running GPU workloads in production. By the end of the session, attendees will understand how to improve GPU utilisation, reduce infrastructure costs, and run AI workloads more efficiently on Kubernetes.

"Sovereign cloud" used to be a procurement label. In 2026 it became a platform architecture requirement — and the cloud-native ecosystem has quietly produced the answer. KubeVirt is GA. Dynamic Resource Allocation shipped. Kueue is the standard for GPU scheduling. Swisscom published the first sovereign Kubernetes reference architecture on architecture.cncf.io. The era of artisanal sovereign stacks is ending.

This talk presents a concrete, layered reference architecture for AI workloads operating under EU jurisdiction, assembled entirely from CNCF projects. We work the problem from four angles — data sovereignty, operational sovereignty, technical sovereignty, jurisdictional sovereignty — and map each to a specific layer: KubeOne and KubeVirt for cluster and VM lifecycle on EU-controlled hardware, Kyverno and OPA for jurisdictional policy enforcement, ArgoCD with signed commits and SLSA provenance for auditable change, KServe and vLLM for inference under sovereign control, Kueue and DRA for GPU scheduling, and an inference gateway pattern that makes model routing declarative rather than buried in application code.

We then map the architecture to the regulations actually driving these decisions: the EU AI Act (now in full enforcement), DORA, the Cyber Resilience Act, and the EU Cloud Sovereignty Framework's SOV-1 through SOV-8 objectives. Every layer ties to a clause. Every clause ties to a control.

Then the honest part. Sovereignty is layered, and most of the layers leak. NVIDIA driver opacity. Model weight provenance. Agent-to-tool authorization without an MCP identity layer. Audit-grade observability lineage. We'll show what the stack ships today, what we paper over with policy, and where the CNCF community needs to push next.

It all started with a simple idea and a single prompt:

"I want to build a harness for agentic development on Kubernetes using a strict Read-Only RBAC role for kubectl, enforcing all writes declaratively through Git and ArgoCD. How do I stop the AI from 'forgetting' the declarative paradigm and going rogue with imperative fixes?"

As platform engineers and architects, we are told that AI agents will revolutionize operations. But when you put an LLM in front of a live Kubernetes cluster, reality hits hard. Give the agent an inch of imperative power, and it will quickly revert to kubectl edit or kubectl patch to fix a failing deployment, completely breaking the Single Source of Truth.

This talk is a transparent, post-mortem-style journey of building a secure, non-imperative development harness using the minimalist pi.dev engine. Instead of relying on fragile prompt engineering to keep the agent in check, we trapped it in an architectural "straitjacket": a strict Read-Only Kubernetes RBAC profile where the only way out is a Git commit reconciled via ArgoCD.

We will walk you through the technical friction points we hit and how we solved them:

The Server-Side Dry-Run Paradox: How do you let a Read-Only agent validate manifests against admission controllers (like Kyverno or OPA) without granting write permissions? (Spoiler: Welcome to Ephemeral-Driven Development via dynamic scratch-agentic-* namespaces).

The State Lag Dilemma: Bridging the gap between the sequential, fast-paced thinking of an LLM and the asynchronous reconciliation loops of ArgoCD using strict API hook systems.

OWASP Kubernetes Top Ten in the Age of AI: Hardening the harness against prompt injections that attempt cluster privilege escalation.

This talk is an engineer-to-engineer retrospective on the boundaries of Kubernetes security when confronted with cognitive automation. We’ll share our code, our architectural scars, and open the floor to the community with the ultimate question: Knowing these constraints, how would you approach the future of AI-driven platform engineering?

Saturation has long been the stepchild of "the Four Golden Signals of SRE". While latency, traffic, and errors are directly measurable through metrics like P99, RPS, and 5xx rates, monitoring just how "full" a service is relies on indirect symptoms such as CPU usage or queue depth. Yet, saturation should ideally rather be the first signal to alert, as once it's reached, other signals - latency and errors - spike fast.

The inability to directly observe and mitigate saturation drives excessive safety margins, chronically low CPU utilization and massive compute waste in latency-sensitive and customer-facing systems. This session introduces an open-source approach extending Envoy proxy and its seamless integration through eBPF and Cilium, to provide direct observability into service saturation, by comparing each instance's live number of concurrent requests to its true concurrency limit. We then explore how such direct visualization of saturation can help reduce MTTR and minimize waste.

What if securing your Kubernetes cluster were as simple as asking straightforward questions? In this lightning talk, we’ll explore a practical way to understand common Kubernetes risks and how to approach them through threat modeling. You’ll learn to identify potential security issues early and think systematically about what could go wrong. Whether you’re new to Kubernetes or a seasoned professional, this talk provides a clear approach to assessing security risks.

GPU demand outgrows GPU supply very quickly, especially once multiple teams start using the same platform for experimentation, model training, and inference. At that point, the real challenge is no longer just scheduling. It is deciding who gets scarce capacity, under what rules, and with what trade-offs.

This talk is about building a practical multi-cluster GPU allocation model for enterprise and research environments. I’ll cover the design decisions behind fairness, quota, isolation, and utilization, and why “first come, first served” breaks down almost immediately in shared environments. I’ll also walk through the operational pain points that are easy to miss in architecture diagrams: fragmented capacity, stranded GPUs, policy drift, and the tension between centralized governance and cluster-level autonomy.

The goal is not to present a perfect control plane. It is to share a realistic platform approach for teams trying to move from ad hoc GPU sharing to something governable, explainable, and efficient. Attendees will leave with a mental model they can use to reason about multi-cluster GPU platforms in their own organizations.