Kubernetes on Oxide: How Customer Needs Shaped Our Integrations

In precocious 2024, customers and prospects were eager to tally Kubernetes connected Oxide, but we had nary supported integrations to thief them do it.

Kubernetes and Oxide are a earthy fit. Kubernetes defines the infrastructure behavior it expects done modular hold points, while Oxide exposes the primitives needed to instrumentality that behaviour done APIs. The instauration for integration was there. What was missing was the package and an knowing of which integrations customers really needed.

That was the business erstwhile I joined Oxide arsenic its first Solutions Software Engineer, focused connected building package to lick customer problems. My first duty was to make it easier to deploy and run Kubernetes connected Oxide.

In my first week, I was handed 2 resources to thief maine get started:

What began pinch those 2 resources grew into a squad effort shaped by a feedback loop. Rather than creation integrations successful the abstract, we followed the problems customers encountered arsenic they moved from provisioning clusters to operating workloads.

This station follows those problems crossed the Kubernetes lifecycle alternatively than in strict chronological order. Different provisioning workflows led america to Rancher, Omni, and Cluster API. Running clusters required infrastructure reconciliation, exposing applications revealed networking gaps, and stateful workloads exposed storage constraints. At each stage, customer workflows exposed the adjacent gap, shaping some the integrations we built and the level activity still ahead.

The first spread we tackled was provisioning. Our contiguous extremity was to unblock the customer who had submitted the Rancher node driver propulsion request. Working through their usage lawsuit would besides springiness america firsthand acquisition creating Kubernetes clusters connected Oxide and thief america uncover the adjacent problems to solve.

No azygous provisioning attack fresh each customers' workflows, truthful we ended up publishing 3 integrations.

Provisioning integrations create and negociate Oxide instances, but they do not reconcile those instances pinch Kubernetes Node objects. Without that reconciliation, a cluster could not reliably find whether an unreachable Kubernetes node was temporarily unavailable aliases whether its backing Oxide instance had been deleted.

We needed a constituent that ran successful each cluster, said to the Oxide API, and continuously reconciled Oxide infrastructure pinch Kubernetes state. Kubernetes provides a modular hold constituent for this purpose: the cloud controller head (CCM). A CCM lets infrastructure-specific controllers merge Kubernetes resources with an infrastructure provider’s API without adding provider-specific codification to Kubernetes itself.

We built the Oxide cloud controller manager to link Kubernetes pinch Oxide. Its node controller keeps Kubernetes Node objects synchronized pinch their backing Oxide instances, recording specifications specified arsenic lawsuit IDs and web addresses, and reporting whether each lawsuit is running, unopen down, aliases nary longer exists. Kubernetes uses this accusation to initialize nodes and safely region them erstwhile their backing instances are deleted.

The CCM does not create instances aliases proviso clusters. That remains the job of provisioning integrations specified arsenic the Rancher node driver, the Omni infrastructure provider, and CAPOx. Instead, it provides a runtime integration shared crossed those provisioning workflows.

Importantly, building the CCM gave america a durable hold constituent wrong each cluster. As Oxide evolves, we tin adhd caller infrastructure-aware controllers to the CCM alternatively than update each provisioning integration.

With that runtime hold constituent successful place, we could reside different furniture of the Kubernetes experience: exposing applications. The CCM architecture also defines a work controller for Kubernetes LoadBalancer services, giving america a place to reside the adjacent customer problem.

One of the capabilities customers expect from cloud-integrated Kubernetes is support for Service objects of type LoadBalancer. When a personification creates one, Kubernetes asks the unreality provider’s work controller to proviso the necessary infrastructure and people its reside successful the Service status. There was conscionable 1 problem: Oxide did not yet connection a autochthonal load balancer.

Oxide did, however, person floating IPs. Floating IPs are addresses from a rack’s outer IP pools that tin beryllium attached to and detached from instances, making those instances reachable from extracurricular their VPCs. Using floating IPs offered a measurement to unblock LoadBalancer services. A floating IP would deliver traffic to a azygous Kubernetes node, and the Kubernetes Service dataplane could administer that postulation to the due pods.

Making that activity required accounting for really Oxide floating IPs look to an instance. They are transparent to the impermanent successful 2 important ways. First, Oxide translates the destination reside of inbound postulation to the instance’s internal IP earlier sending the postulation to the instance. Second, the lawsuit has no network interface configured pinch the floating IP.

The resulting postulation travel looks for illustration this:

That reside translator created a subtle integration problem. The Kubernetes Service dataplane needed to dainty the node’s soul IP arsenic a Service frontend because that was the destination reside packets really carried when they reached the guest. The work controller truthful publishes 2 entries in status.loadBalancer.ingress:

The position entries look for illustration this:

As a result, the kubectl output looks a small unusual:

Users spot some the floating IP and the node’s soul IP successful the EXTERNAL-IP column, moreover though only the floating IP is externally reachable. This is an imperfect abstraction, but it allows america to support a communal Kubernetes workflow while waiting for a autochthonal Oxide load balancer.

This implementation presently supports externalTrafficPolicy: Cluster, which allows the selected node to guardant postulation to a Service endpoint anywhere successful the cluster. If that node disappears, the CCM moves the floating IP to different eligible node and updates the soul reside successful the Service status.

When Oxide introduces a autochthonal load-balancing service, we tin update the service controller to usage it without changing the Kubernetes interface. Customers will continue creating the aforesaid LoadBalancer services and only the infrastructure behind them will change.

To instal the Oxide CCM connected your cluster, spot our CCM guide to get started.

With clusters provisioned, reconciled pinch Oxide, and reachable from outside their VPCs, retention for stateful workloads became the adjacent furniture to address. Kubernetes users petition persistent retention done PersistentVolumeClaim objects and expect a Container Storage Interface (CSI) driver to create, attach, and equine the underlying volumes. Oxide had disks, but Kubernetes had nary autochthonal measurement to negociate their lifecycle.

Without an Oxide CSI driver, customers could deploy a third-party Kubernetes storage strategy specified arsenic Longhorn. Longhorn provides its ain CSI driver and replicates information crossed disks attached to Kubernetes workers. However, utilizing Longhorn meant backing its replicas pinch Oxide distributed disks, which already shop 3 replicas connected chopped sleds.

Layering 1 replicated retention strategy connected different tin create important write fan-out. When a three-replica Longhorn measurement is backed by three-way-replicated Oxide distributed disks, 1 exertion constitute tin instrumentality retired to arsenic galore arsenic nine disk writes. The nonstop beingness constitute amplification depends connected the workload and configuration, but customers wanted to debar that duplicated replication.

The preamble of Oxide section disks provided a measurement to region the 2nd furniture of replication. Local disks person no built-in replication and stay tied to their sled, making them good suited to systems specified arsenic Longhorn that replicate information crossed Kubernetes nodes. Our Rancher showcase uses this attack today. It avoids stacking 2 replicated retention systems, though Longhorn still manages the retention lifecycle alternatively than a autochthonal Oxide integration.

For a autochthonal integration, my teammate Luiz wrote RFD 595 Oxide CSI Plugin. The workflow seemed straightforward connected paper. When a personification creates a PersistentVolumeClaim, the CSI controller creates an Oxide distributed disk. After Kubernetes schedules the pod, the controller attaches that disk to the selected Oxide instance, and the CSI node plugin formats and mounts it for the pod. If the pod is rescheduled onto different node, the controller detaches the disk and reattaches it to the caller node.

Prototyping that workflow instantly exposed a blocker. Oxide requires an instance to beryllium stopped earlier attaching aliases detaching a disk. Kubernetes, however, expects a CSI driver to connect retention to a moving worker after scheduling a pod. Stopping the worker would disrupt each different workload connected the node and could trigger cascading scheduling and attachment operations.

Before we tin merchandise our CSI plugin, we request to adhd support for disk hot-plug throughout the Oxide stack, from the hypervisor each the measurement up to the API. What began arsenic a Kubernetes integration has turned into a task spanning multiple layers of the Oxide package stack.

Disk hot-plug and the Oxide CSI plugin stay nether progressive improvement arsenic of this writing. In the meantime, customers tin usage package specified arsenic Longhorn with Oxide section disks for dynamically provisioned persistent retention without stacking 2 layers of replication. When the autochthonal CSI plugin ships, customers will beryllium capable to usage acquainted Kubernetes retention APIs backed straight by Oxide distributed disks pinch replication and durability built in.

The consequence is not a azygous Kubernetes integration but a increasing ecosystem. Rancher, Omni, and Cluster API supply different paths for provisioning, while the Oxide CCM provides a shared runtime integration for node reconciliation and LoadBalancer services. Customers already usage immoderate of these integrations in production, and we dogfood respective successful our ain accumulation workloads. Together, they supply a coagulated instauration to build on.

Our adjacent measurement is to grow our dogfooding pinch the recently released Cluster API provider. Using it to proviso and run much of our clusters will trial how these integrations activity together time to day.

We still person plentifulness to build and polish. Our near-term activity includes completing disk hot-plug and shipping the CSI plugin, adding autoscaling support, and extending the CCM work controller to support outer subnets. Longer term, as we vessel assets tagging, OIDC support, and autochthonal load balancing, we’ll extend our Kubernetes integrations to return advantage of them.

Building these integrations showed really the architectures of Kubernetes and Oxide complement 1 another. Kubernetes gives infrastructure providers standard extension points, while Oxide exposes infrastructure primitives done APIs. Oxide’s hardware and package co-design lets america reside integration blockers at the furniture wherever they beryllium and transportation the basal changes done the full stack.

This activity besides lets america workout our SDKs and APIs from our customers' perspectives and move customer clash into merchandise improvements. That feedback loop is really we will proceed increasing this ecosystem. Customer needs shaped each integration successful this post, and they will style the adjacent one, too.

To spot the Cluster API and unreality controller head integrations successful action, watch the video below, successful which I deploy a Kubernetes cluster connected Oxide.

Deploy Kubernetes connected Oxide pinch Cluster API