You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit b4069be
Browse filesBrowse the repository at this point in the historyBrowse files
The label is what the RuntimeClass schedules onto; the taint keeps regular runc pods off the node.
203
203
204
-
The taint also keeps the Fluent Bit log collector off the node, and without it agents there produce no runtime logs. Fluent Bit is a DaemonSet rendered by the `observability-logs-opensearch` Helm chart, so its toleration is a chart value. Check whether your install already sets it:
204
+
The taint also keeps the Fluent Bit log collector off the node, and without it agents there produce no runtime logs. Fluent Bit is a DaemonSet rendered by the `observability-logs-opensearch` Helm chart, so its toleration is a chart value. Check whether the release stores it:
205
205
206
206
```bash
207
-
kubectl get daemonset fluent-bit -n openchoreo-observability-plane \
208
-
-o jsonpath='{.spec.template.spec.tolerations}'
207
+
helm get values observability-logs-opensearch -n openchoreo-observability-plane
209
208
```
210
209
211
-
If the output does not include `{"operator":"Exists"}`, set the value on the release:
210
+
Check the release values rather than the live DaemonSet: a toleration patched onto the DaemonSet (as earlier versions of the setup scripts did) shows up on the live object, but is lost when the DaemonSet is recreated.
211
+
212
+
If `fluent-bit.tolerations` does not list `- operator: Exists` with no `key` or `effect`, set the value on the release:
Copy file name to clipboardExpand all lines: docs/guides/isolation-tiers/kata.mdx
+8-46Lines changed: 8 additions & 46 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -45,7 +45,7 @@ Like gVisor, Kata uses a **dedicated node** labeled `kata=true` and tainted `kat
45
45
| Virtualization |`/dev/kvm` present and functional (see warning above) |
46
46
| Architecture | x86_64 |
47
47
| OS | Ubuntu (or similar) with systemd + containerd — **not** Container-Optimized OS |
48
-
| Access | kubectl access to the cluster (the installer runs from your machine, not on the node) |
48
+
| Access |`kubectl` and `helm` access to the cluster (the installer runs from your machine, not on the node) |
49
49
50
50
## Set Up a Kata Node
51
51
@@ -75,7 +75,7 @@ If your platform lets you set taints at node-pool creation, apply `kata=true:NoS
75
75
76
76
### Step 2: Run the installer from a machine with kubectl access
77
77
78
-
Unlike the gVisor installer (which runs on the node), the Kata installer runs from **any machine with kubectl access** and does everything through the Kubernetes API:
78
+
Unlike the gVisor installer (which runs on the node), the Kata installer runs from **any machine with `kubectl` and `helm` access** and does everything through the Kubernetes API:
1.**Pre-flights `/dev/kvm`** on every target node (via a short-lived privileged pod) and aborts if any node lacks it.
87
87
2. Labels the node(s) `kata=true`.
88
-
3.Applies the upstream **kata-deploy**DaemonSet, with a nodeSelector and toleration injected **before** apply — so it only ever lands on the labeled node(s); your other nodes are never reconfigured.
89
-
4.**Auto-wires k3s nodes**: k3s bundles its own containerd (configured only via `config.toml.tmpl`, read at startup), which upstream kata-deploy does not handle — the installer detects k3s nodes and patches the template + restarts the k3s agent for you.
90
-
5.Applies the `kata-qemu` RuntimeClass and taints the node(s) `kata=true:NoSchedule`.
91
-
6. Lets the Fluent Bit log DaemonSet tolerate the taint, so agent logs are collected. It sets the chart value `fluent-bit.tolerations` on your `observability-logs-opensearch` Helm release (a `helm upgrade --reuse-values` at the release's installed chart version), and falls back to patching the DaemonSet only when that release cannot be upgraded with Helm. A patch is lost whenever the DaemonSet is recreated, so the installer prints the `helm upgrade` command to make it durable.
88
+
3.Installs upstream **kata-deploy**from its Helm chart (kata-containers `KATA_VERSION`, default `4.2.0`), scoped to the labeled node(s) so your other nodes are never reconfigured. The chart wires the node's containerd itself on standard Kubernetes, k3s and RKE2; the installer detects which one from the node's kubelet version. Nodes set up by the earlier manifest-based installer are migrated on the first run.
89
+
4.Applies the `kata-qemu` RuntimeClass and boots a test pod under it on every target node. The installer reports success only once that pod has run in its own VM, with a kernel different from the node's.
90
+
5.Taints the node(s) `kata=true:NoSchedule`. Set `KATA_TAINT=false` to leave them schedulable for other pods.
91
+
6. Lets the Fluent Bit log DaemonSet tolerate the taint, so agent logs are collected. It checks the toleration stored in your `observability-logs-opensearch` Helm release and, if it is missing, sets the chart value `fluent-bit.tolerations`(a `helm upgrade --reuse-values` at the release's installed chart version). It patches the DaemonSet only when that release cannot be upgraded with Helm, and then prints the `helm upgrade` command to make it durable, because a patch is lost whenever the DaemonSet is recreated.
The toleration is stored in the `observability-logs-opensearch` release's values, so it must be present on every upgrade that re-renders the Fluent Bit DaemonSet. An upgrade with `--reset-values`, or one that passes new `--set`/`-f` values without `--reuse-values`, drops it unless you pass `--set "fluent-bit.tolerations[0].operator=Exists"` again. Agents on the Kata node then stop producing logs with no error.
Upstream kata-deploy assumes a standalone containerd at `/etc/containerd/config.toml`, which RKE2 nodes do not have. Two extra steps are needed on each RKE2 Kata node:
109
-
110
-
**1. Before running the installer**, create an empty stub so the kata-deploy pod gets past its pre-flight check (it crash-loops with `cp: can't stat '/etc/containerd/config.toml'` otherwise):
**2. After kata-deploy has installed the binaries into `/opt/kata`** (the node gets the `katacontainers.io/kata-runtime=true` label on success), wire the runtime through RKE2's containerd template — kata-deploy wrote its config to the stub, which RKE2 never reads:
The template must contain `{{ template "base" . }}` exactly once. If the file already exists (e.g. it also carries a gVisor or build-related block), merge the runtime sections into one file and rewrite it wholesale; appending a duplicate template line produces an invalid containerd config and the node's kubelet will not come back after the restart — recovery then requires SSH/console access, `kubectl` cannot reach the node.
136
-
:::
137
-
138
-
</details>
139
-
140
105
### Local development
141
106
142
107
There is no macOS-local path for Kata (no nested virtualization, and k3d node containers cannot boot Kata VMs). On a KVM-capable **Linux** host running the repository's dev k3d cluster:
@@ -204,13 +169,10 @@ The RuntimeClass exists but containerd on the node has no `kata-qemu` runtime
204
169
```bash
205
170
kubectl -n kube-system rollout status daemonset/kata-deploy
206
171
kubectl -n kube-system logs -l name=kata-deploy
207
-
208
-
# on the node — confirm the runtime is in containerd's config:
grep -n kata-qemu /etc/containerd/config.toml # other nodes
172
+
kubectl get node <node-name> -L katacontainers.io/kata-runtime # true once kata-deploy has wired the node
211
173
```
212
174
213
-
On k3s nodes, re-running `install-kata.sh` re-applies the containerd template wiring.
175
+
Re-running `install-kata.sh`is safe: it re-applies the kata-deploy chart and boots a test pod on every target node again, failing with the reason if Kata still cannot run there.
The label is what the RuntimeClass schedules onto; the taint keeps regular runc pods off the node.
121
121
122
-
The taint also keeps the Fluent Bit log collector off the node, and without it agents there produce no runtime logs. Fluent Bit is a DaemonSet rendered by the `observability-logs-opensearch` Helm chart, so its toleration is a chart value. Check whether your install already sets it:
122
+
The taint also keeps the Fluent Bit log collector off the node, and without it agents there produce no runtime logs. Fluent Bit is a DaemonSet rendered by the `observability-logs-opensearch` Helm chart, so its toleration is a chart value. Check whether the release stores it:
123
123
124
124
```bash
125
-
kubectl get daemonset fluent-bit -n openchoreo-observability-plane \
126
-
-o jsonpath='{.spec.template.spec.tolerations}'
125
+
helm get values observability-logs-opensearch -n openchoreo-observability-plane
127
126
```
128
127
129
-
If the output does not include `{"operator":"Exists"}`, set the value on the release:
128
+
Check the release values rather than the live DaemonSet: a toleration patched onto the DaemonSet (as earlier versions of the setup scripts did) shows up on the live object, but is lost when the DaemonSet is recreated.
129
+
130
+
If `fluent-bit.tolerations` does not list `- operator: Exists` with no `key` or `effect`, set the value on the release:
0 commit comments