Skip to content

Commit b4069be

Browse files
committed
Check the stored Helm value in the isolation-tier guides and match the Kata guide to the Helm-based installer
1 parent 99b3e03 commit b4069be

3 files changed

Lines changed: 18 additions & 54 deletions

File tree

‎docs/guides/isolation-tiers/gvisor.mdx‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -201,14 +201,15 @@ kubectl taint node <node-name> gvisor=true:NoSchedule --overwrite
201201
202202
The label is what the RuntimeClass schedules onto; the taint keeps regular runc pods off the node.
203203
204-
The taint also keeps the Fluent Bit log collector off the node, and without it agents there produce no runtime logs. Fluent Bit is a DaemonSet rendered by the `observability-logs-opensearch` Helm chart, so its toleration is a chart value. Check whether your install already sets it:
204+
The taint also keeps the Fluent Bit log collector off the node, and without it agents there produce no runtime logs. Fluent Bit is a DaemonSet rendered by the `observability-logs-opensearch` Helm chart, so its toleration is a chart value. Check whether the release stores it:
205205
206206
```bash
207-
kubectl get daemonset fluent-bit -n openchoreo-observability-plane \
208-
-o jsonpath='{.spec.template.spec.tolerations}'
207+
helm get values observability-logs-opensearch -n openchoreo-observability-plane
209208
```
210209
211-
If the output does not include `{"operator":"Exists"}`, set the value on the release:
210+
Check the release values rather than the live DaemonSet: a toleration patched onto the DaemonSet (as earlier versions of the setup scripts did) shows up on the live object, but is lost when the DaemonSet is recreated.
211+
212+
If `fluent-bit.tolerations` does not list `- operator: Exists` with no `key` or `effect`, set the value on the release:
212213
213214
```bash
214215
helm upgrade observability-logs-opensearch \

‎docs/guides/isolation-tiers/kata.mdx‎

Lines changed: 8 additions & 46 deletions
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ Like gVisor, Kata uses a **dedicated node** labeled `kata=true` and tainted `kat
4545
| Virtualization | `/dev/kvm` present and functional (see warning above) |
4646
| Architecture | x86_64 |
4747
| OS | Ubuntu (or similar) with systemd + containerd — **not** Container-Optimized OS |
48-
| Access | kubectl access to the cluster (the installer runs from your machine, not on the node) |
48+
| Access | `kubectl` and `helm` access to the cluster (the installer runs from your machine, not on the node) |
4949

5050
## Set Up a Kata Node
5151

@@ -75,7 +75,7 @@ If your platform lets you set taints at node-pool creation, apply `kata=true:NoS
7575

7676
### Step 2: Run the installer from a machine with kubectl access
7777

78-
Unlike the gVisor installer (which runs on the node), the Kata installer runs from **any machine with kubectl access** and does everything through the Kubernetes API:
78+
Unlike the gVisor installer (which runs on the node), the Kata installer runs from **any machine with `kubectl` and `helm` access** and does everything through the Kubernetes API:
7979

8080
```bash
8181
KATA_NODES="<node-name>" bash "${AMP_DIST}/deployments/setup/install-kata.sh"
@@ -85,10 +85,10 @@ What it does:
8585

8686
1. **Pre-flights `/dev/kvm`** on every target node (via a short-lived privileged pod) and aborts if any node lacks it.
8787
2. Labels the node(s) `kata=true`.
88-
3. Applies the upstream **kata-deploy** DaemonSet, with a nodeSelector and toleration injected **before** apply — so it only ever lands on the labeled node(s); your other nodes are never reconfigured.
89-
4. **Auto-wires k3s nodes**: k3s bundles its own containerd (configured only via `config.toml.tmpl`, read at startup), which upstream kata-deploy does not handle — the installer detects k3s nodes and patches the template + restarts the k3s agent for you.
90-
5. Applies the `kata-qemu` RuntimeClass and taints the node(s) `kata=true:NoSchedule`.
91-
6. Lets the Fluent Bit log DaemonSet tolerate the taint, so agent logs are collected. It sets the chart value `fluent-bit.tolerations` on your `observability-logs-opensearch` Helm release (a `helm upgrade --reuse-values` at the release's installed chart version), and falls back to patching the DaemonSet only when that release cannot be upgraded with Helm. A patch is lost whenever the DaemonSet is recreated, so the installer prints the `helm upgrade` command to make it durable.
88+
3. Installs upstream **kata-deploy** from its Helm chart (kata-containers `KATA_VERSION`, default `4.2.0`), scoped to the labeled node(s) so your other nodes are never reconfigured. The chart wires the node's containerd itself on standard Kubernetes, k3s and RKE2; the installer detects which one from the node's kubelet version. Nodes set up by the earlier manifest-based installer are migrated on the first run.
89+
4. Applies the `kata-qemu` RuntimeClass and boots a test pod under it on every target node. The installer reports success only once that pod has run in its own VM, with a kernel different from the node's.
90+
5. Taints the node(s) `kata=true:NoSchedule`. Set `KATA_TAINT=false` to leave them schedulable for other pods.
91+
6. Lets the Fluent Bit log DaemonSet tolerate the taint, so agent logs are collected. It checks the toleration stored in your `observability-logs-opensearch` Helm release and, if it is missing, sets the chart value `fluent-bit.tolerations` (a `helm upgrade --reuse-values` at the release's installed chart version). It patches the DaemonSet only when that release cannot be upgraded with Helm, and then prints the `helm upgrade` command to make it durable, because a patch is lost whenever the DaemonSet is recreated.
9292

9393
:::tip Verify
9494
```bash
@@ -102,41 +102,6 @@ kubectl get node <node-name> --show-labels # kata=true
102102
The toleration is stored in the `observability-logs-opensearch` release's values, so it must be present on every upgrade that re-renders the Fluent Bit DaemonSet. An upgrade with `--reset-values`, or one that passes new `--set`/`-f` values without `--reuse-values`, drops it unless you pass `--set "fluent-bit.tolerations[0].operator=Exists"` again. Agents on the Kata node then stop producing logs with no error.
103103
:::
104104

105-
<details>
106-
<summary>RKE2 nodes (bundled containerd)</summary>
107-
108-
Upstream kata-deploy assumes a standalone containerd at `/etc/containerd/config.toml`, which RKE2 nodes do not have. Two extra steps are needed on each RKE2 Kata node:
109-
110-
**1. Before running the installer**, create an empty stub so the kata-deploy pod gets past its pre-flight check (it crash-loops with `cp: can't stat '/etc/containerd/config.toml'` otherwise):
111-
112-
```bash
113-
sudo mkdir -p /etc/containerd && sudo touch /etc/containerd/config.toml
114-
```
115-
116-
**2. After kata-deploy has installed the binaries into `/opt/kata`** (the node gets the `katacontainers.io/kata-runtime=true` label on success), wire the runtime through RKE2's containerd template — kata-deploy wrote its config to the stub, which RKE2 never reads:
117-
118-
```bash
119-
sudo mkdir -p /usr/local/bin
120-
sudo ln -sf /opt/kata/bin/containerd-shim-kata-v2 /usr/local/bin/containerd-shim-kata-qemu-v2
121-
sudo tee /var/lib/rancher/rke2/agent/etc/containerd/config.toml.tmpl > /dev/null <<'EOF'
122-
{{ template "base" . }}
123-
124-
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu]
125-
runtime_type = "io.containerd.kata-qemu.v2"
126-
privileged_without_host_devices = true
127-
pod_annotations = ["io.katacontainers.*"]
128-
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu.options]
129-
ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration-qemu.toml"
130-
EOF
131-
sudo systemctl restart rke2-agent
132-
```
133-
134-
:::warning Rewrite, never append
135-
The template must contain `{{ template "base" . }}` exactly once. If the file already exists (e.g. it also carries a gVisor or build-related block), merge the runtime sections into one file and rewrite it wholesale; appending a duplicate template line produces an invalid containerd config and the node's kubelet will not come back after the restart — recovery then requires SSH/console access, `kubectl` cannot reach the node.
136-
:::
137-
138-
</details>
139-
140105
### Local development
141106

142107
There is no macOS-local path for Kata (no nested virtualization, and k3d node containers cannot boot Kata VMs). On a KVM-capable **Linux** host running the repository's dev k3d cluster:
@@ -204,13 +169,10 @@ The RuntimeClass exists but containerd on the node has no `kata-qemu` runtime
204169
```bash
205170
kubectl -n kube-system rollout status daemonset/kata-deploy
206171
kubectl -n kube-system logs -l name=kata-deploy
207-
208-
# on the node — confirm the runtime is in containerd's config:
209-
sudo grep -n kata-qemu /var/lib/rancher/k3s/agent/etc/containerd/config.toml # k3s nodes
210-
grep -n kata-qemu /etc/containerd/config.toml # other nodes
172+
kubectl get node <node-name> -L katacontainers.io/kata-runtime # true once kata-deploy has wired the node
211173
```
212174

213-
On k3s nodes, re-running `install-kata.sh` re-applies the containerd template wiring.
175+
Re-running `install-kata.sh` is safe: it re-applies the kata-deploy chart and boots a test pod on every target node again, failing with the reason if Kata still cannot run there.
214176

215177
</details>
216178

‎versioned_docs/version-v1.0.0/guides/isolation-tiers/gvisor.mdx‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -119,14 +119,15 @@ kubectl taint node <node-name> gvisor=true:NoSchedule --overwrite
119119

120120
The label is what the RuntimeClass schedules onto; the taint keeps regular runc pods off the node.
121121

122-
The taint also keeps the Fluent Bit log collector off the node, and without it agents there produce no runtime logs. Fluent Bit is a DaemonSet rendered by the `observability-logs-opensearch` Helm chart, so its toleration is a chart value. Check whether your install already sets it:
122+
The taint also keeps the Fluent Bit log collector off the node, and without it agents there produce no runtime logs. Fluent Bit is a DaemonSet rendered by the `observability-logs-opensearch` Helm chart, so its toleration is a chart value. Check whether the release stores it:
123123

124124
```bash
125-
kubectl get daemonset fluent-bit -n openchoreo-observability-plane \
126-
-o jsonpath='{.spec.template.spec.tolerations}'
125+
helm get values observability-logs-opensearch -n openchoreo-observability-plane
127126
```
128127

129-
If the output does not include `{"operator":"Exists"}`, set the value on the release:
128+
Check the release values rather than the live DaemonSet: a toleration patched onto the DaemonSet (as earlier versions of the setup scripts did) shows up on the live object, but is lost when the DaemonSet is recreated.
129+
130+
If `fluent-bit.tolerations` does not list `- operator: Exists` with no `key` or `effect`, set the value on the release:
130131

131132
```bash
132133
helm upgrade observability-logs-opensearch \

0 commit comments

Comments
 (0)