kubectl-ko¶

kubectl-ko is a standalone Go kubectl plugin for operating and diagnosing Kube-OVN. It talks to the Kubernetes API and routes node-sensitive operations through the independent kubectl-ko-node-agent DaemonSet. The agent is not a kube-ovn-cni, ovs, ovs-ovn, or ovn-central container; it accesses the node's sockets and namespaces directly. No local Bash, OVN, OVS, tar, or kubectl subprocess is required. kubectl itself is only needed to invoke the plugin as kubectl ko.

Installation¶

Kube-OVN images and the CNI installer install the binary at the existing /usr/local/bin/kubectl-ko path. For a separate workstation, download the kubectl-ko-<os>-<arch>.tar.gz (Linux/macOS) or kubectl-ko-windows-<arch>.zip asset and kubectl-ko-checksums.txt from the matching Kube-OVN GitHub release, verify its SHA256, and put the extracted kubectl-ko on PATH. Linux, macOS and Windows, each on amd64 and arm64, are built independently. Do not copy a Linux pod binary to a macOS or Windows workstation. The Kubernetes nodes and remote OVN/OVS tools remain Linux-based.

The cluster installation script fetches its CLI from the independent agent after the agent DaemonSet rollout completes, selecting the agent container explicitly. It does not copy the CLI from a CNI or OVS/OVN component Pod.

On Windows, use the ZIP matching the workstation architecture. Verify its SHA256 with Get-FileHash, extract it with Expand-Archive, and place kubectl-ko.exe in a directory on PATH alongside the existing kubectl.exe installation. PowerShell can also run it directly:

Get-FileHash .\kubectl-ko-windows-arm64.zip -Algorithm SHA256
Expand-Archive .\kubectl-ko-windows-arm64.zip .\kubectl-ko-windows-arm64
.\kubectl-ko-windows-arm64\kubectl-ko.exe version
.\kubectl-ko-windows-arm64\kubectl-ko.exe --context staging exec nbctl -- show

Use the amd64 archive instead on x64 Windows. kubectl ko version and kubectl plugin list confirm discovery once its directory is on PATH. WSL, Bash and local OVN/OVS installations are not required. Use PowerShell 7.4 or later (or cmd.exe) when redirecting binary stdout such as kubectl ko capture pod/web -- -w -; older PowerShell versions can decode and corrupt native command output.

For a source checkout, run make build-kubectl-ko; the results are dist/images/kubectl-ko and the Linux dist/images/kubectl-ko-node-agent. Install the chart's kubectl-ko-node-agent DaemonSet before using commands that need node access. The module uses replacements, so installing release binaries is preferred to go install ...@version. Both Helm charts deploy the independent agent and use global.registry.imagePullSecrets for private image pulls. The legacy chart includes the agent in full, controlPlaneOnly and dataPlaneOnly installations, so management clusters can execute database commands through the agent too. The v2 chart uses its global image and pull policy, ovsOvn.ovnDirectory, logging.directory and cni.configDirectory to mount the same host data as the running components. The agent is scheduled only on Linux nodes; Windows client support covers the kubectl plugin, while node-side OVN/OVS operations remain Linux-only. On Windows, build from the checkout with go build -o kubectl-ko.exe ./cmd/kubectl-ko. Cross-builds via GOOS=windows GOARCH=amd64 make build-kubectl-ko produce dist/images/kubectl-ko.exe; substitute arm64 for Windows on ARM.

Use kubectl plugin list to detect older copies shadowing the new binary. kubectl ko version prints the local build without accessing a cluster.

Command structure and complete feature inventory¶

The Go CLI uses the following command tree. This is an intentional breaking change: old command names and positional forms are not aliases.

kubectl ko
  exec
    (POD | TYPE/NAME) [-c CONTAINER] [flags] -- COMMAND [args...]
    -l SELECTOR -- COMMAND [args...]
    nbctl | sbctl | ic-nbctl | ic-sbctl -- [TOOL_ARGS...]
    vsctl | ofctl | dpctl | appctl --node NODE -- [TOOL_ARGS...]
  db
    health
    nb
      status | backup | kick SERVER_ID | restore
    sb
      status | backup | kick SERVER_ID
  trace TARGET --target IP [OPTIONS]
  capture TARGET [capture flags] -- [TCPDUMP_ARGS...]
  network
    inspect TARGET [--output table|json]
  diagnose
    cluster
    node NODE
    subnet SUBNET
    connectivity --target PROTOCOL://IP:PORT [--target ...]
    environment
  logs [OPTIONS]
  restart
  perf
    run [OPTIONS]
    recovery --yes
  acl
    decode COOKIE
    listen --node NODE
  version
  completion bash|zsh|fish|powershell

Current target references and diagnostic flags¶

The current trace, capture, and network inspect commands accept one positional TARGET. A bare name means pod/NAME; namespaced resources use KIND/NAME together with -n NAMESPACE. KIND/NAMESPACE/NAME is rejected. Node, localnet, LSP and LRP references are cluster-scoped.

Supported targets include Pods (pod, po, pods), Nodes (node, no, nodes), Services and workloads (service/svc, deployment/deploy, daemonset/ds, statefulset/sts, replicaset/rs, job and aliases), KubeVirt VMs/VMIs (virtualmachine/vm, virtualmachineinstance/vmi), localnet/SUBNET, lsp/NAME, and lrp/NAME. Service and workload targets resolve one running backend Pod.

trace uses trace TARGET --target IP [flags]. --target accepts a bare IPv4/IPv6 address or addr/IP, replacing the legacy --dst-ip form. Use --engine all for OVN followed by OVS, or --engine ovn for OVN-only tracing. Use --source-ip and --source-mac to override synthetic packet fields; localnet sources require both values. --node selects the OVS node for a localnet or localnet LSP source. An unbound ordinary LSP supports OVN-only tracing, while LRP tracing is always OVN-only and defaults to its router-port IP and MAC.

capture uses capture TARGET [flags] -- [CAPTURE_ARGS...]. LRP has no Linux interface, so capture must specify --observe-at lsp/NAME for an adjacent, bound LSP; the resolved observation point is reported on stderr. For Pod, Service/workload and VM/VMI targets, -i IFACE is resolved in the selected Pod network namespace and can select a secondary interface such as net1. -D, -i, -c, -s, -nn and -w - are supported capture arguments.

network inspect uses network inspect TARGET [--output table|json] and supports Pod, Node, Service, workload, VM and VMI targets. Service, workload and VM/VMI results identify both the requested resource and selected Pod; Node targets read the host network namespace. The old network inspect --pod form is removed.

kubectl ko trace -n app svc/api --target addr/10.0.0.8 --protocol tcp --dst-port 443
kubectl ko trace localnet/underlay-a --target 10.0.0.8 --source-ip 10.0.0.2 --source-mac 02:00:00:00:00:02 --node worker-a
kubectl ko trace lsp/web.app --target 10.0.0.8 --engine ovn
kubectl ko trace lrp/router-underlay-a --target 10.0.0.8
kubectl ko capture -n app pod/web -- -i net1 -c 10
kubectl ko capture lrp/router-underlay-a --observe-at lsp/web.app -- -c 10
kubectl ko network inspect node/worker-a
kubectl ko network inspect -n app svc/api --output json

Configuration and argument boundaries¶

Temporary NodePort probes request PreferDualStack and target only the IP families actually allocated to the Service. Single-stack clusters use their available family; dual-stack clusters probe both families. Missing node addresses for an allocated family are reported as errors.

Standard Kubernetes connection flags are accepted before or after subcommands, up to the explicit -- remote-argument separator. The kubeconfig loader supports --kubeconfig, KUBECONFIG, context, TLS, authentication plugins and impersonation. Workload exec uses --namespace/-n, then the kubeconfig namespace, then default; the slash in TYPE/NAME separates the resource type from its name, not the namespace. Other commands that accept a Pod, VM, or VMI name can use namespace/name to override that namespace. The deployment namespace is independently selected by --kube-ovn-namespace, KUBE_OVN_NS, then kube-system.

kubectl ko exec deploy/web -- date
kubectl ko exec -n app pod/web -- date
kubectl ko exec svc/web -c app -- sh -c 'echo ready'
kubectl ko exec -n app -l app=web -- curl http://localhost:8080/healthz
kubectl ko exec nbctl --context staging -- --format=json show
kubectl ko exec nbctl -- -- ls-add example -- lsp-add example example-port
kubectl ko exec vsctl --node worker-a -- --timeout=5 show
kubectl ko trace -n app pod/web --target 10.0.0.8 --protocol tcp --dst-port 443
kubectl ko trace localnet/underlay-a --node worker-a --target 2001:db8::8
kubectl ko capture -n app pod/web -- -w - > capture.pcap
kubectl ko capture -n app vm/database -- -w - > vm.pcap
kubectl ko capture -n app vmi/database -- -c 10
kubectl ko network inspect -n app pod/web
kubectl ko network inspect -n app pod/web --output json
kubectl ko diagnose subnet ovn-default --tcp-port 8100 --udp-port 8101
kubectl ko diagnose connectivity --target tcp://10.0.0.8:8100 --target 'udp://[2001:db8::8]:8101'
kubectl ko db nb backup --output northbound.backup
kubectl ko logs --component ovn --output-dir ./support --strict
kubectl ko perf run --image registry.example/test:v1.13.0 --duration 2s

The separator belongs to the plugin and is removed exactly once. Every argument after it, including another --, --help, --timeout, -n, -c, whitespace or quotes within a single argument, belongs to the remote tool and is preserved. Kubernetes-style resource execution accepts one Pod or workload reference before the separator; -l/--selector is an alternative that selects one running Pod. The resource and selector forms never execute against multiple Pods in one invocation. exec nbctl --help shows local help; exec nbctl -- --help shows remote help. exec nbctl show is rejected instead of guessing the argument boundary.

KubeVirt capture resolves a running VMI's current launcher using its controller UID, node and active Pod status. A positional vm/NAME target verifies that the VMI is controlled by the requested VM; a vmi/NAME target also supports a standalone VMI. A stopped VM or unavailable VMI returns an error. During migration, the source launcher is selected until migration succeeds; a completed failed migration still uses the source. A successful migration selects the target launcher. The selected Pod must be Running and not terminating. Missing or ambiguous launchers and a VMI/Pod change detected before capture fail explicitly; retry after status settles. An established capture stays in that launcher's network namespace and does not follow later migrations. Target diagnostics go to stderr, so -w - preserves pcap stdout. The caller needs permission to get VMIs, list launcher Pods and, for a vm/NAME target, get VMs.

UDP endpoint probes send a health check datagram and require a response. Use a responding health/echo service for UDP; this is not a DNS query or a generic UDP port scan. Subnet probes supply their own matching listeners.

Cluster/node/subnet diagnostics check internal connectivity by default. To also check external ICMP connectivity, repeat --external-address IP, for example diagnose cluster --external-address 1.1.1.1 --external-address 2606:4700:4700::1111. These explicit probes fail the command when unreachable or when a probe Pod has no matching IP family. No public Internet endpoint is required for the default cluster health check.

Each node operation uses Kubernetes pods/exec to start an isolated helper process in the independent agent. A versioned gRPC stream runs over that process's stdin/stdout. Concurrent operations have separate streams and do not restart the agent. It does not execute in Kube-OVN component Pods. Packet capture uses the agent's native Linux AF_PACKET socket and setns(2); it does not start tcpdump, nsenter, or a shell. network inspect resolves the Pod netns from OVSDB through the agent, with a Pod-UID lookup in host process cgroups when no OVS interface exists, then reads Pod and host links with ip -s -j -d addr show; it reports the host-side veth peer by ifindex and resolves macvlan/ipvlan parent NICs in the Pod or host netns using the link namespace identity, name and ifindex, without entering kube-ovn-cni or ovs-ovn. Both JSON and text output include RX/TX statistics for Pod interfaces and resolved host peers or parent NICs. Counter names match iproute2; 64-bit values are preserved, including zero and driver-specific counters. Older stats output is accepted when stats64 is unavailable. Probe Pods may use the normal Kubernetes streaming path for their own short-lived test process. Exec uses WebSocket with SPDY fallback only for supported handshake failures. Completed requests close helper stdin and wait for the outer exec process status before returning; a successful tool result does not hide a helper process failure. Remote exit codes propagate; failed commands are never replayed. Streams have no TTY transformations or stdout banners. Invalid arguments fail before client creation. Cancelling an agent request terminates its Linux command process group, including children such as those started by nsenter, and returns exit 130. Processes that deliberately detach into a different session are outside that group.

Logs and recovery¶

logs keeps the kubectl-ko-log/<node>/{kube-ovn,ovn,openvswitch,linux} layout. Central files on the same node occupy an ovn/central-<pod> child directory. manifest.json records every item and failure. --concurrency, --item-timeout and --max-bytes limit each run; the byte limit is per item. Use --strict to make any failed item fail the command. Archives cannot write outside their collection root. On Unix, extracted files use private permissions. On Windows, use a destination directory protected by your user ACL; Unix permission bits do not configure Windows ACLs. Backup destinations must support hard links (for example, NTFS) for atomic publication without overwriting existing files. Linux state collection and environment checks discover nonterminating Linux nodes independently of CNI Pods. A missing, unready or duplicate agent is an explicit discovery failure; other nodes continue to be checked or collected. XFRM state is collected with nokeys. Treat database and network diagnostics as sensitive local artifacts. IPsec collection locates the CNI Pod's live charon process by Pod UID, pins its root directory and uses the independent agent's strongSwan client to query its control socket. It collects the active ipsec.conf, certificate metadata and status, without executing a tool in the CNI container or exporting keys. Missing or ambiguous source Pods, disabled IPsec and unavailable daemons remain individual IPsec failures in the manifest; other Linux state is still collected.

db nb restore preserves the old operation meaning: reconstruct from a database already on a node, not import an arbitrary local backup. It requires an explicit source node and confirmation; the source must be the first NODE_IPS member, which the image startup script uses to bootstrap the cluster. Recovery supports only the standard central Deployment with literal NODE_IPS membership and a writable central hostPath mount. If OVS does not mount the database directory, temporary hostNetwork helpers use the same hostPath without depending on the OVN Pod network. All helper and database-file checks finish before central is stopped. An existing OVS database mount must match central's hostPath. Run --dry-run first. All central pods must stop before file changes begin. Original files remain in a unique directory on every member; a private local JSON record tracks the last recovery stage. An error stops the procedure and reports this record, the remote directory and original replica count. It does not guess whether it is safe to undo a partially completed recovery. Retain these files for a deliberate recovery or rollback. Remote backup directories use /etc/ovn/.kubectl-ko-recovery-<id> so central startup's file-permission glob cannot remove their directory traversal permissions.

Probe resources have unique names and a run identity. Cleanup uses saved UIDs, not broad label deletion. A cleanup failure reports the exact remaining object without hiding the original error. Diagnostic probe Pods mount a private temporary log directory. Cluster/node/connectivity probes reuse the existing pinger image, service account, tool mounts and network namespace, while executing in a separate short-lived Pod. They retain the original pinger name/IP environment for its Kubernetes checks, revalidate its UID and readiness before execution, and use host PID access plus SYS_ADMIN to enter its network namespace. The independent node agent does not mount a service-account token. No component Pod is an exec target. If the subnet probe DaemonSet fails to become ready, the command collects status and limited container logs from at most eight Pods owned by that DaemonSet before cleanup. Log collection has a five-second deadline and retains the original readiness error. restart, raw OVN/OVS commands, recovery and explicit performance disruption can change live cluster state.

Before Service performance measurements, the temporary IP-wide OVN load balancer must synchronize to all chassis within 30 seconds. This makes qperf's dynamic data ports available before the benchmark starts. A synchronization failure stops measurements and still cleans up the owned load balancer.

Compatibility and migration¶

The supported starting point is a matching CLI and Kube-OVN release. Remote image tools determine feature availability. The independent node agent is required for node sockets, namespaces, captures and host diagnostics; it is deployed by the chart and does not share a component Pod. Existing kubeconfig authorization applies: resource discovery requires get/list, node-agent access needs pods/exec, logs need pods/log, probes need create/delete, and rollout/recovery need workload patch/scale permissions. Exec access is not a read-only database permission.

This rewrite replaces the old command interface. Use logs --component COMPONENT --output-dir DIR in CI and support collectors. Consecutive component collections preserve files already collected in the output directory. Migrate commands using this table:

Bare pod names honor kubeconfig namespace; multiple ready leaders fail instead of selecting arbitrarily; failures retain nonzero exit codes; binary streams do not use a TTY. UDP offered bandwidth defaults to 1G instead of 1000G.

Only the Go executable is shipped and installed. Legacy syntax exists only in version-specific E2E fixtures for older releases. Build scripts and CI invoke the Go binary directly, never through Bash.

Development checks¶

go test -race ./pkg/ko/... ./cmd/kubectl-ko/...
make build-kubectl-ko
make lint

The dedicated workflow tests real exec streams, builds all six workstation platforms and runs file/streaming tests natively on Windows amd64 and arm64. Chart CI builds the complete image from the current checkout, loads it into Kind and installs the v2 chart with that image. It checks credential-free agent readiness on every Linux node, environment output and actual Pod interface statistics. It also executes the installer's CLI bootstrap phase with a temporary local destination, verifies the copy uses the agent container, and compares the copied binary with the current build. It collects OVN and OVS logs with the current CLI's logs --component interface, verifies files on every Linux node, and checks that OVS collection preserves the earlier OVN files. Existing [group:kubectl-ko] E2E exercises trace, capture, logs, diagnostics and backup against a cluster. Environment coverage also installs credential-free agents in a namespace without CNI Pods and verifies every Linux node's environment and interface logs, plus explicit IPsec source failures. The Kind-only serial [group:ha] suite checks restart and leader recovery against NB data and internal connectivity, and database reconstruction against its recovery record, retained originals, NB data and connectivity. It repeats the traffic benchmark and checks probe/LB/multicast cleanup and unchanged central Pod UIDs. These specs use the matching binary from the test cluster image. New CLI paths select the existing full E2E matrix.

PDF Slack Support

Comments