Skip to content

Gateway Netfilter Rules

Kube-OVN uses a gateway Netfilter backend to implement gateway NAT functionality in the default VPC overlay Subnets. The backend can use either iptables or nftables, while this page also retains the rule reference for the iptables backend.

Gateway backend selection

The kube-ovn-cni daemon supports three gateway Netfilter modes:

Mode Behavior
auto Detect the local kube-proxy mode. iptables and ipvs use the iptables backend; nftables uses the nftables backend.
iptables Always use Kube-OVN's iptables and ipset rules.
nftables Always use Kube-OVN's nftables table and rules.

auto is the default. During a cold start it polls the kube-proxy proxyMode endpoint and, if that endpoint is unavailable before the detection timeout, reads the kube-proxy ConfigMap in kube-system. The default endpoint is http://localhost:10249/proxyMode and the default timeout is 30 seconds. At runtime, a mode change is accepted after three consecutive observations so a transient response does not switch the backend.

Set the mode and detection options in the environment before running bash install.sh, or in the kube-ovn-cni DaemonSet:

export GATEWAY_NETFILTER_MODE=auto
export KUBE_PROXY_MODE_ENDPOINT=http://localhost:10249/proxyMode
export GATEWAY_NETFILTER_DETECT_TIMEOUT=30s
args:
- --gateway-netfilter-mode=auto
- --kube-proxy-mode-endpoint=http://localhost:10249/proxyMode
- --gateway-netfilter-detect-timeout=30s

For Helm installations, use the values for the chart in use:

# charts/kube-ovn
networking:
  GATEWAY_NETFILTER_MODE: auto
  KUBE_PROXY_MODE_ENDPOINT: http://localhost:10249/proxyMode
  GATEWAY_NETFILTER_DETECT_TIMEOUT: 30s
# charts/kube-ovn-v2
agent:
  gatewayNetfilterMode: auto
  kubeProxyModeEndpoint: http://localhost:10249/proxyMode
  gatewayNetfilterDetectTimeout: 30s

The nftables backend manages Kube-OVN-owned kube-ovn tables in the IPv4 and IPv6 nftables families, as applicable, for Service and NodePort NAT, subnet NAT, NAT policy, TProxy, filter and mangle rules, and subnet traffic accounting. It does not read or modify kube-proxy-private nftables objects or marks, and it does not flush conntrack entries.

When the backend changes, Kube-OVN prepares and reconciles the target backend before cleaning the old backend. Switching and uninstall cleanup remove only Kube-OVN-owned iptables, ipset, and nftables objects. If cleanup fails, the active backend is retained and cleanup is retried during later reconciliation.

Nftable LoadBalancer Service

The controller flag --enable-nftable-lb-svc enables LoadBalancer Services backed by the VPC NAT gateway's nftable share-DNAT implementation. In current master this flag defaults to true, and it requires the global --enable-lb=true switch.

The corresponding Helm values are:

# charts/kube-ovn
func:
  ENABLE_NFTABLE_LB_SVC: true

# charts/kube-ovn-v2
features:
  enableNftableLoadbalancerService: false

The v2 chart only renders the controller flag when enableNftableLoadbalancerService is true. When it is false, the flag is omitted and the controller default remains true; use the controller argument explicitly if an installation must opt out until the chart exposes an effective opt-out.

When an nftable LoadBalancer Service is terminating, its EIP:port:protocol identity is released immediately. A replacement Service can therefore reuse the identity and be programmed without waiting for the deleted Service's finalizers to disappear.

The ipset used is shown in the following table:

Name(IPv4/IPv6) Type Usage
ovn40services/ovn60services hash:net Service CIDR
ovn40subnets/ovn60subnets hash:net Overlay Subnet CIDR and NodeLocal DNS IP address
ovn40subnets-nat/ovn60subnets-nat hash:net Overlay Subnet CIDRs that enable NatOutgoing
ovn40subnets-distributed-gw/ovn60subnets-distributed-gw hash:net Overlay Subnet CIDRs that use distributed gateway
ovn40other-node/ovn60other-node hash:net Internal IP addresses for other Nodes
ovn40local-pod-ip-nat/ovn60local-pod-ip-nat hash:ip Deprecated
ovn40subnets-nat-policy hash:net All subnet cidrs configured with natOutgoingPolicyRules
ovn40natpr-418e79269dc5-dst hash:net The dstIPs corresponding to the rule in natOutgoingPolicyRules
ovn40natpr-418e79269dc5-src hash:net The srcIPs corresponding to the rule in natOutgoingPolicyRules

The iptables rules (IPv4) used are shown in the following table:

Table Chain Rule Usage Note
filter INPUT -m set --match-set ovn40services src -j ACCEPT Allow k8s service and pod traffic to pass through --
filter INPUT -m set --match-set ovn40services dst -j ACCEPT Allow k8s service and pod traffic to pass through --
filter INPUT -m set --match-set ovn40subnets src -j ACCEPT Allow k8s service and pod traffic to pass through --
filter INPUT -m set --match-set ovn40subnets dst -j ACCEPT Allow k8s service and pod traffic to pass through --
filter FORWARD -m set --match-set ovn40services src -j ACCEPT Allow k8s service and pod traffic to pass through --
filter FORWARD -m set --match-set ovn40services dst -j ACCEPT Allow k8s service and pod traffic to pass through --
filter FORWARD -m set --match-set ovn40subnets src -j ACCEPT Allow k8s service and pod traffic to pass through --
filter FORWARD -m set --match-set ovn40subnets dst -j ACCEPT Allow k8s service and pod traffic to pass through --
filter FORWARD -s 10.16.0.0/16 -m comment --comment "ovn-subnet-gateway,ovn-default" Used to count packets from the subnet to the external network "10.16.0.0/16" is the cidr of the subnet, the "ovn-subnet-gateway" before the "," in comment is used to identify the iptables rule used to count the subnet inbound and outbound gateway packets, and the "ovn-default" after the "," is the name of the subnet
filter FORWARD -d 10.16.0.0/16 -m comment --comment "ovn-subnet-gateway,ovn-default" Used to count packets from the external network accessing the subnet "10.16.0.0/16" is the cidr of the subnet, the "ovn-subnet-gateway" before the "," in comment is used to identify the iptables rule used to count the subnet inbound and outbound gateway packets, and the "ovn-default" after the "," is the name of the subnet
filter OUTPUT -p udp -m udp --dport 6081 -j MARK --set-xmark 0x0 Clear traffic tag to prevent SNAT UDP: bad checksum on VXLAN interface
nat PREROUTING -m comment --comment "kube-ovn prerouting rules" -j OVN-PREROUTING Enter OVN-PREROUTING chain processing --
nat POSTROUTING -m comment --comment "kube-ovn postrouting rules" -j OVN-POSTROUTING Enter OVN-POSTROUTING chain processing --
nat OVN-PREROUTING -i ovn0 -m set --match-set ovn40subnets src -m set --match-set ovn40services dst -j MARK --set-xmark 0x4000/0x4000 Adding masquerade tags to Pod access service traffic Used when the built-in LB is turned off
nat OVN-PREROUTING -p tcp -m addrtype --dst-type LOCAL -m set --match-set KUBE-NODE-PORT-LOCAL-TCP dst -j MARK --set-xmark 0x80000/0x80000 Add specific tags to ExternalTrafficPolicy for Local's Service traffic (TCP) Only used when kube-proxy is using ipvs mode
nat OVN-PREROUTING -p udp -m addrtype --dst-type LOCAL -m set --match-set KUBE-NODE-PORT-LOCAL-UDP dst -j MARK --set-xmark 0x80000/0x80000 Add specific tags to ExternalTrafficPolicy for Local's Service traffic (UDP) Only used when kube-proxy is using ipvs mode
nat OVN-POSTROUTING -m set --match-set ovn40services src -m set --match-set ovn40subnets dst -m mark --mark 0x4000/0x4000 -j SNAT --to-source Use node IP as the source address for access from node to overlay Pods via service IP. Works only when kube-proxy is using ipvs mode
nat OVN-POSTROUTING -m mark --mark 0x4000/0x4000 -j MASQUERADE Perform SNAT for specific tagged traffic --
nat OVN-POSTROUTING -m set --match-set ovn40subnets src -m set --match-set ovn40subnets dst -j MASQUERADE Perform SNAT for Service traffic between Pods passing through the node --
nat OVN-POSTROUTING -m mark --mark 0x80000/0x80000 -m set --match-set ovn40subnets-distributed-gw dst -j RETURN For Service traffic where ExternalTrafficPolicy is Local, if the Endpoint uses a distributed gateway, SNAT is not required. --
nat OVN-POSTROUTING -m mark --mark 0x80000/0x80000 -j MASQUERADE For Service traffic where ExternalTrafficPolicy is Local, if the Endpoint uses a centralized gateway, SNAT is required. --
nat OVN-POSTROUTING -p tcp -m tcp --tcp-flags SYN NONE -m conntrack --ctstate NEW -j RETURN No SNAT is performed when the Pod IP is exposed to the outside world --
nat OVN-POSTROUTING -s 10.16.0.0/16 -m set ! --match-set ovn40subnets dst -j SNAT --to-source 192.168.0.101 When the Pod accesses the network outside the cluster, if the subnet is NatOutgoing and a centralized gateway with the specified IP is used, perform SNAT 10.16.0.0/16 is the Subnet CIDR, 192.168.0.101 is the specified IP of gateway node
nat OVN-POSTROUTING -m set --match-set ovn40subnets-nat src -m set ! --match-set ovn40subnets dst -j MASQUERADE When the Pod accesses the network outside the cluster, if NatOutgoing is enabled on the subnet, perform SNAT --
nat OVN-POSTROUTING -m set --match-set ovn40subnets-nat-policy src -m set ! --match-set ovn40subnets dst -j OVN-NAT-POLICY When Pod accesses the network outside the cluster, if natOutgoingPolicyRules is enabled on the subnet, the packet with the specified policy will perform SNAT ovn40subnets-nat-policy is all subnet segments configured with natOutgoingPolicyRules
nat OVN-POSTROUTING -m mark --mark 0x90001/0x90001 -j MASQUERADE --random-fully When Pod accesses the network outside the cluster, if natOutgoingPolicyRules is enabled on the subnet, the packet with the specified policy will perform SNAT After coming out of OVN-NAT-POLICY, if it is tagged with 0x90001/0x90001, it will do SNAT
nat OVN-POSTROUTING -m mark --mark 0x90002/0x90002 -j RETURN When Pod accesses the network outside the cluster, if natOutgoingPolicyRules is enabled on the subnet, the packet with the specified policy will perform SNAT After coming out of OVN-NAT-POLICY, if it is tagged with 0x90002/0x90002, it will not do SNAT
nat OVN-NAT-POLICY -s 10.0.11.0/24 -m comment --comment natPolicySubnet-net1 -j OVN-NAT-PSUBNET-aa98851157c5 When Pod accesses the network outside the cluster, if natOutgoingPolicyRules is enabled on the subnet, the packet with the specified policy will perform SNAT 10.0.11.0/24 represents the CIDR of the subnet net1, and the rules under the OVN-NAT-PSUBNET-aa98851157c5 chain correspond to the natOutgoingPolicyRules configuration of this subnet
nat OVN-NAT-PSUBNET-xxxxxxxxxxxx -m set --match-set ovn40natpr-418e79269dc5-src src -m set --match-set ovn40natpr-418e79269dc5-dst dst -j MARK --set-xmark 0x90002/0x90002 When Pod accesses the network outside the cluster, if natOutgoingPolicyRules is enabled on the subnet, the packet with the specified policy will perform SNAT 418e79269dc5 indicates the ID of a rule in natOutgoingPolicyRules, which can be viewed through status.natOutgoingPolicyRules[index].RuleID, indicating that srcIPs meets ovn40natpr-418e79269dc5-src, and dstIPS meets ovn40natpr-418e79269dc5- dst will be marked with tag 0x90002
mangle OVN-OUTPUT -d 10.241.39.2/32 -p tcp -m tcp --dport 80 -j MARK --set-xmark 0x90003/0x90003 Introduce kubelet's detection traffic to tproxy with a specific mark
mangle OVN-PREROUTING -d 10.241.39.2/32 -p tcp -m tcp --dport 80 -j TPROXY --on-port 8102 --on-ip 172.18.0.3 --tproxy-mark 0x90004/0x90004 Introduce kubelet's detection traffic to tproxy with a specific mark

PDF Slack Support

Comments