Skip to content

Custom VPC Internal Load Balancer

The Service provided by Kubernetes can be used for load balancing within the cluster. However, there are several issues with using Service as internal load balancing in customize VPC mode:

  1. The Service IP range is a cluster resource, shared by all customize VPCs, and cannot overlap.
  2. Users cannot set internal load balancing IP addresses according to their own preferences.

To address the above issues, Kube-OVN introduced the SwitchLBRule CRD in 1.11, allowing users to set internal load balancing rules within customize VPCs.

SwitchLBRule support the following two ways to set internal load balancing rules within a custom VPC.

Automatically Generate Load Balancing Rules by Selector

Load balancing rules can be generated by using the selector field. The SLR will target any Pod/VM with matching labels. Using the selector field will only detect IPs on the main interface of Pods/VMs and will exclude any secondary interface (e.g when using Multus).

Example of SwitchLBRule is as follows:

apiVersion: kubeovn.io/v1
kind: SwitchLBRule
metadata:
  name:  cjh-slr-nginx
spec:
  vip: 1.1.1.1
  sessionAffinity: ClientIP
  namespace: default
  selector:
    - app:nginx
  ports:
  - name: dns
    port: 8888
    targetPort: 80
    protocol: TCP
  • usage of selector, sessionAffinity, and port is the same as Kubernetes Service.
  • vip: load balancing IP address, please read this section to choose the right address.
  • namespace: namespace of the pod selected by selector.

Kube-OVN will determine the VPC of the selected pods based on the SwitchLBRule definition and set the corresponding L2 LB.

Note

If you wish to manually specify the VPC/Subnet targeted by the SwitchLBRule and avoid auto-detection, use annotation ovn.kubernetes.io/logical_router and ovn.kubernetes.io/logical_switch on the SLR.

Manually Defined Load Balancing Rules by Endpoints

Load balancing rules can be configured by specifying static endpoints to support scenarios where they cannot be automatically generated through selector. For example, to target an endpoint which is on a secondary interface of a Pod/VM.

Example of SwitchLBRule is as follows:

apiVersion: kubeovn.io/v1
kind: SwitchLBRule
metadata:
  name:  cjh-slr-nginx
spec:
  vip: 1.1.1.1
  sessionAffinity: ClientIP
  namespace: default
  endpoints:
    - 192.168.0.101
    - 192.168.0.102
    - 192.168.0.103
  ports:
  - name: dns
    port: 8888
    targetPort: 80
    protocol: TCP

Usage of sessionAffinity, and port is the same as Kubernetes Service.

  • vip: load balancing IP address, please read this section to choose the right address.
  • namespace: namespace of the pods targeted by endpoints.
  • endpoints: load balancing backend IP list.

If both selector and endpoints are configured, the selector configuration will be automatically ignored.

Health checks

OVN supports health checks for load balancer endpoints (for IPv4 load balancers only, OVN doesn't support IPv6 health checks). When health checks are enabled, the load balancer uses only healthy endpoints. Health checks are enabled by default on SLRs whether they use fields selector or endpoints.

Note

If you wish to disable healthchecks, you can use the annotation ovn.kubernetes.io/service_health_check: false on the SLR.

How they work

Kube-OVN uses the implementation of OVN for health checks: [Health Checks](https://www.ovn.org/support/dist-docs/ovn-nb.5.html)

Here's how it happens when an SLR is created with health checks enabled:

  • While creating the SwitchLBRule, Kube-OVN obtains a reusable VIP from the corresponding VPC and Subnet. This VIP will be used as the source of the healthchecks. The VIP custom resource is created in the Subnet to reserve the IP in the IPAM and prevent Pods/VM from using that IP.
  • Kube-OVN associates the corresponding IP_Port_Mappings and Load_Balancer_Health_Check on the load balancer.

The VIP used as the source of the checks will be automatically picked up if it exists in the corresponding Subnet with the same name as the Subnet. If it does not exist, it will be automatically created and deleted after all associated SwitchLBRule are deleted. If you wish to reserve a custom VIP within your Subnet, ensure you're creating a custom resource of type IP with your chosen address before creating the SLR.

SwitchLBRule VIP address selection

When creating a SwitchLBRule, you must provide a VIP in field vip. This virtual IP will receive the traffic to be load balanced on the backends of the SLR (specified using selector or endpoints).

Due to how OVN implements load balancers, this IP address should be outside the CIDR of the subnet in which Pods/VMs will try to join the load balancer.

For example, a load balancer cannot have a VIP 10.0.0.100 if a pod in subnet 10.0.0.0/24 is expected to reach it. In that case, using address 192.168.1.100 would work, as it isn't part of the source CIDR.

If you wish to use a VIP with an address in the source subnet, you must declare a custom resource Vip with type switch_lb_rule before creating the SLR. For more information, head to the documentation on custom VIPs.

Creating a SwitchLBRule

root@server:~# kubectl get po -o wide -n vulpecula
NAME                     READY   STATUS    RESTARTS   AGE     IP          NODE     NOMINATED NODE   READINESS GATES
nginx-78d9578975-f4qn4   1/1     Running   3          4d16h   10.16.0.4   worker   <none>           <none>
nginx-78d9578975-t8tm5   1/1     Running   3          4d16h   10.16.0.6   worker   <none>           <none>

# create slr

root@server:~# cat << END > slr.yaml
apiVersion: kubeovn.io/v1
kind: SwitchLBRule
metadata:
  name:  nginx
  namespace:  vulpecula
spec:
  vip: 1.1.1.1
  sessionAffinity: ClientIP
  namespace: default
  selector:
    - app:nginx
  ports:
  - name: dns
    port: 8888
    targetPort: 80
    protocol: TCP
END

root@server:~# kubectl apply -f slr.yaml
root@server:~# kubectl get slr
NAME              VIP       PORT(S)    SERVICE                       AGE
vulpecula-nginx   1.1.1.1   8888/TCP   default/slr-vulpecula-nginx   3d21h

The vip with the same name of the subnet has been created.

# vip for check

root@server:~# kubectl get vip
NAME          NS    V4IP        MAC                 V6IP    PMAC   SUBNET        READY   TYPE
vulpecula-subnet    10.16.0.2   00:00:00:39:95:C1   <nil>          vulpecula-subnet   true

Query the Load_Balancer_Health_Check and Service_Monitor by commands.

root@server:~# kubectl ko nbctl list Load_Balancer
_uuid               : 3cbb6d43-44aa-4028-962f-30d2dba9f0b8
external_ids        : {}
health_check        : [5bee3f12-6b54-411c-9cc8-c9def8f67356]
ip_port_mappings    : {"10.16.0.4"="nginx-78d9578975-f4qn4.default:10.16.0.2", "10.16.0.6"="nginx-78d9578975-t8tm5.default:10.16.0.2"}
name                : cluster-tcp-session-loadbalancer
options             : {affinity_timeout="10800"}
protocol            : tcp
selection_fields    : [ip_src]
vips                : {"1.1.1.1:8888"="10.16.0.4:80,10.16.0.6:80"}

root@server:~# kubectl ko nbctl list Load_Balancer_Health_Check
_uuid               : 5bee3f12-6b54-411c-9cc8-c9def8f67356
external_ids        : {switch_lb_subnet=vulpecula-subnet}
options             : {failure_count="3", interval="5", success_count="3", timeout="20"}
vip                 : "1.1.1.1:8888"

root@server:~# kubectl ko sbctl list Service_Monitor
_uuid               : 1bddc541-cc49-44ea-9935-a4208f627a91
external_ids        : {}
ip                  : "10.16.0.4"
logical_port        : nginx-78d9578975-f4qn4.default
options             : {failure_count="3", interval="5", success_count="3", timeout="20"}
port                : 80
protocol            : tcp
src_ip              : "10.16.0.2"
src_mac             : "c6:d4:b8:08:54:e7"
status              : online

_uuid               : 84dd24c5-e1b4-4e97-9daa-13687ed59785
external_ids        : {}
ip                  : "10.16.0.6"
logical_port        : nginx-78d9578975-t8tm5.default
options             : {failure_count="3", interval="5", success_count="3", timeout="20"}
port                : 80
protocol            : tcp
src_ip              : "10.16.0.2"
src_mac             : "c6:d4:b8:08:54:e7"
status              : online

At this point, the service response can be successfully obtained through load balancer vip.

root@server:~# kubectl exec -it -n vulpecula nginx-78d9578975-t8tm5 -- curl 1.1.1.1:8888
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>

Updating load balancer service endpoints

Update the service endpoints of the load balancer by deleting the pod.

kubectl delete po nginx-78d9578975-f4qn4
kubectl get po -o wide -n vulpecula
NAME                     READY   STATUS    RESTARTS   AGE     IP          NODE     NOMINATED NODE   READINESS GATES
nginx-78d9578975-lxmvh   1/1     Running   0          31s     10.16.0.8   worker   <none>           <none>
nginx-78d9578975-t8tm5   1/1     Running   3          4d16h   10.16.0.6   worker   <none>           <none>

Query the Load_Balancer_Health_Check and Service_Monitor by commands, the results have undergone corresponding changes.

root@server:~# kubectl ko nbctl list Load_Balancer
_uuid               : 3cbb6d43-44aa-4028-962f-30d2dba9f0b8
external_ids        : {}
health_check        : [5bee3f12-6b54-411c-9cc8-c9def8f67356]
ip_port_mappings    : {"10.16.0.4"="nginx-78d9578975-f4qn4.default:10.16.0.2", "10.16.0.6"="nginx-78d9578975-t8tm5.default:10.16.0.2", "10.16.0.8"="nginx-78d9578975-lxmvh.default:10.16.0.2"}
name                : cluster-tcp-session-loadbalancer
options             : {affinity_timeout="10800"}
protocol            : tcp
selection_fields    : [ip_src]
vips                : {"1.1.1.1:8888"="10.16.0.6:80,10.16.0.8:80"}

root@server:~# kubectl ko nbctl list Load_Balancer_Health_Check
_uuid               : 5bee3f12-6b54-411c-9cc8-c9def8f67356
external_ids        : {switch_lb_subnet=vulpecula-subnet}
options             : {failure_count="3", interval="5", success_count="3", timeout="20"}
vip                 : "1.1.1.1:8888"

root@server:~# kubectl ko sbctl list Service_Monitor
_uuid               : 84dd24c5-e1b4-4e97-9daa-13687ed59785
external_ids        : {}
ip                  : "10.16.0.6"
logical_port        : nginx-78d9578975-t8tm5.default
options             : {failure_count="3", interval="5", success_count="3", timeout="20"}
port                : 80
protocol            : tcp
src_ip              : "10.16.0.2"
src_mac             : "c6:d4:b8:08:54:e7"
status              : online

_uuid               : 5917b7b7-a999-49f2-a42d-da81f1eeb28f
external_ids        : {}
ip                  : "10.16.0.8"
logical_port        : nginx-78d9578975-lxmvh.default
options             : {failure_count="3", interval="5", success_count="3", timeout="20"}
port                : 80
protocol            : tcp
src_ip              : "10.16.0.2"
src_mac             : "c6:d4:b8:08:54:e7"
status              : online

Delete SwitchLBRule and confirm the resource status, Load_Balancer_Health_Check adn Service_Monitor has been deleted, and the corresponding vip has also been deleted.

root@server:~# kubectl delete -f slr.yaml 
switchlbrule.kubeovn.io "vulpecula-nginx" deleted
root@server:~# kubectl get vip
No resources found
root@server:~# kubectl ko sbctl list Service_Monitor
root@server:~# 
root@server:~# kubectl ko nbctl list Load_Balancer_Health_Check
root@server:~#

PDF Slack Support

Comments