OpenELB部署及应用(分配LoadBalancer地址请求)

一、OpenELB介绍

网址： openelb.io

OpenELB 是一个开源的云原生负载均衡器实现，可以在基于裸金属服务器、边缘以及虚拟化的 Kubernetes 环境中使用 LoadBalancer 类型的 Service 对外暴露服务。OpenELB 项目最初由 KubeSphere 社区发起，目前已作为 CNCF 沙箱项目加入 CNCF 基金会，由 OpenELB 开源社区维护与支持。

与 MetalLB 类似，OpenELB 也拥有两种主要工作模式：Layer2 模式和 BGP 模式。OpenELB 的 BGP 模式目前暂不支持 IPv6。
无论是 Layer2 模式还是 BGP 模式，核心思路都是通过某种方式将特定 VIP 的流量引到 k8s 集群中，然后再通过 kube-proxy 将流量转发到后面的特定服务。

1.1 Layer2 模式

Layer2 模式需要我们的 k8s 集群基础环境支持发送 anonymous ARP/NDP packets。因为 OpenELB 是针对裸金属服务器设计的，因此如果是在云环境中部署，需要注意是否满足条件。

1.2 BGP 模式

OpenELB 的 BGP 模式使用的是gobgp实现的 BGP 协议，通过使用 BGP 协议和路由器建立 BGP 连接并实现 ECMP 负载均衡，从而实现高可用的 LoadBalancer。

1.3 注意事项

配置 ARP 参数

部署 Layer2 模式需要把 k8s 集群中的 ipvs 配置打开strictARP，开启之后 k8s 集群中的 kube-proxy 会停止响应 kube-ipvs0 网卡之外的其他网卡的 arp 请求，而由 OpenELB 接手处理（OpenELB和MetalLB处理方法类似）。

strict ARP 开启之后相当于把 将 arp_ignore 设置为 1 并将 arp_announce 设置为 2 启用严格的 ARP，这个原理和 LVS 中的 DR 模式对 RS 的配置一样。

二、OpenELB安装及配置

2.1 需求

• You need to prepare a Kubernetes cluster, and ensure that the Kubernetes version is 1.15 or later. OpenELB requires CustomResourceDefinition (CRD) v1, which is only supported by Kubernetes 1.15 or later. You can use the following methods to deploy a Kubernetes cluster（1.15之后的k8s集群）:
• Use KubeKey (recommended). You can use KubeKey to deploy a Kubernetes cluster with or without KubeSphere（kubesphere部署的集群）.
• Follow official Kubernetes guides.

OpenELB is designed to be used in bare-metal Kubernetes environments. However, you can also use a cloud-based Kubernetes cluster for learning and testing（除了裸金属服务器之外还可以是云服务器搭建的k8s节点）.

2.2 Install OpenELB Using kubectl

可参照官网的安装方式来进行安装（选择不同的集群）：https://openelb.io/docs/getting-started/installation/

1.Log in to the Kubernetes cluster over SSH and run the following command:

# 旧版本
kubectl apply -f https://raw.githubusercontent.com/openelb/openelb/master/deploy/openelb.yaml

# 新版本2024-08-27
wget https://raw.githubusercontent.com/openelb/openelb/release-0.6/deploy/openelb.yaml
kubectl apply -f openelb.yaml

2.Run the following command to check whether the status of openelb-manager is READY: 1/1 and STATUS: Running. If yes, OpenELB has been installed successfully.

# 打开https://raw.githubusercontent.com/openelb/openelb/release-0.6/deploy/openelb.yaml 可以看到创建的资源都在openelb-system名称空间内，所以通过下面的语句查看创建的资源
# kubectl get pods -n openelb-system

2.2 OpenELB配置

Use OpenELB in Layer 2 Mode（二层模式的使用）

2.2.1 需求

• You need to prepare a Kubernetes cluster where OpenELB has been installed. All Kubernetes cluster nodes must be on the same Layer 2 network (under the same router).
• You need to prepare a client machine, which is used to verify whether OpenELB functions properly in Layer 2 mode. The client machine needs to be on the same network as the Kubernetes cluster nodes.
• The Layer 2 mode requires your infrastructure environment to allow anonymous ARP/NDP packets. If OpenELB is installed in a cloud-based Kubernetes cluster for testing, you need to confirm with your cloud vendor whether anonymous ARP/NDP packets are allowed. If not, the Layer 2 mode cannot be used.

2.2.2 配置步骤

Step 1: Enable strictARP for kube-proxy

In Layer 2 mode, you need to enable strictARP for kube-proxy so that all NICs in the Kubernetes cluster stop answering ARP requests from other NICs and OpenELB handles ARP requests instead.

1.Log in to the Kubernetes cluster and run the following command to edit the kube-proxy ConfigMap:

# kubectl edit configmap kube-proxy -n kube-system

2.In the kube-proxy ConfigMap YAML configuration, set data.config.conf.ipvs.strictARP to true.

ipvs:
  strictARP: true

3.Run the following command to restart kube-proxy（重启kube-system名称空间当中的daemonset资源，让配置严格arp模式生效）:

# kubectl rollout restart daemonset kube-proxy -n kube-system

Step 2: Specify the NIC Used for OpenELB（如果是多网卡，那么可以指定出口）

If the node where OpenELB is installed has multiple NICs, you need to specify the NIC used for OpenELB in Layer 2 mode. You can skip this step if the node has only one NIC.

In this example, the master1 node where OpenELB is installed has two NICs (eth0 192.168.0.2 and eth1 192.168.1.2), and eth0 192.168.0.2 will be used for OpenELB.

Run the following command to annotate master1 to specify the NIC:

# kubectl annotate nodes k8s-master01 layer2.openelb.kubesphere.io/v1alpha1="192.168.10.141"

Step 3: Create an Eip Object（创建EIP地址池，用于分配LoadBalancer请求时分配IP地址）

The Eip object functions as an IP address pool for OpenELB.

1.Run the following command to create a YAML file for the Eip object:

# vim layer2-eip.yaml

2.Add the following information to the YAML file:

# 旧版本
apiVersion: network.kubesphere.io/v1alpha2
kind: Eip
metadata:
  name: layer2-eip
spec:
  address: 192.168.10.70-192.168.10.99
  interface: ens33
  protocol: layer2

# 2024-08-27 基于官网最新的地址池配置，简要版本的往下拉一点就是
# 官方配置地址： https://openelb.io/docs/getting-started/configuration/configure-ip-address-pools-using-eip/
apiVersion: network.kubesphere.io/v1alpha2
kind: Eip
metadata:
    name: eip-sample-pool
    annotations:
      eip.openelb.kubesphere.io/is-default-eip: "true"
spec:
    address: 192.168.0.91-192.168.0.100
    # 地址池的优先级
    priority: 100
    # namespace 也可以看需求指定，这个地址池在什么名称空间当中使用
    namespaces:
      - test
      - default
    namespaceSelector:
      kubesphere.io/workspace: workspace
    disable: false
    protocol: layer2
    interface: eth0
    # interface: can_reach:192.168.0.1
# 下面这个不要，因为status是在创建资源后查看是否达到你要求的资源，所以在配置的时候可以不要
status:
    occupied: false
    usage: 1
    poolSize: 10
    used: 
      "192.168.0.91": "default/test-svc"
    firstIP: 192.168.0.91
    lastIP: 192.168.0.100
    ready: true
    v4: true

# 2024-08-27 简要版
apiVersion: network.kubesphere.io/v1alpha2
kind: Eip
metadata:
  name: layer2-eip
spec:
  address: 172.31.73.130-172.31.73.132
  namespaces: 
  - project
  interface: eth0
  protocol: layer2

• The IP addresses specified in spec:address must be on the same network segment as the Kubernetes cluster nodes.
• For details about the fields in the Eip YAML configuration, see Configure IP Address Pools Using Eip.

3.Run the following command to create the Eip object:

# kubectl apply -f layer2-eip.yaml

三、OpenELB使用

3.1 在k8s命令行中使用

3.1.1 Create a Deployment

The following creates a Deployment of two Pods using the luksa/kubia image. Each Pod returns its own Pod name to external requests.

1.Run the following command to create a YAML file for the Deployment:

# vim layer2-openelb.yaml

2.Add the following information to the YAML file:

# 创建普通的dp资源，需要注意你上面创建的EIP在哪个名称空间当中。
apiVersion: apps/v1
kind: Deployment
metadata:
  name: layer2-openelb
spec:
  replicas: 2
  selector:
    matchLabels:
      app: layer2-openelb
  template:
    metadata:
      labels:
        app: layer2-openelb
    spec:
      containers:
        - image: luksa/kubia
          name: kubia
          ports:
            - containerPort: 8080

3.Run the following command to create the Deployment:

# kubectl apply -f layer2-openelb.yaml

3.1.2 Create a Service

1.Run the following command to create a YAML file for the Service:

# vim layer2-svc.yaml

2.Add the following information to the YAML file:

    kind: Service
apiVersion: v1
metadata:
  name: layer2-svc
  annotations:
    lb.kubesphere.io/v1alpha1: openelb
    protocol.openelb.kubesphere.io/v1alpha1: layer2
    eip.openelb.kubesphere.io/v1alpha2: layer2-eip
spec:
  selector:
    app: layer2-openelb
  type: LoadBalancer
  ports:
    - name: http
      port: 80
      targetPort: 8080
  externalTrafficPolicy: Cluster

# 2024-08-27  包含EIP地址池的使用方法
# 这里使用的是简要版的EIP地址池

kind: Service
apiVersion: v1
metadata:
  name: nginx
  namespace: project-test
  # EIP的使用主要是根据注解，如果是在其他的云平台上使用，也可以通过添加注解来达到使用openELB分配的IP地址
  annotations:
    lb.kubesphere.io/v1alpha1: openelb
    eip.openelb.kubesphere.io/v1alpha2: layer2-eip
spec:
  selector:
    app: nginx
  # 选择分配LoadBalancer
  type: LoadBalancer
  ports:
    - name: http
      port: 80
      targetPort: 80
  externalTrafficPolicy: Cluster

• You must set spec:type to LoadBalancer.
• The lb.kubesphere.io/v1alpha1: openelb annotation specifies that the Service uses OpenELB.
• The protocol.openelb.kubesphere.io/v1alpha1: layer2 annotation specifies that OpenELB is used in Layer 2 mode.
• The eip.openelb.kubesphere.io/v1alpha2: layer2-eip annotation specifies the Eip object used by OpenELB. If this annotation is not configured, OpenELB automatically uses the first available Eip object that matches the protocol. You can also delete this annotation and add the spec:loadBalancerIP field (for example, spec:loadBalancerIP: 192.168.0.91) to assign a specific IP address to the Service.
• If spec:externalTrafficPolicy is set to Cluster (default value), OpenELB randomly selects a node from all Kubernetes cluster nodes to handle Service requests. Pods on other nodes can also be reached over kube-proxy.
• If spec:externalTrafficPolicy is set to Local, OpenELB randomly selects a node that contains a Pod in the Kubernetes cluster to handle Service requests. Only Pods on the selected node can be reached.

3.Run the following command to create the Service:

kubectl apply -f layer2-svc.yaml

3.1.3 Verify OpenELB in Layer 2 Mode

In the Kubernetes cluster, run the following command to obtain the external IP address of the Service:

# kubectl get svc