LoadBalancer for bare metal Kubernetes cluster-MetalLB

爱国，求是，奋进！ / 2024-10-14 / 原文

在 Kubernetes 中，对于 LoadBalancer 类型的 Service，k8s 并没有为裸机集群实现负载均衡器，因此我们只有在以下 IaaS 平台（GCP, AWS, Azure）上才能使用 LoadBalancer 类型的 service。

因此裸机集群只能使用 NodePort 或者 externalIPs service 来对面暴露服务，然而这两种方式和 LoadBalancer service 相比都有很大的缺点,而 MetalLB 的出现就是为了解决这个问题。也就是说在裸机的 K8s 集群无法使用 LoadBancer 类型的 Service。否则，您会发现 LoadBancer 的 Service 一直处于 Pending 状态，而 MetalLB 的出现就是为了解决这个问题。

MetalLB

MetalLB 是一款开源软件，它采用标准的路由协议（ARP 或 BGP）实现了裸机 K8s 集群的负载均衡功能。

https://metallb.io/installation/

工作模式：

L2 模式 (ARP)¶

L2 模式下，MetalLB 会通过 memberlist 选举出一个 Leader 节点，此节点负责向本地网络宣告 LoadBalancerIP。从网络的角度来看，这台机器似乎有多个 IP 地址，它会响应来自 LoadBancerIP 的 ARP 请求。 L2 模式最大的优势是它不需要依赖譬如路由器等硬件的依赖便可工作。

优势：通用型，不需要额外的硬件支持
缺点：单节点的带宽限制、稍缓慢的故障转移（10s 左右）

L3 模式 (BGP)¶

在 BGP 模式下，集群中的每个节点都会与路由器建立 BGP Peer，并使用该会话向集群外部通告集群服务的 LoadBalanceIP。 BGP Router 基于每个不同的连接选择一个下一跳（即集群某个节点，这不同于 L2 模式下所有流量先到达某个 Leader 节点）。

优势：负载均衡性更好
缺点：
- 当某个节点故障，所有 BGP 会话将会中断
- Calico BGP 模式无法和 MetaLB L3 模式并存，会存在冲突，详情请参考 ISSUES WITH CALICO

安装 MetalLB

我的网络插件使用的是kube-router, 默认启用了ARP，所以直接安装：

[root@master-01 metallb]# kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.8/config/manifests/metallb-native.yaml
[root@master-01 metallb]# kubectl get pod -A | grep metallb
metallb-system   controller-6dd967fdc7-9lq2p                                   1/1     Running     0             11h
metallb-system   speaker-kjm8c                                                 1/1     Running     0             11h
metallb-system   speaker-knv64                                                 1/1     Running     0             11h
metallb-system   speaker-rcxhc                                                 1/1     Running     0             11h

L2 模式测试

[root@master-01 metallb]# cat IPAddressPool.yaml
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: pool
  namespace: metallb-system
spec:
  addresses:
  # 可分配的 IP 地址,可以指定多个，包括 ipv4、ipv6,我的路由器网段就是下面的网段
  - 192.168.1.240-192.168.1.250 
[root@master-01 metallb]# cat L2Advertisement.yaml
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: example
  namespace: metallb-system
spec:
  ipAddressPools:
  - pool #上一步创建的 ip 地址池，通过名字进行关联
[root@master-01 metallb]# kubectl create -f IPAddressPool.yaml
[root@master-01 metallb]# kubectl create -f L2Advertisement.yaml

Nginx 部署测试

[root@master-01 metallb]# cat nginx-deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: docker.io/nginx:latest
        ports:
        - containerPort: 80
[root@master-01 metallb]# cat nginx-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  selector:
    app: nginx
  ports:
  - name: nginx-port
    protocol: TCP
    port: 80
    targetPort: 80
  type: LoadBalancer
[root@master-01 metallb]# kubectl create -f nginx-deployment.yml
[root@master-01 metallb]# kubectl create -f nginx-svc.yaml
[root@master-01 metallb]# kubectl get deployments.apps nginx-deployment
NAME               READY   UP-TO-DATE   AVAILABLE   AGE
nginx-deployment   3/3     3            3           10h
[root@master-01 metallb]# kubectl get svc
NAME         TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)        AGE
kubernetes   ClusterIP      10.68.0.1      <none>          443/TCP        11h
nginx        LoadBalancer   10.68.74.145   192.168.1.240   80:31054/TCP   10h
[root@master-01 metallb]# kubectl get pod
NAME                                READY   STATUS    RESTARTS   AGE
nginx-deployment-755fbfbc49-gljgb   1/1     Running   0          10h
nginx-deployment-755fbfbc49-l2928   1/1     Running   0          10h
nginx-deployment-755fbfbc49-lvbsp   1/1     Running   0          10h
[root@master-01 metallb]# ip addr show ens33
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0c:29:80:af:bc brd ff:ff:ff:ff:ff:ff
    altname enp2s1
    inet 192.168.1.56/24 brd 192.168.1.255 scope global dynamic noprefixroute ens33
       valid_lft 45382sec preferred_lft 45382sec
    inet6 fe80::20c:29ff:fe80:afbc/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
[root@master-01 metallb]# curl -I 192.168.1.240
HTTP/1.1 200 OK
Server: nginx/1.27.2
Date: Sun, 13 Oct 2024 20:34:22 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Wed, 02 Oct 2024 15:13:19 GMT
Connection: keep-alive
ETag: "66fd630f-267"
Accept-Ranges: bytes

数据库测试

[root@master-01 metallb]# cat mysql.yml
apiVersion: v1
kind: ConfigMap
metadata:
  name: mysql-config
  namespace: db
data:
  my.cnf: |
    [mysqld]
    bind-address = 0.0.0.0
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: mysql
  name: mysql
  namespace: db
spec:
  replicas: 2
  selector:
    matchLabels:
      app: mysql
  template:
    metadata:
      labels:
        app: mysql
    spec:
      volumes:
      - name: mysql-data
        persistentVolumeClaim:
          claimName: mysql-data
      - name: config-volume
        configMap:
          name: mysql-config
      containers:
      - env:
        - name: MYSQL_ROOT_PASSWORD
          value: "Huawei12#$"
        - name: MYSQL_USER
          value: "k8s"
        - name: MYSQL_PASSWORD
          value: "Huawei12#$"
        image: "mysql:5.6"
        imagePullPolicy: IfNotPresent
        name: mysql
        ports:
        - containerPort: 3306
          protocol: TCP
        volumeMounts:
        - name: mysql-data
          mountPath: /var/lib/mysql
        - name: config-volume
          mountPath: /etc/mysql/conf.d
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: mysql-data
spec:
  capacity:
    storage: 2Gi
  accessModes:
    - ReadWriteMany
  nfs:
    path: /mysql
    server: 192.168.1.56
  persistentVolumeReclaimPolicy: Retain
  mountOptions:
    - vers=4
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mysql-data
  namespace: db
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Gi
---
apiVersion: v1
kind: Service
metadata:
  labels:
    expose: "true"
    app: mysql
  name: mysql
  namespace: db
spec:
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 3306
  selector:
    app: mysql

[root@master-01 metallb]# kubectl get pod,svc -n db
NAME                        READY   STATUS    RESTARTS      AGE
pod/mysql-585df6d54-76pqp   1/1     Running   0             3m57s
pod/mysql-585df6d54-xq5qw   1/1     Running   2 (35s ago)   3m57s

NAME            TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)        AGE
service/mysql   LoadBalancer   10.68.157.159   192.168.1.241   80:30771/TCP   3m57s