kubeadm安装k8s高可用集群1.24.4 centos7.8
kubeadm安装k8s高可用集群1.24.4 centos7.8
实验机器准备
部署k8s集群的几点按照用途可以划分为2类角色：
1、master：集群的master节点，集群的初始化节点，最低配置不低于4C8g
2、slave：集群的slave节点，可以是多台，最低配置不低于4C4g
3、主机都配置静态ip
| 主机名 | ip | 配置 |
|---|---|---|
| docker-harbor | 10.20.220.230 | 4c4g |
| k8s-master01 | 10.20.220.231 | 4c4g |
| k8s-master02 | 10.20.220.232 | 4c4g |
| k8s-master03 | 10.20.220.233 | 4c4g |
| k8s-node01 | 10.20.220.234 | 4c4g |
| k8s-node02 | 10.20.220.235 | 4c4g |
| lb01 | 10.20.220.236 | 2c2g |
| lb02 | 10.20.220.237 | 2c2g |
| VIP | 10.20.220.238 | |
| nfs | 10.20.220.239 | 2c2g |
| rancher | 10.20.220.66 | 4c4g |
本次实验在centos7.8系统部署，
#主机信息、节点IP、部署组件
k8s-master01 10.20.220.231 etcd, kube-apiserver, kube-controller-manager, kubectl, kubeadm, kubelet, kube-proxy,
k8s-master002 10.20.220.232 etcd, kube-apiserver, kube-controller-manager, kubectl, kubeadm, kubelet, kube-proxy,
k8s-master03 10.20.220.233 etcd, kube-apiserver, kube-controller-manager, kubectl, kubeadm, kubelet, kube-proxy,
k8s-node1 10.20.220.234 kubectl, kubelet, kube-proxy,
k8s-node2 10.20.220.235 kubectl, kubelet, kube-proxy,
设置代理上网
export https_proxy=http://10.20.201.18:8080
export http_proxy=http://10.20.201.18:8080
1 系统优化
1.1 修改静态ip
#修改登录慢DNS
sudo sed -i '/^UseDNS/ {h; s/.*//; x;}; ${x;/^$/ {s//UseDNS no/;H;}; x;}' /etc/ssh/sshd_config
sudo sed -i '/^GSSAPIAuthentication/ {h; s/.*//; x;}; ${x;/^$/ {s//GSSAPIAuthentication no/;H;}; x;}' /etc/ssh/sshd_config
systemc restart sshd
cp /etc/sysconfig/network-scripts/ifcfg-ens33 /etc/sysconfig/network-scripts/ifcfg-ens33.bak
cat >/etc/sysconfig/network-scripts/ifcfg-ens33 <<EOF
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
IPADDR=10.20.220.241
NETMASK=255.255.255.0
GATEWAY=10.220.20.1
DNS1=223.5.5.5
DNS2=114.114.114.114
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
#IPV6INIT=yes
#IPV6_AUTOCONF=yes
#IPV6_DEFROUTE=yes
#IPV6_FAILURE_FATAL=no
#IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens33
#UUID=2c899347-7114-455d-b0a9-e9446ef8d6bc
DEVICE=ens33
ONBOOT=yes
EOF
#更新网卡
systemctl restart network
1.2 修改为命令行界面
systemctl set-default multi-user.target #修改为命令行
1.3 修改默认yum源
cd /etc/yum.repos.d/
mkdir old
mv CentOS-* old
curl -o /etc/yum.repos.d/Centos-7.repo http://mirrors.aliyun.com/repo/Centos-7.repo
wget -O /etc/yum.repos.d/epel.repo https://mirrors.aliyun.com/repo/epel-7.repo
1.4 修改主机名
hostnamectl set-hostname docker-hub
hostnamectl set-hostname k8s-master01
hostnamectl set-hostname k8s-master02
hostnamectl set-hostname k8s-master03
hostnamectl set-hostname k8s-node01
hostnamectl set-hostname k8s-node02
hostnamectl set-hostname lb01
hostnamectl set-hostname lb02
bash
1.5 hosts解析
cat >/etc/hosts <<'EOF'
10.20.220.231 k8s-master01
10.20.220.232 k8s-master02
10.20.220.233 k8s-master03
10.20.220.234 k8s-node01
10.20.220.235 k8s-node02
EOF
ping -c 2 k8s-master01
ping -c 2 k8s-master02
ping -c 2 k8s-master03
ping -c 2 k8s-node01
ping -c 2 k8s-node02
1.6 设置iptables、selinux
systemctl stop firewalld && systemctl disable firewalld
sed -ri 's#(SELINUX=).*#\1disabled#' /etc/selinux/config
setenforce 0
getenforce 0
如果生产机器，防火墙开启，不能关，也可以，
但是要放行一些端口:
6443：Kubernetes API Server
2379、2380：etcd 服务
10250、10255：kubelet 服务
10257：kube-controller-manager 服务
10259：kube-scheduler 服务
30000-32767：在物理机映射的 NodePort 端口
179、473、4789、9099：Calico 服务端口
如何在防火墙规则里放行端口： firewall-cmd --zone=public --add-port=6443/tcp --permanent #--zone 指定了防火墙规则所属的区域，--add-port 指定了要开放的端口号和协议，-- permanent 表示在重启后也会保留这个规则。
1.7 关闭swap
swapoff -a
#关闭开机启动
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
1.8 设置时间同步
yum install chrony -y
sed -i.bak "s@server@#server@g" /etc/chrony.conf
sed -i "6aserver ntp.aliyun.com iburst" /etc/chrony.conf
systemctl enable chronyd --now && systemctl status chronyd
#查看同步状态，前面 ^*开头表示同步正常
[root@rocky_docker ~]# chronyc sources -v
.-- Source mode '^' = server, '=' = peer, '#' = local clock.
/ .- Source state '*' = current best, '+' = combined, '-' = not combined,
| / 'x' = may be in error, '~' = too variable, '?' = unusable.
|| .- xxxx [ yyyy ] +/- zzzz
|| Reachability register (octal) -. | xxxx = adjusted offset,
|| Log2(Polling interval) --. | | yyyy = measured offset,
|| \ | | zzzz = estimated error.
|| | | \
MS Name/IP address Stratum Poll Reach LastRx Last sample
===============================================================================
^* 203.107.6.88 2 6 137 34 +377us[+5935us] +/- 20ms
#写个定时任务
echo '* * * * * /usr/bin/systemctl restart chronyd' >> /etc/crontab
systemctl status crond.service
systemctl restart crond.service
在 Kubernetes 集群中，各个组件之间的通信和协调都需要依赖时间的同步，如果集群中各节点时 间不一致，可能会导致各种奇怪的问题，例如节点之间无法进行正确的 TLS 握手等。因此，建议在安装 Kubernetes 集群之前，将集群中各个节点的时间同步到相同的时间源，保持时间的一致性。
1.9 修改内核参数
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward=1
vm.max_map_count=262144
EOF
modprobe br_netfilter
sysctl -p /etc/sysctl.d/k8s.conf
modprobe 是一个 Linux 命令，它用于动态地加载内核模块到 Linux 内核中。br_netfilter 是 Linux 内核模块之一，它提供了桥接网络设备和 Netfilter 之间的接口。Netfilter 是 Linux 内核中的一 个框架，它可以在数据包通过网络协议栈时进行修改或过滤。 在 Kubernetes 中，br_netfilter 模块用于实现 Kubernetes 集群中的网络功能。通过加载 br_netfilter 模块，我们可以确保在 Kubernetes 集群中使用的 iptables 规则正确应用。
1）net.bridge.bridge-nf-call-ip6tables: 当数据包经过网桥时，是否需要将 IPv6 数据包传递给 iptables 进行处理。将其设置为 1 表示启用。
2）net.bridge.bridge-nf-call-iptables: 当数据包经过网桥时，是否需要将 IPv4 数据包传递给 iptables 进行处理。将其设置为 1 表示启用。
3）net.ipv4.ip_forward: 是否允许主机转发网络包。将其设置为 1 表示启用。 这些参数是为了让 Linux 系统的网络功能可以更好地支持 Kubernetes 的网络组件（如 flannel、Calico 等），启用这些参数可以确保集群中的 Pod 能够正常通信和访问外部网络。
1.10 挂载nfs
apt install nfs-common -y
mkdir /share/
echo "192.168.199.119:/volume1/test /share nfs defaults 0 0" >>/etc/fstab
mount -a
df -h
2 部署 keepalived 和 HAproxy
yum install keepalived haproxy psmisc -y
2.1 配置 HAProxy
在 IP 为 10.20.220.236 与 10.20.220.237 的服务器上按如下参数配置 HAProxy (两台 lb 机器配置一致即可，注意后端服务地址)。
root@lb02:~# yum install vim -y
root@lb02:~# cat /etc/haproxy/haproxy.cfg
cat > /etc/haproxy/haproxy.cfg <<EOF
# HAProxy Configure /etc/haproxy/haproxy.cfg
global
log 127.0.0.1 local2
chroot /var/lib/haproxy
pidfile /var/run/haproxy.pid
maxconn 4000
user haproxy
group haproxy
daemon
# turn on stats unix socket
stats socket /var/lib/haproxy/stats
#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaults
log global
option httplog
option dontlognull
timeout connect 5000
timeout client 5000
timeout server 5000
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend kube-apiserver
bind *:6443
mode tcp
option tcplog
default_backend kube-apiserver
#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend kube-apiserver
mode tcp
option tcplog
balance roundrobin
default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
server kube-apiserver-1 10.20.220.231:6443 check
server kube-apiserver-2 10.20.220.232:6443 check
server kube-apiserver-3 10.20.220.233:6443 check
EOF
启动之前检查语法是否有问题
haproxy -f /etc/haproxy/haproxy.cfg -c
systemctl enable haproxy --now &&
停止 Haproxy
systemctl stop haproxy.service
2.2 配置 Keepalived
主 HAProxy lb-01 -10.20.220.236 (/etc/keepalived/keepalived.conf)
cat > /etc/keepalived/keepalived.conf <<EOF
global_defs {
notification_email {
}
smtp_connect_timeout 30 #连接超时时间
router_id LVS_DEVEL01 ##相当于给这个服务器起个昵称
vrrp_skip_check_adv_addr
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_script chk_haproxy {
script "killall -0 haproxy"
interval 2
weight 20
}
vrrp_instance haproxy-vip {
state MASTER #主服务器 是MASTER
priority 100 #主服务器优先级要比备服务器高
interface ens33 #实例绑定的网卡
virtual_router_id 60 #定义一个热备组，可以认为这是60号热备组
advert_int 1 #1秒互相通告一次，检查对方通讯
authentication {
auth_type PASS
auth_pass 1111
}
unicast_src_ip 10.20.220.236 #当前机器地址
unicast_peer {
10.20.220.237 #peer中其它机器地址
}
virtual_ipaddress {
#vip地址
10.20.220.238/24
}
track_script {
chk_haproxy
}
}
EOF
备主机 HAProxy lb-1-10.20.220.237 (/etc/keepalived/keepalived.conf)
cat > /etc/keepalived/keepalived.conf <<EOF
global_defs {
notification_email {
}
router_id LVS_DEVEL02 ##相当于给这个服务器起个昵称
vrrp_skip_check_adv_addr
vrrp_garp_interval 0
vrrp_gna_interval 0
}
vrrp_script chk_haproxy {
script "killall -0 haproxy"
interval 2
weight 20
}
vrrp_instance haproxy-vip {
state BACKUP #备份服务器 是 backup
priority 90 #优先级要低（把备份的90修改为100）
interface ens33 #实例绑定的网卡
virtual_router_id 60
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
unicast_src_ip 10.20.220.237 #当前机器地址
unicast_peer {
10.20.220.236 #peer 中其它机器地址
}
virtual_ipaddress {
#加/24
10.20.220.238/24
}
track_script {
chk_haproxy
}
}
EOF
2.2.1 启动服务
systemctl restart haproxy
systemctl restart keepalived && systemctl enable keepalived &&systemctl status keepalived
2.2.2 测试VIP负载
3 安装k8s
本次实验采用kubeadm安装k8s集群
kubeadm 是 Kubernetes 主推的部署工具之一，将k8s的组件打包为了镜像，然后通过kubeadm进行集群初始化创建。
3.1 设置yum源安装containerd（每台机器都操作）
#配置docker源地址
curl -o /etc/yum.repos.d/docker-ce.repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
#可以把docker安装上，用于做镜像
yum install docker-ce-20.10.18 -y
[root@k8s-master ~]# mkdir -p /etc/docker
[root@k8s-master ~]# cat /etc/docker/daemon.json
cat > /etc/docker/daemon.json <<EOF
{
"registry-mirrors" : [
"https://8xpk5wnt.mirror.aliyuncs.com"
]
}
EOF
[root@k8s-master ~]# systemctl daemon-reload &&systemctl enable docker && systemctl start docker
#安装containerd.io，
yum list available | grep containerd.io #查看软件有哪些版本可能查不到1.6.22版本，执行命令直接安装就行
yum install containerd.io-1.6.22 -y #安装指定版本的
mkdir -p /etc/containerd
cd /etc/containerd/
containerd config default > /etc/containerd/config.toml
grep sandbox_image /etc/containerd/config.toml
#替换为阿里源
sed -i "s#k8s.gcr.io/pause#registry.aliyuncs.com/google_containers/pause#g" /etc/containerd/config.toml
sed -i "s#registry.k8s.io/pause#registry.aliyuncs.com/google_containers/pause#g" /etc/containerd/config.toml
配置containerd cgroup 驱动程序systemd：
sed -i 's#SystemdCgroup = false#SystemdCgroup = true#g' /etc/containerd/config.toml
##containerd加速
# 修改配置文件/etc/containerd/config.toml, 145行添加config_path
...
144 [plugins."io.containerd.grpc.v1.cri".registry]
145 config_path = "/etc/containerd/certs.d"
146
147 [plugins."io.containerd.grpc.v1.cri".registry.auths]
148
149 [plugins."io.containerd.grpc.v1.cri".registry.configs]
150
151 [plugins."io.containerd.grpc.v1.cri".registry.headers]
152
153 [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
...
# 创建对应的目录
mkdir -p /etc/containerd/certs.d/docker.io
# 配置加速
# 和你配置多个docker镜像加速源一个意思
cat >/etc/containerd/certs.d/docker.io/hosts.toml <<EOF
server = "https://docker.io"
[host."https://ms9glx6x.mirror.aliyuncs.com"]
capabilities = ["pull","resolve"]
[host."https://docker.mirrors.ustc.edu.cn"]
capabilities = ["pull","resolve"]
[host."https://registry-1.docker.io"]
capabilities = ["pull","resolve","push"]
EOF
#重启服务
systemctl restart containerd.service && systemctl enable containerd.service && systemctl status containerd.service
在 Kubernetes 集群中，containerd 是容器运行时，它的主要作用是负责管理节点上的容器，实 现容器的创建、销毁、运行、暂停、恢复等操作。而 Pod 是 Kubernetes 中最基本的调度单元，一个 Pod 包含一个或多个紧密关联的容器，在 Kubernetes 集群中，当一个 Pod 被调度到一个节点上时， Kubernetes 就会基于 containerd 在 pod 里运行容器。
3.1.1 配置非安全http的私有镜像仓库们这里先不用设置
# 如果要和docker一样，再加一个registry私有仓库的配置如下
# 此处目录必须和个人环境中实际的仓库地址保持一致
mkdir -p /etc/containerd/certs.d/10.20.220.230
cat >/etc/containerd/certs.d/10.20.220.230/hosts.toml <<EOF
server = "http://10.20.220.230"
[host."http://10.20.220.230"]
capabilities = ["pull", "resolve", "push"]
skip_verify = true
EOF
3.2 设置apt源安装k8s源，所有节点
# docker制作的镜像，和containerd，k8s都是通用的
# 和containerd没有向docker一样方便的工具，管理image等，所以还是用docker维护容器居多
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=http://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=0
repo_gpgcheck=0
gpgkey=http://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg
http://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
3.3 安装kubeadm工具，所有节点执行
操作节点： 所有的master和slave节点(
k8s-master,k8s-slave) 需要执行这里采用1.24.10版本
安装方式很多，官网推荐以及CKA也都是走kubeadm安装，二进制方式，还可以借助第三方平：rancher、kubesphere 都可以装 k8s.
# --disableexcludes=kubernetes 作用是允许该k8s仓库的一些包的更新
$ yum install -y kubelet-1.24.4 kubeadm-1.24.4 kubectl-1.24.4 --disableexcludes=kubernetes
## 查看kubeadm 版本
$ kubeadm version
#设置开机启动
systemctl enable kubelet &&systemctl restart kubelet && systemctl status kubelet
注：每个软件包的作用
kubelet： kubelet 是 Kubernetes 集群中的一个核心组件，是每个节点上的代理服务，负责与主 控制节点通信，管理节点上的 Pod 和容器。
kubelet 的主要职责包括：
监控 pod 的状态并按需启动或停止容器、检查容器是否正常运行、与主控制节点通信，将节点状态 和 Pod 状态上报给主控制节点、通过各种插件（如 volume 插件）与其他组件协同工作、管理容器的生 命周期，包括启动、停止、重启等、拉取镜像
kubeadm：用于初始化 k8s 集群的命令行工具
kubectl： 用于和集群通信的命令行，通过 kubectl 可以部署和管理应用，查看各种资源，创建、 删除和更新各种组件
3.4 初始化集群，先在master01节点执行
备注：安装 k8s，物理机网段、pod 网段、service 网段不能冲突
kubeadm config print init-defaults > kubeadm.yaml
apiVersion: kubeadm.k8s.io/v1beta3
bootstrapTokens:
- groups:
- system:bootstrappers:kubeadm:default-node-token
token: abcdef.0123456789abcdef
ttl: 24h0m0s
usages:
- signing
- authentication
kind: InitConfiguration
#localAPIEndpoint:
# advertiseAddress: 1.2.3.4
#bindPort: 6443
nodeRegistration:
criSocket: unix:///var/run/containerd/containerd.sock
imagePullPolicy: IfNotPresent
# name: node
taints: null
---
apiServer:
timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
local:
dataDir: /var/lib/etcd
imageRepository: registry.aliyuncs.com/google_containers #修改国内地址
kind: ClusterConfiguration
kubernetesVersion: 1.24.4
controlPlaneEndpoint: 10.20.220.238:6443 #新增
networking:
dnsDomain: cluster.local
serviceSubnet: 10.96.0.0/12
podSubnet: 10.244.0.0/16 #新增
scheduler: {}
特别提醒：
1、 nodeRegistration: criSocket:
unix:///var/run/containerd/containerd.sock nodeRegistration 是在 Kubernetes 集群中注册节点的一种方式。在这个配置中，criSocket 表 示与此节点相关的容器运行时的地址，此处指的是使用 Containerd 作为容器运行时。
2、--image-repository ：registry.cn-hangzhou.aliyuncs.com/google_containers 表示从 阿里云镜像仓库的的名为"google_containers"的命名空间中拉取容器镜像，因为 kubeadm 安装 k8s，控制节点的组件基于 pod 运行，所以需要镜像，默认的镜像仓库国内无法访问，所以需要指定阿 里云镜像仓库，也可以把安装 k8s1.26 需要的镜像用 ctr -n=k8s.io images import 解压，这样就可 以直接在本地找镜像了。
3、podSubnet: 10.244.0.0/12 #指定 pod 网段 serviceSubnet: 10.96.0.0/12 #指定 Service 网段 在 k8s 集群中，pod 网段，service 网络，物理机网段不能冲突。
Kubeadm 安装 k8s，控制节点和工作节点的组件都是基于 pod 运行的，控制节点组件: apiserver、scheduler、controller-manager、etcd、容器、kubelet、kube-proxy、calico
工作节点：flannel、容器运行时、kube-proxy、coredns、kubelet
3.5 基于kubeadm.yaml文件初始化k8s,
3.5.1 在线方法，只有master01节点执行
# 查看需要使用的镜像列表,若无问题，将得到如下列表
[root@k8s-master01 ~]# kubeadm config images list --config kubeadm.yaml
registry.aliyuncs.com/google_containers/kube-apiserver:v1.24.4
registry.aliyuncs.com/google_containers/kube-controller-manager:v1.24.4
registry.aliyuncs.com/google_containers/kube-scheduler:v1.24.4
registry.aliyuncs.com/google_containers/kube-proxy:v1.24.4
registry.aliyuncs.com/google_containers/pause:3.7
registry.aliyuncs.com/google_containers/etcd:3.5.3-0
registry.aliyuncs.com/google_containers/coredns:v1.8.6
[root@k8s-master01 ~]# kubeadm config images pull --config kubeadm.yaml
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.24.4
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.24.4
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.24.4
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.24.4
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.7
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.5.3-0
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.8.6
把下载的镜像打包
#查看镜像
ctr -n=k8s.io images list
#把所有镜像打包成一个压缩包
ctr -n=k8s.io images export k8s.1.24.4 registry.aliyuncs.com/google_containers/kube-apiserver:v1.24.4 registry.aliyuncs.com/google_containers/kube-controller-manager:v1.24.4 registry.aliyuncs.com/google_containers/kube-scheduler:v1.24.4 registry.aliyuncs.com/google_containers/kube-proxy:v1.24.4 registry.aliyuncs.com/google_containers/pause:3.7 registry.aliyuncs.com/google_containers/etcd:3.5.3-0 registry.aliyuncs.com/google_containers/coredns:v1.8.6
#导入镜像
ctr -n=k8s.io images import k8s.1.24.4
3.5.2 初始化集群过程master01节点
#初始化集群
kubeadm init --config kubeadm.yaml
#初始化过程
[root@k8s-master01 ~]# kubeadm init --config kubeadm.yaml
[init] Using Kubernetes version: v1.24.4
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 10.20.220.231 10.20.220.238]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [k8s-master01 localhost] and IPs [10.20.220.231 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [k8s-master01 localhost] and IPs [10.20.220.231 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 14.546159 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node k8s-master01 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node k8s-master01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: abcdef.0123456789abcdef
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
#安装完成
============================================================
Your Kubernetes control-plane has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
============================================================
#拷贝文件到主机
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
Alternatively, if you are the root user, you can run:
export KUBECONFIG=/etc/kubernetes/admin.conf
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of control-plane nodes by copying certificate authorities
and service account keys on each node and then running the following as root:
============================================================
#加入控制节点
kubeadm join 10.20.220.238:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:54eff049bbe5bf065e5ecfbb55a47535b582b714c9917c97c601a06332c1f465 \
--control-plane
Then you can join any number of worker nodes by running the following on each as root:
============================================================
#加入node节点
kubeadm join 10.20.220.238:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:54eff049bbe5bf065e5ecfbb55a47535b582b714c9917c97c601a06332c1f465
3.5.3 k8s master所有主节点拷贝文件
#创建了k8s集群配置文件
#定制了。默认的ssl证书在哪里，api-server的地址，等信息，在主节点执行
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
mkdir -p $HOME/.kube #另外两台也创建
[root@k8s-master01 ~]# scp -r /etc/kubernetes/admin.conf root@k8s-master03:/root/.kube/config
admin.conf 100% 5641 2.9MB/s 00:00
[root@k8s-master01 ~]# scp -r /etc/kubernetes/admin.conf root@k8s-master02:/root/.kube/config
admin.conf 100% 5641 2.5MB/s 00:00
[root@k8s-master01 ~]#
[root@k8s-master01 ~]# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master01 NotReady control-plane 7m17s v1.24.4
3.5.4 配置另外两台master节点
#master02 master03上操作
cd /root && mkdir -p /etc/kubernetes/pki/etcd &&mkdir -p ~/.kube/
#把 master01节点的证书拷贝到 master02 master03
root@k8s-master01:~# scp /etc/kubernetes/pki/ca.crt k8s-master02:/etc/kubernetes/pki/
root@k8s-master01:~# scp /etc/kubernetes/pki/ca.crt k8s-master03:/etc/kubernetes/pki/
root@k8s-master01:~# scp /etc/kubernetes/pki/ca.key k8s-master03:/etc/kubernetes/pki/
root@k8s-master01:~# scp /etc/kubernetes/pki/ca.key k8s-master02:/etc/kubernetes/pki/
root@k8s-master01:~# scp /etc/kubernetes/pki/sa.key k8s-master02:/etc/kubernetes/pki/
root@k8s-master01:~# scp /etc/kubernetes/pki/sa.key k8s-master03:/etc/kubernetes/pki/
root@k8s-master01:~# scp /etc/kubernetes/pki/sa.pub k8s-master03:/etc/kubernetes/pki/
root@k8s-master01:~# scp /etc/kubernetes/pki/sa.pub k8s-master02:/etc/kubernetes/pki/
root@k8s-master01:~# scp /etc/kubernetes/pki/front-proxy-ca.crt k8s-master02:/etc/kubernetes/pki/
root@k8s-master01:~# scp /etc/kubernetes/pki/front-proxy-ca.crt k8s-master03:/etc/kubernetes/pki/
root@k8s-master01:~# scp /etc/kubernetes/pki/front-proxy-ca.crt k8s-master03:/etc/kubernetes/pki/
root@k8s-master01:~# scp /etc/kubernetes/pki/front-proxy-ca.key k8s-master02:/etc/kubernetes/pki/
root@k8s-master01:~# scp /etc/kubernetes/pki/front-proxy-ca.key k8s-master03:/etc/kubernetes/pki/
root@k8s-master01:~# scp /etc/kubernetes/pki/etcd/ca.crt k8s-master03:/etc/kubernetes/pki/etcd/
root@k8s-master01:~# scp /etc/kubernetes/pki/etcd/ca.crt k8s-master02:/etc/kubernetes/pki/etcd/
root@k8s-master01:~# scp /etc/kubernetes/pki/etcd/ca.key k8s-master02:/etc/kubernetes/pki/etcd/
root@k8s-master01:~# scp /etc/kubernetes/pki/etcd/ca.key k8s-master03:/etc/kubernetes/pki/etcd/
#mster02 master03加入控制节点
#加入控制节点
kubeadm join 10.20.220.238:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:54eff049bbe5bf065e5ecfbb55a47535b582b714c9917c97c601a06332c1f465 \
--control-plane
3.5.5 node节点加入集群
3.5.5.1加入master节点报错解决
解决思路：
1、查看firewalld
2、重启systemctl restart containerd.service
3、重置root@k8s-node01:~# kubeadm reset
4、重新生成token
kubeadm token create --print-join-command
我这里两台，第一台重启containerd、重置 kubeadm ，重启后可以了。第二台使用重新生成的token也可以了。
#在node所有节点执行
kubeadm join 10.20.220.238:6443 --token abcdef.0123456789abcdef \
--discovery-token-ca-cert-hash sha256:54eff049bbe5bf065e5ecfbb55a47535b582b714c9917c97c601a06332c1f465
4 安装k8s命令补全插件
#k8s命令太多了，非常有必要配置补全
yum install bash-completion -y
source /usr/share/bash-completion/bash_completion
source <(kubectl completion bash)
echo "source <(kubectl completion bash)" >> ~/.bashrc
5 安装flannel网络
5.1 下载flannel网络插件，配制文件，yaml以及配置文件
wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
#在k8s主节点上，应用这个yaml,基于这个yaml，在创建具体的pod过程。
#如果需要修改pod的运行网络，就要修改配置文件。
vim /root/flannel/Documentation/kube-flannel.yml #具体位置，就是你刚才git下载到的位置
#修改pod网络，改成和你初始化集群的时候填写的地址
[root@k8s-master Documentation]# grep 'Network' -A 5 kube-flannel.yml
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
---
#这里修改为你的宿主机网卡，因为pol夸主机通信，需要走你的物理网卡
165 containers:
166 - name: kube-flannel
167 image: docker.io/flannel/flannel:v0.22.2
168 command:
169 - /opt/bin/flanneld
170 args:
171 - --ip-masq
172 - --kube-subnet-mgr
173 - --iface=ens33 #如果每台机器上的网卡名不一样都要写上，如eth0，ens33 ，在下一行写上
#使用kubectl命令，应用刚才修改后的yaml文件，读取、以及创建pod资源
root@k8s-master01:~# kubectl apply -f ./kube-flannel.yml
namespace/kube-flannel created
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.apps/kube-flannel-ds created
root@k8s-master01:~#
ed
5.2 确保flannel网络创建正确
#查看是否全部运行
root@k8s-master01:~# kubectl -n kube-flannel get pod -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
kube-flannel-ds-9fzrm 1/1 Running 0 4m28s 10.20.220.232 k8s-master02 <none> <none>
kube-flannel-ds-hswjv 1/1 Running 0 4m28s 10.20.220.231 k8s-master01 <none> <none>
kube-flannel-ds-kdnct 1/1 Running 0 4m28s 10.20.220.234 k8s-node01 <none> <none>
kube-flannel-ds-ld7qn 1/1 Running 0 4m28s 10.20.220.235 k8s-node02 <none> <none>
kube-flannel-ds-vm2s9 1/1 Running 0 4m28s 10.20.220.233 k8s-master03 <none>
5.3 网络插件部署完毕后，所有的pod都通过flanned通信
#可以看到所有节点已经ready了
root@k8s-master01:~# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master01 Ready control-plane 82m v1.24.10
k8s-master002 Ready control-plane 34m v1.24.10
k8s-master03 Ready control-plane 33m v1.24.10
k8s-node01 Ready <none> 28m v1.24.10
k8s-node02 Ready <none> 29m v1.24.10
#表示所有节点都准备好了，现在可以部署pod了
6 给主机打标签
kubectl label node k8s-node01 node-role.kubernetes.io/worker=worker
kubectl label node k8s-node02 node-role.kubernetes.io/worker=worker
root@k8s-master01:~# kubectl get nodes
NAME STATUS ROLES AGE VERSION
k8s-master01 Ready control-plane 83m v1.24.10
k8s-master002 Ready control-plane 35m v1.24.10
k8s-master03 Ready control-plane 34m v1.24.10
k8s-node01 Ready worker 30m v1.24.10
k8s-node02 Ready worker 30m v1.24.10
此时node机器就可以和master机器 通信 了，走kubelet进程
root@k8s-master01:~# netstat -ntlup
激活Internet连接 (仅服务器)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 907/sshd: /usr/sbin
tcp 0 0 127.0.0.1:2379 0.0.0.0:* LISTEN 4818/etcd
tcp 0 0 127.0.0.1:2381 0.0.0.0:* LISTEN 4818/etcd
tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 5007/kube-proxy
tcp 0 0 127.0.0.1:10248 0.0.0.0:* LISTEN 4887/kubelet
tcp 0 0 127.0.0.1:10257 0.0.0.0:* LISTEN 4749/kube-controlle
tcp 0 0 127.0.0.1:10259 0.0.0.0:* LISTEN 4759/kube-scheduler
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 633/systemd-resolve
tcp 0 0 10.20.220.231:2379 0.0.0.0:* LISTEN 4818/etcd
tcp 0 0 10.20.220.231:2380 0.0.0.0:* LISTEN 4818/etcd
tcp 0 0 127.0.0.1:35443 0.0.0.0:* LISTEN 885/containerd
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 871/cupsd
tcp 0 0 127.0.0.1:6010 0.0.0.0:* LISTEN 1360/sshd: ubuntu@p
tcp6 0 0 :::10250 :::* LISTEN 4887/kubelet
tcp6 0 0 :::10256 :::* LISTEN 5007/kube-proxy
tcp6 0 0 :::22 :::* LISTEN 907/sshd: /usr/sbin
tcp6 0 0 :::6443 :::* LISTEN 4747/kube-apiserver
tcp6 0 0 ::1:6010 :::* LISTEN 1360/sshd: ubuntu@p
tcp6 0 0 ::1:631 :::* LISTEN 871/cupsd
udp 0 0 0.0.0.0:631 0.0.0.0:* 1072/cups-browsed
udp 0 0 0.0.0.0:5353 0.0.0.0:* 794/avahi-daemon: r
udp 0 0 0.0.0.0:40424 0.0.0.0:* 794/avahi-daemon: r
udp 0 0 127.0.0.53:53 0.0.0.0:* 633/systemd-resolve
udp 0 0 0.0.0.0:8472 0.0.0.0:* -
udp 0 0 127.0.0.1:323 0.0.0.0:* 12123/chronyd
udp6 0 0 :::5353 :::* 794/avahi-daemon: r
udp6 0 0 :::34108 :::* 794/avahi-daemon: r
udp6 0 0 ::1:323 :::* 12123/chronyd
root@k8s-master01:~#
7 kubectl 帮助命令
7.1发出pod创建请求，运行一个nginx-pod
#创建一个nginx-pod
root@k8s-master01:~# kubectl run nginx --image=nginx
pod/nginx created
#查询pod信息
root@k8s-master01:~# kubectl get pods -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx 1/1 Running 0 50s 10.244.3.2 k8s-node02 <none> <none>
root@k8s-master01:~#
7.2 测试访问nginx页面
#修改nginx首页信息，查看nginx在node1上运行，
root@k8s-master01:~# kubectl get pods -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx 1/1 Running 0 50s 10.244.3.2 k8s-node02 <none> <none>
root@k8s-master01:~#
oot@k8s-master01:~# kubectl exec nginx -it -- sh -c "echo 6666 > /usr/share/nginx/html/index.html"
root@k8s-master01:~# curl 10.244.3.2
6666
root@k8s-master01:~#
8 修改k8s 证书过期,三个master都执行
#查看证书有效时间
root@k8s-master01:~# openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep Not
Not Before: Mar 8 15:06:13 2024 GMT
Not After : Mar 8 15:06:13 2025 GMT
cd /etc/kubernetes/pki
# 备份旧证书
mkdir backup_key; cp -rp ./* backup_key/
# 下载项目，自动化修改证书有效期
git clone https://github.com/yuyicai/update-kube-cert.git
cd update-kube-cert/
bash update-kubeadm-cert.sh all
#若无法clone项目，可以手动在浏览器中打开后，复制update-kubeadm-cert.sh 脚本内容到机器中执行
# 默认10年，自己可以修改。再次验证，查看集群有效期是10年了(10年后你早跑路了，不做这个操作也罢。哈哈)
root@k8s-master01:/etc/kubernetes/pki/update-kube-cert# openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep Not
Not Before: Mar 8 16:37:38 2024 GMT
Not After : Mar 6 16:37:38 2034 GMT
[root@k8s-master update-kube-cert]#
9 测试k8s集群DNS是否正常
[root@k8s-master ~]# kubectl run busybox --image busybox --restart=Never --rm -it busybox -- sh
If you don't see a command prompt, try pressing enter.
/ #
/ #
/ # ping www.baidu.com
PING www.baidu.com (220.181.38.149): 56 data bytes
64 bytes from 220.181.38.149: seq=0 ttl=54 time=3.781 ms
64 bytes from 220.181.38.149: seq=1 ttl=54 time=4.000 ms
^C
--- www.baidu.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 3.781/3.890/4.000 ms
/ #
/ # nslookup kubernetes.default.svc.cluster.local
Server: 10.96.0.10
Address: 10.96.0.10:53
Name: kubernetes.default.svc.cluster.local
Address: 10.96.0.1
#查看这些信息，说明k8s正常
10 安装企业私有仓库Harbor
系统优化这些前面的一样
10.1 安装docker 和docker-compose
yum install docker-ce docker-compose -y
10.2 安装harbor
#创建目录
root@docker-hub:~# cd docker_harbor/
root@docker-hub:~/docker_harbor# ls
#上传压缩包，并解压
harbor-offline-installer-v1.9.0-rc1.tgz
root@docker-hub:~/docker_harbor# tar -xf harbor-offline-installer-v1.9.0-rc1.tgz
root@docker-hub:~/docker_harbor# ls
harbor harbor-offline-installer-v1.9.0-rc1.tgz
#查看文件
root@docker-hub:~/docker_harbor# cd harbor/
root@docker-hub:~/docker_harbor/harbor# ls
harbor.v1.9.0.tar.gz harbor.yml install.sh LICENSE prepare
root@docker-hub:~/docker_harbor/harbor#
harbor.v1.9.0.tar.gz #软件包
harbor.yml
install.sh #安装脚本
LICENSE #授权说明
prepare #脚本
# 修改部分参数
# harbor软件运行再容器里，映射暴露给宿主机去访问
# yaml 字典语法
hostname: 10.20.220.230
harbor_admin_password: Harbor12345
默认用户 admin
#安装
root@docker-hub:~/docker_harbor/harbor# bash install.sh
10.3 创建项目
10.4 修改docker配置，信任自建仓库
#手动写入配置
root@docker-hub:~/docker_harbor/harbor# cat /etc/docker/daemon.json
{
"registry-mirrors": ["https://eih8mpot.mirror.aliyuncs.com"],
"insecure-registries":["http://10.20.220.230"]
}
systemctl daemon-reload
systemctl restart docker.service
#启动所有挂的容器
root@docker-hub:~/docker_harbor/harbor# docker start `docker ps -aq`
cc40c5527bef
3342c0023ff4
8bc4fd3cfdbb
72d41a9bc804
f08982c45055
4d99f94ecdbf
e21d703c3a7a
53e952e02cdd
d79ef3fb2fc1
10.5 推送本地镜像（修改镜像tag）
在项目中标记镜像：
docker tag SOURCE_IMAGE[:TAG] 10.20.220.230/ae/IMAGE[:TAG]
推送镜像到当前项目：
docker push 10.20.220.230/ae/IMAGE[:TAG]
#先下载一个nginx镜像
docker pull nginx
#修改镜像
root@docker-hub:~# docker tag nginx:latest 10.20.220.230/ae/nginx:v1
#登录登录harbor
root@docker-hub:~# docker login 10.20.220.230
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
root@docker-hub:~# docker push 10.20.220.230/ae/nginx:v1
The push refers to repository [10.20.220.230/ae/nginx]
d874fd2bc83b: Pushed
32ce5f6a5106: Pushed
f1db227348d0: Pushed
b8d6e692a25e: Pushed
e379e8aedd4d: Pushed
2edcec3590a4: Pushed
v1: digest: sha256:ee89b00528ff4f02f2405e4ee221743ebc3f8e8dd0bfd5c4c20a2fa2aaa7ede3 size: 1570
root@docker-hub:~#
10.6 设置开机自动启动，不太好使，
cat >> /usr/lib/systemd/system/harbor.service <<EOF
[Unit]
Description=Harbor
After=docker.service systemd-networkd.service systemd-resolved.service
Requires=docker.service
Documentation=http://github.com/vmware/harbor
[Service]
Type=simple
Restart=on-failure
RestartSec=5
ExecStart=/usr/local/bin/docker-compose -f /root/docker_harbor/harbor/docker-compose.yml up
ExecStop=/usr/local/bin/docker-compose -f /root/docker_harbor/harbor/docker-compose.yml down
[Install]
WantedBy=multi-user.target
EOF
cat >> /etc/systemd/system/docker-compose-harbor.service <<EOF
[Unit]
Description=Docker Compose Harbor Service
Requires=docker.service
After=docker.service
[Service]
Type=oneshot
RemainAfterExit=yes
WorkingDirectory=/root/docker_harbor/harbor
ExecStart=/usr/bin/docker-compose up -d
ExecStop=/usr/bin/docker-compose down
TimeoutStartSec=0
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable harbor --now && systemctl status harbor.servicer
10.7 测试k8s下载私有仓库容器
直接使用ctr下载要指定文件配置，要不会报错
链接：containerd/docs/hosts.md 在 main ·容器d/containerd ·GitHub上
root@k8s-master01:~# cat cheng-svc-pod-test.yaml
# 基于deployment创建3个pod，加上标签。
kind: Namespace
apiVersion: v1
metadata:
name: cheng-svc-test
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
namespace: cheng-svc-test
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- image: 10.20.220.230/ae/nginx:v1
name: container-0
resources:
limits:
cpu: 100m
memory: 200Mi
requests:
cpu: 100m
memory: 200Mi
11 设置LoadBalancer模式
LoadBalancer一般都是在公有云上使用，通过负载绑定一个和k8s同网段的ip。咱们自己搭建的环境也可以使用。
链接：MetalLB, bare metal load-balancer for Kubernetes (universe.tf)
11.1 创建pod
#创建目录
mkdir LoadBalancer
#下载yaml文件
wget https://raw.githubusercontent.com/metallb/metallb/v0.14.3/config/manifests/metallb-native.yaml
11.2 使用Layer 2 Configuration模式，这个最简单
1、先创建一个地址池，里面的地址和k8s节点在同一网络
2、在设置L2模式
root@k8s-master01:~/LoadBalancer# cat ippool.yaml
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
name: first-pool
namespace: metallb-system
spec:
addresses:
- 10.20.220.90-10.20.220.95
root@k8s-master01:~/LoadBalancer# kubectl apply -f ippool.yaml
ipaddresspool.metallb.io/first-pool created
root@k8s-master01:~/LoadBalancer# vim l2.yaml
root@k8s-master01:~/LoadBalancer# cat l2.yaml
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
name: example
namespace: metallb-system
root@k8s-master01:~/LoadBalancer# kubectl apply -f l2.yaml
l2advertisement.metallb.io/example created
root@k8s-master01:~/LoadBalancer# kubectl -n metallb-system get ipaddresspools.metallb.io
NAME AUTO ASSIGN AVOID BUGGY IPS ADDRESSES
first-pool true false ["10.20.220.0/24"]
11.3 创建LoadBalancer pod
root@k8s-master01:~# cat cheng-load.yaml
apiVersion: v1
kind: Service
metadata:
name: cheng-loadip # Service的名称 ，svc ip，dns,自定义，做好按业务名称，通俗易懂
namespace: cheng-svc-test #和之前创建的后端放一个名称空间中
spec:
selector: # Label Selector，选择包含app=nginx标签的Pod，后面只要这个命名空间中带这个标签的server IP就会自动发现
app: nginx #之前创建pod中的保持一致
ports:
- name: service
targetPort: 80 # Pod的端口
port: 80 # Service对外暴露的端口，也就是ClusterIP的port
protocol: TCP # 转发协议类型，支持TCP和UDP
type: LoadBalancer # Service的类型
12 安装nfs
系统优化和之前的一样，关闭防火墙、虚拟内存
12.1 创建共享文件夹
root@nfs:~# mkdir /nfs
12.2 安装nfs服务端
yum install nfs-utils rpcbind -y
#创建挂载目录
mkdir -p /nfs/k8s/
12.3 修改共享权限
#安装NFS服务后，会新增一个/etc/exports文件(即/etc/目录下名为exports的文件)，NFS服务根据它的配置来运行。
vim /etc/exports
#设置所有人都可以连接
/nfs/k8s *(rw,sync,no_subtree_check)
#设置权限，不然挂载后不能写
chmod 777 /nfs/k8s/
#重新加载配置
export -ra
#重启服务
systemctl restart nfs-server
systemctl restart rpcbind
12.4安装客户端测试挂载
yum install nfs-utils -y
[root@k8s-master01 ~]# showmount -e 10.20.220.239
Export list for 10.20.220.239:
/nfs/k8s *
[root@k8s-master01 ~]#
root@k8s-master01:~# mkdir /share
[root@k8s-master01 ~]# echo "10.20.220.239:/nfs/k8s /share nfs defaults 0 0 ">> /etc/fstab
[root@k8s-master01 ~]# monut -a
[root@k8s-master01 ~]# df -h
文件系统 容量 已用 可用 已用% 挂载点
/dev/mapper/centos-root 50G 3.3G 47G 7% /
dev/mapper/centos-home 46G 33M 46G 1% /home
10.20.220.239:/nfs/k8s 50G 1.4G 49G 3% /share
[root@k8s-master01 ~]#
[root@k8s-master01 ~]# touch /share/aa
[root@k8s-master01 ~]# ls /share/
aa
[root@k8s-master01 ~]#
13 安装 nfs provisioner，用于配合存储类动态生成 pv
因为nfs是外部的存储，所以需要安装制备器
参考：https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner
1、创建运行 nfs-provisioner 需要的 sa 账号
[root@k8s-slave1 volumes]# cat serviceaccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-provisioner
扩展：什么是 sa？
sa 的全称是 serviceaccount。 serviceaccount 是为了方便 Pod 里面的进程调用 Kubernetes API 或其他外部服务而设计的。
指定了 serviceaccount 之后，我们把 pod 创建出来了，我们在使用这个 pod 时，这个 pod 就有了 我们指定的账户的权限了。
2、对 sa 授权
kubectl create clusterrolebinding nfs-provisioner --clusterrole=cluster-admin --serviceaccount=default:nfs-provisioner
3、安装 nfs-provisioner 程序
在nfs服务器上创建一个共享目录
root@nfs:~# mkdir /nfs/pod_share
root@nfs:~# echo "/nfs/pod_share *(rw,sync,no_subtree_check)" >> /etc/exports
root@nfs:~# exportfs -ra
root@nfs:~# chmod 777 /nfs/pod_share/
#重启服务
systemctl restart nfs-server.service
[root@nfs 课程资料]# cat nfs-deployment.yaml
cat > nfs-deployment.yaml <<EOF
kind: Deployment
apiVersion: apps/v1
metadata:
name: nfs-provisioner
spec:
selector:
matchLabels:
app: nfs-provisioner
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: nfs-provisioner
spec:
serviceAccount: nfs-provisioner
containers:
- name: nfs-provisioner
image: registry.cn-beijing.aliyuncs.com/mydlq/nfs-subdir-external-provisioner:v4.0.0
imagePullPolicy: IfNotPresent
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: example.com/nfs
- name: NFS_SERVER
value: 10.20.220.239 #改成你自己的地址
- name: NFS_PATH
value: /nfs/pod_share #改成你自己的共享目录
volumes:
- name: nfs-client-root
nfs:
server: 10.20.220.239
path: /nfs/pod_share
EOF
挂载报错
这个错误是应为pod所在的节点上没安装nfs客户端，安装后就可以了
yum install nfs-utils -y
13.1 创建 storageclass，动态供给 pv
[root@k8s-slave1 volumes]# cat nfs-storageclass.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: nfs
provisioner: example.com/nfs
13.2 创建 pvc，通过 storageclass 动态生成 pv
[root@k8s-slave1 volumes]# cat pvcclaim.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: test-claim1
spec:
accessModes: ["ReadWriteMany"]
resources:
requests:
storage: 1Gi
storageClassName: nfs
可以看到到 test-claim1 的 pvc 已经成功创建了，绑定的 pv 是pvc-b8526c49-f2ca-4cab-b679-b0d3e65059a4，这个 pv 是由 storageclass 调用 nfs provisioner 自动生成的。
步骤总结：
1、供应商：创建一个 nfs provisioner （只需要创建一次）
2、创建 storageclass，storageclass 指定刚才创建的供应商 （只需要创建一次）
3、创建 pvc，这个 pvc 指定 storageclass
13.3创建 pod，挂载 storageclass 动态生成的 pvc
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
namespace:
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- image: 10.20.220.230/ae/nginx:v1
name: container
volumeMounts:
- name: nginxhtml
mountPath: /usr/share/nginx/html/
volumes:
- name: nginxhtml
persistentVolumeClaim:
claimName: test-claim1
14 rancher
rancher是一个web界面管理k8s集群的工具，可以通过在web界面点点就可以实现所要的资源。
官网：https://ranchermanager.docs.rancher.com
14.1 安装docker
curl -o /etc/yum.repos.d/Centos-7.repo http://mirrors.aliyun.com/repo/Centos-7.repo
curl -o /etc/yum.repos.d/epel.repo https://mirrors.aliyun.com/repo/epel-7.repo
systemctl stop firewalld && systemctl disable firewalld
sed -ri 's#(SELINUX=).*#\1disabled#' /etc/selinux/config
setenforce 0
getenforce 0
swapoff -a
#关闭开机启动
sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
yum install chrony -y
sed -i.bak "s@server@#server@g" /etc/chrony.conf
sed -i "6aserver ntp.aliyun.com iburst" /etc/chrony.conf
systemctl start chronyd &&systemctl status chronyd
modprobe br_netfilter
cat > /etc/sysctl.d/docker.conf <<EOF
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward = 1
EOF
#配置阿里源
yum remove docker docker-common docker-selinux docker-engine -y
curl -o /etc/yum.repos.d/docker-ce.repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
yum install docker-ce docker-compose -y
mkdir -p /etc/docker
cat > /etc/docker/daemon.json <<EOF
{
"registry-mirrors":["https://rsbud4vc.mirror.aliyuncs.com","https://registry.docker-cn.com","https://docker.mirrors.ustc.edu.cn","https://dockerhub.azk8s.cn","http://hub-mirror.c.163.com"],
"insecure-registries":["http://10.20.220.230"]
}
EOF
sudo systemctl daemon-reload &&sudo systemctl restart docker && systemctl enable docker && systemctl status docker
docker info |grep version
14.2 安装rancher
[root@localhost rancher]# cat docker-compose.yaml
version: '3.3'
services:
rancher:
image: 'rancher/rancher:v2.7.3'
ports:
- '443:443'
- '80:80'
restart: unless-stopped
privileged: true
14.3加入集群
#重置密码
https://ranchermanager.docs.rancher.com/zh/faq/technical-items#如何重置管理员密码
$ docker exec -ti <container_id> reset-password
New password for default administrator (user-xxxxx):
<new_password>
curl --insecure -sfL https://10.20.220.66/v3/import/sxlxvj5rtpprzjcb4k6q55qf82nr4x2ghqr5wscgmxgzjh47n2sssb_c-m-445gjd2b.yaml | kubectl apply -f -
15 配置ingress
