Kubernetes 集群安装(Debian 版)

Demon.Lee 2022年12月12日 1,585次浏览

本文演示环境:

demon@ubuntu2204:~$ hostnamectl 
Static hostname: ubuntu2204
Chassis: vm
Virtualization: vmware
Operating System: Ubuntu 22.04.1 LTS              
Kernel: Linux 5.15.0-53-generic
Architecture: arm64
demon@ubuntu2204:~$

网上有那么多 Kubernetes 集群安装文档,为啥还要再写一份?因为那不是自己的。张艺谋说过:眼过千遍,不如手过一遍。计算机是一门实践学科,大多数时候,只有亲手操作一遍,才能真正理解一些东西。

这里先推荐两篇安装文档:

周志明老师:部署 Kubernetes 集群
罗剑锋老师:搭建多节点的 Kubernetes 集群

如果不是安装一个多节点的 Kubernetes 集群,笔者也不知道这玩意如此繁杂。可以说,Kubernetes 对小白一点也不友好。不过,从这个角度来看,把 Kubernetes 比作云原生时代的操作系统也很合理。

值得欣慰的是,Kubernetes 官方提供了一个本地单节点集群的安装工具:minikube


借助于 minikube,我们可以快速把 Kubernetes 玩起来。

来源:网络

不过,minikube 并不是真正的集群,总差点意思。所以,笔者就借助官方推荐的 Kubeadm 工具,在 Ubuntu 22.04 的两台机器上把 Kubernetes 集群搭起来,以下是执行步骤,供大家参考。

Step1: 安装 Containerd

请参考笔者的上一篇文章《容器运行时:containerd

Step2: 安装前准备

1、改主机名

demon@ubuntu2204-master:~$ sudo vi /etc/hostname 
...
demon@ubuntu2204-master:~$ cat /etc/hostname 
ubuntu2204-master
demon@ubuntu2204-master:~$

2、设置流量转发

修改 iptables 的配置,启用“br_netfilter”模块,让 kubernetes 可以检查和转发网络流量。

cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
br_netfilter
EOF

cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward=1 # better than modify /etc/sysctl.conf
EOF

sudo sysctl --system

3、关闭 Linux Swap 分区

基于安全性(如在官方文档中承诺的 Secret 只会在内存中读写,不会落盘)、利于保证节点同步一致性等原因,从 1.8 版开始,Kubernetes 就在它的文档中明确声明了它默认不支持 Swap 分区,在未关闭 Swap 分区的机器中,集群将直接无法启动。

sudo cp /etc/fstab /etc/fstab_bak
sudo swapoff -a
sudo sed -ri '/\sswap\s/s/^#?/#/' /etc/fstab

4、注册 apt 软件源

这里选择阿里云:

sudo apt install -y apt-transport-https ca-certificates curl

curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | sudo apt-key add -

cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF

sudo apt update

Step3: 安装 kubeadm, kubelet, kubectl

这里先解释一下这三个工具的作用:

  • kubeadm:用于引导、安装和启动 Kubernetes 集群的命令行工具,只需要简单的几个命令工具(如 init,join,upgrade,reset 等),便可以快速完成集群的维护管理工作;
  • kubelet:每个节点上的代理程序,负责节点上的大部分操作,比如由它来启停容器等,节点上只有它能与 apiserver 进行通信;
  • kubectl:Kubernetes 客户端工具,用来操控 Kubernetes 集群,比如查看集群中的节点信息等;

如果不指定版本,则默认安装最新版本,笔者这里选择的版本是 1.25.3:

sudo apt install -y kubeadm=1.25.3-00 kubelet=1.25.3-00 kubectl=1.25.3-00

锁定版本,不让它们自动更新:

demon@ubuntu2204-master:~$ sudo apt-mark hold kubeadm kubelet kubectl
kubeadm set on hold.
kubelet set on hold.
kubectl set on hold.
demon@ubuntu2204-master:~$ 

查看安装结果:

demon@ubuntu2204-master:~$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3", GitCommit:"434bfd82814af038ad94d62ebe59b133fcb50506", GitTreeState:"clean", BuildDate:"2022-10-12T10:55:36Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/arm64"}
demon@ubuntu2204-master:~$ 
demon@ubuntu2204-master:~$ kubectl version
WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.3", GitCommit:"434bfd82814af038ad94d62ebe59b133fcb50506", GitTreeState:"clean", BuildDate:"2022-10-12T10:57:26Z", GoVersion:"go1.19.2", Compiler:"gc", Platform:"linux/arm64"}
Kustomize Version: v4.5.7
The connection to the server localhost:8080 was refused - did you specify the right host or port?
demon@ubuntu2204-master:~$ 
demon@ubuntu2204-master:~$ kubelet --version
Kubernetes v1.25.3
demon@ubuntu2204-master:~$

了解后续需要的镜像版本:

demon@ubuntu2204-master:~$ sudo kubeadm config images list --kubernetes-version v1.25.3
registry.k8s.io/kube-apiserver:v1.25.3
registry.k8s.io/kube-controller-manager:v1.25.3
registry.k8s.io/kube-scheduler:v1.25.3
registry.k8s.io/kube-proxy:v1.25.3
registry.k8s.io/pause:3.8
registry.k8s.io/etcd:3.5.4-0
registry.k8s.io/coredns/coredns:v1.9.3
demon@ubuntu2204-master:~$

Step4: 初始化集群控制面

1、启动 kubelet,保证开机执行

$ sudo systemctl start kubelet
$ sudo systemctl enable kubelet

2、开始部署,这里切换到 root 账号,而不是使用 sudo

kubeadm init \
    --image-repository registry.cn-hangzhou.aliyuncs.com/google_containers \
    --pod-network-cidr=10.10.0.0/16 \
    --apiserver-advertise-address=192.168.10.128 \
    --kubernetes-version=v1.25.3 \
    --v=5

解释一下这里的几个参数:

  • --image-repository:从阿里云服务器上拉取上面需要的基础镜像,如果不设置,就得去 Google 服务器拉取;
  • --pod-network-cidr:设置集群中 Pod 的网络地址段,这是给后面安装网络插件 Flannel 使用的;
  • --kubernetes-version:指定 Kubernetes 版本;
  • --v=5:显示详细的跟踪日志,可以参考这里
  • --apiserver-advertise-address:指定 api-server 的 IP 地址,如果有多张网卡,请明确选择哪张网卡。由于 apiserver 在 Kubernetes 集群中有很重要的地位,很多配置(如 ConfigMap 资源等)都直接存储了该地址,后续更改起来十分麻烦,所以要慎重。


安装日志:

root@ubuntu2204-master:~# kubeadm init \
    --image-repository registry.cn-hangzhou.aliyuncs.com/google_containers \
    --pod-network-cidr=10.10.0.0/16 \
    --apiserver-advertise-address=192.168.10.128 \
    --kubernetes-version=v1.25.3 \
>   --v=5
I1116 15:56:26.815643    2036 initconfiguration.go:116] detected and using CRI socket: unix:///var/run/containerd/containerd.sock
I1116 15:56:26.815827    2036 kubelet.go:196] the value of KubeletConfiguration.cgroupDriver is empty; setting it to "systemd"
[init] Using Kubernetes version: v1.25.3
[preflight] Running pre-flight checks
I1116 15:56:26.819499    2036 checks.go:568] validating Kubernetes and kubeadm version
I1116 15:56:26.819566    2036 checks.go:168] validating if the firewall is enabled and active
I1116 15:56:26.825072    2036 checks.go:203] validating availability of port 6443
I1116 15:56:26.825237    2036 checks.go:203] validating availability of port 10259
I1116 15:56:26.825299    2036 checks.go:203] validating availability of port 10257
I1116 15:56:26.825384    2036 checks.go:280] validating the existence of file /etc/kubernetes/manifests/kube-apiserver.yaml
I1116 15:56:26.825516    2036 checks.go:280] validating the existence of file /etc/kubernetes/manifests/kube-controller-manager.yaml
I1116 15:56:26.825620    2036 checks.go:280] validating the existence of file /etc/kubernetes/manifests/kube-scheduler.yaml
I1116 15:56:26.825673    2036 checks.go:280] validating the existence of file /etc/kubernetes/manifests/etcd.yaml
I1116 15:56:26.825728    2036 checks.go:430] validating if the connectivity type is via proxy or direct
I1116 15:56:26.825843    2036 checks.go:469] validating http connectivity to first IP address in the CIDR
I1116 15:56:26.825911    2036 checks.go:469] validating http connectivity to first IP address in the CIDR
I1116 15:56:26.825960    2036 checks.go:104] validating the container runtime
I1116 15:56:26.887241    2036 checks.go:329] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I1116 15:56:26.887388    2036 checks.go:329] validating the contents of file /proc/sys/net/ipv4/ip_forward
I1116 15:56:26.887405    2036 checks.go:644] validating whether swap is enabled or not
I1116 15:56:26.887455    2036 checks.go:370] validating the presence of executable crictl
I1116 15:56:26.887494    2036 checks.go:370] validating the presence of executable conntrack
I1116 15:56:26.887521    2036 checks.go:370] validating the presence of executable ip
I1116 15:56:26.887557    2036 checks.go:370] validating the presence of executable iptables
I1116 15:56:26.887614    2036 checks.go:370] validating the presence of executable mount
I1116 15:56:26.887661    2036 checks.go:370] validating the presence of executable nsenter
I1116 15:56:26.887701    2036 checks.go:370] validating the presence of executable ebtables
I1116 15:56:26.887738    2036 checks.go:370] validating the presence of executable ethtool
I1116 15:56:26.887766    2036 checks.go:370] validating the presence of executable socat
I1116 15:56:26.887809    2036 checks.go:370] validating the presence of executable tc
I1116 15:56:26.887844    2036 checks.go:370] validating the presence of executable touch
I1116 15:56:26.887855    2036 checks.go:516] running all checks
        [WARNING SystemVerification]: missing optional cgroups: blkio
I1116 15:56:26.896081    2036 checks.go:401] checking whether the given node name is valid and reachable using net.LookupHost
I1116 15:56:26.896538    2036 checks.go:610] validating kubelet version
I1116 15:56:26.935223    2036 checks.go:130] validating if the "kubelet" service is enabled and active
I1116 15:56:26.947920    2036 checks.go:203] validating availability of port 10250
I1116 15:56:26.948061    2036 checks.go:203] validating availability of port 2379
I1116 15:56:26.948111    2036 checks.go:203] validating availability of port 2380
I1116 15:56:26.948157    2036 checks.go:243] validating the existence and emptiness of directory /var/lib/etcd
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
I1116 15:56:26.948474    2036 checks.go:832] using image pull policy: IfNotPresent
I1116 15:56:26.964706    2036 checks.go:841] image exists: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.25.3
I1116 15:56:26.978755    2036 checks.go:841] image exists: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.25.3
I1116 15:56:27.000585    2036 checks.go:841] image exists: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.25.3
I1116 15:56:27.017285    2036 checks.go:841] image exists: registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.25.3
I1116 15:56:27.032953    2036 checks.go:841] image exists: registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.8
I1116 15:56:27.048539    2036 checks.go:841] image exists: registry.cn-hangzhou.aliyuncs.com/google_containers/etcd:3.5.4-0
I1116 15:56:27.064166    2036 checks.go:841] image exists: registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:v1.9.3
[certs] Using certificateDir folder "/etc/kubernetes/pki"
I1116 15:56:27.064309    2036 certs.go:112] creating a new certificate authority for ca
[certs] Generating "ca" certificate and key
I1116 15:56:27.115296    2036 certs.go:522] validating certificate period for ca certificate
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local ubuntu2204-master] and IPs [10.96.0.1 192.168.10.128]
[certs] Generating "apiserver-kubelet-client" certificate and key
I1116 15:56:27.397793    2036 certs.go:112] creating a new certificate authority for front-proxy-ca
[certs] Generating "front-proxy-ca" certificate and key
I1116 15:56:27.611526    2036 certs.go:522] validating certificate period for front-proxy-ca certificate
[certs] Generating "front-proxy-client" certificate and key
I1116 15:56:27.756970    2036 certs.go:112] creating a new certificate authority for etcd-ca
[certs] Generating "etcd/ca" certificate and key
I1116 15:56:27.825701    2036 certs.go:522] validating certificate period for etcd/ca certificate
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [localhost ubuntu2204-master] and IPs [192.168.10.128 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [localhost ubuntu2204-master] and IPs [192.168.10.128 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
I1116 15:56:28.545309    2036 certs.go:78] creating new public/private key files for signing service account users
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I1116 15:56:28.708973    2036 kubeconfig.go:103] creating kubeconfig file for admin.conf
[kubeconfig] Writing "admin.conf" kubeconfig file
I1116 15:56:28.784633    2036 kubeconfig.go:103] creating kubeconfig file for kubelet.conf
[kubeconfig] Writing "kubelet.conf" kubeconfig file
I1116 15:56:28.848427    2036 kubeconfig.go:103] creating kubeconfig file for controller-manager.conf
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
I1116 15:56:29.059470    2036 kubeconfig.go:103] creating kubeconfig file for scheduler.conf
[kubeconfig] Writing "scheduler.conf" kubeconfig file
I1116 15:56:29.185240    2036 kubelet.go:66] Stopping the kubelet
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
I1116 15:56:29.363351    2036 manifests.go:99] [control-plane] getting StaticPodSpecs
I1116 15:56:29.363501    2036 certs.go:522] validating certificate period for CA certificate
I1116 15:56:29.363535    2036 manifests.go:125] [control-plane] adding volume "ca-certs" for component "kube-apiserver"
I1116 15:56:29.363538    2036 manifests.go:125] [control-plane] adding volume "etc-ca-certificates" for component "kube-apiserver"
I1116 15:56:29.363541    2036 manifests.go:125] [control-plane] adding volume "etc-pki" for component "kube-apiserver"
I1116 15:56:29.363543    2036 manifests.go:125] [control-plane] adding volume "k8s-certs" for component "kube-apiserver"
I1116 15:56:29.363546    2036 manifests.go:125] [control-plane] adding volume "usr-local-share-ca-certificates" for component "kube-apiserver"
I1116 15:56:29.363549    2036 manifests.go:125] [control-plane] adding volume "usr-share-ca-certificates" for component "kube-apiserver"
I1116 15:56:29.364668    2036 manifests.go:154] [control-plane] wrote static Pod manifest for component "kube-apiserver" to "/etc/kubernetes/manifests/kube-apiserver.yaml"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
I1116 15:56:29.364676    2036 manifests.go:99] [control-plane] getting StaticPodSpecs
I1116 15:56:29.364760    2036 manifests.go:125] [control-plane] adding volume "ca-certs" for component "kube-controller-manager"
I1116 15:56:29.364764    2036 manifests.go:125] [control-plane] adding volume "etc-ca-certificates" for component "kube-controller-manager"
I1116 15:56:29.364766    2036 manifests.go:125] [control-plane] adding volume "etc-pki" for component "kube-controller-manager"
I1116 15:56:29.364769    2036 manifests.go:125] [control-plane] adding volume "flexvolume-dir" for component "kube-controller-manager"
I1116 15:56:29.364771    2036 manifests.go:125] [control-plane] adding volume "k8s-certs" for component "kube-controller-manager"
I1116 15:56:29.364773    2036 manifests.go:125] [control-plane] adding volume "kubeconfig" for component "kube-controller-manager"
I1116 15:56:29.364776    2036 manifests.go:125] [control-plane] adding volume "usr-local-share-ca-certificates" for component "kube-controller-manager"
I1116 15:56:29.364778    2036 manifests.go:125] [control-plane] adding volume "usr-share-ca-certificates" for component "kube-controller-manager"
I1116 15:56:29.365051    2036 manifests.go:154] [control-plane] wrote static Pod manifest for component "kube-controller-manager" to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[control-plane] Creating static Pod manifest for "kube-scheduler"
I1116 15:56:29.365056    2036 manifests.go:99] [control-plane] getting StaticPodSpecs
I1116 15:56:29.365124    2036 manifests.go:125] [control-plane] adding volume "kubeconfig" for component "kube-scheduler"
I1116 15:56:29.365267    2036 manifests.go:154] [control-plane] wrote static Pod manifest for component "kube-scheduler" to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I1116 15:56:29.365641    2036 local.go:65] [etcd] wrote Static Pod manifest for a local etcd member to "/etc/kubernetes/manifests/etcd.yaml"
I1116 15:56:29.365685    2036 waitcontrolplane.go:83] [wait-control-plane] Waiting for the API server to be healthy
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 4.501675 seconds
I1116 15:56:33.870817    2036 uploadconfig.go:110] [upload-config] Uploading the kubeadm ClusterConfiguration to a ConfigMap
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
I1116 15:56:33.876187    2036 uploadconfig.go:124] [upload-config] Uploading the kubelet component config to a ConfigMap
[kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster
I1116 15:56:33.880965    2036 uploadconfig.go:129] [upload-config] Preserving the CRISocket information for the control-plane node
I1116 15:56:33.880980    2036 patchnode.go:31] [patchnode] Uploading the CRI Socket information "unix:///var/run/containerd/containerd.sock" to the Node API object "ubuntu2204-master" as an annotation
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node ubuntu2204-master as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers]
[mark-control-plane] Marking the node ubuntu2204-master as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule]
[bootstrap-token] Using token: xic9xj.h4m8entuwgbz92m4
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
I1116 15:56:34.939896    2036 clusterinfo.go:47] [bootstrap-token] loading admin kubeconfig
I1116 15:56:34.940276    2036 clusterinfo.go:58] [bootstrap-token] copying the cluster from admin.conf to the bootstrap kubeconfig
I1116 15:56:34.940432    2036 clusterinfo.go:70] [bootstrap-token] creating/updating ConfigMap in kube-public namespace
I1116 15:56:34.942150    2036 clusterinfo.go:84] creating the RBAC rules for exposing the cluster-info ConfigMap in the kube-public namespace
I1116 15:56:34.945159    2036 kubeletfinalize.go:90] [kubelet-finalize] Assuming that kubelet client certificate rotation is enabled: found "/var/lib/kubelet/pki/kubelet-client-current.pem"
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
I1116 15:56:34.945791    2036 kubeletfinalize.go:134] [kubelet-finalize] Restarting the kubelet to enable client certificate rotation
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.10.128:6443 --token tu3oa9.o08w1z4c8yfrkoxi \
        --discovery-token-ca-cert-hash sha256:b09777337820602596abd84012da770ec8b1637dbdd157e73e765ba0c7790d2b
root@ubuntu2204-master:~#

3、按照上面日志最后部分给的提示进行 kube config 配置

 mkdir -p $HOME/.kube
 sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
 sudo chown $(id -u):$(id -g) $HOME/.kube/config

如果是 root 用户,可以将下面这行加入 .bashrc,重新登录后生效。

export KUBECONFIG=/etc/kubernetes/admin.conf

Step5: 安装网络插件

kubernetes 将底层容器网络抽象成了 CNI,即容器网络插件:

CNI (Container Network Interface), a Cloud Native Computing Foundation project, consists of a specification and libraries for writing plugins to configure network interfaces in Linux containers, along with a number of supported plugins. CNI concerns itself only with network connectivity of containers and removing allocated resources when the container is deleted. Because of this focus, CNI has a wide range of support and the specification is simple to implement.

官方列出了非常多的网络插件,比如 CalicoWeaveContivCilium 等,不同插件的底层原理和实现方式有所不同,但都遵循 CNI 规范。作为入门学习,笔者这里选择最简单的 Flannel 插件。我们从 Github 获取安装文件 kube-flannel.yml,然后对其进行修改:

net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }

将上面的 Network 调整为我们之前初始化集群时设置的 pod 网段:

net-conf.json: |
    {
      "Network": "10.10.0.0/16",
      "Backend": {
        "Type": "vxlan"
      }
    }

最后我们使用 kubectl apply 进行安装:

kubectl apply -f kube-flannel.yml

Step6:优化(可选)

移除 master 上的污点

移除污点是为了让 master 节点可以部署业务服务[1]

demon@ubuntu2204-master:~$ kubectl describe node ubuntu2204-master
Name:               ubuntu2204-master
Roles:              control-plane
Labels:             beta.kubernetes.io/arch=arm64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=arm64
                    kubernetes.io/hostname=ubuntu2204-master
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=
                    node.kubernetes.io/exclude-from-external-load-balancers=
Annotations:        flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"1a:9b:f1:7d:dd:c4"}
                    flannel.alpha.coreos.com/backend-type: vxlan
                    flannel.alpha.coreos.com/kube-subnet-manager: true
                    flannel.alpha.coreos.com/public-ip: 192.168.10.128
                    kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Wed, 16 Nov 2022 15:56:32 +0800
Taints:             node-role.kubernetes.io/control-plane:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  ubuntu2204-master
  AcquireTime:     <unset>
  RenewTime:       Wed, 16 Nov 2022 21:33:49 +0800
Conditions:
...
...
...

找到 Taints: node-role.kubernetes.io/control-plane:NoSchedule[2],然后将其移除:

demon@ubuntu2204-master:~$ kubectl taint nodes ubuntu2204-master node-role.kubernetes.io/control-plane:NoSchedule-
node/ubuntu2204-master untainted
demon@ubuntu2204-master:~$ 

启用 kubectl 命令自动补全功能

bash:

$ echo 'source <(kubectl completion bash)' >> ~/.bashrc
$ echo 'source /usr/share/bash-completion/bash_completion' >> ~/.bashrc

zsh:

$ echo 'source <(kubectl completion zsh)' >> ~/.zshrc

调整 NodePort 范围

Kubernetes 默认的 NodePort 范围为 30000-32767,本地开发调试可以将其调大一些。

  • 修改 /etc/kubernetes/manifests/kube-apiserver.yaml 文件,在 spec.containers.command 中增加一个参数 --service-node-port-range=5000-32767

  • 重启 apiserver(删除内置的 pod 即可)

export apiserver_pods=$(kubectl get pods --selector=component=kube-apiserver -n kube-system --output=jsonpath={.items..metadata.name})

kubectl delete pod $apiserver_pods -n kube-system
  • 检查结果
$ kubectl describe pod $apiserver_pods -n kube-system | grep 5000
--service-node-port-range=5000-32767

Step7:增加 worker 节点

worker 节点的基础安装部分与 master 节点一致,我们安装到 Step 3 就够了。

接着将 kubelet 修改为默认启动:

sudo systemctl enable kubelet

然后在当前 worker 节点上运行下面的命令加入集群:

kubeadm join 192.168.10.128:6443 --token tu3oa9.o08w1z4c8yfrkoxi \
        --discovery-token-ca-cert-hash sha256:b09777337820602596abd84012da770ec8b1637dbdd157e73e765ba0c7790d2b

该命令来自前面 Step 4kubeadm init 最后的日志:

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.10.128:6443 --token tu3oa9.o08w1z4c8yfrkoxi \
        --discovery-token-ca-cert-hash sha256:b09777337820602596abd84012da770ec8b1637dbdd157e73e765ba0c7790d2b

上面的 token 有效期为 24 小时,如果过期可以使用如下命令重新生成:

kubeadm token create --print-join-command

worker 节点加入日志:

root@ubuntu2204-worker:~# kubeadm join 192.168.10.128:6443 --token tu3oa9.o08w1z4c8yfrkoxi         --discovery-token-ca-cert-hash sha256:b09777337820602596abd84012da770ec8b1637dbdd157e73e765ba0c7790d2b      --v=5
I1117 09:49:35.953178    2006 join.go:416] [preflight] found NodeName empty; using OS hostname as NodeName
I1117 09:49:35.953532    2006 initconfiguration.go:116] detected and using CRI socket: unix:///var/run/containerd/containerd.sock
[preflight] Running pre-flight checks
I1117 09:49:35.953605    2006 preflight.go:92] [preflight] Running general checks
I1117 09:49:35.953661    2006 checks.go:280] validating the existence of file /etc/kubernetes/kubelet.conf
I1117 09:49:35.953673    2006 checks.go:280] validating the existence of file /etc/kubernetes/bootstrap-kubelet.conf
I1117 09:49:35.953710    2006 checks.go:104] validating the container runtime
I1117 09:49:35.974555    2006 checks.go:329] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables
I1117 09:49:35.974601    2006 checks.go:329] validating the contents of file /proc/sys/net/ipv4/ip_forward
I1117 09:49:35.974626    2006 checks.go:644] validating whether swap is enabled or not
I1117 09:49:35.974645    2006 checks.go:370] validating the presence of executable crictl
I1117 09:49:35.974659    2006 checks.go:370] validating the presence of executable conntrack
I1117 09:49:35.974666    2006 checks.go:370] validating the presence of executable ip
I1117 09:49:35.974674    2006 checks.go:370] validating the presence of executable iptables
I1117 09:49:35.974682    2006 checks.go:370] validating the presence of executable mount
I1117 09:49:35.974689    2006 checks.go:370] validating the presence of executable nsenter
I1117 09:49:35.974698    2006 checks.go:370] validating the presence of executable ebtables
I1117 09:49:35.974707    2006 checks.go:370] validating the presence of executable ethtool
I1117 09:49:35.974714    2006 checks.go:370] validating the presence of executable socat
I1117 09:49:35.974720    2006 checks.go:370] validating the presence of executable tc
I1117 09:49:35.974725    2006 checks.go:370] validating the presence of executable touch
I1117 09:49:35.974750    2006 checks.go:516] running all checks
        [WARNING SystemVerification]: missing optional cgroups: blkio
I1117 09:49:35.983005    2006 checks.go:401] checking whether the given node name is valid and reachable using net.LookupHost
I1117 09:49:35.984761    2006 checks.go:610] validating kubelet version
I1117 09:49:36.030854    2006 checks.go:130] validating if the "kubelet" service is enabled and active
I1117 09:49:36.038685    2006 checks.go:203] validating availability of port 10250
I1117 09:49:36.038781    2006 checks.go:280] validating the existence of file /etc/kubernetes/pki/ca.crt
I1117 09:49:36.038789    2006 checks.go:430] validating if the connectivity type is via proxy or direct
I1117 09:49:36.038821    2006 join.go:533] [preflight] Discovering cluster-info
I1117 09:49:36.038862    2006 token.go:80] [discovery] Created cluster-info discovery client, requesting info from "192.168.10.128:6443"
I1117 09:49:36.050731    2006 token.go:118] [discovery] Requesting info from "192.168.10.128:6443" again to validate TLS against the pinned public key
I1117 09:49:36.054719    2006 token.go:135] [discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.10.128:6443"
I1117 09:49:36.054738    2006 discovery.go:52] [discovery] Using provided TLSBootstrapToken as authentication credentials for the join process
I1117 09:49:36.054743    2006 join.go:547] [preflight] Fetching init configuration
I1117 09:49:36.054747    2006 join.go:593] [preflight] Retrieving KubeConfig objects
[preflight] Reading configuration from the cluster...
[preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
I1117 09:49:36.057469    2006 kubelet.go:74] attempting to download the KubeletConfiguration from ConfigMap "kubelet-config"
I1117 09:49:36.059173    2006 interface.go:432] Looking for default routes with IPv4 addresses
I1117 09:49:36.059196    2006 interface.go:437] Default route transits interface "ens160"
I1117 09:49:36.059274    2006 interface.go:209] Interface ens160 is up
I1117 09:49:36.059298    2006 interface.go:257] Interface "ens160" has 2 addresses :[192.168.10.129/24 fe80::b87b:4a27:cd99:16ee/64].
I1117 09:49:36.059309    2006 interface.go:224] Checking addr  192.168.10.129/24.
I1117 09:49:36.059313    2006 interface.go:231] IP found 192.168.10.129
I1117 09:49:36.059316    2006 interface.go:263] Found valid IPv4 address 192.168.10.129 for interface "ens160".
I1117 09:49:36.059320    2006 interface.go:443] Found active IP 192.168.10.129 
I1117 09:49:36.062265    2006 preflight.go:103] [preflight] Running configuration dependant checks
I1117 09:49:36.062285    2006 controlplaneprepare.go:220] [download-certs] Skipping certs download
I1117 09:49:36.062290    2006 kubelet.go:120] [kubelet-start] writing bootstrap kubelet config file at /etc/kubernetes/bootstrap-kubelet.conf
I1117 09:49:36.062705    2006 kubelet.go:135] [kubelet-start] writing CA certificate at /etc/kubernetes/pki/ca.crt
I1117 09:49:36.062997    2006 kubelet.go:156] [kubelet-start] Checking for an existing Node in the cluster with name "ubuntu2204-worker" and status "Ready"
I1117 09:49:36.066958    2006 kubelet.go:171] [kubelet-start] Stopping the kubelet
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Starting the kubelet
[kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap...
I1117 09:49:41.283287    2006 cert_rotation.go:137] Starting client certificate rotation controller
I1117 09:49:41.285117    2006 kubelet.go:219] [kubelet-start] preserving the crisocket information for the node
I1117 09:49:41.285176    2006 patchnode.go:31] [patchnode] Uploading the CRI Socket information "unix:///var/run/containerd/containerd.sock" to the Node API object "ubuntu2204-worker" as an annotation

This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

root@ubuntu2204-worker:~# 

最后,如果想在 worker 节点上运行 kubectl 命令,需要将 master 节点上的 ~/.kube/config 复制过来,然后便可以运行以下命令查看节点信息:

kubectl get nodes

小结

相信通过这些步骤,大家开始能体会到其内部的复杂度了,确实不亚于安装一个操作系统。笔者这里只贴出了主要的执行步骤,其实还有很多可选操作,比如 Metrics Server,Dashboard 等,将来有机会,笔者会在其他文章中进行补充。

不过,话又说回来,这么复杂的东西都是由计算机基础知识构建起来的,比如计算机网络、操作系统等。正如一座庞大的建筑是由一砖一瓦搭起来的,而这一砖一瓦就是基本功。所以,构建基本功是每个计算机工程师、架构师不可或缺的能力。

写到这里,突然想起一句话:所谓才华,就是基本功的溢出。


  1. 这里只是为了学习,所以将污点移除,生产环境不建议移除。另外,我们可以在部署业务时,在 Pod 里设置 tolerations 来“容忍”某些“污点”,也能达到相同的效果。 ↩︎

  2. 在 Kubernetes 1.24 之前,master 使用 node-role.kubernetes.io/master:NoSchedule 来表示不被调度的污点,之后便调整成了 node-role.kubernetes.io/control-plane:NoSchedule↩︎