Comments

Kubernetes Deploy

本文记载Kubernetes通过kubeadm方式安装的详细过程及遇到的问题与解决方法。
更新于2020-08-02

Kubernetes源

考虑到国内的网络环境，采用 阿里云 mirrors.aliyun.com 的源.

https://developer.aliyun.com/mirror/kubernetes
下载地址：https://mirrors.aliyun.com/kubernetes/
官方主页：https://kubernetes.io/

aliyun-kubernetes.repo
aliyun-docker-ce.repo

yum install docker-ce
yum install -y kubelet-1.18.2 kubeadm-1.18.2 kubectl-1.18.2

(1)借助NTP服务设置节点时间精确同步;
(2)通过DNS完成各节点的主机名解析，测试环境主机数量较少时也可以使用hosts文件进行;
(3)关闭各节点的iptables或firewalld服务,并确保它们被禁止随系统引导过程启动;
(4)各节点禁用Selinux(否则在运行容器时可以会遇到各种奇葩报错);
(5)各节点禁用所有的swap设备(生产环境中强烈建议禁用,虽说K8S也支持配置参数来启用swap但这样会降低集群性能，使用”swapoff -a”只是临时关闭交换分区使用,永久关闭需要编辑”/etc/fstab”文件将挂载swap哪一行前面加一个”#”进行注释);
(6)若要使用ipvs模型的proxy，各节点还需要载入ipvs相关的各模块

1. Kubernetes Repo

/etc/yum.repos.d/kubernetes-aliyun.repo

1
2
3
4
5
6
7
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

2. Yum install Kubeadm

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
setenforce 0
yum install -y kubelet-1.18.2 kubeadm-1.18.2 kubectl-1.18.2
yum install -y --nogpgcheck kubelet kubeadm kubectl
systemctl enable kubelet && systemctl start kubelet

[root@master01 ~]# kubeadm config images list
W0727 15:53:36.203783   21851 version.go:102] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W0727 15:53:36.203925   21851 version.go:103] falling back to the local client version: v1.18.2
W0727 15:53:36.204122   21851 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
k8s.gcr.io/kube-apiserver:v1.18.2
k8s.gcr.io/kube-controller-manager:v1.18.2
k8s.gcr.io/kube-scheduler:v1.18.2
k8s.gcr.io/kube-proxy:v1.18.2
k8s.gcr.io/pause:3.2
k8s.gcr.io/etcd:3.4.3-0
k8s.gcr.io/coredns:1.6.7

3. 调整docker

1
2
3
4
5
6
7
8
docker info

cat >/etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "registry-mirrors": ["https://2325obkh.mirror.aliyuncs.com"]
}
EOF

docker自1.13版起会自动设置iptable的FORWARD默认策略为DROP，这可能会影响kubernetes集群依赖的报文转发功能,因此，需要在docker服务启动后，重新将FORWARD链的默认策略设置为ACCEPT。

vim /usr/lib/systemd/system/docker.service”
在”ExecStart=/usr/bin/dockerd -H fd:// –containerd=/run/containerd/containerd.sock”之后
新增一行”ExecStartPost=/usr/sbin/iptables -P FORWARD ACCEPT”(意思是docker服务器启动成功后会执行执行该命令)

1
2
vim /etc/sysconfig/kubelet
KUBELET_EXTRA_ARGS="--fail-swap-on=false"

systemctl daemon-reload

4. 调整系统参数

1
2
3
4
5
yum install chrony
chronyc sourcestats -v
swapoff -a
setenforce=0
systemctl stop firewalld.service

加载ipvs模块

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
vim /etc/sysconfig/modules/ipvs.modules

#!/usr/bin/env bash

ipvs_mods_dir="/usr/lib/modules/$(uname -r)/kernel/net/netfilter/ipvs"

for mod in $(ls $ipvs_mods_dir | grep -o "^[^.]*");do
    /usr/sbin/modinfo -F filename $mod &> /dev/null
    if [ $? -eq 0 ];then
        /sbin/modprobe $mod
    fi
done

# master与node上均执行
chmod +x /etc/sysconfig/modules/ipvs.modules

lsmod | grep ip_vs

kubeadm config print init-defaults

kubeadm config images list
kubeadm config images pull

docker images
docker image ls
docker image tag old_image:version new_image:version
docker image pull
docker image rm

registry.cn-hangzhou.aliyuncs.com__google_containers__kube-proxy__v1.18.2.tar = registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.18.2 =
registry.cn-hangzhou.aliyuncs.com__google_containers__pause__v3.2.tar = registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.2 = k8s.gcr.io/pause:3.2
registry.aliyuncs.com__google_containers__etcd__v3.4.3-0.tar = registry.aliyuncs.com/google_containers/etcd:3.4.3-0 = k8s.gcr.io/etcd:3.4.3-0
registry.cn-hangzhou.aliyuncs.com__google_containers__coredns__v1.6.7.tar = registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.6.7 = k8s.gcr.io/coredns:1.6.7

docker save registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy -o registry.cn-hangzhou.aliyuncs.com__google_containers__kube-proxy__v1.18.2.tar
docker save registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver -o registry.cn-hangzhou.aliyuncs.com__google_containers__kube-apiserver__v1.18.2.tar
docker save registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager -o registry.cn-hangzhou.aliyuncs.com__google_containers__kube-controller-manager__v1.18.2.tar
docker save registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler -o registry.cn-hangzhou.aliyuncs.com__google_containers__kube-scheduler__v1.18.2.tar
docker save registry.cn-hangzhou.aliyuncs.com/google_containers/pause -o registry.cn-hangzhou.aliyuncs.com__google_containers__pause__v3.2.tar
docker save registry.aliyuncs.com/google_containers/etcd -o registry.aliyuncs.com__google_containers__etcd__v3.4.3-0.tar
docker save registry.cn-hangzhou.aliyuncs.com/google_containers/coredns -o registry.cn-hangzhou.aliyuncs.com__google_containers__coredns__v1.6.7.tar

docker image tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-proxy:v1.18.2 k8s.gcr.io/kube-proxy:v1.18.2
docker image tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-apiserver:v1.18.2 k8s.gcr.io/kube-apiserver:v1.18.2
docker image tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-controller-manager:v1.18.2 k8s.gcr.io/kube-controller-manager:v1.18.2
docker image tag registry.cn-hangzhou.aliyuncs.com/google_containers/kube-scheduler:v1.18.2 k8s.gcr.io/kube-scheduler:v1.18.2
docker image tag registry.aliyuncs.com/google_containers/pause:3.2 k8s.gcr.io/pause:3.2
docker image tag registry.aliyuncs.com/google_containers/etcd:3.4.3-0 k8s.gcr.io/etcd:3.4.3-0
docker image tag registry.cn-hangzhou.aliyuncs.com/google_containers/coredns:1.6.7 k8s.gcr.io/coredns:1.6.7

[root@master01 ~]# docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
k8s.gcr.io/kube-proxy v1.18.2 0d40868643c6 3 months ago 117MB
k8s.gcr.io/kube-controller-manager v1.18.2 ace0a8c17ba9 3 months ago 162MB
k8s.gcr.io/kube-scheduler v1.18.2 a3099161e137 3 months ago 95.3MB
k8s.gcr.io/kube-apiserver v1.18.2 6ed75ad404bd 3 months ago 173MB
k8s.gcr.io/pause 3.2 80d28bedfe5d 5 months ago 683kB
k8s.gcr.io/coredns 1.6.7 67da37a9a360 6 months ago 43.8MB
k8s.gcr.io/etcd 3.4.3-0 303ce5db0e90 9 months ago 288MB

docker import registry.aliyuncs.com__google_containers__etcd__v3.4.3-0.tar k8s.gcr.io/etcd:3.4.3-0
docker import registry.cn-hangzhou.aliyuncs.com__google_containers__coredns__v1.6.7.tar k8s.gcr.io/coredns:1.6.7
docker import registry.cn-hangzhou.aliyuncs.com__google_containers__kube-apiserver__v1.18.2.tar k8s.gcr.io/kube-apiserver:v1.18.2
docker import registry.cn-hangzhou.aliyuncs.com__google_containers__kube-controller-manager__v1.18.2.tar k8s.gcr.io/kube-controller-manager:v1.18.2
docker import registry.cn-hangzhou.aliyuncs.com__google_containers__kube-proxy__v1.18.2.tar k8s.gcr.io/kube-proxy:v1.18.2
docker import registry.cn-hangzhou.aliyuncs.com__google_containers__kube-scheduler__v1.18.2.tar k8s.gcr.io/kube-scheduler:v1.18.2
docker import registry.cn-hangzhou.aliyuncs.com__google_containers__pause__v3.2.tar k8s.gcr.io/pause:3.2

初始化CLuster

初始测试dry-run

kubeadm init
–apiserver-advertise-address=192.168.221.134
–image-repository registry.aliyuncs.com/google_containers
–kubernetes-version v1.18.2
–pod-network-cidr=10.244.0.0/16
–dry-run

init cluster

kubeadm init
–apiserver-advertise-address=192.168.221.134
–image-repository registry.aliyuncs.com/google_containers
–kubernetes-version v1.18.2
–pod-network-cidr=10.244.0.0/16

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
[root@master01 tmp]#  kubeadm init --apiserver-advertise-address=192.168.221.134 --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.18.2 --pod-network-cidr=10.244.0.0/16
W0727 16:40:13.862004   32012 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.18.2
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [master01 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.221.134]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [master01 localhost] and IPs [192.168.221.134 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [master01 localhost] and IPs [192.168.221.134 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
W0727 16:40:22.595129   32012 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[control-plane] Creating static Pod manifest for "kube-scheduler"
W0727 16:40:22.596079   32012 manifests.go:225] the default kube-apiserver authorization-mode is "Node,RBAC"; using "Node,RBAC"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 24.507077 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.18" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Skipping phase. Please see --upload-certs
[mark-control-plane] Marking the node master01 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node master01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: a4imyy.ttddd23g8ddh7dv7
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.221.134:6443 --token a4imyy.ttddd23g8ddh7dv7 \
    --discovery-token-ca-cert-hash sha256:4f784226e480a6de709b1f21e6e67ec02050942e634968002c1b86e670db9b0c 
[root@master01 tmp]#

kubeadm init –kubernetes-version=v1.14.2 –pod-network-cidr=10.244.0.0/16 –service-cidr=10.96.0.0/12 –ignore-preflight-errors=Swap

检查kubelet.service 状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
[root@master01 ~]# systemctl status kubelet -l
● kubelet.service - kubelet: The Kubernetes Node Agent
   Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /usr/lib/systemd/system/kubelet.service.d
           └─10-kubeadm.conf
   Active: active (running) since Mon 2020-07-27 16:40:53 CST; 1min 25s ago
     Docs: https://kubernetes.io/docs/
 Main PID: 33413 (kubelet)
    Tasks: 16
   Memory: 74.7M
   CGroup: /system.slice/kubelet.service
           └─33413 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --config=/var/lib/kubelet/config.yaml --cgroup-driver=systemd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.2 --fail-swap-on=false

Jul 27 16:41:54 master01 kubelet[33413]: E0727 16:41:54.116034   33413 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Jul 27 16:41:55 master01 kubelet[33413]: W0727 16:41:55.025914   33413 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
Jul 27 16:41:59 master01 kubelet[33413]: E0727 16:41:59.233911   33413 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Jul 27 16:42:00 master01 kubelet[33413]: W0727 16:42:00.028419   33413 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
Jul 27 16:42:04 master01 kubelet[33413]: E0727 16:42:04.354686   33413 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Jul 27 16:42:05 master01 kubelet[33413]: W0727 16:42:05.029246   33413 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
Jul 27 16:42:09 master01 kubelet[33413]: E0727 16:42:09.444976   33413 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Jul 27 16:42:10 master01 kubelet[33413]: W0727 16:42:10.048128   33413 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
Jul 27 16:42:14 master01 kubelet[33413]: E0727 16:42:14.551861   33413 kubelet.go:2187] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:docker: network plugin is not ready: cni config uninitialized
Jul 27 16:42:15 master01 kubelet[33413]: W0727 16:42:15.049274   33413 cni.go:237] Unable to update cni config: no networks found in /etc/cni/net.d
[root@master01 ~]# 


## 部署Network Plugin

mkdir .kube
cp /etc/kubernetes/admin.conf .kube/config

```shell
[root@master01 ~]# kubectl get nodes
NAME       STATUS     ROLES    AGE   VERSION
master01   NotReady   master   11m   v1.18.2
# 添加一个node后
[root@master01 ~]# kubectl get nodes
NAME              STATUS     ROLES    AGE   VERSION
centos77minimal   NotReady   <none>   41s   v1.18.2
master01          NotReady   master   22m   v1.18.2

node 添加ipvs.modules
node 导入k8s-image.tar
cp master:/etc/sysconfig/kubelet node

kubeadm reset
kubeadm token create –print-join-command
kubectl get pods -n kube-system

查看pod产生的日志

kubectl logs -f calico-node-5rgfg -n kube-system

kubectl describe pod calico-node-5rgfg -n kube-system 查看pod的具体描述

kubectl -n kube-system get ev 查看集群环境输出

flannel

sysctl net.bridge.bridge-nf-call-iptables=1

素丶素丶丶wodetian

忍一时越想越气，退一步越想越亏。