Kubernetes外部etcd部署

资源列表

基础环境

• 关闭防火墙

systemctl stop firewalld
systemctl disable firewalld

• 关闭内核安全机制

setenforce 0
sed -i "s/^SELINUX=.*/SELINUX=disabled/g" /etc/selinux/config

• 修改主机名

hostnamectl set-hostname master
hostnamectl set-hostname node01
hostnamectl set-hostname node02
hostnamectl set-hostname etcd

一、安装部署ETCD

如果是在测试或者开发环境，想要测试和使用etcd服务，只需要部署一个单节点的etcd服务即可。

1.1、二进制部署

• 安装软件

[root@etcd ~]# tar -zxvf etcd-v3.5.9-linux-amd64.tar.gz
[root@etcd ~]# mv etcd-v3.5.9-linux-amd64 /usr/local/etcd

• 添加环境变量

[root@etcd ~]# vim /etc/profile
# 末尾添加即可
export PATH="$PATH:/usr/local/etcd"

# 刷新环境变量
[root@etcd ~]# source /etc/profile

• 验证客户端工具是否可用

[root@etcd ~]# etcdctl version
etcdctl version: 3.5.9
API version: 3.5

1.2、添加system服务管理

# 创建配置文件目录
[root@etcd ~]# mkdir -p /etc/etcd

# 创建数据目录里
[root@etcd ~]# mkdir -p /etc/etcd/data

• 创建system服务配置

[root@etcd ~]# cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=/etc/etcd/etcd.conf
ExecStart=/usr/local/etcd/etcd --config-file=/etc/etcd/etcd.conf
Restart=on-failure
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

[root@etcd ~]# systemctl daemon-reload

二、部署TLS加密

• etcd支持通过TLS协议的加密通信，在实际企业生产环境中，出于安全规范要求，建议开启TLS加密。TLS通道可以用于加密内部的集群通讯，也可以用于加密客户端请求。
• etcd的TLS有两对，一对是etcd和client端的TLS配置。一对是etcd之间的peer的TLS配置。有很多方式可以创建CA证书和私钥，其中比较流行的有两种：
• openssl
• cfssl
• 官方文档推荐使用cfssl生成证书

2.1、下载安装cfssl

• 下载地址： https://github.com/cloudflare/cfssl/releases

[root@etcd ~]# wget https://github.com/cloudflare/cfssl/releases/download/v1.6.3/cfssl_1.6.3_linux_amd64
[root@etcd ~]# wget https://github.com/cloudflare/cfssl/releases/download/v1.6.3/cfssljson_1.6.3_linux_amd64

[root@etcd ~]# mv cfssl_1.6.3_linux_amd64 /usr/bin/cfssl
[root@etcd ~]# mv cfssljson_1.6.3_linux_amd64 /usr/bin/cfssljson

[root@etcd ~]# chmod +x /usr/bin/{cfssl,cfssljson}
[root@etcd ~]# cfssl version
Version: 1.6.3
Runtime: go1.18

2.2、创建默认配置文件

[root@etcd ~]# cfssl print-defaults config > ca-config.json
[root@etcd ~]# cfssl print-defaults csr > ca-csr.json

2.3、证书类型

• 客户端证书用于服务器验证客户端身份
• 服务器证书用于客户端验证服务器端身份
• 对等证书由etcd集群成员使用，同时使用客户端认证和服务器端认证

2.4、创建CA证书

• 由于各个组件都需要配置证书，并且依赖CA证书来签发证书，所以我们首先要生成CA证书以及后续的签发配置文件

[root@etcd ~]# cd /etc/etcd/
[root@etcd etcd]# mkdir -p etcdca
[root@etcd etcd]# cd etcdca/

• 修改ca-config配置

[root@etcd etcdca]# cat ca-config.json
{
  "signing": {
    "default": {
      "expiry": "43800h"
    },
    "profiles": {
      "server": {
        "expiry": "43800h",
        "usages": [
          "signing",
          "key encipherment",
          "server auth"
        ]
      },
      "client": {
        "expiry": "43800h",
        "usages": [
          "signing",
          "key encipherment",
          "client auth"
        ]
      },
      "peer": {
        "expiry": "43800h",
        "usages": [
          "signing",
          "key encipherment",
          "server auth",
          "client auth"
        ]
      }
    }
  }
}

• 配置证书请求

[root@etcd etcdca]# cat ca-csr.json
{
  "CN": "Etcd",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "BeiJing",
      "L": "BeiJing",
      "O": "Etcd",
      "OU": "CA"
    }
  ]
}

• 生成CA证书

[root@etcd etcdca]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca -
[root@etcd etcdca]# ls
ca-config.json ca.csr ca-csr.json ca-key.pem ca.pem

### 生成的文件中有下面三个后面会用到:
# ca-key.pem: CA证书密钥
# ca.pem: CA证书
# ca-config.json: 证书签发配置, 用CA证书来签发其他证书时需要用csr文件字段解释
# CN: Common Name, apiserver从证书中提取该字段作为请求的用户名
# Organization, apiserver从证书中提取该字段作为请求用户所属的组(Group)

• 由于这里是CA证书, 是签发其他证书的根证书, 这个证书密钥不会分发出去作为client证书, 所有组件使用的client证书都是由CA证书签发而来, 所以CA证书的CN和O的名称才是有用的。

2.5、生成服务器端证书

• 注意hosts字段需要加上etcd全部节点的IP/主机名信息以及127.0.0.1

[root@etcd etcdca]# cat server-csr.json
{
  "CN": "server",
  "hosts": [
    "127.0.0.1",
    "192.168.93.104"
  ],
  "key": {
    "algo": "ecdsa",
    "size": 256
  },
  "names": [
    {
      "C": "CN",
      "L": "Shanghai",
      "ST": "Shanghai"
    }
  ]
}

# 创建服务器端证书和私钥
[root@etcd etcdca]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server server-csr.json | cfssljson -bare server

# 生成以下文件
[root@etcd etcdca]# ls server*
server.csr server-csr.json server-key.pem server.pem

2.6、生成客户端证书

[root@etcd etcdca]# cat client-csr.json
{
  "CN": "client",
  "hosts": [
    ""
  ],
  "key": {
    "algo": "ecdsa",
    "size": 256
  },
  "names": [
    {
      "C": "CN",
      "L": "Shanghai",
      "ST": "Shanghai"
    }
  ]
}

# 创建客户端证书和私钥
[root@etcd etcdca]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client-csr.json | cfssljson -bare client

# 生成以下文件
[root@etcd etcdca]# ls client*
client.csr client-csr.json client-key.pem client.pem

2.7、生成对等证书

• peer证书可以统一，也可以分别生成，如果需要统一，则需要在hosts字段加上所有节点的IP/主机名信息，如果分开生成，则hosts字段只需要填写对应节点的IP/主机名信息即可

[root@etcd etcdca]# cat peer-csr.json
{
  "CN": "peer",
  "hosts": [
    "192.168.93.104"
  ],
  "key": {
    "algo": "ecdsa",
    "size": 256
  },
  "names": [
    {
      "C": "CN",
      "L": "Shanghai",
      "ST": "Shanghai"
    }
  ]
}

# 创建对等证书和私钥
[root@etcd etcdca]# cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer peer-csr.json | cfssljson -bare peer

# 生成以下文件
[root@etcd etcdca]# ls peer*
peer.csr peer-csr.json peer-key.pem peer.pem

2.8、更新系统证书库

[root@etcd etcdca]# yum install ca-certificates -y
[root@etcd etcdca]# update-ca-trust

2.9、修改etcd配置并启动etcd

• 主要是将原本的http链接全部改为https，并指定证书密钥地址

[root@etcd etcdca]# cat /etc/etcd/etcd.conf
# 节点名称
name: 'etcd-1'

# 指定节点的数据存储目录
data-dir: '/etc/etcd/data'

# 对外提供服务的地址（客户端连接地址）—— 改为 https
listen-client-urls: 'https://192.168.93.104:2379,https://127.0.0.1:2379'

# 广播给客户端的地址（其他组件如 kube-apiserver 会用这个地址连你）
advertise-client-urls: 'https://192.168.93.104:2379'

# Peer 通信地址（集群内部通信，单节点也需配置）—— 改为 https
listen-peer-urls: 'https://192.168.93.104:2380'
initial-advertise-peer-urls: 'https://192.168.93.104:2380'

# 集群初始化配置（单节点）
initial-cluster: 'etcd-1=https://192.168.93.104:2380'
initial-cluster-token: 'etcd-cluster-1'
initial-cluster-state: 'new'

# 客户端加密
client-transport-security:
  cert-file: "/etc/etcd/etcdca/server.pem"
  key-file: "/etc/etcd/etcdca/server-key.pem"
  client-cert-auth: True
  trusted-ca-file: "/etc/etcd/etcdca/ca.pem"

# 节点加密
peer-transport-security:
  cert-file: "/etc/etcd/etcdca/peer.pem"
  key-file: "/etc/etcd/etcdca/peer-key.pem"
  client-cert-auth: True
  trusted-ca-file: "/etc/etcd/etcdca/ca.pem"

[root@etcd etcdca]# systemctl enable etcd.service --now

2.10、访问验证

[root@etcd etcdca]# etcdctl --endpoints=https://192.168.93.104:2379 \
--cacert=ca.pem --cert=client.pem --key=client-key.pem endpoint status --cluster -w table
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| ENDPOINT                    | ID               | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| https://192.168.93.101:2379 | 8e9e05c52164694d | 3.4.23  | 20 kB   | true      | false      | 3         | 6          | 6                  |        |
+-----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

三、准备环境

• 在正式开始部署kubernetes集群之前，先要进行如下准备工作。基础环境相关配置操作，在三台主机master、node01、node02上都需要执行

3.1、绑定hosts

cat >> /etc/hosts << EOF
192.168.93.101 master
192.168.93.102 node1
192.168.93.103 node2
192.168.93.104 etcd
EOF

3.2、安装常用软件

yum -y install vim lrzsz unzip wget net-tools tree bash-completion telnet ntpdate

3.3、关闭交换分区

• kubeadm不支持swap

# 临时关闭
swapoff -a

# 永久关闭
sed -i '/swap/s/^/#/' /etc/fstab

3.4、配置时间同步器

# 同步阿里云时间
ntpdate ntp.aliyun.com

3.5、桥接IPv4流量传递给iptables的链

# 加载 overlay 内核模块
modprobe overlay

# 加载 br_netfilter 模块
modprobe br_netfilter

# 开启路由转发和网络配置
sed -i 's/net.ipv4.ip_forward=0/net.ipv4.ip_forward=1/g' /etc/sysctl.conf

cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
EOF

# 加载配置
sysctl --system

3.6、安装Runc

• Runc是一个轻量级的容器运行环境

wget https://github.com/opencontainers/runc/releases/download/v1.1.9/runc.amd64
install -m 755 runc.amd64 /usr/local/sbin/runc

3.7、安装cni-plugin

• cni-plugin是容器网络接口插件

mkdir -p /opt/cni/bin
wget https://github.com/containernetworking/plugins/releases/download/v1.3.0/cni-plugins-linux-amd64-v1.3.0.tgz
tar Cxzvf /opt/cni/bin cni-plugins-linux-amd64-v1.3.0.tgz

四、准备Containerd容器运行时

• 三台主机都要操作

4.1、安装Containerd

# 添加 docker 源，containerd也在docker源内的
cat <<EOF | sudo tee /etc/yum.repos.d/docker-ce.repo
[docker]
name=docker-ce
baseurl=https://mirrors.aliyun.com/docker-ce/linux/centos/7/x86_64/stable/
enabled=1
gpgcheck=1
gpgkey=https://mirrors.aliyun.com/docker-ce/linux/centos/gpg
EOF

# 安装Containerd容器
yum -y install containerd.io-1.6.6-3.1.el7.x86_64

4.2、配置Containerd

# 生成配置文件
mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml

# 修改/etc/containerd/config.toml文件中sandbox_image的值
grep 'sandbox_image' /etc/containerd/config.toml
sandbox_image = "registry.aliyuncs.com/google_containers/pause:3.9"

4.3、配置镜像加速器

# 创建所需目录
mkdir -p /etc/containerd/certs.d/docker.io/

# 添加如下内容
vim /etc/containerd/certs.d/docker.io/hosts.toml
####################################################
server = "https://registry-1.docker.io"

[host."https://docker.storehouses.dpdns.org"]
  capabilities = ["pull", "resolve"]
[host."https://docker.1ms.run"]
  capabilities = ["pull", "resolve"]
[host."https://docker.cattt.net"]
  capabilities = ["pull", "resolve"]
[host."https://docker.m.ixdev.cn"]
  capabilities = ["pull", "resolve"]
####################################################

# 在配置文件中找到配置段添加镜像地址目录
cat /etc/containerd/config.toml | grep config_path
config_path = "/etc/containerd/certs.d"

4.4、启动Containerd

systemctl enable containerd
systemctl start containerd

五、部署Kubernetes集群

• 准备好基础环境和Containerd环境后，下面就开始通过kubeadm来部署kubernetes集群

5.1、安装Kubeadm工具

• 所有节点都要操作

# 添加Kubernetes源
cat << EOF >> /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF

# 安装 kubectl：命令行管理工具、kubeadm：安装K8S集群工具、kubelet管理容器工具
yum install -y kubelet-1.28.0 kubeadm-1.28.0 kubectl-1.28.0

# 切记这个时候不要启动，只需要设置为开机自启
systemctl enable kubelet.service

5.2、配置crictl工具

• crictl是CRI兼容的容器运行时命令行接口。你可以使用它来检查和调试Kubernetes节点上的容器运行时和应用程序。crictl和它的源代码在 cri-tools 代码库
• 更好 Containerd后，以上我们常用的docker命令也不再使用了，取而代之的是 crictl 和 ctr 两个命令行客户端
• crictl是遵循CRI接口规范的一个命令行工具，通常用它来检查和管理kubelet节点上的容器运行时和镜像
• ctr是containerd的一个客户端工具

# 所有节点都要操作
cat << EOF >> /etc/crictl.yaml
runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 10
debug: false
EOF

5.3、复制ETCD证书至Master节点

• master操作

[root@master ~]# mkdir -p /etc/kubernetes/pki/etcd/

• etcd操作

[root@etcd etcdca]# pwd
/etc/etcd/etcdca

[root@etcd etcdca]# scp ca.pem root@192.168.93.101:/etc/kubernetes/pki/etcd/ca.crt
[root@etcd etcdca]# scp client.pem root@192.168.93.101:/etc/kubernetes/pki/etcd/apiserver-etcd-client.crt
[root@etcd etcdca]# scp client-key.pem root@192.168.93.101:/etc/kubernetes/pki/etcd/apiserver-etcd-client.key

5.4、初始化Master节点

• 在master节点上操作

# 生成初始化配置文件
[root@master ~]# kubeadm config print init-defaults > init-config.yaml

# 修改初始化配置文件
[root@master ~]# cat init-config.yaml
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
bootstrapTokens:
- token: abcdef.0123456789abcdef
  ttl: 24h0m0s
localAPIEndpoint:
  advertiseAddress: 192.168.93.101
  bindPort: 6443
nodeRegistration:
  criSocket: unix:///var/run/containerd/containerd.sock
  imagePullPolicy: IfNotPresent
  taints: []
---
apiVersion: kubeadm.k8s.io/v1beta3
kind: ClusterConfiguration
apiServer:
  timeoutForControlPlane: 4m0s
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controllerManager: {}
dns: {}
etcd:
  external:
    endpoints:
    - https://192.168.93.104:2379
    caFile: /etc/kubernetes/pki/etcd/ca.crt
    certFile: /etc/kubernetes/pki/etcd/apiserver-etcd-client.crt
    keyFile: /etc/kubernetes/pki/etcd/apiserver-etcd-client.key
imageRepository: registry.aliyuncs.com/google_containers
kubernetesVersion: 1.28.0
networking:
  dnsDomain: cluster.local
  serviceSubnet: 10.96.0.0/12
  podSubnet: 10.244.0.0/16
scheduler: {}
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
failSwapOn: false
address: 0.0.0.0
enableServer: true
cgroupDriver: cgroupfs
---
apiVersion: kubeproxy.config.k8s.io/v1alpha1
kind: KubeProxyConfiguration
mode: ipvs
ipvs:
  strictARP: true

# 拉取所需镜像
[root@master ~]# kubeadm config images pull --config=init-config.yaml
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.28.2
...

# 初始化Master节点
[root@master ~]# kubeadm init --config=init-config.yaml
[init] Using Kubernetes version: v1.28.0
[preflight] Running pre-flight checks
...
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

5.5、master节点复制k8s认证文件到用户的home目录

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

5.6、Node节点加入集群

• 直接把master节点初始化之后的最后回显的token复制粘贴到node节点回车即可，无须做任何配置

# node1
[root@node01 ~]# kubeadm join 192.168.93.101:6443 --token abcdef.0123456789abcdef \
> --discovery-token-ca-cert-hash sha256:0fc85526241963bb2d72d11d9cb5ea17b43d5f79ee27130ee5a7c0840444abab
[preflight] Running pre-flight checks
...
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.

Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

# node2
[root@node02 ~]# kubeadm join 192.168.93.101:6443 --token abcdef.0123456789abcdef \
> --discovery-token-ca-cert-hash sha256:0fc85526241963bb2d72d11d9cb5ea17b43d5f79ee27130ee5a7c0840444abab
...
This node has joined the cluster:

六、部署calico网络插件（CNI）

# 只需在master节点执行即可
[root@master ~]# kubectl apply -f calico.yaml

七、验证集群

7.1、Master打污点

• master节点操作即可
• Euler操作系统安装的K8S集群默认情况下master节点也会接受Pod，为了确保集群的稳定性，须给master节点打污点
• 给节点 master节点增加一个污点，它的键名是 key1，键值是 value1，效果是NoSchedule，此污点动作为该节点不会接受任何Pod的调度

[root@master ~]# kubectl taint nodes master key1=value1:NoSchedule
node/master tainted

7.2、查看节点状态

• master节点操作即可

[root@master ~]# kubectl get node
NAME      STATUS   ROLES           AGE   VERSION
master    Ready    control-plane   10h   v1.28.0
node1     Ready    <none>          10h   v1.28.0
node2     Ready    <none>          10h   v1.28.0

7.3、查看默认Pod状态

• master节点操作即可

[root@master ~]# kubectl get pod -A
NAMESPACE     NAME                                       READY   STATUS    RESTARTS       AGE
kube-system   calico-kube-controllers-658d97c59c-dp9jz   1/1     Running   1 (110s ago)   6m11s
...

7.4、查看组件状态

• master节点操作即可

[root@master ~]# kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE   ERROR
controller-manager   Healthy   ok
etcd-0               Healthy   ok
scheduler            Healthy   ok