IngressNginx v1.14.3 部署笔记 (DaemonSet + NodePort模式)
这是一个整合了 Ingress-NGINX v1.14.3 部署、配置修改以及外部 HAProxy 负载均衡器的完整部署文档。😭 截止发文前，Ingress NGINX 宣布退役，官方宣布尽力维护将持续至2026年3月。下面的内容只是为了纪念。🐶该方案的架构目标是：全节点覆盖（DaemonSet） + 本地流量转发（Local） + 真实 IP 透传（Proxy Protocol）1. 架构与环境说明一个 3 主 3 从的 K8s 集群,k8s 版本: 1.32Ingress 版本: v1.14.3部署模式: DaemonSet (覆盖所有 Node 节点)流量策略: externalTrafficPolicy: Local (仅接收本机流量，防止 SNAT 丢失 IP)IP 透传技术: Proxy Protocol (v2)节点规划:HAProxy (外部 LB): 172.22.33.100 负责K8s API Server 转发 (6443)，负责Ingress HTTPS 流量转发 (443 -> NodePort 32443)K8s master Nodes：172.22.33.101, 172.22.33.102, 172.22.33.103K8s Worker Nodes: 172.22.33.110, 172.22.33.111, 172.22.33.1122. 第一步：部署外部负载均衡器 (HAProxy)我们需要在 K8s 集群外部（或独立节点）部署 HAProxy，负责将流量分发到各个 Worker 节点的 NodePort 上，并封装 Proxy Protocol 包头。2.1 安装 HAProxy在节点 172.22.33.100 上执行：# CentOS/OpenEuler/RHEL yum -y install haproxy2.2 配置 HAProxy编辑 /etc/haproxy/haproxy.cfg关键配置点：后端 server 必须开启 send-proxy-v2，这是 Ingress 能获取真实 IP 的前提。端口需与后续 Ingress Service 定义的 NodePort (32443) 对应，最外侧的 4 层负载均衡只做 443 端口的转发，整个 K8s ingress nginx 使用 https 方式访问。这里 K8s API Server 转发 (6443) 的部署过程忽略，可以参考 k8s 的部署文档： https://blog.srebro.cn/archives/102global log /dev/log local0 warning chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon stats socket /var/lib/haproxy/stats defaults log global option httplog option dontlognull timeout connect 5000 timeout client 50000 timeout server 50000 # K8s API Server 转发 (6443) frontend kube-apiserver bind *:6443 mode tcp option tcplog default_backend kube-apiserver backend kube-apiserver mode tcp option tcplog option tcp-check balance roundrobin default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100 server kube-apiserver-1 172.22.33.101:6443 check server kube-apiserver-2 172.22.33.102:6443 check server kube-apiserver-3 172.22.33.103:6443 check # Ingress HTTPS 流量转发 (443 -> NodePort 32243) frontend kube-https bind *:443 mode tcp option tcplog default_backend kube-https backend kube-https mode tcp option tcplog option tcp-check balance roundrobin default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100 # 关键点：端口 32243 对应 Ingress NodePort # 关键点：send-proxy-v2 启用 Proxy Protocol 协议 server kube-node01 172.22.33.110:32243 send-proxy-v2 check server kube-node02 172.22.33.111:32243 send-proxy-v2 check server kube-node03 172.22.33.112:32243 send-proxy-v2 check2.3 启动服务systemctl enable haproxy systemctl start haproxy systemctl status haproxy3. 第二步：部署 Ingress-NGINX (K8s侧)在 Master 节点操作。首先下载 v1.14.3的官方 YAML：wget https://github.com/kubernetes/ingress-nginx/blob/controller-v1.14.3/deploy/static/provider/cloud/deploy.yaml -O ingress-nginx.yaml # 如果下载失败，可访问我的代理仓库对 ingress-nginx.yaml 进行如下三处关键修改：3.1 修改 ConfigMap (开启 Proxy Protocol)找到 kind: ConfigMap 且 name: ingress-nginx-controller 的部分，修改 data 如下：apiVersion: v1 kind: ConfigMap metadata: labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.14.3 name: ingress-nginx-controller namespace: ingress-nginx data: # 开启 Proxy Protocol 接收，必须与 HAProxy 的 send-proxy-v2 配合 use-proxy-protocol: "true" # 解析完整链路 IP compute-full-forwarded-for: "true" # 指定 Header 名称 forwarded-for-header: "X-Forwarded-For"3.2 修改 Controller (Deployment -> DaemonSet)为了保证每个 Worker 节点都有 Ingress 实例（配合 Local 策略），需将控制器类型改为 DaemonSet。找到 kind: Deployment 且 name: ingress-nginx-controller 的部分：修改 kind: Deployment 为 kind: DaemonSet。修改滚动更新策略： 由 spec.strategy 变成 spec.updateStrategy。修改镜像版本为国内镜像加速地址。docker.cnb.cool/sre-demo/k8s-demo/registry.k8s.io-ingress-nginx-controller:v1.14.3_amd64 , docker.cnb.cool/sre-demo/k8s-demo/registry.k8s.io-ingress-nginx-kube-webhook-certgen:v1.6.7_amd64--- apiVersion: apps/v1 kind: DaemonSet # 1. 修改这里 metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.14.3 name: ingress-nginx-controller namespace: ingress-nginx spec: minReadySeconds: 0 revisionHistoryLimit: 10 selector: matchLabels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx updateStrategy: # 2. 修改滚动更新策略 rollingUpdate: maxUnavailable: 1 type: RollingUpdate template: metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.14.3 spec: automountServiceAccountToken: true containers: - args: - /nginx-ingress-controller - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller - --election-id=ingress-nginx-leader - --controller-class=k8s.io/ingress-nginx - --ingress-class=nginx - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller - --validating-webhook=:8443 - --validating-webhook-certificate=/usr/local/certificates/cert - --validating-webhook-key=/usr/local/certificates/key env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: LD_PRELOAD value: /usr/local/lib/libmimalloc.so image: docker.cnb.cool/sre-demo/k8s-demo/registry.k8s.io-ingress-nginx-controller:v1.14.3_amd64 # 3.0 修改镜像加速地址 #####需要修改3处镜像地址###### imagePullPolicy: IfNotPresent lifecycle: preStop: exec: command: - /wait-shutdown livenessProbe: failureThreshold: 5 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 name: controller ports: - containerPort: 80 name: http protocol: TCP - containerPort: 443 name: https protocol: TCP - containerPort: 8443 name: webhook protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 resources: requests: cpu: 100m memory: 90Mi securityContext: allowPrivilegeEscalation: false capabilities: add: - NET_BIND_SERVICE drop: - ALL readOnlyRootFilesystem: false runAsGroup: 82 runAsNonRoot: true runAsUser: 101 seccompProfile: type: RuntimeDefault volumeMounts: - mountPath: /usr/local/certificates/ name: webhook-cert readOnly: true dnsPolicy: ClusterFirst nodeSelector: kubernetes.io/os: linux serviceAccountName: ingress-nginx terminationGracePeriodSeconds: 300 volumes: - name: webhook-cert secret: secretName: ingress-nginx-admission3.3 修改 Service (NodePort + Local + 端口对齐)确保 Service 暴露 NodePort，且端口与 HAProxy 配置的后端端口一致 (32443)。找到 kind: Service 且 name: ingress-nginx-controller 的部分：apiVersion: v1 kind: Service metadata: name: ingress-nginx-controller namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/version: 1.14.3 spec: type: NodePort # 1. 类型改为 NodePort externalTrafficPolicy: Local # 2. 策略改为 Local (关键：保留源 IP) ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - appProtocol: http name: http port: 80 protocol: TCP targetPort: http nodePort: 32280 # 3. 固定 HTTP 端口 - appProtocol: https name: https port: 443 protocol: TCP targetPort: https nodePort: 32443 # 4. (必须) 固定 HTTPS 端口，与 HAProxy 配置一致 selector: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/component: controller3.4 执行部署$ kubectl apply -f ingress-nginx.yaml4. 验证部署结果4.1 检查 Pod 状态确保每个 Worker 节点上都运行了一个 Ingress Pod。[root@k8s-master01 ingress-nginx-v1.14.3]# kubectl get pods -n ingress-nginx -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES ingress-nginx-controller-jt9xc 1/1 Running 0 8m24s 10.244.58.236 k8s-node02 <none> <none> ingress-nginx-controller-pwvm5 1/1 Running 0 8m24s 10.244.135.163 k8s-node03 <none> <none> ingress-nginx-controller-rskkc 1/1 Running 0 8m24s 10.244.85.241 k8s-node01 <none> <none>4.2 检查 Service 策略确认策略已生效。$ kubectl get svc -n ingress-nginx ingress-nginx-controller -o yaml | grep externalTrafficPolicy # 预期输出：externalTrafficPolicy: Local4.3 真实 IP 验证部署一个测试应用（Demo），并通过域名访问指定「可信代理服务器的 IP 段」，10.244.0.0/16 是k8s 的子网地址，表示信任只信任这段ip的代理头，即任何代理传递的 X-Forwarded-For都被认为是合法的，一般我们会添加上游服务器的 IP 地址。 set_real_ip_from 10.244.0.0/16指定「从哪个 HTTP 头中提取真实客户端 IP ,real_ip_header X-Forwarded-For递归跳过可信代理 IP，取第一个非可信 IP（真实客户端 IP）real_ip_recursive on# main-app.yaml apiVersion: v1 kind: ConfigMap metadata: name: nginx-main-conf namespace: default data: nginx.conf: | user nginx; worker_processes auto; error_log /var/log/nginx/error.log notice; pid /run/nginx.pid; events { worker_connections 1024; } http { include /etc/nginx/mime.types; default_type application/octet-stream; set_real_ip_from 10.244.0.0/16; real_ip_header X-Forwarded-For; real_ip_recursive on; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; sendfile on; keepalive_timeout 65; include /etc/nginx/conf.d/*.conf; } --- apiVersion: apps/v1 kind: Deployment metadata: name: main-app namespace: default spec: replicas: 1 selector: matchLabels: app: main-app template: metadata: labels: app: main-app spec: containers: - name: main-app image: nginx:latest imagePullPolicy: IfNotPresent ports: - containerPort: 80 volumeMounts: - name: nginx-main-conf mountPath: /etc/nginx/nginx.conf subPath: nginx.conf readOnly: true volumes: - name: nginx-main-conf configMap: name: nginx-main-conf --- apiVersion: v1 kind: Service metadata: name: main-app-service namespace: default spec: type: ClusterIP selector: app: main-app ports: - name: http port: 80 targetPort: 80查看服务状态：#apply $ kubectl apply -f main-app.yaml #查看资源 $ kubectl get all -n default NAME READY STATUS RESTARTS AGE pod/main-app-895f5b55d-d6hlf 1/1 Running 0 9s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 18d service/main-app-service ClusterIP 10.106.241.190 <none> 80/TCP 10d NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/main-app 1/1 1 1 9s NAME DESIRED CURRENT READY AGE replicaset.apps/main-app-895f5b55d 1 1 1 9s配置 Ingress 域名配置一个 TLS Secret 证书，将提前准备好的域名证书导入到 k8s 的 default 命名空间下的 secret 中$ kubectl create secret tls srebro.cn-tls \ --cert=srebro.cn.pem \ --key=srebro.cn.key \ --namespace=default创建一个 ingress 规则指定到一个域名上，如 demo.srebro.cn#ingress.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: baseline-sre-test-ingress namespace: default annotations: nginx.ingress.kubernetes.io/use-regex: "true" nginx.ingress.kubernetes.io/rewrite-target: /$2 # 可选：强制 HTTPS 重定向 nginx.ingress.kubernetes.io/ssl-redirect: "true" spec: ingressClassName: nginx tls: - hosts: - demo.srebro.cn secretName: srebro.cn-tls rules: - host: demo.srebro.cn http: paths: - path: /()(.*) pathType: ImplementationSpecific backend: service: name: main-app-service port: number: 80查看ingress资源：#创建ingress $ kubectl apply -f ingress.yaml ingress.networking.k8s.io/baseline-sre-test-ingress created $ kubectl get ingress NAME CLASS HOSTS ADDRESS PORTS AGE baseline-sre-test-ingress nginx demo.srebro.cn 10.104.185.248 80, 443 45s记得修改本地 hosts ， demo.srebro.cn 指向 HAProxy IP 172.22.33.100查看测试应用的日志：kubectl logs -f <业务Pod名称>成功标志：日志中的 $remote_addr 或 X-Forwarded-For 显示的是你客户端的IP，而不是K8s 集群内部地址。5. 其他本文所涉及的资源文件见： https://cnb.cool/sre-demo/k8s-demo/-/tree/main/yaml/v1.32/ingress-nginxingress-nginx 生命周期截止2026.3，后续官方也不会再提供更新服务。https://github.com/kubernetes/ingress-nginx/tree/main 祝好！大家可以切换到其他的 ingress 控制器上，比如traefik,higress,或者其他。
基于 NFS 的 Kubernetes StorageClass 动态供给部署指南
一、StorageClass介绍StorageClass 用于描述集群中的“存储类型”和动态供给策略。它把后端存储（如 NFS、Ceph、云盘）的供给逻辑抽象为一个“provisioner”，并通过 parameters 传递供给参数。应用只需在 PVC 中声明 storageClassName，即可按需创建 PV。provisioner：指定供给器的唯一名称，必须与供给器部署时的环境变量保持一致parameters：供给器的行为配置，例如是否在删除 PVC 时保留数据reclaimPolicy：PV 的回收策略，通常由系统默认或 StorageClass 设定本实验使用 nfs-subdir-external-provisioner 作为动态供给器，它会在 NFS 后端的指定目录下为每个 PVC 创建一个子目录，实现按需供给。apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: nfs-dynamic provisioner: k8s-sigs.io/nfs-subdir-external-provisioner parameters: archiveOnDelete: "false"二、NFS介绍NFS（Network File System）是基于网络的文件系统协议，允许多个主机通过网络共享同一目录。其在 Kubernetes 中常用于共享型持久化存储场景，典型特征是支持 ReadWriteMany（RWX），适合多副本工作负载共享同一数据目录。优点：简易、通用、支持 RWX、和现有基础设施容易集成注意：网络和 NFS 服务器性能会影响 IO；生产环境应考虑权限隔离、备份与高可用先决条件：K8s 节点需安装 NFS 客户端（如 nfs-utils / nfs-common），并能访问 NFS 服务器端口（一般为 2049 等）三、实验介绍本实验在现有 NFS 服务基础上，部署 nfs-subdir-external-provisioner，实现 PVC 的动态供给，并通过示例 Pod 验证挂载与读写。3.1、环境介绍Kubernetes 集群已可用NFS 服务器：172.22.33.100NFS 共享路径：/home/application/nfs/data供给器命名空间：kube-nfs3.2、安装配置nfs服务如果尚未配置 NFS 服务，需在服务器上完成以下步骤（示例）：安装 NFS 服务端（不同发行版命令略有差异）Debian/Ubuntu：apt install -y nfs-kernel-serverCentOS/RHEL：yum install -y nfs-utils创建共享目录：mkdir -p /home/application/nfs/data配置导出（编辑 /etc/exports）：/home/application/nfs/data *(insecure,rw,sync,no_root_squash)启动并刷新导出：systemctl start nfs-serverexportfs -arv验证：showmount -e 172.22.33.100 应显示共享目录确保 K8s 节点能访问nfs 服务器四、PersistenVolume动态供给部署本节基于以下文件完成部署与验证：步骤如下：创建命名空间kubectl create namespace kube-nfs配置 RBAC（见 [rbac.yaml]）该文件创建供给器使用的 ServiceAccount、ClusterRole/Binding，以及 leader 选举所需的 Role/Binding：apiVersion: v1 kind: ServiceAccount metadata: name: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: kube-nfs --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: nfs-client-provisioner-runner rules: - apiGroups: [""] resources: ["nodes"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "list", "watch", "create", "delete"] - apiGroups: [""] resources: ["persistentvolumeclaims"] verbs: ["get", "list", "watch", "update"] - apiGroups: ["storage.k8s.io"] resources: ["storageclasses"] verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["events"] verbs: ["create", "update", "patch"] --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: run-nfs-client-provisioner subjects: - kind: ServiceAccount name: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: kube-nfs roleRef: kind: ClusterRole name: nfs-client-provisioner-runner apiGroup: rbac.authorization.k8s.io --- kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: leader-locking-nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: kube-nfs rules: - apiGroups: [""] resources: ["endpoints"] verbs: ["get", "list", "watch", "create", "update", "patch"] --- kind: RoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: leader-locking-nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: kube-nfs subjects: - kind: ServiceAccount name: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: kube-nfs roleRef: kind: Role name: leader-locking-nfs-client-provisioner apiGroup: rbac.authorization.k8s.io应用：kubectl apply -f rbac.yaml部署供给器（见 [deployment.yaml]） 关键点：serviceAccountName: nfs-client-provisioner 与上一步创建的 SA 对应PROVISIONER_NAME 必须与 StorageClass 的 provisioner 完全一致：k8s-sigs.io/nfs-subdir-external-provisionerNFS_SERVER 与 NFS_PATH 指向实际 NFS 服务volumes.nfs.server/path 与环境变量保持一致示例片段：apiVersion: apps/v1 kind: Deployment metadata: name: nfs-client-provisioner labels: app: nfs-client-provisioner # replace with namespace where provisioner is deployed namespace: kube-nfs spec: replicas: 2 strategy: type: Recreate selector: matchLabels: app: nfs-client-provisioner template: metadata: labels: app: nfs-client-provisioner spec: serviceAccountName: nfs-client-provisioner containers: - name: nfs-client-provisioner #image: registry.k8s.io/sig-storage/nfs-subdir-external-provisioner:v4.0.2 image: docker.cnb.cool/sre-demo/k8s-demo/registry.k8s.io-sig-storage-nfs-subdir-external-provisioner:v4.0.2_amd64 volumeMounts: - name: nfs-client-root mountPath: /persistentvolumes env: - name: PROVISIONER_NAME value: k8s-sigs.io/nfs-subdir-external-provisioner - name: NFS_SERVER value: 172.22.33.100 - name: NFS_PATH value: /home/application/nfs/data volumes: - name: nfs-client-root nfs: server: 172.22.33.100 path: /home/application/nfs/data应用：kubectl apply -f deployment.yaml kubectl -n kube-nfs get podsPod 应处于 Running 状态；若失败，检查镜像拉取、NFS 可达性、节点是否安装 NFS 客户端。创建 StorageClass（见 [class.yaml]）apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: nfs-dynamic provisioner: k8s-sigs.io/nfs-subdir-external-provisioner # or choose another name, must match deployment's env PROVISIONER_NAME' parameters: archiveOnDelete: "false"kubectl apply -f class.yaml kubectl get sc应看到 nfs-dynamic 已创建，且 provisioner 为 k8s-sigs.io/nfs-subdir-external-provisioner。测试 PVC 与挂载（见 [nfs-pvc-test.yaml]）该文件创建一个 ReadWriteMany 的 PVC，并启动一个 Pod 挂载到 /mnt，在容器中创建 /mnt/SUCCESS 文件以验证写入：--- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: test-claim spec: storageClassName: nfs-dynamic accessModes: - ReadWriteMany resources: requests: storage: 1Mi --- kind: Pod apiVersion: v1 metadata: name: test-pod spec: containers: - name: test-pod #image: busybox:stable image: docker.cnb.cool/sre-demo/k8s-demo/busybox:stable_amd64 command: - "/bin/sh" args: - "-c" - "touch /mnt/SUCCESS && exit 0 || exit 1" volumeMounts: - name: nfs-pvc mountPath: "/mnt" restartPolicy: "Never" volumes: - name: nfs-pvc persistentVolumeClaim: claimName: test-claim应用并验证：kubectl apply -f nfs-pvc-test.yaml kubectl get pvc kubectl get pv kubectl get pod test-pod kubectl logs test-podPVC 状态应为 Bound，PV 自动创建并绑定Pod 退出码为 0，日志为空或正常，NFS 目录下应出现对应子目录与 SUCCESS 文件五、其他本文所涉及的 yaml 文件， 见： https://cnb.cool/sre-demo/k8s-demo
Kubernetes 外部 etcd 集群的快速 Docker Compose 部署指南
一、背景在高可用 Kubernetes 部署中，需要单独部署外部 etcd 集群，而不是使用 kubeadm 默认在 master 节点上部署的 etcd。以下是关于这一配置场景的详细记录。二、etcd简介etcd 是一个高可用的分布式键值存储系统，主要用于存储和管理配置信息、服务发现信息以及其他重要的元数据。etcd 是由 CoreOS 开发的，基于 Raft 共识算法来保证数据的一致性和可靠性。2.1 etcd 关键特点分布式：etcd 运行在多个节点上，通过 Raft 协议确保数据一致性。高可用性：即使部分节点出现故障，etcd 也能继续提供服务。一致性保证：使用 Raft 协议来确保数据的一致性，支持强一致性读写。键值存储：数据以键值对的形式存储，适合存储配置和元数据。支持事务：支持原子操作，如事务操作来保证数据的正确性。2.2 etcd 主要功能分布式键值存储：etcd 是一个高可用的分布式键值存储系统，支持高效的键值对存储和读取，并提供强一致性保证。一致性保证：etcd 使用 Raft 协议确保数据一致性，即使部分节点故障，数据仍然保持一致。高可用性：etcd 通过数据复制到多个节点提供高可用性和容错能力，只要大多数节点正常，数据依然可用。事务支持：etcd 支持原子操作和事务，确保对多个键值对的更改安全一致。观察和通知：etcd 支持事件监听和观察功能，客户端可以订阅键值变化，适合动态配置更新。持久化存储：etcd 将数据持久化到磁盘，确保在节点重启或故障时数据不丢失。分布式锁：etcd 提供分布式锁，支持节点间的协调和同步，常用于确保资源唯一性或任务协调。范围查询：etcd 支持对键值对的范围查询，便于批量检索数据。API 接口：etcd 提供 HTTP/gRPC API，支持基本的 CRUD 操作、事务和观察功能。这些功能使 etcd 成为一个强大且可靠的分布式键值存储系统，适合用于需要高一致性和高可用性的场景，特别是在 Kubernetes 这样的集群管理系统中。2.3 etcd 在 Kubernetes 中使用场景在 Kubernetes (K8S) 集群中，etcd 扮演着至关重要的角色。它主要用于存储和管理 Kubernetes 的所有集群状态信息和配置数据。以下是 etcd 在 Kubernetes 中的一些关键使用场景：集群状态存储：Kubernetes 的所有集群状态、配置信息、资源定义等都存储在 etcd 中。包括 Pod、Service、ConfigMap、Secret 等对象的状态和配置都被存储在 etcd 中。高可用性配置：提供高可用性和容错能力，保证即使在集群中的一些节点失效的情况下，Kubernetes 也能继续运行和提供服务。数据一致性保证： 使用 Raft 协议确保数据的一致性，这对于 Kubernetes 来说是非常重要的，因为它需要确保所有操作和状态在集群中的一致性。服务发现：Kubernetes 的服务发现机制依赖于 etcd 存储服务的状态和信息。etcd 记录服务的 IP 地址和端口信息，允许其他组件进行服务发现。动态配置管理：Kubernetes 使用 etcd 来管理配置变化，比如部署新的应用版本或调整资源配置。etcd 的事务支持使得配置更改能够被安全地提交和应用。审计和恢复：etcd 的持久化存储允许 Kubernetes 在发生故障或恢复操作时能够重建集群状态。etcd 的备份和恢复机制是确保 Kubernetes 集群数据安全的重要手段。三、使用docker-compose快速部署etcd参考官方文档：https://github.com/etcd-io/etcd/releases3.1 etcd单节点部署PS: 请勿在K8S 集群中的机器上 使用docker-compose 去部署etcd ，可能会存在子网冲突； 在条件允许的情况下，我们还是申请机器去单独部署etcd3.1.1 环境准备机器IP主机名etcd版本docker 版本docker-compose版本172.22.33.213etcdv3.3.820.10.18v2.10.23.1.2 创建etcd 工作目录，编写docker-compose 文件ETCD_NAME 每个节点的 etcd 服务名称ETCD_ADVERTISE_CLIENT_URLS 填写每个 etcd 容器所在节点对外提供服务的 IPETCD_INITIAL_ADVERTISE_PEER_URLS 填写每个 etcd 容器所在节点对外提供服务的 IPETCD_INITIAL_CLUSTER: ETCD 集群配置包含所有节点的名称和 ETCD_INITIAL_ADVERTISE_PEER_URLS 地址，我这里目前就一个节点ETCD_INITIAL_CLUSTER_STATE:new: 用于首次创建全新 etcd 集群，节点将根据初始配置启动集群。existing: 用于扩展、重启或恢复已有的集群，节点将尝试加入现有集群，而不是重新初始化$ mkdir -p /home/application/etcd/{data,config} $ cat >> /home/application/etcd/docker-compose.yaml << EOF version: '3' services: etcd: container_name: etcd image: quay.srebro.site/coreos/etcd:v3.3.8 environment: - ETCD_NAME=etcd - ETCD_DATA_DIR=/var/etcd - ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379 - ETCD_ADVERTISE_CLIENT_URLS=http://172.22.33.213:2379 - ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380 - ETCD_INITIAL_ADVERTISE_PEER_URLS=http://172.22.33.213:2380 - ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster - ETCD_INITIAL_CLUSTER=etcd=http://172.22.33.213:2380 - ETCD_INITIAL_CLUSTER_STATE=new - ETCD_LOGGER=zap - ETCD_LOG_LeveL=info volumes: - /home/application/etcd/data:/var/etcd - "/etc/localtime:/etc/localtime:ro" ports: - 2379:2379 - 2380:2380 restart: always networks: default: name: etcd-tier driver: bridge EOF3.1.3 创建并启动 etcd 服务$ cd /home/application/etcd/ $ docker-compose up -d $ docker-compose ps NAME COMMAND SERVICE STATUS PORTS etcd "/usr/local/bin/etcd" etcd running 0.0.0.0:2379-2380->2379-2380/tcp, :::2379-2380->2379-2380/tcp3.1.4 etcd 服务可用性测试为了测试 etcd 服务的可用性，需要安装 etcd 客户端工具用于验证测试。$ wget https://github.com/etcd-io/etcd/releases/download/v3.5.15/etcd-v3.5.15-linux-amd64.tar.gz $ tar -xf etcd-v3.5.15-linux-amd64.tar.gz $ cp etcd-v3.5.15-linux-amd64/etcdctl /usr/local/bin $ chmod +x /usr/local/bin/etcdctl #使用 etcdctl 工具快速检查指定的 etcd 节点（或节点集）的健康状态 $ etcdctl --endpoints=172.22.33.213:2379 --write-out=table endpoint health +--------------------+--------+------------+-------+ | ENDPOINT | HEALTH | TOOK | ERROR | +--------------------+--------+------------+-------+ | 172.22.33.213:2379 | true | 3.035915ms | | +--------------------+--------+------------+-------+ #使用 etcdctl 工具查看指定 etcd 节点（或节点集）当前的详细状态信息 $ etcdctl --endpoints=172.22.33.213:2379 --write-out=table endpoint status +--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | 172.22.33.213:2379 | 8e9e05c52164694d | 3.3.8 | 26 MB | true | false | 2 | 659422 | 0 | | +--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ #使用 etcdctl 工具验证测试 member $ etcdctl --endpoints=172.22.33.213:2379 --write-out=table member list +------------------+---------+------+-----------------------+---------------------------+------------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | +------------------+---------+------+-----------------------+---------------------------+------------+ | 8e9e05c52164694d | started | etcd | http://localhost:2380 | http://172.22.33.213:2379 | false | +------------------+---------+------+-----------------------+---------------------------+------------+ # etcdctl 工具验证测试数据读写 $ etcdctl --endpoints=172.22.33.213:2379 put srebro.cn yyds OK $ etcdctl --endpoints=172.22.33.213:2379 get srebro.cn srebro.cn yyds3.2 etcd集群部署PS: 请勿在K8S 集群中的机器上 使用docker-compose 去部署etcd集群 ，可能会存在子网冲突； 在条件允许的情况下，我们还是申请机器去单独部署etcd集群3.2.1 环境准备机器IP主机名etcd版本docker 版本docker-compose版本172.22.33.210etcd-01v3.3.820.10.18v2.10.2172.22.33.211etcd-02v3.3.820.10.18v2.10.2172.22.33.212etcd-03v3.3.820.10.18v2.10.23.2.2 修改主机名，添加hosts解析在每台主机上执行设置主机名： $ hostnamectl set-hostname etcd-01 $ hostnamectl set-hostname etcd-02 $ hostnamectl set-hostname etcd-03 #在所有节点添加hosts： $ cat >> /etc/hosts << EOF 172.22.33.210 etcd-01 172.22.33.212 etcd-02 172.22.33.212 etcd-03 EOF3.2.3 创建etcd 工作目录，编写docker-compose 文件172.22.33.210 etcd-01 主机上每个节点内容不一样，需要分别编写。mkdir -p /home/application/etcd/{data,config} cat >> /home/application/etcd/docker-compose.yaml << EOF version: '3' services: etcd: container_name: etcd image: quay.srebro.site/coreos/etcd:v3.3.8 environment: - ETCD_NAME=etcd01 - ETCD_DATA_DIR=/var/etcd - ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379 - ETCD_ADVERTISE_CLIENT_URLS=http://172.22.33.210:2379 - ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380 - ETCD_INITIAL_ADVERTISE_PEER_URLS=http://172.22.33.210:2380 - ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster - ETCD_INITIAL_CLUSTER=etcd01=http://172.22.33.210:2380,etcd02=http://172.22.33.211:2380,etcd03=http://172.22.33.212:2380 - ETCD_INITIAL_CLUSTER_STATE=new - ETCD_LOGGER=zap - ETCD_LOG_LeveL=info volumes: - /home/application/etcd/data:/var/etcd - "/etc/localtime:/etc/localtime:ro" ports: - 2379:2379 - 2380:2380 restart: always networks: default: name: etcd-tier driver: bridge EOF172.22.33.211 etcd-02 主机上每个节点内容不一样，需要分别编写。mkdir -p /home/application/etcd/{data,config} cat >> /home/application/etcd/docker-compose.yaml << EOF version: '3' services: etcd: container_name: etcd image: quay.srebro.site/coreos/etcd:v3.3.8 environment: - ETCD_NAME=etcd02 - ETCD_DATA_DIR=/var/etcd - ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379 - ETCD_ADVERTISE_CLIENT_URLS=http://172.22.33.211:2379 - ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380 - ETCD_INITIAL_ADVERTISE_PEER_URLS=http://172.22.33.211:2380 - ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster - ETCD_INITIAL_CLUSTER=etcd01=http://172.22.33.210:2380,etcd02=http://172.22.33.211:2380,etcd03=http://172.22.33.212:2380 - ETCD_INITIAL_CLUSTER_STATE=new - ETCD_LOGGER=zap - ETCD_LOG_LeveL=info volumes: - /home/application/etcd/data:/var/etcd - "/etc/localtime:/etc/localtime:ro" ports: - 2379:2379 - 2380:2380 restart: always networks: default: name: etcd-tier driver: bridge EOF172.22.33.212 etcd-03 主机上每个节点内容不一样，需要分别编写。mkdir -p /home/application/etcd/{data,config} cat >> /home/application/etcd/docker-compose.yaml << EOF version: '3' services: etcd: container_name: etcd image: quay.srebro.site/coreos/etcd:v3.3.8 environment: - ETCD_NAME=etcd03 - ETCD_DATA_DIR=/var/etcd - ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379 - ETCD_ADVERTISE_CLIENT_URLS=http://172.22.33.212:2379 - ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380 - ETCD_INITIAL_ADVERTISE_PEER_URLS=http://172.22.33.212:2380 - ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster - ETCD_INITIAL_CLUSTER=etcd01=http://172.22.33.210:2380,etcd02=http://172.22.33.211:2380,etcd03=http://172.22.33.212:2380 - ETCD_INITIAL_CLUSTER_STATE=new - ETCD_LOGGER=zap - ETCD_LOG_LeveL=info volumes: - /home/application/etcd/data:/var/etcd - "/etc/localtime:/etc/localtime:ro" ports: - 2379:2379 - 2380:2380 restart: always networks: default: name: etcd-tier driver: bridge EOF3.2.4 创建并启动 etcd 服务每台机器依次启动etcd 服务$ cd /home/application/etcd/ $ docker-compose up -d $ docker-compose ps NAME COMMAND SERVICE STATUS PORTS etcd "/usr/local/bin/etcd" etcd running 0.0.0.0:2379-2380->2379-2380/tcp, :::2379-2380->2379-2380/tcp3.2.5 etcd 集群服务可用性测试为了测试 etcd 集群服务的可用性，需要安装 etcd 客户端工具用于验证测试。$ wget https://github.com/etcd-io/etcd/releases/download/v3.5.15/etcd-v3.5.15-linux-amd64.tar.gz $ tar -xf etcd-v3.5.15-linux-amd64.tar.gz $ cp etcd-v3.5.15-linux-amd64/etcdctl /usr/local/bin $ chmod +x /usr/local/bin/etcdctl #使用 etcdctl 工具快速检查指定的 etcd 节点（或节点集）的健康状态 $ etcdctl --endpoints=172.22.33.210:2379,172.22.33.211:2379,172.22.33.212:2379 --write-out=table endpoint health +--------------------+--------+------------+-------+ | ENDPOINT | HEALTH | TOOK | ERROR | +--------------------+--------+------------+-------+ | 172.22.33.210:2379 | true | 3.035915ms | | | 172.22.33.211:2379 | true | 3.035915ms | | | 172.22.33.212:2379 | true | 3.035915ms | | +--------------------+--------+------------+-------+ #使用 etcdctl 工具查看指定 etcd 节点（或节点集）当前的详细状态信息 $ etcdctl --endpoints=172.22.33.210:2379,172.22.33.211:2379,172.22.33.212:2379 --write-out=table endpoint status +--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | 172.22.33.210:2379 | 8e9e05c52164694d | 3.3.8 | 26 MB | true | false | 2 | 659422 | 0 | | | 172.22.33.211:2379 | 0e95c521fdoi69ac | 3.3.8 | 26 MB | false| false | 2 | 659422 | 0 | | | 172.22.33.212:2379 | pe9e05dfbrt64680 | 3.3.8 | 26 MB | false| false | 2 | 659422 | 0 | | +--------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ #使用 etcdctl 工具验证测试 member $ etcdctl --endpoints=172.22.33.210:2379,172.22.33.211:2379,172.22.33.212:2379 --write-out=table member list +------------------+---------+------+-----------------------+---------------------------+------------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | +------------------+---------+------+-----------------------+---------------------------+------------+ | 8e9e05c52164694d | started | etcd | http://localhost:2380 | http://172.22.33.210:2379 | true | | 0e95c521fdoi69ac | started | etcd | http://localhost:2380 | http://172.22.33.211:2379 | false| | pe9e05dfbrt64680 | started | etcd | http://localhost:2380 | http://172.22.33.212:2379 | false| +------------------+---------+------+-----------------------+---------------------------+------------+ # etcdctl 工具验证测试数据读写 $ etcdctl --endpoints=172.22.33.210:2379,172.22.33.211:2379,172.22.33.212:2379 put srebro.cn hello OK $ etcdctl --endpoints=172.22.33.210:2379,172.22.33.211:2379,172.22.33.212:2379 get srebro.cn srebro.cn hello四、kubeadm 使用集群外etcd，部署K8S集群kubeadm 安装过程省略，直接贴上kubeadm-init.yaml 配置文件;主要查看etcd 那块的配置；另外需要注意，ETCD 集群 与 apiservice 之间通信，是否采用TLS 加密，我上面部署的etcd集群实则是没有采用TLS 加密的apiVersion: kubeadm.k8s.io/v1beta3 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: 35sesk.5tm3tqc66tzlse8c ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: advertiseAddress: 172.22.33.220 bindPort: 6443 nodeRegistration: criSocket: unix:///var/run/cri-dockerd.sock imagePullPolicy: IfNotPresent name: k8s-master01 taints: - effect: NoSchedule key: node-role.kubernetes.io/master --- apiServer: certSANs: - kubernetes - kubernetes.default - kubernetes.default.svc - kubernetes.default.svc.cluster.local - 172.22.33.220 - 112.11.11.11 - k8s.srebro.cn timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta3 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controlPlaneEndpoint: 172.22.33.220:6443 controllerManager: {} dns: {} etcd: external: endpoints: - "http://172.22.33.210:2379" - "http://172.22.33.211:2379" - "http://172.22.33.212:2379" imageRepository: registry.aliyuncs.com/google_containers kind: ClusterConfiguration kubernetesVersion: v1.24.9 networking: dnsDomain: cluster.local podSubnet: 10.244.0.0/16 serviceSubnet: 10.96.0.0/12 scheduler: {}不使用TLS 加密 ，需要配置如下内容：如果需要使用TLS 加密 ，需要配置如下内容：五、参考https://cloud.tencent.com/developer/article/2394695
一、背景在高可用 Kubernetes 部署中，需要单独部署外部 etcd 集群，而不是使用 kubeadm 默认在 master 节点上部署的 etcd。以下是关于这一配置场景的详细记录。二、etcd简介etcd 是一个高可用的分布式键值存储系统，主要用于存储和管理配置信息、服务发现信息以及其他重要的元数据。etcd 是由 CoreOS 开发的，基于 Raft 共识算法来保证数据的一致性和可靠性。2.1 etcd 关键特点分布式：etcd 运行在多个节点上，通过 Raft 协议确保数据一致性。高可用性：即使部分节点出现故障，etcd 也能继续提供服务。一致性保证：使用 Raft 协议来确保数据的一致性，支持强一致性读写。键值存储：数据以键值对的形式存储，适合存储配置和元数据。支持事务：支持原子操作，如事务操作来保证数据的正确性。2.2 etcd 主要功能分布式键值存储：etcd 是一个高可用的分布式键值存储系统，支持高效的键值对存储和读取，并提供强一致性保证。一致性保证：etcd 使用 Raft 协议确保数据一致性，即使部分节点故障，数据仍然保持一致。高可用性：etcd 通过数据复制到多个节点提供高可用性和容错能力，只要大多数节点正常，数据依然可用。事务支持：etcd 支持原子操作和事务，确保对多个键值对的更改安全一致。观察和通知：etcd 支持事件监听和观察功能，客户端可以订阅键值变化，适合动态配置更新。持久化存储：etcd 将数据持久化到磁盘，确保在节点重启或故障时数据不丢失。分布式锁：etcd 提供分布式锁，支持节点间的协调和同步，常用于确保资源唯一性或任务协调。范围查询：etcd 支持对键值对的范围查询，便于批量检索数据。API 接口：etcd 提供 HTTP/gRPC API，支持基本的 CRUD 操作、事务和观察功能。这些功能使 etcd 成为一个强大且可靠的分布式键值存储系统，适合用于需要高一致性和高可用性的场景，特别是在 Kubernetes 这样的集群管理系统中。2.3 etcd 在 Kubernetes 中使用场景在 Kubernetes (K8S) 集群中，etcd 扮演着至关重要的角色。它主要用于存储和管理 Kubernetes 的所有集群状态信息和配置数据。以下是 etcd 在 Kubernetes 中的一些关键使用场景：集群状态存储：Kubernetes 的所有集群状态、配置信息、资源定义等都存储在 etcd 中。包括 Pod、Service、ConfigMap、Secret 等对象的状态和配置都被存储在 etcd 中。高可用性配置：提供高可用性和容错能力，保证即使在集群中的一些节点失效的情况下，Kubernetes 也能继续运行和提供服务。数据一致性保证： 使用 Raft 协议确保数据的一致性，这对于 Kubernetes 来说是非常重要的，因为它需要确保所有操作和状态在集群中的一致性。服务发现：Kubernetes 的服务发现机制依赖于 etcd 存储服务的状态和信息。etcd 记录服务的 IP 地址和端口信息，允许其他组件进行服务发现。动态配置管理：Kubernetes 使用 etcd 来管理配置变化，比如部署新的应用版本或调整资源配置。etcd 的事务支持使得配置更改能够被安全地提交和应用。审计和恢复：etcd 的持久化存储允许 Kubernetes 在发生故障或恢复操作时能够重建集群状态。etcd 的备份和恢复机制是确保 Kubernetes 集群数据安全的重要手段。三、使用二进制方式部署etcd参考官方文档：https://github.com/etcd-io/etcd/releasesetcd 软件包下载地址： https://github.com/etcd-io/etcd/releases/download/v3.5.15/etcd-v3.5.15-linux-amd64.tar.gz3.1 etcd单节点部署3.1.1 环境准备机器IP主机名etcd版本172.22.33.223etcdv3.3.83.1.2 下载软件包,创建etcd工作目录etcd 软件包下载地址： https://github.com/etcd-io/etcd/releases/download/v3.5.15/etcd-v3.5.15-linux-amd64.tar.gz#创建etcd工作目录 $ mkdir -p /home/application/etcd/{bin,cfg,data,ssl} # 下载软件包并解压 $ wget https://github.com/etcd-io/etcd/releases/download/v3.5.15/etcd-v3.5.15-linux-amd64.tar.gz $ tar -xf etcd-v3.5.15-linux-amd64.tar.gz $ mv etcd-v3.5.15-linux-amd64/{etcdctl,etcd} /home/application/etcd/bin/ #验证etcd版本 $ /home/application/etcd/bin/etcd --version etcd Version: 3.5.15 Git SHA: 9a5533382 Go Version: go1.21.12 Go OS/Arch: linux/amd64 3.1.3 使用cfssl 工具签发etcd 证书cfssl是一个用于生成和管理 TLS/SSL 证书的工具，专门设计用于处理公钥基础设施（PKI）操作。它由 Cloudflare 开发，提供了一个命令行工具和一个 API 服务，可以用来生成、签发和管理证书。cfssl 的主要功能包括：生成证书签名请求（CSR）: 生成包含公钥信息的文件，用于向证书颁发机构（CA）申请证书。签发证书: 作为一个轻量级的证书颁发机构（CA），可以签发符合要求的证书。生成自签名证书: 创建自签名的根证书（Root CA），用于内部系统的信任链。证书管理: 包括查看、更新、撤销和续期证书的功能。支持 JSON API: 提供 HTTP/JSON API，允许通过 RESTful 接口自动化证书管理操作。使用场景自建 CA: 在一些企业环境或开发环境中，使用 cfssl 自建 CA 系统，用于内部服务的证书管理。自动化证书管理: 在 DevOps 环境中，通过 API 自动生成和更新证书。安全的服务间通信: 在分布式系统中，为各个服务生成和分发 TLS 证书，保证服务间通信的安全性。#下载cfssl 签发工具【可能背墙，自行备源】 [root@master1 ~]# curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl [root@master1 ~]# curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson [root@master1 ~]# curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo [root@master1 ~]# chmod +x /usr/local/bin/cfssl*签发证书-配置 CA 并创建 TLS 证书创建 Certificate Authority（CA）， 并为 etcd创建 TLS 证书。# 切换到证书工作目录下 $ cd /home/application/etcd/ssl # 配置 CA 的操作参数和策略。这些参数决定了 CA 的行为，如证书的有效期、证书链的深度、使用的加密算法等 $ cat << EOF | tee ca-config.json { "signing": { "default": { "expiry": "87600h" }, "profiles": { "etcd": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } } EOF # 配置CA 生成证书签名请求（CSR） $ cat << EOF | tee ca-csr.json { "CN": "etcd CA", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Nanjing", "ST": "Nanjing" } ] } EOF #生成 CA 凭证和私钥: $ cfssl gencert -initca ca-csr.json | cfssljson -bare ca #结果将生成以下两个文件： $ ls -l ca*.pem -rw------- 1 root root 1679 8月 26 14:19 ca-key.pem -rw-r--r-- 1 root root 1265 8月 26 14:19 ca.pem签发etcd server证书# 创建etcd 证书请求文件,修改请求文件中hosts字段包含所有etcd节点IP $ cat << EOF | tee server-csr.json { "CN": "etcd", "hosts": [ "172.22.33.223" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Nanjing", "ST": "Nanjing" } ] } EOF # 生成etcd server证书: $ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd server-csr.json | cfssljson -bare server #结果将生成以下两个文件： $ ls -l server*.pem -rw------- 1 root root 1675 8月 26 14:20 server-key.pem -rw-r--r-- 1 root root 1338 8月 26 14:20 server.pem3.1.4 创建配置文件$ cd /home/application/etcd/cfg $ cat >> /home/application/etcd/cfg/etcd.conf << EOF #[Member] #etcd 成员的名称，这是节点在集群中的唯一标识。 ETCD_NAME="etcd" #etcd 数据的存储目录，这里我们自己定了一个数据路径 ETCD_DATA_DIR="/home/application/etcd/data/default.etcd" #etcd 节点监听来自集群内其他节点的通信的地址和端口 ETCD_LISTEN_PEER_URLS="https://172.22.33.223:2380" # 指定 etcd 实例实际监听的地址，等于 etcd 实例接收客户端请求的地址。 ETCD_LISTEN_CLIENT_URLS="https://172.22.33.223:2379,http://127.0.0.1:2379" # 指定 etcd 实例对外公布的地址，客户端和其他 etcd 节点用来连接和访问 etcd 实例。 ETCD_ADVERTISE_CLIENT_URLS="https://172.22.33.223:2379,http://127.0.0.1:2379" #同时支持V2 API ETCD_ENABLE_V2="true" #[Security] ETCD_CERT_FILE="/home/application/etcd/ssl/server.pem" ETCD_KEY_FILE="/home/application/etcd/ssl/server-key.pem" ETCD_TRUSTED_CA_FILE="/home/application/etcd/ssl/ca.pem" ETCD_CLIENT_CERT_AUTH="true" ETCD_PEER_CERT_FILE="/home/application/etcd/ssl/server.pem" ETCD_PEER_KEY_FILE="/home/application/etcd/ssl/server-key.pem" ETCD_PEER_TRUSTED_CA_FILE="/home/application/etcd/ssl/ca.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" EOF3.1.5 创建etcd.service 文件$ cat >> /usr/lib/systemd/system/etcd.service << EOF [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify EnvironmentFile=/home/application/etcd/cfg/etcd.conf ExecStart=/home/application/etcd/bin/etcd Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF3.1.6 启动etcd$ systemctl daemon-reload $ systemctl enable etcd $ systemctl start etcd # 检查etcd 服务是否正常 $ ps -ef | grep etcd | grep -v grep root 7885 1 0 15:23 ? 00:00:00 /home/application/etcd/bin/etcd3.1.7etcd 服务可用性测试使用etcdctl 工具测试，模拟写入数据，验证数据； etcdctl 工具，在安装包里有提供 #使用 etcdctl 工具快速检查指定的 etcd 节点（或节点集）的健康状态 $ /home/application/etcd/bin/etcdctl --endpoints=https://172.22.33.223:2379 \ --cacert=/home/application/etcd/ssl/ca.pem \ --cert=/home/application/etcd/ssl/server.pem \ --key=/home/application/etcd/ssl/server-key.pem \ --write-out=table endpoint health +----------------------------+--------+-------------+-------+ | ENDPOINT | HEALTH | TOOK | ERROR | +----------------------------+--------+-------------+-------+ | https://172.22.33.223:2379 | true | 17.746218ms | | +----------------------------+--------+-------------+-------+ #使用 etcdctl 工具查看指定 etcd 节点（或节点集）当前的详细状态信息 $ /home/application/etcd/bin/etcdctl --endpoints=https://172.22.33.223:2379 \ --cacert=/home/application/etcd/ssl/ca.pem \ --cert=/home/application/etcd/ssl/server.pem \ --key=/home/application/etcd/ssl/server-key.pem \ --write-out=table endpoint status +----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | https://172.22.33.223:2379 | 8e9e05c52164694d | 3.5.15 | 20 kB | true | false | 2 | 5 | 5 | | +----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ #使用 etcdctl 工具验证测试 member $ /home/application/etcd/bin/etcdctl --endpoints=https://172.22.33.223:2379 \ --cacert=/home/application/etcd/ssl/ca.pem \ --cert=/home/application/etcd/ssl/server.pem \ --key=/home/application/etcd/ssl/server-key.pem \ --write-out=table member list --write-out=table member list +------------------+---------+------+-----------------------+--------------------------------------------------+------------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | +------------------+---------+------+-----------------------+--------------------------------------------------+------------+ | 8e9e05c52164694d | started | etcd | http://localhost:2380 | http://127.0.0.1:2379,https://172.22.33.223:2379 | false | +------------------+---------+------+-----------------------+--------------------------------------------------+------------+ # etcdctl 工具验证测试数据读写 $ /home/application/etcd/bin/etcdctl --endpoints=https://172.22.33.223:2379 \ --cacert=/home/application/etcd/ssl/ca.pem \ --cert=/home/application/etcd/ssl/server.pem \ --key=/home/application/etcd/ssl/server-key.pem \ put srebro.cn hello OK $ /home/application/etcd/bin/etcdctl --endpoints=https://172.22.33.223:2379 \ --cacert=/home/application/etcd/ssl/ca.pem \ --cert=/home/application/etcd/ssl/server.pem \ --key=/home/application/etcd/ssl/server-key.pem \ get srebro.cn srebro.cn hello3.2 etcd集群部署3.2.1 环境准备机器IP主机名etcd版本172.22.33.223etcd-01v3.3.8172.22.33.224etcd-02v3.3.8172.22.33.225etcd-03v3.3.83.2.2 修改主机名，添加hosts解析在每台主机上执行设置主机名： $ hostnamectl set-hostname etcd-01 $ hostnamectl set-hostname etcd-02 $ hostnamectl set-hostname etcd-03 #在所有节点添加hosts： $ cat >> /etc/hosts << EOF 172.22.33.223 etcd-01 172.22.33.224 etcd-02 172.22.33.225 etcd-03 EOF3.2.3 下载软件包,创建etcd工作目录在每台主机上执行etcd 软件包下载地址： https://github.com/etcd-io/etcd/releases/download/v3.5.15/etcd-v3.5.15-linux-amd64.tar.gz#创建etcd工作目录 $ mkdir -p /home/application/etcd/{bin,cfg,data,ssl} # 下载软件包并解压 $ wget https://github.com/etcd-io/etcd/releases/download/v3.5.15/etcd-v3.5.15-linux-amd64.tar.gz $ tar -xf etcd-v3.5.15-linux-amd64.tar.gz $ mv etcd-v3.5.15-linux-amd64/{etcdctl,etcd} /home/application/etcd/bin/ #验证etcd版本 $ /home/application/etcd/bin/etcd --version etcd Version: 3.5.15 Git SHA: 9a5533382 Go Version: go1.21.12 Go OS/Arch: linux/amd64 3.2.4 使用cfssl 工具签发etcd 证书在172.22.33.223 主机上执行签发证书的动作只需要在一台服务器上完成，签发之后，拷贝证书即可#下载cfssl 签发工具【可能背墙，自行备源】 [root@master1 ~]# curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl [root@master1 ~]# curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson [root@master1 ~]# curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo [root@master1 ~]# chmod +x /usr/local/bin/cfssl*签发证书-配置 CA 并创建 TLS 证书创建 Certificate Authority（CA）， 并为 etcd创建 TLS 证书。# 切换到证书工作目录下 $ cd /home/application/etcd/ssl # 配置 CA 的操作参数和策略。这些参数决定了 CA 的行为，如证书的有效期、证书链的深度、使用的加密算法等 $ cat << EOF | tee ca-config.json { "signing": { "default": { "expiry": "87600h" }, "profiles": { "etcd": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } } EOF # 配置CA 生成证书签名请求（CSR） $ cat << EOF | tee ca-csr.json { "CN": "etcd CA", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Nanjing", "ST": "Nanjing" } ] } EOF #生成 CA 凭证和私钥: $ cfssl gencert -initca ca-csr.json | cfssljson -bare ca #结果将生成以下两个文件： $ ls -l ca*.pem -rw------- 1 root root 1679 8月 26 14:19 ca-key.pem -rw-r--r-- 1 root root 1265 8月 26 14:19 ca.pem签发etcd server证书修改请求文件中hosts字段包含所有etcd节点IP，172.22.33.223，172.22.33.224 ，172.22.33.225# 创建etcd 证书请求文件 $ cat << EOF | tee server-csr.json { "CN": "etcd", "hosts": [ "172.22.33.223", "172.22.33.224", "172.22.33.225" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Nanjing", "ST": "Nanjing" } ] } EOF # 生成etcd server证书: $ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd server-csr.json | cfssljson -bare server #结果将生成以下两个文件： $ ls -l server*.pem -rw------- 1 root root 1675 8月 26 14:20 server-key.pem -rw-r--r-- 1 root root 1338 8月 26 14:20 server.pem同步etcd 证书到其他节点scp -rp /home/application/etcd/ssl/* 172.22.33.224:/home/application/etcd/ssl/* scp -rp /home/application/etcd/ssl/* 172.22.33.225:/home/application/etcd/ssl/*3.2.5 创建配置文件172.22.33.223 etcd-01 节点$ cd /home/application/etcd/cfg $ cat >> /home/application/etcd/cfg/etcd.conf << EOF #[Member] ETCD_NAME="etcd01" ETCD_DATA_DIR="/home/application/etcd/data/default.etcd" ETCD_LISTEN_PEER_URLS="https://172.22.33.223:2380" ETCD_LISTEN_CLIENT_URLS="https://172.22.33.223:2379,http://127.0.0.1:2379" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.22.33.223:2380" ETCD_ADVERTISE_CLIENT_URLS="https://172.22.33.223:2379" ETCD_INITIAL_CLUSTER="etcd01=https://172.22.33.223:2380,etcd02=https://172.22.33.224:2380,etcd03=https://172.22.33.225:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_ENABLE_V2="true" #[Security] ETCD_CERT_FILE="/home/application/etcd/ssl/server.pem" ETCD_KEY_FILE="/home/application/etcd/ssl/server-key.pem" ETCD_TRUSTED_CA_FILE="/home/application/etcd/ssl/ca.pem" ETCD_CLIENT_CERT_AUTH="true" ETCD_PEER_CERT_FILE="/home/application/etcd/ssl/server.pem" ETCD_PEER_KEY_FILE="/home/application/etcd/ssl/server-key.pem" ETCD_PEER_TRUSTED_CA_FILE="/home/application/etcd/ssl/ca.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" EOF172.22.33.224 etcd-02 节点#[Member] ETCD_NAME="etcd02" ETCD_DATA_DIR="/home/application/etcd/data/default.etcd" ETCD_LISTEN_PEER_URLS="https://172.22.33.224:2380" ETCD_LISTEN_CLIENT_URLS="https://172.22.33.224:2379,http://127.0.0.1:2379" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.22.33.224:2380" ETCD_ADVERTISE_CLIENT_URLS="https://172.22.33.224:2379" ETCD_INITIAL_CLUSTER="etcd01=https://172.22.33.223:2380,etcd02=https://172.22.33.224:2380,etcd03=https://172.22.33.225:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_ENABLE_V2="true" #[Security] ETCD_CERT_FILE="/home/application/etcd/ssl/server.pem" ETCD_KEY_FILE="/home/application/etcd/ssl/server-key.pem" ETCD_TRUSTED_CA_FILE="/home/application/etcd/ssl/ca.pem" ETCD_CLIENT_CERT_AUTH="true" ETCD_PEER_CERT_FILE="/home/application/etcd/ssl/server.pem" ETCD_PEER_KEY_FILE="/home/application/etcd/ssl/server-key.pem" ETCD_PEER_TRUSTED_CA_FILE="/home/application/etcd/ssl/ca.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" EOF172.22.33.225 etcd-03 节点#[Member] ETCD_NAME="etcd03" ETCD_DATA_DIR="/home/application/etcd/data/default.etcd" ETCD_LISTEN_PEER_URLS="https://172.22.33.225:2380" ETCD_LISTEN_CLIENT_URLS="https://172.22.33.225:2379,http://127.0.0.1:2379" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.22.33.225:2380" ETCD_ADVERTISE_CLIENT_URLS="https://172.22.33.225:2379" ETCD_INITIAL_CLUSTER="etcd01=https://172.22.33.223:2380,etcd02=https://172.22.33.224:2380,etcd03=https://172.22.33.225:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_ENABLE_V2="true" #[Security] ETCD_CERT_FILE="/home/application/etcd/ssl/server.pem" ETCD_KEY_FILE="/home/application/etcd/ssl/server-key.pem" ETCD_TRUSTED_CA_FILE="/home/application/etcd/ssl/ca.pem" ETCD_CLIENT_CERT_AUTH="true" ETCD_PEER_CERT_FILE="/home/application/etcd/ssl/server.pem" ETCD_PEER_KEY_FILE="/home/application/etcd/ssl/server-key.pem" ETCD_PEER_TRUSTED_CA_FILE="/home/application/etcd/ssl/ca.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" EOF3.2.6 创建etcd.service 文件在每台主机上执行$ cat >> /usr/lib/systemd/system/etcd.service << EOF [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify EnvironmentFile=/home/application/etcd/cfg/etcd.conf ExecStart=/home/application/etcd/bin/etcd Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF3.2.7 启动etcd在每台主机上执行$ systemctl daemon-reload $ systemctl enable etcd $ systemctl start etcd # 检查etcd 服务是否正常 $ ps -ef | grep etcd | grep -v grep root 7885 1 0 15:23 ? 00:00:00 /home/application/etcd/bin/etcd3.2.8 etcd 集群服务可用性测试使用etcdctl 工具测试，模拟写入数据，验证数据； etcdctl 工具，在安装包里有提供 #使用 etcdctl 工具快速检查指定的 etcd 节点（或节点集）的健康状态 $ /home/application/etcd/bin/etcdctl --endpoints=https://172.22.33.223:2379,https://172.22.33.224:2379,https://172.22.33.225:2379 \ --cacert=/home/application/etcd/ssl/ca.pem \ --cert=/home/application/etcd/ssl/server.pem \ --key=/home/application/etcd/ssl/server-key.pem \ --write-out=table endpoint health +----------------------------+--------+-------------+-------+ | ENDPOINT | HEALTH | TOOK | ERROR | +----------------------------+--------+-------------+-------+ | https://172.22.33.224:2379 | true | 16.120369ms | | | https://172.22.33.223:2379 | true | 16.605784ms | | | https://172.22.33.225:2379 | true | 19.851873ms | | +----------------------------+--------+-------------+-------+ #使用 etcdctl 工具查看指定 etcd 节点（或节点集）当前的详细状态信息 $ /home/application/etcd/bin/etcdctl --endpoints=https://172.22.33.223:2379,https://172.22.33.224:2379,https://172.22.33.225:2379 \ --cacert=/home/application/etcd/ssl/ca.pem \ --cert=/home/application/etcd/ssl/server.pem \ --key=/home/application/etcd/ssl/server-key.pem \ --write-out=table endpoint status +----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | https://172.22.33.223:2379 | b905915d4c303d2 | 3.5.15 | 20 kB | false | false | 2 | 11 | 11 | | | https://172.22.33.224:2379 | 912027a2ec294592 | 3.5.15 | 20 kB | true | false | 2 | 11 | 11 | | | https://172.22.33.225:2379 | 81330706eb2302ba | 3.5.15 | 20 kB | false | false | 2 | 11 | 11 | | +----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ #使用 etcdctl 工具验证测试 member $ /home/application/etcd/bin/etcdctl --endpoints=https://172.22.33.223:2379,https://172.22.33.224:2379,https://172.22.33.225:2379 \ --cacert=/home/application/etcd/ssl/ca.pem \ --cert=/home/application/etcd/ssl/server.pem \ --key=/home/application/etcd/ssl/server-key.pem \ --write-out=table member list +------------------+---------+--------+----------------------------+----------------------------+------------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | +------------------+---------+--------+----------------------------+----------------------------+------------+ | b905915d4c303d2 | started | etcd01 | https://172.22.33.223:2380 | https://172.22.33.223:2379 | false | | 81330706eb2302ba | started | etcd03 | https://172.22.33.225:2380 | https://172.22.33.225:2379 | false | | 912027a2ec294592 | started | etcd02 | https://172.22.33.224:2380 | https://172.22.33.224:2379 | false | +------------------+---------+--------+----------------------------+----------------------------+------------+ # etcdctl 工具验证测试数据读写 $ /home/application/etcd/bin/etcdctl --endpoints=https://172.22.33.223:2379,https://172.22.33.224:2379,https://172.22.33.225:2379 \ --cacert=/home/application/etcd/ssl/ca.pem \ --cert=/home/application/etcd/ssl/server.pem \ --key=/home/application/etcd/ssl/server-key.pem \ put srebro.cn hi OK $ /home/application/etcd/bin/etcdctl --endpoints=https://172.22.33.223:2379,https://172.22.33.224:2379,https://172.22.33.225:2379 \ --cacert=/home/application/etcd/ssl/ca.pem \ --cert=/home/application/etcd/ssl/server.pem \ --key=/home/application/etcd/ssl/server-key.pem \ get srebro.cn srebro.cn OK四、kubeadm 使用集群外etcd，部署K8S集群kubeadm 安装过程省略，直接贴上kubeadm-init.yaml 配置文件;主要查看etcd 那块的配置；另外需要注意，ETCD 集群 与 apiservice 之间通信，是否采用TLS 加密，我上面部署的etcd集群实则是没有采用TLS 加密的apiVersion: kubeadm.k8s.io/v1beta3 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: 35sesk.5tm3tqc66tzlse8c ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: advertiseAddress: 172.22.33.220 bindPort: 6443 nodeRegistration: criSocket: unix:///var/run/cri-dockerd.sock imagePullPolicy: IfNotPresent name: k8s-master01 taints: - effect: NoSchedule key: node-role.kubernetes.io/master --- apiServer: certSANs: - kubernetes - kubernetes.default - kubernetes.default.svc - kubernetes.default.svc.cluster.local - 172.22.33.220 - 112.11.11.11 - k8s.srebro.cn timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta3 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controlPlaneEndpoint: 172.22.33.220:6443 controllerManager: {} dns: {} etcd: external: endpoints: - "http://172.22.33.210:2379" - "http://172.22.33.211:2379" - "http://172.22.33.212:2379" imageRepository: registry.aliyuncs.com/google_containers kind: ClusterConfiguration kubernetesVersion: v1.24.9 networking: dnsDomain: cluster.local podSubnet: 10.244.0.0/16 serviceSubnet: 10.96.0.0/12 scheduler: {}不使用TLS 加密 ，需要配置如下内容：如果需要使用TLS 加密 ，需要配置如下内容：五、参考https://blog.51cto.com/u_15481067/11750804
kubeadm部署 Kubernetes(k8s) 高可用集群 V1.28+ Etcd集群
kubeadm部署 Kubernetes(k8s) 高可用集群 V1.28+ Etcd集群kubeadm是官方社区推出的一个用于快速部署kubernetes集群的工具。1. 安装要求在开始之前，部署Kubernetes集群机器需要满足以下几个条件：10台机器，操作系统Openeuler22.03 LTS SP4硬件配置：2GB或更多RAM，2个CPU或更多CPU，硬盘30GB或更多，docker 数据卷单独挂载集群中所有机器之间网络互通可以访问外网，需要拉取镜像，【镜像代理自行解决】禁止swap分区2. 准备环境角色IP组件功能描述Haproxy172.22.33.210haproxyK8S入口，负载均衡k8s-master01172.22.33.215docker，cri-dockerd，kube-apiserver，kube-scheduler，kube-controller-manager，kubelet，kube-proxy，coredns，calicok8s master01节点k8s-master02172.22.33.216docker，cri-dockerd，kube-apiserver，kube-scheduler，kube-controller-manager，kubelet，kube-proxy，calicok8s master02节点k8s-master03172.22.33.217docker，cri-dockerd，kube-apiserver，kube-scheduler，kube-controller-manager，kubelet，kube-proxy，calicok8s master03节点k8s-node01172.22.33.218docker，cri-dockerd，kubelet，kube-proxy，coredns，calicok8s-node01 工作节点k8s-node02172.22.33.219docker，cri-dockerd，kubelet，kube-proxy，coredns，calicok8s-node02 工作节点k8s-node03172.22.33.220docker，cri-dockerd，kubelet，kube-proxy，coredns，calicok8s-node03 工作节点etcd-01172.22.33.223etcdetcd-01服务器etcd-02172.22.33.224etcdetcd-02服务器etcd-0372.22.33.225etcdetcd-03服务器2.1 环境初始化K8S 集群服务器 都需要配置关闭防火墙： $ systemctl stop firewalld $ systemctl disable firewalld 关闭selinux： $ sed -i 's/enforcing/disabled/' /etc/selinux/config # 永久 $ setenforce 0 # 临时 关闭swap：[临时和永久关闭] #临时关闭 $ swapoff -a #永久关闭 $ sed -ri 's/.*swap.*/#&/' /etc/fstab 设置主机名： $ hostnamectl set-hostname k8s-master01 $ hostnamectl set-hostname k8s-master02 $ hostnamectl set-hostname k8s-master03 $ hostnamectl set-hostname k8s-node01 $ hostnamectl set-hostname k8s-node02 $ hostnamectl set-hostname k8s-node03 #在K8S上所有节点添加hosts： $ cat >> /etc/hosts << EOF 172.22.33.215 k8s-master01 172.22.33.216 k8s-master02 172.22.33.217 k8s-master03 172.22.33.218 k8s-node01 172.22.33.219 k8s-node02 172.22.33.220 k8s-node03 EOF #开启内核路由转发 sed -i 's/net.ipv4.ip_forward=0/net.ipv4.ip_forward=1/g' /etc/sysctl.conf #将桥接的IPv4,IPV6流量传递到iptables的链： $ cat > /etc/sysctl.d/k8s.conf << EOF net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 vm.swappiness = 0 EOF # 生效 $ sysctl --system #每个节点都需要安装 IPVS 的相关工具和加载ipvs内核模块 $ yum install ipvsadm #在所有节点执行以下命令 $ cat > /etc/sysconfig/modules/ipvs.modules <<EOF #!/bin/bash modprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack_ipv4 EOF $ chmod 755 /etc/sysconfig/modules/ipvs.modules $ bash /etc/sysconfig/modules/ipvs.modules #查看IPVS模块加载情况 $ lsmod | grep -e ip_vs -e nf_conntrack_ipv4 #能看到ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack_ipv4 加载成功 时间同步： $ yum install ntpdate -y $ ntpdate ntp.ntsc.ac.cn3. 安装Docker/cri-dockerd/kubeadm/kubelet/kubectl在所有K8S 集群的节点上，执行Kubernetes1.24 之后, 在k8s中使用docker，除了安装docker 以外，还需要=安装cri-dockerd 组件； 不然就使用Containerd3.1.1 docker二进制包下载docker-ce 社区下载地址：$ wget https://mirrors.nju.edu.cn/docker-ce/linux/static/stable/x86_64/docker-20.10.24.tgz3.1.2 解压，拷贝至/usr/bin 下$ tar -xf docker-20.10.24.tgz $ cp docker/* /usr/bin $ which docker3.1.3 编写docker.service文件$ cat > /etc/systemd/system/docker.service <<EOF [Unit] Description=Docker Application Container Engine Documentation=https://docs.docker.com After=network-online.target firewalld.service Wants=network-online.target [Service] Type=notify ExecStart=/usr/bin/dockerd ExecReload=/bin/kill -s HUP $MAINPID LimitNOFILE=65535 LimitNPROC=65535 LimitCORE=65535 TimeoutStartSec=0 Delegate=yes KillMode=process Restart=on-failure StartLimitBurst=3 StartLimitInterval=60s [Install] WantedBy=multi-user.target EOF3.1.4 挂载docker默认存储路径[可选]docker的默认工作路径在/var/lib/docker ，最好不修改默认存储路径，可以做软链接#建立工作目录 $ mkdir /home/application/ #格式化磁盘 $ mkfs.ext4 /dev/sdb #磁盘永久挂载 $ vim /etc/fstab /dev/sdb /home/application ext4 defaults 0 0 #使挂载生效 $ mount -a # 创建docker 工作目录 $ mkdir /home/application/docker #创建软链接 $ ln -s /home/application/docker /var/lib/3.1.5 添加可执行权限$ chmod +x /etc/systemd/system/docker.service3.1.6 启动，加载，开机自启动$ systemctl daemon-reload $ systemctl start docker.service $ systemctl enable docker.service3.1.7 配置镜像加速器$ mkdir -p /etc/docker $ tee /etc/docker/daemon.json <<-'EOF' { "registry-mirrors": ["https://docker.srebro.cn"], "exec-opts": ["native.cgroupdriver=systemd"] } EOF $ systemctl daemon-reload $ systemctl restart docker⚠️tips：如果仓库不是https 的，需要在 /etc/docker/daemon.json 中 添加忽略；比如下方的示例"insecure-registries": [ "https://harbor.srebro.cn", "https://it-docker.pkg.devops.com" ]3.2.1 二进制部署cri-dockerd项目地址: https://github.com/Mirantis/cri-dockerd# 下载 wget https://github.com/Mirantis/cri-dockerd/releases/download/v0.3.14/cri-dockerd-0.3.14.amd64.tgz tar -xf cri-dockerd-0.3.14.amd64.tgz cp cri-dockerd/cri-dockerd /usr/bin/ chmod +x /usr/bin/cri-dockerd # 配置启动文件 cat <<"EOF" > /usr/lib/systemd/system/cri-docker.service [Unit] Description=CRI Interface for Docker Application Container Engine Documentation=https://docs.mirantis.com After=network-online.target firewalld.service docker.service Wants=network-online.target Requires=cri-docker.socket [Service] Type=notify ExecStart=/usr/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.9 ExecReload=/bin/kill -s HUP $MAINPID TimeoutSec=0 RestartSec=2 Restart=always StartLimitBurst=3 StartLimitInterval=60s LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity TasksMax=infinity Delegate=yes KillMode=process [Install] WantedBy=multi-user.target EOF # 生成socket 文件 cat <<"EOF" > /usr/lib/systemd/system/cri-docker.socket [Unit] Description=CRI Docker Socket for the API PartOf=cri-docker.service [Socket] ListenStream=%t/cri-dockerd.sock SocketMode=0660 SocketUser=root SocketGroup=root [Install] WantedBy=sockets.target EOF # 启动CRI-DOCKER systemctl daemon-reload systemctl start cri-docker systemctl enable cri-docker systemctl is-active cri-docker3.2.2 添加阿里云YUM软件源$ cat > /etc/yum.repos.d/kubernetes.repo << EOF [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF3.2.3 安装kubeadm，kubelet和kubectl由于版本更新频繁，这里指定版本号部署：# 查看所有的可用版本 $ yum list kubeadm kubelet kubectl --showduplicates | sort -r #在所有节点安装 $ yum install -y kubelet-1.28.2-0 kubeadm-1.28.2-0 kubectl-1.28.2-0 $ systemctl enable kubelet4. 部署Haproxy负载均衡4.1 yum安装haproxy在172.22.33.210 服务器上部署yum -y install haproxy4.2 修改配置文件$ cat > /etc/haproxy/haproxy.cfg << EOF global log /dev/log local0 warning chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon stats socket /var/lib/haproxy/stats defaults log global option httplog option dontlognull timeout connect 5000 timeout client 50000 timeout server 50000 frontend kube-apiserver bind *:36443 mode tcp option tcplog default_backend kube-apiserver backend kube-apiserver mode tcp option tcplog option tcp-check balance roundrobin default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100 server kube-apiserver-1 172.22.33.215:6443 check # Replace the IP address with your own. server kube-apiserver-2 172.22.33.216:6443 check # Replace the IP address with your own. server kube-apiserver-3 172.22.33.217:6443 check # Replace the IP address with your own. EOF4.3 启动Haproxy$ systemctl enable haproxy $ systemctl start haproxy5.部署etcd 集群5.1 ETCD集群环境机器IP主机名etcd版本172.22.33.223etcd-01v3.3.8172.22.33.224etcd-02v3.3.8172.22.33.225etcd-03v3.3.85.2 修改主机名，添加hosts解析在每台主机上执行设置主机名： $ hostnamectl set-hostname etcd-01 $ hostnamectl set-hostname etcd-02 $ hostnamectl set-hostname etcd-03 #在所有节点添加hosts： $ cat >> /etc/hosts << EOF 172.22.33.223 etcd-01 172.22.33.224 etcd-02 172.22.33.225 etcd-03 EOF5.3 下载软件包,创建etcd工作目录在每台主机上执行etcd 软件包下载地址： https://github.com/etcd-io/etcd/releases/download/v3.5.15/etcd-v3.5.15-linux-amd64.tar.gz#创建etcd工作目录 $ mkdir -p /home/application/etcd/{bin,cfg,data,ssl} # 下载软件包并解压 $ wget https://github.com/etcd-io/etcd/releases/download/v3.5.15/etcd-v3.5.15-linux-amd64.tar.gz $ tar -xf etcd-v3.5.15-linux-amd64.tar.gz $ mv etcd-v3.5.15-linux-amd64/{etcdctl,etcd} /home/application/etcd/bin/ #验证etcd版本 $ /home/application/etcd/bin/etcd --version etcd Version: 3.5.15 Git SHA: 9a5533382 Go Version: go1.21.12 Go OS/Arch: linux/amd64 5.4 使用cfssl 工具签发etcd 证书在172.22.33.223 主机上执行签发证书的动作只需要在一台服务器上完成，签发之后，拷贝证书即可#下载cfssl 签发工具【可能背墙，自行备源】 [root@master1 ~]# curl -L https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 -o /usr/local/bin/cfssl [root@master1 ~]# curl -L https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 -o /usr/local/bin/cfssljson [root@master1 ~]# curl -L https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 -o /usr/local/bin/cfssl-certinfo [root@master1 ~]# chmod +x /usr/local/bin/cfssl*签发证书-配置 CA 并创建 TLS 证书创建 Certificate Authority（CA）， 并为 etcd创建 TLS 证书。# 切换到证书工作目录下 $ cd /home/application/etcd/ssl # 配置 CA 的操作参数和策略。这些参数决定了 CA 的行为，如证书的有效期、证书链的深度、使用的加密算法等 $ cat << EOF | tee ca-config.json { "signing": { "default": { "expiry": "87600h" }, "profiles": { "etcd": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } } EOF # 配置CA 生成证书签名请求（CSR） $ cat << EOF | tee ca-csr.json { "CN": "etcd CA", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Nanjing", "ST": "Nanjing" } ] } EOF #生成 CA 凭证和私钥: $ cfssl gencert -initca ca-csr.json | cfssljson -bare ca #结果将生成以下两个文件： $ ls -l ca*.pem -rw------- 1 root root 1679 8月 26 14:19 ca-key.pem -rw-r--r-- 1 root root 1265 8月 26 14:19 ca.pem签发etcd server证书修改请求文件中hosts字段包含所有etcd节点IP，172.22.33.223，172.22.33.224 ，172.22.33.225# 创建etcd 证书请求文件 $ cat << EOF | tee server-csr.json { "CN": "etcd", "hosts": [ "172.22.33.223", "172.22.33.224", "172.22.33.225" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Nanjing", "ST": "Nanjing" } ] } EOF # 生成etcd server证书: $ cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=etcd server-csr.json | cfssljson -bare server #结果将生成以下两个文件： $ ls -l server*.pem -rw------- 1 root root 1675 8月 26 14:20 server-key.pem -rw-r--r-- 1 root root 1338 8月 26 14:20 server.pem同步etcd 证书到其他节点scp -rp /home/application/etcd/ssl/* 172.22.33.224:/home/application/etcd/ssl/* scp -rp /home/application/etcd/ssl/* 172.22.33.225:/home/application/etcd/ssl/*5.5 创建配置文件172.22.33.223 etcd-01 节点$ cd /home/application/etcd/cfg $ cat >> /home/application/etcd/cfg/etcd.conf << EOF #[Member] ETCD_NAME="etcd01" ETCD_DATA_DIR="/home/application/etcd/data/default.etcd" ETCD_LISTEN_PEER_URLS="https://172.22.33.223:2380" ETCD_LISTEN_CLIENT_URLS="https://172.22.33.223:2379,http://127.0.0.1:2379" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.22.33.223:2380" ETCD_ADVERTISE_CLIENT_URLS="https://172.22.33.223:2379" ETCD_INITIAL_CLUSTER="etcd01=https://172.22.33.223:2380,etcd02=https://172.22.33.224:2380,etcd03=https://172.22.33.225:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_ENABLE_V2="true" #[Security] ETCD_CERT_FILE="/home/application/etcd/ssl/server.pem" ETCD_KEY_FILE="/home/application/etcd/ssl/server-key.pem" ETCD_TRUSTED_CA_FILE="/home/application/etcd/ssl/ca.pem" ETCD_CLIENT_CERT_AUTH="true" ETCD_PEER_CERT_FILE="/home/application/etcd/ssl/server.pem" ETCD_PEER_KEY_FILE="/home/application/etcd/ssl/server-key.pem" ETCD_PEER_TRUSTED_CA_FILE="/home/application/etcd/ssl/ca.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" EOF172.22.33.224 etcd-02 节点#[Member] ETCD_NAME="etcd02" ETCD_DATA_DIR="/home/application/etcd/data/default.etcd" ETCD_LISTEN_PEER_URLS="https://172.22.33.224:2380" ETCD_LISTEN_CLIENT_URLS="https://172.22.33.224:2379,http://127.0.0.1:2379" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.22.33.224:2380" ETCD_ADVERTISE_CLIENT_URLS="https://172.22.33.224:2379" ETCD_INITIAL_CLUSTER="etcd01=https://172.22.33.223:2380,etcd02=https://172.22.33.224:2380,etcd03=https://172.22.33.225:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_ENABLE_V2="true" #[Security] ETCD_CERT_FILE="/home/application/etcd/ssl/server.pem" ETCD_KEY_FILE="/home/application/etcd/ssl/server-key.pem" ETCD_TRUSTED_CA_FILE="/home/application/etcd/ssl/ca.pem" ETCD_CLIENT_CERT_AUTH="true" ETCD_PEER_CERT_FILE="/home/application/etcd/ssl/server.pem" ETCD_PEER_KEY_FILE="/home/application/etcd/ssl/server-key.pem" ETCD_PEER_TRUSTED_CA_FILE="/home/application/etcd/ssl/ca.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" EOF172.22.33.225 etcd-03 节点#[Member] ETCD_NAME="etcd03" ETCD_DATA_DIR="/home/application/etcd/data/default.etcd" ETCD_LISTEN_PEER_URLS="https://172.22.33.225:2380" ETCD_LISTEN_CLIENT_URLS="https://172.22.33.225:2379,http://127.0.0.1:2379" #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://172.22.33.225:2380" ETCD_ADVERTISE_CLIENT_URLS="https://172.22.33.225:2379" ETCD_INITIAL_CLUSTER="etcd01=https://172.22.33.223:2380,etcd02=https://172.22.33.224:2380,etcd03=https://172.22.33.225:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" ETCD_INITIAL_CLUSTER_STATE="new" ETCD_ENABLE_V2="true" #[Security] ETCD_CERT_FILE="/home/application/etcd/ssl/server.pem" ETCD_KEY_FILE="/home/application/etcd/ssl/server-key.pem" ETCD_TRUSTED_CA_FILE="/home/application/etcd/ssl/ca.pem" ETCD_CLIENT_CERT_AUTH="true" ETCD_PEER_CERT_FILE="/home/application/etcd/ssl/server.pem" ETCD_PEER_KEY_FILE="/home/application/etcd/ssl/server-key.pem" ETCD_PEER_TRUSTED_CA_FILE="/home/application/etcd/ssl/ca.pem" ETCD_PEER_CLIENT_CERT_AUTH="true" EOF5.6 创建etcd.service 文件在每台主机上执行$ cat >> /usr/lib/systemd/system/etcd.service << EOF [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify EnvironmentFile=/home/application/etcd/cfg/etcd.conf ExecStart=/home/application/etcd/bin/etcd Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF5.7 启动etcd在每台主机上执行$ systemctl daemon-reload $ systemctl enable etcd $ systemctl start etcd # 检查etcd 服务是否正常 $ ps -ef | grep etcd | grep -v grep root 7885 1 0 15:23 ? 00:00:00 /home/application/etcd/bin/etcd5.8 etcd 集群服务可用性测试使用etcdctl 工具测试，模拟写入数据，验证数据； etcdctl 工具，在安装包里有提供 #使用 etcdctl 工具快速检查指定的 etcd 节点（或节点集）的健康状态 $ /home/application/etcd/bin/etcdctl --endpoints=https://172.22.33.223:2379,https://172.22.33.224:2379,https://172.22.33.225:2379 \ --cacert=/home/application/etcd/ssl/ca.pem \ --cert=/home/application/etcd/ssl/server.pem \ --key=/home/application/etcd/ssl/server-key.pem \ --write-out=table endpoint health +----------------------------+--------+-------------+-------+ | ENDPOINT | HEALTH | TOOK | ERROR | +----------------------------+--------+-------------+-------+ | https://172.22.33.224:2379 | true | 16.120369ms | | | https://172.22.33.223:2379 | true | 16.605784ms | | | https://172.22.33.225:2379 | true | 19.851873ms | | +----------------------------+--------+-------------+-------+ #使用 etcdctl 工具查看指定 etcd 节点（或节点集）当前的详细状态信息 $ /home/application/etcd/bin/etcdctl --endpoints=https://172.22.33.223:2379,https://172.22.33.224:2379,https://172.22.33.225:2379 \ --cacert=/home/application/etcd/ssl/ca.pem \ --cert=/home/application/etcd/ssl/server.pem \ --key=/home/application/etcd/ssl/server-key.pem \ --write-out=table endpoint status +----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS | +----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ | https://172.22.33.223:2379 | b905915d4c303d2 | 3.5.15 | 20 kB | false | false | 2 | 11 | 11 | | | https://172.22.33.224:2379 | 912027a2ec294592 | 3.5.15 | 20 kB | true | false | 2 | 11 | 11 | | | https://172.22.33.225:2379 | 81330706eb2302ba | 3.5.15 | 20 kB | false | false | 2 | 11 | 11 | | +----------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+ #使用 etcdctl 工具验证测试 member $ /home/application/etcd/bin/etcdctl --endpoints=https://172.22.33.223:2379,https://172.22.33.224:2379,https://172.22.33.225:2379 \ --cacert=/home/application/etcd/ssl/ca.pem \ --cert=/home/application/etcd/ssl/server.pem \ --key=/home/application/etcd/ssl/server-key.pem \ --write-out=table member list +------------------+---------+--------+----------------------------+----------------------------+------------+ | ID | STATUS | NAME | PEER ADDRS | CLIENT ADDRS | IS LEARNER | +------------------+---------+--------+----------------------------+----------------------------+------------+ | b905915d4c303d2 | started | etcd01 | https://172.22.33.223:2380 | https://172.22.33.223:2379 | false | | 81330706eb2302ba | started | etcd03 | https://172.22.33.225:2380 | https://172.22.33.225:2379 | false | | 912027a2ec294592 | started | etcd02 | https://172.22.33.224:2380 | https://172.22.33.224:2379 | false | +------------------+---------+--------+----------------------------+----------------------------+------------+ # etcdctl 工具验证测试数据读写 $ /home/application/etcd/bin/etcdctl --endpoints=https://172.22.33.223:2379,https://172.22.33.224:2379,https://172.22.33.225:2379 \ --cacert=/home/application/etcd/ssl/ca.pem \ --cert=/home/application/etcd/ssl/server.pem \ --key=/home/application/etcd/ssl/server-key.pem \ put srebro.cn hi OK $ /home/application/etcd/bin/etcdctl --endpoints=https://172.22.33.223:2379,https://172.22.33.224:2379,https://172.22.33.225:2379 \ --cacert=/home/application/etcd/ssl/ca.pem \ --cert=/home/application/etcd/ssl/server.pem \ --key=/home/application/etcd/ssl/server-key.pem \ get srebro.cn srebro.cn OK5.9 拷贝etcd 集群证书到K8S-master 节点上在所有master 节点上执行#创建etcd证书工作目录 $ mkdir -p /etc/kubernetes/pki/etcd #只需要拿etcd 集群中的任意一个节点上的证书，拷贝过去即可 $ scp -rp 172.22.33.223:/home/application/etcd/ssl/* /etc/kubernetes/pki/etcd/*6. 部署Kubernetes Master官方kubeadm 初始化说明：https://kubernetes.io/zh/docs/reference/setup-tools/kubeadm/kubeadm-init/#config-filehttps://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#initializing-your-control-plane-node6.1 准备好kubeadm-init.yaml初始化文件6.1.1 使用外置ETCD 集群， 完成 kubeadm 初始化⚠️ 包含etcd 证书#在每个master 节点创建K8S,init 的工作目录 $ mkdir -p /etc/kubernetes/init cat >> /etc/kubernetes/init/kubeadm-init.yaml << EOF --- apiVersion: kubeadm.k8s.io/v1beta3 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: 35sesk.5tm3tqc66tzlse8c ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: advertiseAddress: 172.22.33.223 bindPort: 6443 nodeRegistration: criSocket: unix:///var/run/cri-dockerd.sock imagePullPolicy: IfNotPresent name: k8s-master01 taints: - effect: NoSchedule key: node-role.kubernetes.io/master --- apiServer: certSANs: - kubernetes - kubernetes.default - kubernetes.default.svc - kubernetes.default.svc.cluster.local - 172.22.33.210 - 172.22.33.215 - 172.22.33.216 - 172.22.33.217 - 112.94.71.21 - k8s.srebro.cn timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta3 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controlPlaneEndpoint: 172.22.33.210:36443 controllerManager: {} dns: {} etcd: external: endpoints: - "https://172.22.33.223:2379" - "https://172.22.33.224:2379" - "https://172.22.33.225:2379" caFile: /etc/kubernetes/pki/etcd/ca.pem certFile: /etc/kubernetes/pki/etcd/server.pem keyFile: /etc/kubernetes/pki/etcd/server-key.pem imageRepository: registry.aliyuncs.com/google_containers kind: ClusterConfiguration kubernetesVersion: v1.28.2 networking: dnsDomain: cluster.local podSubnet: 10.244.0.0/16 serviceSubnet: 10.96.0.0/12 scheduler: {} EOF6.2 在所有master 节点上，pull 镜像到本地init 之前，先下载镜像到master 机器上#k8s-master01 节点上 $ kubeadm config images pull --config kubeadm-init.yaml [config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.28.2 [config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.28.2 [config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.28.2 [config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.28.2 [config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.9 [config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.10.1 #k8s-master02 节点上 $ kubeadm config images pull --config kubeadm-init.yaml [config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.28.2 [config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.28.2 [config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.28.2 [config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.28.2 [config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.9 [config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.10.1 #k8s-master03 节点上 $ kubeadm config images pull --config kubeadm-init.yaml [config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.28.2 [config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.28.2 [config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.28.2 [config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.28.2 [config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.9 [config/images] Pulled registry.aliyuncs.com/google_containers/coredns:v1.10.1 6.3 在master01 节点上，完成kubeadm 初始化$ cd /etc/kubernetes/init $ kubeadm init --config kubeadm-init.yaml [init] Using Kubernetes version: v1.28.2 [preflight] Running pre-flight checks [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service' [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' W0828 15:30:08.273727 10698 checks.go:835] detected that the sandbox image "registry.aliyuncs.com/google_containers/pause:3.7" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.aliyuncs.com/google_containers/pause:3.9" as the CRI sandbox image. [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [k8s-master01 k8s.srebro.cn kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.22.33.223 172.22.33.226 172.22.33.224 112.94.71.21] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] External etcd mode: Skipping etcd/ca certificate authority generation [certs] External etcd mode: Skipping etcd/server certificate generation [certs] External etcd mode: Skipping etcd/peer certificate generation [certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation [certs] External etcd mode: Skipping apiserver-etcd-client certificate generation [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" W0828 15:30:09.375139 10698 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "admin.conf" kubeconfig file W0828 15:30:09.562271 10698 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "kubelet.conf" kubeconfig file W0828 15:30:09.703979 10698 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "controller-manager.conf" kubeconfig file W0828 15:30:09.998820 10698 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [kubeconfig] Writing "scheduler.conf" kubeconfig file [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 18.596178 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Skipping phase. Please see --upload-certs [mark-control-plane] Marking the node k8s-master01 as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] [mark-control-plane] Marking the node k8s-master01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] [bootstrap-token] Using token: 35sesk.5tm3tqc66tzlse8c [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS W0828 15:30:31.282450 10698 endpoint.go:57] [endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ You can now join any number of control-plane nodes by copying certificate authorities and service account keys on each node and then running the following as root: kubeadm join 172.22.33.210:36443 --token 35sesk.5tm3tqc66tzlse8c \ --discovery-token-ca-cert-hash sha256:5a4476ac560c2c5c577abea0f1310c040385e4c12cfd9038734a68952c9fe815 \ --control-plane Then you can join any number of worker nodes by running the following on each as root: kubeadm join 172.22.33.210:36443 --token 35sesk.5tm3tqc66tzlse8c \ --discovery-token-ca-cert-hash sha256:5a4476ac560c2c5c577abea0f1310c040385e4c12cfd9038734a68952c9fe815 拷贝kubectl使用的连接k8s认证文件到默认路径$ mkdir -p $HOME/.kube $ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config $ sudo chown $(id -u):$(id -g) $HOME/.kube/config6.4 加入 Kubernetes Master 节点因为我们的K8S集群已经初始化过了，再添加其他master 节点到集群中，首要添加是把已经初始化过的证书拷贝到其他master 节点上6.4.1 在master02和master03节点复制相关证书#创建K8S证书工作目录 $ mkdir -p /etc/kubernetes/pki/ $ scp -r 172.22.33.215:/etc/kubernetes/pki/ca.* /etc/kubernetes/pki/ $ scp -r 172.22.33.215:/etc/kubernetes/pki/sa.* /etc/kubernetes/pki/ $ scp -r 172.22.33.215:/etc/kubernetes/pki/front-proxy-ca.* /etc/kubernetes/pki/ #拷贝k8s-admin 文件 $ scp -r 172.22.33.215:/etc/kubernetes/admin.conf /etc/kubernetes/6.4.2 把 master02和master03节点 添加到Kubernetes 集群中使用刚刚 master01 在kubeadm init输出的kubeadm join命令：#加入控制平面 $ kubeadm join 172.22.33.210:36443 --token 35sesk.5tm3tqc66tzlse8c \ --discovery-token-ca-cert-hash sha256:5a4476ac560c2c5c577abea0f1310c040385e4c12cfd9038734a68952c9fe815 \ --control-plane6.4.3 在master01节点上查看master节点个数$ kubectl get node NAME STATUS ROLES AGE VERSION k8s-master01 NotReady control-plane,master 8m26s v1.28.2 k8s-master02 NotReady control-plane,master 2m54s v1.28.2 k8s-master03 NotReady control-plane,master 59s v1.28.2可以看到，所有的master 节点都已经加入到集群中了；NotReady是需要等待CNI网络插件安装好7. 加入Kubernetes Node 节点官方文档： https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-join/在K8S node 节点上操作k8s-node01172.22.33.218k8s-node02172.22.33.219k8s-node03172.22.33.220使用刚刚master01 在kubeadm init输出的kubeadm join命令：$ kubeadm join 172.22.33.210:36443 --token 35sesk.5tm3tqc66tzlse8c \ --discovery-token-ca-cert-hash sha256:5a4476ac560c2c5c577abea0f1310c040385e4c12cfd9038734a68952c9fe815 默认token有效期为24小时，当过期之后，该token就不可用了。这时就需要重新创建token，可以直接在master 节点上，使用命令快捷生成：$ kubeadm token create --print-join-command8. 部署容器网络（CNI）https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#pod-network$ wget https://projectcalico.docs.tigera.io/archive/v3.21/manifests/calico.yaml下载完后还需要修改里面定义Pod网络（CALICO_IPV4POOL_CIDR），与前面kubeadm init指定的--pod-network-cidr=10.244.0.0/16 保持一致； 默认是 192.168.0.0/16- name: CALICO_IPV4POOL_CIDR value: "10.244.0.0/16"修改完后应用清单：$ kubectl apply -f calico.yaml $ kubectl get pods -n kube-system查看所有节点，以及pod，svc情况$ kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master01 Ready control-plane,master 70m v1.28.2 k8s-master02 Ready control-plane,master 64m v1.28.2 k8s-master03 Ready control-plane,master 62m v1.28.2 k8s-node01 Ready <none> 46m v1.28.2 k8s-node02 Ready <none> 46m v1.28.2 k8s-node03 Ready <none> 46m v1.28.2[root@openeuler ~]# kubectl get all -A NAMESPACE NAME READY STATUS RESTARTS AGE default pod/my-web-6b6c9df775-2p67z 1/1 Running 0 45m default pod/my-web-6b6c9df775-76f8w 1/1 Running 0 45m default pod/my-web-6b6c9df775-t2zrm 1/1 Running 0 45m default pod/my-web-6b6c9df775-zs6db 1/1 Running 0 45m kube-system pod/calico-kube-controllers-5bb48c55fd-vvktf 1/1 Running 0 52m kube-system pod/calico-node-7ng4l 1/1 Running 0 52m kube-system pod/calico-node-lrf24 1/1 Running 0 47m kube-system pod/calico-node-qxnxg 1/1 Running 0 52m kube-system pod/calico-node-vwjt4 1/1 Running 0 52m kube-system pod/coredns-7f89b7bc75-cc6gn 1/1 Running 0 70m kube-system pod/coredns-7f89b7bc75-wsg29 1/1 Running 0 70m kube-system pod/etcd-k8s-master01 1/1 Running 0 70m kube-system pod/etcd-k8s-master02 1/1 Running 0 65m kube-system pod/etcd-k8s-master03 1/1 Running 0 62m kube-system pod/kube-apiserver-k8s-master01 1/1 Running 0 70m kube-system pod/kube-apiserver-k8s-master02 1/1 Running 0 65m kube-system pod/kube-apiserver-k8s-master03 1/1 Running 0 62m kube-system pod/kube-controller-manager-k8s-master01 1/1 Running 1 70m kube-system pod/kube-controller-manager-k8s-master02 1/1 Running 0 65m kube-system pod/kube-controller-manager-k8s-master03 1/1 Running 0 62m kube-system pod/kube-proxy-ck9lr 1/1 Running 0 63m kube-system pod/kube-proxy-vzrs8 1/1 Running 0 65m kube-system pod/kube-proxy-wxb78 1/1 Running 0 70m kube-system pod/kube-proxy-z5chd 1/1 Running 0 47m kube-system pod/kube-scheduler-k8s-master01 1/1 Running 1 70m kube-system pod/kube-scheduler-k8s-master02 1/1 Running 0 65m kube-system pod/kube-scheduler-k8s-master03 1/1 Running 0 62m NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 70m kube-system service/kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 70m NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE kube-system daemonset.apps/calico-node 4 4 4 4 4 kubernetes.io/os=linux 52m kube-system daemonset.apps/kube-proxy 4 4 4 4 4 kubernetes.io/os=linux 70m NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE default deployment.apps/my-web 4/4 4 4 45m kube-system deployment.apps/calico-kube-controllers 1/1 1 1 52m kube-system deployment.apps/coredns 2/2 2 2 70m NAMESPACE NAME DESIRED CURRENT READY AGE default replicaset.apps/my-web-6b6c9df775 4 4 4 45m kube-system replicaset.apps/calico-kube-controllers-5bb48c55fd 1 1 1 52m kube-system replicaset.apps/coredns-7f89b7bc75 2 2 2 70m9. 测试kubernetes集群验证Pod工作验证Pod网络通信验证DNS解析在Kubernetes集群中创建一个pod，验证是否正常运行：$ kubectl create deployment nginx --image=nginx $ kubectl expose deployment nginx --port=80 --type=NodePort $ kubectl get pod,svc访问地址：http://NodeIP:Port10. 部署 Dashboard[可选]$ wget https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.3/aio/deploy/recommended.yaml默认Dashboard只能集群内部访问，修改Service为NodePort类型，暴露到外部：$ vi recommended.yaml ... kind: Service apiVersion: v1 metadata: labels: k8s-app: kubernetes-dashboard name: kubernetes-dashboard namespace: kubernetes-dashboard spec: ports: - port: 443 targetPort: 8443 nodePort: 30001 selector: k8s-app: kubernetes-dashboard type: NodePort ... $ kubectl apply -f recommended.yaml $ kubectl get pods -n kubernetes-dashboard NAME READY STATUS RESTARTS AGE dashboard-metrics-scraper-6b4884c9d5-gl8nr 1/1 Running 0 13m kubernetes-dashboard-7f99b75bf4-89cds 1/1 Running 0 13m访问地址：https://NodeIP:30001创建service account并绑定默认cluster-admin管理员集群角色：# 创建用户 $ kubectl create serviceaccount dashboard-admin -n kube-system # 用户授权 $ kubectl create clusterrolebinding dashboard-admin --clusterrole=cluster-admin --serviceaccount=kube-system:dashboard-admin # 获取用户Token $ kubectl describe secrets -n kube-system $(kubectl -n kube-system get secret | awk '/dashboard-admin/{print $1}')使用输出的token登录Dashboard。11. 补充11.1 kubectl 命令补齐$ yum install bash-completion -y $ source /usr/share/bash-completion/bash_completion $ source <(kubectl completion bash) $ kubectl completion bash >/etc/bash_completion.d/kubectl
一、Metrics-Server 介绍​ Metrics-Server是k8s集群采集监控数据的聚合器，如采集node、pod的cpu、内存等数据，从 Kubernetes1.8 开始默认使用Metrics-Server采集数据，并通过Metrics API的形式提供查询，但是，kubeadm安装的k8s集群默认是没有安装Metrics-Server的。⚠️ 注意：​ 在 Kubernetes 1.7 版本引入了聚合层，允许第三方应用程序通过将自己注册到kube-apiserver上，仍然通过 API Server 的 HTTP URL 对新的 API 进行访问和操作。为了实现这个机制，Kubernetes 在 kube-apiserver 服务中引入了一个API 聚合层（API Aggregation Layer），用于将扩展 API 的访问请求转发到用户服务的功能。如果你是kubadm 的部署方式，默认已开启Kubernetes API聚合层。Metrics Server项目地址： https://github.com/kubernetes-sigs/metrics-servermetrics-server版本与k8s版本对应关系: https://github.com/kubernetes-sigs/metrics-server#compatibility-matrix本次部署环境： K8S1.32 ，Metrics Server 0.8.0二、部署Metrics-Server下载 Metrics Server 的部署文件:$ wget https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.8.0/components.yaml修改components.yaml 文件:国内墙，需要替换 registry.k8s.io/metrics-server/metrics-server:v0.8.0 镜像地址为：registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-server:v0.8.0apiVersion: v1 kind: ServiceAccount metadata: labels: k8s-app: metrics-server name: metrics-server namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: k8s-app: metrics-server rbac.authorization.k8s.io/aggregate-to-admin: "true" rbac.authorization.k8s.io/aggregate-to-edit: "true" rbac.authorization.k8s.io/aggregate-to-view: "true" name: system:aggregated-metrics-reader rules: - apiGroups: - metrics.k8s.io resources: - pods - nodes verbs: - get - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: labels: k8s-app: metrics-server name: system:metrics-server rules: - apiGroups: - "" resources: - nodes/metrics verbs: - get - apiGroups: - "" resources: - pods - nodes verbs: - get - list - watch --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: labels: k8s-app: metrics-server name: metrics-server-auth-reader namespace: kube-system roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: extension-apiserver-authentication-reader subjects: - kind: ServiceAccount name: metrics-server namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: labels: k8s-app: metrics-server name: metrics-server:system:auth-delegator roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:auth-delegator subjects: - kind: ServiceAccount name: metrics-server namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: labels: k8s-app: metrics-server name: system:metrics-server roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:metrics-server subjects: - kind: ServiceAccount name: metrics-server namespace: kube-system --- apiVersion: v1 kind: Service metadata: labels: k8s-app: metrics-server name: metrics-server namespace: kube-system spec: ports: - appProtocol: https name: https port: 443 protocol: TCP targetPort: https selector: k8s-app: metrics-server --- apiVersion: apps/v1 kind: Deployment metadata: labels: k8s-app: metrics-server name: metrics-server namespace: kube-system spec: selector: matchLabels: k8s-app: metrics-server strategy: rollingUpdate: maxUnavailable: 0 template: metadata: labels: k8s-app: metrics-server spec: containers: - args: - --cert-dir=/tmp - --secure-port=10250 - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname - --kubelet-use-node-status-port - --metric-resolution=15s - --kubelet-insecure-tls #image: registry.k8s.io/metrics-server/metrics-server:v0.8.0 image: registry.cn-hangzhou.aliyuncs.com/google_containers/metrics-server:v0.8.0 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 3 httpGet: path: /livez port: https scheme: HTTPS periodSeconds: 10 name: metrics-server ports: - containerPort: 10250 name: https protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /readyz port: https scheme: HTTPS initialDelaySeconds: 20 periodSeconds: 10 resources: requests: cpu: 100m memory: 200Mi securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL readOnlyRootFilesystem: true runAsNonRoot: true runAsUser: 1000 seccompProfile: type: RuntimeDefault volumeMounts: - mountPath: /tmp name: tmp-dir nodeSelector: kubernetes.io/os: linux priorityClassName: system-cluster-critical serviceAccountName: metrics-server volumes: - emptyDir: {} name: tmp-dir --- apiVersion: apiregistration.k8s.io/v1 kind: APIService metadata: labels: k8s-app: metrics-server name: v1beta1.metrics.k8s.io spec: group: metrics.k8s.io groupPriorityMinimum: 100 insecureSkipTLSVerify: true service: name: metrics-server namespace: kube-system version: v1beta1 versionPriority: 100部署 Metrics Server :$ kubectl apply -f components.yaml验证 Metrics Server 是否成功部署:$ kubectl get pods -n kube-system -l k8s-app=metrics-server NAME READY STATUS RESTARTS AGE metrics-server-74458669b-wvkkr 1/1 Running 0 130m使用 kubectl top 命令查看 Pods 和 Nodes 的资源使用情况：$ kubectl top pods NAME CPU(cores) MEMORY(bytes) pig-auth-58f7855c76-cft8m 3m 523Mi pig-codegen-7799896dd6-mzfhx 3m 633Mi pig-gateway-65854c6b5c-l7rdc 2m 425Mi pig-gateway-65854c6b5c-t64c9 2m 451Mi pig-monitor-5b5c68bc4f-kt5hn 11m 457Mi pig-mysql-c867c676f-6zs9z 7m 439Mi pig-quartz-657d5c9f77-qx6q5 19m 509Mi pig-quartz-657d5c9f77-sbszt 5m 484Mi pig-redis-6dbccc7854-f786j 2m 8Mi pig-register-798bc58678-rqrrn 7m 647Mi pig-ui-5ff9bbdd7f-ld7mb 0m 3Mi pig-upms-d9dcc697f-7cdhw 17m 575Mi pig-upms-d9dcc697f-klx6z 17m 492Mi $ kubectl top nodes NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% k8s-master01 230m 5% 3026Mi 41% k8s-master02 240m 6% 2399Mi 32% k8s-master03 214m 5% 2221Mi 30% k8s-node01 264m 6% 6663Mi 90%
K8s私有云：Nginx真实客户端IP透传全攻略（HAProxy + Ingress 实战）
一、背景记一次私有云 K8s 项目交付中，Ingress‑NGINX 采用外部 HAProxy 作为四层入口后，发现业务 Pod（如 Nginx）的访问日志记录的并非客户端真实 IP，而是K8s 内部地址或 HAProxy 的地址。要使最终的 Pod 准确采集到客户端真实 IP。实测后发现，需要同时完成链路“透传”：HAProxy 发送 PROXY v2 保留真实 IP，Ingress‑NGINX 打开 透传配置，并由后端应用读取 X‑Forwarded‑For,ingress-nginx 的Service 侧使用 externalTrafficPolicy=Local 保源 IP。可以看我👇的图：二、实战在实战开始之前，我们需要先了解下 K8s service中的externalTrafficPolicy参数2.1 关于service中的externalTrafficPolicy参数两种取值Cluster（默认）：集群内访问会负载到任意就绪后端，可跨节点。Local：仅转发给“源节点的本地后端”，没有本地后端则失败；适合就近访问、降低跨节点带宽/延迟，或依赖本地资源。2.1.1 当 service 的externalTrafficPolicy=Cluster集群内访问会负载到任意就绪后端，可跨节点。目标是任何节点都能访问我们以一个案例：K8s中运行一个Nginx ，以 NodePort 暴露#demo.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx spec: replicas: 1 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: nginx spec: type: NodePort selector: app: nginx externalTrafficPolicy: Cluster ports: - port: 80 targetPort: 80 protocol: TCP nodePort: 31043#apply应用 $ kubectl apply -f demo.yaml deployment.apps/nginx created service/nginx created #查看 pod 状态以及 nodeport 端口 $ kubectl get pod,svc NAME READY STATUS RESTARTS AGE pod/nginx-5869d7778c-4z7kd 1/1 Running 0 15s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 7d service/nginx NodePort 10.111.180.29 <none> 80:31043/TCP 7s访问：http://K8s中任意节点ip:31043现象与结论：master / worker 任意节点的 31043 都能访问。源 IP 通常是节点 IP，不保留真实客户端 IP。2.1.2 当 service 的externalTrafficPolicy=Local仅转发给有源节点的本地后端，没有本地后端则失败我们还是以一个案例：K8s中运行一个Nginx-1 ，以 NodePort 暴露#demo-1.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-1 spec: replicas: 1 selector: matchLabels: app: nginx-1 template: metadata: labels: app: nginx-1 spec: containers: - name: nginx image: nginx ports: - containerPort: 80 --- apiVersion: v1 kind: Service metadata: name: nginx-1 spec: type: NodePort selector: app: nginx-1 externalTrafficPolicy: Local ports: - port: 80 targetPort: 80 protocol: TCP nodePort: 32043#apply应用 $ kubectl apply -f demo-1.yaml deployment.apps/nginx-1 created service/nginx-1 created #查看 pod 状态以及 nodeport 端口 $ kubectl get pods -n default nginx-1-74569994c4-65wjx -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-1-74569994c4-65wjx 1/1 Running 0 3m40s 10.244.135.156 k8s-node03 <none> <none> $ kubectl get svc -n default nginx-1 -o wide NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR nginx-1 NodePort 10.96.39.34 <none> 80:32043/TCP 3m19s app=nginx-1看到 nginx-1 这个 pod 运行在 k8s-node3 节点上，我们试下用不同的节点访问，看下对比的效果现象与结论：只有 k8s-node3 节点的 ip 可以访问到，且保留了真实客户端 IP如需让某节点可达，必须在该节点上跑后端 Pod（role/容忍/DaemonSet）2.2 Ingress-nginx 的 service 修改为 externalTrafficPolicy=Local部署 ingress-nginx 的类型为 deployement，三副本方式；且只部署在所有的 node 节点上，master 节点不参与外部的流量转发，当然也可以调整成DaemonSet。Ingress-nginx 的 service 类型为 Nodeport修改 externalTrafficPolicy: Local--- apiVersion: v1 kind: Service metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.14.0 name: ingress-nginx-controller namespace: ingress-nginx spec: externalTrafficPolicy: Local ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - appProtocol: http name: http port: 80 protocol: TCP targetPort: http nodePort: 32280 - appProtocol: https name: https port: 443 protocol: TCP targetPort: https nodePort: 32443 selector: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx type: NodePort查看ingress 命名空间下的所有资源对象$ kubectl get all -n ingress-nginx NAME READY STATUS RESTARTS AGE pod/ingress-nginx-controller-6f58887cd5-d4mxn 1/1 Running 0 3d17h pod/ingress-nginx-controller-6f58887cd5-t8hhq 1/1 Running 0 3d17h pod/ingress-nginx-controller-6f58887cd5-xszld 1/1 Running 0 3d17h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/ingress-nginx-controller NodePort 10.96.227.186 <none> 80:32280/TCP,443:32443/TCP 3d17h service/ingress-nginx-controller-admission ClusterIP 10.106.60.85 <none> 443/TCP 3d17h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/ingress-nginx-controller 3/3 3 3 3d17h NAME DESIRED CURRENT READY AGE replicaset.apps/ingress-nginx-controller-6f58887cd5 3 3 3 3d17h2.3 Ingress-NGINX 的 ConfigMap 开启 PROXY Protocol 与 X-Forwarded-ForIngress-nginx 的部署内容省略，可直接修改 configmap.data 中定义的内容-- apiVersion: v1 data: use-proxy-protocol: "true" compute-full-forwarded-for: "true" forwarded-for-header: "X-Forwarded-For" kind: ConfigMap metadata: labels: app.kubernetes.io/component: controller app.kubernetes.io/instance: ingress-nginx app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx app.kubernetes.io/version: 1.14.0 name: ingress-nginx-controller namespace: ingress-nginx参数解读use-proxy-protocol：启用 PROXY 协议（v1/v2），让 NGINX 从负载均衡器/代理接收客户端的真实 IP、端口等信息（需上游也支持 PROXY 协议）。 compute-full-forwarded-for：当启用时，NGINX 会将客户端的原始 IP 追加到 X-Forwarded-For头中（而非直接覆盖），保留完整代理链。 forwarded-for-header：指定用于存储客户端 IP 的头部名称（默认是 X-Forwarded-For，可自定义为其他如 X-Real-IP等）。2.4 使用Nginx自带的Realip模块获取用户真实IPNginx 默认只能看到 上一层代理的 IP ，上面的案例我们之所以可以看到真实的 ip 地址，是因为只经过了一层转发。现在我们的访问是客户端 → Haproxy（最外侧）→ Ingress-NGINX → Nginx Pod（你的服务）。每一层代理都会修改请求的元数据（如 IP 头），Nginx 默认只能看到「上一层代理的 IP」（而非真实客户端 IP）。下面的这个三个参数就是为了让你的 Nginx Pod 从多层代理的混乱 IP 中“捞出”真实客户端 IP。这段 Nginx 配置属于 ngx_http_realip_module模块（用于处理“真实客户端 IP”），核心作用是：当 Nginx 前方有代理（如负载均衡器、CDN、Ingress）时，从代理传递的 HTTP 头中提取客户端的真实 IP，并替换 Nginx 默认的 $remote_addr变量（默认 $remote_addr是上一层代理的 IP，而非真实客户端 IP）。#指定「可信代理服务器的 IP 段」，10.244.0.0/16 是k8s 的子网地址，表示信任只信任这段ip的代理头，即任何代理传递的 X-Forwarded-For都被认为是合法的，一般我们会添加上游服务器的 IP 地址。 set_real_ip_from 10.244.0.0/16; #指定「从哪个 HTTP 头中提取真实客户端 IP real_ip_header X-Forwarded-For; #递归跳过可信代理 IP，取第一个非可信 IP（真实客户端 IP） real_ip_recursive on;创建一个 main-app 的案例 ： 包含 configmap，deployment，svc# main-app.yaml apiVersion: v1 kind: ConfigMap metadata: name: nginx-main-conf namespace: default data: nginx.conf: | user nginx; worker_processes auto; error_log /var/log/nginx/error.log notice; pid /run/nginx.pid; events { worker_connections 1024; } http { include /etc/nginx/mime.types; default_type application/octet-stream; set_real_ip_from 10.244.0.0/16; real_ip_header X-Forwarded-For; real_ip_recursive on; log_format main '$remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; sendfile on; keepalive_timeout 65; include /etc/nginx/conf.d/*.conf; } --- apiVersion: apps/v1 kind: Deployment metadata: name: main-app namespace: default spec: replicas: 1 selector: matchLabels: app: main-app template: metadata: labels: app: main-app spec: containers: - name: main-app image: nginx:latest imagePullPolicy: IfNotPresent ports: - containerPort: 80 volumeMounts: - name: nginx-main-conf mountPath: /etc/nginx/nginx.conf subPath: nginx.conf readOnly: true volumes: - name: nginx-main-conf configMap: name: nginx-main-conf --- apiVersion: v1 kind: Service metadata: name: main-app-service namespace: default spec: type: ClusterIP selector: app: main-app ports: - name: http port: 80 targetPort: 80#apply kubectl apply -f main-app.yaml #查看资源 [root@k8s-master01 ~]# kubectl get all -n default NAME READY STATUS RESTARTS AGE pod/main-app-895f5b55d-t8tjq 1/1 Running 0 101s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 7d18h service/main-app-service ClusterIP 10.100.5.238 <none> 80/TCP 101s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/main-app 1/1 1 1 101s NAME DESIRED CURRENT READY AGE replicaset.apps/main-app-895f5b55d 1 1 1 101s2.5 配置 Ingress 域名配置一个 TLS Secret 证书，将证书导入到 k8s 的 default 命名空间下的 secret 中kubectl create secret tls srebro.cn-tls \ --cert=srebro.cn.pem \ --key=srebro.cn.key \ --namespace=default创建一个 ingress 规则指定到一个域名上，如 test.srebro.cn#ingress.yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: baseline-sre-test-ingress namespace: default annotations: nginx.ingress.kubernetes.io/use-regex: "true" nginx.ingress.kubernetes.io/rewrite-target: /$2 # 可选：强制 HTTPS 重定向 nginx.ingress.kubernetes.io/ssl-redirect: "true" spec: ingressClassName: nginx tls: - hosts: - test.srebro.cn secretName: srebro.cn-tls rules: - host: test.srebro.cn http: paths: - path: /()(.*) pathType: ImplementationSpecific backend: service: name: main-app-service port: number: 80#创建ingress $ kubectl apply -f ingress.yaml ingress.networking.k8s.io/baseline-sre-test-ingress created #查看 ingress $ kubectl get ingress -n default NAME CLASS HOSTS ADDRESS PORTS AGE baseline-sre-test-ingress nginx test.srebro.cn 80, 443 22s2.6 HAProxy 配置 send-proxy-v2参数send-proxy-v2是 HAProxy 的一个配置指令，用于 向后端服务发送「代理协议 v2 版本」的元数据。HAProxy 监听本地的 443 端口，并代理到 所有node 节点上的 Nodeport 端口上。frontend kube-https bind *:443 mode tcp option tcplog default_backend kube-https backend kube-https mode tcp option tcplog option tcp-check balance roundrobin default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100 server kube-node01 172.22.33.110:32443 send-proxy-v2 check server kube-node02 172.22.33.111:32443 send-proxy-v2 check server kube-node03 172.22.33.112:32443 send-proxy-v2 check为了方便测试本地电脑，配置 test.srebro.cn 域名解析到 这个 haproxy 服务器172.22.33.100上sudo vim /etc/hosts 172.22.33.100 test.srebro.cn2.7 访问测试页面日志三、结语问题本质：多层代理（HAProxy → Ingress‑NGINX → Nginx Pod）会改变源地址，后端默认只能看到“上一层代理”的 IP，导致真实客户端 IP丢失。策略选择：externalTrafficPolicy=Cluster：任意节点可达，易用但一般不保留真实 IP（跨节点转发+SNAT），externalTrafficPolicy=Local：仅本地后端节点可达，保留源 IP；入口节点必须有 Ingress Pod。注意事项：set_real_ip_from 只信任你控制的网段（如 K8s Pod 网段），避免被伪造头欺骗。生产中建议仅在 worker 暴露入口；master 不承载外部流量。
K8s 节点管理：使用 kubeadm 删除和重新添加节点
一、需求：在 Kubernetes 中，使用 kubeadm 的方式 删除和 重新添加节点二、删除节点从集群中删除节点首先，需要在主节点上将节点从集群中删除。假设要删除的节点名称为 node-1：kubectl delete node node-1在要删除的节点上执行你还需要在要删除的节点上执行以下命令，以清理本地的 K8s 配置和状态：kubeadm reset这个命令会删除 K8s 集群的所有配置文件和数据。执行后，还需要删除节点上的 K8s 配置文件和网络插件的配置，例如：rm -rf /etc/kubernetes rm -rf /var/lib/etcd以及清理 CNI 网络插件的配置，例如：rm -rf /var/lib/cni/三、重新添加节点重新初始化节点确保节点上的 kubeadm 工具已安装，并且网络插件和其他相关工具也已正确配置。然后，使用 kubeadm join 命令重新将节点加入集群。首先，你需要从主节点获取 kubeadm join 命令的令牌。可以通过以下命令生成新的令牌：kubeadm token create --print-join-command这条命令会输出类似以下内容的命令：kubeadm join <master-ip>:<master-port> --token <token> --discovery-token-ca-cert-hash sha256:<hash>在要重新加入的节点上执行输出的 kubeadm join 命令：kubeadm join <master-ip>:<master-port> --token <token> --discovery-token-ca-cert-hash sha256:<hash>验证节点状态重新加入节点后，可以在主节点上检查节点状态以确保它成功加入：kubectl get nodes如果节点状态显示为 Ready，则说明节点已经成功加入集群。注意事项确保在执行 kubeadm reset 命令时，节点上的所有数据和配置都被清除，以避免潜在的冲突。在重新加入节点之前，确保集群和节点的网络配置正常。如果你的集群使用了自定义的网络插件或配置，确保在节点上进行相应的配置。
Velero 快速上手：使用 Velero 实现 Kubernetes 集群备份与迁移
一、velero 简介Velero 是vmware开源的一个云原生的灾难恢复和迁移工具，它本身也是开源的,采用Go语言编写，可以安全的备份、恢复和迁移Kubernetes集群资源数据；Velero 是西班牙语意思是帆船，非常符合Kubernetes社区的命名风格，Velero的开发公司Heptio，已被VMware收购。Velero 支持标准的K8S集群，既可以是私有云平台也可以是公有云，除了灾备之外它还能做资源移转，支持把容器应用从一个集群迁移到另一个集群。Velero 的工作方式就是把kubernetes中的数据备份到对象存储以实现高可用和持久化，默认的备份保存时间为720小时，并在需要的时候进行下载和恢复。github: https://github.com/vmware-tanzu/velero官网：https://velero.io/二、 velero 和 ETCD 备份的区别2.1 velero 备份Velero 是一个开源工具，用于备份和恢复 Kubernetes 集群中的资源和持久化数据。它支持对 Kubernetes 资源（如 Pods、Services、Deployments 等）和持久卷（Persistent Volumes）进行备份和恢复。主要特点：备份 Kubernetes 资源：包括集群的 API 对象和元数据。备份持久化数据：通过与存储供应商（如 AWS S3、Azure Blob、Google Cloud Storage 等）集成，备份 PVC 中的数据。备份计划：可以设置定期备份和备份策略。迁移支持：可以用于跨集群或跨云的迁移。恢复功能：支持恢复整个集群或部分资源。使用场景：需要备份和恢复整个 Kubernetes 集群的配置和持久化数据。需要支持跨集群的迁移。需要定期备份和恢复 Kubernetes 对象和 PVC 数据。2.2 etcd 备份etcd 是 Kubernetes 的核心组件之一，存储了集群的所有状态和配置信息。etcd 备份 是针对 etcd 数据库的备份，主要用于备份和恢复 Kubernetes 的集群状态和配置。主要特点：备份 Kubernetes 状态：包括集群的所有配置和状态数据。原生备份：直接备份 etcd 数据库，而不是通过 Kubernetes API。恢复集群状态：可以恢复整个集群的状态到备份时的状态。使用场景：需要备份和恢复 Kubernetes 集群的核心状态和配置信息。需要恢复集群的状态到某个时间点。需要直接操作 etcd 数据库进行备份和恢复。2.3 对比特性Veleroetcd 备份备份内容Kubernetes 资源和持久卷etcd 数据库（集群状态和配置）备份方式通过 Velero CLI，存储在外部存储使用 etcdctl 工具，存储在指定位置恢复内容Kubernetes 资源和持久卷恢复集群的状态和配置使用场景跨集群迁移、集群资源备份与恢复集群状态恢复、紧急恢复自动化支持定期备份和恢复策略通常手动执行备份和恢复2.4.选择建议使用 Velero 如果你需要备份和恢复整个 Kubernetes 集群的资源和持久化数据，包括跨集群迁移和定期备份。使用 etcd 备份 如果你专注于备份和恢复 Kubernetes 的核心状态和配置，特别是在需要恢复整个集群状态到某个时间点时。有时，你可能需要同时使用这两种备份策略，以确保全面的数据保护和恢复能力。例如，你可以使用 Velero 备份 Kubernetes 资源和持久卷，而使用 etcd 备份来保护集群的核心状态和配置； 本文呢 主要阐述对 velero 的备份详解三、velero整体架构需要部署一个 Velero 服务端到集群，该服务端拥有访问集群资源的权限；通过 velero 二进制程序部署服务端的同时，其也会向 kubernetes 集群中同时注册一些 crd 资源，这些 crd 资源可以对我们备份、还原的资源做抽象；通过 crd 资源创建一个备份请求对象，服务端控制器会根据请求内容完成目标预期的备份，并将备份后的文件上传至配置的远端存储（需要支持 S3 协议，如 Ceph、Minio 都可以）；备份完毕后，可以通过获取特定的 crd 资源来查看备份的记录（文件）；通过 crd 资源创建一个还原请求对象，服务端控制器会根据请求内容对应的备份文件，将备份内容恢复到集群；四、velero备份流程Velero 客户端调用Kubernetes API Server创建Backup任务。Backup 控制器基于watch 机制通过API Server获取到备份任务。Backup 控制器开始执行备份动作，其会通过请求API Server获取需要备份的数据。Backup 控制器将获取到的数据备份到指定的对象存储server端。五、集群迁移示例实验目的： 在两个集群迁移 kubernetes 资源需要进行的动作，使用velero，将 A 集群 default 名称空间及其下的资源迁移到 B 集群软件名称版本备注K8S-A集群V1.20.15 K8S-B集群V1.20.15 minio 单节点RELEASE.2024-06-13T22-53-53Zdocker-compose 部署veleroV1.14.0 大致步骤：部署一个 minio，用来作为 Velero 上传备份文件的远端存储；在两个 kubernetes 集群中都部署一个 Velero 服务端，这两个服务端的远端存储都配置同一个 minio，且指定使用同一个 bucket；在 A 集群中创建一个对 default 名称空间的备份请求，确认备份动作执行完毕；登录 B 集群 查看 A 集群中 备份的记录文件；在B 集群创建一个还原请求，将 A 集群备份的文件还原到 B 集群；在B 集群验证 default 命名空间下的资源 是否正确。六、对象存储minio部署为了方便测试，这里 测试docker-compose 的部署方式6.1 创建minio工作目录$ mkdir -p /home/application/minio6.2 编写docker-compose.yaml 文件$ vim /home/application/minio/docker-compose.yaml version: '3' services: minio: image: minio/minio:latest container_name: minio networks: - srebro ports: - 9000:9000 - 9001:9001 volumes: - /home/application/minio/data:/data - /etc/localtime:/etc/localtime environment: MINIO_ROOT_USER: admin MINIO_ROOT_PASSWORD: 'srebro' command: server /data --console-address :9001 restart: unless-stopped networks: srebro: driver: bridge6.3 运行minio，并创建bucket$ cd /home/application/minio $ docker-compose up -d [root@localhost minio]# docker-compose ps NAME COMMAND SERVICE STATUS PORTS minio "/usr/bin/docker-ent…" minio running 0.0.0.0:9000-9001->9000-9001/tcp, :::9000-9001->9000-9001/tcp 打开web控制台： http://XX.XX.XX.XX:9001docker-compose文件中已经指定了用户名/密码： admin/srebro创建bucket，名称为：velero七、部署velero7.1 下载velero二进制程序分别部署到 A 和 B 集群中$ wget https://github.com/vmware-tanzu/velero/releases/download/v1.14.0/velero-v1.14.0-linux-amd64.tar.gz $ tar -xf velero-v1.14.0-linux-amd64.tar.gz $ mv velero-v1.14.0-linux-amd64/velero /usr/local/bin #查看velero版本 [root@openeuler ~]# velero version Client: Version: v1.14.0 Git commit: 2fc6300f2239f250b40b0488c35feae59520f2d3 <error getting server version: namespaces "velero" not found>7.2 部署服务端到集群7.2.1 创建velero创建远端存储认证文件分别部署到 A 和 B 集群中aws_access_key_id： minio用户名aws_secret_access_key： minio密码$ mkdir -p /home/application/velero/ $ cat > /home/application/velero/velero-auth.txt << EOF [default] aws_access_key_id = admin aws_secret_access_key = srebro EOF7.2.1 部署服务端到A 集群使用 --kubeconfig 选项指定部署到的集群；使用 --namespace 指定部署到的名称空间；使用 s3Url 指定备份使用的远端存储 Url，这里我指定的是 Minio 地址；velero install \ --kubeconfig /root/.kube/config \ --provider aws \ --plugins velero/velero-plugin-for-aws:v1.5.5 \ --bucket velero \ --secret-file /home/application/velero/velero-auth.txt \ --use-volume-snapshots=false \ --namespace velero-system \ --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://172.22.33.204:9000 .......................以下为输出内容......................................................... CustomResourceDefinition/backuprepositories.velero.io: attempting to create resource CustomResourceDefinition/backuprepositories.velero.io: attempting to create resource client CustomResourceDefinition/backuprepositories.velero.io: created CustomResourceDefinition/backups.velero.io: attempting to create resource CustomResourceDefinition/backups.velero.io: attempting to create resource client CustomResourceDefinition/backups.velero.io: created CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource client CustomResourceDefinition/backupstoragelocations.velero.io: created CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource client CustomResourceDefinition/deletebackuprequests.velero.io: created CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource client CustomResourceDefinition/downloadrequests.velero.io: created CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource client CustomResourceDefinition/podvolumebackups.velero.io: created CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource client CustomResourceDefinition/podvolumerestores.velero.io: created CustomResourceDefinition/restores.velero.io: attempting to create resource CustomResourceDefinition/restores.velero.io: attempting to create resource client CustomResourceDefinition/restores.velero.io: created CustomResourceDefinition/schedules.velero.io: attempting to create resource CustomResourceDefinition/schedules.velero.io: attempting to create resource client CustomResourceDefinition/schedules.velero.io: created CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource client CustomResourceDefinition/serverstatusrequests.velero.io: created CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource client CustomResourceDefinition/volumesnapshotlocations.velero.io: created CustomResourceDefinition/datadownloads.velero.io: attempting to create resource CustomResourceDefinition/datadownloads.velero.io: attempting to create resource client CustomResourceDefinition/datadownloads.velero.io: created CustomResourceDefinition/datauploads.velero.io: attempting to create resource CustomResourceDefinition/datauploads.velero.io: attempting to create resource client CustomResourceDefinition/datauploads.velero.io: created Waiting for resources to be ready in cluster... Namespace/velero-system: attempting to create resource Namespace/velero-system: attempting to create resource client Namespace/velero-system: created ClusterRoleBinding/velero-velero-system: attempting to create resource ClusterRoleBinding/velero-velero-system: attempting to create resource client ClusterRoleBinding/velero-velero-system: created ServiceAccount/velero: attempting to create resource ServiceAccount/velero: attempting to create resource client ServiceAccount/velero: created Secret/cloud-credentials: attempting to create resource Secret/cloud-credentials: attempting to create resource client Secret/cloud-credentials: created BackupStorageLocation/default: attempting to create resource BackupStorageLocation/default: attempting to create resource client BackupStorageLocation/default: created Deployment/velero: attempting to create resource Deployment/velero: attempting to create resource client Deployment/velero: created Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero-system' to view the status.7.2.2 部署服务端到B 集群velero install \ --kubeconfig /root/.kube/config \ --provider aws \ --plugins velero/velero-plugin-for-aws:v1.5.5 \ --bucket velero \ --secret-file /home/application/velero/velero-auth.txt \ --use-volume-snapshots=false \ --namespace velero-system \ --backup-location-config region=minio,s3ForcePathStyle="true",s3Url=http://172.22.33.204:9000 ..........................以下为输出内容......................................................... CustomResourceDefinition/backuprepositories.velero.io: attempting to create resource CustomResourceDefinition/backuprepositories.velero.io: attempting to create resource client CustomResourceDefinition/backuprepositories.velero.io: created CustomResourceDefinition/backups.velero.io: attempting to create resource CustomResourceDefinition/backups.velero.io: attempting to create resource client CustomResourceDefinition/backups.velero.io: created CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource client CustomResourceDefinition/backupstoragelocations.velero.io: created CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource client CustomResourceDefinition/deletebackuprequests.velero.io: created CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource client CustomResourceDefinition/downloadrequests.velero.io: created CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource client CustomResourceDefinition/podvolumebackups.velero.io: created CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource client CustomResourceDefinition/podvolumerestores.velero.io: created CustomResourceDefinition/restores.velero.io: attempting to create resource CustomResourceDefinition/restores.velero.io: attempting to create resource client CustomResourceDefinition/restores.velero.io: created CustomResourceDefinition/schedules.velero.io: attempting to create resource CustomResourceDefinition/schedules.velero.io: attempting to create resource client CustomResourceDefinition/schedules.velero.io: created CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource client CustomResourceDefinition/serverstatusrequests.velero.io: created CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource client CustomResourceDefinition/volumesnapshotlocations.velero.io: created CustomResourceDefinition/datadownloads.velero.io: attempting to create resource CustomResourceDefinition/datadownloads.velero.io: attempting to create resource client CustomResourceDefinition/datadownloads.velero.io: created CustomResourceDefinition/datauploads.velero.io: attempting to create resource CustomResourceDefinition/datauploads.velero.io: attempting to create resource client CustomResourceDefinition/datauploads.velero.io: created Waiting for resources to be ready in cluster... Namespace/velero-system: attempting to create resource Namespace/velero-system: attempting to create resource client Namespace/velero-system: created ClusterRoleBinding/velero-velero-system: attempting to create resource ClusterRoleBinding/velero-velero-system: attempting to create resource client ClusterRoleBinding/velero-velero-system: created ServiceAccount/velero: attempting to create resource ServiceAccount/velero: attempting to create resource client ServiceAccount/velero: created Secret/cloud-credentials: attempting to create resource Secret/cloud-credentials: attempting to create resource client Secret/cloud-credentials: created BackupStorageLocation/default: attempting to create resource BackupStorageLocation/default: attempting to create resource client BackupStorageLocation/default: created Deployment/velero: attempting to create resource Deployment/velero: attempting to create resource client Deployment/velero: created Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero-system' to view the status.7.3 在 A,b 集群确认 Velero 服务端已成功启动并就绪：#A集群 [root@k8s-master ~]# kubectl get pods -n velero-system NAME READY STATUS RESTARTS AGE velero-6cc6986575-h6r2k 1/1 Running 0 2m #B集群 [root@openeuler ~]# kubectl get pods -n velero-system NAME READY STATUS RESTARTS AGE velero-6cc6986575-hk6tc 1/1 Running 0 2m八、备份与还原操作8.1 A 集群执行备份动作先查看下A 集群中，default 命名空间下资源信息$ kubectl get all -n default NAME READY STATUS RESTARTS AGE pod/pig-auth-66f5bcfd74-9qhlz 1/1 Running 0 4d pod/pig-codegen-5865cd994b-g4rkd 1/1 Running 0 4d15h pod/pig-gateway-7f754ffdbc-dhf72 1/1 Running 0 4d pod/pig-monitor-5c5d67f57c-5gnwp 1/1 Running 0 4d15h pod/pig-mysql-6c665c56c7-6jdq4 1/1 Running 0 4d15h pod/pig-quartz-76fdbdf497-w9f6g 1/1 Running 0 4d15h pod/pig-redis-554cfcc5cc-kfmv8 1/1 Running 0 4d15h pod/pig-register-777df8f59b-lh7pt 1/1 Running 0 4d15h pod/pig-ui-f48d64f76-wnpcx 1/1 Running 0 4d14h pod/pig-upms-58d6f8448f-8njxd 1/1 Running 0 4d15h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 5d2m service/pig-auth ClusterIP 10.96.16.113 <none> 3000/TCP 4d15h service/pig-codegen ClusterIP 10.108.2.9 <none> 5002/TCP 4d15h service/pig-gateway NodePort 10.110.236.0 <none> 9999:32750/TCP 4d15h service/pig-monitor ClusterIP 10.106.84.163 <none> 5001/TCP 4d15h service/pig-mysql NodePort 10.106.57.25 <none> 3306:30406/TCP 4d15h service/pig-quartz ClusterIP 10.104.94.147 <none> 5007/TCP 4d15h service/pig-redis ClusterIP 10.101.95.155 <none> 6379/TCP 4d15h service/pig-register NodePort 10.108.162.125 <none> 8848:31458/TCP,9848:32186/TCP 4d15h service/pig-ui NodePort 10.97.53.70 <none> 80:32545/TCP 4d14h service/pig-upms ClusterIP 10.100.129.94 <none> 4000/TCP 4d15h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/pig-auth 1/1 1 1 4d15h deployment.apps/pig-codegen 1/1 1 1 4d15h deployment.apps/pig-gateway 1/1 1 1 4d15h deployment.apps/pig-monitor 1/1 1 1 4d15h deployment.apps/pig-mysql 1/1 1 1 4d15h deployment.apps/pig-quartz 1/1 1 1 4d15h deployment.apps/pig-redis 1/1 1 1 4d15h deployment.apps/pig-register 1/1 1 1 4d15h deployment.apps/pig-ui 1/1 1 1 4d14h deployment.apps/pig-upms 1/1 1 1 4d15h NAME DESIRED CURRENT READY AGE replicaset.apps/pig-auth-66f5bcfd74 1 1 1 4d15h replicaset.apps/pig-codegen-5865cd994b 1 1 1 4d15h replicaset.apps/pig-gateway-7f754ffdbc 1 1 1 4d15h replicaset.apps/pig-monitor-5c5d67f57c 1 1 1 4d15h replicaset.apps/pig-mysql-6c665c56c7 1 1 1 4d15h replicaset.apps/pig-quartz-76fdbdf497 1 1 1 4d15h replicaset.apps/pig-redis-554cfcc5cc 1 1 1 4d15h replicaset.apps/pig-register-777df8f59b 1 1 1 4d15h replicaset.apps/pig-ui-f48d64f76 1 1 1 4d14h replicaset.apps/pig-upms-58d6f8448f 1 1 1 4d15h 使用 velero 二进制程序创建备份请求，通过 --namespace 指定 Velero 服务端所在名称空间，--include-namespaces 指定要备份的名称空间：$ DATE=`date +%Y%m%d%H%M%S` $ velero backup create \ default-${DATE} \ --namespace velero-system \ --include-namespaces default \ --kubeconfig /root/.kube/config Backup request "default-20240813102355" submitted successfully. Run `velero backup describe default-20240813102355` or `velero backup logs default-20240813102355` for more details.查看备份：$ kubectl get backups.velero.io -n velero-system NAME AGE default-20240813102355 22s查看备份日志： velero -n velero-system backup logs default-20240813102355登录 Minio 进入名为 velero 的 bucket 可以看到对应的备份文件。8.2 B 集群执行还原操作先确认在 B 集群 确认可以看到刚刚的备份：$ kubectl -n velero-system get backups.velero.io NAME AGE default-20240813102355 4m4s使用 velero 二进制程序创建还原请求，通过 --namespace 指定 Velero 服务端所在名称空间，--from-backup 指定要还原的备份文件：velero restore create \ --namespace velero-system \ --kubeconfig /root/.kube/config \ --from-backup default-20240813102355 --wait Restore request "default-20240813102355-20240813103010" submitted successfully. Waiting for restore to complete. You may safely press ctrl-c to stop waiting - your restore will continue in the background. .... Restore completed with status: Completed. You may check for more information using the commands `velero restore describe default-20240813102355-20240813103010` and `velero restore logs default-20240813102355-20240813103010`.查看还原日志：$ velero -n velero-system restore logs default-20240813102355-20240813103010查看还原后的资源：$ kubectl get all -n default NAME READY STATUS RESTARTS AGE pod/pig-auth-66f5bcfd74-9qhlz 1/1 Running 0 20S pod/pig-codegen-5865cd994b-g4rkd 1/1 Running 0 20S pod/pig-gateway-7f754ffdbc-dhf72 1/1 Running 0 20S pod/pig-monitor-5c5d67f57c-5gnwp 1/1 Running 0 20S pod/pig-mysql-6c665c56c7-6jdq4 1/1 Running 0 20S pod/pig-quartz-76fdbdf497-w9f6g 1/1 Running 0 20S pod/pig-redis-554cfcc5cc-kfmv8 1/1 Running 0 20S pod/pig-register-777df8f59b-lh7pt 1/1 Running 0 20S pod/pig-ui-f48d64f76-wnpcx 1/1 Running 0 20S pod/pig-upms-58d6f8448f-8njxd 1/1 Running 0 20S NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 20S service/pig-auth ClusterIP 10.110.68.56 <none> 3000/TCP 20S service/pig-codegen ClusterIP 10.102.86.142 <none> 5002/TCP 20S service/pig-gateway NodePort 10.98.59.211 <none> 9999:30618/TCP 20S service/pig-monitor ClusterIP 10.102.164.245 <none> 5001/TCP 20S service/pig-mysql NodePort 10.99.227.133 <none> 3306:30785/TCP 20S service/pig-quartz ClusterIP 10.104.167.59 <none> 5007/TCP 20S service/pig-redis ClusterIP 10.99.193.119 <none> 6379/TCP 20S service/pig-register NodePort 10.99.188.165 <none> 8848:32639/TCP,9848:32506/TCP 20S service/pig-ui NodePort 10.105.20.18 <none> 80:31260/TCP 20S service/pig-upms ClusterIP 10.106.203.13 <none> 4000/TCP 20S NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/pig-auth 1/1 1 1 20S deployment.apps/pig-codegen 1/1 1 1 20S deployment.apps/pig-gateway 1/1 1 1 20S deployment.apps/pig-monitor 1/1 1 1 20S deployment.apps/pig-mysql 1/1 1 1 20S deployment.apps/pig-quartz 1/1 1 1 20S deployment.apps/pig-redis 1/1 1 1 20S deployment.apps/pig-register 1/1 1 1 20S deployment.apps/pig-ui 1/1 1 1 20S deployment.apps/pig-upms 1/1 1 1 20S NAME DESIRED CURRENT READY AGE replicaset.apps/pig-auth-66f5bcfd74 1 1 1 20S replicaset.apps/pig-codegen-5865cd994b 1 1 1 20S replicaset.apps/pig-gateway-7f754ffdbc 1 1 1 20S replicaset.apps/pig-monitor-5c5d67f57c 1 1 1 20S replicaset.apps/pig-mysql-6c665c56c7 1 1 1 20S replicaset.apps/pig-quartz-76fdbdf497 1 1 1 20S replicaset.apps/pig-redis-554cfcc5cc 1 1 1 20S replicaset.apps/pig-register-777df8f59b 1 1 1 20S replicaset.apps/pig-ui-f48d64f76 1 1 1 20S replicaset.apps/pig-upms-58d6f8448f 1 1 1 20S九、补充批量备份所有namespacevim all-ns-velero-backup.sh #!/bin/bash NS_NAME=`kubectl get ns | awk '{if (NR>2){print}}' | awk '{print $1}'` DATE=`date +%Y%m%d%H%M%S` cd /home/application/velero/ for i in $NS_NAME;do velero backup create ${i}-ns-backup-${DATE} \ --include-namespaces ${i} \ --kubeconfig=/root/.kube/config \ --namespace velero-system done将备份设置为过期创建备份时，可以通过添加标志 --ttl <DURATION> 来指定 TTL（生存时间）。如果 Velero 发现现有备份资源已过期，它将删除：The TTL flag allows the user to specify the backup retention period with the value specified in hours, minutes and seconds in the form --ttl 24h0m0s. If not specified, a default TTL value of 30 days will be applied.TTL 标志允许用户使用以小时、分钟和秒为单位指定的值指定备份保留期，格式为 --ttl 24h0m0s。如果未指定，将应用默认的 TTL 值 30 天。十、参考https://www.zze.xyz/archives/velero-backup-restore-k8s-resourceshttps://www.cnblogs.com/qiuhom-1874/p/17670945.html
调整k8s node单节点可运行的pod突破110上限官方文档： https://kubernetes.io/zh-cn/docs/reference/config-api/kubelet-config.v1beta1/使用 systemctl status kubelet 查看kubelet.service文件的路径可以看到配置文件在 /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf[root@k8s-node1]# cat /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf # Note: This dropin only works with kubeadm and kubelet v1.11+ [Service] Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf" Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml" # This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env # This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use # the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file. EnvironmentFile=-/etc/sysconfig/kubelet ExecStart= ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS此文件我们发现 KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml直接在KUBELET_CONFIG_ARGS 中添加参数即可修改 /var/lib/kubelet/config.yaml 配置文件，在最下面添加一行 maxPods: xxx 我们修改下maxPods的数值，增大到300（具体按照你node节点资源情况设置，不建议设置的过大）修改完成之后重启下kubelet服务[不会影响正在运行的容器]$ systemctl daemon-reload $ systemctl restart kubelet模拟调整pod 数量，运行500个pod；观察运行情况#调整pod数量到500 [root@k8s-master01 system]# kubectl scale deployment my-nginx-test --replicas=500 deployment.apps/my-nginx-test scaled #两个mode节点 [root@k8s-master01 system]# kubectl get nodes NAME STATUS ROLES AGE VERSION k8s-master01 Ready control-plane,master 24h v1.20.15 k8s-master02 Ready control-plane,master 24h v1.20.15 k8s-node1 Ready <none> 24h v1.20.15 k8s-node2 Ready <none> 53m v1.20.15 #查看run 的pod 数量 [root@k8s-master01 system]# kubectl get pods | grep -i run | wc -l 500