008.Kubernetes二进制部署Nginx实现高可用

一Nginx代理实现kube-apiserver高可用

1.1Nginx实现高可用

基于 nginx 代理的 kube-apiserver 高可用方案。
控制节点的 kube-controller-manager、kube-scheduler 是多实例部署，所以只要有一个实例正常，就可以保证高可用；
集群内的 Pod 使用 K8S 服务域名 kubernetes 访问 kube-apiserver， kube-dns 会自动解析出多个 kube-apiserver 节点的 IP，所以也是高可用的；
在每个节点起一个 nginx 进程，后端对接多个 apiserver 实例，nginx 对它们做健康检查和负载均衡；
kubelet、kube-proxy、controller-manager、scheduler 通过本地的 nginx（监听 127.0.0.1）访问 kube-apiserver，从而实现 kube-apiserver 的高可用；
从而基于 nginx 4 层透明代理功能实现 K8S 节点( master 节点和 worker 节点)高可用访问 kube-apiserver 。

1.2下载编译Nginx

1 [root@k8smaster01 ~]# cd /opt/k8s/work  2 [root@k8smaster01 work]# wget http://nginx.org/download/nginx-1.15.3.tar.gz  3 [root@k8smaster01 work]# tar -xzvf nginx-1.15.3.tar.gz  4 [root@k8smaster01 ~]# cd /opt/k8s/work/nginx-1.15.3/  5 [root@k8smaster01 nginx-1.15.3]# mkdir nginx-prefix  6 [root@k8smaster01 nginx-1.15.3]# ./configure --with-stream --without-http --prefix=$(pwd)/nginx-prefix --without-http_uwsgi_module --without-http_scgi_module --without-http_fastcgi_module  7 [root@k8smaster01 ~]# cd /opt/k8s/work/nginx-1.15.3/  8 [root@k8smaster01 nginx-1.15.3]# make && make install
解释：
  • --with-stream：开启 4 层透明转发(TCP Proxy)功能；
  • --without-xxx：关闭所有其他功能，这样生成的动态链接二进制程序依赖最小。
  • [root@k8smaster01 ~]# cd /opt/k8s/work/nginx-1.15.3/
  • [root@k8smaster01 nginx-1.15.3]# ./nginx-prefix/sbin/nginx -v

1.3验证编译后的Nginx

1 [root@k8smaster01 ~]# cd /opt/k8s/work/nginx-1.15.3  2 [root@k8smaster01 nginx-1.15.3]# ./nginx-prefix/sbin/nginx -v  3 nginx version: nginx/1.15.3  4 [root@k8smaster01 nginx-1.15.3]# ldd ./nginx-prefix/sbin/nginx#查看 nginx 动态链接的库  5         linux-vdso.so.1 =>  (0x00007ffdda980000)  6         libdl.so.2 => /lib64/libdl.so.2 (0x00007feb37300000)  7         libpthread.so.0 => /lib64/libpthread.so.0 (0x00007feb370e4000)  8         libc.so.6 => /lib64/libc.so.6 (0x00007feb36d17000)  9         /lib64/ld-linux-x86-64.so.2 (0x00007feb37504000)
提示：由于只开启了 4 层透明转发功能，所以除了依赖 libc 等操作系统核心 lib 库外，没有对其它 lib 的依赖(如 libz、libssl 等)，以便达到精简编译的目的。

1.4安装和部署Nginx

1 [root@k8smaster01 ~]# cd /opt/k8s/work  2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh  3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}  4   do  5     echo ">>> ${master_ip}"  6     mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}  7   done#创建Nginx目录  8 [root@k8smaster01 ~]# cd /opt/k8s/work  9 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 10 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]} 11   do 12     echo ">>> ${master_ip}" 13     scp /opt/k8s/work/nginx-1.15.3/nginx-prefix/sbin/nginx  root@${master_ip}:/opt/k8s/kube-nginx/sbin/kube-nginx 14     ssh root@${master_ip} "chmod a+x /opt/k8s/kube-nginx/sbin/*" 15     ssh root@${master_ip} "mkdir -p /opt/k8s/kube-nginx/{conf,logs,sbin}" 16   done#分发Nginx二进制

1.5配置Nginx 四层透明转发

1 [root@k8smaster01 ~]# cd /opt/k8s/work  2 [root@k8smaster01 work]# cat > kube-nginx.conf <<EOF  3 worker_processes 1;  4   5 events {  6     worker_connections  1024;  7 }  8   9 stream { 10     upstream backend { 11         hash $remote_addr consistent; 12         server 172.24.8.71:6443        max_fails=3 fail_timeout=30s; 13         server 172.24.8.72:6443        max_fails=3 fail_timeout=30s; 14         server 172.24.8.73:6443        max_fails=3 fail_timeout=30s; 15     } 16  17     server { 18         listen 127.0.0.1:8443; 19         proxy_connect_timeout 1s; 20         proxy_pass backend; 21     } 22 } 23 EOF 24 [root@k8smaster01 ~]# cd /opt/k8s/work 25 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh 26 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]} 27   do 28     echo ">>> ${master_ip}" 29     scp kube-nginx.conf  root@${master_ip}:/opt/k8s/kube-nginx/conf/kube-nginx.conf 30   done#分发Nginx四层透明代理配置文件

1.6配置Nginx system

1 [root@k8smaster01 ~]# cd /opt/k8s/work  2 [root@k8smaster01 work]# cat > kube-nginx.service <<EOF  3 [Unit]  4 Description=kube-apiserver nginx proxy  5 After=network.target  6 After=network-online.target  7 Wants=network-online.target  8   9 [Service] 10 Type=forking 11 ExecStartPre=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -t 12 ExecStart=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx 13 ExecReload=/opt/k8s/kube-nginx/sbin/kube-nginx -c /opt/k8s/kube-nginx/conf/kube-nginx.conf -p /opt/k8s/kube-nginx -s reload 14 PrivateTmp=true 15 Restart=always 16 RestartSec=5 17 StartLimitInterval=0 18 LimitNOFILE=65536 19  20 [Install] 21 WantedBy=multi-user.target 22 EOF

1.7分发Nginx systemd

1 [root@k8smaster01 ~]# cd /opt/k8s/work  2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh  3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}  4   do  5     echo ">>> ${master_ip}"  6     scp kube-nginx.service  root@${master_ip}:/etc/systemd/system/  7   done

二启动并验证

2.1启动Nginx

1 [root@k8smaster01 ~]# cd /opt/k8s/work  2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh  3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}  4   do  5     echo ">>> ${master_ip}"  6     ssh root@${master_ip} "systemctl daemon-reload && systemctl enable kube-nginx && systemctl restart kube-nginx"  7   done

2.2检查Nginx服务

1 [root@k8smaster01 ~]# cd /opt/k8s/work  2 [root@k8smaster01 work]# source /opt/k8s/bin/environment.sh  3 [root@k8smaster01 work]# for master_ip in ${MASTER_IPS[@]}  4   do  5     echo ">>> ${master_ip}"  6     ssh root@${master_ip} "systemctl status kube-nginx |grep 'Active:'"  7   done
赞 (0)

相关推荐