kubeernetes二進制集群部署 一——etcd組件、f

內容要點：

1、環境包準備

2、部署etcd存儲

3、部署flannel網絡組件

一、環境準備：

以下是官方源碼包下載地址：https://github.com/kubernetes/kubernetes/releases?after=v1.13.1

etcd二進制包地址：https://github.com/etcd-io/etcd/releases

二、部署 etcd 存儲：

第一步：部署?master
先準備好兩個腳本文件：
第一個腳本：
vim?etcd-cert.sh
##定義ca證書：
cat?>?ca-config.json?<<EOF
{
??"signing":?{
????"default":?{
??????"expiry":?"87600h"
????},
????"profiles":?{
??????"www":?{
?????????"expiry":?"87600h",
?????????"usages":?[
????????????"signing",
????????????"key?encipherment",
????????????"server?auth",
????????????"client?auth"
????????]
??????}
????}
??}
}
EOF
##實現證書簽名
cat?>?ca-csr.json?<<EOF
{
????"CN":?"etcd?CA",
????"key":?{
????????"algo":?"rsa",
????????"size":?2048
????},
????"names":?[
????????{
????????????"C":?"CN",
????????????"L":?"Beijing",
????????????"ST":?"Beijing"
????????}
????]
}
EOF
cfssl?gencert?-initca?ca-csr.json?|?cfssljson?-bare?ca?-
#-----------------------
##指定etcd三個節點之間的通信驗證
cat?>?server-csr.json?<<EOF
{
????"CN":?"etcd",
????"hosts":?[
????"192.168.109.131",
????"192.168.109.132",
????"192.168.109.138"
????],
????"key":?{
????????"algo":?"rsa",
????????"size":?2048
????},
????"names":?[
????????{
????????????"C":?"CN",
????????????"L":?"BeiJing",
????????????"ST":?"BeiJing"
????????}
????]
}
EOF
##生成?ETCD證書?server-key.pem?和?server.pem
cfssl?gencert?-ca=ca.pem?-ca-key=ca-key.pem?-config=ca-config.json?-profile=www?server-csr.json?|?cfssljson?-bare?server


第二個腳本：
vim?etcd.sh
#!/bin/bash
#?example:?./etcd.sh?etcd01?192.168.1.10?etcd02=https://192.168.1.11:2380,etcd03=https://192.168.1.12:2380
ETCD_NAME=$1
ETCD_IP=$2
ETCD_CLUSTER=$3
WORK_DIR=/opt/etcd
cat?<<EOF?>$WORK_DIR/cfg/etcd
#[Member]
ETCD_NAME="${ETCD_NAME}"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://${ETCD_IP}:2380"
ETCD_LISTEN_CLIENT_URLS="https://${ETCD_IP}:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://${ETCD_IP}:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://${ETCD_IP}:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://${ETCD_IP}:2380,${ETCD_CLUSTER}"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
EOF
cat?<<EOF?>/usr/lib/systemd/system/etcd.service
[Unit]
Description=Etcd?Server
After=network.target
After=network-online.target
Wants=network-online.target
[Service]
Type=notify
EnvironmentFile=${WORK_DIR}/cfg/etcd
ExecStart=${WORK_DIR}/bin/etcd?\
--name=\${ETCD_NAME}?\
--data-dir=\${ETCD_DATA_DIR}?\
--listen-peer-urls=\${ETCD_LISTEN_PEER_URLS}?\
--listen-client-urls=\${ETCD_LISTEN_CLIENT_URLS},http://127.0.0.1:2379?\
--advertise-client-urls=\${ETCD_ADVERTISE_CLIENT_URLS}?\
--initial-advertise-peer-urls=\${ETCD_INITIAL_ADVERTISE_PEER_URLS}?\
--initial-cluster=\${ETCD_INITIAL_CLUSTER}?\
--initial-cluster-token=\${ETCD_INITIAL_CLUSTER_TOKEN}?\
--initial-cluster-state=new?\
--cert-file=${WORK_DIR}/ssl/server.pem?\
--key-file=${WORK_DIR}/ssl/server-key.pem?\
--peer-cert-file=${WORK_DIR}/ssl/server.pem?\
--peer-key-file=${WORK_DIR}/ssl/server-key.pem?\
--trusted-ca-file=${WORK_DIR}/ssl/ca.pem?\
--peer-trusted-ca-file=${WORK_DIR}/ssl/ca.pem
Restart=on-failure
LimitNOFILE=65536
[Install]
WantedBy=multi-user.target
EOF
systemctl?daemon-reload
systemctl?enable?etcd
systemctl?restart?etcd

[root@master?~]#?mkdir?k8s
[root@master?~]#?cd?k8s/
[root@master?k8s]#?ls
etcd-cert.sh??etcd.sh
[root@master?k8s]#?mkdir?etcd-cert
[root@master?k8s]#?mv?etcd-cert.sh?etcd-cert
[root@master?k8s]#?ls
etcd-cert??etcd.sh
[root@master?k8s]#?vim?cfssl.sh
curl?-L?https://pkg.cfssl.org/R1.2/cfssl_linux-amd64?-o?/usr/local/bin/cfssl
curl?-L?https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64?-o?/usr/local/bin/cfssljson
curl?-L?https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64?-o?/usr/local/bin/cfssl-certinfo
chmod?+x?/usr/local/bin/cfssl?/usr/local/bin/cfssljson?/usr/local/bin/cfssl-certinfo
//下載cfssl官方包：
[root@master?k8s]#?bash?cfssl.sh
[root@master?k8s]#?ls?/usr/local/bin/
cfssl??cfssl-certinfo??cfssljson
//cfssl：生成證書工具；
??cfssl-certinfo：查看證書信息；
??cfssljson：通過傳入json文件生成證書

??

[root@localhost?etcd-cert]#?cd?/usr/local/bin/
[root@localhost?bin]#?ls
cfssl??cfssl-certinfo??cfssljson
//cfssl：是生成證書工具；
??cfssljson：通過傳入json文件生成證書；
??cfssl-certinfo：是查看證書信息。
??
[root@localhost?bin]#?chmod?777?cfssl?cfssl-certinfo?cfssljson???//添加權限
接下來是定義生成?ca?證書：
[root@localhost?bin]#?cd?/root/k8s/etcd-cert/
cat?>?ca-config.json?<<EOF
{
??"signing":?{
????"default":?{
??????"expiry":?"87600h"
????},
????"profiles":?{
??????"www":?{
?????????"expiry":?"87600h",
?????????"usages":?[
????????????"signing",
????????????"key?encipherment",
????????????"server?auth",
????????????"client?auth"
????????]??
??????}?
????}
??}
}
EOF
接下來是實現證書簽名：
cat?>?ca-csr.json?<<EOF?
{???
????"CN":?"etcd?CA",
????"key":?{
????????"algo":?"rsa",
????????"size":?2048
????},
????"names":?[
????????{
????????????"C":?"CN",
????????????"L":?"Beijing",
????????????"ST":?"Beijing"
????????}
????]
}
EOF
產生證書，生成?ca-key.pem??ca.pem
[root@localhost?etcd-cert]#?cfssl?gencert?-initca?ca-csr.json?|?cfssljson?-bare?ca?-
2020/02/07?11:29:31?[INFO]?generating?a?new?CA?key?and?certificate?from?CSR
2020/02/07?11:29:31?[INFO]?generate?received?request
2020/02/07?11:29:31?[INFO]?received?CSR
2020/02/07?11:29:31?[INFO]?generating?key:?rsa-2048
2020/02/07?11:29:32?[INFO]?encoded?CSR
2020/02/07?11:29:32?[INFO]?signed?certificate?with?serial?number?50595628178286351983894910684673691034530190819
[root@localhost?etcd-cert]#?ls
ca-config.json??ca.csr??ca-csr.json??ca-key.pem??ca.pem??etcd-cert.sh
指定etcd三個節點之間的通信驗證：
cat?>?server-csr.json?<<EOF
{
????"CN":?"etcd",
????"hosts":?[
????"192.168.109.131",
????"192.168.109.132",
????"192.168.109.138"
????],
????"key":?{
????????"algo":?"rsa",
????????"size":?2048
????},
????"names":?[
????????{
????????????"C":?"CN",
????????????"L":?"BeiJing",
????????????"ST":?"BeiJing"
????????}
????]
}
EOF
[root@localhost?etcd-cert]#?ls
ca-config.json??ca.csr??ca-csr.json??ca-key.pem??ca.pem??etcd-cert.sh??server-csr.json
生成生成ETCD證書?server-key.pem???server.pem：
[root@localhost?etcd-cert]#?cfssl?gencert?-ca=ca.pem?-ca-key=ca-key.pem?-config=ca-config.json?-profile=www?server-csr.json?|?cfssljson?-bare?server
[root@localhost?etcd-cert]#?ls
ca-config.json??ca-csr.json??ca.pem????????server.csr???????server-key.pem
ca.csr??????????ca-key.pem???etcd-cert.sh??server-csr.json??server.pem

第二步：將下載好的軟件包放到 /root/k8s/etcd-cert 目錄下

[root@master?etcd-cert]#?mv?*.tar.gz?../
[root@master?k8s]#?ls
cfssl.sh??etcd-cert??etcd.sh??etcd-v3.3.10-linux-amd64.tar.gz??flannel-v0.10.0-linux-amd64.tar.gz??kubernetes-server-linux-amd64.tar.gz
[root@master?k8s]#?tar?zvxf?etcd-v3.3.10-linux-amd64.tar.gz??//解壓
[root@master?k8s]#?ls?etcd-v3.3.10-linux-amd64
Documentation??etcd??etcdctl??README-etcdctl.md??README.md??READMEv2-etcdctl.md
//配置文件、命令文件、證書：
[root@master?k8s]#?mkdir?/opt/etcd/{cfg,bin,ssl}?-p
[root@master?k8s]#?mv?etcd-v3.3.10-linux-amd64/etcd?etcd-v3.3.10-linux-amd64/etcdctl?/opt/etcd/bin/
//證書拷貝：
[root@master?k8s]#?cp?etcd-cert/*.pem?/opt/etcd/ssl/
//進入卡住狀態等待其他節點加入：
[root@master?k8s]#?bash?etcd.sh?etcd01?192.168.109.138?etcd02=https://192.168.109.131:2380,etcd03=https://192.168.109.132:2380
//此時，我們可以再開啟一個終端，就會發現?etcd進程已經開啟：
[root@master?~]#?ps?-ef?|?grep?etcd

//將證書拷貝到其他節點（提高效率，無需在配置了）
[root@master?k8s]#?scp?-r?/opt/etcd/?root@192.168.109.131:/opt/
[root@master?k8s]#?scp?-r?/opt/etcd/?root@192.168.109.132:/opt/
//啟動腳本拷貝其他節點：
[root@master?k8s]#?scp?/usr/lib/systemd/system/etcd.service?root@192.168.109.131:/usr/lib/systemd/system/
[root@master?k8s]#?scp?/usr/lib/systemd/system/etcd.service?root@192.168.109.132:/usr/lib/systemd/system/

第三步：部署 node

1、修改?node01：
[root@node01?~]#?vim?/opt/etcd/cfg/etcd?
#[Member]
ETCD_NAME="etcd02"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.109.131:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.109.131:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.109.131:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.109.131:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.109.138:2380,etcd02=https://192.168.109.131:2380,etcd03=https://192.168.109.138:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
//啟動：
[root@node01?~]#?systemctl?start?etcd.service?
[root@node01?~]#?systemctl?status?etcd.service?
2、修改?node02：
[root@node02?~]#?vim?/opt/etcd/cfg/etcd?
#[Member]
ETCD_NAME="etcd03"
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"
ETCD_LISTEN_PEER_URLS="https://192.168.109.132:2380"
ETCD_LISTEN_CLIENT_URLS="https://192.168.109.132:2379"
#[Clustering]
ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.109.132:2380"
ETCD_ADVERTISE_CLIENT_URLS="https://192.168.109.132:2379"
ETCD_INITIAL_CLUSTER="etcd01=https://192.168.109.138:2380,etcd02=https://192.168.109.131:2380,etcd03=https://192.168.109.132:2380"
ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster"
ETCD_INITIAL_CLUSTER_STATE="new"
//啟動：
[root@node02?~]#?systemctl?start?etcd.service?
[root@node02?~]#?systemctl?status?etcd.service

第四步：檢測

在 master 上：

[root@master?etcd-cert]#?/opt/etcd/bin/etcdctl?--ca-file=ca.pem?--cert-file=server.pem?--key-file=server-key.pem?--endpoints="https://192.168.220.131:2379,https://192.168.220.140:2379,https://192.168.220.136:2379"?cluster-health

集群狀態健康！

二、部署 flannel 網絡：

首先所有 node 節點需要部署安裝 docker 引擎

1、Flannel 介紹：是 Overlay 網絡的一種，也是將源數據包封裝在另一種網絡包里面進行路由轉發和通信，目前已經支持 UDP、VXLAN、AWX VPC 和 GCE 路由等數據轉發方式。

• Overlay Network：覆蓋網絡，在基礎網絡上疊加的一種虛擬網絡技術模式，該網絡中的主機通過虛擬鏈路連接起來。

• VXLAN：將源數據包封裝到 UDP 中，并使用基礎網絡的 IP/MAC 作為外層報文頭進行封裝，然后在以太網上傳輸，達到目的地后由隧道端點解封并將數據發給目標地址。

2、flannel 網絡配置：

1、寫入分配的子網段到ETCD中，供flannel使用：

[root@localhost?etcd-cert]#?/opt/etcd/bin/etcdctl?--ca-file=ca.pem?--cert-file=server.pem?--key-file=server-key.pem?--endpoints="https://192.168.220.131:2379,https://192.168.220.140:2379,https://192.168.220.136:2379"?set?/coreos.com/network/config?'{?"Network":?"172.17.0.0/16",?"Backend":?{"Type":?"vxlan"}}'

2、查看寫入的信息：

[root@localhost?etcd-cert]#?/opt/etcd/bin/etcdctl?--ca-file=ca.pem?--cert-file=server.pem?--key-file=server-key.pem?--endpoints="https://192.168.220.131:2379,https://192.168.220.140:2379,https://192.168.220.136:2379"?get?/coreos.com/network/config

3、拷貝軟件包到所有的 node 節點（只需要部署在node節點上即可）：

[root@localhost?k8s]#?scp?flannel-v0.10.0-linux-amd64.tar.gz?root@192.168.109.131:/root
[root@localhost?k8s]#?scp?flannel-v0.10.0-linux-amd64.tar.gz?root@192.168.109.132:/root

4、所有 node 節點將壓縮包解壓：

tar?zvxf?flannel-v0.10.0-linux-amd64.tar.gz

5、在node節點上，先創建k8s工作工作目錄：

[root@localhost?~]#?mkdir?/opt/kubernetes/{cfg,bin,ssl}?-p
[root@localhost?~]#?mv?mk-docker-opts.sh?flanneld?/opt/kubernetes/bin/
創建flannel腳本：
[root@localhost?~]#?vim?flannel.sh
#!/bin/bash
ETCD_ENDPOINTS=${1:-"http://127.0.0.1:2379"}
cat?<<EOF?>/opt/kubernetes/cfg/flanneld
FLANNEL_OPTIONS="--etcd-endpoints=${ETCD_ENDPOINTS}?\
-etcd-cafile=/opt/etcd/ssl/ca.pem?\
-etcd-certfile=/opt/etcd/ssl/server.pem?\
-etcd-keyfile=/opt/etcd/ssl/server-key.pem"
EOF
cat?<<EOF?>/usr/lib/systemd/system/flanneld.service
[Unit]
Description=Flanneld?overlay?address?etcd?agent
After=network-online.target?network.target
Before=docker.service
[Service]
Type=notify
EnvironmentFile=/opt/kubernetes/cfg/flanneld
ExecStart=/opt/kubernetes/bin/flanneld?--ip-masq?\$FLANNEL_OPTIONS
ExecStartPost=/opt/kubernetes/bin/mk-docker-opts.sh?-k?DOCKER_NETWORK_OPTIONS?-d?/run/flannel/subnet.env
Restart=on-failure
[Install]
WantedBy=multi-user.target
EOF
systemctl?daemon-reload
systemctl?enable?flanneld
systemctl?restart?flanneld

6、開啟 flannel 網絡功能：

[root@localhost?~]#?bash?flannel.sh?https://192.168.109.138:2379,https://192.168.109.131:2379,https://192.168.109.132:2379

7、配置 docker 連接 flannel：

[root@localhost?~]#?vim?/usr/lib/systemd/system/docker.service
添加和插入以下代碼：
EnvironmentFile=/run/flannel/subnet.env
$DOCKER_NETWORK_OPTIONS

[root@localhost?~]#?cat?/run/flannel/subnet.env?
DOCKER_OPT_BIP="--bip=172.17.93.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=false"
DOCKER_OPT_MTU="--mtu=1450"
//說明：bip指定啟動時的子網
DOCKER_NETWORK_OPTIONS="?--bip=172.17.93.1/24?--ip-masq=false?--mtu=1450"
//重啟docker
[root@localhost?~]#?systemctl?daemon-reload?
[root@localhost?~]#?systemctl?restart?docker

可以用 ifconfig 命令，查看 flannel 網絡：

接下來是，測試ping通對方docker0網卡 證明flannel起到路由作用：

[root@localhost?~]#?docker?run?-it?centos:7?/bin/bash
[root@bad98ca4fe31?/]#?yum?install?-y?net-tools
[root@bad98ca4fe31?/]#?ifconfig

再在另外一個 node 上做一樣的操作，測試是否可以 ping通 兩個node中的centos:7容器

通過下圖，我們可以發現，在 node01 上的容器，是可以 ping 通 node02 上的容器的，代表 flannel 網絡起到了作用。