前言

APISIX 通过 ETCD 存储数据，利用 ETCD 本身的功能实现群集管理和控制。有关 APISIX 的安装配置详情参考“APISIX 网关 CentOS 7 环境安装配置”。

前面文章描述的是单个 ETCD 节点的配置操作，本文通过配置 ETCD 3节点群集的方式实现 APISIX 的群集操作。

注意

截至 apisix 2.10.1 版本对 etcd mtls 证书的支持仍不完善

环境

操作系统：CentOS7，APISIX：2.10.1 ，Dashboard：2.9，ETCD：3.5

序号	IP	主机名称
1	192.168.112.98	etcd1
2	192.168.112.99	etcd2
3	192.168.112.100	etcd3

补充内容

ApiSix 的前置相关组件安装参考“nstall-dependencies.md”

curl https://raw.githubusercontent.com/apache/apisix/master/utils/install-dependencies.sh -sL | bash -

更详细的安装配置可以参考“how-to-build.md”

环境配置

配置主机

在三台机器上分别执行以下操作完成主机名称设置

cat >> /etc/hosts <<EOF
192.168.112.98 etcd1
192.168.112.99 etcd2
192.168.112.100 etcd3
EOF

安全配置

关闭系统的 Selinux 功能和操作系统防火墙

systemctl stop firewalld
systemctl disable firewalld
setenforce 0
vi /etc/selinux/config
SELINUX=disabled

TLS 密钥和证书

使用TLS证书对通信进行加密，并开启即与CA根证书签名的双向认证

mkdir -p /etc/ssl/etcd/ssl/
mkdir -p /var/lib/etcd

创建根 CA 证书

可以通过 openssl 和 cfssl 来创建证书，这里通过 openssl 的方式创建

cd /etc/ssl/etcd/ssl
openssl genrsa -out ca.key 2048
openssl req -x509 -new -nodes -key ca.key -subj "/CN=apisix" -days 3650 -out ca.crt

创建 ETCD 证书和私钥

1、创建配置文件

cat > ./etcd-ca.conf <<EOF
[ req ]
default_bits = 2048
prompt = no
default_md = rsa
req_extensions = req_ext
distinguished_name = dn
    
[ dn ]
C = CN
ST = GuangDong
L = DongGuan
O = etcd
OU = LingFang
CN = etcd
    
[ req_ext ]
subjectAltName = @alt_names
    
[ alt_names ]
DNS.1 = localhost
DNS.2 = etcd1
DNS.3 = etcd2
DNS.4 = etcd3
IP.1 = 127.0.0.1
IP.2 = 192.168.112.98
IP.3 = 192.168.112.99
IP.4 = 192.168.112.100
    
[ v3_ext ]
authorityKeyIdentifier=keyid,issuer:always
basicConstraints=CA:FALSE
keyUsage=keyEncipherment,dataEncipherment
extendedKeyUsage=serverAuth,clientAuth
subjectAltName=@alt_names
EOF

2、生成密钥

openssl genrsa -out etcd.key 2048

3、生成证书签发请求(certificate signing request)

openssl req -new -key etcd.key -out etcd.csr -config etcd-ca.conf

4、生成证书

openssl x509 -req -in etcd.csr -CA /etc/ssl/etcd/ca.crt -CAkey /etc/ssl/etcd/ca.key  -CAcreateserial -out etcd.crt -days 3650 -extensions v3_ext -extfile etcd-ca.conf

5、验证证书

openssl verify -CAfile ca.crt etcd.crt
openssl x509 -text -noout -in /etc/ssl/etcd/ssl/etcd.crt

安装 ETCD

安装证书

将CA证书ca.crt，etcd证书etcd.crt和秘钥etcd.key，拷贝到各节点的/etc/ssl/etcd/ssl/目录中

下载并安装 ETCD

将其中的 etcd 和 etcdctl 两个可执行文件复制到各节点的/usr/local/bin目录

tar zxvf etcd-v3.5.1-linux-amd64.tar.gz
cp ./etcd-v3.5.1-linux-amd64/{etcd,etcdctl} /usr/local/bin/

以 systemd 形式管理 etcd 服务

1、创建 ETCD 配置文件

mkdir -p /etc/etcd
cat > /etc/etcd/etcd.conf <<EOF


# [Member Flags]
# ETCD_ELECTION_TIMEOUT=1000
# ETCD_HEARTBEAT_INTERVAL=100
# 指定etcd的数据目录
ETCD_NAME=etcd1
ETCD_DATA_DIR=/var/lib/etcd/

# [Cluster Flags]
# ETCD_AUTO_COMPACTION_RETENTIO:N=0
ETCD_INITIAL_CLUSTER_STATE=new
ETCD_ADVERTISE_CLIENT_URLS=https://192.168.112.98:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://192.168.112.98:2380
ETCD_LISTEN_CLIENT_URLS=https://192.168.112.98:2379,https://127.0.0.1:2379
ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster
ETCD_LISTEN_PEER_URLS=https://192.168.112.98:2380
ETCD_INITIAL_CLUSTER=etcd1=https://192.168.112.98:2380,etcd2=https://192.168.112.99:2380,etcd3=https://192.168.112.100:2380

# [Proxy Flags]
ETCD_PROXY=off

# [Security flags]
# ETCD_CLIENT_CERT_AUTH=
# ETCD_PEER_CLIENT_CERT_AUTH=
# 指定etcd的公钥证书和私钥
ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.crt
ETCD_CERT_FILE=/etc/ssl/etcd/ssl/etcd.crt
ETCD_KEY_FILE=/etc/ssl/etcd/ssl/etcd.key
# 指定etcd的Peers通信的公钥证书和私钥
ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.crt
ETCD_PEER_CERT_FILE=/etc/ssl/etcd/ssl/etcd.crt
ETCD_PEER_KEY_FILE=/etc/ssl/etcd/ssl/etcd.key

#ETCD_PEER_CLIENT_CERT_AUTH=true
#ETCD_CLIENT_CERT_AUTH=true

# [Profiling flags]
# ETCD_METRICS={{ etcd_metrics }}
EOF

2、创建 systemd 服务启用文件

cat > /usr/lib/systemd/system/etcd.service <<EOF

[Unit]
Description=etcd server
After=network.target
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.conf
ExecStart=/usr/local/bin/etcd
NotifyAccess=all
Restart=always
RestartSec=5s
LimitNOFILE=40000

[Install]
WantedBy=multi-user.target
EOF

其中 ETCD2 和 ETCD3 主机配置文件中的内容需要对应修改

启动 ETCD

只有一个节点启动etcd服务后，etcd会去访问其他两个节点的2380端口，如果一定时间内无法访问，则会服务启动失败。所以应当迅速启动至少两个节点，服务才会成功启动。

systemctl daemon-reload
systemctl enable etcd
systemctl start etcd
systemctl status etcd

检查集群是否健康

etcdctl --cacert=/etc/ssl/etcd/ssl/ca.crt --cert=/etc/ssl/etcd/ssl/etcd.crt  --key=/etc/ssl/etcd/ssl/etcd.key --endpoints="https://192.168.112.98:2379,https://192.168.112.99:2379,https://192.168.112.100:2379" endpoint health

curl --cacert /etc/ssl/etcd/ssl/ca.crt -i https://192.168.112.98:2379/version --cert /etc/ssl/etcd/ssl/etcd.crt --key /etc/ssl/etcd/ssl/etcd.key

etcdctl --cacert=/etc/ssl/etcd/ssl/ca.crt --cert=/etc/ssl/etcd/ssl/etcd.crt  --key=/etc/ssl/etcd/ssl/etcd.key --endpoints="https://192.168.112.99:2379" get / --prefix --keys-only
openssl s_client -connect 192.168.112.98:2379 -key /etc/ssl/etcd/etcd.key  -cert /etc/ssl/etcd/etcd.crt -CAfile /etc/ssl/etcd/ssl/ca.crt

ETCD 参考指南

https://jimmysong.io/kubernetes-handbook/practice/etcd-cluster-installation.html

APISix 配置

主要涉及配置文件和 ETCD 连接部分的内容

config.yaml

apisix:
  admin_key:
    - name: admin
      key: 3cea9e6d29e70e98c09d29ca98bd452a43d057bb3e6a748f3d835492
      role: admin
  port_admin: 9180
  node_listen:
        - 80

  allow_admin:
    - 127.0.0.0/24

  ssl:
    enable: true
    listen:
            - 443
    ssl_trusted_certificate: /etc/ssl/etcd/ssl/ca.crt
    #ssl_protocols: SSLv3 TLSv1.2 TLSv1.3

etcd:
  host:

    - "https://127.0.0.1:2379"
    - "https://192.168.112.98:2379"
    - "https://192.168.112.99:2379"
    - "https://192.168.112.100:2379"

  prefix: /apisix
  timeout: 30
  resync_delay: 5
  health_check_timeout: 10

  tls:
    cert: /etc/ssl/etcd/ssl/etcd.crt
    key: /etc/ssl/etcd/ssl/etcd.key
    verify: true

conf.yaml

conf:
  listen:
    # host: 127.0.0.1     
    port: 9000

  # ssl:
  #   host: 127.0.0.1     # the address on which the `Manager API` should listen for HTTPS.
                          # The default value is 0.0.0.0, if want to specify, please enable it.
  #   port: 9001            # The port on which the `Manager API` should listen for HTTPS.
  #   cert: "/tmp/cert/example.crt" # Path of your SSL cert.
  #   key:  "/tmp/cert/example.key"  # Path of your SSL key.

  allow_list:             # If we don't set any IP list, then any IP access is allowed by default.
    - 127.0.0.1           # The rules are checked in sequence until the first match is found.
    - ::1
    - 0.0.0.0/0
  etcd:
    endpoints:
      - 127.0.0.1:2379
      - 192.168.112.98:2379
      - 192.168.112.99:2379
      - 192.168.112.100:2379
    mtls:
      key_file: "/etc/ssl/etcd/ssl/etcd.key"          # Path of your self-signed client side key
      cert_file: "/etc/ssl/etcd/ssl/etcd.crt"         # Path of your self-signed client side cert
      ca_file: "/etc/ssl/etcd/ssl/ca.crt"           # Path of your self-signed ca cert, the CA is used to sign callers' certificates
    # prefix: /apisix       # apisix config's prefix in etcd, /apisix by default
  log:
    error_log:
      level: warn       # supports levels, lower to higher: debug, info, warn, error, panic, fatal
      file_path:
        logs/error.log  # supports relative path, absolute path, standard output
                       
    access_log:
      file_path:
        logs/access.log  # supports relative path, absolute path, standard output
  max_cpu: 0             

authentication:
  secret:
    secret              # secret for jwt token generation.

  expire_time: 3600     # jwt token expire time, in second
  users:                # yamllint enable rule:comments-indentation
    - username: admin   # username and password for login `manager api`
      password: admin
    - username: user
      password: user

plugins:                          # plugin list (sorted in alphabetical order)
  - api-breaker
  - authz-keycloak
  - basic-auth
  - batch-requests
  - consumer-restriction
  - cors
  # - dubbo-proxy
  - echo
  # - error-log-logger
  # - example-plugin
  - fault-injection
  - grpc-transcode
  - hmac-auth
  - http-logger
  - ip-restriction
  - jwt-auth
  - kafka-logger
  - key-auth
  - limit-conn
  - limit-count
  - limit-req
  # - log-rotate
  # - node-status
  - openid-connect
  - prometheus
  - proxy-cache
  - proxy-mirror
  - proxy-rewrite
  - redirect
  - referer-restriction
  - request-id
  - request-validation
  - response-rewrite
  - serverless-post-function
  - serverless-pre-function
  # - skywalking
  - sls-logger
  - syslog
  - tcp-logger
  - udp-logger
  - uri-blocker
  - wolf-rbac
  - zipkin
  - server-info
  - traffic-split

防火墙配置

受 ApiSix 在匹配 ETCD 群集 mtls 的限制，ETCD 群集不采用证书配置模式，考虑安全通过系统自带的防火墙功能实现对源 IP 地址对目标端口的访问控制。

systemctl start firewalld
firewall-cmd --add-port=80/tcp --permanent
firewall-cmd --add-port=443/tcp --permanent
firewall-cmd --add-port=9000/tcp --permanent
firewall-cmd --add-port=22/tcp --permanent
firewall-cmd --list-rich-rules
firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.112.100" port port="2379" protocol=tcp accept'
firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.112.100" port port="2380" protocol=tcp accept'
firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.112.98" port port="2379" protocol=tcp accept'
firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="192.168.112.98" port port="2380" protocol=tcp accept'

空间快捷链接

页面树结构

前言

环境

补充内容

环境配置

配置主机

安全配置

TLS 密钥和证书

建立证书目录和数据目录

创建根 CA 证书

创建 ETCD 证书和私钥

安装 ETCD

安装证书

下载并安装 ETCD

以 systemd 形式管理 etcd 服务

启动 ETCD

检查集群是否健康

ETCD 参考指南

APISix 配置

config.yaml

conf.yaml

防火墙配置

空间快捷链接

页面树结构

4019-ApiSix 群集配置

前言

环境

补充内容

环境配置

配置主机

安全配置

TLS 密钥和证书

建立证书目录和数据目录

创建根 CA 证书

创建 ETCD 证书和私钥

安装 ETCD

安装证书

下载并安装 ETCD

以 systemd 形式管理 etcd 服务

启动 ETCD

检查集群是否健康

ETCD 参考指南

APISix 配置

config.yaml

conf.yaml

防火墙配置