溫馨提示×
您好，登錄后才能下訂單哦！
點擊 登錄注冊 即表示同意《億速云用戶服務條款》
您好，登錄后才能下訂單哦！
這篇文章主要介紹“如何理解容器部署ELK7.10”，在日常操作中，相信很多人在如何理解容器部署ELK7.10問題上存在疑惑，小編查閱了各式資料，整理出簡單好用的操作方法，希望對大家解答”如何理解容器部署ELK7.10”的疑惑有所幫助！接下來，請跟著小編一起來學習吧！
一、elk架構簡介
首先 logstash 具有日志采集、過濾、篩選等功能，功能完善但同時體量也會比較大，消耗系統資源自然也多。filebeat作為一個輕量級日志采集工具，雖然沒有過濾篩選功能，但是僅僅部署在應用服務器作為我們采集日志的工具可以是說最好的選擇。但我們有些時候可能又需要logstash的過濾篩選功能，所以我們在采集日志時用filebeat，然后交給logstash過濾篩選。
其次，logstash的吞吐量是有限的，一旦短時間內filebeat傳過來的日志過多會產生堆積和堵塞，對日志的采集也會受到影響，所以在filebeat與logstash中間又加了一層kafka消息隊列來緩存或者說解耦，當然redis也是可以的。這樣當眾多filebeat節點采集大量日志直接放到kafka中，logstash慢慢的進行消費，兩邊互不干擾。
至于zookeeper，分布式服務管理神器，監控管理kafka的節點注冊，topic管理等，同時彌補了kafka集群節點對外界無法感知的問題，kafka實際已經自帶了zookeeper，這里將會使用獨立的zookeeper進行管理，方便后期zookeeper集群的擴展。
二、環境
阿里云ECS：5臺部署ES節點，3臺分別部署logstash、kafka、zookeeper和kibana等服務。
阿里云ECS配置：5臺 4核16G SSD磁盤。3臺 4核16G SSD磁盤。都是 Centos7.8系統
安裝 docker 和 docker-compose
ELK版本7.10.1;zookeeper版本3.6.2;kafka版本2.13-2.6.0;
三、系統參數優化
# 最大用戶打開進程數 $ vim /etc/security/limits.d/20-nproc.conf * soft nproc 65535 * hard nproc 65535 # 優化內核，用于 docker 支持 $ modprobe br_netfilter $ cat <<EOF > /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 EOF $ sysctl -p /etc/sysctl.d/k8s.conf # 優化內核，對 es 支持 $ echo 'vm.max_map_count=262144' >> /etc/sysctl.conf # 生效配置 $ sysctl -p
四、部署 docker 和 docker-compose
部署 docker
# 安裝必要的一些系統工具 $ yum install -y yum-utils device-mapper-persistent-data lvm2 # 添加軟件源信息 $ yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo # 更新并安裝 Docker-CE $ yum makecache fast $ yum -y install docker-ce # 配置docker $ systemctl enable docker $ systemctl start docker $ vim /etc/docker/daemon.json {"data-root": "/var/lib/docker", "bip": "10.50.0.1/16", "default-address-pools": [{"base": "10.51.0.1/16", "size": 24}], "registry-mirrors": ["https://4xr1qpsp.mirror.aliyuncs.com"], "log-opts": {"max-size":"500m", "max-file":"3"}} $ sed -i '/ExecStart=/i ExecStartPost=\/sbin\/iptables -P FORWARD ACCEPT' /usr/lib/systemd/system/docker.service $ systemctl enable docker.service $ systemctl daemon-reload $ systemctl restart docker部署 docker-compose
# 安裝 docker-compose $ sudo curl -L "https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose $ chmod +x /usr/local/bin/docker-compose
五、部署 ES
es-master1 操作
# 創建 es 目錄 $ mkdir /data/ELKStack $ mkdir elasticsearch elasticsearch-data elasticsearch-plugins # 容器es用戶 uid 和 gid 都是 1000 $ chown 1000.1000 elasticsearch-data elasticsearch-plugins # 臨時啟動一個es $ docker run --name es-test -it --rm docker.elastic.co/elasticsearch/elasticsearch:7.10.1 bash # 生成證書，證書有效期10年，證書輸入的密碼這里為空 $ bin/elasticsearch-certutil ca --days 3660 $ bin/elasticsearch-certutil cert --ca elastic-stack-ca.p12 --days 3660 # 打開新的窗口，拷貝生成的證書 $ cd /data/ELKStack/elasticsearch $ mkdir es-p12 $ docker cp es-test:/usr/share/elasticsearch/elastic-certificates.p12 ./es-p12 $ docker cp es-test:/usr/share/elasticsearch/elastic-stack-ca.p12 ./es-p12 $ chown -R 1000.1000 ./es-p12 # 創建 docker-compose.yml $ vim docker-compose.yml version: '2.2' services: elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:7.10.1 container_name: es01 environment: - cluster.name=es-docker-cluster - cluster.initial_master_nodes=es01,es02,es03 - bootstrap.memory_lock=true - "ES_JAVA_OPTS=-Xms10000m -Xmx10000m" ulimits: memlock: soft: -1 hard: -1 nofile: soft: 65536 hard: 65536 mem_limit: 13000m cap_add: - IPC_LOCK restart: always # 設置 docker host 網絡模式 network_mode: "host" volumes: - /data/ELKStack/elasticsearch-data:/usr/share/elasticsearch/data - /data/ELKStack/elasticsearch-plugins:/usr/share/elasticsearch/plugins - /data/ELKStack/elasticsearch/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml - /data/ELKStack/elasticsearch/es-p12:/usr/share/elasticsearch/config/es-p12 # 創建 elasticsearch.yml 配置文件 $ vim elasticsearch.yml cluster.name: "es-docker-cluster" node.name: "es01" network.host: 0.0.0.0 node.master: true node.data: true discovery.zen.minimum_master_nodes: 2 http.port: 9200 transport.tcp.port: 9300 # 如果是多節點es，通過ping來健康檢查 discovery.zen.ping.unicast.hosts: ["172.20.166.25:9300", "172.20.166.24:9300", "172.20.166.22:9300", "172.20.166.23:9300", "172.20.166.26:9300"] discovery.zen.fd.ping_timeout: 120s discovery.zen.fd.ping_retries: 6 discovery.zen.fd.ping_interval: 10s cluster.info.update.interval: 1m indices.fielddata.cache.size: 20% indices.breaker.fielddata.limit: 40% indices.breaker.request.limit: 40% indices.breaker.total.limit: 70% indices.memory.index_buffer_size: 20% script.painless.regex.enabled: true # 磁盤分片分配 cluster.routing.allocation.disk.watermark.low: 100gb cluster.routing.allocation.disk.watermark.high: 50gb cluster.routing.allocation.disk.watermark.flood_stage: 30gb # 本地數據分片恢復配置 gateway.recover_after_nodes: 3 gateway.recover_after_time: 5m gateway.expected_nodes: 3 cluster.routing.allocation.node_initial_primaries_recoveries: 8 cluster.routing.allocation.node_concurrent_recoveries: 2 # 允許跨域請求 http.cors.enabled: true http.cors.allow-origin: "*" http.cors.allow-headers: Authorization,X-Requested-With,Content-Length,Content-Type # 開啟xpack xpack.security.enabled: true xpack.monitoring.collection.enabled: true # 開啟集群中https傳輸 xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: es-p12/elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: es-p12/elastic-certificates.p12 # 把 es 配置使用 rsync 同步到其它 es 節點 $ rsync -avp -e ssh /data/ELKStack 172.20.166.24:/data/ $ rsync -avp -e ssh /data/ELKStack 172.20.166.22:/data/ $ rsync -avp -e ssh /data/ELKStack 172.20.166.23:/data/ $ rsync -avp -e ssh /data/ELKStack 172.20.166.26:/data/ # 啟動 es $ docker-compose up -d # 查看 es $ docker-compose ps
es-master2 操作
$ cd /data/ELKStack/elasticsearch # 修改 docker-compose.yml elasticsearch.yml 兩個配置 $ sed -i 's/es01/es02/g' docker-compose.yml elasticsearch.yml # 啟動 es $ docker-compose up -d
es-master3 操作
$ cd /data/ELKStack/elasticsearch # 修改 docker-compose.yml elasticsearch.yml 兩個配置 $ sed -i 's/es01/es03/g' docker-compose.yml elasticsearch.yml # 啟動 es $ docker-compose up -d
es-data1 操作
$ cd /data/ELKStack/elasticsearch # 修改 docker-compose.yml elasticsearch.yml 兩個配置 $ sed -i 's/es01/es04/g' docker-compose.yml elasticsearch.yml # 不做為 es master 節點，只做數據節點 $ sed -i 's/node.master: true/node.master: false/g' elasticsearch.yml # 啟動 es $ docker-compose up -d
es-data2 操作
$ cd /data/ELKStack/elasticsearch # 修改 docker-compose.yml elasticsearch.yml 兩個配置 $ sed -i 's/es01/es05/g' docker-compose.yml elasticsearch.yml # 不做為 es master 節點，只做數據節點 $ sed -i 's/node.master: true/node.master: false/g' elasticsearch.yml # 啟動 es $ docker-compose up -d
設置 es 訪問賬號
# es-master1 操作 $ docker exec -it es01 bash # 設置 elastic，apm_system，kibana，kibana_system，logstash_system，beats_system，remote_monitoring_user 等密碼 # 密碼都設置為 elastic123，這里只是舉例，具體根據需求設置 $ ./bin/elasticsearch-setup-passwords interactive
六、部署 Kibana
logstash4 操作
$ mkdir -p /data/ELKStack/kibana $ cd /data/ELKStack/kibana # 創建 kibana 相關目錄，用于容器掛載 $ mkdir config data plugins $ chown 1000.1000 config data plugins # 創建 docker-compose.yml $ vim docker-compose.yml version: '2' services: kibana: image: docker.elastic.co/kibana/kibana:7.10.1 container_name: kibana restart: always network_mode: "bridge" mem_limit: 2000m environment: SERVER_NAME: kibana.example.com ports: - "5601:5601" volumes: - /data/ELKStack/kibana/config:/usr/share/kibana/config - /data/ELKStack/kibana/data:/usr/share/kibana/data - /data/ELKStack/kibana/plugins:/usr/share/kibana/plugins # 創建 kibana.yml $ vim config/kibana.yml server.name: kibana server.host: "0" elasticsearch.hosts: ["http://172.20.166.25:9200","http://172.20.166.24:9200","http://172.20.166.22:9200"] elasticsearch.username: "kibana" elasticsearch.password: "elastic123" monitoring.ui.container.elasticsearch.enabled: true xpack.security.enabled: true xpack.encryptedSavedObjects.encryptionKey: encryptedSavedObjects12345678909876543210 xpack.security.encryptionKey: encryptionKeysecurity12345678909876543210 xpack.reporting.encryptionKey: encryptionKeyreporting12345678909876543210 i18n.locale: "zh-CN" # 啟動 kibana $ docker-compose up -d
七、部署 Zookeeper
logstash2 操作
# 創建 zookeeper 目錄 $ mkdir /data/ELKStack/zookeeper $ cd /data/ELKStack/zookeeper $ mkdir data datalog $ chown 1000.1000 data datalog # 創建 docker-compose.yml $ vim docker-compose.yml version: '2' services: zoo1: image: zookeeper:3.6.2 restart: always hostname: zoo1 container_name: zoo1 network_mode: "bridge" mem_limit: 2000m ports: - 2181:2181 - 3888:3888 - 2888:2888 volumes: - /data/ELKStack/zookeeper/data:/data - /data/ELKStack/zookeeper/datalog:/datalog - /data/ELKStack/zookeeper/zoo.cfg:/conf/zoo.cfg environment: ZOO_MY_ID: 1 # 表示 ZK服務的 id, 它是1-255 之間的整數, 必須在集群中唯一 ZOO_SERVERS: server.1=0.0.0.0:2888:3888;2181 server.2=172.20.166.28:2888:3888;2181 server.3=172.20.166.29:2888:3888;2181 # ZOOKEEPER_CLIENT_PORT: 2181 # 創建 zoo.cfg 配置 $ vim zoo.cfg tickTime=2000 initLimit=10 syncLimit=5 dataDir=/data dataLogDir=/datalog autopurge.snapRetainCount=3 autopurge.purgeInterval=1 maxClientCnxns=60 server.1= 0.0.0.0:2888:3888;2181 server.2= 172.20.166.28:2888:3888;2181 server.3= 172.20.166.29:2888:3888;2181 # 拷貝配置到 logstash3 logstash4 機器上 $ rsync -avp -e ssh /data/ELKStack/zookeeper 172.20.166.28:/data/ELKStack/ $ rsync -avp -e ssh /data/ELKStack/zookeeper 172.20.166.29:/data/ELKStack/ # 啟動 zookeeper $ docker-compose up -d
logstash3 操作
$ cd /data/ELKStack/zookeeper # 修改 docker-compose.yml 文件 $ vim docker-compose.yml version: '2' services: zoo2: image: zookeeper:3.6.2 restart: always hostname: zoo2 container_name: zoo2 network_mode: "bridge" mem_limit: 2000m ports: - 2181:2181 - 3888:3888 - 2888:2888 volumes: - /data/ELKStack/zookeeper/data:/data - /data/ELKStack/zookeeper/datalog:/datalog - /data/ELKStack/zookeeper/zoo.cfg:/conf/zoo.cfg environment: ZOO_MY_ID: 2 # 表示 ZK服務的 id, 它是1-255 之間的整數, 必須在集群中唯一 ZOO_SERVERS: server.1=172.20.166.27:2888:3888;2181 server.2=0.0.0.0:2888:3888;2181 server.3=172.20.166.29:2888:3888;2181 # ZOOKEEPER_CLIENT_PORT: 2181 # 修改 zoo.cfg $ vim zoo.cfg tickTime=2000 initLimit=10 syncLimit=5 dataDir=/data dataLogDir=/datalog autopurge.snapRetainCount=3 autopurge.purgeInterval=1 maxClientCnxns=60 server.1= 172.20.166.27:2888:3888;2181 server.2= 0.0.0.0:2888:3888;2181 server.3= 172.20.166.29:2888:3888;2181 # 啟動 zookeeper $ docker-compose up -d
logstash4 操作
$ cd /data/ELKStack/zookeeper # 修改 docker-compose.yml 文件 $ vim docker-compose.yml version: '2' services: zoo3: image: zookeeper:3.6.2 restart: always hostname: zoo3 container_name: zoo3 network_mode: "bridge" mem_limit: 2000m ports: - 2181:2181 - 3888:3888 - 2888:2888 volumes: - /data/ELKStack/zookeeper/data:/data - /data/ELKStack/zookeeper/datalog:/datalog - /data/ELKStack/zookeeper/zoo.cfg:/conf/zoo.cfg environment: ZOO_MY_ID: 3 # 表示 ZK服務的 id, 它是1-255 之間的整數, 必須在集群中唯一 ZOO_SERVERS: server.1=172.20.166.27:2888:3888;2181 server.2=172.20.166.28:2888:3888;2181 server.3=0.0.0.0:2888:3888;2181 # ZOOKEEPER_CLIENT_PORT: 2181 # 修改 zoo.cfg $ vim zoo.cfg tickTime=2000 initLimit=10 syncLimit=5 dataDir=/data dataLogDir=/datalog autopurge.snapRetainCount=3 autopurge.purgeInterval=1 maxClientCnxns=60 server.1= 172.20.166.27:2888:3888;2181 server.2= 172.20.166.28:2888:3888;2181 server.3= 0.0.0.0:2888:3888;2181 # 啟動 zookeeper $ docker-compose up -d # 操作 zookeeper $ docker run -it zoo3 bash $ zkCli.sh -server 172.20.166.27:2181,172.20.166.28:2181,172.20.166.29:2181
八、部署 Kafka
logstash2 操作
# 創建 kafka 目錄 $ mkdir -p /data/ELKStack/kafka $ cd /data/ELKStack/kafka # 創建數據目錄，用于存儲kafka容器數據 $ mkdir data # 把kafka配置拷貝到宿主機上 $ docker run --name kafka-test -it --rm wurstmeister/kafka:2.13-2.6.0 bash $ cd /opt/kafka $ tar zcvf /tmp/config.tar.gz config # 打開一個新的窗口 $ docker cp kafka-test:/tmp/config.tar.gz ./ # 解壓配置文件 $ tar xf config.tar.gz # 創建 docker-compose.yml $ vim docker-compose.yml version: '2' services: kafka1: image: wurstmeister/kafka:2.13-2.6.0 restart: always hostname: kafka1 container_name: kafka1 network_mode: "bridge" mem_limit: 5120m ports: - 9092:9092 - 9966:9966 environment: KAFKA_BROKER_ID: 1 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://172.20.166.27:9092 # 宿主機的IP地址而非容器的IP，及暴露出來的端口 KAFKA_ADVERTISED_HOST_NAME: 172.20.166.27 # 外網訪問地址 KAFKA_ADVERTISED_PORT: 9092 # 端口 KAFKA_ZOOKEEPER_CONNECT: 172.20.166.27:2181,172.20.166.28:2181,172.20.166.29:2181 # 連接的zookeeper服務及端口 KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=172.20.166.27 -Dcom.sun.management.jmxremote.rmi.port=9966" JMX_PORT: 9966 # kafka需要監控broker和topic的數據的時候,是需要開啟jmx_port的 KAFKA_HEAP_OPTS: "-Xmx4096M -Xms4096M" volumes: - /data/ELKStack/kafka/data:/kafka # kafka數據文件存儲目錄 - /data/ELKStack/kafka/config:/opt/kafka/config # 優化 kafka server.properties 配置 $ vim config/server.properties # 調大socket，防止報錯 socket.send.buffer.bytes=1024000 socket.receive.buffer.bytes=1024000 socket.request.max.bytes=1048576000 # topic 數據保留多久，默認168小時(7day) log.retention.hours=72 log.cleanup.policy=delete # 拷貝配置到 logstash3 logstash4 機器上 $ rsync -avp -e ssh /data/ELKStack/kafka 172.20.166.28:/data/ELKStack/ $ rsync -avp -e ssh /data/ELKStack/kafka 172.20.166.29:/data/ELKStack/ # 啟動 kafka $ docker-compose up -d
logstash3 操作
$ cd /data/ELKStack/kafka # 修改 docker-compose.yml 文件 $ vim docker-compose.yml version: '2' services: kafka2: image: wurstmeister/kafka:2.13-2.6.0 restart: always hostname: kafka2 container_name: kafka2 network_mode: "bridge" mem_limit: 5120m ports: - 9092:9092 - 9966:9966 environment: KAFKA_BROKER_ID: 2 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://172.20.166.28:9092 # 宿主機的IP地址而非容器的IP，及暴露出來的端口 KAFKA_ADVERTISED_HOST_NAME: 172.20.166.28 # 外網訪問地址 KAFKA_ADVERTISED_PORT: 9092 # 端口 KAFKA_ZOOKEEPER_CONNECT: 172.20.166.27:2181,172.20.166.28:2181,172.20.166.29:2181 # 連接的zookeeper服務及端口 KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=172.20.166.28 -Dcom.sun.management.jmxremote.rmi.port=9966" JMX_PORT: 9966 # kafka需要監控broker和topic的數據的時候,是需要開啟jmx_port的 KAFKA_HEAP_OPTS: "-Xmx4096M -Xms4096M" volumes: - /data/ELKStack/kafka/data:/kafka # kafka數據文件存儲目錄 - /data/ELKStack/kafka/config:/opt/kafka/config # 啟動 kafka $ docker-compose up -d
logstash4 操作
$ cd /data/ELKStack/kafka # 修改 docker-compose.yml 文件 $ vim docker-compose.yml version: '2' services: kafka3: image: wurstmeister/kafka:2.13-2.6.0 restart: always hostname: kafka3 container_name: kafka3 network_mode: "bridge" mem_limit: 5120m ports: - 9092:9092 - 9966:9966 environment: KAFKA_BROKER_ID: 3 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://172.20.166.29:9092 # 宿主機的IP地址而非容器的IP，及暴露出來的端口 KAFKA_ADVERTISED_HOST_NAME: 172.20.166.29 # 外網訪問地址 KAFKA_ADVERTISED_PORT: 9092 # 端口 KAFKA_ZOOKEEPER_CONNECT: 172.20.166.27:2181,172.20.166.28:2181,172.20.166.29:2181 # 連接的zookeeper服務及端口 KAFKA_JMX_OPTS: "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=172.20.166.29 -Dcom.sun.management.jmxremote.rmi.port=9966" JMX_PORT: 9966 # kafka需要監控broker和topic的數據的時候,是需要開啟jmx_port的 KAFKA_HEAP_OPTS: "-Xmx4096M -Xms4096M" volumes: - /data/ELKStack/kafka/data:/kafka # kafka數據文件存儲目錄 - /data/ELKStack/kafka/config:/opt/kafka/config # 啟動 kafka $ docker-compose up -d # 部署 kafka-manager 管理 kafka 平臺 $ mkdir /data/ELKStack/kafka-manager $ cd /data/ELKStack/kafka-manager $ vim docker-compose.yml version: '3.6' services: kafka_manager: restart: always container_name: kafa-manager hostname: kafka-manager network_mode: "bridge" mem_limit: 1024m image: hlebalbau/kafka-manager:3.0.0.5-7e7a22e ports: - "9000:9000" environment: ZK_HOSTS: "172.20.166.27:2181,172.20.166.28:2181,172.20.166.29:2181" APPLICATION_SECRET: "random-secret" KAFKA_MANAGER_AUTH_ENABLED: "true" KAFKA_MANAGER_USERNAME: admin KAFKA_MANAGER_PASSWORD: elastic123 JMX_PORT: 9966 TZ: "Asia/Shanghai" # 啟動 kafka-manager $ docker-compose up -d # 訪問 http://172.20.166.29:9000 ，把上面創建的三臺 kafka 加入管理，這里不在闡述，網上很多配置教程
九、部署 logstash
logstash2 操作
$ mkdir /data/ELKStack/logstash $ cd /data/ELKStack/logstash $ mkdir config data $ chown 1000.1000 config data # 創建 docker-compose.yml $ vim docker-compose.yml version: '2' services: logstash2: image: docker.elastic.co/logstash/logstash:7.10.1 container_name: logstash2 hostname: logstash2 restart: always network_mode: "bridge" mem_limit: 4096m environment: TZ: "Asia/Shanghai" ports: - 5044:5044 volumes: - /data/ELKStack/logstash/config:/config-dir - /data/ELKStack/logstash/logstash.yml:/usr/share/logstash/config/logstash.yml - /data/ELKStack/logstash/data:/usr/share/logstash/data - /etc/localtime:/etc/localtime user: logstash command: bash -c "logstash -f /config-dir --config.reload.automatic" # 創建 logstash.yml $ vim logstash.yml http.host: "0.0.0.0" # 指發送到Elasticsearch的批量請求的大小，值越大，處理則通常更高效，但增加了內存開銷 pipeline.batch.size: 3000 # 指調整Logstash管道的延遲，過了該時間則logstash開始執行過濾器和輸出 pipeline.batch.delay: 200 # 創建 logstash 規則配置 $ vim config/01-input.conf input { # 輸入組件 kafka { # 從kafka消費數據 bootstrap_servers => ["172.20.166.27:9092,172.20.166.28:9092,172.20.166.29:9092"] #topics => "%{[@metadata][topic]}" # 使用kafka傳過來的topic topics_pattern => "elk-.*" # 使用正則匹配topic codec => "json" # 數據格式 consumer_threads => 3 # 消費線程數量 decorate_events => true # 可向事件添加Kafka元數據，比如主題、消息大小的選項，這將向logstash事件中添加一個名為kafka的字段 auto_offset_reset => "latest" # 自動重置偏移量到最新的偏移量 group_id => "logstash-node" # 消費組ID，多個有相同group_id的logstash實例為一個消費組 client_id => "logstash2" # 客戶端ID fetch_max_wait_ms => "1000" # 指當沒有足夠的數據立即滿足fetch_min_bytes時，服務器在回答fetch請求之前將阻塞的最長時間 } } $ vim config/02-output.conf output { # 輸出組件 elasticsearch { # Logstash輸出到es hosts => ["172.20.166.25:9200", "172.20.166.24:9200", "172.20.166.22:9200", "172.20.166.23:9200", "172.20.166.26:9200"] index => "%{[fields][source]}-%{+YYYY-MM-dd}" # 直接在日志中匹配，索引會去掉elk # index => "%{[@metadata][topic]}-%{+YYYY-MM-dd}" # 以日期建索引 user => "elastic" password => "elastic123" } #stdout { # codec => rubydebug #} } $ vim config/03-filter.conf filter { # 當非業務字段時，無traceId則移除 if ([message] =~ "traceId=null") { # 過濾組件，這里只是展示，無實際意義，根據自己的業務需求進行過濾 drop {} } } # 拷貝配置到 logstash3 logstash4 機器上 $ rsync -avp -e ssh /data/ELKStack/logstash 172.20.166.28:/data/ELKStack/ $ rsync -avp -e ssh /data/ELKStack/logstash 172.20.166.29:/data/ELKStack/ # 啟動 logstash $ docker-compose up -dlogstash3 操作
$ cd /data/ELKStack/logstash $ sed -i 's/logstash2/logstash3/g' docker-compose.yml $ sed -i 's/logstash2/logstash3/g' config/01-input.conf # 啟動 logstash $ docker-compose up -d
logstash4 操作
$ cd /data/ELKStack/logstash $ sed -i 's/logstash2/logstash4/g' docker-compose.yml $ sed -i 's/logstash2/logstash4/g' config/01-input.conf # 啟動 logstash $ docker-compose up -d
十、部署 filebeat
# 配置 filebeat yum源，這里以 centos7 為例 $ rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch $ vim /etc/yum.repos.d/elastic.repo [elastic-7.x] name=Elastic repository for 7.x packages baseurl=https://artifacts.elastic.co/packages/7.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md $ yum install -y filebeat-7.10.1 $ systemctl enable filebeat # 配置 $ cd /etc/filebeat/ $ cp -a filebeat.yml filebeat.yml.old $ echo > filebeat.yml # 以收集nginx訪問日志為例 $ vim filebeat.yml filebeat.inputs: # inputs為復數，表名type可以有多個 - type: log # 輸入類型 access: enabled: true # 啟用這個type配置 json.keys_under_root: true # 默認這個值是FALSE的，也就是我們的json日志解析后會被放在json鍵上。設為TRUE，所有的keys就會被放到根節點 json.overwrite_keys: true # 是否要覆蓋原有的key，這是關鍵配置，將keys_under_root設為TRUE后，再將overwrite_keys也設為TRUE，就能把filebeat默認的key值給覆蓋 max_bytes: 20480 # 單條日志的大小限制,建議限制(默認為10M,queue.mem.events * max_bytes 將是占有內存的一部分) paths: - /var/log/nginx/access.log # 監控nginx 的access日志 fields: # 額外的字段 source: nginx-access-prod # 自定義source字段，用于es建議索引（字段名小寫，我記得大寫好像不行） # 自定義es的索引需要把ilm設置為false setup.ilm.enabled: false output.kafka: # 輸出到kafka enabled: true # 該output配置是否啟用 hosts: ["172.20.166.27:9092", "172.20.166.28:9092", "172.20.166.29:9092"] # kafka節點列表 topic: "elk-%{[fields.source]}" # kafka會創建該topic，然后logstash(可以過濾修改)會傳給es作為索引名稱 partition.hash: reachable_only: true # 是否只發往可達分區 compression: gzip # 壓縮 max_message_bytes: 1000000 # Event最大字節數。默認1000000。應小于等于kafka broker message.max.bytes值 required_acks: 1 # kafka ack等級 worker: 1 # kafka output的最大并發數 bulk_max_size: 2048 # 單次發往kafka的最大事件數 logging.to_files: true # 輸出所有日志到file，默認true， 達到日志文件大小限制時，日志文件會自動限制替換，詳細配置：https://www.cnblogs.com/qinwengang/p/10982424.html close_older: 30m # 如果一個文件在某個時間段內沒有發生過更新，則關閉監控的文件handle。默認1h force_close_files: false # 這個選項關閉一個文件,當文件名稱的變化。只在window建議為true # 沒有新日志采集后多長時間關閉文件句柄，默認5分鐘，設置成1分鐘，加快文件句柄關閉 close_inactive: 1m # 傳輸了3h后荏沒有傳輸完成的話就強行關閉文件句柄，這個配置項是解決以上案例問題的key point close_timeout: 3h # 這個配置項也應該配置上，默認值是0表示不清理，不清理的意思是采集過的文件描述在registry文件里永不清理，在運行一段時間后，registry會變大，可能會帶來問題 clean_inactive: 72h # 設置了clean_inactive后就需要設置ignore_older，且要保證ignore_older < clean_inactive ignore_older: 70h # 限制 CPU和內存資源 max_procs: 1 # 限制一個CPU核心,避免過多搶占業務資源 queue.mem.events: 256 # 存儲于內存隊列的事件數，排隊發送 (默認4096) queue.mem.flush.min_events: 128 # 小于 queue.mem.events ,增加此值可提高吞吐量 (默認值2048) # 啟動 filebeat $ systemctl start filebeat十一、部署 curator，定時清理es索引
logstash4 機器操作
# 參考鏈接：https://www.elastic.co/guide/en/elasticsearch/client/curator/current/yum-repository.html # 安裝 curator 服務，以 centos7 為例 $ rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch $ vim /etc/yum.repos.d/elk-curator-5.repo [curator-5] name=CentOS/RHEL 7 repository for Elasticsearch Curator 5.x packages baseurl=https://packages.elastic.co/curator/5/centos/7 gpgcheck=1 gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch enabled=1 $ yum install elasticsearch-curator -y # 創建 curator 配置文件目錄與輸出日志目錄 $ mkdir -p /data/ELKStack/curator/logs $ cd /data/ELKStack/curator $ vim config.yml --- # Remember, leave a key empty if there is no value. None will be a string, # # not a Python "NoneType" client: hosts: ["172.20.166.25", "172.20.166.24", "172.20.166.22", "172.20.166.23", "172.20.166.26"] port: 9200 url_prefix: use_ssl: False certificate: client_cert: client_key: ssl_no_validate: False http_auth: elastic:elastic123 timeout: 150 master_only: False logging: loglevel: INFO logfile: /data/ELKStack/curator/logs/curator.log logformat: default blacklist: ['elasticsearch', 'urllib3'] $ vim action.yml --- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" # # Also remember that all examples have 'disable_action' set to True. If you # want to use this action as a template, be sure to set this to False after # copying it. actions: 1: action: delete_indices description: >- Delete indices older than 30 days. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly. options: ignore_empty_list: True disable_action: False filters: - filtertype: pattern kind: regex value: '^((?!(kibana|json|monitoring|metadata|apm|async|transform|siem|security)).)*$' - filtertype: age source: creation_date direction: older #timestring: '%Yi-%m-%d' unit: days unit_count: 30 2: action: delete_indices description: >- Delete indices older than 15 days. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly. options: ignore_empty_list: True disable_action: False filters: - filtertype: pattern kind: regex value: '^(nginx-).*$' - filtertype: age source: creation_date direction: older #timestring: '%Yi-%m-%d' unit: days unit_count: 15 # 設置定時任務清理es索引 $ crontab -e 0 0 * * * /usr/bin/curator --config /data/ELKStack/curator/config.yml /data/ELKStack/curator/action.yml
到此，關于“如何理解容器部署ELK7.10”的學習就結束了，希望能夠解決大家的疑惑。理論與實踐的搭配能更好的幫助大家學習，快去試試吧！若想繼續學習更多相關知識，請繼續關注億速云網站，小編會繼續努力為大家帶來更多實用的文章！
免責聲明：本站發布的內容（圖片、視頻和文字）以原創、轉載和分享為主，文章觀點不代表本網站立場，如果涉及侵權請聯系站長郵箱：is@yisu.com進行舉報，并提供相關證據，一經查實，將立刻刪除涉嫌侵權內容。
