增加数据缓冲队列

使用Redis作为消息队列，起到数据缓冲作用，也就是峰值处理能力。

filebeat相当于提供者，logstash相当于消费者。

当filebeat采集到日志，会把日志推送到redis中，redis就有数据了，紧接着logstash从redis拿到数据，经过处理推送到es，es通过kibana把日志展现出来。

1.找任意一台机器安装redis

#安装redis
yum -y install epel-release
yum -y install redis

#修改配置文件
vim /etc/redis.conf 
bind 0.0.0.0 
requirepass 123456

#重启服务
systemctl restart redis.service

# 检查redis是否有数据
redis-cli -a 123456
127.0.0.1:6379> keys *
(empty list or set)

2、filebeat配置输出到redis

vim /etc/filebeat/filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/test/product.log
  tags: ["nginx"]
  fields_under_root: true
  fields:
    project: microservice
    app: product


- type: log
  enabled: true
  paths:
    - /var/log/test/gateway.log
  tags: ["nginx"]
  fields_under_root: true
  fields:
    project: microservice
    app: gateway
    
- type: log
  enabled: true
  paths:
    - /var/log/nginx/access.log
  tags: ["nginx"]
  fields_under_root: true
  fields:
    project: microservice
    app: nginx
  multiline.pattern: '^\s'
  multiline.negate: false
  multiline.match: after

#output.logstash:
#  hosts: ["192.168.0.11:5044"]

output.redis:
  hosts: ["192.168.0.12:6379"]
  password: "123456"
  key: "filebeat"
  db: 0
  datatype: "list"

重启服务

systemctl restart filebeat.service

3、检查redis是否有数据

[root@localhost ~]# redis-cli -a 123456
127.0.0.1:6379> keys *
1) "filebeat"

4 、logstash配置从redis 里 读

vim /opt/elk/logstash/conf.d/test.conf

input {
  redis {
    host => "192.168.0.12"
    port => 6379
    password => "123456"
    key => "filebeat"
    db => 0
    data_type => "list"
  }
}
filter {
  json {
    source => "message"
  }

  if [app] == "product" and [project] == "microservice" {
  mutate {
  add_field => {
  "[@metadata][target_index]" => "microservice-product-%{+YYYY.MM}"
	}
  }
} else if [app] == "gateway" and [project] == "microservice" {
  mutate {
  add_field => {
  "[@metadata][target_index]" => "microservice-gateway-%{+YYYY.MM.dd}"
	}
  }
} else if [app] == "nginx" and [project] == "microservice" {
  mutate {
  add_field => {
  "[@metadata][target_index]" => "microservice-nginx-%{+YYYY.MM.dd}"
	}
  }
} else {
  mutate {
  add_field => {
  "[@metadata][target_index]" => "unknown-%{+YYYY}"
	  }
	}
     }
}
output {
  elasticsearch {
  hosts => "192.168.0.11:9200"
  index => "%{[@metadata][target_index]}"
  }
}

热加载配置

kill -HUP <logstash pid>

5 、验证访问

访问nginx

curl 192.168.0.13

在kibana页面上查看

redis学习：

https://www.runoob.com/redis/redis-pub-sub.html

https://blog.csdn.net/liqingtx/article/details/60330555

未来架构扩容思路

如果日志量每天100G以上，还需要增加更多的服务器支撑。

例如扩容：

• Logstash

• Elasticsearch

其他优化点

• 在预算充足情况下，服务器硬件配置尽量高

• 根据业务，规划好索引

• 不用的索引可以删除或者关闭

例如：

#关闭索引
curl -XPOST "http://127.0.0.1:9200/microservice-gateway-2020.11*/_close?pretty" # 开启用_open 
#删除索引
curl -XDELETE "http://127.0.0.1:9200/microservice-gateway-2020.11*"

kafka学习：

https://www.cnblogs.com/qingyunzong/p/9004509.html

https://www.cnblogs.com/bainianminguo/p/12247158.html

https://blog.csdn.net/u012129558/article/details/80065869