Filebeat介绍

Filebeat是一个轻量级的日志采集器，将采集的数据推送到Logstash、ES存储。

Filebeat部署

采用RPM安装

rpm -ivh filebeat-7.9.3-x86_64.rpm

配置文件详解：

# vi /etc/filebeat/filebeat.yml
# 配置不同的输入
- type: log
# 是否启用该输入配置
enabled: false
# 采集的日志文件路径，可以通配
paths:
- /var/log/*.log
# 正则匹配要排除的行，这里以DBG开头的行都过滤掉
#exclude_lines: ['^DBG']
# 正则匹配要采集的行，这里以ERR/WARN开头的行都采集
#include_lines: ['^ERR', '^WARN']
# 排除的文件，默认采集所有
#exclude_files: ['.gz$']
# 添加标签
#tags: ["nginx"]
# 下面fields添加的字段默认是在fields.xxx，可以设置在顶级对象下
# fields_under_root: true
# 自定义添加的字段，一般用于标记日志来源
#fields:
# level: debug
# review: 1

推送到Logstash或ES

#推送到Logstash：
output.logstash:
hosts: ["192.168.0.11:5044"]
#推送到ES：
setup.ilm.enabled: false
setup.template.name: "microservice-product"
setup.template.pattern: "microservice-product-*"
output.elasticsearch:
hosts: ["localhost:9200"]
index: "microservice-product-%{+yyyy.MM.dd}"

配置采集指定日志，修改配置文件(适用于一个服务器上采集多个服务日志）

vim /etc/filebeat/filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/test/product.log
  tags: ["nginx"]
  fields_under_root: true
  fields:
    project: microservice
    app: product


- type: log
  enabled: true
  paths:
    - /var/log/test/gateway.log
  tags: ["nginx"]
  fields_under_root: true
  fields:
    project: microservice
    app: gateway
    
output.logstash:
  hosts: ["192.168.0.11:5044"]

启动filebeat服务

systemctl start filebeat.service 
ps -ef |grep filebeat

logstash配置

修改logstash 配置文件

vim /opt/elk/logstash/conf.d/test.conf

input {
  beats {
   host => "0.0.0.0"
   port => 5044
  }
}
filter {
  if [app] == "product" and [project] == "microservice" {
  mutate {
  add_field => {
  "[@metadata][target_index]" => "microservice-product-%{+YYYY.MM}"
        }
  }
} else if [app] == "gateway" and [project] == "microservice" {
  mutate {
  add_field => {
  "[@metadata][target_index]" => "microservice-gateway-%{+YYYY.MM.dd}"
        }
  }
} else {
  mutate {
  add_field => {
  "[@metadata][target_index]" => "unknown-%{+YYYY}"
          }
        }
  }
}
output {
  elasticsearch {
  hosts => "192.168.0.11:9200"
  index => "%{[@metadata][target_index]}"
  }
}

热加载配置

kill -HUP <logstash pid>

filebeat（部署在要采集日志的机器上）：

模拟数据

mkdir /var/log/test
echo "this is product" > /var/log/test/product.log
echo "this is gateway" > /var/log/test/gateway.log

在kibana页面上查看