http://fluentd.org/

日志收集方案



三种收集方案的优缺点：

ELK分布式日志收集系统的搭建
filebeat+logstash+elasticsearch+kibana搭建一个分布式的日志收集系统

1.linux下filebeat下载安装

deb:安装

 curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.6.1-amd64.deb

sudo dpkg -i filebeat-7.6.1-amd64.deb


启动filebeat

 sudo service filebeat start


filebeat配置文件在/etc/filebeat/filebeat.yml中

#=========================== Filebeat inputs =============================

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/folio/okapi/okapi.log
  tag: ["okapi"]
  fields:
    index: 'filebeat-7.6-okapi'
- type: log
  enabled: true
  paths:
    - /var/lib/okapi/mod-circulation/info.log
  tag: ["circulation"]
  fields:
    index: 'filebeat-7.6-circulation'

  multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
  multiline.negate: true
  multiline.match: after
#============================= Filebeat modules ===============================

filebeat.config.modules: 
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 1

#============================== Kibana =====================================

setup.kibana:

#----------------------------- Logstash output --------------------------------
output.logstash:
  hosts: ["172.168.13.208:5044"]

#================================ Processors=====================================

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~



3.linux下logstash下载安装
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

sudo apt-get install apt-transport-https

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

sudo apt-get update && sudo apt-get install logstash


logstash启动

 sudo systemctl start logstash.service


logstash配置文件在/etc/logstash目录下conf.d 中以config为结尾的配置文件


input {

  beats {
           port => 5044
        }
}
filter {
  grok {
    match => { "message" => "(?<timestamp>%{TIMESTAMP_ISO8601})" }
  }
}

output {

  if [fields][index] == "filebeat-7.6-okapi" {
    elasticsearch {
      hosts => ["172.168.13.208:9200"]
      ## index 默认值是"logstash-%{+YYYY.MM.dd}",不允许包含大写字母
      index => "filebeat-okapi"
      }
   }else if [fields][index] == "filebeat-7.6-circulation" {
    elasticsearch {
      hosts => ["172.168.13.208:9200"]
      index => "filebeat-circulation"
      }
    }
}


3.linux下elasticsearch下载安装
 wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

sudo apt-get install apt-transport-https

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

sudo apt-get update && sudo apt-get install elasticsearch


要将Elasticsearch配置为在系统启动时自动启动，请运行以下命令：

sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable elasticsearch.service


Elasticsearch启动停止

sudo systemctl start elasticsearch.service
sudo systemctl stop elasticsearch.service


elasticsearch配置文件在/etc/elasticsearch/elasticsearch.yml


# ------------------------------------ Node ------------------------------------

node.name: master

# ----------------------------------- Paths ------------------------------------

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

# ---------------------------------- Network -----------------------------------

network.host: 0.0.0.0

# --------------------------------- Discovery ----------------------------------

cluster.initial_master_nodes: ["master"]

# ---------------------------------- Various -----------------------------------

http.cors.enabled: true
http.cors.allow-origin: "*"


3.linux下kibana下载安装
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

sudo apt-get install apt-transport-https

echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

sudo apt-get update && sudo apt-get install kibana


要将Kibana配置为在系统启动时自动启动，请运行以下命令：

sudo /bin/systemctl daemon-reload

sudo /bin/systemctl enable kibana.service


Kibana启动停止

sudo systemctl start kibana.service

sudo systemctl stop kibana.service


logstash配置文件在/etc/kibana/kibana.yml


server.basePath: "/kibana"
#i18n.locale: "en"
i18n.locale: "zh-CN"
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://127.0.0.1:9200/"]
kibana.index: ".kibana"

在分布式的环境中，日志收集通常采用 ELK 或其他的消息中间件，但是部署和使用都比较复杂。
本 Chat 将讲述另外一种非常简便的方式，而且不需要任何的学习成本就可以使用，那就是在 Logback 日志框架的基础上做一些小小的改动，就可以收集不同应用的日志，并输出到不同的日志文件，而且性能也不错。
通过本 Chat 大家将学习到：
全面了解以及使用 Java 的日志系统
分布式环境下的日志收集方式以及对应问题
了解 Logback 的源码及对其修改
共同探讨如何使用简单的技术做复杂的事
阅读全文: http://gitbook.cn/gitchat/activity/5c9db58f27289a155f3fe306
一场场看太麻烦？成为 GitChat 会员，畅享 1000+ 场 Chat ！点击查看