精华内容
下载资源
问答
  • ELK搭建

    2020-09-28 00:59:17
    前言 在前篇文章简单介绍ELK和相关的组件ELK架构介绍,在...ELK搭建 1.ELK架构介绍 2.ElasticSearch搭建 3.Kibana搭建 4.ElasticSearch-head搭建 5.Logstash搭建 6.Kafka搭建 7.Kafka-manager搭建 8.Filebeat搭建 搭建

    前言

    在前篇文章简单介绍ELK和相关的组件ELK架构介绍,在写这篇文章之后本来是不打算搭建ELK的,过了几天吧,公司测试小哥说要看项目日志,而我们后端开发是按模块开发的,所以日志不集中,也没有统一部署到测试服务器上,那么正好找个借口帮公司搭建ELK日志收集系统!

    ELK搭建

    1.ELK架构介绍

    2.ElasticSearch搭建

    3.Kibana搭建

    4.ElasticSearch-head搭建

    5.Logstash搭建

    6.Kafka搭建

    7.Kafka-manager搭建

    8.Filebeat搭建

    搭建好的架构

    在这里插入图片描述

    这里Filebeat是将项目日志从Win/Lin的目录读取并写入Kafka,Kafka其实不一定需要,若是日志打印量特别大的情况可以考虑架设,Logstash则是从Kafka中读取数据存入到ES中,Kafka-Manager负责管理Kafka集群,Head负责ES集群管理,Kibana则读取ES数据负责ES数据可视化操作

    完整链路测试

    1.启动ES
    在这里插入图片描述
    2.启动Kibana
    在这里插入图片描述
    在这里插入图片描述
    3.启动Head
    在这里插入图片描述
    4.启动Kafka
    在这里插入图片描述
    5.启动Kafka-Manager
    在这里插入图片描述

    6…启动Logstash
    这里配置文件为读取Kafka写入ES
    在这里插入图片描述
    7.启动Filebeat
    在这里插入图片描述

    在这里插入图片描述

    展开全文
  • elk搭建

    2020-09-16 10:58:46
    ELK搭建 准备两个linux 环境 关闭防火墙 systemctl stop firewalld setenforce 0 在两个linux 中安装java 上传Java包解压java包 rpm -ivh jdk-8u131-linux-x64_.rpm 验证java java -version 下面对服务端进行...

    ELK搭建

    准备两个linux 环境
    关闭防火墙

    systemctl stop firewalld
    
    setenforce 0
    

    在两个linux 中安装java
    上传Java包解压java包

    rpm -ivh jdk-8u131-linux-x64_.rpm
    

    验证java

    java -version
    

    下面对服务端进行配置
    上传 elesticsearch包对其安装

    yum -y install elasticsearch-6.6.2.rpm
    

    编辑配置文件

    vim /etc/elasticsearch/elasticsearch.yml
    

    自己更改下面的内容在vim里面 全部都是把注释去掉 更改后面的东西
    cluster.name: name
    node.name: node-1
    path.data: /var/lib/elasticsearch
    path.logs: /var/log/elasticsearch
    network.host: 192.168.1.7
    http.port: 9200

    对其启动

    systemctl enable elasticsearch
    
    systemctl start elasticsearch
    

    安装kibana

    yum -y install kibana-6.6.2-x86_64.rpm
    

    编辑配置文件

    vim /etc/kibana/kibana.yml
    

    自己更改下面的内容

    server.port: 5601
    server.host: “192.168.1.7”
    elasticsearch.hosts: [“http://192.168.1.7:9200”]

    开启服务

    systemctl enable kibana
    
    systemctl start kibana
    

    服务端安装完毕 下面开始客户端的安装

    上传logstash对其安装

    yum -y install logstash-6.6.0.rpm
    

    编辑 配置文件

    vim /etc/logstash/conf.d/messages.conf
    
    input {
            file {
                    path => "/var/log/messages"
                    type => "msg-log"
                    start_position => "beginning"
            }
    }
    
    output{
            elasticsearch {
                    hosts => "服务端的ip:9200"
                    index => "msg_log-%{+YYYY.MM.dd}"
            }
    }
    

    开启logstash服务

    systemctl enable logstash
    
    systemctl start logstash
    
    展开全文
  • elk 搭建

    2020-03-18 22:10:19
    1.创建用户 ...2.在根目录下创建目录 elk cd / mkdir elk elk 目录下创建 es cd elk mkdir es 3.更改目录elk的用户权限 chown elsearch:elsearch elk/ -R 4. 下载elk 安装包 下载可以使用迅雷 ...

    1.创建用户

    useradd es

    2.在根目录下创建目录 elk

    cd /

    mkdir elk

    elk 目录下创建 es

    cd elk

    mkdir es

    3.更改目录elk的用户权限

    chown elsearch:elsearch elk/ -R

    4. 下载elk 安装包  下载可以使用迅雷

    检查es 状态

    systemctl status elasticsearch

    安装es的谷歌浏览器插件参考地址

    https://blog.csdn.net/xixiyuguang/article/details/105392140

    https://www.gugeapps.net/webstore/detail/elasticsearch-head/ffmkiejjmecolpfloofpjologoblkegm#download

    5.安装kibana

    下载kibana 7.8.1

    修改 kibana-7.8.1-linux-x86_64/config中的文件kibana.yml

    server.host: "192.168.100.32"

    elasticsearch.hosts: ["http://192.168.100.32:9200"]   连接es

    i18n.locale: "zh-CN" 改成中文

    启动在kibana的bin目录里执行先切换用户

    su es

    ./kibana &

    6.安装logstash

    下载logstash7.8.1 

    解压后进入logstash-7.8.1/config创建文件logstash.conf内容如下

    input {
      file {
        path => "/usr/local/yunshun/logs/service-logistics-account*"
         type => "syslog"     
         start_position => beginning
      }
    }
    filter {}
     
    output {
      elasticsearch {
        hosts =>  ["http://192.168.100.32:9200"]
        action => "index"
        index => "syslog_test-%{+YYYY.MM.dd}"
        #user => "hms"
        #password => "Handmobile"
      }
    }

    启动logstash,后台启动

    ./logstash -f ../config/logstash.conf &

     

     

    https://blog.csdn.net/abtmh02622/article/details/101634560

     

    展开全文
  • ELK 搭建

    2019-06-18 22:05:00
    环境准备两台主机:主机1 elk-master 192.168.93.14主机2 elk-node 192.168.93.19主机hosts192.168.93.14 elk-master 安装elasticsearch和kibana192.168.93.19 elk-node 安装elasticsearch和logstash备注:版本为6.8.0...



    一.环境准备
    两台主机:
    主机1 elk-master 192.168.93.14
    主机2 elk-node 192.168.93.19

    主机hosts
    192.168.93.14 elk-master 安装elasticsearch和kibana
    192.168.93.19 elk-node 安装elasticsearch和logstash

    备注:版本为6.8.0-1

    二.部署
    1.安装jdk
    下载jdk8 地址 https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
    jdk-8u211-linux-x64.tar.gz

    #tar xvf jdk-8u211-linux-x64.tar.gz
    #mv jdk1.8.0_211 /usr/local/jdk1.8
    #vim /etc/profile //添加如下
    export JAVA_HOME=/usr/local/jdk1.8
    export CLASSPATH=.:$JAVA_HOME/lib:$JRE_HOME/lib:$CLASSPATH
    export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH
    export JRE_HOME=$JAVA_HOME/jre

    #source /etc/profile
    #java -version
    java version "1.8.0_211"
    Java(TM) SE Runtime Environment (build 1.8.0_211-b12)
    Java HotSpot(TM) 64-Bit Server VM (build 25.211-b12, mixed mode)

    #ln -s /usr/local/jdk1.8/bin/java /usr/bin/


    2.elasticsearch
    1) 安装
    [root@elk-master ~]# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
    [root@elk-master ~]# vim /etc/yum.repos.d/elasticsearch.repo //添加如下内容
    [elasticsearch-6.x]
    name=Elasticsearch repository for 6.x packages
    baseurl=https://artifacts.elastic.co/packages/6.x/yum
    gpgcheck=1
    gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
    enabled=1
    autorefresh=1
    type=rpm-md
    [root@elk-master ~]# yum install elasticsearch -y

    2)配置
    elasticsearch配置文件主要有两个,一个是/etc/elastcisearch/elasticsearch.yml 主要是配置节点信息,另一个是/etc/sysconfig/elasticsearch 配置elasticsearch本身的信息.
    [root@elk-master ~]# vim /etc/elasticsearch/elasticsearch.yml
    cluster.name: lt-elk
    node.name: elk-master
    node.master: true
    node.data: false
    network.host: 0.0.0.0
    http.port: 9200
    discovery.zen.ping.unicast.hosts: ["192.168.93.14", "192.168.93.19"]
    其他保持默认即可.

    数据节点node配置如下

    cluster.name: lt-elk
    node.name: elk-node
    node.master: false
    node.data: true
    network.host: 0.0.0.0
    http.port: 9200
    discovery.zen.ping.unicast.hosts: ["192.168.93.14", "192.168.93.19"]

    启动服务
    [root@elk-master ~]# systemctl enable elasticsearch.service
    Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.
    [root@elk-master ~]# systemctl start elasticsearch.service

    [root@elk-node ~]# systemctl enable elasticsearch.service
    Created symlink from /etc/systemd/system/multi-user.target.wants/elasticsearch.service to /usr/lib/systemd/system/elasticsearch.service.
    [root@elk-node ~]# systemctl start elasticsearch.service


    查看下端口
    [root@elk-master ~]# netstat -lntp | grep java
    tcp6 0 0 :::9200 :::* LISTEN 21817/java
    tcp6 0 0 :::9300 :::* LISTEN 21817/java
    // 9200 数据传输端口, 9300集群通信端口

    测试下:
    [root@elk-master ~]# curl "http://192.168.93.14:9200"
    {
    "name" : "elk-master",
    "cluster_name" : "lt-elk",
    "cluster_uuid" : "PMzMag2vQsanNWbhzu3MZA",
    "version" : {
    "number" : "6.8.0",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "65b6179",
    "build_date" : "2019-05-15T20:06:13.172855Z",
    "build_snapshot" : false,
    "lucene_version" : "7.7.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
    },
    "tagline" : "You Know, for Search"
    }
    查看下集群
    [root@elk-master ~]# curl "http://192.168.93.14:9200/_cluster/health?pretty"
    {
    "cluster_name" : "lt-elk",
    "status" : "green",
    "timed_out" : false,
    "number_of_nodes" : 2,
    "number_of_data_nodes" : 1,
    "active_primary_shards" : 0,
    "active_shards" : 0,
    "relocating_shards" : 0,
    "initializing_shards" : 0,
    "unassigned_shards" : 0,
    "delayed_unassigned_shards" : 0,
    "number_of_pending_tasks" : 0,
    "number_of_in_flight_fetch" : 0,
    "task_max_waiting_in_queue_millis" : 0,
    "active_shards_percent_as_number" : 100.0
    }

    3.kibana
    安装kibana
    [root@elk-master ~]# yum install kibana -y
    配置
    [root@elk-master ~]# vim /etc/kibana/kibana.yml
    server.port: 5601
    server.host: "192.168.93.14"
    elasticsearch.url: "http://192.168.93.14:9200"
    kibana.index: ".kibana"
    logging.dest: /var/log/kibana.log
    启动
    [root@elk-master ~]# systemctl enable kibana.service
    Created symlink from /etc/systemd/system/multi-user.target.wants/kibana.service to /etc/systemd/system/kibana.service.
    [root@elk-master ~]# systemctl start kibana.service

    4.安装logstash
    [root@elk-node ~]# yum install logstash -y
    [root@elk-node ~]# vim /etc/logstash/conf.d/syslog.conf
    input{
    syslog{
    tyep => "system-log"
    port => 10514
    }
    }
    output{
    stdout{
    codec => rubydebug
    }
    }
    [root@elk-node logstash]# vim /etc/logstash/logstash.yml
    http.host : 192.168.93.19
    [root@elk-node logstash]# systemctl restart logstash
    验证下配置
    [root@elk-node ~]# cd /usr/share/logstash/bin
    [root@elk-node bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit
    Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
    [2019-06-07T11:59:55,496][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
    Configuration OK
    [2019-06-07T12:00:00,480][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
    [root@elk-node bin]#
    // 显示OK,表示验证

    配置rsyslog
    [root@elk-node logstash]# vim /etc/rsyslog.conf
    *.* @192.168.93.19:10514
    [root@elk-node logstash]# systemctl restart rsyslog

    验证配置
    [root@elk-node bin]# cd /usr/share/logstash/bin/
    [root@elk-node bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf
    另找一台机器ssh登录到note上,
    观察输出,类似如下
    {
    "facility_label" => "syslogd",
    "message" => "action 'action 8' resumed (module 'builtin:omfwd') [v8.24.0-34.el7 try http://www.rsyslog.com/e/2359 ]\n",
    "@timestamp" => 2019-06-07T14:21:18.000Z,
    "@version" => "1",
    "timestamp" => "Jun 7 22:21:18",
    "logsource" => "elk-node",
    "host" => "192.168.93.19",
    "facility" => 5,
    "type" => "system-log",
    "program" => "rsyslogd",
    "priority" => 46,
    "severity_label" => "Informational",
    "severity" => 6
    }
    {
    "facility_label" => "security/authorization",
    "message" => "New session 3 of user root.\n",
    "@timestamp" => 2019-06-07T14:21:18.000Z,
    "@version" => "1",
    "timestamp" => "Jun 7 22:21:18",
    "logsource" => "elk-node",
    "host" => "192.168.93.19",
    "facility" => 4,
    "type" => "system-log",
    "program" => "systemd-logind",
    "priority" => 38,
    "severity_label" => "Informational",
    "severity" => 6
    }
    {
    "facility_label" => "system",
    "message" => "Started Session 3 of user root.\n",
    "@timestamp" => 2019-06-07T14:21:18.000Z,
    "@version" => "1",
    "timestamp" => "Jun 7 22:21:18",
    "logsource" => "elk-node",
    "host" => "192.168.93.19",
    "facility" => 3,
    "type" => "system-log",
    "program" => "systemd",
    "priority" => 30,
    "severity_label" => "Informational",
    "severity" => 6
    }
    {
    "facility_label" => "security/authorization",
    "message" => "pam_unix(sshd:session): session opened for user root by (uid=0)\n",
    "@timestamp" => 2019-06-07T14:21:18.000Z,
    "@version" => "1",
    "timestamp" => "Jun 7 22:21:18",
    "logsource" => "elk-node",
    "host" => "192.168.93.19",
    "pid" => "12518",
    "facility" => 10,
    "type" => "system-log",
    "program" => "sshd",
    "priority" => 86,
    "severity_label" => "Informational",
    "severity" => 6
    }
    {
    "facility_label" => "syslogd",
    "message" => "action 'action 8' resumed (module 'builtin:omfwd') [v8.24.0-34.el7 try http://www.rsyslog.com/e/2359 ]\n",
    "@timestamp" => 2019-06-07T14:21:18.000Z,
    "@version" => "1",
    "timestamp" => "Jun 7 22:21:18",
    "logsource" => "elk-node",
    "host" => "192.168.93.19",
    "facility" => 5,
    "type" => "system-log",
    "program" => "rsyslogd",
    "priority" => 46,
    "severity_label" => "Informational",
    "severity" => 6
    }
    {
    "facility_label" => "security/authorization",
    "message" => "Accepted password for root from 192.168.93.14 port 42568 ssh2\n",
    "@timestamp" => 2019-06-07T14:21:18.000Z,
    "@version" => "1",
    "timestamp" => "Jun 7 22:21:18",
    "logsource" => "elk-node",
    "host" => "192.168.93.19",
    "pid" => "12518",
    "facility" => 10,
    "type" => "system-log",
    "program" => "sshd",
    "priority" => 86,
    "severity_label" => "Informational",
    "severity" => 6
    }
    证明配置成功
    修改配置文件,让收集的日志信息输出到master服务器中,而不是当前终端:
    [root@elk-node bin]# vim /etc/logstash/conf.d/syslog.conf
    input{
    syslog{
    type => "system-log"
    port => 10514
    }
    }
    output{
    elasticsearch{
    hosts => ["192.168.93.14:9200"]
    index => "system-log-%{+YYYY.MM}"
    }
    }

    检查配置文件,并启动
    [root@elk-node bin]# ./logstash --path.settings /etc/logstash/ -f /etc/logstash/conf.d/syslog.conf --config.test_and_exit
    Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
    [2019-06-07T22:30:17,139][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
    Configuration OK
    [2019-06-07T22:30:21,755][INFO ][logstash.runner ] Using config.test_and_exit mode. Config Validation Result: OK. Exiting Logstash
    [root@elk-node bin]# systemctl start logstash

    这里要修改下权限,不然虽然能成功启动,但监听不到端口
    //chown logstash /var/log/logstash/logstash-plain.log
    //chown -R logstash /var/lib/logstash/

    检查下端口,如下,启动成功
    [root@elk-node bin]# netstat -lntp | grep 9600
    tcp6 0 0 192.168.93.19:9600 :::* LISTEN 12587/java
    [root@elk-node bin]#
    [root@elk-node bin]# netstat -lntp | grep 10514
    tcp6 0 0 :::10514 :::* LISTEN 12587/java

    5.配置kibana索引
    建立索引index




    完成后,点击Discover




    使用beats采集日志
    [root@elk-node2 ~]# wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.7.0-x86_64.rpm
    [root@elk-node2 ~]# rpm -ivh filebeat-6.7.0-x86_64.rpm
    准备中... ################################# [100%]
    正在升级/安装...
    1:filebeat-6.7.0-1 ################################# [100%]

    修改配置文件
    [root@elk-node2 ~]# vim /etc/filebeat/filebeat.yml
    filebeat.inputs:
    - type: log
    paths:
    - /var/log/messages
    output.elasticsearch:
    hosts: ["192.168.93.14:9200"]

    启动filebeat
    [root@elk-node2 ~]# systemctl start filebeat

    服务端检查
    [root@elk-master ~]# curl "192.168.93.14:9200/_cat/indices?v"
    health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
    green open nginx-log-2019.06.07 q5t0CEG8SPajfnuaqQrTYw 5 1 7 0 92.2kb 46.1kb
    green open system-log-2019.06 hG2UzU9AQ2SpQS6tat-gNQ 5 1 27 0 347.2kb 173.6kb
    green open .kibana_task_manager bkj5j4MmQf6Kf5wb3_UeIA 1 1 2 0 25.1kb 12.5kb
    green open .kibana_1 v5BpxX-JTWqJP3OMyvoClQ 1 1 6 0 57.5kb 28.7kb
    green open filebeat-6.8.0-2019.06.07 MjFxU6A1Q6OhKrU0mNGATw 3 1 0 0 80.6kb 460b

    看到出现filebeat-6.8.0-2019.06.07 索引文件表示正常.






    转载于:https://www.cnblogs.com/luckyleaf/p/11048113.html

    展开全文
  • 昨天在群里面和一起大神讨论一些关于日志系统搭建,所碰到 的一些问题吧导致没有搭建成功,幸好我也是最近研究了一些用ELK来搭建一些日志系统,这个日志系统环境搭建是一个比较初级的,使用ELK搭建的,我们用.Net log4...
  • ELK 是 Elasticsearch、 Logstash 和 Kibana 这三个软件集合的简称,ELK 搭建实时日志分析平台视频教程,本次课程,青云QingCloud 工程师彭科、东升将会从技术角度来分享如果搭建 ELK 实时日志分析平台。
  • elk搭建记录

    2018-11-08 14:20:15
    ELK 搭建记录 为什么用到ELK:一般我们需要进行日志分析场景:直接在日志文件中 grep、awk 就可以获得自己想要的信息。但在规模较大的场景中,此方法效率低下,面临问题包括日志量太大如何归档、文本搜索太慢怎么办...
  • ELK搭建及配置

    2020-06-27 21:12:42
    官网网站 “ELK”是三个开源项目的首字母缩写,这三个项目分别是:Elasticsearch、Logstash 和 Kibana。Elasticsearch 是一个搜索和分析引擎。Logstash 是服务器端数据...一、ELK搭建 EKL快速搭建教程 一、ELK配置 ...
  • 基于docker的ELK搭建

    2021-01-21 10:25:10
    ELK搭建 基于docker (elasticsearch + logstash + kafka+ filebeat + kibana) 链接: link.
  • ELK搭建文档

    2018-10-18 17:17:01
    Linux环境下面搭建ELK步骤,,, 进行日志抽取,备份等
  • ELK搭建和使用

    千次阅读 2018-08-03 16:23:59
    这两天使用公司的一台cent os把ELK搭建了起来,做个记录。 1.在官网down各种安装包-elasticSearch, logstash, kibna https://www.elastic.co/downloads   2.解压并安装各种组件 2-1.elasticSearch 安装目录为...

空空如也

空空如也

1 2 3 4 5 ... 20
收藏数 7,282
精华内容 2,912
关键字:

elk搭建