精华内容
下载资源
问答
  • prometheus+grafana 监控hadoop、yarn

    万次阅读 2018-08-27 10:08:17
    主要是以jmx_exporter、prometheus为主导进行对hadoop的metrics进行收集,通过grafana进行展示、预警。 1、安装jmx_exporter以及配置文件 1、通过...

    主要是以jmx_exporter、prometheus为主导进行对hadoop的metrics进行收集,通过grafana进行展示、预警。

    1、安装jmx_exporter以及配置文件

    1、通过

    阿里云镜像里下载:https://maven.aliyun.com/mvn/search

    搜索下载 jmx_prometheus_javaagent

    2、创建配置文件:xxxx.yml(根据用途不同可以叫不同的名字,在启动时指定配置)

    startDelaySeconds: 0
    hostPort: localhost:1234  #1234为想设置的jmx端口(可设置为未被占用的端口)
    ssl: false
    lowercaseOutputName: false
    lowercaseOutputLabelNames: false

    startDelaySeconds: 0
    hostPort: localhost:1235  #1235为想设置的jmx端口(可设置为未被占用的端口)
    ssl: false
    lowercaseOutputName: false
    lowercaseOutputLabelNames: false

    3、将以上3个文件放到 /usr/local/prometheus_jmx_export_0.3.1 

     并执行 chown -R hadoop:root /usr/local/prometheus_jmx_export_0.3.1 

    4、修改$HADOOP_HOME/etc/hadoop/hadoop-env.sh (提示:端口1234(1235)要与之前设置的jmx端口保持一致)

    ​export HADOOP_NAMENODE_JMX_OPTS="-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false   -Dcom.sun.management.jmxremote.port=1234 -javaagent:/usr/local/prometheus_jmx_export_0.3.1/jmx_prometheus_javaagent-0.3.1.jar=9222:/usr/local/prometheus_jmx_export_0.3.1/namenode.yaml"
    export HADOOP_DATANODE_JMX_OPTS="-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false   -Dcom.sun.management.jmxremote.port=1235 -javaagent:/usr/local/prometheus_jmx_export_0.3.1/jmx_prometheus_javaagent-0.3.1.jar=9322:/usr/local/prometheus_jmx_export_0.3.1/datanode.yaml"

    5、修改$HADOOP_HOME/bin/hdfs 修改 namenode、datanode启动参数如下

    if [ "$COMMAND" = "namenode" ] ; then
      CLASS='org.apache.hadoop.hdfs.server.namenode.NameNode'
      HADOOP_OPTS="$HADOOP_OPTS $HADOOP_NAMENODE_JMX_OPTS $HADOOP_NAMENODE_OPTS"
    .......
    elif [ "$COMMAND" = "datanode" ] ; then
      CLASS='org.apache.hadoop.hdfs.server.datanode.DataNode'
      HADOOP_OPTS="$HADOOP_OPTS $HADOOP_DATANODE_JMX_OPTS"
      if [ "$starting_secure_dn" = "true" ]; then
        HADOOP_OPTS="$HADOOP_OPTS -jvm server $HADOOP_DATANODE_OPTS"
      else
        HADOOP_OPTS="$HADOOP_OPTS -server $HADOOP_DATANODE_OPTS"
      fi

    6、重启 hadoop dfs集群,namenode机器访问 http://xxx:9222/metrics   datanode机器访问 http://xxx:9322/metrics 即可获得metrics信息

    2、安装Prometheus以及配置文件

    1、https://github.com/prometheus/prometheus/releases/download/v2.3.2/prometheus-2.3.2.linux-amd64.tar.gz 下载 prometheus linux版本到 /usr/local/ 下,

    解压 并执行  chown -R hadoop:root prometheus-2.3.2.linux-amd64.tar.gz

    2、修改配置文件 prometheus.yml(注意:以下代码只是在测试上执行的,对多少台机器进行监控就需要配置多少个job,配置文件注意缩进)

    - job_name: hadoop-namenode
      static_configs:
      - targets: ['binamenode01:9222']
    - job_name: hadoop-datanode
      static_configs:
      - targets: ['bidatanode01:9322']

    3、用户hadoop 启动 prometheus 

    cd /usr/local/prometheus-2.3.2.linux-amd64
    ./startPromethous.sh

    4、http://master:9090/targets 查看是否添加成功(prometheus 执行默认端口9090)

    通过点击http://bidatanode01:9222/metrics可以看到metrics数据

    3、安装grafana以及配置文件

    1、下载grafana,解压

    cd /usr/local
    wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana-5.2.2.linux-amd64.tar.gz 
    tar -zxvf grafana-5.2.2.linux-amd64.tar.gz 
    chown -R hadoop:root grafana-5.2.2.linux-amd64

    2、用户 hadoop 启动grafana

    cd /usr/local/grafana-5.2.2/bin/
    nohup ./grafana-server start &

    3、启动后,即可通过http://master:3000/ 来访问了(默认账号密码是admin/admin,grafana默认端口3000)

    4、关联Grafana和Prometheus

    点击Data Sources 

    点击Add data source,填写数据保存

    4、配置grafana预警邮件发送

    1、检查mailx是否安装 

    rpm -qa | grep mailx

    如果检查没有安装 则需要用一下命令安装

    yum -y install mailx

    2、编辑 /usr/local/grafana-5.2.2/conf/defaults.ini

    ...
     
    #################################### SMTP / Emailing #####################
    [smtp]
    enabled = true
    host = smtp.xx.com:587
    user = sys_sender@xx.com
    # 如果密码中包含#或者; 密码需要用三个双引号包围  例如:"""QWER123;4!@#$"""
    password = xxxxxxx #此为邮箱密码
    cert_file =
    key_file =
    skip_verify = true
    from_address = sys_sender@xx.com
    from_name = sys_sender
    ehlo_identity =
    [emails]
    welcome_email_on_sign_up = false
    templates_pattern = emails/*.html
     
    ...
     
    #################################### Alerting ############################
    [alerting]
    # Disable alerting engine & UI features
    enabled = true
    # Makes it possible to turn off alert rule execution but alerting UI is visible
    execute_alerts = true

     
     

    3、测试 grafana 邮件发送

    编辑发送邮件,点击测试 OK

    =======================================================================================

    2018-08-27追加:

    对于yarn的接入也是大同小异 

    对于${HADOOP_HOME}/etc/hadoop/yarn-env.sh 添加 metrics 开启信息并制定端口

    export YARN_JMX_OPTS="-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false   -Dcom.sun.management.jmxremote.port=1236 -javaagent:/usr/local/prometheus_jmx_export_0.3.1/jmx_prometheus_javaagent-0.3.1.jar=9422:/usr/local/prometheus_jmx_export_0.3.1/yarn.yaml"

    然后修改${HADOOP_HOME}/bin/yarn 

    elif [ "$COMMAND" = "resourcemanager" ] ; then
      CLASSPATH=${CLASSPATH}:$YARN_CONF_DIR/rm-config/log4j.properties
      CLASS='org.apache.hadoop.yarn.server.resourcemanager.ResourceManager'
      YARN_OPTS="$YARN_OPTS $YARN_JMX_OPTS $YARN_RESOURCEMANAGER_OPTS"
      if [ "$YARN_RESOURCEMANAGER_HEAPSIZE" != "" ]; then
        JAVA_HEAP_MAX="-Xmx""$YARN_RESOURCEMANAGER_HEAPSIZE""m"
      fi
    ......
    elif [ "$COMMAND" = "nodemanager" ] ; then
      CLASSPATH=${CLASSPATH}:$YARN_CONF_DIR/nm-config/log4j.properties
      CLASS='org.apache.hadoop.yarn.server.nodemanager.NodeManager'
      YARN_OPTS="$YARN_OPTS $YARN_JMX_OPTS -server $YARN_NODEMANAGER_OPTS"
      if [ "$YARN_NODEMANAGER_HEAPSIZE" != "" ]; then
        JAVA_HEAP_MAX="-Xmx""$YARN_NODEMANAGER_HEAPSIZE""m"
      fi

     

    重启 yarn

    添加 prometheus_jmx_export下的yarn.yaml文件

    修改配置文件 prometheus.yml 

    - job_name: yarn
      static_configs:
      - targets: ['binamenode01:9422']
    

    重启 prometheus,即可

    =======================================================================================

    2018-08-29 添加

    对于hbase的监控:

    修改配置文件 $HBASE_HOME/bin/hbase

    在文件

    # figure out which class to run

    位置添加:

    #prometheus jmx export start

    HBASE_JMX_OPTS="-Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false   -Dcom.sun.management.jmxremote.port=1237 -javaagent:/usr/local/prometheus_jmx_export_0.3.1/jmx_prometheus_javaagent-0.3.1.jar=9522:/usr/local/prometheus_jmx_export_0.3.1/hbase.yaml"

    #prometheus jmx export end
    ......
    elif [ "$COMMAND" = "master" ] ; then
      CLASS='org.apache.hadoop.hbase.master.HMaster'
      if [ "$1" != "stop" ] && [ "$1" != "clear" ] ; then
        HBASE_OPTS="$HBASE_OPTS $HBASE_JMX_OPTS $HBASE_MASTER_OPTS"
      fi
    elif [ "$COMMAND" = "regionserver" ] ; then
      CLASS='org.apache.hadoop.hbase.regionserver.HRegionServer'
      if [ "$1" != "stop" ] ; then
        HBASE_OPTS="$HBASE_OPTS $HBASE_JMX_OPTS $HBASE_REGIONSERVER_OPTS"
      fi

    重启 hbase

    添加 prometheus_jmx_export下的hbase.yaml文件

    修改配置文件 prometheus.yml 

    - job_name: hbase
      static_configs:
      - targets: ['binamenode01:9522']
    

    重启 prometheus,即可

    =======================================================================================

    2018-09-01 添加

    kylin 监控添加 

    修改 kylin.sh文件,其启动项 添加 配置

    -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false   -Dcom.sun.management.jmxremote.port=1239 -javaagent:/usr/local/prometheus_jmx_export_0.3.1/jmx_prometheus_javaagent-0.3.1.jar=9722:/usr/local/prometheus_jmx_export_0.3.1/kylin.yaml \
    

    重启 kylin

    添加 prometheus_jmx_export下的kylin.yaml文件

    修改配置文件 prometheus.yml 

    - job_name: hbase
      static_configs:
      - targets: ['binamenode01:9722']
    

    重启 prometheus,即可

    =======================================================================================

    2018-09-01 添加

    hive 监控添加

    修改文件 

    ${HIVE_HOME}/conf/hive-env.sh 添加如下代码

    if [ "$SERVICE" = "hiveserver2" ] ; then
            HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=1240 -javaagent:/usr/local/prometheus_jmx_export_0.3.1/jmx_prometheus_javaagent-0.3.1.jar=9822:/usr/local/prometheus_jmx_export_0.3.1/hive_hiveserver2.yaml"
    fi
    if [ "$SERVICE" = "metastore" ] ; then
            HADOOP_CLIENT_OPTS="$HADOOP_CLIENT_OPTS -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.port=1241 -javaagent:/usr/local/prometheus_jmx_export_0.3.1/jmx_prometheus_javaagent-0.3.1.jar=9922:/usr/local/prometheus_jmx_export_0.3.1/hive_metastore.yaml"
    fi

    添加 prometheus_jmx_export下的hive_metastore.yaml、hive_hiveserver2.yaml文件

     重启 hive的 metastore hiveserver2

    修改配置文件 prometheus.yml  

    - job_name: hbase
      static_configs:
      - targets: ['binamenode01:9822','binamenode01:9922']
    

     重启 prometheus,即可

    引用

    git: https://github.com/prometheus/jmx_exporter

    官网:https://prometheus.io/docs/instrumenting/exporters/#other-monitoring-systems

    展开全文
  • 1.简介 Flink支持多种监控指标的汇报reporter,例如jmx、slf4j、Prometheus、...但Prometheus自动发现Flink on YARN作业的提交、结束以及自动拉取数据是比较困难的。PushGateway作为一个中转组件,通过配置Flink on

    1.简介
    Flink支持多种监控指标的汇报reporter,例如jmx、slf4j、Prometheus、InfluxDB等。
    Grafana+Prometheus是当前比较流行的监控可视化解决方案。如下图为Prometheus及相关组件组成的生态系统。
    在这里插入图片描述
    2.安装配置
    Prometheus是采用拉模式的方式,从exporter拉取监控的数据。但Prometheus自动发现Flink on YARN作业的提交、结束以及自动拉取数据是比较困难的。PushGateway作为一个中转组件,通过配置Flink on YARN作业将监控指标推到PushGateway,Prometheus再从PushGateway拉取数据。
    (1)首先拷贝监控用的jar包Flink的lib目录下
    将flink的opt目录下面的flink-metrics-prometheus-1.7.2.jar包拷贝到flink的lib下:

    cd /opt/flink-1.7.2
    cp opt/flink-metrics-prometheus-1.7.2.jar lib
    

    (2)Prometheus和PushGateway组件下载地址为:

    https://github.com/prometheus/prometheus/releases/download/v2.12.0/prometheus-2.12.0.linux-amd64.tar.gz
    https://github.com/prometheus/pushgateway/releases/download/v0.9.1/pushgateway-0.9.1.linux-amd64.tar.gz
    

    (3)解压文件,并修改
    Prometheus的配置文件prometheus.yml添加配置如下图;
    Prometheus的默认端口是9090,PushGateway的默认端口是9091。

      - job_name: 'flink'
        file_sd_configs:
         - files:
           - configs/flink.json
    

    configs/flink.json的配置如下图:

    [
     {
      "targets": ["172.30.11.15:9091"]
     }
    ]
    

    (4)启动PushGateway和Prometheus。

    nohup ./pushgateway --web.listen-address :9091 > /var/log/pushgateway.log 2>&1 &
    nohup ./prometheus --config.file=prometheus.yml > /var/log/prometheus.log 2>&1 &
    

    若Prometheus为生产环境热启动,只需要在服务器上执行如下命令即可:

    curl -X POST http://172.30.11.15:9090/-/reload
    

    (5)编辑flink-conf.yaml,添加Flink与PushGateway集成的参数

    metrics.reporter.promgateway.class: org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter
    # 这里写PushGateway的主机名与端口号
    metrics.reporter.promgateway.host: 172.30.11.15
    metrics.reporter.promgateway.port: 9091
    # Flink metric在前端展示的标签(前缀)与随机后缀
    metrics.reporter.promgateway.jobName: flink-metrics-ppg
    metrics.reporter.promgateway.randomJobNameSuffix: true
    metrics.reporter.promgateway.deleteOnShutdown: false
    

    (6)配置grafana
    Grafana默认端口:3000,点击Create your first data source添加Prometheus数据源。如下图所示:
    在这里插入图片描述
    (7)启动flink
    (8)添加监控图表
    点击New Dashboard->Add Query按钮,即可看到Flink下的监控指标。
    在这里插入图片描述
    选好监控指标,就可以从图表中看到监控数据的折线了。

    展开全文
  • python脚本实现,访问官方提供的地址,获取队列资源使用指标,并将指标进行json解析,之后再转换为prometheus认识的数据格式,暴露于端口下 #coding=utf-8 ''' 通过访问官方提供的yarn restful api界面,获取yarn资源...

    python脚本实现,访问官方提供的地址,获取队列资源使用指标,并将指标进行json解析,之后再转换为prometheus认识的数据格式,暴露于端口下
    #coding=utf-8

    '''
    通过访问官方提供的yarn restful api界面,获取yarn资源指标
    分析json取的所需指标
    以pormetheus的数据格式将指标暴露在指定端口下
    
    入参:yarn的ip:port export暴露的机器IP(建议为本机IP) export暴露的端口号
    '''
    
    import prometheus_client
    from prometheus_client import Gauge
    from prometheus_client.core import CollectorRegistry
    from flask import Response, Flask
    
    import json
    import urllib.request
    import sys
    
    
    hjson = json.loads('{}')
    
    app = Flask(__name__)
    
    REGISTRY = CollectorRegistry(auto_describe=False)
    
    def getMetric(ipport):
    
        html = urllib.request.urlopen(r'http://'+ipport+'/ws/v1/cluster/scheduler')
    
        data = html.read().decode("utf-8")
    
        global hjson
        hjson = json.loads(data)
        queuelist = []
        coresList = []
        memList = []
    
    
        for i in range(0,len(hjson['scheduler']['schedulerInfo']['rootQueue']['childQueues']['queue'])):
            queuename = hjson['scheduler']['schedulerInfo']['rootQueue']['childQueues']['queue'][i]['queueName']
            usevCores = hjson['scheduler']['schedulerInfo']['rootQueue']['childQueues']['queue'][i]['usedResources']['vCores']
            usermemory = hjson['scheduler']['schedulerInfo']['rootQueue']['childQueues']['queue'][i]['usedResources']['memory']
            maxvCores = hjson['scheduler']['schedulerInfo']['rootQueue']['childQueues']['queue'][i]['maxResources']['vCores']
            maxmemory = hjson['scheduler']['schedulerInfo']['rootQueue']['childQueues']['queue'][i]['maxResources']['memory']
            usermemory_percentage = 0.0000
            usevCores_percentage = 0.0000
            if(maxvCores != 0):
                usevCores_percentage = round(usevCores/maxvCores,4)
            if(maxmemory != 0):
                usermemory_percentage = round(usermemory/maxmemory,4)
            queuelist.append(queuename.split(".")[1])
            coresList.append(float(usevCores_percentage))
            memList.append(float(usermemory_percentage))
        
        usememory_percentage_temp = Gauge(
            name='vcores_percentage',
            documentation= 'vcores_percentage',
            labelnames=['queue_name'],
            registry=REGISTRY)
        usevCores_percentage_temp = Gauge(
            'memory_percentage',
            'memory_percentage',
            labelnames=['queue_name'],
            registry=REGISTRY)
    
        for i in range(0,len(hjson['scheduler']['schedulerInfo']['rootQueue']['childQueues']['queue'])):
            usememory_percentage_temp.labels(queue_name=str(queuelist[i])).set(float(memList[i]))
            usevCores_percentage_temp.labels(queue_name=str(queuelist[i])).set(float(coresList[i]))
    
    
    
    
    
    @app.route('/metrics')
    def r_value():
        return Response(prometheus_client.generate_latest(REGISTRY),
                        mimetype="text/plain")
    
    
    @app.route('/')
    def index():
        return "namenode jmx metric address: ip:port/metrics"
    
    
    
    if __name__ == "__main__":
    	#yarn的ip和端口
        # yarn_ipport = str(sys.argv[1])
        #展示的ip,建议本机ip
        # showIp = str(sys.argv[2])
        #展示的端口
        # showPort = int(sys.argv[3])
    
    	#yarn的ip和端口
        yarn_ipport = "127.0.0.1:8001"
        #展示的ip,建议本机ipx
        showIp = "localhost"
        #展示的端口
        showPort = 8000
        getMetric(yarn_ipport)
    
        app.run(host=showIp,port=showPort,debug=True)
    

    效果图如下

    在这里插入图片描述

    展开全文
  • Flink支持多种汇报监控指标(metrics)的reporter,如JMX、SLF4J、InfluxDB、Prometheus等。 Prometheus+Grafana是目前相当流行的监控+可视化一体方案,容易上手。下图示出Prometheus及其周边组件组成的生态系统。 ...

    实时作业要保证7 x 24运行,除了要在业务逻辑和编码上下功夫之外,好的监控系统也是必不可少的。Flink支持多种汇报监控指标(metrics)的reporter,如JMX、SLF4J、InfluxDB、Prometheus等。

    Prometheus+Grafana是目前相当流行的监控+可视化一体方案,容易上手。下图示出Prometheus及其周边组件组成的生态系统。

    195230-5324926d0cb914fc.png

    本文简述整个配置流程。

    Prometheus+PushGateway安装配置

    为什么还要使用PushGateway呢?从上面的图可以发现,Prometheus在正常情况下是采用拉模式从产生metric的作业或者exporter(比如专门监控主机的NodeExporter)拉取监控数据。但是我们要监控的是Flink on YARN作业,想要让Prometheus自动发现作业的提交、结束以及自动拉取数据显然是比较困难的。PushGateway就是一个中转组件,通过配置Flink on YARN作业将metric推到PushGateway,Prometheus再从PushGateway拉取就可以了。

    首先将与Prometheus监控相关的JAR包拷贝到Flink的lib目录下。

    cd /opt/flink-1.9.0
    cp opt/flink-metrics-prometheus-1.9.0.jar lib
    

    来到Prometheus官网的下载页面,分别下载本体以及PushGateway组件。也可以直接wget到服务器上。

    wget -c https://github.com/prometheus/prometheus/releases/download/v2.12.0/prometheus-2.12.0.linux-amd64.tar.gz
    wget -c https://github.com/prometheus/pushgateway/releases/download/v0.9.1/pushgateway-0.9.1.linux-amd64.tar.gz
    

    解压之后,编辑Prometheus的配置文件prometheus.yml。主要是修改监控间隔,以及添加PushGateway的监控配置。Prometheus的默认端口是9090,PushGateway的是9091。

    global:
      scrape_interval:     60s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
      evaluation_interval: 60s # Evaluate rules every 15 seconds. The default is every 1 minute.
      # scrape_timeout is set to the global default (10s).
    
    # Alertmanager configuration
    alerting:
      alertmanagers:
      - static_configs:
        - targets:
          # - alertmanager:9093
    
    # A scrape configuration containing exactly one endpoint to scrape:
    # Here it's Prometheus itself.
    scrape_configs:
      # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
      - job_name: 'prometheus'
        static_configs:
        - targets: ['localhost:9090']
          labels:
            instance: prometheus
    
      - job_name: 'pushgateway'
        static_configs:
        - targets: ['my-flink002:9091']
          labels:
            instance: pushgateway
    

    然后用nohup的方式启动PushGateway和Prometheus。

    nohup ./pushgateway --web.listen-address :9091 > /var/log/pushgateway.log 2>&1 &
    
    nohup ./prometheus --config.file=prometheus.yml > /var/log/prometheus.log 2>&1 &
    

    浏览器访问Prometheus所在节点的9090端口,查看Targets一项。如果两个项目都为UP状态,就是配置好了。

    195230-b6e4ea2d39c5138d.png

    还得记得编辑flink-conf.yaml,在其中加上Flink与PushGateway集成的参数。

    metrics.reporter.promgateway.class: org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporter
    # 这里写PushGateway的主机名与端口号
    metrics.reporter.promgateway.host: my-flink002
    metrics.reporter.promgateway.port: 9091
    # Flink metric在前端展示的标签(前缀)与随机后缀
    metrics.reporter.promgateway.jobName: flink-metrics-ppg
    metrics.reporter.promgateway.randomJobNameSuffix: true
    metrics.reporter.promgateway.deleteOnShutdown: false
    

    Grafana安装配置

    从Grafana官网直接下载二进制包非常慢,所以要借助清华的镜像源。在/etc/yum.repos.d目录下新建grafana.repo,写入如下内容。

    [grafana]
    name=grafana
    baseurl=https://mirrors.tuna.tsinghua.edu.cn/grafana/yum/rpm
    repo_gpgcheck=0
    enabled=1
    gpgcheck=0
    

    然后用yum安装。

    yum makecache
    yum -y install grafana
    

    Grafana的配置文件位于/etc/grafana/grafana.ini,但是我们可以不用改,直接启动服务。

    service grafana-server start
    

    Grafana默认跑在3000端口,用浏览器访问之,然后点击Create your first data source添加Prometheus数据源。具体的填写方法如下图所示。

    195230-a82dc28fe19d59a6.png

    运行示例程序来产生一些监控数据。

    nc -l 17777
    
    /opt/flink-1.9.0/bin/flink run \
    --detached \
    --jobmanager yarn-cluster \
    --yarnname "example-socket-window-wordcount" \
    /opt/flink-1.9.0/examples/streaming/SocketWindowWordCount.jar \
    --port 17777
    

    最后添加一个用于查看监控图表的dashboard。点击New Dashboard->Add Query按钮,就可以直接看到Flink下的各个metric,还可以添加多个metric。

    195230-201875f558414707.png

    选完之后,在上面的图表中就可以看到监控数据的折线了。

    展开全文
  • 日萌社 人工智能AI:Keras PyTorch MXNet ... 拷贝jar包:cp /data/flink-1.10.1/opt/flink-metrics-prometheus-1.10.1.jar /data/flink-1.10.1/lib 修改配置文件 vim /data/flink-1.10.1/conf/flink-conf...
  • You also need Node.js and Yarn installed in order to build the frontend assets. You can directly use the go tool to download and install the prometheus and promtool binaries into your GOPATH: $ go get...
  • 简述 flink metric用来对外暴露系统内部的一些运行指标,比如flink框架运行时的JVM相关配置,或者基于flink开发的项目。flink提供了Counter, Gauge, Histogram ...今天主要讲的是使用Prometheus+PushGateway+Grafana...
  • Prometheus Pushgateway The Prometheus Pushgateway exists to allow ephemeral and batch jobs to expose their metrics to Prometheus. Since these kinds of jobs may not exist long enough to be scraped, ...
  • prometheus

    2021-07-06 17:52:59
    prometheus_javaagent-0.15.0.jar=11104:/usr/monitor/collectors/conf/config.yaml " 2、HADOOP_HOME/bin/yarn中添加: 在对应的YARN_OPTS后面加上$HADOOP_RESOURCEMANAGER_JMX_OPTS或$HADOOP_NODEMANAGER_JMX_OPTS ...
  • yarn add @willsoto/nestjs-prometheus prom-client npm install @willsoto/nestjs-prometheus prom-client 用法 import { Module } from "@nestjs/common" ; import { PrometheusModule } from "@willsoto/nestjs-...
  • Prometheus模板示例

    2019-11-16 21:37:00
    完整译文请访问:http://www.coderdocument.com/docs/prometheus/v2.14/prometheus/configuration/template_examples.html。 Prometheus支持模板化告警的注解和标签以及服务控制台页面。模板能够对本地数据库运行...
  • $ yarn add @qfed/prometheus 在线链接[] Getting started iimport React from "react" import ReactDOM from "react-dom" import { Button } from "@qfed/prometheus" class ButtonDemo extends React.Component { ...
  • Yarn Build Clone the repository and build manually: make build Usage usage: prometheus-webhook-dingtalk [<flags>] Flags: -h, --help Show context-sensitive help (also try --help-long and --...

空空如也

空空如也

1 2 3 4 5 ... 20
收藏数 632
精华内容 252
关键字:

prometheusyarn