storm 订阅
英文单词,风暴的意思,同时还是游戏人物,日用品,播放软件,实时平台,歌曲等的名称。 展开全文
英文单词,风暴的意思,同时还是游戏人物,日用品,播放软件,实时平台,歌曲等的名称。
信息
公司性质
股份制
员工数
50
年营业额
30000000
总部地点
浙江省宁波市鄞州区惠风西路115号城南商务大厦B1202-1室
外文名称
STORM
经营范围
户外野营
成立时间
2014年1月1日
公司名称
宁波开普乐户外科技有限公司
storm单词释义
storm英[stɔːm] 美[stɔrm]风暴的英文名,是一个天体的大气的任何不稳定状态,尤其影响天体的表面,通常会带来恶劣天气。风暴可能伴随著强风、雷和闪电(例如雷暴)、强降水或随风在大气中移动的物质(例如沙尘暴、暴风雪和雹暴)。
收起全文
精华内容
下载资源
问答
  • Storm

    2018-09-05 16:57:58
    Storm

    学习进展记录:

    • 2019-8-21:解决ui无法显示输出log的问题;开始阅读github中starter的demo代码;解决在idea中远程调试Storm程序的问题;
    • 2019-8-27,手打、阅读完Storm的官方Topology demo;
    • 接下来学习,Storm与其他项目的集成,输入和输出;

    2020-5-9,Strom似乎被Flink替代了,不再需要学习和使用了

    资料:

    专有名词:

    • Storm(风暴):一种实时计算引擎,可提供毫秒级实时计算;相较而言,Spark Streaming仅可提供秒级准实时计算;
    • Java为其主要开发语言。将Storm代码极其依赖打成jar包,然后调用storm cli命令开启运行;
    • Topologies:Storm的计算图例,包含进程逻辑、数据流向,有些类似于Hadoop的MapReduce;但MR任务是会结束的,topologies任务则不会结束(除非手动kill);
    • Nodes:分为 master node 和 worker nodes 两类;
    • Nimbus(云雨):任务分配、监控Storm集群;
    • Supervisor:Nimbus将集群的任务分配给各个node,Supervisor进行监听,从而开启、关闭该节点的进程;
    • Storm Cluster极其健壮、稳定,比Spark稳定;
    • Strom Stream:
      • spout(龙卷风):将数据源(API / MQ)转化为Storm Stream;
      • bolt(闪电):将Stream进行函数计算、生成新流、下发数据等;
      • spout和bolt组成的工作网络,称之为Topology;
    • DRPC:Distributed Remote Procedure Call,分布式远程调用;主要作用在于,利用Storm来完成那些计算密集型函数(CPU密集型函数)的实时并行计算;
    • 输入:
    • 输出:Hive、

    Tips:

    • 运行Storm时,storm nimbus这个进程一定要开启;
    • 重启zookeeper中storm文件,可以清除所有Storm的topology;
    • 在Storm cli提交jar包运行之后,一般要过一分钟左右,Storm UI才会有显示;
    • 在集群上启动Storm时,要在主节点开启nimbus、ui进程,在从节点开启supervisor进程,且在所有节点均开启logviewer进程,才能在Storm UI上查看log;
    • 若想在本地idea直接运行集群的Storm,可将storm.yaml拷贝当java项目的src文件目录下即可;

    Storm CLI

    # 详细见:http://storm.apache.org/releases/2.0.0/Command-line-client.html
    storm jar <topology-jar-path> <class-name> <args>  # 在Storm集群上运行jar包,<args>为java 主函数的参数
    storm sql <sql-file> <topology-name>  # 运行sql
    storm list  #  显示目前运行的所有topology
    storm kill <topology-name>  # 中止指定topology的计算
    storm activate <topology-name>  # 激活某个topology
    storm deactivate <topology-name>  # 失活某个topology
    storm classpath  # 打印路径
    storm server_classpath  # 打印路径
    storm nimbus  # 开启nimbus进程
    storm supervisor  # 开启supervisor进程
    storm ui  # 开启storm的web页面进程
    storm drpc  # 开启drpc进程
    storm logviewer  # 开启logviewer进程,从而在storm ui上浏览log
    storm version  # Storm版本号
    storm help  # 帮助
    

    踩坑记录

    Q:启动Storm UI时,报错org.apache.storm.utils.NimbusLeaderNotFoundException: Could not find leader nimbus from seed hosts ["bigdata85"]. Did you specify a valid list of nimbus hosts for config nimbus.seeds?
    A:https://blog.csdn.net/henianyou/article/details/73733637
    Q:在集群中运行Storm的工程时,明明在Maven中添加了pom依赖,但报错Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/storm/starter/spout/RandomSentenceSpout
    A:将RandomSentenceSpout方法所在的jar文件,storm-starter-1.1.10.jar拷贝到集群中的$STORM_HOME/lib路径下即可
    Q:在idea本地运行Storm项目,报错java.lang.ClassNotFoundException: org.apache.storm.topology.IRichSpout
    A:删除pom.xml中的<scope>provided</scope>;打jar包在集群上运行,参数为 provided;若直接在本地运行,删除即可;


    Storm工程模板

    /**
     *     <dependencies>
     *         <dependency>
     *             <groupId>org.apache.storm</groupId>
     *             <artifactId>storm-core</artifactId>
     *             <version>1.1.1</version>
     *             <!--<scope>provided</scope>-->
     *             <!-- 打jar包在集群上运行,参数为 provided;若直接在本地运行,删除即可 -->
     *         </dependency>
     *
     *     </dependencies>
     */
    
    /**
     * @ Name        : Storm_demo
     * @ Description : TODO
     * @ Author      : yangsong
     * @ Date        : 2019-8-27 11:35
     * @ Version     : 1.0
     **/
    
    import org.apache.storm.Config;
    import org.apache.storm.LocalCluster;
    import org.apache.storm.StormSubmitter;
    import org.apache.storm.spout.SpoutOutputCollector;
    import org.apache.storm.task.OutputCollector;
    import org.apache.storm.task.TopologyContext;
    import org.apache.storm.topology.BasicOutputCollector;
    import org.apache.storm.topology.OutputFieldsDeclarer;
    import org.apache.storm.topology.TopologyBuilder;
    import org.apache.storm.topology.base.BaseBasicBolt;
    import org.apache.storm.topology.base.BaseRichBolt;
    import org.apache.storm.topology.base.BaseRichSpout;
    import org.apache.storm.tuple.Fields;
    import org.apache.storm.tuple.Values;
    import org.apache.storm.tuple.Tuple;
    
    import java.util.Map;
    
    public class Storm_demo {
        public static class Split extends BaseRichSpout {  // 根据业务逻辑重写Spout
            SpoutOutputCollector _collector;
            int _base = 0;
            int _i = 0;
    
            @Override
            public void open(Map conf, TopologyContext context, SpoutOutputCollector collector) {
                _collector = collector;
                _base = context.getThisTaskIndex();
            }
    
            @Override
            public void nextTuple() {
                Values v = new Values(_base, _i);
                _collector.emit(v, "ACK");
                _i++;
            }
    
            @Override
            public void declareOutputFields(OutputFieldsDeclarer declarer) {
                // declarer.declare(new Fields("fields_spout"));
            }
    
            // @Override
            // 重写函数
        }
    
        public static class Bolt1 extends BaseBasicBolt {  // 根据业务逻辑重写Bolt
            @Override
            public void declareOutputFields(OutputFieldsDeclarer declarer) {
                // Empty
                // declarer.declare(new Fields("blot1", "bolt1_info"));
            }
    
            @Override
            public void execute(Tuple tuple, BasicOutputCollector collector) {
                // 根据业务逻辑重写
            }
    
            // @Override
            // 重写函数
        }
    
        public static class Bolt2 extends BaseBasicBolt {  // 根据业务逻辑重写Bolt
            @Override
            public void declareOutputFields(OutputFieldsDeclarer declarer) {
                // Empty
                declarer.declare(new Fields("blot2", "bolt2_info"));
            }
    
            @Override
            public void execute(Tuple tuple, BasicOutputCollector collector) {
                // 根据业务逻辑重写
            }
    
            // @Override
            // 重写函数
        }
    
        public static class Bolt3 extends BaseRichBolt {  // 根据业务逻辑重写Bolt
            OutputCollector _collector;
    
            @Override
            public void declareOutputFields(OutputFieldsDeclarer declarer) {
                // Empty
                declarer.declare(new Fields("blot3", "blot3_info"));
            }
    
            @Override
            public void prepare(Map conf, TopologyContext context, OutputCollector collector) {
                _collector = collector;
            }
    
            @Override
            public void execute(Tuple tuple) {
                // 根据业务逻辑重写
            }
    
            // @Override
            // 重写函数
        }
    
        public static void main(String[] args) throws Exception {
            TopologyBuilder builder = new TopologyBuilder();
            builder.setSpout("spout_name", new Split(), 3);  // 创建Spout
            builder.setBolt("bolt1_name", new Bolt1(), 3).shuffleGrouping("spout_name");  // 创建Bolt
            builder.setBolt("bolt2_name", new Bolt2(), 3).shuffleGrouping("bolt1_name");  // 创建Bolt
            builder.setBolt("bolt3_name", new Bolt3(), 3).shuffleGrouping("bolt2_name");  // 创建Bolt
    
            Config conf = new Config();  // 集群配置
            conf.setDebug(true);  // 设置Storm记录Spout、Bolt产生的每条信息
    
            if (args != null && args.length > 0) {  // 若输入参数正常
                conf.setNumWorkers(3);  // 设置worker为3
    
                StormSubmitter.submitTopologyWithProgressBar(args[0], conf, builder.createTopology());  // 提交Topology
            } else {
                conf.setMaxTaskParallelism(3);
    
                LocalCluster cluster = new LocalCluster();
                cluster.submitTopology("topology_name", conf, builder.createTopology());
    
                Thread.sleep(10000);
                // cluster.killTopology();  // kill掉topology
                cluster.shutdown();
            }
        }
    }
    
    展开全文
  • storm

    2019-10-23 18:51:14
    一、安装storm 将压缩包apache-storm-1.2.2.tar.gz拖入hdp-4 的sftp>下: 在linux下解压: tar -zvxf apache-storm-1.2.2.tar.gz 给storm改名,将apache-storm-1.2.2改为storm-1.2.2: mv apache-storm-...

    一、安装storm

    将压缩包apache-storm-1.2.2.tar.gz拖入hdp-4 的sftp>下:

    在linux下解压:

    tar -zvxf apache-storm-1.2.2.tar.gz

    给storm改名,将apache-storm-1.2.2改为storm-1.2.2:

    mv  apache-storm-1.2.2  storm-1.2.2

    修改配置环境(在[root@hdp-4 etc]#下):

    vi profile

    修改内容:

    export STORM_HOME=/root/apps/storm-1.2.2
    export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin:$STORM_HOME/bin

    1

     修改完后重新加载配置文件:使storm立即生效

    source /etc/profile
    storm verson

    如下:

     1

     修改storm配置文件(在hdp-4下的cd storm-1.2.2/conf/下):

    vi storm.yaml

    配置文件内容为:

    storm.zookeeper.servers:
         - "hdp-1"
         - "hdp-2"
         - "hdp-4"
    storm.local.dir: "/root/stormdata"
    nimbus.seeds: ["hdp-4"]
    supervisor.slots.ports:
        - 6700
        - 6701
        - 6702
        - 6703

    将storm复制到hdp-1与hdp-2中,并依此修改上面的配置:

    scp -r /root/apps/storm-1.2.2 hdp-1:/root/apps/storm-1.2.2

    在hdp-1的cd apps/storm-1.2.2/bin下启动storm服务到后台:

     storm nimbus >/dev/null 2>&1 &

    再输入命令访问web界面(master):

    storm ui

    在hdp-4或者hdp-2输入命令启动另一个服务:

    storm supervisor >/dev/null 2>&1 &

    在浏览器搜索hdp-1:8080:

    1

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

    展开全文
  • STORM

    2011-06-30 22:57:00
    We have been doing realtime processing for a long time at BackType. Weve recently developed a new system for doing realtime processing called Storm to replace our old system of queues and workers

    We have been doing realtime processing for a long time at BackType. We've recently developed a new system for doing realtime processing called Storm to replace our old system of queues and workers. Storm is a distributed, reliable, and fault-tolerant stream processing system. Its use cases are so broad that we consider it to be a fundamental new primitive for data processing. That's why we call it the Hadoop of realtime: it does for realtime processing what Hadoop does for batch processing. We are planning to open-source Storm sometime in the next few months. 

    Like most people doing realtime processing, our old system was a complicated graph of queues and workers. Worker processes would take messages off a queue and then update a database and/or fire off new messages to other queues. 

    There was a lot of pain in doing realtime processing this way. We found that we spent most of our time worrying about where to send messages, where to receive messages from, deploying workers, and deploying queues. Worst of all, the system wasn't fault tolerant: we had to make sure our queues and workers stayed up.

    Storm solves these issues completely. It abstracts the message passing away, automatically parallelizes the stream computation on a cluster of machines, and lets you focus on your realtime processing logic. Even more interesting, Storm enables a whole new range of applications we didn't anticipate when we initially designed it.

    Properties of Storm

    Here are the key properties of Storm:

    1. Simple programming model: Just like how MapReduce dramatically lowers the complexity for doing parallel batch processing, Storm's programming model dramatically lowers the complexity for doing realtime processing. 

    2. Runs any programming language: Even though Storm runs on the JVM (and is written in Clojure), you can use any programming language on top of Storm. We've added support for Ruby and Python, and support can easily be added for any language -- all you need to do is code a ~100 line library which implements a simple communication protocol with Storm.

    3. Fault-tolerant: To launch a processing topology on Storm, all you have to do is provide a jar containing all your code. Storm then distributes that jar, assigns workers across the cluster to execute the topology, monitors the topology, and automatically reassigns workers that go down.

    4. Horizontally scalable: All computations are done in parallel. To scale a realtime computation, all you have to do is add more machines and Storm takes care of the rest.

    5. Reliable: Storm guarantees that each message will be fully processed at least once. Messages will be processed exactly once as long as there are no errors.

    6. Fast: Storm is built with speed in mind. ZeroMQ is used for the underlying message passing, and care has been taken so that messages are processed extremely quickly.

    Use cases for Storm

    There are three broad use cases for Storm:

    1. Stream processing: This is the traditional realtime processing use case: process messages and update a variety of databases.

    2. Continuous computation: Storm can be used to do a continuous computation and stream out the results as they're computed. For example, we used Storm the other day to compute trending users on Twitter off of the Twitter firehose. Every second, Storm streams out the 50 users with the most retweets in the last few minutes with perfect accuracy. We stream this information directly into a webpage which visualizes and animates the trending users in realtime.

    3. Distributed RPC: Distributed RPC is perhaps the most unexpected and most compelling use case for Storm. There are a lot of queries that are both hard to precompute and too intense to compute on the fly on a single machine. Traditionally, you have to do an approximation of some sort to lower the cost of a query like this. Storm gives the capability to parallelize an intense query so that you can compute it in realtime.

    An example of a query that is only possible with distributed RPC is "reach": computing the number of unique people exposed to a URL on Twitter. To compute reach, you need to get all the people who tweeted the URL, get all the followers of all those people, unique that set of followers, and then count the number of uniques. It's an intense computation that potentially involves thousands of database calls and tens of millions of follower records. It can take minutes or worse to compute on a single machine. With Storm, you can do every step of the reach computation in parallel and compute reach for any URL in seconds (and less than a second for most URLs).

    The idea behind distributed RPC is that you run a processing topology on Storm that implements the RPC function and waits for RPC invocations. An RPC invocation is a message containing the parameters of the RPC request and information of where Storm should send the results. The topology picks up messages, computes the RPC call in parallel, and returns the results to the return address.

    Summary

    Storm is already enormously useful for us at BackType. It reduces a ton of complexity in our realtime processing and lets us do things we couldn't do before. We look forward to the day we open source it.

    You should follow the BackType tech team on Twitter here.

    展开全文
  • Stormstorm集群搭建

    万次阅读 2019-12-16 22:35:59
    storm集群搭建 首先需要搭建zookeeper集群 核心配置文件 storm.zookeeper.servers: - "s1" - "s2" - "s3" nimbus.host: "s1" 其中s1、s2、s3为zookeeper集群。nimbus.host指定nimbus所在主机 启动storm集群 在...

    storm集群搭建

    首先需要搭建zookeeper集群

    核心配置文件

    storm.zookeeper.servers:
         - "s1"
         - "s2"
         - "s3"
    nimbus.host: "s1"
    

    其中s1、s2、s3为zookeeper集群。nimbus.host指定nimbus所在主机

    启动storm集群

    nimbus所在主机上启动

    storm nimbus &
    storm ui &
    

    在storm集群中其他主机上启动supervisor

    storm supervisor &
    
    展开全文
  • Storm的简要介绍及Storm集群的安装部署和启动停止。
  • Storm介绍

    万次阅读 2019-01-17 09:22:06
    Storm介绍 Storm是Twitter开源的分布式实时大数据处理框架,最早开源于github,从0.9.1版本之后,归于Apache社区,被业界称为实时版Hadoop。随着越来越多的场景对Hadoop的MapReduce高延迟无法容忍,比如网站统计、...
  • Storm 03_Storm 架构设计

    2019-10-05 22:11:23
    Storm 任务提交流程 Storm 本地目录树 Storm Zookeeper目录树
  • Storm教程

    2019-01-16 10:19:10
    Storm教程 Storm介绍 Storm核心概念 Storm集群体系结构 Storm工作流程 Storm分布式消息系统 Storm 安装 Storm示例 Storm Trident Storm Twitter 在雅虎金融中使用Storm Storm应用程序 ...
  • Storm组件本地目录树   Storm zookeeper目录树   Storm任务提交的过程    
  • Apache Storm

    千次阅读 2019-12-28 23:43:12
    Apache Storm 一、概述 http://storm.apache.org/ Apache Storm是一款免费开源的分布式实时计算的框架(流处理) Apache Storm可以非常容易并且可靠的处理无界的流数据,进行实时的分析处理。 Apache Storm支持多种...
  • storm心得

    2019-10-12 12:14:44
    /zywa/storm/bin/storm nimbus & 2、启动supervisor /zywa/storm/bin/storm supervisor & 3、启动 ui /zywa/storm/bin/storm ui & 4、启动任务 Topology cd /zywa/storm && ....
  • Storm之——Storm2.0.0配置详解

    千次阅读 2019-08-10 23:23:09
    转载请注明出处:... Storm2.0.0中defaults.yaml文件的配置如下(链接为:https://github.com/apache/storm/blob/v2.0.0/conf/defaults.yaml): # Licensed to the Apache Soft...
  • Storm 安装

    万次阅读 2018-10-10 19:13:20
    修改 conf目录下的storm.yaml文件,集群的所有服务器统一配置 #zookeeper地址,默认端口2181 storm.zookeeper.servers: - "test1" - "test2" - "test3" #nimbus服务器 nimbus....
  • 浅析Storm

    千次阅读 2019-05-18 17:09:06
    浅析StormStorm基本概念Storm体系架构topology运行机制 Storm基本概念   Hadoop等离线批量处理主要针对数据挖掘、数据分析、数据统计、BI分析等。而Storm等分布式实时计算系统可以支持一些在线的业务系统、分布式...
  • 安装Storm 0.93

    千次阅读 2015-01-24 15:54:13
    Storm
  • Storm0.9.2 集群安装部署Storm0.9.2 集群安装部署1. 部署环境准备2. 部署 Storm 集群2.1 搭建 Zookeeper 集群2.2 依赖软件安装2.3 安装 Storm 集群2.4 修改 storm.yaml 配置文件2.5 启动 Storm 各后台进程 Storm...
  • storm安装

    2019-10-13 16:10:21
    1、启动zk集群 2、每台机器要求java 7+ 以及 python 2.6.6+ 3、下载storm : ...4、将storm-1.2.2.tar.gz上传到master上hadoop-jrq用户下的bigdata目录下,用下面的命令解压: ta...
  • Storm安装

    2019-12-09 15:37:46
    Storm安装 搭建介绍 软件名 版本 apache-zookeeper apache-zookeeper-3.5.5.tar apache-storm apache-storm-2.1.0.tar IP HostName 192.168.159.111 master 192.168.159.112 slave1 192.168...
  • Storm学习笔记 - Storm初识

    千次阅读 2018-10-26 10:06:42
    Storm学习笔记 - Storm初识 1. Strom是什么? Storm是一个开源免费的分布式计算框架,可以实时处理大量的数据流。 2. Storm的特点 高性能,低延迟。 分布式:可解决数据量大,单机搞不定的场景。 可扩展...
  • 上一章节讲了如何手写一个storm wordcount程序:34-Storm学习-Storm教程:纯手敲WordCount程序 蕴含了很多的知识点,如下: (1)Spout (2)Bolt (3)OutputCollector,Declarer (4)Topology (5)设置worker...
  • 一篇文带你快速起步Apache Storm

    万次阅读 2020-09-21 21:57:03
    Storm是什么 应用场景 Storm与Hadoop的关系 Storm怎么用 示例1:统计单词出现的次数 核心概念 集群架构 示例2:统计通话记录 1.Storm是什么 Storm 是一个分布式实时大数据...
  • Storm六之storm安装

    2017-01-16 14:06:38
    我们现在来看看如何在自己的机器上安装storm。 主要分为三步: 1.安装java环境 2.安装zookeeper环境 3.安装storm   步骤1和2省略,不会请自行百度。 下面开始安装storm的步骤(此时java和zookeeper需要先装好...
  • storm简介

    千次阅读 2018-10-09 14:45:50
    Storm是一个分布式的,可靠的,容错的数据流处理系统。Storm集群的输入流由一个被称作spout的组件管理,spout把数据传递给bolt, bolt要么把数据保存到某种存储器,要么把数据传递给其它的bolt。一个Storm集群就是在...

空空如也

空空如也

1 2 3 4 5 ... 20
收藏数 23,146
精华内容 9,258
关键字:

storm