Fwd: Storm with Drools Complex Event Processing rules

[Nishu]

Hi,

I am using Storm with Drools Fusion for adding CEP rules to raise system
alerts . I need to raise alerts for system if condition is matched for last
10 records or 10 minutes.
 Bolt is receiving one event in tuple from Spout. How can I make it work
where bolt can read multiple events(lets say 10 events or last 10 minute
events) and how to call CEP rules on those event to raise an alert?

I checked following thread also :
https://groups.google.com/forum/#!topic/storm-user/VqBPnheDtQ8

-- 
Thanks,
Nishu Tayal

Re: why storm still use it's own zookeeper node not my zookeeper cluster?

[Niels Basjes]

Where is the storm.yaml placed?
Do you have a ~/.storm/storm.yaml with overrides?

Niels
On Apr 24, 2014 8:05 AM, "ch huang"  wrote:

> hi,maillist:
> i am a storm newbie ,when i start storm nimbus and suervisor
> ,i check log and find the following info
>
> 2014-04-08 09:26:02 o.a.z.ZooKeeper [INFO] Client
> environment:zookeeper.version=3.3.3-1073969, built on 02/23/2011 22:27 GMT
> 2014-04-08 09:26:02 o.a.z.s.ZooKeeperServer [INFO] Server
> environment:zookeeper.version=3.3.3-1073969, built on 02/23/2011 22:27 GMT
>  but my zookeeper is  3.4.5 ,
>
> # rpm -qa|grep zookeeper
> zookeeper-server-3.4.5+23-1.cdh4.4.0.p0.24.el6.noarch
> zookeeper-3.4.5+23-1.cdh4.4.0.p0.24.el6.noarch
> my storm.yaml is
>
> storm.zookeeper.servers:
> - 192.168.10.220
> - 192.168.10.221
> - 192.168.10.223
> but why storm still use it's build-in zookeeper not my zookeeper cluster?
>

Storm dies when zookeeper followers are not available

[Gaurav Sehgal]

We have a cluster deployment of Storm and Zookeeper. Where storm is running
on 2 machines and zookeeper is running on 3 machines.

Now if the zookeeper followers are not available, though the leader is
available storm keeps throwing an exception that it is not able to connect
to zookeeper; and ultimately dies.

Has anyone encountered this issue?

Regards,
Gaurav

Re: Storm with Drools Complex Event Processing rules

[Eugene]

Hi Nishi,
I will point to you some resources that solve similar problems:
1. Book: Storm Real-time processing Cookbook  has
chapter "Rule-based analysis of the log stream". This recipe integrates the
JBoss Library and Drools into a bolt and allow to capture event based on
rule. Next you need sliding window analysis to see if your events frequency
exceed some threshold. The same book has chapter how to implement "rolling
window topology" or better yet,
2. brand new book "Storm Blueprints " explain how to
do sliding windows and alerting using Trident ( which I prefer over plain
storm semantics). Using those 2 resources it should be easy to craft
solution for your use case.
good luck



On Thu, Apr 24, 2014 at 3:36 AM, Nishu  wrote:

> Hi,
>
> I am using Storm with Drools Fusion for adding CEP rules to raise system
> alerts . I need to raise alerts for system if condition is matched for last
> 10 records or 10 minutes.
>  Bolt is receiving one event in tuple from Spout. How can I make it work
> where bolt can read multiple events(lets say 10 events or last 10 minute
> events) and how to call CEP rules on those event to raise an alert?
>
> I checked following thread also :
> https://groups.google.com/forum/#!topic/storm-user/VqBPnheDtQ8
>
> --
> Thanks,
> Nishu Tayal
>
>


-- 
Eugene Dvorkin
Software Engineer
New York City Storm User Group - organizer
WebMD
email: edvor...@gmail.com
phone: 646-250-9649
eugenedvorkin.com
Connect with me on:
LinkedIn 
Twitter

RE: Logging levels

[Gibson, Richard]

If you want to adjust on a per topology basis create a logback config and 
package in uber jar

The config should be an included logback config file rather than stand alone, 
so root tag is 
After that all storm cluster.xml files should be updated to include




From: Lin Zhao [mailto:l...@groupon.com]
Sent: 21 April 2014 18:10
To: user@storm.incubator.apache.org
Subject: Re: Logging levels

You should probably add a new logger and appender for your topology such that 
it doesn't include storm's infrastructure logging.

e. g.


myApp.log

  %d{-MM-dd HH:mm:ss} %c{1} [%p] %m%n

  





On Mon, Apr 21, 2014 at 9:42 AM, Software Dev 
mailto:static.void@gmail.com>> wrote:
Is there any way to adjust this per topology or project as opposed to
system wide?

On Sun, Apr 20, 2014 at 11:23 PM, 朱春来 
mailto:zhuchunlai...@gmail.com>> wrote:
> Try to modify the property file of log4j which is in the $STROM_HOME/log4j
>
>
> 2014-04-19 6:59 GMT+08:00 Software Dev 
> mailto:static.void@gmail.com>>:
>
>> How can one change the log levels.. the output is insane!
>
>
>
>
> --
> Thanks,
>
> Chunlai



--
Lin Zhao

https://wiki.groupondev.com/Message_Bus
3101 Park Blvd, Palo Alto, CA 94306

storm starter ExclamationTopology

[Bilal Al Fartakh]

~/src/storm-0.8.1/bin/storm jar
/root/src/storm-starter/target/storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar
storm.starter.ExclamationTopology demo

*I tried to run this and it said that the problem is with the nimbus
connection , but my storm client (and supervisor in the same time ) is
connected with my nimbus (shown in Strom UI )*

Running: java -client -Dstorm.options= -Dstorm.home=/root/src/storm-0.8.1
-Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -cp
/root/src/storm-0.8.1/storm-0.8.1.jar:/root/src/storm-0.8.1/lib/asm-4.0.jar:/root/src/storm-0.8.1/lib/commons-codec-1.4.jar:/root/src/storm-0.8.1/lib/carbonite-1.5.0.jar:/root/src/storm-0.8.1/lib/kryo-2.17.jar:/root/src/storm-0.8.1/lib/clout-0.4.1.jar:/root/src/storm-0.8.1/lib/clojure-1.4.0.jar:/root/src/storm-0.8.1/lib/ring-servlet-0.3.11.jar:/root/src/storm-0.8.1/lib/hiccup-0.3.6.jar:/root/src/storm-0.8.1/lib/disruptor-2.10.1.jar:/root/src/storm-0.8.1/lib/tools.cli-0.2.2.jar:/root/src/storm-0.8.1/lib/snakeyaml-1.9.jar:/root/src/storm-0.8.1/lib/joda-time-2.0.jar:/root/src/storm-0.8.1/lib/jetty-util-6.1.26.jar:/root/src/storm-0.8.1/lib/commons-exec-1.1.jar:/root/src/storm-0.8.1/lib/jetty-6.1.26.jar:/root/src/storm-0.8.1/lib/servlet-api-2.5.jar:/root/src/storm-0.8.1/lib/jzmq-2.1.0.jar:/root/src/storm-0.8.1/lib/curator-framework-1.0.1.jar:/root/src/storm-0.8.1/lib/httpclient-4.1.1.jar:/root/src/storm-0.8.1/lib/slf4j-log4j12-1.5.8.jar:/root/src/storm-0.8.1/lib/clj-time-0.4.1.jar:/root/src/storm-0.8.1/lib/commons-lang-2.5.jar:/root/src/storm-0.8.1/lib/libthrift7-0.7.0.jar:/root/src/storm-0.8.1/lib/log4j-1.2.16.jar:/root/src/storm-0.8.1/lib/servlet-api-2.5-20081211.jar:/root/src/storm-0.8.1/lib/tools.logging-0.2.3.jar:/root/src/storm-0.8.1/lib/ring-core-0.3.10.jar:/root/src/storm-0.8.1/lib/minlog-1.2.jar:/root/src/storm-0.8.1/lib/objenesis-1.2.jar:/root/src/storm-0.8.1/lib/jline-0.9.94.jar:/root/src/storm-0.8.1/lib/commons-io-1.4.jar:/root/src/storm-0.8.1/lib/ring-jetty-adapter-0.3.11.jar:/root/src/storm-0.8.1/lib/jgrapht-0.8.3.jar:/root/src/storm-0.8.1/lib/json-simple-1.1.jar:/root/src/storm-0.8.1/lib/tools.macro-0.1.0.jar:/root/src/storm-0.8.1/lib/commons-fileupload-1.2.1.jar:/root/src/storm-0.8.1/lib/compojure-0.6.4.jar:/root/src/storm-0.8.1/lib/httpcore-4.1.jar:/root/src/storm-0.8.1/lib/commons-logging-1.1.1.jar:/root/src/storm-0.8.1/lib/guava-13.0.jar:/root/src/storm-0.8.1/lib/curator-client-1.0.1.jar:/root/src/storm-0.8.1/lib/math.numeric-tower-0.0.1.jar:/root/src/storm-0.8.1/lib/junit-3.8.1.jar:/root/src/storm-0.8.1/lib/slf4j-api-1.5.8.jar:/root/src/storm-0.8.1/lib/reflectasm-1.07-shaded.jar:/root/src/storm-0.8.1/lib/core.incubator-0.1.0.jar:/root/src/storm-0.8.1/lib/zookeeper-3.3.3.jar:/root/src/storm-starter/target/storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar:/root/.storm:/root/src/storm-0.8.1/bin
-Dstorm.jar=/root/src/storm-starter/target/storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar
storm.starter.ExclamationTopology demo Exception in thread "main"
java.lang.RuntimeException:
org.apache.thrift7.transport.TTransportException:
java.net.ConnectException: Connection refused at
backtype.storm.utils.NimbusClient.(NimbusClient.java:36) at
backtype.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:17)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:53) at
storm.starter.ExclamationTopology.main(ExclamationTopology.java:59) Caused
by: org.apache.thrift7.transport.TTransportException:
java.net.ConnectException: Connection refused at
org.apache.thrift7.transport.TSocket.open(TSocket.java:183) at
org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81)
at backtype.storm.utils.NimbusClient.(NimbusClient.java:34) ... 3 more
Caused by: java.net.ConnectException: Connection refused at
java.net.PlainSocketImpl.socketConnect(Native Method) at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at
java.net.Socket.connect(Socket.java:579) at
org.apache.thrift7.transport.TSocket.open(TSocket.java:178)

-- 
*Al Fartakh Bilal*

Re: storm starter ExclamationTopology

[Bilal Al Fartakh]

and the question is , what should I fix dear experts ? :)


2014-04-24 16:23 GMT+00:00 Bilal Al Fartakh :

> ~/src/storm-0.8.1/bin/storm jar
> /root/src/storm-starter/target/storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar
> storm.starter.ExclamationTopology demo
>
> *I tried to run this and it said that the problem is with the nimbus
> connection , but my storm client (and supervisor in the same time ) is
> connected with my nimbus (shown in Strom UI )*
>
> Running: java -client -Dstorm.options= -Dstorm.home=/root/src/storm-0.8.1
> -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -cp
> /root/src/storm-0.8.1/storm-0.8.1.jar:/root/src/storm-0.8.1/lib/asm-4.0.jar:/root/src/storm-0.8.1/lib/commons-codec-1.4.jar:/root/src/storm-0.8.1/lib/carbonite-1.5.0.jar:/root/src/storm-0.8.1/lib/kryo-2.17.jar:/root/src/storm-0.8.1/lib/clout-0.4.1.jar:/root/src/storm-0.8.1/lib/clojure-1.4.0.jar:/root/src/storm-0.8.1/lib/ring-servlet-0.3.11.jar:/root/src/storm-0.8.1/lib/hiccup-0.3.6.jar:/root/src/storm-0.8.1/lib/disruptor-2.10.1.jar:/root/src/storm-0.8.1/lib/tools.cli-0.2.2.jar:/root/src/storm-0.8.1/lib/snakeyaml-1.9.jar:/root/src/storm-0.8.1/lib/joda-time-2.0.jar:/root/src/storm-0.8.1/lib/jetty-util-6.1.26.jar:/root/src/storm-0.8.1/lib/commons-exec-1.1.jar:/root/src/storm-0.8.1/lib/jetty-6.1.26.jar:/root/src/storm-0.8.1/lib/servlet-api-2.5.jar:/root/src/storm-0.8.1/lib/jzmq-2.1.0.jar:/root/src/storm-0.8.1/lib/curator-framework-1.0.1.jar:/root/src/storm-0.8.1/lib/httpclient-4.1.1.jar:/root/src/storm-0.8.1/lib/slf4j-log4j12-1.5.8.jar:/root/src/storm-0.8.1/lib/clj-time-0.4.1.jar:/root/src/storm-0.8.1/lib/commons-lang-2.5.jar:/root/src/storm-0.8.1/lib/libthrift7-0.7.0.jar:/root/src/storm-0.8.1/lib/log4j-1.2.16.jar:/root/src/storm-0.8.1/lib/servlet-api-2.5-20081211.jar:/root/src/storm-0.8.1/lib/tools.logging-0.2.3.jar:/root/src/storm-0.8.1/lib/ring-core-0.3.10.jar:/root/src/storm-0.8.1/lib/minlog-1.2.jar:/root/src/storm-0.8.1/lib/objenesis-1.2.jar:/root/src/storm-0.8.1/lib/jline-0.9.94.jar:/root/src/storm-0.8.1/lib/commons-io-1.4.jar:/root/src/storm-0.8.1/lib/ring-jetty-adapter-0.3.11.jar:/root/src/storm-0.8.1/lib/jgrapht-0.8.3.jar:/root/src/storm-0.8.1/lib/json-simple-1.1.jar:/root/src/storm-0.8.1/lib/tools.macro-0.1.0.jar:/root/src/storm-0.8.1/lib/commons-fileupload-1.2.1.jar:/root/src/storm-0.8.1/lib/compojure-0.6.4.jar:/root/src/storm-0.8.1/lib/httpcore-4.1.jar:/root/src/storm-0.8.1/lib/commons-logging-1.1.1.jar:/root/src/storm-0.8.1/lib/guava-13.0.jar:/root/src/storm-0.8.1/lib/curator-client-1.0.1.jar:/root/src/storm-0.8.1/lib/math.numeric-tower-0.0.1.jar:/root/src/storm-0.8.1/lib/junit-3.8.1.jar:/root/src/storm-0.8.1/lib/slf4j-api-1.5.8.jar:/root/src/storm-0.8.1/lib/reflectasm-1.07-shaded.jar:/root/src/storm-0.8.1/lib/core.incubator-0.1.0.jar:/root/src/storm-0.8.1/lib/zookeeper-3.3.3.jar:/root/src/storm-starter/target/storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar:/root/.storm:/root/src/storm-0.8.1/bin
> -Dstorm.jar=/root/src/storm-starter/target/storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar
> storm.starter.ExclamationTopology demo Exception in thread "main"
> java.lang.RuntimeException:
> org.apache.thrift7.transport.TTransportException:
> java.net.ConnectException: Connection refused at
> backtype.storm.utils.NimbusClient.(NimbusClient.java:36) at
> backtype.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:17)
> at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:53) at
> storm.starter.ExclamationTopology.main(ExclamationTopology.java:59) Caused
> by: org.apache.thrift7.transport.TTransportException:
> java.net.ConnectException: Connection refused at
> org.apache.thrift7.transport.TSocket.open(TSocket.java:183) at
> org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81)
> at backtype.storm.utils.NimbusClient.(NimbusClient.java:34) ... 3 more
> Caused by: java.net.ConnectException: Connection refused at
> java.net.PlainSocketImpl.socketConnect(Native Method) at
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> at
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at
> java.net.Socket.connect(Socket.java:579) at
> org.apache.thrift7.transport.TSocket.open(TSocket.java:178)
>
> --
> *Al Fartakh Bilal*
>



-- 
*Al Fartakh Bilal*

Re: storm starter ExclamationTopology

[Nishu]

In your storm cluster , you need to verify first, nimbus is running
properly or not. Check nimbus.log in $STORM_HOME/logs directory for error
logs.
Also check nimbus.host parameter in ~/.storm/storm.yaml.


On Thu, Apr 24, 2014 at 9:56 PM, Bilal Al Fartakh
wrote:

> and the question is , what should I fix dear experts ? :)
>
>
> 2014-04-24 16:23 GMT+00:00 Bilal Al Fartakh :
>
>  ~/src/storm-0.8.1/bin/storm jar
>> /root/src/storm-starter/target/storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar
>> storm.starter.ExclamationTopology demo
>>
>> *I tried to run this and it said that the problem is with the nimbus
>> connection , but my storm client (and supervisor in the same time ) is
>> connected with my nimbus (shown in Strom UI )*
>>
>> Running: java -client -Dstorm.options= -Dstorm.home=/root/src/storm-0.8.1
>> -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -cp
>> /root/src/storm-0.8.1/storm-0.8.1.jar:/root/src/storm-0.8.1/lib/asm-4.0.jar:/root/src/storm-0.8.1/lib/commons-codec-1.4.jar:/root/src/storm-0.8.1/lib/carbonite-1.5.0.jar:/root/src/storm-0.8.1/lib/kryo-2.17.jar:/root/src/storm-0.8.1/lib/clout-0.4.1.jar:/root/src/storm-0.8.1/lib/clojure-1.4.0.jar:/root/src/storm-0.8.1/lib/ring-servlet-0.3.11.jar:/root/src/storm-0.8.1/lib/hiccup-0.3.6.jar:/root/src/storm-0.8.1/lib/disruptor-2.10.1.jar:/root/src/storm-0.8.1/lib/tools.cli-0.2.2.jar:/root/src/storm-0.8.1/lib/snakeyaml-1.9.jar:/root/src/storm-0.8.1/lib/joda-time-2.0.jar:/root/src/storm-0.8.1/lib/jetty-util-6.1.26.jar:/root/src/storm-0.8.1/lib/commons-exec-1.1.jar:/root/src/storm-0.8.1/lib/jetty-6.1.26.jar:/root/src/storm-0.8.1/lib/servlet-api-2.5.jar:/root/src/storm-0.8.1/lib/jzmq-2.1.0.jar:/root/src/storm-0.8.1/lib/curator-framework-1.0.1.jar:/root/src/storm-0.8.1/lib/httpclient-4.1.1.jar:/root/src/storm-0.8.1/lib/slf4j-log4j12-1.5.8.jar:/root/src/storm-0.8.1/lib/clj-time-0.4.1.jar:/root/src/storm-0.8.1/lib/commons-lang-2.5.jar:/root/src/storm-0.8.1/lib/libthrift7-0.7.0.jar:/root/src/storm-0.8.1/lib/log4j-1.2.16.jar:/root/src/storm-0.8.1/lib/servlet-api-2.5-20081211.jar:/root/src/storm-0.8.1/lib/tools.logging-0.2.3.jar:/root/src/storm-0.8.1/lib/ring-core-0.3.10.jar:/root/src/storm-0.8.1/lib/minlog-1.2.jar:/root/src/storm-0.8.1/lib/objenesis-1.2.jar:/root/src/storm-0.8.1/lib/jline-0.9.94.jar:/root/src/storm-0.8.1/lib/commons-io-1.4.jar:/root/src/storm-0.8.1/lib/ring-jetty-adapter-0.3.11.jar:/root/src/storm-0.8.1/lib/jgrapht-0.8.3.jar:/root/src/storm-0.8.1/lib/json-simple-1.1.jar:/root/src/storm-0.8.1/lib/tools.macro-0.1.0.jar:/root/src/storm-0.8.1/lib/commons-fileupload-1.2.1.jar:/root/src/storm-0.8.1/lib/compojure-0.6.4.jar:/root/src/storm-0.8.1/lib/httpcore-4.1.jar:/root/src/storm-0.8.1/lib/commons-logging-1.1.1.jar:/root/src/storm-0.8.1/lib/guava-13.0.jar:/root/src/storm-0.8.1/lib/curator-client-1.0.1.jar:/root/src/storm-0.8.1/lib/math.numeric-tower-0.0.1.jar:/root/src/storm-0.8.1/lib/junit-3.8.1.jar:/root/src/storm-0.8.1/lib/slf4j-api-1.5.8.jar:/root/src/storm-0.8.1/lib/reflectasm-1.07-shaded.jar:/root/src/storm-0.8.1/lib/core.incubator-0.1.0.jar:/root/src/storm-0.8.1/lib/zookeeper-3.3.3.jar:/root/src/storm-starter/target/storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar:/root/.storm:/root/src/storm-0.8.1/bin
>> -Dstorm.jar=/root/src/storm-starter/target/storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar
>> storm.starter.ExclamationTopology demo Exception in thread "main"
>> java.lang.RuntimeException:
>> org.apache.thrift7.transport.TTransportException:
>> java.net.ConnectException: Connection refused at
>> backtype.storm.utils.NimbusClient.(NimbusClient.java:36) at
>> backtype.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:17)
>> at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:53) at
>> storm.starter.ExclamationTopology.main(ExclamationTopology.java:59) Caused
>> by: org.apache.thrift7.transport.TTransportException:
>> java.net.ConnectException: Connection refused at
>> org.apache.thrift7.transport.TSocket.open(TSocket.java:183) at
>> org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81)
>> at backtype.storm.utils.NimbusClient.(NimbusClient.java:34) ... 3 more
>> Caused by: java.net.ConnectException: Connection refused at
>> java.net.PlainSocketImpl.socketConnect(Native Method) at
>> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
>> at
>> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
>> at
>> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
>> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at
>> java.net.Socket.connect(Socket.java:579) at
>> org.apache.thrift7.transport.TSocket.open(TSocket.java:178)
>>
>> --
>> *Al Fartakh Bilal*
>>
>
>
>
> --
> *Al Fartakh Bilal*
>



-- 
with regards,
Nishu Tayal

Re: storm starter ExclamationTopology

[Derek Dagit]

In your storm cluster , you need to verify first, nimbus is running
properly or not. Check nimbus.log in $STORM_HOME/logs directory for error
logs.
Also check nimbus.host parameter in ~/.storm/storm.yaml.


Yeah, that's what I was writing in my reply.  I'll go ahead and add below:


Start nimbus again, and make sure it is up.

If your nimbus host is the same host, try (assuming from here nimbus port is 
6627):

```
$ telnet localhost 6627
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
```

If you see that, then nimbus is up-and-running (accepting connections at least).
Check:
- storm.yaml files have correct nimbus.host and nimbus.thrift.port
- firewall settings
- routing (What interface did nimbus open a port on? `netstat -lnt | grep 6627`)


If not, check:
- Make sure likewise ZooKeeper is running.
- logs/nimbus.log (Is there some other issue?)


--
Derek

On 4/24/14, 11:48, Nishu wrote:

In your storm cluster , you need to verify first, nimbus is running
properly or not. Check nimbus.log in $STORM_HOME/logs directory for error
logs.
Also check nimbus.host parameter in ~/.storm/storm.yaml.


On Thu, Apr 24, 2014 at 9:56 PM, Bilal Al Fartakh
wrote:


and the question is , what should I fix dear experts ? :)


2014-04-24 16:23 GMT+00:00 Bilal Al Fartakh :

  ~/src/storm-0.8.1/bin/storm jar

/root/src/storm-starter/target/storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar
storm.starter.ExclamationTopology demo

*I tried to run this and it said that the problem is with the nimbus
connection , but my storm client (and supervisor in the same time ) is
connected with my nimbus (shown in Strom UI )*

Running: java -client -Dstorm.options= -Dstorm.home=/root/src/storm-0.8.1
-Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib -cp
/root/src/storm-0.8.1/storm-0.8.1.jar:/root/src/storm-0.8.1/lib/asm-4.0.jar:/root/src/storm-0.8.1/lib/commons-codec-1.4.jar:/root/src/storm-0.8.1/lib/carbonite-1.5.0.jar:/root/src/storm-0.8.1/lib/kryo-2.17.jar:/root/src/storm-0.8.1/lib/clout-0.4.1.jar:/root/src/storm-0.8.1/lib/clojure-1.4.0.jar:/root/src/storm-0.8.1/lib/ring-servlet-0.3.11.jar:/root/src/storm-0.8.1/lib/hiccup-0.3.6.jar:/root/src/storm-0.8.1/lib/disruptor-2.10.1.jar:/root/src/storm-0.8.1/lib/tools.cli-0.2.2.jar:/root/src/storm-0.8.1/lib/snakeyaml-1.9.jar:/root/src/storm-0.8.1/lib/joda-time-2.0.jar:/root/src/storm-0.8.1/lib/jetty-util-6.1.26.jar:/root/src/storm-0.8.1/lib/commons-exec-1.1.jar:/root/src/storm-0.8.1/lib/jetty-6.1.26.jar:/root/src/storm-0.8.1/lib/servlet-api-2.5.jar:/root/src/storm-0.8.1/lib/jzmq-2.1.0.jar:/root/src/storm-0.8.1/lib/curator-framework-1.0.1.jar:/root/src/storm-0.8.1/lib/httpclient-4.1.1.jar:/root/src/storm-0.8.1/lib/slf4j-log4j12-1.5.8.jar:/root/src/storm-0.8.1/lib/clj-time-0.4.1.jar:/roo

t/src/storm-0.8.1/lib/commons-lang-2.5.jar:/root/src/storm-0.8.1/lib/libthrift7-0.7.0.jar:/root/src/storm-0.8.1/lib/log4j-1.2.16.jar:/root/src/storm-0.8.1/lib/servlet-api-2.5-20081211.jar:/root/src/storm-0.8.1/lib/tools.logging-0.2.3.jar:/root/src/storm-0.8.1/lib/ring-core-0.3.10.jar:/root/src/storm-0.8.1/lib/minlog-1.2.jar:/root/src/storm-0.8.1/lib/objenesis-1.2.jar:/root/src/storm-0.8.1/lib/jline-0.9.94.jar:/root/src/storm-0.8.1/lib/commons-io-1.4.jar:/root/src/storm-0.8.1/lib/ring-jetty-adapter-0.3.11.jar:/root/src/storm-0.8.1/lib/jgrapht-0.8.3.jar:/root/src/storm-0.8.1/lib/json-simple-1.1.jar:/root/src/storm-0.8.1/lib/tools.macro-0.1.0.jar:/root/src/storm-0.8.1/lib/commons-fileupload-1.2.1.jar:/root/src/storm-0.8.1/lib/compojure-0.6.4.jar:/root/src/storm-0.8.1/lib/httpcore-4.1.jar:/root/src/storm-0.8.1/lib/commons-logging-1.1.1.jar:/root/src/storm-0.8.1/lib/guava-13.0.jar:/root/src/storm-0.8.1/lib/curator-client-1.0.1.jar:/root/src/storm-0.8.1/lib/math.numeric-tower-0.0.1.jar:/roo
t/src/storm-0.8.1/lib/junit-3.8.1.jar:/root/src/storm-0.8.1/lib/slf4j-api-1.5.8.jar:/root/src/storm-0.8.1/lib/reflectasm-1.07-shaded.jar:/root/src/storm-0.8.1/lib/core.incubator-0.1.0.jar:/root/src/storm-0.8.1/lib/zookeeper-3.3.3.jar:/root/src/storm-starter/target/storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar:/root/.storm:/root/src/storm-0.8.1/bin

-Dstorm.jar=/root/src/storm-starter/target/storm-starter-0.0.1-SNAPSHOT-jar-with-dependencies.jar
storm.starter.ExclamationTopology demo Exception in thread "main"
java.lang.RuntimeException:
org.apache.thrift7.transport.TTransportException:
java.net.ConnectException: Connection refused at
backtype.storm.utils.NimbusClient.(NimbusClient.java:36) at
backtype.storm.utils.NimbusClient.getConfiguredClient(NimbusClient.java:17)
at backtype.storm.StormSubmitter.submitTopology(StormSubmitter.java:53) at
storm.starter.ExclamationTopology.main(ExclamationTopology.java:59) Caused
by: org.apache.thrift7.transport.TTransportException:
java.net.ConnectException: Connection refused at
org.apache.thrift7.transport.TSocket.open(TSocket.java:183) at
org.apache.thrift7.transport.TFramedTransport.open(TFramedTransport.java:81)
at backtype.storm.utils.NimbusClient

time aggregated data and shared structures efficiency

[Xavier Daull]

I have already developed a Python script (not using storm) which transforms
a stream of millions of prices history of different items (provided in 1
common csv) and output dedicate streams for each item with enriched data in
real-time. This script computes and aggregates in real-time latest item
price with past data to get moving average and slop over different
timeframes (month/week/day/hour) and add to it latest data from nearest
items (neighbours). The goal is to feed models for price prediction. In
order to manage time aggregated data and nearest neighbours data I use a
shared buffer of recent data needed for aggregation, latest computed data
for each item and some shared timestamp indexes.

I am wondering if I would really benefit from moving this script to storm
and how.

My first understanding of storm is I should:
- create a dedicated spout class to fetch prices data.
- create a dedicated bolt class to aggregate data (moving average / slopes
/ cross aggregated data between items).

Where should I put my shared buffers and data required to efficiently
aggregate and compute my time aggregated data and nearest neighbours data ?

Will the topology impact performance compared to in-memory data management
? My current script, even if it is in Python, highly benefits from
efficient buffered computation (no recompute, use delta average...), few
data manipulation, minimum access to memory and computation.

Thank you for your advice.
Xavier

Re: Storm with Drools Complex Event Processing rules

[Nishu]

Hi Eugene,
Thanks for the links. These links are quite useful to go forward in this
direction.
As these books are paid if you have copy of these books available, can you
please share on my mail ID?

Thanks,
Nishu


On Thu, Apr 24, 2014 at 6:59 PM, Eugene  wrote:

> Hi Nishi,
> I will point to you some resources that solve similar problems:
> 1. Book: Storm Real-time processing Cookbook  has
> chapter "Rule-based analysis of the log stream". This recipe integrates the
> JBoss Library and Drools into a bolt and allow to capture event based on
> rule. Next you need sliding window analysis to see if your events frequency
> exceed some threshold. The same book has chapter how to implement "rolling
> window topology" or better yet,
> 2. brand new book "Storm Blueprints " explain how
> to do sliding windows and alerting using Trident ( which I prefer over
> plain storm semantics). Using those 2 resources it should be easy to craft
> solution for your use case.
> good luck
>
>
>
> On Thu, Apr 24, 2014 at 3:36 AM, Nishu  wrote:
>
>> Hi,
>>
>> I am using Storm with Drools Fusion for adding CEP rules to raise system
>> alerts . I need to raise alerts for system if condition is matched for last
>> 10 records or 10 minutes.
>>  Bolt is receiving one event in tuple from Spout. How can I make it work
>> where bolt can read multiple events(lets say 10 events or last 10 minute
>> events) and how to call CEP rules on those event to raise an alert?
>>
>> I checked following thread also :
>> https://groups.google.com/forum/#!topic/storm-user/VqBPnheDtQ8
>>
>> --
>> Thanks,
>> Nishu Tayal
>>
>>
>
>
> --
> Eugene Dvorkin
> Software Engineer
> New York City Storm User Group - organizer
> WebMD
> email: edvor...@gmail.com
> phone: 646-250-9649
> eugenedvorkin.com
> Connect with me on:
> LinkedIn  
> Twitter
>



-- 
with regards,
Nishu Tayal

Trident Stream Join with Self

[Charles LeDoux]

Is it possible to join a trident stream with itself?

My particular use case is that I want to take the cross product of all the
incoming tuples for a batch and then only keep the joined tuples containing
a known value.

I believe the SQL for what I am trying to accomplish is:

SELECT * FROM table AS t1 JOIN table AS t2 ON field1 WHERE t1.field2 =
"known value";

My intention was to do a self join on my stream and then run the now joined
stream through a filter.

Thanks,
Charles

-- 
PhD Candidate; University Fellow
University of Louisiana at Lafayette
Center for Advanced Computer Studies
http://charlesledoux.com

Flush aggregated data every X seconds

[Raphael Hsieh]

Is there a way in Storm Trident to aggregate data over a certain time
period and have it flush the data out to an external data store after that
time period is up ?

Trident does not have the functionality of Tick Tuples yet, so I cannot use
that. Everything I've been researching leads to believe that this is not
possible in Storm/Trident, however this seems to me to be a fairly standard
use case of any streaming map reduce library.

For example,
If I am receiving a stream of integers
I want to aggregate all those integers over a period of 1 second, then
persist it into an external datastore.

This is not in order to count how much it will add up to over X amount of
time, rather I would like to minimize the read/write/updates I do to said
datastore.

There are many ways in order to reduce these variables, however all of them
force me to modify my schema in ways that are unpleasant. Also, I would
rather not have my final external datastore be my scratch space, where my
program is reading/updating/writing and checking to make sure that the
transaction id's line up.
Instead I want that scratch work to be done separately, then the final
result stored into a final database that no longer needs to do constant
updating.

Thanks
-- 
Raphael Hsieh

Re: Flush aggregated data every X seconds

[Corey Nolet]

Raphael, in your case it sounds like a "TickSpout" could be useful where
you emit a tuple every n time slices and then sleep until needing to emit
another. I'm not sure how that'd work in a Trident aggregator, however.

I'm not sure if this is something Nathan or the community would approve of,
but I've been writing my own framework for doing sliding/tumbling windows
in Storm that allow aggregations and triggering/eviction by count, time,
and other policies like "when the time difference between the first item
and the last item in a window is less than x". The bolts could easily be
ripped out for doing your own aggregations.

It's located here: https://github.com/calrissian/flowbox

It's very much so in the proof of concept stage. My other requirement (and
the reason I cared so much to implement this) was that the rules need to be
dynamic and the topology needs to be static as to make the best use of
resources while users are defining that they need.



On Thu, Apr 24, 2014 at 11:27 PM, Raphael Hsieh wrote:

> Is there a way in Storm Trident to aggregate data over a certain time
> period and have it flush the data out to an external data store after that
> time period is up ?
>
> Trident does not have the functionality of Tick Tuples yet, so I cannot
> use that. Everything I've been researching leads to believe that this is
> not possible in Storm/Trident, however this seems to me to be a fairly
> standard use case of any streaming map reduce library.
>
> For example,
> If I am receiving a stream of integers
> I want to aggregate all those integers over a period of 1 second, then
> persist it into an external datastore.
>
> This is not in order to count how much it will add up to over X amount of
> time, rather I would like to minimize the read/write/updates I do to said
> datastore.
>
> There are many ways in order to reduce these variables, however all of
> them force me to modify my schema in ways that are unpleasant. Also, I
> would rather not have my final external datastore be my scratch space,
> where my program is reading/updating/writing and checking to make sure that
> the transaction id's line up.
> Instead I want that scratch work to be done separately, then the final
> result stored into a final database that no longer needs to do constant
> updating.
>
> Thanks
> --
> Raphael Hsieh
>
>
>
>

Re: Flush aggregated data every X seconds

[Raphael Hsieh]

Thank you very much for your quick reply Corey,
Unfortunately I don't believe the TickSpout exists within Trident yet. I
have seen the threads discussing the implementation of a sliding window and
I've read Michael Noll's
blogabout
it as well. I don't need a sliding window, as much as just multiple
window chunks if that makes sense haha.

What I'm thinking about resorting to is increasing my Batch size to be much
larger than the throughput of the spout, then at the end of my topology,
doing an aggregation such that everything aggregates to a single tuple, and
running a ".each" on that single tuple with the function just sleeping for
X time.

My theory is that this should allow the stream to back up enough such that
the next batch takes (roughly) the entire next X time amount of data.

Can anyone validate that this technique will work ?


On Thu, Apr 24, 2014 at 8:36 PM, Corey Nolet  wrote:

> Raphael, in your case it sounds like a "TickSpout" could be useful where
> you emit a tuple every n time slices and then sleep until needing to emit
> another. I'm not sure how that'd work in a Trident aggregator, however.
>
> I'm not sure if this is something Nathan or the community would approve
> of, but I've been writing my own framework for doing sliding/tumbling
> windows in Storm that allow aggregations and triggering/eviction by count,
> time, and other policies like "when the time difference between the first
> item and the last item in a window is less than x". The bolts could easily
> be ripped out for doing your own aggregations.
>
> It's located here: https://github.com/calrissian/flowbox
>
> It's very much so in the proof of concept stage. My other requirement (and
> the reason I cared so much to implement this) was that the rules need to be
> dynamic and the topology needs to be static as to make the best use of
> resources while users are defining that they need.
>
>
>
> On Thu, Apr 24, 2014 at 11:27 PM, Raphael Hsieh wrote:
>
>> Is there a way in Storm Trident to aggregate data over a certain time
>> period and have it flush the data out to an external data store after that
>> time period is up ?
>>
>> Trident does not have the functionality of Tick Tuples yet, so I cannot
>> use that. Everything I've been researching leads to believe that this is
>> not possible in Storm/Trident, however this seems to me to be a fairly
>> standard use case of any streaming map reduce library.
>>
>> For example,
>> If I am receiving a stream of integers
>> I want to aggregate all those integers over a period of 1 second, then
>> persist it into an external datastore.
>>
>> This is not in order to count how much it will add up to over X amount of
>> time, rather I would like to minimize the read/write/updates I do to said
>> datastore.
>>
>> There are many ways in order to reduce these variables, however all of
>> them force me to modify my schema in ways that are unpleasant. Also, I
>> would rather not have my final external datastore be my scratch space,
>> where my program is reading/updating/writing and checking to make sure that
>> the transaction id's line up.
>> Instead I want that scratch work to be done separately, then the final
>> result stored into a final database that no longer needs to do constant
>> updating.
>>
>> Thanks
>> --
>> Raphael Hsieh
>>
>>
>>
>>
>
>


-- 
Raphael Hsieh

Re: Storm dies when zookeeper followers are not available

[Srinath C]

Once you lose the zookeeper quorum, I have seen that the workers keep
throwing exceptions that they are not able to connect to the zookeeper. But
I haven't seen them die because of this. I have even seen them recover once
the quorum is restored.

But if the worker process gets killed, they don't get relaunched because
since the communication between nimbus and supervisor is broken.


On Thu, Apr 24, 2014 at 4:32 PM, Gaurav Sehgal  wrote:

> We have a cluster deployment of Storm and Zookeeper. Where storm is
> running on 2 machines and zookeeper is running on 3 machines.
>
> Now if the zookeeper followers are not available, though the leader is
> available storm keeps throwing an exception that it is not able to connect
> to zookeeper; and ultimately dies.
>
> Has anyone encountered this issue?
>
> Regards,
> Gaurav
>
>