Re: Storm with ActiveMQ

2014-07-02 Thread siddharth ubale 
Hi Peter,

Thank you for helping me out!!!
I went through the provided link and it has helped me a lot. Now i do get
that there will be a intra worker messaging and another messaging system
for pulling data into storm.
Can you also tell me whether i need to have specific knowledge about the
external queing system(RabbitMQ,ActiveMQ,kafka) or it will just be a
connector which i could use to ingest data. I wanna know whether i need to
develop specific skills for the queing systems like RabbitMQ?

Thanks,
Siddharth ubale


On Wed, Jul 2, 2014 at 6:33 PM, Richards Peter hbkricha...@gmail.com
wrote:

 Hi,

 I think you are slightly confused. Let me try to explain it in a simple
 manner. Storm has a transport layer for communication between spouts and
 bolts. At present (release =storm 0.9) there are two transport layer
 implementations provided by storm - zeromq and netty based
 implementations.You can select anyone of them. Installation of zeromq needs
 some special steps. It is already documented in storm project. I haven't
 tried Netty based implementation. However netty based implementation is
 supposed to give you better performance.
 http://yahooeng.tumblr.com/post/64758709722/making-storm-fly-with-netty

 Now let me try to answer your question related to ActiveMQ. For most of
 the realtime usecases people like to process messages from a queue. For
 such usecases the messages are read from these queues by the spout. You can
 use any queuing system(Kafka/RabbitMQ/ActiveMQ/...) for this activity. All
 that you need is a consumer in your spout to read these messages.

 Regards,
 Richards Peter.

Re: Storm with ActiveMQ

2014-07-02 Thread Richards Peter 
Hi,

Well that depends on how you pull data into your storm cluster. If you are
pulling it from a queue, you will need to know how the queuing system
allows you to retrieve the data. You may have standards such as JMS or even
specific implementations as in kafka. Some people even pull data from file
systems and databases. Please take a call based on type of data source,
acceptable latency in fetching data and the cost of having such a system.

Regards,
Richards Peter.

Re: Storm with ActiveMQ

2014-07-02 Thread Michael Ritsema 
We use ActiveMQ with storm in production with a patched version of the JMS
Trident Spout.


On Wed, Jul 2, 2014 at 11:27 AM, Richards Peter hbkricha...@gmail.com
wrote:

 Hi,

 Well that depends on how you pull data into your storm cluster. If you are
 pulling it from a queue, you will need to know how the queuing system
 allows you to retrieve the data. You may have standards such as JMS or even
 specific implementations as in kafka. Some people even pull data from file
 systems and databases. Please take a call based on type of data source,
 acceptable latency in fetching data and the cost of having such a system.

 Regards,
 Richards Peter.