We do have both MQ and Storm in-house. However, we actually have another java process running to read from MQ (polling and stopping every x seconds when the queue is empty) and push message to kafka. We thought it was easier to use KafkaSpout then creating another spout. Not saying this is the right way to do it, but we got by with this :)
Thanks, Baek > On Jan 21, 2016, at 6:26 AM, Sameer Kirange <sam.kira...@gmail.com> wrote: > > Hi, > I'm quite new to world of IBM Websphere MQ and need advise on one of the > Integration scenario. > In this scenario, we need to integrate MQ with Apache Storm, We need to read > the MQ in Apache Storm spout so that message can be processed in bolt. > Has anybody worked on this integration and is able to successfully receive > the messages from IBM MQ into Storm for processing ? If yes can some pointers > be provided on how to achieve it ? > > I understand from the various mailing lists and the blog pages from > hortwonworks that that IBM MQ does not support the integration with Apache > Storm. It states following reasons for the same. > "We ran into issues with IBM MQ-Series with respect to how messages are > acknowledged by IBM > MQ. IBM-MQ requires the thread that receives the message be the same thread > that acks it. > Storm's framework cannot support this requirement, as the receiving and > acking thread by > design are different threads to achieve higher throughput. > However when I searched further i came to know from the reply of Nathan Marz > that > That's not accurate. A spout task's nextTuple, ack, and fail methods are all > called by the > exact same thread. You're confusing Storm acks with acks that go back to the > source message > queue. Storm acks are about detecting tuple DAG completion and have nothing > to do with the > ack to the source message queue > Are there any guidelines available to achieve the integration of IBM MQ and > Apache Storm ? > Thanks and Regards, > Sameer >
