I tried this once and failed, following are my findings:
IBM MQ has a strict limitation the thread that receives the message has to be the thread that acks it. This does not work with storm's threading model where one thread reads the message and sends it to the bolt another thread invokes spout's ack method when the bolt acks the message and only then the spout can ack the message. Based on http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q031020_.htm?lang=en we tried looking into proprietary IBM MQ java client. The acking mechanism relies on following 3 APIS http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#begin() http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#commit() http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#backout() We could use these with one caveat. There is no way to cherry pick what messages, or range of messages like JMS where you can pick the message with lowest id, will get committed. This means in the spout we will keep an active list of pending messages, when ack method in spout is called for a message we remove the message from the pending list and call commit only if the pending list is empty. This works as long as the producer is slow or the consumers are slow but if both of them are really fast the pending list may never become empty causing lot of memory overhead on the box as well as server side overhead of keeping all the messages around. we could introduce a pseudo upper limit in spout where we stop handing out messages once pending message list reaches some upper bound but that will just slow down the topology. In summary, it seems to be impossible to implement a IBM MQ spout that does not run the risk of message loss and be performant. Thanks Parth From: jeremy p <athomewithagroove...@gmail.com<mailto:athomewithagroove...@gmail.com>> Reply-To: "user@storm.apache.org<mailto:user@storm.apache.org>" <user@storm.apache.org<mailto:user@storm.apache.org>> Date: Tuesday, March 31, 2015 at 11:41 AM To: "user@storm.apache.org<mailto:user@storm.apache.org>" <user@storm.apache.org<mailto:user@storm.apache.org>> Subject: Using Storm with IBM MQ Series Hello all, According to this webinar recap, Storm cannot work with IBM MQ Series (also known as Websphere MQ) : http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/ This is a real bummer for me, because my company uses IBM MQ Series, and we have a new project that Storm would be perfect for. Is there currently an effort underway to make IBM MQ Series interoperate with Storm? Do you think it's even possible to make Storm and IBM MQ Series work together? I'd really like to use Storm, if I can.
