Hi Aleksey, I am guessing you are using proprietary IBM-MQ Apis instead of using the JMS API. Is that right? Is this code open sourced?
Thanks Parth From: "aleksey.di...@ubs.com<mailto:aleksey.di...@ubs.com>" <aleksey.di...@ubs.com<mailto:aleksey.di...@ubs.com>> Reply-To: "user@storm.apache.org<mailto:user@storm.apache.org>" <user@storm.apache.org<mailto:user@storm.apache.org>> Date: Wednesday, April 1, 2015 at 8:48 AM To: "user@storm.apache.org<mailto:user@storm.apache.org>" <user@storm.apache.org<mailto:user@storm.apache.org>> Subject: RE: Using Storm with IBM MQ Series Hello Jeremy, I'm using Storm with IBM MQ for my project a half of a year. In MQ you have no per message acknowledgment – that the problem. You could ack all messages received in current session only altogether. That makes Parth to make his trick with pending list committing only when list is empty. We did we do – made one spout process one message only at the moment: 1. Create TRANSACTED session in spout 2. Create one consumer for this session in spout. 3. Receive message. 4. Emit to bolts. 5. When get ack – commit session. 6. When get fail – rollback session. 7. Start once again from point 2. One session has one message in that case, so we have per message ack. It makes us to scale topology throughput adding more spouts instead of using TOPOLOGY_MAX_SPOUT_PENDING. Personally myself don't like this solution, but I have nothing better now. I have never met limitation of different threads in current solution. HTH, Aleksey. From: jeremy p [mailto:athomewithagroove...@gmail.com] Sent: Wednesday, April 01, 2015 11:19 AM To: user Subject: Re: Using Storm with IBM MQ Series Nathan : thanks for the response! It appears that the engineer who gave this webinar ran into the same problem as Parth : http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/ He reports that he was unable to make MQ Series work with Storm because : "We ran into issues with IBM MQ-Series with respect to how messages are acknowledged by IBM MQ. IBM-MQ requires the thread that receives the message be the same thread that acks it. Storm’s framework cannot support this requirement, as the receiving and acking thread by design are different threads to achieve higher throughput." If threading wasn't the issue, how would you explain Storm's behavior in this situation? --Jeremy On Wed, Apr 1, 2015 at 12:11 AM, Parth Brahmbhatt <pbrahmbh...@hortonworks.com<mailto:pbrahmbh...@hortonworks.com>> wrote: Thanks for correcting me Nathan. Jeremy, in that case you can give storm-jms<https://github.com/ptgoetz/storm-jms> a shot and see if it works for you. Thanks Parth From: Nathan Marz <nat...@nathanmarz.com<mailto:nat...@nathanmarz.com>> Reply-To: "user@storm.apache.org<mailto:user@storm.apache.org>" <user@storm.apache.org<mailto:user@storm.apache.org>> Date: Tuesday, March 31, 2015 at 6:11 PM To: "user@storm.apache.org<mailto:user@storm.apache.org>" <user@storm.apache.org<mailto:user@storm.apache.org>> Subject: Re: Using Storm with IBM MQ Series That's not accurate. A spout task's nextTuple, ack, and fail methods are all called by the exact same thread. You're confusing Storm acks with acks that go back to the source message queue. Storm acks are about detecting tuple DAG completion and have nothing to do with the ack to the source message queue. So if you weren't able to get it to work then it was a different problem. On Tue, Mar 31, 2015 at 2:48 PM, Parth Brahmbhatt <pbrahmbh...@hortonworks.com<mailto:pbrahmbh...@hortonworks.com>> wrote: I tried this once and failed, following are my findings: IBM MQ has a strict limitation the thread that receives the message has to be the thread that acks it. This does not work with storm's threading model where one thread reads the message and sends it to the bolt another thread invokes spout's ack method when the bolt acks the message and only then the spout can ack the message. Based on http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q031020_.htm?lang=en we tried looking into proprietary IBM MQ java client. The acking mechanism relies on following 3 APIS http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#begin() http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#commit() http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#backout() We could use these with one caveat. There is no way to cherry pick what messages, or range of messages like JMS where you can pick the message with lowest id, will get committed. This means in the spout we will keep an active list of pending messages, when ack method in spout is called for a message we remove the message from the pending list and call commit only if the pending list is empty. This works as long as the producer is slow or the consumers are slow but if both of them are really fast the pending list may never become empty causing lot of memory overhead on the box as well as server side overhead of keeping all the messages around. we could introduce a pseudo upper limit in spout where we stop handing out messages once pending message list reaches some upper bound but that will just slow down the topology. In summary, it seems to be impossible to implement a IBM MQ spout that does not run the risk of message loss and be performant. Thanks Parth From: jeremy p <athomewithagroove...@gmail.com<mailto:athomewithagroove...@gmail.com>> Reply-To: "user@storm.apache.org<mailto:user@storm.apache.org>" <user@storm.apache.org<mailto:user@storm.apache.org>> Date: Tuesday, March 31, 2015 at 11:41 AM To: "user@storm.apache.org<mailto:user@storm.apache.org>" <user@storm.apache.org<mailto:user@storm.apache.org>> Subject: Using Storm with IBM MQ Series Hello all, According to this webinar recap, Storm cannot work with IBM MQ Series (also known as Websphere MQ) : http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/ This is a real bummer for me, because my company uses IBM MQ Series, and we have a new project that Storm would be perfect for. Is there currently an effort underway to make IBM MQ Series interoperate with Storm? Do you think it's even possible to make Storm and IBM MQ Series work together? I'd really like to use Storm, if I can. -- Twitter: @nathanmarz http://nathanmarz.com<http://nathanmarz.com/>
