Hi Jeremy, I was the engineer who gave that webinar. And the statement you quoted is misleading (IIRC, that’s not the wording I used when the question was asked).
Nathan is correct, the nextTuple(), ack() and fail() methods are all called from the same thread. At the moment I don’t have access to the specific errors that were encountered when using the storm-jms spout with IBM-MQ, but it was related that JMS implementation enforcing a strict policy around threads and JMS message acknowledgement (not Storm acks). I’ll try to dig up more details to refresh my memory and get back to you. -Taylor On Apr 1, 2015, at 11:37 AM, Parth Brahmbhatt <pbrahmbh...@hortonworks.com> wrote: > Jeremy, > > Taylor and I work for the same company and the webinar contains the same info > because it is coming from the same source. We will investigate further what > caused the IBM-MQ integration failure but we can not promise a timeline. Once > again apologies for providing inaccurate info in first place. > > Thanks > Parth > > From: jeremy p <athomewithagroove...@gmail.com> > Reply-To: user <user@storm.apache.org> > Date: Wednesday, April 1, 2015 at 8:19 AM > To: user <user@storm.apache.org> > Subject: Re: Using Storm with IBM MQ Series > > Nathan : thanks for the response! It appears that the engineer who gave this > webinar ran into the same problem as Parth : > http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/ > > He reports that he was unable to make MQ Series work with Storm because : > "We ran into issues with IBM MQ-Series with respect to how messages are > acknowledged by IBM MQ. IBM-MQ requires the thread that receives the message > be the same thread that acks it. Storm’s framework cannot support this > requirement, as the receiving and acking thread by design are different > threads to achieve higher throughput." > > If threading wasn't the issue, how would you explain Storm's behavior in this > situation? > > --Jeremy > > On Wed, Apr 1, 2015 at 12:11 AM, Parth Brahmbhatt > <pbrahmbh...@hortonworks.com> wrote: > Thanks for correcting me Nathan. > > Jeremy, in that case you can give storm-jms a shot and see if it works for > you. > > Thanks > Parth > > From: Nathan Marz <nat...@nathanmarz.com> > Reply-To: "user@storm.apache.org" <user@storm.apache.org> > Date: Tuesday, March 31, 2015 at 6:11 PM > To: "user@storm.apache.org" <user@storm.apache.org> > Subject: Re: Using Storm with IBM MQ Series > > That's not accurate. A spout task's nextTuple, ack, and fail methods are all > called by the exact same thread. You're confusing Storm acks with acks that > go back to the source message queue. Storm acks are about detecting tuple DAG > completion and have nothing to do with the ack to the source message queue. > > So if you weren't able to get it to work then it was a different problem. > > On Tue, Mar 31, 2015 at 2:48 PM, Parth Brahmbhatt > <pbrahmbh...@hortonworks.com> wrote: > I tried this once and failed, following are my findings: > > IBM MQ has a strict limitation the thread that receives the message has to be > the thread that acks it. This does not work with storm's threading model > where one thread reads the message and sends it to the bolt another thread > invokes spout's ack method when the bolt acks the message and only then the > spout can ack the message. > Based on > http://www-01.ibm.com/support/knowledgecenter/SSFKSJ_8.0.0/com.ibm.mq.dev.doc/q031020_.htm?lang=en > we tried looking into proprietary IBM MQ java client. The acking mechanism > relies on following 3 APIS > http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#begin() > http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#commit() > http://www-01.ibm.com/support/knowledgecenter/api/content/SSFKSJ_8.0.0/com.ibm.mq.javadoc.doc/WMQJavaClasses/com/ibm/mq/MQQueueManager.html#backout() > We could use these with one caveat. There is no way to cherry pick what > messages, or range of messages like JMS where you can pick the message with > lowest id, will get committed. > This means in the spout we will keep an active list of pending messages, when > ack method in spout is called for a message we remove the message from the > pending list and call commit only if the pending list is empty. This works as > long as the producer is slow or the consumers are slow but if both of them > are really fast the pending list may never become empty causing lot of memory > overhead on the box as well as server side overhead of keeping all the > messages around. > we could introduce a pseudo upper limit in spout where we stop handing out > messages once pending message list reaches some upper bound but that will > just slow down the topology. > In summary, it seems to be impossible to implement a IBM MQ spout that does > not run the risk of message loss and be performant. > > Thanks > Parth > > From: jeremy p <athomewithagroove...@gmail.com> > Reply-To: "user@storm.apache.org" <user@storm.apache.org> > Date: Tuesday, March 31, 2015 at 11:41 AM > To: "user@storm.apache.org" <user@storm.apache.org> > Subject: Using Storm with IBM MQ Series > > Hello all, > > According to this webinar recap, Storm cannot work with IBM MQ Series (also > known as Websphere MQ) : > http://hortonworks.com/blog/discover-hdp-2-2-apache-kafka-apache-storm-stream-data-processing/ > > This is a real bummer for me, because my company uses IBM MQ Series, and we > have a new project that Storm would be perfect for. Is there currently an > effort underway to make IBM MQ Series interoperate with Storm? Do you think > it's even possible to make Storm and IBM MQ Series work together? > > I'd really like to use Storm, if I can. > > > > -- > Twitter: @nathanmarz > http://nathanmarz.com >
signature.asc
Description: Message signed with OpenPGP using GPGMail
