Hello, I can also add that storm kafka client 1.2.0-SNAPSHOT also works pretty well with Storm 1.1.0 : we patched in production our "logs centralization to HBase" topology to use the Spout from storm kafka client 1.2.0-SNAPSHOT and we have super-stable behavior with ~25000 line of logs per second processed with relatively small setup (4 spouts, and a 4-data nodes Hadoop cluster).
I must also confess that, so far, this setup is likely the one we're going to use for up *very soon" next production upgrade, because to our experience, we have very strange behavior of Nimbus UI showing wacky capacity statistics. For example, in our logs topology, our main bolt shows a capacity of 492.583 with a Storm cluster fully based on 1.2.0-SNAPSHOT, while the same topology fed by the same data on a Storm cluster 1.1.0 (but based on Storm kafka client 1.2.0 SNAPSHOT) gets a 0.142 capacity. So far, we don't know if the capacity computed by Storm 1.1.0 is completely erroneous and underrated, or if there's a 1.2.0-SNAPSHOT regression on capacity computation. I have been running out of time to build a sample, so I'm for the moment cowardly considering keeping our cluster at 1.1.0 version with storm kafka client 1.2.0-SNAPSHOT. Disclaimer : I haven't rebuilt Storm 1.2.0 SNAPSHOT full distrib using maven for a long (1,5 month) while, so maybe this weird behavior of "capacity" was fixed in the meantime... hope I'll be able to find time to at least rebuild (upgrading our preproduction test is easy then) or maybe some RC binaries will be available soon? Best regards, Alexandre Vermeerbergen 2018-01-14 15:18 GMT+01:00 Stig Rohde Døssing <stigdoess...@gmail.com>: > I think we're planning to release 1.2.0 once > https://github.com/apache/storm/pull/2203 has been merged. > > Yes, storm-kafka-client 1.2.0-SNAPSHOT will work with Storm 1.1.1. > > 2018-01-14 15:08 GMT+01:00 chandan singh <cks07...@gmail.com>: > >> Stig, >> >> I saw lots of changes in storm-kafka-client 1.x when compared to 1.1.1. >> Should I assume that storm-kafka-client from 1.x will work fine with storm >> release 1.1.1? >> >> Thanks >> Chandan >> >> On Sun, Jan 14, 2018 at 7:29 PM, chandan singh <cks07...@gmail.com> wrote: >> >> > Thanks Stig. >> > >> > I have considered that option but it is more easy to convince others to >> > upgrade to a new release. I did read another mail thread discussing 1.2.0 >> > release but felt 1.1.2 is more close to release than the former. Release >> of >> > 1.2.0 seemed pending in need of more consensus. Do we have an estimate >> when >> > is it coming out? >> > >> > On Sun, Jan 14, 2018 at 2:07 PM, Stig Rohde Døssing < >> > stigdoess...@gmail.com> wrote: >> > >> >> Chandan, >> >> >> >> I don't believe STORM-2549 is fixed in 1.1.2. We're hoping to release >> >> 1.2.0 >> >> soon, but if you need it now it's really easy to check out the Storm >> code >> >> and build storm-kafka-client yourself. Check out the 1.x-branch and do >> >> "mvn >> >> clean install -DskipTests" from the project root. The storm-kafka-client >> >> jar will be in external/storm-kafka-client/target. >> >> >> >> 2018-01-14 9:01 GMT+01:00 chandan singh <cks07...@gmail.com>: >> >> >> >> > +1 Any idea when is 1.1.2v coming out. Under tremendous pressure to >> >> upgrade >> >> > storm-kafka-client due to STORM-2549 >> >> > <https://issues.apache.org/jira/browse/STORM-2549>. Is there any help >> >> > needed towards that goal? >> >> > >> >> > On Tue, Jan 9, 2018 at 5:59 PM, Satish Duggana < >> >> satish.dugg...@gmail.com> >> >> > wrote: >> >> > >> >> > > +1 to start release process for 1.1.2v >> >> > > >> >> > > On Tue, Jan 9, 2018 at 2:15 AM, Stig Rohde Døssing < >> >> > stigdoess...@gmail.com >> >> > > > >> >> > > wrote: >> >> > > >> >> > > > +1 for starting 1.1.2 release process. >> >> > > > >> >> > > > 2018-01-08 20:27 GMT+01:00 P. Taylor Goetz <ptgo...@gmail.com>: >> >> > > > >> >> > > > > +1 >> >> > > > > >> >> > > > > If there are no remaining issues to be included, we can start >> the >> >> > > release >> >> > > > > process. >> >> > > > > >> >> > > > > -Taylor >> >> > > > > >> >> > > > > > On Jan 7, 2018, at 7:07 PM, Jungtaek Lim <kabh...@gmail.com> >> >> > wrote: >> >> > > > > > >> >> > > > > > Bump, does someone have issues which are necessary to be >> >> included >> >> > in >> >> > > > > Storm >> >> > > > > > 1.1.2? If not I think we should start release phase for 1.1.2 >> >> soon. >> >> > > > > > >> >> > > > > > -Jungtaek Lim (HeartSaVioR) >> >> > > > > > >> >> > > > > > 2017년 12월 28일 (목) 오후 3:16, Jungtaek Lim <kabh...@gmail.com>님이 >> >> 작성: >> >> > > > > > >> >> > > > > >> I have been really busy so couldn't care about releases, and >> >> now I >> >> > > got >> >> > > > > >> some time period to track again. >> >> > > > > >> >> >> > > > > >> We have been delaying new release, since we have been >> focusing >> >> on >> >> > > > 1.2.0 >> >> > > > > >> and issues relevant in storm-kafka-client have been raised >> >> > > > continuously. >> >> > > > > >> (though things looks like going to be less critical) >> >> > > > > >> >> >> > > > > >> But other than storm-kafka-client issues, I think Storm 1.1.2 >> >> is >> >> > > ready >> >> > > > > to >> >> > > > > >> be released, and we should release Storm 1.1.2 regardless of >> >> Storm >> >> > > > 1.2.0 >> >> > > > > >> because we have fixed another critical issues in core >> >> > > (STORM-2231[1], >> >> > > > > >> STORM-2682[2]) which are published to 1.0.5 but no release in >> >> > 1.1.x >> >> > > > > version >> >> > > > > >> line yet. >> >> > > > > >> (Noting that some bugfixes on storm-kafka-client are not >> ported >> >> > back >> >> > > > to >> >> > > > > >> 1.1.x version line because of heavy divergence.) >> >> > > > > >> >> >> > > > > >> It may not be good time to discuss since it is year-end now, >> >> but >> >> > > would >> >> > > > > >> like to remind this so that we could start the process at >> least >> >> > > > earlier >> >> > > > > in >> >> > > > > >> next year. >> >> > > > > >> >> >> > > > > >> Thanks, >> >> > > > > >> Jungtaek Lim (HeartSaVioR) >> >> > > > > >> >> >> > > > > >> 1. https://issues.apache.org/jira/browse/STORM-2231 >> >> > > > > >> 2. https://issues.apache.org/jira/browse/STORM-2682 >> >> > > > > >> >> >> > > > > >> 2017년 10월 19일 (목) 오전 1:19, Stig Rohde Døssing < >> >> > > stigdoess...@gmail.com >> >> > > > > >님이 >> >> > > > > >> 작성: >> >> > > > > >> >> >> > > > > >>> Looks like Hugo is working on it >> >> > > > > >>> https://issues.apache.org/jira/browse/STORM-2781 >> >> > > > > >>> >> >> > > > > >>> 2017-10-18 4:22 GMT+02:00 Jungtaek Lim <kabh...@gmail.com>: >> >> > > > > >>> >> >> > > > > >>>>> I'm hoping the delay for 1.2.0 will be very short. The >> >> changes >> >> > we >> >> > > > > >>>> discussed >> >> > > > > >>>>> were minor, and had to do with renaming some of the new >> >> methods >> >> > > and >> >> > > > > >>>>> constants. It would be good to do before 1.2.0 because the >> >> > > renames >> >> > > > > are >> >> > > > > >>>>> breaking changes. >> >> > > > > >>>> >> >> > > > > >>>> Suppose the changes will be minor, then why not go ahead >> >> making >> >> > > the >> >> > > > > >>> change? >> >> > > > > >>>> I guess it doesn't need much efforts to do. >> >> > > > > >>>> (Will Hugo submit the patch?) >> >> > > > > >>>> >> >> > > > > >>>> I'm +1 to get it before 1.2.0, and also hope that the delay >> >> will >> >> > > be >> >> > > > > very >> >> > > > > >>>> short. >> >> > > > > >>>> >> >> > > > > >>>> 2017년 10월 14일 (토) 오후 6:05, Alexandre Vermeerbergen < >> >> > > > > >>>> avermeerber...@gmail.com>님이 >> >> > > > > >>>> 작성: >> >> > > > > >>>> >> >> > > > > >>>>> +1 for a short delay until 1.2.0 is avaible :) >> >> > > > > >>>>> >> >> > > > > >>>>> 2017-10-14 8:48 GMT+02:00 Stig Rohde Døssing < >> >> > > > stigdoess...@gmail.com >> >> > > > > >>>> : >> >> > > > > >>>>> >> >> > > > > >>>>>> I'm hoping the delay for 1.2.0 will be very short. The >> >> changes >> >> > > we >> >> > > > > >>>>> discussed >> >> > > > > >>>>>> were minor, and had to do with renaming some of the new >> >> > methods >> >> > > > and >> >> > > > > >>>>>> constants. It would be good to do before 1.2.0 because >> the >> >> > > renames >> >> > > > > >>> are >> >> > > > > >>>>>> breaking changes. >> >> > > > > >>>>>> >> >> > > > > >>>>>> 2017-10-14 5:33 GMT+02:00 Arun Mahadevan < >> ar...@apache.org >> >> >: >> >> > > > > >>>>>> >> >> > > > > >>>>>>> I was hoping we will get 1.2.0 out along with 1.1.2. The >> >> > > pending >> >> > > > > >>>> issues >> >> > > > > >>>>>> in >> >> > > > > >>>>>>> the epic https://issues.apache.org/ >> jira/browse/STORM-2710 >> >> > > seems >> >> > > > > >>> to >> >> > > > > >>>>> have >> >> > > > > >>>>>>> been addressed. Can you add the new issue to the epic? >> >> > > > > >>>>>>> >> >> > > > > >>>>>>> If its not something critical we can do it in a minor >> >> release >> >> > > > post >> >> > > > > >>>>> 1.2.0. >> >> > > > > >>>>>>> >> >> > > > > >>>>>>> Thanks, >> >> > > > > >>>>>>> Arun >> >> > > > > >>>>>>> >> >> > > > > >>>>>>> >> >> > > > > >>>>>>> On 10/14/17, 3:50 AM, "Hugo Da Cruz Louro" < >> >> > > > > >>> hlo...@hortonworks.com> >> >> > > > > >>>>>> wrote: >> >> > > > > >>>>>>> >> >> > > > > >>>>>>>> I am +1 to releasing 1.1.2 right away. I am in the >> >> middle of >> >> > > one >> >> > > > > >>>>> review >> >> > > > > >>>>>>> but I will finish it in the next day, such that we can >> get >> >> > this >> >> > > > > >>>> merged >> >> > > > > >>>>>> soon. >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> However, we need to hold onto releasing 1.2.0 until >> some >> >> of >> >> > > the >> >> > > > > >>>>> changes >> >> > > > > >>>>>>> for ProcessingGuarantee that got in this patch< >> >> > > > > >>> https://github.com/ >> >> > > > > >>>>>>> apache/storm/commit/48f6969027e7b02a5b9220577189d3 >> >> > 911aa2226d> >> >> > > > are >> >> > > > > >>>>> fixed. >> >> > > > > >>>>>>> I briefly discussed [1] this issue with @Stig on >> Gitter, I >> >> > will >> >> > > > > >>>> submit >> >> > > > > >>>>> a >> >> > > > > >>>>>>> patch with the change. >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> Thanks, >> >> > > > > >>>>>>>> Hugo >> >> > > > > >>>>>>>> [1] - We did not have a technical discussion. I just >> >> asked a >> >> > > > > >>> couple >> >> > > > > >>>> of >> >> > > > > >>>>>>> clarifying questions and then the idea surged that we >> >> should >> >> > > > > >>> improve >> >> > > > > >>>>> some >> >> > > > > >>>>>>> of the changes in this patch<https://github.com/ >> >> > > > > >>>> apache/storm/commit/ >> >> > > > > >>>>>>> 48f6969027e7b02a5b9220577189d3911aa2226d>. I will >> create >> >> a >> >> > > JIRA, >> >> > > > > >>> and >> >> > > > > >>>>> all >> >> > > > > >>>>>>> the discussion go through either JIRA or dev email list. >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> On Oct 10, 2017, at 12:48 PM, Stig Rohde Døssing < >> >> > > > > >>>>>> stigdoess...@gmail.com< >> >> > > > > >>>>>>> mailto:stigdoess...@gmail.com>> wrote: >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> Thanks Jungtaek, that sounds like a good plan. Here's >> the >> >> > new >> >> > > PR >> >> > > > > >>> for >> >> > > > > >>>>>> 2607 >> >> > > > > >>>>>>>> https://github.com/apache/storm/pull/2367. >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> Beginning release next week sounds good to me. >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> 2017-10-10 17:42 GMT+02:00 Arun Mahadevan < >> >> ar...@apache.org >> >> > > > > >>> <mailto: >> >> > > > > >>>>>> arunm >> >> > > > > >>>>>>> @apache.org>>: >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> +1 for addressing the pending reviews and getting 1.2.0 >> >> out >> >> > > > soon. >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> On 10/10/17, 6:14 AM, "Jungtaek Lim" < >> kabh...@gmail.com >> >> > > <mailto: >> >> > > > > >>> kabh >> >> > > > > >>>>>>> w...@gmail.com>> wrote: >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> Stig, >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> Let's just handle all the issues pending Storm 1.1.2. >> For >> >> > > > pending >> >> > > > > >>>>> issues >> >> > > > > >>>>>>>> on >> >> > > > > >>>>>>>> Storm 1.2.0, I already handled all the things. >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> For STORM-2607, could you just take over and craft a >> new >> >> > pull >> >> > > > > >>>> request? >> >> > > > > >>>>>> We >> >> > > > > >>>>>>>> are waiting more than 2 months after requesting simple >> >> > rebase >> >> > > > > >>> (sadly >> >> > > > > >>>>> it >> >> > > > > >>>>>> is >> >> > > > > >>>>>>>> not done yet), which I don't think it's acceptable. >> That >> >> > issue >> >> > > > > >>>>> relates a >> >> > > > > >>>>>>>> bug which we should handle it in time. >> >> > > > > >>>>>>>> (The patch includes your work indeed.) >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> For STORM-2549, let's see someone could review in this >> >> week. >> >> > > > I'll >> >> > > > > >>>> try >> >> > > > > >>>>> to >> >> > > > > >>>>>>>> get it too. >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> Then I think we can start release phase for Storm 1.1.2 >> >> and >> >> > > > > >>> 1.2.0 at >> >> > > > > >>>>>> next >> >> > > > > >>>>>>>> week. Opinions anyone? >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> Thanks, >> >> > > > > >>>>>>>> Jungtaek Lim (HeartSaVioR) >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> 2017년 10월 10일 (화) 오전 4:02, Stig Rohde Døssing < >> >> > > > > >>>> stigdoess...@gmail.com >> >> > > > > >>>>> < >> >> > > > > >>>>>>> mailto:stigdoess...@gmail.com>>님이 >> >> > > > > >>>>>>>> 작성: >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> Maybe we would be better off releasing 1.1.2 as is, and >> >> > > > > >>> postponing >> >> > > > > >>>> the >> >> > > > > >>>>>>>> other issues to 1.2.0? I don't think we should delay >> the >> >> fix >> >> > > for >> >> > > > > >>>>>>>> https://issues.apache.org/jira/browse/STORM-2682 for >> >> much >> >> > > > > >>> longer. >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> 2017-09-22 14:50 GMT+02:00 Alexandre Vermeerbergen < >> >> > > > > >>>>>>>> avermeerber...@gmail.com<mailto:avermeerbergen@gmail. >> com >> >> > >> >> > > > > >>>>>>>> : >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> Hello, >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> I don't know if that help, but we're still waiting with >> >> lots >> >> > > of >> >> > > > > >>>>>>>> expectations https://issues.apache.org/ >> >> > jira/browse/STORM-2648 >> >> > > > > >>> with >> >> > > > > >>>>>>>> Storm >> >> > > > > >>>>>>>> 1.2.0 ! >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> Best regards, >> >> > > > > >>>>>>>> Alexandre Vermeerbergen >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> 2017-09-22 12:24 GMT+02:00 Jungtaek Lim < >> >> kabh...@gmail.com >> >> > > > > >>> <mailto: >> >> > > > > >>>>> kabh >> >> > > > > >>>>>>> w...@gmail.com>>: >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> Looks like three weeks went by from initiating the >> >> thread. >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> I'm seeing some issues pending for review and all of >> them >> >> > are >> >> > > > > >>>>>>>> regarding >> >> > > > > >>>>>>>> storm-kafka-client. >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> Remaining issues are below: >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> Storm 1.1.2 >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> https://issues.apache.org/jira/browse/STORM-2549 >> >> > > > > >>>>>>>> https://issues.apache.org/jira/browse/STORM-2607 >> >> > > > > >>>>>>>> https://issues.apache.org/jira/browse/STORM-2666 >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> Storm 1.2.0 >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> https://issues.apache.org/jira/browse/STORM-2648 >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> Please note that above issues are 'effectively' blocker >> >> for >> >> > > > > >>>>>>>> releases. >> >> > > > > >>>>>>>> Like >> >> > > > > >>>>>>>> I said Storm 1.1.1 has critical issue which is fixed >> and >> >> > will >> >> > > be >> >> > > > > >>>>>>>> available >> >> > > > > >>>>>>>> at Storm 1.1.2, so at least I'd like to see the >> progress >> >> on >> >> > > > Storm >> >> > > > > >>>>>>>> 1.1.2, >> >> > > > > >>>>>>>> and ideally with Storm 1.2.0 since there's only one >> issue >> >> > left >> >> > > > on >> >> > > > > >>>>>>>> epic. >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> Please finish reviewing if you are in reviewing one or >> >> more >> >> > of >> >> > > > > >>> them. >> >> > > > > >>>>>>>> I'll >> >> > > > > >>>>>>>> try to start reviewing them but take some times since >> I'm >> >> > not >> >> > > > > >>>>>>>> familiar >> >> > > > > >>>>>>>> with >> >> > > > > >>>>>>>> that module. >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> Thanks, >> >> > > > > >>>>>>>> Jungtaek Lim (HeartSaVioR) >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> 2017년 8월 30일 (수) 오전 2:45, P. Taylor Goetz < >> >> > ptgo...@gmail.com >> >> > > > > >>>> <mailto: >> >> > > > > >>>>>> ptgo >> >> > > > > >>>>>>> e...@gmail.com>>님이 작성: >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> It looks to me like 1.0.5 is ready for a release >> >> candidate >> >> > > > (still >> >> > > > > >>>>>>>> some >> >> > > > > >>>>>>>> ongoing work for 1.1.2, but likely soon). >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> Is there anything else we would want to include in >> 1.0.5 >> >> or >> >> > > > > >>>>>>>> should we >> >> > > > > >>>>>>>> go >> >> > > > > >>>>>>>> ahead with a release? >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> -Taylor >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> On Aug 25, 2017, at 3:26 AM, Jungtaek Lim < >> >> > kabh...@gmail.com >> >> > > > > >>>> <mailto: >> >> > > > > >>>>>> kabh >> >> > > > > >>>>>>> w...@gmail.com>> >> >> > > > > >>>>>>>> wrote: >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> Hi devs, >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> We received a bug report (STORM-2682 >> >> > > > > >>>>>>>> <https://issues.apache.org/jira/browse/STORM-2682>) on >> >> > Storm >> >> > > > > >>>>>>>> 1.0.4 >> >> > > > > >>>>>>>> and >> >> > > > > >>>>>>>> 1.1.1 which prevents Storm cluster from update. >> >> Personally >> >> > it >> >> > > > > >>>>>>>> looks >> >> > > > > >>>>>>>> like >> >> > > > > >>>>>>>> pretty critical, and hopefully it is fixed now. >> >> > > > > >>>>>>>> So maybe we would like to have another bug fix releases >> >> > > quickly >> >> > > > > >>>>>>>> for >> >> > > > > >>>>>>>> affected 1.x version lines. What do you think? >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> Also please enumerate the issues if you would want to >> >> > include >> >> > > > > >>>>>>>> any >> >> > > > > >>>>>>>> bug >> >> > > > > >>>>>>>> fix >> >> > > > > >>>>>>>> issues to the new bug fix releases, so that we can >> create >> >> > epic >> >> > > > > >>>>>>>> issues >> >> > > > > >>>>>>>> and >> >> > > > > >>>>>>>> track them to make releases happening sooner. >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> Thanks, >> >> > > > > >>>>>>>> Jungtaek Lim (HeartSaVioR) >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>>> >> >> > > > > >>>>>>> >> >> > > > > >>>>>>> >> >> > > > > >>>>>> >> >> > > > > >>>>> >> >> > > > > >>>> >> >> > > > > >>> >> >> > > > > >> >> >> > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> >> > >> > >>
