hey Jungtaek, Thanks for continuing to pursue this!
The issue for Storm not working on Mesos is due to a fundamental change to the core scheduling logic in Storm: - https://issues.apache.org/jira/browse/STORM-2126?focusedCommentId=16136150&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16136150 The yet-to-be-ironed-out solution that Bobby was brainstorming about isn't a short term fix as far as I understand it. I believe it to be many many months (years?) out for it to actually be workable. Per my naive understanding of the proposal, we'd probably have to completely rewrite the Storm-on-Mesos framework. So it's probably the right long-term solution, but it isn't anything that should impact this discussion. > The thing is, even users pick storm-kafka-client 1.1.x/1.2.0 and include it into their topology jar, it will also not work with Storm 1.0.x. It even can't compile. FWIW, I'm pretty sure that I was able to successfully run storm-kafka-client-1.1.x on a 1.0.5 storm cluster, but only after shading in storm-core-1.1.x to the topology uber jar. There was *at least* a change to some timer-related class in storm-core in 1.1.x (something about milliseconds IIRC -- it's been 1.5 months since I did it, need to revisit the process I followed). I'm happy to help with backporting / stomping storm-kafka-client in 1.0.x. Maybe we can talk offline about it? - Erik On Mon, Feb 5, 2018 at 4:20 PM, Jungtaek Lim <kabh...@gmail.com> wrote: > UPDATE: Looks like we changed some parts of storm-core while fixing > storm-kafka-client issues (especially went in 1.1.0), hence overwriting > also incurs changes of storm-core. It doesn't look like a big deal for > 1.1.x-branch, but there looks like needed many changes for 1.0.x-branch. > > The thing is, even users pick storm-kafka-client 1.1.x/1.2.0 and include it > into their topology jar, it will also not work with Storm 1.0.x. It even > can't compile. > > 1.0.x version line was long lived (22 months) even we released Storm 1.1.0 > at 11 months ago. Instead of struggling 1.0.x-branch to up to date, I'd > like to suggest that we define 1.0.x-branch as deprecated with guiding to > update to latest 1.1.x version or 1.2.0 (after release), and try to resolve > storm-mesos issue with Storm 1.1.0 ASAP to resolve Erik's concern. > > Makes sense? I'll continue working on 1.1.x-branch and update anyway. > > -Jungtaek Lim (HeartSaVioR) > > 2018년 2월 6일 (화) 오전 7:53, Jungtaek Lim <kabh...@gmail.com>님이 작성: > > > OK. No more opinion/vote in 5 days. I'll treat consensus was made, and go > > ahead making change: overwrite storm-kafka-client 1.2.0 to two branches > > 1.1.x/1.0.x. > > > > -Jungtaek Lim (HeartSaVioR) > > > > 2018년 2월 1일 (목) 오전 10:48, Jungtaek Lim <kabh...@gmail.com>님이 작성: > > > >> This discussion got 4 +1 (binding) and no -1. Moreover two active > >> maintainers for storm-kafka-client (Hugo and Stig) voted +1. > >> > >> Do we want to hold on for hearing more voices, or treating above > opinions > >> as consensus and reflect the change? > >> > >> Btw, I think we need to sort out the sequences between two topics: > >> separating storm-kafka-client as independent release cycle, and this. I > >> guess some of us agreed former topic doesn't related to current RC, but > I > >> think this topic can be (should be) reflected to current RC ongoing. > >> > >> -Jungtaek Lim (HeartSaVioR) > >> > >> 2018년 2월 1일 (목) 오전 4:08, Hugo Da Cruz Louro <hlo...@hortonworks.com>님이 > >> 작성: > >> > >>> +1 to replace storm-kafka-client in 1.0.x branch. > >>> Hugo > >>> > >>> > On Jan 31, 2018, at 11:03 AM, Stig Rohde Døssing < > >>> stigdoess...@gmail.com> wrote: > >>> > > >>> > +1 to replace storm-kafka-client in 1.0.x branch. Breaking semantic > >>> > versioning is really nasty, but I think it is the lesser evil in this > >>> case. > >>> > > >>> > 2018-01-31 5:14 GMT+01:00 Harsha <st...@harsha.io>: > >>> > > >>> >> +1 to replace storm-kafka-client in 1.0.x branch > >>> >> -Harsha > >>> >> On Tue, Jan 30, 2018, at 7:04 PM, Jungtaek Lim wrote: > >>> >>> Bump up this thread so that we could reach consensus earlier. Given > >>> that > >>> >> we > >>> >>> got concern related to this, I think it is ideal to release > >>> 1.1.x/1.0.x > >>> >>> with making decision and applying the change if we want. > >>> >>> > >>> >>> 2018년 1월 30일 (화) 오전 9:25, Jungtaek Lim <kabh...@gmail.com>님이 작성: > >>> >>> > >>> >>>> Erik's concern brought from 1.0.6 RC1, because they can't use > Storm > >>> >> 1.1.0 > >>> >>>> or higher (Storm 1.1.0 broke storm-mesos.). While he could take an > >>> >>>> workaround to use storm-kafka-client 1.2.0 or 1.1.2 (if we decide > to > >>> >>>> replace) with Storm 1.0.6, it would be better if we don't allow > >>> leaving > >>> >>>> storm-kafka-client in 1.0.x in inconsistent state. > >>> >>>> > >>> >>>> IMHO, breaking backward compatibility is worse, but leaving broken > >>> >> thing > >>> >>>> is worst. Hence I'm +1 to replace all, with noticing that it may > >>> bring > >>> >>>> backward incompatibility in release announce. > >>> >>>> > >>> >>>> -Jungtaek Lim (HeartSaVioR) > >>> >>>> > >>> >>>> 2018년 1월 30일 (화) 오전 4:49, P. Taylor Goetz <ptgo...@gmail.com>님이 > 작성: > >>> >>>> > >>> >>>>> As I mentioned else thread I’m open to this but would defer to > >>> >> community > >>> >>>>> consensus. > >>> >>>>> > >>> >>>>> If there’s concern about doing this for 1.0.x, one option would > be > >>> >> skip > >>> >>>>> that version line and only apply it to 1.2.0 and 1.1.x. > >>> >>>>> > >>> >>>>> -Taylor > >>> >>>>> > >>> >>>>>> On Jan 29, 2018, at 12:12 AM, Jungtaek Lim <kabh...@gmail.com> > >>> >> wrote: > >>> >>>>>> > >>> >>>>>> Hi devs, > >>> >>>>>> > >>> >>>>>> This is initial post to separate out discussion topic from vote > >>> >> thread, > >>> >>>>> and > >>> >>>>>> continue discussing. > >>> >>>>>> > >>> >>>>>> Background of the topic: > >>> >>>>>> 1. Only 1.x-branch of storm-kafka-client got stabilized. > >>> >> (relatively) > >>> >>>>>> 2. We would avoid to port back patches to 1.1.x and 1.0.x > because > >>> >>>>> they're > >>> >>>>>> diverged too much. > >>> >>>>>> > >>> >>>>>> Downside: > >>> >>>>>> Backward compatibility might be broken for 1.1.x and 1.0.x. Not > >>> >> sure for > >>> >>>>>> 1.1.x, but at least 1.0.x, since supported Kafka client version > is > >>> >>>>>> different, and if my memory is right, we already applied > backward > >>> >>>>>> incompatible change into storm-kafka-client 1.1.0. > >>> >>>>>> > >>> >>>>>> Please put your opinion regarding topic. You're encouraged to > copy > >>> >> your > >>> >>>>>> previous post in vote thread which helps to centralize opinions > in > >>> >>>>> current > >>> >>>>>> thread. > >>> >>>>>> > >>> >>>>>> Thanks, > >>> >>>>>> Jungtaek Lim (HeartSaVioR) > >>> >>>>> > >>> >>>>> > >>> >> > >>> > >>> >
