Hello Stig,
Here's my first feedback on this Storm 1.2.0 preview on my Supervision
system based on Storm : I have 11 topologies KO (not even able to start),
and 4 topologies which seem to be unaffected.
Details:
- I used the binaries posted on dropbox by Jungteak and your binary for
storm-kafka-client
- Rebuilt all our topologies using these storm-core.jar &
storm-kafka-client.jar from these storm-1.2.0-snapshot (and I got
absolutely no error at build time).
Meaning that I have new "big jar" files including this newer
storm-kafka-client.jar (except for 1 topology which depends on another
build system)
- The 11 topologies which fail to start show up the following trace when
they are submitted:
Running: java -client -Ddaemon.name= -Dstorm.options=
-Dstorm.home=/usr/local/Storm/storm-stable
-Dstorm.log.dir=/usr/local/Storm/storm-stable/logs
-Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib
-Dstorm.conf.file= -cp
/usr/local/Storm/storm-stable/*:/usr/local/Storm/storm-stable/lib/*:/usr/local/Storm/storm-stable/extlib/*:/usr/local/Storm/StormTopologiesBJ.jar:/usr/local/Storm/storm-stable/conf:/usr/local/Storm/storm-stable/bin
-Dstorm.jar=/usr/local/Storm/StormTopologiesBJ.jar -Dstorm.dependency.jars=
-Dstorm.dependency.artifacts={}
com.acme.storm.evaluator.SLAEventsInterceptorTopology slaEventsInterceptor
Latest
1022 [main] INFO c.a.s.e.SLAEventsInterceptorTopology - Topology :
'slaEventsInterceptor', KafkaOffsetTimeStrategy = 'Latest'
1026 [main] WARN c.a.s.u.StormUtils - Couldn't read
/usr/local/Storm/flux/slaEventsInterceptor_kafkaConsumer.properties file:
/usr/local/Storm/flux/slaEventsInterceptor_kafkaConsumer.properties (No
such file or directory)
1037 [main] WARN o.a.s.k.s.KafkaSpoutConfig - Do not set
enable.auto.commit manually. Instead use
KafkaSpoutConfig.Builder.setProcessingGuarantee. This will be treated as an
error in the next major release. For now the spout will be configured to
behave like it would have in pre-1.2.0 releases.
Exception in thread "main" java.lang.ClassCastException: java.lang.String
cannot be cast to java.lang.Boolean
at
org.apache.storm.kafka.spout.KafkaSpoutConfig.setAutoCommitMode(KafkaSpoutConfig.java:721)
at
org.apache.storm.kafka.spout.KafkaSpoutConfig.<init>(KafkaSpoutConfig.java:97)
at
org.apache.storm.kafka.spout.KafkaSpoutConfig$Builder.build(KafkaSpoutConfig.java:671)
at
com.acme.storm.evaluator.SLAEventsInterceptorTopology.main(SLAEventsInterceptorTopology.java:118)
The line part of our application which triggered above stack trace is the
following one:
KafkaSpoutConfig<String, Event> spoutConfigForMainTopic =
KafkaSpoutConfig
.builder(elasticKafkaBrokers, KafkaTopics.MAIN)
.setValue(EventKafkaDeserializer.class)
.setGroupId(consumerId + "_" + KafkaTopics.MAIN)
.setFirstPollOffsetStrategy(strategy)
.setProp(kafkaConsumerProp)
.setRecordTranslator(
new SLAEventsInterceptorKafkaRecordTranslator())
.build();
I understand that there's a WARN message, but if it's a warning, then why
do we exit with such a brutal exception?
Best regards,
Alexandre Vermeerbergen
2017-11-18 14:54 GMT+01:00 Stig Rohde Døssing <stigdoess...@gmail.com>:
> Alexandre,
>
> I've uploaded the 1.2.0-SNAPSHOT storm-kafka-client jar here
> https://drive.google.com/file/d/1DgJWjhWwczYgZS82YGd63V3GT2G_
> v9fd/view?usp=sharing.
> You can probably keep the 1.1.0 versions of storm-hbase and storm-hdfs,
> since the core Storm API hasn't changed.
>
> If you want to build the jars yourself, clone the repository at
> https://github.com/apache/storm and check out the 1.x-branch branch. You
> can build the entire project by running "mvn clean install -DskipTests" in
> the root. The individual module jars will be available in the target
> directories for each module, e.g. storm-kafka-client.jar will be in
> external/storm-kafka-client/target.
>
> If you need to build the regular Storm distribution (the tarball Jungtaek
> sent you), you can go to storm-dist/binary and run "mvn clean install
> -Dgpg.skip". The tar/zip will be in storm-dist/binary/target.
>
> Further reference in case you need it
> https://github.com/apache/storm/blob/master/DEVELOPER.
> md#build-the-code-and-run-the-tests
> .
>
> 2017-11-18 12:57 GMT+01:00 Alexandre Vermeerbergen <
> avermeerber...@gmail.com
> >:
>
> > Hello Jungtaek,
> >
> > Thanks for the link to 1.2.0 preview binaries.
> > However, since we heavily depend on Storm Kafka Client, would you please
> > either add it or remind me how to build the jars of this external lib, at
> > same "snapshot" version as the rest ?
> > On a side note, some of our topologies also rely on storm hbase & storm
> > hdfs : does it matters if these later ones stay in 1.1.0 version for this
> > test?
> > Indeed, I want to focus on Storm 1.2.0 "core" + Storm Kafka Client 1.2.0
> >
> > Best regards,
> > Alexandre Vermeerbergen
> >
> >
> > 2017-11-18 7:41 GMT+01:00 Jungtaek Lim <kabh...@gmail.com>:
> >
> > > Alexandre,
> > >
> > > https://www.dropbox.com/s/mg2gnunk24oesyc/apache-storm-
> > > 1.2.0-SNAPSHOT.tar.gz?dl=0
> > >
> > > Above link is custom binary distribution of current 1.x-branch
> (SNAPSHOT
> > of
> > > 1.2.0). Could you run the test in your environment first and let us
> know
> > > about the result? Regardless of including metrics V2 to 1.2.0 or not,
> > your
> > > test report should be valuable for us.
> > >
> > > And please refer my analysis of current metrics:
> > > https://cwiki.apache.org/confluence/display/STORM/
> > Limitations+of+current+
> > > metrics+feature
> > > to
> > > see why we want to move toward to Metrics V2. Current PR on Metrics V2
> is
> > > for initial state and it could cover only some of issues in list, but I
> > > expect we will address another issues as well based on the PR.
> > >
> > > Thanks in advance!
> > > Jungtaek Lim (HeartSaVioR)
> > >
> > > 2017년 11월 18일 (토) 오전 6:40, Alexandre Vermeerbergen <
> > > avermeerber...@gmail.com>님이
> > > 작성:
> > >
> > > > Hello Hugo,
> > > >
> > > > As I already posted, I was getting ready to test the upcoming RC, so
> > > yeah :
> > > > as soon as I have some binaries for testing, I can run any Storm
> 1.2.0
> > > > preview on my pre-production (well stressed) environment. Is there
> some
> > > URL
> > > > from where I can download such "snapshot" binaries?
> > > >
> > > >
> > > > Best regards,
> > > > Alexandre Vermeerbergen
> > > >
> > > > PS: By the way, I have no interest in the metrics-related
> enhancements
> > > you
> > > > mentionned - I can understand it is of interest for people aware of
> it
> > > and
> > > > who are expecting "something" related to it - but I would be sad to
> > see a
> > > > postponing of the imminent Storm 1.2.0 release for something we do
> not
> > > use
> > > > at all.
> > > > PPS: I would love to learn more about these "metrics", maybe it's
> > > something
> > > > worth which never quite understood... any link with clear explanation
> > of
> > > > this feature?
> > > >
> > > >
> > > > 2017-11-17 17:44 GMT+01:00 Hugo Da Cruz Louro <
> hlo...@hortonworks.com
> > >:
> > > >
> > > > > I am also in agreement that we should not delay the 1.2.0 release
> for
> > > too
> > > > > long, but in order to release it a few things need to be kept in
> > mind:
> > > > >
> > > > > - If we want to avoid releasing 1.3.0 soon or at all, that means
> > that
> > > we
> > > > > should include in 1.2.0 as many important features (e.g. metricsV2)
> > as
> > > > > possible within a reasonable time frame. Also to avoid as much back
> > > > porting
> > > > > as possible one should really get all features that we foresee we
> > want
> > > to
> > > > > maintain in 1.2.0 right away, and then simply maintain them there.
> > > > However
> > > > > it’s a fact of life that if a bug is found and it is a blocker, it
> > will
> > > > > have to be back ported.
> > > > >
> > > > > - STORM-2153 details that the new metrics requirements were driven
> by
> > > the
> > > > > users. Therefore if this feature is highly important and sought
> > after,
> > > it
> > > > > may not make much sense to release 1.2.0 without it. If we do
> release
> > > > 1.2.0
> > > > > without the metrics, does it mean that metrics will go only in
> 2.0? I
> > > > would
> > > > > conjecture that most production deployments will take a while to
> > > upgrade
> > > > to
> > > > > 2.0 even after it is released. That means that they will still be
> > > running
> > > > > Storm without the benefit of the new metrics.
> > > > >
> > > > > - Several fixes have got in storm-kafka-client. There have been a
> > lot
> > > of
> > > > > changes and I wonder to which degree they have been system tested
> in
> > > > > addition the existing unit tests
> > > > >
> > > > > @Alexandre, since you are using storm-kafka-client and have filed
> > some
> > > > > bugs and driven some feature requests, I would like to ask if you
> > could
> > > > > help us within what is reasonably possible for you with the
> > following:
> > > > >
> > > > > - system test the latest storm-kafka-client changes in your
> > > > > test/pre-production environment
> > > > > - provide some information on about your setup,
> > > > > - how you are using storm-kafka-client
> > > > > - how it is performing
> > > > > - Kafka brokers, number of topics/partitions, storm
> > parallelism,
> > > > and
> > > > > some info about your network
> > > > > - if it is reasonable to do so share some of your tests such that
> we
> > > can
> > > > > also test it at our end.
> > > > > - Tell us specific things that you would like us to test
> > > > >
> > > > > Thanks,
> > > > > Hugo
> > > > >
> > > > > > On Nov 16, 2017, at 11:49 PM, Stig Rohde Døssing <
> > > > stigdoess...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > I agree with Jungtaek, if metrics v2 can go in very soon it
> should
> > go
> > > > in,
> > > > > > otherwise I'd rather release 1.2.0 now and work on getting 2.0.0
> > > ready
> > > > > for
> > > > > > release.
> > > > > >
> > > > > > 1.x and master have drifted pretty far apart, and it's causing a
> > lot
> > > of
> > > > > > porting work at this point. 1.x still has a lot of Clojure code,
> > > > > storm-core
> > > > > > hasn't been split yet in that branch and it's also targeting JDK
> > 1.7.
> > > > > It's
> > > > > > seems rare at this point that a PR cherry picks cleanly onto 1.x
> > from
> > > > > > master.
> > > > > >
> > > > > > I've linked the list of issues that are only fixed in 2.0.0 and
> not
> > > > 1.x,
> > > > > to
> > > > > > illustrate how far ahead 2.0.0 is.
> > > > > >
> > > > > > https://pste.eu/p/9CJT.html
> > > > > >
> > > > > > 2017-11-17 5:41 GMT+01:00 Arun Iyer <ai...@hortonworks.com>:
> > > > > >
> > > > > >> Hi Taylor,
> > > > > >>
> > > > > >> Is it https://github.com/apache/storm/pull/2203 ?
> > > > > >>
> > > > > >>
> > > > > >> I think it would be great to get this in 1.2 release. Can we try
> > to
> > > > > >> address the issues in a week or so and get this in?
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Arun
> > > > > >>
> > > > > >> On 11/17/17, 7:13 AM, "P. Taylor Goetz" <ptgo...@gmail.com>
> > wrote:
> > > > > >>
> > > > > >>> The original idea for the 1.2 release was that would introduce
> > the
> > > > > >> metrics v2 work (and there is additional work required there,
> so I
> > > > > >> understand a desire not to delay the release). Do we want to
> stick
> > > > with
> > > > > >> that, or deviate? If the latter, would we do a 1.3 release for
> > > > metrics?
> > > > > >>>
> > > > > >>> As far as a 1.1.2 release I’m fine with releasing that at any
> > time.
> > > > > >>>
> > > > > >>> -Taylor
> > > > > >>>
> > > > > >>>> On Nov 15, 2017, at 6:24 PM, Jungtaek Lim <kabh...@gmail.com>
> > > > wrote:
> > > > > >>>>
> > > > > >>>> I think we could start release phase for both 1.1.2 and 1.2.0
> > when
> > > > > >>>> https://github.com/apache/storm/pull/2423 will be merged.
> > > > > >>>>
> > > > > >>>> Thanks,
> > > > > >>>> Jungtaek Lim (HeartSaVioR)
> > > > > >>>>
> > > > > >>>> 2017년 11월 16일 (목) 오전 7:00, Alexandre Vermeerbergen <
> > > > > >> avermeerber...@gmail.com>님이
> > > > > >>>> 작성:
> > > > > >>>>
> > > > > >>>>> Hello,
> > > > > >>>>>
> > > > > >>>>> I'd love to see a Storm 1.2.0 released: it's a perfect
> schedule
> > > for
> > > > > me
> > > > > >> for
> > > > > >>>>> test any Release Candidate that might be available if it
> > happens
> > > > > soon.
> > > > > >>>>>
> > > > > >>>>> Best regards,
> > > > > >>>>> Alexandre
> > > > > >>>>>
> > > > > >>>>> 2017-11-15 7:29 GMT+01:00 Arun Mahadevan <ar...@apache.org>:
> > > > > >>>>>
> > > > > >>>>>> Hi,
> > > > > >>>>>>
> > > > > >>>>>> Looks like we are only waiting on
> https://issues.apache.org/
> > > > > >>>>>> jira/browse/STORM-2546 .
> > > > > >>>>>>
> > > > > >>>>>> Are there any other issues which are blockers for Storm
> 1.2.0?
> > > > Would
> > > > > >> be
> > > > > >>>>>> great to see the 1.2.0 release out soon as it has a lot of
> > > > critical
> > > > > >>>>> fixes.
> > > > > >>>>>>
> > > > > >>>>>> Thanks,
> > > > > >>>>>> Arun
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>
> > > > > >>>
> > > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>
