Hello, +1: I have finally (after having understood how to generate the "external" .jar files) been able to test our Supervision topologies (10 topologies) on a small cluster: * 1 VM dedicated to Nimbus * 2 VM dedicated to Supervisor * 3 VM dedicated to Zookeeper * Connected to a Kafka 0.10.1.1 cluster using storm-kafka spout * All running with Java 8 update 121 & CentOS 6.5
We have no functional regressions and our topologies's performances seem equal to same setup based on Storm 1.0.3 RTM Thanks to all who answered my naive questions about the external .jar files. I hope the documentation that'll come with Storm 1.1.0 will help newcomers avoiding the pitfalls I experienced with this RC3. Best regards, Alexandre 2017-03-27 12:54 GMT+02:00 Jungtaek Lim <[email protected]>: > Sorry I was not aware of the requirement of build for Flux. > > Unfortunately there seemed to be build error on Flux because Flux test case > requires "node" (NodeJS). > I think we would be better to remove the requirement since it might also > blocks tests running on ASF Jenkins. > > You can do "mvn clean install -DskipTests" to ignore the test, or install > "node" to make your machine fulfill the requirement. > > Please let me know if neither work for you. > > - Jungtaek Lim (HeartSaVioR) > > 2017년 3월 27일 (월) 오후 7:33, Alexandre Vermeerbergen < > [email protected]>님이 > 작성: > > > More about this: > > > > - I can see in110rc3 source directories that external/flux/flux-core > > exists, however 'mvn clean install' did not generate flux-core-1.1.10.jar > > in external/flux/flux-core/target/ but it doesn't seem like there's a > build > > error > > > > - I have attached the logs (gzip) of the output of 'mvn clean install' > if > > it can help finding out why flux-core-1.1.10.jar hasn't been generated > > > > Best regards, > > Alexandre Vermeerbergen > > > > > > 2017-03-27 11:43 GMT+02:00 Alexandre Vermeerbergen < > > [email protected]>: > > > > Hello, > > > > I was able to build storm-kakfa-1.1.0.jar using "mvn clean install" > > started from root directory of expanded archive of storm-1.1.0 RC3 source > > distribution. > > > > But now my tests are failing before I'm missing the following JAR file: > > > > flux-core-1.1.0.jar > > > > I am sure that it was part of Storm 1.1.0rc2. Why isn't this file part of > > RC3 ? > > It is deprecated? > > > > Note: we copy flux-core*.jar and flux-wrappers*.jar into > > > > storm-stable/extlib > > > > to use flux in our topologies. > > > > best regards, > > Alexandre Vermeerbergen > > > > > > 2017-03-27 7:58 GMT+02:00 Jungtaek Lim <[email protected]>: > > > > You can just build it with "mvn clean package". > > > > Btw, since we just removed shading from external libraries (previously > some > > modules did, and some others didn't), it may not enough to just copy > built > > jar. You also need to copy transitive dependencies too. That's why we > > recommend using dependency management tools. If really needed, we may > > introduce another build profile for create uber shaded jar. > > > > Thanks, > > Jungtaek Lim (HeartSaVioR) > > > > 2017년 3월 27일 (월) 오후 2:46, Alexandre Vermeerbergen < > > [email protected]>님이 > > 작성: > > > > > Hello, > > > > > > Where is the documentation explaining how to generate the .jar files > that > > > used to be in external/*/* directories? > > > > > > I haven't tested our apps with 1.1.0rc3, I would like to do it with > using > > > instructions for generating storm-kafka.jar / storm-hbase.jar / ... > > binary > > > artifacts. > > > > > > Best regards, > > > Alexandre > > > > > > > > > 2017-03-27 5:15 GMT+02:00 Jungtaek Lim <[email protected]>: > > > > > > > Since there's no -1 vote for binding/non-binding, please put opinions > > on > > > > what Arun stated. > > > > > > > > Personally I don't think it's blocker for release. The main reason of > > > > change - size issue for binary dist. - was discussed enough, and > > though > > > we > > > > discussed shortly we were in consensus that we remove binary jars > from > > > > binary dist. > > > > > > > > We may want to apply exception to the storm-starter, but that's > > optional > > > > for me. > > > > > > > > - Jungtaek Lim (HeartSaVioR) > > > > > > > > 2017년 3월 25일 (토) 오전 3:01, Hugo Da Cruz Louro <[email protected] > > >님이 > > > > 작성: > > > > > > > > > +1 (binding) > > > > > - unzipped .zip sources and binaries > > > > > - mvn clean install in unzipped sources > > > > > - created uber jar for storm-solr and storm-kafka-client > > > > > - tested storm-solr and storm-kafka-client in local cluster mode > and > > > > > remode cluster mode > > > > > > > > > > The issue<https://issues.apache.org/jira/browse/STORM-2432> I > > > mentioned > > > > > earlier can safely be downgraded to critical as after further > > > evaluation > > > > I > > > > > consider there are workarounds. Furthermore the fix can easily be > > > > > incorporated in minor release as I stated in the follow up > discussion > > > > > emails. > > > > > > > > > > Thanks > > > > > Hugo > > > > > > > > > > > > > > > > > > > > On Mar 24, 2017, at 12:11 AM, Jungtaek Lim <[email protected] > > <mailto: > > > > > [email protected]>> wrote: > > > > > > > > > > 1. I'm OK to contain uber jar only for storm-starter, as example > for > > > > > starter. > > > > > The size should be smaller enough if we remove connector > dependencies > > > in > > > > > storm-starter. Currently it depends on storm-hdfs, storm-hbase, > > > > > storm-redis, and we could move out some topologies to remove > > > > dependencies. > > > > > > > > > > 2. Yeah maintaining doc page for connector in website is fine for > me. > > > > More > > > > > clearly, either is fine. > > > > > > > > > > 3. SQL runner in Storm SQL compiles SQL to trident topology, so in > > > order > > > > to > > > > > run SQL runner, binary dist. needs to have dependencies for SQL > > runner. > > > > Not > > > > > same as other connectors. > > > > > We might want to make uber jar for Storm SQL core and put to binary > > > dist. > > > > > to make it clear. > > > > > > > > > > Thanks, > > > > > Jungtaek Lim (HeartSaVioR) > > > > > > > > > > 2017년 3월 24일 (금) 오후 3:52, Arun Mahadevan <[email protected]<mailto: > > > > > [email protected]>>님이 작성: > > > > > > > > > > Could you cast your vote? If you are still not satisfied with > > excluding > > > > > jars you can cast -0 or even -1. > > > > > > > > > > I am not fully convinced with the current binary distribution. > > > > > > > > > > 1. Why do we expect users to build the examples source from the > > binary > > > > > distribution? Since these are most likely to be used by new users > > they > > > > will > > > > > find it difficult if the build breaks. I checked other > distributions > > > like > > > > > spark and they have the example jar inside the binary, but the size > > is > > > > > pretty small. If we remove the shading and only keep the > > > > storm-starter.jar > > > > > the size will be pretty small. Other option is to release a > separate > > > > binary > > > > > (like apache-storm-examples-xyz.jar) with just the example jars. > > > > > 2. Keeping the connectors out of the binary is good. We should also > > > > remove > > > > > the directories (with only the README.md). The users can find this > > info > > > > > from the website. > > > > > 3. Storm-sql jars are kept in the binary. Is there some reason? May > > be > > > > > this should also be removed from the binary to be consistent. > > > > > > > > > > Thanks, > > > > > Arun > > > > > > > > > > On 3/23/17, 7:34 PM, "Jungtaek Lim" <[email protected]<mailto: > > > > > [email protected]>> wrote: > > > > > > > > > > +1 to the latter. > > > > > > > > > > I'm in favor of documenting the change to release note, and also > docs > > > so > > > > > that website can be reflected. The users who are affected to the > > change > > > > > wouldn't be much, since using dependency management tool (Maven, > > > Gradle, > > > > > and so on) has been recommended for creating topology jar. > > > > > > > > > > For me it's not a blocker for release. > > > > > > > > > > Arun, I initiated another thread to discuss moving non-connectors > to > > > the > > > > > top directory. > > > > > Could you cast your vote? If you are still not satisfied with > > excluding > > > > > jars you can cast -0 or even -1. > > > > > > > > > > - Jungtaek Lim (HeartSaVioR) > > > > > > > > > > 2017년 3월 23일 (목) 오후 10:43, P. Taylor Goetz <[email protected] > > <mailto: > > > > > [email protected]>>님이 작성: > > > > > > > > > > Do we want to cancel this RC in order to better document the > changes, > > > or > > > > > will documenting it in the release announcement suffice for now > > > > > (provided > > > > > documentation is added for subsequent releases)? > > > > > > > > > > I’m partial to the latter, but am open to others’ opinions. > > > > > > > > > > -Taylor > > > > > > > > > > > > > > > On Mar 22, 2017, at 9:49 AM, Bobby Evans > <[email protected] > > > > > <mailto:[email protected]> > > > > > > > > > > wrote: > > > > > > > > > > +1 I built form the tag and ran using a single node cluster. > > > > > The examples and external components are excluded because they are > > > > > huge. Because of shading they we distribute the same copy of them > > > > > multiple > > > > > times. > > > > > I agree with Alexandre. We should document this change better, > > > > > because > > > > > it is confusing for people to get a release that used to have these > > in > > > > > it, > > > > > but does not any more. > > > > > > > > > > > > > > > - Bobby > > > > > > > > > > On Tuesday, March 21, 2017, 10:46:38 PM CDT, Arun Mahadevan < > > > > > [email protected]<mailto:[email protected]>> wrote:Verified the > > > artifacts. > > > > > Compiled examples and > > > > > ran > > > > > some sample topologies. Looks good. > > > > > > > > > > BTW, why are the external modules excluded from the binaries (the > > .zip > > > > > and .tar.gz). Isn’t it better if the binary distribution includes > > them? > > > > > Maybe it was already discussed but I am missing it. The sql > directory > > > > > however seems to include the jars so it looks inconsistent. > > > > > > > > > > - Arun > > > > > > > > > > > > > > > On 3/22/17, 12:56 AM, "P. Taylor Goetz" <[email protected] > <mailto: > > > > > [email protected]>> wrote: > > > > > > > > > > This is a call to vote on releasing Apache Storm 1.1.0 (rc3) > > > > > > > > > > Full list of changes in this release: > > > > > > > > > > > > > > > > > > > > > > > > > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_ > > > > plain;f=CHANGELOG.md;h=68fbab3c4f91359bd397d93a157830542839b002;hb= > > > > e40d213de7067f7d3aa4d4992b81890d8ed6ff31 > > > > > > > > > > The tag/commit to be voted upon is v1.1.0: > > > > > > > > > > > > > > > > > > > > > > > > > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=tree;h= > > > > > > > > > 7fa62404feb6b86b3143c851b46237580720eb6b;hb= > e40d213de7067f7d3aa4d4992b8189 > > > > 0d8ed6ff31 > > > > > > > > > > The source archive being voted upon can be found here: > > > > > > > > > > > > > > > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1. > > > > 1.0-rc3/apache-storm-1.1.0-src.tar.gz > > > > > > > > > > Other release files, signatures and digests can be found here: > > > > > > > > > > https://dist.apache.org/repos/dist/dev/storm/apache-storm-1. > 1.0-rc3/ > > > > > > > > > > The release artifacts are signed with the following key: > > > > > > > > > > > > > > > > > > > > > > > > > https://git-wip-us.apache.org/repos/asf?p=storm.git;a=blob_ > > > > plain;f=KEYS;hb=22b832708295fa2c15c4f3c70ac0d2bc6fded4bd > > > > > > > > > > The Nexus staging repository for this release is: > > > > > > > > > > > > > > > > > https://repository.apache.org/content/repositories/orgapachestorm-1047 > > > > > > > > > > Please vote on releasing this package as Apache Storm 1.1.0. > > > > > > > > > > When voting, please list the actions taken to verify the release. > > > > > > > > > > This vote will be open for at least 72 hours. > > > > > > > > > > [ ] +1 Release this package as Apache Storm 1.1.0 > > > > > [ ] 0 No opinion > > > > > [ ] -1 Do not release this package because... > > > > > > > > > > Thanks to everyone who contributed to this release. > > > > > > > > > > -Taylor > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
