Looks like the formatting on my prev email got dropped. But observation is that storm-kafka-monitor and storm-druid are the biggest new contributors. -roshan
On 3/1/17, 5:23 PM, "Roshan Naik" <[email protected]> wrote: Once all of the shaded examples are included the size will go up further. But currently as they are not part of the tar.gz … something else is the culprit for the bloat. Below is a comparative listing of 1.0.3 vs 1.1.0 binary releases .. of files that are larger than 4MB. @Jungtaek Lim<mailto:[email protected]> : I am thinking, since the code for the examples can be easily viewed online … it would be valuable to have the executable topologies made available to the user as part of the binary release … rather have them figure out how to build it correctly before trying them out. -roshan ➜ apache-storm-1.0.3 > find . -type f -size +4096 -exec ls -lh {} \; -rw-r--r--@ 1 roshan staff 70M Feb 7 12:33 ./examples/storm-starter/storm-starter-topologies-1.0.3.jar -rwxr-xr-x@ 1 roshan staff 65M Feb 7 12:30 ./external/flux/flux-examples-1.0.3.jar -rwxr-xr-x@ 1 roshan staff 3.5M Feb 7 12:32 ./external/sql/storm-sql-core/calcite-core-1.4.0-incubating.jar -rwxr-xr-x@ 1 roshan staff 2.1M Feb 7 12:32 ./external/sql/storm-sql-core/guava-16.0.1.jar -rwxr-xr-x@ 1 roshan staff 7.3M Feb 7 12:30 ./external/storm-eventhubs/storm-eventhubs-1.0.3-jar-with-dependencies.jar -rwxr-xr-x@ 1 roshan staff 5.6M Feb 7 12:33 ./external/storm-jms/storm-jms-examples-1.0.3-jar-with-dependencies.jar -rwxr-xr-x@ 1 roshan staff 9.9M Feb 7 12:33 ./external/storm-mqtt/storm-mqtt-examples-1.0.3.jar -rw-r--r--@ 1 roshan staff 3.7M Nov 4 10:02 ./lib/clojure-1.7.0.jar -rw-r--r--@ 1 roshan staff 19M Feb 7 12:26 ./lib/storm-core-1.0.3.jar -rw-r--r--@ 1 roshan staff 2.4M Feb 7 12:26 ./lib/storm-rename-hack-1.0.3.jar ➜ apache-storm-1.1.0 > find . -type f -size +4096 -exec ls -lh {} \; -rwxr-xr-x@ 1 roshan staff 8.0M Feb 24 12:23 ./examples/storm-pmml-examples/storm-pmml-examples-1.1.0.jar -rwxr-xr-x@ 1 roshan staff 60M Feb 24 12:20 ./examples/storm-starter/storm-starter-topologies-1.1.0.jar -rwxr-xr-x@ 1 roshan staff 66M Feb 24 12:11 ./external/flux/flux-examples-1.1.0.jar -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:16 ./external/sql/storm-sql-core/calcite-core-1.11.0.jar -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:16 ./external/sql/storm-sql-core/guava-16.0.1.jar -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:12 ./external/sql/storm-sql-runtime/calcite-core-1.11.0.jar -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:12 ./external/sql/storm-sql-runtime/guava-16.0.1.jar -rwxr-xr-x@ 1 roshan staff 78M Feb 24 12:18 ./external/storm-druid/storm-druid-1.1.0.jar -rwxr-xr-x@ 1 roshan staff 7.3M Feb 24 12:11 ./external/storm-eventhubs/storm-eventhubs-1.1.0-jar-with-dependencies.jar -rwxr-xr-x@ 1 roshan staff 5.6M Feb 24 12:20 ./external/storm-jms/storm-jms-examples-1.1.0-jar-with-dependencies.jar -rwxr-xr-x@ 1 roshan staff 6.7M Feb 24 12:18 ./external/storm-submit-tools/storm-submit-tools-1.1.0.jar -rwxr-xr-x@ 1 roshan staff 3.7M Nov 4 10:02 ./lib/clojure-1.7.0.jar -rwxr-xr-x@ 1 roshan staff 20M Feb 24 12:07 ./lib/storm-core-1.1.0.jar -rwxr-xr-x@ 1 roshan staff 2.4M Feb 24 12:07 ./lib/storm-rename-hack-1.1.0.jar -rwxr-xr-x@ 1 roshan staff 18M Feb 24 12:19 ./toollib/storm-kafka-monitor-1.1.0.jar On 3/1/17, 4:43 PM, "Jungtaek Lim" <[email protected]> wrote: About STORM-2249, since examples are shading their dependencies, binary dist will grow much bigger. I've left some comments regarding that. Btw, I have another view of this. Showing example codes is more important than just let users execute some topologies. That's what example modules are for. We need to include source as well. If we need to pick one, source code would be better. STORM-2343 seems better to add to 1.1.0. I just am not enough familiar with storm-kafka-client so not sure I can review that, but I'll try to. I feel it's not that make release dragged. Let's add to 1.1.0 epic. Let's make minimum merge before another RC vote. Addressing STORM-2389 (and maybe STORM-2343) is enough for me. Others are not that critical. Thanks, Jungtaek Lim (HeartSaVioR) On Thu, Mar 2, 2017 at 7:00 AM, Hugo Da Cruz Louro <[email protected]> wrote: > Roshan, does this PR<https://github.com/apache/storm/pull/1831> and JIRA< > https://issues.apache.org/jira/browse/STORM-2249> address the missing > jars problem that you mentioned. I had created it in December 2016, but > there is an ongoing discussion if we should indeed put the jars in the > examples location or not. > > On a different note, this storm-kafka-client/KafkaSpout PR< > https://github.com/apache/storm/pull/1924> fixes a bug with the number of > uncommitted offsets that is quite important. It is not a blocker, but it is > quite critical. I am going to do one last pass reviewing today. It would be > good if we could have this PR included with the release. Can anyone else > review it as well ? > > Thanks, > Hugo > > On Mar 1, 2017, at 9:14 AM, P. Taylor Goetz <[email protected]<mailto:ptgo > [email protected]>> wrote: > > Yeah, I don’t think the file size is a killer/blocker. It’s largely due to > shaded examples, etc. But it’s something to keep an eye on. Our binary > releases shouldn’t have to be that big. > > -Taylor > > On Mar 1, 2017, at 12:09 PM, Roshan Naik <[email protected]<mailto: > [email protected]>> wrote: > > Have filed Jiras so for the 3 issues mentioned. Not sure if we need a JIRA > for the file size getting bloated by that much. > Somebody better familiar with the matter may want to take about that? > -roshan > > > On 3/1/17, 8:13 AM, "P. Taylor Goetz" <[email protected]<mailto:ptgo > [email protected]>> wrote: > > Thanks for bringing these up Roshan. Feel free to file JIRA tickets for > these issues and assign the “Release Apache Storm 1.1.0” epic so they can > be tracked for this release. > > -Taylor > > On Mar 1, 2017, at 9:27 AM, Roshan Naik <[email protected]<mailto: > [email protected]>> wrote: > > Found these additional issues: > > > > 1- BUG: Even if topology.eventlogger.executors=0, the event_logger bolt > is instantiated … previously observed to cause ~10% degradation in perf > even with logging disabled. > > 2- Missing Jars: The storm-*-examples jars are missing in the binary > distro (other than a storm-pmml-examples.jar, storm-jms-examples.jar & > flux-examples.jar). > > 3- Minor: HdfsSpoutTopology example has not been moved into > storm-hdfs-examples from storm-starter > > > > Another side observation … v1.0.3 tar.gz downloadable was 190MB. This > v1.1.0 tar.gz downloadable is 297MB !! …. even though some of the example > topologies didn’t make it. > > > > -roshan > > > > > > > > -- Name : Jungtaek Lim Blog : http://medium.com/@heartsavior Twitter : http://twitter.com/heartsavior LinkedIn : http://www.linkedin.com/in/heartsavior
