I guess it might be good time to think why we add all connectors to the binary distribution.
Spark and Flink don't include them to binary dist. They even moved some or most of connectors out of repo, have been maintaining them in Apache Bahir. (Personally this is something I'm in favor of. We have lots of connectors and many of them are outdated - clear example is storm-elasticsearch.) If we are assuming online then we don't even need to think about users touching binary dist. version of connectors. Users have been including them via build tools' dependency management, or even starting 1.1.0, users can include them via '--artifact' option. I also was just one of users for Storm, and I haven't use them directly. How much UX gets worse when we remove connectors to binary dist? It only helps some users who are not connected to the internet, and IMHO it's a rare case. I would like to see the opposite approach, removing all connectors (or just keeping storm-kafka/storm-kafka-client and some more preferred things) and its relevant examples from binary dist. What do you think about it? - Jungtaek Lim (HeartSaVioR) 2017년 3월 2일 (목) 오전 10:23, Roshan Naik <ros...@hortonworks.com>님이 작성: Once all of the shaded examples are included the size will go up further. But currently as they are not part of the tar.gz … something else is the culprit for the bloat. Below is a comparative listing of 1.0.3 vs 1.1.0 binary releases .. of files that are larger than 4MB. @Jungtaek Lim<mailto:j...@hortonworks.com> : I am thinking, since the code for the examples can be easily viewed online … it would be valuable to have the executable topologies made available to the user as part of the binary release … rather have them figure out how to build it correctly before trying them out. -roshan ➜ apache-storm-1.0.3 > find . -type f -size +4096 -exec ls -lh {} \; -rw-r--r--@ 1 roshan staff 70M Feb 7 12:33 ./examples/storm-starter/storm-starter-topologies-1.0.3.jar -rwxr-xr-x@ 1 roshan staff 65M Feb 7 12:30 ./external/flux/flux-examples-1.0.3.jar -rwxr-xr-x@ 1 roshan staff 3.5M Feb 7 12:32 ./external/sql/storm-sql-core/calcite-core-1.4.0-incubating.jar -rwxr-xr-x@ 1 roshan staff 2.1M Feb 7 12:32 ./external/sql/storm-sql-core/guava-16.0.1.jar -rwxr-xr-x@ 1 roshan staff 7.3M Feb 7 12:30 ./external/storm-eventhubs/storm-eventhubs-1.0.3-jar-with-dependencies.jar -rwxr-xr-x@ 1 roshan staff 5.6M Feb 7 12:33 ./external/storm-jms/storm-jms-examples-1.0.3-jar-with-dependencies.jar -rwxr-xr-x@ 1 roshan staff 9.9M Feb 7 12:33 ./external/storm-mqtt/storm-mqtt-examples-1.0.3.jar -rw-r--r--@ 1 roshan staff 3.7M Nov 4 10:02 ./lib/clojure-1.7.0.jar -rw-r--r--@ 1 roshan staff 19M Feb 7 12:26 ./lib/storm-core-1.0.3.jar -rw-r--r--@ 1 roshan staff 2.4M Feb 7 12:26 ./lib/storm-rename-hack-1.0.3.jar ➜ apache-storm-1.1.0 > find . -type f -size +4096 -exec ls -lh {} \; -rwxr-xr-x@ 1 roshan staff 8.0M Feb 24 12:23 ./examples/storm-pmml-examples/storm-pmml-examples-1.1.0.jar -rwxr-xr-x@ 1 roshan staff 60M Feb 24 12:20 ./examples/storm-starter/storm-starter-topologies-1.1.0.jar -rwxr-xr-x@ 1 roshan staff 66M Feb 24 12:11 ./external/flux/flux-examples-1.1.0.jar -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:16 ./external/sql/storm-sql-core/calcite-core-1.11.0.jar -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:16 ./external/sql/storm-sql-core/guava-16.0.1.jar -rwxr-xr-x@ 1 roshan staff 4.0M Feb 24 12:12 ./external/sql/storm-sql-runtime/calcite-core-1.11.0.jar -rwxr-xr-x@ 1 roshan staff 2.1M Feb 24 12:12 ./external/sql/storm-sql-runtime/guava-16.0.1.jar -rwxr-xr-x@ 1 roshan staff 78M Feb 24 12:18 ./external/storm-druid/storm-druid-1.1.0.jar -rwxr-xr-x@ 1 roshan staff 7.3M Feb 24 12:11 ./external/storm-eventhubs/storm-eventhubs-1.1.0-jar-with-dependencies.jar -rwxr-xr-x@ 1 roshan staff 5.6M Feb 24 12:20 ./external/storm-jms/storm-jms-examples-1.1.0-jar-with-dependencies.jar -rwxr-xr-x@ 1 roshan staff 6.7M Feb 24 12:18 ./external/storm-submit-tools/storm-submit-tools-1.1.0.jar -rwxr-xr-x@ 1 roshan staff 3.7M Nov 4 10:02 ./lib/clojure-1.7.0.jar -rwxr-xr-x@ 1 roshan staff 20M Feb 24 12:07 ./lib/storm-core-1.1.0.jar -rwxr-xr-x@ 1 roshan staff 2.4M Feb 24 12:07 ./lib/storm-rename-hack-1.1.0.jar -rwxr-xr-x@ 1 roshan staff 18M Feb 24 12:19 ./toollib/storm-kafka-monitor-1.1.0.jar On 3/1/17, 4:43 PM, "Jungtaek Lim" <kabh...@gmail.com> wrote: About STORM-2249, since examples are shading their dependencies, binary dist will grow much bigger. I've left some comments regarding that. Btw, I have another view of this. Showing example codes is more important than just let users execute some topologies. That's what example modules are for. We need to include source as well. If we need to pick one, source code would be better. STORM-2343 seems better to add to 1.1.0. I just am not enough familiar with storm-kafka-client so not sure I can review that, but I'll try to. I feel it's not that make release dragged. Let's add to 1.1.0 epic. Let's make minimum merge before another RC vote. Addressing STORM-2389 (and maybe STORM-2343) is enough for me. Others are not that critical. Thanks, Jungtaek Lim (HeartSaVioR) On Thu, Mar 2, 2017 at 7:00 AM, Hugo Da Cruz Louro < hlo...@hortonworks.com> wrote: > Roshan, does this PR<https://github.com/apache/storm/pull/1831> and JIRA< > https://issues.apache.org/jira/browse/STORM-2249> address the missing > jars problem that you mentioned. I had created it in December 2016, but > there is an ongoing discussion if we should indeed put the jars in the > examples location or not. > > On a different note, this storm-kafka-client/KafkaSpout PR< > https://github.com/apache/storm/pull/1924> fixes a bug with the number of > uncommitted offsets that is quite important. It is not a blocker, but it is > quite critical. I am going to do one last pass reviewing today. It would be > good if we could have this PR included with the release. Can anyone else > review it as well ? > > Thanks, > Hugo > > On Mar 1, 2017, at 9:14 AM, P. Taylor Goetz <ptgo...@gmail.com<mailto: ptgo > e...@gmail.com>> wrote: > > Yeah, I don’t think the file size is a killer/blocker. It’s largely due to > shaded examples, etc. But it’s something to keep an eye on. Our binary > releases shouldn’t have to be that big. > > -Taylor > > On Mar 1, 2017, at 12:09 PM, Roshan Naik <ros...@hortonworks.com <mailto: > ros...@hortonworks.com>> wrote: > > Have filed Jiras so for the 3 issues mentioned. Not sure if we need a JIRA > for the file size getting bloated by that much. > Somebody better familiar with the matter may want to take about that? > -roshan > > > On 3/1/17, 8:13 AM, "P. Taylor Goetz" <ptgo...@gmail.com<mailto:ptgo > e...@gmail.com>> wrote: > > Thanks for bringing these up Roshan. Feel free to file JIRA tickets for > these issues and assign the “Release Apache Storm 1.1.0” epic so they can > be tracked for this release. > > -Taylor > > On Mar 1, 2017, at 9:27 AM, Roshan Naik <ros...@hortonworks.com <mailto: > ros...@hortonworks.com>> wrote: > > Found these additional issues: > > > > 1- BUG: Even if topology.eventlogger.executors=0, the event_logger bolt > is instantiated … previously observed to cause ~10% degradation in perf > even with logging disabled. > > 2- Missing Jars: The storm-*-examples jars are missing in the binary > distro (other than a storm-pmml-examples.jar, storm-jms-examples.jar & > flux-examples.jar). > > 3- Minor: HdfsSpoutTopology example has not been moved into > storm-hdfs-examples from storm-starter > > > > Another side observation … v1.0.3 tar.gz downloadable was 190MB. This > v1.1.0 tar.gz downloadable is 297MB !! …. even though some of the example > topologies didn’t make it. > > > > -roshan > > > > > > > > -- Name : Jungtaek Lim Blog : http://medium.com/@heartsavior Twitter : http://twitter.com/heartsavior LinkedIn : http://www.linkedin.com/in/heartsavior
