[
https://issues.apache.org/jira/browse/STORM-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15984440#comment-15984440
]
Erik Weathers edited comment on STORM-2191 at 4/26/17 9:25 AM:
---------------------------------------------------------------
A few thoughts:
[~revans2]: I dug a bit into the kryo clashes and it's because of using
{{carbonite}} but also having a direct dependency on {{kryo}}. Though
carbonite's kryo dependency gets ignored because of our own kryo dependency
existing, carbonite's dependency on {{chill-java}} pulls in {{kryo-shaded}}.
kryo-shaded is just kryo with some reflectasm stuff shaded in. I imagine we
can work around this problem by either:
# excluding kryo-shaded from the carbonite dependency in storm's pom. - OR -
# excluding kryo from the carbonite dependency and adding an explicit
dependency on kryo-shaded to storm's pom.
Notably, the general {{lib/}} and the {{lib-webapp/}} dependencies show another
conflict when I run [Bobby's script from
above|https://issues.apache.org/jira/browse/STORM-2191?focusedCommentId=15982927&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15982927]
\[1]: {{javax/servlet}}.
----
[~revans2] & [~sriharsha]: I still am coming back to my doubt about the "full
jar" expansion being any better than the wildcard. Both rely on running on the
same host file system. The current mechanism of doing "full jar" expansion
uses both Python and Java methods to get the list of files from the filesystem.
My proposed mechanism of having the JVM do the wildcard expansion would switch
this to doing whatever the JVM implementation does for listing files, which
would presumably be just using the same filesystem primitives. I'm not seeing
how the "full jar" expansion produces a better, more predictable ordering of
the jars.
----
Finally, I had another wacky idea: is there any possibility of building an
uber jar of storm's dependencies? i.e., instead of all these individual 3rd
party dependency jars that are being enumerated, we might be able to have a
"storm-dependencies.jar" uber jar that just slaps all of the non-shaded
dependencies into a single jar without relocating anything.
----
\[1] I had to slightly modify Bobby's script. {{ls}} on my computer shows a
"*" suffix for executable files, so I used {{find}} instead.
was (Author: erikdw):
A few thoughts:
[~revans2]: I dug a bit into the kryo clashes and it's because of using
{{carbonite}} but also having a direct dependency on {{kryo}}. Though
carbonite's kryo dependency gets ignored because of our own kryo dependency
existing, carbonite's dependency on {{chill-java}} pulls in {{kryo-shaded}}.
kryo-shaded is just kryo with some reflectasm stuff shaded in. I imagine we
can work around this problem by either:
# excluding kryo-shaded from the carbonite dependency in storm's pom. - OR -
# excluding kryo from the carbonite dependency and adding an explicit
dependency on kryo-shaded to storm's pom.
Notably, the general {{lib/}} and the {{lib-webapp/}} dependencies show another
conflict when I run [Bobby's script from
above|https://issues.apache.org/jira/browse/STORM-2191?focusedCommentId=15982927&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15982927]
\[1]: {{javax/servlet}}.
----
[~revans2] & [~sriharsha]: I still am coming back to my doubt about the "full
jar" expansion being any better than the wildcard. Both rely on running on the
same host file system. The current mechanism of doing "full jar" expansion
uses both Python and Java methods to get the list of files from the filesystem.
The proposed (by me) mechanism of having the JVM do the wildcard expansion
would switch this to doing whatever the JVM implementation does for listing
files, which would presumably be just using the same filesystem primitives.
I'm not seeing how the "full jar" expansion produces a better, more predictable
ordering of the jars.
----
Finally, I had another wacky idea: is there any possibility of building an
uber jar of storm's dependencies? i.e., instead of all these individual 3rd
party dependency jars that are being enumerated, we might be able to have a
"storm-dependencies.jar" uber jar that just slaps all of the non-shaded
dependencies into a single jar without relocating anything.
----
\[1] I had to slightly modify Bobby's script. {{ls}} on my computer shows a
"*" suffix for executable files, so I used {{find}} instead.
> shorten classpaths in worker and LogWriter commands
> ---------------------------------------------------
>
> Key: STORM-2191
> URL: https://issues.apache.org/jira/browse/STORM-2191
> Project: Apache Storm
> Issue Type: Task
> Components: storm-core
> Affects Versions: 1.0.2
> Reporter: Erik Weathers
> Priority: Minor
> Labels: cli, command-line
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When launching the worker daemon and its wrapping LogWriter daemon, the
> commands can become so long that they eclipse the default Linux limit of 4096
> bytes. That results in commands that are cut off in {{ps}} output, and
> prevents easily inspecting the system to see even what processes are running.
> The specific scenario in which this problem can be easily triggered: *running
> Storm on Mesos*.
> h5. Details on why it happens:
> # using the default Mesos containerizer instead of Docker containers, which
> causes the storm-mesos package to be unpacked into the Mesos executor sandbox.
> # The ["expand all jars on
> classpath"|https://github.com/apache/storm/blob/6dc6407a01d032483edebb1c1b4d8b69a304d81c/bin/storm.py#L114-L140]
> functionality in the {{bin/storm.py}} script causes every one of the jars
> that storm bundles into its lib directory to be explicitly listed in the
> command.
> #* e.g., say the mesos work dir is {{/var/run/mesos/work_dir/}}
> #* and say that the original classpath argument in the supervisor cmd
> includes the following for the {{lib/}} dir in the binary storm package:
> #**
> {{/var/run/mesos/work_dir/slaves/2357b762-6653-4052-ab9e-f1354d78991b-S12/frameworks/20160509-084241-1086985738-5050-32231-0000/executors/STORM_TOPOLOGY_ID/runs/e6a1407e-73fd-4be4-8d00-e882117b3391/storm-mesos-0.1.7-storm0.9.6-mesos0.28.2/lib/*}}
> #* That leads to a hugely expanded classpath argument for the LogWriter and
> Worker daemons that get launched:
> #**
> {{/var/run/mesos/work_dir/slaves/2357b762-6653-4052-ab9e-f1354d78991b-S12/frameworks/20160509-084241-1086985738-5050-32231-0000/executors/STORM_TOPOLOGY_ID/runs/e6a1407e-73fd-4be4-8d00-e882117b3391/storm-mesos-0.1.7-storm0.9.6-mesos0.28.2/lib/asm-4.0.jar:/var/run/mesos/work_dir/slaves/2357b762-6653-4052-ab9e-f1354d78991b-S12/frameworks/20160509-084241-1086985738-5050-32231-0000/executors/STORM_TOPOLOGY_ID/runs/e6a1407e-73fd-4be4-8d00-e882117b3391/storm-mesos-0.1.7-storm0.9.6-mesos0.28.2/lib/carbonite-1.4.0.jar:...}}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
