[
https://issues.apache.org/jira/browse/STORM-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709779#comment-15709779
]
Erik Weathers commented on STORM-2191:
--------------------------------------
[~revans2], [~ptgoetz], [~kabhwan]: I'm working on fixing this issue, but I've
noticed that in newer (as compared to 0.9.x) storm versions the
{{get_jars_full}} logic has spread into the [core Storm Java
code|https://github.com/apache/storm/blob/master/storm-core/src/jvm/org/apache/storm/daemon/supervisor/BasicContainer.java#L330-L356]
now too, and there's [*more*
uses|https://github.com/apache/storm/blob/285668742742da6316d38c1ef492c7c873b5a649/bin/storm.py#L131-L138]
of {{get_jars_full}} within {{bin/storm.py}}. I do not understand the
purpose for this logic and intend to submit a PR that replaces it with simply
putting a wildcard in the dir that is being expanded. Without this the
commands get obscenely long (as explained in the Description of this ticket).
Can you please let me know if this would be unacceptable? I *really* don't
want to have to forever fork and hack the core storm repo in order to support
running the storm jar from a deep directly (as is necessitated when using
non-dockerized Mesos).
Thanks!!
- Erik
> shorten classpaths in worker and LogWriter commands
> ---------------------------------------------------
>
> Key: STORM-2191
> URL: https://issues.apache.org/jira/browse/STORM-2191
> Project: Apache Storm
> Issue Type: Task
> Components: storm-core
> Affects Versions: 1.0.2
> Reporter: Erik Weathers
> Priority: Minor
> Labels: cli, command-line
>
> When launching the worker daemon and its wrapping LogWriter daemon, the
> commands can become so long that they eclipse the default Linux limit of 4096
> bytes. That results in commands that are cut off in {{ps}} output, and
> prevents easily inspecting the system to see even what processes are running.
> The specific scenario in which this problem can be easily triggered: *running
> Storm on Mesos*.
> h5. Details on why it happens:
> # using the default Mesos containerizer instead of Docker containers, which
> causes the storm-mesos package to be unpacked into the Mesos executor sandbox.
> # The ["expand all jars on
> classpath"|https://github.com/apache/storm/blob/6dc6407a01d032483edebb1c1b4d8b69a304d81c/bin/storm.py#L114-L140]
> functionality in the {{bin/storm.py}} script causes every one of the jars
> that storm bundles into its lib directory to be explicitly listed in the
> command.
> #* e.g., say the mesos work dir is {{/var/run/mesos/work_dir/}}
> #* and say that the original classpath argument in the supervisor cmd
> includes the following for the {{lib/}} dir in the binary storm package:
> #**
> {{/var/run/mesos/work_dir/slaves/2357b762-6653-4052-ab9e-f1354d78991b-S12/frameworks/20160509-084241-1086985738-5050-32231-0000/executors/STORM_TOPOLOGY_ID/runs/e6a1407e-73fd-4be4-8d00-e882117b3391/storm-mesos-0.1.7-storm0.9.6-mesos0.28.2/lib/*}}
> #* That leads to a hugely expanded classpath argument for the LogWriter and
> Worker daemons that get launched:
> #**
> {{/var/run/mesos/work_dir/slaves/2357b762-6653-4052-ab9e-f1354d78991b-S12/frameworks/20160509-084241-1086985738-5050-32231-0000/executors/STORM_TOPOLOGY_ID/runs/e6a1407e-73fd-4be4-8d00-e882117b3391/storm-mesos-0.1.7-storm0.9.6-mesos0.28.2/lib/asm-4.0.jar:/var/run/mesos/work_dir/slaves/2357b762-6653-4052-ab9e-f1354d78991b-S12/frameworks/20160509-084241-1086985738-5050-32231-0000/executors/STORM_TOPOLOGY_ID/runs/e6a1407e-73fd-4be4-8d00-e882117b3391/storm-mesos-0.1.7-storm0.9.6-mesos0.28.2/lib/carbonite-1.4.0.jar:...}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
