[ https://issues.apache.org/jira/browse/STORM-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15981936#comment-15981936 ]
Erik Weathers edited comment on STORM-2191 at 4/24/17 9:21 PM: --------------------------------------------------------------- [~sriharsha] I think you're suggesting possibly allowing the "classpath wildcarding" via an option. Can you please confirm if I understood correctly? One thing I was thinking about w.r.t. this idea of the wildcard being dangerous: we are already exposed to funkiness from OS filesystem variances in the way that we resolve the classpaths today, are we not? i.e., [{{get_jars_full}}|https://github.com/apache/storm/blob/7e183c010ee1ab342c923596d0474e4093571a43/bin/storm.py#L121] and [{{getFullJars}}|https://github.com/apache/storm/blob/7e183c010ee1ab342c923596d0474e4093571a43/storm-server/src/main/java/org/apache/storm/daemon/supervisor/BasicContainer.java#L342] are using OS filesystem calls to get the list of files. So I'm not sure how it would be *worse* to use wildcards. was (Author: erikdw): [~sriharsha] I think you're suggesting possibly allowing the "classpath wildcarding" via an option. Can you please confirm if I understood correctly? One thing I was thinking about w.r.t. this idea of the wildcard being dangerous: we are already exposed to funkiness from OS filesystem variances in the way that we resolve the classpaths today, are we not? i.e., [{{get_jars_full}}|https://github.com/apache/storm/blob/7e183c010ee1ab342c923596d0474e4093571a43/bin/storm.py#L121] and [{{getFullJars}}|https://github.com/apache/storm/blob/7e183c010ee1ab342c923596d0474e4093571a43/storm-server/src/main/java/org/apache/storm/daemon/supervisor/BasicContainer.java#L342]. So I'm not sure how it would be *worse* to use wildcards. > shorten classpaths in worker and LogWriter commands > --------------------------------------------------- > > Key: STORM-2191 > URL: https://issues.apache.org/jira/browse/STORM-2191 > Project: Apache Storm > Issue Type: Task > Components: storm-core > Affects Versions: 1.0.2 > Reporter: Erik Weathers > Priority: Minor > Labels: cli, command-line > > When launching the worker daemon and its wrapping LogWriter daemon, the > commands can become so long that they eclipse the default Linux limit of 4096 > bytes. That results in commands that are cut off in {{ps}} output, and > prevents easily inspecting the system to see even what processes are running. > The specific scenario in which this problem can be easily triggered: *running > Storm on Mesos*. > h5. Details on why it happens: > # using the default Mesos containerizer instead of Docker containers, which > causes the storm-mesos package to be unpacked into the Mesos executor sandbox. > # The ["expand all jars on > classpath"|https://github.com/apache/storm/blob/6dc6407a01d032483edebb1c1b4d8b69a304d81c/bin/storm.py#L114-L140] > functionality in the {{bin/storm.py}} script causes every one of the jars > that storm bundles into its lib directory to be explicitly listed in the > command. > #* e.g., say the mesos work dir is {{/var/run/mesos/work_dir/}} > #* and say that the original classpath argument in the supervisor cmd > includes the following for the {{lib/}} dir in the binary storm package: > #** > {{/var/run/mesos/work_dir/slaves/2357b762-6653-4052-ab9e-f1354d78991b-S12/frameworks/20160509-084241-1086985738-5050-32231-0000/executors/STORM_TOPOLOGY_ID/runs/e6a1407e-73fd-4be4-8d00-e882117b3391/storm-mesos-0.1.7-storm0.9.6-mesos0.28.2/lib/*}} > #* That leads to a hugely expanded classpath argument for the LogWriter and > Worker daemons that get launched: > #** > {{/var/run/mesos/work_dir/slaves/2357b762-6653-4052-ab9e-f1354d78991b-S12/frameworks/20160509-084241-1086985738-5050-32231-0000/executors/STORM_TOPOLOGY_ID/runs/e6a1407e-73fd-4be4-8d00-e882117b3391/storm-mesos-0.1.7-storm0.9.6-mesos0.28.2/lib/asm-4.0.jar:/var/run/mesos/work_dir/slaves/2357b762-6653-4052-ab9e-f1354d78991b-S12/frameworks/20160509-084241-1086985738-5050-32231-0000/executors/STORM_TOPOLOGY_ID/runs/e6a1407e-73fd-4be4-8d00-e882117b3391/storm-mesos-0.1.7-storm0.9.6-mesos0.28.2/lib/carbonite-1.4.0.jar:...}} -- This message was sent by Atlassian JIRA (v6.3.15#6346)