GitHub user erikdw opened a pull request:
https://github.com/apache/storm/pull/2101
[STORM-2191] shorten classpaths by using wildcards
Backport of PR #2094 to 1.x-branch.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/erikdw/storm STORM-2191__1.x-branch
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/storm/pull/2101.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2101
----
commit 9767255b2627119f14f930f6b39fe5d5e7fd663d
Author: Erik Weathers <[email protected]>
Date: 2017-04-28T07:50:18Z
[STORM-2191] shorten classpaths by using wildcards
This commit resolves cherry-pick conflicts for backporting change from
storm master branch.
Related commit in master: 2ceef0dce21895c4ef0f0b8eb5bc8248fad25e13
Instead of fully enumerating all JARs in the lib directories, we just use
a Java classpath wildcard to allow the JVM to autodiscover all JARs in the
classpath. This affects both the Worker and LogWriter processes, as well
as Storm daemons such as the Nimbus, UI, Logviewer, and Supervisor.
This change results in shorter commands, so that you can actually see
the full content of the Worker command in `ps` output on Linux.
Prior to this change Worker commands were easily longer than 4096 bytes,
which is the default Linux kernel limit for commands being recorded into
the process table. Longer commands get truncated, though they do get
executed.
An example of the change in Worker classpath length can be seen here:
Before:
```
-cp
STORM_DIR/lib-worker/asm-5.0.3.jar:STORM_DIR/lib-worker/carbonite-1.5.0.jar:STORM_DIR/lib-worker/chill-java-0.8.0.jar:STORM_DIR/lib-worker/clojure-1.7.0.jar:STORM_DIR/lib-worker/commons-codec-1.6.jar:STORM_DIR/lib-worker/commons-collections-3.2.2.jar:STORM_DIR/lib-worker/commons-io-2.5.jar:STORM_DIR/lib-worker/commons-lang-2.5.jar:STORM_DIR/lib-worker/commons-logging-1.1.3.jar:STORM_DIR/lib-worker/curator-client-2.12.0.jar:STORM_DIR/lib-worker/curator-framework-2.12.0.jar:STORM_DIR/lib-worker/disruptor-3.3.2.jar:STORM_DIR/lib-worker/guava-16.0.1.jar:STORM_DIR/lib-worker/httpclient-4.3.3.jar:STORM_DIR/lib-worker/httpcore-4.4.1.jar:STORM_DIR/lib-worker/jgrapht-core-0.9.0.jar:STORM_DIR/lib-worker/jline-0.9.94.jar:STORM_DIR/lib-worker/json-simple-1.1.jar:STORM_DIR/lib-worker/kryo-3.0.3.jar:STORM_DIR/lib-worker/kryo-shaded-3.0.3.jar:STORM_DIR/lib-worker/libthrift-0.9.3.jar:STORM_DIR/lib-worker/log4j-api-2.8.jar:STORM_DIR/lib-worker/log4j-core-2.8.jar:STORM_DIR/lib-worker/log4j-ove
r-slf4j-1.6.6.jar:STORM_DIR/lib-worker/log4j-slf4j-impl-2.8.jar:STORM_DIR/lib-worker/minlog-1.3.0.jar:STORM_DIR/lib-worker/netty-3.9.0.Final.jar:STORM_DIR/lib-worker/objenesis-2.1.jar:STORM_DIR/lib-worker/reflectasm-1.10.1.jar:STORM_DIR/lib-worker/servlet-api-2.5.jar:STORM_DIR/lib-worker/slf4j-api-1.7.21.jar:STORM_DIR/lib-worker/snakeyaml-1.11.jar:STORM_DIR/lib-worker/storm-client-2.0.0-SNAPSHOT.jar:STORM_DIR/lib-worker/sysout-over-slf4j-1.0.2.jar:STORM_DIR/lib-worker/zookeeper-3.4.6.jar:STORM_DIR/conf:STORM_DIR/storm-local/supervisor/stormdist/foo-topology-1-1-1493359573/stormjar.jar
```
After:
```
-cp
STORM_DIR/lib-worker/*:STORM_DIR/extlib/*:STORM_DIR/conf:STORM_DIR/storm-local/supervisor/stormdist/foo-topology-1-1-1493359573/stormjar.jar
```
This change also includes additional documentation about the use of
classpaths in
Storm and provides some guidance for using the various features for using
external
libraries.
For more details on this problem and a discussion about this solution's
merits, please see
[STORM-2191](https://issues.apache.org/jira/browse/STORM-2191).
commit b04011ee8c40b8971592e74e85c18778989bb932
Author: Erik Weathers <[email protected]>
Date: 2017-05-04T21:57:56Z
[STORM-2191] address revans2's suggestions, and fix typo (extra ']') in
markdown in Setting-up-a-Storm-cluster.md
commit b780df2312835021de4c29c673696cfac7dff767
Author: Erik Weathers <[email protected]>
Date: 2017-05-04T22:01:12Z
[STORM-2191] add link to the new Classpath-handling doc page from the index
page
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
