Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/storm/pull/1608#discussion_r74367408
--- Diff: bin/storm.py ---
@@ -232,44 +271,91 @@ def jar(jarfile, klass, *args):
The process is configured so that StormSubmitter
(http://storm.apache.org/releases/current/javadocs/org/apache/storm/StormSubmitter.html)
will upload the jar at topology-jar-path when the topology is
submitted.
+
+ When you want to ship other jars which is not included to application
jar, you can pass them to --jars option with comma-separated string.
+ For example, --jars "your-local-jar.jar,your-local-jar2.jar" will load
your-local-jar.jar and your-local-jar2.jar.
+ And when you want to ship maven artifacts and its transitive
dependencies, you can pass them to --artifacts with comma-separated string.
+ You can also exclude some dependencies like what you're doing in maven
pom.
+ Please add exclusion artifacts with '^' separated string after the
artifact.
+ For example, --artifacts
"redis.clients:jedis:2.9.0,org.apache.kafka:kafka_2.10:0.8.2.2^org.slf4j:slf4j-log4j12"
will load jedis and kafka artifact and all of transitive dependencies but
exclude slf4j-log4j12 from kafka.
+
+ Complete example of both options is here: `./bin/storm jar
example/storm-starter/storm-starter-topologies-*.jar
org.apache.storm.starter.RollingTopWords blobstore-remote2 remote --jars
"./external/storm-redis/storm-redis-1.1.0.jar,./external/storm-kafka/storm-kafka-1.1.0.jar"
--artifacts
"redis.clients:jedis:2.9.0,org.apache.kafka:kafka_2.10:0.8.2.2^org.slf4j:slf4j-log4j12"`
+
+ When you pass jars and/or artifacts options, StormSubmitter will
upload them when the topology is submitted, and they will be included to
classpath of both the process which runs the class, and also workers for that
topology.
"""
+ global DEP_JARS_OPTS, DEP_ARTIFACTS_OPTS
+
+ local_jars = DEP_JARS_OPTS
+ artifact_to_file_jars = resolve_dependencies(DEP_ARTIFACTS_OPTS)
+
transform_class = confvalue("client.jartransformer.class",
[CLUSTER_CONF_DIR])
if (transform_class != None and transform_class != "nil"):
tmpjar = os.path.join(tempfile.gettempdir(),
uuid.uuid1().hex+".jar")
exec_storm_class("org.apache.storm.daemon.ClientJarTransformerRunner",
args=[transform_class, jarfile, tmpjar], fork=True, daemon=False)
+ extra_jars = [tmpjar, USER_CONF_DIR, STORM_BIN_DIR]
+ extra_jars.extend(local_jars)
+ extra_jars.extend(artifact_to_file_jars.values())
topology_runner_exit_code = exec_storm_class(
klass,
jvmtype="-client",
- extrajars=[tmpjar, USER_CONF_DIR, STORM_BIN_DIR],
+ extrajars=extra_jars,
args=args,
daemon=False,
fork=True,
- jvmopts=JAR_JVM_OPTS + ["-Dstorm.jar=" + tmpjar])
+ jvmopts=JAR_JVM_OPTS + ["-Dstorm.jar=" + tmpjar] +
+ ["-Dstorm.dependency.jars=" +
",".join(local_jars)] +
+ ["-Dstorm.dependency.artifacts=" +
json.dumps(artifact_to_file_jars)])
os.remove(tmpjar)
sys.exit(topology_runner_exit_code)
else:
+ extra_jars=[jarfile, USER_CONF_DIR, STORM_BIN_DIR]
+ extra_jars.extend(local_jars)
+ extra_jars.extend(artifact_to_file_jars.values())
exec_storm_class(
klass,
jvmtype="-client",
- extrajars=[jarfile, USER_CONF_DIR, STORM_BIN_DIR],
+ extrajars=extra_jars,
args=args,
daemon=False,
- jvmopts=JAR_JVM_OPTS + ["-Dstorm.jar=" + jarfile])
+ jvmopts=JAR_JVM_OPTS + ["-Dstorm.jar=" + jarfile] +
+ ["-Dstorm.dependency.jars=" + ",".join(local_jars)] +
+ ["-Dstorm.dependency.artifacts=" +
json.dumps(artifact_to_file_jars)])
def sql(sql_file, topology_name):
"""Syntax: [storm sql sql-file topology-name]
Compiles the SQL statements into a Trident topology and submits it to
Storm.
+
+ --jars and --artifacts options available for jar are also applied to
sql command.
+ Please refer "help jar" to see how to use --jars and --artifacts
options.
+ You normally want to pass these options since you need to set data
source to your sql which is an external storage in many cases.
"""
+ global DEP_JARS_OPTS, DEP_ARTIFACTS_OPTS
+
+ local_jars = DEP_JARS_OPTS
+ artifact_to_file_jars = resolve_dependencies(DEP_ARTIFACTS_OPTS)
+
+ sql_core_jars = get_jars_full(STORM_DIR +
"/external/sql/storm-sql-core")
--- End diff --
Definitely. I've also stated that thing to somewhere other pull requests.
`external` has various kinds of modules, and we need to arrange that.
Maybe we can leave kind of connectors as it is (we can even rename on
this), and get flux, sql, kafka-monitor (this module is referred from script in
bin, too), and more to out of external.
Since this issue is not only regarding to storm-sql I would like to
initiate the discussion before making the change, and handle this to another
issue. What do you think?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
