Hello! After building my project that comtaint the topology, I have a big fat jar of 75MB. I have dependencies on HBase, OpenTSDB and Kafka. I would like to reduce the size of the the jar due to the fact that we can a lot of instances of the topology running (aprox 100).
I have read in two posts: http://qnalist.com/questions/4712134/is-there-a-way-to-add-a-custom-jar-or-directory-of-jars-to-the-storm-classpath-without-copying-the-jar-s-to-storm-lib-folder and http://programmers.stackexchange.com/questions/238711/why-does-storm-not-supply-a-mechanism-for-supplying-topology-necessary-dependent I found two solutions. One is to put the commons libararies in Storm_install_dir/lib and the second one is to put them in folder USER_CONF_DIR defined in storm.yaml. In both acases we should take care to keep a single version of dependent libraries. This could be a potential issue, due to the fact that their might be common libraries that can interfere with storm libraries (such as zookeeper or appache-commons). Also, I'm running Storm from Ambari. In the above scenario: - what could be the best solution to keep the common dependent libraries? - If I configure the USER_CONF_DIR in Ambari will it be propagated to all cluster machines? I look forward for your answers. Thanks . Florin
