Resurrecting this thread. Were we able to come to a resolution on this? Or is it still causing problems?
Thanks, James 05.10.2016, 12:55, "[email protected]" <[email protected]>: > Right, I will try that. I guess my main question, though, is why would > that need more time? Is it a function of CPU, memory, disk, or otherwise? > Obviously if the timeout is the issue it's probably related to what I'm > sending into the cluster and how resources are allocated in relation to the > workload but I plan to scale this up roughly 10x and don't want to end up > with a 600 second timeout. > > Jon > > On Wed, Oct 5, 2016, 15:34 Otto Fowler <[email protected]> wrote: > > What you would have to do to try this is to edit > /usr/local/monit/stop_bro_topology.sh and add in a -w parameter. > to read: > > storm kill bro -w 60 > > save that and then try to stop if from monit again. > > If that works, then we will know. > > On October 5, 2016 at 15:18:43, [email protected] ([email protected]) wrote: > > topology.message.timeout.secs is set to 30 according to ambari. > > What you requested is essentially what I did before, except instead of > using `storm kill` I used monit to kill all of the topologies, and I waited > multiple minutes before stopping nimbus/supervisor/drpc, then waited a > couple more minutes before grabbing the prior metrics/processes. I've seen > this multiple times across the past week and each time I see it I shut > everything down, do a `kill -9` on the storm processes left behind, make > sure no processes are running as the storm user on my entire cluster, then > bring things up in ambari, then monit once ambari says > nimbus/supervisor/drpc is up. > > Jon > > On Wed, Oct 5, 2016 at 3:09 PM P. Taylor Goetz <[email protected]> wrote: > >> To build on what Otto said: >> >> What’s your topology.message.timeout.secs configuration value? The > > default >> is 30. >> >> If you issue the `storm kill` command without the -w option, Storm will >> first deactivate the topology, then wait topology.message.timeout.secs >> before actually shutting down the topology. >> >> Also, if you have topologies deployed, and stop ninmubs, supervisors, > > etc. >> the workers may continue to run (depending on how you have monit setup). >> >> Can you try the following: >> >> 1. Kill the topology and wait topology.message.timeout.sec so Storm >> properly shuts down the topology. >> 2. Stop nimbus/supervisor/etc. >> 3. Check to see if you still have threads left behind. >> >> -Taylor >> >> > On Oct 5, 2016, at 2:31 PM, Otto Fowler <[email protected]> > > wrote: >> > >> > The shutdown topology scripts only do a storm kill <item>. >> > They probably need to -w WAITTIME. >> > >> > >> > -- >> > >> > Sent with Airmail >> > >> > On October 5, 2016 at 13:53:33, [email protected] ([email protected]) >> wrote: >> > >> > Hi All, >> > >> > While tuning Metron I have been doing a large amount of storm topology >> > restarts, as I tune the parallelism and memory use of bolts. I have >> > noticed multiple times that some threads will be left behind on > > restart, >> > which eventually causes all topologies to fail to start due to >> > "java.lang.OutOfMemoryError: unable to create new native thread". I am >> > only using the bro, indexing, and enrichments topologies right now, and > > I >> > have seen all of them leave hanging processes behind. Has anybody else >> > seen anything like this? >> > >> > I'm exclusively using monit to do topology restarts, and Ambari to >> restart >> > supervisors/nimbus/UI/DRPC processes. When it is currently registering > > as >> > stopped in both monit and Ambari, I see the following threads left >> behind: >> >> [jzeolla@node1 storm]$ ps -austorm | wc -l >> >> 6 >> >> [jzeolla@node2 storm]$ ps -austorm | wc -l >> >> 12 >> >> [jzeolla@node3 storm]$ ps -austorm | wc -l >> >> 9 >> >> [jzeolla@node4 storm]$ ps -austorm | wc -l >> >> 11 >> >> [jzeolla@node5 storm]$ ps -austorm | wc -l >> >> 9 >> > >> > Some sample processes: >> >> storm 14257 0.1 0.2 6279184 293912 ? Sl Oct03 3:44 >> > /usr/jdk64/jdk1.8.0_60/bin/java -server -Ddaemon.name=logviewer >> > -Dstorm.options= -Dstorm.home=/usr/hdp/2.4.0.0-169/storm >> > -Dstorm.log.dir=/var/log/storm >> > > > -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > >> > >> > -Dstorm.conf.file= -cp >> > > > /usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/ > >> > >> > 2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0. > > 0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm:/usr/hdp/current/storm-supervisor/conf > >> > >> > -Xmx128m -Dlogfile.name=logviewer.log >> > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/cluster.xml >> > backtype.storm.daemon.logviewer >> >> storm 32072 0.0 0.1 37337056 177112 ? Sl Oct02 2:02 >> > /usr/jdk64/jdk1.8.0_60/bin/java -cp >> > > > /usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/ > >> > >> > 2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0. > > 0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > >> > >> > -Dlogfile.name=bro-13-1475456606-worker-6704.log >> > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.id=bro-13-1475456606 >> > -Dworker.id=f10b3f2f-3ef7-4e8c-b9d3-10e7fabdc4d7 -Dworker.port=6704 >> > -Dstorm.log.dir=/var/log/storm >> > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml >> > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > >> > >> > backtype.storm.LogWriter /usr/jdk64/jdk1.8.0_60/bin/java -server > > -Xmx20G >> > > > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6704_JVM > >> > >> > > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > >> > >> > -Dlogfile.name=bro-13-1475456606-worker-6704.log >> > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= >> -Dstorm.options= >> > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 >> > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml >> > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > >> > >> > -Dstorm.id=bro-13-1475456606 >> > -Dworker.id=f10b3f2f-3ef7-4e8c-b9d3-10e7fabdc4d7 -Dworker.port=6704 -cp >> > > > /usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/ > >> > >> > 2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0. > > 0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > >> > >> > backtype.storm.daemon.worker bro-13-1475456606 >> > 05236488-ed9a-4969-a74f-cb471d0e8fdf 6704 >> > f10b3f2f-3ef7-4e8c-b9d3-10e7fabdc4d7 >> >> storm 32127 121 7.4 32422464 9799588 ? Sl Oct02 4337:42 >> > /usr/jdk64/jdk1.8.0_60/bin/java -server -Xmx20G >> > > > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6704_JVM > >> > >> > > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > >> > >> > -Dlogfile.name=bro-13-1475456606-worker-6704.log >> > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= >> -Dstorm.options= >> > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 >> > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml >> > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > >> > >> > -Dstorm.id=bro-13-1475456606 >> > -Dworker.id=f10b3f2f-3ef7-4e8c-b9d3-10e7fabdc4d7 -Dworker.port=6704 -cp >> > > > /usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/ > >> > >> > 2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0. > > 0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > >> > >> > backtype.storm.daemon.worker bro-13-1475456606 >> > 05236488-ed9a-4969-a74f-cb471d0e8fdf 6704 >> > f10b3f2f-3ef7-4e8c-b9d3-10e7fabdc4d7 >> >> storm 47882 0.0 0.1 37338556 188920 ? Sl Oct04 0:47 >> > /usr/jdk64/jdk1.8.0_60/bin/java -cp >> > > > /usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/ > >> > >> > 2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0. > > 0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/indexing-16-1475589543/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > >> > >> > -Dlogfile.name=indexing-16-1475589543-worker-6700.log >> > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm > > -Dstorm.id=indexing-16-1475589543 >> > -Dworker.id=5e17c5c3-e6ab-4b48-a48a-dc540cda7087 -Dworker.port=6700 >> > -Dstorm.log.dir=/var/log/storm >> > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml >> > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > >> > >> > backtype.storm.LogWriter /usr/jdk64/jdk1.8.0_60/bin/java -server -Xmx2G >> > > > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6700_JVM > >> > >> > > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/indexing-16-1475589543/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/indexing-16-1475589543/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > >> > >> > -Dlogfile.name=indexing-16-1475589543-worker-6700.log >> > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= >> -Dstorm.options= >> > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 >> > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml >> > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > >> > >> > -Dstorm.id=indexing-16-1475589543 >> > -Dworker.id=5e17c5c3-e6ab-4b48-a48a-dc540cda7087 -Dworker.port=6700 -cp >> > > > /usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/ > >> > >> > 2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0. > > 0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/indexing-16-1475589543/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > >> > >> > backtype.storm.daemon.worker indexing-16-1475589543 >> > 05236488-ed9a-4969-a74f-cb471d0e8fdf 6700 >> > 5e17c5c3-e6ab-4b48-a48a-dc540cda7087 >> >> storm 47978 4.7 0.5 19996036 740372 ? Sl Oct04 65:04 >> > /usr/jdk64/jdk1.8.0_60/bin/java -server -Xmx2G >> > > > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6700_JVM > >> > >> > > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/indexing-16-1475589543/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/indexing-16-1475589543/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > >> > >> > -Dlogfile.name=indexing-16-1475589543-worker-6700.log >> > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= >> -Dstorm.options= >> > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 >> > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml >> > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > >> > >> > -Dstorm.id=indexing-16-1475589543 >> > -Dworker.id=5e17c5c3-e6ab-4b48-a48a-dc540cda7087 -Dworker.port=6700 -cp >> > > > /usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/ > >> > >> > 2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0. > > 0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/indexing-16-1475589543/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > >> > >> > backtype.storm.daemon.worker indexing-16-1475589543 >> > 05236488-ed9a-4969-a74f-cb471d0e8fdf 6700 >> > 5e17c5c3-e6ab-4b48-a48a-dc540cda7087 >> >> storm 21301 0.1 0.2 6347824 298632 ? Sl Oct03 3:44 >> > /usr/jdk64/jdk1.8.0_60/bin/java -server -Ddaemon.name=logviewer >> > -Dstorm.options= -Dstorm.home=/usr/hdp/2.4.0.0-169/storm >> > -Dstorm.log.dir=/var/log/storm >> > > > -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > >> > >> > -Dstorm.conf.file= -cp >> > > > /usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/ > >> > >> > > > 2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm:/usr/hdp/current/storm-supervisor/conf > >> > >> > -Xmx128m -Dlogfile.name=logviewer.log >> > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/cluster.xml >> > backtype.storm.daemon.logviewer >> >> storm 14046 102 19.3 42125944 25539856 ? Sl Oct03 2655:13 >> > /usr/jdk64/jdk1.8.0_60/bin/java -server -Xmx20G >> > > > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6700_JVM > >> > >> > > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/enrichment-12-1475456272/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/enrichment-12-1475456272/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > >> > >> > -Dlogfile.name=enrichment-12-1475456272-worker-6700.log >> > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= >> -Dstorm.options= >> > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 >> > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml >> > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > >> > >> > -Dstorm.id=enrichment-12-1475456272 >> > -Dworker.id=5d586063-e405-4a67-a644-53a0ac943ab0 -Dworker.port=6700 -cp >> > > > /usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/ > >> > >> > > > 2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/enrichment-12-1475456272/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > >> > >> > backtype.storm.daemon.worker enrichment-12-1475456272 >> > eff7f971-ebe2-4f53-beab-99b061bfde10 6700 >> > 5d586063-e405-4a67-a644-53a0ac943ab0 >> >> storm 6544 0.0 0.1 37339132 180872 ? Sl Oct02 3:22 >> > /usr/jdk64/jdk1.8.0_60/bin/java -cp >> > > > /usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/ > >> > >> > > > 2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > >> > >> > -Dlogfile.name=bro-13-1475456606-worker-6704.log >> > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.id=bro-13-1475456606 >> > -Dworker.id=f917291c-bd1e-4a9c-a52d-b9c5fef0567e -Dworker.port=6704 >> > -Dstorm.log.dir=/var/log/storm >> > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml >> > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > >> > >> > backtype.storm.LogWriter /usr/jdk64/jdk1.8.0_60/bin/java -server > > -Xmx20G >> > > > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6704_JVM > >> > >> > > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > >> > >> > -Dlogfile.name=bro-13-1475456606-worker-6704.log >> > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= >> -Dstorm.options= >> > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 >> > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml >> > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > >> > >> > -Dstorm.id=bro-13-1475456606 >> > -Dworker.id=f917291c-bd1e-4a9c-a52d-b9c5fef0567e -Dworker.port=6704 -cp >> > > > /usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/ > >> > >> > > > 2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > >> > >> > backtype.storm.daemon.worker bro-13-1475456606 >> > eff7f971-ebe2-4f53-beab-99b061bfde10 6704 >> > f917291c-bd1e-4a9c-a52d-b9c5fef0567e >> >> storm 6592 192 9.8 34986568 12987704 ? Sl Oct02 6866:30 >> > /usr/jdk64/jdk1.8.0_60/bin/java -server -Xmx20G >> > > > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6704_JVM > >> > >> > > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > >> > >> > -Dlogfile.name=bro-13-1475456606-worker-6704.log >> > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= >> -Dstorm.options= >> > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 >> > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml >> > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > >> > >> > -Dstorm.id=bro-13-1475456606 >> > -Dworker.id=f917291c-bd1e-4a9c-a52d-b9c5fef0567e -Dworker.port=6704 -cp >> > > > /usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/ > >> > >> > > > 2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/bro-13-1475456606/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > >> > >> > backtype.storm.daemon.worker bro-13-1475456606 >> > eff7f971-ebe2-4f53-beab-99b061bfde10 6704 >> > f917291c-bd1e-4a9c-a52d-b9c5fef0567e >> >> storm 13998 0.0 0.1 37339104 184140 ? Sl Oct03 1:57 >> > /usr/jdk64/jdk1.8.0_60/bin/java -cp >> > > > /usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/ > >> > >> > > > 2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/enrichment-12-1475456272/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > >> > >> > -Dlogfile.name=enrichment-12-1475456272-worker-6700.log >> > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm >> -Dstorm.id=enrichment-12-1475456272 >> > -Dworker.id=5d586063-e405-4a67-a644-53a0ac943ab0 -Dworker.port=6700 >> > -Dstorm.log.dir=/var/log/storm >> > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml >> > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > >> > >> > backtype.storm.LogWriter /usr/jdk64/jdk1.8.0_60/bin/java -server > > -Xmx20G >> > > > -javaagent:/usr/hdp/current/storm-client/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar=host=localhost,port=8650,wireformat31x=true,mode=multicast,config=/usr/hdp/current/storm-client/contrib/storm-jmxetric/conf/jmxetric-conf.xml,process=Worker_6700_JVM > >> > >> > > > -Djava.library.path=/Data_1/hadoop/storm/supervisor/stormdist/enrichment-12-1475456272/resources/Linux-amd64:/Data_1/hadoop/storm/supervisor/stormdist/enrichment-12-1475456272/resources:/usr/local/lib:/opt/local/lib:/usr/lib:/usr/hdp/current/storm-client/lib > >> > >> > -Dlogfile.name=enrichment-12-1475456272-worker-6700.log >> > -Dstorm.home=/usr/hdp/2.4.0.0-169/storm -Dstorm.conf.file= >> -Dstorm.options= >> > -Dstorm.log.dir=/var/log/storm -Dlogging.sensitivity=S3 >> > -Dlog4j.configurationFile=/usr/hdp/2.4.0.0-169/storm/log4j2/worker.xml >> > > > -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector > >> > >> > -Dstorm.id=enrichment-12-1475456272 >> > -Dworker.id=5d586063-e405-4a67-a644-53a0ac943ab0 -Dworker.port=6700 -cp >> > > > /usr/hdp/2.4.0.0-169/storm/lib/clout-1.0.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/hadoop-auth-2.7.1.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-devel-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/javax.servlet-2.5.0.v201103041518.jar:/usr/hdp/2.4.0.0-169/storm/lib/tigris-0.1.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/servlet-api-2.5.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-core-1.1.5.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/java.classpath-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/asm-4.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/gmetric4j-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/hiccup-0.3.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/compojure-1.1.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/tools.logging-0.2.3.jar:/usr/hdp/2.4.0.0-169/storm/lib/disruptor-2.10.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clojure-1.6.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-core-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/cheshire-5.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-servlet-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/oncrpc-1.0.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/hdp/2.4.0.0-169/storm/lib/reflectasm-1.07-shaded.jar:/usr/hdp/2.4.0.0-169/storm/lib/jline-0.9.94.jar:/usr/hdp/2.4.0.0-169/storm/lib/jackson-dataformat-smile-2.3.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-jetty-adapter-1.3.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/storm-core-0.10.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/lib/ns-tracker-0.2.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/ring-json-0.3.1.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/tools.namespace-0.2.4.jar:/usr/hdp/2.4.0.0-169/storm/lib/commons-codec-1.6.jar:/usr/hdp/ > >> > >> > 2.4.0. >> > > > 0-169/storm/lib/clj-time-0.8.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/log4j-api-2.1.jar:/usr/hdp/2.4.0.0-169/storm/lib/clj-stacktrace-0.2.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/slf4j-api-1.7.7.jar:/usr/hdp/2.4.0.0-169/storm/lib/minlog-1.2.jar:/usr/hdp/2.4.0.0-169/storm/lib/core.incubator-0.1.0.jar:/usr/hdp/2.4.0.0-169/storm/lib/kryo-2.21.jar:/usr/hdp/2.4.0.0-169/storm/lib/zookeeper.jar:/usr/hdp/ > >> > >> > > > 2.4.0.0-169/storm/lib/log4j-core-2.1.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-storm-plugin-shim-0.5.0.2.4.0.0-169.jar:/usr/hdp/2.4.0.0-169/storm/extlib-daemon/ranger-plugin-classloader-0.5.0.2.4.0.0-169.jar:/usr/hdp/current/storm-supervisor/conf:/usr/hdp/current/storm-supervisor/contrib/storm-jmxetric/lib/jmxetric-1.0.4.jar:/Data_1/hadoop/storm/supervisor/stormdist/enrichment-12-1475456272/stormjar.jar:/etc/hbase/conf:/etc/hadoop/conf > >> > >> > backtype.storm.daemon.worker enrichment-12-1475456272 >> > eff7f971-ebe2-4f53-beab-99b061bfde10 6700 >> > 5d586063-e405-4a67-a644-53a0ac943ab0 >> > >> > >> > Logs showing the inability to start a topology: >> >> 2016-10-04 10:10:57.787 b.s.m.n.Client [INFO] creating Netty Client, >> > connecting to pgh-dat-t53.iso.local.cmu.edu:6702, bufferSize: 5242880 >> >> 2016-10-04 10:10:57.787 o.a.s.c.r.ExponentialBackoffRetry [WARN] >> > maxRetries too large (30). Pinning to 29 >> >> 2016-10-04 10:10:57.787 b.s.u.StormBoundedExponentialBackoffRetry > > [INFO] >> > The baseSleepTimeMs [100] the maxSleepTimeMs [1000] the maxRetries [30] >> >> 2016-10-04 10:10:57.787 b.s.m.n.Client [INFO] connection attempt 1 to >> > Netty-Client-pgh-dat-t53.iso.local.cmu.edu/172.19.247.79:6702 scheduled >> to >> > run in 0 ms >> >> 2016-10-04 10:10:57.790 b.s.m.n.Client [INFO] connection established > > to >> > Netty-Client-pgh-dat-t53.iso.local.cmu.edu/172.19.247.79:6703 >> >> 2016-10-04 10:10:57.790 b.s.m.n.Client [INFO] connection established > > to >> > Netty-Client-pgh-dat-t53.iso.local.cmu.edu/172.19.247.79:6702 >> >> 2016-10-04 10:10:57.790 b.s.m.n.Client [INFO] connection established > > to >> > Netty-Client-pgh-dat-t53.iso.local.cmu.edu/172.19.247.79:6701 >> >> 2016-10-04 10:10:58.030 b.s.d.executor [INFO] Loading executor >> > enrichmentJoinBolt:[2 2] >> >> 2016-10-04 10:10:58.064 b.s.d.executor [INFO] Loaded executor tasks >> > enrichmentJoinBolt:[2 2] >> >> 2016-10-04 10:10:58.071 b.s.d.executor [INFO] Finished loading > > executor >> > enrichmentJoinBolt:[2 2] >> >> 2016-10-04 10:10:58.081 b.s.d.executor [INFO] Loading executor >> > simpleHBaseThreatIntelBolt:[22 22] >> >> 2016-10-04 10:10:58.095 b.s.d.executor [INFO] Loaded executor tasks >> > simpleHBaseThreatIntelBolt:[22 22] >> >> 2016-10-04 10:10:58.097 b.s.d.executor [INFO] Finished loading > > executor >> > simpleHBaseThreatIntelBolt:[22 22] >> >> 2016-10-04 10:10:58.104 b.s.d.executor [INFO] Loading executor >> > kafkaSpout:[18 18] >> >> 2016-10-04 10:10:58.112 b.s.d.executor [INFO] Loaded executor tasks >> > kafkaSpout:[18 18] >> >> 2016-10-04 10:10:58.118 b.s.d.executor [INFO] Finished loading > > executor >> > kafkaSpout:[18 18] >> >> 2016-10-04 10:10:58.125 b.s.d.executor [INFO] Loading executor >> > geoEnrichmentBolt:[6 6] >> >> 2016-10-04 10:10:58.135 b.s.d.executor [INFO] Loaded executor tasks >> > geoEnrichmentBolt:[6 6] >> >> 2016-10-04 10:10:58.137 b.s.d.executor [ERROR] >> >> java.lang.OutOfMemoryError: unable to create new native thread >> >> at java.lang.Thread.start0(Native Method) ~[?:1.8.0_60] >> >> at java.lang.Thread.start(Thread.java:714) ~[?:1.8.0_60] >> >> at backtype.storm.util$async_loop.doInvoke(util.clj:491) >> > ~[storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> >> at clojure.lang.RestFn.invoke(RestFn.java:559) >> > ~[clojure-1.6.0.jar:?] >> >> at backtype.storm.daemon.executor$fn__7245.invoke(executor.clj:820) >> > ~[storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> >> at clojure.lang.MultiFn.invoke(MultiFn.java:236) >> > ~[clojure-1.6.0.jar:?] >> >> at >> > > > backtype.storm.daemon.executor$mk_executor$fn__7116.invoke(executor.clj:351) > >> > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> >> at >> > backtype.storm.daemon.executor$mk_executor.invoke(executor.clj:350) >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> >> at >> > > > backtype.storm.daemon.worker$fn__7719$exec_fn__1353__auto__$reify__7721$iter__7726__7730$fn__7731.invoke(worker.clj:541) > >> > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> >> at clojure.lang.LazySeq.sval(LazySeq.java:40) [clojure-1.6.0.jar:?] >> >> at clojure.lang.LazySeq.seq(LazySeq.java:49) [clojure-1.6.0.jar:?] >> >> at clojure.lang.Cons.next(Cons.java:39) [clojure-1.6.0.jar:?] >> >> at clojure.lang.RT.next(RT.java:598) [clojure-1.6.0.jar:?] >> >> at clojure.core$next.invoke(core.clj:64) [clojure-1.6.0.jar:?] >> >> at clojure.core$dorun.invoke(core.clj:2856) [clojure-1.6.0.jar:?] >> >> at clojure.core$doall.invoke(core.clj:2871) [clojure-1.6.0.jar:?] >> >> at >> > > > backtype.storm.daemon.worker$fn__7719$exec_fn__1353__auto__$reify__7721.run(worker.clj:541) > >> > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> >> at java.security.AccessController.doPrivileged(Native Method) >> > ~[?:1.8.0_60] >> >> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_60] >> >> at >> > > > backtype.storm.daemon.worker$fn__7719$exec_fn__1353__auto____7720.invoke(worker.clj:515) > >> > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> >> at clojure.lang.AFn.applyToHelper(AFn.java:178) >> > [clojure-1.6.0.jar:?] >> >> at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.6.0.jar:?] >> >> at clojure.core$apply.invoke(core.clj:624) [clojure-1.6.0.jar:?] >> >> at >> > > > backtype.storm.daemon.worker$fn__7719$mk_worker__7805.doInvoke(worker.clj:491) > >> > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> >> at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.6.0.jar:?] >> >> at backtype.storm.daemon.worker$_main.invoke(worker.clj:641) >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> >> at clojure.lang.AFn.applyToHelper(AFn.java:165) >> > [clojure-1.6.0.jar:?] >> >> at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.6.0.jar:?] >> >> at backtype.storm.daemon.worker.main(Unknown Source) >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> >> 2016-10-04 10:10:58.169 b.s.util [ERROR] Halting process: ("Worker >> died") >> >> java.lang.RuntimeException: ("Worker died") >> >> at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:332) >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> >> at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.6.0.jar:?] >> >> at >> > backtype.storm.daemon.worker$fn__7818$fn__7819.invoke(worker.clj:636) >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> >> at >> > > > backtype.storm.daemon.executor$mk_executor_data$fn__7078$fn__7079.invoke(executor.clj:256) > >> > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> >> at >> > > > backtype.storm.daemon.executor$mk_executor$fn__7116.invoke(executor.clj:350) > >> > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> >> at >> > backtype.storm.daemon.executor$mk_executor.invoke(executor.clj:350) >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> >> at >> > > > backtype.storm.daemon.worker$fn__7719$exec_fn__1353__auto__$reify__7721$iter__7726__7730$fn__7731.invoke(worker.clj:541) > >> > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> >> at clojure.lang.LazySeq.sval(LazySeq.java:40) [clojure-1.6.0.jar:?] >> >> at clojure.lang.LazySeq.seq(LazySeq.java:49) [clojure-1.6.0.jar:?] >> >> at clojure.lang.Cons.next(Cons.java:39) [clojure-1.6.0.jar:?] >> >> at clojure.lang.RT.next(RT.java:598) [clojure-1.6.0.jar:?] >> >> at clojure.core$next.invoke(core.clj:64) [clojure-1.6.0.jar:?] >> >> at clojure.core$dorun.invoke(core.clj:2856) [clojure-1.6.0.jar:?] >> >> at clojure.core$doall.invoke(core.clj:2871) [clojure-1.6.0.jar:?] >> >> at >> > > > backtype.storm.daemon.worker$fn__7719$exec_fn__1353__auto__$reify__7721.run(worker.clj:541) > >> > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> >> at java.security.AccessController.doPrivileged(Native Method) >> > ~[?:1.8.0_60] >> >> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_60] >> >> at >> > > > backtype.storm.daemon.worker$fn__7719$exec_fn__1353__auto____7720.invoke(worker.clj:515) > >> > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> >> at clojure.lang.AFn.applyToHelper(AFn.java:178) >> > [clojure-1.6.0.jar:?] >> >> at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.6.0.jar:?] >> >> at clojure.core$apply.invoke(core.clj:624) [clojure-1.6.0.jar:?] >> >> at >> > > > backtype.storm.daemon.worker$fn__7719$mk_worker__7805.doInvoke(worker.clj:491) > >> > >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> >> at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.6.0.jar:?] >> >> at backtype.storm.daemon.worker$_main.invoke(worker.clj:641) >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> >> at clojure.lang.AFn.applyToHelper(AFn.java:165) >> > [clojure-1.6.0.jar:?] >> >> at clojure.lang.AFn.applyTo(AFn.java:144) [clojure-1.6.0.jar:?] >> >> at backtype.storm.daemon.worker.main(Unknown Source) >> > [storm-core-0.10.0.2.4.0.0-169.jar:0.10.0.2.4.0.0-169] >> > -- >> > >> > Jon >> >> -- > > Jon > > -- > > Jon ------------------- Thank you, James Sirota PPMC- Apache Metron (Incubating) jsirota AT apache DOT org
