Hi all, I'm running a 1.0.1 storm cluster with 3 nodes and i'm having a problem :
When I killed my topology, some (a lot) worker process are still runing on my 3 nodes : For example : s -ef |grep storm storm 5247 1 0 12:12 pts/0 00:00:03 /usr/lib/jvm/java-8-oracle/bin/java -cp /usr/local/storm/lib/objenesis-2.1.jar:/usr/local/storm/lib/slf4j-api-1.7.7.jar:/usr/local/storm/lib/storm-core-1.0.1.jar:/usr/local/storm/lib/disruptor-3.3.2.jar:/usr/local/storm/lib/minlog-1.3.0.jar:/usr/local/storm/lib/kryo-3.0.3.jar:/usr/local/storm/lib/clojure-1.7.0.jar:/usr/local/storm/lib/log4j-core-2.1.jar:/usr/local/storm/lib/storm-rename-hack-1.0.1.jar:/usr/local/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/local/storm/lib/reflectasm-1.10.1.jar:/usr/local/storm/lib/asm-5.0.3.jar:/usr/local/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/storm/lib/log4j-api-2.1.jar:/usr/local/storm/lib/servlet-api-2.5.jar:/usr/local/storm/conf:/var/lib/storm/supervisor/stormdist/monitoring_ingestion-1-1464351024/stormjar.jar -Xmx64m -Dlogfile.name=worker.log -Dstorm.home=/usr/local/storm -Dworkers.artifacts=/usr/local/storm/logs/workers-artifacts -Dstorm.id=monitoring_ingestion-1-1464351024 -Dworker.id=db3aa712-bcbb-468a-a838-5ced93dfbcf1 -Dworker.port=6702 -Dstorm.log.dir=/usr/local/storm/logs -Dlog4j.configurationFile=/usr/local/storm/log4j2/worker.xml -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector org.apache.storm.LogWriter /usr/lib/jvm/java-8-oracle/bin/java -server -Xmx768m -XX:+PrintGCDetails -Xloggc:artifacts/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=artifacts/heapdump -Djava.library.path=/var/lib/storm/supervisor/stormdist/monitoring_ingestion-1-1464351024/resources/Linux-amd64:/var/lib/storm/supervisor/stormdist/monitoring_ingestion-1-1464351024/resources:/usr/local/lib:/opt/local/lib:/usr/lib -Dlogfile.name=worker.log -Dstorm.home=/usr/local/storm -Dworkers.artifacts=/usr/local/storm/logs/workers-artifacts -Dstorm.conf.file= -Dstorm.options= -Dstorm.log.dir=/usr/local/storm/logs -Djava.io.tmpdir=/var/lib/storm/workers/db3aa712-bcbb-468a-a838-5ced93dfbcf1/tmp -Dlogging.sensitivity=S3 -Dlog4j.configurationFile=/usr/local/storm/log4j2/worker.xml -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector -Dstorm.id=monitoring_ingestion-1-1464351024 -Dworker.id=db3aa712-bcbb-468a-a838-5ced93dfbcf1 -Dworker.port=6702 -cp /usr/local/storm/lib/objenesis-2.1.jar:/usr/local/storm/lib/slf4j-api-1.7.7.jar:/usr/local/storm/lib/storm-core-1.0.1.jar:/usr/local/storm/lib/disruptor-3.3.2.jar:/usr/local/storm/lib/minlog-1.3.0.jar:/usr/local/storm/lib/kryo-3.0.3.jar:/usr/local/storm/lib/clojure-1.7.0.jar:/usr/local/storm/lib/log4j-core-2.1.jar:/usr/local/storm/lib/storm-rename-hack-1.0.1.jar:/usr/local/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/local/storm/lib/reflectasm-1.10.1.jar:/usr/local/storm/lib/asm-5.0.3.jar:/usr/local/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/storm/lib/log4j-api-2.1.jar:/usr/local/storm/lib/servlet-api-2.5.jar:/usr/local/storm/conf:/var/lib/storm/supervisor/stormdist/monitoring_ingestion-1-1464351024/stormjar.jar org.apache.storm.daemon.worker monitoring_ingestion-1-1464351024 28255ed2-ecc8-488c-8d57-92018bccc4c6 6702 db3aa712-bcbb-468a-a838-5ced93dfbcf1 storm 5256 1 0 12:12 pts/0 00:00:03 /usr/lib/jvm/java-8-oracle/bin/java -cp /usr/local/storm/lib/objenesis-2.1.jar:/usr/local/storm/lib/slf4j-api-1.7.7.jar:/usr/local/storm/lib/storm-core-1.0.1.jar:/usr/local/storm/lib/disruptor-3.3.2.jar:/usr/local/storm/lib/minlog-1.3.0.jar:/usr/local/storm/lib/kryo-3.0.3.jar:/usr/local/storm/lib/clojure-1.7.0.jar:/usr/local/storm/lib/log4j-core-2.1.jar:/usr/local/storm/lib/storm-rename-hack-1.0.1.jar:/usr/local/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/local/storm/lib/reflectasm-1.10.1.jar:/usr/local/storm/lib/asm-5.0.3.jar:/usr/local/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/storm/lib/log4j-api-2.1.jar:/usr/local/storm/lib/servlet-api-2.5.jar:/usr/local/storm/conf:/var/lib/storm/supervisor/stormdist/monitoring_ingestion-1-1464351024/stormjar.jar -Xmx64m -Dlogfile.name=worker.log -Dstorm.home=/usr/local/storm -Dworkers.artifacts=/usr/local/storm/logs/workers-artifacts -Dstorm.id=monitoring_ingestion-1-1464351024 -Dworker.id=d8d5ee43-2247-4082-8e7d-3260e6156a77 -Dworker.port=6703 -Dstorm.log.dir=/usr/local/storm/logs -Dlog4j.configurationFile=/usr/local/storm/log4j2/worker.xml -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector org.apache.storm.LogWriter /usr/lib/jvm/java-8-oracle/bin/java -server -Xmx768m -XX:+PrintGCDetails -Xloggc:artifacts/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=artifacts/heapdump -Djava.library.path=/var/lib/storm/supervisor/stormdist/monitoring_ingestion-1-1464351024/resources/Linux-amd64:/var/lib/storm/supervisor/stormdist/monitoring_ingestion-1-1464351024/resources:/usr/local/lib:/opt/local/lib:/usr/lib -Dlogfile.name=worker.log -Dstorm.home=/usr/local/storm -Dworkers.artifacts=/usr/local/storm/logs/workers-artifacts -Dstorm.conf.file= -Dstorm.options= -Dstorm.log.dir=/usr/local/storm/logs -Djava.io.tmpdir=/var/lib/storm/workers/d8d5ee43-2247-4082-8e7d-3260e6156a77/tmp -Dlogging.sensitivity=S3 -Dlog4j.configurationFile=/usr/local/storm/log4j2/worker.xml -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector -Dstorm.id=monitoring_ingestion-1-1464351024 -Dworker.id=d8d5ee43-2247-4082-8e7d-3260e6156a77 -Dworker.port=6703 -cp /usr/local/storm/lib/objenesis-2.1.jar:/usr/local/storm/lib/slf4j-api-1.7.7.jar:/usr/local/storm/lib/storm-core-1.0.1.jar:/usr/local/storm/lib/disruptor-3.3.2.jar:/usr/local/storm/lib/minlog-1.3.0.jar:/usr/local/storm/lib/kryo-3.0.3.jar:/usr/local/storm/lib/clojure-1.7.0.jar:/usr/local/storm/lib/log4j-core-2.1.jar:/usr/local/storm/lib/storm-rename-hack-1.0.1.jar:/usr/local/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/local/storm/lib/reflectasm-1.10.1.jar:/usr/local/storm/lib/asm-5.0.3.jar:/usr/local/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/storm/lib/log4j-api-2.1.jar:/usr/local/storm/lib/servlet-api-2.5.jar:/usr/local/storm/conf:/var/lib/storm/supervisor/stormdist/monitoring_ingestion-1-1464351024/stormjar.jar org.apache.storm.daemon.worker monitoring_ingestion-1-1464351024 28255ed2-ecc8-488c-8d57-92018bccc4c6 6703 d8d5ee43-2247-4082-8e7d-3260e6156a77 storm 5317 5247 17 12:12 pts/0 00:16:00 /usr/lib/jvm/java-8-oracle/bin/java -server -Xmx768m -XX:+PrintGCDetails -Xloggc:artifacts/gc.log -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=1M -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=artifacts/heapdump -Djava.library.path=/var/lib/storm/supervisor/stor dist/monitoring_ingestion-1-1464351024/resources/Linux-amd64:/var/lib/storm/supervisor/stormdist/monitoring_ingestion-1-1464351024/resources:/usr/local/lib:/opt/local/lib:/usr/lib -Dlogfile.name=worker.log -Dstorm.home=/usr/local/storm -Dworkers.artifacts=/usr/local/storm/logs/workers-artifacts -Dstorm.conf.file= -Dstorm.options= -Dstorm.log.dir=/usr/local/storm/logs -Djava.io.tmpdir=/var/lib/storm/workers/db3aa712-bcbb-468a-a838-5ced93dfbcf1/tmp -Dlogging.sensitivity=S3 -Dlog4j.configurationFile=/usr/local/storm/log4j2/worker.xml -DLog4jContextSelector=org.apache.logging.log4j.core.selector.BasicContextSelector -Dstorm.id=monitoring_ingestion-1-1464351024 -Dworker.id=db3aa712-bcbb-468a-a838-5ced93dfbcf1 -Dworker.port=6702 -cp /usr/local/storm/lib/objenesis-2.1.jar:/usr/local/storm/lib/slf4j-api-1.7.7.jar:/usr/local/storm/lib/storm-core-1.0.1.jar:/usr/local/storm/lib/disruptor-3.3.2.jar:/usr/local/storm/lib/minlog-1.3.0.jar:/usr/local/storm/lib/kryo-3.0.3.jar:/usr/local/storm/lib/clojure-1.7.0.jar:/usr/local/storm/lib/log4j-core-2.1.jar:/usr/local/storm/lib/storm-rename-hack-1.0.1.jar:/usr/local/storm/lib/log4j-slf4j-impl-2.1.jar:/usr/local/storm/lib/reflectasm-1.10.1.jar:/usr/local/storm/lib/asm-5.0.3.jar:/usr/local/storm/lib/log4j-over-slf4j-1.6.6.jar:/usr/local/storm/lib/log4j-api-2.1.jar:/usr/local/storm/lib/servlet-api-2.5.jar:/usr/local/storm/conf:/var/lib/storm/supervisor/stormdist/monitoring_ingestion-1-1464351024/stormjar.jar org.apache.storm.daemon.worker monitoring_ingestion-1-1464351024 28255ed2-ecc8-488c-8d57-92018bccc4c6 6702 db3aa712-bcbb-468a-a838-5ced93dfbcf1 I'm getting 4 remaining process on one of my node (4 slots on this one) with the command ps -ef |grep storm.daemon.worker | wc -l (minus the grep itself) And the web GUI didn't display any running task.. As a result I get some remaining connections and unacked messages in my source (RabbitMQ) : [image: Images intégrées 2] And when i'm (re)submitting a new topology, I got some serious performance issue on my cluster.. can anyone help me with this problem ? Thanks in advance !
