Damien DESMARETS created STORM-840: -------------------------------------- Summary: My supervisor crashes when I kill a topology Key: STORM-840 URL: https://issues.apache.org/jira/browse/STORM-840 Project: Apache Storm Issue Type: Bug Affects Versions: 0.9.4 Environment: I have a test cluster of 3 servers base on Debian. Each server use a docker running storm inside.
2 servers are only supervisor. 1 server is nimbus+UI+supervisor. I use Oracle JVM 8u45. Reporter: Damien DESMARETS Hello, I run 3 topologies inside my cluster. Sometimes, when I kill one of them (not one specific). One supervisor goes down and restart. After few restart, it become stable. The topology process is in "Zombie state" in the process list. In version 0.9.3, all the supervisors crashed and couldn't restart. To resolve this, I had to "rm -fr <storm-local-dir>/workers/" So I migrate to 0.9.4 (I thought that was STORM-682). Now it continues but no all the times, but occasionally. I have these logs inside supervisor.log: 2015-05-29 15:01:42 b.s.d.supervisor [INFO] Removing code for storm id nlp-11-1432906756 2015-05-29 15:01:42 b.s.d.supervisor [INFO] Removing code for storm id nlp-11-1432906756 2015-05-29 15:01:42 b.s.d.supervisor [INFO] Shutting down and clearing state for id 355af307-fafc-43a8-865d-0dfbf9baee33. Current supervisor time: 1432911702. State: :disallowed, Heartbeat: #backtype.storm.daemon.common.WorkerHeartbeat{:time-secs 1432911702, :storm-id "nlp-11-1432906756", :executors #{[2 2] [3 3] [-1 -1] [1 1]}, :port 6700} 2015-05-29 15:01:42 b.s.d.supervisor [INFO] Shutting down and clearing state for id 355af307-fafc-43a8-865d-0dfbf9baee33. Current supervisor time: 1432911702. State: :disallowed, Heartbeat: #backtype.storm.daemon.common.WorkerHeartbeat{:time-secs 1432911702, :storm-id "nlp-11-1432906756", :executors #{[2 2] [3 3] [-1 -1] [1 1]}, :port 6700} 2015-05-29 15:01:42 b.s.d.supervisor [INFO] Shutting down 90f0964b-c48c-4cbc-9d1c-57119c56e99c:355af307-fafc-43a8-865d-0dfbf9baee33 2015-05-29 15:01:42 b.s.d.supervisor [INFO] Shutting down 90f0964b-c48c-4cbc-9d1c-57119c56e99c:355af307-fafc-43a8-865d-0dfbf9baee33 2015-05-29 15:01:42 b.s.event [ERROR] Error when processing event java.io.IOException: Cannot run program "kill" (in directory "."): error=2, No such file or directory at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) ~[na:1.8.0_45] at java.lang.Runtime.exec(Runtime.java:620) ~[na:1.8.0_45] at org.apache.commons.exec.launcher.Java13CommandLauncher.exec(Java13CommandLauncher.java:58) ~[commons-exec-1.1.jar:1.1] at org.apache.commons.exec.DefaultExecutor.launch(DefaultExecutor.java:254) ~[commons-exec-1.1.jar:1.1] at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:319) ~[commons-exec-1.1.jar:1.1] at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:160) ~[commons-exec-1.1.jar:1.1] at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:147) ~[commons-exec-1.1.jar:1.1] at backtype.storm.util$exec_command_BANG_.invoke(util.clj:386) ~[storm-core-0.9.4.jar:0.9.4] at backtype.storm.util$send_signal_to_process.invoke(util.clj:415) ~[storm-core-0.9.4.jar:0.9.4] at backtype.storm.util$kill_process_with_sig_term.invoke(util.clj:426) ~[storm-core-0.9.4.jar:0.9.4] at backtype.storm.daemon.supervisor$shutdown_worker.invoke(supervisor.clj:197) ~[storm-core-0.9.4.jar:0.9.4] at backtype.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:267) ~[storm-core-0.9.4.jar:0.9.4] at clojure.lang.AFn.applyToHelper(AFn.java:161) [clojure-1.5.1.jar:na] at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na] at clojure.core$apply.invoke(core.clj:619) ~[clojure-1.5.1.jar:na] at clojure.core$partial$fn__4190.doInvoke(core.clj:2396) ~[clojure-1.5.1.jar:na] at clojure.lang.RestFn.invoke(RestFn.java:397) ~[clojure-1.5.1.jar:na] at backtype.storm.event$event_manager$fn__2809.invoke(event.clj:40) ~[storm-core-0.9.4.jar:0.9.4] at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45] Caused by: java.io.IOException: error=2, No such file or directory at java.lang.UNIXProcess.forkAndExec(Native Method) ~[na:1.8.0_45] at java.lang.UNIXProcess.<init>(UNIXProcess.java:248) ~[na:1.8.0_45] at java.lang.ProcessImpl.start(ProcessImpl.java:134) ~[na:1.8.0_45] at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) ~[na:1.8.0_45] ... 19 common frames omitted 2015-05-29 15:01:42 b.s.event [ERROR] Error when processing event java.io.IOException: Cannot run program "kill" (in directory "."): error=2, No such file or directory at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) ~[na:1.8.0_45] at java.lang.Runtime.exec(Runtime.java:620) ~[na:1.8.0_45] at org.apache.commons.exec.launcher.Java13CommandLauncher.exec(Java13CommandLauncher.java:58) ~[commons-exec-1.1.jar:1.1] at org.apache.commons.exec.DefaultExecutor.launch(DefaultExecutor.java:254) ~[commons-exec-1.1.jar:1.1] at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:319) ~[commons-exec-1.1.jar:1.1] at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:160) ~[commons-exec-1.1.jar:1.1] at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:147) ~[commons-exec-1.1.jar:1.1] at backtype.storm.util$exec_command_BANG_.invoke(util.clj:386) ~[storm-core-0.9.4.jar:0.9.4] at backtype.storm.util$send_signal_to_process.invoke(util.clj:415) ~[storm-core-0.9.4.jar:0.9.4] at backtype.storm.util$kill_process_with_sig_term.invoke(util.clj:426) ~[storm-core-0.9.4.jar:0.9.4] at backtype.storm.daemon.supervisor$shutdown_worker.invoke(supervisor.clj:197) ~[storm-core-0.9.4.jar:0.9.4] at backtype.storm.daemon.supervisor$sync_processes.invoke(supervisor.clj:267) ~[storm-core-0.9.4.jar:0.9.4] at clojure.lang.AFn.applyToHelper(AFn.java:161) [clojure-1.5.1.jar:na] at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.5.1.jar:na] at clojure.core$apply.invoke(core.clj:619) ~[clojure-1.5.1.jar:na] at clojure.core$partial$fn__4190.doInvoke(core.clj:2396) ~[clojure-1.5.1.jar:na] at clojure.lang.RestFn.invoke(RestFn.java:397) ~[clojure-1.5.1.jar:na] at backtype.storm.event$event_manager$fn__2809.invoke(event.clj:40) ~[storm-core-0.9.4.jar:0.9.4] at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45] Caused by: java.io.IOException: error=2, No such file or directory at java.lang.UNIXProcess.forkAndExec(Native Method) ~[na:1.8.0_45] at java.lang.UNIXProcess.<init>(UNIXProcess.java:248) ~[na:1.8.0_45] at java.lang.ProcessImpl.start(ProcessImpl.java:134) ~[na:1.8.0_45] at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) ~[na:1.8.0_45] ... 19 common frames omitted 2015-05-29 15:01:42 b.s.util [ERROR] Halting process: ("Error when processing an event") java.lang.RuntimeException: ("Error when processing an event") at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325) [storm-core-0.9.4.jar:0.9.4] at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.5.1.jar:na] at backtype.storm.event$event_manager$fn__2809.invoke(event.clj:48) [storm-core-0.9.4.jar:0.9.4] at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45] 2015-05-29 15:01:42 b.s.util [ERROR] Halting process: ("Error when processing an event") java.lang.RuntimeException: ("Error when processing an event") at backtype.storm.util$exit_process_BANG_.doInvoke(util.clj:325) [storm-core-0.9.4.jar:0.9.4] at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.5.1.jar:na] at backtype.storm.event$event_manager$fn__2809.invoke(event.clj:48) [storm-core-0.9.4.jar:0.9.4] at clojure.lang.AFn.run(AFn.java:24) [clojure-1.5.1.jar:na] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45] 2015-05-29 15:01:42 b.s.d.supervisor [INFO] Shutting down supervisor 90f0964b-c48c-4cbc-9d1c-57119c56e99c 2015-05-29 15:01:42 b.s.d.supervisor [INFO] Shutting down supervisor 90f0964b-c48c-4cbc-9d1c-57119c56e99c 2015-05-29 15:01:42 b.s.event [INFO] Event manager interrupted 2015-05-29 15:01:42 b.s.event [INFO] Event manager interrupted 2015-05-29 15:01:53 o.a.s.z.ZooKeeper [INFO] Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2015-05-29 15:01:53 o.a.s.z.ZooKeeper [INFO] Client environment:host.name=storm-supervisor-01 2015-05-29 15:01:53 o.a.s.z.ZooKeeper [INFO] Client environment:java.version=1.8.0_45 2015-05-29 15:01:53 o.a.s.z.ZooKeeper [INFO] Client environment:java.vendor=Oracle Corporation 2015-05-29 15:01:53 o.a.s.z.ZooKeeper [INFO] Client environment:java.home=/usr/lib/jvm/jre-8-oracle-x64/jre 2015-05-29 15:01:53 o.a.s.z.ZooKeeper [INFO] Client environment:java.class.path=/usr/share/apache-storm-0.9.4/lib/zookeeper-3.4.6.jar:/usr/share/apache-storm-0.9.4/lib/hiccup-0.3.6.jar:/usr/share/apache-storm-0.9.4/lib/chill-java-0.3.5.jar:/usr/share/apache-storm-0.9.4/lib/commons-exec-1.1.jar:/usr/share/apache-storm-0.9.4/lib/tools.macro-0.1.0.jar:/usr/share/apache-storm-0.9.4/lib/jgrapht-core-0.9.0.jar:/usr/share/apache-storm-0.9.4/lib/ring-servlet-0.3.11.jar:/usr/share/apache-storm-0.9.4/lib/clout-1.0.1.jar:/usr/share/apache-storm-0.9.4/lib/storm-core-0.9.4.jar:/usr/share/apache-storm-0.9.4/lib/asm-4.0.jar:/usr/share/apache-storm-0.9.4/lib/tools.cli-0.2.4.jar:/usr/share/apache-storm-0.9.4/lib/disruptor-2.10.1.jar:/usr/share/apache-storm-0.9.4/lib/log4j-over-slf4j-1.6.6.jar:/usr/share/apache-storm-0.9.4/lib/clj-time-0.4.1.jar:/usr/share/apache-storm-0.9.4/lib/slf4j-api-1.7.5.jar:/usr/share/apache-storm-0.9.4/lib/clojure-1.5.1.jar:/usr/share/apache-storm-0.9.4/lib/core.incubator-0.1.0.jar:/usr/share/apache-storm-0.9.4/lib/json-simple-1.1.jar:/usr/share/apache-storm-0.9.4/lib/logback-classic-1.0.13.jar:/usr/share/apache-storm-0.9.4/lib/servlet-api-2.5.jar:/usr/share/apache-storm-0.9.4/lib/logback-core-1.0.13.jar:/usr/share/apache-storm-0.9.4/lib/jetty-6.1.26.jar:/usr/share/apache-storm-0.9.4/lib/clj-stacktrace-0.2.2.jar:/usr/share/apache-storm-0.9.4/lib/ring-devel-0.3.11.jar:/usr/share/apache-storm-0.9.4/lib/minlog-1.2.jar:/usr/share/apache-storm-0.9.4/lib/kryo-2.21.jar:/usr/share/apache-storm-0.9.4/lib/compojure-1.1.3.jar:/usr/share/apache-storm-0.9.4/lib/commons-codec-1.6.jar:/usr/share/apache-storm-0.9.4/lib/tools.logging-0.2.3.jar:/usr/share/apache-storm-0.9.4/lib/ring-jetty-adapter-0.3.11.jar:/usr/share/apache-storm-0.9.4/lib/jetty-util-6.1.26.jar:/usr/share/apache-storm-0.9.4/lib/joda-time-2.0.jar:/usr/share/apache-storm-0.9.4/lib/jline-2.11.jar:/usr/share/apache-storm-0.9.4/lib/commons-logging-1.1.3.jar:/usr/share/apache-storm-0.9.4/lib/reflectasm-1.07-shaded.jar:/usr/share/apache-storm-0.9.4/lib/carbonite-1.4.0.jar:/usr/share/apache-storm-0.9.4/lib/snakeyaml-1.11.jar:/usr/share/apache-storm-0.9.4/lib/objenesis-1.2.jar:/usr/share/apache-storm-0.9.4/lib/ring-core-1.1.5.jar:/usr/share/apache-storm-0.9.4/lib/commons-io-2.4.jar:/usr/share/apache-storm-0.9.4/lib/commons-fileupload-1.2.1.jar:/usr/share/apache-storm-0.9.4/lib/math.numeric-tower-0.0.1.jar:/usr/share/apache-storm-0.9.4/lib/commons-lang-2.5.jar:/usr/share/apache-storm-0.9.4/conf 2015-05-29 15:01:53 o.a.s.z.ZooKeeper [INFO] Client environment:java.library.path=/usr/local/lib:/opt/local/lib:/usr/lib 2015-05-29 15:01:53 o.a.s.z.ZooKeeper [INFO] Client environment:java.io.tmpdir=/tmp 2015-05-29 15:01:53 o.a.s.z.ZooKeeper [INFO] Client environment:java.compiler=<NA> 2015-05-29 15:01:53 o.a.s.z.ZooKeeper [INFO] Client environment:os.name=Linux 2015-05-29 15:01:53 o.a.s.z.ZooKeeper [INFO] Client environment:os.arch=amd64 2015-05-29 15:01:53 o.a.s.z.ZooKeeper [INFO] Client environment:os.version=3.16.0-0.bpo.4-amd64 ... -- This message was sent by Atlassian JIRA (v6.3.4#6332)