Hi, all I'm running a topology on storm cluster of 0.9.0.1 with netty as transport layer, this error occurs : Netty client failed to create a selector due to* too many open files exception*, the worker continuously halting with initialization error.
I checked the ulimit -n(> 130000) which is much bigger than currently opened fds (sudo lsof | grep java | wc -l) which is about 6000 at most. By the way,this topology works fine with storm cluster of 0.8.0. What's the problem? here is the stack trace: ------------------------------------------------------------- 2014-03-04 20:24:14 b.s.m.TransportFactory [INFO] Storm peer transport plugin:backtype.storm.messaging.netty.Context 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1] 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1] 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1] 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1] 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1] 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1] 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1] 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [2] 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1] 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1] 2014-03-04 20:24:14 b.s.m.n.Client [INFO] Reconnect ... [1] 2014-03-04 20:24:14 b.s.d.worker [ERROR] Error on initialization of server mk-worker org.jboss.netty.channel.ChannelException: Failed to create a selector. at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:337) ~[netty-3.6.3.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.(AbstractNioSelector.java:95) ~[netty-3.6.3.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorker.(AbstractNioWorker.java:51) ~[netty-3.6.3.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorker.(NioWorker.java:45) ~[netty-3.6.3.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:45) ~[netty-3.6.3.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorkerPool.createWorker(NioWorkerPool.java:28) ~[netty-3.6.3.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.newWorker(AbstractNioWorkerPool.java:99) ~[netty-3.6.3.Final.jar:na] at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.init(AbstractNioWorkerPool.java:69) ~[netty-3.6.3.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorkerPool.(NioWorkerPool.java:39) ~[netty-3.6.3.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioWorkerPool.(NioWorkerPool.java:33) ~[netty-3.6.3.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.(NioClientSocketChannelFactory.java:152) ~[netty-3.6.3.Final.jar:na] at org.jboss.netty.channel.socket.nio.NioClientSocketChannelFactory.(NioClientSocketChannelFactory.java:134) ~[netty-3.6.3.Final.jar:na] at backtype.storm.messaging.netty.Client.(Client.java:54) ~[storm-netty-0.9.0.1.jar:na] at backtype.storm.messaging.netty.Context.connect(Context.java:36) ~[storm-netty-0.9.0.1.jar:na] at backtype.storm.daemon.worker$mk_refresh_connections$this__5827$iter__5834__5838$fn__5839.invoke(worker.clj:250) ~[storm-core-0.9.0.1.jar:na] at clojure.lang.LazySeq.sval(LazySeq.java:42) ~[clojure-1.4.0.jar:na] at clojure.lang.LazySeq.seq(LazySeq.java:60) ~[clojure-1.4.0.jar:na] at clojure.lang.Cons.next(Cons.java:39) ~[clojure-1.4.0.jar:na] at clojure.lang.RT.next(RT.java:587) ~[clojure-1.4.0.jar:na] at clojure.core$next.invoke(core.clj:64) ~[clojure-1.4.0.jar:na] at clojure.core$dorun.invoke(core.clj:2726) ~[clojure-1.4.0.jar:na] at clojure.core$doall.invoke(core.clj:2741) ~[clojure-1.4.0.jar:na] at backtype.storm.daemon.worker$mk_refresh_connections$this__5827.invoke(worker.clj:244) ~[storm-core-0.9.0.1.jar:na] at backtype.storm.daemon.worker$fn__5882$exec_fn__1229__auto____5883.invoke(worker.clj:357) ~[storm-core-0.9.0.1.jar:na] at clojure.lang.AFn.applyToHelper(AFn.java:185) [clojure-1.4.0.jar:na] at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na] at clojure.core$apply.invoke(core.clj:601) ~[clojure-1.4.0.jar:na] at backtype.storm.daemon.worker$fn__5882$mk_worker__5938.doInvoke(worker.clj:329) [storm-core-0.9.0.1.jar:na] at clojure.lang.RestFn.invoke(RestFn.java:512) [clojure-1.4.0.jar:na] at backtype.storm.daemon.worker$_main.invoke(worker.clj:439) [storm-core-0.9.0.1.jar:na] at clojure.lang.AFn.applyToHelper(AFn.java:172) [clojure-1.4.0.jar:na] at clojure.lang.AFn.applyTo(AFn.java:151) [clojure-1.4.0.jar:na] at backtype.storm.daemon.worker.main(Unknown Source) [storm-core-0.9.0.1.jar:na] * Caused by: java.io.IOException: Too many open files* at sun.nio.ch.IOUtil.initPipe(Native Method) ~[na:1.6.0_38] at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:49) ~[na:1.6.0_38] at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18) ~[na:1.6.0_38] at java.nio.channels.Selector.open(Selector.java:209) ~[na:1.6.0_38] at org.jboss.netty.channel.socket.nio.AbstractNioSelector.openSelector(AbstractNioSelector.java:335) ~[netty-3.6.3.Final.jar:na] ... 32 common frames omitted 2014-03-04 20:24:14 b.s.util [INFO] Halting process: ("Error on initialization") -------------------------------------------------------------------------------------------------------------------- Thanks -- ====================================================== Gvain Email: jh.li...@gmail.com