thanks but i should run it with -0.8.2 however that i updgraded it before and got that it couldn't bind port i need to try that
It was indeed a port conflict, but not with another ZMQ process. It turns out our ephemeral port range was messed up on the machines: $ cat /proc/sys/net/ipv4/ip_local_port_range 1024 65000 but the defuakt i have is 32768 61000 but don't know what are the right numbers ? On Thu, Dec 17, 2015 at 12:03 PM, 刘键(Basti Liu) <[email protected]> wrote: > Hi Sam, > > Not sure what the reason is why ZMQ cause this error. > If you are using a new version Storm(e.g 0.9.x), could you try to use > Netty for message transport, since ZMQ is no longer used in Storm now? > Please make sure following configuration is correct. > - storm.messaging.transport: "backtype.storm.messaging.netty.Context" > > Regards > Basti > > -----Original Message----- > From: sam mohel [mailto:[email protected]] > Sent: Thursday, December 17, 2015 5:08 PM > To: [email protected] > Subject: Re: Problem with storm since 4 months > > i edited to make topology.debug true and got in the supervisor log file > still hasn't start and in the worker log file > > 2015-12-17 07:52:23 task [INFO] Emitting: b-7 __system ["startup"] > 2015-12-17 07:52:23 executor [INFO] Loaded executor tasks b-7:[33 33] > 2015-12-17 07:52:23 executor [INFO] Preparing bolt b-7:(33) > 2015-12-17 07:52:23 executor [INFO] Finished loading executor b-7:[33 33] > 2015-12-17 07:52:23 worker [INFO] Launching receive-thread for > 5587bcc1-05d4-4d92-ae3d-2a8503cef259:6706 > 2015-12-17 07:52:23 executor [INFO] Prepared bolt b-7:(33) > > after finished loading got alot of this lines > > 2015-12-17 07:52:27 executor [INFO] Processing received message > source: __system:-1, stream: __tick, id: {}, [5] > 2015-12-17 07:52:27 executor [INFO] Processing received message > source: __system:-1, stream: __tick, id: {}, [5] > > Got in the storm ui zeros in emitted and transfered > > i executed the command that launch worker and supervisor got > > 2015-12-17 07:59:04 executor [INFO] Prepared bolt b-7:(33) > 2015-12-17 07:59:04 util [ERROR] Async loop died! > org.zeromq.ZMQException: Address already in use(0x62) at > org.zeromq.ZMQ$Socket.bind(Native Method) at zilch.mq$bind.invoke(mq.clj:69) > at backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj:57) > at > backtype.storm.messaging.loader$launch_receive_thread_BANG_$fn__1629.invoke(loader.clj:26) > at backtype.storm.util$async_loop$fn__465.invoke(util.clj:375) > at clojure.lang.AFn.run(AFn.java:24) > at java.lang.Thread.run(Thread.java:701) > 2015-12-17 07:59:04 util [INFO] Halting process: > > > i read that " It was indeed a port conflict, but not with another ZMQ > process. It turns out our ephemeral port range was messed up on the > machines: > > > $ cat /proc/sys/net/ipv4/ip_local_port_range 1024 65000" i'm on ubuntu > 14.04 i tried it to put 6706 in reserved port but problem still > > > On Thu, Dec 17, 2015 at 9:21 AM, 刘键(Basti Liu) <[email protected]> > wrote: > > > Hi Sam, > > > > If the worker(pid=2621) belongs to the topology you just submitted, it > > means the worker has bound the port "6703" successfully. > > So there should not be any "binding error". Is there any other problems? > > > > Regards > > Basti > > > > -----Original Message----- > > From: sam mohel [mailto:[email protected]] > > Sent: Thursday, December 17, 2015 2:11 PM > > To: [email protected] > > Subject: Re: Problem with storm since 4 months > > > > can i find help ? > > > > On Fri, Dec 11, 2015 at 6:32 AM, sam mohel <[email protected]> wrote: > > > > > this topology that has problem , i mean this i'm now submitted it > > > > > > On Fri, Dec 11, 2015 at 5:45 AM, 刘键(Basti Liu) > > > <[email protected]> > > > wrote: > > > > > >> This worker (pid=2621) belongs to topology " fsd-1-1449794574". > > >> Please check if this topology has already been killed. If so, just > > >> kill this process. > > >> > > >> Regards > > >> Basti > > >> -----Original Message----- > > >> From: sam mohel [mailto:[email protected]] > > >> Sent: Friday, December 11, 2015 11:18 AM > > >> To: [email protected] > > >> Subject: Re: Problem with storm since 4 months > > >> > > >> is that right command ps aux |grep 2621 > > >> user 2621 7.8 2.7 3444276 108056 pts/12 Sl+ 02:42 12:01 java > > >> -server -Djava.net.preferIPv4Stack=true > > >> -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib > > >> -Dlogfile.name=worker-6703.log -Dstorm.home=/home/user/storm-0.8.2 > > >> -Dlog4j.configuration=storm.log.properties -cp > > >> /home/user/storm-0.8.2/storm-0.8.2.jar:/home/user/storm-0.8.2/lib/c > > >> om > > >> mons-exec-1.1.jar:/home/user/storm-0.8.2/lib/carbonite-1.5.0.jar:/h > > >> om > > >> e/user/storm-0.8.2/lib/ring-jetty-adapter-0.3.11.jar:/home/user/sto > > >> rm > > >> -0.8.2/lib/minlog-1.2.jar:/home/user/storm-0.8.2/lib/hiccup-0.3.6.j > > >> ar > > >> :/home/user/storm-0.8.2/lib/commons-lang-2.5.jar:/home/user/storm-0 > > >> .8 > > >> .2/lib/math.numeric-tower-0.0.1.jar:/home/user/storm-0.8.2/lib/serv > > >> le > > >> t-api-2.5-20081211.jar:/home/user/storm-0.8.2/lib/slf4j-log4j12-1.5 > > >> .8 > > >> .jar:/home/user/storm-0.8.2/lib/commons-logging-1.1.1.jar:/home/use > > >> r/ > > >> storm-0.8.2/lib/tools.logging-0.2.3.jar:/home/user/storm-0.8.2/lib/ > > >> lo > > >> g4j-1.2.16.jar:/home/user/storm-0.8.2/lib/clout-1.0.1.jar:/home/use > > >> r/ > > >> storm-0.8.2/lib/httpcore-4.1.jar:/home/user/storm-0.8.2/lib/servlet > > >> -a > > >> pi-2.5.jar:/home/user/storm-0.8.2/lib/objenesis-1.2.jar:/home/user/ > > >> st > > >> orm-0.8.2/lib/clojure-1.4.0.jar:/home/user/storm-0.8.2/lib/json-sim > > >> pl > > >> e-1.1.jar:/home/user/storm-0.8.2/lib/First-Story-Detection-1.0-SNAP > > >> SH > > >> OT.jar:/home/user/storm-0.8.2/lib/httpclient-4.1.1.jar:/home/user/s > > >> to > > >> rm-0.8.2/lib/jzmq-2.1.0.jar:/home/user/storm-0.8.2/lib/core.incubat > > >> or > > >> -0.1.0.jar:/home/user/storm-0.8.2/lib/junit-3.8.1.jar:/home/user/st > > >> or > > >> m-0.8.2/lib/guava-13.0.jar:/home/user/storm-0.8.2/lib/jgrapht-0.8.3 > > >> .j > > >> ar:/home/user/storm-0.8.2/lib/commons-io-1.4.jar:/home/user/storm-0 > > >> .8 > > >> .2/lib/curator-framework-1.0.1.jar:/home/user/storm-0.8.2/lib/ring- > > >> se > > >> rvlet-0.3.11.jar:/home/user/storm-0.8.2/lib/tools.macro-0.1.0.jar:/ > > >> ho > > >> me/user/storm-0.8.2/lib/asm-4.0.jar:/home/user/storm-0.8.2/lib/jett > > >> y- > > >> 6.1.26.jar:/home/user/storm-0.8.2/lib/reflectasm-1.07-shaded.jar:/h > > >> om > > >> e/user/storm-0.8.2/lib/joda-time-2.0.jar:/home/user/storm-0.8.2/lib > > >> /j > > >> line-0.9.94.jar:/home/user/storm-0.8.2/lib/libthrift7-0.7.0.jar:/ho > > >> me > > >> /user/storm-0.8.2/lib/tools.cli-0.2.2.jar:/home/user/storm-0.8.2/li > > >> b/ > > >> commons-fileupload-1.2.1.jar:/home/user/storm-0.8.2/lib/zookeeper-3 > > >> .3 > > >> .3.jar:/home/user/storm-0.8.2/lib/First-Story-Detection-1.0-SNAPSHO > > >> T- > > >> jar-with-dependencies.jar:/home/user/storm-0.8.2/lib/slf4j-api-1.5.8. > > >> jar:/home/user/storm-0.8.2/lib/kryo-2.17.jar:/home/user/storm-0.8.2 > > >> /l > > >> ib/compojure-1.1.3.jar:/home/user/storm-0.8.2/lib/curator-client-1.0. > > >> 1.jar:/home/user/storm-0.8.2/lib/jetty-util-6.1.26.jar:/home/user/s > > >> to > > >> rm-0.8.2/lib/ring-core-1.1.5.jar:/home/user/storm-0.8.2/lib/snakeya > > >> ml > > >> -1.9.jar:/home/user/storm-0.8.2/lib/disruptor-2.10.1.jar:/home/user > > >> /s > > >> torm-0.8.2/lib/commons-codec-1.4.jar:/home/user/storm-0.8.2/lib/clj > > >> -t > > >> ime-0.4.1.jar:/home/user/storm-0.8.2/log4j:/home/user/storm-0.8.2/c > > >> on > > >> f:/home/user/storms/storm/supervisor/stormdist/fsd-1-1449794574/sto > > >> rm jar.jar backtype.storm.daemon.worker fsd-1-1449794574 > > >> 1953edf9-8f3e-4bd8-870a-05370272b097 6703 > > >> ff8603ef-5fb9-47c7-90d0-01813ace4188 > > >> user 3573 0.0 0.0 16016 2256 pts/28 S+ 05:16 0:00 grep > > >> --color=auto 2621 > > >> > > >> > > >> On Fri, Dec 11, 2015 at 5:16 AM, sam mohel <[email protected]> > wrote: > > >> > > >> > please , how can i check it ? > > >> > > > >> > On Fri, Dec 11, 2015 at 5:12 AM, 刘键(Basti Liu) > > >> > <[email protected]> > > >> > wrote: > > >> > > > >> >> Okay. Please check which application this process belongs to, > > >> >> and whether is it OK to kill it. > > >> >> > > >> >> Regards > > >> >> Basti > > >> >> -----Original Message----- > > >> >> From: sam mohel [mailto:[email protected]] > > >> >> Sent: Friday, December 11, 2015 11:06 AM > > >> >> To: [email protected] > > >> >> Subject: Re: Problem with storm since 4 months > > >> >> > > >> >> i used > > >> >> > > >> >> netstat -ntape > > >> >> > > >> >> and got > > >> >> Code: > > >> >> > > >> >> tcp 0 0 0.0.0.0:6703 0.0.0.0:* > > >> >> LISTEN 1000 22776 2621/java > > >> >> > > >> >> then used this command with PID of PORT = 2621 > > >> >> Code: > > >> >> > > >> >> lsof -P -w -n -p 2621 -a -i > > >> >> > > >> >> and got > > >> >> Code: > > >> >> > > >> >> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME > > >> >> java 2621 user 76u IPv4 18906 0t0 TCP > > >> >> 127.0.0.1:50036->127.0.0.1:2181 (ESTABLISHED) > > >> >> java 2621 user 83u IPv4 22773 0t0 TCP > > >> >> 127.0.0.1:56733->127.0.0.1:3773 (ESTABLISHED) > > >> >> java 2621 user 86u IPv4 22776 0t0 TCP *:6703 (LISTEN) > > >> >> > > >> >> > > >> >> On Fri, Dec 11, 2015 at 4:44 AM, 刘键(Basti Liu) > > >> >> <[email protected]> > > >> >> wrote: > > >> >> > > >> >> > Hi Sam, > > >> >> > > > >> >> > Have you tried "netstat" and "lsof" commands recommended by Erik? > > >> >> > What is the result? > > >> >> > We need to find which process bound this port, then kill the > > >> >> > process to release this port. > > >> >> > > > >> >> > Regards > > >> >> > Basti > > >> >> > > > >> >> > -----Original Message----- > > >> >> > From: sam mohel [mailto:[email protected]] > > >> >> > Sent: Friday, December 11, 2015 8:15 AM > > >> >> > To: [email protected] > > >> >> > Subject: Re: Problem with storm since 4 months > > >> >> > > > >> >> > > now i converted the change > > >> >> > > > >> >> > but problem still > > >> >> > > > >> >> > > i mean in the link you sent , he said "the ephermal port > > >> >> > > range is > > >> >> > specified in /proc/sys/net/ipv4/ip_local_ port_range. You can > > >> >> > probably extend it to run from 16k to 64k." > > >> >> > > > >> >> > how can i do that ? > > >> >> > > > >> >> > > how can i check if there is anothe dump worker working , > > >> >> > or > > >> >> > is there another way to kill any worker if it is dump or not > > >> >> "completely" > > >> >> > and start from scratch > > >> >> > > > >> >> > thanks for your patience and your time > > >> >> > > > >> >> > > > >> >> > On Thu, Dec 10, 2015 at 7:38 PM, Erik Weathers < > > >> >> > [email protected]> wrote: > > >> >> > > > >> >> > > responses inline. > > >> >> > > > > >> >> > > >Ah, so, you shouldn't have touched the "local port range" > > >> >> > > >setting since > > >> >> > > you > > >> >> > > don't fully understand ephemeral ports and TCP yet! :-) > > >> >> > > > > >> >> > > I'll give a brief synopsis: > > >> >> > > > > >> >> > > Say you're making a connection from a client to a server > > >> >> > > that listens on port 80. The client needs to have a port of > > >> >> > > its own to receive the response packets from the server. The > > >> >> > > port that is allocated for it by the TCP stack is a "random" > > >> >> > > port, which is called an "ephemeral" port in > > >> >> > this context. > > >> >> > > So with your original default config (32768 61000) the port > > >> would > > >> >> be > > >> >> > > chosen from an available port on the machine that lies > > >> >> > > within that range of > > >> >> > > 32768->61000. That's good, it wouldn't conflict with the > > >> >> > > 32768->default TCP ports > > >> >> > > being listened to by the Storm Worker processes (67xx). So 1. > > >> >> > > isn't your problem. But now you've made it possible for it > > >> >> > > to become a problem, since now the range of ports that can > > >> >> > > be given as an ephemeral port overlaps with the default > > >> >> > > Storm Worker ports > > >> >> > > (1024->65535 includes 67xx). So you should revert that > > >> >> > > config > > >> change. > > >> >> > > > > >> >> > > Thanks for your replying , now i reverted the change > > >> >> > > > >> >> > > > > >> >> > > >Sorry, I don't understand what you are asking. What thing > > >> >> > > >is > > "16" > > >> >> > > >that > > >> >> > > you > > >> >> > > are trying to extend? > > >> >> > > > > >> >> > > i mean in the link you sent , he adviced to extend from 16 > > >> >> > > to > > >> >> > > 64 > > >> >> > > >You can kill topologies from the Nimbus UI (web page). Or > > >> >> > > >with the > > >> >> > > bin/storm command. > > >> >> > > > > >> >> > > > > >> >> > > >The commands are *solely* intended to figure out what is > > >> conflicting. > > >> >> > > Your > > >> >> > > logs claim that there is something holding onto 67xx which > > >> >> > > prevents > > >> >> the > > >> >> > > Storm Worker from launching. So if that is happening you > should > > >> >> > > *immediately* try to figure out what is actually holding > > >> >> > > onto the port and preventing your Storm Worker from launching. > > >> >> > > > > >> >> > > > > >> >> > > >Not sure what you mean here. > > >> >> > > > > >> >> > > Please note that Storm often suffers from "cascading" > > >> >> > > failures, where there are a lot of exceptions and errors > > >> >> > > that aren't actually the root cause of the problem. Often > > >> >> > > you need to spend time and effort looking at lots of logs > > >> >> > > and tracing back to the real root > > >> >> cause. > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > > >> >> > > > Really , Thanks for your time > > >> >> > > > > > >> >> > > > On Thu, Dec 10, 2015 at 6:22 AM, Erik Weathers < > > >> >> > > > [email protected]> wrote: > > >> >> > > > > > >> >> > > > > Regarding Basti's suggestion (1.) that your host's > > >> >> > > > > configured ephemeral ports might be conflicting with the > > >> >> > > > > storm worker ports, here's how you > > >> >> > > > can > > >> >> > > > > check your "local port range" setting: > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > https://serverfault.com/questions/261663/on-linux-how-can-i- > > >> >> > > te > > >> >> > > ll- how -m any-ephemeral-ports-are-left-available > > >> >> > > > > > > >> >> > > > > % cat /proc/sys/net/ipv4/ip_local_port_range > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > It's possible that there is a zombie worker process > > >> >> > > > > holding onto port > > >> >> > > > 6703. > > >> >> > > > > I would try to identify the process like so: > > >> >> > > > > > > >> >> > > > > % sudo netstat -ap --numeric-ports --extend | grep -w > > >> >> > > > > LISTEN > > >> >> > > > > | grep -w > > >> >> > > > 6703 > > >> >> > > > > > > >> >> > > > > Alternatively you can try a global lsof search: > > >> >> > > > > > > >> >> > > > > % sudo lsof | grep TCP | grep -w LISTEN | grep -w 6703 > > >> >> > > > > > > >> >> > > > > - Erik > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > On Wed, Dec 9, 2015 at 7:37 PM, 刘键(Basti Liu) < > > >> >> > > [email protected]> > > >> >> > > > > wrote: > > >> >> > > > > > > >> >> > > > > > Hi Sam, > > >> >> > > > > > > > >> >> > > > > > You can try to find which process has bound this port > > >> >> > > > > > by "netstat > > >> >> > > -anp" > > >> >> > > > > > first. > > >> >> > > > > > > > >> >> > > > > > Generally, there are following cases for the binding > error. > > >> >> > > > > > 1. "local port range" is not set to exclude the port > > >> >> > > > > > range used in > > >> >> > > > Storm. > > >> >> > > > > > 2. The previous worker was not killed correctly. > > >> >> > > > > > 3. There is bug of assignment in some scenarios. Same > > >> >> > > > > > port was > > >> >> > > assigned > > >> >> > > > > to > > >> >> > > > > > two workers. > > >> >> > > > > > > > >> >> > > > > > Regards > > >> >> > > > > > Basti > > >> >> > > > > > > > >> >> > > > > > -----Original Message----- > > >> >> > > > > > From: sam mohel [mailto:[email protected]] > > >> >> > > > > > Sent: Thursday, December 10, 2015 7:16 AM > > >> >> > > > > > To: [email protected] > > >> >> > > > > > Subject: Re: Problem with storm since 4 months > > >> >> > > > > > > > >> >> > > > > > i tried to use storm-0.9.5 but problem changed with > > >> >> > > > > > > > >> >> > > > > > cannot bind port 6703 i think it's same problem > > >> >> > > > > > > > >> >> > > > > > On Wed, Dec 9, 2015 at 8:42 PM, Harsha > > >> >> > > > > > <[email protected]> > > >> wrote: > > >> >> > > > > > > > >> >> > > > > > > Sam, > > >> >> > > > > > > you might be using very old version of > > >> >> > > > > > > storm since its > > >> >> > > > > showing > > >> >> > > > > > > ZeroMQ. Can you try using newer version > > >> >> > > > > > > storm without > > >> >> > > zero > > >> >> > > > > mq. > > >> >> > > > > > > -Harsha > > >> >> > > > > > > > > >> >> > > > > > > On Wed, Dec 9, 2015, at 10:19 AM, sam mohel wrote: > > >> >> > > > > > > > I have this problem since 4months when I submitted > > >> >> > > > > > > > topology I got this in the worker log file [ERROR] > > >> >> > > > > > > > Async > > >> >> loop died! > > >> >> > > > > > org.zeromq.ZMQException: > > >> >> > > > > > > > Address already in use(0x62) at > > >> >> > > > > > > > org.zeromq.ZMQ$Socket.bind(Native Method) at > > >> >> > > > > > > > zilch.mq$bind.invoke(mq.clj:69) at > > >> >> > > > > > > > backtype.storm.messaging.zmq.ZMQContext.bind(zmq.c > > >> >> > > > > > > > lj > > >> >> > > > > > > > :57 > > >> >> > > > > > > > )at > > >> >> > > > > > > > > > >> >> > > > > > > > > >> >> > > > backtype.storm.messaging.loader$launch_receive_thread_BANG > > >> >> > > > _$ > > >> >> > > > fn_ > > >> >> > > > _16 > > >> >> > > > 29 > > >> >> > > > .i > > >> >> > > > > > > nvoke(loader.clj:26) > > >> >> > > > > > > > at > > >> >> > > > > > > > > backtype.storm.util$async_loop$fn__465.invoke(util.clj: > > >> >> > > > > > > > 375 > > >> >> > > > > > > > ) at clojure.lang.AFn.run(AFn.java:24) at > > >> >> > > > java.lang.Thread.run(Unknown > > >> >> > > > > > > > Source) > > >> >> > > > > > > > > > >> >> > > > > > > > when i tried to connect port 6703 and 6702 > > >> >> > > > > > > > > > >> >> > > > > > > > And supervisor log file hadn't still start > > >> >> > > > > > > > > > >> >> > > > > > > > > > >> >> > > > > > > > I searched everywhere but cannot find any solution > > >> >> > > > > > > > I hope you can > > >> >> > > > > > > > > >> >> > > > > > > > >> >> > > > > > > > >> >> > > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > > >> >> > > >> >> > > >> > > > >> > > >> > > > > > > > > >
