Hi Sam, If the worker(pid=2621) belongs to the topology you just submitted, it means the worker has bound the port "6703" successfully. So there should not be any "binding error". Is there any other problems?
Regards Basti -----Original Message----- From: sam mohel [mailto:[email protected]] Sent: Thursday, December 17, 2015 2:11 PM To: [email protected] Subject: Re: Problem with storm since 4 months can i find help ? On Fri, Dec 11, 2015 at 6:32 AM, sam mohel <[email protected]> wrote: > this topology that has problem , i mean this i'm now submitted it > > On Fri, Dec 11, 2015 at 5:45 AM, 刘键(Basti Liu) > <[email protected]> > wrote: > >> This worker (pid=2621) belongs to topology " fsd-1-1449794574". >> Please check if this topology has already been killed. If so, just >> kill this process. >> >> Regards >> Basti >> -----Original Message----- >> From: sam mohel [mailto:[email protected]] >> Sent: Friday, December 11, 2015 11:18 AM >> To: [email protected] >> Subject: Re: Problem with storm since 4 months >> >> is that right command ps aux |grep 2621 >> user 2621 7.8 2.7 3444276 108056 pts/12 Sl+ 02:42 12:01 java >> -server -Djava.net.preferIPv4Stack=true >> -Djava.library.path=/usr/local/lib:/opt/local/lib:/usr/lib >> -Dlogfile.name=worker-6703.log -Dstorm.home=/home/user/storm-0.8.2 >> -Dlog4j.configuration=storm.log.properties -cp >> /home/user/storm-0.8.2/storm-0.8.2.jar:/home/user/storm-0.8.2/lib/com >> mons-exec-1.1.jar:/home/user/storm-0.8.2/lib/carbonite-1.5.0.jar:/hom >> e/user/storm-0.8.2/lib/ring-jetty-adapter-0.3.11.jar:/home/user/storm >> -0.8.2/lib/minlog-1.2.jar:/home/user/storm-0.8.2/lib/hiccup-0.3.6.jar >> :/home/user/storm-0.8.2/lib/commons-lang-2.5.jar:/home/user/storm-0.8 >> .2/lib/math.numeric-tower-0.0.1.jar:/home/user/storm-0.8.2/lib/servle >> t-api-2.5-20081211.jar:/home/user/storm-0.8.2/lib/slf4j-log4j12-1.5.8 >> .jar:/home/user/storm-0.8.2/lib/commons-logging-1.1.1.jar:/home/user/ >> storm-0.8.2/lib/tools.logging-0.2.3.jar:/home/user/storm-0.8.2/lib/lo >> g4j-1.2.16.jar:/home/user/storm-0.8.2/lib/clout-1.0.1.jar:/home/user/ >> storm-0.8.2/lib/httpcore-4.1.jar:/home/user/storm-0.8.2/lib/servlet-a >> pi-2.5.jar:/home/user/storm-0.8.2/lib/objenesis-1.2.jar:/home/user/st >> orm-0.8.2/lib/clojure-1.4.0.jar:/home/user/storm-0.8.2/lib/json-simpl >> e-1.1.jar:/home/user/storm-0.8.2/lib/First-Story-Detection-1.0-SNAPSH >> OT.jar:/home/user/storm-0.8.2/lib/httpclient-4.1.1.jar:/home/user/sto >> rm-0.8.2/lib/jzmq-2.1.0.jar:/home/user/storm-0.8.2/lib/core.incubator >> -0.1.0.jar:/home/user/storm-0.8.2/lib/junit-3.8.1.jar:/home/user/stor >> m-0.8.2/lib/guava-13.0.jar:/home/user/storm-0.8.2/lib/jgrapht-0.8.3.j >> ar:/home/user/storm-0.8.2/lib/commons-io-1.4.jar:/home/user/storm-0.8 >> .2/lib/curator-framework-1.0.1.jar:/home/user/storm-0.8.2/lib/ring-se >> rvlet-0.3.11.jar:/home/user/storm-0.8.2/lib/tools.macro-0.1.0.jar:/ho >> me/user/storm-0.8.2/lib/asm-4.0.jar:/home/user/storm-0.8.2/lib/jetty- >> 6.1.26.jar:/home/user/storm-0.8.2/lib/reflectasm-1.07-shaded.jar:/hom >> e/user/storm-0.8.2/lib/joda-time-2.0.jar:/home/user/storm-0.8.2/lib/j >> line-0.9.94.jar:/home/user/storm-0.8.2/lib/libthrift7-0.7.0.jar:/home >> /user/storm-0.8.2/lib/tools.cli-0.2.2.jar:/home/user/storm-0.8.2/lib/ >> commons-fileupload-1.2.1.jar:/home/user/storm-0.8.2/lib/zookeeper-3.3 >> .3.jar:/home/user/storm-0.8.2/lib/First-Story-Detection-1.0-SNAPSHOT- >> jar-with-dependencies.jar:/home/user/storm-0.8.2/lib/slf4j-api-1.5.8. >> jar:/home/user/storm-0.8.2/lib/kryo-2.17.jar:/home/user/storm-0.8.2/l >> ib/compojure-1.1.3.jar:/home/user/storm-0.8.2/lib/curator-client-1.0. >> 1.jar:/home/user/storm-0.8.2/lib/jetty-util-6.1.26.jar:/home/user/sto >> rm-0.8.2/lib/ring-core-1.1.5.jar:/home/user/storm-0.8.2/lib/snakeyaml >> -1.9.jar:/home/user/storm-0.8.2/lib/disruptor-2.10.1.jar:/home/user/s >> torm-0.8.2/lib/commons-codec-1.4.jar:/home/user/storm-0.8.2/lib/clj-t >> ime-0.4.1.jar:/home/user/storm-0.8.2/log4j:/home/user/storm-0.8.2/con >> f:/home/user/storms/storm/supervisor/stormdist/fsd-1-1449794574/storm >> jar.jar backtype.storm.daemon.worker fsd-1-1449794574 >> 1953edf9-8f3e-4bd8-870a-05370272b097 6703 >> ff8603ef-5fb9-47c7-90d0-01813ace4188 >> user 3573 0.0 0.0 16016 2256 pts/28 S+ 05:16 0:00 grep >> --color=auto 2621 >> >> >> On Fri, Dec 11, 2015 at 5:16 AM, sam mohel <[email protected]> wrote: >> >> > please , how can i check it ? >> > >> > On Fri, Dec 11, 2015 at 5:12 AM, 刘键(Basti Liu) >> > <[email protected]> >> > wrote: >> > >> >> Okay. Please check which application this process belongs to, and >> >> whether is it OK to kill it. >> >> >> >> Regards >> >> Basti >> >> -----Original Message----- >> >> From: sam mohel [mailto:[email protected]] >> >> Sent: Friday, December 11, 2015 11:06 AM >> >> To: [email protected] >> >> Subject: Re: Problem with storm since 4 months >> >> >> >> i used >> >> >> >> netstat -ntape >> >> >> >> and got >> >> Code: >> >> >> >> tcp 0 0 0.0.0.0:6703 0.0.0.0:* >> >> LISTEN 1000 22776 2621/java >> >> >> >> then used this command with PID of PORT = 2621 >> >> Code: >> >> >> >> lsof -P -w -n -p 2621 -a -i >> >> >> >> and got >> >> Code: >> >> >> >> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME >> >> java 2621 user 76u IPv4 18906 0t0 TCP >> >> 127.0.0.1:50036->127.0.0.1:2181 (ESTABLISHED) >> >> java 2621 user 83u IPv4 22773 0t0 TCP >> >> 127.0.0.1:56733->127.0.0.1:3773 (ESTABLISHED) >> >> java 2621 user 86u IPv4 22776 0t0 TCP *:6703 (LISTEN) >> >> >> >> >> >> On Fri, Dec 11, 2015 at 4:44 AM, 刘键(Basti Liu) >> >> <[email protected]> >> >> wrote: >> >> >> >> > Hi Sam, >> >> > >> >> > Have you tried "netstat" and "lsof" commands recommended by Erik? >> >> > What is the result? >> >> > We need to find which process bound this port, then kill the >> >> > process to release this port. >> >> > >> >> > Regards >> >> > Basti >> >> > >> >> > -----Original Message----- >> >> > From: sam mohel [mailto:[email protected]] >> >> > Sent: Friday, December 11, 2015 8:15 AM >> >> > To: [email protected] >> >> > Subject: Re: Problem with storm since 4 months >> >> > >> >> > > now i converted the change >> >> > >> >> > but problem still >> >> > >> >> > > i mean in the link you sent , he said "the ephermal port range >> >> > > is >> >> > specified in /proc/sys/net/ipv4/ip_local_ port_range. You can >> >> > probably extend it to run from 16k to 64k." >> >> > >> >> > how can i do that ? >> >> > >> >> > > how can i check if there is anothe dump worker working , >> >> > or >> >> > is there another way to kill any worker if it is dump or not >> >> "completely" >> >> > and start from scratch >> >> > >> >> > thanks for your patience and your time >> >> > >> >> > >> >> > On Thu, Dec 10, 2015 at 7:38 PM, Erik Weathers < >> >> > [email protected]> wrote: >> >> > >> >> > > responses inline. >> >> > > >> >> > > >Ah, so, you shouldn't have touched the "local port range" >> >> > > >setting since >> >> > > you >> >> > > don't fully understand ephemeral ports and TCP yet! :-) >> >> > > >> >> > > I'll give a brief synopsis: >> >> > > >> >> > > Say you're making a connection from a client to a server that >> >> > > listens on port 80. The client needs to have a port of its own >> >> > > to receive the response packets from the server. The port that >> >> > > is allocated for it by the TCP stack is a "random" port, which >> >> > > is called an "ephemeral" port in >> >> > this context. >> >> > > So with your original default config (32768 61000) the port >> would >> >> be >> >> > > chosen from an available port on the machine that lies within >> >> > > that range of >> >> > > 32768->61000. That's good, it wouldn't conflict with the >> >> > > 32768->default TCP ports >> >> > > being listened to by the Storm Worker processes (67xx). So 1. >> >> > > isn't your problem. But now you've made it possible for it to >> >> > > become a problem, since now the range of ports that can be >> >> > > given as an ephemeral port overlaps with the default Storm >> >> > > Worker ports >> >> > > (1024->65535 includes 67xx). So you should revert that config >> change. >> >> > > >> >> > > Thanks for your replying , now i reverted the change >> >> > >> >> > > >> >> > > >Sorry, I don't understand what you are asking. What thing is "16" >> >> > > >that >> >> > > you >> >> > > are trying to extend? >> >> > > >> >> > > i mean in the link you sent , he adviced to extend from 16 to >> >> > > 64 >> >> > > >You can kill topologies from the Nimbus UI (web page). Or >> >> > > >with the >> >> > > bin/storm command. >> >> > > >> >> > > >> >> > > >The commands are *solely* intended to figure out what is >> conflicting. >> >> > > Your >> >> > > logs claim that there is something holding onto 67xx which >> >> > > prevents >> >> the >> >> > > Storm Worker from launching. So if that is happening you should >> >> > > *immediately* try to figure out what is actually holding onto >> >> > > the port and preventing your Storm Worker from launching. >> >> > > >> >> > > >> >> > > >Not sure what you mean here. >> >> > > >> >> > > Please note that Storm often suffers from "cascading" >> >> > > failures, where there are a lot of exceptions and errors that >> >> > > aren't actually the root cause of the problem. Often you need >> >> > > to spend time and effort looking at lots of logs and tracing >> >> > > back to the real root >> >> cause. >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > > >> >> > > > Really , Thanks for your time >> >> > > > >> >> > > > On Thu, Dec 10, 2015 at 6:22 AM, Erik Weathers < >> >> > > > [email protected]> wrote: >> >> > > > >> >> > > > > Regarding Basti's suggestion (1.) that your host's >> >> > > > > configured ephemeral ports might be conflicting with the >> >> > > > > storm worker ports, here's how you >> >> > > > can >> >> > > > > check your "local port range" setting: >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > >> >> > > https://serverfault.com/questions/261663/on-linux-how-can-i-te >> >> > > ll- how -m any-ephemeral-ports-are-left-available >> >> > > > > >> >> > > > > % cat /proc/sys/net/ipv4/ip_local_port_range >> >> > > > > >> >> > > > > >> >> > > > > It's possible that there is a zombie worker process >> >> > > > > holding onto port >> >> > > > 6703. >> >> > > > > I would try to identify the process like so: >> >> > > > > >> >> > > > > % sudo netstat -ap --numeric-ports --extend | grep -w >> >> > > > > LISTEN >> >> > > > > | grep -w >> >> > > > 6703 >> >> > > > > >> >> > > > > Alternatively you can try a global lsof search: >> >> > > > > >> >> > > > > % sudo lsof | grep TCP | grep -w LISTEN | grep -w 6703 >> >> > > > > >> >> > > > > - Erik >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > On Wed, Dec 9, 2015 at 7:37 PM, 刘键(Basti Liu) < >> >> > > [email protected]> >> >> > > > > wrote: >> >> > > > > >> >> > > > > > Hi Sam, >> >> > > > > > >> >> > > > > > You can try to find which process has bound this port by >> >> > > > > > "netstat >> >> > > -anp" >> >> > > > > > first. >> >> > > > > > >> >> > > > > > Generally, there are following cases for the binding error. >> >> > > > > > 1. "local port range" is not set to exclude the port >> >> > > > > > range used in >> >> > > > Storm. >> >> > > > > > 2. The previous worker was not killed correctly. >> >> > > > > > 3. There is bug of assignment in some scenarios. Same >> >> > > > > > port was >> >> > > assigned >> >> > > > > to >> >> > > > > > two workers. >> >> > > > > > >> >> > > > > > Regards >> >> > > > > > Basti >> >> > > > > > >> >> > > > > > -----Original Message----- >> >> > > > > > From: sam mohel [mailto:[email protected]] >> >> > > > > > Sent: Thursday, December 10, 2015 7:16 AM >> >> > > > > > To: [email protected] >> >> > > > > > Subject: Re: Problem with storm since 4 months >> >> > > > > > >> >> > > > > > i tried to use storm-0.9.5 but problem changed with >> >> > > > > > >> >> > > > > > cannot bind port 6703 i think it's same problem >> >> > > > > > >> >> > > > > > On Wed, Dec 9, 2015 at 8:42 PM, Harsha <[email protected]> >> wrote: >> >> > > > > > >> >> > > > > > > Sam, >> >> > > > > > > you might be using very old version of storm >> >> > > > > > > since its >> >> > > > > showing >> >> > > > > > > ZeroMQ. Can you try using newer version >> >> > > > > > > storm without >> >> > > zero >> >> > > > > mq. >> >> > > > > > > -Harsha >> >> > > > > > > >> >> > > > > > > On Wed, Dec 9, 2015, at 10:19 AM, sam mohel wrote: >> >> > > > > > > > I have this problem since 4months when I submitted >> >> > > > > > > > topology I got this in the worker log file [ERROR] >> >> > > > > > > > Async >> >> loop died! >> >> > > > > > org.zeromq.ZMQException: >> >> > > > > > > > Address already in use(0x62) at >> >> > > > > > > > org.zeromq.ZMQ$Socket.bind(Native Method) at >> >> > > > > > > > zilch.mq$bind.invoke(mq.clj:69) at >> >> > > > > > > > backtype.storm.messaging.zmq.ZMQContext.bind(zmq.clj >> >> > > > > > > > :57 >> >> > > > > > > > )at >> >> > > > > > > > >> >> > > > > > > >> >> > > > backtype.storm.messaging.loader$launch_receive_thread_BANG_$ >> >> > > > fn_ >> >> > > > _16 >> >> > > > 29 >> >> > > > .i >> >> > > > > > > nvoke(loader.clj:26) >> >> > > > > > > > at >> >> > > > > > > > backtype.storm.util$async_loop$fn__465.invoke(util.clj: >> >> > > > > > > > 375 >> >> > > > > > > > ) at clojure.lang.AFn.run(AFn.java:24) at >> >> > > > java.lang.Thread.run(Unknown >> >> > > > > > > > Source) >> >> > > > > > > > >> >> > > > > > > > when i tried to connect port 6703 and 6702 >> >> > > > > > > > >> >> > > > > > > > And supervisor log file hadn't still start >> >> > > > > > > > >> >> > > > > > > > >> >> > > > > > > > I searched everywhere but cannot find any solution I >> >> > > > > > > > hope you can >> >> > > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > >> >> > > > >> >> > > >> >> > >> >> > >> >> >> >> >> > >> >> >
