A shard-server was heavily loaded yesterday & ultimately crashed with an
OOM.
I tried to restart the shard-server but it quit with the following error
INFO 20160425_05:10:02:556_PDT [main] thrift.ThriftBlurShardServer:
Setting up Shard Server
INFO 20160425_05:10:02:581_PDT [main] thrift.ThriftServer: ulimit:
core file size (blocks, -c) unlimited
INFO 20160425_05:10:02:581_PDT [main] thrift.ThriftServer: ulimit:
data seg size (kbytes, -d) unlimited
INFO 20160425_05:10:02:581_PDT [main] thrift.ThriftServer: ulimit:
scheduling priority (-e) 0
INFO 20160425_05:10:02:581_PDT [main] thrift.ThriftServer: ulimit:
file size (blocks, -f) unlimited
INFO 20160425_05:10:02:581_PDT [main] thrift.ThriftServer: ulimit:
pending signals (-i) 1031474
INFO 20160425_05:10:02:582_PDT [main] thrift.ThriftServer: ulimit:
max locked memory (kbytes, -l) 64
INFO 20160425_05:10:02:582_PDT [main] thrift.ThriftServer: ulimit:
max memory size (kbytes, -m) unlimited
INFO 20160425_05:10:02:582_PDT [main] thrift.ThriftServer: ulimit:
open files (-n) 65536
INFO 20160425_05:10:02:582_PDT [main] thrift.ThriftServer: ulimit:
pipe size (512 bytes, -p) 8
INFO 20160425_05:10:02:582_PDT [main] thrift.ThriftServer: ulimit:
POSIX message queues (bytes, -q) 819200
INFO 20160425_05:10:02:582_PDT [main] thrift.ThriftServer: ulimit:
real-time priority (-r) 0
INFO 20160425_05:10:02:582_PDT [main] thrift.ThriftServer: ulimit:
stack size (kbytes, -s) 10240
INFO 20160425_05:10:02:582_PDT [main] thrift.ThriftServer: ulimit:
cpu time (seconds, -t) unlimited
INFO 20160425_05:10:02:582_PDT [main] thrift.ThriftServer: ulimit:
max user processes (-u) 3047
INFO 20160425_05:10:02:582_PDT [main] thrift.ThriftServer: ulimit:
virtual memory (kbytes, -v) unlimited
INFO 20160425_05:10:02:582_PDT [main] thrift.ThriftServer: ulimit:
file locks (-x) unlimited
INFO 20160425_05:10:02:686_PDT [main] utils.GCWatcherJdk7:
GCWatcherJdk7 was setup.
ERROR 20160425_05:10:02:719_PDT [main]
concurrent.SimpleUncaughtExceptionHandler: Unknown error in thread
[Thread[main,5,main]]
org.apache.blur.thirdparty.thrift_0_9_0.transport.TTransportException:
Could not create ServerSocket on address /0.0.0.0:40020.
at
org.apache.blur.thirdparty.thrift_0_9_0.transport.TNonblockingServerSocket.<init>(TNonblockingServerSocket.java:91)
at
org.apache.blur.thirdparty.thrift_0_9_0.transport.TNonblockingServerSocket.<init>(TNonblockingServerSocket.java:73)
at
org.apache.blur.thrift.ThriftServer.getTNonblockingServerSocket(ThriftServer.java:246)
at
org.apache.blur.thrift.ThriftBlurShardServer.createServer(ThriftBlurShardServer.java:155)
at
org.apache.blur.thrift.ThriftBlurShardServer.main(ThriftBlurShardServer.java:139)
The shard-server process was killed but thrift port seemed to be hanging on
(Guess it was on TIMED_WAIT) & not released.
I also saw HttpJettyServer reporting the same issue when I attempted
restart for a second time...
*Caused by: java.net.BindException: Address already in use*
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)
at java.net.ServerSocket.bind(ServerSocket.java:376)
at java.net.ServerSocket.(ServerSocket.java:237)
at java.net.ServerSocket.(ServerSocket.java:181)
at
org.mortbay.jetty.bio.SocketConnector.newServerSocket(SocketConnector.java:80)
at org.mortbay.jetty.bio.SocketConnector.open(SocketConnector.java:73)
at org.mortbay.jetty.AbstractConnector.doStart(AbstractConnector.java:283)
at org.mortbay.jetty.bio.SocketConnector.doStart(SocketConnector.java:147)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.Server.doStart(Server.java:235)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.apache.blur.gui.HttpJettyServer.(HttpJettyServer.java:93)
Under heavy loads will ports be hanging around even after process gets
killed?
Any known work-arounds for this?
--
Ravi
