Jetty can't start the SelectChannelConnector
Hi all, I configured the Hadoop in a cluster and the NameNode and JobTracker are running ok, but the DataNode and TaskTracker Doesn't start, they stop and keep waiting when they are going to start the Jetty I observed that Jetty can't start the _SelectChannelConnector_ Is there any Jetty configuration that should be changed ? There is no log message in the NN and JT when I try to start the DN and TT. The kernel I'm using is: Linux bl05 2.6.32.10 #2 SMP Tue Apr 6 12:33:42 BRT 2010 x86_64 GNU/Linux This is the message when I start the DN. It happens with TT too. ram...@bl05:~/hadoop-0.20.1+169.56$ ./bin/hadoop datanode 10/04/06 16:24:14 INFO datanode.DataNode: STARTUP_MSG: / STARTUP_MSG: Starting DataNode STARTUP_MSG: host = bl05.ctinfra.ufpr.br/192.168.1.115 STARTUP_MSG: args = [] STARTUP_MSG: version = 0.20.1+169.56 STARTUP_MSG: build = -r 8e662cb065be1c4bc61c55e6bff161e09c1d36f3; compiled by 'chad' on Tue Feb 2 13:27:17 PST 2010 / 10/04/06 16:24:14 INFO datanode.DataNode: Registered FSDatasetStatusMBean 10/04/06 16:24:14 INFO datanode.DataNode: Opened info server at 50010 10/04/06 16:24:14 INFO datanode.DataNode: Balancing bandwith is 1048576 bytes/s 10/04/06 16:24:14 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 10/04/06 16:24:14 INFO http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50075 10/04/06 16:24:14 INFO http.HttpServer: listener.getLocalPort() returned 50075 webServer.getConnectors()[0].getLocalPort() returned 50075 10/04/06 16:24:14 INFO http.HttpServer: Jetty bound to port 50075 10/04/06 16:24:14 INFO mortbay.log: jetty-6.1.14 Thanks in Advance, Edson Ramiro
Re: Jetty can't start the SelectChannelConnector
Hi Todd, I'm getting this behavior in another cluster too, there the same thing happens. and as I don't have a jstack installed in the first cluster and I'm not the admin, I'm sending the results of the second cluster. These are the results: [erl...@cohiba ~ ]$ jstack -l 22510 22510: well-known file is not secure [erl...@cohiba ~ ]$ jstack -l 3836 3836: well-known file is not secure The jstack -F result is in thread_dump files and the jstack -m result is in java_native_frames. The files ending with nn are the namenode results and the files ending with dn are the datanode results. Thanks, Edson Ramiro On 6 April 2010 18:19, Todd Lipcon t...@cloudera.com wrote: Hi Edson, Can you please run jstack on the daemons in question and paste the output here? -Todd On Tue, Apr 6, 2010 at 12:44 PM, Edson Ramiro erlfi...@gmail.com wrote: Hi all, I configured the Hadoop in a cluster and the NameNode and JobTracker are running ok, but the DataNode and TaskTracker Doesn't start, they stop and keep waiting when they are going to start the Jetty I observed that Jetty can't start the _SelectChannelConnector_ Is there any Jetty configuration that should be changed ? There is no log message in the NN and JT when I try to start the DN and TT. The kernel I'm using is: Linux bl05 2.6.32.10 #2 SMP Tue Apr 6 12:33:42 BRT 2010 x86_64 GNU/Linux This is the message when I start the DN. It happens with TT too. ram...@bl05:~/hadoop-0.20.1+169.56$ ./bin/hadoop datanode 10/04/06 16:24:14 INFO datanode.DataNode: STARTUP_MSG: / STARTUP_MSG: Starting DataNode STARTUP_MSG: host = bl05.ctinfra.ufpr.br/192.168.1.115 STARTUP_MSG: args = [] STARTUP_MSG: version = 0.20.1+169.56 STARTUP_MSG: build = -r 8e662cb065be1c4bc61c55e6bff161e09c1d36f3; compiled by 'chad' on Tue Feb 2 13:27:17 PST 2010 / 10/04/06 16:24:14 INFO datanode.DataNode: Registered FSDatasetStatusMBean 10/04/06 16:24:14 INFO datanode.DataNode: Opened info server at 50010 10/04/06 16:24:14 INFO datanode.DataNode: Balancing bandwith is 1048576 bytes/s 10/04/06 16:24:14 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 10/04/06 16:24:14 INFO http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50075 10/04/06 16:24:14 INFO http.HttpServer: listener.getLocalPort() returned 50075 webServer.getConnectors()[0].getLocalPort() returned 50075 10/04/06 16:24:14 INFO http.HttpServer: Jetty bound to port 50075 10/04/06 16:24:14 INFO mortbay.log: jetty-6.1.14 Thanks in Advance, Edson Ramiro -- Todd Lipcon Software Engineer, Cloudera
Re: Jetty can't start the SelectChannelConnector
Hi Edson, Your attachments did not come through - can you put them on pastebin? -Todd On Tue, Apr 6, 2010 at 3:37 PM, Edson Ramiro erlfi...@gmail.com wrote: Hi Todd, I'm getting this behavior in another cluster too, there the same thing happens. and as I don't have a jstack installed in the first cluster and I'm not the admin, I'm sending the results of the second cluster. These are the results: [erl...@cohiba ~ ]$ jstack -l 22510 22510: well-known file is not secure [erl...@cohiba ~ ]$ jstack -l 3836 3836: well-known file is not secure The jstack -F result is in thread_dump files and the jstack -m result is in java_native_frames. The files ending with nn are the namenode results and the files ending with dn are the datanode results. Thanks, Edson Ramiro On 6 April 2010 18:19, Todd Lipcon t...@cloudera.com wrote: Hi Edson, Can you please run jstack on the daemons in question and paste the output here? -Todd On Tue, Apr 6, 2010 at 12:44 PM, Edson Ramiro erlfi...@gmail.com wrote: Hi all, I configured the Hadoop in a cluster and the NameNode and JobTracker are running ok, but the DataNode and TaskTracker Doesn't start, they stop and keep waiting when they are going to start the Jetty I observed that Jetty can't start the _SelectChannelConnector_ Is there any Jetty configuration that should be changed ? There is no log message in the NN and JT when I try to start the DN and TT. The kernel I'm using is: Linux bl05 2.6.32.10 #2 SMP Tue Apr 6 12:33:42 BRT 2010 x86_64 GNU/Linux This is the message when I start the DN. It happens with TT too. ram...@bl05:~/hadoop-0.20.1+169.56$ ./bin/hadoop datanode 10/04/06 16:24:14 INFO datanode.DataNode: STARTUP_MSG: / STARTUP_MSG: Starting DataNode STARTUP_MSG: host = bl05.ctinfra.ufpr.br/192.168.1.115 STARTUP_MSG: args = [] STARTUP_MSG: version = 0.20.1+169.56 STARTUP_MSG: build = -r 8e662cb065be1c4bc61c55e6bff161e09c1d36f3; compiled by 'chad' on Tue Feb 2 13:27:17 PST 2010 / 10/04/06 16:24:14 INFO datanode.DataNode: Registered FSDatasetStatusMBean 10/04/06 16:24:14 INFO datanode.DataNode: Opened info server at 50010 10/04/06 16:24:14 INFO datanode.DataNode: Balancing bandwith is 1048576 bytes/s 10/04/06 16:24:14 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 10/04/06 16:24:14 INFO http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50075 10/04/06 16:24:14 INFO http.HttpServer: listener.getLocalPort() returned 50075 webServer.getConnectors()[0].getLocalPort() returned 50075 10/04/06 16:24:14 INFO http.HttpServer: Jetty bound to port 50075 10/04/06 16:24:14 INFO mortbay.log: jetty-6.1.14 Thanks in Advance, Edson Ramiro -- Todd Lipcon Software Engineer, Cloudera -- Todd Lipcon Software Engineer, Cloudera
Re: Jetty can't start the SelectChannelConnector
Thanks Todd, but could you please explain why entropy is important ? Edson Ramiro On 6 April 2010 20:09, Todd Lipcon t...@cloudera.com wrote: Not enough entropy on your system - you need to generate entropy or fake some using this technique: http://www.chrissearle.org/blog/technical/increase_entropy_26_kernel_linux_box -Todd On Tue, Apr 6, 2010 at 4:05 PM, Edson Ramiro erlfi...@gmail.com wrote: ok, [erl...@cohiba ~ ]$ cat java_native_frames_dn Attaching to process ID 3836, please wait... Debugger attached successfully. Server compiler detected. JVM version is 11.2-b01 Deadlock Detection: No deadlocks found. - 3864 - 0xf775a4f1__libc_read + 0x41 0xf702298creadBytes + 0xdc 0xf701e717Java_java_io_FileInputStream_readBytes + 0x47 0xf400b4aa* java.io.FileInputStream.readBytes(byte[], int, int) bci:0 (Interpreted frame) 0xf4003f69* java.io.FileInputStream.read(byte[], int, int) bci:4 line:199 (Interpreted frame) 0xf4003f69* java.io.BufferedInputStream.read1(byte[], int, int) bci:39 line:256 (Interpreted frame) 0xf4003f69* java.io.BufferedInputStream.read(byte[], int, int) bci:49 line:317 (Interpreted frame) 0xf4003f69* java.io.BufferedInputStream.fill() bci:175 line:218 (Interpreted frame) 0xf400408d* java.io.BufferedInputStream.read1(byte[], int, int) bci:44 line:258 (Interpreted frame) 0xf4003f69* java.io.BufferedInputStream.read(byte[], int, int) bci:49 line:317 (Interpreted frame) 0xf4003f69* sun.security.provider.SeedGenerator$URLSeedGenerator.getSeedByte() bci:12 line:453 (Interpreted frame) 0xf4003e61* sun.security.provider.SeedGenerator.getSeedBytes(byte[]) bci:11 line:123 (Interpreted frame) 0xf400408d* sun.security.provider.SeedGenerator.generateSeed(byte[]) bci:4 line:118 (Interpreted frame) 0xf400408d* sun.security.provider.SecureRandom.engineGenerateSeed(int) bci:5 line:114 (Interpreted frame) 0xf4003f27* sun.security.provider.SecureRandom.engineNextBytes(byte[]) bci:40 line:171 (Interpreted frame) 0xf400408d* java.security.SecureRandom.nextBytes(byte[]) bci:5 line:433 (Interpreted frame) 0xf400408d* java.security.SecureRandom.next(int) bci:17 line:455 (Interpreted frame) 0xf4003f69* java.util.Random.nextLong() bci:3 line:284 (Interpreted frame) 0xf4003fab* org.mortbay.jetty.servlet.HashSessionIdManager.doStart() bci:73 line:139 (Interpreted frame) 0xf400408d* org.mortbay.component.AbstractLifeCycle.start() bci:31 line:50 (Interpreted frame) 0xf4004569* org.mortbay.jetty.servlet.AbstractSessionManager.doStart() bci:96 line:168 (Interpreted frame) 0xf400408d* org.mortbay.jetty.servlet.HashSessionManager.doStart() bci:12 line:67 (Interpreted frame) 0xf400408d* org.mortbay.component.AbstractLifeCycle.start() bci:31 line:50 (Interpreted frame) 0xf4004569* org.mortbay.jetty.servlet.SessionHandler.doStart() bci:4 line:115 (Interpreted frame) 0xf400408d* org.mortbay.component.AbstractLifeCycle.start() bci:31 line:50 (Interpreted frame) 0xf4004569* org.mortbay.jetty.handler.HandlerWrapper.doStart() bci:11 line:130 (Interpreted frame) 0xf400408d* org.mortbay.jetty.handler.ContextHandler.startContext() bci:1 line:537 (Interpreted frame) 0xf400408d* org.mortbay.jetty.servlet.Context.startContext() bci:1 line:136 (Interpreted frame) 0xf400408d* org.mortbay.jetty.webapp.WebAppContext.startContext() bci:123 line:1234 (Interpreted frame) 0xf400408d* org.mortbay.jetty.handler.ContextHandler.doStart() bci:140 line:517 (Interpreted frame) 0xf400408d* org.mortbay.jetty.webapp.WebAppContext.doStart() bci:170 line:460 (Interpreted frame) 0xf400408d* org.mortbay.component.AbstractLifeCycle.start() bci:31 line:50 (Interpreted frame) 0xf4004569* org.mortbay.jetty.handler.HandlerCollection.doStart() bci:32 line:152 (Interpreted frame) 0xf400408d* org.mortbay.jetty.handler.ContextHandlerCollection.doStart() bci:5 line:156 (Interpreted frame) 0xf400408d* org.mortbay.component.AbstractLifeCycle.start() bci:31 line:50 (Interpreted frame) 0xf4004569* org.mortbay.jetty.handler.HandlerWrapper.doStart() bci:11 line:130 (Interpreted frame) 0xf400408d* org.mortbay.jetty.Server.doStart() bci:201 line:222 (Interpreted frame) 0xf400408d* org.mortbay.component.AbstractLifeCycle.start() bci:31 line:50 (Interpreted frame) 0xf400408d* org.apache.hadoop.http.HttpServer.start() bci:383 line:461 (Interpreted frame) 0xf400408d* org.apache.hadoop.hdfs.server.datanode.DataNode.startDataNode(org.apache.hadoop.conf.Configuration, java.util.AbstractList) bci:916 line:375 (Interpreted frame) 0xf400408d* org.apache.hadoop.hdfs.server.datanode.DataNode.init(org.apache.hadoop.conf.Configuration, java.util.AbstractList) bci:158 line:216
