Hi,
I've been running hadoop 0.20.2 for a while now on 2 different clusters
that I setup. Now on this new cluster I can't get the namenode to stay
up. It exits with a IOException "incomplete hdfs uri" and prints the uri:
hdfs://rmsi_combined.rmsi.com:54310 -which looks complete to me.
All the usual suspects are ok, DNS works both ways, passphraseless ssh
works fine, 54310 is open, so I don't know what to tell the IT folks to
fix. in core-site.xml, fs.default.name is set to the ip address:
hdfs://10.2.50.235:54310 (same problem if it is set to the hostname)
The only way to get it to work is to set fs.default.name to
hdfs://localhost:54310, but that isn't going to let me run a multi-node
cluster.
The IOException boils up from the uri.getHost() call in
DistributedFileSystem.initialize() as shown in the attached log snippit.
The exception is thrown if the host returned by getHost() is null.
This has got to be some kind of permission problem somewhere.
Anyone have any ideas where to look?
Thanks!
-Bill
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = rmsi_combined/127.0.0.1
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.2
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
************************************************************/
2011-03-30 07:32:37,748 INFO org.apache.hadoop.ipc.metrics.RpcMetrics: Initializing RPC Metrics with hostName=NameNode, port=54310
2011-03-30 07:32:37,759 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Namenode up at: rmsi_combined.rmsi.com/10.2.50.235:54310
2011-03-30 07:32:37,763 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=NameNode, sessionId=null
2011-03-30 07:32:37,765 INFO org.apache.hadoop.hdfs.server.namenode.metrics.NameNodeMetrics: Initializing NameNodeMeterics using context object:org.apache.hadoop.metrics.spi.NullContext
2011-03-30 07:32:37,845 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner=dmc2,dmc2
2011-03-30 07:32:37,845 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup=supergroup
2011-03-30 07:32:37,845 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled=false
2011-03-30 07:32:37,858 INFO org.apache.hadoop.hdfs.server.namenode.metrics.FSNamesystemMetrics: Initializing FSNamesystemMetrics using context object:org.apache.hadoop.metrics.spi.NullContext
2011-03-30 07:32:37,861 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered FSNamesystemStatusMBean
2011-03-30 07:32:37,912 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files = 1
2011-03-30 07:32:37,919 INFO org.apache.hadoop.hdfs.server.common.Storage: Number of files under construction = 0
2011-03-30 07:32:37,919 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 94 loaded in 0 seconds.
2011-03-30 07:32:37,920 INFO org.apache.hadoop.hdfs.server.common.Storage: Edits file /hdfs/name/current/edits of size 4 edits # 0 loaded in 0 seconds.
2011-03-30 07:32:37,931 INFO org.apache.hadoop.hdfs.server.common.Storage: Image file of size 94 saved in 0 seconds.
2011-03-30 07:32:37,967 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading FSImage in 146 msecs
2011-03-30 07:32:37,969 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Total number of blocks = 0
2011-03-30 07:32:37,969 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of invalid blocks = 0
2011-03-30 07:32:37,969 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of under-replicated blocks = 0
2011-03-30 07:32:37,969 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of over-replicated blocks = 0
2011-03-30 07:32:37,969 INFO org.apache.hadoop.hdfs.StateChange: STATE* Leaving safe mode after 0 secs.
2011-03-30 07:32:37,970 INFO org.apache.hadoop.hdfs.StateChange: STATE* Network topology has 0 racks and 0 datanodes
2011-03-30 07:32:37,970 INFO org.apache.hadoop.hdfs.StateChange: STATE* UnderReplicatedBlocks has 0 blocks
2011-03-30 07:32:38,151 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2011-03-30 07:32:38,245 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50070
2011-03-30 07:32:38,247 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50070 webServer.getConnectors()[0].getLocalPort() returned 50070
2011-03-30 07:32:38,247 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50070
2011-03-30 07:32:38,247 INFO org.mortbay.log: jetty-6.1.14
2011-03-30 07:32:38,663 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50070
2011-03-30 07:32:38,663 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Web-server up at: 0.0.0.0:50070
2011-03-30 07:32:38,663 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2011-03-30 07:32:38,665 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 54310: starting
2011-03-30 07:32:38,665 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 54310: starting
2011-03-30 07:32:38,666 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 54310: starting
2011-03-30 07:32:38,666 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 54310: starting
2011-03-30 07:32:38,666 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 54310: starting
2011-03-30 07:32:38,673 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 54310: starting
2011-03-30 07:32:38,673 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 54310: starting
2011-03-30 07:32:38,673 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 54310: starting
2011-03-30 07:32:38,673 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 54310: starting
2011-03-30 07:32:38,674 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 54310: starting
2011-03-30 07:32:38,674 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 54310: starting
2011-03-30 07:32:38,674 INFO org.apache.hadoop.ipc.Server: IPC Server handler 10 on 54310: starting
2011-03-30 07:32:38,674 INFO org.apache.hadoop.ipc.Server: IPC Server handler 11 on 54310: starting
2011-03-30 07:32:38,674 INFO org.apache.hadoop.ipc.Server: IPC Server handler 12 on 54310: starting
2011-03-30 07:32:38,675 INFO org.apache.hadoop.ipc.Server: IPC Server handler 13 on 54310: starting
2011-03-30 07:32:38,675 INFO org.apache.hadoop.ipc.Server: IPC Server handler 14 on 54310: starting
2011-03-30 07:32:38,675 INFO org.apache.hadoop.ipc.Server: IPC Server handler 15 on 54310: starting
2011-03-30 07:32:38,675 INFO org.apache.hadoop.ipc.Server: IPC Server handler 16 on 54310: starting
2011-03-30 07:32:38,676 INFO org.apache.hadoop.ipc.Server: IPC Server handler 17 on 54310: starting
2011-03-30 07:32:38,676 INFO org.apache.hadoop.ipc.Server: IPC Server handler 18 on 54310: starting
2011-03-30 07:32:38,676 INFO org.apache.hadoop.ipc.Server: IPC Server handler 19 on 54310: starting
2011-03-30 07:32:38,676 INFO org.apache.hadoop.ipc.Server: IPC Server handler 20 on 54310: starting
2011-03-30 07:32:38,676 INFO org.apache.hadoop.ipc.Server: IPC Server handler 21 on 54310: starting
2011-03-30 07:32:38,677 INFO org.apache.hadoop.ipc.Server: IPC Server handler 22 on 54310: starting
2011-03-30 07:32:38,677 INFO org.apache.hadoop.ipc.Server: IPC Server handler 23 on 54310: starting
2011-03-30 07:32:38,677 INFO org.apache.hadoop.ipc.Server: IPC Server handler 24 on 54310: starting
2011-03-30 07:32:38,677 INFO org.apache.hadoop.ipc.Server: IPC Server handler 25 on 54310: starting
2011-03-30 07:32:38,678 INFO org.apache.hadoop.ipc.Server: IPC Server handler 26 on 54310: starting
2011-03-30 07:32:38,678 INFO org.apache.hadoop.ipc.Server: IPC Server handler 27 on 54310: starting
2011-03-30 07:32:38,678 INFO org.apache.hadoop.ipc.Server: IPC Server handler 28 on 54310: starting
2011-03-30 07:32:38,678 INFO org.apache.hadoop.ipc.Server: IPC Server handler 29 on 54310: starting
2011-03-30 07:32:38,678 INFO org.apache.hadoop.ipc.Server: IPC Server handler 30 on 54310: starting
2011-03-30 07:32:38,687 INFO org.apache.hadoop.ipc.Server: IPC Server handler 31 on 54310: starting
2011-03-30 07:32:38,783 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: ReplicationMonitor thread received InterruptedException.java.lang.InterruptedException: sleep interrupted
2011-03-30 07:32:38,785 INFO org.apache.hadoop.hdfs.server.namenode.DecommissionManager: Interrupted Monitor
java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at org.apache.hadoop.hdfs.server.namenode.DecommissionManager$Monitor.run(DecommissionManager.java:65)
at java.lang.Thread.run(Thread.java:619)
2011-03-30 07:32:38,786 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of transactions: 0 Total time for transactions(ms): 0Number of transactions batched in Syncs: 0 Number of syncs: 0 SyncTimes(ms): 0
2011-03-30 07:32:38,792 INFO org.apache.hadoop.ipc.Server: Stopping server on 54310
2011-03-30 07:32:38,793 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 54310: exiting
2011-03-30 07:32:38,793 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 54310: exiting
2011-03-30 07:32:38,793 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 54310: exiting
2011-03-30 07:32:38,793 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 54310: exiting
2011-03-30 07:32:38,794 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 54310: exiting
2011-03-30 07:32:38,794 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 54310: exiting
2011-03-30 07:32:38,794 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 54310: exiting
2011-03-30 07:32:38,794 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 54310: exiting
2011-03-30 07:32:38,794 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 54310: exiting
2011-03-30 07:32:38,794 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 54310: exiting
2011-03-30 07:32:38,795 INFO org.apache.hadoop.ipc.Server: IPC Server handler 10 on 54310: exiting
2011-03-30 07:32:38,795 INFO org.apache.hadoop.ipc.Server: IPC Server handler 11 on 54310: exiting
2011-03-30 07:32:38,795 INFO org.apache.hadoop.ipc.Server: IPC Server handler 12 on 54310: exiting
2011-03-30 07:32:38,795 INFO org.apache.hadoop.ipc.Server: IPC Server handler 13 on 54310: exiting
2011-03-30 07:32:38,795 INFO org.apache.hadoop.ipc.Server: IPC Server handler 14 on 54310: exiting
2011-03-30 07:32:38,796 INFO org.apache.hadoop.ipc.Server: IPC Server handler 15 on 54310: exiting
2011-03-30 07:32:38,796 INFO org.apache.hadoop.ipc.Server: IPC Server handler 16 on 54310: exiting
2011-03-30 07:32:38,796 INFO org.apache.hadoop.ipc.Server: IPC Server handler 17 on 54310: exiting
2011-03-30 07:32:38,796 INFO org.apache.hadoop.ipc.Server: IPC Server handler 18 on 54310: exiting
2011-03-30 07:32:38,796 INFO org.apache.hadoop.ipc.Server: IPC Server handler 19 on 54310: exiting
2011-03-30 07:32:38,796 INFO org.apache.hadoop.ipc.Server: IPC Server handler 20 on 54310: exiting
2011-03-30 07:32:38,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 21 on 54310: exiting
2011-03-30 07:32:38,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 22 on 54310: exiting
2011-03-30 07:32:38,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 23 on 54310: exiting
2011-03-30 07:32:38,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 24 on 54310: exiting
2011-03-30 07:32:38,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 25 on 54310: exiting
2011-03-30 07:32:38,797 INFO org.apache.hadoop.ipc.Server: IPC Server handler 26 on 54310: exiting
2011-03-30 07:32:38,798 INFO org.apache.hadoop.ipc.Server: IPC Server handler 27 on 54310: exiting
2011-03-30 07:32:38,798 INFO org.apache.hadoop.ipc.Server: IPC Server handler 28 on 54310: exiting
2011-03-30 07:32:38,798 INFO org.apache.hadoop.ipc.Server: IPC Server handler 29 on 54310: exiting
2011-03-30 07:32:38,798 INFO org.apache.hadoop.ipc.Server: IPC Server handler 30 on 54310: exiting
2011-03-30 07:32:38,798 INFO org.apache.hadoop.ipc.Server: IPC Server handler 31 on 54310: exiting
2011-03-30 07:32:38,799 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 54310
2011-03-30 07:32:38,799 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2011-03-30 07:32:38,800 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.io.IOException: Incomplete HDFS URI, no host: hdfs://rmsi_combined.rmsi.com:54310
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:78)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
at org.apache.hadoop.fs.Trash.<init>(Trash.java:62)
at org.apache.hadoop.hdfs.server.namenode.NameNode.startTrashEmptier(NameNode.java:208)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:204)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:279)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:956)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:965)
2011-03-30 07:32:38,801 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at rmsi_combined/127.0.0.1
************************************************************/
