[ https://issues.apache.org/jira/browse/HDFS-1592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035533#comment-13035533 ]
Bharath Mundlapudi commented on HDFS-1592: ------------------------------------------ Eli, thanks for your review and comments. Yes, I have tested against trunk. How did you test this? Did you configure volumes tolerated correctly? The expected behavior is - if volumes failed are more than volumes tolerated, BPOfferService daemon will fail to start. Also, note that, i have filed another Jira for - if all BPService exit due to some reason, Datanode should exit. This is a bug in the current code. Please see the following four tests i have performed and their outcome on trunk. Case 1: One disk failure (/grid/2) and Vol Tolerated = 0. Outcome: BP Service should exit. 11/05/18 07:48:56 WARN common.Util: Path /grid/0/testing/hadoop-logs/dfs/data should be specified as a URI in configuration files. Please update hdfs configuration. 11/05/18 07:48:56 WARN common.Util: Path /grid/1/testing/hadoop-logs/dfs/data should be specified as a URI in configuration files. Please update hdfs configuration. 11/05/18 07:48:56 WARN common.Util: Path /grid/2/testing/hadoop-logs/dfs/data should be specified as a URI in configuration files. Please update hdfs configuration. 11/05/18 07:48:56 WARN common.Util: Path /grid/3/testing/hadoop-logs/dfs/data should be specified as a URI in configuration files. Please update hdfs configuration. 11/05/18 07:48:56 WARN datanode.DataNode: Invalid directory in: dfs.datanode.data.dir: java.io.FileNotFoundException: File file:/grid/2/testing/hadoop-logs/dfs/data does not exist. at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:424) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:315) at org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:131) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:148) at org.apache.hadoop.hdfs.server.datanode.DataNode.getDataDirsFromURIs(DataNode.java:2154) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2133) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2074) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2097) at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2240) at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2250) 11/05/18 07:48:56 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 11/05/18 07:48:56 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 11/05/18 07:48:56 INFO impl.MetricsSystemImpl: DataNode metrics system started 11/05/18 07:48:56 INFO impl.MetricsSystemImpl: Registered source UgiMetrics 11/05/18 07:48:56 INFO datanode.DataNode: Opened info server at 50010 11/05/18 07:48:56 INFO datanode.DataNode: Balancing bandwith is 1048576 bytes/s 11/05/18 07:48:56 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 11/05/18 07:48:56 INFO http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) 11/05/18 07:48:56 INFO http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50075 11/05/18 07:48:56 INFO http.HttpServer: listener.getLocalPort() returned 50075 webServer.getConnectors()[0].getLocalPort() returned 50075 11/05/18 07:48:56 INFO http.HttpServer: Jetty bound to port 50075 11/05/18 07:48:56 INFO mortbay.log: jetty-6.1.14 11/05/18 07:48:56 WARN mortbay.log: Can't reuse /tmp/Jetty_0_0_0_0_50075_datanode____hwtdwq, using /tmp/Jetty_0_0_0_0_50075_datanode____hwtdwq_6441176730816569391 11/05/18 07:49:01 INFO mortbay.log: Started SelectChannelConnector@0.0.0.0:50075 11/05/18 07:49:01 INFO ipc.Server: Starting Socket Reader #1 for port 50020 11/05/18 07:49:01 INFO ipc.Server: Starting Socket Reader #2 for port 50020 11/05/18 07:49:01 INFO ipc.Server: Starting Socket Reader #3 for port 50020 11/05/18 07:49:01 INFO ipc.Server: Starting Socket Reader #4 for port 50020 11/05/18 07:49:01 INFO ipc.Server: Starting Socket Reader #5 for port 50020 11/05/18 07:49:01 INFO impl.MetricsSystemImpl: Registered source RpcActivityForPort50020 11/05/18 07:49:01 INFO impl.MetricsSystemImpl: Registered source RpcDetailedActivityForPort50020 11/05/18 07:49:01 INFO impl.MetricsSystemImpl: Registered source JvmMetrics 11/05/18 07:49:01 INFO impl.MetricsSystemImpl: Registered source DataNodeActivity-hadooplab40.yst.corp.yahoo.com-50010 11/05/18 07:49:01 INFO datanode.DataNode: DatanodeRegistration(hadooplab40.yst.corp.yahoo.com:50010, storageID=, infoPort=50075, ipcPort=50020, storageInfo=lv=0;cid=;nsid=0;c=0)In BPOfferService.run, data = null;bp=null 11/05/18 07:49:01 INFO ipc.Server: IPC Server Responder: starting 11/05/18 07:49:01 INFO ipc.Server: IPC Server listener on 50020: starting 11/05/18 07:49:01 INFO ipc.Server: IPC Server handler 0 on 50020: starting 11/05/18 07:49:01 INFO ipc.Server: IPC Server handler 1 on 50020: starting 11/05/18 07:49:01 INFO ipc.Server: IPC Server handler 2 on 50020: starting 11/05/18 07:49:01 INFO datanode.DataNode: handshake: namespace info = lv=-35;cid=test;nsid=413952175;c=0;bpid=BP-1694914230-10.72.86.55-1305704227822 11/05/18 07:49:01 INFO common.Storage: Locking is disabled 11/05/18 07:49:01 INFO common.Storage: Locking is disabled 11/05/18 07:49:01 INFO common.Storage: Locking is disabled 11/05/18 07:49:01 INFO datanode.DataNode: setting up storage: nsid=0;bpid=BP-1694914230-10.72.86.55-1305704227822;lv=-35;nsInfo=lv=-35;cid=test;nsid=413952175;c=0;bpid=BP-1694914230-10.72.86.55-1305704227822 11/05/18 07:49:01 FATAL datanode.DataNode: DatanodeRegistration(hadooplab40.yst.corp.yahoo.com:50010, storageID=DS-340618566-10.72.86.55-50010-1305704313207, infoPort=50075, ipcPort=50020, storageInfo=lv=-35;cid=test;nsid=413952175;c=0) initialization failed for block pool BP-1694914230-10.72.86.55-1305704227822 org.apache.hadoop.util.DiskChecker$DiskErrorException: Invalid value for volumes required - validVolsRequired: 4, Current valid volumes: 3, volsConfigured: 4, volFailuresTolerated: 0 at org.apache.hadoop.hdfs.server.datanode.FSDataset.<init>(FSDataset.java:1160) at org.apache.hadoop.hdfs.server.datanode.DataNode.initFsDataSet(DataNode.java:1420) at org.apache.hadoop.hdfs.server.datanode.DataNode.access$1100(DataNode.java:169) at org.apache.hadoop.hdfs.server.datanode.DataNode$BPOfferService.setupBPStorage(DataNode.java:804) at org.apache.hadoop.hdfs.server.datanode.DataNode$BPOfferService.setupBP(DataNode.java:774) at org.apache.hadoop.hdfs.server.datanode.DataNode$BPOfferService.run(DataNode.java:1191) at java.lang.Thread.run(Thread.java:619) 11/05/18 07:49:01 WARN datanode.DataNode: DatanodeRegistration(hadooplab40.yst.corp.yahoo.com:50010, storageID=DS-340618566-10.72.86.55-50010-1305704313207, infoPort=50075, ipcPort=50020, storageInfo=lv=-35;cid=test;nsid=413952175;c=0) ending block pool service for: BP-1694914230-10.72.86.55-1305704227822 Case 2: One disk failure (/grid/2) and Vol Tolerated = 1. Outcome: BP Service should not exit 11/05/18 08:48:39 WARN common.Util: Path /grid/0/testing/hadoop-logs/dfs/data should be specified as a URI in configuration files. Please update hdfs configuration. 11/05/18 08:48:39 WARN common.Util: Path /grid/1/testing/hadoop-logs/dfs/data should be specified as a URI in configuration files. Please update hdfs configuration. 11/05/18 08:48:39 WARN common.Util: Path /grid/2/testing/hadoop-logs/dfs/data should be specified as a URI in configuration files. Please update hdfs configuration. 11/05/18 08:48:39 WARN common.Util: Path /grid/3/testing/hadoop-logs/dfs/data should be specified as a URI in configuration files. Please update hdfs configuration. 11/05/18 08:48:39 WARN datanode.DataNode: Invalid directory in: dfs.datanode.data.dir: java.io.FileNotFoundException: File file:/grid/2/testing/hadoop-logs/dfs/data does not exist. at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:424) at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:315) at org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:131) at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:148) at org.apache.hadoop.hdfs.server.datanode.DataNode.getDataDirsFromURIs(DataNode.java:2154) at org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:2133) at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2074) at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2097) at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2240) at org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:2250) 11/05/18 08:48:40 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 11/05/18 08:48:40 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 11/05/18 08:48:40 INFO impl.MetricsSystemImpl: DataNode metrics system started 11/05/18 08:48:40 INFO impl.MetricsSystemImpl: Registered source UgiMetrics 11/05/18 08:48:40 INFO datanode.DataNode: Opened info server at 50010 11/05/18 08:48:40 INFO datanode.DataNode: Balancing bandwith is 1048576 bytes/s 11/05/18 08:48:40 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 11/05/18 08:48:40 INFO http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) 11/05/18 08:48:40 INFO http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50075 11/05/18 08:48:40 INFO http.HttpServer: listener.getLocalPort() returned 50075 webServer.getConnectors()[0].getLocalPort() returned 50075 11/05/18 08:48:40 INFO http.HttpServer: Jetty bound to port 50075 11/05/18 08:48:40 INFO mortbay.log: jetty-6.1.14 11/05/18 08:48:40 WARN mortbay.log: Can't reuse /tmp/Jetty_0_0_0_0_50075_datanode____hwtdwq, using /tmp/Jetty_0_0_0_0_50075_datanode____hwtdwq_4334063446071982759 11/05/18 08:48:40 INFO mortbay.log: Started SelectChannelConnector@0.0.0.0:50075 11/05/18 08:48:40 INFO ipc.Server: Starting Socket Reader #1 for port 50020 11/05/18 08:48:40 INFO ipc.Server: Starting Socket Reader #2 for port 50020 11/05/18 08:48:40 INFO ipc.Server: Starting Socket Reader #3 for port 50020 11/05/18 08:48:40 INFO ipc.Server: Starting Socket Reader #4 for port 50020 11/05/18 08:48:40 INFO ipc.Server: Starting Socket Reader #5 for port 50020 11/05/18 08:48:40 INFO impl.MetricsSystemImpl: Registered source RpcActivityForPort50020 11/05/18 08:48:40 INFO impl.MetricsSystemImpl: Registered source RpcDetailedActivityForPort50020 11/05/18 08:48:40 INFO impl.MetricsSystemImpl: Registered source JvmMetrics 11/05/18 08:48:40 INFO impl.MetricsSystemImpl: Registered source DataNodeActivity-hadooplab40.yst.corp.yahoo.com-50010 11/05/18 08:48:40 INFO datanode.DataNode: DatanodeRegistration(hadooplab40.yst.corp.yahoo.com:50010, storageID=, infoPort=50075, ipcPort=50020, storageInfo=lv=0;cid=;nsid=0;c=0)In BPOfferService.run, data = null;bp=null 11/05/18 08:48:40 INFO ipc.Server: IPC Server Responder: starting 11/05/18 08:48:40 INFO ipc.Server: IPC Server listener on 50020: starting 11/05/18 08:48:40 INFO ipc.Server: IPC Server handler 0 on 50020: starting 11/05/18 08:48:40 INFO ipc.Server: IPC Server handler 1 on 50020: starting 11/05/18 08:48:40 INFO ipc.Server: IPC Server handler 2 on 50020: starting 11/05/18 08:48:40 INFO datanode.DataNode: handshake: namespace info = lv=-35;cid=test;nsid=413952175;c=0;bpid=BP-1694914230-10.72.86.55-1305704227822 11/05/18 08:48:40 INFO common.Storage: Locking is disabled 11/05/18 08:48:40 INFO common.Storage: Locking is disabled 11/05/18 08:48:40 INFO common.Storage: Locking is disabled 11/05/18 08:48:40 INFO datanode.DataNode: setting up storage: nsid=0;bpid=BP-1694914230-10.72.86.55-1305704227822;lv=-35;nsInfo=lv=-35;cid=test;nsid=413952175;c=0;bpid=BP-1694914230-10.72.86.55-1305704227822 11/05/18 08:48:40 INFO datanode.DataNode: FSDataset added volume - /grid/0/testing/hadoop-logs/dfs/data/current 11/05/18 08:48:40 INFO datanode.DataNode: FSDataset added volume - /grid/1/testing/hadoop-logs/dfs/data/current 11/05/18 08:48:40 INFO datanode.DataNode: FSDataset added volume - /grid/3/testing/hadoop-logs/dfs/data/current 11/05/18 08:48:40 INFO datanode.DataNode: Registered FSDatasetState MBean 11/05/18 08:48:40 INFO datanode.DataNode: Adding block pool BP-1694914230-10.72.86.55-1305704227822 11/05/18 08:48:40 INFO datanode.DirectoryScanner: Periodic Directory Tree Verification scan starting at 1305719925918 with interval 21600000 11/05/18 08:48:40 INFO datanode.DataNode: in register: sid=DS-340618566-10.72.86.55-50010-1305704313207;SI=lv=-35;cid=test;nsid=413952175;c=0 11/05/18 08:48:40 INFO datanode.DataNode: bpReg after =lv=-35;cid=test;nsid=413952175;c=0;sid=DS-340618566-10.72.86.55-50010-1305704313207;name=127.0.0.1:50010 11/05/18 08:48:40 INFO datanode.DataNode: in register:;bpDNR=lv=-35;cid=test;nsid=413952175;c=0 11/05/18 08:48:40 INFO datanode.DataNode: For namenode localhost/127.0.0.1:8020 using BLOCKREPORT_INTERVAL of 21600000msec Initial delay: 0msec; heartBeatInterval=3000 11/05/18 08:48:40 INFO datanode.DataNode: BlockReport of 0 blocks got processed in 4 msecs 11/05/18 08:48:40 INFO datanode.DataNode: sent block report, processed command:org.apache.hadoop.hdfs.server.protocol.DatanodeCommand$Finalize@135ae7e 11/05/18 08:48:40 INFO datanode.BlockPoolSliceScanner: Periodic Block Verification scan initialized with interval 1814400000. 11/05/18 08:48:40 INFO datanode.DataBlockScanner: Added bpid=BP-1694914230-10.72.86.55-1305704227822 to blockPoolScannerMap, new size=1 11/05/18 08:48:45 INFO datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00% Case 3: All good volumes and Vol Tolerated = 1. Outcome: BP Service should not exit. 11/05/18 09:18:56 WARN common.Util: Path /grid/0/testing/hadoop-logs/dfs/data should be specified as a URI in configuration files. Please update hdfs configuration. 11/05/18 09:18:56 WARN common.Util: Path /grid/1/testing/hadoop-logs/dfs/data should be specified as a URI in configuration files. Please update hdfs configuration. 11/05/18 09:18:56 WARN common.Util: Path /grid/2/testing/hadoop-logs/dfs/data should be specified as a URI in configuration files. Please update hdfs configuration. 11/05/18 09:18:56 WARN common.Util: Path /grid/3/testing/hadoop-logs/dfs/data should be specified as a URI in configuration files. Please update hdfs configuration. 11/05/18 09:18:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 11/05/18 09:18:56 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 11/05/18 09:18:56 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 11/05/18 09:18:56 INFO impl.MetricsSystemImpl: DataNode metrics system started 11/05/18 09:18:56 INFO impl.MetricsSystemImpl: Registered source UgiMetrics 11/05/18 09:18:56 INFO datanode.DataNode: Opened info server at 50010 11/05/18 09:18:56 INFO datanode.DataNode: Balancing bandwith is 1048576 bytes/s 11/05/18 09:18:56 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 11/05/18 09:18:56 INFO http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) 11/05/18 09:18:56 INFO http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50075 11/05/18 09:18:56 INFO http.HttpServer: listener.getLocalPort() returned 50075 webServer.getConnectors()[0].getLocalPort() returned 50075 11/05/18 09:18:56 INFO http.HttpServer: Jetty bound to port 50075 11/05/18 09:18:56 INFO mortbay.log: jetty-6.1.14 11/05/18 09:18:56 WARN mortbay.log: Can't reuse /tmp/Jetty_0_0_0_0_50075_datanode____hwtdwq, using /tmp/Jetty_0_0_0_0_50075_datanode____hwtdwq_5832726280495656689 11/05/18 09:18:56 INFO mortbay.log: Started SelectChannelConnector@0.0.0.0:50075 11/05/18 09:18:57 INFO ipc.Server: Starting Socket Reader #1 for port 50020 11/05/18 09:18:57 INFO ipc.Server: Starting Socket Reader #2 for port 50020 11/05/18 09:18:57 INFO ipc.Server: Starting Socket Reader #3 for port 50020 11/05/18 09:18:57 INFO ipc.Server: Starting Socket Reader #4 for port 50020 11/05/18 09:18:57 INFO ipc.Server: Starting Socket Reader #5 for port 50020 11/05/18 09:18:57 INFO impl.MetricsSystemImpl: Registered source RpcActivityForPort50020 11/05/18 09:18:57 INFO impl.MetricsSystemImpl: Registered source RpcDetailedActivityForPort50020 11/05/18 09:18:57 INFO impl.MetricsSystemImpl: Registered source JvmMetrics 11/05/18 09:18:57 INFO impl.MetricsSystemImpl: Registered source DataNodeActivity-hadooplab40.yst.corp.yahoo.com-50010 11/05/18 09:18:57 INFO datanode.DataNode: DatanodeRegistration(hadooplab40.yst.corp.yahoo.com:50010, storageID=, infoPort=50075, ipcPort=50020, storageInfo=lv=0;cid=;nsid=0;c=0)In BPOfferService.run, data = null;bp=null 11/05/18 09:18:57 INFO ipc.Server: IPC Server Responder: starting 11/05/18 09:18:57 INFO ipc.Server: IPC Server listener on 50020: starting 11/05/18 09:18:57 INFO ipc.Server: IPC Server handler 1 on 50020: starting 11/05/18 09:18:57 INFO ipc.Server: IPC Server handler 0 on 50020: starting 11/05/18 09:18:57 INFO ipc.Server: IPC Server handler 2 on 50020: starting 11/05/18 09:18:57 INFO datanode.DataNode: handshake: namespace info = lv=-35;cid=test;nsid=413952175;c=0;bpid=BP-1694914230-10.72.86.55-1305704227822 11/05/18 09:18:57 INFO common.Storage: Storage directory /grid/2/testing/hadoop-logs/dfs/data is not formatted. 11/05/18 09:18:57 INFO common.Storage: Formatting ... 11/05/18 09:18:57 INFO common.Storage: Locking is disabled 11/05/18 09:18:57 INFO common.Storage: Locking is disabled 11/05/18 09:18:57 INFO common.Storage: Locking is disabled 11/05/18 09:18:57 INFO common.Storage: Storage directory /grid/2/testing/hadoop-logs/dfs/data/current/BP-1694914230-10.72.86.55-1305704227822 is not formatted. 11/05/18 09:18:57 INFO common.Storage: Formatting ... 11/05/18 09:18:57 INFO common.Storage: Formatting block pool BP-1694914230-10.72.86.55-1305704227822 directory /grid/2/testing/hadoop-logs/dfs/data/current/BP-1694914230-10.72.86.55-1305704227822/current 11/05/18 09:18:57 INFO common.Storage: Locking is disabled 11/05/18 09:18:57 INFO datanode.DataNode: setting up storage: nsid=413952175;bpid=BP-1694914230-10.72.86.55-1305704227822;lv=-35;nsInfo=lv=-35;cid=test;nsid=413952175;c=0;bpid=BP-1694914230-10.72.86.55-1305704227822 11/05/18 09:18:57 INFO datanode.DataNode: FSDataset added volume - /grid/0/testing/hadoop-logs/dfs/data/current 11/05/18 09:18:57 INFO datanode.DataNode: FSDataset added volume - /grid/1/testing/hadoop-logs/dfs/data/current 11/05/18 09:18:57 INFO datanode.DataNode: FSDataset added volume - /grid/2/testing/hadoop-logs/dfs/data/current 11/05/18 09:18:57 INFO datanode.DataNode: FSDataset added volume - /grid/3/testing/hadoop-logs/dfs/data/current 11/05/18 09:18:57 INFO datanode.DataNode: Registered FSDatasetState MBean 11/05/18 09:18:57 INFO datanode.DataNode: Adding block pool BP-1694914230-10.72.86.55-1305704227822 11/05/18 09:18:57 INFO datanode.DirectoryScanner: Periodic Directory Tree Verification scan starting at 1305728372371 with interval 21600000 11/05/18 09:18:57 INFO datanode.DataNode: in register: sid=DS-340618566-10.72.86.55-50010-1305704313207;SI=lv=-35;cid=test;nsid=413952175;c=0 11/05/18 09:18:57 INFO datanode.DataNode: bpReg after =lv=-35;cid=test;nsid=413952175;c=0;sid=DS-340618566-10.72.86.55-50010-1305704313207;name=127.0.0.1:50010 11/05/18 09:18:57 INFO datanode.DataNode: in register:;bpDNR=lv=-35;cid=test;nsid=413952175;c=0 11/05/18 09:18:57 INFO datanode.DataNode: For namenode localhost/127.0.0.1:8020 using BLOCKREPORT_INTERVAL of 21600000msec Initial delay: 0msec; heartBeatInterval=3000 11/05/18 09:18:57 INFO datanode.DataNode: BlockReport of 0 blocks got processed in 4 msecs 11/05/18 09:18:57 INFO datanode.DataNode: sent block report, processed command:org.apache.hadoop.hdfs.server.protocol.DatanodeCommand$Finalize@8de972 11/05/18 09:18:57 INFO datanode.BlockPoolSliceScanner: Periodic Block Verification scan initialized with interval 1814400000. 11/05/18 09:18:57 INFO datanode.DataBlockScanner: Added bpid=BP-1694914230-10.72.86.55-1305704227822 to blockPoolScannerMap, new size=1 11/05/18 09:19:02 INFO datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00% Case 4: All good volumes and Vol Tolerated = 0. Outcome: BP Service should not exit. 11/05/18 09:24:16 WARN common.Util: Path /grid/0/testing/hadoop-logs/dfs/data should be specified as a URI in configuration files. Please update hdfs configuration. 11/05/18 09:24:16 WARN common.Util: Path /grid/1/testing/hadoop-logs/dfs/data should be specified as a URI in configuration files. Please update hdfs configuration. 11/05/18 09:24:16 WARN common.Util: Path /grid/2/testing/hadoop-logs/dfs/data should be specified as a URI in configuration files. Please update hdfs configuration. 11/05/18 09:24:16 WARN common.Util: Path /grid/3/testing/hadoop-logs/dfs/data should be specified as a URI in configuration files. Please update hdfs configuration. 11/05/18 09:24:16 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 11/05/18 09:24:16 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s). 11/05/18 09:24:16 INFO impl.MetricsSystemImpl: DataNode metrics system started 11/05/18 09:24:16 INFO impl.MetricsSystemImpl: Registered source UgiMetrics 11/05/18 09:24:16 INFO datanode.DataNode: Opened info server at 50010 11/05/18 09:24:16 INFO datanode.DataNode: Balancing bandwith is 1048576 bytes/s 11/05/18 09:24:16 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 11/05/18 09:24:16 INFO http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter) 11/05/18 09:24:16 INFO http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50075 11/05/18 09:24:16 INFO http.HttpServer: listener.getLocalPort() returned 50075 webServer.getConnectors()[0].getLocalPort() returned 50075 11/05/18 09:24:16 INFO http.HttpServer: Jetty bound to port 50075 11/05/18 09:24:16 INFO mortbay.log: jetty-6.1.14 11/05/18 09:24:16 WARN mortbay.log: Can't reuse /tmp/Jetty_0_0_0_0_50075_datanode____hwtdwq, using /tmp/Jetty_0_0_0_0_50075_datanode____hwtdwq_5258458250806180443 11/05/18 09:24:17 INFO mortbay.log: Started SelectChannelConnector@0.0.0.0:50075 11/05/18 09:24:17 INFO ipc.Server: Starting Socket Reader #1 for port 50020 11/05/18 09:24:17 INFO ipc.Server: Starting Socket Reader #2 for port 50020 11/05/18 09:24:17 INFO ipc.Server: Starting Socket Reader #3 for port 50020 11/05/18 09:24:17 INFO ipc.Server: Starting Socket Reader #4 for port 50020 11/05/18 09:24:17 INFO ipc.Server: Starting Socket Reader #5 for port 50020 11/05/18 09:24:17 INFO impl.MetricsSystemImpl: Registered source RpcActivityForPort50020 11/05/18 09:24:17 INFO impl.MetricsSystemImpl: Registered source RpcDetailedActivityForPort50020 11/05/18 09:24:17 INFO impl.MetricsSystemImpl: Registered source JvmMetrics 11/05/18 09:24:17 INFO impl.MetricsSystemImpl: Registered source DataNodeActivity-hadooplab40.yst.corp.yahoo.com-50010 11/05/18 09:24:17 INFO datanode.DataNode: DatanodeRegistration(hadooplab40.yst.corp.yahoo.com:50010, storageID=, infoPort=50075, ipcPort=50020, storageInfo=lv=0;cid=;nsid=0;c=0)In BPOfferService.run, data = null;bp=null 11/05/18 09:24:17 INFO ipc.Server: IPC Server Responder: starting 11/05/18 09:24:17 INFO ipc.Server: IPC Server listener on 50020: starting 11/05/18 09:24:17 INFO ipc.Server: IPC Server handler 0 on 50020: starting 11/05/18 09:24:17 INFO ipc.Server: IPC Server handler 1 on 50020: starting 11/05/18 09:24:17 INFO ipc.Server: IPC Server handler 2 on 50020: starting 11/05/18 09:24:17 INFO datanode.DataNode: handshake: namespace info = lv=-35;cid=test;nsid=413952175;c=0;bpid=BP-1694914230-10.72.86.55-1305704227822 11/05/18 09:24:17 INFO common.Storage: Locking is disabled 11/05/18 09:24:17 INFO common.Storage: Locking is disabled 11/05/18 09:24:17 INFO common.Storage: Locking is disabled 11/05/18 09:24:17 INFO common.Storage: Locking is disabled 11/05/18 09:24:17 INFO datanode.DataNode: setting up storage: nsid=0;bpid=BP-1694914230-10.72.86.55-1305704227822;lv=-35;nsInfo=lv=-35;cid=test;nsid=413952175;c=0;bpid=BP-1694914230-10.72.86.55-1305704227822 11/05/18 09:24:17 INFO datanode.DataNode: FSDataset added volume - /grid/0/testing/hadoop-logs/dfs/data/current 11/05/18 09:24:17 INFO datanode.DataNode: FSDataset added volume - /grid/1/testing/hadoop-logs/dfs/data/current 11/05/18 09:24:17 INFO datanode.DataNode: FSDataset added volume - /grid/2/testing/hadoop-logs/dfs/data/current 11/05/18 09:24:17 INFO datanode.DataNode: FSDataset added volume - /grid/3/testing/hadoop-logs/dfs/data/current 11/05/18 09:24:17 INFO datanode.DataNode: Registered FSDatasetState MBean 11/05/18 09:24:17 INFO datanode.DataNode: Adding block pool BP-1694914230-10.72.86.55-1305704227822 11/05/18 09:24:17 INFO datanode.DirectoryScanner: Periodic Directory Tree Verification scan starting at 1305719970633 with interval 21600000 11/05/18 09:24:17 INFO datanode.DataNode: in register: sid=DS-340618566-10.72.86.55-50010-1305704313207;SI=lv=-35;cid=test;nsid=413952175;c=0 11/05/18 09:24:17 INFO datanode.DataNode: bpReg after =lv=-35;cid=test;nsid=413952175;c=0;sid=DS-340618566-10.72.86.55-50010-1305704313207;name=127.0.0.1:50010 11/05/18 09:24:17 INFO datanode.DataNode: in register:;bpDNR=lv=-35;cid=test;nsid=413952175;c=0 11/05/18 09:24:17 INFO datanode.DataNode: For namenode localhost/127.0.0.1:8020 using BLOCKREPORT_INTERVAL of 21600000msec Initial delay: 0msec; heartBeatInterval=3000 11/05/18 09:24:17 INFO datanode.DataNode: BlockReport of 0 blocks got processed in 4 msecs 11/05/18 09:24:17 INFO datanode.DataNode: sent block report, processed command:org.apache.hadoop.hdfs.server.protocol.DatanodeCommand$Finalize@18c5e67 11/05/18 09:24:17 INFO datanode.BlockPoolSliceScanner: Periodic Block Verification scan initialized with interval 1814400000. 11/05/18 09:24:17 INFO datanode.DataBlockScanner: Added bpid=BP-1694914230-10.72.86.55-1305704227822 to blockPoolScannerMap, new size=1 11/05/18 09:24:22 INFO datanode.BlockPoolSliceScanner: Starting a new period : work left in prev period : 0.00% > Datanode startup doesn't honor volumes.tolerated > ------------------------------------------------- > > Key: HDFS-1592 > URL: https://issues.apache.org/jira/browse/HDFS-1592 > Project: Hadoop HDFS > Issue Type: Bug > Affects Versions: 0.20.204.0 > Reporter: Bharath Mundlapudi > Assignee: Bharath Mundlapudi > Fix For: 0.20.204.0, 0.23.0 > > Attachments: HDFS-1592-1.patch, HDFS-1592-2.patch, > HDFS-1592-rel20.patch > > > Datanode startup doesn't honor volumes.tolerated for hadoop 20 version. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira