Datanode support different Namespace
Hi, all We try to use hadoop-2.0.5-alpha, using two namespaces, one is for hbase cluster, and the other one is for common use.At the same time, we use Quorum Journal policy as HA. GS-CIX-SEV0001, GS-CIX-SEV0002, namenodes in hbasecluster namespace GS-CIX-SEV0003, GS-CIX-SEV0004, namenodes in commoncluster namespace. GS-CIX-SEV0001~GS-CIX-SEV0008 , 8 machines used as Datanode After launching the hdfs cluster all, there is something which makes me confused, that each namespace has half of the datanodes. NameNode 'GS-CIX-SEV0004:9100' Started:Thu Jul 04 10:28:00 CST 2013 Version:2.0.5-alpha, 1488459 Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha Cluster ID: CID-15c48d78-2137-4c6e-aacf-0edbf2bb3db7 Block Pool ID: BP-1792015895-10.100.2.3-1372904504940 Browse the filesystem NameNode Logs Go back to DFS home Live Datanodes : 4 Last AdminConfigured Used Non DFS Remaining Used Used Remaining Block Block Pool Failed Node Contact State Capacity (GB) Used (GB) (%) (%) (%)Blocks Pool Used (%) Volumes (GB) (GB) Used (GB) Blocks GS-CIX-SEV0001 1 In Service 888.07 0.00 116.04772.03 0.00 ┌┐ 86.93 0 0.00 0.000 └┘ GS-CIX-SEV0002 1 In Service 888.07 0.00 135.50752.57 0.00 ┌┐ 84.74 0 0.00 0.000 └┘ GS-CIX-SEV0005 1 In Service 888.07 0.0097.61790.46 0.00 ┌┐ 89.01 0 0.00 0.000 └┘ GS-CIX-SEV0006 1 In Service 888.07 0.00 122.30765.77 0.00 ┌┐ 86.23 0 0.00 0.000 └┘ Another Namespace's NameNode: NameNode 'GS-CIX-SEV0001:9100' Started:Thu Jul 04 10:19:03 CST 2013 Version:2.0.5-alpha, 1488459 Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha Cluster ID: CID-1a53483d-000e-4726-aef1-f500bedb1df6 Block Pool ID: BP-1142418822-10.100.2.1-1372904314309 Browse the filesystem NameNode Logs Go back to DFS home Live Datanodes : 4 Last AdminConfigured Used Non DFS Remaining Used Used Remaining Block Block Pool Failed Node Contact State Capacity (GB) Used (GB) (%) (%) (%)Blocks Pool Used (%) Volumes (GB) (GB) Used (GB) Blocks GS-CIX-SEV0003 0 In Service 888.07 0.00 150.54737.53 0.00 ┌┐ 83.05 0 0.00 0.000 └┘ GS-CIX-SEV0004 0 In Service 888.07 0.00 177.22710.85 0.00 ┌┐ 80.04 0 0.00 0.000 └┘ GS-CIX-SEV0007 0 In Service 888.07 0.0062.91825.16 0.00 ┌┐ 92.92 0 0.00 0.000 └┘ GS-CIX-SEV0008 0 In Service 888.07 0.00 125.25762.82 0.00 ┌┐ 85.90 0 0.00 0.000 And check the DN(GS-CIX-SEV0001)'s log, it prints like this: 2013-07-04 10:34:51,699 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool BP-1142418822-10.100.2.1-1372904314309 (storage id DS-1677272131-10.100.2.1-50010-1372905291690) service to GS-CIX-SEV0001/ 10.100.2.1:9100 java.io.IOException: Inconsistent storage IDs. Name-node returned DS811369792. Expecting DS-1677272131-10.100.2.1-50010-1372905291690 at org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:731) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:308) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:632) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.connectToNNAndHandshake(BPServiceActor.java:225) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:664) at java.lang.Thread.run(Thread.java:662) It is proved that one datanode has been required to attached to only one namespace? Any views about it will be thankful.
Re: Datanode support different Namespace
This is because you don't use the same clusterID. all data nodes and namenodes should use the same clusterID. On Thu, Jul 4, 2013 at 3:12 PM, Bing Jiang jiangbinglo...@gmail.com wrote: Hi, all We try to use hadoop-2.0.5-alpha, using two namespaces, one is for hbase cluster, and the other one is for common use.At the same time, we use Quorum Journal policy as HA. GS-CIX-SEV0001, GS-CIX-SEV0002, namenodes in hbasecluster namespace GS-CIX-SEV0003, GS-CIX-SEV0004, namenodes in commoncluster namespace. GS-CIX-SEV0001~GS-CIX-SEV0008 , 8 machines used as Datanode After launching the hdfs cluster all, there is something which makes me confused, that each namespace has half of the datanodes. NameNode 'GS-CIX-SEV0004:9100' Started:Thu Jul 04 10:28:00 CST 2013 Version:2.0.5-alpha, 1488459 Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha Cluster ID: CID-15c48d78-2137-4c6e-aacf-0edbf2bb3db7 Block Pool ID: BP-1792015895-10.100.2.3-1372904504940 Browse the filesystem NameNode Logs Go back to DFS home Live Datanodes : 4 Last AdminConfigured Used Non DFS Remaining Used Used Remaining Block Block Pool Failed Node Contact State Capacity (GB) Used (GB) (%) (%) (%)Blocks Pool Used (%) Volumes (GB) (GB) Used (GB) Blocks GS-CIX-SEV0001 1 In Service 888.07 0.00 116.04772.03 0.00 ┌┐ 86.93 0 0.00 0.000 └┘ GS-CIX-SEV0002 1 In Service 888.07 0.00 135.50752.57 0.00 ┌┐ 84.74 0 0.00 0.000 └┘ GS-CIX-SEV0005 1 In Service 888.07 0.0097.61790.46 0.00 ┌┐ 89.01 0 0.00 0.000 └┘ GS-CIX-SEV0006 1 In Service 888.07 0.00 122.30765.77 0.00 ┌┐ 86.23 0 0.00 0.000 └┘ Another Namespace's NameNode: NameNode 'GS-CIX-SEV0001:9100' Started:Thu Jul 04 10:19:03 CST 2013 Version:2.0.5-alpha, 1488459 Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha Cluster ID: CID-1a53483d-000e-4726-aef1-f500bedb1df6 Block Pool ID: BP-1142418822-10.100.2.1-1372904314309 Browse the filesystem NameNode Logs Go back to DFS home Live Datanodes : 4 Last AdminConfigured Used Non DFS Remaining Used Used Remaining Block Block Pool Failed Node Contact State Capacity (GB) Used (GB) (%) (%) (%)Blocks Pool Used (%) Volumes (GB) (GB) Used (GB) Blocks GS-CIX-SEV0003 0 In Service 888.07 0.00 150.54737.53 0.00 ┌┐ 83.05 0 0.00 0.000 └┘ GS-CIX-SEV0004 0 In Service 888.07 0.00 177.22710.85 0.00 ┌┐ 80.04 0 0.00 0.000 └┘ GS-CIX-SEV0007 0 In Service 888.07 0.0062.91825.16 0.00 ┌┐ 92.92 0 0.00 0.000 └┘ GS-CIX-SEV0008 0 In Service 888.07 0.00 125.25762.82 0.00 ┌┐ 85.90 0 0.00 0.000 And check the DN(GS-CIX-SEV0001)'s log, it prints like this: 2013-07-04 10:34:51,699 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool BP-1142418822-10.100.2.1-1372904314309 (storage id DS-1677272131-10.100.2.1-50010-1372905291690) service to GS-CIX-SEV0001/ 10.100.2.1:9100 java.io.IOException: Inconsistent storage IDs. Name-node returned DS811369792. Expecting DS-1677272131-10.100.2.1-50010-1372905291690 at org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:731) at org.apache.hadoop.hdfs.server.datanode.BPOfferService.registrationSucceeded(BPOfferService.java:308) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.register(BPServiceActor.java:632) at
Re: Datanode support different Namespace
Additional, If these are two new clusters, then on each namenode, using hdfs namenode -format -clusterID yourID But if you want to upgrade these two clusters from NonHA to HA, then using bin/start-dfs.sh -upgrade -clusterID yourID On Thu, Jul 4, 2013 at 3:14 PM, Azuryy Yu azury...@gmail.com wrote: This is because you don't use the same clusterID. all data nodes and namenodes should use the same clusterID. On Thu, Jul 4, 2013 at 3:12 PM, Bing Jiang jiangbinglo...@gmail.comwrote: Hi, all We try to use hadoop-2.0.5-alpha, using two namespaces, one is for hbase cluster, and the other one is for common use.At the same time, we use Quorum Journal policy as HA. GS-CIX-SEV0001, GS-CIX-SEV0002, namenodes in hbasecluster namespace GS-CIX-SEV0003, GS-CIX-SEV0004, namenodes in commoncluster namespace. GS-CIX-SEV0001~GS-CIX-SEV0008 , 8 machines used as Datanode After launching the hdfs cluster all, there is something which makes me confused, that each namespace has half of the datanodes. NameNode 'GS-CIX-SEV0004:9100' Started:Thu Jul 04 10:28:00 CST 2013 Version:2.0.5-alpha, 1488459 Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha Cluster ID: CID-15c48d78-2137-4c6e-aacf-0edbf2bb3db7 Block Pool ID: BP-1792015895-10.100.2.3-1372904504940 Browse the filesystem NameNode Logs Go back to DFS home Live Datanodes : 4 Last AdminConfigured Used Non DFS Remaining Used Used Remaining Block Block Pool Failed Node Contact State Capacity (GB) Used (GB) (%) (%) (%)Blocks Pool Used (%) Volumes (GB) (GB) Used (GB) Blocks GS-CIX-SEV0001 1 In Service 888.07 0.00 116.04772.03 0.00 ┌┐ 86.93 0 0.00 0.000 └┘ GS-CIX-SEV0002 1 In Service 888.07 0.00 135.50752.57 0.00 ┌┐ 84.74 0 0.00 0.000 └┘ GS-CIX-SEV0005 1 In Service 888.07 0.0097.61790.46 0.00 ┌┐ 89.01 0 0.00 0.000 └┘ GS-CIX-SEV0006 1 In Service 888.07 0.00 122.30765.77 0.00 ┌┐ 86.23 0 0.00 0.000 └┘ Another Namespace's NameNode: NameNode 'GS-CIX-SEV0001:9100' Started:Thu Jul 04 10:19:03 CST 2013 Version:2.0.5-alpha, 1488459 Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha Cluster ID: CID-1a53483d-000e-4726-aef1-f500bedb1df6 Block Pool ID: BP-1142418822-10.100.2.1-1372904314309 Browse the filesystem NameNode Logs Go back to DFS home Live Datanodes : 4 Last AdminConfigured Used Non DFS Remaining Used Used Remaining Block Block Pool Failed Node Contact State Capacity (GB) Used (GB) (%) (%) (%)Blocks Pool Used (%) Volumes (GB) (GB) Used (GB) Blocks GS-CIX-SEV0003 0 In Service 888.07 0.00 150.54737.53 0.00 ┌┐ 83.05 0 0.00 0.000 └┘ GS-CIX-SEV0004 0 In Service 888.07 0.00 177.22710.85 0.00 ┌┐ 80.04 0 0.00 0.000 └┘ GS-CIX-SEV0007 0 In Service 888.07 0.0062.91825.16 0.00 ┌┐ 92.92 0 0.00 0.000 └┘ GS-CIX-SEV0008 0 In Service 888.07 0.00 125.25762.82 0.00 ┌┐ 85.90 0 0.00 0.000 And check the DN(GS-CIX-SEV0001)'s log, it prints like this: 2013-07-04 10:34:51,699 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool BP-1142418822-10.100.2.1-1372904314309 (storage id DS-1677272131-10.100.2.1-50010-1372905291690) service to GS-CIX-SEV0001/ 10.100.2.1:9100 java.io.IOException: Inconsistent storage IDs. Name-node returned DS811369792. Expecting DS-1677272131-10.100.2.1-50010-1372905291690 at org.apache.hadoop.hdfs.server.datanode.DataNode.bpRegistrationSucceeded(DataNode.java:731) at
Re: Datanode support different Namespace
If not set cluster id in formatting the Namenode, is there a policy in hdfs to guarantee the even of distributing DataNodes into different Namespace, or just randomly? 2013/7/4 Azuryy Yu azury...@gmail.com Additional, If these are two new clusters, then on each namenode, using hdfs namenode -format -clusterID yourID But if you want to upgrade these two clusters from NonHA to HA, then using bin/start-dfs.sh -upgrade -clusterID yourID On Thu, Jul 4, 2013 at 3:14 PM, Azuryy Yu azury...@gmail.com wrote: This is because you don't use the same clusterID. all data nodes and namenodes should use the same clusterID. On Thu, Jul 4, 2013 at 3:12 PM, Bing Jiang jiangbinglo...@gmail.comwrote: Hi, all We try to use hadoop-2.0.5-alpha, using two namespaces, one is for hbase cluster, and the other one is for common use.At the same time, we use Quorum Journal policy as HA. GS-CIX-SEV0001, GS-CIX-SEV0002, namenodes in hbasecluster namespace GS-CIX-SEV0003, GS-CIX-SEV0004, namenodes in commoncluster namespace. GS-CIX-SEV0001~GS-CIX-SEV0008 , 8 machines used as Datanode After launching the hdfs cluster all, there is something which makes me confused, that each namespace has half of the datanodes. NameNode 'GS-CIX-SEV0004:9100' Started:Thu Jul 04 10:28:00 CST 2013 Version:2.0.5-alpha, 1488459 Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha Cluster ID: CID-15c48d78-2137-4c6e-aacf-0edbf2bb3db7 Block Pool ID: BP-1792015895-10.100.2.3-1372904504940 Browse the filesystem NameNode Logs Go back to DFS home Live Datanodes : 4 Last AdminConfigured Used Non DFS Remaining Used Used Remaining Block Block Pool Failed Node Contact State Capacity (GB) Used (GB) (%) (%) (%)Blocks Pool Used (%) Volumes (GB) (GB) Used (GB) Blocks GS-CIX-SEV0001 1 In Service 888.07 0.00 116.04772.03 0.00 ┌┐ 86.93 0 0.00 0.000 └┘ GS-CIX-SEV0002 1 In Service 888.07 0.00 135.50752.57 0.00 ┌┐ 84.74 0 0.00 0.000 └┘ GS-CIX-SEV0005 1 In Service 888.07 0.0097.61790.46 0.00 ┌┐ 89.01 0 0.00 0.000 └┘ GS-CIX-SEV0006 1 In Service 888.07 0.00 122.30765.77 0.00 ┌┐ 86.23 0 0.00 0.000 └┘ Another Namespace's NameNode: NameNode 'GS-CIX-SEV0001:9100' Started:Thu Jul 04 10:19:03 CST 2013 Version:2.0.5-alpha, 1488459 Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha Cluster ID: CID-1a53483d-000e-4726-aef1-f500bedb1df6 Block Pool ID: BP-1142418822-10.100.2.1-1372904314309 Browse the filesystem NameNode Logs Go back to DFS home Live Datanodes : 4 Last AdminConfigured Used Non DFS Remaining Used Used Remaining Block Block Pool Failed Node Contact State Capacity (GB) Used (GB) (%) (%) (%)Blocks Pool Used (%) Volumes (GB) (GB) Used (GB) Blocks GS-CIX-SEV0003 0 In Service 888.07 0.00 150.54737.53 0.00 ┌┐ 83.05 0 0.00 0.000 └┘ GS-CIX-SEV0004 0 In Service 888.07 0.00 177.22710.85 0.00 ┌┐ 80.04 0 0.00 0.000 └┘ GS-CIX-SEV0007 0 In Service 888.07 0.0062.91825.16 0.00 ┌┐ 92.92 0 0.00 0.000 └┘ GS-CIX-SEV0008 0 In Service 888.07 0.00 125.25762.82 0.00 ┌┐ 85.90 0 0.00 0.000 And check the DN(GS-CIX-SEV0001)'s log, it prints like this: 2013-07-04 10:34:51,699 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool BP-1142418822-10.100.2.1-1372904314309 (storage id DS-1677272131-10.100.2.1-50010-1372905291690) service to GS-CIX-SEV0001/ 10.100.2.1:9100 java.io.IOException: Inconsistent storage
Re: Datanode support different Namespace
It's random. On Jul 4, 2013 3:33 PM, Bing Jiang jiangbinglo...@gmail.com wrote: If not set cluster id in formatting the Namenode, is there a policy in hdfs to guarantee the even of distributing DataNodes into different Namespace, or just randomly? 2013/7/4 Azuryy Yu azury...@gmail.com Additional, If these are two new clusters, then on each namenode, using hdfs namenode -format -clusterID yourID But if you want to upgrade these two clusters from NonHA to HA, then using bin/start-dfs.sh -upgrade -clusterID yourID On Thu, Jul 4, 2013 at 3:14 PM, Azuryy Yu azury...@gmail.com wrote: This is because you don't use the same clusterID. all data nodes and namenodes should use the same clusterID. On Thu, Jul 4, 2013 at 3:12 PM, Bing Jiang jiangbinglo...@gmail.comwrote: Hi, all We try to use hadoop-2.0.5-alpha, using two namespaces, one is for hbase cluster, and the other one is for common use.At the same time, we use Quorum Journal policy as HA. GS-CIX-SEV0001, GS-CIX-SEV0002, namenodes in hbasecluster namespace GS-CIX-SEV0003, GS-CIX-SEV0004, namenodes in commoncluster namespace. GS-CIX-SEV0001~GS-CIX-SEV0008 , 8 machines used as Datanode After launching the hdfs cluster all, there is something which makes me confused, that each namespace has half of the datanodes. NameNode 'GS-CIX-SEV0004:9100' Started:Thu Jul 04 10:28:00 CST 2013 Version:2.0.5-alpha, 1488459 Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha Cluster ID: CID-15c48d78-2137-4c6e-aacf-0edbf2bb3db7 Block Pool ID: BP-1792015895-10.100.2.3-1372904504940 Browse the filesystem NameNode Logs Go back to DFS home Live Datanodes : 4 Last AdminConfigured Used Non DFS Remaining Used Used Remaining Block Block Pool Failed Node Contact State Capacity (GB) Used (GB) (%) (%) (%)Blocks Pool Used (%) Volumes (GB) (GB) Used (GB) Blocks GS-CIX-SEV0001 1 In Service 888.07 0.00 116.04772.03 0.00 ┌┐ 86.93 0 0.00 0.000 └┘ GS-CIX-SEV0002 1 In Service 888.07 0.00 135.50752.57 0.00 ┌┐ 84.74 0 0.00 0.000 └┘ GS-CIX-SEV0005 1 In Service 888.07 0.0097.61790.46 0.00 ┌┐ 89.01 0 0.00 0.000 └┘ GS-CIX-SEV0006 1 In Service 888.07 0.00 122.30765.77 0.00 ┌┐ 86.23 0 0.00 0.000 └┘ Another Namespace's NameNode: NameNode 'GS-CIX-SEV0001:9100' Started:Thu Jul 04 10:19:03 CST 2013 Version:2.0.5-alpha, 1488459 Compiled: 2013-06-01T04:05Z by jenkins from branch-2.0.5-alpha Cluster ID: CID-1a53483d-000e-4726-aef1-f500bedb1df6 Block Pool ID: BP-1142418822-10.100.2.1-1372904314309 Browse the filesystem NameNode Logs Go back to DFS home Live Datanodes : 4 Last AdminConfigured Used Non DFS Remaining Used Used Remaining Block Block Pool Failed Node Contact State Capacity (GB) Used (GB) (%) (%) (%)Blocks Pool Used (%) Volumes (GB) (GB) Used (GB) Blocks GS-CIX-SEV0003 0 In Service 888.07 0.00 150.54737.53 0.00 ┌┐ 83.05 0 0.00 0.000 └┘ GS-CIX-SEV0004 0 In Service 888.07 0.00 177.22710.85 0.00 ┌┐ 80.04 0 0.00 0.000 └┘ GS-CIX-SEV0007 0 In Service 888.07 0.0062.91825.16 0.00 ┌┐ 92.92 0 0.00 0.000 └┘ GS-CIX-SEV0008 0 In Service 888.07 0.00 125.25762.82 0.00 ┌┐ 85.90 0 0.00 0.000 And check the DN(GS-CIX-SEV0001)'s log, it prints like this: 2013-07-04 10:34:51,699 FATAL org.apache.hadoop.hdfs.server.datanode.DataNode: Initialization failed for block pool Block pool BP-1142418822-10.100.2.1-1372904314309 (storage id DS-1677272131-10.100.2.1-50010-1372905291690)
