Datanode install on cluster Dockers

[林家銘]

Hi

I install three datanodes ( Hadoop 2.7.1) in a multiple Docker cluster
which is connected by Flannel.
The bridge on each physical node are

physical host 1:
IP address for flannel0: 172.16.91.0
IP address for docker0:  172.16.91.1

physical host 2:
IP address for flannel0: 172.16.46.0
IP address for docker0:  172.16.46.1

physical host 3:
IP address for flannel0: 172.16.31.0
IP address for docker0:  172.16.31.1

There are two datanodes are installed on "physical host 1", with IPs
172.16.91.19 and 172.16.91.20
One on "physical host 2" with IP 172.16.46.22
Namenode is installed on "physical host 2" with IP 172.16.46.21

And the problem is when I check the HDFS health status there is only
two datanodes appear.

[hdfs@amb1-host-dev bin]$ hadoop dfsadmin -report
Name: 
172.16.46.22:50010(k8s_POD-afaa08c2_amb4-rc-dev-n35n9_default_42432432-71d9-11e5-8d52-000c2900be4e_5fee9eed)
Hostname: amb4-host-dev
Decommission Status : Normal
Configured Capacity: 87571161088 (81.56 GB)
DFS Used: 33554432 (32 MB)
Non DFS Used: 32210010112 (30.00 GB)
DFS Remaining: 55327596544 (51.53 GB)
DFS Used%: 0.04%
DFS Remaining%: 63.18%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Wed Oct 14 00:16:42 UTC 2015


Name: 172.16.91.0:50010 (172.16.91.0)   <---here
is the problem
Hostname: amb3-host-dev
Decommission Status : Normal
Configured Capacity: 87571161088 (81.56 GB)
DFS Used: 28672 (28 KB)
Non DFS Used: 36738072576 (34.21 GB)
DFS Remaining: 50833059840 (47.34 GB)
DFS Used%: 0.00%
DFS Remaining%: 58.05%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 2
Last contact: Wed Oct 14 00:16:41 UTC 2015

Why there is only one datanode on "physical host2", and why HDFS
regards the whole physical host as a datanode?

Re: Passing instance of a class to Mapper

[Chris Nauroth]

Yes, the grep example job from the Hadoop codebase is a good demo.

Here we can see the grep job setting up Configuration with the regex to match:

https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/Grep.java

The RegexMapper class then consumes this by overriding setup to read the regex 
back out of Configuration:

https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/map/RegexMapper.java

--Chris Nauroth

From: , Saurav mailto:sda...@paypal.com>>
Reply-To: "user@hadoop.apache.org" 
mailto:user@hadoop.apache.org>>
Date: Tuesday, October 13, 2015 at 1:04 PM
To: "user@hadoop.apache.org" 
mailto:user@hadoop.apache.org>>
Subject: Re: Passing instance of a class to Mapper

Thanks very much Chris!
I will try it out. Do you have any examples showing this  ?

--


From: Chris Nauroth
Reply-To: "user@hadoop.apache.org"
Date: Tuesday, October 13, 2015 at 12:07 PM
To: "user@hadoop.apache.org"
Subject: Re: Passing instance of a class to Mapper

Hello Saurav,

You are correct that it generally is not possible to pass an instance of a 
class directly to a mapper (or reducer).  This is because the mapper tasks 
execute on arbitrary nodes in the Hadoop cluster, running in different JVM 
processes from the JVM running the client that submits the job.

A typical solution is for the client to populate the Configuration object with 
relevant primitive data type values.

http://hadoop.apache.org/docs/r2.7.1/api/org/apache/hadoop/conf/Configuration.html

This configuration propagates to all map and reduce tasks of the job.  The 
Mapper can override the setup function to do one-time initialization at the 
start of the task.

http://hadoop.apache.org/docs/r2.7.1/api/org/apache/hadoop/mapreduce/Mapper.html#setup(org.apache.hadoop.mapreduce.Mapper.Context)

As part of this one-time initialization, you can read the values back out of 
the Configuration.  As I said earlier, these will be only primitive types like 
String or int.  If it's helpful, your setup method can use the primitive values 
read from configuration to reconstruct an instance of any class that you want.

I hope this helps.

--Chris Nauroth

From: , Saurav mailto:sda...@paypal.com>>
Reply-To: "user@hadoop.apache.org" 
mailto:user@hadoop.apache.org>>
Date: Monday, October 12, 2015 at 11:14 PM
To: "user@hadoop.apache.org" 
mailto:user@hadoop.apache.org>>
Subject: Passing instance of a class to Mapper

Hello,

I am trying to pass an instance of a class to a Mapper. However, I understand 
Hadoop does not allow this.
Any workaround to make this happen ?

Regards,
Saurav Datta

Data Engineer| Desk - (408)967-7360| Cell - (408)666-1722

Re: Passing instance of a class to Mapper

[Datta, Saurav]

Thanks very much Chris!
I will try it out. Do you have any examples showing this  ?

--


From: Chris Nauroth
Reply-To: "user@hadoop.apache.org"
Date: Tuesday, October 13, 2015 at 12:07 PM
To: "user@hadoop.apache.org"
Subject: Re: Passing instance of a class to Mapper

Hello Saurav,

You are correct that it generally is not possible to pass an instance of a 
class directly to a mapper (or reducer).  This is because the mapper tasks 
execute on arbitrary nodes in the Hadoop cluster, running in different JVM 
processes from the JVM running the client that submits the job.

A typical solution is for the client to populate the Configuration object with 
relevant primitive data type values.

http://hadoop.apache.org/docs/r2.7.1/api/org/apache/hadoop/conf/Configuration.html

This configuration propagates to all map and reduce tasks of the job.  The 
Mapper can override the setup function to do one-time initialization at the 
start of the task.

http://hadoop.apache.org/docs/r2.7.1/api/org/apache/hadoop/mapreduce/Mapper.html#setup(org.apache.hadoop.mapreduce.Mapper.Context)

As part of this one-time initialization, you can read the values back out of 
the Configuration.  As I said earlier, these will be only primitive types like 
String or int.  If it's helpful, your setup method can use the primitive values 
read from configuration to reconstruct an instance of any class that you want.

I hope this helps.

--Chris Nauroth

From: , Saurav mailto:sda...@paypal.com>>
Reply-To: "user@hadoop.apache.org" 
mailto:user@hadoop.apache.org>>
Date: Monday, October 12, 2015 at 11:14 PM
To: "user@hadoop.apache.org" 
mailto:user@hadoop.apache.org>>
Subject: Passing instance of a class to Mapper

Hello,

I am trying to pass an instance of a class to a Mapper. However, I understand 
Hadoop does not allow this.
Any workaround to make this happen ?

Regards,
Saurav Datta

Data Engineer| Desk - (408)967-7360| Cell - (408)666-1722

Re: Passing instance of a class to Mapper

[Chris Nauroth]

Hello Saurav,

You are correct that it generally is not possible to pass an instance of a 
class directly to a mapper (or reducer).  This is because the mapper tasks 
execute on arbitrary nodes in the Hadoop cluster, running in different JVM 
processes from the JVM running the client that submits the job.

A typical solution is for the client to populate the Configuration object with 
relevant primitive data type values.

http://hadoop.apache.org/docs/r2.7.1/api/org/apache/hadoop/conf/Configuration.html

This configuration propagates to all map and reduce tasks of the job.  The 
Mapper can override the setup function to do one-time initialization at the 
start of the task.

http://hadoop.apache.org/docs/r2.7.1/api/org/apache/hadoop/mapreduce/Mapper.html#setup(org.apache.hadoop.mapreduce.Mapper.Context)

As part of this one-time initialization, you can read the values back out of 
the Configuration.  As I said earlier, these will be only primitive types like 
String or int.  If it's helpful, your setup method can use the primitive values 
read from configuration to reconstruct an instance of any class that you want.

I hope this helps.

--Chris Nauroth

From: , Saurav mailto:sda...@paypal.com>>
Reply-To: "user@hadoop.apache.org" 
mailto:user@hadoop.apache.org>>
Date: Monday, October 12, 2015 at 11:14 PM
To: "user@hadoop.apache.org" 
mailto:user@hadoop.apache.org>>
Subject: Passing instance of a class to Mapper

Hello,

I am trying to pass an instance of a class to a Mapper. However, I understand 
Hadoop does not allow this.
Any workaround to make this happen ?

Regards,
Saurav Datta

Data Engineer| Desk - (408)967-7360| Cell - (408)666-1722

Re: unsubscribe

[Rich Haase]

Please see https://hadoop.apache.org/mailing_lists.html for unsubscribe 
instructions.

On Oct 13, 2015, at 3:55 AM, shanthi k 
mailto:kshanthi...@gmail.com>> wrote:


Who is thiz?

On 13 Oct 2015 15:18, "MANISH SINGLA" 
mailto:coolmanishh...@gmail.com>> wrote:


--
Regards
Manish Singla

Re: unsubscribe

[shanthi k]

Who is thiz?
On 13 Oct 2015 15:18, "MANISH SINGLA"  wrote:

>
>
> --
> Regards
> Manish Singla
>

unsubscribe

[MANISH SINGLA]

-- 
Regards
Manish Singla