Re: HDFS Federation with HA

2018-10-10 Thread Rams Venkatesh 
Please email to user-unsubscr...@hadoop.apache.org to get yourself removed from 
the distribution list


> On 10 Oct 2018, at 3:14 pm, Pankaj Kumar  
> wrote:
> 
> Remove Me.
>  
> ---
> Thanks & Regards
> Pankaj Kumar
> IDC Technologies Inc.
> Work: 408-457-9381- Ext-4102
> Mailto: pankaj.ku...@idctechnologies.com
> -
> Empowering Technologies Services
> Remote Services | IT Services | BPO |
> IT Consulting | Staffing Solutions |
> -
>  
> From: rahul patwari [mailto:rahulpatwari8...@gmail.com] 
> Sent: Wednesday, October 10, 2018 3:56 AM
> To: tasan...@yahoo-corp.jp
> Cc: user@hadoop.apache.org
> Subject: Re: HDFS Federation with HA
>  
> Hi Takanobu,
>  
> Thanks for the reply. So, does that mean that 3 dedicated journal nodes are 
> required for each nameservice? or can we have the same journal nodes for all 
> the nameservices?
>  
> Thanks,
> Rahul 
>  
> On Wed, Oct 10, 2018 at 1:00 PM Takanobu Asanuma  
> wrote:
> Hi Rahul,
> 
> Yes, it is possible. Here are the sample configurations.
> 
> ===
> dfs.nameservices=ns1,ns2
> dfs.ha.namenodes.ns1=nn1,nn2
> dfs.ha.namenodes.ns2=nn1,nn2
> 
> dfs.namenode.rpc-address.ns1.nn1=ns1nn1:8020
> dfs.namenode.rpc-address.ns1.nn2=ns1nn2:8020
> dfs.namenode.http-address.ns1.nn1=ns1nn1:9870
> dfs.namenode.http-address.ns1.nn2=ns1nn2:9870
> 
> dfs.namenode.rpc-address.ns2.nn1=ns2nn1:8020
> dfs.namenode.rpc-address.ns2.nn2=ns2nn2:8020
> dfs.namenode.http-address.ns2.nn1=ns2nn1:9870
> dfs.namenode.http-address.ns2.nn2=ns2nn2:9870
> 
> dfs.client.failover.proxy.provider.ns1=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
> dfs.client.failover.proxy.provider.ns2=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
> 
> dfs.namenode.shared.edits.dir.ns1=qjournal://ns1jn1:8485;ns1jn2:8485;ns1jn3:8485/ns1
> dfs.namenode.shared.edits.dir.ns2=qjournal://ns2jn1:8485;ns2jn2:8485;ns2jn3:8485/ns2
> ===
> 
> Regards,
> - Takanobu
> 
> on 2018/10/10 3:07, "rahul patwari" wrote:
> 
> Hi,
> 
> 
> Is it possible in the current architecture of HDFS to have multiple 
> nameservices(federation), with Active and Backup Namenode(HA) for each of the 
> nameservices?
> 
> 
> Thanks,
> Rahul 
> 
>

RE: HDFS Federation with HA

2018-10-10 Thread Pankaj Kumar 
Remove Me.

 

---

Thanks & Regards
Pankaj Kumar

IDC Technologies Inc.

Work: 408-457-9381- Ext-4102

Mailto: pankaj.ku...@idctechnologies.com

-

Empowering Technologies Services

Remote Services | IT Services | BPO |

IT Consulting | Staffing Solutions |

-

 

From: rahul patwari [mailto:rahulpatwari8...@gmail.com] 
Sent: Wednesday, October 10, 2018 3:56 AM
To: tasan...@yahoo-corp.jp
Cc: user@hadoop.apache.org
Subject: Re: HDFS Federation with HA

 

Hi Takanobu,

 

Thanks for the reply. So, does that mean that 3 dedicated journal nodes are 
required for each nameservice? or can we have the same journal nodes for all 
the nameservices?

 

Thanks,

Rahul 

 

On Wed, Oct 10, 2018 at 1:00 PM Takanobu Asanuma  wrote:

Hi Rahul,

Yes, it is possible. Here are the sample configurations.

===
dfs.nameservices=ns1,ns2
dfs.ha.namenodes.ns1=nn1,nn2
dfs.ha.namenodes.ns2=nn1,nn2

dfs.namenode.rpc-address.ns1.nn1=ns1nn1:8020
dfs.namenode.rpc-address.ns1.nn2=ns1nn2:8020
dfs.namenode.http-address.ns1.nn1=ns1nn1:9870
dfs.namenode.http-address.ns1.nn2=ns1nn2:9870

dfs.namenode.rpc-address.ns2.nn1=ns2nn1:8020
dfs.namenode.rpc-address.ns2.nn2=ns2nn2:8020
dfs.namenode.http-address.ns2.nn1=ns2nn1:9870
dfs.namenode.http-address.ns2.nn2=ns2nn2:9870

dfs.client.failover.proxy.provider.ns1=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
dfs.client.failover.proxy.provider.ns2=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

dfs.namenode.shared.edits.dir.ns1=qjournal://ns1jn1:8485;ns1jn2:8485;ns1jn3:8485/ns1
dfs.namenode.shared.edits.dir.ns2=qjournal://ns2jn1:8485;ns2jn2:8485;ns2jn3:8485/ns2
===

Regards,
- Takanobu

on 2018/10/10 3:07, "rahul patwari" wrote:

Hi,


Is it possible in the current architecture of HDFS to have multiple 
nameservices(federation), with Active and Backup Namenode(HA) for each of the 
nameservices?


Thanks,
Rahul

Re: HDFS Federation with HA

2018-10-10 Thread Takanobu Asanuma 
Hi Rahul,

Actually, you can use the same journal nodes for other nameservices if you 
specify the different directories.

===
dfs.namenode.shared.edits.dir.ns1=qjournal://ns1jn1:8485;ns1jn2:8485;ns1jn3:8485/ns1
dfs.namenode.shared.edits.dir.ns2=qjournal://ns1jn1:8485;ns1jn2:8485;ns1jn3:8485/ns2
===

Regards,
- Takanobu

on 2018/10/10 16:56, "rahul patwari" wrote:

Hi Takanobu,


Thanks for the reply. So, does that mean that 3 dedicated journal nodes are 
required for each nameservice? or can we have the same journal nodes for all 
the nameservices?


Thanks,
Rahul 


On Wed, Oct 10, 2018 at 1:00 PM Takanobu Asanuma  
wrote:


Hi Rahul,

Yes, it is possible. Here are the sample configurations.

===
dfs.nameservices=ns1,ns2
dfs.ha.namenodes.ns1=nn1,nn2
dfs.ha.namenodes.ns2=nn1,nn2

dfs.namenode.rpc-address.ns1.nn1=ns1nn1:8020
dfs.namenode.rpc-address.ns1.nn2=ns1nn2:8020
dfs.namenode.http-address.ns1.nn1=ns1nn1:9870
dfs.namenode.http-address.ns1.nn2=ns1nn2:9870

dfs.namenode.rpc-address.ns2.nn1=ns2nn1:8020
dfs.namenode.rpc-address.ns2.nn2=ns2nn2:8020
dfs.namenode.http-address.ns2.nn1=ns2nn1:9870
dfs.namenode.http-address.ns2.nn2=ns2nn2:9870


dfs.client.failover.proxy.provider.ns1=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

dfs.client.failover.proxy.provider.ns2=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider


dfs.namenode.shared.edits.dir.ns1=qjournal://ns1jn1:8485;ns1jn2:8485;ns1jn3:8485/ns1

dfs.namenode.shared.edits.dir.ns2=qjournal://ns2jn1:8485;ns2jn2:8485;ns2jn3:8485/ns2
===

Regards,
- Takanobu

on 2018/10/10 3:07, "rahul patwari" wrote:

Hi,


Is it possible in the current architecture of HDFS to have multiple 
nameservices(federation), with Active and Backup Namenode(HA) for each of the 
nameservices?


Thanks,
Rahul

Re: HDFS Federation with HA

2018-10-10 Thread rahul patwari 
Hi Takanobu,

Thanks for the reply. So, does that mean that 3 dedicated journal nodes are
required for each nameservice? or can we have the same journal nodes for
all the nameservices?

Thanks,
Rahul

On Wed, Oct 10, 2018 at 1:00 PM Takanobu Asanuma 
wrote:

> Hi Rahul,
>
> Yes, it is possible. Here are the sample configurations.
>
> ===
> dfs.nameservices=ns1,ns2
> dfs.ha.namenodes.ns1=nn1,nn2
> dfs.ha.namenodes.ns2=nn1,nn2
>
> dfs.namenode.rpc-address.ns1.nn1=ns1nn1:8020
> dfs.namenode.rpc-address.ns1.nn2=ns1nn2:8020
> dfs.namenode.http-address.ns1.nn1=ns1nn1:9870
> dfs.namenode.http-address.ns1.nn2=ns1nn2:9870
>
> dfs.namenode.rpc-address.ns2.nn1=ns2nn1:8020
> dfs.namenode.rpc-address.ns2.nn2=ns2nn2:8020
> dfs.namenode.http-address.ns2.nn1=ns2nn1:9870
> dfs.namenode.http-address.ns2.nn2=ns2nn2:9870
>
>
> dfs.client.failover.proxy.provider.ns1=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
>
> dfs.client.failover.proxy.provider.ns2=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
>
>
> dfs.namenode.shared.edits.dir.ns1=qjournal://ns1jn1:8485;ns1jn2:8485;ns1jn3:8485/ns1
>
> dfs.namenode.shared.edits.dir.ns2=qjournal://ns2jn1:8485;ns2jn2:8485;ns2jn3:8485/ns2
> ===
>
> Regards,
> - Takanobu
>
> on 2018/10/10 3:07, "rahul patwari" wrote:
>
> Hi,
>
>
> Is it possible in the current architecture of HDFS to have multiple
> nameservices(federation), with Active and Backup Namenode(HA) for each of
> the nameservices?
>
>
> Thanks,
> Rahul
>
>
>
>

Re: HDFS Federation with HA

2018-10-10 Thread Takanobu Asanuma 
Hi Rahul,

Yes, it is possible. Here are the sample configurations.

===
dfs.nameservices=ns1,ns2
dfs.ha.namenodes.ns1=nn1,nn2
dfs.ha.namenodes.ns2=nn1,nn2

dfs.namenode.rpc-address.ns1.nn1=ns1nn1:8020
dfs.namenode.rpc-address.ns1.nn2=ns1nn2:8020
dfs.namenode.http-address.ns1.nn1=ns1nn1:9870
dfs.namenode.http-address.ns1.nn2=ns1nn2:9870

dfs.namenode.rpc-address.ns2.nn1=ns2nn1:8020
dfs.namenode.rpc-address.ns2.nn2=ns2nn2:8020
dfs.namenode.http-address.ns2.nn1=ns2nn1:9870
dfs.namenode.http-address.ns2.nn2=ns2nn2:9870

dfs.client.failover.proxy.provider.ns1=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider
dfs.client.failover.proxy.provider.ns2=org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider

dfs.namenode.shared.edits.dir.ns1=qjournal://ns1jn1:8485;ns1jn2:8485;ns1jn3:8485/ns1
dfs.namenode.shared.edits.dir.ns2=qjournal://ns2jn1:8485;ns2jn2:8485;ns2jn3:8485/ns2
===

Regards,
- Takanobu

on 2018/10/10 3:07, "rahul patwari" wrote:

Hi,


Is it possible in the current architecture of HDFS to have multiple 
nameservices(federation), with Active and Backup Namenode(HA) for each of the 
nameservices?


Thanks,
Rahul

HDFS Federation with HA

2018-10-09 Thread rahul patwari 
Hi,

Is it possible in the current architecture of HDFS to have multiple
nameservices(federation), with Active and Backup Namenode(HA) for each of
the nameservices?

Thanks,
Rahul