Recently, I discovered a situation while using hadoop.
When I use C API to send a request to the HDFS federation mode router node RPC
port, such as writing a file, after the client sends the completed request, the
Hadoop side needs a 20-minute delay before the file has a byte size, and the
file
Hello.I have a version 3.3.4 Hadoop cluster with over a thousand datanodes. I want to use HDFS Router based Federation. May I ask you a few questions.The first question is, is it better to deploy the router on the namenode server or separately? I want to add a set of Namenodes to the existing HDFS
>
> From: rahul patwari [mailto:rahulpatwari8...@gmail.com]
> Sent: Wednesday, October 10, 2018 3:56 AM
> To: tasan...@yahoo-corp.jp
> Cc: user@hadoop.apache.org
> Subject: Re: HDFS Federation with HA
>
> Hi Takanobu,
>
> Thanks for the reply. So, does tha
ting | Staffing Solutions |
-
From: rahul patwari [mailto:rahulpatwari8...@gmail.com]
Sent: Wednesday, October 10, 2018 3:56 AM
To: tasan...@yahoo-corp.jp
Cc: user@hadoop.apache.org
Subject: Re: HDFS Federation with HA
Hi Takanobu,
Th
Hi Rahul,
Actually, you can use the same journal nodes for other nameservices if you
specify the different directories.
===
dfs.namenode.shared.edits.dir.ns1=qjournal://ns1jn1:8485;ns1jn2:8485;ns1jn3:8485/ns1
dfs.namenode.shared.edits.dir.ns2=qjournal://ns
Hi Takanobu,
Thanks for the reply. So, does that mean that 3 dedicated journal nodes are
required for each nameservice? or can we have the same journal nodes for
all the nameservices?
Thanks,
Rahul
On Wed, Oct 10, 2018 at 1:00 PM Takanobu Asanuma
wrote:
> Hi Rahul,
>
> Yes, it is possible. Her
Hi Rahul,
Yes, it is possible. Here are the sample configurations.
===
dfs.nameservices=ns1,ns2
dfs.ha.namenodes.ns1=nn1,nn2
dfs.ha.namenodes.ns2=nn1,nn2
dfs.namenode.rpc-address.ns1.nn1=ns1nn1:8020
dfs.namenode.rpc-address.ns1.nn2=ns1nn2:8020
dfs.namenode
Hi,
Is it possible in the current architecture of HDFS to have multiple
nameservices(federation), with Active and Backup Namenode(HA) for each of
the nameservices?
Thanks,
Rahul
On Sun, Jun 5, 2016 at 1:10 PM, Kun Ren wrote:
>
>> Hi Genius,
>>
>> I just configured HDFS Federation, and try to use it(2 namenodes, one is
>> for /my, another is for /your). When I run the command:
>> hdfs dfs -ls /,
>>
>> I can get:
>> -
Perhaps use the "viewfs://" protocol prepended to your path?
On Sun, Jun 5, 2016 at 1:10 PM, Kun Ren wrote:
> Hi Genius,
>
> I just configured HDFS Federation, and try to use it(2 namenodes, one is
> for /my, another is for /your). When I run the command:
> hdfs dfs -
Hi Genius,
I just configured HDFS Federation, and try to use it(2 namenodes, one is
for /my, another is for /your). When I run the command:
hdfs dfs -ls /,
I can get:
-r-xr-xr-x - hadoop hadoop 0 2016-06-05 20:05 /my
-r-xr-xr-x - hadoop hadoop 0 2016-06-05 20:05 /your
ses
> different file systems, then typically the rename either degrades to a
> non-atomic copy-delete or the call simply fails fast.
>
> --Chris Nauroth
>
> From: Kun Ren
> Date: Wednesday, May 25, 2016 at 1:58 PM
>
> To: Chris Nauroth
> Cc: "user@hadoop.apache.org
mail.com>>
Date: Wednesday, May 25, 2016 at 1:58 PM
To: Chris Nauroth mailto:cnaur...@hortonworks.com>>
Cc: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>"
mailto:user@hadoop.apache.org>>
Subject: Re: HDFS Federation
Yes, it is rename. Thanks for the expl
3 AM
> To: Chris Nauroth
> Cc: "user@hadoop.apache.org"
> Subject: Re: HDFS Federation
>
> Thanks a lot, Chris.
>
> I remembered that Federation doesn't support some cross-namenodes
> operations, if so, which operations that Federation doesn't support an
>>
Date: Wednesday, May 25, 2016 at 11:03 AM
To: Chris Nauroth mailto:cnaur...@hortonworks.com>>
Cc: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>"
mailto:user@hadoop.apache.org>>
Subject: Re: HDFS Federation
Thanks a lot, Chris.
I remembered that Fed
rks with federation. The command would copy the file
> from NameNode 1/block pool 1 to NameNode 2/block pool 2.
>
> --Chris Nauroth
>
> From: Kun Ren
> Date: Wednesday, May 25, 2016 at 8:57 AM
> To: "user@hadoop.apache.org"
> Subject: HDFS Federation
>
> Hi G
lt;mailto:user@hadoop.apache.org>"
mailto:user@hadoop.apache.org>>
Subject: HDFS Federation
Hi Genius,
Does HDFS Federation support the cross namenodes operations?
For example:
./bin/hdfs dfs -cp input1/a.xml input2/b.xml
Supposed that input1 belongs namenode 1, and input 2 belongs namenode
Hi Genius,
Does HDFS Federation support the cross namenodes operations?
For example:
./bin/hdfs dfs -cp input1/a.xml input2/b.xml
Supposed that input1 belongs namenode 1, and input 2 belongs namenode 2,
does Federation support this operation? And if not, why?
Thanks.
node by setting different HTTP/RPC port and fsimage/edits directory
>> for each NameNode. I haven't tried this, so perhaps it is not
>> possible.
>>
>> Regards,
>> Akira
>>
>> On 4/28/16 09:30, Kun Ren wrote:
>>
>> Hi G
C port and fsimage/edits directory
for each NameNode. I haven't tried this, so perhaps it is not possible.
Regards,
Akira
On 4/28/16 09:30, Kun Ren wrote:
Hi Genius,
I have two questions about the HDFS Federation:
(1) Since there are multiple nam
in one node by
> setting different HTTP/RPC port and fsimage/edits directory for each
> NameNode. I haven't tried this, so perhaps it is not possible.
>
> Regards,
> Akira
>
> On 4/28/16 09:30, Kun Ren wrote:
>
>> Hi Genius,
>>
>> I have two questions about
/edits directory for each
NameNode. I haven't tried this, so perhaps it is not possible.
Regards,
Akira
On 4/28/16 09:30, Kun Ren wrote:
Hi Genius,
I have two questions about the HDFS Federation:
(1) Since there are multiple namenodes, there should be some code that
analysis the client re
Hi Genius,
I have two questions about the HDFS Federation:
(1) Since there are multiple namenodes, there should be some code that
analysis the client request and transfer the request to the appropriate
namenode, could you please point to me where I can find the related code?
(2) .Also just
who is this?
On Sun, Aug 23, 2015 at 8:54 PM, Todd wrote:
> Hi, our hadoop cluster is using HDFS Federation, but when use the
> following command to report the HDFS status
> It gives me the following message that viewfs is NOT HDFS filesystem
> Then how can I proceed to report the
com>]
*Sent:* Monday, 24 August, 2015 11:54 AM
*To:* user@hadoop.apache.org <mailto:user@hadoop.apache.org>
*Subject:* Unable to use ./hdfs dfsadmin -report with HDFS Federation
Hi, our hadoop cluster is using HDFS Federation, but when use the
following command to repo
tula
From: Todd [bit1...@163.com]
Sent: Monday, August 24, 2015 9:33 AM
To: user@hadoop.apache.org
Subject: Re:RE: Unable to use ./hdfs dfsadmin -report with HDFS Federation
Thanks Chinnappan for the quick reply!
Looks that the command hdfs dfsadmin -report should work
che.org
Subject: Unable to use ./hdfs dfsadmin -report with HDFS Federation
Hi, our hadoop cluster is using HDFS Federation, but when use the following
command to report the HDFS status
It gives me the following message that viewfs is NOT HDFS filesystem
Then how can I proceed to report the hdf
hdfs dfsadmin -report
The above command will show entire cluster status
From: Todd [mailto:bit1...@163.com]
Sent: Monday, 24 August, 2015 11:54 AM
To: user@hadoop.apache.org
Subject: Unable to use ./hdfs dfsadmin -report with HDFS Federation
Hi, our hadoop cluster is using HDFS Federation, but
Hi, our hadoop cluster is using HDFS Federation, but when use the following
command to report the HDFS status
It gives me the following message that viewfs is NOT HDFS filesystem
Then how can I proceed to report the hdfs status
Thanks!
[admin@namenode1]$ ./hdfs dfsadmin -report
report
Hi seok srry was unable to answer was your problem solved It was a long
time back right :(
On Wed, Sep 11, 2013 at 3:07 PM, Oh Seok Keun wrote:
> Hello~ I am Rho working in korea
>
> I am trying to install HDFS Federation( with 2.1.0 beta version ) and to
> test
> After 2.1.0 (
Hello,
According to CDH documentation
(http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM4Ent/4.5.3/Cloudera-Manager-Enterprise-Edition-User-Guide/cmeeug_topic_5_13.html),
"/" can NOT be used as mount point. Is there some explicit reason for this?
Because of this limitation, if
this is a short
duration lock and hence will make from the multiple clients perspective,
the creation of files as simultaneous.
If you are talking about a single client, with a single thread, then it
would be sequential.
Hope that makes sense.
>
> I need this answer to estimat
ent directory) run in sequential because WRITE-once
> control govened by single NM?
>
> I need this answer to estimate the necessity of moving to HDFS federation.
>
> Thanks
>
> --
> --Anfernee
>
>
>
--
--Anfernee
govened
by single NM?
I need this answer to estimate the necessity of moving to HDFS federation.
Thanks
--
--Anfernee
ntrol govened by single NM?
I need this answer to estimate the necessity of moving to HDFS federation.
Thanks
--
--Anfernee
Thanks Suresh.
I followed the link it's clear now.
But client side configuration is not covered on the doc.
Thanks & Regards,
B Anil Kumar.
On Thu, Jan 23, 2014 at 11:44 PM, Suresh Srinivas wrote:
> Have you looked at -
> http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-hdfs/Fe
Have you looked at -
http://hadoop.apache.org/docs/r2.2.0/hadoop-project-dist/hadoop-hdfs/Federation.html
?
On Thu, Jan 23, 2014 at 9:35 AM, AnilKumar B wrote:
> Hi,
>
> We tried setting up HDFS name node federation set up with 2 name nodes. I
> am facing few issues.
>
> Can any one help me in
Hi,
We tried setting up HDFS name node federation set up with 2 name nodes. I
am facing few issues.
Can any one help me in understanding below points?
1) how can we configure different namespaces to different name node? Where
exactly we need to configure this?
2) After formatting each NN with o
Hi Manickam,
Can you explain what you're really trying to achieve/construct here?
YARN has little to do with Federation. One typically runs federation
to divide the namespace across multiple NameNodes (lowering load
compared to a single NN). YARN and MR though would run across the
whole cluster i
Guys,
I have installed federated cluster with two name nodes and 3 data nodes. Now i
want to have separate RM for yarn. Can i install like that? Then will my name
node
work like NM? How it will work?
Can i install hive in any one of the name node in federated cluster?
Pls help me to understan
>
> I'm not able to follow the page completely.
> Can you pls help me to get some clear step by step or little bit more
> details in the configuration side?
>
Have you setup a non-federated cluster before. If you have, the page should
be easy to follow. If you have not setup a non-federated cluste
, 23 Sep 2013 10:02:42 -0700
Subject: Re: HDFS federation Configuration
From: sur...@hortonworks.com
To: user@hadoop.apache.org
I'm not able to follow the page completely.
Can you pls help me to get some clear step by step or little bit more details
in the configuration side?
Have you se
Hi,
Thanks. I have done that.
Thanks,
Manickam P
Date: Mon, 23 Sep 2013 10:02:42 -0700
Subject: Re: HDFS federation Configuration
From: sur...@hortonworks.com
To: user@hadoop.apache.org
I'm not able to follow the page completely.
Can you pls help me to get some clear step by step or l
data nodes.
Thanks,
Manickam P
Date: Thu, 19 Sep 2013 12:28:04 -0700
Subject: Re: HDFS federation Configuration
From: sur...@hortonworks.com
To: user@hadoop.apache.org
Have you looked at -
http://hadoop.apache.org/docs/r2.1.0-beta/hadoop-project-dist/hadoop-hdfs/Federation.html
Let me know i
Hi,
Except "hadoop.tmp.dir" I am not defined any thing in core-site.xmlCan you
please let me what exactly I should include in core-site.xml
Thanks,Sandeep.
Date: Sat, 21 Sep 2013 23:46:47 +0530
Subject: Re: Can you help me to install HDFS Federation and test?
From: visioner.sa...@gm
ut is "ls: test : No
> such file or directory"
>
>
> Thanks,
> Sandeep.
>
>
> --
> Date: Wed, 18 Sep 2013 16:58:50 +0530
>
> Subject: Re: Can you help me to install HDFS Federation and test?
> From: visioner.sa...@gmail.com
> T
Guys,
I need some tutorials to configure fedration. Can you pls suggest me some?
Thanks,
Manickam P
Have you looked at -
http://hadoop.apache.org/docs/r2.1.0-beta/hadoop-project-dist/hadoop-hdfs/Federation.html
Let me know if the document is not clear or needs improvements.
Regards,
Suresh
On Thu, Sep 19, 2013 at 12:01 PM, Manickam P wrote:
> Guys,
>
> I need some tutorials to configure f
is "ls: test : No such file or
directory"
Thanks,Sandeep.
Date: Wed, 18 Sep 2013 16:58:50 +0530
Subject: Re: Can you help me to install HDFS Federation and test?
From: visioner.sa...@gmail.com
To: user@hadoop.apache.org
It shud be visible from every namenode machine have you tried thi
--
> Date: Tue, 17 Sep 2013 17:56:00 +0530
>
> Subject: Re: Can you help me to install HDFS Federation and test?
> From: visioner.sa...@gmail.com
> To: user@hadoop.apache.org
>
> 1.> make sure to check hadoop logs once u start u r datanode at
>
their own FS configured.
(There may be some caveats that apply with use of JHS and where the MR
staging dir goes, but it can be fixed via consistent configs)
On Tue, Sep 17, 2013 at 8:33 PM, Manickam P wrote:
> Hello Experts,
>
> I know that hdfs federation helps adding multiple name
Guys,
I am a bit confused with the package availability and compatibility of the
upgraded Hadoop components - HDFS Federation and YARN (MRv2)
My questions areWhich download contains HDFS federation?Any good documentation
to get daemons running on HDFS federation?Is Yarn compatible with older
explain with bit detailed.
Thanks,Sandeep.
Date: Tue, 17 Sep 2013 17:56:00 +0530
Subject: Re: Can you help me to install HDFS Federation and test?
From: visioner.sa...@gmail.com
To: user@hadoop.apache.org
1.> make sure to check hadoop logs once u start u r datanode at
/home/hadoop/hadoop-vers
Hello Experts,
I know that hdfs federation helps adding multiple name nodes. In this case how
this Yarn works? Where will be the Resource Manger if i have multiple name
nodes? (Normally in Master Node I guess) Can we execute yarn applications in
hadoop cluster without having this hdfs
:44 PM, Sandeep L wrote:
> Hi,
>
> I tried to install HDFS federation with the help of document given by you.
>
> I have small issue.
> I used 2 slave in setup, both will act as namenode and datanode.
> Now the issue is when I am looking at home pages of both namenodes only
>
Hi,
I tried to install HDFS federation with the help of document given by you.
I have small issue.I used 2 slave in setup, both will act as namenode and
datanode.Now the issue is when I am looking at home pages of both namenodes
only one datanode is appearing.As per my understanding 2 datanodes
Reading the HDFS Federation documentation, it seems that it provides
support for multiple NameNodes in a single cluster. The DataNodes are
shared across all NameNodes. It seems one can take a single hadoop
cluster, and add HDFS Federation, but I do not see any way to take multiple
hadoop
Hello~ I am Rho working in korea
I am trying to install HDFS Federation( with 2.1.0 beta version ) and to
test
After 2.1.0 ( for federation test ) install I have a big trouble when file
putting test
I command to hadoop
Can you help me to install HDFS Federation and test?
./bin/hadoop fs -put
7; ZKFCs.
>>
>> Do post back if there are still some more doubts.
>>
>> On Mon, Apr 8, 2013 at 10:53 AM, Azuryy Yu wrote:
>> > Hi dears,
>> > I deployed Hadoopv2 with HA enabled using QJM, so my question is:
>> >
>> > 1) if we also c
, and
> won't collide with another NSs' ZKFCs.
>
> Do post back if there are still some more doubts.
>
> On Mon, Apr 8, 2013 at 10:53 AM, Azuryy Yu wrote:
> > Hi dears,
> > I deployed Hadoopv2 with HA enabled using QJM, so my question is:
> >
> >
arent znode name on the ZK instance, and
won't collide with another NSs' ZKFCs.
Do post back if there are still some more doubts.
On Mon, Apr 8, 2013 at 10:53 AM, Azuryy Yu wrote:
> Hi dears,
> I deployed Hadoopv2 with HA enabled using QJM, so my question is:
>
> 1) if we also
Hi dears,
I deployed Hadoopv2 with HA enabled using QJM, so my question is:
1) if we also configured HDFS federation, such as:
NN1 is active, NN2 is standby
NN3 is active, NN4 is standby
they are configured as HDFS federation, then,
Can these four NNs using the same Journal nodes and
'm testing the HDFS Federation.I have 2 namenodes in my cluster.
>>>> I find that it is Rolling Edit Log continuously on one
>>>> namenode,however the other one changes nothing.
>>>>
>>>> My question :
>>>> 1. Is it the right situation.
>>>> 2. I have thought that the two namenodes shoud keep concurrency.
>>>> why they get differences.
>>>> 3. is there any advanced docs about HDFS Federation.
>>>>
>>>> thanks.
>>>>
>>>> Regards.
>>>>
>>>>
>>>>
>>>
>>>
>>
>
--
Harsh J
have 2 namenodes in my cluster.
>>> I find that it is Rolling Edit Log continuously on one
>>> namenode,however the other one changes nothing.
>>>
>>> My question :
>>> 1. Is it the right situation.
>>> 2. I have thought that the two namenodes shoud keep concurrency.
>>> why they get differences.
>>> 3. is there any advanced docs about HDFS Federation.
>>>
>>> thanks.
>>>
>>> Regards.
>>>
>>>
>>>
>>>
>>
>>
>
> I find that it is Rolling Edit Log continuously on one
>> namenode,however the other one changes nothing.
>>
>> My question :
>> 1. Is it the right situation.
>> 2. I have thought that the two namenodes shoud keep concurrency.
>> why they get differences.
>> 3. is there any advanced docs about HDFS Federation.
>>
>> thanks.
>>
>> Regards.
>>
>>
>>
>>
>
>
amenodes shoud keep concurrency.
> why they get differences.
> 3. is there any advanced docs about HDFS Federation.
>
> thanks.
>
> Regards.
>
>
>
>
rote:
> >>> Hi All
> >>>
> >>> I'm testing the HDFS Federation.I have 2 namenodes in my cluster.
> >>> I find that it is Rolling Edit Log continuously on one
> namenode,however
> >>> the other one changes nothing.
> >>>
> >>> My question :
> >>> 1. Is it the right situation.
> >>> 2. I have thought that the two namenodes shoud keep concurrency.
> >>> why they get differences.
> >>> 3. is there any advanced docs about HDFS Federation.
> >>>
> >>> thanks.
> >>>
> >>> Regards.
> >>
> >>
> >>
> >> --
> >> Harsh J
> >>
>
>
>
> --
> Harsh J
>
e HDFS Federation.I have 2 namenodes in my cluster.
>>> I find that it is Rolling Edit Log continuously on one namenode,however
>>> the other one changes nothing.
>>>
>>> My question :
>>> 1. Is it the right situation.
>>> 2. I have thought that the two namenodes shoud keep concurrency.
>>> why they get differences.
>>> 3. is there any advanced docs about HDFS Federation.
>>>
>>> thanks.
>>>
>>> Regards.
>>
>>
>>
>> --
>> Harsh J
>>
--
Harsh J
e HDFS Federation.I have 2 namenodes in my cluster.
>> I find that it is Rolling Edit Log continuously on one namenode,however
>> the other one changes nothing.
>>
>> My question :
>> 1. Is it the right situation.
>> 2. I have thought that the two namenode
> why they get differences.
> 3. is there any advanced docs about HDFS Federation.
>
> thanks.
>
> Regards.
>
>
>
--
Harsh J
or procedure for configuring HDFS FEDERATION
0.23.3 in a cluster
any hints friends...
On Fri, Oct 19, 2012 at 12:54 PM, Visioner Sadak
wrote:
Is there any simple links tutorials for installing HDFS federation 0.23.3 in a
cluster set up
any hints friends...
On Fri, Oct 19, 2012 at 12:54 PM, Visioner Sadak
wrote:
> Is there any simple links tutorials for installing HDFS federation 0.23.3
> in a cluster set up
>
Is there any simple links tutorials for installing HDFS federation 0.23.3
in a cluster set up
got tht Anil Thanks a ton friends for your help..
On Wed, Oct 17, 2012 at 8:35 PM, Anil Gupta wrote:
> Hi visioner,
> It won't work on pseudo distributed because you need to run at least 2 NN
> and if you run 2NN then u need to configure them separately on different
> ports. It's little tr
Hi visioner,
It won't work on pseudo distributed because you need to run at least 2 NN and
if you run 2NN then u need to configure them separately on different ports.
It's little tricky doing on one node. Multiple VM is the cleanest solution.
Best Regards,
Anil
On Oct 17, 2012, at 12:04 AM, Vis
tht means i will need a cluster set up only right pseudo distr mode wont
wrk
On Wed, Oct 17, 2012 at 5:15 AM, lohit wrote:
> You can try out federation by create 3 different conf directories and
> starting 3 different NameNodes out of those configurations. These
> configurations should make
You can try out federation by create 3 different conf directories and
starting 3 different NameNodes out of those configurations. These
configurations should make sure they have different directories and port
numbers. If you want to just give it a try, it is easier to spawn 3 VMs and
use them as se
I have a single linux node in which i installed 0.23.3 in pseudo-mode can i
test federation configuration and functionality using this or will i have
to install hadoop in a cluster with atleast 3 linux nodes ..incase of
my single linux box i have 3 ips with me.
78 matches
Mail list logo
