Re: HDFS Federation

[Chris Nauroth]

You're correct that multi-phase commit protocols seek to address this class of 
problem.  However, there is currently no such multi-phase commit protocol 
implemented within HDFS, and I'd be reluctant to introduce it at the scale of 
HDFS.  It definitely would be a very different operational model for HDFS 
compared to current state.

The situation with rename across NameNodes is somewhat analogous to renames 
across different volumes/different mount points in typical local file systems.  
It's typical for the OS to guarantee atomicity of rename, as long as the rename 
is performed within the same file system.  If multiple file systems on 
different volumes are mounted, and the rename crosses different file systems, 
then typically the rename either degrades to a non-atomic copy-delete or the 
call simply fails fast.

--Chris Nauroth

From: Kun Ren mailto:ren.h...@gmail.com>>
Date: Wednesday, May 25, 2016 at 1:58 PM
To: Chris Nauroth mailto:cnaur...@hortonworks.com>>
Cc: "user@hadoop.apache.org" 
mailto:user@hadoop.apache.org>>
Subject: Re: HDFS Federation

Yes, it is rename. Thanks for the explanation, make sense.

The rename example you gave(across namenodes) is like a distributed 
transaction,  standard protocol like two-phase commit can be used to support 
distributed transactions in Database world, so is that possible to support the 
"atomic distributed rename" using something like two-phase commit protocol? Or 
even it is possible, the implementation should be very difficult or the 
performance is bad?

On Wed, May 25, 2016 at 4:26 PM, Chris Nauroth 
mailto:cnaur...@hortonworks.com>> wrote:
You might be thinking of renames.  For example, when using ViewFs with 
Federation, the client-side mount configuration might have path /data1 backed 
by NameNode1 and path /data2 backed by NameNode2.  Renaming /data1/file1 to 
/data2/file1 would not be supported, because it would need to be copied across 
NameNodes, and then HDFS would not be able to satisfy its promise that rename 
is atomic.  There are more details about this in the ViewFs guide.

http://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/ViewFs.html

--Chris Nauroth

From: Kun Ren mailto:ren.h...@gmail.com>>
Date: Wednesday, May 25, 2016 at 11:03 AM
To: Chris Nauroth mailto:cnaur...@hortonworks.com>>
Cc: "user@hadoop.apache.org" 
mailto:user@hadoop.apache.org>>
Subject: Re: HDFS Federation

Thanks a lot, Chris.

I remembered that Federation doesn't support some cross-namenodes operations, 
if so,  which operations that Federation doesn't support and why? Thanks again.

On Wed, May 25, 2016 at 12:17 PM, Chris Nauroth 
mailto:cnaur...@hortonworks.com>> wrote:
Hello Kun,

Yes, this command works with federation.  The command would copy the file from 
NameNode 1/block pool 1 to NameNode 2/block pool 2.

--Chris Nauroth

From: Kun Ren mailto:ren.h...@gmail.com>>
Date: Wednesday, May 25, 2016 at 8:57 AM
To: "user@hadoop.apache.org" 
mailto:user@hadoop.apache.org>>
Subject: HDFS Federation

Hi Genius,

Does HDFS Federation support the cross namenodes operations?

For example:

./bin/hdfs dfs -cp input1/a.xml input2/b.xml

Supposed that input1 belongs namenode 1, and input 2 belongs namenode 2, does 
Federation support this operation? And if not, why?

Thanks.

Re: YARN cluster underutilization

[Shubh hadoopExp]

Hey,

OFFSWITCH allocation means if the data locality is maintained or not. It has no 
relation with heartbeat! Heartbeat is just used to clear the pipelining of 
Container request.

-Shubh


> On May 25, 2016, at 3:30 PM, Guttadauro, Jeff  
> wrote:
> 
> Interesting stuff!  I did not know about this handling of OFFSWITCH requests. 
>  
> To get around this, would you recommend reducing the heartbeat interval, 
> perhaps to 250ms to get a 4x improvement in container allocation rate (or is 
> it not quite as simple as that)?  Maybe doing this in combination with using 
> a greater number of smaller nodes would help?  Would overloading the 
> ResourceManager be a concern if doing that?  Should I bump up the 
> “YARN_RESOURCEMANAGER_HEAPSIZE” configuration property (current default for 
> m3.xlarge is 2396M), or would you suggest any other knobs to turn to help RM 
> handle it?
>  
> Thanks again for all your help, Sunil!
>  
> From: Sunil Govind [mailto:sunil.gov...@gmail.com] 
> Sent: Wednesday, May 25, 2016 1:07 PM
> To: Guttadauro, Jeff ; user@hadoop.apache.org
> Subject: Re: YARN cluster underutilization
>  
> Hi Jeff,
>  
>  I do see the yarn.resourcemanager.nodemanagers.heartbeat-interval-ms 
> property set to 1000 in the job configuration
> >> Ok, This make sense.. node heartbeat seems default.
>  
> If there are no locality specified in resource requests (using 
> ResourceRequest.ANY) , then YARN will allocate only one container per node 
> heartbeat. So your container allocation rate is slower considering 600k 
> requests and only 20 nodes. And if more number of containers are also getting 
> released fast (I could see that some containers lifetime is 80 to 90 secs), 
> then this will become more complex and container allocation rate will be 
> slower.
>  
> YARN-4963  is trying to make 
> more allocation per heartbeat for NODE_OFFSWITCH (ANY) requests. But its not 
> yet available in any release.
>  
> I guess you can investigate more in this line to confirm this points. 
>  
> Thanks
> Sunil
>  
>  
> On Wed, May 25, 2016 at 11:00 PM Guttadauro, Jeff  > wrote:
> Thanks for digging into the log, Sunil, and making some interesting 
> observations!
>  
> The heartbeat interval hasn’t been changed from its default, and I do see the 
> yarn.resourcemanager.nodemanagers.heartbeat-interval-ms property set to 1000 
> in the job configuration.  I was searching in the log for heartbeat interval 
> information, but I didn’t find anything.  Where do you look in the log for 
> the heartbeats?
>  
> Also, you are correct about there being no data locality, as all the input 
> data is in S3.  The utilization has been fluctuating, but I can’t really see 
> a pattern or tell why.  It actually started out pretty low in the 20-30% 
> range and then managed to get up into the 50-70% range after a while, but 
> that was short-lived, as it went back down into the 20-30% range for quite a 
> while.  While writing this, I saw it surprisingly hit 80%!!  First time I’ve 
> seen it that high in the 20 hours it’s been running…  Although looks like it 
> may be headed back down.  I’m perplexed.  Wouldn’t you generally expect 
> fairly stable utilization over the course of the job?  (This is the only job 
> running.)
>  
> Thanks,
> -Jeff
>  
> From: Sunil Govind [mailto:sunil.gov...@gmail.com 
> ] 
> Sent: Wednesday, May 25, 2016 11:55 AM
> 
> To: Guttadauro, Jeff  >; user@hadoop.apache.org 
> 
> Subject: Re: YARN cluster underutilization
>  
> Hi Jeff.
>  
> Thanks for sharing this information. I have some observations from this logs.
>  
> - I think the node heartbeat is around 2/3 seconds here. Is it changed due to 
> some other reasons?
> - And all mappers Resource Request seems to be asking for type ANY (there is 
> no data locality). pls correct me if I am wrong.
>  
> If the resource request type is ANY, only one container will be allocated per 
> heartbeat for a node. Here node heartbeat delay is also more. And I can see 
> that containers are released very fast too. So when u started you 
> application, are you seeing more better resource utilization? And once 
> containers started to get released/completed, you are seeing under 
> utilization. 
>  
> Pls look into this line. It may be a reason.
>  
> Thanks
> Sunil
>  
> On Wed, May 25, 2016 at 9:59 PM Guttadauro, Jeff  > wrote:
> Thanks for your thoughts thus far, Sunil.  Most grateful for any additional 
> help you or others can offer.  To answer your questions,
>  
> 1.   This is a custom M/R job, which uses mappers only (no reduce phase) 
> to process GPS probe data and filter based on inclusion within a provided 
> polygon.  There is actually a lot of upfront work done in the driver to make 
> that task as simple as can be (identifies a list of tiles 

Re: YARN cluster underutilization

[Sunil Govind]

Hi Jeff,

I am not very sure about reducing heartbeat interval. This may put more
pressure on RM. We can wait for opinion from others in this point.

I think a good option is to add node locality in resource requests rather
than keeping it OFF_SWITCH. This could help.

Thanks
Sunil

On Thu, May 26, 2016 at 1:00 AM Guttadauro, Jeff 
wrote:

> Interesting stuff!  I did not know about this handling of OFFSWITCH
> requests.
>
>
>
> To get around this, would you recommend reducing the heartbeat interval,
> perhaps to 250ms to get a 4x improvement in container allocation rate (or
> is it not quite as simple as that)?  Maybe doing this in combination with
> using a greater number of smaller nodes would help?  Would overloading the
> ResourceManager be a concern if doing that?  Should I bump up the
> “YARN_RESOURCEMANAGER_HEAPSIZE” configuration property (current default for
> m3.xlarge is 2396M), or would you suggest any other knobs to turn to help
> RM handle it?
>
>
>
> Thanks again for all your help, Sunil!
>
>
>
> *From:* Sunil Govind [mailto:sunil.gov...@gmail.com]
> *Sent:* Wednesday, May 25, 2016 1:07 PM
>
>
> *To:* Guttadauro, Jeff ; user@hadoop.apache.org
> *Subject:* Re: YARN cluster underutilization
>
>
>
> Hi Jeff,
>
>
>
>  I do see the yarn.resourcemanager.nodemanagers.heartbeat-interval-ms
> property set to 1000 in the job configuration
>
> >> Ok, This make sense.. node heartbeat seems default.
>
>
>
> If there are no locality specified in resource requests (using
> ResourceRequest.ANY) , then YARN will allocate only one container per node
> heartbeat. So your container allocation rate is slower considering 600k
> requests and only 20 nodes. And if more number of containers are also
> getting released fast (I could see that some containers lifetime is 80 to
> 90 secs), then this will become more complex and container allocation rate
> will be slower.
>
>
>
> YARN-4963  is trying to
> make more allocation per heartbeat for NODE_OFFSWITCH (ANY) requests. But
> its not yet available in any release.
>
>
>
> I guess you can investigate more in this line to confirm this points.
>
>
>
> Thanks
>
> Sunil
>
>
>
>
>
> On Wed, May 25, 2016 at 11:00 PM Guttadauro, Jeff <
> jeff.guttada...@here.com> wrote:
>
> Thanks for digging into the log, Sunil, and making some interesting
> observations!
>
>
>
> The heartbeat interval hasn’t been changed from its default, and I do see
> the yarn.resourcemanager.nodemanagers.heartbeat-interval-ms property set to
> 1000 in the job configuration.  I was searching in the log for heartbeat
> interval information, but I didn’t find anything.  Where do you look in the
> log for the heartbeats?
>
>
>
> Also, you are correct about there being no data locality, as all the input
> data is in S3.  The utilization has been fluctuating, but I can’t really
> see a pattern or tell why.  It actually started out pretty low in the
> 20-30% range and then managed to get up into the 50-70% range after a
> while, but that was short-lived, as it went back down into the 20-30% range
> for quite a while.  While writing this, I saw it surprisingly hit 80%!!
> First time I’ve seen it that high in the 20 hours it’s been running…
>  Although looks like it may be headed back down.  I’m perplexed.  Wouldn’t
> you generally expect fairly stable utilization over the course of the job?
> (This is the only job running.)
>
>
>
> Thanks,
>
> -Jeff
>
>
>
> *From:* Sunil Govind [mailto:sunil.gov...@gmail.com]
> *Sent:* Wednesday, May 25, 2016 11:55 AM
>
>
> *To:* Guttadauro, Jeff ; user@hadoop.apache.org
> *Subject:* Re: YARN cluster underutilization
>
>
>
> Hi Jeff.
>
>
>
> Thanks for sharing this information. I have some observations from this
> logs.
>
>
>
> - I think the node heartbeat is around 2/3 seconds here. Is it changed due
> to some other reasons?
>
> - And all mappers Resource Request seems to be asking for type ANY (there
> is no data locality). pls correct me if I am wrong.
>
>
>
> If the resource request type is ANY, only one container will be allocated
> per heartbeat for a node. Here node heartbeat delay is also more. And I can
> see that containers are released very fast too. So when u started you
> application, are you seeing more better resource utilization? And once
> containers started to get released/completed, you are seeing under
> utilization.
>
>
>
> Pls look into this line. It may be a reason.
>
>
>
> Thanks
>
> Sunil
>
>
>
> On Wed, May 25, 2016 at 9:59 PM Guttadauro, Jeff 
> wrote:
>
> Thanks for your thoughts thus far, Sunil.  Most grateful for any
> additional help you or others can offer.  To answer your questions,
>
>
>
> 1.   This is a custom M/R job, which uses mappers only (no reduce
> phase) to process GPS probe data and filter based on inclusion within a
> provided polygon.  There is actually a lot of upfront work done in the
> driver to make that task as simple as can be (identifies a list of tiles
> that are completely

What is the recommended way to append to files on hdfs?

[Dmitry Goldenberg]

I'm having trouble figuring out a safe way to append to files in HDFS.

I'm using a small, 3-node Hadoop cluster (CDH v.5.3.9 to be specific). Our
process is a data pipeliner which is multi-threaded (8 threads) and it has
a stage which appends lines of delimited text to files in a dedicated
directory on HDFS. I'm using locks to synchronize access of the threads to
the buffered writers which append the data.

My first issue is deciding on the approach generally.

Approach A is to open the file, append to it, then close it for every line
appended. This seems slow and would seem to create too many small blocks,
or at least I see some such sentiment in various posts.

Approach B is to cache the writers but periodically refresh them to make
sure the list of writers doesn't grow unbounded (currently, it's one writer
per each input file processed by the pipeliner). This seems like a more
efficient approach but I imagine having open streams over a period of time
however controlled may be an issue, especially for output file readers (?)

Beyond this, my real issues are two. I am using the FileSystem Java Hadoop
API to do the appending and am intermittently getting these 2 types of
exceptions:

org.apache.hadoop.ipc.RemoteException: failed to create file
/output/acme_20160524_1.txt for DFSClient_NONMAPREDUCE_271210261_1 for
client XXX.XX.XXX.XX because current leaseholder is trying to recreate file.

org.apache.hadoop.ipc.RemoteException:
BP-182165-XXX.XX.XXX.XX-146307410:blk_1073760252_54540 does not
exist or is not under Constructionblk_1073760252_545
40{blockUCState=UNDER_RECOVERY, primaryNodeIndex=1,
replicas=[ReplicaUnderConstruction[[DISK]DS-ccdf4e55-234b-4e17-955f-daaed1afdd92:NORMAL|RBW],
ReplicaUnderConst
ruction[[DISK]DS-1f66db61-759f-4c5d-bb3b-f78c260e338f:NORMAL|RBW]]}

Anyone have any ideas on either of those?

For the first problem, I've tried instrumenting logic discussed in this post

but
didn't seem to help.

I'm also interested in the role of the dfs.support.append property, if at
all applicable.

The code is more or less as follows, for getting the output stream

userGroupInfo = UserGroupInformation.*createRemoteUser*("hdfs");

Configuration conf = new Configuration();

conf.set(key1, val1);



conf.set(keyN, valN);

fileSystem = userGroupInfo.doAs(*new* PrivilegedExceptionAction()
{

*public* FileSystem run() *throws* Exception {

*return* FileSystem.*get*(conf);

}

  });

*org.apache.hadoop.fs.path.Path file = ...*

*public* OutputStream getOutputStream(*boolean* append) *throws* IOException
{

OutputStream os = *null*;

*synchronized* (file) {

  // If the file exists

  *if* (isFile()) {

// See if we're to append or to overwrite

os = (append)

  ? fs.append(file) : fs.create(file, *true*);

  }

  // Appending to a non-existent file

  *else* *if* (append) {

// Create the file first

// otherwise, "failed to append to non-existent file" exception

FSDataOutputStream dos = fs.create(file);

dos.close();

// Open it for appending

os = fs.append(file);

  }

  // Creating a new file

  *else* {

os = fs.create(file);

  }

}

*return* os;

  }

Re: "Big Data" Problem

[Deepak Goel]

I totally agree with everyone and some final parting thoughts on everyone's
answers:

1. I do not know of any work laws in India (I am out of India) which deals
on ethics. Even if they are there, they must be in books.

2. I cannot personally afford to consult a lawyer on this. As for my
company, I am too small time a guy to suggest them anything on legal
matters. I can always voice my thought out but that's about it. Quitting
for ethical reasons could be one option, but then i would be just plainly
running away from the problem like an ostrich (Also my & my family survival
depends on this job)

3. About whether this is a appropriate question for this mailing list is
for everyone to decide. I had checked Hadoop's instructions on the mailing
list and this is what they had to say:

*The user@ mailing list is the preferred mailing list for end-user
questions and discussion.*
http://hadoop.apache.org/mailing_lists.html

So it does not really draws the line between what can be asked and what
cannot be asked. Anyways I agree it was my mistake to post in this forum
and I am sorry for that (It is just that I haven't eaten and slept for the
past couple of days because of this matter and one can hardly understand
'courtly and well-bred behaviour' when I am so pale)

4. I did read and had a discussion about what can be used as information in
corporate banking. But it is too vague, subjective. Does anyone has a link
on this? Also from personal experience, I have seen companies especially
banks go to lengths which are completely unethical and illegal (like
recovery of loans and bad debts from poor people). And they don't stop at
anything. I am not blaming any particular bank or person, but I guess it is
just the nature of business. It is for us an individual to decide what kind
of technology we are building and how it can be misused. Will it be used
for war? Will it be used for terrorism? Will it be used harassing poor &
unfortunate people? How can we build technology which will provide
safeguards against these matters for the sake of humanity and life as a
whole?

*Overall, I am sorry once again for posting this discussion on this group.
I will sort this matter out on my own. Thank you everyone for your effort
and time.*

Deepak

Hey

Namaskara~Nalama~Guten Tag~Bonjour


   --
Keigu

Deepak
73500 12833
www.simtree.net, dee...@simtree.net
deic...@gmail.com

LinkedIn: www.linkedin.com/in/deicool
Skype: thumsupdeicool
Google talk: deicool
Blog: http://loveandfearless.wordpress.com
Facebook: http://www.facebook.com/deicool

"Contribute to the world, environment and more : http://www.gridrepublic.org
"

On Wed, May 25, 2016 at 7:16 PM, Tom Deutsch  wrote:

> I would politely remind you that, per the note that Chris sent, this is
> not really the best discussion for this community.
>
> I would also suggest that you may want to better understand permissible
> use of information in a corporate banking context before suggesting
> something is unethical.
>
> 
> Tom Deutsch, MBA
> Global Industry Cognitive Solutions CTO and Transformation
> http://www.ibmbigdatahub.com/blog/author/tom-deutsch
> Twitter: @thomasdeutsch
> Data Management Blog: ibmdatamag.com/author/tdeutsch/
> LinkedIn: http://www.linkedin.com/profile/view?id=833160
>
> Co-author;
> Harness the Power of Big Data
> Understanding Big Data: Analytics for Enterprise Class Hadoop and
> Streaming Data
> Handbook of Human Computing
>
> Employer Advisory Board member at Harrisburg University of Science and
> Technology
> Advisory Board Member for Data Science at California State University,
> Fullerton.
>
> [image: Inactive hide details for Deepak Goel ---05/24/2016 08:16:40
> PM---I did discuss with the product owner. It seems, it is a busin]Deepak
> Goel ---05/24/2016 08:16:40 PM---I did discuss with the product owner. It
> seems, it is a business requirement and we cannot question
>
> From: Deepak Goel 
> To: Suvro Choudhury 
> Cc: Chris Nauroth , "user@hadoop.apache.org" <
> user@hadoop.apache.org>
> Date: 05/24/2016 08:16 PM
> Subject: Re: "Big Data" Problem
> --
>
>
>
> I did discuss with the product owner. It seems, it is a business
> requirement and we cannot question the customer on whether their need is
> valid or not (Or is it legal or ethical).
>
> I was wondering, (on a technical level!), can't we incorporate a change in
> Hadoop, whereby it allows the parent entity (in this case the corporate
> account) to only connect to the child entity (personal account) if there is
> a confirmation received from the customer~child entity (through email, sms,
> whatsapp, etc).
>
> I am sure this is not a one~off case and would be applicable everywhere
> (In most business needs where we are connecting different systems together
> and are bound to raise legal & moral questions)
>
>
>
> Hey
>
> Namaskara~Nalama~Guten Tag~Bonjour
>
>
>--
> Keigu
>
> Deepak
> 73500 12833
> *www.simtree.net* 

Re: HDFS Federation

[Kun Ren]

Yes, it is rename. Thanks for the explanation, make sense.

The rename example you gave(across namenodes) is like a distributed
transaction,  standard protocol like two-phase commit can be used to
support distributed transactions in Database world, so is that possible to
support the "atomic distributed rename" using something like two-phase
commit protocol? Or even it is possible, the implementation should be very
difficult or the performance is bad?

On Wed, May 25, 2016 at 4:26 PM, Chris Nauroth 
wrote:

> You might be thinking of renames.  For example, when using ViewFs with
> Federation, the client-side mount configuration might have path /data1
> backed by NameNode1 and path /data2 backed by NameNode2.  Renaming
> /data1/file1 to /data2/file1 would not be supported, because it would need
> to be copied across NameNodes, and then HDFS would not be able to satisfy
> its promise that rename is atomic.  There are more details about this in
> the ViewFs guide.
>
>
> http://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/ViewFs.html
>
> --Chris Nauroth
>
> From: Kun Ren 
> Date: Wednesday, May 25, 2016 at 11:03 AM
> To: Chris Nauroth 
> Cc: "user@hadoop.apache.org" 
> Subject: Re: HDFS Federation
>
> Thanks a lot, Chris.
>
> I remembered that Federation doesn't support some cross-namenodes
> operations, if so,  which operations that Federation doesn't support and
> why? Thanks again.
>
> On Wed, May 25, 2016 at 12:17 PM, Chris Nauroth 
> wrote:
>
>> Hello Kun,
>>
>> Yes, this command works with federation.  The command would copy the file
>> from NameNode 1/block pool 1 to NameNode 2/block pool 2.
>>
>> --Chris Nauroth
>>
>> From: Kun Ren 
>> Date: Wednesday, May 25, 2016 at 8:57 AM
>> To: "user@hadoop.apache.org" 
>> Subject: HDFS Federation
>>
>> Hi Genius,
>>
>> Does HDFS Federation support the cross namenodes operations?
>>
>> For example:
>>
>> ./bin/hdfs dfs -cp input1/a.xml input2/b.xml
>>
>> Supposed that input1 belongs namenode 1, and input 2 belongs namenode 2,
>> does Federation support this operation? And if not, why?
>>
>> Thanks.
>>
>
>

Re: question on long running YARN app via Kerberos

[Chen Song]

Can someone shed some light on this? This is just to confirm my very high
level understanding of jobs running on Kerberos enabled Yarn cluster is
correct.

On Tue, May 24, 2016 at 3:05 PM, Chen Song  wrote:

> Hi
>
> I am working on running a long lived app on a secure Yarn cluster. After
> some reading on this domain, I want to make sure my understanding on the
> life cycle of an app on Kerberos-enabled Yarn is correct as below.
>
> 1. Client kinit to login into KDC and add the HDFS delegation tokens to
> the launcher context before submitting the application.
> 2. Once the resource is allocated for the Application Master, the Node
> Manager will localizes app resources from HDFS using the HDFS delegation
> tokens in the launcher context. Same rule applies to Container localization
> too.
> 3. If the app is expected to run continuously over 7 (the default
> token-max-life-time) days, then application developer needs to develop a
> way to renew and recreate HDFS delegation token and distribute them among
> Containers. Some strategies can be found here as per Steve,
> https://steveloughran.gitbooks.io/kerberos_and_hadoop/content/sections/yarn.html
> .
> 4. When Container or Application Master fails after token-max-life-time
> elapses. The original HDFS delegation token stored in the launcher context
> will be invalid no matter what. To mitigate this problem, users can set up
> RM as the proxy user to renew HDFS delegation token on behalf of the user,
> as per https://issues.apache.org/jira/browse/YARN-2704. After applying
> this, RM will periodically renew and recreate HDFS delegation tokens and
> update the launcher context as well as all NMs running Containers.
> 5. Assuming 4 is working, technically and theoretically, users can still
> have their app running beyond 7 days even without implementing 3, if my
> understanding is correct. The reason is that once Containers or AM fail b/c
> original HDFS delegation tokens become invalid, once restarts, the new
> Containers or AM will have the valid renewed or recreated HDFS delegation
> token from the launcher context. Of course, this is not scalable and only
> alleviates the problem a bit.
> 6. There seems to have some issues when applying (1-4) in HA Hadoop
> cluster (For example, https://issues.apache.org/jira/browse/HDFS-9276).
> So I assume this is not working for HA Hadoop cluster.
>
> It would be great if someone with insights can let me know if my
> understandings are correct.
>
> --
> Chen Song
>
>


-- 
Chen Song

Re: HDFS Federation

[Chris Nauroth]

You might be thinking of renames.  For example, when using ViewFs with 
Federation, the client-side mount configuration might have path /data1 backed 
by NameNode1 and path /data2 backed by NameNode2.  Renaming /data1/file1 to 
/data2/file1 would not be supported, because it would need to be copied across 
NameNodes, and then HDFS would not be able to satisfy its promise that rename 
is atomic.  There are more details about this in the ViewFs guide.

http://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/ViewFs.html

--Chris Nauroth

From: Kun Ren mailto:ren.h...@gmail.com>>
Date: Wednesday, May 25, 2016 at 11:03 AM
To: Chris Nauroth mailto:cnaur...@hortonworks.com>>
Cc: "user@hadoop.apache.org" 
mailto:user@hadoop.apache.org>>
Subject: Re: HDFS Federation

Thanks a lot, Chris.

I remembered that Federation doesn't support some cross-namenodes operations, 
if so,  which operations that Federation doesn't support and why? Thanks again.

On Wed, May 25, 2016 at 12:17 PM, Chris Nauroth 
mailto:cnaur...@hortonworks.com>> wrote:
Hello Kun,

Yes, this command works with federation.  The command would copy the file from 
NameNode 1/block pool 1 to NameNode 2/block pool 2.

--Chris Nauroth

From: Kun Ren mailto:ren.h...@gmail.com>>
Date: Wednesday, May 25, 2016 at 8:57 AM
To: "user@hadoop.apache.org" 
mailto:user@hadoop.apache.org>>
Subject: HDFS Federation

Hi Genius,

Does HDFS Federation support the cross namenodes operations?

For example:

./bin/hdfs dfs -cp input1/a.xml input2/b.xml

Supposed that input1 belongs namenode 1, and input 2 belongs namenode 2, does 
Federation support this operation? And if not, why?

Thanks.

RE: YARN cluster underutilization

[Guttadauro, Jeff]

Interesting stuff!  I did not know about this handling of OFFSWITCH requests.

To get around this, would you recommend reducing the heartbeat interval, 
perhaps to 250ms to get a 4x improvement in container allocation rate (or is it 
not quite as simple as that)?  Maybe doing this in combination with using a 
greater number of smaller nodes would help?  Would overloading the 
ResourceManager be a concern if doing that?  Should I bump up the 
“YARN_RESOURCEMANAGER_HEAPSIZE” configuration property (current default for 
m3.xlarge is 2396M), or would you suggest any other knobs to turn to help RM 
handle it?

Thanks again for all your help, Sunil!

From: Sunil Govind [mailto:sunil.gov...@gmail.com]
Sent: Wednesday, May 25, 2016 1:07 PM
To: Guttadauro, Jeff ; user@hadoop.apache.org
Subject: Re: YARN cluster underutilization

Hi Jeff,

 I do see the yarn.resourcemanager.nodemanagers.heartbeat-interval-ms property 
set to 1000 in the job configuration
>> Ok, This make sense.. node heartbeat seems default.

If there are no locality specified in resource requests (using 
ResourceRequest.ANY) , then YARN will allocate only one container per node 
heartbeat. So your container allocation rate is slower considering 600k 
requests and only 20 nodes. And if more number of containers are also getting 
released fast (I could see that some containers lifetime is 80 to 90 secs), 
then this will become more complex and container allocation rate will be slower.

YARN-4963 is trying to make 
more allocation per heartbeat for NODE_OFFSWITCH (ANY) requests. But its not 
yet available in any release.

I guess you can investigate more in this line to confirm this points.

Thanks
Sunil


On Wed, May 25, 2016 at 11:00 PM Guttadauro, Jeff 
mailto:jeff.guttada...@here.com>> wrote:
Thanks for digging into the log, Sunil, and making some interesting 
observations!

The heartbeat interval hasn’t been changed from its default, and I do see the 
yarn.resourcemanager.nodemanagers.heartbeat-interval-ms property set to 1000 in 
the job configuration.  I was searching in the log for heartbeat interval 
information, but I didn’t find anything.  Where do you look in the log for the 
heartbeats?

Also, you are correct about there being no data locality, as all the input data 
is in S3.  The utilization has been fluctuating, but I can’t really see a 
pattern or tell why.  It actually started out pretty low in the 20-30% range 
and then managed to get up into the 50-70% range after a while, but that was 
short-lived, as it went back down into the 20-30% range for quite a while.  
While writing this, I saw it surprisingly hit 80%!!  First time I’ve seen it 
that high in the 20 hours it’s been running…  Although looks like it may be 
headed back down.  I’m perplexed.  Wouldn’t you generally expect fairly stable 
utilization over the course of the job?  (This is the only job running.)

Thanks,
-Jeff

From: Sunil Govind 
[mailto:sunil.gov...@gmail.com]
Sent: Wednesday, May 25, 2016 11:55 AM

To: Guttadauro, Jeff 
mailto:jeff.guttada...@here.com>>; 
user@hadoop.apache.org
Subject: Re: YARN cluster underutilization

Hi Jeff.

Thanks for sharing this information. I have some observations from this logs.

- I think the node heartbeat is around 2/3 seconds here. Is it changed due to 
some other reasons?
- And all mappers Resource Request seems to be asking for type ANY (there is no 
data locality). pls correct me if I am wrong.

If the resource request type is ANY, only one container will be allocated per 
heartbeat for a node. Here node heartbeat delay is also more. And I can see 
that containers are released very fast too. So when u started you application, 
are you seeing more better resource utilization? And once containers started to 
get released/completed, you are seeing under utilization.

Pls look into this line. It may be a reason.

Thanks
Sunil

On Wed, May 25, 2016 at 9:59 PM Guttadauro, Jeff 
mailto:jeff.guttada...@here.com>> wrote:
Thanks for your thoughts thus far, Sunil.  Most grateful for any additional 
help you or others can offer.  To answer your questions,


1.   This is a custom M/R job, which uses mappers only (no reduce phase) to 
process GPS probe data and filter based on inclusion within a provided polygon. 
 There is actually a lot of upfront work done in the driver to make that task 
as simple as can be (identifies a list of tiles that are completely inside the 
polygon and those that fall across an edge, for which more processing would be 
needed), but the job would still be more compute-intensive than wordcount, for 
example.



2.   I’m running almost 84k mappers for this job.  This is actually down 
from ~600k mappers, since one other thing I’ve done is increased the 
mapreduce.input.fileinputformat.split.minsize to 536870912 (512M) for the job.  
Data is in S3, so loss of locality isn’t really a concern.


3.

Hadoop Security - NN fails to connect to JN on different servers

[Arulanand Dayalan]

Hi,
I had setup Hadoop HA Cluster(QJM) with kerberos. Cluster has three journal 
nodes. Namenode is able to connectto journal node running on the Namenode. For 
other Journal nodes its not able to connect and throws below exception.Can you 
please provide inputs for resolving the issue.
2016-05-25 23:09:47,526 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: 
STARTUP_MSG:/STARTUP_MSG:
 Starting NameNodeSTARTUP_MSG:   host = 
server1.cehdev.com/10.67.169.45STARTUP_MSG:   args = []STARTUP_MSG:   version = 
2.5.2STARTUP_MSG:   classpath = STARTUP_MSG:   build = 
https://git-wip-us.apache.org/repos/asf/hadoop.git -r 
cc72e9b000545b86b75a61f4835eb86d57bfafc0; compiled by 'jenkins' on 
2014-11-14T23:45ZSTARTUP_MSG:   java = 
1.7.0_79/2016-05-25 
23:09:47,550 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: registered 
UNIX signal handlers for [TERM, HUP, INT]2016-05-25 23:09:47,557 INFO 
org.apache.hadoop.hdfs.server.namenode.NameNode: createNameNode []2016-05-25 
23:09:48,147 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded 
properties from hadoop-metrics2.properties2016-05-25 23:09:48,307 INFO 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 
10 second(s).2016-05-25 23:09:48,307 INFO 
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system 
started2016-05-25 23:09:48,309 INFO 
org.apache.hadoop.hdfs.server.namenode.NameNode: fs.defaultFS is 
hdfs://hdcluster2016-05-25 23:09:48,310 INFO 
org.apache.hadoop.hdfs.server.namenode.NameNode: Clients are to use hdcluster 
to access this namenode/service.2016-05-25 23:09:49,331 INFO 
org.apache.hadoop.security.UserGroupInformation: Login successful for user 
hdfs/server1.cehdev@cehdev.com using keytab file 
/etc/security/keytab/hdfs.service.keytab2016-05-25 23:09:49,392 INFO 
org.apache.hadoop.hdfs.DFSUtil: Starting web server as: 
HTTP/server1.cehdev.com@CEHDEV.COM2016-05-25 23:09:49,392 INFO 
org.apache.hadoop.hdfs.DFSUtil: Starting Web-server for hdfs at: 
http://server1.cehdev.com:500702016-05-25 23:09:49,480 INFO org.mortbay.log: 
Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via 
org.mortbay.log.Slf4jLog2016-05-25 23:09:49,487 INFO 
org.apache.hadoop.http.HttpRequestLog: Http request log for 
http.requests.namenode is not defined2016-05-25 23:09:49,504 INFO 
org.apache.hadoop.http.HttpServer2: Added global filter 'safety' 
(class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)2016-05-25 
23:09:49,510 INFO org.apache.hadoop.http.HttpServer2: Added filter 
static_user_filter 
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to 
context hdfs2016-05-25 23:09:49,510 INFO org.apache.hadoop.http.HttpServer2: 
Added filter static_user_filter 
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to 
context logs2016-05-25 23:09:49,511 INFO org.apache.hadoop.http.HttpServer2: 
Added filter static_user_filter 
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to 
context static2016-05-25 23:09:49,563 INFO org.apache.hadoop.http.HttpServer2: 
Added filter 'org.apache.hadoop.hdfs.web.AuthFilter' 
(class=org.apache.hadoop.hdfs.web.AuthFilter)2016-05-25 23:09:49,566 INFO 
org.apache.hadoop.http.HttpServer2: addJerseyResourcePackage: 
packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
 pathSpec=/webhdfs/v1/*2016-05-25 23:09:49,573 INFO 
org.apache.hadoop.http.HttpServer2: Adding Kerberos (SPNEGO) filter to 
getDelegationToken2016-05-25 23:09:49,575 INFO 
org.apache.hadoop.http.HttpServer2: Adding Kerberos (SPNEGO) filter to 
renewDelegationToken2016-05-25 23:09:49,577 INFO 
org.apache.hadoop.http.HttpServer2: Adding Kerberos (SPNEGO) filter to 
cancelDelegationToken2016-05-25 23:09:49,579 INFO 
org.apache.hadoop.http.HttpServer2: Adding Kerberos (SPNEGO) filter to 
fsck2016-05-25 23:09:49,581 INFO org.apache.hadoop.http.HttpServer2: Adding 
Kerberos (SPNEGO) filter to imagetransfer2016-05-25 23:09:49,593 INFO 
org.apache.hadoop.http.HttpServer2: Jetty bound to port 500702016-05-25 
23:09:49,593 INFO org.mortbay.log: jetty-6.1.262016-05-25 23:09:49,954 INFO 
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: 
Login using keytab /etc/security/keytab/hdfs.service.keytab, for principal 
HTTP/server1.cehdev.com@CEHDEV.COM2016-05-25 23:09:49,965 WARN 
org.apache.hadoop.security.authentication.server.AuthenticationFilter: 
'signature.secret' configuration not set, using a random value as 
secret2016-05-25 23:09:49,967 INFO 
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler: 
Login using keytab /etc/security/keytab/hdfs.service.keytab, for principal 
HTTP/server1.cehdev.com@CEHDEV.COM2016-05-25 23:09:49,973 WARN 
org.apache.hadoop.security.authentication.server.AuthenticationFilter: 
'signature.secret'

Re: YARN cluster underutilization

[Sunil Govind]

Hi Jeff,

 I do see the yarn.resourcemanager.nodemanagers.heartbeat-interval-ms
property set to 1000 in the job configuration
>> Ok, This make sense.. node heartbeat seems default.

If there are no locality specified in resource requests (using
ResourceRequest.ANY) , then YARN will allocate only one container per node
heartbeat. So your container allocation rate is slower considering 600k
requests and only 20 nodes. And if more number of containers are also
getting released fast (I could see that some containers lifetime is 80 to
90 secs), then this will become more complex and container allocation rate
will be slower.

YARN-4963  is trying to
make more allocation per heartbeat for NODE_OFFSWITCH (ANY) requests. But
its not yet available in any release.

I guess you can investigate more in this line to confirm this points.

Thanks
Sunil


On Wed, May 25, 2016 at 11:00 PM Guttadauro, Jeff 
wrote:

> Thanks for digging into the log, Sunil, and making some interesting
> observations!
>
>
>
> The heartbeat interval hasn’t been changed from its default, and I do see
> the yarn.resourcemanager.nodemanagers.heartbeat-interval-ms property set to
> 1000 in the job configuration.  I was searching in the log for heartbeat
> interval information, but I didn’t find anything.  Where do you look in the
> log for the heartbeats?
>
>
>
> Also, you are correct about there being no data locality, as all the input
> data is in S3.  The utilization has been fluctuating, but I can’t really
> see a pattern or tell why.  It actually started out pretty low in the
> 20-30% range and then managed to get up into the 50-70% range after a
> while, but that was short-lived, as it went back down into the 20-30% range
> for quite a while.  While writing this, I saw it surprisingly hit 80%!!
> First time I’ve seen it that high in the 20 hours it’s been running…
>  Although looks like it may be headed back down.  I’m perplexed.  Wouldn’t
> you generally expect fairly stable utilization over the course of the job?
> (This is the only job running.)
>
>
>
> Thanks,
>
> -Jeff
>
>
>
> *From:* Sunil Govind [mailto:sunil.gov...@gmail.com]
> *Sent:* Wednesday, May 25, 2016 11:55 AM
>
>
> *To:* Guttadauro, Jeff ; user@hadoop.apache.org
> *Subject:* Re: YARN cluster underutilization
>
>
>
> Hi Jeff.
>
>
>
> Thanks for sharing this information. I have some observations from this
> logs.
>
>
>
> - I think the node heartbeat is around 2/3 seconds here. Is it changed due
> to some other reasons?
>
> - And all mappers Resource Request seems to be asking for type ANY (there
> is no data locality). pls correct me if I am wrong.
>
>
>
> If the resource request type is ANY, only one container will be allocated
> per heartbeat for a node. Here node heartbeat delay is also more. And I can
> see that containers are released very fast too. So when u started you
> application, are you seeing more better resource utilization? And once
> containers started to get released/completed, you are seeing under
> utilization.
>
>
>
> Pls look into this line. It may be a reason.
>
>
>
> Thanks
>
> Sunil
>
>
>
> On Wed, May 25, 2016 at 9:59 PM Guttadauro, Jeff 
> wrote:
>
> Thanks for your thoughts thus far, Sunil.  Most grateful for any
> additional help you or others can offer.  To answer your questions,
>
>
>
> 1.   This is a custom M/R job, which uses mappers only (no reduce
> phase) to process GPS probe data and filter based on inclusion within a
> provided polygon.  There is actually a lot of upfront work done in the
> driver to make that task as simple as can be (identifies a list of tiles
> that are completely inside the polygon and those that fall across an edge,
> for which more processing would be needed), but the job would still be more
> compute-intensive than wordcount, for example.
>
>
>
> 2.   I’m running almost 84k mappers for this job.  This is actually
> down from ~600k mappers, since one other thing I’ve done is increased the
> mapreduce.input.fileinputformat.split.minsize to 536870912 (512M) for the
> job.  Data is in S3, so loss of locality isn’t really a concern.
>
>
>
> 3.   For NodeManager configuration, I’m using EMR’s default
> configuration for the m3.xlarge instance type, which is
> yarn.scheduler.minimum-allocation-mb=32,
> yarn.scheduler.maximum-allocation-mb=11520, and
> yarn.nodemanager.resource.memory-mb=11520.  YARN dashboard shows min/max
> allocations of /.
>
>
>
> 4.   Capacity Scheduler [MEMORY]
>
>
>
> 5.   I’ve attached 2500 lines from the RM log.  Happy to grab more,
> but they are pretty big, and I thought that might be sufficient.
>
>
>
> Any guidance is much appreciated!
>
> -Jeff
>
>
>
> *From:* Sunil Govind [mailto:sunil.gov...@gmail.com]
> *Sent:* Wednesday, May 25, 2016 10:55 AM
> *To:* Guttadauro, Jeff ; user@hadoop.apache.org
> *Subject:* Re: YARN cluster underutilization
>
>
>
> Hi Jeff,
>
>
>
> It looks like to you are allocating more memory for AM contain

Re: HDFS Federation

[Kun Ren]

Thanks a lot, Chris.

I remembered that Federation doesn't support some cross-namenodes
operations, if so,  which operations that Federation doesn't support and
why? Thanks again.

On Wed, May 25, 2016 at 12:17 PM, Chris Nauroth 
wrote:

> Hello Kun,
>
> Yes, this command works with federation.  The command would copy the file
> from NameNode 1/block pool 1 to NameNode 2/block pool 2.
>
> --Chris Nauroth
>
> From: Kun Ren 
> Date: Wednesday, May 25, 2016 at 8:57 AM
> To: "user@hadoop.apache.org" 
> Subject: HDFS Federation
>
> Hi Genius,
>
> Does HDFS Federation support the cross namenodes operations?
>
> For example:
>
> ./bin/hdfs dfs -cp input1/a.xml input2/b.xml
>
> Supposed that input1 belongs namenode 1, and input 2 belongs namenode 2,
> does Federation support this operation? And if not, why?
>
> Thanks.
>

RE: YARN cluster underutilization

[Guttadauro, Jeff]

Thanks for digging into the log, Sunil, and making some interesting 
observations!

The heartbeat interval hasn’t been changed from its default, and I do see the 
yarn.resourcemanager.nodemanagers.heartbeat-interval-ms property set to 1000 in 
the job configuration.  I was searching in the log for heartbeat interval 
information, but I didn’t find anything.  Where do you look in the log for the 
heartbeats?

Also, you are correct about there being no data locality, as all the input data 
is in S3.  The utilization has been fluctuating, but I can’t really see a 
pattern or tell why.  It actually started out pretty low in the 20-30% range 
and then managed to get up into the 50-70% range after a while, but that was 
short-lived, as it went back down into the 20-30% range for quite a while.  
While writing this, I saw it surprisingly hit 80%!!  First time I’ve seen it 
that high in the 20 hours it’s been running…  Although looks like it may be 
headed back down.  I’m perplexed.  Wouldn’t you generally expect fairly stable 
utilization over the course of the job?  (This is the only job running.)

Thanks,
-Jeff

From: Sunil Govind [mailto:sunil.gov...@gmail.com]
Sent: Wednesday, May 25, 2016 11:55 AM
To: Guttadauro, Jeff ; user@hadoop.apache.org
Subject: Re: YARN cluster underutilization

Hi Jeff.

Thanks for sharing this information. I have some observations from this logs.

- I think the node heartbeat is around 2/3 seconds here. Is it changed due to 
some other reasons?
- And all mappers Resource Request seems to be asking for type ANY (there is no 
data locality). pls correct me if I am wrong.

If the resource request type is ANY, only one container will be allocated per 
heartbeat for a node. Here node heartbeat delay is also more. And I can see 
that containers are released very fast too. So when u started you application, 
are you seeing more better resource utilization? And once containers started to 
get released/completed, you are seeing under utilization.

Pls look into this line. It may be a reason.

Thanks
Sunil

On Wed, May 25, 2016 at 9:59 PM Guttadauro, Jeff 
mailto:jeff.guttada...@here.com>> wrote:
Thanks for your thoughts thus far, Sunil.  Most grateful for any additional 
help you or others can offer.  To answer your questions,


1.   This is a custom M/R job, which uses mappers only (no reduce phase) to 
process GPS probe data and filter based on inclusion within a provided polygon. 
 There is actually a lot of upfront work done in the driver to make that task 
as simple as can be (identifies a list of tiles that are completely inside the 
polygon and those that fall across an edge, for which more processing would be 
needed), but the job would still be more compute-intensive than wordcount, for 
example.



2.   I’m running almost 84k mappers for this job.  This is actually down 
from ~600k mappers, since one other thing I’ve done is increased the 
mapreduce.input.fileinputformat.split.minsize to 536870912 (512M) for the job.  
Data is in S3, so loss of locality isn’t really a concern.


3.   For NodeManager configuration, I’m using EMR’s default configuration 
for the m3.xlarge instance type, which is 
yarn.scheduler.minimum-allocation-mb=32, 
yarn.scheduler.maximum-allocation-mb=11520, and 
yarn.nodemanager.resource.memory-mb=11520.  YARN dashboard shows min/max 
allocations of /.


4.   Capacity Scheduler [MEMORY]


5.   I’ve attached 2500 lines from the RM log.  Happy to grab more, but 
they are pretty big, and I thought that might be sufficient.

Any guidance is much appreciated!
-Jeff

From: Sunil Govind 
[mailto:sunil.gov...@gmail.com]
Sent: Wednesday, May 25, 2016 10:55 AM
To: Guttadauro, Jeff 
mailto:jeff.guttada...@here.com>>; 
user@hadoop.apache.org
Subject: Re: YARN cluster underutilization

Hi Jeff,

It looks like to you are allocating more memory for AM container. Mostly you 
might not need 6Gb (as per the log). Could you please help  to provide some 
more information.

1. What type of mapreduce application (wordcount etc) are you running? Some AMs 
may be CPU intensive and some may not be. So based on the type application, 
memory/cpu can be tuned for better utilization.
2. How many mappers (reducers) are you trying to run here?
3. You have mentioned that each node has 8 cores and 15GB, but how much is 
actually configured for NM?
4. Which scheduler are you using?
5. Its better to attach RM log if possible.

Thanks
Sunil

On Wed, May 25, 2016 at 8:58 PM Guttadauro, Jeff 
mailto:jeff.guttada...@here.com>> wrote:
Hi, all.

I have an M/R (map-only) job that I’m running on a Hadoop 2.7.1 YARN cluster 
that is being quite underutilized (utilization of around 25-30%).  The EMR 
cluster is 1 master + 20 core m3.xlarge nodes, which have 8 cores each and 15G 
total memory (with 11.25G of that available to YARN).  I’ve configured mapper 
memory with the following properties, which should allow for 8 conta

Re: YARN cluster underutilization

[Sunil Govind]

Hi Jeff.

Thanks for sharing this information. I have some observations from this
logs.

- I think the node heartbeat is around 2/3 seconds here. Is it changed due
to some other reasons?
- And all mappers Resource Request seems to be asking for type ANY (there
is no data locality). pls correct me if I am wrong.

If the resource request type is ANY, only one container will be allocated
per heartbeat for a node. Here node heartbeat delay is also more. And I can
see that containers are released very fast too. So when u started you
application, are you seeing more better resource utilization? And once
containers started to get released/completed, you are seeing under
utilization.

Pls look into this line. It may be a reason.

Thanks
Sunil

On Wed, May 25, 2016 at 9:59 PM Guttadauro, Jeff 
wrote:

> Thanks for your thoughts thus far, Sunil.  Most grateful for any
> additional help you or others can offer.  To answer your questions,
>
>
>
> 1.   This is a custom M/R job, which uses mappers only (no reduce
> phase) to process GPS probe data and filter based on inclusion within a
> provided polygon.  There is actually a lot of upfront work done in the
> driver to make that task as simple as can be (identifies a list of tiles
> that are completely inside the polygon and those that fall across an edge,
> for which more processing would be needed), but the job would still be more
> compute-intensive than wordcount, for example.
>
>
>
> 2.   I’m running almost 84k mappers for this job.  This is actually
> down from ~600k mappers, since one other thing I’ve done is increased the
> mapreduce.input.fileinputformat.split.minsize to 536870912 (512M) for the
> job.  Data is in S3, so loss of locality isn’t really a concern.
>
>
>
> 3.   For NodeManager configuration, I’m using EMR’s default
> configuration for the m3.xlarge instance type, which is
> yarn.scheduler.minimum-allocation-mb=32,
> yarn.scheduler.maximum-allocation-mb=11520, and
> yarn.nodemanager.resource.memory-mb=11520.  YARN dashboard shows min/max
> allocations of /.
>
>
>
> 4.   Capacity Scheduler [MEMORY]
>
>
>
> 5.   I’ve attached 2500 lines from the RM log.  Happy to grab more,
> but they are pretty big, and I thought that might be sufficient.
>
>
>
> Any guidance is much appreciated!
>
> -Jeff
>
>
>
> *From:* Sunil Govind [mailto:sunil.gov...@gmail.com]
> *Sent:* Wednesday, May 25, 2016 10:55 AM
> *To:* Guttadauro, Jeff ; user@hadoop.apache.org
> *Subject:* Re: YARN cluster underutilization
>
>
>
> Hi Jeff,
>
>
>
> It looks like to you are allocating more memory for AM container. Mostly
> you might not need 6Gb (as per the log). Could you please help  to provide
> some more information.
>
>
>
> 1. What type of mapreduce application (wordcount etc) are you running?
> Some AMs may be CPU intensive and some may not be. So based on the type
> application, memory/cpu can be tuned for better utilization.
>
> 2. How many mappers (reducers) are you trying to run here?
>
> 3. You have mentioned that each node has 8 cores and 15GB, but how much is
> actually configured for NM?
>
> 4. Which scheduler are you using?
>
> 5. Its better to attach RM log if possible.
>
>
>
> Thanks
>
> Sunil
>
>
>
> On Wed, May 25, 2016 at 8:58 PM Guttadauro, Jeff 
> wrote:
>
> Hi, all.
>
>
>
> I have an M/R (map-only) job that I’m running on a Hadoop 2.7.1 YARN
> cluster that is being quite underutilized (utilization of around 25-30%).
> The EMR cluster is 1 master + 20 core m3.xlarge nodes, which have 8 cores
> each and 15G total memory (with 11.25G of that available to YARN).  I’ve
> configured mapper memory with the following properties, which should allow
> for 8 containers running map tasks per node:
>
>
>
> mapreduce.map.memory.mb1440
> 
>
> mapreduce.map.java.opts-Xmx1024m
> 
>
>
>
> It was suggested that perhaps my AppMaster was having trouble keeping up
> with creating all the mapper containers and that I bulk up its resource
> allocation.  So I did, as shown below, providing it 6G container memory (5G
> task memory), 3 cores, and 60 task listener threads.
>
>
>
> yarn.app.mapreduce.am.job.task.listener.thread-count60
> 
>
> yarn.app.mapreduce.am.resource.cpu-vcores3
> 
>
> yarn.app.mapreduce.am.resource.mb6400
> 
>
> yarn.app.mapreduce.am.command-opts-Xmx5120m
> 
>
>
>
> Taking a look at the node on which the AppMaster is running, I'm seeing
> plenty of CPU idle time and free memory, yet there are still nodes with no
> utilization (0 running containers).  The log indicates that the AppMaster
> has way more memory (physical/virtual) than it appears to need with
> repeated log messages like this:
>
>
>
> 2016-05-25 13:59:04,615 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
> (Container Monitor): Memory usage of ProcessTree 11265 for container-id
> container_1464122327865_0002_01_01: 1.6 GB of 6.3 GB physical memory
> used; 6.1 GB of 31.3 GB virtual memory used
>
>
>
> Can you please help me figure out whe

Re: HDFS Federation

[Chris Nauroth]

Hello Kun,

Yes, this command works with federation.  The command would copy the file from 
NameNode 1/block pool 1 to NameNode 2/block pool 2.

--Chris Nauroth

From: Kun Ren mailto:ren.h...@gmail.com>>
Date: Wednesday, May 25, 2016 at 8:57 AM
To: "user@hadoop.apache.org" 
mailto:user@hadoop.apache.org>>
Subject: HDFS Federation

Hi Genius,

Does HDFS Federation support the cross namenodes operations?

For example:

./bin/hdfs dfs -cp input1/a.xml input2/b.xml

Supposed that input1 belongs namenode 1, and input 2 belongs namenode 2, does 
Federation support this operation? And if not, why?

Thanks.

Re: YARN cluster underutilization

[Sunil Govind]

Hi Jeff,

It looks like to you are allocating more memory for AM container. Mostly
you might not need 6Gb (as per the log). Could you please help  to provide
some more information.

1. What type of mapreduce application (wordcount etc) are you running? Some
AMs may be CPU intensive and some may not be. So based on the type
application, memory/cpu can be tuned for better utilization.
2. How many mappers (reducers) are you trying to run here?
3. You have mentioned that each node has 8 cores and 15GB, but how much is
actually configured for NM?
4. Which scheduler are you using?
5. Its better to attach RM log if possible.

Thanks
Sunil

On Wed, May 25, 2016 at 8:58 PM Guttadauro, Jeff 
wrote:

> Hi, all.
>
>
>
> I have an M/R (map-only) job that I’m running on a Hadoop 2.7.1 YARN
> cluster that is being quite underutilized (utilization of around 25-30%).
> The EMR cluster is 1 master + 20 core m3.xlarge nodes, which have 8 cores
> each and 15G total memory (with 11.25G of that available to YARN).  I’ve
> configured mapper memory with the following properties, which should allow
> for 8 containers running map tasks per node:
>
>
>
> mapreduce.map.memory.mb1440
> 
>
> mapreduce.map.java.opts-Xmx1024m
> 
>
>
>
> It was suggested that perhaps my AppMaster was having trouble keeping up
> with creating all the mapper containers and that I bulk up its resource
> allocation.  So I did, as shown below, providing it 6G container memory (5G
> task memory), 3 cores, and 60 task listener threads.
>
>
>
> yarn.app.mapreduce.am.job.task.listener.thread-count60
> 
>
> yarn.app.mapreduce.am.resource.cpu-vcores3
> 
>
> yarn.app.mapreduce.am.resource.mb6400
> 
>
> yarn.app.mapreduce.am.command-opts-Xmx5120m
> 
>
>
>
> Taking a look at the node on which the AppMaster is running, I'm seeing
> plenty of CPU idle time and free memory, yet there are still nodes with no
> utilization (0 running containers).  The log indicates that the AppMaster
> has way more memory (physical/virtual) than it appears to need with
> repeated log messages like this:
>
>
>
> 2016-05-25 13:59:04,615 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
> (Container Monitor): Memory usage of ProcessTree 11265 for container-id
> container_1464122327865_0002_01_01: 1.6 GB of 6.3 GB physical memory
> used; 6.1 GB of 31.3 GB virtual memory used
>
>
>
> Can you please help me figure out where to go from here to troubleshoot,
> or any other things to try?
>
>
>
> Thanks!
>
> -Jeff
>
>
>

HDFS Federation

[Kun Ren]

Hi Genius,

Does HDFS Federation support the cross namenodes operations?

For example:

./bin/hdfs dfs -cp input1/a.xml input2/b.xml

Supposed that input1 belongs namenode 1, and input 2 belongs namenode 2,
does Federation support this operation? And if not, why?

Thanks.

YARN cluster underutilization

[Guttadauro, Jeff]

Hi, all.

I have an M/R (map-only) job that I'm running on a Hadoop 2.7.1 YARN cluster 
that is being quite underutilized (utilization of around 25-30%).  The EMR 
cluster is 1 master + 20 core m3.xlarge nodes, which have 8 cores each and 15G 
total memory (with 11.25G of that available to YARN).  I've configured mapper 
memory with the following properties, which should allow for 8 containers 
running map tasks per node:

mapreduce.map.memory.mb1440   

mapreduce.map.java.opts-Xmx1024m
  

It was suggested that perhaps my AppMaster was having trouble keeping up with 
creating all the mapper containers and that I bulk up its resource allocation.  
So I did, as shown below, providing it 6G container memory (5G task memory), 3 
cores, and 60 task listener threads.

yarn.app.mapreduce.am.job.task.listener.thread-count60
  
yarn.app.mapreduce.am.resource.cpu-vcores3
  
yarn.app.mapreduce.am.resource.mb6400
  
yarn.app.mapreduce.am.command-opts-Xmx5120m
  


Taking a look at the node on which the AppMaster is running, I'm seeing plenty 
of CPU idle time and free memory, yet there are still nodes with no utilization 
(0 running containers).  The log indicates that the AppMaster has way more 
memory (physical/virtual) than it appears to need with repeated log messages 
like this:



2016-05-25 13:59:04,615 INFO 
org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl
 (Container Monitor): Memory usage of ProcessTree 11265 for container-id 
container_1464122327865_0002_01_01: 1.6 GB of 6.3 GB physical memory used; 
6.1 GB of 31.3 GB virtual memory used



Can you please help me figure out where to go from here to troubleshoot, or any 
other things to try?

Thanks!
-Jeff

Re: "Big Data" Problem

[Tom Deutsch]

I would politely remind you that, per the note that Chris sent, this is not
really the best discussion for this community.

I would also suggest that you may want to better understand permissible use
of information in a corporate banking context before suggesting something
is unethical.


Tom Deutsch, MBA
Global Industry Cognitive Solutions CTO and Transformation
http://www.ibmbigdatahub.com/blog/author/tom-deutsch
Twitter: @thomasdeutsch
Data Management Blog: ibmdatamag.com/author/tdeutsch/
LinkedIn: http://www.linkedin.com/profile/view?id=833160

Co-author;
Harness the Power of Big Data
Understanding Big Data: Analytics for Enterprise Class Hadoop and Streaming
Data
Handbook of Human Computing

Employer Advisory Board member at Harrisburg University of Science and
Technology
Advisory Board Member for Data Science at California State University,
Fullerton.



From:   Deepak Goel 
To: Suvro Choudhury 
Cc: Chris Nauroth ,
"user@hadoop.apache.org" 
Date:   05/24/2016 08:16 PM
Subject:Re: "Big Data" Problem



I did discuss with the product owner. It seems, it is a business
requirement and we cannot question the customer on whether their need is
valid or not (Or is it legal or ethical).

I was wondering, (on a technical level!), can't we incorporate a change in
Hadoop, whereby it allows the parent entity (in this case the corporate
account) to only connect to the child entity (personal account) if there is
a confirmation received from the customer~child entity (through email, sms,
whatsapp, etc).

I am sure this is not a one~off case and would be applicable everywhere (In
most business needs where we are connecting different systems together and
are bound to raise legal & moral questions)



Hey

Namaskara~Nalama~Guten Tag~Bonjour


   --
Keigu

Deepak
73500 12833
www.simtree.net, dee...@simtree.net
deic...@gmail.com

LinkedIn: www.linkedin.com/in/deicool
Skype: thumsupdeicool
Google talk: deicool
Blog: http://loveandfearless.wordpress.com
Facebook: http://www.facebook.com/deicool

"Contribute to the world, environment and more :
http://www.gridrepublic.org
"

On Wed, May 25, 2016 at 7:55 AM, Suvro Choudhury  wrote:
  IMHO, you should discuss this with your product owner /business customer
  since this relates to the requirement. This forum is for purely technical
  questions, although I agree as human being we need to be sensitive about
  these.
  You can also try posting to Quora and see what folks think.
  Cheers!


  On May 24, 2016 2:29 PM, "Deepak Goel"  wrote:
   Hmm..i understand what you are saying Chris :)

   However it is like that. You are building a knife (Hadoop in this case)
   and you would know that you would kill someone by it. Would you build
   the knife at all (So it is about Hadoop also in some sense. But as
   pointed about by you, it is not a technical question. However it is
   about usage of Hadoop)?

   Anyways sorry if I have taken up time of everyone :-(

   Hey

   Namaskara~Nalama~Guten Tag~Bonjour


      --
   Keigu

   Deepak
   73500 12833
   www.simtree.net, dee...@simtree.net
   deic...@gmail.com

   LinkedIn: www.linkedin.com/in/deicool
   Skype: thumsupdeicool
   Google talk: deicool
   Blog: http://loveandfearless.wordpress.com
   Facebook: http://www.facebook.com/deicool

   "Contribute to the world, environment and more :
   http://www.gridrepublic.org
   "

   On Tue, May 24, 2016 at 10:31 PM, Chris Nauroth <
   cnaur...@hortonworks.com> wrote:
 Hello Deepak,

 This is a fascinating question, but it's not the right forum.  This
 list is for questions on usage of Apache Hadoop.  I don't see anything
 about Hadoop in your question.  Even if the software in question
 involves Hadoop, it's not a question about usage.  Unfortunately, I'm
 not aware of a more appropriate forum for this question, so I don't
 have a recommendation for where to take this.

 --Chris Nauroth

 From: Deepak Goel 
 Date: Monday, May 23, 2016 at 2:59 PM
 To: "user@hadoop.apache.org" 
 Subject: "Big Data" Problem


 Hey

 Namaskara~Nalama~Guten Tag~Bonjour

 (Sorry, as this might not be the right question for this forum.
 However am not sure which other forum would be able to address this
 problem)

 I work for a "Big Data" company and we are supposed to provide the
 technology to a bank which will link the corporate accounts and
 savings accounts of a customer.

 So if a bank has given a corporate loan to a company and the company
 does not dole out some payments from its savings account, then the
 bank should warn itself and take action against the company as there
 might be a probability that the company might default on its loan.

 Now we would be providing the technology however it is not completely
 ethical to link the corporate account to the savings account of the
 customer company. However the bank is sayin