RE: how to add a shareable node label?

[Frank Luo]

Sunil,

You confirmed my understanding. I got the understanding by reading the docs and 
haven’t really tried 2.8 or 3.0-alphal.

My situation is that I am in a multi-tenant env, and  got several very powerful 
machines with expensive licenses to run a particular linux job, let’s say 
Job_A. But the job is executed infrequently, so I want to let other jobs to use 
the machines when Job_A is not running. In the meaning time, I am not powerful 
enough to force all other jobs to be preemptable. As matter of fact, I know 
they have Hadoop jobs inserting into sql-server, or just pure linux jobs that 
are not preemptable in nature. So preempt jobs is not an option for me.

I hope it makes sense.

Frank

From: Sunil Govind [mailto:sunil.gov...@gmail.com]
Sent: Thursday, October 06, 2016 2:15 PM
To: Frank Luo ; user@hadoop.apache.org
Subject: Re: how to add a shareable node label?

HI Frank

Ideally those containers will be preempted if there are unsatisfied demand for 
"configured label".

I could explain this:
"labelA" has few empty resources.  All nodes under "default" label is used. 
Hence a new application which is submitted to "default" label has to wait. But 
if "labelA" is non-exclusive and there are some free resources, this new 
application can run on "labelA".
However if there are some more new apps submitted to "labelA", and if there are 
no more resources available in "labelA", then it may preempt containers from 
the app which was sharing containers earlier.

May be you could share some more information so tht it may become more clear. 
Also I suppose you are running this in hadoop 3 alpha1 release. please correct 
me if I m wrong.

Thanks
Sunil

On Thu, Oct 6, 2016 at 9:44 PM Frank Luo 
> wrote:
Thanks Sunil.


>  3. If there is any future ask for those resources , we will preempt the non 
> labeled apps and give them back to labeled apps.

Unfortunately, I am still not able to use it, because of the preemptive 
behavior. The jobs that steals labelled resources are not preemptable, and I’d 
rather waiting instead of killing.

From: Sunil Govind 
[mailto:sunil.gov...@gmail.com]
Sent: Thursday, October 06, 2016 1:59 AM

To: Frank Luo >; 
user@hadoop.apache.org
Subject: Re: how to add a shareable node label?

Hi Frank
I think as of today this is not possible. You could try and experience the 
"non-exlusive" feature of node-label which will officially come in 2.8 soon. Or 
you can try it in "Hadoop 3 alpha1" release too if its fine to check. 
YARN-3214 has the details for 
the nodelabel sharing concept.

Thanks
Sunil

On Wed, Oct 5, 2016 at 8:14 PM Frank Luo 
> wrote:
Sunil, thanks for responding.

So is there any way to dedicate one kind of jobs to certain machines, then 
having those machines be shared if no dedicated job running?

From: Sunil Govind 
[mailto:sunil.gov...@gmail.com]
Sent: Wednesday, October 05, 2016 12:50 AM
To: Frank Luo >; 
user@hadoop.apache.org; 
u...@yarn.apache.org

Subject: Re: how to add a shareable node label?

Hi Frank,

As far as I checked, all labels are "exclusive" in 2.7. In upcoming 2.8 
release, we can get "non-exclusive" or sharable node labels.

Thanks
Sunil

On Wed, Oct 5, 2016 at 8:40 AM Frank Luo 
> wrote:
I am using Hadoop 2.7.3, when I run:
$ yarn rmadmin -addToClusterNodeLabels "Label1(exclusive=false)"

I got an error as:

… addToClusterNodeLabels: java.io.IOException: label name should only contains 
{0-9, a-z, A-Z, -, _} and should not started with {-,_}

If I just use “Label1”, it will work fine, but I want a shareable one.

Anyone knows a better way to do it?

Access the Q2 2016 Digital Marketing Report for a fresh set of trends and 
benchmarks in digital 
marketing

Download our latest report titled “The Case for Change: Exploring the Myths of 
Customer-Centric Transformation” 


This email and any attachments transmitted with it are intended for use by the 
intended recipient(s) only. If you have received this email in error, please 
notify the sender immediately and then delete it. If you are not the intended 
recipient, you must not keep, use, disclose, copy or distribute this email 
without the author’s prior permission. We take precautions to minimize the risk 
of transmitting software viruses, but we advise you to perform your own virus 
checks on any attachment to this message. We cannot accept liability for any 
loss or damage caused by software viruses. The information contained in this 

RE: how to add a shareable node label?

[Frank Luo]

Thanks Sunil.


Ø  3. If there is any future ask for those resources , we will preempt the non 
labeled apps and give them back to labeled apps.

Unfortunately, I am still not able to use it, because of the preemptive 
behavior. The jobs that steals labelled resources are not preemptable, and I’d 
rather waiting instead of killing.

From: Sunil Govind [mailto:sunil.gov...@gmail.com]
Sent: Thursday, October 06, 2016 1:59 AM
To: Frank Luo ; user@hadoop.apache.org
Subject: Re: how to add a shareable node label?

Hi Frank
I think as of today this is not possible. You could try and experience the 
"non-exlusive" feature of node-label which will officially come in 2.8 soon. Or 
you can try it in "Hadoop 3 alpha1" release too if its fine to check. 
YARN-3214 has the details for 
the nodelabel sharing concept.

Thanks
Sunil

On Wed, Oct 5, 2016 at 8:14 PM Frank Luo 
> wrote:
Sunil, thanks for responding.

So is there any way to dedicate one kind of jobs to certain machines, then 
having those machines be shared if no dedicated job running?

From: Sunil Govind 
[mailto:sunil.gov...@gmail.com]
Sent: Wednesday, October 05, 2016 12:50 AM
To: Frank Luo >; 
user@hadoop.apache.org; 
u...@yarn.apache.org

Subject: Re: how to add a shareable node label?

Hi Frank,

As far as I checked, all labels are "exclusive" in 2.7. In upcoming 2.8 
release, we can get "non-exclusive" or sharable node labels.

Thanks
Sunil

On Wed, Oct 5, 2016 at 8:40 AM Frank Luo 
> wrote:
I am using Hadoop 2.7.3, when I run:
$ yarn rmadmin -addToClusterNodeLabels "Label1(exclusive=false)"

I got an error as:

… addToClusterNodeLabels: java.io.IOException: label name should only contains 
{0-9, a-z, A-Z, -, _} and should not started with {-,_}

If I just use “Label1”, it will work fine, but I want a shareable one.

Anyone knows a better way to do it?

Access the Q2 2016 Digital Marketing Report for a fresh set of trends and 
benchmarks in digital 
marketing

Download our latest report titled “The Case for Change: Exploring the Myths of 
Customer-Centric Transformation” 


This email and any attachments transmitted with it are intended for use by the 
intended recipient(s) only. If you have received this email in error, please 
notify the sender immediately and then delete it. If you are not the intended 
recipient, you must not keep, use, disclose, copy or distribute this email 
without the author’s prior permission. We take precautions to minimize the risk 
of transmitting software viruses, but we advise you to perform your own virus 
checks on any attachment to this message. We cannot accept liability for any 
loss or damage caused by software viruses. The information contained in this 
communication may be confidential and may be subject to the attorney-client 
privilege.

Access the Q2 2016 Digital Marketing Report for a fresh set of trends and 
benchmarks in digital 
marketing

Download our latest report titled “The Case for Change: Exploring the Myths of 
Customer-Centric Transformation” 


This email and any attachments transmitted with it are intended for use by the 
intended recipient(s) only. If you have received this email in error, please 
notify the sender immediately and then delete it. If you are not the intended 
recipient, you must not keep, use, disclose, copy or distribute this email 
without the author’s prior permission. We take precautions to minimize the risk 
of transmitting software viruses, but we advise you to perform your own virus 
checks on any attachment to this message. We cannot accept liability for any 
loss or damage caused by software viruses. The information contained in this 
communication may be confidential and may be subject to the attorney-client 
privilege.

Access the Q2 2016 Digital Marketing Report for a fresh set of trends and 
benchmarks in digital 
marketing

Download our latest report titled “The Case for Change: Exploring the Myths of 
Customer-Centric Transformation” 


This email and any attachments transmitted with it are intended for use by the 
intended recipient(s) only. If you have received this email in error, please 
notify the sender immediately and then delete it. If you are not the intended 
recipient, you must not keep, use, disclose, copy or distribute this email 
without the author’s prior permission. We take precautions to minimize the risk 
of transmitting software viruses, but 

Hadoop: precomputing data

[venito camelas]

I'm designing a prototype using *Hadoop* for video processing to do face
recognition. I thought of 2 ways of doing it.

*Approach 1:*

I was thinking of doing something in 2 steps:

   1. A map that receives frames and if a face is found it gets stored for
   the next step.
   2. A map that receives the frames from step 1 (all frames containing 1
   face at least) and does face recognition.

Step 1 would be ran only once while step 2 runs every time I want recognize
a new face.


*Approach 2:*

The other approach I thought about is to do face recognition to all the
data every time

The first approach saves time because I don't have to process faceless
frames every time I want to do face recognition, it also uses more disk
space (and it could be a lot of space).


I'm not sure whats better. Is it a bad thing to leave that precomputed
frames there forever?

S3AFileSystem & read-after-write consistency

[Dave Maughan]

Hi,

I'm investigating S3's read-after-write consistency model with
S3AFileSystem and something is not quite clear to me, so I'm hoping someone
more knowledgeable can clarify it for me.

Amazon state (
http://docs.aws.amazon.com/AmazonS3/latest/dev/Introduction.html):

*"Amazon S3 provides read-after-write consistency for PUTS of new
objects in your S3 bucket in all regions with one caveat. **The caveat is
that if you make a HEAD or GET request to the key name (to find if the
object exists) before creating the object, Amazon S3 provides eventual
consistency for read-after-write".*

In S3FileSystem, create -> exists -> getFileStatus ->
AmazonS3Client.getObjectMetadata (HEAD).

Does this mean that currently, S3AFileSystem cannot take advantage of S3's
read-after-write consistency?

Thanks
- Dave

Block Replace Operation in Mover/Balancer

[Senthil Kumar]

Hi All ,  Recently we have upgraded our Cluster to Hadoop 2.7.1 version.



When i try to run Mover , it throws  below exception :

2016-10-06 00:24:44,643 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: opCopyBlock
BP-1267816582-10.114.118.11-1392844883031:blk_12046985669_160995799
received exception java.net.SocketException: Original Exception :
java.io.IOException: Connection reset by peer
2016-10-06 00:31:49,846 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: opCopyBlock
BP-1267816582-10.114.118.11-1392844883031:blk_12046985669_160995799
received exception java.net.SocketException: Original Exception :
java.io.IOException: Connection reset by peer
2016-10-06 02:31:37,153 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: opCopyBlock
BP-1267816582-10.114.118.11-1392844883031:blk_12046985669_160995799
received exception java.net.SocketException: Original Exception :
java.io.IOException: Broken pipe
2016-10-06 02:43:10,751 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Copied
BP-1267816582-10.114.118.11-1392844883031:blk_12046985669_160995799 to /
10.115.70.28:35730
2016-10-06 02:54:18,866 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: opCopyBlock
BP-1267816582-10.114.118.11-1392844883031:blk_12046985669_160995799
received
exception java.net.SocketException: Original Exception :
java.io.IOException: Connection reset by peer



Not sure why Client closing the connection before operation completes ?.

Here is the piece of Code which actually Replace the Blocks in Mover ..
/** Send a reportedBlock replace request to the output stream */
private void sendRequest(DataOutputStream out, ExtendedBlock eb,
Token accessToken) throws IOException {
  new Sender(out).replaceBlock(eb, target.storageType, accessToken,
  source.getDatanodeInfo().getDatanodeUuid(), proxySource.datanode);
}


Did anyone face this problem while running mover or balancer ??

--Senthil

Newbie Ambari Question

[Deepak Goel]

Hey

Namaskara~Nalama~Guten Tag~Bonjour

Sorry, Is this the right forum for asking a question "Ambari Hadoop
Installation" from Hortonworks?

Thanks
Deepak
   --
Keigu

Deepak
73500 12833
www.simtree.net, dee...@simtree.net
deic...@gmail.com

LinkedIn: www.linkedin.com/in/deicool
Skype: thumsupdeicool
Google talk: deicool
Blog: http://loveandfearless.wordpress.com
Facebook: http://www.facebook.com/deicool

"Contribute to the world, environment and more : http://www.gridrepublic.org
"

Re: how to add a shareable node label?

[Sunil Govind]

Hi Frank
I think as of today this is not possible. You could try and experience the
"non-exlusive" feature of node-label which will officially come in 2.8
soon. Or you can try it in "Hadoop 3 alpha1" release too if its fine to
check. YARN-3214  has the
details for the nodelabel sharing concept.

Thanks
Sunil

On Wed, Oct 5, 2016 at 8:14 PM Frank Luo  wrote:

> Sunil, thanks for responding.
>
>
>
> So is there any way to dedicate one kind of jobs to certain machines, then
> having those machines be shared if no dedicated job running?
>
>
>
> *From:* Sunil Govind [mailto:sunil.gov...@gmail.com]
> *Sent:* Wednesday, October 05, 2016 12:50 AM
> *To:* Frank Luo ; user@hadoop.apache.org;
> u...@yarn.apache.org
>
>
> *Subject:* Re: how to add a shareable node label?
>
>
>
> Hi Frank,
>
>
>
> As far as I checked, all labels are "exclusive" in 2.7. In upcoming 2.8
> release, we can get "non-exclusive" or sharable node labels.
>
>
>
> Thanks
>
> Sunil
>
>
>
> On Wed, Oct 5, 2016 at 8:40 AM Frank Luo  wrote:
>
> I am using Hadoop 2.7.3, when I run:
>
> $ yarn rmadmin -addToClusterNodeLabels "Label1(exclusive=false)"
>
>
>
> I got an error as:
>
> … addToClusterNodeLabels: java.io.IOException: label name should only
> contains {0-9, a-z, A-Z, -, _} and should not started with {-,_}
>
>
>
> If I just use “Label1”, it will work fine, but I want a shareable one.
>
>
>
> Anyone knows a better way to do it?
>
> *Access the Q2 2016 Digital Marketing Report for a fresh set of trends and
> benchmarks in digital marketing*
> 
>
> *Download our latest report titled “The Case for Change: Exploring the
> Myths of Customer-Centric Transformation” *
> 
>
> This email and any attachments transmitted with it are intended for use by
> the intended recipient(s) only. If you have received this email in error,
> please notify the sender immediately and then delete it. If you are not the
> intended recipient, you must not keep, use, disclose, copy or distribute
> this email without the author’s prior permission. We take precautions to
> minimize the risk of transmitting software viruses, but we advise you to
> perform your own virus checks on any attachment to this message. We cannot
> accept liability for any loss or damage caused by software viruses. The
> information contained in this communication may be confidential and may be
> subject to the attorney-client privilege.
>
> *Access the Q2 2016 Digital Marketing Report for a fresh set of trends and
> benchmarks in digital marketing*
> 
>
> *Download our latest report titled “The Case for Change: Exploring the
> Myths of Customer-Centric Transformation” *
> 
>
> This email and any attachments transmitted with it are intended for use by
> the intended recipient(s) only. If you have received this email in error,
> please notify the sender immediately and then delete it. If you are not the
> intended recipient, you must not keep, use, disclose, copy or distribute
> this email without the author’s prior permission. We take precautions to
> minimize the risk of transmitting software viruses, but we advise you to
> perform your own virus checks on any attachment to this message. We cannot
> accept liability for any loss or damage caused by software viruses. The
> information contained in this communication may be confidential and may be
> subject to the attorney-client privilege.
>