Cpu Metric in JobHistory logs processed with Rumen

2016-11-30 Thread Tomas Delvechio 
Hello, I gathered stats of my jobs in a Hadoop Cluster.  I took the
JobHistory logs and process them with Rumen. In the json file, for
each task attemp, there are a field named "cpuUsages".

Example:

"cpuUsages" : [ 6028, 3967, 3597, 3354, 3225, 3454, 3589, 4316, 42632,
102, 103, 103 ]

I need to know the unit of measurement of this numbers. Some official
or academic reference for that?

thanks for advance.


-- 
Tomas Delvechio

-
To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: user-h...@hadoop.apache.org

IllegalArgumentException: /tmp is a symbolic link to a directory STATUS: 500

2016-11-30 Thread kelly chen 
Hi,
I used AWS EMR to create a hadoop vm and do my work.I got the following error:
com.google.common.util.concurrent.UncheckedExecutionException: 
java.lang.IllegalArgumentException: /tmp is a symbolic link to a directory 
STATUS: 500
The /tmp directory in an AWS EMR EC2 instance is a symbolic link and caused 
this error.
>From my google search:From 
>http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/core-default.html"Setting
> this to false causes an exception to be thrown upon encountering a symlink." 
>However, the default value is already set to true.Appreciate anyone facing 
>this issue can share their experience. Thanks! Kelly

Save the date: ApacheCon Miami, May 15-19, 2017

2016-11-30 Thread Rich Bowen 
Dear Apache enthusiast,

ApacheCon and Apache Big Data will be held at the Intercontinental in
Miami, Florida, May 16-18, 2017. Submit your talks, and register, at
http://apachecon.com/  Talks aimed at the Big Data section of the event
should go to
http://events.linuxfoundation.org/events/apache-big-data-north-america/program/cfp
while other talks should go to
http://events.linuxfoundation.org/events/apachecon-north-america/program/cfp


ApacheCon is the best place to meet the people that develop the software
that you use and rely on. It’s also a great opportunity to deepen your
involvement in the project, and perhaps make the leap to contributing.
And we find that user case studies, showcasing how you use Apache
projects to solve real world problems, are very popular at this event.
So, do consider whether you have a use case that might make a good
presentation.

ApacheCon will have many different ways that you can participate:

Technical Content: We’ll have three days of technical sessions covering
many of the projects at the ASF. We’ll be publishing a schedule of talks
on March 9th, so that you can plan what you’ll be attending

BarCamp: The Apache BarCamp is a standard feature of ApacheCon - an
un-conference style event, where the schedule is determined on-site by
the attendees, and anything is fair game.

Lightning Talks: Even if you don’t give a full-length talk, the
Lightning Talks are five minute presentations on any topic related to
the ASF, and can be given by any attendee. If there’s something you’re
passionate about, consider giving a Lightning Talk.

Sponsor: It costs money to put on a conference, and this is a great
opportunity for companies involved in Apache projects, or who benefit
from Apache code - your employers - to get their name and products in
front of the community. Sponsors can start any any monetary level, and
can sponsor everything from the conference badge lanyard, through larger
items such as video recordings and evening events. For more information
on sponsoring ApacheCon, see http://apachecon.com/sponsor/

So, get your tickets today at http://apachecon.com/ and submit your
talks. ApacheCon Miami is going to be our best ApacheCon yet, and you,
and your project, can’t afford to miss it.

-- 
Rich Bowen - rbo...@apache.org
VP, Conferences
http://apachecon.com
@apachecon


-
To unsubscribe, e-mail: hdfs-user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-user-h...@hadoop.apache.org

Save the date: ApacheCon Miami, May 15-19, 2017

2016-11-30 Thread Rich Bowen 
Dear Apache enthusiast,

ApacheCon and Apache Big Data will be held at the Intercontinental in
Miami, Florida, May 16-18, 2017. Submit your talks, and register, at
http://apachecon.com/  Talks aimed at the Big Data section of the event
should go to
http://events.linuxfoundation.org/events/apache-big-data-north-america/program/cfp
while other talks should go to
http://events.linuxfoundation.org/events/apachecon-north-america/program/cfp


ApacheCon is the best place to meet the people that develop the software
that you use and rely on. It’s also a great opportunity to deepen your
involvement in the project, and perhaps make the leap to contributing.
And we find that user case studies, showcasing how you use Apache
projects to solve real world problems, are very popular at this event.
So, do consider whether you have a use case that might make a good
presentation.

ApacheCon will have many different ways that you can participate:

Technical Content: We’ll have three days of technical sessions covering
many of the projects at the ASF. We’ll be publishing a schedule of talks
on March 9th, so that you can plan what you’ll be attending

BarCamp: The Apache BarCamp is a standard feature of ApacheCon - an
un-conference style event, where the schedule is determined on-site by
the attendees, and anything is fair game.

Lightning Talks: Even if you don’t give a full-length talk, the
Lightning Talks are five minute presentations on any topic related to
the ASF, and can be given by any attendee. If there’s something you’re
passionate about, consider giving a Lightning Talk.

Sponsor: It costs money to put on a conference, and this is a great
opportunity for companies involved in Apache projects, or who benefit
from Apache code - your employers - to get their name and products in
front of the community. Sponsors can start any any monetary level, and
can sponsor everything from the conference badge lanyard, through larger
items such as video recordings and evening events. For more information
on sponsoring ApacheCon, see http://apachecon.com/sponsor/

So, get your tickets today at http://apachecon.com/ and submit your
talks. ApacheCon Miami is going to be our best ApacheCon yet, and you,
and your project, can’t afford to miss it.

-- 
Rich Bowen - rbo...@apache.org
VP, Conferences
http://apachecon.com
@apachecon


-
To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: user-h...@hadoop.apache.org

Save the date: ApacheCon Miami, May 15-19, 2017

2016-11-30 Thread Rich Bowen 
Dear Apache enthusiast,

ApacheCon and Apache Big Data will be held at the Intercontinental in
Miami, Florida, May 16-18, 2017. Submit your talks, and register, at
http://apachecon.com/  Talks aimed at the Big Data section of the event
should go to
http://events.linuxfoundation.org/events/apache-big-data-north-america/program/cfp
while other talks should go to
http://events.linuxfoundation.org/events/apachecon-north-america/program/cfp


ApacheCon is the best place to meet the people that develop the software
that you use and rely on. It’s also a great opportunity to deepen your
involvement in the project, and perhaps make the leap to contributing.
And we find that user case studies, showcasing how you use Apache
projects to solve real world problems, are very popular at this event.
So, do consider whether you have a use case that might make a good
presentation.

ApacheCon will have many different ways that you can participate:

Technical Content: We’ll have three days of technical sessions covering
many of the projects at the ASF. We’ll be publishing a schedule of talks
on March 9th, so that you can plan what you’ll be attending

BarCamp: The Apache BarCamp is a standard feature of ApacheCon - an
un-conference style event, where the schedule is determined on-site by
the attendees, and anything is fair game.

Lightning Talks: Even if you don’t give a full-length talk, the
Lightning Talks are five minute presentations on any topic related to
the ASF, and can be given by any attendee. If there’s something you’re
passionate about, consider giving a Lightning Talk.

Sponsor: It costs money to put on a conference, and this is a great
opportunity for companies involved in Apache projects, or who benefit
from Apache code - your employers - to get their name and products in
front of the community. Sponsors can start any any monetary level, and
can sponsor everything from the conference badge lanyard, through larger
items such as video recordings and evening events. For more information
on sponsoring ApacheCon, see http://apachecon.com/sponsor/

So, get your tickets today at http://apachecon.com/ and submit your
talks. ApacheCon Miami is going to be our best ApacheCon yet, and you,
and your project, can’t afford to miss it.

-- 
Rich Bowen - rbo...@apache.org
VP, Conferences
http://apachecon.com
@apachecon


-
To unsubscribe, e-mail: common-user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-user-h...@hadoop.apache.org

Save the date: ApacheCon Miami, May 15-19, 2017

2016-11-30 Thread Rich Bowen 
Dear Apache enthusiast,

ApacheCon and Apache Big Data will be held at the Intercontinental in
Miami, Florida, May 16-18, 2017. Submit your talks, and register, at
http://apachecon.com/  Talks aimed at the Big Data section of the event
should go to
http://events.linuxfoundation.org/events/apache-big-data-north-america/program/cfp
while other talks should go to
http://events.linuxfoundation.org/events/apachecon-north-america/program/cfp


ApacheCon is the best place to meet the people that develop the software
that you use and rely on. It’s also a great opportunity to deepen your
involvement in the project, and perhaps make the leap to contributing.
And we find that user case studies, showcasing how you use Apache
projects to solve real world problems, are very popular at this event.
So, do consider whether you have a use case that might make a good
presentation.

ApacheCon will have many different ways that you can participate:

Technical Content: We’ll have three days of technical sessions covering
many of the projects at the ASF. We’ll be publishing a schedule of talks
on March 9th, so that you can plan what you’ll be attending

BarCamp: The Apache BarCamp is a standard feature of ApacheCon - an
un-conference style event, where the schedule is determined on-site by
the attendees, and anything is fair game.

Lightning Talks: Even if you don’t give a full-length talk, the
Lightning Talks are five minute presentations on any topic related to
the ASF, and can be given by any attendee. If there’s something you’re
passionate about, consider giving a Lightning Talk.

Sponsor: It costs money to put on a conference, and this is a great
opportunity for companies involved in Apache projects, or who benefit
from Apache code - your employers - to get their name and products in
front of the community. Sponsors can start any any monetary level, and
can sponsor everything from the conference badge lanyard, through larger
items such as video recordings and evening events. For more information
on sponsoring ApacheCon, see http://apachecon.com/sponsor/

So, get your tickets today at http://apachecon.com/ and submit your
talks. ApacheCon Miami is going to be our best ApacheCon yet, and you,
and your project, can’t afford to miss it.

-- 
Rich Bowen - rbo...@apache.org
VP, Conferences
http://apachecon.com
@apachecon


-
To unsubscribe, e-mail: mapreduce-user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-user-h...@hadoop.apache.org

Re: yarnClient.getContainerReport exception for KILLED containers

2016-11-30 Thread AJAY GUPTA 
The following property in my cluster was set to false. This needs to be
true for being able to read data from timeline server.

  The setting that controls whether yarn system metrics is
  published on the timeline server or not by RM.
  yarn.resourcemanager.system-metrics-publisher.enabled
  true


To add, below configuration also needs to be true to retrieve data using
yarn client.

  Indicate to clients whether to query generic application
  data from timeline history-service or not. If not enabled then application
  data is queried only from Resource Manager.
  yarn.timeline-service.generic-application-history.enabled
  true


Thanks,
Ajay

On Wed, Nov 30, 2016 at 8:22 PM, AJAY GUPTA  wrote:

> Hi
>
> I am running Hadoop 2.6.0 on a cluster setup. I have setup the timeline
> server on this cluster.
> My code wants to fetch details of KILLED containers for an application. I
> have used the yarnClient.getContainerReport() call to get
> details of containers. This call throws an exception for KILLED containers,
> despite timeline server being up.
> Is there some configuration I am missing here which is resulting in this
> issue or is this expected in hadoop 2.6.0?
>
> I have a local setup with Hadoop 2.7.2. It is able to give details of
> KILLED containers without any exception.
>
> Thanks,
> Ajay
>

Hadoop 2.7.3: hosts with different hardware

2016-11-30 Thread Sebastian Rinke 
Dear all,

Is there a way to specify a different number of vcores and memory for
node managers on different hosts.

I was only able to find the property yarn.nodemanager.resource.cpu-vcores of 
Yarn,
but this setting then applies to all node managers.

Thank you for your help.

Best,
Sebastian
-
To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org
For additional commands, e-mail: user-h...@hadoop.apache.org

yarnClient.getContainerReport exception for KILLED containers

2016-11-30 Thread AJAY GUPTA 
Hi

I am running Hadoop 2.6.0 on a cluster setup. I have setup the timeline
server on this cluster.
My code wants to fetch details of KILLED containers for an application. I
have used the yarnClient.getContainerReport() call to get
details of containers. This call throws an exception for KILLED containers,
despite timeline server being up.
Is there some configuration I am missing here which is resulting in this
issue or is this expected in hadoop 2.6.0?

I have a local setup with Hadoop 2.7.2. It is able to give details of
KILLED containers without any exception.

Thanks,
Ajay