[jira] [Commented] (MAPREDUCE-4680) Job history cleaner should only check timestamps of files in old enough directories
[
https://issues.apache.org/jira/browse/MAPREDUCE-4680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766152#comment-13766152
]
Hadoop QA commented on MAPREDUCE-4680:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12602936/MAPREDUCE-4680.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4000//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4000//console
This message is automatically generated.
> Job history cleaner should only check timestamps of files in old enough
> directories
> ---
>
> Key: MAPREDUCE-4680
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4680
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.0.0-alpha
>Reporter: Sandy Ryza
>Assignee: Robert Kanter
> Attachments: MAPREDUCE-4680.patch, MAPREDUCE-4680.patch
>
>
> Job history files are stored in /mm/dd folders. Currently, the job
> history cleaner checks the modification date of each file in every one of
> these folders to see whether it's past the maximum age. The load on HDFS
> could be reduced by only checking the ages of files in directories that are
> old enough, as determined by their name.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4680) Job history cleaner should only check timestamps of files in old enough directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-4680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-4680: - Attachment: MAPREDUCE-4680.patch New patch suppresses the 3 new javac warnings (caused by a test) and fixes the test failure. > Job history cleaner should only check timestamps of files in old enough > directories > --- > > Key: MAPREDUCE-4680 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4680 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.0.0-alpha >Reporter: Sandy Ryza >Assignee: Robert Kanter > Attachments: MAPREDUCE-4680.patch, MAPREDUCE-4680.patch > > > Job history files are stored in /mm/dd folders. Currently, the job > history cleaner checks the modification date of each file in every one of > these folders to see whether it's past the maximum age. The load on HDFS > could be reduced by only checking the ages of files in directories that are > old enough, as determined by their name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5379) Include token tracking ids in jobconf
[
https://issues.apache.org/jira/browse/MAPREDUCE-5379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13766097#comment-13766097
]
Hadoop QA commented on MAPREDUCE-5379:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12602922/mr-5379-3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3999//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3999//console
This message is automatically generated.
> Include token tracking ids in jobconf
> -
>
> Key: MAPREDUCE-5379
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5379
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: job submission, security
>Affects Versions: 2.1.0-beta
>Reporter: Sandy Ryza
>Assignee: Karthik Kambatla
> Attachments: MAPREDUCE-5379-1.patch, MAPREDUCE-5379-2.patch,
> MAPREDUCE-5379.patch, mr-5379-3.patch
>
>
> HDFS-4680 enables audit logging delegation tokens. By storing the tracking
> ids in the job conf, we can enable tracking what files each job touches.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5379) Include token tracking ids in jobconf
[ https://issues.apache.org/jira/browse/MAPREDUCE-5379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5379: Description: HDFS-4680 enables audit logging delegation tokens. By storing the tracking ids in the job conf, we can enable tracking what files each job touches. (was: Making a job's FS delegation token ID accessible will allow external services to associate it with the file system operations it performs.) > Include token tracking ids in jobconf > - > > Key: MAPREDUCE-5379 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5379 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: job submission, security >Affects Versions: 2.1.0-beta >Reporter: Sandy Ryza >Assignee: Karthik Kambatla > Attachments: MAPREDUCE-5379-1.patch, MAPREDUCE-5379-2.patch, > MAPREDUCE-5379.patch, mr-5379-3.patch > > > HDFS-4680 enables audit logging delegation tokens. By storing the tracking > ids in the job conf, we can enable tracking what files each job touches. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5379) Include token tracking ids in jobconf
[ https://issues.apache.org/jira/browse/MAPREDUCE-5379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5379: Summary: Include token tracking ids in jobconf (was: Include FS delegation token ID in job conf) > Include token tracking ids in jobconf > - > > Key: MAPREDUCE-5379 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5379 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: job submission, security >Affects Versions: 2.1.0-beta >Reporter: Sandy Ryza >Assignee: Karthik Kambatla > Attachments: MAPREDUCE-5379-1.patch, MAPREDUCE-5379-2.patch, > MAPREDUCE-5379.patch, mr-5379-3.patch > > > Making a job's FS delegation token ID accessible will allow external services > to associate it with the file system operations it performs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-5379) Include FS delegation token ID in job conf
[ https://issues.apache.org/jira/browse/MAPREDUCE-5379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla reassigned MAPREDUCE-5379: --- Assignee: Karthik Kambatla (was: Sandy Ryza) > Include FS delegation token ID in job conf > -- > > Key: MAPREDUCE-5379 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5379 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: job submission, security >Affects Versions: 2.1.0-beta >Reporter: Sandy Ryza >Assignee: Karthik Kambatla > Attachments: MAPREDUCE-5379-1.patch, MAPREDUCE-5379-2.patch, > MAPREDUCE-5379.patch, mr-5379-3.patch > > > Making a job's FS delegation token ID accessible will allow external services > to associate it with the file system operations it performs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5379) Include FS delegation token ID in job conf
[ https://issues.apache.org/jira/browse/MAPREDUCE-5379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5379: Attachment: mr-5379-3.patch Taking this over. The patch uses the tracking ids introduced in HDFS-4680 for tracking the file system operations the job performs. > Include FS delegation token ID in job conf > -- > > Key: MAPREDUCE-5379 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5379 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: job submission, security >Affects Versions: 2.1.0-beta >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5379-1.patch, MAPREDUCE-5379-2.patch, > MAPREDUCE-5379.patch, mr-5379-3.patch > > > Making a job's FS delegation token ID accessible will allow external services > to associate it with the file system operations it performs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4421) Remove dependency on deployed MR jars
[ https://issues.apache.org/jira/browse/MAPREDUCE-4421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13765961#comment-13765961 ] Hitesh Shah commented on MAPREDUCE-4421: s/Configuration/Jobconf/ in the previous comment. > Remove dependency on deployed MR jars > - > > Key: MAPREDUCE-4421 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4421 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 2.0.0-alpha >Reporter: Arun C Murthy >Assignee: Jason Lowe > Attachments: MAPREDUCE-4421.patch, MAPREDUCE-4421.patch > > > Currently MR AM depends on MR jars being deployed on all nodes via implicit > dependency on YARN_APPLICATION_CLASSPATH. > We should stop adding mapreduce jars to YARN_APPLICATION_CLASSPATH and, > probably, just rely on adding a shaded MR jar along with job.jar to the > dist-cache. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5332) Support token-preserving restart of history server
[
https://issues.apache.org/jira/browse/MAPREDUCE-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13765935#comment-13765935
]
Hadoop QA commented on MAPREDUCE-5332:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12602867/MAPREDUCE-5332-5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapreduce.TestMRJobClient
The following test timeouts occurred in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapreduce.v2.TestUberAM
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3998//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3998//console
This message is automatically generated.
> Support token-preserving restart of history server
> --
>
> Key: MAPREDUCE-5332
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5332
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: jobhistoryserver
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: MAPREDUCE-5332-2.patch, MAPREDUCE-5332-3.patch,
> MAPREDUCE-5332-4.patch, MAPREDUCE-5332-5.patch, MAPREDUCE-5332-5.patch,
> MAPREDUCE-5332.patch
>
>
> To better support rolling upgrades through a cluster, the history server
> needs the ability to restart without losing track of delegation tokens.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5164) command "mapred job" and "mapred queue" omit HADOOP_CLIENT_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13765503#comment-13765503 ] Hudson commented on MAPREDUCE-5164: --- SUCCESS: Integrated in Hadoop-trunk-Commit #4403 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4403/]) MAPREDUCE-5164. mapred job and queue commands omit HADOOP_CLIENT_OPTS. Contributed by Nemon Lou. (devaraj: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1522595) * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred * /hadoop/common/trunk/hadoop-mapreduce-project/bin/mapred.cmd > command "mapred job" and "mapred queue" omit HADOOP_CLIENT_OPTS > - > > Key: MAPREDUCE-5164 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5164 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.3-alpha >Reporter: Nemon Lou >Assignee: Nemon Lou > Fix For: 2.1.1-beta > > Attachments: MAPREDUCE-5164.patch, MAPREDUCE-5164.patch, > MAPREDUCE-5164.patch, MAPREDUCE-5164.patch > > > HADOOP_CLIENT_OPTS does not take effect when type "mapred job -list" and > "mapred queue -list". > The mapred script omit it -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4421) Remove dependency on deployed MR jars
[ https://issues.apache.org/jira/browse/MAPREDUCE-4421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13765889#comment-13765889 ] Hitesh Shah commented on MAPREDUCE-4421: [~jlowe] Had a few questions/comments related to the implementation/patch: - Why does classpath need to include all of common, hdfs and yarn jar locations? Assuming that MR is running on a YARN-based cluster, shouldn't the location of the core dependencies come from the cluster deployment i.e. via the env that the NM sets for a container. I believe the only jars that MR should have in its uploaded tarball should be the client jars. I understand that there is no clear boundary for client-side only jars for common and hdfs today ( for For YARN, I believe it should be simple to split out the client-side requirements ) but it is something we should aim for or assume that the jars deployed on the cluster are compatible. - I guess the underlying question is why use the full hadoop tarball and not just the mapreduce-only tarball? If MR is trully a user-land library, it should be treated as such and have a separate deployment approach. - I would vote to make the tar-ball in HDFS be the only way to run MR on YARN. Obviously, this cannot be done for 2.x but we should move to this model on trunk and not support the current approach at all there. Comments? - The other point is related to configs. Configuration still loads mapred-site and mapred-default files and new Configuration objects are created on the cluster. Are these files still expected on the cluster? job.xml does override these but cluster configs could still have final params. If this is meant to be addressed in a follow-up jira to ensure all MR configs come from the client, you can ignore this point for now. - How do you see framework name extracted from the path to be used? Is it just a safety check to ensure that it is found in the classpath? Will it have any relation to a version? A minor nit - framework name seems confusing in relation to the framework name in use from earlier i.e yarn vs local framework. - Description in the default-xml for mapreduce.application.framework.path does not mention the need for the URI fragment and how the fragment is used as a sanity check to the classpath. - Regarding versions, it seems like users will need to do 2 things. Change the location of the tarball on HDFS and modify the classpath. Users will need to know the exact structure of the classpath. In such a scenario, do defaults even make sense? On the other hand, if we define a common standard i.e. a base path for all MR tarballs, with each tarball in a defined structure ( possibly with version info added on later on for the code to infer the structure of the tarball ), all the user would need to do is specify the base path ( which could have a default value ) and a version which again has a default value. The latter approach would require the code to construct the necessary classpath if the upload path is in use. Do you have any comments on which of the 2 approaches makes more sense? The former is way more flexible but a bit more complex. The latter brittle/inflexible with respect to changing tarball structures but likely more easier to enforce a standard on. > Remove dependency on deployed MR jars > - > > Key: MAPREDUCE-4421 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4421 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 2.0.0-alpha >Reporter: Arun C Murthy >Assignee: Jason Lowe > Attachments: MAPREDUCE-4421.patch, MAPREDUCE-4421.patch > > > Currently MR AM depends on MR jars being deployed on all nodes via implicit > dependency on YARN_APPLICATION_CLASSPATH. > We should stop adding mapreduce jars to YARN_APPLICATION_CLASSPATH and, > probably, just rely on adding a shaded MR jar along with job.jar to the > dist-cache. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5332) Support token-preserving restart of history server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-5332: -- Attachment: MAPREDUCE-5332-5.patch Wow, that's a lot of test breakage. None of the test failures appear to be related to this change. Many of them are failing with OOM errors due to too many threads, suspect this is caused by lingering AMs like what was reported in MAPREDUCE-5501 and YARN-1183. Also, I'm able to reproduce many of the failures on trunk without this patch. Uploading the same patch again to see if we can get a clean(er) run this time. > Support token-preserving restart of history server > -- > > Key: MAPREDUCE-5332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5332 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobhistoryserver >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: MAPREDUCE-5332-2.patch, MAPREDUCE-5332-3.patch, > MAPREDUCE-5332-4.patch, MAPREDUCE-5332-5.patch, MAPREDUCE-5332-5.patch, > MAPREDUCE-5332.patch > > > To better support rolling upgrades through a cluster, the history server > needs the ability to restart without losing track of delegation tokens. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4680) Job history cleaner should only check timestamps of files in old enough directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-4680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-4680: - Status: Patch Available (was: Open) > Job history cleaner should only check timestamps of files in old enough > directories > --- > > Key: MAPREDUCE-4680 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4680 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.0.0-alpha >Reporter: Sandy Ryza >Assignee: Robert Kanter > Attachments: MAPREDUCE-4680.patch > > > Job history files are stored in /mm/dd folders. Currently, the job > history cleaner checks the modification date of each file in every one of > these folders to see whether it's past the maximum age. The load on HDFS > could be reduced by only checking the ages of files in directories that are > old enough, as determined by their name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4680) Job history cleaner should only check timestamps of files in old enough directories
[ https://issues.apache.org/jira/browse/MAPREDUCE-4680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Kanter updated MAPREDUCE-4680: - Attachment: MAPREDUCE-4680.patch The patch makes it now only gather the directories that could possibly be old enough. > Job history cleaner should only check timestamps of files in old enough > directories > --- > > Key: MAPREDUCE-4680 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4680 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 2.0.0-alpha >Reporter: Sandy Ryza >Assignee: Robert Kanter > Attachments: MAPREDUCE-4680.patch > > > Job history files are stored in /mm/dd folders. Currently, the job > history cleaner checks the modification date of each file in every one of > these folders to see whether it's past the maximum age. The load on HDFS > could be reduced by only checking the ages of files in directories that are > old enough, as determined by their name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5332) Support token-preserving restart of history server
[
https://issues.apache.org/jira/browse/MAPREDUCE-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13765651#comment-13765651
]
Hadoop QA commented on MAPREDUCE-5332:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12602813/MAPREDUCE-5332-5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
org.apache.hadoop.mapred.TestMiniMRClasspath
org.apache.hadoop.mapred.TestLazyOutput
org.apache.hadoop.mapred.TestMiniMRChildTask
org.apache.hadoop.mapred.TestJobSysDirWithDFS
org.apache.hadoop.mapreduce.TestMRJobClient
org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
The following test timeouts occurred in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:
org.apache.hadoop.mapreduce.v2.TestUberAM
org.apache.hadoop.mapred.TestClusterMapReduceTestCase
org.apache.hadoop.mapred.TestMerge
org.apache.hadoop.mapred.TestMiniMRClientCluster
org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
org.apache.hadoop.mapred.TestReduceFetch
org.apache.hadoop.mapred.TestJobName
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3995//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3995//console
This message is automatically generated.
> Support token-preserving restart of history server
> --
>
> Key: MAPREDUCE-5332
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5332
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: jobhistoryserver
>Reporter: Jason Lowe
>Assignee: Jason Lowe
> Attachments: MAPREDUCE-5332-2.patch, MAPREDUCE-5332-3.patch,
> MAPREDUCE-5332-4.patch, MAPREDUCE-5332-5.patch, MAPREDUCE-5332.patch
>
>
> To better support rolling upgrades through a cluster, the history server
> needs the ability to restart without losing track of delegation tokens.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5329) APPLICATION_INIT is never sent to AuxServices other than the builtin ShuffleHandler
[ https://issues.apache.org/jira/browse/MAPREDUCE-5329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avner BenHanoch updated MAPREDUCE-5329: --- Fix Version/s: trunk Target Version/s: trunk, 2.1.1-beta Affects Version/s: (was: 2.0.4-alpha) 2.1.0-beta 2.0.6-alpha Status: Patch Available (was: Open) > APPLICATION_INIT is never sent to AuxServices other than the builtin > ShuffleHandler > --- > > Key: MAPREDUCE-5329 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5329 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.6-alpha, 2.1.0-beta >Reporter: Avner BenHanoch > Fix For: trunk > > Attachments: MAPREDUCE-5329.patch > > > APPLICATION_INIT is never sent to AuxServices other than the built-in > ShuffleHandler. This means that 3rd party ShuffleProvider(s) will not be > able to function, because APPLICATION_INIT enables the AuxiliaryService to > map jobId->userId. This is needed for properly finding the MOFs of a job per > reducers' requests. > NOTE: The built-in ShuffleHandler does get APPLICATION_INIT events due to > hard-coded expression in hadoop code. The current TaskAttemptImpl.java code > explicitly call: serviceData.put (ShuffleHandler.MAPREDUCE_SHUFFLE_SERVICEID, > ...) and ignores any additional AuxiliaryService. As a result, only the > built-in ShuffleHandler will get APPLICATION_INIT events. Any 3rd party > AuxillaryService will never get APPLICATION_INIT events. > I think a solution can be in one of two ways: > 1. Change TaskAttemptImpl.java to loop on all Auxiliary Services and register > each of them, by calling serviceData.put (…) in loop. > 2. Change AuxServices.java similar to the fix in: MAPREDUCE-2668 > "APPLICATION_STOP is never sent to AuxServices". This means that in case the > 'handle' method gets APPLICATION_INIT event it will demultiplex it to all Aux > Services regardless of the value in event.getServiceID(). > I prefer the 2nd solution. I am welcoming any ideas. I can provide the > needed patch for any option that people like. > See [Pluggable Shuffle in Hadoop > documentation|http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5329) APPLICATION_INIT is never sent to AuxServices other than the builtin ShuffleHandler
[
https://issues.apache.org/jira/browse/MAPREDUCE-5329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13765570#comment-13765570
]
Hadoop QA commented on MAPREDUCE-5329:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12602814/MAPREDUCE-5329.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3996//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3996//console
This message is automatically generated.
> APPLICATION_INIT is never sent to AuxServices other than the builtin
> ShuffleHandler
> ---
>
> Key: MAPREDUCE-5329
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5329
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
>Affects Versions: 2.1.0-beta, 2.0.6-alpha
>Reporter: Avner BenHanoch
> Fix For: trunk
>
> Attachments: MAPREDUCE-5329.patch
>
>
> APPLICATION_INIT is never sent to AuxServices other than the built-in
> ShuffleHandler. This means that 3rd party ShuffleProvider(s) will not be
> able to function, because APPLICATION_INIT enables the AuxiliaryService to
> map jobId->userId. This is needed for properly finding the MOFs of a job per
> reducers' requests.
> NOTE: The built-in ShuffleHandler does get APPLICATION_INIT events due to
> hard-coded expression in hadoop code. The current TaskAttemptImpl.java code
> explicitly call: serviceData.put (ShuffleHandler.MAPREDUCE_SHUFFLE_SERVICEID,
> ...) and ignores any additional AuxiliaryService. As a result, only the
> built-in ShuffleHandler will get APPLICATION_INIT events. Any 3rd party
> AuxillaryService will never get APPLICATION_INIT events.
> I think a solution can be in one of two ways:
> 1. Change TaskAttemptImpl.java to loop on all Auxiliary Services and register
> each of them, by calling serviceData.put (…) in loop.
> 2. Change AuxServices.java similar to the fix in: MAPREDUCE-2668
> "APPLICATION_STOP is never sent to AuxServices". This means that in case the
> 'handle' method gets APPLICATION_INIT event it will demultiplex it to all Aux
> Services regardless of the value in event.getServiceID().
> I prefer the 2nd solution. I am welcoming any ideas. I can provide the
> needed patch for any option that people like.
> See [Pluggable Shuffle in Hadoop
> documentation|http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html]
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5329) APPLICATION_INIT is never sent to AuxServices other than the builtin ShuffleHandler
[ https://issues.apache.org/jira/browse/MAPREDUCE-5329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avner BenHanoch updated MAPREDUCE-5329: --- Attachment: MAPREDUCE-5329.patch > APPLICATION_INIT is never sent to AuxServices other than the builtin > ShuffleHandler > --- > > Key: MAPREDUCE-5329 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5329 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am >Affects Versions: 2.0.4-alpha >Reporter: Avner BenHanoch > Attachments: MAPREDUCE-5329.patch > > > APPLICATION_INIT is never sent to AuxServices other than the built-in > ShuffleHandler. This means that 3rd party ShuffleProvider(s) will not be > able to function, because APPLICATION_INIT enables the AuxiliaryService to > map jobId->userId. This is needed for properly finding the MOFs of a job per > reducers' requests. > NOTE: The built-in ShuffleHandler does get APPLICATION_INIT events due to > hard-coded expression in hadoop code. The current TaskAttemptImpl.java code > explicitly call: serviceData.put (ShuffleHandler.MAPREDUCE_SHUFFLE_SERVICEID, > ...) and ignores any additional AuxiliaryService. As a result, only the > built-in ShuffleHandler will get APPLICATION_INIT events. Any 3rd party > AuxillaryService will never get APPLICATION_INIT events. > I think a solution can be in one of two ways: > 1. Change TaskAttemptImpl.java to loop on all Auxiliary Services and register > each of them, by calling serviceData.put (…) in loop. > 2. Change AuxServices.java similar to the fix in: MAPREDUCE-2668 > "APPLICATION_STOP is never sent to AuxServices". This means that in case the > 'handle' method gets APPLICATION_INIT event it will demultiplex it to all Aux > Services regardless of the value in event.getServiceID(). > I prefer the 2nd solution. I am welcoming any ideas. I can provide the > needed patch for any option that people like. > See [Pluggable Shuffle in Hadoop > documentation|http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5332) Support token-preserving restart of history server
[ https://issues.apache.org/jira/browse/MAPREDUCE-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-5332: -- Attachment: MAPREDUCE-5332-5.patch Updating the patch to use temporary files when creating key and token files. This prevents the recovery from seeing a partially-written file if we crash in the middle of a write. Also extended the unit tests to check for correct behavior on redundant key and token stores. > Support token-preserving restart of history server > -- > > Key: MAPREDUCE-5332 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5332 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: jobhistoryserver >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: MAPREDUCE-5332-2.patch, MAPREDUCE-5332-3.patch, > MAPREDUCE-5332-4.patch, MAPREDUCE-5332-5.patch, MAPREDUCE-5332.patch > > > To better support rolling upgrades through a cluster, the history server > needs the ability to restart without losing track of delegation tokens. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5164) command "mapred job" and "mapred queue" omit HADOOP_CLIENT_OPTS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-5164: - Resolution: Fixed Fix Version/s: 2.1.1-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk, branch-2.1-beta and branch-2. Thanks Nemon Lou. > command "mapred job" and "mapred queue" omit HADOOP_CLIENT_OPTS > - > > Key: MAPREDUCE-5164 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5164 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.3-alpha >Reporter: Nemon Lou >Assignee: Nemon Lou > Fix For: 2.1.1-beta > > Attachments: MAPREDUCE-5164.patch, MAPREDUCE-5164.patch, > MAPREDUCE-5164.patch, MAPREDUCE-5164.patch > > > HADOOP_CLIENT_OPTS does not take effect when type "mapred job -list" and > "mapred queue -list". > The mapred script omit it -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
