[jira] [Commented] (MAPREDUCE-5334) TestContainerLauncherImpl is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688766#comment-13688766 ] Hudson commented on MAPREDUCE-5334: --- Integrated in Hadoop-trunk-Commit #3986 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3986/]) MAPREDUCE-5334. Fix failing unit tests - TestContainerLauncher, TestContainerLauncherImpl. Contributed by Vinod Kumar Vavilapalli. (Revision 1494820) Result = SUCCESS sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494820 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncherImpl.java > TestContainerLauncherImpl is failing > > > Key: MAPREDUCE-5334 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5334 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Fix For: 2.1.0-beta > > Attachments: MAPREDUCE-5334.1.txt, MAPREDUCE-5334.2.txt, > MAPREDUCE-5334.txt > > > Broken by YARN-694, but Jenkins didn't report it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5221) Reduce side Combiner is not used when using the new API
[ https://issues.apache.org/jira/browse/MAPREDUCE-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688750#comment-13688750 ] Tsuyoshi OZAWA commented on MAPREDUCE-5221: --- [~kkambatl]: You're right. However, conf.getCombinerClass() is removed in the latest patch because CombinerRunner wraps the switch of new or old API. It's done in MAPREDUCE-5294. > Reduce side Combiner is not used when using the new API > --- > > Key: MAPREDUCE-5221 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5221 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 2.0.4-alpha >Reporter: Siddharth Seth >Assignee: Tsuyoshi OZAWA > Attachments: MAPREDUCE-5221.1.patch, MAPREDUCE-5221.2.patch, > MAPREDUCE-5221.3.patch, MAPREDUCE-5221.4.patch > > > If a combiner is specified using o.a.h.mapreduce.Job.setCombinerClass - this > will silently ignored on the reduce side since the reduce side usage is only > aware of the old api combiner. > This doesn't fail the job - since the new combiner key does not deprecate the > old key. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5334) TestContainerLauncherImpl is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated MAPREDUCE-5334: -- Resolution: Fixed Fix Version/s: 2.1.0-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk, branch-2 and branch-2.1-beta. Thanks Vinod > TestContainerLauncherImpl is failing > > > Key: MAPREDUCE-5334 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5334 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Fix For: 2.1.0-beta > > Attachments: MAPREDUCE-5334.1.txt, MAPREDUCE-5334.2.txt, > MAPREDUCE-5334.txt > > > Broken by YARN-694, but Jenkins didn't report it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5334) TestContainerLauncherImpl is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688703#comment-13688703 ] Siddharth Seth commented on MAPREDUCE-5334: --- +1. The patch looks good. > TestContainerLauncherImpl is failing > > > Key: MAPREDUCE-5334 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5334 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Attachments: MAPREDUCE-5334.1.txt, MAPREDUCE-5334.2.txt, > MAPREDUCE-5334.txt > > > Broken by YARN-694, but Jenkins didn't report it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5334) TestContainerLauncherImpl is failing
[
https://issues.apache.org/jira/browse/MAPREDUCE-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688662#comment-13688662
]
Hadoop QA commented on MAPREDUCE-5334:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588721/MAPREDUCE-5334.2.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3789//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3789//console
This message is automatically generated.
> TestContainerLauncherImpl is failing
>
>
> Key: MAPREDUCE-5334
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5334
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Vinod Kumar Vavilapalli
>Priority: Blocker
> Attachments: MAPREDUCE-5334.1.txt, MAPREDUCE-5334.2.txt,
> MAPREDUCE-5334.txt
>
>
> Broken by YARN-694, but Jenkins didn't report it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5247) FileInputFormat should filter files with '._COPYING_' sufix
[
https://issues.apache.org/jira/browse/MAPREDUCE-5247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688650#comment-13688650
]
Kousuke Saruta commented on MAPREDUCE-5247:
---
I found the jira where the code which "._COPYING_" temporary file being created
is added.
https://issues.apache.org/jira/browse/HADOOP-7771
In that jira, they discussed NPE problem when using copyToLocal and the reason
why the "._COPYING_" is created is to copy file persistently.
So, I think the temporary file is not necessarily assigned the name with
"._COPYING_" suffix.
> FileInputFormat should filter files with '._COPYING_' sufix
> ---
>
> Key: MAPREDUCE-5247
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5247
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Stan Rosenberg
>
> FsShell copy/put creates staging files with '._COPYING_' suffix. These files
> should be considered hidden by FileInputFormat. (A simple fix is to add the
> following conjunct to the existing hiddenFilter:
> {code}
> !name.endsWith("._COPYING_")
> {code}
> After upgrading to CDH 4.2.0 we encountered this bug. We have a legacy data
> loader which uses 'hadoop fs -put' to load data into hourly partitions. We
> also have intra-hourly jobs which are scheduled to execute several times per
> hour using the same hourly partition as input. Thus, as the new data is
> continuously loaded, these staging files (i.e., ._COPYING_) are breaking our
> jobs (since when copy/put completes staging files are moved).
> As a workaround, we've defined a custom input path filter and loaded it with
> "mapred.input.pathFilter.class".
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5334) TestContainerLauncherImpl is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5334: --- Attachment: MAPREDUCE-5334.2.txt Fixing the test-failure that is newly reported. > TestContainerLauncherImpl is failing > > > Key: MAPREDUCE-5334 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5334 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Attachments: MAPREDUCE-5334.1.txt, MAPREDUCE-5334.2.txt, > MAPREDUCE-5334.txt > > > Broken by YARN-694, but Jenkins didn't report it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5334) TestContainerLauncherImpl is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5334: --- Status: Patch Available (was: Open) > TestContainerLauncherImpl is failing > > > Key: MAPREDUCE-5334 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5334 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Attachments: MAPREDUCE-5334.1.txt, MAPREDUCE-5334.2.txt, > MAPREDUCE-5334.txt > > > Broken by YARN-694, but Jenkins didn't report it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5334) TestContainerLauncherImpl is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5334: --- Status: Open (was: Patch Available) > TestContainerLauncherImpl is failing > > > Key: MAPREDUCE-5334 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5334 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Attachments: MAPREDUCE-5334.1.txt, MAPREDUCE-5334.txt > > > Broken by YARN-694, but Jenkins didn't report it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2351) mapred.job.tracker.history.completed.location should support an arbitrary filesystem URI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chelsey Chang updated MAPREDUCE-2351: - Attachment: HADOOP-472.branch-1-win.3.patch Hi, Could someone please backport the newly attached patch to branch-1-win? It has pretty much the same changes as MAPREDUCE-2351.patch with a small difference. The new testcase testDoneFolderOnNotDefaultFileSystem fails on Windows because Windows default permission is not 755. So in addition to the changes in MAPREDUCE-2341.patch, the permission check for history_done_folder's "sub-directories" on Windows is skipped, given that users can still access the information via the Web-UI. > mapred.job.tracker.history.completed.location should support an arbitrary > filesystem URI > > > Key: MAPREDUCE-2351 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2351 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Tom White >Assignee: Tom White > Fix For: 0.23.0 > > Attachments: HADOOP-472.branch-1-win.3.patch, MAPREDUCE-2351.patch > > > Currently, mapred.job.tracker.history.completed.location is resolved relative > to the default filesystem. If not set it defaults to history/done in the > local log directory. There is no way to set it to another local filesystem > location (with a file:// URI) or an arbitrary Hadoop filesystem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5130) Add missing job config options to mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5130: -- Status: Patch Available (was: Open) > Add missing job config options to mapred-default.xml > > > Key: MAPREDUCE-5130 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5130 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5130-1.patch, MAPREDUCE-5130-1.patch, > MAPREDUCE-5130-2.patch, MAPREDUCE-5130-3.patch, MAPREDUCE-5130-4.patch, > MAPREDUCE-5130-5.patch, MAPREDUCE-5130.patch, MAPREDUCE-5130.patch > > > I came across that mapreduce.map.child.java.opts and > mapreduce.reduce.child.java.opts were missing in mapred-default.xml. I'll do > a fuller sweep to see what else is missing before posting a patch. > List so far: > mapreduce.map/reduce.child.java.opts > mapreduce.map/reduce.memory.mb > mapreduce.job.jvm.numtasks > mapreduce.input.lineinputformat.linespermap > mapreduce.task.combine.progress.records > mapreduce.map/reduce.env -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5130) Add missing job config options to mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-5130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5130: -- Status: Open (was: Patch Available) > Add missing job config options to mapred-default.xml > > > Key: MAPREDUCE-5130 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5130 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Affects Versions: 2.0.4-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Attachments: MAPREDUCE-5130-1.patch, MAPREDUCE-5130-1.patch, > MAPREDUCE-5130-2.patch, MAPREDUCE-5130-3.patch, MAPREDUCE-5130-4.patch, > MAPREDUCE-5130-5.patch, MAPREDUCE-5130.patch, MAPREDUCE-5130.patch > > > I came across that mapreduce.map.child.java.opts and > mapreduce.reduce.child.java.opts were missing in mapred-default.xml. I'll do > a fuller sweep to see what else is missing before posting a patch. > List so far: > mapreduce.map/reduce.child.java.opts > mapreduce.map/reduce.memory.mb > mapreduce.job.jvm.numtasks > mapreduce.input.lineinputformat.linespermap > mapreduce.task.combine.progress.records > mapreduce.map/reduce.env -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5317) Stale files left behind for failed jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-5317:
--
Status: Open (was: Patch Available)
FAIL_ABORT was intended to mirror KILL_ABORT and cover the state where we are
awaiting the completion of the abortJob call from the committer thread. In
that sense it may make more sense and be easier to understand what's happening
in each state if FAIL_WAIT was a separate state from FAIL_ABORT. It would
mirror the KILL_ABORT / KILL_WAIT states that exist in the job kill path.
Speaking of the KILL_WAIT state, if we're using a timeout for FAIL_WAIT it
should be applied to KILL_WAIT as well. Could just rename the event to
something like JOB_WAIT_TIMEOUT.
The JOB_FAIL_WAIT_TIMEDOUT event needs to be handled in more than just the
FAIL_WAIT state. The event could be backed up in the event queue and delivered
*after* we leave the state, so it needs to be at least ignored in any
downstream states from FAIL_WAIT.
The findbugs warning also seems legit.
> Stale files left behind for failed jobs
> ---
>
> Key: MAPREDUCE-5317
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5317
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.8, 2.0.4-alpha, 3.0.0
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: MAPREDUCE-5317.patch, MAPREDUCE-5317.patch
>
>
> Courtesy [~amar_kamat]!
> {quote}
> We are seeing _temporary files left behind in the output folder if the job
> fails.
> The job were failed due to hitting quota issue.
> I simply ran the randomwriter (from hadoop examples) with the default setting.
> That failed and left behind some stray files.
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5334) TestContainerLauncherImpl is failing
[
https://issues.apache.org/jira/browse/MAPREDUCE-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688425#comment-13688425
]
Hadoop QA commented on MAPREDUCE-5334:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588678/MAPREDUCE-5334.1.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:
org.apache.hadoop.mapreduce.v2.app.launcher.TestContainerLauncher
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3788//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3788//console
This message is automatically generated.
> TestContainerLauncherImpl is failing
>
>
> Key: MAPREDUCE-5334
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5334
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Vinod Kumar Vavilapalli
>Priority: Blocker
> Attachments: MAPREDUCE-5334.1.txt, MAPREDUCE-5334.txt
>
>
> Broken by YARN-694, but Jenkins didn't report it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5247) FileInputFormat should filter files with '._COPYING_' sufix
[
https://issues.apache.org/jira/browse/MAPREDUCE-5247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688399#comment-13688399
]
Stan Rosenberg commented on MAPREDUCE-5247:
---
I agree. It appears this change is confined to FsShell and nothing else. Do
we know why this particular file naming convention was chosen or was it just an
oversight?
> FileInputFormat should filter files with '._COPYING_' sufix
> ---
>
> Key: MAPREDUCE-5247
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5247
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Stan Rosenberg
>
> FsShell copy/put creates staging files with '._COPYING_' suffix. These files
> should be considered hidden by FileInputFormat. (A simple fix is to add the
> following conjunct to the existing hiddenFilter:
> {code}
> !name.endsWith("._COPYING_")
> {code}
> After upgrading to CDH 4.2.0 we encountered this bug. We have a legacy data
> loader which uses 'hadoop fs -put' to load data into hourly partitions. We
> also have intra-hourly jobs which are scheduled to execute several times per
> hour using the same hourly partition as input. Thus, as the new data is
> continuously loaded, these staging files (i.e., ._COPYING_) are breaking our
> jobs (since when copy/put completes staging files are moved).
> As a workaround, we've defined a custom input path filter and loaded it with
> "mapred.input.pathFilter.class".
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5334) TestContainerLauncherImpl is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5334: --- Status: Patch Available (was: Open) > TestContainerLauncherImpl is failing > > > Key: MAPREDUCE-5334 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5334 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Attachments: MAPREDUCE-5334.1.txt, MAPREDUCE-5334.txt > > > Broken by YARN-694, but Jenkins didn't report it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5334) TestContainerLauncherImpl is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5334: --- Attachment: MAPREDUCE-5334.1.txt > TestContainerLauncherImpl is failing > > > Key: MAPREDUCE-5334 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5334 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Attachments: MAPREDUCE-5334.1.txt, MAPREDUCE-5334.txt > > > Broken by YARN-694, but Jenkins didn't report it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5334) TestContainerLauncherImpl is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5334: --- Status: Open (was: Patch Available) Good point, adding the timeouts. > TestContainerLauncherImpl is failing > > > Key: MAPREDUCE-5334 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5334 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Attachments: MAPREDUCE-5334.txt > > > Broken by YARN-694, but Jenkins didn't report it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5334) TestContainerLauncherImpl is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688380#comment-13688380 ] Jason Lowe commented on MAPREDUCE-5334: --- One quick comment -- we should add a timeout parameter to the @Test annotation to avoid having this test fail the build in the future if it hangs again. > TestContainerLauncherImpl is failing > > > Key: MAPREDUCE-5334 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5334 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Attachments: MAPREDUCE-5334.txt > > > Broken by YARN-694, but Jenkins didn't report it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5334) TestContainerLauncherImpl is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5334: --- Attachment: MAPREDUCE-5334.txt Patch fixing the test issue.. > TestContainerLauncherImpl is failing > > > Key: MAPREDUCE-5334 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5334 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Attachments: MAPREDUCE-5334.txt > > > Broken by YARN-694, but Jenkins didn't report it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5334) TestContainerLauncherImpl is failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5334: --- Status: Patch Available (was: Open) > TestContainerLauncherImpl is failing > > > Key: MAPREDUCE-5334 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5334 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Attachments: MAPREDUCE-5334.txt > > > Broken by YARN-694, but Jenkins didn't report it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5247) FileInputFormat should filter files with '._COPYING_' sufix
[
https://issues.apache.org/jira/browse/MAPREDUCE-5247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688335#comment-13688335
]
Kousuke Saruta commented on MAPREDUCE-5247:
---
Stan, I think we should modify FsShell to create a file assigned the name with
underscore prefix for a file being created so that FileInputFormat can ignore
the file rather than modify FileInputFormat to handle a ._COPYING_ suffix file
as a hidden file.
It's just HDFS matter and I think the specification change shouldn't affects
MapReduce.
How do you think?
> FileInputFormat should filter files with '._COPYING_' sufix
> ---
>
> Key: MAPREDUCE-5247
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5247
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Stan Rosenberg
>
> FsShell copy/put creates staging files with '._COPYING_' suffix. These files
> should be considered hidden by FileInputFormat. (A simple fix is to add the
> following conjunct to the existing hiddenFilter:
> {code}
> !name.endsWith("._COPYING_")
> {code}
> After upgrading to CDH 4.2.0 we encountered this bug. We have a legacy data
> loader which uses 'hadoop fs -put' to load data into hourly partitions. We
> also have intra-hourly jobs which are scheduled to execute several times per
> hour using the same hourly partition as input. Thus, as the new data is
> continuously loaded, these staging files (i.e., ._COPYING_) are breaking our
> jobs (since when copy/put completes staging files are moved).
> As a workaround, we've defined a custom input path filter and loaded it with
> "mapred.input.pathFilter.class".
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5334) TestContainerLauncherImpl is failing
Vinod Kumar Vavilapalli created MAPREDUCE-5334: -- Summary: TestContainerLauncherImpl is failing Key: MAPREDUCE-5334 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5334 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Vinod Kumar Vavilapalli Assignee: Vinod Kumar Vavilapalli Priority: Blocker Broken by YARN-694, but Jenkins didn't report it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5333) Add test that verifies MRAM works correctly when sending requests with non-normalized capabilities
Alejandro Abdelnur created MAPREDUCE-5333: - Summary: Add test that verifies MRAM works correctly when sending requests with non-normalized capabilities Key: MAPREDUCE-5333 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5333 Project: Hadoop Map/Reduce Issue Type: Test Components: mr-am Affects Versions: 2.1.0-beta Reporter: Alejandro Abdelnur This is a follow on MAPREDUCE-5310 to ensure nothing broke after we removed normalization on the MRAM side. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-5333) Add test that verifies MRAM works correctly when sending requests with non-normalized capabilities
[ https://issues.apache.org/jira/browse/MAPREDUCE-5333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Yan reassigned MAPREDUCE-5333: -- Assignee: Wei Yan > Add test that verifies MRAM works correctly when sending requests with > non-normalized capabilities > -- > > Key: MAPREDUCE-5333 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5333 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: mr-am >Affects Versions: 2.1.0-beta >Reporter: Alejandro Abdelnur >Assignee: Wei Yan > > This is a follow on MAPREDUCE-5310 to ensure nothing broke after we removed > normalization on the MRAM side. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5102) fix coverage org.apache.hadoop.mapreduce.lib.db and org.apache.hadoop.mapred.lib.db
[
https://issues.apache.org/jira/browse/MAPREDUCE-5102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688142#comment-13688142
]
Hadoop QA commented on MAPREDUCE-5102:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588615/MAPREDUCE-5102-trunk-v1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3786//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3786//console
This message is automatically generated.
> fix coverage org.apache.hadoop.mapreduce.lib.db and
> org.apache.hadoop.mapred.lib.db
>
>
> Key: MAPREDUCE-5102
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5102
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha
>Reporter: Aleksey Gorshkov
>Assignee: Aleksey Gorshkov
> Attachments: MAPREDUCE-5102-branch-0.23.patch,
> MAPREDUCE-5102-branch-0.23-v1.patch, MAPREDUCE-5102-trunk.patch,
> MAPREDUCE-5102-trunk-v1.patch
>
>
> fix coverage org.apache.hadoop.mapreduce.lib.db and
> org.apache.hadoop.mapred.lib.db
> patch MAPREDUCE-5102-trunk.patch for trunk and branch-2
> patch MAPREDUCE-5102-branch-0.23.patch for branch-0.23 only
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5084) fix coverage org.apache.hadoop.mapreduce.v2.app.webapp and org.apache.hadoop.mapreduce.v2.hs.webapp
[ https://issues.apache.org/jira/browse/MAPREDUCE-5084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688138#comment-13688138 ] Hudson commented on MAPREDUCE-5084: --- Integrated in Hadoop-trunk-Commit #3980 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3980/]) MAPREDUCE-5084. fix coverage org.apache.hadoop.mapreduce.v2.app.webapp and org.apache.hadoop.mapreduce.v2.hs.webapp (Aleksey Gorshkov via jeagles) (Revision 1494676) Result = SUCCESS jeagles : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494676 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/AppControllerForTest.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/AppForTest.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TasksBlockForTest.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestAppController.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/TestBlocks.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/webapp/ViewForTest.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/yarn * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/yarn/webapp * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/yarn/webapp/view * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/yarn/webapp/view/BlockForTest.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestBlocks.java > fix coverage org.apache.hadoop.mapreduce.v2.app.webapp and > org.apache.hadoop.mapreduce.v2.hs.webapp > > > Key: MAPREDUCE-5084 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5084 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha >Reporter: Aleksey Gorshkov >Assignee: Aleksey Gorshkov > Fix For: 3.0.0, 0.23.9, 2.3.0 > > Attachments: MAPREDUCE-5084-branch-0.23-v1.patch, > MAPREDUCE-5084-trunk.patch, MAPREDUCE-5084-trunk-v1.patch > > > fix coverage org.apache.hadoop.mapreduce.v2.app.webapp and > org.apache.hadoop.mapreduce.v2.hs.webapp > The same patch for trunk, branch-2 and branch-0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5084) fix coverage org.apache.hadoop.mapreduce.v2.app.webapp and org.apache.hadoop.mapreduce.v2.hs.webapp
[ https://issues.apache.org/jira/browse/MAPREDUCE-5084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5084: --- Resolution: Fixed Fix Version/s: 2.3.0 0.23.9 3.0.0 Status: Resolved (was: Patch Available) > fix coverage org.apache.hadoop.mapreduce.v2.app.webapp and > org.apache.hadoop.mapreduce.v2.hs.webapp > > > Key: MAPREDUCE-5084 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5084 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha >Reporter: Aleksey Gorshkov >Assignee: Aleksey Gorshkov > Fix For: 3.0.0, 0.23.9, 2.3.0 > > Attachments: MAPREDUCE-5084-branch-0.23-v1.patch, > MAPREDUCE-5084-trunk.patch, MAPREDUCE-5084-trunk-v1.patch > > > fix coverage org.apache.hadoop.mapreduce.v2.app.webapp and > org.apache.hadoop.mapreduce.v2.hs.webapp > The same patch for trunk, branch-2 and branch-0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5084) fix coverage org.apache.hadoop.mapreduce.v2.app.webapp and org.apache.hadoop.mapreduce.v2.hs.webapp
[ https://issues.apache.org/jira/browse/MAPREDUCE-5084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688095#comment-13688095 ] Jonathan Eagles commented on MAPREDUCE-5084: +1. Patch looks good. Great to see these coverage additions, Aleksey. The following table shows classes are moved to acceptable level. *org.apache.hadoop.mapreduce.v2.app.webapp* AppController7.4% -> 78.1% ConfBlock 21.1% -> 82.7% TasksBlock 21.9% -> 68.4% *org.apache.hadoop.mapreduce.v2.hs.webapp* HsTasksBlock 6.6% -> 71.2% HsTaskPage.AttemptsBlock 9.8% -> 85.9% HsJobsBlock 0% -> 93.8% HsController 9.5% -> 100% > fix coverage org.apache.hadoop.mapreduce.v2.app.webapp and > org.apache.hadoop.mapreduce.v2.hs.webapp > > > Key: MAPREDUCE-5084 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5084 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha >Reporter: Aleksey Gorshkov >Assignee: Aleksey Gorshkov > Attachments: MAPREDUCE-5084-branch-0.23-v1.patch, > MAPREDUCE-5084-trunk.patch, MAPREDUCE-5084-trunk-v1.patch > > > fix coverage org.apache.hadoop.mapreduce.v2.app.webapp and > org.apache.hadoop.mapreduce.v2.hs.webapp > The same patch for trunk, branch-2 and branch-0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5102) fix coverage org.apache.hadoop.mapreduce.lib.db and org.apache.hadoop.mapred.lib.db
[ https://issues.apache.org/jira/browse/MAPREDUCE-5102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688078#comment-13688078 ] Aleksey Gorshkov commented on MAPREDUCE-5102: - sorry mixed up. patch MAPREDUCE-5102-branch-0.23-v1.patch for trunk and branch-2 patch MAPREDUCE-5102-trunk-v1.patch patch for branch-0.23 > fix coverage org.apache.hadoop.mapreduce.lib.db and > org.apache.hadoop.mapred.lib.db > > > Key: MAPREDUCE-5102 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5102 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha >Reporter: Aleksey Gorshkov >Assignee: Aleksey Gorshkov > Attachments: MAPREDUCE-5102-branch-0.23.patch, > MAPREDUCE-5102-branch-0.23-v1.patch, MAPREDUCE-5102-trunk.patch, > MAPREDUCE-5102-trunk-v1.patch > > > fix coverage org.apache.hadoop.mapreduce.lib.db and > org.apache.hadoop.mapred.lib.db > patch MAPREDUCE-5102-trunk.patch for trunk and branch-2 > patch MAPREDUCE-5102-branch-0.23.patch for branch-0.23 only -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5102) fix coverage org.apache.hadoop.mapreduce.lib.db and org.apache.hadoop.mapred.lib.db
[ https://issues.apache.org/jira/browse/MAPREDUCE-5102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Gorshkov updated MAPREDUCE-5102: Attachment: (was: HADOOP-9598-branch-0.23-v1.patch) > fix coverage org.apache.hadoop.mapreduce.lib.db and > org.apache.hadoop.mapred.lib.db > > > Key: MAPREDUCE-5102 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5102 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha >Reporter: Aleksey Gorshkov >Assignee: Aleksey Gorshkov > Attachments: MAPREDUCE-5102-branch-0.23.patch, > MAPREDUCE-5102-branch-0.23-v1.patch, MAPREDUCE-5102-trunk.patch, > MAPREDUCE-5102-trunk-v1.patch > > > fix coverage org.apache.hadoop.mapreduce.lib.db and > org.apache.hadoop.mapred.lib.db > patch MAPREDUCE-5102-trunk.patch for trunk and branch-2 > patch MAPREDUCE-5102-branch-0.23.patch for branch-0.23 only -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5102) fix coverage org.apache.hadoop.mapreduce.lib.db and org.apache.hadoop.mapred.lib.db
[ https://issues.apache.org/jira/browse/MAPREDUCE-5102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Gorshkov updated MAPREDUCE-5102: Attachment: (was: HADOOP-9598-trunk-v1.patch) > fix coverage org.apache.hadoop.mapreduce.lib.db and > org.apache.hadoop.mapred.lib.db > > > Key: MAPREDUCE-5102 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5102 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha >Reporter: Aleksey Gorshkov >Assignee: Aleksey Gorshkov > Attachments: MAPREDUCE-5102-branch-0.23.patch, > MAPREDUCE-5102-branch-0.23-v1.patch, MAPREDUCE-5102-trunk.patch, > MAPREDUCE-5102-trunk-v1.patch > > > fix coverage org.apache.hadoop.mapreduce.lib.db and > org.apache.hadoop.mapred.lib.db > patch MAPREDUCE-5102-trunk.patch for trunk and branch-2 > patch MAPREDUCE-5102-branch-0.23.patch for branch-0.23 only -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5102) fix coverage org.apache.hadoop.mapreduce.lib.db and org.apache.hadoop.mapred.lib.db
[ https://issues.apache.org/jira/browse/MAPREDUCE-5102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Gorshkov updated MAPREDUCE-5102: Attachment: MAPREDUCE-5102-trunk-v1.patch MAPREDUCE-5102-branch-0.23-v1.patch > fix coverage org.apache.hadoop.mapreduce.lib.db and > org.apache.hadoop.mapred.lib.db > > > Key: MAPREDUCE-5102 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5102 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha >Reporter: Aleksey Gorshkov >Assignee: Aleksey Gorshkov > Attachments: MAPREDUCE-5102-branch-0.23.patch, > MAPREDUCE-5102-branch-0.23-v1.patch, MAPREDUCE-5102-trunk.patch, > MAPREDUCE-5102-trunk-v1.patch > > > fix coverage org.apache.hadoop.mapreduce.lib.db and > org.apache.hadoop.mapred.lib.db > patch MAPREDUCE-5102-trunk.patch for trunk and branch-2 > patch MAPREDUCE-5102-branch-0.23.patch for branch-0.23 only -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5102) fix coverage org.apache.hadoop.mapreduce.lib.db and org.apache.hadoop.mapred.lib.db
[ https://issues.apache.org/jira/browse/MAPREDUCE-5102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13688068#comment-13688068 ] Jonathan Eagles commented on MAPREDUCE-5102: Looks like patch for HADOOP-9598 was attached instead of patch for this JIRA. The approach to a mock connection object seems to be the right way to go. > fix coverage org.apache.hadoop.mapreduce.lib.db and > org.apache.hadoop.mapred.lib.db > > > Key: MAPREDUCE-5102 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5102 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha >Reporter: Aleksey Gorshkov >Assignee: Aleksey Gorshkov > Attachments: HADOOP-9598-branch-0.23-v1.patch, > HADOOP-9598-trunk-v1.patch, MAPREDUCE-5102-branch-0.23.patch, > MAPREDUCE-5102-trunk.patch > > > fix coverage org.apache.hadoop.mapreduce.lib.db and > org.apache.hadoop.mapred.lib.db > patch MAPREDUCE-5102-trunk.patch for trunk and branch-2 > patch MAPREDUCE-5102-branch-0.23.patch for branch-0.23 only -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5316) job -list-attempt-ids command does not handle illegal task-state
[
https://issues.apache.org/jira/browse/MAPREDUCE-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687999#comment-13687999
]
Hudson commented on MAPREDUCE-5316:
---
Integrated in Hadoop-Mapreduce-trunk #1462 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1462/])
MAPREDUCE-5316. job -list-attempt-ids command does not handle illegal
task-state. Contributed by Ashwin Shankar (Revision 1494285)
Result = FAILURE
jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494285
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/tools/TestCLI.java
> job -list-attempt-ids command does not handle illegal task-state
>
>
> Key: MAPREDUCE-5316
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5316
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 3.0.0, 2.0.4-alpha, 0.23.8
>Reporter: Ashwin Shankar
>Assignee: Ashwin Shankar
> Fix For: 0.23.9, 2.3.0
>
> Attachments: MAPREDUCE-5316-1-b23.txt, MAPREDUCE-5316-1.txt
>
>
> Courtesy : [~mikanboy]
> {quote}
> job -list-attempt-ids command should handle illegal argument for
> the same way as . Right now only illegal is handle by
> an exception being thrown. Illegal on the other hand does not
> throw exception. For example is a user mistype 'completed' as 'complete', they
> may wrongly think there are no completed tasks, instead of being notified of
> the illegal that was used.
> 1) illegal handled.
> [philips@gwbl2003:4095 ~/svn/HadoopQEAutomation/branch-23]$
> /home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
> job_1345673924741_0086 map completed
> 12/08/24 15:16:31 WARN conf.Configuration: mapred.used.genericoptionsparser is
> deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> Exception in thread "main" java.lang.IllegalArgumentException: No enum const
> class org.apache.hadoop.mapreduce.TaskType.map
> at java.lang.Enum.valueOf(Enum.java:196)
> at org.apache.hadoop.mapreduce.TaskType.valueOf(TaskType.java:27)
> at org.apache.hadoop.mapreduce.tools.CLI.displayTasks(CLI.java:553)
> at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:309)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
> at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1242)
> 1) illegal not handled.
> /home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
> job_1345673924741_0086 MAP complete
> 12/08/24 15:15:47 WARN conf.Configuration: mapred.used.genericoptionsparser is
> deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> {quote}
> Also,we could make input task-state to be case-insensitive to be consistent
> with
> task-type behavior(MAPREDUCE-4019).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5194) Heed interrupts during Fetcher shutdown
[
https://issues.apache.org/jira/browse/MAPREDUCE-5194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687992#comment-13687992
]
Hudson commented on MAPREDUCE-5194:
---
Integrated in Hadoop-Mapreduce-trunk #1462 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1462/])
MAPREDUCE-5194. Heed interrupts during Fetcher shutdown. (Revision 1494416)
Result = FAILURE
cdouglas :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494416
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
> Heed interrupts during Fetcher shutdown
> ---
>
> Key: MAPREDUCE-5194
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5194
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: task
>Reporter: Chris Douglas
>Assignee: Chris Douglas
>Priority: Minor
> Fix For: 2.1.0-beta
>
> Attachments: M5194-1.patch, M5194-2.patch, M5194.patch
>
>
> In the current implementation, {{Fetcher}} instances usually exit gracefully
> when the shuffle succeeds. When it fails, threads are interrupted, but may
> continue running harmlessly until the JVM shuts down.
> However, to generate consistent checkpoints, these threads should exit
> cleanly to quiesce the state of the shuffle.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5332) Support token-preserving restart of history server
Jason Lowe created MAPREDUCE-5332: - Summary: Support token-preserving restart of history server Key: MAPREDUCE-5332 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5332 Project: Hadoop Map/Reduce Issue Type: New Feature Components: jobhistoryserver Reporter: Jason Lowe To better support rolling upgrades through a cluster, the history server needs the ability to restart without losing track of delegation tokens. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5194) Heed interrupts during Fetcher shutdown
[
https://issues.apache.org/jira/browse/MAPREDUCE-5194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687947#comment-13687947
]
Hudson commented on MAPREDUCE-5194:
---
Integrated in Hadoop-Hdfs-trunk #1435 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1435/])
MAPREDUCE-5194. Heed interrupts during Fetcher shutdown. (Revision 1494416)
Result = FAILURE
cdouglas :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494416
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
> Heed interrupts during Fetcher shutdown
> ---
>
> Key: MAPREDUCE-5194
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5194
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: task
>Reporter: Chris Douglas
>Assignee: Chris Douglas
>Priority: Minor
> Fix For: 2.1.0-beta
>
> Attachments: M5194-1.patch, M5194-2.patch, M5194.patch
>
>
> In the current implementation, {{Fetcher}} instances usually exit gracefully
> when the shuffle succeeds. When it fails, threads are interrupted, but may
> continue running harmlessly until the JVM shuts down.
> However, to generate consistent checkpoints, these threads should exit
> cleanly to quiesce the state of the shuffle.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5316) job -list-attempt-ids command does not handle illegal task-state
[
https://issues.apache.org/jira/browse/MAPREDUCE-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687954#comment-13687954
]
Hudson commented on MAPREDUCE-5316:
---
Integrated in Hadoop-Hdfs-trunk #1435 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1435/])
MAPREDUCE-5316. job -list-attempt-ids command does not handle illegal
task-state. Contributed by Ashwin Shankar (Revision 1494285)
Result = FAILURE
jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494285
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/tools/TestCLI.java
> job -list-attempt-ids command does not handle illegal task-state
>
>
> Key: MAPREDUCE-5316
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5316
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 3.0.0, 2.0.4-alpha, 0.23.8
>Reporter: Ashwin Shankar
>Assignee: Ashwin Shankar
> Fix For: 0.23.9, 2.3.0
>
> Attachments: MAPREDUCE-5316-1-b23.txt, MAPREDUCE-5316-1.txt
>
>
> Courtesy : [~mikanboy]
> {quote}
> job -list-attempt-ids command should handle illegal argument for
> the same way as . Right now only illegal is handle by
> an exception being thrown. Illegal on the other hand does not
> throw exception. For example is a user mistype 'completed' as 'complete', they
> may wrongly think there are no completed tasks, instead of being notified of
> the illegal that was used.
> 1) illegal handled.
> [philips@gwbl2003:4095 ~/svn/HadoopQEAutomation/branch-23]$
> /home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
> job_1345673924741_0086 map completed
> 12/08/24 15:16:31 WARN conf.Configuration: mapred.used.genericoptionsparser is
> deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> Exception in thread "main" java.lang.IllegalArgumentException: No enum const
> class org.apache.hadoop.mapreduce.TaskType.map
> at java.lang.Enum.valueOf(Enum.java:196)
> at org.apache.hadoop.mapreduce.TaskType.valueOf(TaskType.java:27)
> at org.apache.hadoop.mapreduce.tools.CLI.displayTasks(CLI.java:553)
> at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:309)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
> at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1242)
> 1) illegal not handled.
> /home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
> job_1345673924741_0086 MAP complete
> 12/08/24 15:15:47 WARN conf.Configuration: mapred.used.genericoptionsparser is
> deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> {quote}
> Also,we could make input task-state to be case-insensitive to be consistent
> with
> task-type behavior(MAPREDUCE-4019).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5316) job -list-attempt-ids command does not handle illegal task-state
[
https://issues.apache.org/jira/browse/MAPREDUCE-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687933#comment-13687933
]
Hudson commented on MAPREDUCE-5316:
---
Integrated in Hadoop-Hdfs-0.23-Build #643 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/643/])
MAPREDUCE-5316. job -list-attempt-ids command does not handle illegal
task-state. Contributed by Ashwin Shankar (Revision 1494293)
Result = SUCCESS
jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494293
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/tools/TestCLI.java
> job -list-attempt-ids command does not handle illegal task-state
>
>
> Key: MAPREDUCE-5316
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5316
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 3.0.0, 2.0.4-alpha, 0.23.8
>Reporter: Ashwin Shankar
>Assignee: Ashwin Shankar
> Fix For: 0.23.9, 2.3.0
>
> Attachments: MAPREDUCE-5316-1-b23.txt, MAPREDUCE-5316-1.txt
>
>
> Courtesy : [~mikanboy]
> {quote}
> job -list-attempt-ids command should handle illegal argument for
> the same way as . Right now only illegal is handle by
> an exception being thrown. Illegal on the other hand does not
> throw exception. For example is a user mistype 'completed' as 'complete', they
> may wrongly think there are no completed tasks, instead of being notified of
> the illegal that was used.
> 1) illegal handled.
> [philips@gwbl2003:4095 ~/svn/HadoopQEAutomation/branch-23]$
> /home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
> job_1345673924741_0086 map completed
> 12/08/24 15:16:31 WARN conf.Configuration: mapred.used.genericoptionsparser is
> deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> Exception in thread "main" java.lang.IllegalArgumentException: No enum const
> class org.apache.hadoop.mapreduce.TaskType.map
> at java.lang.Enum.valueOf(Enum.java:196)
> at org.apache.hadoop.mapreduce.TaskType.valueOf(TaskType.java:27)
> at org.apache.hadoop.mapreduce.tools.CLI.displayTasks(CLI.java:553)
> at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:309)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
> at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1242)
> 1) illegal not handled.
> /home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
> job_1345673924741_0086 MAP complete
> 12/08/24 15:15:47 WARN conf.Configuration: mapred.used.genericoptionsparser is
> deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> {quote}
> Also,we could make input task-state to be case-insensitive to be consistent
> with
> task-type behavior(MAPREDUCE-4019).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5317) Stale files left behind for failed jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687876#comment-13687876
]
Hadoop QA commented on MAPREDUCE-5317:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588568/MAPREDUCE-5317.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3785//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3785//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3785//console
This message is automatically generated.
> Stale files left behind for failed jobs
> ---
>
> Key: MAPREDUCE-5317
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5317
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.0.0, 2.0.4-alpha, 0.23.8
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: MAPREDUCE-5317.patch, MAPREDUCE-5317.patch
>
>
> Courtesy [~amar_kamat]!
> {quote}
> We are seeing _temporary files left behind in the output folder if the job
> fails.
> The job were failed due to hitting quota issue.
> I simply ran the randomwriter (from hadoop examples) with the default setting.
> That failed and left behind some stray files.
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5317) Stale files left behind for failed jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687856#comment-13687856
]
Ravi Prakash commented on MAPREDUCE-5317:
-
Thanks a lot for your detailed review Jason. Thanks also for summarizing all
the problems.
Daryn and Kihwal also pointed out to me that another problem with using
non-recursive create is that the existing behavior of some OutputCommitters may
be to depend on the recursive create in setupTask. I agree with you, that we
should use non-recursive create but can we please pursue that in another JIRA
since it changes pre-existing behavior?
This patch simply provides a best-effort guarantee that abortJob will be called
after all the tasks are finished/killed. I think that is a reasonable
expectation on the part of our users who choose to write their own
OutputCommitters. In the case that the timeout expires, we may still be left
with stale files if some tasks are still alive and create the directory, but I
feel that is a bullet we have to bite.
I've incorporated all your comments. I took the liberty of renaming FAIL_ABORT
to FAIL_WAIT because it made a lot more sense to me. Thanks a lot again! :)
> Stale files left behind for failed jobs
> ---
>
> Key: MAPREDUCE-5317
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5317
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.0.0, 2.0.4-alpha, 0.23.8
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: MAPREDUCE-5317.patch, MAPREDUCE-5317.patch
>
>
> Courtesy [~amar_kamat]!
> {quote}
> We are seeing _temporary files left behind in the output folder if the job
> fails.
> The job were failed due to hitting quota issue.
> I simply ran the randomwriter (from hadoop examples) with the default setting.
> That failed and left behind some stray files.
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5316) job -list-attempt-ids command does not handle illegal task-state
[
https://issues.apache.org/jira/browse/MAPREDUCE-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687853#comment-13687853
]
Hudson commented on MAPREDUCE-5316:
---
Integrated in Hadoop-Yarn-trunk #245 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/245/])
MAPREDUCE-5316. job -list-attempt-ids command does not handle illegal
task-state. Contributed by Ashwin Shankar (Revision 1494285)
Result = FAILURE
jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494285
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/tools/TestCLI.java
> job -list-attempt-ids command does not handle illegal task-state
>
>
> Key: MAPREDUCE-5316
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5316
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Affects Versions: 3.0.0, 2.0.4-alpha, 0.23.8
>Reporter: Ashwin Shankar
>Assignee: Ashwin Shankar
> Fix For: 0.23.9, 2.3.0
>
> Attachments: MAPREDUCE-5316-1-b23.txt, MAPREDUCE-5316-1.txt
>
>
> Courtesy : [~mikanboy]
> {quote}
> job -list-attempt-ids command should handle illegal argument for
> the same way as . Right now only illegal is handle by
> an exception being thrown. Illegal on the other hand does not
> throw exception. For example is a user mistype 'completed' as 'complete', they
> may wrongly think there are no completed tasks, instead of being notified of
> the illegal that was used.
> 1) illegal handled.
> [philips@gwbl2003:4095 ~/svn/HadoopQEAutomation/branch-23]$
> /home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
> job_1345673924741_0086 map completed
> 12/08/24 15:16:31 WARN conf.Configuration: mapred.used.genericoptionsparser is
> deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> Exception in thread "main" java.lang.IllegalArgumentException: No enum const
> class org.apache.hadoop.mapreduce.TaskType.map
> at java.lang.Enum.valueOf(Enum.java:196)
> at org.apache.hadoop.mapreduce.TaskType.valueOf(TaskType.java:27)
> at org.apache.hadoop.mapreduce.tools.CLI.displayTasks(CLI.java:553)
> at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:309)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
> at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1242)
> 1) illegal not handled.
> /home/gs/gridre/yroot.theoden/share/hadoop/bin/mapred job -list-attempt-ids
> job_1345673924741_0086 MAP complete
> 12/08/24 15:15:47 WARN conf.Configuration: mapred.used.genericoptionsparser is
> deprecated. Instead, use mapreduce.client.genericoptionsparser.used
> {quote}
> Also,we could make input task-state to be case-insensitive to be consistent
> with
> task-type behavior(MAPREDUCE-4019).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5194) Heed interrupts during Fetcher shutdown
[
https://issues.apache.org/jira/browse/MAPREDUCE-5194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687846#comment-13687846
]
Hudson commented on MAPREDUCE-5194:
---
Integrated in Hadoop-Yarn-trunk #245 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/245/])
MAPREDUCE-5194. Heed interrupts during Fetcher shutdown. (Revision 1494416)
Result = FAILURE
cdouglas :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1494416
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/Fetcher.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/task/reduce/OnDiskMapOutput.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestFetcher.java
> Heed interrupts during Fetcher shutdown
> ---
>
> Key: MAPREDUCE-5194
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5194
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: task
>Reporter: Chris Douglas
>Assignee: Chris Douglas
>Priority: Minor
> Fix For: 2.1.0-beta
>
> Attachments: M5194-1.patch, M5194-2.patch, M5194.patch
>
>
> In the current implementation, {{Fetcher}} instances usually exit gracefully
> when the shuffle succeeds. When it fails, threads are interrupted, but may
> continue running harmlessly until the JVM shuts down.
> However, to generate consistent checkpoints, these threads should exit
> cleanly to quiesce the state of the shuffle.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5317) Stale files left behind for failed jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-5317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash updated MAPREDUCE-5317:
Attachment: MAPREDUCE-5317.patch
Updated patch addressing Jason's comments
> Stale files left behind for failed jobs
> ---
>
> Key: MAPREDUCE-5317
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5317
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 3.0.0, 2.0.4-alpha, 0.23.8
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: MAPREDUCE-5317.patch, MAPREDUCE-5317.patch
>
>
> Courtesy [~amar_kamat]!
> {quote}
> We are seeing _temporary files left behind in the output folder if the job
> fails.
> The job were failed due to hitting quota issue.
> I simply ran the randomwriter (from hadoop examples) with the default setting.
> That failed and left behind some stray files.
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-914) MapReduce source code has significant number of JavaDocs errors and warnings
[ https://issues.apache.org/jira/browse/MAPREDUCE-914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arijit Banerjee reassigned MAPREDUCE-914: - Assignee: (was: Arijit Banerjee) > MapReduce source code has significant number of JavaDocs errors and warnings > > > Key: MAPREDUCE-914 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-914 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Konstantin Boudnik > Labels: newbie > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-914) MapReduce source code has significant number of JavaDocs errors and warnings
[ https://issues.apache.org/jira/browse/MAPREDUCE-914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arijit Banerjee reassigned MAPREDUCE-914: - Assignee: Arijit Banerjee > MapReduce source code has significant number of JavaDocs errors and warnings > > > Key: MAPREDUCE-914 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-914 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Konstantin Boudnik >Assignee: Arijit Banerjee > Labels: newbie > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5102) fix coverage org.apache.hadoop.mapreduce.lib.db and org.apache.hadoop.mapred.lib.db
[
https://issues.apache.org/jira/browse/MAPREDUCE-5102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13687704#comment-13687704
]
Hadoop QA commented on MAPREDUCE-5102:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12588548/HADOOP-9598-trunk-v1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3784//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3784//console
This message is automatically generated.
> fix coverage org.apache.hadoop.mapreduce.lib.db and
> org.apache.hadoop.mapred.lib.db
>
>
> Key: MAPREDUCE-5102
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5102
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha
>Reporter: Aleksey Gorshkov
>Assignee: Aleksey Gorshkov
> Attachments: HADOOP-9598-branch-0.23-v1.patch,
> HADOOP-9598-trunk-v1.patch, MAPREDUCE-5102-branch-0.23.patch,
> MAPREDUCE-5102-trunk.patch
>
>
> fix coverage org.apache.hadoop.mapreduce.lib.db and
> org.apache.hadoop.mapred.lib.db
> patch MAPREDUCE-5102-trunk.patch for trunk and branch-2
> patch MAPREDUCE-5102-branch-0.23.patch for branch-0.23 only
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
