[jira] [Resolved] (MAPREDUCE-5081) Backport DistCpV2 and the related JIRAs to branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE resolved MAPREDUCE-5081. --- Resolution: Fixed Fix Version/s: 1.2.0 Hadoop Flags: Reviewed Thanks Suresh for reviewing the patch. I have committed this. > Backport DistCpV2 and the related JIRAs to branch-1 > --- > > Key: MAPREDUCE-5081 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5081 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: distcp >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 1.2.0 > > Attachments: DistCp.java.diff, m5081_20130328b.patch, > m5081_20130328.patch, m5981_20130321b.patch, m5981_20130321.patch, > m5981_20130323.patch > > > Here is a list of DistCpV2 JIRAs: > - MAPREDUCE-2765: DistCpV2 main jira > - HADOOP-8703: turn CRC checking off for 0 byte size > - HDFS-3054: distcp -skipcrccheck has no effect. > - HADOOP-8431: Running distcp without args throws IllegalArgumentException > - HADOOP-8775: non-positive value to -bandwidth > - MAPREDUCE-4654: TestDistCp is ignored > - HADOOP-9022: distcp fails to copy file if -m 0 specified > - HADOOP-9025: TestCopyListing failing > - MAPREDUCE-5075: DistCp leaks input file handles > - distcp part of HADOOP-8341: Fix findbugs issues in hadoop-tools > - MAPREDUCE-5014: custom CopyListing -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5081) Backport DistCpV2 and the related JIRAs to branch-1
[
https://issues.apache.org/jira/browse/MAPREDUCE-5081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated MAPREDUCE-5081:
--
Attachment: m5081_20130328b.patch
Oops, I accidentally added a new vaidya entry to site.xml.
{code}
+
+
{code}
m5081_20130328b.patch: removes the new vaidya entry.
> Backport DistCpV2 and the related JIRAs to branch-1
> ---
>
> Key: MAPREDUCE-5081
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5081
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: distcp
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: DistCp.java.diff, m5081_20130328b.patch,
> m5081_20130328.patch, m5981_20130321b.patch, m5981_20130321.patch,
> m5981_20130323.patch
>
>
> Here is a list of DistCpV2 JIRAs:
> - MAPREDUCE-2765: DistCpV2 main jira
> - HADOOP-8703: turn CRC checking off for 0 byte size
> - HDFS-3054: distcp -skipcrccheck has no effect.
> - HADOOP-8431: Running distcp without args throws IllegalArgumentException
> - HADOOP-8775: non-positive value to -bandwidth
> - MAPREDUCE-4654: TestDistCp is ignored
> - HADOOP-9022: distcp fails to copy file if -m 0 specified
> - HADOOP-9025: TestCopyListing failing
> - MAPREDUCE-5075: DistCp leaks input file handles
> - distcp part of HADOOP-8341: Fix findbugs issues in hadoop-tools
> - MAPREDUCE-5014: custom CopyListing
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5081) Backport DistCpV2 and the related JIRAs to branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13616027#comment-13616027 ] Suresh Srinivas commented on MAPREDUCE-5081: bq. Are you sure that you diff the correct file since there two DistCp.java files? Let me post the diff. Sorry, I must have compared the wrong files. bq. In branch-1, other xml files like core-default.xml and hdfs-default.xml also do not have license header. So let's don't fix the new xml here and fix them with other xml files together later. Sounds good. +1 for the patch. > Backport DistCpV2 and the related JIRAs to branch-1 > --- > > Key: MAPREDUCE-5081 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5081 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: distcp >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: DistCp.java.diff, m5081_20130328.patch, > m5981_20130321b.patch, m5981_20130321.patch, m5981_20130323.patch > > > Here is a list of DistCpV2 JIRAs: > - MAPREDUCE-2765: DistCpV2 main jira > - HADOOP-8703: turn CRC checking off for 0 byte size > - HDFS-3054: distcp -skipcrccheck has no effect. > - HADOOP-8431: Running distcp without args throws IllegalArgumentException > - HADOOP-8775: non-positive value to -bandwidth > - MAPREDUCE-4654: TestDistCp is ignored > - HADOOP-9022: distcp fails to copy file if -m 0 specified > - HADOOP-9025: TestCopyListing failing > - MAPREDUCE-5075: DistCp leaks input file handles > - distcp part of HADOOP-8341: Fix findbugs issues in hadoop-tools > - MAPREDUCE-5014: custom CopyListing -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5110) Long task launch delays can lead to multiple parallel attempts of the task
[
https://issues.apache.org/jira/browse/MAPREDUCE-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karthik Kambatla updated MAPREDUCE-5110:
Attachment: mr-5110.patch
Uploading a patch that fixes the issue. It
# adds a config parameter {{mapred.tasktracker.task.launch.timeout}} with a
default value of 2 minutes. Also, adds this to mapred-default.xml
# updates {{TT#markUnresponsiveTasks()}} to address tasks in UNASSIGNED state
for longer than the timeout above.
# modifies JT to not expire UNASSIGNED tasks; MAPREDUCE-2217 added this to
address the case where the task launch would hang, but that doesn't help in the
case where task launch just takes really long. Leaving the check there can lead
to inappropriate error messages for the tasks. Also,
{{markUnresponsiveTasks()}} and {{transmitHeartBeat()}} are in the same thread:
if the TT were unable to fail the UNASSIGNED task, it wouldn't be able to send
a heartbeat either and will eventually be marked lost.
To validate the patch, I ran the same setup as above and verified that the
first attempt is killed before launching the subsequent attempt.
> Long task launch delays can lead to multiple parallel attempts of the task
> --
>
> Key: MAPREDUCE-5110
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5110
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
>Affects Versions: 1.1.2
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
> Attachments: expose-mr-5110.patch, mr-5110.patch
>
>
> If a task takes too long to launch, the JT expires the task and schedules
> another attempt. The earlier attempt can start after the later attempt
> leading to two parallel attempts running at the same time. This is
> particularly an issue if the user turns off speculation and expects a single
> attempt of a task to run at any point in time.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5110) Long task launch delays can lead to multiple parallel attempts of the task
[ https://issues.apache.org/jira/browse/MAPREDUCE-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13616000#comment-13616000 ] Karthik Kambatla commented on MAPREDUCE-5110: - The issue can be reproduced by applying the above patch on a 4 node cluster (8 map/reduce slots) with mapred.tasktracker.expiry.interval set to 1000 and running terasort on 20 GB data. > Long task launch delays can lead to multiple parallel attempts of the task > -- > > Key: MAPREDUCE-5110 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5110 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 1.1.2 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: expose-mr-5110.patch > > > If a task takes too long to launch, the JT expires the task and schedules > another attempt. The earlier attempt can start after the later attempt > leading to two parallel attempts running at the same time. This is > particularly an issue if the user turns off speculation and expects a single > attempt of a task to run at any point in time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5111) Unnecessary RM address DNS lookup can cause unnecessary slowness on every JHS page load
Sandy Ryza created MAPREDUCE-5111: - Summary: Unnecessary RM address DNS lookup can cause unnecessary slowness on every JHS page load Key: MAPREDUCE-5111 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5111 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobhistoryserver Affects Versions: 2.0.3-alpha Reporter: Sandy Ryza Assignee: Sandy Ryza When I run the job history server locally, every page load takes in the 10s of seconds. I profiled the process and discovered that all the extra time was spent inside YarnConfiguration#getRMWebAppURL, trying to resolve 0.0.0.0 to a hostname. When I changed my yarn.resourcemanager.address to localhost, the page load times decreased drastically. There's no that we need to perform this resolution on every page load. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5111) RM address DNS lookup can cause unnecessary slowness on every JHS page load
[ https://issues.apache.org/jira/browse/MAPREDUCE-5111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandy Ryza updated MAPREDUCE-5111: -- Summary: RM address DNS lookup can cause unnecessary slowness on every JHS page load (was: Unnecessary RM address DNS lookup can cause unnecessary slowness on every JHS page load ) > RM address DNS lookup can cause unnecessary slowness on every JHS page load > > > Key: MAPREDUCE-5111 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5111 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: jobhistoryserver >Affects Versions: 2.0.3-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > > When I run the job history server locally, every page load takes in the 10s > of seconds. I profiled the process and discovered that all the extra time > was spent inside YarnConfiguration#getRMWebAppURL, trying to resolve 0.0.0.0 > to a hostname. When I changed my yarn.resourcemanager.address to localhost, > the page load times decreased drastically. > There's no that we need to perform this resolution on every page load. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5081) Backport DistCpV2 and the related JIRAs to branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated MAPREDUCE-5081: -- Attachment: m5081_20130328.patch m5081_20130328.patch: - keeps distcp (version 1) unchanged and adds a new distcp2 command; - also updates the doc. > Backport DistCpV2 and the related JIRAs to branch-1 > --- > > Key: MAPREDUCE-5081 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5081 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: distcp >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: DistCp.java.diff, m5081_20130328.patch, > m5981_20130321b.patch, m5981_20130321.patch, m5981_20130323.patch > > > Here is a list of DistCpV2 JIRAs: > - MAPREDUCE-2765: DistCpV2 main jira > - HADOOP-8703: turn CRC checking off for 0 byte size > - HDFS-3054: distcp -skipcrccheck has no effect. > - HADOOP-8431: Running distcp without args throws IllegalArgumentException > - HADOOP-8775: non-positive value to -bandwidth > - MAPREDUCE-4654: TestDistCp is ignored > - HADOOP-9022: distcp fails to copy file if -m 0 specified > - HADOOP-9025: TestCopyListing failing > - MAPREDUCE-5075: DistCp leaks input file handles > - distcp part of HADOOP-8341: Fix findbugs issues in hadoop-tools > - MAPREDUCE-5014: custom CopyListing -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5069) add concrete common implementations of CombineFileInputFormat
[ https://issues.apache.org/jira/browse/MAPREDUCE-5069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615979#comment-13615979 ] Sandy Ryza commented on MAPREDUCE-5069: --- Sangjin, I think you're more likely to get a response from a committer by shooting an email to mapreduce-dev list. Another nit about the patch: it might make sense to rename RecordReaderWrapper to something like CombineFileRecordReaderWrapper so that it's clear it's something specific to CombineFileInputFormat. > add concrete common implementations of CombineFileInputFormat > - > > Key: MAPREDUCE-5069 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5069 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, mrv2 >Affects Versions: 2.0.3-alpha >Reporter: Sangjin Lee >Priority: Minor > Attachments: MAPREDUCE-5069-1.patch, MAPREDUCE-5069-2.patch, > MAPREDUCE-5069-3.patch, MAPREDUCE-5069-4.patch, MAPREDUCE-5069.patch > > > CombineFileInputFormat is abstract, and its specific equivalents to > TextInputFormat, SequenceFileInputFormat, etc. are currently not in the > hadoop code base. > These sound like very common need wherever CombineFileInputFormat is used, > and different folks would write the same code over and over to achieve the > same goal. It sounds very natural for hadoop to provide at least the text and > sequence file implementations of the CombineFileInputFormat class. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5081) Backport DistCpV2 and the related JIRAs to branch-1
[
https://issues.apache.org/jira/browse/MAPREDUCE-5081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615978#comment-13615978
]
Tsz Wo (Nicholas), SZE commented on MAPREDUCE-5081:
---
{quote}
> 1. sslConfig.xml, distcp-default.xml is missing Apache license header.
Will do.
{quote}
In branch-1, other xml files like core-default.xml and hdfs-default.xml also do
not have license header. So let's don't fix the new xml here and fix them with
other xml files together later.
> Backport DistCpV2 and the related JIRAs to branch-1
> ---
>
> Key: MAPREDUCE-5081
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5081
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: distcp
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Attachments: DistCp.java.diff, m5981_20130321b.patch,
> m5981_20130321.patch, m5981_20130323.patch
>
>
> Here is a list of DistCpV2 JIRAs:
> - MAPREDUCE-2765: DistCpV2 main jira
> - HADOOP-8703: turn CRC checking off for 0 byte size
> - HDFS-3054: distcp -skipcrccheck has no effect.
> - HADOOP-8431: Running distcp without args throws IllegalArgumentException
> - HADOOP-8775: non-positive value to -bandwidth
> - MAPREDUCE-4654: TestDistCp is ignored
> - HADOOP-9022: distcp fails to copy file if -m 0 specified
> - HADOOP-9025: TestCopyListing failing
> - MAPREDUCE-5075: DistCp leaks input file handles
> - distcp part of HADOOP-8341: Fix findbugs issues in hadoop-tools
> - MAPREDUCE-5014: custom CopyListing
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5081) Backport DistCpV2 and the related JIRAs to branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated MAPREDUCE-5081: -- Attachment: DistCp.java.diff DistCp.java.diff: diff between branch-1 tools/distcp2/DistCp.java (not tools/DistCp.java) and trunk tools/DistCp.java diff b-1/src/tools/org/apache/hadoop/tools/distcp2/DistCp.java t3/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java > DistCp.java.diff > Backport DistCpV2 and the related JIRAs to branch-1 > --- > > Key: MAPREDUCE-5081 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5081 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: distcp >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: DistCp.java.diff, m5981_20130321b.patch, > m5981_20130321.patch, m5981_20130323.patch > > > Here is a list of DistCpV2 JIRAs: > - MAPREDUCE-2765: DistCpV2 main jira > - HADOOP-8703: turn CRC checking off for 0 byte size > - HDFS-3054: distcp -skipcrccheck has no effect. > - HADOOP-8431: Running distcp without args throws IllegalArgumentException > - HADOOP-8775: non-positive value to -bandwidth > - MAPREDUCE-4654: TestDistCp is ignored > - HADOOP-9022: distcp fails to copy file if -m 0 specified > - HADOOP-9025: TestCopyListing failing > - MAPREDUCE-5075: DistCp leaks input file handles > - distcp part of HADOOP-8341: Fix findbugs issues in hadoop-tools > - MAPREDUCE-5014: custom CopyListing -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5081) Backport DistCpV2 and the related JIRAs to branch-1
[ https://issues.apache.org/jira/browse/MAPREDUCE-5081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615961#comment-13615961 ] Tsz Wo (Nicholas), SZE commented on MAPREDUCE-5081: --- > 1. sslConfig.xml, distcp-default.xml is missing Apache license header. Will do. > 2. There is a lot difference in DistCp.java in trunk and DistCp.java in this > patch? Are you sure that you diff the correct file since there two DistCp.java files? Let me post the diff. > 3. ... Why not leave the old distcp as is and add a new command for distcp2? Sure, I can add a new command for distcp2. > Backport DistCpV2 and the related JIRAs to branch-1 > --- > > Key: MAPREDUCE-5081 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5081 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: distcp >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: m5981_20130321b.patch, m5981_20130321.patch, > m5981_20130323.patch > > > Here is a list of DistCpV2 JIRAs: > - MAPREDUCE-2765: DistCpV2 main jira > - HADOOP-8703: turn CRC checking off for 0 byte size > - HDFS-3054: distcp -skipcrccheck has no effect. > - HADOOP-8431: Running distcp without args throws IllegalArgumentException > - HADOOP-8775: non-positive value to -bandwidth > - MAPREDUCE-4654: TestDistCp is ignored > - HADOOP-9022: distcp fails to copy file if -m 0 specified > - HADOOP-9025: TestCopyListing failing > - MAPREDUCE-5075: DistCp leaks input file handles > - distcp part of HADOOP-8341: Fix findbugs issues in hadoop-tools > - MAPREDUCE-5014: custom CopyListing -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5110) Long task launch delays can lead to multiple parallel attempts of the task
[ https://issues.apache.org/jira/browse/MAPREDUCE-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5110: Attachment: expose-mr-5110.patch Uploading a test case (expose-mr-5110.patch) that exposes the failure. > Long task launch delays can lead to multiple parallel attempts of the task > -- > > Key: MAPREDUCE-5110 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5110 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 1.1.2 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Attachments: expose-mr-5110.patch > > > If a task takes too long to launch, the JT expires the task and schedules > another attempt. The earlier attempt can start after the later attempt > leading to two parallel attempts running at the same time. This is > particularly an issue if the user turns off speculation and expects a single > attempt of a task to run at any point in time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5110) Long task launch delays can lead to multiple parallel attempts of the task
Karthik Kambatla created MAPREDUCE-5110: --- Summary: Long task launch delays can lead to multiple parallel attempts of the task Key: MAPREDUCE-5110 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5110 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 1.1.2 Reporter: Karthik Kambatla Assignee: Karthik Kambatla If a task takes too long to launch, the JT expires the task and schedules another attempt. The earlier attempt can start after the later attempt leading to two parallel attempts running at the same time. This is particularly an issue if the user turns off speculation and expects a single attempt of a task to run at any point in time. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5109) Job view-acl should apply to job listing too
Arun C Murthy created MAPREDUCE-5109: Summary: Job view-acl should apply to job listing too Key: MAPREDUCE-5109 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5109 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Arun C Murthy Assignee: Arun C Murthy Job view-acl should apply to job listing too, currently it only applies to job details pages. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2389) Spurious EOFExceptions reading SpillRecord index files
[ https://issues.apache.org/jira/browse/MAPREDUCE-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615850#comment-13615850 ] Jordan Mendelson commented on MAPREDUCE-2389: - Is there any update to this? We are hitting this on EMR, with 20-30 exceptions per job. > Spurious EOFExceptions reading SpillRecord index files > -- > > Key: MAPREDUCE-2389 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2389 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: tasktracker >Affects Versions: 0.22.0 > Environment: Seen on RHEL 5.5, RHEL 6.0, local dirs on ext3, Java > 6u20 and 6u24 >Reporter: Todd Lipcon > Attachments: stap-output.txt > > > In large jobs, I see around 1 shuffle fetch out of every million fetches fail > with an EOFException reading the SpillRecord index file. After lots of > investigation, including systemtap, it looks like the read() syscall is > actually returning a premature "0" result for no reason, so this is likely a > kernel or filesystem bug which is exacerbated by some workload the TT does. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5108) Review support for Binary Compatibility for MR applications via YARN
Arun C Murthy created MAPREDUCE-5108: Summary: Review support for Binary Compatibility for MR applications via YARN Key: MAPREDUCE-5108 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5108 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 2.0.3-alpha Reporter: Arun C Murthy Assignee: Zhijie Shen Priority: Blocker Fix For: 2.0.5-beta As we get ready to ship out a beta/stable version of hadoop-2, it makes sense to spend time reviewing support for existing MR applications (hadoop-1) to migrate seamlessly. We've done various pieces of work over time, let's track progress and document things clearly. [~zjshen] has done a bunch of testing and results look very promising so far. The aim is to support applications using org.apache.hadoop.mapred.* api in a binary compatible manner in hadoop-2 - thus, users can just take existing MR applications jars, point them at YARN clusters and things just work. Clearly, we might have some corner cases (haven't seen many so far), including semantics (not just apis); however the intent is to, at least, document them throughly if not actually fix them as feasible. Also, it's clear that we will *not* be able to support org.apache.hadoop.mapreduce api due to the interface changes we made in hadoop-0.21 (sigh), but given that vast majority of users use the mapred api, it's a reasonable step to easing migration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5066) JobTracker should set a timeout when calling into job.end.notification.url
[ https://issues.apache.org/jira/browse/MAPREDUCE-5066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615757#comment-13615757 ] Ivan Mitic commented on MAPREDUCE-5066: --- bq. You'll need to port this to branch-1 and branch-2(trunk). Thanks for reviewing Arun! Will attach branch-1 and branch-2 compatible patches in a bit. bq. Also, if you could minimize formatting changes it will make it easier to review (for future ref). Thanks! Sounds good, thanks. I noticed that formatting was incorrect so I intentionally made this change :) Will avoid doing formatting changes with bugfixes going forward. > JobTracker should set a timeout when calling into job.end.notification.url > -- > > Key: MAPREDUCE-5066 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5066 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 1-win, 2.0.3-alpha, 1.3.0 >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Attachments: MAPREDUCE-5066.branch-1-win.2.patch, > MAPREDUCE-5066.branch-1-win.patch > > > In current code, timeout is not specified when JobTracker (JobEndNotifier) > calls into the notification URL. When the given URL points to a server that > will not respond for a long time, job notifications are completely stuck > (given that we have only a single thread processing all notifications). We've > seen this cause noticeable delays in job execution in components that rely on > job end notifications (like Oozie workflows). > I propose we introduce a configurable timeout option and set a default to a > reasonably small value. > If we want, we can also introduce a configurable number of workers processing > the notification queue (not sure if this is needed though at this point). > I will prepare a patch soon. Please comment back. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5066) JobTracker should set a timeout when calling into job.end.notification.url
[ https://issues.apache.org/jira/browse/MAPREDUCE-5066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615753#comment-13615753 ] Arun C Murthy commented on MAPREDUCE-5066: -- Ivan - the patch looks good. You'll need to port this to branch-1 and branch-2(trunk). Also, if you could minimize formatting changes it will make it easier to review (for future ref). Thanks! > JobTracker should set a timeout when calling into job.end.notification.url > -- > > Key: MAPREDUCE-5066 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5066 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 1-win, 2.0.3-alpha, 1.3.0 >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Attachments: MAPREDUCE-5066.branch-1-win.2.patch, > MAPREDUCE-5066.branch-1-win.patch > > > In current code, timeout is not specified when JobTracker (JobEndNotifier) > calls into the notification URL. When the given URL points to a server that > will not respond for a long time, job notifications are completely stuck > (given that we have only a single thread processing all notifications). We've > seen this cause noticeable delays in job execution in components that rely on > job end notifications (like Oozie workflows). > I propose we introduce a configurable timeout option and set a default to a > reasonably small value. > If we want, we can also introduce a configurable number of workers processing > the notification queue (not sure if this is needed though at this point). > I will prepare a patch soon. Please comment back. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5007) fix coverage org.apache.hadoop.mapreduce.v2.hs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-5007: - Assignee: Aleksey Gorshkov > fix coverage org.apache.hadoop.mapreduce.v2.hs > -- > > Key: MAPREDUCE-5007 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5007 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.7 >Reporter: Aleksey Gorshkov >Assignee: Aleksey Gorshkov > Attachments: MAPREDUCE-5007-branch-0.23.patch, > MAPREDUCE-5007-branch-2.patch, MAPREDUCE-5007-trunk.patch > > > fix coverage org.apache.hadoop.mapreduce.v2.hs > MAPREDUCE-5007-trunk.patch patch for trunk > MAPREDUCE-5007-branch-2.patch patch for branch-2 > MAPREDUCE-5007-branch-0.23.patch patch for branch-0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5098) Fix findbugs warnings in gridmix
[ https://issues.apache.org/jira/browse/MAPREDUCE-5098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5098: Status: Open (was: Patch Available) > Fix findbugs warnings in gridmix > > > Key: MAPREDUCE-5098 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5098 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/gridmix >Affects Versions: 2.0.3-alpha >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > Labels: findbugs > Attachments: mr-5098.patch, mr-5098.patch > > > Work on MAPREDUCE-5077 has exposed a bunch of findbugs warnings in gridmix > code. > https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3459//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-gridmix.html > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5107) MR AM needs to be able to handle unavailable HDFS
[ https://issues.apache.org/jira/browse/MAPREDUCE-5107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-5107: -- Issue Type: Improvement (was: Bug) > MR AM needs to be able to handle unavailable HDFS > - > > Key: MAPREDUCE-5107 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5107 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Bikas Saha > > Stuff like not starting tasks when HDFS is unavailable. Not terminating tasks > that are spinning on HDFS to become available. These features exist in > branch-1 JT. Some of this may already be in place in the current app master. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5107) MR AM needs to be able to handle unavailable HDFS
Bikas Saha created MAPREDUCE-5107: - Summary: MR AM needs to be able to handle unavailable HDFS Key: MAPREDUCE-5107 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5107 Project: Hadoop Map/Reduce Issue Type: Bug Reporter: Bikas Saha Stuff like not starting tasks when HDFS is unavailable. Not terminating tasks that are spinning on HDFS to become available. These features exist in branch-1 JT. Some of this may already be in place in the current app master. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4875) coverage fixing for org.apache.hadoop.mapred
[
https://issues.apache.org/jira/browse/MAPREDUCE-4875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615546#comment-13615546
]
Hadoop QA commented on MAPREDUCE-4875:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12575695/MAPREDUCE-4875-branch-0.23-b.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3474//console
This message is automatically generated.
> coverage fixing for org.apache.hadoop.mapred
>
>
> Key: MAPREDUCE-4875
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4875
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
>Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6
>Reporter: Aleksey Gorshkov
>Assignee: Aleksey Gorshkov
> Fix For: 3.0.0, 2.0.5-beta
>
> Attachments: MAPREDUCE-4875-branch-0.23-b.patch,
> MAPREDUCE-4875-branch-0.23.patch, MAPREDUCE-4875-trunk-b.patch,
> MAPREDUCE-4875-trunk.patch
>
>
> added some tests for org.apache.hadoop.mapred
> MAPREDUCE-4875-trunk.patch for trunk and branch-2
> MAPREDUCE-4875-branch-0.23.patch for branch-0.23
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4875) coverage fixing for org.apache.hadoop.mapred
[ https://issues.apache.org/jira/browse/MAPREDUCE-4875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Klochkov updated MAPREDUCE-4875: --- Target Version/s: 0.23.6 Status: Patch Available (was: Reopened) > coverage fixing for org.apache.hadoop.mapred > > > Key: MAPREDUCE-4875 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4875 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 2.0.3-alpha, 3.0.0, 0.23.6 >Reporter: Aleksey Gorshkov >Assignee: Aleksey Gorshkov > Fix For: 3.0.0, 2.0.5-beta > > Attachments: MAPREDUCE-4875-branch-0.23-b.patch, > MAPREDUCE-4875-branch-0.23.patch, MAPREDUCE-4875-trunk-b.patch, > MAPREDUCE-4875-trunk.patch > > > added some tests for org.apache.hadoop.mapred > MAPREDUCE-4875-trunk.patch for trunk and branch-2 > MAPREDUCE-4875-branch-0.23.patch for branch-0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5006) streaming tests failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615475#comment-13615475 ] Chris Nauroth commented on MAPREDUCE-5006: -- Thank you, Sandy and Tom! > streaming tests failing > --- > > Key: MAPREDUCE-5006 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5006 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Affects Versions: 2.0.5-beta >Reporter: Alejandro Abdelnur >Assignee: Sandy Ryza > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5006-1.patch, MAPREDUCE-5006-2.patch, > MAPREDUCE-5006.patch > > > The following 2 tests are failing in trunk > * org.apache.hadoop.streaming.TestStreamReduceNone > * org.apache.hadoop.streaming.TestStreamXmlRecordReader -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5106) mapreduce.jobtracker.split.metainfo.maxsize can be set at job level
[ https://issues.apache.org/jira/browse/MAPREDUCE-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615365#comment-13615365 ] Abhishek Gayakwad commented on MAPREDUCE-5106: -- I am using 0.20.2 version, where config is loaded from job.xml this.localJobFile = default_conf.getLocalPath(JobTracker.SUBDIR+"/"+jobId + ".xml"); conf = new JobConf(localJobFile); I have tested this as well. It is okay to close this issue if it is fixed in 1.0. > mapreduce.jobtracker.split.metainfo.maxsize can be set at job level > > > Key: MAPREDUCE-5106 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5106 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.2 >Reporter: Abhishek Gayakwad >Priority: Minor > > mapreduce.jobtracker.split.metainfo.maxsize gives an impression that this > property can be set at JT level only and will require JT restart in case of > any modification. But actually this can be set at individual job level. > So either this property should be named properly or should be restricted to > JT level only. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4991) coverage for gridmix
[ https://issues.apache.org/jira/browse/MAPREDUCE-4991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4991: - Status: Open (was: Patch Available) > coverage for gridmix > > > Key: MAPREDUCE-4991 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4991 > Project: Hadoop Map/Reduce > Issue Type: Test >Affects Versions: 2.0.3-alpha, 3.0.0, 0.23.7 >Reporter: Aleksey Gorshkov > Attachments: MAPREDUCE-4991-branch-0.23.patch, > MAPREDUCE-4991-branch-2.patch, MAPREDUCE-4991-trunk.patch > > > fix coverage for GridMix > MAPREDUCE-4991-trunk.patch patch for thunk > MAPREDUCE-4991-branch-2.patch for branch-2 and > MAPREDUCE-4991-branch-0.23.patch for branch-0.23 > known fail > -org.apache.hadoop.mapred.gridmix.TestGridmixSummary.testExecutionSummarizer. > It is for next issue -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4875) coverage fixing for org.apache.hadoop.mapred
[ https://issues.apache.org/jira/browse/MAPREDUCE-4875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615281#comment-13615281 ] Hudson commented on MAPREDUCE-4875: --- Integrated in Hadoop-Mapreduce-trunk #1385 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1385/]) MAPREDUCE-4875. coverage fixing for org.apache.hadoop.mapred (Aleksey Gorshkov via bobby) (Revision 1461146) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461146 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptListenerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/QueueManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskStatus.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestClock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobConf.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestOldMethodsJobID.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestSkipBadRecords.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestTaskLog.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestTaskLogAppender.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/resources/mapred-queues.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestIFile.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMultiFileSplit.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestNetworkedJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestQueueConfigurationParser.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestStatisticsCollector.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestTextInputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestTextOutputFormat.java > coverage fixing for org.apache.hadoop.mapred > > > Key: MAPREDUCE-4875 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4875 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 >Reporter: Aleksey Gorshkov >Assignee: Aleksey Gorshkov > Fix For: 3.0.0, 2.0.5-beta > > Attachments: MAPREDUCE-4875-branch-0.23-b.patch, > MAPREDUCE-4875-branch-0.23.patch, MAPREDUCE-4875-trunk-b.patch, > MAPREDUCE-4875-trunk.patch > > > added some tests for org.apache.hadoop.mapred > MAPREDUCE-4875-trunk.patch for trunk and branch-2 > MAPREDUCE-4875-branch-0.23.patch for branch-0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA a
[jira] [Commented] (MAPREDUCE-5077) Cleanup: mapreduce.util.ResourceCalculatorPlugin and related code should be removed
[
https://issues.apache.org/jira/browse/MAPREDUCE-5077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615278#comment-13615278
]
Hudson commented on MAPREDUCE-5077:
---
Integrated in Hadoop-Mapreduce-trunk #1385 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1385/])
MAPREDUCE-5077. Remove mapreduce.util.ResourceCalculatorPlugin and related
code. Contributed by Karthik Kambatla. (Revision 1461251)
Result = SUCCESS
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461251
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/LinuxResourceCalculatorPlugin.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ProcfsBasedProcessTree.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceCalculatorPlugin.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/util/TestLinuxResourceCalculatorPlugin.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/DummyResourceCalculatorPlugin.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/LinuxMemoryCalculatorPlugin.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/MemoryCalculatorPlugin.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/TestProcfsBasedProcessTree.java
*
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/LoadJob.java
*
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/CumulativeCpuUsageEmulatorPlugin.java
*
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/ResourceUsageEmulatorPlugin.java
*
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/ResourceUsageMatcher.java
*
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/TotalHeapUsageEmulatorPlugin.java
*
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/DummyResourceCalculatorPlugin.java
*
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestGridmixMemoryEmulation.java
*
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestResourceUsageEmulators.java
> Cleanup: mapreduce.util.ResourceCalculatorPlugin and related code should be
> removed
> ---
>
> Key: MAPREDUCE-5077
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5077
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.0.3-alpha
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
>Priority: Minor
> Fix For: 2.0.5-beta
>
> Attachments: mr-5077-branch2.patch, mr-5077.patch, mr-5077.patch,
> mr-5077.patch
>
>
> {{ResourceCalculatorPlugin}} and {{ProcfsBasedProcessTree}} have moved to
> yarn.util and the mapreduce.util versions don't seem to be used anymore.
> Should remove related code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5006) streaming tests failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615280#comment-13615280 ] Hudson commented on MAPREDUCE-5006: --- Integrated in Hadoop-Mapreduce-trunk #1385 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1385/]) MAPREDUCE-5006. Fix failing streaming tests due to MAPREDUCE-4994. Contributed by Sandy Ryza. (Revision 1461541) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461541 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalClientProtocolProvider.java * /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TestStreamReduceNone.java * /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TestStreamXmlRecordReader.java > streaming tests failing > --- > > Key: MAPREDUCE-5006 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5006 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Affects Versions: 2.0.5-beta >Reporter: Alejandro Abdelnur >Assignee: Sandy Ryza > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5006-1.patch, MAPREDUCE-5006-2.patch, > MAPREDUCE-5006.patch > > > The following 2 tests are failing in trunk > * org.apache.hadoop.streaming.TestStreamReduceNone > * org.apache.hadoop.streaming.TestStreamXmlRecordReader -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4994) -jt generic command line option does not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615276#comment-13615276 ] Hudson commented on MAPREDUCE-4994: --- Integrated in Hadoop-Mapreduce-trunk #1385 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1385/]) MAPREDUCE-5006. Fix failing streaming tests due to MAPREDUCE-4994. Contributed by Sandy Ryza. (Revision 1461541) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461541 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalClientProtocolProvider.java * /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TestStreamReduceNone.java * /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TestStreamXmlRecordReader.java > -jt generic command line option does not work > - > > Key: MAPREDUCE-4994 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4994 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.0.2-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-4994-1.patch, MAPREDUCE-4994-addendum-1.patch, > MAPREDUCE-4994-addendum.patch, MAPREDUCE-4994.patch > > > hadoop jar myjar.jar MyDriver -fs file:/// -jt local input.txt output/ > should run a job using the local file system and the local job runner. > Instead it tries to connect to a jobtracker. > hadoop jar myjar.jar MyDriver -fs file:/// -jt host:port input.txt output/ > does not use the given host/port > This appears to be because Cluster#initialize, which loads the > ClientProtocol, contains no special handling for mapred.job.tracker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4875) coverage fixing for org.apache.hadoop.mapred
[ https://issues.apache.org/jira/browse/MAPREDUCE-4875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615209#comment-13615209 ] Hudson commented on MAPREDUCE-4875: --- Integrated in Hadoop-Hdfs-trunk #1357 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1357/]) MAPREDUCE-4875. coverage fixing for org.apache.hadoop.mapred (Aleksey Gorshkov via bobby) (Revision 1461146) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461146 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptListenerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/QueueManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskStatus.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestClock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobConf.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestOldMethodsJobID.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestSkipBadRecords.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestTaskLog.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestTaskLogAppender.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/resources/mapred-queues.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestIFile.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMultiFileSplit.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestNetworkedJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestQueueConfigurationParser.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestStatisticsCollector.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestTextInputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestTextOutputFormat.java > coverage fixing for org.apache.hadoop.mapred > > > Key: MAPREDUCE-4875 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4875 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 >Reporter: Aleksey Gorshkov >Assignee: Aleksey Gorshkov > Fix For: 3.0.0, 2.0.5-beta > > Attachments: MAPREDUCE-4875-branch-0.23-b.patch, > MAPREDUCE-4875-branch-0.23.patch, MAPREDUCE-4875-trunk-b.patch, > MAPREDUCE-4875-trunk.patch > > > added some tests for org.apache.hadoop.mapred > MAPREDUCE-4875-trunk.patch for trunk and branch-2 > MAPREDUCE-4875-branch-0.23.patch for branch-0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrat
[jira] [Commented] (MAPREDUCE-5077) Cleanup: mapreduce.util.ResourceCalculatorPlugin and related code should be removed
[
https://issues.apache.org/jira/browse/MAPREDUCE-5077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615207#comment-13615207
]
Hudson commented on MAPREDUCE-5077:
---
Integrated in Hadoop-Hdfs-trunk #1357 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1357/])
MAPREDUCE-5077. Remove mapreduce.util.ResourceCalculatorPlugin and related
code. Contributed by Karthik Kambatla. (Revision 1461251)
Result = FAILURE
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461251
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/LinuxResourceCalculatorPlugin.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ProcfsBasedProcessTree.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceCalculatorPlugin.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/util/TestLinuxResourceCalculatorPlugin.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/DummyResourceCalculatorPlugin.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/LinuxMemoryCalculatorPlugin.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/MemoryCalculatorPlugin.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/TestProcfsBasedProcessTree.java
*
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/LoadJob.java
*
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/CumulativeCpuUsageEmulatorPlugin.java
*
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/ResourceUsageEmulatorPlugin.java
*
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/ResourceUsageMatcher.java
*
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/TotalHeapUsageEmulatorPlugin.java
*
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/DummyResourceCalculatorPlugin.java
*
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestGridmixMemoryEmulation.java
*
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestResourceUsageEmulators.java
> Cleanup: mapreduce.util.ResourceCalculatorPlugin and related code should be
> removed
> ---
>
> Key: MAPREDUCE-5077
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5077
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.0.3-alpha
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
>Priority: Minor
> Fix For: 2.0.5-beta
>
> Attachments: mr-5077-branch2.patch, mr-5077.patch, mr-5077.patch,
> mr-5077.patch
>
>
> {{ResourceCalculatorPlugin}} and {{ProcfsBasedProcessTree}} have moved to
> yarn.util and the mapreduce.util versions don't seem to be used anymore.
> Should remove related code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (MAPREDUCE-4875) coverage fixing for org.apache.hadoop.mapred
[ https://issues.apache.org/jira/browse/MAPREDUCE-4875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Gorshkov reopened MAPREDUCE-4875: - patch for branch-0.23 attached > coverage fixing for org.apache.hadoop.mapred > > > Key: MAPREDUCE-4875 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4875 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 >Reporter: Aleksey Gorshkov >Assignee: Aleksey Gorshkov > Fix For: 3.0.0, 2.0.5-beta > > Attachments: MAPREDUCE-4875-branch-0.23-b.patch, > MAPREDUCE-4875-branch-0.23.patch, MAPREDUCE-4875-trunk-b.patch, > MAPREDUCE-4875-trunk.patch > > > added some tests for org.apache.hadoop.mapred > MAPREDUCE-4875-trunk.patch for trunk and branch-2 > MAPREDUCE-4875-branch-0.23.patch for branch-0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4875) coverage fixing for org.apache.hadoop.mapred
[ https://issues.apache.org/jira/browse/MAPREDUCE-4875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615192#comment-13615192 ] Aleksey Gorshkov commented on MAPREDUCE-4875: - I've attached updated patch for branch-0.23 MAPREDUCE-4875-branch-0.23-b.patch Could you apply this patch to branch-0.23 > coverage fixing for org.apache.hadoop.mapred > > > Key: MAPREDUCE-4875 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4875 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 >Reporter: Aleksey Gorshkov >Assignee: Aleksey Gorshkov > Fix For: 3.0.0, 2.0.5-beta > > Attachments: MAPREDUCE-4875-branch-0.23-b.patch, > MAPREDUCE-4875-branch-0.23.patch, MAPREDUCE-4875-trunk-b.patch, > MAPREDUCE-4875-trunk.patch > > > added some tests for org.apache.hadoop.mapred > MAPREDUCE-4875-trunk.patch for trunk and branch-2 > MAPREDUCE-4875-branch-0.23.patch for branch-0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4875) coverage fixing for org.apache.hadoop.mapred
[ https://issues.apache.org/jira/browse/MAPREDUCE-4875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aleksey Gorshkov updated MAPREDUCE-4875: Attachment: MAPREDUCE-4875-branch-0.23-b.patch > coverage fixing for org.apache.hadoop.mapred > > > Key: MAPREDUCE-4875 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4875 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 >Reporter: Aleksey Gorshkov >Assignee: Aleksey Gorshkov > Fix For: 3.0.0, 2.0.5-beta > > Attachments: MAPREDUCE-4875-branch-0.23-b.patch, > MAPREDUCE-4875-branch-0.23.patch, MAPREDUCE-4875-trunk-b.patch, > MAPREDUCE-4875-trunk.patch > > > added some tests for org.apache.hadoop.mapred > MAPREDUCE-4875-trunk.patch for trunk and branch-2 > MAPREDUCE-4875-branch-0.23.patch for branch-0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5075) DistCp leaks input file handles
[
https://issues.apache.org/jira/browse/MAPREDUCE-5075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615176#comment-13615176
]
Hudson commented on MAPREDUCE-5075:
---
Integrated in Hadoop-Hdfs-0.23-Build #566 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/566/])
MAPREDUCE-5075. DistCp leaks input file handles since ThrottledInputStream
does not close the wrapped InputStream. (Chris Nauroth via tgraves) (Revision
1461305)
Result = SUCCESS
tgraves :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461305
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
*
/hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/util/ThrottledInputStream.java
*
/hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestIntegration.java
> DistCp leaks input file handles
> ---
>
> Key: MAPREDUCE-5075
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5075
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: distcp
>Affects Versions: 3.0.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Fix For: 0.23.7, 2.0.5-beta
>
> Attachments: MAPREDUCE-5075.1.patch
>
>
> DistCp wraps the {{InputStream}} for each input file it reads in an instance
> of {{ThrottledInputStream}}. This class does not close the wrapped
> {{InputStream}}. {{RetriableFileCopyCommand}} guarantees that the
> {{ThrottledInputStream}} gets closed, but without closing the underlying
> wrapped stream, it still leaks a file handle.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5106) mapreduce.jobtracker.split.metainfo.maxsize can be set at job level
[ https://issues.apache.org/jira/browse/MAPREDUCE-5106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J resolved MAPREDUCE-5106. Resolution: Not A Problem Looking at branch-1's implementation (JobInProgress), this is not a current problem. Resolving as Not a Problem. The config is taken from the JobTracker.getConf() call so the two cannot be mixed as reported. What version were you using? > mapreduce.jobtracker.split.metainfo.maxsize can be set at job level > > > Key: MAPREDUCE-5106 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5106 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.20.2 >Reporter: Abhishek Gayakwad >Priority: Minor > > mapreduce.jobtracker.split.metainfo.maxsize gives an impression that this > property can be set at JT level only and will require JT restart in case of > any modification. But actually this can be set at individual job level. > So either this property should be named properly or should be restricted to > JT level only. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5006) streaming tests failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615143#comment-13615143 ] Hudson commented on MAPREDUCE-5006: --- Integrated in Hadoop-trunk-Commit #3533 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3533/]) MAPREDUCE-5006. Fix failing streaming tests due to MAPREDUCE-4994. Contributed by Sandy Ryza. (Revision 1461541) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461541 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalClientProtocolProvider.java * /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TestStreamReduceNone.java * /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TestStreamXmlRecordReader.java > streaming tests failing > --- > > Key: MAPREDUCE-5006 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5006 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Affects Versions: 2.0.5-beta >Reporter: Alejandro Abdelnur >Assignee: Sandy Ryza > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5006-1.patch, MAPREDUCE-5006-2.patch, > MAPREDUCE-5006.patch > > > The following 2 tests are failing in trunk > * org.apache.hadoop.streaming.TestStreamReduceNone > * org.apache.hadoop.streaming.TestStreamXmlRecordReader -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4994) -jt generic command line option does not work
[ https://issues.apache.org/jira/browse/MAPREDUCE-4994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615142#comment-13615142 ] Hudson commented on MAPREDUCE-4994: --- Integrated in Hadoop-trunk-Commit #3533 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3533/]) MAPREDUCE-5006. Fix failing streaming tests due to MAPREDUCE-4994. Contributed by Sandy Ryza. (Revision 1461541) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461541 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapred/LocalClientProtocolProvider.java * /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TestStreamReduceNone.java * /hadoop/common/trunk/hadoop-tools/hadoop-streaming/src/test/java/org/apache/hadoop/streaming/TestStreamXmlRecordReader.java > -jt generic command line option does not work > - > > Key: MAPREDUCE-4994 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4994 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client >Affects Versions: 2.0.2-alpha >Reporter: Sandy Ryza >Assignee: Sandy Ryza > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-4994-1.patch, MAPREDUCE-4994-addendum-1.patch, > MAPREDUCE-4994-addendum.patch, MAPREDUCE-4994.patch > > > hadoop jar myjar.jar MyDriver -fs file:/// -jt local input.txt output/ > should run a job using the local file system and the local job runner. > Instead it tries to connect to a jobtracker. > hadoop jar myjar.jar MyDriver -fs file:/// -jt host:port input.txt output/ > does not use the given host/port > This appears to be because Cluster#initialize, which loads the > ClientProtocol, contains no special handling for mapred.job.tracker. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5006) streaming tests failing
[ https://issues.apache.org/jira/browse/MAPREDUCE-5006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-5006: - Resolution: Fixed Fix Version/s: 2.0.5-beta Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Sorry I forgot to commit this one. I've just committed this to branch-2. Thanks Sandy! > streaming tests failing > --- > > Key: MAPREDUCE-5006 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5006 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: contrib/streaming >Affects Versions: 2.0.5-beta >Reporter: Alejandro Abdelnur >Assignee: Sandy Ryza > Fix For: 2.0.5-beta > > Attachments: MAPREDUCE-5006-1.patch, MAPREDUCE-5006-2.patch, > MAPREDUCE-5006.patch > > > The following 2 tests are failing in trunk > * org.apache.hadoop.streaming.TestStreamReduceNone > * org.apache.hadoop.streaming.TestStreamXmlRecordReader -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4875) coverage fixing for org.apache.hadoop.mapred
[ https://issues.apache.org/jira/browse/MAPREDUCE-4875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615109#comment-13615109 ] Hudson commented on MAPREDUCE-4875: --- Integrated in Hadoop-Yarn-trunk #168 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/168/]) MAPREDUCE-4875. coverage fixing for org.apache.hadoop.mapred (Aleksey Gorshkov via bobby) (Revision 1461146) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461146 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptListenerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobEndNotifier.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/QueueManager.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskStatus.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestClock.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobConf.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobInfo.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestOldMethodsJobID.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestSkipBadRecords.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestTaskLog.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestTaskLogAppender.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/resources/mapred-queues.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestIFile.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestMultiFileSplit.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestNetworkedJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestQueueConfigurationParser.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestStatisticsCollector.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestTextInputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestTextOutputFormat.java > coverage fixing for org.apache.hadoop.mapred > > > Key: MAPREDUCE-4875 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4875 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: test >Affects Versions: 3.0.0, 2.0.3-alpha, 0.23.6 >Reporter: Aleksey Gorshkov >Assignee: Aleksey Gorshkov > Fix For: 3.0.0, 2.0.5-beta > > Attachments: MAPREDUCE-4875-branch-0.23.patch, > MAPREDUCE-4875-trunk-b.patch, MAPREDUCE-4875-trunk.patch > > > added some tests for org.apache.hadoop.mapred > MAPREDUCE-4875-trunk.patch for trunk and branch-2 > MAPREDUCE-4875-branch-0.23.patch for branch-0.23 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: ht
[jira] [Commented] (MAPREDUCE-5077) Cleanup: mapreduce.util.ResourceCalculatorPlugin and related code should be removed
[
https://issues.apache.org/jira/browse/MAPREDUCE-5077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615107#comment-13615107
]
Hudson commented on MAPREDUCE-5077:
---
Integrated in Hadoop-Yarn-trunk #168 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/168/])
MAPREDUCE-5077. Remove mapreduce.util.ResourceCalculatorPlugin and related
code. Contributed by Karthik Kambatla. (Revision 1461251)
Result = SUCCESS
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1461251
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/LinuxResourceCalculatorPlugin.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ProcfsBasedProcessTree.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/util/ResourceCalculatorPlugin.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/util/TestLinuxResourceCalculatorPlugin.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/DummyResourceCalculatorPlugin.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/LinuxMemoryCalculatorPlugin.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/MemoryCalculatorPlugin.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/util/TestProcfsBasedProcessTree.java
*
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/LoadJob.java
*
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/CumulativeCpuUsageEmulatorPlugin.java
*
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/ResourceUsageEmulatorPlugin.java
*
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/ResourceUsageMatcher.java
*
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/main/java/org/apache/hadoop/mapred/gridmix/emulators/resourceusage/TotalHeapUsageEmulatorPlugin.java
*
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/DummyResourceCalculatorPlugin.java
*
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestGridmixMemoryEmulation.java
*
/hadoop/common/trunk/hadoop-tools/hadoop-gridmix/src/test/java/org/apache/hadoop/mapred/gridmix/TestResourceUsageEmulators.java
> Cleanup: mapreduce.util.ResourceCalculatorPlugin and related code should be
> removed
> ---
>
> Key: MAPREDUCE-5077
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5077
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.0.3-alpha
>Reporter: Karthik Kambatla
>Assignee: Karthik Kambatla
>Priority: Minor
> Fix For: 2.0.5-beta
>
> Attachments: mr-5077-branch2.patch, mr-5077.patch, mr-5077.patch,
> mr-5077.patch
>
>
> {{ResourceCalculatorPlugin}} and {{ProcfsBasedProcessTree}} have moved to
> yarn.util and the mapreduce.util versions don't seem to be used anymore.
> Should remove related code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5019) Fair scheduler should allow peremption on reducer only
[ https://issues.apache.org/jira/browse/MAPREDUCE-5019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615072#comment-13615072 ] Damien Hardy commented on MAPREDUCE-5019: - I just discover the option : mapred.reduce.slowstart.completed.maps (default is 0.05) that could be used to prevent big job from staling smaller ones waiting for reducer slots. Trying with a value of 0.8 (80% of mappers have to be terminated before starting reducers) > Fair scheduler should allow peremption on reducer only > -- > > Key: MAPREDUCE-5019 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5019 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv1, scheduler >Affects Versions: 2.0.2-alpha > Environment: CDH4.1.2 >Reporter: Damien Hardy >Priority: Minor > Labels: scheduler, scheduling > > Fair scheduler is very good. > But having a big MR job running lots of mapper and reducer( 10M + 10R ) > Then a small MR on the same pool (1M + 1R) > having slots for 10 mapper and 10 reducer > - The big job take all the map slots > - The small job wait for a map slot > - 1rst big job map task finish > - the small job take the map slot it needs > - meanwhile all the reducer of the big job take all the reducer slot to copy > and sort > - the small job end is map and wait for the all maps to end and for 1 > reducer to end before accessing for a reducer slot. > - all the reducer stalled after sorting waiting for the mapper to end one > by one... > If I have a big job and a lot of small, I don't want new small arriving and > killing running map tasks of big job to get a slot. > I think it could be useful that the small job can kill a reducer tasks (and > only reducer) to end before the big job finish all its map tasks and a > reducer. > rules can be : a job having all its map finished and waiting for reducer slot > can kill reducer tasks from a job that still have map slot running (assuming > they are just waiting for copy and sort) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
