[jira] [Updated] (MAPREDUCE-5287) Create a generic InputFormat wrapping any other InputFormat, to control the number of map tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nicu marasoiu updated MAPREDUCE-5287: - Priority: Minor (was: Major) Remaining Estimate: 4h Original Estimate: 4h Create a generic InputFormat wrapping any other InputFormat, to control the number of map tasks --- Key: MAPREDUCE-5287 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5287 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, performance Reporter: nicu marasoiu Priority: Minor Original Estimate: 4h Remaining Estimate: 4h I wrote a generic InputFormat that wraps any other InputFormat, and creates CompositeInputSplits to reduce the number of map tasks in a controllable manner while preserving data locality. A correspondent CompositeRecordReader is written to iterate through underlying RecordReaders as created by the underlying InputFormat for each underlying raw split. An application to this is to group TableSplits when the raw splits are coming from multiple regions and are filtered with key ranges. We use this to shard/distribute a time based incremental access to an hbase table. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5287) Create a generic InputFormat wrapping any other InputFormat, to control the number of map tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nicu marasoiu updated MAPREDUCE-5287: - Assignee: Owen O'Malley Create a generic InputFormat wrapping any other InputFormat, to control the number of map tasks --- Key: MAPREDUCE-5287 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5287 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, performance Reporter: nicu marasoiu Assignee: Owen O'Malley Priority: Minor Original Estimate: 4h Remaining Estimate: 4h I wrote a generic InputFormat that wraps any other InputFormat, and creates CompositeInputSplits to reduce the number of map tasks in a controllable manner while preserving data locality. A correspondent CompositeRecordReader is written to iterate through underlying RecordReaders as created by the underlying InputFormat for each underlying raw split. An application to this is to group TableSplits when the raw splits are coming from multiple regions and are filtered with key ranges. We use this to shard/distribute a time based incremental access to an hbase table. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5304) mapreduce.Job killTask/failTask/getTaskCompletionEvents methods have incompatible signature changes
[
https://issues.apache.org/jira/browse/MAPREDUCE-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13682113#comment-13682113
]
Hudson commented on MAPREDUCE-5304:
---
Integrated in Hadoop-Yarn-trunk #239 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/239/])
MAPREDUCE-5304. mapreduce.Job killTask/failTask/getTaskCompletionEvents
methods have incompatible signature changes. (kkambatl via tucu) (Revision
1492360)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1492360
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskCompletionEvent.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java
mapreduce.Job killTask/failTask/getTaskCompletionEvents methods have
incompatible signature changes
---
Key: MAPREDUCE-5304
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5304
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Affects Versions: 2.0.4-alpha
Reporter: Alejandro Abdelnur
Assignee: Karthik Kambatla
Priority: Blocker
Fix For: 2.1.0-beta
Attachments: mr-5304-1.patch
Pointed out by [~zjshen] in MAPREDUCE-4942.
In {{o.a.h.mapreduce.Job}} class, the following changed from Hadoop 1 to
Hadoop 2.
boolean failTask(TaskAttemptID): Change in return type from void to boolean.
boolean killTask(TaskAttemptID): Change in return type from void to boolean.
TaskCompletionEvent[] getTaskCompletionEvents(int): Change in return type
from org.apache.hadoop.mapred.TaskCompletionEvent[] to
org.apache.hadoop.mapreduce.TaskCompletionEvent[].
Using same rational as in other JIRAs, we should fix this to ensure Hadoop 1
to Hadoop 2 source compatibility (taking 0.23.x releases as a casualty as
there is not right way for everybody because we screwed up :( ). Flagging it
as incompatible change because of 0.23.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5259) TestTaskLog fails on Windows because of path separators missmatch
[
https://issues.apache.org/jira/browse/MAPREDUCE-5259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13682116#comment-13682116
]
Hudson commented on MAPREDUCE-5259:
---
Integrated in Hadoop-Yarn-trunk #239 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/239/])
MAPREDUCE-5259. TestTaskLog fails on Windows because of path separators
missmatch. Contributed by Ivan Mitic. (Revision 1492364)
Result = SUCCESS
cnauroth :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1492364
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestTaskLog.java
TestTaskLog fails on Windows because of path separators missmatch
-
Key: MAPREDUCE-5259
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5259
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Ivan Mitic
Assignee: Ivan Mitic
Fix For: 3.0.0, 2.1.0-beta
Attachments: MAPREDUCE-5259.patch
Test failure:
{noformat}
Running org.apache.hadoop.mapred.TestTaskLog
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.516 sec
FAILURE!
testTaskLog(org.apache.hadoop.mapred.TestTaskLog) Time elapsed: 409 sec
FAILURE!
junit.framework.AssertionFailedError: null
at junit.framework.Assert.fail(Assert.java:47)
at junit.framework.Assert.assertTrue(Assert.java:20)
at junit.framework.Assert.assertTrue(Assert.java:27)
at org.apache.hadoop.mapred.TestTaskLog.testTaskLog(TestTaskLog.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-5287) Create a generic InputFormat wrapping any other InputFormat, to control the number of map tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nicu marasoiu reassigned MAPREDUCE-5287: Assignee: nicu marasoiu (was: Owen O'Malley) Create a generic InputFormat wrapping any other InputFormat, to control the number of map tasks --- Key: MAPREDUCE-5287 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5287 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, performance Reporter: nicu marasoiu Assignee: nicu marasoiu Priority: Minor Original Estimate: 4h Remaining Estimate: 4h I wrote a generic InputFormat that wraps any other InputFormat, and creates CompositeInputSplits to reduce the number of map tasks in a controllable manner while preserving data locality. A correspondent CompositeRecordReader is written to iterate through underlying RecordReaders as created by the underlying InputFormat for each underlying raw split. An application to this is to group TableSplits when the raw splits are coming from multiple regions and are filtered with key ranges. We use this to shard/distribute a time based incremental access to an hbase table. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (MAPREDUCE-5287) Create a generic InputFormat wrapping any other InputFormat, to control the number of map tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on MAPREDUCE-5287 started by nicu marasoiu. Create a generic InputFormat wrapping any other InputFormat, to control the number of map tasks --- Key: MAPREDUCE-5287 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5287 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, performance Reporter: nicu marasoiu Assignee: nicu marasoiu Priority: Minor Attachments: AggregatingInputFormat.java, CompositeInputSplit.java, CompositeRecordReader.java Original Estimate: 4h Remaining Estimate: 4h I wrote a generic InputFormat that wraps any other InputFormat, and creates CompositeInputSplits to reduce the number of map tasks in a controllable manner while preserving data locality. A correspondent CompositeRecordReader is written to iterate through underlying RecordReaders as created by the underlying InputFormat for each underlying raw split. An application to this is to group TableSplits when the raw splits are coming from multiple regions and are filtered with key ranges. We use this to shard/distribute a time based incremental access to an hbase table. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5287) Create a generic InputFormat wrapping any other InputFormat, to control the number of map tasks
[ https://issues.apache.org/jira/browse/MAPREDUCE-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nicu marasoiu updated MAPREDUCE-5287: - Attachment: CompositeRecordReader.java CompositeInputSplit.java AggregatingInputFormat.java implementation of the split-merging inputformat which is optimal in both distribution and data locality only when single locations are used as in hbase. for multiple locations another algorithm I can implement quite easy if interest exists Create a generic InputFormat wrapping any other InputFormat, to control the number of map tasks --- Key: MAPREDUCE-5287 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5287 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv1, performance Reporter: nicu marasoiu Assignee: nicu marasoiu Priority: Minor Attachments: AggregatingInputFormat.java, CompositeInputSplit.java, CompositeRecordReader.java Original Estimate: 4h Remaining Estimate: 4h I wrote a generic InputFormat that wraps any other InputFormat, and creates CompositeInputSplits to reduce the number of map tasks in a controllable manner while preserving data locality. A correspondent CompositeRecordReader is written to iterate through underlying RecordReaders as created by the underlying InputFormat for each underlying raw split. An application to this is to group TableSplits when the raw splits are coming from multiple regions and are filtered with key ranges. We use this to shard/distribute a time based incremental access to an hbase table. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5304) mapreduce.Job killTask/failTask/getTaskCompletionEvents methods have incompatible signature changes
[
https://issues.apache.org/jira/browse/MAPREDUCE-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13682217#comment-13682217
]
Hudson commented on MAPREDUCE-5304:
---
Integrated in Hadoop-Hdfs-trunk #1429 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1429/])
MAPREDUCE-5304. mapreduce.Job killTask/failTask/getTaskCompletionEvents
methods have incompatible signature changes. (kkambatl via tucu) (Revision
1492360)
Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1492360
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskCompletionEvent.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java
mapreduce.Job killTask/failTask/getTaskCompletionEvents methods have
incompatible signature changes
---
Key: MAPREDUCE-5304
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5304
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Affects Versions: 2.0.4-alpha
Reporter: Alejandro Abdelnur
Assignee: Karthik Kambatla
Priority: Blocker
Fix For: 2.1.0-beta
Attachments: mr-5304-1.patch
Pointed out by [~zjshen] in MAPREDUCE-4942.
In {{o.a.h.mapreduce.Job}} class, the following changed from Hadoop 1 to
Hadoop 2.
boolean failTask(TaskAttemptID): Change in return type from void to boolean.
boolean killTask(TaskAttemptID): Change in return type from void to boolean.
TaskCompletionEvent[] getTaskCompletionEvents(int): Change in return type
from org.apache.hadoop.mapred.TaskCompletionEvent[] to
org.apache.hadoop.mapreduce.TaskCompletionEvent[].
Using same rational as in other JIRAs, we should fix this to ensure Hadoop 1
to Hadoop 2 source compatibility (taking 0.23.x releases as a casualty as
there is not right way for everybody because we screwed up :( ). Flagging it
as incompatible change because of 0.23.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5259) TestTaskLog fails on Windows because of path separators missmatch
[
https://issues.apache.org/jira/browse/MAPREDUCE-5259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13682221#comment-13682221
]
Hudson commented on MAPREDUCE-5259:
---
Integrated in Hadoop-Hdfs-trunk #1429 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1429/])
MAPREDUCE-5259. TestTaskLog fails on Windows because of path separators
missmatch. Contributed by Ivan Mitic. (Revision 1492364)
Result = FAILURE
cnauroth :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1492364
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestTaskLog.java
TestTaskLog fails on Windows because of path separators missmatch
-
Key: MAPREDUCE-5259
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5259
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Ivan Mitic
Assignee: Ivan Mitic
Fix For: 3.0.0, 2.1.0-beta
Attachments: MAPREDUCE-5259.patch
Test failure:
{noformat}
Running org.apache.hadoop.mapred.TestTaskLog
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.516 sec
FAILURE!
testTaskLog(org.apache.hadoop.mapred.TestTaskLog) Time elapsed: 409 sec
FAILURE!
junit.framework.AssertionFailedError: null
at junit.framework.Assert.fail(Assert.java:47)
at junit.framework.Assert.assertTrue(Assert.java:20)
at junit.framework.Assert.assertTrue(Assert.java:27)
at org.apache.hadoop.mapred.TestTaskLog.testTaskLog(TestTaskLog.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5304) mapreduce.Job killTask/failTask/getTaskCompletionEvents methods have incompatible signature changes
[
https://issues.apache.org/jira/browse/MAPREDUCE-5304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13682266#comment-13682266
]
Hudson commented on MAPREDUCE-5304:
---
Integrated in Hadoop-Mapreduce-trunk #1456 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1456/])
MAPREDUCE-5304. mapreduce.Job killTask/failTask/getTaskCompletionEvents
methods have incompatible signature changes. (kkambatl via tucu) (Revision
1492360)
Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1492360
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskCompletionEvent.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java
mapreduce.Job killTask/failTask/getTaskCompletionEvents methods have
incompatible signature changes
---
Key: MAPREDUCE-5304
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5304
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Affects Versions: 2.0.4-alpha
Reporter: Alejandro Abdelnur
Assignee: Karthik Kambatla
Priority: Blocker
Fix For: 2.1.0-beta
Attachments: mr-5304-1.patch
Pointed out by [~zjshen] in MAPREDUCE-4942.
In {{o.a.h.mapreduce.Job}} class, the following changed from Hadoop 1 to
Hadoop 2.
boolean failTask(TaskAttemptID): Change in return type from void to boolean.
boolean killTask(TaskAttemptID): Change in return type from void to boolean.
TaskCompletionEvent[] getTaskCompletionEvents(int): Change in return type
from org.apache.hadoop.mapred.TaskCompletionEvent[] to
org.apache.hadoop.mapreduce.TaskCompletionEvent[].
Using same rational as in other JIRAs, we should fix this to ensure Hadoop 1
to Hadoop 2 source compatibility (taking 0.23.x releases as a casualty as
there is not right way for everybody because we screwed up :( ). Flagging it
as incompatible change because of 0.23.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5259) TestTaskLog fails on Windows because of path separators missmatch
[
https://issues.apache.org/jira/browse/MAPREDUCE-5259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13682269#comment-13682269
]
Hudson commented on MAPREDUCE-5259:
---
Integrated in Hadoop-Mapreduce-trunk #1456 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1456/])
MAPREDUCE-5259. TestTaskLog fails on Windows because of path separators
missmatch. Contributed by Ivan Mitic. (Revision 1492364)
Result = SUCCESS
cnauroth :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1492364
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestTaskLog.java
TestTaskLog fails on Windows because of path separators missmatch
-
Key: MAPREDUCE-5259
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5259
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Ivan Mitic
Assignee: Ivan Mitic
Fix For: 3.0.0, 2.1.0-beta
Attachments: MAPREDUCE-5259.patch
Test failure:
{noformat}
Running org.apache.hadoop.mapred.TestTaskLog
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.516 sec
FAILURE!
testTaskLog(org.apache.hadoop.mapred.TestTaskLog) Time elapsed: 409 sec
FAILURE!
junit.framework.AssertionFailedError: null
at junit.framework.Assert.fail(Assert.java:47)
at junit.framework.Assert.assertTrue(Assert.java:20)
at junit.framework.Assert.assertTrue(Assert.java:27)
at org.apache.hadoop.mapred.TestTaskLog.testTaskLog(TestTaskLog.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5310) MRAM should not normalize allocation request capabilities
[ https://issues.apache.org/jira/browse/MAPREDUCE-5310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13682337#comment-13682337 ] Tom White commented on MAPREDUCE-5310: -- +1 As discussed in YARN-689, normalization of requests is a scheduler implementation artifact, so should be done there (as it already is) and not in the AM. MRAM should not normalize allocation request capabilities - Key: MAPREDUCE-5310 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5310 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: MAPREDUCE-5310.patch, MAPREDUCE-5310.patch The MRAM is assuming knowledge of the scheduler internals to normalize allocation request capabilities. Per discussions in YARN-689 and YARN-769 it should not do that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5298) Move MapReduce services to YARN-117 stricter lifecycle
[ https://issues.apache.org/jira/browse/MAPREDUCE-5298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved MAPREDUCE-5298. Resolution: Fixed Fix Version/s: 2.1.0-beta Hadoop Flags: Reviewed Committed this to trunk, branch-2 and branch-2.1 along with YARN-117/YARN-530. Closing this. Move MapReduce services to YARN-117 stricter lifecycle -- Key: MAPREDUCE-5298 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5298 Project: Hadoop Map/Reduce Issue Type: New Feature Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Steve Loughran Assignee: Steve Loughran Fix For: 2.1.0-beta Attachments: MAPREDUCE-5298-016.patch, MAPREDUCE-5298-018.patch, MAPREDUCE-5298-019.patch, MAPREDUCE-5298-020.patch, MAPREDUCE-5298-021.patch, MAPREDUCE-5298-022.patch, MAPREDUCE-5298-023.patch The MR services need to be in sync with the YARN-117 lifecycle enhancements -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5298) Move MapReduce services to YARN-117 stricter lifecycle
[ https://issues.apache.org/jira/browse/MAPREDUCE-5298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13682374#comment-13682374 ] Hudson commented on MAPREDUCE-5298: --- Integrated in Hadoop-trunk-Commit #3911 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3911/]) YARN-530. Defined Service model strictly, implemented AbstractService for robust subclassing and migrated yarn-common services. Contributed by Steve Loughran. YARN-117. Migrated rest of YARN to the new service model. Contributed by Steve Louhran. MAPREDUCE-5298. Moved MapReduce services to YARN-530 stricter lifecycle. Contributed by Steve Loughran. (Revision 1492718) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1492718 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/dev-support/findbugs-exclude.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/LocalContainerLauncher.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/TaskAttemptListenerImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryCopyService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/jobhistory/JobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/TaskHeartbeatHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/commit/CommitterEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/local/LocalContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMCommunicator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerRequestor.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/speculate/DefaultSpeculator.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/jobhistory/TestJobHistoryEventHandler.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRAppBenchmark.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestFail.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CachedHistoryStorage.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/HistoryClientService.java *
[jira] [Commented] (MAPREDUCE-5259) TestTaskLog fails on Windows because of path separators missmatch
[
https://issues.apache.org/jira/browse/MAPREDUCE-5259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13682389#comment-13682389
]
Ivan Mitic commented on MAPREDUCE-5259:
---
Thanks Chris for the review and commit!
TestTaskLog fails on Windows because of path separators missmatch
-
Key: MAPREDUCE-5259
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5259
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Ivan Mitic
Assignee: Ivan Mitic
Fix For: 3.0.0, 2.1.0-beta
Attachments: MAPREDUCE-5259.patch
Test failure:
{noformat}
Running org.apache.hadoop.mapred.TestTaskLog
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.516 sec
FAILURE!
testTaskLog(org.apache.hadoop.mapred.TestTaskLog) Time elapsed: 409 sec
FAILURE!
junit.framework.AssertionFailedError: null
at junit.framework.Assert.fail(Assert.java:47)
at junit.framework.Assert.assertTrue(Assert.java:20)
at junit.framework.Assert.assertTrue(Assert.java:27)
at org.apache.hadoop.mapred.TestTaskLog.testTaskLog(TestTaskLog.java:54)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
at
org.junit.internal.runners.statements.FailOnTimeout$1.run(FailOnTimeout.java:28)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5199) AppTokens file can/should be removed
[ https://issues.apache.org/jira/browse/MAPREDUCE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5199: --- Attachment: MAPREDUCE-5199-20130613.patch Same patch reattached. AppTokens file can/should be removed Key: MAPREDUCE-5199 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5199 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: security Affects Versions: 3.0.0, 2.1.0-beta Reporter: Vinod Kumar Vavilapalli Assignee: Daryn Sharp Priority: Blocker Attachments: MAPREDUCE-5199-20130613.patch, MAPREDUCE-5199.patch, MAPREDUCE-5199.patch, MAPREDUCE-5199.patch, MAPREDUCE-5199.patch, MAPREDUCE-5199.patch All the required tokens are propagated to AMs and containers via startContainer(), no need for explicitly creating the app-token file that we have today.. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5199) AppTokens file can/should be removed
[ https://issues.apache.org/jira/browse/MAPREDUCE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5199: --- Status: Open (was: Patch Available) This looks good. +1. Will reattach the same patch again for one final run through Jenkins. AppTokens file can/should be removed Key: MAPREDUCE-5199 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5199 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: security Affects Versions: 3.0.0, 2.1.0-beta Reporter: Vinod Kumar Vavilapalli Assignee: Daryn Sharp Priority: Blocker Attachments: MAPREDUCE-5199-20130613.patch, MAPREDUCE-5199.patch, MAPREDUCE-5199.patch, MAPREDUCE-5199.patch, MAPREDUCE-5199.patch, MAPREDUCE-5199.patch All the required tokens are propagated to AMs and containers via startContainer(), no need for explicitly creating the app-token file that we have today.. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5199) AppTokens file can/should be removed
[ https://issues.apache.org/jira/browse/MAPREDUCE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5199: --- Hadoop Flags: Incompatible change,Reviewed Status: Patch Available (was: Open) AppTokens file can/should be removed Key: MAPREDUCE-5199 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5199 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: security Affects Versions: 3.0.0, 2.1.0-beta Reporter: Vinod Kumar Vavilapalli Assignee: Daryn Sharp Priority: Blocker Attachments: MAPREDUCE-5199-20130613.patch, MAPREDUCE-5199.patch, MAPREDUCE-5199.patch, MAPREDUCE-5199.patch, MAPREDUCE-5199.patch, MAPREDUCE-5199.patch All the required tokens are propagated to AMs and containers via startContainer(), no need for explicitly creating the app-token file that we have today.. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5321) Enable better parallelism in the Fair Scheduler
Sandy Ryza created MAPREDUCE-5321: - Summary: Enable better parallelism in the Fair Scheduler Key: MAPREDUCE-5321 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5321 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Sandy Ryza Assignee: Sandy Ryza Currently, the Fair Scheduler is locked on pretty much every operation, node updates, application additions and removals, every time the update thread runs, and every time the RM queries it for information. Most of this locking is unnecessary, especially as only the core scheduling operations like application additions, removals, and node updates need a consistent view of scheduler state. We can probably increase parallelism by using concurrent data structures when applicable, as well as keeping a slightly stale view to serve via the RM APIs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Remove slot millis computation logic and deprecate counter constants
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13682594#comment-13682594 ] Jason Lowe commented on MAPREDUCE-5311: --- I like the idea of a memory-millis counter that uses the granted container size to remove the need to know how it was rounded to that value. However I'm not sure it's best to have slot-millis suddenly start returning something that isn't slot-millis. That could throw off calculations from downstream tools that assume a particular slot size (e.g.: min-allocation from today's setup). Probably better to simply not provide slot-millis when it cannot be calculated and supply memory-millis as an alternative. Remove slot millis computation logic and deprecate counter constants Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5251) Reducer should not implicate map attempt if it has insufficient space to fetch map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5251: -- Attachment: MAPREDUCE-5251-2.txt Reducer should not implicate map attempt if it has insufficient space to fetch map output - Key: MAPREDUCE-5251 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5251 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.7, 2.0.4-alpha Reporter: Jason Lowe Assignee: Ashwin Shankar Attachments: MAPREDUCE-5251-2.txt A job can fail if a reducer happens to run on a node with insufficient space to hold a map attempt's output. The reducer keeps reporting the map attempt as bad, and if the map attempt ends up being re-launched too many times before the reducer decides maybe it is the real problem the job can fail. In that scenario it would be better to re-launch the reduce attempt and hopefully it will run on another node that has sufficient space to complete the shuffle. Reporting the map attempt is bad and relaunching the map task doesn't change the fact that the reducer can't hold the output. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5199) AppTokens file can/should be removed
[
https://issues.apache.org/jira/browse/MAPREDUCE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13682610#comment-13682610
]
Hadoop QA commented on MAPREDUCE-5199:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12587677/MAPREDUCE-5199-20130613.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3766//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3766//console
This message is automatically generated.
AppTokens file can/should be removed
Key: MAPREDUCE-5199
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5199
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: security
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Vinod Kumar Vavilapalli
Assignee: Daryn Sharp
Priority: Blocker
Attachments: MAPREDUCE-5199-20130613.patch, MAPREDUCE-5199.patch,
MAPREDUCE-5199.patch, MAPREDUCE-5199.patch, MAPREDUCE-5199.patch,
MAPREDUCE-5199.patch
All the required tokens are propagated to AMs and containers via
startContainer(), no need for explicitly creating the app-token file that we
have today..
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5311) Remove slot millis computation logic and deprecate counter constants
[ https://issues.apache.org/jira/browse/MAPREDUCE-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13682629#comment-13682629 ] Alejandro Abdelnur commented on MAPREDUCE-5311: --- I'd prefer to remove it as well. Remove slot millis computation logic and deprecate counter constants Key: MAPREDUCE-5311 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5311 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: MAPREDUCE-5311.patch Per discussion in MAPREDUCE-5310 and comments in the code we should remove all the related logic and just leave the counter constant for backwards compatibility and deprecate the counter constants. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5251) Reducer should not implicate map attempt if it has insufficient space to fetch map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashwin Shankar updated MAPREDUCE-5251: -- Target Version/s: 0.23.8, 2.0.4-alpha Status: Patch Available (was: Open) The issue here is was that any IOException that arises from Fetcher.copyMapOutput is always blamed on map causing the map to be re-run. But we can get an exception like DiskErrorException which is due to lack of disk space on the reduce node and has nothing to do with map. The map task should not be re-executed in this case. To solve this,I've added code to catch this DiskErrorException,blame the reduce and report this to the shuffle scheduler ,which in turn kills this reduce task and would hopefully re-run it on another node which has disk space. I manually tested this by changing the source and stubbing it to reserve 1 Peta-byte during the shuffle phase. This throws the DiskErrorException as expected,blames the reduce and most importantly doesn't run the map task again. I've also added a unit test to simulate this scenario. Reducer should not implicate map attempt if it has insufficient space to fetch map output - Key: MAPREDUCE-5251 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5251 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 2.0.4-alpha, 0.23.7 Reporter: Jason Lowe Assignee: Ashwin Shankar Attachments: MAPREDUCE-5251-2.txt A job can fail if a reducer happens to run on a node with insufficient space to hold a map attempt's output. The reducer keeps reporting the map attempt as bad, and if the map attempt ends up being re-launched too many times before the reducer decides maybe it is the real problem the job can fail. In that scenario it would be better to re-launch the reduce attempt and hopefully it will run on another node that has sufficient space to complete the shuffle. Reporting the map attempt is bad and relaunching the map task doesn't change the fact that the reducer can't hold the output. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5251) Reducer should not implicate map attempt if it has insufficient space to fetch map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13682631#comment-13682631 ] Ashwin Shankar commented on MAPREDUCE-5251: --- Also please note that this patch doesn't directly apply to branch-23. I'll upload a separate patch once all the comments are resolved for this current patch. Reducer should not implicate map attempt if it has insufficient space to fetch map output - Key: MAPREDUCE-5251 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5251 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.7, 2.0.4-alpha Reporter: Jason Lowe Assignee: Ashwin Shankar Attachments: MAPREDUCE-5251-2.txt A job can fail if a reducer happens to run on a node with insufficient space to hold a map attempt's output. The reducer keeps reporting the map attempt as bad, and if the map attempt ends up being re-launched too many times before the reducer decides maybe it is the real problem the job can fail. In that scenario it would be better to re-launch the reduce attempt and hopefully it will run on another node that has sufficient space to complete the shuffle. Reporting the map attempt is bad and relaunching the map task doesn't change the fact that the reducer can't hold the output. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5199) AppTokens file can/should be removed
[ https://issues.apache.org/jira/browse/MAPREDUCE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-5199: --- Resolution: Fixed Fix Version/s: 2.1.0-beta Hadoop Flags: Reviewed (was: Incompatible change,Reviewed) Status: Resolved (was: Patch Available) Committed this to trunk, branch-2 and branch-2.1. Thanks Daryn! AppTokens file can/should be removed Key: MAPREDUCE-5199 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5199 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: security Affects Versions: 3.0.0, 2.1.0-beta Reporter: Vinod Kumar Vavilapalli Assignee: Daryn Sharp Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5199-20130613.patch, MAPREDUCE-5199.patch, MAPREDUCE-5199.patch, MAPREDUCE-5199.patch, MAPREDUCE-5199.patch, MAPREDUCE-5199.patch All the required tokens are propagated to AMs and containers via startContainer(), no need for explicitly creating the app-token file that we have today.. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-4069) Cleanup task tokens interface
[ https://issues.apache.org/jira/browse/MAPREDUCE-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli resolved MAPREDUCE-4069. Resolution: Duplicate Fixed by MAPREDUCE-5199. Cleanup task tokens interface - Key: MAPREDUCE-4069 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4069 Project: Hadoop Map/Reduce Issue Type: Improvement Components: mrv2 Affects Versions: 0.23.2 Reporter: Jason Lowe Priority: Minor This tracks a couple of cleanup issues that were identified in MAPREDUCE-4043: * It seems unnecessary to pass the job token and credentials separately when we always combine the job token into the credentials before building the container launch context. The TaskImpl and TaskAttemptImpl constructors could simply take credentials with the job token already added rather than separate job token and credential parameters. * It's unclear whether we still need the appTokens file that is placed into HDFS by the job submitter, localized by the NM, and finally read in by the AM. I believe the AM's credentials sent in the AM's container launch context already contains the same information. If that's the case, we should remove the code related to the appTokens file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5199) AppTokens file can/should be removed
[ https://issues.apache.org/jira/browse/MAPREDUCE-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13682681#comment-13682681 ] Hudson commented on MAPREDUCE-5199: --- Integrated in Hadoop-trunk-Commit #3914 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3914/]) MAPREDUCE-5199. Removing ApplicationTokens file as it is no longer needed. Contributed by Daryn Sharp. (Revision 1492848) Result = SUCCESS vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1492848 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/JobImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/MRApp.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestMRAppMaster.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRecovery.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestStagingCleanup.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestYARNRunner.java AppTokens file can/should be removed Key: MAPREDUCE-5199 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5199 Project: Hadoop Map/Reduce Issue Type: Sub-task Components: security Affects Versions: 3.0.0, 2.1.0-beta Reporter: Vinod Kumar Vavilapalli Assignee: Daryn Sharp Priority: Blocker Fix For: 2.1.0-beta Attachments: MAPREDUCE-5199-20130613.patch, MAPREDUCE-5199.patch, MAPREDUCE-5199.patch, MAPREDUCE-5199.patch, MAPREDUCE-5199.patch, MAPREDUCE-5199.patch All the required tokens are propagated to AMs and containers via startContainer(), no need for explicitly creating the app-token file that we have today.. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4019) -list-attempt-ids is not working
[
https://issues.apache.org/jira/browse/MAPREDUCE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13682706#comment-13682706
]
Jason Lowe commented on MAPREDUCE-4019:
---
+1, will commit shortly.
-list-attempt-ids is not working
-
Key: MAPREDUCE-4019
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4019
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: B Anil Kumar
Assignee: Ashwin Shankar
Priority: Minor
Attachments: MAPREDUCE-4019-4.txt, MAPREDUCE-4019-6.txt,
MAPREDUCE-4019.patch, MAPREDUCE-4019.patch, MAPREDUCE-4019.txt
while executing {noformat}bin/mapred job -list-attempt-ids job_id map
running{noformat}, we are getting IllegalArgumentexception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5251) Reducer should not implicate map attempt if it has insufficient space to fetch map output
[ https://issues.apache.org/jira/browse/MAPREDUCE-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-5251: -- Target Version/s: 2.1.0-beta, 0.23.9 (was: 2.0.4-alpha, 0.23.8) Reducer should not implicate map attempt if it has insufficient space to fetch map output - Key: MAPREDUCE-5251 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5251 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.7, 2.0.4-alpha Reporter: Jason Lowe Assignee: Ashwin Shankar Attachments: MAPREDUCE-5251-2.txt A job can fail if a reducer happens to run on a node with insufficient space to hold a map attempt's output. The reducer keeps reporting the map attempt as bad, and if the map attempt ends up being re-launched too many times before the reducer decides maybe it is the real problem the job can fail. In that scenario it would be better to re-launch the reduce attempt and hopefully it will run on another node that has sufficient space to complete the shuffle. Reporting the map attempt is bad and relaunching the map task doesn't change the fact that the reducer can't hold the output. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5251) Reducer should not implicate map attempt if it has insufficient space to fetch map output
[
https://issues.apache.org/jira/browse/MAPREDUCE-5251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13682727#comment-13682727
]
Hadoop QA commented on MAPREDUCE-5251:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12587690/MAPREDUCE-5251-2.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3767//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3767//console
This message is automatically generated.
Reducer should not implicate map attempt if it has insufficient space to
fetch map output
-
Key: MAPREDUCE-5251
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5251
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.7, 2.0.4-alpha
Reporter: Jason Lowe
Assignee: Ashwin Shankar
Attachments: MAPREDUCE-5251-2.txt
A job can fail if a reducer happens to run on a node with insufficient space
to hold a map attempt's output. The reducer keeps reporting the map attempt
as bad, and if the map attempt ends up being re-launched too many times
before the reducer decides maybe it is the real problem the job can fail.
In that scenario it would be better to re-launch the reduce attempt and
hopefully it will run on another node that has sufficient space to complete
the shuffle. Reporting the map attempt is bad and relaunching the map task
doesn't change the fact that the reducer can't hold the output.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4019) -list-attempt-ids is not working
[
https://issues.apache.org/jira/browse/MAPREDUCE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashwin Shankar updated MAPREDUCE-4019:
--
Attachment: MAPREDUCE-4019-b23-2.txt
MAPREDUCE-4019-b23-2.txt is the patch for branch-23. Please note that Jenkins
will complain since this won't apply to trunk.
-list-attempt-ids is not working
-
Key: MAPREDUCE-4019
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4019
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: B Anil Kumar
Assignee: Ashwin Shankar
Priority: Minor
Attachments: MAPREDUCE-4019-4.txt, MAPREDUCE-4019-6.txt,
MAPREDUCE-4019-b23-2.txt, MAPREDUCE-4019.patch, MAPREDUCE-4019.patch,
MAPREDUCE-4019.txt
while executing {noformat}bin/mapred job -list-attempt-ids job_id map
running{noformat}, we are getting IllegalArgumentexception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4019) -list-attempt-ids is not working
[
https://issues.apache.org/jira/browse/MAPREDUCE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13682746#comment-13682746
]
Hadoop QA commented on MAPREDUCE-4019:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12587706/MAPREDUCE-4019-b23-2.txt
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3768//console
This message is automatically generated.
-list-attempt-ids is not working
-
Key: MAPREDUCE-4019
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4019
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: B Anil Kumar
Assignee: Ashwin Shankar
Priority: Minor
Attachments: MAPREDUCE-4019-4.txt, MAPREDUCE-4019-6.txt,
MAPREDUCE-4019-b23-2.txt, MAPREDUCE-4019.patch, MAPREDUCE-4019.patch,
MAPREDUCE-4019.txt
while executing {noformat}bin/mapred job -list-attempt-ids job_id map
running{noformat}, we are getting IllegalArgumentexception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4019) -list-attempt-ids is not working
[
https://issues.apache.org/jira/browse/MAPREDUCE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13682757#comment-13682757
]
Hudson commented on MAPREDUCE-4019:
---
Integrated in Hadoop-trunk-Commit #3915 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/3915/])
MAPREDUCE-4019. -list-attempt-ids is not working. Contributed by Ashwin
Shankar, Devaraj K, and B Anil Kumar (Revision 1492868)
Result = SUCCESS
jlowe : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1492868
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/tools/CLI.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/tools
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/tools/TestCLI.java
-list-attempt-ids is not working
-
Key: MAPREDUCE-4019
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4019
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: B Anil Kumar
Assignee: Ashwin Shankar
Priority: Minor
Attachments: MAPREDUCE-4019-4.txt, MAPREDUCE-4019-6.txt,
MAPREDUCE-4019-b23-2.txt, MAPREDUCE-4019.patch, MAPREDUCE-4019.patch,
MAPREDUCE-4019.txt
while executing {noformat}bin/mapred job -list-attempt-ids job_id map
running{noformat}, we are getting IllegalArgumentexception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4019) -list-attempt-ids is not working
[
https://issues.apache.org/jira/browse/MAPREDUCE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13682779#comment-13682779
]
Jason Lowe commented on MAPREDUCE-4019:
---
+1 for the 0.23 patch which will also apply to branch-2. There are a couple of
the whitespace changes that didn't make it into the 0.23 patch, but they're
small nits that I can cleanup during the commmit.
-list-attempt-ids is not working
-
Key: MAPREDUCE-4019
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4019
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: B Anil Kumar
Assignee: Ashwin Shankar
Priority: Minor
Attachments: MAPREDUCE-4019-4.txt, MAPREDUCE-4019-6.txt,
MAPREDUCE-4019-b23-2.txt, MAPREDUCE-4019.patch, MAPREDUCE-4019.patch,
MAPREDUCE-4019.txt
while executing {noformat}bin/mapred job -list-attempt-ids job_id map
running{noformat}, we are getting IllegalArgumentexception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4019) -list-attempt-ids is not working
[
https://issues.apache.org/jira/browse/MAPREDUCE-4019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Lowe updated MAPREDUCE-4019:
--
Resolution: Fixed
Fix Version/s: 0.23.9
2.1.0-beta
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks, Ashwin, Devaraj, and B Anil! I committed this to trunk, branch-2,
branch-2.1-beta, and branch-0.23.
-list-attempt-ids is not working
-
Key: MAPREDUCE-4019
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4019
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: B Anil Kumar
Assignee: Ashwin Shankar
Priority: Minor
Fix For: 2.1.0-beta, 0.23.9
Attachments: MAPREDUCE-4019-4.txt, MAPREDUCE-4019-6.txt,
MAPREDUCE-4019-b23-2.txt, MAPREDUCE-4019.patch, MAPREDUCE-4019.patch,
MAPREDUCE-4019.txt
while executing {noformat}bin/mapred job -list-attempt-ids job_id map
running{noformat}, we are getting IllegalArgumentexception.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5310) MRAM should not normalize allocation request capabilities
[ https://issues.apache.org/jira/browse/MAPREDUCE-5310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13682911#comment-13682911 ] Alejandro Abdelnur commented on MAPREDUCE-5310: --- planning to commit this by FRI noon PST. MRAM should not normalize allocation request capabilities - Key: MAPREDUCE-5310 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5310 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.0.4-alpha Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: MAPREDUCE-5310.patch, MAPREDUCE-5310.patch The MRAM is assuming knowledge of the scheduler internals to normalize allocation request capabilities. Per discussions in YARN-689 and YARN-769 it should not do that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5322) If input path does not exist application/job id is getting assigned.
Nishan Shetty created MAPREDUCE-5322: Summary: If input path does not exist application/job id is getting assigned. Key: MAPREDUCE-5322 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5322 Project: Hadoop Map/Reduce Issue Type: Bug Components: resourcemanager Affects Versions: 2.0.2-alpha, 2.0.1-alpha Reporter: Nishan Shetty Priority: Minor 1.Run job by giving input as some path which does not exist 2.Application/job is is getting assigned. 2013-06-12 16:00:24,494 INFO org.apache.hadoop.yarn.server.resourcemanager.ClientRMService: Allocated new applicationId: 12 Suggestion Before assiging job/app id input path check can be made. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5319) Job.xml file does not has 'user.name' property for Hadoop2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuan Gong updated MAPREDUCE-5319:
-
Attachment: MR-5319.patch
The job.xml is generated in JobSubmitter:: submitJobInternal(), but the
user.name property did not set there. So, it just copy it from mapred-site.xml
that is why we saw ${user.name} in job.xml.
Job.xml file does not has 'user.name' property for Hadoop2
--
Key: MAPREDUCE-5319
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5319
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.1.0-beta
Reporter: yeshavora
Assignee: Xuan Gong
Fix For: 2.1.0-beta
Attachments: MR-5319.patch
Run a sleep job and look for job.xml file generated by sleep job.
It does not contain user.name property.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-5319) Job.xml file does not has 'user.name' property for Hadoop2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuan Gong updated MAPREDUCE-5319: - Status: Patch Available (was: Open) Job.xml file does not has 'user.name' property for Hadoop2 -- Key: MAPREDUCE-5319 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5319 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: yeshavora Assignee: Xuan Gong Fix For: 2.1.0-beta Attachments: MR-5319.patch Run a sleep job and look for job.xml file generated by sleep job. It does not contain user.name property. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5319) Job.xml file does not has 'user.name' property for Hadoop2
[
https://issues.apache.org/jira/browse/MAPREDUCE-5319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13683106#comment-13683106
]
Hadoop QA commented on MAPREDUCE-5319:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12587772/MR-5319.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3769//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/3769//console
This message is automatically generated.
Job.xml file does not has 'user.name' property for Hadoop2
--
Key: MAPREDUCE-5319
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5319
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.1.0-beta
Reporter: yeshavora
Assignee: Xuan Gong
Fix For: 2.1.0-beta
Attachments: MR-5319.patch
Run a sleep job and look for job.xml file generated by sleep job.
It does not contain user.name property.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-5319) Job.xml file does not has 'user.name' property for Hadoop2
[ https://issues.apache.org/jira/browse/MAPREDUCE-5319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13683126#comment-13683126 ] Xuan Gong commented on MAPREDUCE-5319: -- simple patch Job.xml file does not has 'user.name' property for Hadoop2 -- Key: MAPREDUCE-5319 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5319 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: yeshavora Assignee: Xuan Gong Fix For: 2.1.0-beta Attachments: MR-5319.patch Run a sleep job and look for job.xml file generated by sleep job. It does not contain user.name property. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
