[jira] [Updated] (MAPREDUCE-5621) mr-jobhistory-daemon.sh doesn't have to execute mkdir and chown all the time
[ https://issues.apache.org/jira/browse/MAPREDUCE-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shinichi Yamashita updated MAPREDUCE-5621: -- Attachment: MAPREDUCE-5621.patch I attach a patch file. > mr-jobhistory-daemon.sh doesn't have to execute mkdir and chown all the time > > > Key: MAPREDUCE-5621 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5621 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 3.0.0 >Reporter: Shinichi Yamashita >Assignee: Shinichi Yamashita >Priority: Minor > Attachments: MAPREDUCE-5621.patch > > > mr-jobhistory-daemon.sh executes mkdir and chown command to output the log > files. > This is always executed with or without a directory. In addition, this is > executed not only starting daemon but also stopping daemon. > It add "if" like hadoop-daemon.sh and yarn-daemon.sh and should control it. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5621) mr-jobhistory-daemon.sh doesn't have to execute mkdir and chown all the time
[ https://issues.apache.org/jira/browse/MAPREDUCE-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shinichi Yamashita updated MAPREDUCE-5621: -- Status: Patch Available (was: Open) > mr-jobhistory-daemon.sh doesn't have to execute mkdir and chown all the time > > > Key: MAPREDUCE-5621 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5621 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver >Affects Versions: 3.0.0 >Reporter: Shinichi Yamashita >Assignee: Shinichi Yamashita >Priority: Minor > Attachments: MAPREDUCE-5621.patch > > > mr-jobhistory-daemon.sh executes mkdir and chown command to output the log > files. > This is always executed with or without a directory. In addition, this is > executed not only starting daemon but also stopping daemon. > It add "if" like hadoop-daemon.sh and yarn-daemon.sh and should control it. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5610) TestSleepJob fails in jdk7
[
https://issues.apache.org/jira/browse/MAPREDUCE-5610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13821001#comment-13821001
]
Hadoop QA commented on MAPREDUCE-5610:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12613516/MAPREDUCE-5610.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-gridmix.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4194//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4194//console
This message is automatically generated.
> TestSleepJob fails in jdk7
> --
>
> Key: MAPREDUCE-5610
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5610
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: MAPREDUCE-5610.patch, MAPREDUCE-5610.patch,
> MAPREDUCE-5610.patch, MAPREDUCE-5610.patch
>
>
> In jdk7 tests methods in a class do not run in file order, but rather in
> random order. TestSleepJob hosts are not initialized and a
> NullPointerException is thrown unless testRandomLocation was run first.
> This can be easily seen by running tests individually.
> org.apache.hadoop.mapred.gridmix.TestSleepJob#testStressSubmit
> org.apache.hadoop.mapred.gridmix.TestSleepJob#testReplaySubmit
> org.apache.hadoop.mapred.gridmix.TestSleepJob#testSerialSubmit
> org.apache.hadoop.mapred.gridmix.TestSleepJob#testMapTasksOnlySleepJobs
> org.apache.hadoop.mapred.gridmix.TestSleepJob#testRandomLocation
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-5622) MRAppMaster doesn't assign all allocated NODE_LOCAL containers to node-local maps
[
https://issues.apache.org/jira/browse/MAPREDUCE-5622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13820992#comment-13820992
]
Devaraj K commented on MAPREDUCE-5622:
--
RM Log:
{code:xml}
yarn-hdpcrm-resourcemanager-ocean24.log.1:2013-11-11 13:58:36,738 DEBUG
org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp:
allocate: applicationAttemptId=appattempt_1383828930654_0030_01
container=container_1383828930654_0030_01_73 host=ocean21 type=NODE_LOCAL
yarn-hdpcrm-resourcemanager-ocean24.log.1:2013-11-11 13:58:36,743 DEBUG
org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp:
allocate: applicationAttemptId=appattempt_1383828930654_0030_01
container=container_1383828930654_0030_01_74 host=ocean21 type=NODE_LOCAL
{code}
\\
NM Log:
{code:xml}
2013-11-11 13:57:38,904 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container
(Container: [ContainerId: container_1383828930654_0030_01_73, NodeId:
ocean21:23240, NodeHttpAddress: ocean21:23239, Resource: , Priority: 20, State: NEW, Token: null, Status: container_id {,
app_attempt_id {, application_id {, id: 30, cluster_timestamp: 1383828930654,
}, attemptId: 1, }, id: 73, }, state: C_NEW, ]) to task
attempt_1383828930654_0030_m_61_0 on node ocean21:23240
2013-11-11 13:57:38,904 DEBUG [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned based on
rack match /default-rack
2013-11-11 13:57:38,905 INFO [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container
(Container: [ContainerId: container_1383828930654_0030_01_74, NodeId:
ocean21:23240, NodeHttpAddress: ocean21:23239, Resource: , Priority: 20, State: NEW, Token: null, Status: container_id {,
app_attempt_id {, application_id {, id: 30, cluster_timestamp: 1383828930654,
}, attemptId: 1, }, id: 74, }, state: C_NEW, ]) to task
attempt_1383828930654_0030_m_63_0 on node ocean21:23240
2013-11-11 13:57:38,905 DEBUG [RMCommunicator Allocator]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned based on
rack match /default-rack
{code}
\\
Here RM says allocating the container as NODE_LOCAL but AM is assigning based
on rack match.
> MRAppMaster doesn't assign all allocated NODE_LOCAL containers to node-local
> maps
> -
>
> Key: MAPREDUCE-5622
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5622
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Affects Versions: 2.2.0
>Reporter: Devaraj K
>
> MRAppMaster will request containers for all the splits to launch map tasks,
> RM will give Node Local containers for all these if available. When the RM
> gives all containers as Node Local, MR AM may assign these NODE-LOCAL
> containers to non-local maps.
> \\
> \\
> |node1|split1|split2| |split4|
> |node2| |split2|split3| |
> |node3|split1|split2|split3|split4|
> |node4|split1| |split3|split4|
> \\
> Consider this instance, assume RM has given one NODE LOCAL container on each
> node to process all the splits as local maps. While assigning, if the AM
> gives node1-container for split1, node2-container for split3, node3-container
> for split3 and node4-container can be given to only split2 which is not
> local.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (MAPREDUCE-5622) MRAppMaster doesn't assign all allocated NODE_LOCAL containers to node-local maps
Devaraj K created MAPREDUCE-5622: Summary: MRAppMaster doesn't assign all allocated NODE_LOCAL containers to node-local maps Key: MAPREDUCE-5622 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5622 Project: Hadoop Map/Reduce Issue Type: Bug Components: applicationmaster Affects Versions: 2.2.0 Reporter: Devaraj K MRAppMaster will request containers for all the splits to launch map tasks, RM will give Node Local containers for all these if available. When the RM gives all containers as Node Local, MR AM may assign these NODE-LOCAL containers to non-local maps. \\ \\ |node1|split1|split2| |split4| |node2| |split2|split3| | |node3|split1|split2|split3|split4| |node4|split1| |split3|split4| \\ Consider this instance, assume RM has given one NODE LOCAL container on each node to process all the splits as local maps. While assigning, if the AM gives node1-container for split1, node2-container for split3, node3-container for split3 and node4-container can be given to only split2 which is not local. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5593) Cleanup code for AssignMapsWithLocality() in RMContainerAllocator
[ https://issues.apache.org/jira/browse/MAPREDUCE-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13820906#comment-13820906 ] Junping Du commented on MAPREDUCE-5593: --- The test failure seems to be unrelated with the patch (also failed on clean trunk branch in my local env) and may caused by MAPREDUCE-5427. Will do more investigation soon. > Cleanup code for AssignMapsWithLocality() in RMContainerAllocator > - > > Key: MAPREDUCE-5593 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5593 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster >Reporter: Junping Du >Assignee: Junping Du >Priority: Minor > Attachments: MAPREDUCE-5593-v3.patch, MAPREDUCE-5593-v4.patch, > YARN-1163-v1.patch, YARN-1163-v2.patch > > > In RMContainerAllocator, AssignMapsWithLocality() is a very important method > to assign map tasks on allocated containers with conforming different level > of locality (dataLocal, rackLocal, etc.). However, this method messed with > different code logic to handle different type of locality but have lots of > similar behaviours. This is hard to maintain as well as do extension with > other locality type, so we need some more clear code here. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5610) TestSleepJob fails in jdk7
[ https://issues.apache.org/jira/browse/MAPREDUCE-5610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5610: --- Attachment: MAPREDUCE-5610.patch Thanks for the feedback, Jason. Incorporated the changes and have uploaded a new patch. > TestSleepJob fails in jdk7 > -- > > Key: MAPREDUCE-5610 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5610 > Project: Hadoop Map/Reduce > Issue Type: Test >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-5610.patch, MAPREDUCE-5610.patch, > MAPREDUCE-5610.patch, MAPREDUCE-5610.patch > > > In jdk7 tests methods in a class do not run in file order, but rather in > random order. TestSleepJob hosts are not initialized and a > NullPointerException is thrown unless testRandomLocation was run first. > This can be easily seen by running tests individually. > org.apache.hadoop.mapred.gridmix.TestSleepJob#testStressSubmit > org.apache.hadoop.mapred.gridmix.TestSleepJob#testReplaySubmit > org.apache.hadoop.mapred.gridmix.TestSleepJob#testSerialSubmit > org.apache.hadoop.mapred.gridmix.TestSleepJob#testMapTasksOnlySleepJobs > org.apache.hadoop.mapred.gridmix.TestSleepJob#testRandomLocation -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5593) Cleanup code for AssignMapsWithLocality() in RMContainerAllocator
[ https://issues.apache.org/jira/browse/MAPREDUCE-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-5593: -- Status: Open (was: Patch Available) > Cleanup code for AssignMapsWithLocality() in RMContainerAllocator > - > > Key: MAPREDUCE-5593 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5593 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster >Reporter: Junping Du >Assignee: Junping Du >Priority: Minor > Attachments: MAPREDUCE-5593-v3.patch, MAPREDUCE-5593-v4.patch, > YARN-1163-v1.patch, YARN-1163-v2.patch > > > In RMContainerAllocator, AssignMapsWithLocality() is a very important method > to assign map tasks on allocated containers with conforming different level > of locality (dataLocal, rackLocal, etc.). However, this method messed with > different code logic to handle different type of locality but have lots of > similar behaviours. This is hard to maintain as well as do extension with > other locality type, so we need some more clear code here. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5613) DefaultSpeculator holds and checks hashmap that is always empty
[
https://issues.apache.org/jira/browse/MAPREDUCE-5613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sandy Ryza updated MAPREDUCE-5613:
--
Summary: DefaultSpeculator holds and checks hashmap that is always empty
(was: DefaultSpeculator.statusUpdate wastes CPU for task attempt id hashing and
lookups in an unused empty CHM)
> DefaultSpeculator holds and checks hashmap that is always empty
> ---
>
> Key: MAPREDUCE-5613
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5613
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Reporter: Gera Shegalov
> Attachments: MAPREDUCE-5613.v01.patch
>
>
> The only way pendingSpeculations is used:
> {code}
> // If the task is already known to be speculation-bait, don't do
> anything
> if (pendingSpeculations.get(task) != null) {
>
> if (pendingSpeculations.get(task).get()) {
>
> return;
>
> }
>
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-5613) DefaultSpeculator.statusUpdate wastes CPU for task attempt id hashing and lookups in an unused empty CHM
[
https://issues.apache.org/jira/browse/MAPREDUCE-5613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13820844#comment-13820844
]
Sandy Ryza commented on MAPREDUCE-5613:
---
+1
> DefaultSpeculator.statusUpdate wastes CPU for task attempt id hashing and
> lookups in an unused empty CHM
>
>
> Key: MAPREDUCE-5613
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5613
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: applicationmaster
>Reporter: Gera Shegalov
> Attachments: MAPREDUCE-5613.v01.patch
>
>
> The only way pendingSpeculations is used:
> {code}
> // If the task is already known to be speculation-bait, don't do
> anything
> if (pendingSpeculations.get(task) != null) {
>
> if (pendingSpeculations.get(task).get()) {
>
> return;
>
> }
>
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (MAPREDUCE-5621) mr-jobhistory-daemon.sh doesn't have to execute mkdir and chown all the time
Shinichi Yamashita created MAPREDUCE-5621: - Summary: mr-jobhistory-daemon.sh doesn't have to execute mkdir and chown all the time Key: MAPREDUCE-5621 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5621 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 3.0.0 Reporter: Shinichi Yamashita Assignee: Shinichi Yamashita Priority: Minor mr-jobhistory-daemon.sh executes mkdir and chown command to output the log files. This is always executed with or without a directory. In addition, this is executed not only starting daemon but also stopping daemon. It add "if" like hadoop-daemon.sh and yarn-daemon.sh and should control it. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (MAPREDUCE-5620) distcp1 -delete fails when target directory contains files with percent signs
[ https://issues.apache.org/jira/browse/MAPREDUCE-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang resolved MAPREDUCE-5620. Resolution: Invalid Turns out this was due to running distcp1 with hadoop 2's FsShell. I couldn't repro this on a pure branch-1 setup, so resolving as invalid. > distcp1 -delete fails when target directory contains files with percent signs > - > > Key: MAPREDUCE-5620 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5620 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 1.2.1 >Reporter: Andrew Wang >Assignee: Andrew Wang > > Debugging a distcp1 issue, it fails to delete extra files in the target > directory when there is a percent sign in the filename. I'm pretty sure this > is an issue with how percent encoding is handled in FsShell (reproduced with > just "hadoop fs -rmr"), but we can also fix this in distcp1 by using > FileSystem instead of FsShell. This is what distcp2 does. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5610) TestSleepJob fails in jdk7
[
https://issues.apache.org/jira/browse/MAPREDUCE-5610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13820667#comment-13820667
]
Jason Lowe commented on MAPREDUCE-5610:
---
Thanks, Jon. One nit is that the conditions are now more complex than
necessary, as this code:
{code}
if (numLocations < 0) numLocations=0;
if ((numLocations >= 0) && (hosts == null)) {
{code}
should just be simplified to:
{code}
if (numLocations < 0) numLocations=0;
if (hosts == null) {
{code}
It's more clear that we want to initialize hosts whenever it's null and not
dependent upon some other condition as well.
> TestSleepJob fails in jdk7
> --
>
> Key: MAPREDUCE-5610
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5610
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: MAPREDUCE-5610.patch, MAPREDUCE-5610.patch,
> MAPREDUCE-5610.patch
>
>
> In jdk7 tests methods in a class do not run in file order, but rather in
> random order. TestSleepJob hosts are not initialized and a
> NullPointerException is thrown unless testRandomLocation was run first.
> This can be easily seen by running tests individually.
> org.apache.hadoop.mapred.gridmix.TestSleepJob#testStressSubmit
> org.apache.hadoop.mapred.gridmix.TestSleepJob#testReplaySubmit
> org.apache.hadoop.mapred.gridmix.TestSleepJob#testSerialSubmit
> org.apache.hadoop.mapred.gridmix.TestSleepJob#testMapTasksOnlySleepJobs
> org.apache.hadoop.mapred.gridmix.TestSleepJob#testRandomLocation
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-5618) Allow setting lineage information
[ https://issues.apache.org/jira/browse/MAPREDUCE-5618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13820605#comment-13820605 ] Karthik Kambatla commented on MAPREDUCE-5618: - Re-purposing this JIRA to track MR-side changes to use the applicationLineage in YARN. > Allow setting lineage information > - > > Key: MAPREDUCE-5618 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5618 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am >Affects Versions: 2.2.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > > MR AM sets the applicationType to be "MAPREDUCE". Downstream projects like > Pig, Hive, Oozie might want to set this to a different value for their > error-handling, query-tracking etc. Making this pluggable should help this > cause. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5618) Allow setting lineage information
[ https://issues.apache.org/jira/browse/MAPREDUCE-5618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Kambatla updated MAPREDUCE-5618: Summary: Allow setting lineage information (was: Make applicationType pluggable) > Allow setting lineage information > - > > Key: MAPREDUCE-5618 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5618 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am >Affects Versions: 2.2.0 >Reporter: Karthik Kambatla >Assignee: Karthik Kambatla > > MR AM sets the applicationType to be "MAPREDUCE". Downstream projects like > Pig, Hive, Oozie might want to set this to a different value for their > error-handling, query-tracking etc. Making this pluggable should help this > cause. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5600) ConcurrentModificationException on /tasktracker.jsp
[
https://issues.apache.org/jira/browse/MAPREDUCE-5600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13820550#comment-13820550
]
Kousuke Saruta commented on MAPREDUCE-5600:
---
I think branch-1 has the same issue.
When TaskTracker#getTasksFromRunningJobs is called, we'll get a iterator of
"runningJobs"
{code}
/**
* Get the list of tasks from running jobs on this task tracker.
* @return a copy of the list of TaskStatus objects
*/
synchronized List getTasksFromRunningJobs() {
List result = new ArrayList(tasks.size());
for (Map.Entry item : runningJobs.entrySet()) {
...
}
}
return result;
}
{code}
On the other hand, TaskTracker#addTaskToJob can be called during iteration.
addTaskToJob method modifies "runningJob" so it can cause
ConcurrentModificationException.
{code}
private RunningJob addTaskToJob(JobID jobId,
TaskInProgress tip) {
synchronized (runningJobs) {
...
runningJobs.put(jobId, rJob);
...
}
}
{code}
When we call getTasksFromRunningJobs, we get a synchronized monitor for the
instance of TaskTracker, but, when we call addTaskToJob,
we get a synchronized monitor for "runningJob". So, we may need to modify
TaskTracker to get an appropriate monitor when calling getTasksFromRunningJobs.
> ConcurrentModificationException on /tasktracker.jsp
> ---
>
> Key: MAPREDUCE-5600
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5600
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: tasktracker
> Environment: Found this in the MR1 bundled with
> hadoop-2.0.0-mr1-cdh4.1.3 (which I think is based on some 0.20 version).
>Reporter: Benoit Sigoure
>
> If you request {{/tasktracker.jsp}} frequently on a TaskTracker that's busy,
> every once in a while you'll get this:
> {code}
> 2013-10-29 13:25:55,524 ERROR org.mortbay.log: /tasktracker.jsp
> java.util.ConcurrentModificationException
> at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1117)
> at java.util.TreeMap$EntryIterator.next(TreeMap.java:1153)
> at java.util.TreeMap$EntryIterator.next(TreeMap.java:1148)
> at
> org.apache.hadoop.mapred.TaskTracker.getTasksFromRunningJobs(TaskTracker.java:3991)
> at
> org.apache.hadoop.mapred.tasktracker_jsp._jspService(tasktracker_jsp.java:98)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:98)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
> at
> org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:1056)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
> at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (MAPREDUCE-5620) distcp1 -delete fails when target directory contains files with percent signs
Andrew Wang created MAPREDUCE-5620: -- Summary: distcp1 -delete fails when target directory contains files with percent signs Key: MAPREDUCE-5620 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5620 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.2.1 Reporter: Andrew Wang Assignee: Andrew Wang Debugging a distcp1 issue, it fails to delete extra files in the target directory when there is a percent sign in the filename. I'm pretty sure this is an issue with how percent encoding is handled in FsShell (reproduced with just "hadoop fs -rmr"), but we can also fix this in distcp1 by using FileSystem instead of FsShell. This is what distcp2 does. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5610) TestSleepJob fails in jdk7
[
https://issues.apache.org/jira/browse/MAPREDUCE-5610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13820502#comment-13820502
]
Hadoop QA commented on MAPREDUCE-5610:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12613413/MAPREDUCE-5610.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-tools/hadoop-gridmix.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4193//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4193//console
This message is automatically generated.
> TestSleepJob fails in jdk7
> --
>
> Key: MAPREDUCE-5610
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5610
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: MAPREDUCE-5610.patch, MAPREDUCE-5610.patch,
> MAPREDUCE-5610.patch
>
>
> In jdk7 tests methods in a class do not run in file order, but rather in
> random order. TestSleepJob hosts are not initialized and a
> NullPointerException is thrown unless testRandomLocation was run first.
> This can be easily seen by running tests individually.
> org.apache.hadoop.mapred.gridmix.TestSleepJob#testStressSubmit
> org.apache.hadoop.mapred.gridmix.TestSleepJob#testReplaySubmit
> org.apache.hadoop.mapred.gridmix.TestSleepJob#testSerialSubmit
> org.apache.hadoop.mapred.gridmix.TestSleepJob#testMapTasksOnlySleepJobs
> org.apache.hadoop.mapred.gridmix.TestSleepJob#testRandomLocation
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-1176) FixedLengthInputFormat and FixedLengthRecordReader
[
https://issues.apache.org/jira/browse/MAPREDUCE-1176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13820470#comment-13820470
]
BitsOfInfo commented on MAPREDUCE-1176:
---
Glad to see this finally making it in. Thanks to those who picked it up and
helped push it through.
> FixedLengthInputFormat and FixedLengthRecordReader
> --
>
> Key: MAPREDUCE-1176
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1176
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
>Affects Versions: 2.2.0
> Environment: Any
>Reporter: BitsOfInfo
>Assignee: Mariappan Asokan
> Fix For: 2.3.0
>
> Attachments: MAPREDUCE-1176-v1.patch, MAPREDUCE-1176-v2.patch,
> MAPREDUCE-1176-v3.patch, MAPREDUCE-1176-v4.patch, mapreduce-1176_v1.patch,
> mapreduce-1176_v2.patch, mapreduce-1176_v3.patch
>
>
> Hello,
> I would like to contribute the following two classes for incorporation into
> the mapreduce.lib.input package. These two classes can be used when you need
> to read data from files containing fixed length (fixed width) records. Such
> files have no CR/LF (or any combination thereof), no delimiters etc, but each
> record is a fixed length, and extra data is padded with spaces. The data is
> one gigantic line within a file.
> Provided are two classes first is the FixedLengthInputFormat and its
> corresponding FixedLengthRecordReader. When creating a job that specifies
> this input format, the job must have the
> "mapreduce.input.fixedlengthinputformat.record.length" property set as follows
> myJobConf.setInt("mapreduce.input.fixedlengthinputformat.record.length",[myFixedRecordLength]);
> OR
> myJobConf.setInt(FixedLengthInputFormat.FIXED_RECORD_LENGTH,
> [myFixedRecordLength]);
> This input format overrides computeSplitSize() in order to ensure that
> InputSplits do not contain any partial records since with fixed records there
> is no way to determine where a record begins if that were to occur. Each
> InputSplit passed to the FixedLengthRecordReader will start at the beginning
> of a record, and the last byte in the InputSplit will be the last byte of a
> record. The override of computeSplitSize() delegates to FileInputFormat's
> compute method, and then adjusts the returned split size by doing the
> following: (Math.floor(fileInputFormatsComputedSplitSize / fixedRecordLength)
> * fixedRecordLength)
> This suite of fixed length input format classes, does not support compressed
> files.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-1176) FixedLengthInputFormat and FixedLengthRecordReader
[
https://issues.apache.org/jira/browse/MAPREDUCE-1176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13820381#comment-13820381
]
Mariappan Asokan commented on MAPREDUCE-1176:
-
Thanks Sandy!
-- Asokan
> FixedLengthInputFormat and FixedLengthRecordReader
> --
>
> Key: MAPREDUCE-1176
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1176
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
>Affects Versions: 2.2.0
> Environment: Any
>Reporter: BitsOfInfo
>Assignee: Mariappan Asokan
> Fix For: 2.3.0
>
> Attachments: MAPREDUCE-1176-v1.patch, MAPREDUCE-1176-v2.patch,
> MAPREDUCE-1176-v3.patch, MAPREDUCE-1176-v4.patch, mapreduce-1176_v1.patch,
> mapreduce-1176_v2.patch, mapreduce-1176_v3.patch
>
>
> Hello,
> I would like to contribute the following two classes for incorporation into
> the mapreduce.lib.input package. These two classes can be used when you need
> to read data from files containing fixed length (fixed width) records. Such
> files have no CR/LF (or any combination thereof), no delimiters etc, but each
> record is a fixed length, and extra data is padded with spaces. The data is
> one gigantic line within a file.
> Provided are two classes first is the FixedLengthInputFormat and its
> corresponding FixedLengthRecordReader. When creating a job that specifies
> this input format, the job must have the
> "mapreduce.input.fixedlengthinputformat.record.length" property set as follows
> myJobConf.setInt("mapreduce.input.fixedlengthinputformat.record.length",[myFixedRecordLength]);
> OR
> myJobConf.setInt(FixedLengthInputFormat.FIXED_RECORD_LENGTH,
> [myFixedRecordLength]);
> This input format overrides computeSplitSize() in order to ensure that
> InputSplits do not contain any partial records since with fixed records there
> is no way to determine where a record begins if that were to occur. Each
> InputSplit passed to the FixedLengthRecordReader will start at the beginning
> of a record, and the last byte in the InputSplit will be the last byte of a
> record. The override of computeSplitSize() delegates to FileInputFormat's
> compute method, and then adjusts the returned split size by doing the
> following: (Math.floor(fileInputFormatsComputedSplitSize / fixedRecordLength)
> * fixedRecordLength)
> This suite of fixed length input format classes, does not support compressed
> files.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-5610) TestSleepJob fails in jdk7
[ https://issues.apache.org/jira/browse/MAPREDUCE-5610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5610: --- Attachment: (was: MAPREDUCE-5610.patch) > TestSleepJob fails in jdk7 > -- > > Key: MAPREDUCE-5610 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5610 > Project: Hadoop Map/Reduce > Issue Type: Test >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-5610.patch, MAPREDUCE-5610.patch, > MAPREDUCE-5610.patch > > > In jdk7 tests methods in a class do not run in file order, but rather in > random order. TestSleepJob hosts are not initialized and a > NullPointerException is thrown unless testRandomLocation was run first. > This can be easily seen by running tests individually. > org.apache.hadoop.mapred.gridmix.TestSleepJob#testStressSubmit > org.apache.hadoop.mapred.gridmix.TestSleepJob#testReplaySubmit > org.apache.hadoop.mapred.gridmix.TestSleepJob#testSerialSubmit > org.apache.hadoop.mapred.gridmix.TestSleepJob#testMapTasksOnlySleepJobs > org.apache.hadoop.mapred.gridmix.TestSleepJob#testRandomLocation -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5610) TestSleepJob fails in jdk7
[ https://issues.apache.org/jira/browse/MAPREDUCE-5610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5610: --- Attachment: MAPREDUCE-5610.patch > TestSleepJob fails in jdk7 > -- > > Key: MAPREDUCE-5610 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5610 > Project: Hadoop Map/Reduce > Issue Type: Test >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-5610.patch, MAPREDUCE-5610.patch, > MAPREDUCE-5610.patch > > > In jdk7 tests methods in a class do not run in file order, but rather in > random order. TestSleepJob hosts are not initialized and a > NullPointerException is thrown unless testRandomLocation was run first. > This can be easily seen by running tests individually. > org.apache.hadoop.mapred.gridmix.TestSleepJob#testStressSubmit > org.apache.hadoop.mapred.gridmix.TestSleepJob#testReplaySubmit > org.apache.hadoop.mapred.gridmix.TestSleepJob#testSerialSubmit > org.apache.hadoop.mapred.gridmix.TestSleepJob#testMapTasksOnlySleepJobs > org.apache.hadoop.mapred.gridmix.TestSleepJob#testRandomLocation -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5427) TestRMContainerAllocator.testUpdatedNodes fails on jdk7
[
https://issues.apache.org/jira/browse/MAPREDUCE-5427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13820370#comment-13820370
]
Jonathan Eagles commented on MAPREDUCE-5427:
When a new patch is updated to address the test failures can the
testUpdatedNodes be moved to be the first test like in the patch I added. This
will prove that the test is truly working on jdk6 and jdk7.
> TestRMContainerAllocator.testUpdatedNodes fails on jdk7
> ---
>
> Key: MAPREDUCE-5427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5427
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 2.1.0-beta, 2.0.5-alpha
>Reporter: Nemon Lou
>Assignee: Sandy Ryza
> Labels: java7
> Attachments: MAPREDUCE-5427.patch, MAPREDUCE-5427.patch
>
>
> {code}
> ---
> Test set: org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
> ---
> Tests run: 14, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 42.777 sec
> <<< FAILURE!
> testUpdatedNodes(org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator)
> Time elapsed: 0.14 sec <<< FAILURE!
> junit.framework.AssertionFailedError: null
> at junit.framework.Assert.fail(Assert.java:47)
> at junit.framework.Assert.assertTrue(Assert.java:20)
> at junit.framework.Assert.assertTrue(Assert.java:27)
> at
> org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator.testUpdatedNodes(TestRMContainerAllocator.java:747)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-5610) TestSleepJob fails in jdk7
[ https://issues.apache.org/jira/browse/MAPREDUCE-5610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5610: --- Attachment: MAPREDUCE-5610.patch > TestSleepJob fails in jdk7 > -- > > Key: MAPREDUCE-5610 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5610 > Project: Hadoop Map/Reduce > Issue Type: Test >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-5610.patch, MAPREDUCE-5610.patch, > MAPREDUCE-5610.patch > > > In jdk7 tests methods in a class do not run in file order, but rather in > random order. TestSleepJob hosts are not initialized and a > NullPointerException is thrown unless testRandomLocation was run first. > This can be easily seen by running tests individually. > org.apache.hadoop.mapred.gridmix.TestSleepJob#testStressSubmit > org.apache.hadoop.mapred.gridmix.TestSleepJob#testReplaySubmit > org.apache.hadoop.mapred.gridmix.TestSleepJob#testSerialSubmit > org.apache.hadoop.mapred.gridmix.TestSleepJob#testMapTasksOnlySleepJobs > org.apache.hadoop.mapred.gridmix.TestSleepJob#testRandomLocation -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (MAPREDUCE-5593) Cleanup code for AssignMapsWithLocality() in RMContainerAllocator
[
https://issues.apache.org/jira/browse/MAPREDUCE-5593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13820369#comment-13820369
]
Hadoop QA commented on MAPREDUCE-5593:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12613332/MAPREDUCE-5593-v4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4192//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4192//console
This message is automatically generated.
> Cleanup code for AssignMapsWithLocality() in RMContainerAllocator
> -
>
> Key: MAPREDUCE-5593
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5593
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: applicationmaster
>Reporter: Junping Du
>Assignee: Junping Du
>Priority: Minor
> Attachments: MAPREDUCE-5593-v3.patch, MAPREDUCE-5593-v4.patch,
> YARN-1163-v1.patch, YARN-1163-v2.patch
>
>
> In RMContainerAllocator, AssignMapsWithLocality() is a very important method
> to assign map tasks on allocated containers with conforming different level
> of locality (dataLocal, rackLocal, etc.). However, this method messed with
> different code logic to handle different type of locality but have lots of
> similar behaviours. This is hard to maintain as well as do extension with
> other locality type, so we need some more clear code here.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (MAPREDUCE-5427) TestRMContainerAllocator.testUpdatedNodes fails on jdk7
[
https://issues.apache.org/jira/browse/MAPREDUCE-5427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13820360#comment-13820360
]
Hadoop QA commented on MAPREDUCE-5427:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12613386/MAPREDUCE-5427.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4191//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4191//console
This message is automatically generated.
> TestRMContainerAllocator.testUpdatedNodes fails on jdk7
> ---
>
> Key: MAPREDUCE-5427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5427
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 2.1.0-beta, 2.0.5-alpha
>Reporter: Nemon Lou
>Assignee: Sandy Ryza
> Labels: java7
> Attachments: MAPREDUCE-5427.patch, MAPREDUCE-5427.patch
>
>
> {code}
> ---
> Test set: org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
> ---
> Tests run: 14, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 42.777 sec
> <<< FAILURE!
> testUpdatedNodes(org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator)
> Time elapsed: 0.14 sec <<< FAILURE!
> junit.framework.AssertionFailedError: null
> at junit.framework.Assert.fail(Assert.java:47)
> at junit.framework.Assert.assertTrue(Assert.java:20)
> at junit.framework.Assert.assertTrue(Assert.java:27)
> at
> org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator.testUpdatedNodes(TestRMContainerAllocator.java:747)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethod
[jira] [Commented] (MAPREDUCE-5610) TestSleepJob fails in jdk7
[
https://issues.apache.org/jira/browse/MAPREDUCE-5610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13820348#comment-13820348
]
Hadoop QA commented on MAPREDUCE-5610:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12613392/MAPREDUCE-5610.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-tools/hadoop-gridmix:
org.apache.hadoop.mapred.gridmix.TestSleepJob
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4190//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4190//console
This message is automatically generated.
> TestSleepJob fails in jdk7
> --
>
> Key: MAPREDUCE-5610
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5610
> Project: Hadoop Map/Reduce
> Issue Type: Test
>Reporter: Jonathan Eagles
>Assignee: Jonathan Eagles
> Attachments: MAPREDUCE-5610.patch, MAPREDUCE-5610.patch
>
>
> In jdk7 tests methods in a class do not run in file order, but rather in
> random order. TestSleepJob hosts are not initialized and a
> NullPointerException is thrown unless testRandomLocation was run first.
> This can be easily seen by running tests individually.
> org.apache.hadoop.mapred.gridmix.TestSleepJob#testStressSubmit
> org.apache.hadoop.mapred.gridmix.TestSleepJob#testReplaySubmit
> org.apache.hadoop.mapred.gridmix.TestSleepJob#testSerialSubmit
> org.apache.hadoop.mapred.gridmix.TestSleepJob#testMapTasksOnlySleepJobs
> org.apache.hadoop.mapred.gridmix.TestSleepJob#testRandomLocation
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Updated] (MAPREDUCE-5610) TestSleepJob fails in jdk7
[ https://issues.apache.org/jira/browse/MAPREDUCE-5610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-5610: --- Attachment: MAPREDUCE-5610.patch > TestSleepJob fails in jdk7 > -- > > Key: MAPREDUCE-5610 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5610 > Project: Hadoop Map/Reduce > Issue Type: Test >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-5610.patch, MAPREDUCE-5610.patch > > > In jdk7 tests methods in a class do not run in file order, but rather in > random order. TestSleepJob hosts are not initialized and a > NullPointerException is thrown unless testRandomLocation was run first. > This can be easily seen by running tests individually. > org.apache.hadoop.mapred.gridmix.TestSleepJob#testStressSubmit > org.apache.hadoop.mapred.gridmix.TestSleepJob#testReplaySubmit > org.apache.hadoop.mapred.gridmix.TestSleepJob#testSerialSubmit > org.apache.hadoop.mapred.gridmix.TestSleepJob#testMapTasksOnlySleepJobs > org.apache.hadoop.mapred.gridmix.TestSleepJob#testRandomLocation -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Updated] (MAPREDUCE-5427) TestRMContainerAllocator.testUpdatedNodes fails on jdk7
[
https://issues.apache.org/jira/browse/MAPREDUCE-5427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Eagles updated MAPREDUCE-5427:
---
Attachment: MAPREDUCE-5427.patch
This patch still fails for me. To show this more clearly, I have re added this
patch with testUpdatedNodes as the first test to force the build to run that
test first on jdk6. This patch fails on both jdk6 and jdk7.
> TestRMContainerAllocator.testUpdatedNodes fails on jdk7
> ---
>
> Key: MAPREDUCE-5427
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5427
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 2.1.0-beta, 2.0.5-alpha
>Reporter: Nemon Lou
>Assignee: Sandy Ryza
> Labels: java7
> Attachments: MAPREDUCE-5427.patch, MAPREDUCE-5427.patch
>
>
> {code}
> ---
> Test set: org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
> ---
> Tests run: 14, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 42.777 sec
> <<< FAILURE!
> testUpdatedNodes(org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator)
> Time elapsed: 0.14 sec <<< FAILURE!
> junit.framework.AssertionFailedError: null
> at junit.framework.Assert.fail(Assert.java:47)
> at junit.framework.Assert.assertTrue(Assert.java:20)
> at junit.framework.Assert.assertTrue(Assert.java:27)
> at
> org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator.testUpdatedNodes(TestRMContainerAllocator.java:747)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:31)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:236)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:134)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:113)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
> at
> org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
> at
> org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:103)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:74)
> {code}
--
This message was sent by Atlassian JIRA
(v6.1#6144)
