date:20120628


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj K reassigned MAPREDUCE-4380:


Assignee: Devaraj K

 Empty Userlogs directory is getting created under logs directory
 

 Key: MAPREDUCE-4380
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4380
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, nodemanager
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Minor

 Empty Userlogs directory is getting created under logs directory.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4346) Adding a refined version of JobTracker.getAllJobs() and exposing through the JobClient


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403022#comment-13403022
 ] 

Hudson commented on MAPREDUCE-4346:
---

Integrated in Hadoop-Hdfs-trunk #1090 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1090/])
Reverting MAPREDUCE-4346 r1353757 (Revision 1354656)

 Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1354656
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestJobClient.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestJobClientGetJob.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Cluster.java


 Adding a refined version of JobTracker.getAllJobs() and exposing through the 
 JobClient
 --

 Key: MAPREDUCE-4346
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4346
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv1
Reporter: Ahmed Radwan
Assignee: Ahmed Radwan
 Attachments: MAPREDUCE-4346.patch, MAPREDUCE-4346_rev2.patch, 
 MAPREDUCE-4346_rev3.patch, MAPREDUCE-4346_rev4.patch


 The current implementation for JobTracker.getAllJobs() returns all submitted 
 jobs in any state, in addition to retired jobs. This list can be long and 
 represents an unneeded overhead especially in the case of clients only 
 interested in jobs in specific state(s). 
 It is beneficial to include a refined version where only jobs having specific 
 statuses are returned and retired jobs are optional to include. 
 I'll be uploading an initial patch momentarily.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4372) Deadlock in Resource Manager between SchedulerEventDispatcher.EventProcessor and Shutdown hook manager


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403027#comment-13403027
 ] 

Hudson commented on MAPREDUCE-4372:
---

Integrated in Hadoop-Hdfs-trunk #1090 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1090/])
MAPREDUCE-4372. Deadlock in Resource Manager (Devaraj K via bobby) 
(Revision 1354531)

 Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1354531
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java


 Deadlock in Resource Manager between SchedulerEventDispatcher.EventProcessor 
 and Shutdown hook manager
 --

 Key: MAPREDUCE-4372
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4372
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, resourcemanager
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
 Fix For: 2.0.1-alpha, 3.0.0

 Attachments: MAPREDUCE-4372-1.patch, MAPREDUCE-4372.patch, 
 rm-threaddump.out


 Please find the attached resource manager thread dump for the issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4380) Empty Userlogs directory is getting created under logs directory


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj K updated MAPREDUCE-4380:
-

Attachment: MAPREDUCE-4380.patch

Attaching the patch to fix this issue.

 Empty Userlogs directory is getting created under logs directory
 

 Key: MAPREDUCE-4380
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4380
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, nodemanager
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Minor
 Attachments: MAPREDUCE-4380.patch


 Empty Userlogs directory is getting created under logs directory.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4380) Empty Userlogs directory is getting created under logs directory


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj K updated MAPREDUCE-4380:
-

Status: Patch Available  (was: Open)

 Empty Userlogs directory is getting created under logs directory
 

 Key: MAPREDUCE-4380
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4380
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, nodemanager
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Minor
 Attachments: MAPREDUCE-4380.patch


 Empty Userlogs directory is getting created under logs directory.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4380) Empty Userlogs directory is getting created under logs directory

[
https://issues.apache.org/jira/browse/MAPREDUCE-4380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403044#comment-13403044
]

Hadoop QA commented on MAPREDUCE-4380:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12533814/MAPREDUCE-4380.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2525//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2525//console

This message is automatically generated.

Empty Userlogs directory is getting created under logs directory

Key: MAPREDUCE-4380
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4380
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, nodemanager
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Minor
Attachments: MAPREDUCE-4380.patch

Empty Userlogs directory is getting created under logs directory.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4346) Adding a refined version of JobTracker.getAllJobs() and exposing through the JobClient

2012-06-28 Thread Alejandro Abdelnur (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403095#comment-13403095
 ] 

Alejandro Abdelnur commented on MAPREDUCE-4346:
---

FYI the Hudson integration comment above is incorrect as it is for another JIRA 
for which I've used the wrong ID number in the commit.

 Adding a refined version of JobTracker.getAllJobs() and exposing through the 
 JobClient
 --

 Key: MAPREDUCE-4346
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4346
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv1
Reporter: Ahmed Radwan
Assignee: Ahmed Radwan
 Attachments: MAPREDUCE-4346.patch, MAPREDUCE-4346_rev2.patch, 
 MAPREDUCE-4346_rev3.patch, MAPREDUCE-4346_rev4.patch


 The current implementation for JobTracker.getAllJobs() returns all submitted 
 jobs in any state, in addition to retired jobs. This list can be long and 
 represents an unneeded overhead especially in the case of clients only 
 interested in jobs in specific state(s). 
 It is beneficial to include a refined version where only jobs having specific 
 statuses are returned and retired jobs are optional to include. 
 I'll be uploading an initial patch momentarily.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4346) Adding a refined version of JobTracker.getAllJobs() and exposing through the JobClient


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403121#comment-13403121
 ] 

Hudson commented on MAPREDUCE-4346:
---

Integrated in Hadoop-Mapreduce-trunk #1123 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1123/])
Reverting MAPREDUCE-4346 r1353757 (Revision 1354656)

 Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1354656
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestJobClient.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/test/java/org/apache/hadoop/mapred/TestJobClientGetJob.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Cluster.java


 Adding a refined version of JobTracker.getAllJobs() and exposing through the 
 JobClient
 --

 Key: MAPREDUCE-4346
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4346
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv1
Reporter: Ahmed Radwan
Assignee: Ahmed Radwan
 Attachments: MAPREDUCE-4346.patch, MAPREDUCE-4346_rev2.patch, 
 MAPREDUCE-4346_rev3.patch, MAPREDUCE-4346_rev4.patch


 The current implementation for JobTracker.getAllJobs() returns all submitted 
 jobs in any state, in addition to retired jobs. This list can be long and 
 represents an unneeded overhead especially in the case of clients only 
 interested in jobs in specific state(s). 
 It is beneficial to include a refined version where only jobs having specific 
 statuses are returned and retired jobs are optional to include. 
 I'll be uploading an initial patch momentarily.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4372) Deadlock in Resource Manager between SchedulerEventDispatcher.EventProcessor and Shutdown hook manager


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403126#comment-13403126
 ] 

Hudson commented on MAPREDUCE-4372:
---

Integrated in Hadoop-Mapreduce-trunk #1123 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1123/])
MAPREDUCE-4372. Deadlock in Resource Manager (Devaraj K via bobby) 
(Revision 1354531)

 Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1354531
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java


 Deadlock in Resource Manager between SchedulerEventDispatcher.EventProcessor 
 and Shutdown hook manager
 --

 Key: MAPREDUCE-4372
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4372
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, resourcemanager
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
 Fix For: 2.0.1-alpha, 3.0.0

 Attachments: MAPREDUCE-4372-1.patch, MAPREDUCE-4372.patch, 
 rm-threaddump.out


 Please find the attached resource manager thread dump for the issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4365) Shipping Profiler Libraries by DistributedCache


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403191#comment-13403191
 ] 

Robert Joseph Evans commented on MAPREDUCE-4365:


@Jie on 1.0 that may work, but I don't know if we are exploding the job.jar for 
2.0.  I think we need to have a JIRA for creating the symlinks before launching 
at least in 2.0.

 Shipping Profiler Libraries by DistributedCache
 ---

 Key: MAPREDUCE-4365
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4365
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
Affects Versions: 1.0.3
Reporter: Jie Li

 Hadoop profiling is great for performance tuning and debugging, but currently 
 we can only use Java built-in profilers such as HProf, and for other 
 profilers we need to install them on all slave nodes first, which is 
 inconvenient for large clusters and sometimes impossible for production 
 clusters. 
 Supporting shipping profiler libraries using DistributedCache will solve this 
 problem. For example, in mapred.task.profile.params, we specify a profiler 
 library from the DistributedCache using special place holders such as 
 foo.jar, and Hadoop can look at the DistributedCache to replace foo.jar 
 with the localized path before launching the child jvm.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4376) TestClusterMRNotification times out


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kihwal Lee updated MAPREDUCE-4376:
--

Fix Version/s: 3.0.0
   2.0.1-alpha

 TestClusterMRNotification times out
 ---

 Key: MAPREDUCE-4376
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4376
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, test
Affects Versions: 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Kihwal Lee
 Fix For: 2.0.1-alpha, 3.0.0


 The TestClusterMRNotification test is often timing out.  git bisect tests 
 narrowed it down to MAPREDUCE-3921, as the test consistently passes before 
 that change and times out most of the time after picking up that change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4376) TestClusterMRNotification times out


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kihwal Lee updated MAPREDUCE-4376:
--

Attachment: mapreduce-4376.patch

What this patch does:
- Fixes the NPE bug in {{RMContainerAllocator}}.
- Improves {{UtilsForTests}} by making the kill/fail job runner to timeout.
- Improves {{NotificationTestCase}} by having it check for more failure 
conditions.

{{TestJobHistory}}, {{TestJobInProgressListener}} and {{TestJobKillAndFail}} 
also call the kill/fail job runner in {{UtilsForTests}}. They were all tested 
okay with the new timeout.

 TestClusterMRNotification times out
 ---

 Key: MAPREDUCE-4376
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4376
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, test
Affects Versions: 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Kihwal Lee
 Fix For: 2.0.1-alpha, 3.0.0

 Attachments: mapreduce-4376.patch


 The TestClusterMRNotification test is often timing out.  git bisect tests 
 narrowed it down to MAPREDUCE-3921, as the test consistently passes before 
 that change and times out most of the time after picking up that change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4376) TestClusterMRNotification times out


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403237#comment-13403237
 ] 

Kihwal Lee commented on MAPREDUCE-4376:
---

- Also verified that the timeout works when the bug fix is missing.

{noformat}
---
Test set: org.apache.hadoop.mapred.TestClusterMRNotification
---
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 77.437 sec  
FAILURE!
testMR(org.apache.hadoop.mapred.TestClusterMRNotification)  Time elapsed: 
77.365 sec   ERROR!
java.io.IOException: Job cleanup didn't start in 30 seconds
at 
org.apache.hadoop.mapred.UtilsForTests.runJobKill(UtilsForTests.java:676)
at 
org.apache.hadoop.mapred.NotificationTestCase.testMR(NotificationTestCase.java:174)
{noformat}

 TestClusterMRNotification times out
 ---

 Key: MAPREDUCE-4376
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4376
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, test
Affects Versions: 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Kihwal Lee
 Fix For: 2.0.1-alpha, 3.0.0

 Attachments: mapreduce-4376.patch


 The TestClusterMRNotification test is often timing out.  git bisect tests 
 narrowed it down to MAPREDUCE-3921, as the test consistently passes before 
 that change and times out most of the time after picking up that change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4376) TestClusterMRNotification times out


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kihwal Lee updated MAPREDUCE-4376:
--

Status: Patch Available  (was: Open)

 TestClusterMRNotification times out
 ---

 Key: MAPREDUCE-4376
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4376
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, test
Affects Versions: 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Kihwal Lee
 Fix For: 2.0.1-alpha, 3.0.0

 Attachments: mapreduce-4376.patch


 The TestClusterMRNotification test is often timing out.  git bisect tests 
 narrowed it down to MAPREDUCE-3921, as the test consistently passes before 
 that change and times out most of the time after picking up that change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4328) Add the option to quiesce the JobTracker

2012-06-28 Thread Tom White (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403261#comment-13403261
]

Tom White commented on MAPREDUCE-4328:
--

3. Doesn't accept new job submissions.

To be clear - the client would get a failure, right? The current patch doesn't
do that yet as far as I can see.

A few other pieces of feedback on the patch:

* The -refreshNodes option in MRAdmin was deleted from the usage message.
* Rather than putting markup in the JobTracker (in getSafeModeText()), do the
formatting in the JSP or a utility class like JSPUtil (which already exists).
* Change JobTracker's getSafeMode() method to isInSafeMode(), to mirror
NameNode.
* MRAdmin introduced a couple of unneeded imports: DistributedFileSystem,
org.mortbay.log.Log

Add the option to quiesce the JobTracker

Key: MAPREDUCE-4328
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4328
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv1
Affects Versions: 1.0.3
Reporter: Arun C Murthy
Assignee: Arun C Murthy
Attachments: MAPREDUCE-4328.patch

In several failure scenarios it would be very handy to have an option to
quiesce the JobTracker.
Recently, we saw a case where the NameNode had to be rebooted at a customer
due to a random hardware failure - in such a case it would have been nice to
not lose jobs by quiescing the JobTracker.

[jira] [Commented] (MAPREDUCE-4376) TestClusterMRNotification times out

[
https://issues.apache.org/jira/browse/MAPREDUCE-4376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403318#comment-13403318
]

Hadoop QA commented on MAPREDUCE-4376:
--

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12533846/mapreduce-4376.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 2 new or modified test
files.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2526//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2526//console

This message is automatically generated.

TestClusterMRNotification times out
---

Key: MAPREDUCE-4376
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4376
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, test
Affects Versions: 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Kihwal Lee
Fix For: 2.0.1-alpha, 3.0.0

Attachments: mapreduce-4376.patch

The TestClusterMRNotification test is often timing out. git bisect tests
narrowed it down to MAPREDUCE-3921, as the test consistently passes before
that change and times out most of the time after picking up that change.

[jira] [Updated] (MAPREDUCE-4355) Add RunningJob.getJobStatus()


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated MAPREDUCE-4355:


Attachment: MR-4355_mr2.patch

Updated patch for MR2.
- RunningJob.getJobStatus()
- TestNetworkedJob uses MiniMRClientCluster

 Add RunningJob.getJobStatus()
 -

 Key: MAPREDUCE-4355
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4355
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: mrv1, mrv2
Affects Versions: 1.0.3, 2.0.0-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Fix For: 1.1.0, 2.0.1-alpha

 Attachments: MR-4355_mr1.patch, MR-4355_mr2.patch


 Usecase: Read the start/end-time of a particular job.
 Currently, one has to iterate through JobClient.getAllJobStatuses() and 
 iterate through them. JobClient.getJob(JobID) returns RunningJob, which 
 doesn't hold the job's start time.
 Adding RunningJob.getJobStatus() solves the issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4355) Add RunningJob.getJobStatus()


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated MAPREDUCE-4355:


Attachment: (was: MR-4355_mr2.patch)

 Add RunningJob.getJobStatus()
 -

 Key: MAPREDUCE-4355
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4355
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: mrv1, mrv2
Affects Versions: 1.0.3, 2.0.0-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Fix For: 1.1.0, 2.0.1-alpha

 Attachments: MR-4355_mr1.patch, MR-4355_mr2.patch


 Usecase: Read the start/end-time of a particular job.
 Currently, one has to iterate through JobClient.getAllJobStatuses() and 
 iterate through them. JobClient.getJob(JobID) returns RunningJob, which 
 doesn't hold the job's start time.
 Adding RunningJob.getJobStatus() solves the issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4355) Add RunningJob.getJobStatus()


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated MAPREDUCE-4355:


Attachment: (was: MR-4355_mr1.patch)

 Add RunningJob.getJobStatus()
 -

 Key: MAPREDUCE-4355
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4355
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: mrv1, mrv2
Affects Versions: 1.0.3, 2.0.0-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Fix For: 1.1.0, 2.0.1-alpha

 Attachments: MR-4355_mr1.patch, MR-4355_mr2.patch


 Usecase: Read the start/end-time of a particular job.
 Currently, one has to iterate through JobClient.getAllJobStatuses() and 
 iterate through them. JobClient.getJob(JobID) returns RunningJob, which 
 doesn't hold the job's start time.
 Adding RunningJob.getJobStatus() solves the issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4369) Fix streaming job failures with WindowsResourceCalculatorPlugin


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1340#comment-1340
 ] 

Bikas Saha commented on MAPREDUCE-4369:
---

Let me look at that. I was thinking about revisiting that too.

 Fix streaming job failures with WindowsResourceCalculatorPlugin
 ---

 Key: MAPREDUCE-4369
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4369
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Bikas Saha
Assignee: Bikas Saha
 Attachments: MAPREDUCE-4369.branch-1-win.1.patch


 Some streaming jobs use local mode job runs that do not start tasks trackers. 
 In these cases, the jvm context is not setup and hence local mode execution 
 causes the code to crash.
 Fix is to not not use ResourceCalculatorPlugin in such cases or make the 
 local job run creating dummy jvm contexts. Choosing the first option because 
 thats the current implicit behavior in Linux. The ProcfsBasedProcessTree 
 (used inside the LinuxResourceCalculatorPlugin) does no real work when the 
 process pid is not setup correctly. This is what happens when local job mode 
 runs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4369) Fix streaming job failures with WindowsResourceCalculatorPlugin

[
https://issues.apache.org/jira/browse/MAPREDUCE-4369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403336#comment-13403336
]

Bikas Saha commented on MAPREDUCE-4369:
---

Actually, the cause of this bug was a null JVMContext and not an invalid
ProcessPid. I should have kept the other process pid changes separate from this
bug fix.

Fix streaming job failures with WindowsResourceCalculatorPlugin
---

Key: MAPREDUCE-4369
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4369
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Bikas Saha
Assignee: Bikas Saha
Attachments: MAPREDUCE-4369.branch-1-win.1.patch

Some streaming jobs use local mode job runs that do not start tasks trackers.
In these cases, the jvm context is not setup and hence local mode execution
causes the code to crash.
Fix is to not not use ResourceCalculatorPlugin in such cases or make the
local job run creating dummy jvm contexts. Choosing the first option because
thats the current implicit behavior in Linux. The ProcfsBasedProcessTree
(used inside the LinuxResourceCalculatorPlugin) does no real work when the
process pid is not setup correctly. This is what happens when local job mode
runs.

[jira] [Commented] (MAPREDUCE-4369) Fix streaming job failures with WindowsResourceCalculatorPlugin

[
https://issues.apache.org/jira/browse/MAPREDUCE-4369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403360#comment-13403360
]

Ivan Mitic commented on MAPREDUCE-4369:
---

bq. Actually, the cause of this bug was a null JVMContext and not an invalid
ProcessPid. I should have kept the other process pid changes separate from this
bug fix.
Right, there are two fixes in your change. I still believe that it would be
useful to refactor {{ResourceCalculatorPlugin#getResourceCalculatorPlugin()}}
to accept processPid and explicitly pass it on the call site.

On a related note, I'm not a fan of the fallback to
{{System.getenv().get(JVM_PID)}} on Windows, that's why I asked for the list
of tests :) Might be fine at this point, but it is something we should take a
look at later.

Fix streaming job failures with WindowsResourceCalculatorPlugin
---

[jira] [Created] (MAPREDUCE-4381) Make PROGRESS_INTERVAL of org.apache.hadoop.mapred.Task a tunable

2012-06-28 Thread Shrinivas Joshi (JIRA)

Shrinivas Joshi created MAPREDUCE-4381:
--

 Summary: Make PROGRESS_INTERVAL of org.apache.hadoop.mapred.Task a 
tunable
 Key: MAPREDUCE-4381
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4381
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: task, tasktracker
Reporter: Shrinivas Joshi
Priority: Minor


Currently PROGRESS_INTERVAL is a hard-coded value and is set to 3000 msec. We 
tried making it a tunable and experimented with different values. In some cases 
setting it to a smaller value like 1000 msec helps significantly improve 
performance of short running jobs such as piEstimator. This is because the task 
threads do not end up blocking for as many as 3 seconds for their last progress 
update event. We also noticed close to 14% improvement on Mahout KMeans 
iteration jobs which take more than 5 minutes on the test cluster that we are 
using. Please let me know if this seems to be a good idea. I have an initial 
patch that I have attached here. This is based on branch-1 tree. It may need 
some rework on MRv2 based branches I think. Also note that I have not changed 
the variable naming style for PROGRESS_INTERVAL even though it is not a public 
static final anymore. I can revise the patch if there are no objections to this 
idea. 
Thanks.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Moved] (MAPREDUCE-4382) RMContainerImpl State Machine Doesn't Handle RELEASED State from AM Properly

2012-06-28 Thread Jason Lowe (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe moved HADOOP-8539 to MAPREDUCE-4382:
---

Affects Version/s: (was: 0.23.3)
   0.23.3
  Key: MAPREDUCE-4382  (was: HADOOP-8539)
  Project: Hadoop Map/Reduce  (was: Hadoop Common)

 RMContainerImpl State Machine Doesn't Handle RELEASED State from AM Properly
 

 Key: MAPREDUCE-4382
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4382
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.23.3
Reporter: Jack Dintruff
 Attachments: NewStateMachine.jpg, OldStateMachine.jpg


 Encountered this error with an RMContainerImpl unable to go from the RUNNING 
 state to the RELEASED state.  Upon further inspection, the state machine has 
 no means of dealing with this and since the RELEASED state can be sent from 
 the AM at any time, it should be able to handle this in both the RUNNING 
 state and the ALLOCATED state.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4381) Make PROGRESS_INTERVAL of org.apache.hadoop.mapred.Task a tunable

2012-06-28 Thread Shrinivas Joshi (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shrinivas Joshi updated MAPREDUCE-4381:
---

Attachment: progress_interval.patch

 Make PROGRESS_INTERVAL of org.apache.hadoop.mapred.Task a tunable
 -

 Key: MAPREDUCE-4381
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4381
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: task, tasktracker
Reporter: Shrinivas Joshi
Priority: Minor
 Attachments: progress_interval.patch


 Currently PROGRESS_INTERVAL is a hard-coded value and is set to 3000 msec. We 
 tried making it a tunable and experimented with different values. In some 
 cases setting it to a smaller value like 1000 msec helps significantly 
 improve performance of short running jobs such as piEstimator. This is 
 because the task threads do not end up blocking for as many as 3 seconds for 
 their last progress update event. We also noticed close to 14% improvement on 
 Mahout KMeans iteration jobs which take more than 5 minutes on the test 
 cluster that we are using. Please let me know if this seems to be a good 
 idea. I have an initial patch that I have attached here. This is based on 
 branch-1 tree. It may need some rework on MRv2 based branches I think. Also 
 note that I have not changed the variable naming style for PROGRESS_INTERVAL 
 even though it is not a public static final anymore. I can revise the patch 
 if there are no objections to this idea. 
 Thanks.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4382) RMContainerImpl State Machine Doesn't Handle RELEASED State from AM Properly

2012-06-28 Thread Jason Lowe (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe updated MAPREDUCE-4382:
--

Component/s: resourcemanager
 mrv2

 RMContainerImpl State Machine Doesn't Handle RELEASED State from AM Properly
 

 Key: MAPREDUCE-4382
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4382
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, resourcemanager
Affects Versions: 0.23.3
Reporter: Jack Dintruff
 Attachments: NewStateMachine.jpg, OldStateMachine.jpg


 Encountered this error with an RMContainerImpl unable to go from the RUNNING 
 state to the RELEASED state.  Upon further inspection, the state machine has 
 no means of dealing with this and since the RELEASED state can be sent from 
 the AM at any time, it should be able to handle this in both the RUNNING 
 state and the ALLOCATED state.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4376) TestClusterMRNotification times out


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403399#comment-13403399
 ] 

Robert Joseph Evans commented on MAPREDUCE-4376:


The changes look good to me.  All of the changes are to test code, and Jenkins 
gave it a +1 so I give it a +1 too.  Thanks for the fixes Kihwal I'll check 
them in.

 TestClusterMRNotification times out
 ---

 Key: MAPREDUCE-4376
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4376
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, test
Affects Versions: 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Kihwal Lee
 Fix For: 2.0.1-alpha, 3.0.0

 Attachments: mapreduce-4376.patch


 The TestClusterMRNotification test is often timing out.  git bisect tests 
 narrowed it down to MAPREDUCE-3921, as the test consistently passes before 
 that change and times out most of the time after picking up that change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4383) HadoopPipes.cc compile fails with HadoopPipes.cc:1049:16: error: ‘sleep’ was not declared in this scope


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy Isaacson updated MAPREDUCE-4383:
-

Attachment: mapreduce-4383.txt

Attaching one-line patch that fixes my build.

 HadoopPipes.cc compile fails with HadoopPipes.cc:1049:16: error: ‘sleep’ was 
 not declared in this scope
 -

 Key: MAPREDUCE-4383
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4383
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: pipes
Affects Versions: 2.0.1-alpha, 3.0.0
Reporter: Andy Isaacson
Priority: Minor
 Attachments: mapreduce-4383.txt


 Since MAPREDUCE-4267 I've seen mvn -Pnative compile failing with:
 {quote}
  [exec] /usr/bin/c++-g -Wall -O2 -D_REENTRANT -D_FILE_OFFSET_BITS=64 
 -I/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/utils/api
  
 -I/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/api
  -I/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src-o 
 CMakeFiles/hadooppipes.dir/main/native/pipes/impl/HadoopPipes.cc.o -c 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc
  [exec] 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:
  In member function ‘std::string 
 HadoopPipes::BinaryProtocol::createDigest(std::string, std::string)’:
  [exec] 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:439:21:
  warning: value computed is not used [-Wunused-value]
  [exec] 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:
  In function ‘void* HadoopPipes::ping(void*)’:
  [exec] 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:1049:16:
  error: ‘sleep’ was not declared in this scope
  [exec] 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:1067:30:
  error: ‘close’ was not declared in this scope
  [exec] 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:
  In function ‘bool HadoopPipes::runTask(const HadoopPipes::Factory)’:
  [exec] 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:1162:28:
  error: ‘close’ was not declared in this scope
  [exec] make[2]: *** 
 [CMakeFiles/hadooppipes.dir/main/native/pipes/impl/HadoopPipes.cc.o] Error 1
 {quote}
 I believe the failure is new simply because I wasn't compiling pipes before.
 The fix is pretty simple, just include unistd.h in HadoopPipes.cc.
 My environment is debian unstable, amd64, g++ 4.7.0-6, openjdk-6-jdk 
 6b24-1.11.1-6.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4383) HadoopPipes.cc compile fails with HadoopPipes.cc:1049:16: error: ‘sleep’ was not declared in this scope


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy Isaacson updated MAPREDUCE-4383:
-

Status: Patch Available  (was: Open)

 HadoopPipes.cc compile fails with HadoopPipes.cc:1049:16: error: ‘sleep’ was 
 not declared in this scope
 -

 Key: MAPREDUCE-4383
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4383
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: pipes
Affects Versions: 2.0.1-alpha, 3.0.0
Reporter: Andy Isaacson
Priority: Minor
 Attachments: mapreduce-4383.txt


 Since MAPREDUCE-4267 I've seen mvn -Pnative compile failing with:
 {quote}
  [exec] /usr/bin/c++-g -Wall -O2 -D_REENTRANT -D_FILE_OFFSET_BITS=64 
 -I/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/utils/api
  
 -I/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/api
  -I/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src-o 
 CMakeFiles/hadooppipes.dir/main/native/pipes/impl/HadoopPipes.cc.o -c 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc
  [exec] 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:
  In member function ‘std::string 
 HadoopPipes::BinaryProtocol::createDigest(std::string, std::string)’:
  [exec] 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:439:21:
  warning: value computed is not used [-Wunused-value]
  [exec] 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:
  In function ‘void* HadoopPipes::ping(void*)’:
  [exec] 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:1049:16:
  error: ‘sleep’ was not declared in this scope
  [exec] 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:1067:30:
  error: ‘close’ was not declared in this scope
  [exec] 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:
  In function ‘bool HadoopPipes::runTask(const HadoopPipes::Factory)’:
  [exec] 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:1162:28:
  error: ‘close’ was not declared in this scope
  [exec] make[2]: *** 
 [CMakeFiles/hadooppipes.dir/main/native/pipes/impl/HadoopPipes.cc.o] Error 1
 {quote}
 I believe the failure is new simply because I wasn't compiling pipes before.
 The fix is pretty simple, just include unistd.h in HadoopPipes.cc.
 My environment is debian unstable, amd64, g++ 4.7.0-6, openjdk-6-jdk 
 6b24-1.11.1-6.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4376) TestClusterMRNotification times out


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403409#comment-13403409
 ] 

Hudson commented on MAPREDUCE-4376:
---

Integrated in Hadoop-Hdfs-trunk-Commit #2472 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2472/])
MAPREDUCE-4376. TestClusterMRNotification times out (Kihwal Lee via bobby) 
(Revision 1355124)

 Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1355124
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/NotificationTestCase.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/UtilsForTests.java


 TestClusterMRNotification times out
 ---

 Key: MAPREDUCE-4376
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4376
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, test
Affects Versions: 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Kihwal Lee
 Fix For: 2.0.1-alpha, 3.0.0

 Attachments: mapreduce-4376.patch


 The TestClusterMRNotification test is often timing out.  git bisect tests 
 narrowed it down to MAPREDUCE-3921, as the test consistently passes before 
 that change and times out most of the time after picking up that change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4376) TestClusterMRNotification times out


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403413#comment-13403413
 ] 

Hudson commented on MAPREDUCE-4376:
---

Integrated in Hadoop-Common-trunk-Commit #2404 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2404/])
MAPREDUCE-4376. TestClusterMRNotification times out (Kihwal Lee via bobby) 
(Revision 1355124)

 Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1355124
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/NotificationTestCase.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/UtilsForTests.java


 TestClusterMRNotification times out
 ---

 Key: MAPREDUCE-4376
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4376
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, test
Affects Versions: 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Kihwal Lee
 Fix For: 2.0.1-alpha, 3.0.0

 Attachments: mapreduce-4376.patch


 The TestClusterMRNotification test is often timing out.  git bisect tests 
 narrowed it down to MAPREDUCE-3921, as the test consistently passes before 
 that change and times out most of the time after picking up that change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4374) Fix child task environment variable config and add support for Windows

2012-06-28 Thread Vinod Kumar Vavilapalli (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403423#comment-13403423
]

Vinod Kumar Vavilapalli commented on MAPREDUCE-4374:

Just checked that the CLASSPATH separator on Windows is indeed ';', so yeah, it
is natural to use it. Same for '%'.

Good to see that we are not breaking any compatibility for existing users and
only defining new behaviour for users on Windows.

bq. Why not use existing syntax, i.e. $ and ':' (e.g. '$x=a:b'), to set
environment variables on Windows?
Didn't understand this part, the existing syntax is
mapred.child.env=MY_PATH=/tmp, is that how you set vars on Windows?

Fix child task environment variable config and add support for Windows
--

Key: MAPREDUCE-4374
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4374
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 1-win
Reporter: Chuan Liu
Assignee: Chuan Liu
Priority: Minor
Attachments: MAPREDUCE-4374-branch-1-win.patch

In HADOOP-2838, a new feature was introduced to set environment variables via
the Hadoop config 'mapred.child.env' for child tasks. There are some further
fixes and improvements around this feature, e.g. HADOOP-5981 were a bug fix;
MAPREDUCE-478 broke the config into 'mapred.map.child.env' and
'mapred.reduce.child.env'. However the current implementation is still not
complete. It does not match its documentation or original intend as I
believe. Also, by using ‘:’ (colon) and ‘;’ (semicolon) in the configuration
syntax, we will have problems using them on Windows because ‘:’ appears very
often in Windows path as in “C:\”, and environment variables are used very
often to hold path names. The Jira is created to fix the problem and provide
support on Windows.

[jira] [Commented] (MAPREDUCE-4383) HadoopPipes.cc compile fails with HadoopPipes.cc:1049:16: error: ‘sleep’ was not declared in this scope

[
https://issues.apache.org/jira/browse/MAPREDUCE-4383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403427#comment-13403427
]

Hadoop QA commented on MAPREDUCE-4383:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12533878/mapreduce-4383.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in .

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2528//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2528//console

This message is automatically generated.

HadoopPipes.cc compile fails with HadoopPipes.cc:1049:16: error: ‘sleep’ was
not declared in this scope
-

Key: MAPREDUCE-4383
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4383
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: pipes
Affects Versions: 2.0.1-alpha, 3.0.0
Reporter: Andy Isaacson
Priority: Minor
Attachments: mapreduce-4383.txt

Since MAPREDUCE-4267 I've seen mvn -Pnative compile failing with:
{quote}
[exec] /usr/bin/c++-g -Wall -O2 -D_REENTRANT -D_FILE_OFFSET_BITS=64
-I/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/utils/api

-I/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/api
-I/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src-o
CMakeFiles/hadooppipes.dir/main/native/pipes/impl/HadoopPipes.cc.o -c
/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc
[exec]
/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:
In member function ‘std::string
HadoopPipes::BinaryProtocol::createDigest(std::string, std::string)’:
[exec]
/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:439:21:
warning: value computed is not used [-Wunused-value]
[exec]
/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:
In function ‘void* HadoopPipes::ping(void*)’:
[exec]
/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:1049:16:
error: ‘sleep’ was not declared in this scope
[exec]
/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:1067:30:
error: ‘close’ was not declared in this scope
[exec]
/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:
In function ‘bool HadoopPipes::runTask(const HadoopPipes::Factory)’:
[exec]
/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:1162:28:
error: ‘close’ was not declared in this scope
[exec] make[2]: ***
[CMakeFiles/hadooppipes.dir/main/native/pipes/impl/HadoopPipes.cc.o] Error 1
{quote}
I believe the failure is new simply because I wasn't compiling pipes before.
The fix is pretty simple, just include unistd.h in HadoopPipes.cc.
My environment is debian unstable, amd64, g++ 4.7.0-6, openjdk-6-jdk
6b24-1.11.1-6.

[jira] [Commented] (MAPREDUCE-4383) HadoopPipes.cc compile fails with HadoopPipes.cc:1049:16: error: ‘sleep’ was not declared in this scope


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403430#comment-13403430
 ] 

Andy Isaacson commented on MAPREDUCE-4383:
--

This patch doesn't include any tests because it's a compile fix.

 HadoopPipes.cc compile fails with HadoopPipes.cc:1049:16: error: ‘sleep’ was 
 not declared in this scope
 -

 Key: MAPREDUCE-4383
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4383
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: pipes
Affects Versions: 2.0.1-alpha, 3.0.0
Reporter: Andy Isaacson
Priority: Minor
 Attachments: mapreduce-4383.txt


 Since MAPREDUCE-4267 I've seen mvn -Pnative compile failing with:
 {quote}
  [exec] /usr/bin/c++-g -Wall -O2 -D_REENTRANT -D_FILE_OFFSET_BITS=64 
 -I/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/utils/api
  
 -I/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/api
  -I/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src-o 
 CMakeFiles/hadooppipes.dir/main/native/pipes/impl/HadoopPipes.cc.o -c 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc
  [exec] 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:
  In member function ‘std::string 
 HadoopPipes::BinaryProtocol::createDigest(std::string, std::string)’:
  [exec] 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:439:21:
  warning: value computed is not used [-Wunused-value]
  [exec] 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:
  In function ‘void* HadoopPipes::ping(void*)’:
  [exec] 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:1049:16:
  error: ‘sleep’ was not declared in this scope
  [exec] 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:1067:30:
  error: ‘close’ was not declared in this scope
  [exec] 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:
  In function ‘bool HadoopPipes::runTask(const HadoopPipes::Factory)’:
  [exec] 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:1162:28:
  error: ‘close’ was not declared in this scope
  [exec] make[2]: *** 
 [CMakeFiles/hadooppipes.dir/main/native/pipes/impl/HadoopPipes.cc.o] Error 1
 {quote}
 I believe the failure is new simply because I wasn't compiling pipes before.
 The fix is pretty simple, just include unistd.h in HadoopPipes.cc.
 My environment is debian unstable, amd64, g++ 4.7.0-6, openjdk-6-jdk 
 6b24-1.11.1-6.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (MAPREDUCE-4383) HadoopPipes.cc compile fails with HadoopPipes.cc:1049:16: error: ‘sleep’ was not declared in this scope


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andy Isaacson reassigned MAPREDUCE-4383:


Assignee: Andy Isaacson

 HadoopPipes.cc compile fails with HadoopPipes.cc:1049:16: error: ‘sleep’ was 
 not declared in this scope
 -

 Key: MAPREDUCE-4383
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4383
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: pipes
Affects Versions: 2.0.1-alpha, 3.0.0
Reporter: Andy Isaacson
Assignee: Andy Isaacson
Priority: Minor
 Attachments: mapreduce-4383.txt


 Since MAPREDUCE-4267 I've seen mvn -Pnative compile failing with:
 {quote}
  [exec] /usr/bin/c++-g -Wall -O2 -D_REENTRANT -D_FILE_OFFSET_BITS=64 
 -I/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/utils/api
  
 -I/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/api
  -I/home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src-o 
 CMakeFiles/hadooppipes.dir/main/native/pipes/impl/HadoopPipes.cc.o -c 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc
  [exec] 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:
  In member function ‘std::string 
 HadoopPipes::BinaryProtocol::createDigest(std::string, std::string)’:
  [exec] 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:439:21:
  warning: value computed is not used [-Wunused-value]
  [exec] 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:
  In function ‘void* HadoopPipes::ping(void*)’:
  [exec] 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:1049:16:
  error: ‘sleep’ was not declared in this scope
  [exec] 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:1067:30:
  error: ‘close’ was not declared in this scope
  [exec] 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:
  In function ‘bool HadoopPipes::runTask(const HadoopPipes::Factory)’:
  [exec] 
 /home/adi/w/apache-hadoop-trunk/hadoop-tools/hadoop-pipes/src/main/native/pipes/impl/HadoopPipes.cc:1162:28:
  error: ‘close’ was not declared in this scope
  [exec] make[2]: *** 
 [CMakeFiles/hadooppipes.dir/main/native/pipes/impl/HadoopPipes.cc.o] Error 1
 {quote}
 I believe the failure is new simply because I wasn't compiling pipes before.
 The fix is pretty simple, just include unistd.h in HadoopPipes.cc.
 My environment is debian unstable, amd64, g++ 4.7.0-6, openjdk-6-jdk 
 6b24-1.11.1-6.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4371) Check for cyclic dependencies in Jobcontrol job DAG


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403432#comment-13403432
 ] 

Robert Joseph Evans commented on MAPREDUCE-4371:


The change looks good to me and the javadoc warnings appear to be caused by a 
different patch.  +1 I'll check this in for you.  I am going to put it in trunk 
because that is what the target version is set to.  If you want this to go into 
branch-2 I think that should be fine, just post another comment to let me know.

 Check for cyclic dependencies in Jobcontrol job DAG
 ---

 Key: MAPREDUCE-4371
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4371
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv1
Affects Versions: 3.0.0
Reporter: madhukara phatak
 Attachments: MAPREDUCE-4371-1.patch, MAPREDUCE-4371.patch


 In current implementation of JobControl, whenever there is a cyclic 
 dependency between the jobs it throws a Stack overflow exception. This jira 
 adds a cyclic check to jobcontrol.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4371) Check for cyclic dependencies in Jobcontrol job DAG


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Joseph Evans updated MAPREDUCE-4371:
---

   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Thanks Madhukara,

I put this into trunk.  Keep up the good work.

 Check for cyclic dependencies in Jobcontrol job DAG
 ---

 Key: MAPREDUCE-4371
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4371
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv1
Affects Versions: 3.0.0
Reporter: madhukara phatak
 Fix For: 3.0.0

 Attachments: MAPREDUCE-4371-1.patch, MAPREDUCE-4371.patch


 In current implementation of JobControl, whenever there is a cyclic 
 dependency between the jobs it throws a Stack overflow exception. This jira 
 adds a cyclic check to jobcontrol.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4384) Race conditions in IndexCache

Kihwal Lee created MAPREDUCE-4384:
-

 Summary: Race conditions in IndexCache
 Key: MAPREDUCE-4384
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4384
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.0.0-alpha
Reporter: Kihwal Lee
 Fix For: 0.23.3, 2.0.1-alpha, 3.0.0


TestIndexCache is intermittently failing due to a race condition. Up on 
inspection of IndexCache implementation, more potential issues have been 
discovered.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4355) Add RunningJob.getJobStatus()

[
https://issues.apache.org/jira/browse/MAPREDUCE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403437#comment-13403437
]

Hadoop QA commented on MAPREDUCE-4355:
--

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12533864/MR-4355_mr2.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 2 new or modified test
files.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2527//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2527//console

This message is automatically generated.

Add RunningJob.getJobStatus()
-

Key: MAPREDUCE-4355
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4355
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: mrv1, mrv2
Affects Versions: 1.0.3, 2.0.0-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
Fix For: 1.1.0, 2.0.1-alpha

Attachments: MR-4355_mr1.patch, MR-4355_mr2.patch

Usecase: Read the start/end-time of a particular job.
Currently, one has to iterate through JobClient.getAllJobStatuses() and
iterate through them. JobClient.getJob(JobID) returns RunningJob, which
doesn't hold the job's start time.
Adding RunningJob.getJobStatus() solves the issue.

[jira] [Commented] (MAPREDUCE-4371) Check for cyclic dependencies in Jobcontrol job DAG


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403447#comment-13403447
 ] 

Hudson commented on MAPREDUCE-4371:
---

Integrated in Hadoop-Common-trunk-Commit #2405 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2405/])
MAPREDUCE-4371. Check for cyclic dependencies in Jobcontrol job DAG 
(madhukara phatak via bobby) (Revision 1355139)

 Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1355139
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/jobcontrol/JobControl.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/jobcontrol
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/jobcontrol/TestJobControl.java


 Check for cyclic dependencies in Jobcontrol job DAG
 ---

 Key: MAPREDUCE-4371
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4371
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv1
Affects Versions: 3.0.0
Reporter: madhukara phatak
 Fix For: 3.0.0

 Attachments: MAPREDUCE-4371-1.patch, MAPREDUCE-4371.patch


 In current implementation of JobControl, whenever there is a cyclic 
 dependency between the jobs it throws a Stack overflow exception. This jira 
 adds a cyclic check to jobcontrol.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4371) Check for cyclic dependencies in Jobcontrol job DAG


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403446#comment-13403446
 ] 

Hudson commented on MAPREDUCE-4371:
---

Integrated in Hadoop-Hdfs-trunk-Commit #2473 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2473/])
MAPREDUCE-4371. Check for cyclic dependencies in Jobcontrol job DAG 
(madhukara phatak via bobby) (Revision 1355139)

 Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1355139
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/jobcontrol/JobControl.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/jobcontrol
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/jobcontrol/TestJobControl.java


 Check for cyclic dependencies in Jobcontrol job DAG
 ---

 Key: MAPREDUCE-4371
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4371
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv1
Affects Versions: 3.0.0
Reporter: madhukara phatak
 Fix For: 3.0.0

 Attachments: MAPREDUCE-4371-1.patch, MAPREDUCE-4371.patch


 In current implementation of JobControl, whenever there is a cyclic 
 dependency between the jobs it throws a Stack overflow exception. This jira 
 adds a cyclic check to jobcontrol.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4385) FairScheduler.maxTasksToAssign() should check for fairscheduler.assignmultiple.maps TaskTracker.availableSlots

Karthik Kambatla created MAPREDUCE-4385:
---

 Summary: FairScheduler.maxTasksToAssign() should check for 
fairscheduler.assignmultiple.maps  TaskTracker.availableSlots
 Key: MAPREDUCE-4385
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4385
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Affects Versions: 1.0.3
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla


FairScheduler.maxTasksToAssign() can potentially return a value greater than 
the available slots. Currently, we rely on canAssignMaps()/canAssignReduces() 
to reject such requests.

These additional calls can be avoided by check against the available slots in 
maxTasksToAssign().

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4385) FairScheduler.maxTasksToAssign() should check for fairscheduler.assignmultiple.maps TaskTracker.availableSlots


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated MAPREDUCE-4385:


Attachment: MR-4385.patch

Uploading a straight-forward patch that returns the minimum of availableSlots() 
and the number of tasks requested as per config file.

Testing:
- Didn't add additional tests
- Ran TestFairScheduler

 FairScheduler.maxTasksToAssign() should check for 
 fairscheduler.assignmultiple.maps  TaskTracker.availableSlots
 

 Key: MAPREDUCE-4385
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4385
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Affects Versions: 1.0.3
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Attachments: MR-4385.patch


 FairScheduler.maxTasksToAssign() can potentially return a value greater than 
 the available slots. Currently, we rely on canAssignMaps()/canAssignReduces() 
 to reject such requests.
 These additional calls can be avoided by check against the available slots in 
 maxTasksToAssign().

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4384) Race conditions in IndexCache


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403464#comment-13403464
 ] 

Kihwal Lee commented on MAPREDUCE-4384:
---

When {{TestIndexCache}} failed, the log contained a warning message, Map 
ID not found in queue!!. The queue is used to figure out what to drop in 
its the FIFO cache replacement policy. This message indicates that the cache 
entry was freed by a removeMap() call, but the corresponding entry was not 
found in the queue.

This can happen if {{removeMap()}} is called while the cache entry is being 
loaded. If a new incomplete entry is added to the cache between 
{{cache.get(mapId)}} and {[cache.remove{{mapId}} in {{removeMap()}}, the new 
entry will be removed from the cache. Further, if {{totalMemoryUsed}} is 
updated before the entry is fully loaded, it will end up subtracting zero from 
the usage. When the loading is complete in {{readIndexFileToCache()}}, 
{{totalMemoryUsed}} will be incremented, but since it was already removed from 
the cache, there is no way it can be decremented. Hence the discrepancy in 
memory usage tracking.

This issue can be fixed by adding one more condition to the first check in 
{{removeMap()}}

{noformat}
   IndexInformation info = cache.get(mapId);
 - if ((info != null)  (info.getSize() == 0)) {
 + if (info == null || ((info != null)  (info.getSize() == 0))) {
 return;
}
{noformat}


Another potential issue is in {{readIndexFileToCache()}}. When two different 
threads are trying to add the same entry to the cache, there can be a deadlock. 
When Thread A puts a new {{IndexInformation}} object in the cache,  Thread B 
can come in a bit late and do {{wait()}} on this object to be fully ready. The 
{{wait()}} is inside the {{synchronized(info)}} block and {{info}} is the new 
object it just found in the cache.  But Thread A also tries to update the same 
object and do {{notifyAll()}} inside a synchronized() block on it. This results 
in a deadlock.



 Race conditions in IndexCache
 -

 Key: MAPREDUCE-4384
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4384
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.0.0-alpha
Reporter: Kihwal Lee
 Fix For: 0.23.3, 2.0.1-alpha, 3.0.0


 TestIndexCache is intermittently failing due to a race condition. Up on 
 inspection of IndexCache implementation, more potential issues have been 
 discovered.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4371) Check for cyclic dependencies in Jobcontrol job DAG


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403477#comment-13403477
 ] 

Hudson commented on MAPREDUCE-4371:
---

Integrated in Hadoop-Mapreduce-trunk-Commit #2423 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2423/])
MAPREDUCE-4371. Check for cyclic dependencies in Jobcontrol job DAG 
(madhukara phatak via bobby) (Revision 1355139)

 Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1355139
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/jobcontrol/JobControl.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/jobcontrol
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/jobcontrol/TestJobControl.java


 Check for cyclic dependencies in Jobcontrol job DAG
 ---

 Key: MAPREDUCE-4371
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4371
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv1
Affects Versions: 3.0.0
Reporter: madhukara phatak
 Fix For: 3.0.0

 Attachments: MAPREDUCE-4371-1.patch, MAPREDUCE-4371.patch


 In current implementation of JobControl, whenever there is a cyclic 
 dependency between the jobs it throws a Stack overflow exception. This jira 
 adds a cyclic check to jobcontrol.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4376) TestClusterMRNotification times out


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403475#comment-13403475
 ] 

Hudson commented on MAPREDUCE-4376:
---

Integrated in Hadoop-Mapreduce-trunk-Commit #2423 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2423/])
MAPREDUCE-4376. TestClusterMRNotification times out (Kihwal Lee via bobby) 
(Revision 1355124)

 Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1355124
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/NotificationTestCase.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/UtilsForTests.java


 TestClusterMRNotification times out
 ---

 Key: MAPREDUCE-4376
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4376
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, test
Affects Versions: 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Kihwal Lee
 Fix For: 2.0.1-alpha, 3.0.0

 Attachments: mapreduce-4376.patch


 The TestClusterMRNotification test is often timing out.  git bisect tests 
 narrowed it down to MAPREDUCE-3921, as the test consistently passes before 
 that change and times out most of the time after picking up that change.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4014) Make yarn-common services robust

2012-06-28 Thread Siddharth Seth (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Siddharth Seth updated MAPREDUCE-4014:
--

Status: Open  (was: Patch Available)

 Make yarn-common services robust
 

 Key: MAPREDUCE-4014
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4014
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: mrv2
Affects Versions: 0.23.1, 0.24.0
Reporter: Steve Loughran
Assignee: Steve Loughran
  Labels: yarn
 Attachments: MAPREDUCE-4014.patch


 Review the yarn common services ({{CompositeService}}, 
 {{AbstractLivelinessMonitor}} and make their service startup _and especially 
 shutdown_ more robust against out-of-lifecycle invocation and partially 
 complete initialization.
 Write tests for these where possible. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4014) Make yarn-common services robust

2012-06-28 Thread Siddharth Seth (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403517#comment-13403517
]

Siddharth Seth commented on MAPREDUCE-4014:
---

Steve, apologies for the delayed feedback. I was wondering which parts of
MAPREDUCE-3939 are planned for this patch.
Static methods to choreograph of lifecycle operations seems to be covered.
Will the others, specifically AbstractService doesn't prevent duplicate state
change requests, AbstractService state change doesn't defend against race
conditions, state model prevents stopped state being entered if you could not
successfully start the service be separate patches? Definitely looks like they
should go in.

bq. state model prevents stopped state being entered if you could not
successfully start the service.
I don't believe resources which require an explicit release are meant to be
obtained in the init() stage - but that may not always be the case. I'd agree
with allowing stop() from any state, as well as a CompositeService attempting
to stop all child services when told to stop() (instead of just the ones which
have started).
Currently, a failed start() on a composite service will stop() services which
had started and move them to STOPPED state, attempt to stop() the failed
service but leave it in INITED state, and leave remaining services in INITED
state - which doesn't seem correct.

Feedback on this patch
- {{interruptAndJoinThread(Thread target)}} - could have a joinTimeout
parameter as well.
- stopIPCServer, stopWebApp, interrupt* - should these be in AbstractService ?
or a separate helper class.
- toString() in AbstractService - the text is missing a closing quote.

Make yarn-common services robust

Key: MAPREDUCE-4014
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4014
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2
Affects Versions: 0.23.1, 0.24.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Labels: yarn
Attachments: MAPREDUCE-4014.patch

Review the yarn common services ({{CompositeService}},
{{AbstractLivelinessMonitor}} and make their service startup _and especially
shutdown_ more robust against out-of-lifecycle invocation and partially
complete initialization.
Write tests for these where possible.

[jira] [Commented] (MAPREDUCE-4384) Race conditions in IndexCache


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403567#comment-13403567
 ] 

Kihwal Lee commented on MAPREDUCE-4384:
---

branch-1 has similar problem, but its usage is different. While 
{{ShuffleHandler}} does not call {{IndexCache#removeMap()}} in 2.0, 
{{TaskTracker}} does call it. The race seems very rare, if ever actually 
happens. I will work on a branch-1 version, once the trunk version is done.

 Race conditions in IndexCache
 -

 Key: MAPREDUCE-4384
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4384
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.0.0-alpha
Reporter: Kihwal Lee
 Fix For: 0.23.3, 2.0.1-alpha, 3.0.0


 TestIndexCache is intermittently failing due to a race condition. Up on 
 inspection of IndexCache implementation, more potential issues have been 
 discovered.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (MAPREDUCE-4384) Race conditions in IndexCache


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kihwal Lee reassigned MAPREDUCE-4384:
-

Assignee: Kihwal Lee

 Race conditions in IndexCache
 -

 Key: MAPREDUCE-4384
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4384
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.0.0-alpha
Reporter: Kihwal Lee
Assignee: Kihwal Lee
 Fix For: 0.23.3, 2.0.1-alpha, 3.0.0


 TestIndexCache is intermittently failing due to a race condition. Up on 
 inspection of IndexCache implementation, more potential issues have been 
 discovered.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4384) Race conditions in IndexCache


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kihwal Lee updated MAPREDUCE-4384:
--

Attachment: mapreduce-4384.patch

A new test case tries to create the new entry race condition. I ran it many 
times, but only got it to happen once.

 Race conditions in IndexCache
 -

 Key: MAPREDUCE-4384
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4384
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.0.0-alpha
Reporter: Kihwal Lee
Assignee: Kihwal Lee
 Fix For: 0.23.3, 2.0.1-alpha, 3.0.0

 Attachments: mapreduce-4384.patch


 TestIndexCache is intermittently failing due to a race condition. Up on 
 inspection of IndexCache implementation, more potential issues have been 
 discovered.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4384) Race conditions in IndexCache


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kihwal Lee updated MAPREDUCE-4384:
--

Status: Patch Available  (was: Open)

 Race conditions in IndexCache
 -

 Key: MAPREDUCE-4384
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4384
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.0.0-alpha
Reporter: Kihwal Lee
Assignee: Kihwal Lee
 Fix For: 0.23.3, 2.0.1-alpha, 3.0.0

 Attachments: mapreduce-4384.patch


 TestIndexCache is intermittently failing due to a race condition. Up on 
 inspection of IndexCache implementation, more potential issues have been 
 discovered.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4369) Fix streaming job failures with WindowsResourceCalculatorPlugin

[
https://issues.apache.org/jira/browse/MAPREDUCE-4369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Bikas Saha updated MAPREDUCE-4369:
--

Attachment: MAPREDUCE-4369.branch-1-win.2.patch

Removing the implicit JVM_PID usage and explicitly calling
Plugin.setProcessId() at call sites.

Fix streaming job failures with WindowsResourceCalculatorPlugin
---

[jira] [Commented] (MAPREDUCE-4342) Distributed Cache gives inconsistent result if cache files get deleted from task tracker

2012-06-28 Thread Konstantin Shvachko (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403585#comment-13403585
 ] 

Konstantin Shvachko commented on MAPREDUCE-4342:


+1 for branch 0.22 patch

 Distributed Cache gives inconsistent result if cache files get deleted from 
 task tracker 
 -

 Key: MAPREDUCE-4342
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4342
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.22.0, 1.0.3, trunk
Reporter: Mayank Bansal
Assignee: Mayank Bansal
 Attachments: MAPREDUCE-4342-22-1.patch, MAPREDUCE-4342-22-2.patch, 
 MAPREDUCE-4342-22-3.patch, MAPREDUCE-4342-22.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4342) Distributed Cache gives inconsistent result if cache files get deleted from task tracker

2012-06-28 Thread Konstantin Shvachko (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403589#comment-13403589
 ] 

Konstantin Shvachko commented on MAPREDUCE-4342:


I just committed this to branch 0.22. Thank you Mayank.
Is it applicable for trunk? If so could you please attach a patch.

 Distributed Cache gives inconsistent result if cache files get deleted from 
 task tracker 
 -

 Key: MAPREDUCE-4342
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4342
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.22.0, 1.0.3, trunk
Reporter: Mayank Bansal
Assignee: Mayank Bansal
 Attachments: MAPREDUCE-4342-22-1.patch, MAPREDUCE-4342-22-2.patch, 
 MAPREDUCE-4342-22-3.patch, MAPREDUCE-4342-22.patch




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4384) Race conditions in IndexCache

[
https://issues.apache.org/jira/browse/MAPREDUCE-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403617#comment-13403617
]

Hadoop QA commented on MAPREDUCE-4384:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12533908/mapreduce-4384.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test
files.

-1 javac. The applied patch generated 2071 javac compiler warnings (more
than the trunk's current 2070 warnings).

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

-1 core tests. The patch failed these unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient:

org.apache.hadoop.mapred.jobcontrol.TestLocalJobControl

org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2529//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2529//artifact/trunk/trunk/patchprocess/diffJavacWarnings.txt
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2529//console

This message is automatically generated.

Race conditions in IndexCache
-

Key: MAPREDUCE-4384
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4384
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: nodemanager
Affects Versions: 2.0.0-alpha
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Fix For: 0.23.3, 2.0.1-alpha, 3.0.0

Attachments: mapreduce-4384.patch

TestIndexCache is intermittently failing due to a race condition. Up on
inspection of IndexCache implementation, more potential issues have been
discovered.

[jira] [Updated] (MAPREDUCE-4355) Add RunningJob.getJobStatus()


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated MAPREDUCE-4355:


Attachment: MR-4355_mr2.patch

Patch with updated test:
- TestNetworkedJob defines its own MR job, independent of other tests

 Add RunningJob.getJobStatus()
 -

 Key: MAPREDUCE-4355
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4355
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: mrv1, mrv2
Affects Versions: 1.0.3, 2.0.0-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Fix For: 1.1.0, 2.0.1-alpha

 Attachments: MR-4355_mr1.patch, MR-4355_mr2.patch


 Usecase: Read the start/end-time of a particular job.
 Currently, one has to iterate through JobClient.getAllJobStatuses() and 
 iterate through them. JobClient.getJob(JobID) returns RunningJob, which 
 doesn't hold the job's start time.
 Adding RunningJob.getJobStatus() solves the issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4355) Add RunningJob.getJobStatus()


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Karthik Kambatla updated MAPREDUCE-4355:


Attachment: (was: MR-4355_mr2.patch)

 Add RunningJob.getJobStatus()
 -

 Key: MAPREDUCE-4355
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4355
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: mrv1, mrv2
Affects Versions: 1.0.3, 2.0.0-alpha
Reporter: Karthik Kambatla
Assignee: Karthik Kambatla
 Fix For: 1.1.0, 2.0.1-alpha

 Attachments: MR-4355_mr1.patch, MR-4355_mr2.patch


 Usecase: Read the start/end-time of a particular job.
 Currently, one has to iterate through JobClient.getAllJobStatuses() and 
 iterate through them. JobClient.getJob(JobID) returns RunningJob, which 
 doesn't hold the job's start time.
 Adding RunningJob.getJobStatus() solves the issue.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4355) Add RunningJob.getJobStatus()

[
https://issues.apache.org/jira/browse/MAPREDUCE-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403664#comment-13403664
]

Hadoop QA commented on MAPREDUCE-4355:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12533921/MR-4355_mr2.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test
files.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl
org.apache.hadoop.mapred.jobcontrol.TestLocalJobControl

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2530//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2530//console

This message is automatically generated.

Add RunningJob.getJobStatus()
-

Attachments: MR-4355_mr1.patch, MR-4355_mr2.patch

[jira] [Created] (MAPREDUCE-4386) Investigate possible abstractions for shell commands bash -c and cmd /c

Ivan Mitic created MAPREDUCE-4386:
-

 Summary: Investigate possible abstractions for shell commands 
bash -c and cmd /c
 Key: MAPREDUCE-4386
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4386
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Ivan Mitic


Currently, we have multiple files within the codebase that have {{if (WINDOWS) 
cmd /c … else bash –c …}}. Instead, we should try to scope this down to 
only Shell.java and expose the needed functionality to other places. We might 
not be able to remove all such occurrences, but it looks like there is room for 
improvement (check the discussion on MAPREDUCE-4322).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4322) Fix command-line length abort issues on Windows


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403670#comment-13403670
 ] 

Ivan Mitic commented on MAPREDUCE-4322:
---

FYI, I opened MAPREDUCE-4386 for better abstractions around different shells.

 Fix command-line length abort issues on Windows
 ---

 Key: MAPREDUCE-4322
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4322
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: tasktracker
 Environment: Windows, downstream applications with long aggregate 
 classpaths
Reporter: John Gordon
Assignee: Ivan Mitic
 Attachments: MAPREDUCE-4322-branch-1-win(2).patch, 
 MAPREDUCE-4322-branch-1-win(3).patch, MAPREDUCE-4322-branch-1-win(4).patch, 
 MAPREDUCE-4322-branch-1-win(5).patch, MAPREDUCE-4322-branch-1-win.patch

   Original Estimate: 12h
  Remaining Estimate: 12h

 When a task is started on the tasktracker, it creates a small batch file to 
 invoke java and runs that batch.  Within the batch file, the invocation of 
 Java currently has -classpath ${CLASSPATH} inline to the command.  That line 
 often exceeds 8000 characters.  This is ok for most linux distributions 
 because the line limit env variable is often set much higher than this.  
 However, for Windows this cause cmd to abort execution.  This surfaces in 
 Hadoop as an unknown failure mode for the task.
 I think the easiest and most natural way to fix this is to push the 
 -classpath option into a config file to take the longest variable part of the 
 line and put it somewhere that scales better.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4369) Fix streaming job failures with WindowsResourceCalculatorPlugin


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403674#comment-13403674
 ] 

Ivan Mitic commented on MAPREDUCE-4369:
---

Thanks Bikas, change looks much better.

I have two additional questions:
1. I see that you still kept the setProcessPid() separate. Is there a reason 
for going with this model instead of passing it as part of 
{{ResourceCalculatorPlugin.getResourceCalculatorPlugin()}}?
2. In Task.java, you now initialize {{resourceCalculator}} to {{null}} if 
{{jvmContext}} is {{null}}. Before your original change for MAPREDUCE-4203, we 
would have the {{resourceCalculator}} with {{System.getenv(JVM_PID)}} as the 
{{processPid}}. Is there a scenario where the old behavior is desired?

 Fix streaming job failures with WindowsResourceCalculatorPlugin
 ---

 Key: MAPREDUCE-4369
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4369
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Bikas Saha
Assignee: Bikas Saha
 Attachments: MAPREDUCE-4369.branch-1-win.1.patch, 
 MAPREDUCE-4369.branch-1-win.2.patch


 Some streaming jobs use local mode job runs that do not start tasks trackers. 
 In these cases, the jvm context is not setup and hence local mode execution 
 causes the code to crash.
 Fix is to not not use ResourceCalculatorPlugin in such cases or make the 
 local job run creating dummy jvm contexts. Choosing the first option because 
 thats the current implicit behavior in Linux. The ProcfsBasedProcessTree 
 (used inside the LinuxResourceCalculatorPlugin) does no real work when the 
 process pid is not setup correctly. This is what happens when local job mode 
 runs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4379) Node Manager throws java.lang.OutOfMemoryError: Java heap space due to org.apache.hadoop.fs.LocalDirAllocator.contexts


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj K updated MAPREDUCE-4379:
-

Priority: Blocker  (was: Critical)

 Node Manager throws java.lang.OutOfMemoryError: Java heap space due to 
 org.apache.hadoop.fs.LocalDirAllocator.contexts
 --

 Key: MAPREDUCE-4379
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4379
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, nodemanager
Affects Versions: 0.23.3, 2.0.0-alpha, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Blocker

 {code:xml}
 Exception in thread Container Monitor java.lang.OutOfMemoryError: Java heap 
 space
   at java.io.BufferedReader.init(BufferedReader.java:80)
   at java.io.BufferedReader.init(BufferedReader.java:91)
   at 
 org.apache.hadoop.yarn.util.ProcfsBasedProcessTree.constructProcessInfo(ProcfsBasedProcessTree.java:410)
   at 
 org.apache.hadoop.yarn.util.ProcfsBasedProcessTree.getProcessTree(ProcfsBasedProcessTree.java:171)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl$MonitoringThread.run(ContainersMonitorImpl.java:389)
   Exception in thread LocalizerRunner for 
 container_1340690914008_10890_01_03 java.lang.OutOfMemoryError: Java 
 heap space
   at java.util.Arrays.copyOfRange(Arrays.java:3209)
   at java.lang.String.init(String.java:215)
   at 
 com.sun.org.apache.xerces.internal.xni.XMLString.toString(XMLString.java:185)
   at 
 com.sun.org.apache.xerces.internal.parsers.AbstractDOMParser.characters(AbstractDOMParser.java:1188)
   at 
 com.sun.org.apache.xerces.internal.xinclude.XIncludeHandler.characters(XIncludeHandler.java:1084)
   at 
 com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:464)
   at 
 com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:808)
   at 
 com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:737)
   at 
 com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:119)
   at 
 com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:235)
   at 
 com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:284)
   at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:180)
   at 
 org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:1738)
   at 
 org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:1689)
   at 
 org.apache.hadoop.conf.Configuration.getProps(Configuration.java:1635)
   at org.apache.hadoop.conf.Configuration.set(Configuration.java:722)
   at 
 org.apache.hadoop.conf.Configuration.setStrings(Configuration.java:1300)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.initDirs(ContainerLocalizer.java:375)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ContainerLocalizer.runLocalization(ContainerLocalizer.java:127)
   at 
 org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.startLocalizer(DefaultContainerExecutor.java:103)
   at 
 org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService$LocalizerRunner.run(ResourceLocalizationService.java:862)
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4371) Check for cyclic dependencies in Jobcontrol job DAG


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403692#comment-13403692
 ] 

Kihwal Lee commented on MAPREDUCE-4371:
---

{{TestLocalJobControl}} and {{TestMapReduceJobControl}} are failing in trunk 
due to NPE from JobControl#isCircular(). 

 Check for cyclic dependencies in Jobcontrol job DAG
 ---

 Key: MAPREDUCE-4371
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4371
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv1
Affects Versions: 3.0.0
Reporter: madhukara phatak
 Fix For: 3.0.0

 Attachments: MAPREDUCE-4371-1.patch, MAPREDUCE-4371.patch


 In current implementation of JobControl, whenever there is a cyclic 
 dependency between the jobs it throws a Stack overflow exception. This jira 
 adds a cyclic check to jobcontrol.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4384) Race conditions in IndexCache


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13403694#comment-13403694
 ] 

Kihwal Lee commented on MAPREDUCE-4384:
---

The two test failures seem to be caused by MAPREDUCE-4371. It reliably fails 
without this patch applied.

 Race conditions in IndexCache
 -

 Key: MAPREDUCE-4384
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4384
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 2.0.0-alpha
Reporter: Kihwal Lee
Assignee: Kihwal Lee
 Fix For: 0.23.3, 2.0.1-alpha, 3.0.0

 Attachments: mapreduce-4384.patch


 TestIndexCache is intermittently failing due to a race condition. Up on 
 inspection of IndexCache implementation, more potential issues have been 
 discovered.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4369) Fix streaming job failures with WindowsResourceCalculatorPlugin