date:20120312


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated MAPREDUCE-3995:
--

Status: Open  (was: Patch Available)

 Add support for  static service lifecycle listeners .
 -

 Key: MAPREDUCE-3995
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3995
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: mrv2
Affects Versions: 0.23.1, 0.24.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
 Attachments: MAPREDUCE-3995.patch, MAPREDUCE-3995.patch, 
 MAPREDUCE-3995.patch, MAPREDUCE-3995.patch


 Add support to {{AbstractService}} that allow callers to register listeners 
 for all instances. The existing listener interface could be used. This allows 
 management tools to hook into the events.
 The static listeners would be invoked for all state changes except creation 
 (base class shouldn't be handing out references to itself at this point).
 These static events could all be async, pushed through a shared 
 {{ConcurrentLinkedQueue}}; failures logged at warn and the rest of the 
 listeners invoked.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3995) Add support for static service lifecycle listeners .


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated MAPREDUCE-3995:
--

Attachment: MAPREDUCE-3995.patch

latest diff

 Add support for  static service lifecycle listeners .
 -

 Key: MAPREDUCE-3995
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3995
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: mrv2
Affects Versions: 0.23.1, 0.24.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
 Attachments: MAPREDUCE-3995.patch, MAPREDUCE-3995.patch, 
 MAPREDUCE-3995.patch, MAPREDUCE-3995.patch


 Add support to {{AbstractService}} that allow callers to register listeners 
 for all instances. The existing listener interface could be used. This allows 
 management tools to hook into the events.
 The static listeners would be invoked for all state changes except creation 
 (base class shouldn't be handing out references to itself at this point).
 These static events could all be async, pushed through a shared 
 {{ConcurrentLinkedQueue}}; failures logged at warn and the rest of the 
 listeners invoked.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3995) Add support for static service lifecycle listeners .


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated MAPREDUCE-3995:
--

Status: Patch Available  (was: Open)

Patch with test. I'm submitting to see what Jenkins says, but I've realised 
that adding the listeners to {{AbstractService}} isn't the right place to add a 
list of static listeners.

It would be better to move the code to the {{ServiceOperations}} class of 
MAPREDUCE-3970, and then invoke it from {{AbstractService}}. Any implementation 
of {{Service}} that did not extend that base class would still have the ability 
to invoke the event list.

Other enhancements
# clone the list before making the notifications to make the iterator robust 
against concurrency problems.
# add a package-scoped {{resetGlobalListeners()}} method for testing, to ensure 
the list is empty after each test run. 

Such a patch will have to wait for MAPREDUCE-3970, so I'll mark this as a 
dependency.

 Add support for  static service lifecycle listeners .
 -

 Key: MAPREDUCE-3995
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3995
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: mrv2
Affects Versions: 0.23.1, 0.24.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
 Attachments: MAPREDUCE-3995.patch, MAPREDUCE-3995.patch, 
 MAPREDUCE-3995.patch, MAPREDUCE-3995.patch


 Add support to {{AbstractService}} that allow callers to register listeners 
 for all instances. The existing listener interface could be used. This allows 
 management tools to hook into the events.
 The static listeners would be invoked for all state changes except creation 
 (base class shouldn't be handing out references to itself at this point).
 These static events could all be async, pushed through a shared 
 {{ConcurrentLinkedQueue}}; failures logged at warn and the rest of the 
 listeners invoked.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3995) Add support for static service lifecycle listeners .


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227406#comment-13227406
 ] 

Hadoop QA commented on MAPREDUCE-3995:
--

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12517994/MAPREDUCE-3995.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 4 new or modified tests.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 eclipse:eclipse.  The patch built with eclipse:eclipse.

+1 findbugs.  The patch does not introduce any new Findbugs (version 1.3.9) 
warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed unit tests in .

+1 contrib tests.  The patch passed contrib unit tests.

Test results: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2035//testReport/
Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2035//console

This message is automatically generated.

 Add support for  static service lifecycle listeners .
 -

 Key: MAPREDUCE-3995
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3995
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: mrv2
Affects Versions: 0.23.1, 0.24.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
 Attachments: MAPREDUCE-3995.patch, MAPREDUCE-3995.patch, 
 MAPREDUCE-3995.patch, MAPREDUCE-3995.patch


 Add support to {{AbstractService}} that allow callers to register listeners 
 for all instances. The existing listener interface could be used. This allows 
 management tools to hook into the events.
 The static listeners would be invoked for all state changes except creation 
 (base class shouldn't be handing out references to itself at this point).
 These static events could all be async, pushed through a shared 
 {{ConcurrentLinkedQueue}}; failures logged at warn and the rest of the 
 listeners invoked.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3829) [Gridmix] Gridmix should give better error message when input-data directory already exists and -generate option is given

2012-03-12 Thread Ravi Gummadi (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Gummadi updated MAPREDUCE-3829:


Status: Open  (was: Patch Available)

 [Gridmix] Gridmix should give better error message when input-data directory 
 already exists and -generate option is given
 -

 Key: MAPREDUCE-3829
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3829
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: contrib/gridmix
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
 Attachments: 3829.v0.patch, 3829.v1.3.patch, 3829.v1.patch


 Instead of throwing exception messages on to the console, Gridmix should give 
 better error message when input-data directory already exists and -generate 
 option is given.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3829) [Gridmix] Gridmix should give better error message when input-data directory already exists and -generate option is given

2012-03-12 Thread Ravi Gummadi (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Gummadi updated MAPREDUCE-3829:


Attachment: 3829.v2.patch

Attaching new patch with minor code changes and more testcases added.

 [Gridmix] Gridmix should give better error message when input-data directory 
 already exists and -generate option is given
 -

 Key: MAPREDUCE-3829
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3829
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: contrib/gridmix
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
 Attachments: 3829.v0.patch, 3829.v1.3.patch, 3829.v1.patch, 
 3829.v2.patch


 Instead of throwing exception messages on to the console, Gridmix should give 
 better error message when input-data directory already exists and -generate 
 option is given.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3829) [Gridmix] Gridmix should give better error message when input-data directory already exists and -generate option is given

2012-03-12 Thread Amar Kamat (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227421#comment-13227421
 ] 

Amar Kamat commented on MAPREDUCE-3829:
---

Looks good to me. +1

 [Gridmix] Gridmix should give better error message when input-data directory 
 already exists and -generate option is given
 -

 Key: MAPREDUCE-3829
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3829
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: contrib/gridmix
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
 Attachments: 3829.v0.patch, 3829.v1.3.patch, 3829.v1.patch, 
 3829.v2.patch


 Instead of throwing exception messages on to the console, Gridmix should give 
 better error message when input-data directory already exists and -generate 
 option is given.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3829) [Gridmix] Gridmix should give better error message when input-data directory already exists and -generate option is given

2012-03-12 Thread Ravi Gummadi (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Gummadi updated MAPREDUCE-3829:


Release Note: Makes Gridmix emit out correct error message when the input 
data directory already exists and -generate option is used. Makes Gridmix exit 
with proper exit codes when Gridmix fails in args-processing, startup/setup.  
(was: Makes Gridmix emit out correct error message when the input data 
directory already exists and -generate option is used. Makes Gridmix exit with 
proper exit codes when Gridmix fails in startup/setup.)
Hadoop Flags: Reviewed
  Status: Patch Available  (was: Open)

 [Gridmix] Gridmix should give better error message when input-data directory 
 already exists and -generate option is given
 -

 Key: MAPREDUCE-3829
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3829
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: contrib/gridmix
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
 Attachments: 3829.v0.patch, 3829.v1.3.patch, 3829.v1.patch, 
 3829.v2.patch


 Instead of throwing exception messages on to the console, Gridmix should give 
 better error message when input-data directory already exists and -generate 
 option is given.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3829) [Gridmix] Gridmix should give better error message when input-data directory already exists and -generate option is given

2012-03-12 Thread Ravi Gummadi (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227432#comment-13227432
 ] 

Ravi Gummadi commented on MAPREDUCE-3829:
-

Unit tests and test-patch passed on my local machine.

I just committed the patch to trunk. Thanks Amar for the review.

 [Gridmix] Gridmix should give better error message when input-data directory 
 already exists and -generate option is given
 -

 Key: MAPREDUCE-3829
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3829
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: contrib/gridmix
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
 Attachments: 3829.v0.patch, 3829.v1.3.patch, 3829.v1.patch, 
 3829.v2.patch


 Instead of throwing exception messages on to the console, Gridmix should give 
 better error message when input-data directory already exists and -generate 
 option is given.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3829) [Gridmix] Gridmix should give better error message when input-data directory already exists and -generate option is given


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227433#comment-13227433
 ] 

Hudson commented on MAPREDUCE-3829:
---

Integrated in Hadoop-Hdfs-trunk-Commit #1941 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1941/])
MAPREDUCE-3829. [Gridmix] Gridmix should give better error message when 
input data directory already exists and -generate option is given.(ravigummadi) 
(Revision 1299625)

 Result = FAILURE
ravigummadi : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1299625
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/DistributedCacheEmulator.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixExitCodes.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java


 [Gridmix] Gridmix should give better error message when input-data directory 
 already exists and -generate option is given
 -

 Key: MAPREDUCE-3829
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3829
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: contrib/gridmix
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
 Attachments: 3829.v0.patch, 3829.v1.3.patch, 3829.v1.patch, 
 3829.v2.patch


 Instead of throwing exception messages on to the console, Gridmix should give 
 better error message when input-data directory already exists and -generate 
 option is given.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3829) [Gridmix] Gridmix should give better error message when input-data directory already exists and -generate option is given


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227435#comment-13227435
 ] 

Hudson commented on MAPREDUCE-3829:
---

Integrated in Hadoop-Common-trunk-Commit #1866 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1866/])
MAPREDUCE-3829. [Gridmix] Gridmix should give better error message when 
input data directory already exists and -generate option is given.(ravigummadi) 
(Revision 1299625)

 Result = SUCCESS
ravigummadi : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1299625
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/DistributedCacheEmulator.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixExitCodes.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java


 [Gridmix] Gridmix should give better error message when input-data directory 
 already exists and -generate option is given
 -

 Key: MAPREDUCE-3829
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3829
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: contrib/gridmix
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
 Attachments: 3829.v0.patch, 3829.v1.3.patch, 3829.v1.patch, 
 3829.v2.patch


 Instead of throwing exception messages on to the console, Gridmix should give 
 better error message when input-data directory already exists and -generate 
 option is given.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3967) DefaultContainerExecutor cannot launch container under windows

2012-03-12 Thread Steve Loughran (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227440#comment-13227440
 ] 

Steve Loughran commented on MAPREDUCE-3967:
---

Looks like the shell path should be generated using 
{{FileUtils.makeShellPath()}}. 

 DefaultContainerExecutor cannot launch container under windows
 --

 Key: MAPREDUCE-3967
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3967
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: nodemanager
Affects Versions: 0.23.3
 Environment: Apache Maven 3.0.4 (r1232337; 2012-01-17 16:44:56+0800)
 Java version: 1.7.0_02, vendor: Oracle Corporation
 Java home: C:\Program Files (x86)\Java\jdk1.7.0_02\jre
 Default locale: zh_CN, platform encoding: GBK
 OS name: windows 7, version: 6.1, arch: x86, family: windows
Reporter: Changming Sun
   Original Estimate: 72h
  Remaining Estimate: 72h

 DefaultContainerExecutor cannot launch container under windows, because bash 
 cannot find the WRAPPER_LAUNCH_SCRIPT.
 Path wrapperScriptDst = new Path(containerWorkDir, WRAPPER_LAUNCH_SCRIPT);
  String[] command = {bash, -c,
   wrapperScriptDst.toUri().getPath().toString()};
   LOG.info(launchContainer:  + Arrays.toString(command));
 Suppose that  the value of 'wrapperScriptDst' is 
 C:\hadoop\default_container_executor.sh
 Then wrapperScriptDst.toUri().getPath().toString() will be 
 /C:/hadoop/default_container_executor.sh, which is a wrong path

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3829) [Gridmix] Gridmix should give better error message when input-data directory already exists and -generate option is given


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227443#comment-13227443
 ] 

Hudson commented on MAPREDUCE-3829:
---

Integrated in Hadoop-Mapreduce-trunk-Commit #1875 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1875/])
MAPREDUCE-3829. [Gridmix] Gridmix should give better error message when 
input data directory already exists and -generate option is given.(ravigummadi) 
(Revision 1299625)

 Result = ABORTED
ravigummadi : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1299625
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/DistributedCacheEmulator.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixExitCodes.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java


 [Gridmix] Gridmix should give better error message when input-data directory 
 already exists and -generate option is given
 -

 Key: MAPREDUCE-3829
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3829
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: contrib/gridmix
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
 Attachments: 3829.v0.patch, 3829.v1.3.patch, 3829.v1.patch, 
 3829.v2.patch


 Instead of throwing exception messages on to the console, Gridmix should give 
 better error message when input-data directory already exists and -generate 
 option is given.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3829) [Gridmix] Gridmix should give better error message when input-data directory already exists and -generate option is given

[
https://issues.apache.org/jira/browse/MAPREDUCE-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227444#comment-13227444
]

Hadoop QA commented on MAPREDUCE-3829:
--

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12518000/3829.v2.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 8 new or modified tests.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in .

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2036//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2036//console

This message is automatically generated.

[Gridmix] Gridmix should give better error message when input-data directory
already exists and -generate option is given
-

Key: MAPREDUCE-3829
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3829
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: contrib/gridmix
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
Attachments: 3829.v0.patch, 3829.v1.3.patch, 3829.v1.patch,
3829.v2.patch

Instead of throwing exception messages on to the console, Gridmix should give
better error message when input-data directory already exists and -generate
option is given.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3829) [Gridmix] Gridmix should give better error message when input-data directory already exists and -generate option is given


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227470#comment-13227470
 ] 

Hudson commented on MAPREDUCE-3829:
---

Integrated in Hadoop-Hdfs-trunk #982 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/982/])
MAPREDUCE-3829. [Gridmix] Gridmix should give better error message when 
input data directory already exists and -generate option is given.(ravigummadi) 
(Revision 1299625)

 Result = SUCCESS
ravigummadi : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1299625
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/DistributedCacheEmulator.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixExitCodes.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java


 [Gridmix] Gridmix should give better error message when input-data directory 
 already exists and -generate option is given
 -

 Key: MAPREDUCE-3829
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3829
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: contrib/gridmix
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
 Attachments: 3829.v0.patch, 3829.v1.3.patch, 3829.v1.patch, 
 3829.v2.patch


 Instead of throwing exception messages on to the console, Gridmix should give 
 better error message when input-data directory already exists and -generate 
 option is given.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3869) Distributed shell application fails with NoClassDefFoundError

2012-03-12 Thread Devaraj K (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj K updated MAPREDUCE-3869:
-

Attachment: MAPREDUCE-3869-1.patch

 Distributed shell application fails with NoClassDefFoundError
 -

 Key: MAPREDUCE-3869
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3869
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.1
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Blocker
 Fix For: 0.23.2

 Attachments: MAPREDUCE-3869-1.patch, MAPREDUCE-3869.patch


 Distributed shell application always fails to start the application master 
 with the following error.
 \\
 {code:xml}
 12/02/16 05:35:25 FATAL distributedshell.ApplicationMaster: Error running 
 ApplicationMaster
 java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/ipc/YarnRPC
   at 
 org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.init(ApplicationMaster.java:252)
   at 
 org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.main(ApplicationMaster.java:195)
 Caused by: java.lang.ClassNotFoundException: 
 org.apache.hadoop.yarn.ipc.YarnRPC
   at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
   at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
   ... 2 more
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3869) Distributed shell application fails with NoClassDefFoundError

2012-03-12 Thread Devaraj K (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj K updated MAPREDUCE-3869:
-

Status: Patch Available  (was: Open)

Thanks Hitesh and Vinod for looking into the patch.

I'm sorry for the late response.

I have updated the patch with the suggested change.

 Distributed shell application fails with NoClassDefFoundError
 -

 Key: MAPREDUCE-3869
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3869
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.1
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Blocker
 Fix For: 0.23.2

 Attachments: MAPREDUCE-3869-1.patch, MAPREDUCE-3869.patch


 Distributed shell application always fails to start the application master 
 with the following error.
 \\
 {code:xml}
 12/02/16 05:35:25 FATAL distributedshell.ApplicationMaster: Error running 
 ApplicationMaster
 java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/ipc/YarnRPC
   at 
 org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.init(ApplicationMaster.java:252)
   at 
 org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.main(ApplicationMaster.java:195)
 Caused by: java.lang.ClassNotFoundException: 
 org.apache.hadoop.yarn.ipc.YarnRPC
   at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
   at java.security.AccessController.doPrivileged(Native Method)
   at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
   at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
   at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
   ... 2 more
 {code}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3829) [Gridmix] Gridmix should give better error message when input-data directory already exists and -generate option is given


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227536#comment-13227536
 ] 

Hudson commented on MAPREDUCE-3829:
---

Integrated in Hadoop-Mapreduce-trunk #1017 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1017/])
MAPREDUCE-3829. [Gridmix] Gridmix should give better error message when 
input data directory already exists and -generate option is given.(ravigummadi) 
(Revision 1299625)

 Result = SUCCESS
ravigummadi : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1299625
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/DistributedCacheEmulator.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/java/org/apache/hadoop/mapred/gridmix/Gridmix.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestDistCacheEmulation.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixExitCodes.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/contrib/gridmix/src/test/org/apache/hadoop/mapred/gridmix/TestGridmixSubmission.java


 [Gridmix] Gridmix should give better error message when input-data directory 
 already exists and -generate option is given
 -

 Key: MAPREDUCE-3829
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3829
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: contrib/gridmix
Reporter: Ravi Gummadi
Assignee: Ravi Gummadi
 Attachments: 3829.v0.patch, 3829.v1.3.patch, 3829.v1.patch, 
 3829.v2.patch


 Instead of throwing exception messages on to the console, Gridmix should give 
 better error message when input-data directory already exists and -generate 
 option is given.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3869) Distributed shell application fails with NoClassDefFoundError

2012-03-12 Thread Uma Maheswara Rao G (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227574#comment-13227574
]

Hadoop QA commented on MAPREDUCE-3869:
--

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12518022/MAPREDUCE-3869-1.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

-1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in .

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2037//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2037//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-applications-distributedshell.html
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2037//console

This message is automatically generated.

Distributed shell application fails with NoClassDefFoundError
-

Key: MAPREDUCE-3869
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3869
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.1
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Blocker
Fix For: 0.23.2

Attachments: MAPREDUCE-3869-1.patch, MAPREDUCE-3869.patch

Distributed shell application always fails to start the application master
with the following error.
\\
{code:xml}
12/02/16 05:35:25 FATAL distributedshell.ApplicationMaster: Error running
ApplicationMaster
java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/ipc/YarnRPC
at
org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.init(ApplicationMaster.java:252)
at
org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster.main(ApplicationMaster.java:195)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.yarn.ipc.YarnRPC
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
... 2 more
{code}

[jira] [Updated] (MAPREDUCE-3996) zookeeper artifact is missing from the hadoop-dist assembly

2012-03-12 Thread Roman Shaposhnik (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roman Shaposhnik updated MAPREDUCE-3996:


Status: Patch Available  (was: Open)

 zookeeper artifact is missing from the hadoop-dist assembly
 ---

 Key: MAPREDUCE-3996
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3996
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.1
Reporter: Roman Shaposhnik
 Fix For: 0.23.2

 Attachments: MAPREDUCE-3996.patch.txt


 According to maven, zookeeper happens to be a dependency of 
 hadoop-yarn-server-common. Yet it is missing from the final distribution 
 assembly (and hence from the binary tarball)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3996) zookeeper artifact is missing from the hadoop-dist assembly

2012-03-12 Thread Roman Shaposhnik (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Roman Shaposhnik updated MAPREDUCE-3996:


Attachment: MAPREDUCE-3996.patch.txt

The alternative is to depend on hadoop-yarn-server-common which seems to be an 
overkill

 zookeeper artifact is missing from the hadoop-dist assembly
 ---

 Key: MAPREDUCE-3996
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3996
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.1
Reporter: Roman Shaposhnik
 Fix For: 0.23.2

 Attachments: MAPREDUCE-3996.patch.txt


 According to maven, zookeeper happens to be a dependency of 
 hadoop-yarn-server-common. Yet it is missing from the final distribution 
 assembly (and hence from the binary tarball)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3974) TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227641#comment-13227641
 ] 

Uma Maheswara Rao G commented on MAPREDUCE-3974:


Suresh, 0.23 build failing after HA changes merged to 23 branch.
Unfortunately I am not a committer for mapreduce project. Could you please 
merge this to 23 branch as well?

{quote}
[javac] Compiling 98 source files to 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Build/trunk/hadoop-mapreduce-project/build/test/mapred/classes
   [javac] 
/home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-0.23-Build/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSubmitJob.java:153:
 cannot find symbol
   [javac] symbol  : constructor 
ClientNamenodeProtocolTranslatorPB(java.net.InetSocketAddress,org.apache.hadoop.conf.Configuration,org.apache.hadoop.security.UserGroupInformation)
   [javac] location: class 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB
   [javac] return new 
ClientNamenodeProtocolTranslatorPB(NameNode.getAddress(conf),
   [javac]^
   [javac] Note: Some input files use or override a deprecated API.
   [javac] Note: Recompile with -Xlint:deprecation for details.
   [javac] 1 error

BUILD FAILED
{quote}

Thanks
Uma

 TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge
 

 Key: MAPREDUCE-3974
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3974
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.23.3
Reporter: Arun C Murthy
Assignee: Aaron T. Myers
Priority: Blocker
 Fix For: 0.24.0

 Attachments: MAPREDUCE-3974.patch


 TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge.
 'ant compile-tests' doesn't work (since it's ant for MR1).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3971) Job History web services need to have limits on the number of itmes they can return.

2012-03-12 Thread Robert Joseph Evans (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227643#comment-13227643
]

Robert Joseph Evans commented on MAPREDUCE-3971:

Bhallamudi,

I agree that there are a number of optimizations that can be made and that we
can guarantee ordering if we do them correctly. I am just not positive what
those optimizations should be yet. I made a proposal on MAPREDUCE-3973 to
possibly change the back end all together. If you have feedback on it I would
be very grateful to hear it. Depending on how that proposal is received the
way this JIRA is implemented could be very different.

Job History web services need to have limits on the number of itmes they can
return.

Key: MAPREDUCE-3971
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3971
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2
Affects Versions: 0.23.2
Reporter: Robert Joseph Evans

The Job History web services canput a very large load on the job history
server. We should put in a limit on the number of entries that can be
returned by the web service, and also add in the ability to modify the
starting location in the list, so that all entries can still be downlaoded.
Just not all at once.

[jira] [Updated] (MAPREDUCE-3971) Job History web services need to have limits on the number of items they can return.

2012-03-12 Thread Robert Joseph Evans (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Joseph Evans updated MAPREDUCE-3971:
---

Summary: Job History web services need to have limits on the number of 
items they can return.  (was: Job History web services need to have limits on 
the number of itmes they can return.)

 Job History web services need to have limits on the number of items they can 
 return.
 

 Key: MAPREDUCE-3971
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3971
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: mrv2
Affects Versions: 0.23.2
Reporter: Robert Joseph Evans

 The Job History web services canput a very large load on the job history 
 server.  We should put in a limit on the number of entries that can be 
 returned by the web service, and also add in the ability to modify the 
 starting location in the list, so that all entries can still be downlaoded.  
 Just not all at once.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-1109) ConcurrentModificationException in jobtracker.jsp

2012-03-12 Thread Tom White (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-1109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227652#comment-13227652
 ] 

Tom White commented on MAPREDUCE-1109:
--

+1

Can you run test-patch manually please?

 ConcurrentModificationException in jobtracker.jsp
 -

 Key: MAPREDUCE-1109
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1109
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobtracker
Affects Versions: 0.20.1
Reporter: dhruba borthakur
Assignee: dhruba borthakur
 Attachments: MAPREDUCE-1109.patch


 The jobtracker.jsp invoked methods tracker.runningJobs(), 
 tracker.completedJobs() and tracker.failedJobs() but these methods are not 
 synchronized and can cause ConcurrentModificationException if the JT is 
 concurrently changing the contents of these datastructures.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3729) Commit build failing TestJobClientGetJob, TestMRWithDistributedCache, TestLocalModeWithNewApis

2012-03-12 Thread Robert Joseph Evans (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227667#comment-13227667
]

Robert Joseph Evans commented on MAPREDUCE-3729:

I have been thinking about this and how we can fix the issue in the code. I am
a bit confused about how /tmp/hadoop-jenkins is created by a user that is not
jenkins? unless the other person explicitly set user.name to be jenkins, or
they manually created that directory before running. Vinod or Giri, did you
look at the directories before this happened?

If I don't know what caused the bad directory t begin with I really don't have
much hope of understanding how to prevent it in the future. If the directory
was manually created then there is nothing we can do. If the user explicitly
set user.name to jenkins we can detect that, and blow up.

We cannot delete the directory after a job finishes in local mode because more
jobs could still be running.

Commit build failing TestJobClientGetJob, TestMRWithDistributedCache,
TestLocalModeWithNewApis
--

Key: MAPREDUCE-3729
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3729
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, test
Affects Versions: 0.23.0
Reporter: Vinod Kumar Vavilapalli
Assignee: Robert Joseph Evans
Priority: Critical
Fix For: 0.23.1

See https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1671//testReport/.

[jira] [Resolved] (MAPREDUCE-3984) TestDBJob and TestDataDrivenDBInputFormat timeout

2012-03-12 Thread Thomas Graves (Resolved) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Graves resolved MAPREDUCE-3984.
--

Resolution: Duplicate

dup of MAPREDUCE-3621

 TestDBJob and TestDataDrivenDBInputFormat timeout
 -

 Key: MAPREDUCE-3984
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3984
 Project: Hadoop Map/Reduce
  Issue Type: Test
  Components: mrv2
Affects Versions: 0.23.2
Reporter: Robert Joseph Evans

 org.apache.hadoop.mapreduce.lib.db.TestDBJob.testRun and 
 org.apache.hadoop.mapreduce.lib.db.TestDataDrivenDBInputFormat.testDateSplits 
 bith timeout.  They look like they timeout trying to shutdown the DB that is 
 used as part of the testing.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3996) zookeeper artifact is missing from the hadoop-dist assembly

[
https://issues.apache.org/jira/browse/MAPREDUCE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227675#comment-13227675
]

Hadoop QA commented on MAPREDUCE-3996:
--

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12518036/MAPREDUCE-3996.patch.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in .

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2038//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2038//console

This message is automatically generated.

zookeeper artifact is missing from the hadoop-dist assembly
---

Key: MAPREDUCE-3996
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3996
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.1
Reporter: Roman Shaposhnik
Fix For: 0.23.2

Attachments: MAPREDUCE-3996.patch.txt

According to maven, zookeeper happens to be a dependency of
hadoop-yarn-server-common. Yet it is missing from the final distribution
assembly (and hence from the binary tarball)

[jira] [Commented] (MAPREDUCE-3996) zookeeper artifact is missing from the hadoop-dist assembly

2012-03-12 Thread Roman Shaposhnik (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227694#comment-13227694
 ] 

Roman Shaposhnik commented on MAPREDUCE-3996:
-

It seems that the HA merge that happened over the weekened now pulls ZK as a 
dependency for common and thus makes it available in the overall hadoop-dist 
assembly. At this point the big issue of not having ZK artifact there at all is 
solved. The lesser issue of making hadoop-mapreduce assembly self-contained is 
still a problem, but I'm not sure whether it is worth addressing. Feel free to 
disregard this patch if you think hadoop-mapreduce being self-contained is not 
a requirement.

 zookeeper artifact is missing from the hadoop-dist assembly
 ---

 Key: MAPREDUCE-3996
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3996
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.1
Reporter: Roman Shaposhnik
 Fix For: 0.23.2

 Attachments: MAPREDUCE-3996.patch.txt


 According to maven, zookeeper happens to be a dependency of 
 hadoop-yarn-server-common. Yet it is missing from the final distribution 
 assembly (and hence from the binary tarball)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3682) Tracker URL says AM tasks run on localhost


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash updated MAPREDUCE-3682:


Attachment: MAPREDUCE-3682.patch

Using NetUtils.getHostname() which returns hostname/ipaddress, and then 
splitting using / to get hostname
Simple enough change. Not including a test because its simple enough.

 Tracker URL says AM tasks run on localhost
 --

 Key: MAPREDUCE-3682
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3682
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.1
Reporter: David Capwell
Assignee: Hitesh Shah
 Attachments: MAPREDUCE-3682.patch


 If you look at the task page, it will show you the node the task ran on.  For 
 jobs that run in UberAM they point to http://localhost: and logs points 
 to http://localhost:/node/containerlogs/$container_id/
 This was run on a multi node cluster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (MAPREDUCE-3682) Tracker URL says AM tasks run on localhost

2012-03-12 Thread Ravi Prakash (Assigned) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash reassigned MAPREDUCE-3682:
---

Assignee: Ravi Prakash  (was: Hitesh Shah)

 Tracker URL says AM tasks run on localhost
 --

 Key: MAPREDUCE-3682
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3682
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.1
Reporter: David Capwell
Assignee: Ravi Prakash
 Attachments: MAPREDUCE-3682.patch


 If you look at the task page, it will show you the node the task ran on.  For 
 jobs that run in UberAM they point to http://localhost: and logs points 
 to http://localhost:/node/containerlogs/$container_id/
 This was run on a multi node cluster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3682) Tracker URL says AM tasks run on localhost

2012-03-12 Thread Hitesh Shah (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227721#comment-13227721
 ] 

Hitesh Shah commented on MAPREDUCE-3682:


@Ravi, thanks for taking this over. From an uber-AM point of view, the task 
logs need to point to the AM's logs. 

 Tracker URL says AM tasks run on localhost
 --

 Key: MAPREDUCE-3682
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3682
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.1
Reporter: David Capwell
Assignee: Ravi Prakash
 Attachments: MAPREDUCE-3682.patch


 If you look at the task page, it will show you the node the task ran on.  For 
 jobs that run in UberAM they point to http://localhost: and logs points 
 to http://localhost:/node/containerlogs/$container_id/
 This was run on a multi node cluster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3974) TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge

2012-03-12 Thread Thomas Graves (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Graves updated MAPREDUCE-3974:
-

   Resolution: Fixed
Fix Version/s: (was: 0.24.0)
   0.23.3
   Status: Resolved  (was: Patch Available)

I committed this to branch-0.23

 TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge
 

 Key: MAPREDUCE-3974
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3974
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.23.3
Reporter: Arun C Murthy
Assignee: Aaron T. Myers
Priority: Blocker
 Fix For: 0.23.3

 Attachments: MAPREDUCE-3974.patch


 TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge.
 'ant compile-tests' doesn't work (since it's ant for MR1).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3974) TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge

2012-03-12 Thread Aaron T. Myers (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227752#comment-13227752
 ] 

Aaron T. Myers commented on MAPREDUCE-3974:
---

Thanks a lot for taking care of this, Tom.

 TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge
 

 Key: MAPREDUCE-3974
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3974
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.23.3
Reporter: Arun C Murthy
Assignee: Aaron T. Myers
Priority: Blocker
 Fix For: 0.23.3

 Attachments: MAPREDUCE-3974.patch


 TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge.
 'ant compile-tests' doesn't work (since it's ant for MR1).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3974) TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227755#comment-13227755
 ] 

Hudson commented on MAPREDUCE-3974:
---

Integrated in Hadoop-Hdfs-0.23-Commit #664 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/664/])
merge  -r 1297661:1297662 from trunk to branch-0.23. FIXES: MAPREDUCE-3974 
(Revision 1299773)

 Result = SUCCESS
tgraves : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1299773
Files : 
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSubmitJob.java


 TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge
 

 Key: MAPREDUCE-3974
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3974
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.23.3
Reporter: Arun C Murthy
Assignee: Aaron T. Myers
Priority: Blocker
 Fix For: 0.23.3

 Attachments: MAPREDUCE-3974.patch


 TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge.
 'ant compile-tests' doesn't work (since it's ant for MR1).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3974) TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227756#comment-13227756
 ] 

Hudson commented on MAPREDUCE-3974:
---

Integrated in Hadoop-Hdfs-trunk-Commit #1942 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1942/])
MAPREDUCE-3974. Change to release 23.3 (Revision 1299775)

 Result = SUCCESS
tgraves : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1299775
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt


 TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge
 

 Key: MAPREDUCE-3974
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3974
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.23.3
Reporter: Arun C Murthy
Assignee: Aaron T. Myers
Priority: Blocker
 Fix For: 0.23.3

 Attachments: MAPREDUCE-3974.patch


 TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge.
 'ant compile-tests' doesn't work (since it's ant for MR1).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3974) TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227759#comment-13227759
 ] 

Hudson commented on MAPREDUCE-3974:
---

Integrated in Hadoop-Common-0.23-Commit #673 (See 
[https://builds.apache.org/job/Hadoop-Common-0.23-Commit/673/])
merge  -r 1297661:1297662 from trunk to branch-0.23. FIXES: MAPREDUCE-3974 
(Revision 1299773)

 Result = SUCCESS
tgraves : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1299773
Files : 
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSubmitJob.java


 TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge
 

 Key: MAPREDUCE-3974
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3974
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.23.3
Reporter: Arun C Murthy
Assignee: Aaron T. Myers
Priority: Blocker
 Fix For: 0.23.3

 Attachments: MAPREDUCE-3974.patch


 TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge.
 'ant compile-tests' doesn't work (since it's ant for MR1).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3974) TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227766#comment-13227766
 ] 

Hudson commented on MAPREDUCE-3974:
---

Integrated in Hadoop-Common-trunk-Commit #1867 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1867/])
MAPREDUCE-3974. Change to release 23.3 (Revision 1299775)

 Result = SUCCESS
tgraves : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1299775
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt


 TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge
 

 Key: MAPREDUCE-3974
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3974
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.23.3
Reporter: Arun C Murthy
Assignee: Aaron T. Myers
Priority: Blocker
 Fix For: 0.23.3

 Attachments: MAPREDUCE-3974.patch


 TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge.
 'ant compile-tests' doesn't work (since it's ant for MR1).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3974) TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227797#comment-13227797
 ] 

Hudson commented on MAPREDUCE-3974:
---

Integrated in Hadoop-Mapreduce-trunk-Commit #1876 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1876/])
MAPREDUCE-3974. Change to release 23.3 (Revision 1299775)

 Result = ABORTED
tgraves : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1299775
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt


 TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge
 

 Key: MAPREDUCE-3974
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3974
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.23.3
Reporter: Arun C Murthy
Assignee: Aaron T. Myers
Priority: Blocker
 Fix For: 0.23.3

 Attachments: MAPREDUCE-3974.patch


 TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge.
 'ant compile-tests' doesn't work (since it's ant for MR1).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3974) TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge

2012-03-12 Thread Robert Joseph Evans (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227798#comment-13227798
 ] 

Hudson commented on MAPREDUCE-3974:
---

Integrated in Hadoop-Mapreduce-0.23-Commit #681 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/681/])
merge  -r 1297661:1297662 from trunk to branch-0.23. FIXES: MAPREDUCE-3974 
(Revision 1299773)

 Result = ABORTED
tgraves : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1299773
Files : 
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapred/TestSubmitJob.java


 TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge
 

 Key: MAPREDUCE-3974
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3974
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.23.3
Reporter: Arun C Murthy
Assignee: Aaron T. Myers
Priority: Blocker
 Fix For: 0.23.3

 Attachments: MAPREDUCE-3974.patch


 TestSubmitJob in MR1 tests doesn't compile after HDFS-1623 merge.
 'ant compile-tests' doesn't work (since it's ant for MR1).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3928) App GUI Cluster-Applications UI has confusing job status report

2012-03-12 Thread Jason Lowe (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227856#comment-13227856
 ] 

Jason Lowe commented on MAPREDUCE-3928:
---

Yes, I think the confusion here is that a user is primarily interested in 
whether their job succeeded, failed, was killed, etc.  The filter links on the 
apps page implies one can quickly filter the apps to one of those states, yet 
apps that have failed or killed often don't show up under the FAILED or KILLED 
filters.  It's not clear on the web page that those only filter State instead 
of the FinalState, and many users may not understand or appreciate the 
distinction between State and FinalState (at least for MapReduce apps).

For example, it can be a bit confusing why a job that's killed via mapred job 
-kill shows up as KILLED / KILLED, while another job where the AM process is 
killed directly is listed as FINISHED / KILLED.



 App GUI Cluster-Applications UI has confusing job status report
 

 Key: MAPREDUCE-3928
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3928
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: applicationmaster
Affects Versions: 0.23.1
Reporter: patrick white

 The 0.23.1 Application GUI has some potential usability issues and confusion
 points with respect to job status, from a user's perspective. 
 Currently, when starting from the App UI base link of
 http://RM_HOST:8088/cluster, the main window shows All Applications, and 
 in
 the upper left corner is the Cluster pulldown, opened up showing the
 following list:
 About
 Nodes
 Applications
 - New
 - Submitted
 - Accepted
 - Running
 - Finished
 - Failed
 - Killed
 Scheduler
 When jobs are submitted, they show up in the pre-final states, like Accepted
 and Running. however after completion, no matter on the final outcome of a job
 (SUCCESS, KILLED or FAILED), all jobs are listed under FINISHED, this is okay
 since any completion result qualifies as Finished, but the FAILED and KILLED
 jobs do not appear in the corresponding links for FAILED or KILLED. 
 Clicking on FAILED or KILLED reports No data available in table, even with
 Failed or Killed jobs showing up in the FINISHED link. This can be very
 confusing to a user checking on their jobs, especially for someone using the
 direct URL links, such as:
 http://RM_HOST:8088/cluster/apps/FAILED
 http://RM_HOST:8088/cluster/apps/KILLED
 Another potential issue is that the RM and AM each have their own
 interpretation of a job's result, so the State and FinalStatus reported in the
 Cluster Metrics display may not align with the defined states in the Cluster
 pulldown. It would be useful to clearly delineate areas of the GUI wrt the
 component visible states of a user's job.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-3999) Tracking link gives an error if the AppMaster hasn't started yet

2012-03-12 Thread Ravi Prakash (Created) (JIRA)

Tracking link gives an error if the AppMaster hasn't started yet


 Key: MAPREDUCE-3999
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3999
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, webapps
Affects Versions: 0.23.1
Reporter: Ravi Prakash
Assignee: Ravi Prakash


Courtesy [~sseth]
{quote}
The MRAppMaster died before writing anything.

Steps to generate the error:
1. Setup a queue with 1 max active application per user
2. Submit a long running job to this queue.
3. Submit another job to the queue as the same user. Access the tracking URL
for job 2 directly or via Oozie (not via the RM link - which is rewritten once
the app starts).

This would exist in situations where the queue doesn't have enough capacity -
or for the small period of time between app submission and AM start.
{quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3999) Tracking link gives an error if the AppMaster hasn't started yet

[
https://issues.apache.org/jira/browse/MAPREDUCE-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227876#comment-13227876
]

Robert Joseph Evans commented on MAPREDUCE-3999:

If the proxy can tell from the state of the application that the AM has not
launched yet, we should return an appropriate page. Either way the error the
MRAppMaster died before writing anything does not seem to be an appropriate
error message in all cases that it is being returned for.

Tracking link gives an error if the AppMaster hasn't started yet

Key: MAPREDUCE-3999
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3999
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, webapps
Affects Versions: 0.23.1
Reporter: Ravi Prakash
Assignee: Ravi Prakash

Courtesy [~sseth]
{quote}
The MRAppMaster died before writing anything.
Steps to generate the error:
1. Setup a queue with 1 max active application per user
2. Submit a long running job to this queue.
3. Submit another job to the queue as the same user. Access the tracking URL
for job 2 directly or via Oozie (not via the RM link - which is rewritten once
the app starts).
This would exist in situations where the queue doesn't have enough capacity -
or for the small period of time between app submission and AM start.
{quote}

[jira] [Updated] (MAPREDUCE-3999) Tracking link gives an error if the AppMaster hasn't started yet


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash updated MAPREDUCE-3999:


Status: Patch Available  (was: Open)

 Tracking link gives an error if the AppMaster hasn't started yet
 

 Key: MAPREDUCE-3999
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3999
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, webapps
Affects Versions: 0.23.1
Reporter: Ravi Prakash
Assignee: Ravi Prakash
 Attachments: MAPREDUCE-3999.patch


 Courtesy [~sseth]
 {quote}
 The MRAppMaster died before writing anything.
 Steps to generate the error:
 1. Setup a queue with 1 max active application per user
 2. Submit a long running job to this queue.
 3. Submit another job to the queue as the same user. Access the tracking URL
 for job 2 directly or via Oozie (not via the RM link - which is rewritten once
 the app starts).
 This would exist in situations where the queue doesn't have enough capacity -
 or for the small period of time between app submission and AM start.
 {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3999) Tracking link gives an error if the AppMaster hasn't started yet


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash updated MAPREDUCE-3999:


Attachment: MAPREDUCE-3999.patch

This fixes the issue. Now I am checking the FinalApplicationStatus too to 
figure out if the AM has completed / hasn't started

 Tracking link gives an error if the AppMaster hasn't started yet
 

 Key: MAPREDUCE-3999
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3999
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, webapps
Affects Versions: 0.23.1
Reporter: Ravi Prakash
Assignee: Ravi Prakash
 Attachments: MAPREDUCE-3999.patch


 Courtesy [~sseth]
 {quote}
 The MRAppMaster died before writing anything.
 Steps to generate the error:
 1. Setup a queue with 1 max active application per user
 2. Submit a long running job to this queue.
 3. Submit another job to the queue as the same user. Access the tracking URL
 for job 2 directly or via Oozie (not via the RM link - which is rewritten once
 the app starts).
 This would exist in situations where the queue doesn't have enough capacity -
 or for the small period of time between app submission and AM start.
 {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3999) Tracking link gives an error if the AppMaster hasn't started yet

2012-03-12 Thread Ravi Prakash (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227884#comment-13227884
 ] 

Ravi Prakash commented on MAPREDUCE-3999:
-

@Bobby: I'm still using the 404 response which I think is reasonable. Please 
let me know if that isn't ideal.

 Tracking link gives an error if the AppMaster hasn't started yet
 

 Key: MAPREDUCE-3999
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3999
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, webapps
Affects Versions: 0.23.1
Reporter: Ravi Prakash
Assignee: Ravi Prakash
 Attachments: MAPREDUCE-3999.patch


 Courtesy [~sseth]
 {quote}
 The MRAppMaster died before writing anything.
 Steps to generate the error:
 1. Setup a queue with 1 max active application per user
 2. Submit a long running job to this queue.
 3. Submit another job to the queue as the same user. Access the tracking URL
 for job 2 directly or via Oozie (not via the RM link - which is rewritten once
 the app starts).
 This would exist in situations where the queue doesn't have enough capacity -
 or for the small period of time between app submission and AM start.
 {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3992) Reduce fetcher doesn't verify HTTP status code of response

2012-03-12 Thread Todd Lipcon (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated MAPREDUCE-3992:
---

Attachment: mr-3992.txt

Simple patch against trunk. I haven't tried this on an MR2 cluster, but I 
tested the equivalent 0.20-based patch. Unfortunately it's difficult to 
engineer a unit test case for this.

I'll upload a patch for branch-1 as well if this is accepted.

 Reduce fetcher doesn't verify HTTP status code of response
 --

 Key: MAPREDUCE-3992
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3992
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv1
Affects Versions: 0.23.1, 0.24.0, 1.0.1
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Attachments: mr-3992.txt


 Currently, the reduce fetch code doesn't check the HTTP status code of the 
 response. This can lead to the following situation:
 - the map output servlet gets an IOException after setting the headers but 
 before the first call to flush()
 - this causes it to send a response with a non-OK result code, including the 
 exception text as the response body (response.sendError() does this if the 
 response isn't committed)
 - it will still include the response headers indicating it's a valid response
 In the case of a merge-to-memory, the compression codec might then try to 
 interpret the HTML response as compressed data, resulting in either a huge 
 allocation (OOME) or some other nasty error. This bug seems to be present in 
 MR1, but haven't checked trunk/MR2 yet.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3992) Reduce fetcher doesn't verify HTTP status code of response

2012-03-12 Thread Todd Lipcon (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Todd Lipcon updated MAPREDUCE-3992:
---

 Target Version/s: 0.24.0, 1.0.2, 0.23.3  (was: 1.0.2)
Affects Version/s: 0.24.0
   0.23.1
   Status: Patch Available  (was: Open)

 Reduce fetcher doesn't verify HTTP status code of response
 --

 Key: MAPREDUCE-3992
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3992
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv1
Affects Versions: 1.0.1, 0.23.1, 0.24.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
 Attachments: mr-3992.txt


 Currently, the reduce fetch code doesn't check the HTTP status code of the 
 response. This can lead to the following situation:
 - the map output servlet gets an IOException after setting the headers but 
 before the first call to flush()
 - this causes it to send a response with a non-OK result code, including the 
 exception text as the response body (response.sendError() does this if the 
 response isn't committed)
 - it will still include the response headers indicating it's a valid response
 In the case of a merge-to-memory, the compression codec might then try to 
 interpret the HTML response as compressed data, resulting in either a huge 
 allocation (OOME) or some other nasty error. This bug seems to be present in 
 MR1, but haven't checked trunk/MR2 yet.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3999) Tracking link gives an error if the AppMaster hasn't started yet


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash updated MAPREDUCE-3999:


Status: Open  (was: Patch Available)

 Tracking link gives an error if the AppMaster hasn't started yet
 

 Key: MAPREDUCE-3999
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3999
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, webapps
Affects Versions: 0.23.1
Reporter: Ravi Prakash
Assignee: Ravi Prakash
 Attachments: MAPREDUCE-3999.patch


 Courtesy [~sseth]
 {quote}
 The MRAppMaster died before writing anything.
 Steps to generate the error:
 1. Setup a queue with 1 max active application per user
 2. Submit a long running job to this queue.
 3. Submit another job to the queue as the same user. Access the tracking URL
 for job 2 directly or via Oozie (not via the RM link - which is rewritten once
 the app starts).
 This would exist in situations where the queue doesn't have enough capacity -
 or for the small period of time between app submission and AM start.
 {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3999) Tracking link gives an error if the AppMaster hasn't started yet


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash updated MAPREDUCE-3999:


Attachment: MAPREDUCE-3999.patch

Removing a stray logging statement =D

 Tracking link gives an error if the AppMaster hasn't started yet
 

 Key: MAPREDUCE-3999
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3999
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, webapps
Affects Versions: 0.23.1
Reporter: Ravi Prakash
Assignee: Ravi Prakash
 Attachments: MAPREDUCE-3999.patch, MAPREDUCE-3999.patch


 Courtesy [~sseth]
 {quote}
 The MRAppMaster died before writing anything.
 Steps to generate the error:
 1. Setup a queue with 1 max active application per user
 2. Submit a long running job to this queue.
 3. Submit another job to the queue as the same user. Access the tracking URL
 for job 2 directly or via Oozie (not via the RM link - which is rewritten once
 the app starts).
 This would exist in situations where the queue doesn't have enough capacity -
 or for the small period of time between app submission and AM start.
 {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3999) Tracking link gives an error if the AppMaster hasn't started yet


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash updated MAPREDUCE-3999:


Status: Patch Available  (was: Open)

 Tracking link gives an error if the AppMaster hasn't started yet
 

 Key: MAPREDUCE-3999
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3999
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, webapps
Affects Versions: 0.23.1
Reporter: Ravi Prakash
Assignee: Ravi Prakash
 Attachments: MAPREDUCE-3999.patch, MAPREDUCE-3999.patch


 Courtesy [~sseth]
 {quote}
 The MRAppMaster died before writing anything.
 Steps to generate the error:
 1. Setup a queue with 1 max active application per user
 2. Submit a long running job to this queue.
 3. Submit another job to the queue as the same user. Access the tracking URL
 for job 2 directly or via Oozie (not via the RM link - which is rewritten once
 the app starts).
 This would exist in situations where the queue doesn't have enough capacity -
 or for the small period of time between app submission and AM start.
 {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3995) Add support for static service lifecycle listeners .

2012-03-12 Thread Robert Joseph Evans (Commented) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Loughran updated MAPREDUCE-3995:
--

Status: Open  (was: Patch Available)

will rework and resub once MR-3970 is committed

 Add support for  static service lifecycle listeners .
 -

 Key: MAPREDUCE-3995
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3995
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: mrv2
Affects Versions: 0.23.1, 0.24.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
 Attachments: MAPREDUCE-3995.patch, MAPREDUCE-3995.patch, 
 MAPREDUCE-3995.patch, MAPREDUCE-3995.patch


 Add support to {{AbstractService}} that allow callers to register listeners 
 for all instances. The existing listener interface could be used. This allows 
 management tools to hook into the events.
 The static listeners would be invoked for all state changes except creation 
 (base class shouldn't be handing out references to itself at this point).
 These static events could all be async, pushed through a shared 
 {{ConcurrentLinkedQueue}}; failures logged at warn and the rest of the 
 listeners invoked.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3999) Tracking link gives an error if the AppMaster hasn't started yet


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13227938#comment-13227938
 ] 

Robert Joseph Evans commented on MAPREDUCE-3999:


A few comments:

* In the patch if the path is N/A and the status is SUCCEEDED the proxy will 
still try to load N/A.
* For some reason the proxy which is part of yarn, and not MR is returning an 
error message that is MR specific.  

Because we will always return a 404 error in this case, I would prefer to see 
something like

{code}
if(applicationReport.getOriginalTrackingUrl().equals(N/A)) {
String message;
switch(applicationReport.getFinalApplicationStatus()) {
case FAILED:
case KILLED:
case SUCCEEDED:
message = The requested application exited before setting a 
tracking URL.;
break;
case UNDEFINED:
message = The requested application does not appear to be running 
yet, and has not set a tracking URL.;
break;
default:
//This should never happen, but just to be safe
message = The requested application has not set a tracking URL.;
break;
}
noFound(resp, message);
return;
}
{code}

If you would prefer to throw an exception in the default case that would be 
good too. 

 Tracking link gives an error if the AppMaster hasn't started yet
 

 Key: MAPREDUCE-3999
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3999
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, webapps
Affects Versions: 0.23.1
Reporter: Ravi Prakash
Assignee: Ravi Prakash
 Attachments: MAPREDUCE-3999.patch, MAPREDUCE-3999.patch


 Courtesy [~sseth]
 {quote}
 The MRAppMaster died before writing anything.
 Steps to generate the error:
 1. Setup a queue with 1 max active application per user
 2. Submit a long running job to this queue.
 3. Submit another job to the queue as the same user. Access the tracking URL
 for job 2 directly or via Oozie (not via the RM link - which is rewritten once
 the app starts).
 This would exist in situations where the queue doesn't have enough capacity -
 or for the small period of time between app submission and AM start.
 {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3999) Tracking link gives an error if the AppMaster hasn't started yet

2012-03-12 Thread Robert Joseph Evans (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Joseph Evans updated MAPREDUCE-3999:
---

Status: Open  (was: Patch Available)

 Tracking link gives an error if the AppMaster hasn't started yet
 

 Key: MAPREDUCE-3999
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3999
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, webapps
Affects Versions: 0.23.1
Reporter: Ravi Prakash
Assignee: Ravi Prakash
 Attachments: MAPREDUCE-3999.patch, MAPREDUCE-3999.patch


 Courtesy [~sseth]
 {quote}
 The MRAppMaster died before writing anything.
 Steps to generate the error:
 1. Setup a queue with 1 max active application per user
 2. Submit a long running job to this queue.
 3. Submit another job to the queue as the same user. Access the tracking URL
 for job 2 directly or via Oozie (not via the RM link - which is rewritten once
 the app starts).
 This would exist in situations where the queue doesn't have enough capacity -
 or for the small period of time between app submission and AM start.
 {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3672) Killed maps shouldn't be counted towards JobCounter.NUM_FAILED_MAPS

2012-03-12 Thread Anupam Seth (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anupam Seth updated MAPREDUCE-3672:
---

Attachment: MAPREDUCE-3672-branch-23.patch

Attaching patch, kindly review and let me know of any comments!

 Killed maps shouldn't be counted towards JobCounter.NUM_FAILED_MAPS
 ---

 Key: MAPREDUCE-3672
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3672
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mr-am, mrv2
Affects Versions: 0.23.0
Reporter: Vinod Kumar Vavilapalli
Assignee: Anupam Seth
 Fix For: 0.24.0

 Attachments: MAPREDUCE-3672-branch-23.patch


 We count maps that are killed, say by speculator, towards 
 JobCounter.NUM_FAILED_MAPS. We should instead have a separate JobCounter for 
 killed maps.
 Same with reduces too.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3999) Tracking link gives an error if the AppMaster hasn't started yet

[
https://issues.apache.org/jira/browse/MAPREDUCE-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Ravi Prakash updated MAPREDUCE-3999:

Attachment: MAPREDUCE-3999.patch

Thanks for the review and comments Bobby!

I haven't included an exception. Didn't seem it was warranted.

Tracking link gives an error if the AppMaster hasn't started yet

[jira] [Updated] (MAPREDUCE-3999) Tracking link gives an error if the AppMaster hasn't started yet


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ravi Prakash updated MAPREDUCE-3999:


Status: Patch Available  (was: Open)

 Tracking link gives an error if the AppMaster hasn't started yet
 

 Key: MAPREDUCE-3999
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3999
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, webapps
Affects Versions: 0.23.1
Reporter: Ravi Prakash
Assignee: Ravi Prakash
 Attachments: MAPREDUCE-3999.patch, MAPREDUCE-3999.patch, 
 MAPREDUCE-3999.patch


 Courtesy [~sseth]
 {quote}
 The MRAppMaster died before writing anything.
 Steps to generate the error:
 1. Setup a queue with 1 max active application per user
 2. Submit a long running job to this queue.
 3. Submit another job to the queue as the same user. Access the tracking URL
 for job 2 directly or via Oozie (not via the RM link - which is rewritten once
 the app starts).
 This would exist in situations where the queue doesn't have enough capacity -
 or for the small period of time between app submission and AM start.
 {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3992) Reduce fetcher doesn't verify HTTP status code of response

[
https://issues.apache.org/jira/browse/MAPREDUCE-3992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13228004#comment-13228004
]

Hadoop QA commented on MAPREDUCE-3992:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12518085/mr-3992.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in .

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2039//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2039//console

This message is automatically generated.

Reduce fetcher doesn't verify HTTP status code of response
--

Key: MAPREDUCE-3992
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3992
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1
Affects Versions: 0.23.1, 0.24.0, 1.0.1
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Attachments: mr-3992.txt

Currently, the reduce fetch code doesn't check the HTTP status code of the
response. This can lead to the following situation:
- the map output servlet gets an IOException after setting the headers but
before the first call to flush()
- this causes it to send a response with a non-OK result code, including the
exception text as the response body (response.sendError() does this if the
response isn't committed)
- it will still include the response headers indicating it's a valid response
In the case of a merge-to-memory, the compression codec might then try to
interpret the HTML response as compressed data, resulting in either a huge
allocation (OOME) or some other nasty error. This bug seems to be present in
MR1, but haven't checked trunk/MR2 yet.

[jira] [Commented] (MAPREDUCE-3999) Tracking link gives an error if the AppMaster hasn't started yet

[
https://issues.apache.org/jira/browse/MAPREDUCE-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13228003#comment-13228003
]

Hadoop QA commented on MAPREDUCE-3999:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12518087/MAPREDUCE-3999.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in .

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2040//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2040//console

This message is automatically generated.

Tracking link gives an error if the AppMaster hasn't started yet

[jira] [Commented] (MAPREDUCE-3837) Hadoop 22 Job tracker is not able to recover job in case of crash and after that no user can submit job.

2012-03-12 Thread Arun C Murthy (Commented) (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-3837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13228016#comment-13228016
]

Arun C Murthy commented on MAPREDUCE-3837:
--

Apologies for the late response, I missed this.

Thanks for the clarification Mayank, Tucu Konst. I agree it's much more
palatable without all the complexities of trying to recover jobs from
point-of-crash.

Couple of questions:
a) How does it work in a secure setting?
b) We should at least add some docs on this feature.

Makes sense?

Hadoop 22 Job tracker is not able to recover job in case of crash and after
that no user can submit job.

Key: MAPREDUCE-3837
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3837
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.22.0
Reporter: Mayank Bansal
Assignee: Mayank Bansal
Fix For: 0.24.0, 0.22.1, 0.23.2

Attachments: PATCH-HADOOP-1-MAPREDUCE-3837-1.patch,
PATCH-HADOOP-1-MAPREDUCE-3837-2.patch, PATCH-HADOOP-1-MAPREDUCE-3837.patch,
PATCH-MAPREDUCE-3837.patch, PATCH-TRUNK-MAPREDUCE-3837.patch

If job tracker is crashed while running , and there were some jobs are
running , so if job tracker's property mapreduce.jobtracker.restart.recover
is true then it should recover the job.
However the current behavior is as follows
jobtracker try to restore the jobs but it can not . And after that jobtracker
closes its handle to hdfs and nobody else can submit job.
Thanks,
Mayank

[jira] [Commented] (MAPREDUCE-3942) Randomize master key generation for ApplicationTokenSecretManager and roll it every so often

2012-03-12 Thread Siddharth Seth (Commented) (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13228029#comment-13228029
 ] 

Siddharth Seth commented on MAPREDUCE-3942:
---

The patch looks good. It needs to be upmerged though. 
Is the additional synchronization in ApplicationTokenIdentifier really required 
? Doesn't look like any of the synchronized methods are called concurrently.

 Randomize master key generation for ApplicationTokenSecretManager and roll it 
 every so often
 

 Key: MAPREDUCE-3942
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3942
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: mrv2, security
Affects Versions: 0.23.0
Reporter: Vinod Kumar Vavilapalli
Assignee: Vinod Kumar Vavilapalli
 Fix For: 0.23.2

 Attachments: MAPREDUCE-3942-20120307.1.txt, 
 MAPREDUCE-3942-20120307.txt


  - Master key for authentication of AMs need to be automatically generated.
  - The key needs to be rolled every so often but AMs with old keys should 
 continue to be able to talk to the RM.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3999) Tracking link gives an error if the AppMaster hasn't started yet

[
https://issues.apache.org/jira/browse/MAPREDUCE-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13228070#comment-13228070
]

Hadoop QA commented on MAPREDUCE-3999:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12518106/MAPREDUCE-3999.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in .

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2041//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2041//console

This message is automatically generated.

Tracking link gives an error if the AppMaster hasn't started yet

[jira] [Updated] (MAPREDUCE-3988) mapreduce.job.local.dir doesn't point to a single directory on a node.

2012-03-12 Thread Eric Payne (Updated) (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Payne updated MAPREDUCE-3988:
--

Status: Patch Available  (was: Open)

 mapreduce.job.local.dir doesn't point to a single directory on a node.
 --

 Key: MAPREDUCE-3988
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3988
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 0.23.0
Reporter: Vinod Kumar Vavilapalli
Assignee: Eric Payne
 Attachments: MAPREDUCE-3988-1.txt


 After MAPREDUCE-3975, mapreduce.job.local.dir is set correctly for the tasks 
 but it doesn't point to the same directory for all tasks running on the node.
 It is a public API. Either we should point to a single directory or point it 
 to all directories and change the documentation to say that it points to all 
 dirs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-3988) mapreduce.job.local.dir doesn't point to a single directory on a node.

2012-03-12 Thread Eric Payne (Updated) (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-3988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Eric Payne updated MAPREDUCE-3988:
--

Attachment: MAPREDUCE-3988-1.txt

This patch uses the LocalDirAllocator.getLocalPathToRead() API to first check
for existence of a scratch directory before using the
LocalDirAllocator.getLocalPathForWrite() API if the directory does not exist.

There are a couple of race conditions with this implementation because of the
fact that each YarnChild task attempt is running in its own JVM. This patch
addresses one of the race conditions.

The race addressed by this patch is when 2 or more tasks at the same time
detect there is no mapreduce.job.local.dir and try to create it in the same
directory at the same time. Whichever one loses the race will get a
FileAlreadyExistsException, which is ignored.

There is still a race condition wherein 2 tasks running on the same node could
both see that there is no mapreduce.job.local.dir and then try to create it in
2 different directories; for eg, one in /grid/0 and one in /grid/1.

We could go a step further in addressing the second race and try to hash the
attempt ID and map it to the same directory, but that seems overly complicated.

mapreduce.job.local.dir doesn't point to a single directory on a node.
--

Key: MAPREDUCE-3988
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3988
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Vinod Kumar Vavilapalli
Assignee: Eric Payne
Attachments: MAPREDUCE-3988-1.txt

After MAPREDUCE-3975, mapreduce.job.local.dir is set correctly for the tasks
but it doesn't point to the same directory for all tasks running on the node.
It is a public API. Either we should point to a single directory or point it
to all directories and change the documentation to say that it points to all
dirs.

[jira] [Commented] (MAPREDUCE-3988) mapreduce.job.local.dir doesn't point to a single directory on a node.