date:20120712


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Radwan updated MAPREDUCE-4422:


Attachment: MAPREDUCE-4422.patch

 YARN_APPLICATION_CLASSPATH needs a documented default value in 
 YarnConfiguration
 

 Key: MAPREDUCE-4422
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4422
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Arun C Murthy
Assignee: Ahmed Radwan
 Attachments: MAPREDUCE-4422.patch


 MAPREDUCE-3505 allowed YARN_APPLICATION_CLASSPATH to be configurable.
 However, we didn't add a default value to YarnConfiguration, as-is the norm.
 Ran into it while investigating MAPREDUCE-4421.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4431) killing already completed application gives ambiguous message as Application has been killed

2012-07-12 Thread Nishan Shetty (JIRA)

Nishan Shetty created MAPREDUCE-4431:


 Summary: killing already completed application gives ambiguous 
message as Application has been killed
 Key: MAPREDUCE-4431
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4431
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Reporter: Nishan Shetty
Priority: Minor


If we try to kill the already completed application by the following command it 
gives ambiguous message as Application has been killed

./mapred job -kill already completed application id


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4422) YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13412633#comment-13412633
 ] 

Ahmed Radwan commented on MAPREDUCE-4422:
-

Here is the patch adding the default value (from yarn-default.xml) to 
YarnConfiguration as well. Sorry, I missed adding it while working on 
MAPREDUCE-3505.

 YARN_APPLICATION_CLASSPATH needs a documented default value in 
 YarnConfiguration
 

 Key: MAPREDUCE-4422
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4422
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Arun C Murthy
Assignee: Ahmed Radwan
 Attachments: MAPREDUCE-4422.patch


 MAPREDUCE-3505 allowed YARN_APPLICATION_CLASSPATH to be configurable.
 However, we didn't add a default value to YarnConfiguration, as-is the norm.
 Ran into it while investigating MAPREDUCE-4421.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4422) YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Radwan updated MAPREDUCE-4422:


Status: Patch Available  (was: Open)

 YARN_APPLICATION_CLASSPATH needs a documented default value in 
 YarnConfiguration
 

 Key: MAPREDUCE-4422
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4422
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Arun C Murthy
Assignee: Ahmed Radwan
 Attachments: MAPREDUCE-4422.patch


 MAPREDUCE-3505 allowed YARN_APPLICATION_CLASSPATH to be configurable.
 However, we didn't add a default value to YarnConfiguration, as-is the norm.
 Ran into it while investigating MAPREDUCE-4421.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Assigned] (MAPREDUCE-4420) ./mapred queue -info queuename -showJobs displays containers and memory as zero always


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj K reassigned MAPREDUCE-4420:


Assignee: Devaraj K

 ./mapred queue -info queuename -showJobs displays containers and memory as 
 zero always
 

 Key: MAPREDUCE-4420
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4420
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 2.0.0-alpha
Reporter: Nishan Shetty
Assignee: Devaraj K
 Attachments: screenshot-1.jpg


 ./mapred queue -info queuename -showJobs displays containers and memory as 
 zero always.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4431) killing already completed job gives ambiguous message as Killed job job id

2012-07-12 Thread Nishan Shetty (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nishan Shetty updated MAPREDUCE-4431:
-

Description: 
If we try to kill the already completed job by the following command it gives 
ambiguous message as Killed job job id

./mapred job -kill already completed job id


  was:
If we try to kill the already completed application by the following command it 
gives ambiguous message as Application has been killed

./mapred job -kill already completed application id


Summary: killing already completed job gives ambiguous message as 
Killed job job id  (was: killing already completed application gives 
ambiguous message as Application has been killed)

 killing already completed job gives ambiguous message as Killed job job id
 --

 Key: MAPREDUCE-4431
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4431
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Reporter: Nishan Shetty
Priority: Minor

 If we try to kill the already completed job by the following command it gives 
 ambiguous message as Killed job job id
 ./mapred job -kill already completed job id

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4419) ./mapred queue -info queuename -showJobs displays all the jobs irrespective of queuename


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj K updated MAPREDUCE-4419:
-

Attachment: MAPREDUCE-4419.patch

 ./mapred queue -info queuename -showJobs displays all the jobs irrespective 
 of queuename 
 -

 Key: MAPREDUCE-4419
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4419
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Nishan Shetty
Assignee: Devaraj K
 Attachments: MAPREDUCE-4419.patch, screenshot-1.jpg, screenshot-2.jpg


 ./mapred queue -info queuename -showJobs shows all the jobs irrespective of 
 queuename
 In Queue name field all the jobs are showing as default queue but they are 
 submitted to the configured queue(see screenshots attached).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4419) ./mapred queue -info queuename -showJobs displays all the jobs irrespective of queuename


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj K updated MAPREDUCE-4419:
-

Affects Version/s: 3.0.0
   Status: Patch Available  (was: Open)

Attaching the patch to fix this issue.

 ./mapred queue -info queuename -showJobs displays all the jobs irrespective 
 of queuename 
 -

 Key: MAPREDUCE-4419
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4419
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Nishan Shetty
Assignee: Devaraj K
 Attachments: MAPREDUCE-4419.patch, screenshot-1.jpg, screenshot-2.jpg


 ./mapred queue -info queuename -showJobs shows all the jobs irrespective of 
 queuename
 In Queue name field all the jobs are showing as default queue but they are 
 submitted to the configured queue(see screenshots attached).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4419) ./mapred queue -info queuename -showJobs displays all the jobs irrespective of queuename


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj K updated MAPREDUCE-4419:
-

Target Version/s: 2.0.1-alpha, 3.0.0

 ./mapred queue -info queuename -showJobs displays all the jobs irrespective 
 of queuename 
 -

 Key: MAPREDUCE-4419
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4419
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Nishan Shetty
Assignee: Devaraj K
 Attachments: MAPREDUCE-4419.patch, screenshot-1.jpg, screenshot-2.jpg


 ./mapred queue -info queuename -showJobs shows all the jobs irrespective of 
 queuename
 In Queue name field all the jobs are showing as default queue but they are 
 submitted to the configured queue(see screenshots attached).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4431) killing already completed job gives ambiguous message as Killed job job id

2012-07-12 Thread Nishan Shetty (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13412635#comment-13412635
 ] 

Nishan Shetty commented on MAPREDUCE-4431:
--

{noformat}
12/07/12 14:27:35 INFO mapred.ClientServiceDelegate: Application state is 
completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
12/07/12 14:27:35 INFO mapred.ResourceMgrDelegate: Killing application 
application_1342080430464_0001
Killed job job_1342080430464_0001
{noformat}

 killing already completed job gives ambiguous message as Killed job job id
 --

 Key: MAPREDUCE-4431
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4431
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2
Reporter: Nishan Shetty
Priority: Minor

 If we try to kill the already completed job by the following command it gives 
 ambiguous message as Killed job job id
 ./mapred job -kill already completed job id

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4422) YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration

[
https://issues.apache.org/jira/browse/MAPREDUCE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13412644#comment-13412644
]

Hadoop QA commented on MAPREDUCE-4422:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12536196/MAPREDUCE-4422.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common.

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2578//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2578//console

This message is automatically generated.

YARN_APPLICATION_CLASSPATH needs a documented default value in
YarnConfiguration

Key: MAPREDUCE-4422
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4422
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Arun C Murthy
Assignee: Ahmed Radwan
Attachments: MAPREDUCE-4422.patch

MAPREDUCE-3505 allowed YARN_APPLICATION_CLASSPATH to be configurable.
However, we didn't add a default value to YarnConfiguration, as-is the norm.
Ran into it while investigating MAPREDUCE-4421.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4419) ./mapred queue -info queuename -showJobs displays all the jobs irrespective of queuename

[
https://issues.apache.org/jira/browse/MAPREDUCE-4419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13412647#comment-13412647
]

Hadoop QA commented on MAPREDUCE-4419:
--

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12536197/MAPREDUCE-4419.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test
files.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2577//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2577//console

This message is automatically generated.

./mapred queue -info queuename -showJobs displays all the jobs irrespective
of queuename
-

Key: MAPREDUCE-4419
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4419
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Nishan Shetty
Assignee: Devaraj K
Attachments: MAPREDUCE-4419.patch, screenshot-1.jpg, screenshot-2.jpg

./mapred queue -info queuename -showJobs shows all the jobs irrespective of
queuename
In Queue name field all the jobs are showing as default queue but they are
submitted to the configured queue(see screenshots attached).

[jira] [Updated] (MAPREDUCE-4380) Empty Userlogs directory is getting created under logs directory


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Devaraj K updated MAPREDUCE-4380:
-

Target Version/s: 2.0.1-alpha, 3.0.0

 Empty Userlogs directory is getting created under logs directory
 

 Key: MAPREDUCE-4380
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4380
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: mrv2, nodemanager
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Priority: Minor
 Attachments: MAPREDUCE-4380.patch


 Empty Userlogs directory is getting created under logs directory.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4422) YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13412693#comment-13412693
 ] 

Harsh J commented on MAPREDUCE-4422:


The yarn-default contains spaces, but am guessing that those missing here is no 
big deal?:

{code}
$HADOOP_CONF_DIR, $HADOOP_COMMON_HOME/share/hadoop/common/*, 
$HADOOP_COMMON_HOME/share/hadoop/common/lib/*, 
$HADOOP_HDFS_HOME/share/hadoop/hdfs/*, 
$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*, $YARN_HOME/share/hadoop/mapreduce/*, 
$YARN_HOME/share/hadoop/mapreduce/lib/*
{code}

Regardless of the result, lets make it consistent in both places. I feel its 
better to do that with an area like the CP config.

 YARN_APPLICATION_CLASSPATH needs a documented default value in 
 YarnConfiguration
 

 Key: MAPREDUCE-4422
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4422
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Arun C Murthy
Assignee: Ahmed Radwan
 Attachments: MAPREDUCE-4422.patch


 MAPREDUCE-3505 allowed YARN_APPLICATION_CLASSPATH to be configurable.
 However, we didn't add a default value to YarnConfiguration, as-is the norm.
 Ran into it while investigating MAPREDUCE-4421.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2220) Fix new API FileOutputFormat-related typos in mapred-default.xml


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13412743#comment-13412743
 ] 

Hudson commented on MAPREDUCE-2220:
---

Integrated in Hadoop-Hdfs-0.23-Build #311 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/311/])
svn merge -c 1358788 FIXES: MAPREDUCE-2220. Fix new API 
FileOutputFormat-related typos in mapred-default.xml. Contributed by Rui Kubo. 
(harsh) (Revision 1360253)

 Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1360253
Files : 
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml


 Fix new API FileOutputFormat-related typos in mapred-default.xml
 

 Key: MAPREDUCE-2220
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2220
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: documentation
Affects Versions: 0.21.0
Reporter: Rui KUBO
Assignee: Rui KUBO
Priority: Minor
 Fix For: 2.0.1-alpha

 Attachments: mapred-default.diff, mapred-default2.diff, 
 mapred-default3.diff


 there're two typos:
  * mapreduce.output.fileoutputformat.compression.type instead of 
 mapreduce.output.fileoutputformat.compress.type
  * mapreduce.output.fileoutputformat.compression.codec instead of 
 mapreduce.output.fileoutputformat.compress.codec
 in mapred-default. Trivial patch to fix.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4432) Confusing warning message when GenericOptionsParser is not used


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gabriel Reid updated MAPREDUCE-4432:


Attachment: MAPREDUCE-4432.patch

Patch attached to update the warning message

 Confusing warning message when GenericOptionsParser is not used
 ---

 Key: MAPREDUCE-4432
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4432
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Gabriel Reid
Priority: Trivial
 Attachments: MAPREDUCE-4432.patch


 The warning that is issued in JobSubmitter -- Use GenericOptionsParser for 
 parsing the arguments. Applications should implement Tool for the same. -- 
 is confusing and (probably) grammatically incorrect.
 This can be improved by having an updated warning message which gives clearer 
 directions on what can be improved in the application to avoid the warning in 
 the future.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4432) Confusing warning message when GenericOptionsParser is not used

Gabriel Reid created MAPREDUCE-4432:
---

 Summary: Confusing warning message when GenericOptionsParser is 
not used
 Key: MAPREDUCE-4432
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4432
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Gabriel Reid
Priority: Trivial
 Attachments: MAPREDUCE-4432.patch

The warning that is issued in JobSubmitter -- Use GenericOptionsParser for 
parsing the arguments. Applications should implement Tool for the same. -- is 
confusing and (probably) grammatically incorrect.

This can be improved by having an updated warning message which gives clearer 
directions on what can be improved in the application to avoid the warning in 
the future.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4428) A failed job is not available under job history if the job is killed right around the time job is notified as failed

[
https://issues.apache.org/jira/browse/MAPREDUCE-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13412773#comment-13412773
]

Robert Joseph Evans commented on MAPREDUCE-4428:

It looks like someone killed your application

{noformat}
2012-07-11 03:04:28,481 INFO
org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop
IP=10.202.50.180OPERATION=Kill Application Request
TARGET=ClientRMService RESULT=SUCCESS APPID=application_1341894680756_0017
2012-07-11 03:04:28,481 INFO
org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl:
application_1341894680756_0017 State change from RUNNING to KILLED
{noformat}

This caused the RM to forget about the application, and it happened just as
your application was about to fail, so the AM asked to unregister, but the RM
said I don't know who you are, when in reality it should have said didn't I try
to kill you? I don't know who tried to kill this application or really why it
went to the RM instead of the AM. The issue here is that normally for
mapreduce job -kill the client is first going to request that the AM commit
suicide. That way it can put the logs where they are supposed to be, before it
tries to ask the RM to kill the application. If you do a yarn application kill
there is no guarantee what the AM will or will not be able to do before it is
killed. If the AM had been slower the NodeManager would have just sent a kill
-9 to the AM, and then it would not have had any chance at putting the logs in
the correct place. You should probably look at who was on 10.202.50.180 and
what they were doing that might have asked the RM to kill
this AM.

Fixing this in the general case so that the job history logs always are copied
to the correct place is going to be difficult. This is because we have to
insert something that will always run after the AM has exited, it is probably
best to make it so it will only run after the AM has exited badly, even for a
kill. It is possible, just not that simple of a fix. It is even more
difficult if we want to handle the case where the node appears to go down just
as the AM is crashing. there are lots of corner cases that potentially make
this very difficult to get right.

A failed job is not available under job history if the job is killed right
around the time job is notified as failed
-

Key: MAPREDUCE-4428
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4428
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver, jobtracker
Affects Versions: 2.0.0-alpha
Reporter: Rahul Jain
Attachments: appMaster_bad.txt, appMaster_good.txt, resrcmgr_bad.txt

We have observed this issue consistently running hadoop CDH4 version (based
upon 2.0 alpha release):
In case our hadoop client code gets a notification for a completed job (
using RunningJob object job, with (job.isComplete()
job.isSuccessful()==false)
the hadoop client code does an unconditional job.killJob() to terminate the
job.
With earlier hadoop versions (verified on hadoop 0.20.2 version), we still
have full access to job logs afterwards through hadoop console. However, when
using MapReduceV2, the failed hadoop job no longer shows up under jobhistory
server. Also, the tracking URL of the job still points to the non-existent
Application master http port.
Once we removed the call to job.killJob() for failed jobs from our hadoop
client code, we were able to access the job in job history with mapreduce V2
as well. Therefore this appears to be a race condition in the job management
wrt. job history for failed jobs.
We do have the application master and node manager logs collected for this
scenario if that'll help isolate the problem and the fix better.

[jira] [Commented] (MAPREDUCE-4428) A failed job is not available under job history if the job is killed right around the time job is notified as failed


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13412781#comment-13412781
 ] 

Robert Joseph Evans commented on MAPREDUCE-4428:


Just as an FYI if the user does an mapreduce job -kill and the AM reports back 
that it is no running, then it will ask the RM to kill the job instead of 
asking the AM to commit suicide.  So if you lost two races then it could be 
caused by someone running mapreduce job -kill.

 A failed job is not available under job history if the job is killed right 
 around the time job is notified as failed 
 -

 Key: MAPREDUCE-4428
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4428
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobhistoryserver, jobtracker
Affects Versions: 2.0.0-alpha
Reporter: Rahul Jain
 Attachments: appMaster_bad.txt, appMaster_good.txt, resrcmgr_bad.txt


 We have observed this issue consistently running hadoop CDH4 version (based 
 upon 2.0 alpha release):
 In case our hadoop client code gets a notification for a completed job ( 
 using RunningJob object job, with (job.isComplete()  
 job.isSuccessful()==false)
 the hadoop client code does an unconditional job.killJob() to terminate the 
 job.
 With earlier hadoop versions (verified on hadoop 0.20.2 version), we still  
 have full access to job logs afterwards through hadoop console. However, when 
 using MapReduceV2, the failed hadoop job no longer shows up under jobhistory 
 server. Also, the tracking URL of the job still points to the non-existent 
 Application master http port.
 Once we removed the call to job.killJob() for failed jobs from our hadoop 
 client code, we were able to access the job in job history with mapreduce V2 
 as well. Therefore this appears to be a race condition in the job management 
 wrt. job history for failed jobs.
 We do have the application master and node manager logs collected for this 
 scenario if that'll help isolate the problem and the fix better.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4416) Some tests fail if Clover is enabled


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kihwal Lee updated MAPREDUCE-4416:
--

Attachment: mapreduce-4416.patch.txt

 Some tests fail if Clover is enabled
 

 Key: MAPREDUCE-4416
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4416
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Kihwal Lee
Priority: Critical
 Fix For: 2.0.1-alpha, 3.0.0

 Attachments: mapreduce-4416.patch.txt


 There are number of tests running under hadoop-mapreduce-client-jobclient 
 that fail if Clover is enabled. Whenever a job is launched, AM doesn't start 
 because it can't locate the clover jar file.
 I thought MAPREDUCE-4253 had something to do with this, but I can reproduce 
 the issue on an older revision. Although unrelated, MAPREDUCE-4253 does have 
 a problem and it has been reported to the jira.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4416) Some tests fail if Clover is enabled


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kihwal Lee updated MAPREDUCE-4416:
--

Status: Patch Available  (was: Open)

 Some tests fail if Clover is enabled
 

 Key: MAPREDUCE-4416
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4416
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Kihwal Lee
Priority: Critical
 Fix For: 2.0.1-alpha, 3.0.0

 Attachments: mapreduce-4416.patch.txt


 There are number of tests running under hadoop-mapreduce-client-jobclient 
 that fail if Clover is enabled. Whenever a job is launched, AM doesn't start 
 because it can't locate the clover jar file.
 I thought MAPREDUCE-4253 had something to do with this, but I can reproduce 
 the issue on an older revision. Although unrelated, MAPREDUCE-4253 does have 
 a problem and it has been reported to the jira.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4416) Some tests fail if Clover is enabled

[
https://issues.apache.org/jira/browse/MAPREDUCE-4416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Kihwal Lee updated MAPREDUCE-4416:
--

Description:
There are number of tests running under hadoop-mapreduce-client-jobclient that
fail if Clover is enabled. Whenever a job is launched, AM doesn't start because
it can't locate the clover jar file.

I thought MAPREDUCE-4253 had something to do with this, but I can reproduce the
issue on an older revision. Although unrelated, MAPREDUCE-4253 does have a
problem and it has been reported to the jira.

was:
There are number of tests running under hadoop-mapreduce-client-jobclient that
fail if Clover is enabled. Whenever a job is launched, AM doesn't start because
it can't locate the clover jar file.

I thought MAPREDUCE-4253 had something to do this, but I can reproduce the
issue on an older revision. Although unrelated, MAPREDUCE-4253 does have a
problem and it has been reported to the jira.

Some tests fail if Clover is enabled

Key: MAPREDUCE-4416
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4416
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: client, mrv2
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Kihwal Lee
Priority: Critical
Fix For: 2.0.1-alpha, 3.0.0

Attachments: mapreduce-4416.patch.txt

There are number of tests running under hadoop-mapreduce-client-jobclient
that fail if Clover is enabled. Whenever a job is launched, AM doesn't start
because it can't locate the clover jar file.
I thought MAPREDUCE-4253 had something to do with this, but I can reproduce
the issue on an older revision. Although unrelated, MAPREDUCE-4253 does have
a problem and it has been reported to the jira.

[jira] [Updated] (MAPREDUCE-3451) Port Fair Scheduler to MR2


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun C Murthy updated MAPREDUCE-3451:
-

Status: Open  (was: Patch Available)

Patrick I took a brief look, mostly looks great! I agree having different 
schedulers is very useful.

Minor nits: I don't think it's worth modifying common infrastructure such as 
SchedulerApp right now - maybe you can just extend them where necessary?

 Port Fair Scheduler to MR2
 --

 Key: MAPREDUCE-3451
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: mrv2, scheduler
Reporter: Patrick Wendell
Assignee: Patrick Wendell
 Attachments: MAPREDUCE-3451.v1.patch.txt, 
 MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, 
 MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, 
 MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch


 The Fair Scheduler is in widespread use today in MR1 clusters, but not yet 
 ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and 
 will be updated to include design considerations and progress.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13412848#comment-13412848
 ] 

Arun C Murthy commented on MAPREDUCE-3451:
--

In fact, we should probably make SchedulerApp etc. an interface and have 
different implementations for CS, FS etc. to minimize interference. Thoughts?

 Port Fair Scheduler to MR2
 --

 Key: MAPREDUCE-3451
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: mrv2, scheduler
Reporter: Patrick Wendell
Assignee: Patrick Wendell
 Attachments: MAPREDUCE-3451.v1.patch.txt, 
 MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, 
 MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, 
 MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch


 The Fair Scheduler is in widespread use today in MR1 clusters, but not yet 
 ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and 
 will be updated to include design considerations and progress.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4432) Confusing warning message when GenericOptionsParser is not used


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13412856#comment-13412856
 ] 

Harsh J commented on MAPREDUCE-4432:


Gabriel,

Thanks for filing this! I agree we should improve it. Your improvement is much 
nicer to read and understand as well :)

Given that GenericOptionsParser is a non-public API class, and Tool was the 
real to-be-used class for users, I think we should not suggest GOP at all and 
rather show the message just with Tool?

 Confusing warning message when GenericOptionsParser is not used
 ---

 Key: MAPREDUCE-4432
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4432
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Gabriel Reid
Priority: Trivial
 Attachments: MAPREDUCE-4432.patch


 The warning that is issued in JobSubmitter -- Use GenericOptionsParser for 
 parsing the arguments. Applications should implement Tool for the same. -- 
 is confusing and (probably) grammatically incorrect.
 This can be improved by having an updated warning message which gives clearer 
 directions on what can be improved in the application to avoid the warning in 
 the future.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4416) Some tests fail if Clover is enabled

[
https://issues.apache.org/jira/browse/MAPREDUCE-4416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13412878#comment-13412878
]

Hadoop QA commented on MAPREDUCE-4416:
--

+1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12536228/mapreduce-4416.patch.txt
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test
files.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2579//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2579//console

This message is automatically generated.

Some tests fail if Clover is enabled

Attachments: mapreduce-4416.patch.txt

[jira] [Commented] (MAPREDUCE-4416) Some tests fail if Clover is enabled


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13412889#comment-13412889
 ] 

Robert Joseph Evans commented on MAPREDUCE-4416:


+1 the patch looks good, the tests pass, and clover now works.  I verified this 
manually.

 Some tests fail if Clover is enabled
 

 Key: MAPREDUCE-4416
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4416
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Kihwal Lee
Priority: Critical
 Fix For: 2.0.1-alpha, 3.0.0

 Attachments: mapreduce-4416.patch.txt


 There are number of tests running under hadoop-mapreduce-client-jobclient 
 that fail if Clover is enabled. Whenever a job is launched, AM doesn't start 
 because it can't locate the clover jar file.
 I thought MAPREDUCE-4253 had something to do with this, but I can reproduce 
 the issue on an older revision. Although unrelated, MAPREDUCE-4253 does have 
 a problem and it has been reported to the jira.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4416) Some tests fail if Clover is enabled


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Joseph Evans updated MAPREDUCE-4416:
---

Resolution: Fixed
  Assignee: Kihwal Lee
Status: Resolved  (was: Patch Available)

Thanks Kihwal,

I put this into trunk, and branch-2

 Some tests fail if Clover is enabled
 

 Key: MAPREDUCE-4416
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4416
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Priority: Critical
 Fix For: 2.0.1-alpha, 3.0.0

 Attachments: mapreduce-4416.patch.txt


 There are number of tests running under hadoop-mapreduce-client-jobclient 
 that fail if Clover is enabled. Whenever a job is launched, AM doesn't start 
 because it can't locate the clover jar file.
 I thought MAPREDUCE-4253 had something to do with this, but I can reproduce 
 the issue on an older revision. Although unrelated, MAPREDUCE-4253 does have 
 a problem and it has been reported to the jira.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4416) Some tests fail if Clover is enabled


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13412895#comment-13412895
 ] 

Hudson commented on MAPREDUCE-4416:
---

Integrated in Hadoop-Hdfs-trunk-Commit #2522 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2522/])
MAPREDUCE-4416. Some tests fail if Clover is enabled (Kihwal Lee via bobby) 
(Revision 1360735)

 Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1360735
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestIndexCache.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/pom.xml
* /hadoop/common/trunk/hadoop-project/pom.xml


 Some tests fail if Clover is enabled
 

 Key: MAPREDUCE-4416
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4416
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Priority: Critical
 Fix For: 2.0.1-alpha, 3.0.0

 Attachments: mapreduce-4416.patch.txt


 There are number of tests running under hadoop-mapreduce-client-jobclient 
 that fail if Clover is enabled. Whenever a job is launched, AM doesn't start 
 because it can't locate the clover jar file.
 I thought MAPREDUCE-4253 had something to do with this, but I can reproduce 
 the issue on an older revision. Although unrelated, MAPREDUCE-4253 does have 
 a problem and it has been reported to the jira.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4416) Some tests fail if Clover is enabled


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13412896#comment-13412896
 ] 

Hudson commented on MAPREDUCE-4416:
---

Integrated in Hadoop-Common-trunk-Commit #2456 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2456/])
MAPREDUCE-4416. Some tests fail if Clover is enabled (Kihwal Lee via bobby) 
(Revision 1360735)

 Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1360735
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestIndexCache.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/pom.xml
* /hadoop/common/trunk/hadoop-project/pom.xml


 Some tests fail if Clover is enabled
 

 Key: MAPREDUCE-4416
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4416
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Priority: Critical
 Fix For: 2.0.1-alpha, 3.0.0

 Attachments: mapreduce-4416.patch.txt


 There are number of tests running under hadoop-mapreduce-client-jobclient 
 that fail if Clover is enabled. Whenever a job is launched, AM doesn't start 
 because it can't locate the clover jar file.
 I thought MAPREDUCE-4253 had something to do with this, but I can reproduce 
 the issue on an older revision. Although unrelated, MAPREDUCE-4253 does have 
 a problem and it has been reported to the jira.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4393) PaaS on YARN: an YARN application to demonstrate that YARN can be used as a PaaS


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jaigak Song updated MAPREDUCE-4393:
---

Attachment: MAPREDUCE4393.patch

Finally I could successfully run 'dec-support.sh' with a positive overall 
result. 



 PaaS on YARN: an YARN application to demonstrate that YARN can be used as a 
 PaaS
 

 Key: MAPREDUCE-4393
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4393
 Project: Hadoop Map/Reduce
  Issue Type: Task
  Components: examples
Affects Versions: 0.23.1
Reporter: Jaigak Song
Assignee: Jaigak Song
 Fix For: 3.0.0

 Attachments: HADOOPasPAAS_Architecture.pdf, MAPREDUCE-4393.patch, 
 MAPREDUCE-4393.patch, MAPREDUCE-4393.patch, MAPREDUCE4393.patch

   Original Estimate: 336h
  Remaining Estimate: 336h

 This application is to demonstrate that YARN can be used for non-mapreduce 
 applications. As Hadoop has already been adopted and deployed widely and its 
 deployment in future will be highly increased, we thought that it's a good 
 potential to be used as PaaS.  
 I have implemented a proof of concept to demonstrate that YARN can be used as 
 a PaaS (Platform as a Service). I have done a gap analysis against VMware's 
 Cloud Foundry and tried to achieve as many PaaS functionalities as possible 
 on YARN.
 I'd like to check in this POC as a YARN example application.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4393) PaaS on YARN: an YARN application to demonstrate that YARN can be used as a PaaS

[
https://issues.apache.org/jira/browse/MAPREDUCE-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13412923#comment-13412923
]

Hadoop QA commented on MAPREDUCE-4393:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12536236/MAPREDUCE4393.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 6 new or modified test
files.

-1 javac. The patch appears to cause the build to fail.

Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2580//console

This message is automatically generated.

PaaS on YARN: an YARN application to demonstrate that YARN can be used as a
PaaS

Key: MAPREDUCE-4393
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4393
Project: Hadoop Map/Reduce
Issue Type: Task
Components: examples
Affects Versions: 0.23.1
Reporter: Jaigak Song
Assignee: Jaigak Song
Fix For: 3.0.0

Attachments: HADOOPasPAAS_Architecture.pdf, MAPREDUCE-4393.patch,
MAPREDUCE-4393.patch, MAPREDUCE-4393.patch, MAPREDUCE4393.patch

Original Estimate: 336h
Remaining Estimate: 336h

This application is to demonstrate that YARN can be used for non-mapreduce
applications. As Hadoop has already been adopted and deployed widely and its
deployment in future will be highly increased, we thought that it's a good
potential to be used as PaaS.
I have implemented a proof of concept to demonstrate that YARN can be used as
a PaaS (Platform as a Service). I have done a gap analysis against VMware's
Cloud Foundry and tried to achieve as many PaaS functionalities as possible
on YARN.
I'd like to check in this POC as a YARN example application.

[jira] [Commented] (MAPREDUCE-4416) Some tests fail if Clover is enabled


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13412934#comment-13412934
 ] 

Hudson commented on MAPREDUCE-4416:
---

Integrated in Hadoop-Mapreduce-trunk-Commit #2475 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2475/])
MAPREDUCE-4416. Some tests fail if Clover is enabled (Kihwal Lee via bobby) 
(Revision 1360735)

 Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1360735
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestIndexCache.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/pom.xml
* /hadoop/common/trunk/hadoop-project/pom.xml


 Some tests fail if Clover is enabled
 

 Key: MAPREDUCE-4416
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4416
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Priority: Critical
 Fix For: 2.0.1-alpha, 3.0.0

 Attachments: mapreduce-4416.patch.txt


 There are number of tests running under hadoop-mapreduce-client-jobclient 
 that fail if Clover is enabled. Whenever a job is launched, AM doesn't start 
 because it can't locate the clover jar file.
 I thought MAPREDUCE-4253 had something to do with this, but I can reproduce 
 the issue on an older revision. Although unrelated, MAPREDUCE-4253 does have 
 a problem and it has been reported to the jira.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2

[
https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13412941#comment-13412941
]

Patrick Wendell commented on MAPREDUCE-3451:

Hey Arun - I think extending it is a better move for now.

If we decided to move SchedulerApp to an interface, 95% of the implementation
would be the same between the FS/CS. The only differences are in the way delay
scheduling works for the FairScheduler.

As far as I can tell - the idea behind SchedulerApp is to deal with application
lifecycle management from the perspective of the scheduler. That's helpful to
have and made writing the FS simpler. This is likely to be useful for future
schedulers well. In terms of maximizing code sharing I would be in favor of
keeping it collectively managed.

Port Fair Scheduler to MR2
--

Key: MAPREDUCE-3451
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: mrv2, scheduler
Reporter: Patrick Wendell
Assignee: Patrick Wendell
Attachments: MAPREDUCE-3451.v1.patch.txt,
MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt,
MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch,
MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch

The Fair Scheduler is in widespread use today in MR1 clusters, but not yet
ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and
will be updated to include design considerations and progress.

[jira] [Commented] (MAPREDUCE-4428) A failed job is not available under job history if the job is killed right around the time job is notified as failed

2012-07-12 Thread Rahul Jain (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13412950#comment-13412950
 ] 

Rahul Jain commented on MAPREDUCE-4428:
---

Robert,

To make the user perspective clear here:

This grid is a single user managed grid, no other process was running at the 
time and no one else tried to do kill on the job here.

The sequence is:

a) The job creator application submitted the job to hadoop grid.

b) Max retry count was set to 1 for both mappers and reducers; so as soon as a 
task failed, the system (AM?) decided to kill all other tasks.

c) The submitter application is waiting in a sleep loop, waking up every 1 
second to check the status of the task
calling: JobClient.getJob()

d) When the above condition happens, the application receives the running job 
status as completed , failed (isSuccessful()=false, isComplete()=true on 
RunningJob object)

e) The application issues a killJob() on the running job object at this time

f) As a result, nothing is accessible in job history from hadoop console, even 
the AM container logs cannot be accessed.

Removing (e) from the above sequence make logs accessible again. As I 
mentioned, with older version of map-reduce, we never encountered the issues of 
logs getting lost. I believe we need to handle the case of user initiated 
'KILL' of the job better in MapReduceV2; 90% of the time we look at map-reduce 
logs only for failed and killed jobs; so this functionality should work 
reliably as much as possible.





 A failed job is not available under job history if the job is killed right 
 around the time job is notified as failed 
 -

 Key: MAPREDUCE-4428
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4428
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: jobhistoryserver, jobtracker
Affects Versions: 2.0.0-alpha
Reporter: Rahul Jain
 Attachments: appMaster_bad.txt, appMaster_good.txt, resrcmgr_bad.txt


 We have observed this issue consistently running hadoop CDH4 version (based 
 upon 2.0 alpha release):
 In case our hadoop client code gets a notification for a completed job ( 
 using RunningJob object job, with (job.isComplete()  
 job.isSuccessful()==false)
 the hadoop client code does an unconditional job.killJob() to terminate the 
 job.
 With earlier hadoop versions (verified on hadoop 0.20.2 version), we still  
 have full access to job logs afterwards through hadoop console. However, when 
 using MapReduceV2, the failed hadoop job no longer shows up under jobhistory 
 server. Also, the tracking URL of the job still points to the non-existent 
 Application master http port.
 Once we removed the call to job.killJob() for failed jobs from our hadoop 
 client code, we were able to access the job in job history with mapreduce V2 
 as well. Therefore this appears to be a race condition in the job management 
 wrt. job history for failed jobs.
 We do have the application master and node manager logs collected for this 
 scenario if that'll help isolate the problem and the fix better.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4428) A failed job is not available under job history if the job is killed right around the time job is notified as failed

2012-07-12 Thread Rahul Jain (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Rahul Jain updated MAPREDUCE-4428:
--

Attachment: am_failed_counter_limits.txt

Am attaching another case (not directly related to the ticket) that illustrates
the point on more reliable way of getting job history.

Our map-reduce jobs generate counters beyond the standard 120 limit imposed by
yarn. We increased the counters to 1000 but the jobs went over that limit as
well.

As you can see in attached file am_failed_counter_limits.txt, the AM
encountered exceptions trying to handle the large # of counters; at the end no
job history was accessible either.

Note that we did increase counters for the job submitter task to 2000, however
the only way to have AM use a higher # of counters is to change the global yarn
settings and restart yarn.

A failed job is not available under job history if the job is killed right
around the time job is notified as failed
-

Key: MAPREDUCE-4428
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4428
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobhistoryserver, jobtracker
Affects Versions: 2.0.0-alpha
Reporter: Rahul Jain
Attachments: am_failed_counter_limits.txt, appMaster_bad.txt,
appMaster_good.txt, resrcmgr_bad.txt

[jira] [Updated] (MAPREDUCE-3451) Port Fair Scheduler to MR2


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Wendell updated MAPREDUCE-3451:
---

Attachment: MAPREDUCE-3451.v8.patch

This patch extends rather than modifies the SchedulerApp class for use in the 
Fair Scheduler.

I cleaned up some tabs/whitespace issues in SchedulerApp.java and those are 
retained in this patch. But there are no code changes to SchedulerApp.java.

 Port Fair Scheduler to MR2
 --

 Key: MAPREDUCE-3451
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: mrv2, scheduler
Reporter: Patrick Wendell
Assignee: Patrick Wendell
 Attachments: MAPREDUCE-3451.v1.patch.txt, 
 MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, 
 MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, 
 MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch


 The Fair Scheduler is in widespread use today in MR1 clusters, but not yet 
 ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and 
 will be updated to include design considerations and progress.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3868) Reenable Raid

2012-07-12 Thread Eli Collins (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413067#comment-13413067
 ] 

Eli Collins commented on MAPREDUCE-3868:


TestRaidNode is still consistently failing, I filed HDFS-3648 for this.  

Weiyan/Scott - can you guys address the findbugs and test failures soon?

 Reenable Raid
 -

 Key: MAPREDUCE-3868
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3868
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: contrib/raid
Reporter: Scott Chen
Assignee: Weiyan Wang
 Attachments: MAPREDUCE-3868-1.patch, MAPREDUCE-3868-2.patch, 
 MAPREDUCE-3868-3.patch, MAPREDUCE-3868.patch, MAPREDUCE-3868v1.patch, 
 MAPREDUCE-3868v1.sh


 Currently Raid is outdated and not compiled. Make it compile.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4422) YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413073#comment-13413073
 ] 

Ahmed Radwan commented on MAPREDUCE-4422:
-

Thanks Harsh for the review!
Yes, it is not an issue, but I agree about consistency. I have updated the 
patch accordingly.

 YARN_APPLICATION_CLASSPATH needs a documented default value in 
 YarnConfiguration
 

 Key: MAPREDUCE-4422
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4422
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Arun C Murthy
Assignee: Ahmed Radwan
 Attachments: MAPREDUCE-4422.patch, MAPREDUCE-4422_rev2.patch


 MAPREDUCE-3505 allowed YARN_APPLICATION_CLASSPATH to be configurable.
 However, we didn't add a default value to YarnConfiguration, as-is the norm.
 Ran into it while investigating MAPREDUCE-4421.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4422) YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Radwan updated MAPREDUCE-4422:


Attachment: MAPREDUCE-4422_rev2.patch

 YARN_APPLICATION_CLASSPATH needs a documented default value in 
 YarnConfiguration
 

 Key: MAPREDUCE-4422
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4422
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Arun C Murthy
Assignee: Ahmed Radwan
 Attachments: MAPREDUCE-4422.patch, MAPREDUCE-4422_rev2.patch


 MAPREDUCE-3505 allowed YARN_APPLICATION_CLASSPATH to be configurable.
 However, we didn't add a default value to YarnConfiguration, as-is the norm.
 Ran into it while investigating MAPREDUCE-4421.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4432) Confusing warning message when GenericOptionsParser is not used


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413077#comment-13413077
 ] 

Gabriel Reid commented on MAPREDUCE-4432:
-

Good point about GOP -- I'll upload a new patch that only mentions Tool and 
ToolRunner.

 Confusing warning message when GenericOptionsParser is not used
 ---

 Key: MAPREDUCE-4432
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4432
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Gabriel Reid
Priority: Trivial
 Attachments: MAPREDUCE-4432.patch


 The warning that is issued in JobSubmitter -- Use GenericOptionsParser for 
 parsing the arguments. Applications should implement Tool for the same. -- 
 is confusing and (probably) grammatically incorrect.
 This can be improved by having an updated warning message which gives clearer 
 directions on what can be improved in the application to avoid the warning in 
 the future.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4432) Confusing warning message when GenericOptionsParser is not used


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gabriel Reid updated MAPREDUCE-4432:


Attachment: MAPREDUCE-4432.patch

Updated patch that doesn't mention GenericOptionsParser

 Confusing warning message when GenericOptionsParser is not used
 ---

 Key: MAPREDUCE-4432
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4432
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Reporter: Gabriel Reid
Priority: Trivial
 Attachments: MAPREDUCE-4432.patch, MAPREDUCE-4432.patch


 The warning that is issued in JobSubmitter -- Use GenericOptionsParser for 
 parsing the arguments. Applications should implement Tool for the same. -- 
 is confusing and (probably) grammatically incorrect.
 This can be improved by having an updated warning message which gives clearer 
 directions on what can be improved in the application to avoid the warning in 
 the future.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4422) YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413088#comment-13413088
 ] 

Harsh J commented on MAPREDUCE-4422:


Thanks Ahmed. +1, pending Jenkins.

 YARN_APPLICATION_CLASSPATH needs a documented default value in 
 YarnConfiguration
 

 Key: MAPREDUCE-4422
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4422
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Reporter: Arun C Murthy
Assignee: Ahmed Radwan
 Attachments: MAPREDUCE-4422.patch, MAPREDUCE-4422_rev2.patch


 MAPREDUCE-3505 allowed YARN_APPLICATION_CLASSPATH to be configurable.
 However, we didn't add a default value to YarnConfiguration, as-is the norm.
 Ran into it while investigating MAPREDUCE-4421.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4422) YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration

[
https://issues.apache.org/jira/browse/MAPREDUCE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413092#comment-13413092
]

Hadoop QA commented on MAPREDUCE-4422:
--

-1 overall. Here are the results of testing the latest attachment

http://issues.apache.org/jira/secure/attachment/12536259/MAPREDUCE-4422_rev2.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common.

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2581//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2581//console

This message is automatically generated.

YARN_APPLICATION_CLASSPATH needs a documented default value in
YarnConfiguration

MAPREDUCE-3505 allowed YARN_APPLICATION_CLASSPATH to be configurable.
However, we didn't add a default value to YarnConfiguration, as-is the norm.
Ran into it while investigating MAPREDUCE-4421.

[jira] [Updated] (MAPREDUCE-4422) YARN_APPLICATION_CLASSPATH needs a documented default value in YarnConfiguration


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J updated MAPREDUCE-4422:
---

  Component/s: nodemanager
 Target Version/s: 2.0.1-alpha
Affects Version/s: 2.0.0-alpha
 Hadoop Flags: Reviewed

Committing shortly.

 YARN_APPLICATION_CLASSPATH needs a documented default value in 
 YarnConfiguration
 

 Key: MAPREDUCE-4422
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4422
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: nodemanager
Affects Versions: 2.0.0-alpha
Reporter: Arun C Murthy
Assignee: Ahmed Radwan
 Attachments: MAPREDUCE-4422.patch, MAPREDUCE-4422_rev2.patch


 MAPREDUCE-3505 allowed YARN_APPLICATION_CLASSPATH to be configurable.
 However, we didn't add a default value to YarnConfiguration, as-is the norm.
 Ran into it while investigating MAPREDUCE-4421.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (MAPREDUCE-4433) Use Time#now and Time#monotonicNow instead of System#currentTimeMillis

2012-07-12 Thread Eli Collins (JIRA)

Eli Collins created MAPREDUCE-4433:
--

 Summary: Use Time#now and Time#monotonicNow instead of 
System#currentTimeMillis
 Key: MAPREDUCE-4433
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4433
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
Affects Versions: 2.0.0-alpha
Reporter: Eli Collins
Priority: Minor


hadoop-auth should be updated per HDFS-3641, I didn't do so in that patch as 
hadoop-common depends on hadoop-auth so we can't introduce a dependency on a 
common method. Given it's two one-line methods that we won't update let's just 
create a small Util (or Time) class in hadoop-auth and copy the methods into 
there.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4393) PaaS on YARN: an YARN application to demonstrate that YARN can be used as a PaaS

[
https://issues.apache.org/jira/browse/MAPREDUCE-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jaigak Song updated MAPREDUCE-4393:
---

Attachment: MAPREDUCE4393.patch

Please bear with me as I'm new to this Hadoop development environment. The
attached patch works fine (i.e. test-patch.sh produced +1 overall result) on my
ubuntu machine. Let's see how it goes this time.

PaaS on YARN: an YARN application to demonstrate that YARN can be used as a
PaaS

Attachments: HADOOPasPAAS_Architecture.pdf, MAPREDUCE-4393.patch,
MAPREDUCE-4393.patch, MAPREDUCE-4393.patch, MAPREDUCE4393.patch,
MAPREDUCE4393.patch

Original Estimate: 336h
Remaining Estimate: 336h

[jira] [Updated] (MAPREDUCE-4416) Some tests fail if Clover is enabled


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kihwal Lee updated MAPREDUCE-4416:
--

Attachment: mapreduce-4416.patch.txt

 Some tests fail if Clover is enabled
 

 Key: MAPREDUCE-4416
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4416
 Project: Hadoop Map/Reduce
  Issue Type: Bug
  Components: client, mrv2
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Priority: Critical
 Fix For: 2.0.1-alpha, 3.0.0

 Attachments: mapreduce-4416.patch.txt, mapreduce-4416.patch.txt


 There are number of tests running under hadoop-mapreduce-client-jobclient 
 that fail if Clover is enabled. Whenever a job is launched, AM doesn't start 
 because it can't locate the clover jar file.
 I thought MAPREDUCE-4253 had something to do with this, but I can reproduce 
 the issue on an older revision. Although unrelated, MAPREDUCE-4253 does have 
 a problem and it has been reported to the jira.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4283) Display tail of aggregated logs by default

2012-07-12 Thread Jason Lowe (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413146#comment-13413146
 ] 

Jason Lowe commented on MAPREDUCE-4283:
---

Pinging for review, would like to see this feature integrated soon as we keep 
running into large aggregated logs and have no good way to view a portion of 
them without downloading the entire log.

 Display tail of aggregated logs by default
 --

 Key: MAPREDUCE-4283
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4283
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: jobhistoryserver, mrv2
Affects Versions: 0.23.3, 2.0.1-alpha
Reporter: Jason Lowe
Assignee: Jason Lowe
 Attachments: MAPREDUCE-4283.patch


 Similar to the manner in which the nodemanager webUI displays container logs, 
 it would be very useful if the historyserver showed the trailing 4K or so of 
 the aggregated logs with a link to see the full log.
 When debugging issues the relevant errors are usually at the end of the log, 
 so showing just the last few K can enable quick diagnosis without waiting for 
 what can be many megabytes of log data to download. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4393) PaaS on YARN: an YARN application to demonstrate that YARN can be used as a PaaS

[
https://issues.apache.org/jira/browse/MAPREDUCE-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413148#comment-13413148
]

Hadoop QA commented on MAPREDUCE-4393:
--

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12536269/MAPREDUCE4393.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 6 new or modified test
files.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-paas/hadoop-yarn-applications-paas-client

hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-paas/hadoop-yarn-applications-paas-container

hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-paas/hadoop-yarn-applications-paas-master

hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-paas/hadoop-yarn-applications-paas-zkclient.

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2582//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2582//console

This message is automatically generated.

PaaS on YARN: an YARN application to demonstrate that YARN can be used as a
PaaS

Attachments: HADOOPasPAAS_Architecture.pdf, MAPREDUCE-4393.patch,
MAPREDUCE-4393.patch, MAPREDUCE-4393.patch, MAPREDUCE4393.patch,
MAPREDUCE4393.patch

Original Estimate: 336h
Remaining Estimate: 336h

[jira] [Updated] (MAPREDUCE-4157) ResourceManager should not kill apps that are well behaved

2012-07-12 Thread Jason Lowe (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe updated MAPREDUCE-4157:
--

Attachment: MAPREDUCE-4157.patch

Updated patch so it applies cleanly to trunk.

 ResourceManager should not kill apps that are well behaved
 --

 Key: MAPREDUCE-4157
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4157
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: mrv2
Affects Versions: 2.0.0-alpha
Reporter: Jason Lowe
Assignee: Jason Lowe
 Attachments: MAPREDUCE-4157.patch, MAPREDUCE-4157.patch


 Currently when the ApplicationMaster unregisters with the ResourceManager, 
 the RM kills (via the NMs) all the active containers for an application.  
 This introduces a race where the AM may be trying to clean up and may not 
 finish before it is killed.  The RM should give the AM a chance to exit 
 cleanly on its own rather than always race with a pending kill on shutdown.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4157) ResourceManager should not kill apps that are well behaved

[
https://issues.apache.org/jira/browse/MAPREDUCE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413193#comment-13413193
]

Hadoop QA commented on MAPREDUCE-4157:
--

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12536275/MAPREDUCE-4157.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 14 new or modified test
files.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2583//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2583//console

This message is automatically generated.

ResourceManager should not kill apps that are well behaved
--

Key: MAPREDUCE-4157
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4157
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 2.0.0-alpha
Reporter: Jason Lowe
Assignee: Jason Lowe
Attachments: MAPREDUCE-4157.patch, MAPREDUCE-4157.patch

Currently when the ApplicationMaster unregisters with the ResourceManager,
the RM kills (via the NMs) all the active containers for an application.
This introduces a race where the AM may be trying to clean up and may not
finish before it is killed. The RM should give the AM a chance to exit
cleanly on its own rather than always race with a pending kill on shutdown.

[jira] [Commented] (MAPREDUCE-4349) Distributed Cache gives inconsistent result if cache Archive files get deleted from task tracker

2012-07-12 Thread Mayank Bansal (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413217#comment-13413217
 ] 

Mayank Bansal commented on MAPREDUCE-4349:
--

MAPREDUCE-4342 fixes this issue. I will add the test case to verify the event.

Thanks,
Mayank

 Distributed Cache gives inconsistent result if cache Archive files get 
 deleted from task tracker 
 -

 Key: MAPREDUCE-4349
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4349
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 0.22.0, 1.0.3, trunk
Reporter: Mayank Bansal
Assignee: Mayank Bansal



--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4393) PaaS on YARN: an YARN application to demonstrate that YARN can be used as a PaaS

2012-07-12 Thread Bikas Saha (JIRA)

[
https://issues.apache.org/jira/browse/MAPREDUCE-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413241#comment-13413241
]

Bikas Saha commented on MAPREDUCE-4393:
---

I took a pass at the changes. I have some
1) The Client and AppMaster look very similar to DistributedShell. It might be
useful to see if some of the common portions could be abstracted out.
2) How about using the AM itself as the information repo about active PAAS
containers instead of storing information in ZK? The AM knows exactly what is
running. If there is some information that the containers need to post then
that can be posted to the AM itself. Thereafter, the AM can be queried for the
same information that ZK is giving.
3) The AM could open a port to listen for new commands from the PAAS client. So
starting new instances can be done via the currently running AM instead of
starting new AM's.

PaaS on YARN: an YARN application to demonstrate that YARN can be used as a
PaaS

Attachments: HADOOPasPAAS_Architecture.pdf, MAPREDUCE-4393.patch,
MAPREDUCE-4393.patch, MAPREDUCE-4393.patch, MAPREDUCE4393.patch,
MAPREDUCE4393.patch

Original Estimate: 336h
Remaining Estimate: 336h

[jira] [Commented] (MAPREDUCE-4393) PaaS on YARN: an YARN application to demonstrate that YARN can be used as a PaaS

[
https://issues.apache.org/jira/browse/MAPREDUCE-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413274#comment-13413274
]

Jaigak Song commented on MAPREDUCE-4393:

Bikas, thanks for your comments!
Regarding #1, some shell related portion and some more can be abstracted out,
but I don't see much value out of that in my opinion.

For #2, I agree on that AM can be used, but first, #3 should be implemented
before that in order to have one place to maintain available containers of the
same application type. Besides, ZooKeeper seems a better choice at this point
as the PaaS implementation has Routers (which is not part of the patch due to
some dependency), which are supposed to distribute incoming requests by
utilizing the information of which containers are available for which
application type. If there are multiple AM's (e.g. hundreds or thousands of
AM's) for different application types, ZooKeeper is much simpler to use and can
be better performant because of its asynchronous characteristics.

For #3, I received the same suggestion from Arun Murthy a couple of weeks ago
and I put it as an enhancement to my documentation to distribute soon.
Certainly we can improve the application later. This implementation was a POC
done within a couple of weeks.

PaaS on YARN: an YARN application to demonstrate that YARN can be used as a
PaaS

Attachments: HADOOPasPAAS_Architecture.pdf, MAPREDUCE-4393.patch,
MAPREDUCE-4393.patch, MAPREDUCE-4393.patch, MAPREDUCE4393.patch,
MAPREDUCE4393.patch

Original Estimate: 336h
Remaining Estimate: 336h

[jira] [Commented] (MAPREDUCE-4393) PaaS on YARN: an YARN application to demonstrate that YARN can be used as a PaaS

[
https://issues.apache.org/jira/browse/MAPREDUCE-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413294#comment-13413294
]

Kihwal Lee commented on MAPREDUCE-4393:
---

I think use of ZK is fine since it won't be pretty for routers to poll status
from RM (to get the list of AMs) and AM (to get updates on app instances).
Multiple AMs can run on the same node, so a predefined port number cannot be
used. Then there has to be a way to discover the port number. Having ZK in the
picture certainly helps.

But depending on the requirement on router, all external dependencies (router
zk) can be substituted with another YARN app! PaaS System App? If we do this,
the PaaS app can be made to talk to any one of the two types of management
system.

PaaS on YARN: an YARN application to demonstrate that YARN can be used as a
PaaS

Attachments: HADOOPasPAAS_Architecture.pdf, MAPREDUCE-4393.patch,
MAPREDUCE-4393.patch, MAPREDUCE-4393.patch, MAPREDUCE4393.patch,
MAPREDUCE4393.patch

Original Estimate: 336h
Remaining Estimate: 336h

[jira] [Commented] (MAPREDUCE-4393) PaaS on YARN: an YARN application to demonstrate that YARN can be used as a PaaS


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413304#comment-13413304
 ] 

Jaigak Song commented on MAPREDUCE-4393:


One of the requirements for PaaS (at least I have) is that even if AM crashes, 
all the application containers should keep running if possible. In this sense, 
ZK or a more reliable component is better for tracking available instances 
instead of AM doing that.  


 PaaS on YARN: an YARN application to demonstrate that YARN can be used as a 
 PaaS
 

 Key: MAPREDUCE-4393
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4393
 Project: Hadoop Map/Reduce
  Issue Type: Task
  Components: examples
Affects Versions: 0.23.1
Reporter: Jaigak Song
Assignee: Jaigak Song
 Fix For: 3.0.0

 Attachments: HADOOPasPAAS_Architecture.pdf, MAPREDUCE-4393.patch, 
 MAPREDUCE-4393.patch, MAPREDUCE-4393.patch, MAPREDUCE4393.patch, 
 MAPREDUCE4393.patch

   Original Estimate: 336h
  Remaining Estimate: 336h

 This application is to demonstrate that YARN can be used for non-mapreduce 
 applications. As Hadoop has already been adopted and deployed widely and its 
 deployment in future will be highly increased, we thought that it's a good 
 potential to be used as PaaS.  
 I have implemented a proof of concept to demonstrate that YARN can be used as 
 a PaaS (Platform as a Service). I have done a gap analysis against VMware's 
 Cloud Foundry and tried to achieve as many PaaS functionalities as possible 
 on YARN.
 I'd like to check in this POC as a YARN example application.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4435) Expose JobTracker metrics for number of reducers in shuffle vs. sort vs. reduce phase

2012-07-12 Thread Eirik Bakke (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eirik Bakke updated MAPREDUCE-4435:
---

Attachment: mapreduce.patch

 Expose JobTracker metrics for number of reducers in shuffle vs. sort vs. 
 reduce phase
 -

 Key: MAPREDUCE-4435
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4435
 Project: Hadoop Map/Reduce
  Issue Type: Improvement
  Components: jobtracker, tasktracker
Reporter: Eirik Bakke
 Attachments: mapreduce.patch


 We'd like to be able to show our Cloudera Manager users some more detailed 
 metrics about the number of reducers running at any given time--specifically, 
 how many reducers are running in each of the three possible phases (shuffle, 
 sort, and reduce). This would require the addition of some new overridable 
 methods to the JobTrackerInstrumentation API, plus a little bit of code to 
 actually call them from the JobTracker class. The necessary information seems 
 to already be available in the TaskStatus object. The attached patch (which 
 I've tested on hadoop-common/branch-1.0) shows one way to do it.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4435) Expose JobTracker metrics for number of reducers in shuffle vs. sort vs. reduce phase

[
https://issues.apache.org/jira/browse/MAPREDUCE-4435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413330#comment-13413330
]

Arun C Murthy commented on MAPREDUCE-4435:
--

Eirik, seems reasonable.

Could you please add a test case?

Also, a minor improvement would be to store the MapPhase,Integer in
TaskTrackerStatus to save a recompute - not that it matters a whole lot... :)

Expose JobTracker metrics for number of reducers in shuffle vs. sort vs.
reduce phase
-

Key: MAPREDUCE-4435
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4435
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobtracker, tasktracker
Reporter: Eirik Bakke
Attachments: mapreduce.patch

We'd like to be able to show our Cloudera Manager users some more detailed
metrics about the number of reducers running at any given time--specifically,
how many reducers are running in each of the three possible phases (shuffle,
sort, and reduce). This would require the addition of some new overridable
methods to the JobTrackerInstrumentation API, plus a little bit of code to
actually call them from the JobTracker class. The necessary information seems
to already be available in the TaskStatus object. The attached patch (which
I've tested on hadoop-common/branch-1.0) shows one way to do it.

[jira] [Commented] (MAPREDUCE-4393) PaaS on YARN: an YARN application to demonstrate that YARN can be used as a PaaS

[
https://issues.apache.org/jira/browse/MAPREDUCE-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413337#comment-13413337
]

Kihwal Lee commented on MAPREDUCE-4393:
---

I didn't mean that the manager AM is responsible for launching app AMs. I think
it can be a separate yarn app. They don't even have to be any start-up
dependency among them, if we design communication protocol well. This also
makes restart easy.

If we can (re)launch the manager AM on one of the predefined set of hosts, most
of the requirements can be met. By storing system state in the hdfs and
reading back on restart, it can go back in sync fast and offer service again.
Routers can be provisioned similarly, but they will acquire state information
from the manager AM. The service discovery is simplified by the fact that they
will be on specific hosts. If a VIP is used to deal with service up/down or
migration among the given set of hosts, the service discovery is further
simplified. Since they are independent app instances or independent yarn apps,
a crash/restart of one thing won't force termination of others.

The one thing I am not sure about is the ability to specifying a specific set
of candidate hosts for launching AM. If not supported already, we can launch AM
on a random host and then launch containers on a specific set of hosts, but
that lowers the reliability. Or maybe the AM can be anywhere and the container
launched from it will only be used for service discovery.

I am not insisting on doing this now, but it will be nice if everything is
contained in YARN so that setting up is simpler and it is easily demoable.

PaaS on YARN: an YARN application to demonstrate that YARN can be used as a
PaaS

Attachments: HADOOPasPAAS_Architecture.pdf, MAPREDUCE-4393.patch,
MAPREDUCE-4393.patch, MAPREDUCE-4393.patch, MAPREDUCE4393.patch,
MAPREDUCE4393.patch

Original Estimate: 336h
Remaining Estimate: 336h

[jira] [Updated] (MAPREDUCE-3451) Port Fair Scheduler to MR2


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Wendell updated MAPREDUCE-3451:
---

Attachment: MAPREDUCE-3451.v9.patch

This patch moves a TestFSSchedulerApp into the fair/ test subdirectory.

It also reverts whitespace fixes in SchedulerApp.java (so that file is now 
totally unchanged from trunk). That is out of scope for this JIRA.

 Port Fair Scheduler to MR2
 --

 Key: MAPREDUCE-3451
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: mrv2, scheduler
Reporter: Patrick Wendell
Assignee: Patrick Wendell
 Attachments: MAPREDUCE-3451.v1.patch.txt, 
 MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, 
 MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, 
 MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, 
 MAPREDUCE-3451.v9.patch


 The Fair Scheduler is in widespread use today in MR1 clusters, but not yet 
 ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and 
 will be updated to include design considerations and progress.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service

[
https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413344#comment-13413344
]

Arun C Murthy commented on MAPREDUCE-4049:
--

Avner, apologies for taking this long.

The patch looks reasonable, and small(!) which is great.

The concern I have is that this patch introduces an interface (i.e.
ShuffleProvider/Consumer) which isn't present in hadoop-2.x. Should we do both
hadoop-2 and hadoop-1 simultaneously? Else, this 'feature' will break as soon
as we upgrade to hadoop-2.x.

Other nits:
# We should get TaskTracker.MapOutputServlet to implement ShuffleProvider
interface, else it's very easy to break an interface if no one in the core
implements it. For e.g. I have no idea about ShuffleProvider.taskDone or
ShuffleProvider.jobDone are used.
# Minor nits: ShuffleProvider is mis-spelt in a couple of places.
# We should add the new configs for provider/consumer in mapred-default.xml

Again, apologies it took me so long to get to your patch and thanks for being
super-patient! I'd like to work with you to get this committed asap!

plugin for generic shuffle service
--

Key: MAPREDUCE-4049
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: performance, task, tasktracker
Affects Versions: 1.0.3, 1.1.0, 2.0.0-alpha, 3.0.0
Reporter: Avner BenHanoch
Labels: merge, plugin, rdma, shuffle
Attachments: HADOOP-1.0.2.patch, HADOOP-1.0.x.patch,
HADOOP-1.1.patch, HADOOP-1.x.y-review-oriented.patch, Hadoop Shuffle Consumer
Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, mapred-site.xml

Support generic shuffle service as set of two plugins: ShuffleProvider
ShuffleConsumer.
This will satisfy the following needs:
# Better shuffle and merge performance. For example: we are working on
shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE,
or Infiniband) instead of using the current HTTP shuffle. Based on the fast
RDMA shuffle, the plugin can also utilize a suitable merge approach during
the intermediate merges. Hence, getting much better performance.
# Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden
dependency of NodeManager with a specific version of mapreduce shuffle
(currently targeted to 0.24.0).
References:
# Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu
from Auburn University with others,
[http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf]
# I am attaching 2 documents with suggested Top Level Design for both plugins
(currently, based on 1.0 branch)

[jira] [Commented] (MAPREDUCE-4393) PaaS on YARN: an YARN application to demonstrate that YARN can be used as a PaaS


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413348#comment-13413348
 ] 

Jaigak Song commented on MAPREDUCE-4393:


Personally I don't like everything being contained in YARN considering 
enterprise environments. Certainly it can be one of the options Hadoop YARN may 
provide, but too much tight integrations (but I don't think you mean this) can 
be a dislike in enterprise environments. 

 PaaS on YARN: an YARN application to demonstrate that YARN can be used as a 
 PaaS
 

 Key: MAPREDUCE-4393
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4393
 Project: Hadoop Map/Reduce
  Issue Type: Task
  Components: examples
Affects Versions: 0.23.1
Reporter: Jaigak Song
Assignee: Jaigak Song
 Fix For: 3.0.0

 Attachments: HADOOPasPAAS_Architecture.pdf, MAPREDUCE-4393.patch, 
 MAPREDUCE-4393.patch, MAPREDUCE-4393.patch, MAPREDUCE4393.patch, 
 MAPREDUCE4393.patch

   Original Estimate: 336h
  Remaining Estimate: 336h

 This application is to demonstrate that YARN can be used for non-mapreduce 
 applications. As Hadoop has already been adopted and deployed widely and its 
 deployment in future will be highly increased, we thought that it's a good 
 potential to be used as PaaS.  
 I have implemented a proof of concept to demonstrate that YARN can be used as 
 a PaaS (Platform as a Service). I have done a gap analysis against VMware's 
 Cloud Foundry and tried to achieve as many PaaS functionalities as possible 
 on YARN.
 I'd like to check in this POC as a YARN example application.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2

2012-07-12 Thread Alejandro Abdelnur (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413379#comment-13413379
 ] 

Alejandro Abdelnur commented on MAPREDUCE-3451:
---

+1

 Port Fair Scheduler to MR2
 --

 Key: MAPREDUCE-3451
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: mrv2, scheduler
Reporter: Patrick Wendell
Assignee: Patrick Wendell
 Attachments: MAPREDUCE-3451.v1.patch.txt, 
 MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, 
 MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, 
 MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, 
 MAPREDUCE-3451.v9.patch


 The Fair Scheduler is in widespread use today in MR1 clusters, but not yet 
 ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and 
 will be updated to include design considerations and progress.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4334) Add support for CPU isolation/monitoring of containers


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413380#comment-13413380
 ] 

Arun C Murthy commented on MAPREDUCE-4334:
--

Andrew, what are the security implications here? Does the NM need superuser 
privs to create/move cgroups?

 Add support for CPU isolation/monitoring of containers
 --

 Key: MAPREDUCE-4334
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4334
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Arun C Murthy
Assignee: Andrew Ferguson
 Attachments: MAPREDUCE-4334-pre1.patch, 
 MAPREDUCE-4334-pre2-with_cpu.patch, MAPREDUCE-4334-pre2.patch, 
 MAPREDUCE-4334-pre3-with_cpu.patch, MAPREDUCE-4334-pre3.patch


 Once we get in MAPREDUCE-4327, it will be important to actually enforce 
 limits on CPU consumption of containers. 
 Several options spring to mind:
 # taskset (RHEL5+)
 # cgroups (RHEL6+)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4351) Make ContainersMonitor pluggable


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413383#comment-13413383
 ] 

Arun C Murthy commented on MAPREDUCE-4351:
--

It seems like we have a couple of things we need to support:

a) Ability for NM to monitor containers for resource usage (e.g. a container 
might use only 1G even if it was allocated 2G etc.).
b) Ability for NM to enforce the resource limits.

Thus, it might be better for this to be called ContainerResourceEnforcer or 
some such? We'd like ContainerMonitor to merely monitor containers.

 Make ContainersMonitor pluggable
 

 Key: MAPREDUCE-4351
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4351
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: mrv2, nodemanager
Reporter: Andrew Ferguson
Assignee: Andrew Ferguson
 Attachments: MAPREDUCE-4351-v1.patch, MAPREDUCE-4351-v2.patch, 
 MAPREDUCE-4351-v3.patch, MAPREDUCE-4351-v4.patch, MAPREDUCE-4351-v4.patch, 
 MAPREDUCE-4351-v5.patch, MAPREDUCE-4351-v5.patch, MAPREDUCE-4351-v6.patch


 Make the existing ContainersManager pluggable, just as the ContainerExecutor 
 is currently. This will allow us to add container resource enforcement using 
 other techniques (such as cgroups) in an extensible fashion.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4351) Make ContainersMonitor pluggable


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun C Murthy updated MAPREDUCE-4351:
-

Status: Open  (was: Patch Available)

 Make ContainersMonitor pluggable
 

 Key: MAPREDUCE-4351
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4351
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: mrv2, nodemanager
Reporter: Andrew Ferguson
Assignee: Andrew Ferguson
 Attachments: MAPREDUCE-4351-v1.patch, MAPREDUCE-4351-v2.patch, 
 MAPREDUCE-4351-v3.patch, MAPREDUCE-4351-v4.patch, MAPREDUCE-4351-v4.patch, 
 MAPREDUCE-4351-v5.patch, MAPREDUCE-4351-v5.patch, MAPREDUCE-4351-v6.patch


 Make the existing ContainersManager pluggable, just as the ContainerExecutor 
 is currently. This will allow us to add container resource enforcement using 
 other techniques (such as cgroups) in an extensible fashion.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (MAPREDUCE-3451) Port Fair Scheduler to MR2

2012-07-12 Thread Alejandro Abdelnur (JIRA)


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alejandro Abdelnur resolved MAPREDUCE-3451.
---

   Resolution: Fixed
Fix Version/s: 2.0.1-alpha

Nice job, thanks Patrick. Committed to trunk and branch-2.

 Port Fair Scheduler to MR2
 --

 Key: MAPREDUCE-3451
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: mrv2, scheduler
Reporter: Patrick Wendell
Assignee: Patrick Wendell
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-3451.v1.patch.txt, 
 MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, 
 MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, 
 MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, 
 MAPREDUCE-3451.v9.patch


 The Fair Scheduler is in widespread use today in MR1 clusters, but not yet 
 ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and 
 will be updated to include design considerations and progress.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4334) Add support for CPU isolation/monitoring of containers

2012-07-12 Thread Andrew Ferguson (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413396#comment-13413396
 ] 

Andrew Ferguson commented on MAPREDUCE-4334:


@Arun:  no, the NM does not need superuser privs. in my comment above [1], the 
line $ sudo cgcreate -a hadoop_user_name -g memory:hadoop-yarn is run when 
installing Hadoop. This creates a branch of the memory hierarchy called 
hadoop-yarn which is owned by the user hadoop_user_name (which would be the 
user running the NM). This allows the NM to create and move cgroups without 
superuser privs.

The one complication is only the superuser or the owner of a process may move a 
process into a cgroup. As the LinuxContainerExecutor runs processes under 
different user accounts, we will need to either augment it, or use a similar 
tool to move such processes into a cgroup created by the NM user. 

Let me know if you'd like further clarification.



[1] 
https://issues.apache.org/jira/browse/MAPREDUCE-4334?focusedCommentId=13399014page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13399014

 Add support for CPU isolation/monitoring of containers
 --

 Key: MAPREDUCE-4334
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4334
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Arun C Murthy
Assignee: Andrew Ferguson
 Attachments: MAPREDUCE-4334-pre1.patch, 
 MAPREDUCE-4334-pre2-with_cpu.patch, MAPREDUCE-4334-pre2.patch, 
 MAPREDUCE-4334-pre3-with_cpu.patch, MAPREDUCE-4334-pre3.patch


 Once we get in MAPREDUCE-4327, it will be important to actually enforce 
 limits on CPU consumption of containers. 
 Several options spring to mind:
 # taskset (RHEL5+)
 # cgroups (RHEL6+)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413398#comment-13413398
 ] 

Hudson commented on MAPREDUCE-3451:
---

Integrated in Hadoop-Common-trunk-Commit #2460 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2460/])
MAPREDUCE-3451. Port Fair Scheduler to MR2 (pwendell via tucu) (Revision 
1361020)

 Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1361020
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/Resources.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerEventType.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfigurationException.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueSchedulable.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerApp.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerEventLog.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FifoAppComparator.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/NewJobWeightBooster.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/Schedulable.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingAlgorithms.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingMode.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/WeightAdjuster.java
*

[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413401#comment-13413401
 ] 

Hudson commented on MAPREDUCE-3451:
---

Integrated in Hadoop-Hdfs-trunk-Commit #2526 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2526/])
MAPREDUCE-3451. Port Fair Scheduler to MR2 (pwendell via tucu) (Revision 
1361020)

 Result = SUCCESS
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1361020
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/Resources.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerEventType.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfigurationException.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueSchedulable.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerApp.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerEventLog.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FifoAppComparator.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/NewJobWeightBooster.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/Schedulable.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingAlgorithms.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingMode.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/WeightAdjuster.java
*

[jira] [Commented] (MAPREDUCE-4351) Make ContainersMonitor pluggable

2012-07-12 Thread Andrew Ferguson (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413404#comment-13413404
 ] 

Andrew Ferguson commented on MAPREDUCE-4351:


indeed, ContainersMonitor is not a great name for the current class. in my 
mind, it has four jobs:

1) monitor resource usage
2) enforce resource limits
3) notify YARN of killed or failed tasks
4) delete temporary files created by container executor

while I see arguments to split each of these jobs into a separate class, I also 
see reasons to keep them together (a la the current ContainersMonitor): jobs #1 
and #3 are similar (poll /proc), jobs #1 and #2 are related (in the current 
implementation, the results of #1 are used for #2), and #2 and #4 are related, 
as you may need to revert some resource settings after the job is complete.

thoughts?

thanks!

 Make ContainersMonitor pluggable
 

 Key: MAPREDUCE-4351
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4351
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
  Components: mrv2, nodemanager
Reporter: Andrew Ferguson
Assignee: Andrew Ferguson
 Attachments: MAPREDUCE-4351-v1.patch, MAPREDUCE-4351-v2.patch, 
 MAPREDUCE-4351-v3.patch, MAPREDUCE-4351-v4.patch, MAPREDUCE-4351-v4.patch, 
 MAPREDUCE-4351-v5.patch, MAPREDUCE-4351-v5.patch, MAPREDUCE-4351-v6.patch


 Make the existing ContainersManager pluggable, just as the ContainerExecutor 
 is currently. This will allow us to add container resource enforcement using 
 other techniques (such as cgroups) in an extensible fashion.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413424#comment-13413424
 ] 

Hudson commented on MAPREDUCE-3451:
---

Integrated in Hadoop-Mapreduce-trunk-Commit #2479 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2479/])
MAPREDUCE-3451. Port Fair Scheduler to MR2 (pwendell via tucu) (Revision 
1361020)

 Result = FAILURE
tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1361020
Files : 
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/resource/Resources.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerEventType.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfigurationException.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AppSchedulable.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueueSchedulable.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSSchedulerApp.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerEventLog.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FifoAppComparator.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/NewJobWeightBooster.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/QueueManager.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/Schedulable.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingAlgorithms.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/SchedulingMode.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/WeightAdjuster.java
*

[jira] [Commented] (MAPREDUCE-4334) Add support for CPU isolation/monitoring of containers


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413438#comment-13413438
 ] 

Arun C Murthy commented on MAPREDUCE-4334:
--

Andrew, thanks, I missed that comment.

I'm concerned asking admins to setup croups etc. via cgcreate prior to 
deploying Hadoop clusters etc., particularly on all nodes, is almost a 
non-starter.

Preventing such onerous requirements on cluster setup is a key goal - something 
which initially led to creation of LinuxTaskController etc.

I'd strongly urge we implement this functionality via LinuxContainerExecutor - 
there-by allowing us to write low-level platform specific code (RHEL5 v/s RHEL6 
etc.) in a single place and not rely on tedious Java code for the same.

Thoughts?

 Add support for CPU isolation/monitoring of containers
 --

 Key: MAPREDUCE-4334
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4334
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Arun C Murthy
Assignee: Andrew Ferguson
 Attachments: MAPREDUCE-4334-pre1.patch, 
 MAPREDUCE-4334-pre2-with_cpu.patch, MAPREDUCE-4334-pre2.patch, 
 MAPREDUCE-4334-pre3-with_cpu.patch, MAPREDUCE-4334-pre3.patch


 Once we get in MAPREDUCE-4327, it will be important to actually enforce 
 limits on CPU consumption of containers. 
 Several options spring to mind:
 # taskset (RHEL5+)
 # cgroups (RHEL6+)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413457#comment-13413457
 ] 

Arun C Murthy commented on MAPREDUCE-3451:
--

bq. As far as I can tell - the idea behind SchedulerApp is to deal with 
application lifecycle management from the perspective of the scheduler. 

I wish I had a chance to respond to Patrick before this was committed... IAC, 
we should plan to allow the schedulers to deviate in significant ways (which is 
the whole point of having multiple schedulers) and, as a result, minimizing 
interference is a key goal.

Maybe we should open a jira to make SchedulerApp an interface?

I originally kept it common across FifoScheduler and CapacityScheduler since, 
for the degenerate case of a single queue, they are identical (currently) and 
planned to pull it out as a separate one once they deviated.

 Port Fair Scheduler to MR2
 --

 Key: MAPREDUCE-3451
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: mrv2, scheduler
Reporter: Patrick Wendell
Assignee: Patrick Wendell
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-3451.v1.patch.txt, 
 MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, 
 MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, 
 MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, 
 MAPREDUCE-3451.v9.patch


 The Fair Scheduler is in widespread use today in MR1 clusters, but not yet 
 ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and 
 will be updated to include design considerations and progress.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Updated] (MAPREDUCE-4432) Confusing warning message when GenericOptionsParser is not used


 [ 
https://issues.apache.org/jira/browse/MAPREDUCE-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Harsh J updated MAPREDUCE-4432:
---

 Target Version/s: 2.0.1-alpha
Affects Version/s: 2.0.0-alpha
 Hadoop Flags: Reviewed
   Status: Patch Available  (was: Open)

This looks good, thank you! +1 pending jenkins result.

 Confusing warning message when GenericOptionsParser is not used
 ---

 Key: MAPREDUCE-4432
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4432
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Gabriel Reid
Priority: Trivial
 Attachments: MAPREDUCE-4432.patch, MAPREDUCE-4432.patch


 The warning that is issued in JobSubmitter -- Use GenericOptionsParser for 
 parsing the arguments. Applications should implement Tool for the same. -- 
 is confusing and (probably) grammatically incorrect.
 This can be improved by having an updated warning message which gives clearer 
 directions on what can be improved in the application to avoid the warning in 
 the future.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413477#comment-13413477
 ] 

Patrick Wendell commented on MAPREDUCE-3451:


We have several pieces of follow-up around this, and one of them can be looking 
at moving SchedulerApp to an interface (this would involve changing the other 
two schedulers as well to some extent).

I'd like to see that change occur in tandem with better documentation for 
common scheduling components like SchedulerApp. The design goals of the RM 
scheduling code are currently opaque to anyone trying to add new code. Better 
documentation would make it clearer how to hook in a new scheduler the right 
way.

 Port Fair Scheduler to MR2
 --

 Key: MAPREDUCE-3451
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: mrv2, scheduler
Reporter: Patrick Wendell
Assignee: Patrick Wendell
 Fix For: 2.0.1-alpha

 Attachments: MAPREDUCE-3451.v1.patch.txt, 
 MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt, 
 MAPREDUCE-3451.v4.patch.txt, MAPREDUCE-3451.v5.patch, 
 MAPREDUCE-3451.v6.patch, MAPREDUCE-3451.v7.patch, MAPREDUCE-3451.v8.patch, 
 MAPREDUCE-3451.v9.patch


 The Fair Scheduler is in widespread use today in MR1 clusters, but not yet 
 ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and 
 will be updated to include design considerations and progress.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4432) Confusing warning message when GenericOptionsParser is not used

[
https://issues.apache.org/jira/browse/MAPREDUCE-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413480#comment-13413480
]

Hadoop QA commented on MAPREDUCE-4432:
--

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12536261/MAPREDUCE-4432.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 javac. The applied patch does not increase the total number of javac
compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.

+1 release audit. The applied patch does not increase the total number of
release audit warnings.

+1 core tests. The patch passed unit tests in
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core.

+1 contrib tests. The patch passed contrib unit tests.

Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2584//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2584//console

This message is automatically generated.

Confusing warning message when GenericOptionsParser is not used
---

Key: MAPREDUCE-4432
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4432
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Gabriel Reid
Priority: Trivial
Attachments: MAPREDUCE-4432.patch, MAPREDUCE-4432.patch

The warning that is issued in JobSubmitter -- Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the same. --
is confusing and (probably) grammatically incorrect.
This can be improved by having an updated warning message which gives clearer
directions on what can be improved in the application to avoid the warning in
the future.

[jira] [Commented] (MAPREDUCE-4432) Confusing warning message when GenericOptionsParser is not used


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413491#comment-13413491
 ] 

Gabriel Reid commented on MAPREDUCE-4432:
-

Justification for the lack of new tests on this patch is that the patch is only 
changing the wording of a warning message, and therefore not a candidate for 
automated testing.

 Confusing warning message when GenericOptionsParser is not used
 ---

 Key: MAPREDUCE-4432
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4432
 Project: Hadoop Map/Reduce
  Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Gabriel Reid
Priority: Trivial
 Attachments: MAPREDUCE-4432.patch, MAPREDUCE-4432.patch


 The warning that is issued in JobSubmitter -- Use GenericOptionsParser for 
 parsing the arguments. Applications should implement Tool for the same. -- 
 is confusing and (probably) grammatically incorrect.
 This can be improved by having an updated warning message which gives clearer 
 directions on what can be improved in the application to avoid the warning in 
 the future.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4417) add support for encrypted shuffle

2012-07-12 Thread Alejandro Abdelnur (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413498#comment-13413498
 ] 

Alejandro Abdelnur commented on MAPREDUCE-4417:
---

When looking at encryption on the wire for the shuffle the alternatives that 
popped up where transport encryption (HTTPS) and data/spills encryption (doable 
via a codec).

Using HTTPS requires improving the Fetcher/ShuffleHandler (Netty/JDK-URL) to 
use HTTPS and configuring certificates. It is a well understood/standard/proven 
technology and gives you end to end confidentiality, integrity, server 
authentication (and optionally client authentication), in an out of box manner 
without room to get things wrong. The server certificates private keys are out 
of reach from job tasks (they are used by the NM, similar to Kerberos keytabs). 

Using a codec, requires (leveraging a existing plugin point) a compression 
codec implementation that adds cipher-streams wrappers to the original streams 
and in addition could delegate to a real compression codec (in order not to 
lose compression if doing encryption). This requires us choosing a Cipher 
implementation by hand (which I'm not an expert on) and I'm not sure which one 
would be the best choice and what are the weaknesses of each one of them 
(http://en.wikipedia.org/wiki/Stream_cipher#Comparison_Of_Stream_Ciphers). 
Using a cipher on its own will provide confidentiality but it would not provide 
integrity or man-in-the-middle protection (unless we end up implementing 
something like TLS). In addition, both ends are controlled by job tasks, thus 
it becomes the responsibility of the user to create/distribute/protect the 
secrets that are basis of confidentiality. In addition, with the codec approach 
the HTTP shuffle requests/response headers go in the clear which could enable a 
man-in-the-middle attach.


 add support for encrypted shuffle
 -

 Key: MAPREDUCE-4417
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4417
 Project: Hadoop Map/Reduce
  Issue Type: New Feature
  Components: mrv2, security
Affects Versions: 2.0.0-alpha
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
 Fix For: 2.0.1-alpha


 Currently Shuffle fetches go on the clear. While Kerberos provides 
 comprehensive authentication for the cluster, it does not provide 
 confidentiality. 
 When processing sensitive data confidentiality may be desired (at the expense 
 of job performance and resources utilization for doing encryption).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4334) Add support for CPU isolation/monitoring of containers

2012-07-12 Thread Todd Lipcon (JIRA)


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413499#comment-13413499
 ] 

Todd Lipcon commented on MAPREDUCE-4334:


bq. Preventing such onerous requirements on cluster setup is a key goal - 
something which initially led to creation of LinuxTaskController etc.

How is it onerous? Packages could easily do this as part of the install on 
platforms where it's supported.

It seems equivalent to the installation of the LTC itself, which requires root 
to make it setuid, right?

Andrew: do the cgroups persist cross-reboot, or does that cgcreate command need 
to go the startup scripts?

 Add support for CPU isolation/monitoring of containers
 --

 Key: MAPREDUCE-4334
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4334
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Arun C Murthy
Assignee: Andrew Ferguson
 Attachments: MAPREDUCE-4334-pre1.patch, 
 MAPREDUCE-4334-pre2-with_cpu.patch, MAPREDUCE-4334-pre2.patch, 
 MAPREDUCE-4334-pre3-with_cpu.patch, MAPREDUCE-4334-pre3.patch


 Once we get in MAPREDUCE-4327, it will be important to actually enforce 
 limits on CPU consumption of containers. 
 Several options spring to mind:
 # taskset (RHEL5+)
 # cgroups (RHEL6+)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4334) Add support for CPU isolation/monitoring of containers


[ 
https://issues.apache.org/jira/browse/MAPREDUCE-4334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13413515#comment-13413515
 ] 

Arun C Murthy commented on MAPREDUCE-4334:
--

bq. How is it onerous? Packages could easily do this as part of the install on 
platforms where it's supported.

This doesn't make sense. What if CPU isolation is disabled? Do you still want 
'packages' to make it part of the install?

Clearly, we need to support taskset for platforms on which cgroups isn't 
supported e.g. RHEL5. For taskset you need super-user privs - would you prefer 
packages to do it too?

Yes, LTC is a pain, but using it consistently (e.g. for both cgroups and 
taskset) seems better than having multiple steps forced on the admin (LCE + 
cgroups + taskset etc.).

 Add support for CPU isolation/monitoring of containers
 --

 Key: MAPREDUCE-4334
 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4334
 Project: Hadoop Map/Reduce
  Issue Type: Sub-task
Reporter: Arun C Murthy
Assignee: Andrew Ferguson
 Attachments: MAPREDUCE-4334-pre1.patch, 
 MAPREDUCE-4334-pre2-with_cpu.patch, MAPREDUCE-4334-pre2.patch, 
 MAPREDUCE-4334-pre3-with_cpu.patch, MAPREDUCE-4334-pre3.patch


 Once we get in MAPREDUCE-4327, it will be important to actually enforce 
 limits on CPU consumption of containers. 
 Several options spring to mind:
 # taskset (RHEL5+)
 # cgroups (RHEL6+)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-4334) Add support for CPU isolation/monitoring of containers