[jira] [Updated] (MAPREDUCE-2984) Throwing NullPointerException when we open the container page
[
https://issues.apache.org/jira/browse/MAPREDUCE-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-2984:
-
Priority: Minor (was: Major)
Affects Version/s: (was: 0.24.0)
Fix Version/s: (was: 0.24.0)
> Throwing NullPointerException when we open the container page
> -
>
> Key: MAPREDUCE-2984
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2984
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
>Affects Versions: 0.23.0
>Reporter: Devaraj K
>Assignee: Devaraj K
>Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2984-1.patch, MAPREDUCE-2984.patch
>
>
> {code:xml}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.api.records.ContainerId.compareTo(ContainerId.java:97)
> at
> org.apache.hadoop.yarn.api.records.ContainerId.compareTo(ContainerId.java:23)
> at
> java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:819)
> at
> java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1640)
> at
> org.apache.hadoop.yarn.server.nodemanager.webapp.ContainerPage$ContainerBlock.render(ContainerPage.java:70)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:64)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:74)
> at org.apache.hadoop.yarn.webapp.View.render(View.java:210)
> {code}
> {code:xml}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.nodemanager.webapp.ContainerPage$ContainerBlock.render(ContainerPage.java:71)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:64)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:74)
> at org.apache.hadoop.yarn.webapp.View.render(View.java:210)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:47)
> at
> org.apache.hadoop.yarn.webapp.hamlet.HamletImpl$EImp._v(HamletImpl.java:117)
> at org.apache.hadoop.yarn.webapp.hamlet.Hamlet$TD._(Hamlet.java:843)
> at
> org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:54)
> at org.apache.hadoop.yarn.webapp.view.HtmlPage.render(HtmlPage.java:80)
> at org.apache.hadoop.yarn.webapp.Controller.render(Controller.java:210)
> at
> org.apache.hadoop.yarn.server.nodemanager.webapp.NMController.container(NMController.java:62)
> ... 30 more
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2984) Throwing NullPointerException when we open the container page
[
https://issues.apache.org/jira/browse/MAPREDUCE-2984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-2984:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Devaraj!
> Throwing NullPointerException when we open the container page
> -
>
> Key: MAPREDUCE-2984
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2984
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, nodemanager
>Affects Versions: 0.23.0
>Reporter: Devaraj K
>Assignee: Devaraj K
>Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2984-1.patch, MAPREDUCE-2984.patch
>
>
> {code:xml}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.api.records.ContainerId.compareTo(ContainerId.java:97)
> at
> org.apache.hadoop.yarn.api.records.ContainerId.compareTo(ContainerId.java:23)
> at
> java.util.concurrent.ConcurrentSkipListMap.doGet(ConcurrentSkipListMap.java:819)
> at
> java.util.concurrent.ConcurrentSkipListMap.get(ConcurrentSkipListMap.java:1640)
> at
> org.apache.hadoop.yarn.server.nodemanager.webapp.ContainerPage$ContainerBlock.render(ContainerPage.java:70)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:64)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:74)
> at org.apache.hadoop.yarn.webapp.View.render(View.java:210)
> {code}
> {code:xml}
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.nodemanager.webapp.ContainerPage$ContainerBlock.render(ContainerPage.java:71)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.render(HtmlBlock.java:64)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlBlock.renderPartial(HtmlBlock.java:74)
> at org.apache.hadoop.yarn.webapp.View.render(View.java:210)
> at
> org.apache.hadoop.yarn.webapp.view.HtmlPage$Page.subView(HtmlPage.java:47)
> at
> org.apache.hadoop.yarn.webapp.hamlet.HamletImpl$EImp._v(HamletImpl.java:117)
> at org.apache.hadoop.yarn.webapp.hamlet.Hamlet$TD._(Hamlet.java:843)
> at
> org.apache.hadoop.yarn.webapp.view.TwoColumnLayout.render(TwoColumnLayout.java:54)
> at org.apache.hadoop.yarn.webapp.view.HtmlPage.render(HtmlPage.java:80)
> at org.apache.hadoop.yarn.webapp.Controller.render(Controller.java:210)
> at
> org.apache.hadoop.yarn.server.nodemanager.webapp.NMController.container(NMController.java:62)
> ... 30 more
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2791) [MR-279] Missing/incorrect info on job -status CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2791: - Status: Open (was: Patch Available) Devaraj - sorry if I wasn't clear. In MRv2 there is no history url until the job completes. Once the job completes there is no 'tracking url' since the MR AppMaster goes away. Thus, we can use the same field i.e. trackingUrl without adding another one. Makes sense? > [MR-279] Missing/incorrect info on job -status CLI > --- > > Key: MAPREDUCE-2791 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2791 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Devaraj K >Priority: Blocker > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-2791-1.patch, MAPREDUCE-2791-1.patch, > MAPREDUCE-2791-2.patch, MAPREDUCE-2791.patch > > > There are a couple of details missing/incorrect on the job -status command > line output for completed jobs: > 1. Incorrect job file > 2. map() completion is always 0 > 3. reduce() completion is always set to 0 > 4. history URL is empty > 5. Missing launched map tasks > 6. Missing launched reduce tasks -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3098) Report Application status as well as ApplicationMaster status in GetApplicationReportResponse
[ https://issues.apache.org/jira/browse/MAPREDUCE-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3098: - Priority: Blocker (was: Major) Hitesh - while you are at this, do you want to go ahead and make 'finalState' and enum too? > Report Application status as well as ApplicationMaster status in > GetApplicationReportResponse > -- > > Key: MAPREDUCE-3098 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3098 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Blocker > Fix For: 0.23.0 > > > Currently, an application report received by the client from the RM/ASM for a > given application returns the status of the application master. It does not > return the status of the application i.e. whether that particular job > succeeded or failed. > The AM status would be one of FINISHED (SUCCEEDED should be renamed to > FINISHED as AM state does not indicate overall success/failure), FAILED or > KILLED. > The final state sent by the AM to the RM in the > FinishApplicationMasterRequest should be exposed to the client as > ApplicationState. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3098) Report Application status as well as ApplicationMaster status in GetApplicationReportResponse
[ https://issues.apache.org/jira/browse/MAPREDUCE-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3098: - Issue Type: Sub-task (was: Bug) Parent: MAPREDUCE-2894 > Report Application status as well as ApplicationMaster status in > GetApplicationReportResponse > -- > > Key: MAPREDUCE-3098 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3098 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Blocker > Fix For: 0.23.0 > > > Currently, an application report received by the client from the RM/ASM for a > given application returns the status of the application master. It does not > return the status of the application i.e. whether that particular job > succeeded or failed. > The AM status would be one of FINISHED (SUCCEEDED should be renamed to > FINISHED as AM state does not indicate overall success/failure), FAILED or > KILLED. > The final state sent by the AM to the RM in the > FinishApplicationMasterRequest should be exposed to the client as > ApplicationState. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2889) Add docs for writing new application frameworks
[ https://issues.apache.org/jira/browse/MAPREDUCE-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2889: - Issue Type: Sub-task (was: Improvement) Parent: MAPREDUCE-2890 > Add docs for writing new application frameworks > --- > > Key: MAPREDUCE-2889 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2889 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation, mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Priority: Critical > Fix For: 0.23.0 > > > We need to add docs for writing new application frameworks, including > examples, javadocs and sample apps. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2890) Documentation for MRv2
[ https://issues.apache.org/jira/browse/MAPREDUCE-2890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2890: - Comment: was deleted (was: Some info in the wiki (http://wiki.apache.org/hadoop/WritingYarnApps) we need to convert to site docs.) > Documentation for MRv2 > -- > > Key: MAPREDUCE-2890 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2890 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation, mrv2 >Reporter: Arun C Murthy >Priority: Blocker > Fix For: 0.23.0 > > > Let's use this jira to track docs for all of MRv2. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3098) Report Application status as well as ApplicationMaster status in GetApplicationReportResponse
[ https://issues.apache.org/jira/browse/MAPREDUCE-3098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3098: - Comment: was deleted (was: There is MAPREUDCE-2892, lets check with Anupam to ensure he hasn't started on it. Makes sense? Thanks.) > Report Application status as well as ApplicationMaster status in > GetApplicationReportResponse > -- > > Key: MAPREDUCE-3098 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3098 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Blocker > Fix For: 0.23.0 > > > Currently, an application report received by the client from the RM/ASM for a > given application returns the status of the application master. It does not > return the status of the application i.e. whether that particular job > succeeded or failed. > The AM status would be one of FINISHED (SUCCEEDED should be renamed to > FINISHED as AM state does not indicate overall success/failure), FAILED or > KILLED. > The final state sent by the AM to the RM in the > FinishApplicationMasterRequest should be exposed to the client as > ApplicationState. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3099) Add docs for setting up a single node MRv2 cluster.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3099: - Issue Type: Sub-task (was: Improvement) Parent: MAPREDUCE-2890 > Add docs for setting up a single node MRv2 cluster. > --- > > Key: MAPREDUCE-3099 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3099 > Project: Hadoop Map/Reduce > Issue Type: Sub-task >Affects Versions: 0.23.0 >Reporter: Mahadev konar >Assignee: Mahadev konar > Fix For: 0.23.0 > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3054) Unable to kill submitted jobs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-3054:
-
Status: Open (was: Patch Available)
javadoc warning
> Unable to kill submitted jobs
> -
>
> Key: MAPREDUCE-3054
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3054
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0
>Reporter: Siddharth Seth
>Assignee: Mahadev konar
>Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3054.patch, MAPREDUCE-3054.patch,
> MAPREDUCE-3054.patch, MAPREDUCE-3054.patch, MAPREDUCE-3054.patch,
> MAPREDUCE-3054.patch
>
>
> Found by Philip Su
> The "mapred job -kill" command
> appears to succeed, but listing the jobs again shows that the job supposedly
> killed is still there.
> {code}
> mapred job -list
> Total jobs:2
> JobId State StartTime UserNameQueue Priority
> SchedulingInfo
> job_1316203984216_0002 PREP1316204924937 hadoopqadefault NORMAL
> job_1316203984216_0001 PREP1316204031206 hadoopqadefault NORMAL
> mapred job -kill job_1316203984216_0002
> Killed job job_1316203984216_0002
> mapred job -list
> Total jobs:2
> JobId State StartTime UserNameQueue Priority
> SchedulingInfo
> job_1316203984216_0002 PREP1316204924937 hadoopqadefault NORMAL
> job_1316203984216_0001 PREP1316204031206 hadoopqadefault NORMAL
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3041) Enhance YARN Client-RM protocol to provide access to information such as cluster's Min/Max Resource capabilities similar to that of AM-RM protocol
[ https://issues.apache.org/jira/browse/MAPREDUCE-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3041: - Status: Open (was: Patch Available) +1 looks good. Minor nit - can we go ahead and rename getNewApplicationId to getNewApplication? Similarly for response/request too... thanks. > Enhance YARN Client-RM protocol to provide access to information such as > cluster's Min/Max Resource capabilities similar to that of AM-RM protocol > -- > > Key: MAPREDUCE-3041 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3041 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MR-3041.1.patch > > > To request a container to launch an application master, the client needs to > know the min/max resource capabilities so as to be able to make a proper > resource request when submitting a new application. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3067) Container exit status not set properly to launched process's exit code on successful completion of process
[ https://issues.apache.org/jira/browse/MAPREDUCE-3067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3067: - Resolution: Fixed Status: Resolved (was: Patch Available) +1, lgtm. I just committed this. Thanks Hitesh! > Container exit status not set properly to launched process's exit code on > successful completion of process > -- > > Key: MAPREDUCE-3067 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3067 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MR-3067.1.patch, MR-3067.2.patch > > > When testing the distributed shell sample app master, the container exit > status was being returned incorrectly. > 11/09/21 11:32:58 INFO DistributedShell.ApplicationMaster: Got container > status for containerID= container_1316629955324_0001_01_02, > state=COMPLETE, exitStatus=-1000, diagnostics= -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3041) Enhance YARN Client-RM protocol to provide access to information such as cluster's Min/Max Resource capabilities similar to that of AM-RM protocol
[ https://issues.apache.org/jira/browse/MAPREDUCE-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3041: - Status: Open (was: Patch Available) Hitesh, can you pls rebase? Can't apply the patch. Also, could you please run 'mvn install' to make sure the integration tests pass? Thanks. > Enhance YARN Client-RM protocol to provide access to information such as > cluster's Min/Max Resource capabilities similar to that of AM-RM protocol > -- > > Key: MAPREDUCE-3041 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3041 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MR-3041.1.patch, MR-3041.2.patch > > > To request a container to launch an application master, the client needs to > know the min/max resource capabilities so as to be able to make a proper > resource request when submitting a new application. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3021) all yarn webapps use same base name of "yarn/"
[ https://issues.apache.org/jira/browse/MAPREDUCE-3021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3021: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks Thomas! > all yarn webapps use same base name of "yarn/" > -- > > Key: MAPREDUCE-3021 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3021 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Thomas Graves > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3021.patch, MAPREDUCE-3021.patch > > > All of the yarn webapps (resource manager, node manager, app master, job > history) use the same base url of /yarn/. This doesn't lend itself very well > to filters be able to differentiate them to say allow some to be not > authenticated and other to be authenticated. Perhaps we should rename them > based on component. > There are also things in the code that hardcode paths to "/yarn" that should > be fixed up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2791) [MR-279] Missing/incorrect info on job -status CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2791: - Fix Version/s: (was: 0.24.0) Status: Open (was: Patch Available) Devaraj, the patch looks good. Thanks! Can you please check why the patch can't be applied by hudson? Maybe it's got to do with ^M? If not, can you please run: # mvn clean install to run all tests (incl. integration tests) and paste output? # AlPlease run old tests i.e ant test to make sure. Also, can you pls paste the latest output? Thanks! Meanwhile I'll run this on a secure cluster to test. > [MR-279] Missing/incorrect info on job -status CLI > --- > > Key: MAPREDUCE-2791 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2791 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Devaraj K >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2791-1.patch, MAPREDUCE-2791-1.patch, > MAPREDUCE-2791-2.patch, MAPREDUCE-2791-3.patch, MAPREDUCE-2791.patch > > > There are a couple of details missing/incorrect on the job -status command > line output for completed jobs: > 1. Incorrect job file > 2. map() completion is always 0 > 3. reduce() completion is always set to 0 > 4. history URL is empty > 5. Missing launched map tasks > 6. Missing launched reduce tasks -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3041) Enhance YARN Client-RM protocol to provide access to information such as cluster's Min/Max Resource capabilities similar to that of AM-RM protocol
[ https://issues.apache.org/jira/browse/MAPREDUCE-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3041: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks Hitesh! > Enhance YARN Client-RM protocol to provide access to information such as > cluster's Min/Max Resource capabilities similar to that of AM-RM protocol > -- > > Key: MAPREDUCE-3041 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3041 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MR-3041.1.patch, MR-3041.2.patch, MR-3041.3.patch > > > To request a container to launch an application master, the client needs to > know the min/max resource capabilities so as to be able to make a proper > resource request when submitting a new application. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3078) Application's progress isn't updated from AM to RM.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3078: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks Vinod! > Application's progress isn't updated from AM to RM. > --- > > Key: MAPREDUCE-3078 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3078 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, mrv2, resourcemanager >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3078-20110927.txt > > > It helps to be able to monitor the application-progress from the RM UI itself. > Bits of it is already there, even the AM-RM API (in AllocateRequest). We just > need to make sure the progress is produced and consumed properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2996) Log uberized information into JobHistory and use the same via CompletedJob
[ https://issues.apache.org/jira/browse/MAPREDUCE-2996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2996: - Status: Open (was: Patch Available) Cancelling patch while Sid's comments are being addressed... > Log uberized information into JobHistory and use the same via CompletedJob > -- > > Key: MAPREDUCE-2996 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2996 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Eagles >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2996.patch > > > We always print the uberized info on the UI to be false irrespective of > whether it is uberized or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2791) [MR-279] Missing/incorrect info on job -status CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2791: - Status: Open (was: Patch Available) > [MR-279] Missing/incorrect info on job -status CLI > --- > > Key: MAPREDUCE-2791 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2791 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Devaraj K >Priority: Blocker > Fix For: 0.24.0 > > Attachments: MAPREDUCE-2791-1.patch, MAPREDUCE-2791-1.patch, > MAPREDUCE-2791-2.patch, MAPREDUCE-2791-3.patch, MAPREDUCE-2791-4.patch, > MAPREDUCE-2791.patch, mvn install.log > > > There are a couple of details missing/incorrect on the job -status command > line output for completed jobs: > 1. Incorrect job file > 2. map() completion is always 0 > 3. reduce() completion is always set to 0 > 4. history URL is empty > 5. Missing launched map tasks > 6. Missing launched reduce tasks -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2791) [MR-279] Missing/incorrect info on job -status CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2791: - Fix Version/s: (was: 0.24.0) 0.23.0 Status: Open (was: Patch Available) Unfortunately, the patch doesn't apply anymore. I'll upload rebased one presently... > [MR-279] Missing/incorrect info on job -status CLI > --- > > Key: MAPREDUCE-2791 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2791 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Devaraj K >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2791-1.patch, MAPREDUCE-2791-1.patch, > MAPREDUCE-2791-2.patch, MAPREDUCE-2791-3.patch, MAPREDUCE-2791-4.patch, > MAPREDUCE-2791-5.patch, MAPREDUCE-2791.patch, mvn install.log > > > There are a couple of details missing/incorrect on the job -status command > line output for completed jobs: > 1. Incorrect job file > 2. map() completion is always 0 > 3. reduce() completion is always set to 0 > 4. history URL is empty > 5. Missing launched map tasks > 6. Missing launched reduce tasks -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2791) [MR-279] Missing/incorrect info on job -status CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2791: - Attachment: MAPREDUCE-2791.patch Rebased. I'll commit once hudson +1s it. Thanks for being patient Devaraj! > [MR-279] Missing/incorrect info on job -status CLI > --- > > Key: MAPREDUCE-2791 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2791 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Devaraj K >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2791-1.patch, MAPREDUCE-2791-1.patch, > MAPREDUCE-2791-2.patch, MAPREDUCE-2791-3.patch, MAPREDUCE-2791-4.patch, > MAPREDUCE-2791-5.patch, MAPREDUCE-2791.patch, MAPREDUCE-2791.patch, mvn > install.log > > > There are a couple of details missing/incorrect on the job -status command > line output for completed jobs: > 1. Incorrect job file > 2. map() completion is always 0 > 3. reduce() completion is always set to 0 > 4. history URL is empty > 5. Missing launched map tasks > 6. Missing launched reduce tasks -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2791) [MR-279] Missing/incorrect info on job -status CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2791: - Status: Patch Available (was: Open) > [MR-279] Missing/incorrect info on job -status CLI > --- > > Key: MAPREDUCE-2791 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2791 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Devaraj K >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2791-1.patch, MAPREDUCE-2791-1.patch, > MAPREDUCE-2791-2.patch, MAPREDUCE-2791-3.patch, MAPREDUCE-2791-4.patch, > MAPREDUCE-2791-5.patch, MAPREDUCE-2791.patch, MAPREDUCE-2791.patch, mvn > install.log > > > There are a couple of details missing/incorrect on the job -status command > line output for completed jobs: > 1. Incorrect job file > 2. map() completion is always 0 > 3. reduce() completion is always set to 0 > 4. history URL is empty > 5. Missing launched map tasks > 6. Missing launched reduce tasks -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2791) [MR-279] Missing/incorrect info on job -status CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2791: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks Devaraj! > [MR-279] Missing/incorrect info on job -status CLI > --- > > Key: MAPREDUCE-2791 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2791 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Devaraj K >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2791-1.patch, MAPREDUCE-2791-1.patch, > MAPREDUCE-2791-2.patch, MAPREDUCE-2791-3.patch, MAPREDUCE-2791-4.patch, > MAPREDUCE-2791-5.patch, MAPREDUCE-2791.patch, MAPREDUCE-2791.patch, mvn > install.log > > > There are a couple of details missing/incorrect on the job -status command > line output for completed jobs: > 1. Incorrect job file > 2. map() completion is always 0 > 3. reduce() completion is always set to 0 > 4. history URL is empty > 5. Missing launched map tasks > 6. Missing launched reduce tasks -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2996) Log uberized information into JobHistory and use the same via CompletedJob
[ https://issues.apache.org/jira/browse/MAPREDUCE-2996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2996: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks Jonathan! > Log uberized information into JobHistory and use the same via CompletedJob > -- > > Key: MAPREDUCE-2996 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2996 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Eagles >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2996.patch, MAPREDUCE-2996.patch, > MAPREDUCE-2996.patch > > > We always print the uberized info on the UI to be false irrespective of > whether it is uberized or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3056) Jobs are failing when those are submitted by other users
[ https://issues.apache.org/jira/browse/MAPREDUCE-3056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3056: - Status: Open (was: Patch Available) Agree with atm. Please provide a test case. Thanks! > Jobs are failing when those are submitted by other users > > > Key: MAPREDUCE-3056 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3056 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Devaraj K >Assignee: Devaraj K >Priority: Blocker > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-3056.patch > > > MR cluster is started by the user 'root'. If any other users other than > 'root' submit a job, it is failing always. > Find the conatiner logs in the comments section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3050) YarnScheduler needs to expose Resource Usage Information
[ https://issues.apache.org/jira/browse/MAPREDUCE-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3050: - Resolution: Fixed Fix Version/s: (was: 0.24.0) Target Version/s: 0.23.0 (was: 0.23.0, 0.24.0) Status: Resolved (was: Patch Available) I just committed this. Thanks Robert! > YarnScheduler needs to expose Resource Usage Information > > > Key: MAPREDUCE-3050 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3050 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, resourcemanager >Affects Versions: 0.23.0, 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MR-3050.txt, MR-3050.txt, MR-3050.txt, MR-3050.txt, > MR-3050.txt, MR-3050.txt, MR-3050.txt > > > Before the recent refactor The nodes had information in them about how much > resources they were using. This information is not hidden inside > SchedulerNode. Similarly resource usage information about an application, or > in aggregate is only available through the Scheduler and there is not > interface to pull it out. > We need to expose APIs to get Resource and Container information from the > scheduler, in aggregate across the entire cluster, per application, per node, > and ideally also per queue if applicable (although there are no JIRAs I am > aware of that need this right now). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2719) MR-279: Write a shell command application
[ https://issues.apache.org/jira/browse/MAPREDUCE-2719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2719: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks Hitesh! > MR-279: Write a shell command application > - > > Key: MAPREDUCE-2719 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2719 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Reporter: Sharad Agarwal >Assignee: Hitesh Shah > Fix For: 0.23.0 > > Attachments: MR-2179.1.patch, MR-2179.2.patch, MR-2179.3.patch, > MR-2719.4.post3098.patch, MR-2719.5.patch, MR-2719.6.patch, mr-2719.wip.patch > > > With nextgen hadoop (mrv2), it is simple to write non-MR applications. Write > an AplicationMaster (also corresponding simple client), to submit and run a > shell command application in the cluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3013) Remove YarnConfiguration.YARN_SECURITY_INFO
[ https://issues.apache.org/jira/browse/MAPREDUCE-3013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3013: - Status: Open (was: Patch Available) Sorry, you need to rebase this. > Remove YarnConfiguration.YARN_SECURITY_INFO > --- > > Key: MAPREDUCE-3013 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3013 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3013-20110930.txt > > > We don't need this anymore since RPC client uses SecurityUtil to pick it up > via going through the providers for SecurityInfo interface. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3113) the scripts yarn-daemon.sh and yarn are not working properly
[
https://issues.apache.org/jira/browse/MAPREDUCE-3113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-3113:
-
Resolution: Fixed
Fix Version/s: (was: 0.24.0)
0.23.0
Status: Resolved (was: Patch Available)
I just committed this. Thanks XieXianshan!
> the scripts yarn-daemon.sh and yarn are not working properly
>
>
> Key: MAPREDUCE-3113
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3113
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Affects Versions: 0.24.0
>Reporter: XieXianshan
>Assignee: XieXianshan
>Priority: Minor
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3113.patch
>
>
> When we execute them on any path but $YARN_HOME with bash -x option,it is
> giving the error as follows:
> (Of course we should set the path variable of that scritps into the .bashrc
> or profile in advance)
> {code}
> /usr/share/hadoop/hadoop-mapreduce-0.24.0-SNAPSHOT/bin/yarn: line 55:
> /usr/share/hadoop/yarn-config.sh: No such file or directory
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2889) Add docs for writing new application frameworks
[ https://issues.apache.org/jira/browse/MAPREDUCE-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2889: - Attachment: MAPREDUCE-2889.patch Looks great Hitesh, thanks! I just did minor edits, ensured 80-char limit etc. > Add docs for writing new application frameworks > --- > > Key: MAPREDUCE-2889 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2889 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation, mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Hitesh Shah >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2889.patch, MR-2889.1.patch, MR-2889.2.patch > > > We need to add docs for writing new application frameworks, including > examples, javadocs and sample apps. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2889) Add docs for writing new application frameworks
[ https://issues.apache.org/jira/browse/MAPREDUCE-2889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2889: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks Hitesh! > Add docs for writing new application frameworks > --- > > Key: MAPREDUCE-2889 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2889 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation, mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Hitesh Shah >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2889.patch, MR-2889.1.patch, MR-2889.2.patch > > > We need to add docs for writing new application frameworks, including > examples, javadocs and sample apps. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3132) Running a set of methods in a Single Test Class
[ https://issues.apache.org/jira/browse/MAPREDUCE-3132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3132: - Status: Open (was: Patch Available) Jonathan - this is great, thanks! Can I ask you to go ahead and make this change for all of Hadoop to be consistent? Thanks. > Running a set of methods in a Single Test Class > --- > > Key: MAPREDUCE-3132 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3132 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: build >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-3132.patch, MAPREDUCE-3132.patch > > > Instead of running every test method in a class, limit to specific testing > methods as describe in the link below. > http://maven.apache.org/plugins/maven-surefire-plugin/examples/single-test.html > Upgrade to the latest version of maven-surefire-plugin that has this feature. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3134) Add documentation for CapacityScheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3134: - Attachment: MAPREDUCE-3134.patch Docs for CS. > Add documentation for CapacityScheduler > --- > > Key: MAPREDUCE-3134 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3134 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation, mrv2, scheduler >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3134.patch > > > Add documentation for CapacityScheduler in MRv2 similar to > http://hadoop.apache.org/common/docs/stable/capacity_scheduler.html. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3134) Add documentation for CapacityScheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3134: - Status: Patch Available (was: Open) > Add documentation for CapacityScheduler > --- > > Key: MAPREDUCE-3134 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3134 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation, mrv2, scheduler >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3134.patch > > > Add documentation for CapacityScheduler in MRv2 similar to > http://hadoop.apache.org/common/docs/stable/capacity_scheduler.html. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3136) Add docs for setting up real-world MRv2 clusters
[ https://issues.apache.org/jira/browse/MAPREDUCE-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3136: - Priority: Blocker (was: Major) > Add docs for setting up real-world MRv2 clusters > > > Key: MAPREDUCE-3136 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3136 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation, mrv2 >Reporter: Arun C Murthy >Priority: Blocker > Fix For: 0.23.0 > > > Add docs for setting up real-world MRv2 clusters - MR portion of > http://hadoop.apache.org/common/docs/stable/cluster_setup.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3134) Add documentation for CapacityScheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3134: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. > Add documentation for CapacityScheduler > --- > > Key: MAPREDUCE-3134 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3134 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation, mrv2, scheduler >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3134.patch > > > Add documentation for CapacityScheduler in MRv2 similar to > http://hadoop.apache.org/common/docs/stable/capacity_scheduler.html. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3131) Docs and Scripts for setting up single node MRV2 cluster.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3131: - Status: Open (was: Patch Available) Prashanth, thanks for taking this up. No doubt, this is very useful for folks new to the project. Thinking a bit more, maybe we can add a bin/single-node directory where we check-in these scripts. To make them even more useful you can provide a way to run at least 2 (or more) DataNodes and 2 (or more) NodeManagers. You can also provide a documentation patch to hadoop-yarn-site/src/site/apt/SingleCluster.apt.vm. (mvn site:site to generate updated docs). Thoughts? > Docs and Scripts for setting up single node MRV2 cluster. > -- > > Key: MAPREDUCE-3131 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3131 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation, mrv2 >Affects Versions: 0.24.0 >Reporter: Prashant Sharma >Priority: Trivial > Labels: documentation, hadoop > Fix For: 0.24.0 > > Attachments: MAPREDUCE-3131.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > Scripts to run a single node cluster with a default configuration. Takes care > of running all the daemons including hdfs and yarn. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2792) [MR-279] Replace IP addresses with hostnames
[ https://issues.apache.org/jira/browse/MAPREDUCE-2792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2792: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks Vinod! > [MR-279] Replace IP addresses with hostnames > > > Key: MAPREDUCE-2792 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2792 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Vinod Kumar Vavilapalli >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2792-20111003.1.txt, > MAPREDUCE-2792-20111003.txt > > > Currently, all the logs, UI, CLI have IP addresses of the NM/RM, which are > difficult to manage. It will be useful to have hostnames like in 0.20.x for > easier debugging and maintenance purpose. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3138) Allow for applications to deal with MAPREDUCE-954
[ https://issues.apache.org/jira/browse/MAPREDUCE-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3138: - Status: Patch Available (was: Open) > Allow for applications to deal with MAPREDUCE-954 > - > > Key: MAPREDUCE-3138 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3138 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client, mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Owen O'Malley >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3138.patch > > > MAPREDUCE-954 changed the context-objs api to interfaces. This breaks Pig. We > need a bridge for them to move to 0.23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3138) Allow for applications to deal with MAPREDUCE-954
[ https://issues.apache.org/jira/browse/MAPREDUCE-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3138: - Attachment: MAPREDUCE-3138.patch Attaching patch on Owen's behalf. It's incredibly hard to unit test this without multiple versions of hadoop. Via PIG-2125 pig-dev@ have used this for nearly 2 months now and has no issues. > Allow for applications to deal with MAPREDUCE-954 > - > > Key: MAPREDUCE-3138 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3138 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client, mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Owen O'Malley >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3138.patch > > > MAPREDUCE-954 changed the context-objs api to interfaces. This breaks Pig. We > need a bridge for them to move to 0.23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3138) Allow for applications to deal with MAPREDUCE-954
[ https://issues.apache.org/jira/browse/MAPREDUCE-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3138: - Attachment: MAPREDUCE-3138.patch Minor update - added javadoc to the class itself. > Allow for applications to deal with MAPREDUCE-954 > - > > Key: MAPREDUCE-3138 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3138 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client, mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Owen O'Malley >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3138.patch, MAPREDUCE-3138.patch > > > MAPREDUCE-954 changed the context-objs api to interfaces. This breaks Pig. We > need a bridge for them to move to 0.23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3138) Allow for applications to deal with MAPREDUCE-954
[ https://issues.apache.org/jira/browse/MAPREDUCE-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3138: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks Owen! > Allow for applications to deal with MAPREDUCE-954 > - > > Key: MAPREDUCE-3138 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3138 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client, mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Owen O'Malley >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3138.patch, MAPREDUCE-3138.patch > > > MAPREDUCE-954 changed the context-objs api to interfaces. This breaks Pig. We > need a bridge for them to move to 0.23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2702: - Attachment: MAPREDUCE-2702.patch Sharad, I realize you are probably busy, so I went ahead and updated the patch with my comments. If are ok with it, I'll commit it. Thanks! > [MR-279] OutputCommitter changes for MR Application Master recovery > --- > > Key: MAPREDUCE-2702 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal >Priority: Blocker > Attachments: MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, > mr2702_v3.patch, mr2702_v4.patch > > > In MR AM recovers from a crash, it only reruns the non completed tasks. The > completed tasks (along with their output, if any) needs to be recovered from > the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2702: - Status: Patch Available (was: Open) > [MR-279] OutputCommitter changes for MR Application Master recovery > --- > > Key: MAPREDUCE-2702 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal >Priority: Blocker > Attachments: MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, > mr2702_v3.patch, mr2702_v4.patch > > > In MR AM recovers from a crash, it only reruns the non completed tasks. The > completed tasks (along with their output, if any) needs to be recovered from > the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[
https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-2702:
-
Status: Open (was: Patch Available)
Oh, just realized that the app-attempt dir was
outputPath/_temporary/${app-attempt-id} rather than
outputPath/_${app-attempt-id}, I'll fix that too.
> [MR-279] OutputCommitter changes for MR Application Master recovery
> ---
>
> Key: MAPREDUCE-2702
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2
>Reporter: Sharad Agarwal
>Assignee: Sharad Agarwal
>Priority: Blocker
> Attachments: MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch,
> mr2702_v3.patch, mr2702_v4.patch
>
>
> In MR AM recovers from a crash, it only reruns the non completed tasks. The
> completed tasks (along with their output, if any) needs to be recovered from
> the previous life. This would require some changes in OutputCommitter.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2702: - Status: Patch Available (was: Open) Never mind, I realized you do need a outputPath/_temporary to reliably blow away all outputs on job failures... > [MR-279] OutputCommitter changes for MR Application Master recovery > --- > > Key: MAPREDUCE-2702 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal >Priority: Blocker > Attachments: MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, > mr2702_v3.patch, mr2702_v4.patch > > > In MR AM recovers from a crash, it only reruns the non completed tasks. The > completed tasks (along with their output, if any) needs to be recovered from > the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3083) Distributed Cache Handling of public directories with private files
[ https://issues.apache.org/jira/browse/MAPREDUCE-3083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3083: - Priority: Blocker (was: Minor) Assignee: Siddharth Seth Sid, is this a blocker? > Distributed Cache Handling of public directories with private files > --- > > Key: MAPREDUCE-3083 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3083 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distributed-cache, mrv2 >Affects Versions: 0.23.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth >Priority: Blocker > > 0.23 distributed cache fails to localize with public directory containing > private files. 0.20 handles this a private directory. (seems to handles all > directories as private) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3127) Unable to restrict users based on resourcemanager.admin.acls value set
[ https://issues.apache.org/jira/browse/MAPREDUCE-3127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3127: - Priority: Blocker (was: Major) Assignee: Arun C Murthy > Unable to restrict users based on resourcemanager.admin.acls value set > -- > > Key: MAPREDUCE-3127 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3127 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Amol Kekre >Assignee: Arun C Murthy >Priority: Blocker > > Setting the following property in yarn-site.xml with user ids to restrict > ability to run > 'rmadmin -refreshQueues is not honoured > > yarn.server.resourcemanager.admin.acls > hadoop1 > > > > Should it be the same for rmadmin -refreshNodes? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3058) Sometimes task keeps on running while its Syslog says that it is shutdown
[
https://issues.apache.org/jira/browse/MAPREDUCE-3058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-3058:
-
Priority: Critical (was: Major)
> Sometimes task keeps on running while its Syslog says that it is shutdown
> -
>
> Key: MAPREDUCE-3058
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3058
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/gridmix, mrv2
>Affects Versions: 0.23.0
>Reporter: Karam Singh
>Assignee: Vinod Kumar Vavilapalli
>Priority: Critical
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3058-20110923.txt
>
>
> While running GridMixV3, one of the jobs got stuck for 15 hrs. After clicking
> on the Job-page, found one of its reduces to be stuck. Looking at syslog of
> the stuck reducer, found this:
> Task-logs' head:
> {code}
> 2011-09-19 17:57:22,002 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period
> at 10 second(s).
> 2011-09-19 17:57:22,002 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: ReduceTask metrics system
> started
> {code}
> Task-logs' tail:
> {code}
> 2011-09-19 18:06:49,818 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Bad connect ack with
> firstBadLink as
> 2011-09-19 18:06:49,818 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery
> for block
> BP-1405370709--1316452621953:blk_-7004355226367468317_79871 in
> pipeline , : bad datanode
> 2011-09-19 18:06:49,818 DEBUG
> org.apache.hadoop.hdfs.protocol.datatransfer.DataTransferProtocol:
> lastAckedSeqno = 26870
> 2011-09-19 18:06:49,820 DEBUG org.apache.hadoop.ipc.Client: IPC Client
> (26613121) connection to from gridperf sending #454
> 2011-09-19 18:06:49,826 DEBUG org.apache.hadoop.ipc.Client: IPC Client
> (26613121) connection to < from gridperf got value #454
> 2011-09-19 18:06:49,827 DEBUG org.apache.hadoop.ipc.RPC: Call:
> getAdditionalDatanode 8
> 2011-09-19 18:06:49,827 DEBUG org.apache.hadoop.hdfs.DFSClient: Connecting to
> datanode
> 2011-09-19 18:06:49,827 DEBUG org.apache.hadoop.hdfs.DFSClient: Send buf size
> 131071
> 2011-09-19 18:06:49,833 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
> Exception
> java.io.EOFException: Premature EOF: no length prefix available
> at
> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:158)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.transfer(DFSOutputStream.java:860)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:838)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:929)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:740)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:415)
> 2011-09-19 18:06:49,837 WARN org.apache.hadoop.mapred.YarnChild: Exception
> running child : java.io.EOFException: Premature EOF: no length prefix
> available
> at
> org.apache.hadoop.hdfs.protocol.HdfsProtoUtil.vintPrefixed(HdfsProtoUtil.java:158)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.transfer(DFSOutputStream.java:860)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:838)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:929)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:740)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:415)
> 2011-09-19 18:06:49,837 DEBUG org.apache.hadoop.ipc.Client: IPC Client
> (26613121) connection to from job_1316452677984_0862 sending #455
> 2011-09-19 18:06:49,839 DEBUG org.apache.hadoop.ipc.Client: IPC Client
> (26613121) connection to from job_1316452677984_0862 got value
> #455
> 2011-09-19 18:06:49,840 DEBUG org.apache.hadoop.ipc.RPC: Call: statusUpdate 3
> 2011-09-19 18:06:49,840 INFO org.apache.hadoop.mapred.Task: Runnning cleanup
> for the task
> 2011-09-19 18:06:49,840 DEBUG org.apache.hadoop.ipc.Client: IPC Client
> (26613121) connection to from gridperf sending #456
> 2011-09-19 18:06:49,858 DEBUG org.apache.hadoop.ipc.Client: IPC Client
> (26613121) connection to from gridperf got value #456
> 2011-09-19 18:06:49,858 DEBUG org.apache.hadoop.ipc.RPC: Call: delete 18
> 2011-09-19 18:06:49,858 DEBUG org.apache.hadoop.ipc.Client: IPC Client
> (26613121) connection to from job_1316452677984_0862 sending #457
> 2011-0
[jira] [Updated] (MAPREDUCE-2666) MR-279: Need to retrieve shuffle port number on AplicationMaster restart
[ https://issues.apache.org/jira/browse/MAPREDUCE-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2666: - Priority: Blocker (was: Major) > MR-279: Need to retrieve shuffle port number on AplicationMaster restart > > > Key: MAPREDUCE-2666 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2666 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.0 > > > MAPREDUCE-2652 allows ShuffleHandler to return the port it is operating on. > In the case of an ApplicationMaster crash where it needs to be restarted that > information is lost. We either need to re-query it from each of the > NodeManagers or to persist it to the JobHistory logs and retrieve it again. > The job history logs is probably the simpler solution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2788) LeafQueue.assignContainer() can cause a crash if request.getCapability().getMemory() == 0
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2788: - Priority: Critical (was: Major) > LeafQueue.assignContainer() can cause a crash if > request.getCapability().getMemory() == 0 > - > > Key: MAPREDUCE-2788 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Ahmed Radwan >Assignee: Ahmed Radwan >Priority: Critical > Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch > > > The assignContainer() method in > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue > can cause the scheduler to crash if the ResourseRequest capability memory == > 0 (divide by zero). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2702: - Status: Open (was: Patch Available) > [MR-279] OutputCommitter changes for MR Application Master recovery > --- > > Key: MAPREDUCE-2702 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal >Priority: Blocker > Attachments: MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, > mr2702_v3.patch, mr2702_v4.patch > > > In MR AM recovers from a crash, it only reruns the non completed tasks. The > completed tasks (along with their output, if any) needs to be recovered from > the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2702: - Status: Patch Available (was: Open) > [MR-279] OutputCommitter changes for MR Application Master recovery > --- > > Key: MAPREDUCE-2702 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal >Priority: Blocker > Attachments: MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, > mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch > > > In MR AM recovers from a crash, it only reruns the non completed tasks. The > completed tasks (along with their output, if any) needs to be recovered from > the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2702: - Attachment: MAPREDUCE-2702.patch Updated patch, I've incorporated f/b on better docs for the api. Regarding return value v/s exception, it's odd to throw an exception when it's expected that many impls won't deal with it - a default impl of throwing an exception is odd too - I've kept it as-is. I've changed names of private apis from get(Job,Task)TempDirName to get(Job,Task)AttemptBaseDirName. As Sharad suggested, I'll open a new jira for the old api. > [MR-279] OutputCommitter changes for MR Application Master recovery > --- > > Key: MAPREDUCE-2702 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal >Priority: Blocker > Attachments: MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, > mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch > > > In MR AM recovers from a crash, it only reruns the non completed tasks. The > completed tasks (along with their output, if any) needs to be recovered from > the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2702: - Status: Open (was: Patch Available) Good point Sharad, I missed that. I'll add it back. > [MR-279] OutputCommitter changes for MR Application Master recovery > --- > > Key: MAPREDUCE-2702 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal >Priority: Blocker > Attachments: MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, > mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, mr2702_v4.patch > > > In MR AM recovers from a crash, it only reruns the non completed tasks. The > completed tasks (along with their output, if any) needs to be recovered from > the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2702: - Attachment: MAPREDUCE-2702.patch Added isRecoverySupported. > [MR-279] OutputCommitter changes for MR Application Master recovery > --- > > Key: MAPREDUCE-2702 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, > MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, > mr2702_v4.patch > > > In MR AM recovers from a crash, it only reruns the non completed tasks. The > completed tasks (along with their output, if any) needs to be recovered from > the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2702) [MR-279] OutputCommitter changes for MR Application Master recovery
[ https://issues.apache.org/jira/browse/MAPREDUCE-2702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2702: - Fix Version/s: 0.23.0 Status: Patch Available (was: Open) > [MR-279] OutputCommitter changes for MR Application Master recovery > --- > > Key: MAPREDUCE-2702 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2702 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2 >Reporter: Sharad Agarwal >Assignee: Sharad Agarwal >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2702.patch, MAPREDUCE-2702.patch, > MAPREDUCE-2702.patch, mr2702_v1.patch, mr2702_v2.patch, mr2702_v3.patch, > mr2702_v4.patch > > > In MR AM recovers from a crash, it only reruns the non completed tasks. The > completed tasks (along with their output, if any) needs to be recovered from > the previous life. This would require some changes in OutputCommitter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3068: - Component/s: mrv2 Priority: Blocker (was: Major) > Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers > --- > > Key: MAPREDUCE-3068 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Chris Riccomini >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff > > > This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should > all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3148) Port MAPREDUCE-2702 to old mapred api
[ https://issues.apache.org/jira/browse/MAPREDUCE-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3148: - Priority: Blocker (was: Major) > Port MAPREDUCE-2702 to old mapred api > - > > Key: MAPREDUCE-3148 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3148 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Fix For: 0.23.0 > > > Port MAPREDUCE-2702 to old mapred api -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3148) Port MAPREDUCE-2702 to old mapred api
[ https://issues.apache.org/jira/browse/MAPREDUCE-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3148: - Attachment: MAPREDUCE-3148.patch Done, need to add tests. > Port MAPREDUCE-2702 to old mapred api > - > > Key: MAPREDUCE-3148 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3148 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3148.patch > > > Port MAPREDUCE-2702 to old mapred api -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3087) CLASSPATH not the same after MAPREDUCE-2880
[ https://issues.apache.org/jira/browse/MAPREDUCE-3087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3087: - Status: Open (was: Patch Available) Ravi, thanks for digging through this. Patch looks ok, not sure if there is a better way. OTOH, I don't get the changes to pom deps? Also, pls fix the javac warning. Thanks. > CLASSPATH not the same after MAPREDUCE-2880 > --- > > Key: MAPREDUCE-3087 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3087 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ravi Prakash >Assignee: Ravi Prakash > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3087.patch, MAPREDUCE-3087.patch, > notworking.classpath.uniq, working.classpath.uniq > > > After MAPREDUCE-2880, my classpath was missing key jar files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3033) JobClient requires mapreduce.jobtracker.address config even when mapreduce.framework.name is set to yarn
[
https://issues.apache.org/jira/browse/MAPREDUCE-3033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-3033:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Hitesh!
> JobClient requires mapreduce.jobtracker.address config even when
> mapreduce.framework.name is set to yarn
>
>
> Key: MAPREDUCE-3033
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3033
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: job submission, mrv2
>Affects Versions: 0.23.0
>Reporter: Karam Singh
>Assignee: Hitesh Shah
>Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MR-3033.1.patch, MR-3033.2.patch
>
>
> If mapreduce.jobtracker.address is not set in mapred-site.xml and
> mapreduce.framework.name is set yarn, job submission fails :
> Tried to submit sleep job with maps 1 task. Job submission failed with
> following exception -:
> {code}
> 11/09/19 13:19:20 INFO ipc.YarnRPC: Creating YarnRPC for
> org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC
> 11/09/19 13:19:20 INFO mapred.ResourceMgrDelegate: Connecting to
> ResourceManager at :8040
> 11/09/19 13:19:20 INFO ipc.HadoopYarnRPC: Creating a HadoopYarnProtoRpc proxy
> for protocol interface org.apache.hadoop.yarn.api.ClientRMProtocol
> 11/09/19 13:19:20 INFO mapred.ResourceMgrDelegate: Connected to
> ResourceManager at :8040
> 11/09/19 13:19:21 INFO mapred.ResourceMgrDelegate: DEBUG ---
> getStagingAreaDir: dir=/user//.staging
> 11/09/19 13:19:21 INFO mapreduce.JobSubmitter: Cleaning up the staging area
> /user//.staging/job_1316435926198_0004
> java.lang.RuntimeException: Not a host:port pair: local
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:148)
> at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:132)
> at org.apache.hadoop.mapred.Master.getMasterAddress(Master.java:42)
> at org.apache.hadoop.mapred.Master.getMasterPrincipal(Master.java:47)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:104)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:90)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:83)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:346)
> at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1072)
> at org.apache.hadoop.mapreduce.Job$2.run(Job.java:1069)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1135)
> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1069)
> at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1089)
> at org.apache.hadoop.mapreduce.SleepJob.run(SleepJob.java:262)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
> at org.apache.hadoop.mapreduce.SleepJob.main(SleepJob.java:194)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
> at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
> at
> org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
> at
> org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:189)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2802) [MR-279] Jobhistory filenames should have jobID to help in better parsing
[ https://issues.apache.org/jira/browse/MAPREDUCE-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2802: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks Jonathan! (Sid - thanks for the review too!) > [MR-279] Jobhistory filenames should have jobID to help in better parsing > -- > > Key: MAPREDUCE-2802 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2802 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Jonathan Eagles >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2802.patch > > > For jobID such as job_1312933838300_0007, jobhistory file names are named as > job%5F1312933838300%5F0007__ramya___1_1_SUCCEEDED.jhist > It would be easier for parsing if the jobIDs were a part of the filenames. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2789) [MR:279] Update the scheduling info on CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2789: - Status: Open (was: Patch Available) Eric, sorry it took me this long to review this. Some comments: # Rename JobReport to ApplicationResourceUsageReport or some such - we have tried to avoid 'job' in YARN # Move the JobReport generation to RMAppAttemptImpl and call that from RMAppImpl # I don't get why you have readFields/writeFields in org.apache.hadoop.yarn.api.records.JobResourceInfo interface - we should avoid this. None of the other interface in org.apache.hadoop.yarn.api.records have this. > [MR:279] Update the scheduling info on CLI > -- > > Key: MAPREDUCE-2789 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2789 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Eric Payne > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-2789.v1.txt > > > The scheduling information such as number of containers running, memory usage > and reservations per job is not available on bin/mapred job -list CLI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2876) ContainerAllocationExpirer appears to use the incorrect configs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2876: - Target Version/s: 0.23.0 (was: 0.24.0, 0.23.0) Affects Version/s: (was: 0.24.0) Status: Open (was: Patch Available) Cancelling patch while Mahadev's comments are addressed. > ContainerAllocationExpirer appears to use the incorrect configs > --- > > Key: MAPREDUCE-2876 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2876 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Robert Joseph Evans >Assignee: Anupam Seth >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2876-branch-0_23.patch > > > ContainerAllocationExpirer sets the expiration interval to be > RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL but uses > AMLIVELINESS_MONITORING_INTERVAL as the interval. This is very different > from what AMLivelinessMonitor does. > There should be two configs RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL > for the monitoring interval and RMConfig.CONTAINER_EXPIRY_INTERVAL for the > expiry. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2876) ContainerAllocationExpirer appears to use the incorrect configs
[ https://issues.apache.org/jira/browse/MAPREDUCE-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2876: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks Anupam! > ContainerAllocationExpirer appears to use the incorrect configs > --- > > Key: MAPREDUCE-2876 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2876 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Robert Joseph Evans >Assignee: Anupam Seth >Priority: Critical > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2876-branch-0_23.patch, > MAPREDUCE-2876-branch-0_23.patch > > > ContainerAllocationExpirer sets the expiration interval to be > RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL but uses > AMLIVELINESS_MONITORING_INTERVAL as the interval. This is very different > from what AMLivelinessMonitor does. > There should be two configs RMConfig.CONTAINER_LIVELINESS_MONITORING_INTERVAL > for the monitoring interval and RMConfig.CONTAINER_EXPIRY_INTERVAL for the > expiry. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3153) TestFileOutputCommitter.testFailAbort() is failing on trunk on Jenkins
[ https://issues.apache.org/jira/browse/MAPREDUCE-3153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3153: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks Mahadev! > TestFileOutputCommitter.testFailAbort() is failing on trunk on Jenkins > -- > > Key: MAPREDUCE-3153 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3153 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, test >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Mahadev konar > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3153.patch > > > This mostly is caused by MAPREDUCE-2702. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3123) Symbolic links with special chars causing container/task.sh to fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-3123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3123: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks Hitesh! > Symbolic links with special chars causing container/task.sh to fail > --- > > Key: MAPREDUCE-3123 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3123 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Hitesh Shah >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MR-3123.1.patch, MR-3123.2.patch, MR-3123.3.patch > > > the following job throws an exception when you have the special characters in > it. > hadoop jar hadoop-streaming.jar -Dmapreduce.job.acl-view-job=* > -Dmapreduce.job.queuename=queue1 -files > file:///homes/user/hadoop/Streaming/data/streaming-980//InputDir#testlink!@$&*()-_+= > -input Streaming/streaming-980/input.txt -mapper 'xargs cat' -reducer cat > -output Streaming/streaming-980/Output -jobconf > mapred.job.name=streamingTest-980 -jobconf mapreduce.job.acl-view-job=* > Exception: > 2011-09-27 20:58:48,903 INFO > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: > launchContainer: > [container-executor, hadoopuser, 1, application_1317077272567_0239, > container_1317077272567 0239_01_01, > tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01, > tmp/mapred-local/nmPrivate/application_1317077272567_0239/container_1317077272567_0239_01 > 01/task.sh, > tmp/mapred-local/nmPrivate/container_1317077272567_0239_01_01/container_1317077272567_0239_01_01.tokens]110922-tests.jar:hadoop-mapreduce-p2011-09-27 > 20:58:48,944 WARN > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code > from container is : 2 > > 2011-09-27 > 20:58:48,946 WARN > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exception > from container-launch : > > > org.apache.hadoop.util.Shell$ExitCodeException: > /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01/task.sh: > line 26: syntax error near unexpected token `-_+=' > /tmp/mapred-local/usercache/hadoopuser/appcache/application_1317077272567_0239/container_1317077272567_0239_01_01/task.sh: > line 26: `ln -sf > /tmp/mapred-local/usercache/hadoopqa/filecache/-1888139433818483070/InputDir > test > ink!@$&*()-_+='kson-jaxrs-1.7.1.jar:/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.7.3/jackson-mapper-asl-1.7.3.jar:.m2/repository/org/codehaus/jackson/jackson-xc/1.7.1/jackson-xc-1.7.1.jar: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:261) > > > >at org.apache.hadoop.util.Shell.run(Shell.java:188) > > > > at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:381) > > > > at > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:174) > > > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:197) > > > > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:62) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:619) >
[jira] [Updated] (MAPREDUCE-3154) Validate the Jobs Output Specification as the first statement in JobSubmitter.submitJobInternal(Job, Cluster) method
[
https://issues.apache.org/jira/browse/MAPREDUCE-3154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-3154:
-
Resolution: Fixed
Fix Version/s: 0.23.0
Status: Resolved (was: Patch Available)
I just committed this. Thanks Abhijit!
> Validate the Jobs Output Specification as the first statement in
> JobSubmitter.submitJobInternal(Job, Cluster) method
>
>
> Key: MAPREDUCE-3154
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3154
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client, mrv2
>Affects Versions: 0.23.0, 0.24.0
> Environment: mrv2
>Reporter: Abhijit Suresh Shingate
>Assignee: Abhijit Suresh Shingate
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3154.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> Presently the output specification is validated after getting new JobId from
> ClientRMService, Copying the job jar, Configuration file, archives etc.
> Instead of that move following Job Output specification validation call to
> the begining of JobSubmitter.submitJobInternal(Job, Cluster) method.
> {code}
> checkSpecs(job);
> {code}
> This will avoid unnecessary work in case of invalid output specs.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3158) Fix trunk build failures
[ https://issues.apache.org/jira/browse/MAPREDUCE-3158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3158: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks Hitesh! > Fix trunk build failures > > > Key: MAPREDUCE-3158 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3158 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah >Assignee: Hitesh Shah > Fix For: 0.23.0 > > Attachments: MR-3158.1.patch > > > https://builds.apache.org/view/G-L/view/Hadoop/job/Hadoop-Mapreduce-trunk-Commit/1060/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3148) Port MAPREDUCE-2702 to old mapred api
[ https://issues.apache.org/jira/browse/MAPREDUCE-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3148: - Component/s: mrv2 > Port MAPREDUCE-2702 to old mapred api > - > > Key: MAPREDUCE-3148 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3148 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3148.patch > > > Port MAPREDUCE-2702 to old mapred api -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3148) Port MAPREDUCE-2702 to old mapred api
[ https://issues.apache.org/jira/browse/MAPREDUCE-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3148: - Status: Patch Available (was: Open) > Port MAPREDUCE-2702 to old mapred api > - > > Key: MAPREDUCE-3148 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3148 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3148.patch, MAPREDUCE-3148.patch > > > Port MAPREDUCE-2702 to old mapred api -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3148) Port MAPREDUCE-2702 to old mapred api
[ https://issues.apache.org/jira/browse/MAPREDUCE-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3148: - Attachment: MAPREDUCE-3148.patch Updated patch with tests - also ensured MRv1 tests work too. > Port MAPREDUCE-2702 to old mapred api > - > > Key: MAPREDUCE-3148 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3148 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3148.patch, MAPREDUCE-3148.patch > > > Port MAPREDUCE-2702 to old mapred api -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3148) Port MAPREDUCE-2702 to old mapred api
[ https://issues.apache.org/jira/browse/MAPREDUCE-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3148: - Status: Open (was: Patch Available) Need to fix findbugs warnings via an exclude... this is the standard old v/s new api mess. > Port MAPREDUCE-2702 to old mapred api > - > > Key: MAPREDUCE-3148 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3148 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3148.patch, MAPREDUCE-3148.patch > > > Port MAPREDUCE-2702 to old mapred api -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3148) Port MAPREDUCE-2702 to old mapred api
[ https://issues.apache.org/jira/browse/MAPREDUCE-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3148: - Attachment: MAPREDUCE-3148.patch Fixed findbugs warnings. > Port MAPREDUCE-2702 to old mapred api > - > > Key: MAPREDUCE-3148 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3148 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3148.patch, MAPREDUCE-3148.patch, > MAPREDUCE-3148.patch > > > Port MAPREDUCE-2702 to old mapred api -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3148) Port MAPREDUCE-2702 to old mapred api
[ https://issues.apache.org/jira/browse/MAPREDUCE-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3148: - Status: Patch Available (was: Open) > Port MAPREDUCE-2702 to old mapred api > - > > Key: MAPREDUCE-3148 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3148 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3148.patch, MAPREDUCE-3148.patch, > MAPREDUCE-3148.patch > > > Port MAPREDUCE-2702 to old mapred api -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3126) mr job stuck because reducers using all slots and mapper isn't scheduled
[ https://issues.apache.org/jira/browse/MAPREDUCE-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3126: - Status: Patch Available (was: Open) > mr job stuck because reducers using all slots and mapper isn't scheduled > > > Key: MAPREDUCE-3126 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3126 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Arun C Murthy >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3126.patch > > > The command in MAPREDUCE-3124 run and this job got hung with 1 Map task > waiting for resources and 7 Reducers running (2 waiting). The mapper got > scheduler, then AM scheduled the reducers, the map task failed and tried to > start a new attempt but reducers were using all the slots. > I will try to add some more info from the logs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3126) mr job stuck because reducers using all slots and mapper isn't scheduled
[ https://issues.apache.org/jira/browse/MAPREDUCE-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3126: - Attachment: MAPREDUCE-3126.patch Patch to fix corner case in CS. > mr job stuck because reducers using all slots and mapper isn't scheduled > > > Key: MAPREDUCE-3126 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3126 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Arun C Murthy >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3126.patch > > > The command in MAPREDUCE-3124 run and this job got hung with 1 Map task > waiting for resources and 7 Reducers running (2 waiting). The mapper got > scheduler, then AM scheduled the reducers, the map task failed and tried to > start a new attempt but reducers were using all the slots. > I will try to add some more info from the logs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3126) mr job stuck because reducers using all slots and mapper isn't scheduled
[ https://issues.apache.org/jira/browse/MAPREDUCE-3126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3126: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. > mr job stuck because reducers using all slots and mapper isn't scheduled > > > Key: MAPREDUCE-3126 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3126 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Arun C Murthy >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3126.patch > > > The command in MAPREDUCE-3124 run and this job got hung with 1 Map task > waiting for resources and 7 Reducers running (2 waiting). The mapper got > scheduler, then AM scheduled the reducers, the map task failed and tried to > start a new attempt but reducers were using all the slots. > I will try to add some more info from the logs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3148) Port MAPREDUCE-2702 to old mapred api
[ https://issues.apache.org/jira/browse/MAPREDUCE-3148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3148: - Resolution: Fixed Status: Resolved (was: Patch Available) Since this was a straight backport I committed this. > Port MAPREDUCE-2702 to old mapred api > - > > Key: MAPREDUCE-3148 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3148 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3148.patch, MAPREDUCE-3148.patch, > MAPREDUCE-3148.patch > > > Port MAPREDUCE-2702 to old mapred api -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2789) [MR:279] Update the scheduling info on CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2789: - Status: Open (was: Patch Available) Eric, we shouldn't use the *PBImpl directly - we need to go through the RecordFactory. Also, the readLock is imp. - we may not need it right now, but it's important to be future proof, someone might access fields in RMAppAttemptImpl there eventually... There are also some 80+ char lines. > [MR:279] Update the scheduling info on CLI > -- > > Key: MAPREDUCE-2789 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2789 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Eric Payne > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-2789.v1.txt, MAPREDUCE-2789.v2.txt, > MAPREDUCE-2789.v3.txt, MAPREDUCE-2789.v4.txt > > > The scheduling information such as number of containers running, memory usage > and reservations per job is not available on bin/mapred job -list CLI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2789) [MR:279] Update the scheduling info on CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2789: - Status: Patch Available (was: Open) > [MR:279] Update the scheduling info on CLI > -- > > Key: MAPREDUCE-2789 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2789 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Eric Payne > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-2789.patch, MAPREDUCE-2789.v1.txt, > MAPREDUCE-2789.v2.txt, MAPREDUCE-2789.v3.txt, MAPREDUCE-2789.v4.txt > > > The scheduling information such as number of containers running, memory usage > and reservations per job is not available on bin/mapred job -list CLI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2789) [MR:279] Update the scheduling info on CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2789: - Attachment: MAPREDUCE-2789.patch Patch incorporating my own minor nits. > [MR:279] Update the scheduling info on CLI > -- > > Key: MAPREDUCE-2789 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2789 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Eric Payne > Fix For: 0.23.0, 0.24.0 > > Attachments: MAPREDUCE-2789.patch, MAPREDUCE-2789.v1.txt, > MAPREDUCE-2789.v2.txt, MAPREDUCE-2789.v3.txt, MAPREDUCE-2789.v4.txt > > > The scheduling information such as number of containers running, memory usage > and reservations per job is not available on bin/mapred job -list CLI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2666) MR-279: Need to retrieve shuffle port number on ApplicationMaster restart
[ https://issues.apache.org/jira/browse/MAPREDUCE-2666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2666: - Resolution: Fixed Target Version/s: 0.23.0 (was: 0.24.0, 0.23.0) Status: Resolved (was: Patch Available) I just committed this. Thanks Jonathan! Thanks to Bobby and Vinod for reviews. > MR-279: Need to retrieve shuffle port number on ApplicationMaster restart > - > > Key: MAPREDUCE-2666 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2666 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Robert Joseph Evans >Assignee: Jonathan Eagles >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2666.patch, MAPREDUCE-2666.patch > > > MAPREDUCE-2652 allows ShuffleHandler to return the port it is operating on. > In the case of an ApplicationMaster crash where it needs to be restarted that > information is lost. We either need to re-query it from each of the > NodeManagers or to persist it to the JobHistory logs and retrieve it again. > The job history logs is probably the simpler solution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2789) [MR:279] Update the scheduling info on CLI
[ https://issues.apache.org/jira/browse/MAPREDUCE-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2789: - Resolution: Fixed Fix Version/s: (was: 0.24.0) Status: Resolved (was: Patch Available) I just committed this. Thanks Eric! > [MR:279] Update the scheduling info on CLI > -- > > Key: MAPREDUCE-2789 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2789 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Eric Payne > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2789.patch, MAPREDUCE-2789.v1.txt, > MAPREDUCE-2789.v2.txt, MAPREDUCE-2789.v3.txt, MAPREDUCE-2789.v4.txt > > > The scheduling information such as number of containers running, memory usage > and reservations per job is not available on bin/mapred job -list CLI. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3170) Trunk nightly commit builds are failing.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3170: - Component/s: mrv1 build Affects Version/s: (was: 0.23.0) 0.22.0 Fix Version/s: (was: 0.23.0) 0.22.0 > Trunk nightly commit builds are failing. > > > Key: MAPREDUCE-3170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3170 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build, mrv1 >Affects Versions: 0.22.0 >Reporter: Mahadev konar >Assignee: Hitesh Shah >Priority: Critical > Fix For: 0.22.0 > > Attachments: MR-3170.1.patch, MR-3170.2.patch > > > Looks like the trunk commit builds are failing after MAPREDUCE-3148 and > MAPREDUCE-3126 were committed. I suspect its MAPREDUCE-3148. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3170) Trunk nightly commit builds are failing.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3170: - Component/s: mrv2 Affects Version/s: (was: 0.22.0) 0.23.0 Fix Version/s: (was: 0.22.0) 0.23.0 > Trunk nightly commit builds are failing. > > > Key: MAPREDUCE-3170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3170 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build, mrv1, mrv2 >Affects Versions: 0.23.0 >Reporter: Mahadev konar >Assignee: Hitesh Shah >Priority: Critical > Fix For: 0.23.0 > > Attachments: MR-3170.1.patch, MR-3170.2.patch > > > Looks like the trunk commit builds are failing after MAPREDUCE-3148 and > MAPREDUCE-3126 were committed. I suspect its MAPREDUCE-3148. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3170) Trunk nightly commit builds are failing.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3170: - Resolution: Fixed Status: Resolved (was: Patch Available) I just committed this. Thanks Hitesh! > Trunk nightly commit builds are failing. > > > Key: MAPREDUCE-3170 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3170 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build, mrv1, mrv2 >Affects Versions: 0.23.0 >Reporter: Mahadev konar >Assignee: Hitesh Shah >Priority: Critical > Fix For: 0.23.0 > > Attachments: MR-3170.1.patch, MR-3170.2.patch > > > Looks like the trunk commit builds are failing after MAPREDUCE-3148 and > MAPREDUCE-3126 were committed. I suspect its MAPREDUCE-3148. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3174) app master UI goes away when app finishes - not very user friendly
[ https://issues.apache.org/jira/browse/MAPREDUCE-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3174: - Priority: Major (was: Blocker) I'm not sure this is a blocker - yes we can improve this, but doesn't block a release. > app master UI goes away when app finishes - not very user friendly > -- > > Key: MAPREDUCE-3174 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3174 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves > > A user can go to the application master UI to see the stats on the app, but > as soon as the app finishes that UI goes away and user is left with nothing. > A redirect to history server or similar would be much better. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3152) Miscellaneous web UI issues
[ https://issues.apache.org/jira/browse/MAPREDUCE-3152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3152: - Priority: Major (was: Blocker) > Miscellaneous web UI issues > --- > > Key: MAPREDUCE-3152 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3152 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli > Fix For: 0.23.0 > > > We need to fix the following issues on YARN web-UI: > - Remove the "Note" column from the application list. When a failure > happens, this "Note" spoils the table layout. > - When the Application is still not running, the Tracking UI should be title > "UNASSIGNED", for some reason it is titled "ApplicationMaster" but > (correctly) links to "#". > - The per-application page has all the RM related information like version, > start-time etc. Must be some accidental change by one of the patches. > - The diagnostics for a failed app on the per-application page don't retain > new lines and wrap'em around - looks hard to read. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3124) mapper failed with failed to load native libs
[
https://issues.apache.org/jira/browse/MAPREDUCE-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-3124:
-
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this trivial change. Thanks John!
> mapper failed with failed to load native libs
> -
>
> Key: MAPREDUCE-3124
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3124
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Thomas Graves
>Assignee: John George
>Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-3124.patch, MAPREDUCE-3124.patch,
> MAPREDUCE-3124.patch
>
>
> hadoop jar hadoop-mapreduce-examples-*.jar sort -Dmapreduce.job.acl-view
> -job=* -Dmapreduce.map.output.compress=true
> -Dmapreduce.map.output.compress.codec=org.apache.hadoop.io.compress.GzipCodec
> -Dmapreduce.output.fileoutputformat.compress=true
> -Dmapreduce.output.fileoutputformat.compression.type=NONE -Dmap
> reduce.output.fileoutputformat.compression.codec=org.apache.hadoop.io.compress.GzipCodec
> -outKey
> org.apache.hadoop.io.Text -outValue org.apache.hadoop.io.Text
> Compression/textinput Compression/textoutput-1317315994
> This will fail with native libs not found error unless
> -Dmapred.child.java.opts='-Djava.library.path=${HADOOP_COMMON_HOME}/lib/native/Linux-i386-32'
> is added.
> The error in container log:
> 2011-09-29 17:06:56,787 DEBUG org.apache.hadoop.util.NativeCodeLoader: Trying
> to load the custom-built native-hadoop
> library...2011-09-29 17:06:56,787 DEBUG
> org.apache.hadoop.util.NativeCodeLoader: Failed to load native-hadoop with
> error: java.lang.UnsatisfiedLinkError: no hadoop in
> java.library.path2011-09-29 17:06:56,787 DEBUG
> org.apache.hadoop.util.NativeCodeLoader:
> java.library.path=/share/gridjdk-1.6.0_21/jre/lib/i386/server:/share/gridjdk-1.6.0_21/jre/lib/i386:/share/gridjdk-1.6.0_21/jre/../lib/i386:/tmp/mapred-local/usercache/hadoopqa/appcache/application_1317314754104_0012/container_1317314754104_0012_01_02:/current/lib:/usr/java/packages/lib/i386:/lib:/usr/lib2011-09-29
> 17:06:56,787 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform...
> using builtin-java classes where applicable
> Also note that the error that shows up at the application master for this is
> terrible:
> Container killed by the ApplicationMaster. Container killed on request. Exit
> code is 137 Too Many fetch failures.Failing the attempt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2775) [MR-279] Decommissioned node does not shutdown
[ https://issues.apache.org/jira/browse/MAPREDUCE-2775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2775: - Status: Open (was: Patch Available) Cancelling patch while Vinod's comments are addressed. > [MR-279] Decommissioned node does not shutdown > -- > > Key: MAPREDUCE-2775 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2775 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ramya Sunil >Assignee: Devaraj K >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2775-1.patch, MAPREDUCE-2775.patch > > > A Nodemanager which is decommissioned by an admin via refreshnodes does not > automatically shutdown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3068) Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers
[ https://issues.apache.org/jira/browse/MAPREDUCE-3068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3068: - Status: Open (was: Patch Available) Chris - this looks great, thanks for the nice refactor to use it in both NM and MR apps. One nit is that we want the 'putIfAbsent' semantics i.e. allow containers to override HADOOP_*_HOME etc. Could you please fix it? Also, can you please look at the javadoc warnings? Thanks! > Should set MALLOC_ARENA_MAX for all YARN daemons and AMs/Containers > --- > > Key: MAPREDUCE-3068 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3068 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Chris Riccomini >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3068.diff, MAPREDUCE-3068.diff, > MAPREDUCE-3068.diff > > > This is same as HADOOP-7154 but for yarn. RM, NM, AM and containers should > all have this. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3173) MRV2 UI doesn't work properly without internet
[
https://issues.apache.org/jira/browse/MAPREDUCE-3173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated MAPREDUCE-3173:
-
Priority: Major (was: Critical)
Not sure this is a blocker.
> MRV2 UI doesn't work properly without internet
> --
>
> Key: MAPREDUCE-3173
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3173
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.0, 0.24.0
>Reporter: Devaraj K
>Assignee: Devaraj K
>
> When we try access the MRV2 UI, it is always giving the below message in the
> UI even if the java script enabled in the browser.
> {code:xml}
> This page works best with javascript enabled.
> {code}
> It is trying to download these below css/js files from internet and finally
> ending up with the above message. For loading the page also it is taking long
> time.
> {code:title=JQueryUI.java|borderStyle=solid}
> html.
>
> link(join("https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.9/themes/";,
> getTheme(), "/jquery-ui.css")).
> link("/static/dt-1.7.5/css/jui-dt.css").
>
> script("https://ajax.googleapis.com/ajax/libs/jquery/1.4.4/jquery.min.js";).
>
> script("https://ajax.googleapis.com/ajax/libs/jqueryui/1.8.9/jquery-ui.min.js";).
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3144) Augment JobHistory to include information needed for serving aggregated logs.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3144: - Priority: Critical (was: Major) > Augment JobHistory to include information needed for serving aggregated logs. > - > > Key: MAPREDUCE-3144 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3144 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Siddharth Seth >Priority: Critical > Fix For: 0.23.0 > > Attachments: MR3144.patch, MR3144_v2.txt, MR3144_v3.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3136) Add docs for setting up real-world MRv2 clusters
[ https://issues.apache.org/jira/browse/MAPREDUCE-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3136: - Attachment: MAPREDUCE-3136.patch WIP patch for preview. > Add docs for setting up real-world MRv2 clusters > > > Key: MAPREDUCE-3136 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3136 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation, mrv2 >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3136.patch > > > Add docs for setting up real-world MRv2 clusters - MR portion of > http://hadoop.apache.org/common/docs/stable/cluster_setup.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3165) Ensure logging option is set on child command line
[ https://issues.apache.org/jira/browse/MAPREDUCE-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3165: - Fix Version/s: 0.23.0 Status: Open (was: Patch Available) Cancelling patch since Todd is improving it a tad. > Ensure logging option is set on child command line > -- > > Key: MAPREDUCE-3165 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3165 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster, mrv2 >Reporter: Arun C Murthy >Assignee: Todd Lipcon >Priority: Blocker > Fix For: 0.23.0 > > Attachments: mr-3165.txt > > > Currently the logging config is set in env in MapReduceChildJVM - we need to > set it on command line. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2788) LeafQueue.assignContainer() can cause a crash if request.getCapability().getMemory() == 0
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2788: - Status: Open (was: Patch Available) Ahmed - not sure when this happened, but looks like CS.allocate already normalizes the incoming request. Thus, we might only need to fix FifoScheduler.normalizeRequest? We should pull out normalizeRequest to SchedulerUtils and use it from both places? > LeafQueue.assignContainer() can cause a crash if > request.getCapability().getMemory() == 0 > - > > Key: MAPREDUCE-2788 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Ahmed Radwan >Assignee: Ahmed Radwan >Priority: Critical > Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, > MAPREDUCE-2788_rev3.patch > > > The assignContainer() method in > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue > can cause the scheduler to crash if the ResourseRequest capability memory == > 0 (divide by zero). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2788) Normalize requests in FifoScheduler.allocate to prevent NPEs later
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-2788: - Priority: Major (was: Critical) Summary: Normalize requests in FifoScheduler.allocate to prevent NPEs later (was: LeafQueue.assignContainer() can cause a crash if request.getCapability().getMemory() == 0) Changing the title to reflect the existing bug. > Normalize requests in FifoScheduler.allocate to prevent NPEs later > -- > > Key: MAPREDUCE-2788 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Ahmed Radwan >Assignee: Ahmed Radwan > Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, > MAPREDUCE-2788_rev3.patch > > > The assignContainer() method in > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue > can cause the scheduler to crash if the ResourseRequest capability memory == > 0 (divide by zero). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3136) Add docs for setting up real-world MRv2 clusters
[ https://issues.apache.org/jira/browse/MAPREDUCE-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3136: - Attachment: MAPREDUCE-3136.patch Completed, need to add more info. on secure clusters. > Add docs for setting up real-world MRv2 clusters > > > Key: MAPREDUCE-3136 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3136 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: documentation, mrv2 >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Fix For: 0.23.0 > > Attachments: MAPREDUCE-3136.patch, MAPREDUCE-3136.patch > > > Add docs for setting up real-world MRv2 clusters - MR portion of > http://hadoop.apache.org/common/docs/stable/cluster_setup.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
