[jira] [Updated] (MAPREDUCE-4169) Container Logs appear in unsorted order
[ https://issues.apache.org/jira/browse/MAPREDUCE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-4169: --- Target Version/s: 0.23.3, 2.0.0 (was: 2.0.0, 0.23.3) Status: Patch Available (was: Open) > Container Logs appear in unsorted order > --- > > Key: MAPREDUCE-4169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4169 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Minor > Attachments: MAPREDUCE-4169.patch > > > container logs (stdout, stderr, syslog) in the nodemanager ui and jobhistory > ui appear in unsorted order where the order displayed is based on what file > was created first. This jira will have the results be displayed in a > consistent order. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4169) Container Logs appear in unsorted order
[ https://issues.apache.org/jira/browse/MAPREDUCE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-4169: --- Attachment: MAPREDUCE-4169.patch > Container Logs appear in unsorted order > --- > > Key: MAPREDUCE-4169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4169 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Minor > Attachments: MAPREDUCE-4169.patch > > > container logs (stdout, stderr, syslog) in the nodemanager ui and jobhistory > ui appear in unsorted order where the order displayed is based on what file > was created first. This jira will have the results be displayed in a > consistent order. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4154) streaming MR job succeeds even if the streaming command fails
[ https://issues.apache.org/jira/browse/MAPREDUCE-4154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Owen O'Malley updated MAPREDUCE-4154: - Target Version/s: 1.1.0, 1.0.3 (was: 1.1.0) I think we should put this into 1.0.3. Thoughts? > streaming MR job succeeds even if the streaming command fails > - > > Key: MAPREDUCE-4154 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4154 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 1.0.2 >Reporter: Thejas M Nair >Assignee: Devaraj Das > Fix For: 1.1.0 > > Attachments: streaming.patch > > > Hadoop 1.0.1 behaves as expected - The task fails for streaming MR job if the > streaming command fails. But it succeeds in hadoop 1.0.2 . -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4026) Lower minimum-allocation-mb to sensible defaults
[ https://issues.apache.org/jira/browse/MAPREDUCE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4026: --- Attachment: MAPREDUCE-4026.patch Only TestRMWebServices failure seems related to this. Fixed that test to not use hardcodes either. Fixed test passes now. The other failures stem out of port binding issues (mis-closed min-YARN clusters?) and a couple of others with kerberos-related issues. > Lower minimum-allocation-mb to sensible defaults > > > Key: MAPREDUCE-4026 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, scheduler >Affects Versions: 2.0.0 >Reporter: Harsh J >Assignee: Harsh J > Attachments: MAPREDUCE-4026.patch, MAPREDUCE-4026.patch, > MAPREDUCE-4026.patch, MAPREDUCE-4026.patch > > > The CapacityScheduler's minimum-allocation-mb is set to 1024. > The FIFO's minimum-allocation-mb meanwhile, is 128. > I propose changing the formers' minimum to that amount as well. 1024 is way > too much as a default, wastes "slots" on NMs - and I also do not see why CS > has to deviate that settings from the FIFO default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4026) Lower minimum-allocation-mb to sensible defaults
[ https://issues.apache.org/jira/browse/MAPREDUCE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4026: --- Attachment: MAPREDUCE-4026.patch Ah earlier patch lacked all sub commits. Re-upping a new one. > Lower minimum-allocation-mb to sensible defaults > > > Key: MAPREDUCE-4026 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, scheduler >Affects Versions: 2.0.0 >Reporter: Harsh J >Assignee: Harsh J > Attachments: MAPREDUCE-4026.patch, MAPREDUCE-4026.patch, > MAPREDUCE-4026.patch > > > The CapacityScheduler's minimum-allocation-mb is set to 1024. > The FIFO's minimum-allocation-mb meanwhile, is 128. > I propose changing the formers' minimum to that amount as well. 1024 is way > too much as a default, wastes "slots" on NMs - and I also do not see why CS > has to deviate that settings from the FIFO default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4026) Lower minimum-allocation-mb to sensible defaults
[ https://issues.apache.org/jira/browse/MAPREDUCE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4026: --- Attachment: MAPREDUCE-4026.patch Didn't realize there was a hardcoded test in TestFifoScheduler. Fixed test to not hardcode now, and rather reuse a constant. > Lower minimum-allocation-mb to sensible defaults > > > Key: MAPREDUCE-4026 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, scheduler >Affects Versions: 2.0.0 >Reporter: Harsh J >Assignee: Harsh J > Attachments: MAPREDUCE-4026.patch, MAPREDUCE-4026.patch > > > The CapacityScheduler's minimum-allocation-mb is set to 1024. > The FIFO's minimum-allocation-mb meanwhile, is 128. > I propose changing the formers' minimum to that amount as well. 1024 is way > too much as a default, wastes "slots" on NMs - and I also do not see why CS > has to deviate that settings from the FIFO default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4027) Document the minimum-allocation-mb configurations
[ https://issues.apache.org/jira/browse/MAPREDUCE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4027: --- Attachment: MAPREDUCE-4027.patch Docs inline with MAPREDUCE-4026 changes. > Document the minimum-allocation-mb configurations > - > > Key: MAPREDUCE-4027 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4027 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 0.23.3 >Reporter: Harsh J >Assignee: Harsh J >Priority: Minor > Attachments: MAPREDUCE-4027.patch > > > None of the current yarn.scheduler.fifo.minimum-allocation-mb and > yarn.scheduler.capacity.minimum-allocation-mb are documented anywhere. > Without knowledge of these params, one can't change the default allocations. > And the default allocations are pretty high btw (MAPREDUCE-4026). > We should document these in the Cluster Setup page at least. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4027) Document the minimum-allocation-mb configurations
[ https://issues.apache.org/jira/browse/MAPREDUCE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4027: --- Target Version/s: 2.0.0, trunk (was: trunk, 2.0.0) Status: Patch Available (was: Open) > Document the minimum-allocation-mb configurations > - > > Key: MAPREDUCE-4027 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4027 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 0.23.3 >Reporter: Harsh J >Assignee: Harsh J >Priority: Minor > Attachments: MAPREDUCE-4027.patch > > > None of the current yarn.scheduler.fifo.minimum-allocation-mb and > yarn.scheduler.capacity.minimum-allocation-mb are documented anywhere. > Without knowledge of these params, one can't change the default allocations. > And the default allocations are pretty high btw (MAPREDUCE-4026). > We should document these in the Cluster Setup page at least. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4026) Lower minimum-allocation-mb to sensible defaults
[ https://issues.apache.org/jira/browse/MAPREDUCE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4026: --- Target Version/s: 2.0.0 (was: trunk, 2.0.0) Status: Patch Available (was: Open) > Lower minimum-allocation-mb to sensible defaults > > > Key: MAPREDUCE-4026 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, scheduler >Affects Versions: 0.23.3 >Reporter: Harsh J >Assignee: Harsh J > Attachments: MAPREDUCE-4026.patch > > > The CapacityScheduler's minimum-allocation-mb is set to 1024. > The FIFO's minimum-allocation-mb meanwhile, is 128. > I propose changing the formers' minimum to that amount as well. 1024 is way > too much as a default, wastes "slots" on NMs - and I also do not see why CS > has to deviate that settings from the FIFO default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4026) Lower minimum-allocation-mb to sensible defaults
[ https://issues.apache.org/jira/browse/MAPREDUCE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4026: --- Affects Version/s: (was: 0.23.3) 2.0.0 > Lower minimum-allocation-mb to sensible defaults > > > Key: MAPREDUCE-4026 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, scheduler >Affects Versions: 2.0.0 >Reporter: Harsh J >Assignee: Harsh J > Attachments: MAPREDUCE-4026.patch > > > The CapacityScheduler's minimum-allocation-mb is set to 1024. > The FIFO's minimum-allocation-mb meanwhile, is 128. > I propose changing the formers' minimum to that amount as well. 1024 is way > too much as a default, wastes "slots" on NMs - and I also do not see why CS > has to deviate that settings from the FIFO default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4026) Lower minimum-allocation-mb to sensible defaults
[ https://issues.apache.org/jira/browse/MAPREDUCE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated MAPREDUCE-4026: --- Attachment: MAPREDUCE-4026.patch Patch that lowers the min-alloc MB to 128 in both existing FIFO and CS schedulers. > Lower minimum-allocation-mb to sensible defaults > > > Key: MAPREDUCE-4026 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, scheduler >Affects Versions: 2.0.0 >Reporter: Harsh J >Assignee: Harsh J > Attachments: MAPREDUCE-4026.patch > > > The CapacityScheduler's minimum-allocation-mb is set to 1024. > The FIFO's minimum-allocation-mb meanwhile, is 128. > I propose changing the formers' minimum to that amount as well. 1024 is way > too much as a default, wastes "slots" on NMs - and I also do not see why CS > has to deviate that settings from the FIFO default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3674) If invoked with no queueName request param, jobqueue_details.jsp injects a null queue name into schedulers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated MAPREDUCE-3674:
---
Target Version/s: (was: 1.1.0)
> If invoked with no queueName request param, jobqueue_details.jsp injects a
> null queue name into schedulers.
> ---
>
> Key: MAPREDUCE-3674
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3674
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Affects Versions: 1.0.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Critical
> Fix For: 1.1.0
>
> Attachments: MAPREDUCE-3674.patch, MAPREDUCE-3674.patch,
> MAPREDUCE-3674.patch, MAPREDUCE-3674.patch, hadoop-findbugs-report.html
>
>
> When you access /jobqueue_details.jsp manually, instead of via a link, it has
> queueName set to null internally and this goes for a lookup into the
> scheduling info maps as well.
> As a result, if using FairScheduler, a Pool with String name = null gets
> created and this brings the scheduler down. I have not tested what happens to
> the CapacityScheduler, but ideally if no queueName is set in that jsp, it
> should fall back to 'default'. Otherwise, this brings down the JobTracker
> completely.
> FairScheduler must also add a check to not create a pool with 'null' name.
> The following is the strace that ensues:
> {code}
> ERROR org.mortbay.log: /jobqueue_details.jsp
> java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.jobqueue_005fdetails_jsp._jspService(jobqueue_005fdetails_jsp.java:71)
>
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>
> at
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:829)
>
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9001, call
> heartbeat from XYZ:MNOP: error: java.io.IOException:
> java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:95)
>
> at
> org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:68)
>
> at java.util.Arrays.mergeSort(Unknown Source)
> at java.util.Arrays.sort(Unknown Source)
> at java.util.Collections.sort(Unknown Source)
> at org.apache.hadoop.mapred.FairScheduler.assignTasks(FairScheduler.java:435)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3226)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)
> at java.security.AccessController.doPrivileged(Native
[jira] [Updated] (MAPREDUCE-3674) If invoked with no queueName request param, jobqueue_details.jsp injects a null queue name into schedulers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated MAPREDUCE-3674:
---
Resolution: Fixed
Fix Version/s: 1.1.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks for the review Tom and Todd, committed for 1.1.0.
> If invoked with no queueName request param, jobqueue_details.jsp injects a
> null queue name into schedulers.
> ---
>
> Key: MAPREDUCE-3674
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3674
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Affects Versions: 1.0.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Critical
> Fix For: 1.1.0
>
> Attachments: MAPREDUCE-3674.patch, MAPREDUCE-3674.patch,
> MAPREDUCE-3674.patch, MAPREDUCE-3674.patch, hadoop-findbugs-report.html
>
>
> When you access /jobqueue_details.jsp manually, instead of via a link, it has
> queueName set to null internally and this goes for a lookup into the
> scheduling info maps as well.
> As a result, if using FairScheduler, a Pool with String name = null gets
> created and this brings the scheduler down. I have not tested what happens to
> the CapacityScheduler, but ideally if no queueName is set in that jsp, it
> should fall back to 'default'. Otherwise, this brings down the JobTracker
> completely.
> FairScheduler must also add a check to not create a pool with 'null' name.
> The following is the strace that ensues:
> {code}
> ERROR org.mortbay.log: /jobqueue_details.jsp
> java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.jobqueue_005fdetails_jsp._jspService(jobqueue_005fdetails_jsp.java:71)
>
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>
> at
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:829)
>
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9001, call
> heartbeat from XYZ:MNOP: error: java.io.IOException:
> java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:95)
>
> at
> org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:68)
>
> at java.util.Arrays.mergeSort(Unknown Source)
> at java.util.Arrays.sort(Unknown Source)
> at java.util.Collections.sort(Unknown Source)
> at org.apache.hadoop.mapred.FairScheduler.assignTasks(FairScheduler.java:435)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3226)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
> at org.apache.hadoop.ipc.
[jira] [Updated] (MAPREDUCE-3193) FileInputFormat doesn't read files recursively in the input path dir
[ https://issues.apache.org/jira/browse/MAPREDUCE-3193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-3193: - Affects Version/s: 0.23.2 > FileInputFormat doesn't read files recursively in the input path dir > > > Key: MAPREDUCE-3193 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3193 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, mrv2 >Affects Versions: 0.23.2, 1.0.2, 2.0.0, 3.0.0 >Reporter: Ramgopal N >Assignee: Devaraj K > Attachments: MAPREDUCE-3193-1.patch, MAPREDUCE-3193-2.patch, > MAPREDUCE-3193-2.patch, MAPREDUCE-3193.patch, MAPREDUCE-3193.security.patch > > > java.io.FileNotFoundException is thrown,if input file is more than one folder > level deep and the job is getting failed. > Example:Input file is /r1/r2/input.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3451) Port Fair Scheduler to MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-3451: - Hadoop Flags: Reviewed Status: Patch Available (was: Open) > Port Fair Scheduler to MR2 > -- > > Key: MAPREDUCE-3451 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2, scheduler >Reporter: Patrick Wendell >Assignee: Patrick Wendell > Attachments: MAPREDUCE-3451.v1.patch.txt, > MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt > > > The Fair Scheduler is in widespread use today in MR1 clusters, but not yet > ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and > will be updated to include design considerations and progress. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3958) RM: Remove RMNodeState and replace it with NodeState
[ https://issues.apache.org/jira/browse/MAPREDUCE-3958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-3958: -- Attachment: MAPREDUCE-3958-2.patch Done > RM: Remove RMNodeState and replace it with NodeState > > > Key: MAPREDUCE-3958 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3958 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Bikas Saha >Assignee: Bikas Saha > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3958-1.patch, MAPREDUCE-3958-2.patch, > MAPREDUCE-3958.patch > > > RMNodeState is being sent over the wire after MAPREDUCE-3353. This has been > done by cloning the enum into NodeState in yarn protocol records. > That makes RMNodeState redundant and it should be replaced with NodeState. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4163) consistently set the bind address
[
https://issues.apache.org/jira/browse/MAPREDUCE-4163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated MAPREDUCE-4163:
---
Attachment: MAPREDUCE-3659-1.patch
Remove now defunct code.
> consistently set the bind address
> -
>
> Key: MAPREDUCE-4163
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4163
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.0, 0.24.0, 2.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: MAPREDUCE-3659-1.patch, MAPREDUCE-3659.patch
>
>
> Use {{NetUtils.getConnectAddress}} for determining the bind address used for
> setting a token's service.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3947) yarn.app.mapreduce.am.resource.mb not documented
[ https://issues.apache.org/jira/browse/MAPREDUCE-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3947: --- Resolution: Fixed Fix Version/s: 3.0.0 2.0.0 0.23.3 Assignee: Devaraj K Status: Resolved (was: Patch Available) Thanks Devaraj, I put this into trunk, branch-2, and branch-0.23 > yarn.app.mapreduce.am.resource.mb not documented > > > Key: MAPREDUCE-3947 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3947 > Project: Hadoop Map/Reduce > Issue Type: Bug >Affects Versions: 0.24.0, 0.23.3 >Reporter: Todd Lipcon >Assignee: Devaraj K >Priority: Minor > Labels: mrv2 > Fix For: 0.23.3, 2.0.0, 3.0.0 > > Attachments: MAPREDUCE-3947.patch > > > This configuration is useful but doesn't appear to be documented anywhere. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3867) MiniMRYarn/MiniYarn uses fixed ports
[ https://issues.apache.org/jira/browse/MAPREDUCE-3867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-3867: -- Resolution: Fixed Fix Version/s: 2.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) committed to trunk and branch-2 > MiniMRYarn/MiniYarn uses fixed ports > > > Key: MAPREDUCE-3867 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3867 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.24.0, 0.23.2 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Fix For: 2.0.0 > > Attachments: MAPREDUCE-3867.patch, MAPREDUCE-3867.patch, > MAPREDUCE-3867.patch > > > This presents issues if there are other processes using those ports. Also, if > multitasking among dev environments using Mini* things start to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3867) MiniMRYarn/MiniYarn uses fixed ports
[ https://issues.apache.org/jira/browse/MAPREDUCE-3867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-3867: -- Summary: MiniMRYarn/MiniYarn uses fixed ports (was: MiniMRYarn/MiniYarn should uses fixed ports) > MiniMRYarn/MiniYarn uses fixed ports > > > Key: MAPREDUCE-3867 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3867 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.24.0, 0.23.2 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: MAPREDUCE-3867.patch, MAPREDUCE-3867.patch, > MAPREDUCE-3867.patch > > > This presents issues if there are other processes using those ports. Also, if > multitasking among dev environments using Mini* things start to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3867) MiniMRYarn/MiniYarn should uses fixed ports
[ https://issues.apache.org/jira/browse/MAPREDUCE-3867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-3867: -- Summary: MiniMRYarn/MiniYarn should uses fixed ports (was: MiniMRYarn/MiniYarn use fixed ports) > MiniMRYarn/MiniYarn should uses fixed ports > --- > > Key: MAPREDUCE-3867 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3867 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.24.0, 0.23.2 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: MAPREDUCE-3867.patch, MAPREDUCE-3867.patch, > MAPREDUCE-3867.patch > > > This presents issues if there are other processes using those ports. Also, if > multitasking among dev environments using Mini* things start to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3958) RM: Remove RMNodeState and replace it with NodeState
[ https://issues.apache.org/jira/browse/MAPREDUCE-3958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3958: --- Status: Open (was: Patch Available) Could you please upmerge, the patch is out of date. > RM: Remove RMNodeState and replace it with NodeState > > > Key: MAPREDUCE-3958 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3958 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Bikas Saha >Assignee: Bikas Saha > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3958-1.patch, MAPREDUCE-3958.patch > > > RMNodeState is being sent over the wire after MAPREDUCE-3353. This has been > done by cloning the enum into NodeState in yarn protocol records. > That makes RMNodeState redundant and it should be replaced with NodeState. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4129) Lots of unneeded counters log messages
[
https://issues.apache.org/jira/browse/MAPREDUCE-4129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-4129:
---
Resolution: Fixed
Fix Version/s: (was: 0.23.2)
3.0.0
2.0.0
0.23.3
Status: Resolved (was: Patch Available)
Thanks Ahmed, +1 the patch looks good. I don't think we really need any new
tests because we are just changing log messages. I put this into trunk,
branch-2, and branch-0.23.
> Lots of unneeded counters log messages
> --
>
> Key: MAPREDUCE-4129
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4129
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1
>Reporter: Ahmed Radwan
>Assignee: Ahmed Radwan
> Fix For: 0.23.3, 2.0.0, 3.0.0
>
> Attachments: MAPREDUCE-4129.patch, MAPREDUCE-4129_rev2.patch
>
>
> Huge number of the same WARN messages are written. We only need to write each
> distinct message once. The messages are of the form:
> {code}
> 2012-04-05 03:55:04,166 WARN mapreduce.Counters: Group {oldGroup} is
> deprecated. Use {newGroup} instead
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4048) NullPointerException exception while accessing the Application Master UI
[
https://issues.apache.org/jira/browse/MAPREDUCE-4048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-4048:
---
Status: Open (was: Patch Available)
> NullPointerException exception while accessing the Application Master UI
>
>
> Key: MAPREDUCE-4048
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4048
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.0.0, 3.0.0
>Reporter: Devaraj K
>Assignee: Devaraj K
> Attachments: MAPREDUCE-4048.patch
>
>
> {code:xml}
> 2012-03-21 10:21:31,838 ERROR [2145015588@qtp-957250718-801]
> org.apache.hadoop.yarn.webapp.Dispatcher: error handling URI:
> /mapreduce/attempts/job_1332261815858_2_8/m/KILLED
> java.lang.reflect.InvocationTargetException
> at sun.reflect.GeneratedMethodAccessor50.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.yarn.webapp.Dispatcher.service(Dispatcher.java:150)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at
> com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
> at
> com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
> at
> com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
> at
> com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:900)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
> ...
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> Caused by: java.lang.NullPointerException
> at com.google.common.base.Joiner.toString(Joiner.java:317)
> at com.google.common.base.Joiner.appendTo(Joiner.java:97)
> at com.google.common.base.Joiner.appendTo(Joiner.java:127)
> at com.google.common.base.Joiner.join(Joiner.java:158)
> at com.google.common.base.Joiner.join(Joiner.java:166)
> at
> org.apache.hadoop.yarn.util.StringHelper.join(StringHelper.java:102)
> at
> org.apache.hadoop.mapreduce.v2.app.webapp.AppController.badRequest(AppController.java:319)
> at
> org.apache.hadoop.mapreduce.v2.app.webapp.AppController.attempts(AppController.java:286)
> ... 36 more
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4093) Improve RM WebApp start up when proxy address is not set
[
https://issues.apache.org/jira/browse/MAPREDUCE-4093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-4093:
---
Resolution: Fixed
Fix Version/s: 3.0.0
2.0.0
Status: Resolved (was: Patch Available)
Thanks Devaraj, I put this into branch-2, and trunk
> Improve RM WebApp start up when proxy address is not set
>
>
> Key: MAPREDUCE-4093
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4093
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Affects Versions: 2.0.0, 3.0.0
>Reporter: Devaraj K
>Assignee: Devaraj K
> Fix For: 2.0.0, 3.0.0
>
> Attachments: MAPREDUCE-4093.patch
>
>
> {code:title=ResourceManager.java|borderStyle=solid}
> protected void startWepApp() {
> Builder builder =
> WebApps.$for("cluster", ApplicationMasterService.class, masterService,
> "ws").at(
> this.conf.get(YarnConfiguration.RM_WEBAPP_ADDRESS,
> YarnConfiguration.DEFAULT_RM_WEBAPP_ADDRESS));
> if(YarnConfiguration.getRMWebAppHostAndPort(conf).
> equals(YarnConfiguration.getProxyHostAndPort(conf))) {
> AppReportFetcher fetcher = new AppReportFetcher(conf,
> getClientRMService());
> builder.withServlet(ProxyUriUtils.PROXY_SERVLET_NAME,
> ProxyUriUtils.PROXY_PATH_SPEC, WebAppProxyServlet.class);
> builder.withAttribute(WebAppProxy.FETCHER_ATTRIBUTE, fetcher);
> String proxy = YarnConfiguration.getProxyHostAndPort(conf);
> String[] proxyParts = proxy.split(":");
> builder.withAttribute(WebAppProxy.PROXY_HOST_ATTRIBUTE, proxyParts[0]);
> }
> webApp = builder.start(new RMWebApp(this));
> }
> {code}
> In the above code, YarnConfiguration.getProxyHostAndPort(conf) is invoking
> twice. getProxyHostAndPort() internally invokes getRMWebAppHostAndPort()
> which resolves RM web app address when proxy address is not set.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4079) Allow MR AppMaster to limit ephemeral port range.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4079: --- Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 3.0.0, 2.0.0, 0.23.3) Status: Open (was: Patch Available) > Allow MR AppMaster to limit ephemeral port range. > - > > Key: MAPREDUCE-4079 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4079 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am, mrv2 >Affects Versions: 0.23.2, 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Attachments: MR-4079-full-branch-0.23.txt, > MR-4079-full-branch-0.23.txt, MR-4079-full-branch-0.23.txt, > MR-4079-trunk.txt, MR-4079-trunk.txt, MR-4079-trunk.txt, MR-4079-trunk.txt, > MR-4079-trunk.txt > > > Having the MapReduce Application Masters bind to any ephemeral port makes it > very difficult to setup ACLs. mapreduce.job.am-access-disabled from > MAPREDUCE-3251 is not a practical permanent solution for all jobs. > Especially for tools like pig where they are not aware of > mapreduce.job.am-access-disabled and may deal with it properly. > We should add in a config option that would allow someone to restrict the > range of ports that the MR-AM can bind to. It will slow down startup in some > cases because we will have to probe for open ports instead of just asking the > OS to find one for us. But we can make that conditional on this config so > users who do not set this config do not see any performance degradation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4079) Allow MR AppMaster to limit ephemeral port range.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4079: --- Attachment: MR-4079-trunk.txt Kicking Jenkins again. > Allow MR AppMaster to limit ephemeral port range. > - > > Key: MAPREDUCE-4079 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4079 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am, mrv2 >Affects Versions: 0.23.2, 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Attachments: MR-4079-full-branch-0.23.txt, > MR-4079-full-branch-0.23.txt, MR-4079-full-branch-0.23.txt, > MR-4079-trunk.txt, MR-4079-trunk.txt, MR-4079-trunk.txt, MR-4079-trunk.txt, > MR-4079-trunk.txt > > > Having the MapReduce Application Masters bind to any ephemeral port makes it > very difficult to setup ACLs. mapreduce.job.am-access-disabled from > MAPREDUCE-3251 is not a practical permanent solution for all jobs. > Especially for tools like pig where they are not aware of > mapreduce.job.am-access-disabled and may deal with it properly. > We should add in a config option that would allow someone to restrict the > range of ports that the MR-AM can bind to. It will slow down startup in some > cases because we will have to probe for open ports instead of just asking the > OS to find one for us. But we can make that conditional on this config so > users who do not set this config do not see any performance degradation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4079) Allow MR AppMaster to limit ephemeral port range.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4079: --- Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 3.0.0, 2.0.0, 0.23.3) Status: Patch Available (was: Open) > Allow MR AppMaster to limit ephemeral port range. > - > > Key: MAPREDUCE-4079 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4079 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am, mrv2 >Affects Versions: 0.23.2, 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Attachments: MR-4079-full-branch-0.23.txt, > MR-4079-full-branch-0.23.txt, MR-4079-full-branch-0.23.txt, > MR-4079-trunk.txt, MR-4079-trunk.txt, MR-4079-trunk.txt, MR-4079-trunk.txt, > MR-4079-trunk.txt > > > Having the MapReduce Application Masters bind to any ephemeral port makes it > very difficult to setup ACLs. mapreduce.job.am-access-disabled from > MAPREDUCE-3251 is not a practical permanent solution for all jobs. > Especially for tools like pig where they are not aware of > mapreduce.job.am-access-disabled and may deal with it properly. > We should add in a config option that would allow someone to restrict the > range of ports that the MR-AM can bind to. It will slow down startup in some > cases because we will have to probe for open ports instead of just asking the > OS to find one for us. But we can make that conditional on this config so > users who do not set this config do not see any performance degradation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4165) Committing is misspelled as commiting in task logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4165: --- Resolution: Fixed Fix Version/s: 3.0.0 2.0.0 0.23.3 Target Version/s: 0.23.3, 2.0.0 (was: 2.0.0, 0.23.3) Status: Resolved (was: Patch Available) Thanks Jon, +1 I put this in trunk, branch-2, and branch-0.23 > Committing is misspelled as commiting in task logs > -- > > Key: MAPREDUCE-4165 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4165 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Trivial > Fix For: 0.23.3, 2.0.0, 3.0.0 > > Attachments: MAPREDUCE-4165.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4163) consistently set the bind address
[
https://issues.apache.org/jira/browse/MAPREDUCE-4163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated MAPREDUCE-4163:
---
Status: Patch Available (was: Open)
> consistently set the bind address
> -
>
> Key: MAPREDUCE-4163
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4163
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.0, 0.24.0, 2.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: MAPREDUCE-3659.patch
>
>
> Use {{NetUtils.getConnectAddress}} for determining the bind address used for
> setting a token's service.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4163) consistently set the bind address
[
https://issues.apache.org/jira/browse/MAPREDUCE-4163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated MAPREDUCE-4163:
---
Attachment: MAPREDUCE-3659.patch
> consistently set the bind address
> -
>
> Key: MAPREDUCE-4163
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4163
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.0, 0.24.0, 2.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: MAPREDUCE-3659.patch
>
>
> Use {{NetUtils.getConnectAddress}} for determining the bind address used for
> setting a token's service.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4163) consistently set the bind address
[
https://issues.apache.org/jira/browse/MAPREDUCE-4163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated MAPREDUCE-4163:
---
Attachment: (was: HADOOP-3659.patch)
> consistently set the bind address
> -
>
> Key: MAPREDUCE-4163
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4163
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.0, 0.24.0, 2.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>
> Use {{NetUtils.getConnectAddress}} for determining the bind address used for
> setting a token's service.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4163) consistently set the bind address
[
https://issues.apache.org/jira/browse/MAPREDUCE-4163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated MAPREDUCE-4163:
---
Attachment: HADOOP-3659.patch
> consistently set the bind address
> -
>
> Key: MAPREDUCE-4163
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4163
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.0, 0.24.0, 2.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>
> Use {{NetUtils.getConnectAddress}} for determining the bind address used for
> setting a token's service.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4159) Job is running in Uber mode after setting "mapreduce.job.ubertask.maxreduces" to zero
[ https://issues.apache.org/jira/browse/MAPREDUCE-4159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4159: --- Resolution: Fixed Fix Version/s: 3.0.0 2.0.0 0.23.3 Status: Resolved (was: Patch Available) Thanks Devaraj, I just pulled this into trunk, branch-2, and branch-0.23 > Job is running in Uber mode after setting "mapreduce.job.ubertask.maxreduces" > to zero > - > > Key: MAPREDUCE-4159 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4159 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.0, 3.0.0 >Reporter: Nishan Shetty >Assignee: Devaraj K > Fix For: 0.23.3, 2.0.0, 3.0.0 > > Attachments: MAPREDUCE-4159-1.patch, MAPREDUCE-4159.patch > > > 1.Configure "mapreduce.job.ubertask.enable" to true > 2.Configure "mapreduce.job.ubertask.maxreduces" to 0(zero) > 3.Run job such that it has one reducer(more than > "mapreduce.job.ubertask.maxreduces" value) > Observe that job is running in Uber mode instead of normal mode(non uber mode) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3867) MiniMRYarn/MiniYarn use fixed ports
[ https://issues.apache.org/jira/browse/MAPREDUCE-3867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated MAPREDUCE-3867: -- Attachment: MAPREDUCE-3867.patch Nailed the TestDistributedShell failure. It needed some special classpath setting and yarn-site.xml for the minicluster. Also this testcase was using Java assert instead Junit Assert, and when running from IDE the test was wrongly passing as the IDE does not have assertions on. Now all tests using minicluster pass. Though the patch won't apply because it is multiproject (mapreduce/tools) For the reviewer, the steps to test it are: * from trunk/ : mvn install -DskipTests -Dmaven.javadoc.skip=true * from trunk/hadoop-mapreduce-project/ : mvn test -offline * from trunk/hadoop-tools/ : mvn test -offline > MiniMRYarn/MiniYarn use fixed ports > --- > > Key: MAPREDUCE-3867 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3867 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: test >Affects Versions: 0.24.0, 0.23.2 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: MAPREDUCE-3867.patch, MAPREDUCE-3867.patch, > MAPREDUCE-3867.patch > > > This presents issues if there are other processes using those ports. Also, if > multitasking among dev environments using Mini* things start to fail. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4133) MR over viewfs is broken
[
https://issues.apache.org/jira/browse/MAPREDUCE-4133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John George updated MAPREDUCE-4133:
---
Status: Patch Available (was: Open)
trying again...
> MR over viewfs is broken
>
>
> Key: MAPREDUCE-4133
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4133
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.23.3, 2.0.0
>Reporter: John George
>Assignee: John George
> Attachments: MR-4133.patch, MR-4133.patch, MR-4133.patch
>
>
> After the changes in HADOOP-8014 went in, MR programs using viewfs broke.
> This is because, viewfs now expects getDefaultBlockSize,
> getDefaultReplication, and getServerDefaults to pass in a {{path}} as an
> argument. In the existing MR source, these are called with no arguments.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4133) MR over viewfs is broken
[
https://issues.apache.org/jira/browse/MAPREDUCE-4133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John George updated MAPREDUCE-4133:
---
Attachment: MR-4133.patch
> MR over viewfs is broken
>
>
> Key: MAPREDUCE-4133
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4133
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.23.3, 2.0.0
>Reporter: John George
>Assignee: John George
> Attachments: MR-4133.patch, MR-4133.patch, MR-4133.patch
>
>
> After the changes in HADOOP-8014 went in, MR programs using viewfs broke.
> This is because, viewfs now expects getDefaultBlockSize,
> getDefaultReplication, and getServerDefaults to pass in a {{path}} as an
> argument. In the existing MR source, these are called with no arguments.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4133) MR over viewfs is broken
[
https://issues.apache.org/jira/browse/MAPREDUCE-4133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John George updated MAPREDUCE-4133:
---
Status: Open (was: Patch Available)
> MR over viewfs is broken
>
>
> Key: MAPREDUCE-4133
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4133
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.23.3, 2.0.0
>Reporter: John George
>Assignee: John George
> Attachments: MR-4133.patch, MR-4133.patch, MR-4133.patch
>
>
> After the changes in HADOOP-8014 went in, MR programs using viewfs broke.
> This is because, viewfs now expects getDefaultBlockSize,
> getDefaultReplication, and getServerDefaults to pass in a {{path}} as an
> argument. In the existing MR source, these are called with no arguments.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4074) Client continuously retries to RM When RM goes down before launching Application Master
[
https://issues.apache.org/jira/browse/MAPREDUCE-4074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves updated MAPREDUCE-4074:
-
Resolution: Fixed
Fix Version/s: 0.23.3
Target Version/s: 0.23.3
Status: Resolved (was: Patch Available)
+1. Thanks xieguiming! I've committed this to trunk, branch-2, and branch-0.23.
> Client continuously retries to RM When RM goes down before launching
> Application Master
> ---
>
> Key: MAPREDUCE-4074
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4074
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.23.1
>Reporter: Devaraj K
> Fix For: 0.23.3
>
> Attachments: MAPREDUCE-4074-1.patch, MAPREDUCE-4074-2.patch,
> MAPREDUCE-4074-3.patch, MAPREDUCE-4074.patch
>
>
> Client continuously tries to RM and logs the below messages when the RM goes
> down before launching App Master.
> I feel exception should be thrown or break the loop after finite no of
> retries.
> {code:xml}
> 28/03/12 07:15:03 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 0 time(s).
> 28/03/12 07:15:04 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 1 time(s).
> 28/03/12 07:15:05 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 2 time(s).
> 28/03/12 07:15:06 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 3 time(s).
> 28/03/12 07:15:07 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 4 time(s).
> 28/03/12 07:15:08 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 5 time(s).
> 28/03/12 07:15:09 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 6 time(s).
> 28/03/12 07:15:10 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 7 time(s).
> 28/03/12 07:15:11 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 8 time(s).
> 28/03/12 07:15:12 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 9 time(s).
> 28/03/12 07:15:13 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 0 time(s).
> 28/03/12 07:15:14 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 1 time(s).
> 28/03/12 07:15:15 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 2 time(s).
> 28/03/12 07:15:16 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 3 time(s).
> 28/03/12 07:15:17 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 4 time(s).
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4074) Client continuously retries to RM When RM goes down before launching Application Master
[
https://issues.apache.org/jira/browse/MAPREDUCE-4074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xieguiming updated MAPREDUCE-4074:
--
Status: Patch Available (was: Open)
Hi Thomas,
I have complete the modification by your opinions.
please check, thanks.
> Client continuously retries to RM When RM goes down before launching
> Application Master
> ---
>
> Key: MAPREDUCE-4074
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4074
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.23.1
>Reporter: Devaraj K
> Attachments: MAPREDUCE-4074-1.patch, MAPREDUCE-4074-2.patch,
> MAPREDUCE-4074-3.patch, MAPREDUCE-4074.patch
>
>
> Client continuously tries to RM and logs the below messages when the RM goes
> down before launching App Master.
> I feel exception should be thrown or break the loop after finite no of
> retries.
> {code:xml}
> 28/03/12 07:15:03 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 0 time(s).
> 28/03/12 07:15:04 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 1 time(s).
> 28/03/12 07:15:05 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 2 time(s).
> 28/03/12 07:15:06 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 3 time(s).
> 28/03/12 07:15:07 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 4 time(s).
> 28/03/12 07:15:08 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 5 time(s).
> 28/03/12 07:15:09 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 6 time(s).
> 28/03/12 07:15:10 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 7 time(s).
> 28/03/12 07:15:11 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 8 time(s).
> 28/03/12 07:15:12 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 9 time(s).
> 28/03/12 07:15:13 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 0 time(s).
> 28/03/12 07:15:14 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 1 time(s).
> 28/03/12 07:15:15 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 2 time(s).
> 28/03/12 07:15:16 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 3 time(s).
> 28/03/12 07:15:17 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 4 time(s).
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4074) Client continuously retries to RM When RM goes down before launching Application Master
[
https://issues.apache.org/jira/browse/MAPREDUCE-4074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
xieguiming updated MAPREDUCE-4074:
--
Attachment: MAPREDUCE-4074-3.patch
> Client continuously retries to RM When RM goes down before launching
> Application Master
> ---
>
> Key: MAPREDUCE-4074
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4074
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.23.1
>Reporter: Devaraj K
> Attachments: MAPREDUCE-4074-1.patch, MAPREDUCE-4074-2.patch,
> MAPREDUCE-4074-3.patch, MAPREDUCE-4074.patch
>
>
> Client continuously tries to RM and logs the below messages when the RM goes
> down before launching App Master.
> I feel exception should be thrown or break the loop after finite no of
> retries.
> {code:xml}
> 28/03/12 07:15:03 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 0 time(s).
> 28/03/12 07:15:04 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 1 time(s).
> 28/03/12 07:15:05 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 2 time(s).
> 28/03/12 07:15:06 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 3 time(s).
> 28/03/12 07:15:07 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 4 time(s).
> 28/03/12 07:15:08 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 5 time(s).
> 28/03/12 07:15:09 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 6 time(s).
> 28/03/12 07:15:10 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 7 time(s).
> 28/03/12 07:15:11 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 8 time(s).
> 28/03/12 07:15:12 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 9 time(s).
> 28/03/12 07:15:13 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 0 time(s).
> 28/03/12 07:15:14 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 1 time(s).
> 28/03/12 07:15:15 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 2 time(s).
> 28/03/12 07:15:16 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 3 time(s).
> 28/03/12 07:15:17 INFO ipc.Client: Retrying connect to server:
> linux-f330.site/10.18.40.182:8032. Already tried 4 time(s).
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4159) Job is running in Uber mode after setting "mapreduce.job.ubertask.maxreduces" to zero
[ https://issues.apache.org/jira/browse/MAPREDUCE-4159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-4159: - Affects Version/s: 3.0.0 Status: Patch Available (was: Open) Thanks a lot Robert for looking into the patch. I have updated the patch with the optimization. > Job is running in Uber mode after setting "mapreduce.job.ubertask.maxreduces" > to zero > - > > Key: MAPREDUCE-4159 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4159 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.0, 3.0.0 >Reporter: Nishan Shetty >Assignee: Devaraj K > Attachments: MAPREDUCE-4159-1.patch, MAPREDUCE-4159.patch > > > 1.Configure "mapreduce.job.ubertask.enable" to true > 2.Configure "mapreduce.job.ubertask.maxreduces" to 0(zero) > 3.Run job such that it has one reducer(more than > "mapreduce.job.ubertask.maxreduces" value) > Observe that job is running in Uber mode instead of normal mode(non uber mode) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4159) Job is running in Uber mode after setting "mapreduce.job.ubertask.maxreduces" to zero
[ https://issues.apache.org/jira/browse/MAPREDUCE-4159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj K updated MAPREDUCE-4159: - Attachment: MAPREDUCE-4159-1.patch > Job is running in Uber mode after setting "mapreduce.job.ubertask.maxreduces" > to zero > - > > Key: MAPREDUCE-4159 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4159 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.0 >Reporter: Nishan Shetty >Assignee: Devaraj K > Attachments: MAPREDUCE-4159-1.patch, MAPREDUCE-4159.patch > > > 1.Configure "mapreduce.job.ubertask.enable" to true > 2.Configure "mapreduce.job.ubertask.maxreduces" to 0(zero) > 3.Run job such that it has one reducer(more than > "mapreduce.job.ubertask.maxreduces" value) > Observe that job is running in Uber mode instead of normal mode(non uber mode) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4157) ResourceManager should not kill apps that are well behaved
[ https://issues.apache.org/jira/browse/MAPREDUCE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-4157: -- Target Version/s: 2.0.0 Status: Patch Available (was: Open) > ResourceManager should not kill apps that are well behaved > -- > > Key: MAPREDUCE-4157 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4157 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 2.0.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: MAPREDUCE-4157.patch > > > Currently when the ApplicationMaster unregisters with the ResourceManager, > the RM kills (via the NMs) all the active containers for an application. > This introduces a race where the AM may be trying to clean up and may not > finish before it is killed. The RM should give the AM a chance to exit > cleanly on its own rather than always race with a pending kill on shutdown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4165) Committing is misspelled as commiting in task logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-4165: --- Target Version/s: 0.23.3, 2.0.0 (was: 2.0.0) Affects Version/s: 0.23.3 > Committing is misspelled as commiting in task logs > -- > > Key: MAPREDUCE-4165 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4165 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Trivial > Attachments: MAPREDUCE-4165.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4157) ResourceManager should not kill apps that are well behaved
[ https://issues.apache.org/jira/browse/MAPREDUCE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe updated MAPREDUCE-4157: -- Attachment: MAPREDUCE-4157.patch Patch to add a FINISHING application state as described for the initial patch in MAPREDUCE-4099. Instead of transitioning from RUNNING to FINISHED, the application transitions from RUNNING to FINISHING when it unregisters with the RM. The RM then gives it so many seconds for the AM container to exit on its own. If it fails to do so, it is killed. In either case, the app moves from the FINISHING state to the FINISHED state. The timeout for the FINISHING state is the same as the liveliness monitor (i.e.: yarn.am.liveness-monitor.expiry-interval-ms) but we could easily add another property if we think these need to be configured separately. > ResourceManager should not kill apps that are well behaved > -- > > Key: MAPREDUCE-4157 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4157 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 2.0.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: MAPREDUCE-4157.patch > > > Currently when the ApplicationMaster unregisters with the ResourceManager, > the RM kills (via the NMs) all the active containers for an application. > This introduces a race where the AM may be trying to clean up and may not > finish before it is killed. The RM should give the AM a chance to exit > cleanly on its own rather than always race with a pending kill on shutdown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4165) Committing is misspelled as commiting in task logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-4165: --- Status: Patch Available (was: Open) > Committing is misspelled as commiting in task logs > -- > > Key: MAPREDUCE-4165 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4165 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Trivial > Attachments: MAPREDUCE-4165.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4165) Committing is misspelled as commiting in task logs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-4165: --- Attachment: MAPREDUCE-4165.patch > Committing is misspelled as commiting in task logs > -- > > Key: MAPREDUCE-4165 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4165 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Trivial > Attachments: MAPREDUCE-4165.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-434) local map-reduce job limited to single reducer
[ https://issues.apache.org/jira/browse/MAPREDUCE-434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-434: -- Target Version/s: 2.0.0, 3.0.0 (was: 0.23.2) > local map-reduce job limited to single reducer > -- > > Key: MAPREDUCE-434 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-434 > Project: Hadoop Map/Reduce > Issue Type: Bug > Environment: local job tracker >Reporter: Yoram Arnon >Assignee: Aaron Kimball >Priority: Minor > Attachments: MAPREDUCE-434.2.patch, MAPREDUCE-434.3.patch, > MAPREDUCE-434.4.patch, MAPREDUCE-434.5.patch, MAPREDUCE-434.6.patch, > MAPREDUCE-434.patch, MAPREDUCE-434.patch > > > when mapred.job.tracker is set to 'local', my setNumReduceTasks call is > ignored, and the number of reduce tasks is set at 1. > This prevents me from locally debugging my partition function, which tries to > partition based on the number of reduce tasks. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3213) AM should kill all running tasks on a node when it gets marked as blacklisted
[ https://issues.apache.org/jira/browse/MAPREDUCE-3213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3213: --- Target Version/s: 2.0.0, 3.0.0 (was: 0.23.1) > AM should kill all running tasks on a node when it gets marked as blacklisted > - > > Key: MAPREDUCE-3213 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3213 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah > Fix For: 0.24.0 > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3320) Error conditions in web apps should stop pages from rendering.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3320: --- Target Version/s: 2.0.0, 3.0.0 (was: 0.23.1, 0.24.0) > Error conditions in web apps should stop pages from rendering. > -- > > Key: MAPREDUCE-3320 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3320 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > Fix For: 0.24.0 > > > There are several places in the web apps where an error condition should > short circuit the page from rendering, but it does not. Ideally the web app > framework should be extended to support exceptions similar to Jersey that can > have an HTTP return code associated with them. Then all of the places that > produce custom error pages can just throw these exceptions instead. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3658) Improvements to CapacityScheduler documentation
[ https://issues.apache.org/jira/browse/MAPREDUCE-3658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3658: --- Target Version/s: 2.0.0, 3.0.0 (was: 0.23.1) > Improvements to CapacityScheduler documentation > --- > > Key: MAPREDUCE-3658 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3658 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Yoram Arnon >Assignee: Yoram Arnon >Priority: Minor > Labels: documentation > Fix For: 0.24.0 > > Attachments: MAPREDUCE-3658, MAPREDUCE-3658 > > Original Estimate: 3h > Remaining Estimate: 3h > > There are some typos and some cases of incorrect English. > Also, the descriptions of yarn.scheduler.capacity..capacity, > yarn.scheduler.capacity..maximum-capacity, > yarn.scheduler.capacity..user-limit-factor, > yarn.scheduler.capacity.maximum-applications are not very clear to the > uninitiated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3079) usercache//appcache/ directory not removed when using DefaultContainerExecutor
[ https://issues.apache.org/jira/browse/MAPREDUCE-3079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3079: --- Target Version/s: 2.0.0, 3.0.0 (was: 0.23.1) > usercache//appcache/ directory not removed when using > DefaultContainerExecutor > --- > > Key: MAPREDUCE-3079 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3079 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves > Fix For: 0.24.0 > > > Running with the DefaultContainerExecutor it appears that the > usercache//appcache/ directory itself is not removed when the > app finishes. All the directories under it are properly removed though. > The nodemanager log file indicates that it tries to delete it: > 11/09/23 15:17:56 INFO nodemanager.DefaultContainerExecutor: Deleting > absolute path : > /home/hadoop/mapred/tmp/mapred-local/usercache/tgraves/appcache/application_1316722920862_0003 > This doesn't appear to happen with the LinuxContainerExecutor. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2584) Check for serializers early, and give out more information regarding missing serializers
[ https://issues.apache.org/jira/browse/MAPREDUCE-2584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-2584: --- Target Version/s: 2.0.0, 3.0.0 (was: 0.24.0, 0.23.1) > Check for serializers early, and give out more information regarding missing > serializers > > > Key: MAPREDUCE-2584 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2584 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: task >Affects Versions: 0.20.2 >Reporter: Harsh J >Assignee: Harsh J > Labels: serializers, tasks > Attachments: 0.20-security-MAPREDUCE-2584.r5.diff, > MAPREDUCE-2584.r2.diff, MAPREDUCE-2584.r3.diff, MAPREDUCE-2584.r4.diff, > MAPREDUCE-2584.r5.diff > > > As discussed on HADOOP-7328, MapReduce can handle serializers in a much > better way in case of bad configuration, improper imports (Some odd Text > class instead of the Writable Text set as key), etc.. > This issue covers the MapReduce parts of the improvements (made to IFile, > MapOutputBuffer, etc. and possible early-check of serializer availability > pre-submit) that provide more information than just an NPE as is the current > case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3314) AM should have a way of telling RM to not allocate containers on some nodes
[ https://issues.apache.org/jira/browse/MAPREDUCE-3314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3314: --- Target Version/s: 2.0.0, 3.0.0 (was: 0.23.1) > AM should have a way of telling RM to not allocate containers on some nodes > --- > > Key: MAPREDUCE-3314 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3314 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Hitesh Shah > Fix For: 0.24.0 > > > The AM blacklists certain nodes due to task failures but the RM does not know > of this and can allocate blacklisted nodes to the AM again and again. > RM should support a protocol that allows AM to inform the RM of nodes that it > should not allocate to it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3502) Review all Service.stop() operations and make sure that they work before a service is started
[ https://issues.apache.org/jira/browse/MAPREDUCE-3502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3502: --- Target Version/s: 2.0.0, 3.0.0 (was: 0.24.0, 0.23.1) > Review all Service.stop() operations and make sure that they work before a > service is started > - > > Key: MAPREDUCE-3502 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3502 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Steve Loughran >Assignee: Steve Loughran > Attachments: MAPREDUCE-3502.patch, MAPREDUCE-3502.patch > > Original Estimate: 24h > Time Spent: 2.5h > Remaining Estimate: 21.5h > > MAPREDUCE-3431 has shown that some of the key services's shutdown operations > are not robust against being invoked before the service is started. They need > to be by > # not calling other things if the other things are null > # not being re-entrant (i.e. make synchronized if possible), > Maybe > # have a StopService operation that only stops a service if it is live > # factor out the is-running test from the base service class and make it a > pre-check for all the child services, so they bail out sooner rather than > later. This would be the best as it would be the one guaranteed to work > consistently across all instances, so only one or two would need testing > my first iteration will skip the sync though it's something to consider. > Testing: try to create each instance; call stop() straight after > construction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3289) Make use of fadvise in the NM's shuffle handler
[ https://issues.apache.org/jira/browse/MAPREDUCE-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3289: --- Target Version/s: 2.0.0, 3.0.0 (was: 0.23.1) > Make use of fadvise in the NM's shuffle handler > --- > > Key: MAPREDUCE-3289 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3289 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2, nodemanager, performance >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: mr-3289.txt > > > Using the new NativeIO fadvise functions, we can make the NodeManager > prefetch map output before it's send over the socket, and drop it out of the > fs cache once it's been sent (since it's very rare for an output to have to > be re-sent). This improves IO efficiency and reduces cache pollution. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3739) Document all missing default configuration values in the relevant *default.xml files.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3739: --- Target Version/s: 2.0.0, 3.0.0 (was: 0.24.0, 0.23.1) > Document all missing default configuration values in the relevant > *default.xml files. > - > > Key: MAPREDUCE-3739 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3739 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0, 0.24.0 >Reporter: Hitesh Shah > > There seem to be quite a few configuration settings that are used in the code > but missing from the *default.xml files. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3871) Allow symlinking in LocalJobRunner DistributedCache
[ https://issues.apache.org/jira/browse/MAPREDUCE-3871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3871: --- Target Version/s: 2.0.0, 3.0.0 (was: 0.23.2) > Allow symlinking in LocalJobRunner DistributedCache > --- > > Key: MAPREDUCE-3871 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3871 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: distributed-cache >Reporter: Tom White >Assignee: Tom White > Attachments: MAPREDUCE-3871.patch > > > Currently the LocalJobRunner doesn't create symlinks for files in the > DistributedCache. It is safe to create symlinks if files of the same name > don't exist. LocalJobRunner should also delete the symlinks when the job has > completed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3768) MR-2450 introduced a significant performance regression (Hive)
[
https://issues.apache.org/jira/browse/MAPREDUCE-3768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-3768:
---
Target Version/s: 2.0.0, 3.0.0 (was: 0.23.1)
> MR-2450 introduced a significant performance regression (Hive)
> --
>
> Key: MAPREDUCE-3768
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3768
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1
>Reporter: Eli Collins
> Attachments: stopcommunicatorpatch.txt
>
>
> MAPREDUCE-2450 introduced, or at least triggers, a significant performance
> regression in Hive. With MR-2450 the execution time of TestCliDriver.skewjoin
> goes from 2 minutes to 15 minutes. Reverting this change from the build fixes
> the issue.
> Here's the relevant query:
> {noformat}
> FROM src src1 JOIN src src2 ON (src1.key = src2.key)
> INSERT OVERWRITE TABLE dest_j1 SELECT src1.key, src2.value;
> {noformat}
> You can reproduce this by running the following from Hive 8.0 against Hadoop
> built from branch-23.
> {noformat}
> ant very-clean package test -Dtestcase=TestCliDriver -Dqfile=skewjoin.q
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3461) Move hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site to hadoop-site
[ https://issues.apache.org/jira/browse/MAPREDUCE-3461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3461: --- Target Version/s: 2.0.0, 3.0.0 (was: 0.23.1) > Move hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site to hadoop-site > - > > Key: MAPREDUCE-3461 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3461 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: documentation >Affects Versions: 0.23.0 >Reporter: Arun C Murthy > > Currently hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site has both HDFS > and MR docs, we should move it to top-level hadoop-site. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4039) Sort Avoidance
[
https://issues.apache.org/jira/browse/MAPREDUCE-4039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-4039:
---
Target Version/s: 2.0.0, 3.0.0 (was: 0.23.2)
> Sort Avoidance
> --
>
> Key: MAPREDUCE-4039
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4039
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv2
>Affects Versions: 0.23.2
>Reporter: anty.rao
>Assignee: anty
>Priority: Minor
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-4039-branch-0.23.2.patch,
> MAPREDUCE-4039-branch-0.23.2.patch
>
>
> Inspired by
> [Tenzing|http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en//pubs/archive/37200.pdf],
> in 5.1 MapReduce Enhanceemtns:
> {quote}*Sort Avoidance*. Certain operators such as hash join
> and hash aggregation require shuffling, but not sorting. The
> MapReduce API was enhanced to automatically turn off
> sorting for these operations. When sorting is turned off, the
> mapper feeds data to the reducer which directly passes the
> data to the Reduce() function bypassing the intermediate
> sorting step. This makes many SQL operators significantly
> more ecient.{quote}
> There are a lot of applications which need aggregation only, not
> sorting.Using sorting to achieve aggregation is costly and inefficient.
> Without sorting, up application can make use of hash table or hash map to do
> aggregation efficiently.But application should bear in mind that reduce
> memory is limited, itself is committed to manage memory of reduce, guard
> against out of memory. Map-side combiner is not supported, you can also do
> hash aggregation in map side as a workaround.
> the following is the main points of sort avoidance implementation
> # add a configuration parameter ??mapreduce.sort.avoidance??, boolean type,
> to turn on/off sort avoidance workflow.Two type of workflow are coexist
> together.
> # key/value pairs emitted by map function is sorted by partition only, using
> a more efficient sorting algorithm: counting sort.
> # map-side merge, use a kind of byte merge, which just concatenate bytes from
> generated spills, read in bytes, write out bytes, without overhead of
> key/value serialization/deserailization, comparison, which current version
> incurs.
> # reduce can start up as soon as there is any map output available, in
> contrast to sort workflow which must wait until all map outputs are fetched
> and merged.
> # map output in memory can be directly consumed by reduce.When reduce can't
> catch up with the speed of incoming map outputs, in-memory merge thread will
> kick in, merging in-memory map outputs onto disk.
> # sequentially read in on-disk files to feed reduce, in contrast to currently
> implementation which read multiple files concurrently, result in many disk
> seek. Map output in memory take precedence over on disk files in feeding
> reduce function.
> I have already implement this feature based on hadoop CDH3U3 and done some
> performance evaluation, you can reference to
> [https://github.com/hanborq/hadoop] for details. Now,I'm willing to port it
> into yarn. Welcome for commenting.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3223) Remove MR1 configs from mapred-default.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-3223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3223: --- Target Version/s: 2.0.0, 3.0.0 (was: 0.23.1) > Remove MR1 configs from mapred-default.xml > -- > > Key: MAPREDUCE-3223 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3223 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation, mrv2 >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Attachments: mr-3223.txt > > > All of the MRv1 configs are still in mapred-default.xml. This is confusing > when trying to make config changes. Since a lot of the input/output format > tests still depend on MR1, I'd like to move these to src/test/mapred-site.xml > for now, and once that dependency is broken, we can remove them entirely. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3302) Remove the last dependency call from org.apache.hadoop.record package in MR.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-3302:
---
Target Version/s: 2.0.0, 3.0.0 (was: 0.23.1)
> Remove the last dependency call from org.apache.hadoop.record package in MR.
>
>
> Key: MAPREDUCE-3302
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3302
> Project: Hadoop Map/Reduce
> Issue Type: Task
> Components: client
>Affects Versions: 0.24.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Minor
> Attachments: MAPREDUCE-3302.patch
>
>
> SecureShuffleUtils provides the following helper:
> {code}
> /**
>* verify that hash equals to HMacHash(msg)
>* @param newHash
>* @return true if is the same
>*/
> private static boolean verifyHash(byte[] hash, byte[] msg, SecretKey key) {
> byte[] msg_hash = generateByteHash(msg, key);
> return Utils.compareBytes(msg_hash, 0, msg_hash.length, hash, 0,
> hash.length) == 0;
> }
> {code}
> The {{Utils}} class used there is {{org.apache.hadoop.record.Utils}}. With
> the {{record}} common package going away via HADOOP-7781, the internal (and
> also deprecated on the whole) {{compareBytes}} utility must be moved
> elsewhere.
> The {{Utils#compareBytes}} contains:
> {code}
> /** Lexicographic order of binary data. */
> public static int compareBytes(byte[] b1, int s1, int l1,
> byte[] b2, int s2, int l2) {
> return WritableComparator.compareBytes(b1, s1, l1, b2, s2, l2);
> }
> {code}
> Which looks like it can be replaced inline, as it appears to be a dummy
> wrapper call. I'll put up a patch with this inline replacement shortly for
> review.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3498) yarn rmadmin help message contains reference to hadoop cli and JT
[ https://issues.apache.org/jira/browse/MAPREDUCE-3498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3498: --- Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 0.23.1) > yarn rmadmin help message contains reference to hadoop cli and JT > - > > Key: MAPREDUCE-3498 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3498 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves > > has option to specify a job tracker and the last line for general command > line syntax had "bin/hadoop command [genericOptions] [commandOptions]" > ran "yarn rmadmin" to get usage: > RMAdmin > Usage: java RMAdmin >[-refreshQueues] >[-refreshNodes] >[-refreshUserToGroupsMappings] >[-refreshSuperUserGroupsConfiguration] >[-refreshAdminAcls] >[-refreshServiceAcl] >[-help [cmd]] > Generic options supported are > -conf specify an application configuration file > -D use value for given property > -fs specify a namenode > -jt specify a job tracker > -files specify comma separated files to be > copied to the map reduce cluster > -libjars specify comma separated jar files > to include in the classpath. > -archives specify comma separated > archives to be unarchived on the compute machines. > The general command line syntax is > bin/hadoop command [genericOptions] [commandOptions] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3908) jobhistory server trying to load job conf file from wrong location
[ https://issues.apache.org/jira/browse/MAPREDUCE-3908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3908: --- Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 0.23.2) > jobhistory server trying to load job conf file from wrong location > -- > > Key: MAPREDUCE-3908 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3908 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 >Affects Versions: 0.23.0 >Reporter: Thomas Graves > > I have seen a few instance where I try to click on the job configuration link > from the job history server web ui and it gives a 500 message. Looking at > the job history server log file it shows an exception like: > 2012-02-23 22:16:32,519 ERROR org.apache.hadoop.yarn.webapp.View: Error while > reading > hdfs://host.com:9000/home/hadoop/mapred/history/done_intermediate/user/job_1330033607650_0001_conf.xml > java.io.FileNotFoundException: File does not exist: > /home/hadoop/mapred/history/done_intermediate/user/job_1330033607650_0001_conf.xml > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:746) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:709) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:681) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:302) > at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:2 > If I go look in hdfs, it doesn't exist in the done_intermediate directory > anymore, it exists in the done directory structure. > hdfs://host.com:9000/home/hadoop/mapred/history/done/2012/02/23/00/job_1330033607650_0001_conf.xml > I'm not exactly sure how to reproduce this, but I definitely see it every > once in a while. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3659) Host-based token support
[ https://issues.apache.org/jira/browse/MAPREDUCE-3659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3659: --- Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 0.24.0, 0.23.1) > Host-based token support > > > Key: MAPREDUCE-3659 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3659 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: security >Affects Versions: 0.23.1, 0.24.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > > Need to port the 205 host-based token support into MR and yarn. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3120) JobHistory is not providing correct count failed,killed task
[ https://issues.apache.org/jira/browse/MAPREDUCE-3120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3120: --- Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 0.23.1) > JobHistory is not providing correct count failed,killed task > > > Key: MAPREDUCE-3120 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3120 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Subroto Sanyal >Assignee: Subroto Sanyal >Priority: Critical > Fix For: 0.24.0 > > Attachments: JobFail.PNG > > > Please refer the attachment JobFail.PNG. > Here the Job (WordCount) Failed as all Map Attempts were > killed(intensionally) but, still the Table in UI shows 0 Killed Attempts and > no reason for Failure is also available. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3623) Authorization of NM <=> RM with simple authentication mistakenly attempts kerberos when yarn.nodemanager.principal is defined
[
https://issues.apache.org/jira/browse/MAPREDUCE-3623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-3623:
---
Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 0.24.0, 0.23.1)
> Authorization of NM <=> RM with simple authentication mistakenly attempts
> kerberos when yarn.nodemanager.principal is defined
> -
>
> Key: MAPREDUCE-3623
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3623
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Jonathan Eagles
>
> MAPREDUCE-3617 addresses the default values of yarn.nodemanager.principal and
> yarn.resourcemanager.principal
> I have enabled authorization with simple authentication. NM <=> RM still
> attempts kerberos authentication. If simple authentication is enabled
> yarn.nodemanager.principal and yarn.resourcemanager.principal values should
> be ignored and simple authentication should be used.
> {code:xml|title=core-site.xml snippet}
>
> hadoop.security.authentication
> simple
>
>
>
> hadoop.security.authorization
> true
>
>
> {code}
> {code:xml|title=yarn-site.xml snippet}
>
> The Kerberos principal for the resource manager.
> yarn.resourcemanager.principal
> rm/sightbusy-lx@LOCALHOST
>
>
> The kerberos principal for the node manager.
> yarn.nodemanager.principal
> nm/sightbusy-lx@LOCALHOST
>
> {code}
> {noformat:title=nodemanager.out snippet}
> 2012-01-03 16:40:00,793 INFO nodemanager.NodeStatusUpdaterImpl
> (NodeStatusUpdaterImpl.java:registerWithRM(176)) - Connected to
> ResourceManager at machine.example.com:8025
> 2012-01-03 16:40:00,845 ERROR service.CompositeService
> (CompositeService.java:start(72)) - Error starting services
> org.apache.hadoop.yarn.server.nodemanager.NodeManager
> org.apache.avro.AvroRuntimeException:
> java.lang.reflect.UndeclaredThrowableException
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:149)
> at
> org.apache.hadoop.yarn.service.CompositeService.start(CompositeService.java:68)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.start(NodeManager.java:167)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:242)
> Caused by: java.lang.reflect.UndeclaredThrowableException
> at
> org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:66)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:182)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.start(NodeStatusUpdaterImpl.java:145)
> ... 3 more
> Caused by: com.google.protobuf.ServiceException:
> org.apache.hadoop.security.authorize.AuthorizationException: User user
> (auth:SIMPLE) is not authorized for protocol interface
> org.apache.hadoop.yarn.proto.ResourceTracker$ResourceTrackerService$BlockingInterface,
> expected client Kerberos principal is nm/sightbusy-lx@LOCALHOST
> at
> org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:139)
> at $Proxy24.registerNodeManager(Unknown Source)
> at
> org.apache.hadoop.yarn.server.api.impl.pb.client.ResourceTrackerPBClientImpl.registerNodeManager(ResourceTrackerPBClientImpl.java:59)
> ... 5 more
> Caused by: org.apache.hadoop.security.authorize.AuthorizationException: User
> user (auth:SIMPLE) is not authorized for protocol interface
> org.apache.hadoop.yarn.proto.ResourceTracker$ResourceTrackerService$BlockingInterface,
> expected client Kerberos principal is nm/sightbusy-lx@LOCALHOST
> at org.apache.hadoop.ipc.Client.call(Client.java:1085)
> at
> org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:136)
> ... 7 more
> 2012-01-03 16:40:00,846 WARN event.AsyncDispatcher
> (AsyncDispatcher.java:run(78)) - AsyncDispatcher thread interrupted
> java.lang.InterruptedException
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:1961)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1996)
> at
> java.util.concurrent.LinkedBlockingQ
[jira] [Updated] (MAPREDUCE-3707) mapreduce/yarn source jars not included in dist tarball
[ https://issues.apache.org/jira/browse/MAPREDUCE-3707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3707: --- Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 0.23.1) > mapreduce/yarn source jars not included in dist tarball > --- > > Key: MAPREDUCE-3707 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3707 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves > > the mapreduce and yarn sources jars don't get included into the distribution > tarball. It seems they get built by default just aren't assembled. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3917) Use java.net.preferIPv4Stack to force IPv4 in yarn
[ https://issues.apache.org/jira/browse/MAPREDUCE-3917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3917: --- Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 0.23.2) > Use java.net.preferIPv4Stack to force IPv4 in yarn > -- > > Key: MAPREDUCE-3917 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3917 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves > > HADOOP-6056 made the changes for hadoop cli to use java.net.preferIPv4Stack > to force IPv4. We should do the same things for the yarn commands. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3406) Add node information to bin/mapred job -list-attempt-ids and other improvements
[ https://issues.apache.org/jira/browse/MAPREDUCE-3406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3406: --- Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 0.23.1) > Add node information to bin/mapred job -list-attempt-ids and other > improvements > --- > > Key: MAPREDUCE-3406 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3406 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Ravi Prakash >Assignee: Ravi Prakash > Fix For: 0.24.0 > > > From [~rramya] > Providing the NM information where the containers are scheduled in bin/mapred > job -list-attempt-ids will be helpful in automation, debugging and to avoid > grepping through the AM logs. > From my own observation, the list-attempt-ids should list the attempt ids and > not require the arguments. The arguments if given, can be used to filter the > results. From the usage: > bq. [-list-attempt-ids ]. Valid values for > are MAP REDUCE JOB_SETUP JOB_CLEANUP TASK_CLEANUP. Valid values > for are running, completed -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3835) RM capacity scheduler web UI doesn't show active users
[ https://issues.apache.org/jira/browse/MAPREDUCE-3835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3835: --- Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 0.23.1) > RM capacity scheduler web UI doesn't show active users > -- > > Key: MAPREDUCE-3835 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3835 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Priority: Minor > > On the jobtracker, the web ui showed the active users for each queue and how > much resources each of those users were using. That currently isn't being > displayed on the RM capacity scheduler web ui. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3889) job client tries to use /tasklog interface, but that doesn't exist anymore
[ https://issues.apache.org/jira/browse/MAPREDUCE-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3889: --- Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 0.23.2) > job client tries to use /tasklog interface, but that doesn't exist anymore > -- > > Key: MAPREDUCE-3889 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3889 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: Thomas Graves >Priority: Critical > > if you specify -Dmapreduce.client.output.filter=SUCCEEDED option when > running a job it tries to fetch task logs to print out on the client side > from a url like: > http://nodemanager:8080/tasklog?plaintext=true&attemptid=attempt_1329857083014_0003_r_00_0&filter=stdout > It always errors on this request with: Required param job, map and reduce > We saw this error when using distcp and the distcp failed. I'm not sure if it > is mandatory for distcp or just informational purposes. I'm guessing the > latter. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3267) MR2 reduce tasks showing >100% complete
[ https://issues.apache.org/jira/browse/MAPREDUCE-3267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3267: --- Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 0.23.1) > MR2 reduce tasks showing >100% complete > --- > > Key: MAPREDUCE-3267 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3267 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, task >Affects Versions: 0.23.0 >Reporter: Todd Lipcon >Assignee: Ravi Prakash > > My job is currently showing >100% reduce completion. Some reduce tasks are > much higher than 100% complete. they appear to be in the "last merge pass" > stage -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3755) Add the equivalent of JobStatus to end of JobHistory file
[ https://issues.apache.org/jira/browse/MAPREDUCE-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3755: --- Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 0.23.2) > Add the equivalent of JobStatus to end of JobHistory file > -- > > Key: MAPREDUCE-3755 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3755 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: jobhistoryserver, mrv2 >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Bikas Saha >Priority: Critical > Fix For: 0.23.2 > > > In MR1 we have the notion of CompletedJobStatus store to aid fast responses > to job.getStatus. We need the equivalent for MR2, an option is to add the > jobStatus to the end of the JobHistory file to which the JHS can easily jump > ahead to and serve the query, it should also cache this for a fair number of > recently completed jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3893) allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
[ https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3893: --- Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 0.23.2) > allow capacity scheduler configs maximum-applications and > maximum-am-resource-percent configurable on a per queue basis > --- > > Key: MAPREDUCE-3893 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Eric Payne >Priority: Critical > Attachments: MAPREDUCE-3893-1.txt > > > The capacity scheduler configs for maximum-applications and > maximum-am-resource-percent are currently configured globally and then made > proportional to each queue based on its capacity. There are times when this > may not work well. some exampless - if you have a queue that is running on > uberAM jobs, the jobs a queue is running always has a small number of > containers, and then you have the opposite where in a queue with very small > capacity, you may want to limit the am resources even more so you don't end > up deadlocked with all your capacity being used for app masters. > I think we should make those configurable on a per queue basis. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3825) MR should not be getting duplicate tokens for a MR Job.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3825: --- Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 0.24.0, 0.23.1) > MR should not be getting duplicate tokens for a MR Job. > --- > > Key: MAPREDUCE-3825 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3825 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: security >Affects Versions: 0.23.1, 0.24.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: MAPREDUCE-3825.patch, TokenCache.pdf, solution4.patch > > > This is the counterpart to HADOOP-7967. > MR gets tokens for all input, output and the default filesystem when a MR job > is submitted. > The APIs in FileSystem make it challenging to avoid duplicate tokens when > there are file systems that have embedded > filesystems. > Here is the original description that Daryn wrote: > The token cache currently tries to assume a filesystem's token service key. > The assumption generally worked while there was a one to one mapping of > filesystem to token. With the advent of multi-token filesystems like viewfs, > the token cache will try to use a service key (ie. for viewfs) that will > never exist (because it really gets the mounted fs tokens). > The descriop -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3850) Avoid redundant calls for tokens in TokenCache
[
https://issues.apache.org/jira/browse/MAPREDUCE-3850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-3850:
---
Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 0.24.0, 0.23.1)
> Avoid redundant calls for tokens in TokenCache
> --
>
> Key: MAPREDUCE-3850
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3850
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: security
>Affects Versions: 0.23.1, 0.24.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: MAPREDUCE-3850-1.patch, MAPREDUCE-3850.patch,
> MAPREDUCE-3850.patch
>
>
> The {{TokenCache}} will repeatedly call the same filesystem for tokens. This
> is inefficient and can easily be changed to only call each filesystem once.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3838) MapReduce job submission time has increased in 0.23 when compared to 0.20.206
[ https://issues.apache.org/jira/browse/MAPREDUCE-3838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3838: --- Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 0.23.2) > MapReduce job submission time has increased in 0.23 when compared to 0.20.206 > - > > Key: MAPREDUCE-3838 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3838 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: client >Affects Versions: 0.23.0 >Reporter: Amar Kamat > Labels: gridmix, job-submit-time, yarn > Fix For: 0.23.2 > > > While running Gridmix on 0.23, we found that the job submission time has > increased when compared to 0.20.206. > Here are some stats: > ||Submit-Time||Total number of jobs in YARN|| Total number of jobs in FRED|| > |> 25secs|3 |1 | > |> 20secs| 6 | 2 | > |> 15secs| 14 | 4 | > |> 10secs| 24 | 4 | > |> 5secs | 67 | 28| > Note that Gridmix was run using the same trace. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3350) Per-app RM page should have the list of application-attempts like on the app JHS page
[ https://issues.apache.org/jira/browse/MAPREDUCE-3350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3350: --- Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 0.24.0, 0.23.1) > Per-app RM page should have the list of application-attempts like on the app > JHS page > - > > Key: MAPREDUCE-3350 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3350 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, webapps >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Jonathan Eagles >Priority: Critical > Fix For: 0.24.0 > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4030) If the nodemanager on which the maptask is executed is going down before the mapoutput is consumed by the reducer,then the job is failing with shuffle error
[ https://issues.apache.org/jira/browse/MAPREDUCE-4030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4030: --- Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 0.23.2) > If the nodemanager on which the maptask is executed is going down before the > mapoutput is consumed by the reducer,then the job is failing with shuffle > error > > > Key: MAPREDUCE-4030 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4030 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Reporter: Nishan Shetty >Assignee: Devaraj K > > My cluster has 2 NM's. > The value of "mapreduce.job.reduce.slowstart.completedmaps" is set to 1. > When the job execution is in progress and Mappers has finished about 99% > completion,one of the NM has gone down. > The job has failed with the following trace > "Error: org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: error > in shuffle in fetcher#1 at > org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:123) at > org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:371) at > org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:148) at > java.security.AccessController.doPrivileged(Native Method) at > javax.security.auth.Subject.doAs(Subject.java:396) at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:143) Caused by: > java.io.IOException: Exceeded MAX_FAILED_UNIQUE_FETCHES; bailing-out. at > org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler.checkReducerHealth(ShuffleScheduler.java:253) > at > org.apache.hadoop.mapreduce.task.reduce.ShuffleScheduler.copyFailed(ShuffleScheduler.java:187) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:240) > at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:152) " -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3550) RM web proxy should handle redirect of web services urls
[ https://issues.apache.org/jira/browse/MAPREDUCE-3550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3550: --- Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 0.23.2) > RM web proxy should handle redirect of web services urls > > > Key: MAPREDUCE-3550 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3550 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > > the RM web proxy should handle the web services urls added in MAPREDUCE-2863. > The proxy does handle passing the web service urls to the AM, it just > doesn't handle redirecting it after the AM goes away. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3905) Allow per job log aggregation configuration
[ https://issues.apache.org/jira/browse/MAPREDUCE-3905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3905: --- Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 0.23.2) > Allow per job log aggregation configuration > --- > > Key: MAPREDUCE-3905 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3905 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Siddharth Seth >Assignee: Siddharth Seth > > Currently, if log aggregation is enabled for a cluster - logs for all jobs > will be aggregated - leading to a whole bunch of files on hdfs which users > may not want. > Users should be able to control this along with the aggregation policy - > failed only, all, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4079) Allow MR AppMaster to limit ephemeral port range.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4079: --- Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 2.0.0, 0.23.2) > Allow MR AppMaster to limit ephemeral port range. > - > > Key: MAPREDUCE-4079 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4079 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am, mrv2 >Affects Versions: 0.23.2, 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Attachments: MR-4079-full-branch-0.23.txt, > MR-4079-full-branch-0.23.txt, MR-4079-full-branch-0.23.txt, > MR-4079-trunk.txt, MR-4079-trunk.txt, MR-4079-trunk.txt, MR-4079-trunk.txt > > > Having the MapReduce Application Masters bind to any ephemeral port makes it > very difficult to setup ACLs. mapreduce.job.am-access-disabled from > MAPREDUCE-3251 is not a practical permanent solution for all jobs. > Especially for tools like pig where they are not aware of > mapreduce.job.am-access-disabled and may deal with it properly. > We should add in a config option that would allow someone to restrict the > range of ports that the MR-AM can bind to. It will slow down startup in some > cases because we will have to probe for open ports instead of just asking the > OS to find one for us. But we can make that conditional on this config so > users who do not set this config do not see any performance degradation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3751) Simplify job submission in gridmix
[ https://issues.apache.org/jira/browse/MAPREDUCE-3751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3751: --- Target Version/s: 1.1.0, 2.0.0 (was: 1.1.0, 0.23.1) > Simplify job submission in gridmix > -- > > Key: MAPREDUCE-3751 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3751 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: contrib/gridmix, mrv2 >Affects Versions: 0.23.0, 1.0.0 >Reporter: Arun C Murthy > > Currently gridmix tries to gauge cluster load etc. and throttles job > submission. This makes it unpredictable and also is hard to support across > MR1 and MR2. > I propose we simplify it to be: > # Replay mode - Just submit jobs in the interval as in the original trace. > # Stress mode - Compress the interval with a given factor for all jobs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3993) reduce fetch catch clause should catch RTEs as well
[ https://issues.apache.org/jira/browse/MAPREDUCE-3993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3993: --- Target Version/s: 1.0.3, 2.0.0 (was: 1.0.3, 0.23.3) > reduce fetch catch clause should catch RTEs as well > --- > > Key: MAPREDUCE-3993 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3993 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, mrv2 >Affects Versions: 0.23.1, 1.0.2 >Reporter: Todd Lipcon > > When using a compression codec for intermediate compression, some cases of > corrupt data can cause the codec to throw exceptions other than IOException > (eg java.lang.InternalError). This will currently cause the whole reduce task > to fail, instead of simply treating it like another case of a failed fetch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4029) NodeManager status web page should express 'last update' times as seconds ago
[ https://issues.apache.org/jira/browse/MAPREDUCE-4029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4029: --- Target Version/s: 2.0.0, trunk (was: 0.23.3, 0.24.0) > NodeManager status web page should express 'last update' times as seconds ago > - > > Key: MAPREDUCE-4029 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4029 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: webapps >Affects Versions: 0.23.1 >Reporter: Harsh J > > The 'Last health update' field on the MR2 apps' nodes page (at > http://host:8088/cluster/nodes) is a timestamp right now, which isn't really > informative for what the field means. It ought to be in seconds-ago from > now(), like was the case in JobTracker for heartbeats. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3937) docs need updating on how to start proxyserver
[ https://issues.apache.org/jira/browse/MAPREDUCE-3937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3937: --- Target Version/s: 2.0.0, trunk (was: 0.23.3) > docs need updating on how to start proxyserver > -- > > Key: MAPREDUCE-3937 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3937 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Thomas Graves >Priority: Minor > > The docs > (http://hadoop.apache.org/common/docs/r0.23.1/hadoop-yarn/hadoop-yarn-site/ClusterSetup.html) > on how to start the proxyserver are wrong. It says to use yarn start, which > should be use yarn_daemon.sh start proxyserver. > Also just running "yarn" doesn't show the proxyserver as an option. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4026) Lower minimum-allocation-mb to sensible defaults
[ https://issues.apache.org/jira/browse/MAPREDUCE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4026: --- Target Version/s: 2.0.0, trunk (was: 0.23.3) > Lower minimum-allocation-mb to sensible defaults > > > Key: MAPREDUCE-4026 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2, scheduler >Affects Versions: 0.23.3 >Reporter: Harsh J >Assignee: Harsh J > > The CapacityScheduler's minimum-allocation-mb is set to 1024. > The FIFO's minimum-allocation-mb meanwhile, is 128. > I propose changing the formers' minimum to that amount as well. 1024 is way > too much as a default, wastes "slots" on NMs - and I also do not see why CS > has to deviate that settings from the FIFO default. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3939) Enhance YARN service model
[
https://issues.apache.org/jira/browse/MAPREDUCE-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-3939:
---
Target Version/s: 2.0.0, trunk (was: 0.23.3, 0.24.0)
> Enhance YARN service model
> --
>
> Key: MAPREDUCE-3939
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3939
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
>Affects Versions: 0.24.0, 0.23.2
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>
> Having played the YARN service model, there are some issues
> that I've identified based on past work and initial use.
> This JIRA issue is an overall one to cover the issues, with solutions pushed
> out to separate JIRAs.
> h2. state model prevents stopped state being entered if you could not
> successfully start the service.
> In the current lifecycle you cannot stop a service unless it was successfully
> started, but
> * {{init()}} may acquire resources that need to be explicitly released
> * if the {{start()}} operation fails partway through, the {{stop()}}
> operation may be needed to release resources.
> *Fix:* make {{stop()}} a valid state transition from all states and require
> the implementations to be able to stop safely without requiring all fields to
> be non null.
> Before anyone points out that the {{stop()}} operations assume that all
> fields are valid; and if called before a {{start()}} they will NPE;
> MAPREDUCE-3431 shows that this problem arises today, MAPREDUCE-3502 is a fix
> for this. It is independent of the rest of the issues in this doc but it will
> aid making {{stop()}} execute from all states other than "stopped".
> MAPREDUCE-3502 is too big a patch and needs to be broken down for easier
> review and take up; this can be done with issues linked to this one.
> h2. AbstractService doesn't prevent duplicate state change requests.
> The {{ensureState()}} checks to verify whether or not a state transition is
> allowed from the current state are performed in the base {{AbstractService}}
> class -yet subclasses tend to call this *after* their own {{init()}},
> {{start()}} & {{stop()}} operations. This means that these operations can be
> performed out of order, and even if the outcome of the call is an exception,
> all actions performed by the subclasses will have taken place. MAPREDUCE-3877
> demonstrates this.
> This is a tricky one to address. In HADOOP-3128 I used a base class instead
> of an interface and made the {{init()}}, {{start()}} & {{stop()}} methods
> {{final}}. These methods would do the checks, and then invoke protected inner
> methods, {{innerStart()}}, {{innerStop()}}, etc. It should be possible to
> retrofit the same behaviour to everything that extends {{AbstractService}}
> -something that must be done before the class is considered stable (because
> once the lifecycle methods are declared final, all subclasses that are out of
> the source tree will need fixing by the respective developers.
> h2. AbstractService state change doesn't defend against race conditions.
> There's no concurrency locks on the state transitions. Whatever fix for wrong
> state calls is added should correct this to prevent re-entrancy, such as
> {{stop()}} being called from two threads.
> h2. Static methods to choreograph of lifecycle operations
> Helper methods to move things through lifecycles. init->start is common,
> stop-if-service!=null another. Some static methods can execute these, and
> even call {{stop()}} if {{init()}} raises an exception. These could go into a
> class {{ServiceOps}} in the same package. These can be used by those services
> that wrap other services, and help manage more robust shutdowns.
> h2. state transition failures are something that registered service listeners
> may wish to be informed of.
> When a state transition fails a {{RuntimeException}} can be thrown -and the
> service listeners are not informed as the notification point isn't reached.
> They may wish to know this, especially for management and diagnostics.
> *Fix:* extend {{ServiceStateChangeListener}} with a callback such as
> {{stateChangeFailed(Service service,Service.State targeted-state,
> RuntimeException e)}} that is invoked from the (final) state change methods
> in the {{AbstractService}} class (once they delegate to their inner
> {{innerStart()}}, {{innerStop()}} methods; make a no-op on the existing
> implementations of the interface.
> h2. Service listener failures not handled
> Is this an error an error or not? Log and igno
[jira] [Updated] (MAPREDUCE-3990) MRBench allows Long-sized input-lines value but parses CLI argument as an Integer
[
https://issues.apache.org/jira/browse/MAPREDUCE-3990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-3990:
---
Target Version/s: 2.0.0, trunk (was: 0.23.3, 0.24.0)
> MRBench allows Long-sized input-lines value but parses CLI argument as an
> Integer
> -
>
> Key: MAPREDUCE-3990
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3990
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
>Affects Versions: 0.23.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Trivial
> Attachments: MAPREDUCE-3990.patch
>
>
> MRBench has the following method:
> {code}
> public void generateTextFile(FileSystem fs, Path inputFile, long numLines,
> Order sortOrder) { ... }
> {code}
> The method is already set to accept a long datatype for numLines, for
> generating very large amount of data.
> However, in {{MRBench#run(...)}}, the inputLines CLI parameter is parsed via
> an Integer.parseInt, causing numbers passed > Integer.MAX_VALUE to throw
> NumberFormatExceptions as a result.
> The parsing should be Long.parseLong and the inputLines datatype should be
> switched to the same type as passed to the method (long).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4027) Document the minimum-allocation-mb configurations
[ https://issues.apache.org/jira/browse/MAPREDUCE-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4027: --- Target Version/s: 2.0.0, trunk (was: 0.23.3, 0.24.0) > Document the minimum-allocation-mb configurations > - > > Key: MAPREDUCE-4027 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4027 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 0.23.3 >Reporter: Harsh J >Assignee: Harsh J >Priority: Minor > > None of the current yarn.scheduler.fifo.minimum-allocation-mb and > yarn.scheduler.capacity.minimum-allocation-mb are documented anywhere. > Without knowledge of these params, one can't change the default allocations. > And the default allocations are pretty high btw (MAPREDUCE-4026). > We should document these in the Cluster Setup page at least. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3971) Job History web services need to have limits on the number of items they can return.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3971: --- Target Version/s: 2.0.0, trunk (was: 0.23.3) > Job History web services need to have limits on the number of items they can > return. > > > Key: MAPREDUCE-3971 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3971 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.2 >Reporter: Robert Joseph Evans > > The Job History web services canput a very large load on the job history > server. We should put in a limit on the number of entries that can be > returned by the web service, and also add in the ability to modify the > starting location in the list, so that all entries can still be downlaoded. > Just not all at once. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3967) DefaultContainerExecutor cannot launch container under windows
[
https://issues.apache.org/jira/browse/MAPREDUCE-3967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-3967:
---
Target Version/s: 2.0.0, trunk (was: 0.23.3)
> DefaultContainerExecutor cannot launch container under windows
> --
>
> Key: MAPREDUCE-3967
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3967
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: nodemanager
>Affects Versions: 0.23.3
> Environment: Apache Maven 3.0.4 (r1232337; 2012-01-17 16:44:56+0800)
> Java version: 1.7.0_02, vendor: Oracle Corporation
> Java home: C:\Program Files (x86)\Java\jdk1.7.0_02\jre
> Default locale: zh_CN, platform encoding: GBK
> OS name: "windows 7", version: "6.1", arch: "x86", family: "windows"
>Reporter: Changming Sun
> Original Estimate: 72h
> Remaining Estimate: 72h
>
> DefaultContainerExecutor cannot launch container under windows, because bash
> cannot find the WRAPPER_LAUNCH_SCRIPT.
> Path wrapperScriptDst = new Path(containerWorkDir, WRAPPER_LAUNCH_SCRIPT);
> String[] command = {"bash", "-c",
> wrapperScriptDst.toUri().getPath().toString()};
> LOG.info("launchContainer: " + Arrays.toString(command));
> Suppose that the value of 'wrapperScriptDst' is
> "C:\hadoop\default_container_executor.sh"
> Then wrapperScriptDst.toUri().getPath().toString() will be
> "/C:/hadoop/default_container_executor.sh", which is a wrong path
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3957) yarn daemonlog usage wrong
[ https://issues.apache.org/jira/browse/MAPREDUCE-3957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3957: --- Target Version/s: 2.0.0, trunk (was: 0.23.3) > yarn daemonlog usage wrong > --- > > Key: MAPREDUCE-3957 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3957 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Priority: Minor > > $ yarn daemonlog > USAGES: > java org.apache.hadoop.log.LogLevel -getlevel > java org.apache.hadoop.log.LogLevel -setlevel > The usage shouldn't print java org.apache.hadoop.log.LogLevel -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3685) There are some bugs in implementation of MergeManager
[ https://issues.apache.org/jira/browse/MAPREDUCE-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3685: --- Target Version/s: 2.0.0, trunk (was: 0.23.3) > There are some bugs in implementation of MergeManager > - > > Key: MAPREDUCE-3685 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3685 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.1 >Reporter: anty.rao >Assignee: anty >Priority: Minor > Attachments: MAPREDUCE-3685-branch-0.23.1.patch, > MAPREDUCE-3685-branch-0.23.1.patch, MAPREDUCE-3685-branch-0.23.1.patch, > MAPREDUCE-3685.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3994) Port TestEmptyJob to maven, possibly using the MiniMRYarnCluster
[ https://issues.apache.org/jira/browse/MAPREDUCE-3994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-3994: --- Target Version/s: 2.0.0, trunk (was: 0.23.3) > Port TestEmptyJob to maven, possibly using the MiniMRYarnCluster > > > Key: MAPREDUCE-3994 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3994 > Project: Hadoop Map/Reduce > Issue Type: Test > Components: mrv2, test >Affects Versions: 0.23.2 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans > > TestEmptyJob started failing recently. MAPREDUCE-3982 fixes that failure, > but it would have been good to have the test as part of maven so that the > failure would have been caught long before it was checked in. > The test currently uses the MRMiniCluster, but it does some things with > synchronization that are not currently available in the MiniMRYarnCluster. > Before just moving the test over, we need to better understand what the > intention of the test is, and if it should just be deleted, if there are > other tests that cover it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4056) Remove MR1 src/test/system
[ https://issues.apache.org/jira/browse/MAPREDUCE-4056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4056: --- Target Version/s: 2.0.0 (was: 0.23.3) > Remove MR1 src/test/system > -- > > Key: MAPREDUCE-4056 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4056 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: test >Reporter: Eli Collins >Assignee: Eli Collins >Priority: Minor > > hadoop-mapreduce-project/src/test/system is MR1 specific (eg built against > the JT/TT), is already maintained in branch-1 can be removed from trunk/23. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4159) Job is running in Uber mode after setting "mapreduce.job.ubertask.maxreduces" to zero
[ https://issues.apache.org/jira/browse/MAPREDUCE-4159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4159: --- Status: Open (was: Patch Available) > Job is running in Uber mode after setting "mapreduce.job.ubertask.maxreduces" > to zero > - > > Key: MAPREDUCE-4159 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4159 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 2.0.0 >Reporter: Nishan Shetty >Assignee: Devaraj K > Attachments: MAPREDUCE-4159.patch > > > 1.Configure "mapreduce.job.ubertask.enable" to true > 2.Configure "mapreduce.job.ubertask.maxreduces" to 0(zero) > 3.Run job such that it has one reducer(more than > "mapreduce.job.ubertask.maxreduces" value) > Observe that job is running in Uber mode instead of normal mode(non uber mode) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
