[jira] [Commented] (MAPREDUCE-4198) Update RackResolver and related tests to resolve host with new API from HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual host besides a li
[ https://issues.apache.org/jira/browse/MAPREDUCE-4198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262386#comment-13262386 ] Junping Du commented on MAPREDUCE-4198: --- This patch depends on new interface added in HADOOP-8304. > Update RackResolver and related tests to resolve host with new API from > HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual > host besides a list of host) > --- > > Key: MAPREDUCE-4198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4198 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 1.0.0, 2.0.0 >Reporter: Junping Du >Assignee: Junping Du > Attachments: MAPREDUCE-4198-v2.patch, MAPREDUCE-4198.patch > > > HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual > host besides a list of host) will induce a new API to resolve individual host > rather than a list of host. Here is update on MapReduce part to use new API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4195) With invalid queueName request param, jobqueue_details.jsp shows NPE
[
https://issues.apache.org/jira/browse/MAPREDUCE-4195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262379#comment-13262379
]
Harsh J commented on MAPREDUCE-4195:
Thanks for filing this Gera. Usually one's never supposed to access this URL
directly (its not documented to). Otherwise your patch looks fine to me, to
avoid null objects in the JSP.
You've not granted ASF the rights to use your patch. If you're willing to
contribute this into our repos, please re-upload with the grant added (it
appears as a radio option in the bottom of attach-file dialog).
Many thanks for doing this!
> With invalid queueName request param, jobqueue_details.jsp shows NPE
>
>
> Key: MAPREDUCE-4195
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4195
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Affects Versions: 1.0.0
>Reporter: Gera Shegalov
>Assignee: Harsh J
>Priority: Critical
> Fix For: 1.1.0
>
> Attachments: MR-4195
>
>
> When you access /jobqueue_details.jsp manually, instead of via a link, it has
> queueName set to null internally and this goes for a lookup into the
> scheduling info maps as well.
> As a result, if using FairScheduler, a Pool with String name = null gets
> created and this brings the scheduler down. I have not tested what happens to
> the CapacityScheduler, but ideally if no queueName is set in that jsp, it
> should fall back to 'default'. Otherwise, this brings down the JobTracker
> completely.
> FairScheduler must also add a check to not create a pool with 'null' name.
> The following is the strace that ensues:
> {code}
> ERROR org.mortbay.log: /jobqueue_details.jsp
> java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.jobqueue_005fdetails_jsp._jspService(jobqueue_005fdetails_jsp.java:71)
>
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>
> at
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:829)
>
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9001, call
> heartbeat from XYZ:MNOP: error: java.io.IOException:
> java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:95)
>
> at
> org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:68)
>
> at java.util.Arrays.mergeSort(Unknown Source)
> at java.util.Arrays.sort(Unknown Source)
> at java.util.Collections.sort(Unknown Source)
> at org.apache.hadoop.mapred.FairScheduler.assignTasks(FairScheduler.java:435)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3226)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.s
[jira] [Assigned] (MAPREDUCE-4195) With invalid queueName request param, jobqueue_details.jsp shows NPE
[
https://issues.apache.org/jira/browse/MAPREDUCE-4195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J reassigned MAPREDUCE-4195:
--
Assignee: (was: Harsh J)
> With invalid queueName request param, jobqueue_details.jsp shows NPE
>
>
> Key: MAPREDUCE-4195
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4195
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Affects Versions: 1.0.0
>Reporter: Gera Shegalov
>Priority: Critical
> Fix For: 1.1.0
>
> Attachments: MR-4195
>
>
> When you access /jobqueue_details.jsp manually, instead of via a link, it has
> queueName set to null internally and this goes for a lookup into the
> scheduling info maps as well.
> As a result, if using FairScheduler, a Pool with String name = null gets
> created and this brings the scheduler down. I have not tested what happens to
> the CapacityScheduler, but ideally if no queueName is set in that jsp, it
> should fall back to 'default'. Otherwise, this brings down the JobTracker
> completely.
> FairScheduler must also add a check to not create a pool with 'null' name.
> The following is the strace that ensues:
> {code}
> ERROR org.mortbay.log: /jobqueue_details.jsp
> java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.jobqueue_005fdetails_jsp._jspService(jobqueue_005fdetails_jsp.java:71)
>
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>
> at
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:829)
>
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9001, call
> heartbeat from XYZ:MNOP: error: java.io.IOException:
> java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:95)
>
> at
> org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:68)
>
> at java.util.Arrays.mergeSort(Unknown Source)
> at java.util.Arrays.sort(Unknown Source)
> at java.util.Collections.sort(Unknown Source)
> at org.apache.hadoop.mapred.FairScheduler.assignTasks(FairScheduler.java:435)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3226)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jir
[jira] [Commented] (MAPREDUCE-4198) Update RackResolver and related tests to resolve host with new API from HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual host besides a li
[ https://issues.apache.org/jira/browse/MAPREDUCE-4198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262378#comment-13262378 ] Hadoop QA commented on MAPREDUCE-4198: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524391/MAPREDUCE-4198-v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 2 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapreduce.TestYarnClientProtocolProvider org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2313//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2313//console This message is automatically generated. > Update RackResolver and related tests to resolve host with new API from > HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual > host besides a list of host) > --- > > Key: MAPREDUCE-4198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4198 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 1.0.0, 2.0.0 >Reporter: Junping Du >Assignee: Junping Du > Attachments: MAPREDUCE-4198-v2.patch, MAPREDUCE-4198.patch > > > HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual > host besides a list of host) will induce a new API to resolve individual host > rather than a list of host. Here is update on MapReduce part to use new API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4198) Update RackResolver and related tests to resolve host with new API from HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual host besides a list
[ https://issues.apache.org/jira/browse/MAPREDUCE-4198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-4198: -- Status: Patch Available (was: Open) > Update RackResolver and related tests to resolve host with new API from > HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual > host besides a list of host) > --- > > Key: MAPREDUCE-4198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4198 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 1.0.0, 2.0.0 >Reporter: Junping Du >Assignee: Junping Du > Attachments: MAPREDUCE-4198-v2.patch, MAPREDUCE-4198.patch > > > HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual > host besides a list of host) will induce a new API to resolve individual host > rather than a list of host. Here is update on MapReduce part to use new API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4198) Update RackResolver and related tests to resolve host with new API from HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual host besides a list
[ https://issues.apache.org/jira/browse/MAPREDUCE-4198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-4198: -- Attachment: MAPREDUCE-4198-v2.patch This is an update patch as previous one include whole changes with common > Update RackResolver and related tests to resolve host with new API from > HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual > host besides a list of host) > --- > > Key: MAPREDUCE-4198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4198 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 1.0.0, 2.0.0 >Reporter: Junping Du >Assignee: Junping Du > Attachments: MAPREDUCE-4198-v2.patch, MAPREDUCE-4198.patch > > > HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual > host besides a list of host) will induce a new API to resolve individual host > rather than a list of host. Here is update on MapReduce part to use new API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4198) Update RackResolver and related tests to resolve host with new API from HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual host besides a li
[ https://issues.apache.org/jira/browse/MAPREDUCE-4198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262357#comment-13262357 ] Hadoop QA commented on MAPREDUCE-4198: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524390/MAPREDUCE-4198.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2312//console This message is automatically generated. > Update RackResolver and related tests to resolve host with new API from > HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual > host besides a list of host) > --- > > Key: MAPREDUCE-4198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4198 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 1.0.0, 2.0.0 >Reporter: Junping Du >Assignee: Junping Du > Attachments: MAPREDUCE-4198.patch > > > HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual > host besides a list of host) will induce a new API to resolve individual host > rather than a list of host. Here is update on MapReduce part to use new API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4198) Update RackResolver and related tests to resolve host with new API from HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual host besides a list
[ https://issues.apache.org/jira/browse/MAPREDUCE-4198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-4198: -- Status: Open (was: Patch Available) > Update RackResolver and related tests to resolve host with new API from > HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual > host besides a list of host) > --- > > Key: MAPREDUCE-4198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4198 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 1.0.0, 2.0.0 >Reporter: Junping Du >Assignee: Junping Du > Attachments: MAPREDUCE-4198.patch > > > HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual > host besides a list of host) will induce a new API to resolve individual host > rather than a list of host. Here is update on MapReduce part to use new API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4198) Update RackResolver and related tests to resolve host with new API from HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual host besides a li
[ https://issues.apache.org/jira/browse/MAPREDUCE-4198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262354#comment-13262354 ] Junping Du commented on MAPREDUCE-4198: --- Sorry. This code looks wrong. Let me check again. > Update RackResolver and related tests to resolve host with new API from > HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual > host besides a list of host) > --- > > Key: MAPREDUCE-4198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4198 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 1.0.0, 2.0.0 >Reporter: Junping Du >Assignee: Junping Du > Attachments: MAPREDUCE-4198.patch > > > HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual > host besides a list of host) will induce a new API to resolve individual host > rather than a list of host. Here is update on MapReduce part to use new API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4198) Update RackResolver and related tests to resolve host with new API from HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual host besides a lis
[ https://issues.apache.org/jira/browse/MAPREDUCE-4198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins reassigned MAPREDUCE-4198: -- Assignee: Junping Du > Update RackResolver and related tests to resolve host with new API from > HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual > host besides a list of host) > --- > > Key: MAPREDUCE-4198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4198 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 1.0.0, 2.0.0 >Reporter: Junping Du >Assignee: Junping Du > Attachments: MAPREDUCE-4198.patch > > > HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual > host besides a list of host) will induce a new API to resolve individual host > rather than a list of host. Here is update on MapReduce part to use new API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4198) Update RackResolver and related tests to resolve host with new API from HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual host besides a list
[ https://issues.apache.org/jira/browse/MAPREDUCE-4198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-4198: -- Attachment: MAPREDUCE-4198.patch This code is separated from HADOOP-8304's original patch. > Update RackResolver and related tests to resolve host with new API from > HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual > host besides a list of host) > --- > > Key: MAPREDUCE-4198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4198 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 1.0.0, 2.0.0 >Reporter: Junping Du > Attachments: MAPREDUCE-4198.patch > > > HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual > host besides a list of host) will induce a new API to resolve individual host > rather than a list of host. Here is update on MapReduce part to use new API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4198) Update RackResolver and related tests to resolve host with new API from HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual host besides a list
[ https://issues.apache.org/jira/browse/MAPREDUCE-4198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated MAPREDUCE-4198: -- Status: Patch Available (was: Open) > Update RackResolver and related tests to resolve host with new API from > HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual > host besides a list of host) > --- > > Key: MAPREDUCE-4198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4198 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Affects Versions: 1.0.0, 2.0.0 >Reporter: Junping Du > Attachments: MAPREDUCE-4198.patch > > > HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual > host besides a list of host) will induce a new API to resolve individual host > rather than a list of host. Here is update on MapReduce part to use new API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4198) Update RackResolver and related tests to resolve host with new API from HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual host besides a list
Junping Du created MAPREDUCE-4198: - Summary: Update RackResolver and related tests to resolve host with new API from HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual host besides a list of host) Key: MAPREDUCE-4198 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4198 Project: Hadoop Map/Reduce Issue Type: Improvement Affects Versions: 1.0.0, 2.0.0 Reporter: Junping Du HADOOP-8304 (DNSToSwitchMapping should add interface to resolve individual host besides a list of host) will induce a new API to resolve individual host rather than a list of host. Here is update on MapReduce part to use new API. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4197) Include the hsqldb jar in the hadoop-mapreduce tar file
[
https://issues.apache.org/jira/browse/MAPREDUCE-4197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262306#comment-13262306
]
Ravi Prakash commented on MAPREDUCE-4197:
-
Hi Eli! Its actually the mapreduce-examples which need the hsqldb Server class.
However, hadoop-mapreduce-dist.xml (in hadoop-assemblies) seems to package only
the dependencies in hadoop-mapreduce/pom.xml .
(hadoop-mapreduce-examples/pom.xml already has hsqldb listed as a dependency,
but its still not getting packaged in the tarball)
The test results from Hadoop QA are spurious. I ran both of them on my local
machine and they passed. I suspect they failed because .m2 cache needs to be
refreshed after MAPREDUCE-3942 went in. No tests were included because this is
a change only in the pom.xml file.
> Include the hsqldb jar in the hadoop-mapreduce tar file
> ---
>
> Key: MAPREDUCE-4197
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4197
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.23.1
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: MAPREDUCE-4197.patch
>
>
> Courtesy Brahma
> {quote}
> In the previuos hadoop releases(20.XX) hsqldb was provided.
> But in hadoop-2.0.0 it is not present.Is it intentionally deleted or missing?
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4169) Container Logs appear in unsorted order
[ https://issues.apache.org/jira/browse/MAPREDUCE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262295#comment-13262295 ] Jonathan Eagles commented on MAPREDUCE-4169: Current test failures are preexisting. > Container Logs appear in unsorted order > --- > > Key: MAPREDUCE-4169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4169 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Minor > Attachments: MAPREDUCE-4169.patch, MAPREDUCE-4169.patch, > MAPREDUCE-4169.patch > > > container logs (stdout, stderr, syslog) in the nodemanager ui and jobhistory > ui appear in unsorted order where the order displayed is based on what file > was created first. This jira will have the results be displayed in a > consistent order. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4169) Container Logs appear in unsorted order
[ https://issues.apache.org/jira/browse/MAPREDUCE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-4169: --- Target Version/s: 0.23.3, 2.0.0 (was: 2.0.0, 0.23.3) Status: Patch Available (was: Open) Apparently, I was too eager and looked at the old test results. > Container Logs appear in unsorted order > --- > > Key: MAPREDUCE-4169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4169 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Minor > Attachments: MAPREDUCE-4169.patch, MAPREDUCE-4169.patch, > MAPREDUCE-4169.patch > > > container logs (stdout, stderr, syslog) in the nodemanager ui and jobhistory > ui appear in unsorted order where the order displayed is based on what file > was created first. This jira will have the results be displayed in a > consistent order. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4197) Include the hsqldb jar in the hadoop-mapreduce tar file
[
https://issues.apache.org/jira/browse/MAPREDUCE-4197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262284#comment-13262284
]
Eli Collins commented on MAPREDUCE-4197:
Given that it's only used by the MR client we only need to update
hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml right?
> Include the hsqldb jar in the hadoop-mapreduce tar file
> ---
>
> Key: MAPREDUCE-4197
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4197
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.23.1
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: MAPREDUCE-4197.patch
>
>
> Courtesy Brahma
> {quote}
> In the previuos hadoop releases(20.XX) hsqldb was provided.
> But in hadoop-2.0.0 it is not present.Is it intentionally deleted or missing?
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4197) Include the hsqldb jar in the hadoop-mapreduce tar file
[
https://issues.apache.org/jira/browse/MAPREDUCE-4197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262260#comment-13262260
]
Hadoop QA commented on MAPREDUCE-4197:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12524367/MAPREDUCE-4197.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.TestContainerManagerSecurity
org.apache.hadoop.mapreduce.TestYarnClientProtocolProvider
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2311//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2311//console
This message is automatically generated.
> Include the hsqldb jar in the hadoop-mapreduce tar file
> ---
>
> Key: MAPREDUCE-4197
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4197
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.23.1
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: MAPREDUCE-4197.patch
>
>
> Courtesy Brahma
> {quote}
> In the previuos hadoop releases(20.XX) hsqldb was provided.
> But in hadoop-2.0.0 it is not present.Is it intentionally deleted or missing?
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4169) Container Logs appear in unsorted order
[ https://issues.apache.org/jira/browse/MAPREDUCE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262225#comment-13262225 ] Hadoop QA commented on MAPREDUCE-4169: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524357/MAPREDUCE-4169.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapreduce.TestYarnClientProtocolProvider org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2310//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2310//console This message is automatically generated. > Container Logs appear in unsorted order > --- > > Key: MAPREDUCE-4169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4169 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Minor > Attachments: MAPREDUCE-4169.patch, MAPREDUCE-4169.patch, > MAPREDUCE-4169.patch > > > container logs (stdout, stderr, syslog) in the nodemanager ui and jobhistory > ui appear in unsorted order where the order displayed is based on what file > was created first. This jira will have the results be displayed in a > consistent order. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4197) Include the hsqldb jar in the hadoop-mapreduce tar file
[
https://issues.apache.org/jira/browse/MAPREDUCE-4197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash updated MAPREDUCE-4197:
Status: Patch Available (was: Open)
> Include the hsqldb jar in the hadoop-mapreduce tar file
> ---
>
> Key: MAPREDUCE-4197
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4197
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.23.1
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: MAPREDUCE-4197.patch
>
>
> Courtesy Brahma
> {quote}
> In the previuos hadoop releases(20.XX) hsqldb was provided.
> But in hadoop-2.0.0 it is not present.Is it intentionally deleted or missing?
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4197) Include the hsqldb jar in the hadoop-mapreduce tar file
[
https://issues.apache.org/jira/browse/MAPREDUCE-4197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ravi Prakash updated MAPREDUCE-4197:
Attachment: MAPREDUCE-4197.patch
This patch adds hsqldb to the dependencies of hadoop-mapreduce. This list of
dependencies is then used to assemble the tar file in
hadoop-assemblies->hadoop-mapreduce-dist.xml
Can someone please review and commit this small change?
> Include the hsqldb jar in the hadoop-mapreduce tar file
> ---
>
> Key: MAPREDUCE-4197
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4197
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 0.23.1
>Reporter: Ravi Prakash
>Assignee: Ravi Prakash
> Attachments: MAPREDUCE-4197.patch
>
>
> Courtesy Brahma
> {quote}
> In the previuos hadoop releases(20.XX) hsqldb was provided.
> But in hadoop-2.0.0 it is not present.Is it intentionally deleted or missing?
> {quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4197) Include the hsqldb jar in the hadoop-mapreduce tar file
Ravi Prakash created MAPREDUCE-4197:
---
Summary: Include the hsqldb jar in the hadoop-mapreduce tar file
Key: MAPREDUCE-4197
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4197
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 0.23.1
Reporter: Ravi Prakash
Assignee: Ravi Prakash
Courtesy Brahma
{quote}
In the previuos hadoop releases(20.XX) hsqldb was provided.
But in hadoop-2.0.0 it is not present.Is it intentionally deleted or missing?
{quote}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4169) Container Logs appear in unsorted order
[ https://issues.apache.org/jira/browse/MAPREDUCE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-4169: --- Target Version/s: 0.23.3, 2.0.0 (was: 2.0.0, 0.23.3) Status: Open (was: Patch Available) Canceling to address test failures > Container Logs appear in unsorted order > --- > > Key: MAPREDUCE-4169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4169 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Minor > Attachments: MAPREDUCE-4169.patch, MAPREDUCE-4169.patch, > MAPREDUCE-4169.patch > > > container logs (stdout, stderr, syslog) in the nodemanager ui and jobhistory > ui appear in unsorted order where the order displayed is based on what file > was created first. This jira will have the results be displayed in a > consistent order. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4169) Container Logs appear in unsorted order
[ https://issues.apache.org/jira/browse/MAPREDUCE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262150#comment-13262150 ] Jonathan Eagles commented on MAPREDUCE-4169: Addressed Bobby's comments as part of MAPREDUCE-4194. Posting new patch. > Container Logs appear in unsorted order > --- > > Key: MAPREDUCE-4169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4169 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Minor > Attachments: MAPREDUCE-4169.patch, MAPREDUCE-4169.patch, > MAPREDUCE-4169.patch > > > container logs (stdout, stderr, syslog) in the nodemanager ui and jobhistory > ui appear in unsorted order where the order displayed is based on what file > was created first. This jira will have the results be displayed in a > consistent order. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4169) Container Logs appear in unsorted order
[ https://issues.apache.org/jira/browse/MAPREDUCE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-4169: --- Target Version/s: 0.23.3, 2.0.0 (was: 2.0.0, 0.23.3) Status: Patch Available (was: Open) > Container Logs appear in unsorted order > --- > > Key: MAPREDUCE-4169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4169 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Minor > Attachments: MAPREDUCE-4169.patch, MAPREDUCE-4169.patch, > MAPREDUCE-4169.patch > > > container logs (stdout, stderr, syslog) in the nodemanager ui and jobhistory > ui appear in unsorted order where the order displayed is based on what file > was created first. This jira will have the results be displayed in a > consistent order. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4169) Container Logs appear in unsorted order
[ https://issues.apache.org/jira/browse/MAPREDUCE-4169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-4169: --- Attachment: MAPREDUCE-4169.patch > Container Logs appear in unsorted order > --- > > Key: MAPREDUCE-4169 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4169 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Minor > Attachments: MAPREDUCE-4169.patch, MAPREDUCE-4169.patch, > MAPREDUCE-4169.patch > > > container logs (stdout, stderr, syslog) in the nodemanager ui and jobhistory > ui appear in unsorted order where the order displayed is based on what file > was created first. This jira will have the results be displayed in a > consistent order. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4138) Reduce memory usage of counters due to non-static nested classes
[ https://issues.apache.org/jira/browse/MAPREDUCE-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262147#comment-13262147 ] Robert Joseph Evans commented on MAPREDUCE-4138: +1 if you want me to commit this tom just let me know. > Reduce memory usage of counters due to non-static nested classes > > > Key: MAPREDUCE-4138 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4138 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Tom White >Assignee: Tom White > Attachments: MAPREDUCE-4138.patch, MAPREDUCE-4138.patch > > > FrameworkCounter is a non-static nested class of FrameworkCounterGroup which > means it retains a reference to the outer class, which isn't really needed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3613) web service calls header contains 2 content types
[ https://issues.apache.org/jira/browse/MAPREDUCE-3613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262143#comment-13262143 ] Hudson commented on MAPREDUCE-3613: --- Integrated in Hadoop-Mapreduce-trunk-Commit #2151 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2151/]) MAPREDUCE-3613. web service calls header contains 2 content types (tgraves) (Revision 1330560) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330560 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMWebServices.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java > web service calls header contains 2 content types > - > > Key: MAPREDUCE-3613 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3613 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Fix For: 0.23.3 > > Attachments: MAPREDUCE-3613.patch > > > when doing requesting info from the web services rest API, curl seems to > return content-type of both text and json or xml: > > Accept: application/xml > > > < HTTP/1.1 200 OK > < Content-Type: text/plain; charset=utf-8 > < Content-Type: application/xml -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4148) MapReduce should not have a compile-time dependency on HDFS
[
https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262139#comment-13262139
]
Daryn Sharp commented on MAPREDUCE-4148:
Very nice! I've been wanting to do something similar to allow tokens to decode
their identifiers for quite awhile now. Thoughts/suggestions:
{{Token#decodeIdentifier()}} would be _really_ useful and prevent callers from
having to know about the factory. This would very nearly abstract away the
details. It could either delegate to the factory as-is, or just query the
factory for the class, or maybe even {{Token}} can host the kind/class
registration method. There are a number of other places in the code, ex.
{{AbstractDelegationTokenSecretManager#renewToken(Token)}} that could benefit
from such a method.
I'm not too fond of
{{AbstractDelegationTokenIdentifier#stringifyToken(Token)}}. It creates a
circular relationship. A {{TokenIdentifier}} really shouldn't have to know
about a {{Token}} wrapper. How removing the inversion of knowledge, and update
{{Token#toString()}} to use {{token.decodeIdentifier()}}? If the value is null
because the class isn't available, it can print the raw bytes like it does now.
{{TokenIdentifierFactory.createIdentifier}} is using
{{ReflectionUtils.newInstance(Configuration)}} whose main purpose is to invoke
{{setConf(conf)}} and/or MR's {{configure(conf)}} which isn't applicable in
this case. How about directly using {{class.newInstance()}}?
I think there's pitfalls with using static class inits for the factory
registration. The identifier class has to be loaded (sorry if I'm stating the
obvious: not just imported, but something referenced from it) before a token
can be decoded. In general this probably means a token has to be created
before other tokens can be decoded. Perhaps the static blocks should become a
static class method that's invoked by the secret manager, since we know the
secret manager is instantiated before token manipulation. Although, that
limits token ident decoding only to tokens owned by the daemon, which would
leave a client out of luck. Maybe you could get fancy with a class loader.
> MapReduce should not have a compile-time dependency on HDFS
> ---
>
> Key: MAPREDUCE-4148
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Tom White
>Assignee: Tom White
> Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch
>
>
> MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token
> debug information). We should remove this dependency and MapReduce's
> compile-time dependency on HDFS.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3613) web service calls header contains 2 content types
[ https://issues.apache.org/jira/browse/MAPREDUCE-3613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262128#comment-13262128 ] Hudson commented on MAPREDUCE-3613: --- Integrated in Hadoop-Common-trunk-Commit #2134 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2134/]) MAPREDUCE-3613. web service calls header contains 2 content types (tgraves) (Revision 1330560) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330560 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMWebServices.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java > web service calls header contains 2 content types > - > > Key: MAPREDUCE-3613 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3613 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Fix For: 0.23.3 > > Attachments: MAPREDUCE-3613.patch > > > when doing requesting info from the web services rest API, curl seems to > return content-type of both text and json or xml: > > Accept: application/xml > > > < HTTP/1.1 200 OK > < Content-Type: text/plain; charset=utf-8 > < Content-Type: application/xml -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3613) web service calls header contains 2 content types
[ https://issues.apache.org/jira/browse/MAPREDUCE-3613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262125#comment-13262125 ] Hudson commented on MAPREDUCE-3613: --- Integrated in Hadoop-Hdfs-trunk-Commit #2208 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2208/]) MAPREDUCE-3613. web service calls header contains 2 content types (tgraves) (Revision 1330560) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330560 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/webapp/AMWebServices.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/webapp/NMWebServices.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/RMWebServices.java > web service calls header contains 2 content types > - > > Key: MAPREDUCE-3613 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3613 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Fix For: 0.23.3 > > Attachments: MAPREDUCE-3613.patch > > > when doing requesting info from the web services rest API, curl seems to > return content-type of both text and json or xml: > > Accept: application/xml > > > < HTTP/1.1 200 OK > < Content-Type: text/plain; charset=utf-8 > < Content-Type: application/xml -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3613) web service calls header contains 2 content types
[ https://issues.apache.org/jira/browse/MAPREDUCE-3613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-3613: - Resolution: Fixed Fix Version/s: 0.23.3 Status: Resolved (was: Patch Available) thanks for the review Bobby! I've committed this to trunk, branch-2, and branch-0.23 > web service calls header contains 2 content types > - > > Key: MAPREDUCE-3613 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3613 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Fix For: 0.23.3 > > Attachments: MAPREDUCE-3613.patch > > > when doing requesting info from the web services rest API, curl seems to > return content-type of both text and json or xml: > > Accept: application/xml > > > < HTTP/1.1 200 OK > < Content-Type: text/plain; charset=utf-8 > < Content-Type: application/xml -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4196) nodemanager doesn't cleanup old logs/local files after restart
Thomas Graves created MAPREDUCE-4196: Summary: nodemanager doesn't cleanup old logs/local files after restart Key: MAPREDUCE-4196 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4196 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, nodemanager Affects Versions: 0.23.3 Reporter: Thomas Graves The nodemanager doesn't ever remove old logs or local files when it is restarted. This is very noticeable if you are using yarn.nodemanager.delete.debug-delay-sec set to > 0 and restart your nodemanager. This could happen in other situations though also, for instance if the node manager crashes/is restarted with running containers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4194) ConcurrentModificationError in DirectoryCollection
[ https://issues.apache.org/jira/browse/MAPREDUCE-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262091#comment-13262091 ] Hudson commented on MAPREDUCE-4194: --- Integrated in Hadoop-Mapreduce-trunk-Commit #2150 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2150/]) MAPREDUCE-4194. ConcurrentModificationError in DirectoryCollection (Jonathan Eagles via bobby) (Revision 1330552) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330552 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java > ConcurrentModificationError in DirectoryCollection > -- > > Key: MAPREDUCE-4194 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4194 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Fix For: 0.23.3, 2.0.0, 3.0.0 > > Attachments: MAPREDUCE-4194.patch, MAPREDUCE-4194.patch > > > As found as part of work on MAPREDUCE-4169, it is possible for a > ConcurrentModificationException to be thrown upon disk failure. > DirectoryCollection hands out its internal list structure that is accessed > across multiple threads. Upon disk failure its internal list is modified, > invalidating all current iterators to that structure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4194) ConcurrentModificationError in DirectoryCollection
[ https://issues.apache.org/jira/browse/MAPREDUCE-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262079#comment-13262079 ] Hudson commented on MAPREDUCE-4194: --- Integrated in Hadoop-Common-trunk-Commit #2133 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2133/]) MAPREDUCE-4194. ConcurrentModificationError in DirectoryCollection (Jonathan Eagles via bobby) (Revision 1330552) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330552 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java > ConcurrentModificationError in DirectoryCollection > -- > > Key: MAPREDUCE-4194 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4194 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Fix For: 0.23.3, 2.0.0, 3.0.0 > > Attachments: MAPREDUCE-4194.patch, MAPREDUCE-4194.patch > > > As found as part of work on MAPREDUCE-4169, it is possible for a > ConcurrentModificationException to be thrown upon disk failure. > DirectoryCollection hands out its internal list structure that is accessed > across multiple threads. Upon disk failure its internal list is modified, > invalidating all current iterators to that structure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3883) Document yarn.nodemanager.delete.debug-delay-sec configuration property
[
https://issues.apache.org/jira/browse/MAPREDUCE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262082#comment-13262082
]
Thomas Graves commented on MAPREDUCE-3883:
--
Thanks Eugene, great description. could you update it to wrap at 80 characters.
It also affects how long the application container logs are saved, so you might
comment on that too. The location of the logs are detailed in
yarn.nodemanager.log-dirs.
> Document yarn.nodemanager.delete.debug-delay-sec configuration property
> ---
>
> Key: MAPREDUCE-3883
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3883
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: documentation, mrv2
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
>Priority: Minor
> Labels: newbie
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3883.patch
>
>
> If you are using Yarn's nodemanager, you can add to your configuration:
> {noformat}
>
> yarn.nodemanager.delete.debug-delay-sec
> 1000
>
> {noformat}
> to save the environmental directories of the applications (by default in
> {{/tmp/nm-local-dir}}) that the nodemanager starts so that you can examine
> them later.
> Set the above value long enough so that you have time to examine the contents
> before the nodemanager's {{DeletionService}} removes them.
> This setting is defined in
> {{hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java}},
> and used in
> {{hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java}}.
> It's a useful configuration setting for developers, but seems not to be
> documented currently (see
> http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/yarn-default.xml).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3883) Document yarn.nodemanager.delete.debug-delay-sec configuration property
[
https://issues.apache.org/jira/browse/MAPREDUCE-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves updated MAPREDUCE-3883:
-
Target Version/s: 2.0.0
Status: Open (was: Patch Available)
> Document yarn.nodemanager.delete.debug-delay-sec configuration property
> ---
>
> Key: MAPREDUCE-3883
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3883
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: documentation, mrv2
>Reporter: Eugene Koontz
>Assignee: Eugene Koontz
>Priority: Minor
> Labels: newbie
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3883.patch
>
>
> If you are using Yarn's nodemanager, you can add to your configuration:
> {noformat}
>
> yarn.nodemanager.delete.debug-delay-sec
> 1000
>
> {noformat}
> to save the environmental directories of the applications (by default in
> {{/tmp/nm-local-dir}}) that the nodemanager starts so that you can examine
> them later.
> Set the above value long enough so that you have time to examine the contents
> before the nodemanager's {{DeletionService}} removes them.
> This setting is defined in
> {{hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java}},
> and used in
> {{hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DeletionService.java}}.
> It's a useful configuration setting for developers, but seems not to be
> documented currently (see
> http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/yarn-default.xml).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4194) ConcurrentModificationError in DirectoryCollection
[ https://issues.apache.org/jira/browse/MAPREDUCE-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262076#comment-13262076 ] Jonathan Eagles commented on MAPREDUCE-4194: Thanks, Bobby! > ConcurrentModificationError in DirectoryCollection > -- > > Key: MAPREDUCE-4194 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4194 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Fix For: 0.23.3, 2.0.0, 3.0.0 > > Attachments: MAPREDUCE-4194.patch, MAPREDUCE-4194.patch > > > As found as part of work on MAPREDUCE-4169, it is possible for a > ConcurrentModificationException to be thrown upon disk failure. > DirectoryCollection hands out its internal list structure that is accessed > across multiple threads. Upon disk failure its internal list is modified, > invalidating all current iterators to that structure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4194) ConcurrentModificationError in DirectoryCollection
[ https://issues.apache.org/jira/browse/MAPREDUCE-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262073#comment-13262073 ] Hudson commented on MAPREDUCE-4194: --- Integrated in Hadoop-Hdfs-trunk-Commit #2207 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2207/]) MAPREDUCE-4194. ConcurrentModificationError in DirectoryCollection (Jonathan Eagles via bobby) (Revision 1330552) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330552 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DirectoryCollection.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java > ConcurrentModificationError in DirectoryCollection > -- > > Key: MAPREDUCE-4194 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4194 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Fix For: 0.23.3, 2.0.0, 3.0.0 > > Attachments: MAPREDUCE-4194.patch, MAPREDUCE-4194.patch > > > As found as part of work on MAPREDUCE-4169, it is possible for a > ConcurrentModificationException to be thrown upon disk failure. > DirectoryCollection hands out its internal list structure that is accessed > across multiple threads. Upon disk failure its internal list is modified, > invalidating all current iterators to that structure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4194) ConcurrentModificationError in DirectoryCollection
[ https://issues.apache.org/jira/browse/MAPREDUCE-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4194: --- Resolution: Fixed Fix Version/s: 3.0.0 2.0.0 0.23.3 Target Version/s: 0.23.3, 2.0.0 (was: 2.0.0, 0.23.3) Status: Resolved (was: Patch Available) Thanks Jon, I put this into trunk, branch-2, and branch-0.23. > ConcurrentModificationError in DirectoryCollection > -- > > Key: MAPREDUCE-4194 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4194 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Fix For: 0.23.3, 2.0.0, 3.0.0 > > Attachments: MAPREDUCE-4194.patch, MAPREDUCE-4194.patch > > > As found as part of work on MAPREDUCE-4169, it is possible for a > ConcurrentModificationException to be thrown upon disk failure. > DirectoryCollection hands out its internal list structure that is accessed > across multiple threads. Upon disk failure its internal list is modified, > invalidating all current iterators to that structure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3613) web service calls header contains 2 content types
[ https://issues.apache.org/jira/browse/MAPREDUCE-3613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262059#comment-13262059 ] Thomas Graves commented on MAPREDUCE-3613: -- I agree with you totally. I wanted to change it in HttpServer but saw that as a bigger effort that we could do later. > web service calls header contains 2 content types > - > > Key: MAPREDUCE-3613 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3613 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Attachments: MAPREDUCE-3613.patch > > > when doing requesting info from the web services rest API, curl seems to > return content-type of both text and json or xml: > > Accept: application/xml > > > < HTTP/1.1 200 OK > < Content-Type: text/plain; charset=utf-8 > < Content-Type: application/xml -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4194) ConcurrentModificationError in DirectoryCollection
[ https://issues.apache.org/jira/browse/MAPREDUCE-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262047#comment-13262047 ] Jonathan Eagles commented on MAPREDUCE-4194: Tests above are unrelated. > ConcurrentModificationError in DirectoryCollection > -- > > Key: MAPREDUCE-4194 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4194 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-4194.patch, MAPREDUCE-4194.patch > > > As found as part of work on MAPREDUCE-4169, it is possible for a > ConcurrentModificationException to be thrown upon disk failure. > DirectoryCollection hands out its internal list structure that is accessed > across multiple threads. Upon disk failure its internal list is modified, > invalidating all current iterators to that structure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3613) web service calls header contains 2 content types
[ https://issues.apache.org/jira/browse/MAPREDUCE-3613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262038#comment-13262038 ] Robert Joseph Evans commented on MAPREDUCE-3613: I don't really like the fix because it requires all of the jersey methods to be modified, in exactly the same way, but that is the exact same fix as HDFS-2441 did so I guess it is OK. The root of the issue appears to be that HttpServer.QuotingInputFilter is trying to guess what the content type is based off of the URI of the request, and always setting it no matter what. This seems to be so that we can set the charset of the response to be utf-8. This is coupled with Jersey also outputting the content type instead of overwriting the one that was set previously. Also it makes it more difficult for them to be called as regular methods.I really would prefer to see something with the server where we could fix some of this in HttpServer itself so that we don't get the automatic content type for web services. But that is likely to be a bigger change and would be a bit more cross project. Sadly I tried upgrading to Jersey 1.12 and it does not fix the issue. So reluctantly I give this a +1. > web service calls header contains 2 content types > - > > Key: MAPREDUCE-3613 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3613 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Attachments: MAPREDUCE-3613.patch > > > when doing requesting info from the web services rest API, curl seems to > return content-type of both text and json or xml: > > Accept: application/xml > > > < HTTP/1.1 200 OK > < Content-Type: text/plain; charset=utf-8 > < Content-Type: application/xml -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4195) With invalid queueName request param, jobqueue_details.jsp shows NPE
[
https://issues.apache.org/jira/browse/MAPREDUCE-4195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gera Shegalov updated MAPREDUCE-4195:
-
Attachment: MR-4195
The fix in MAPREDUCE-3674 was incomplete. In
src/webapps/job/jobqueue_details.jsp, a null check is needed for schedInfo,
when the queueName is invalid. This fix makes sure that a meaningful info is
printed instead of NPE.
> With invalid queueName request param, jobqueue_details.jsp shows NPE
>
>
> Key: MAPREDUCE-4195
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4195
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Affects Versions: 1.0.0
>Reporter: Gera Shegalov
>Assignee: Harsh J
>Priority: Critical
> Fix For: 1.1.0
>
> Attachments: MR-4195
>
>
> When you access /jobqueue_details.jsp manually, instead of via a link, it has
> queueName set to null internally and this goes for a lookup into the
> scheduling info maps as well.
> As a result, if using FairScheduler, a Pool with String name = null gets
> created and this brings the scheduler down. I have not tested what happens to
> the CapacityScheduler, but ideally if no queueName is set in that jsp, it
> should fall back to 'default'. Otherwise, this brings down the JobTracker
> completely.
> FairScheduler must also add a check to not create a pool with 'null' name.
> The following is the strace that ensues:
> {code}
> ERROR org.mortbay.log: /jobqueue_details.jsp
> java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.jobqueue_005fdetails_jsp._jspService(jobqueue_005fdetails_jsp.java:71)
>
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>
> at
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:829)
>
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9001, call
> heartbeat from XYZ:MNOP: error: java.io.IOException:
> java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:95)
>
> at
> org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:68)
>
> at java.util.Arrays.mergeSort(Unknown Source)
> at java.util.Arrays.sort(Unknown Source)
> at java.util.Collections.sort(Unknown Source)
> at org.apache.hadoop.mapred.FairScheduler.assignTasks(FairScheduler.java:435)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3226)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)
> {code}
--
This mes
[jira] [Commented] (MAPREDUCE-4194) ConcurrentModificationError in DirectoryCollection
[ https://issues.apache.org/jira/browse/MAPREDUCE-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13262026#comment-13262026 ] Hadoop QA commented on MAPREDUCE-4194: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524324/MAPREDUCE-4194.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.mapreduce.TestYarnClientProtocolProvider +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2309//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2309//console This message is automatically generated. > ConcurrentModificationError in DirectoryCollection > -- > > Key: MAPREDUCE-4194 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4194 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-4194.patch, MAPREDUCE-4194.patch > > > As found as part of work on MAPREDUCE-4169, it is possible for a > ConcurrentModificationException to be thrown upon disk failure. > DirectoryCollection hands out its internal list structure that is accessed > across multiple threads. Upon disk failure its internal list is modified, > invalidating all current iterators to that structure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4195) With invalid queueName request param, jobqueue_details.jsp shows NPE
Gera Shegalov created MAPREDUCE-4195:
Summary: With invalid queueName request param,
jobqueue_details.jsp shows NPE
Key: MAPREDUCE-4195
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4195
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 1.0.0
Reporter: Gera Shegalov
Assignee: Harsh J
Priority: Critical
Fix For: 1.1.0
When you access /jobqueue_details.jsp manually, instead of via a link, it has
queueName set to null internally and this goes for a lookup into the scheduling
info maps as well.
As a result, if using FairScheduler, a Pool with String name = null gets
created and this brings the scheduler down. I have not tested what happens to
the CapacityScheduler, but ideally if no queueName is set in that jsp, it
should fall back to 'default'. Otherwise, this brings down the JobTracker
completely.
FairScheduler must also add a check to not create a pool with 'null' name.
The following is the strace that ensues:
{code}
ERROR org.mortbay.log: /jobqueue_details.jsp
java.lang.NullPointerException
at
org.apache.hadoop.mapred.jobqueue_005fdetails_jsp._jspService(jobqueue_005fdetails_jsp.java:71)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:829)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9001, call heartbeat
from XYZ:MNOP: error: java.io.IOException: java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
at
org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:95)
at
org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:68)
at java.util.Arrays.mergeSort(Unknown Source)
at java.util.Arrays.sort(Unknown Source)
at java.util.Collections.sort(Unknown Source)
at org.apache.hadoop.mapred.FairScheduler.assignTasks(FairScheduler.java:435)
at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3226)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4102) job counters not available in Jobhistory webui for killed jobs
[ https://issues.apache.org/jira/browse/MAPREDUCE-4102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4102: - Status: Open (was: Patch Available) Sorry looks like we took to long to get to this, the patch no longer applies. Can you please upmerge to latest? > job counters not available in Jobhistory webui for killed jobs > -- > > Key: MAPREDUCE-4102 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4102 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: webapps >Affects Versions: 0.23.2, 2.0.0 >Reporter: Thomas Graves > Attachments: MAPREDUCE-4102-1.patch, MAPREDUCE-4102.patch > > > Run a simple wordcount or sleep, and kill the job before it finishes. Go to > the job history web ui and click the "Counters" link for that job. It > displays "500 error". > The job history log has: > Caused by: com.google.inject.ProvisionException: Guice provision errors: > 2012-04-03 19:42:53,148 ERROR org.apache.hadoop.yarn.webapp.Dispatcher: error > handling URI: /jobhistory/jobcounters/job_1333482028750_0001 > java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > ... > .. > ... > 1) Error injecting constructor, java.lang.NullPointerException > at > org.apache.hadoop.mapreduce.v2.app.webapp.CountersBlock.(CountersBlock.java:56) > while locating org.apache.hadoop.mapreduce.v2.app.webapp.CountersBlock > ... > .. > ... > Caused by: java.lang.NullPointerExceptionat > org.apache.hadoop.mapreduce.counters.AbstractCounters.incrAllCounters(AbstractCounters.java:328) > at > org.apache.hadoop.mapreduce.v2.app.webapp.CountersBlock.getCounters(CountersBlock.java:188) > at > org.apache.hadoop.mapreduce.v2.app.webapp.CountersBlock.(CountersBlock.java:57) > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > There are task counters available if you drill down into successful tasks > though. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3940) ContainerTokens should have an expiry interval
[ https://issues.apache.org/jira/browse/MAPREDUCE-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261977#comment-13261977 ] Hadoop QA commented on MAPREDUCE-3940: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524315/MAPREDUCE-3940-20120425.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs org.apache.hadoop.mapred.TestClientRedirect org.apache.hadoop.mapreduce.TestYarnClientProtocolProvider org.apache.hadoop.mapreduce.security.TestJHSSecurity +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2307//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2307//console This message is automatically generated. > ContainerTokens should have an expiry interval > -- > > Key: MAPREDUCE-3940 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3940 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3940-20120308.txt, > MAPREDUCE-3940-20120416.txt, MAPREDUCE-3940-20120425.txt > > > - RM should generate the expiry time for a container > - A ContainerToken should have its expire time encoded > - NMs should reject containers with expired tokens. > - Expiry interval for a ContainerToken is same as the expiry interval for a > container. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3674) If invoked with no queueName request param, jobqueue_details.jsp injects a null queue name into schedulers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261976#comment-13261976
]
Harsh J commented on MAPREDUCE-3674:
Hello Gera,
Could you create a new issue for your patch and what it aims to fix (and what
was missed in this original one here)?
This JIRA has been resolved, so please open a new one with your patch and link
back to this one as a relation.
Thanks! :)
> If invoked with no queueName request param, jobqueue_details.jsp injects a
> null queue name into schedulers.
> ---
>
> Key: MAPREDUCE-3674
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3674
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Affects Versions: 1.0.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Critical
> Fix For: 1.1.0
>
> Attachments: MAPREDUCE-3674.patch, MAPREDUCE-3674.patch,
> MAPREDUCE-3674.patch, MAPREDUCE-3674.patch, MAPREDUCE-3674.patch,
> hadoop-findbugs-report.html
>
>
> When you access /jobqueue_details.jsp manually, instead of via a link, it has
> queueName set to null internally and this goes for a lookup into the
> scheduling info maps as well.
> As a result, if using FairScheduler, a Pool with String name = null gets
> created and this brings the scheduler down. I have not tested what happens to
> the CapacityScheduler, but ideally if no queueName is set in that jsp, it
> should fall back to 'default'. Otherwise, this brings down the JobTracker
> completely.
> FairScheduler must also add a check to not create a pool with 'null' name.
> The following is the strace that ensues:
> {code}
> ERROR org.mortbay.log: /jobqueue_details.jsp
> java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.jobqueue_005fdetails_jsp._jspService(jobqueue_005fdetails_jsp.java:71)
>
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>
> at
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:829)
>
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9001, call
> heartbeat from XYZ:MNOP: error: java.io.IOException:
> java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:95)
>
> at
> org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:68)
>
> at java.util.Arrays.mergeSort(Unknown Source)
> at java.util.Arrays.sort(Unknown Source)
> at java.util.Collections.sort(Unknown Source)
> at org.apache.hadoop.mapred.FairScheduler.assignTasks(FairScheduler.java:435)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3226)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)
> at java.security.AccessController.doPr
[jira] [Commented] (MAPREDUCE-4138) Reduce memory usage of counters due to non-static nested classes
[ https://issues.apache.org/jira/browse/MAPREDUCE-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261955#comment-13261955 ] Hadoop QA commented on MAPREDUCE-4138: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12524313/MAPREDUCE-4138.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.TestContainerManagerSecurity org.apache.hadoop.mapreduce.TestYarnClientProtocolProvider +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2306//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2306//console This message is automatically generated. > Reduce memory usage of counters due to non-static nested classes > > > Key: MAPREDUCE-4138 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4138 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Tom White >Assignee: Tom White > Attachments: MAPREDUCE-4138.patch, MAPREDUCE-4138.patch > > > FrameworkCounter is a non-static nested class of FrameworkCounterGroup which > means it retains a reference to the outer class, which isn't really needed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4148) MapReduce should not have a compile-time dependency on HDFS
[ https://issues.apache.org/jira/browse/MAPREDUCE-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-4148: - Attachment: MAPREDUCE-4148.patch > MapReduce should not have a compile-time dependency on HDFS > --- > > Key: MAPREDUCE-4148 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4148 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Tom White >Assignee: Tom White > Attachments: MAPREDUCE-4148.patch, MAPREDUCE-4148.patch > > > MapReduce depends on HDFS's DelegationTokenIdentifier (for printing token > debug information). We should remove this dependency and MapReduce's > compile-time dependency on HDFS. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4194) ConcurrentModificationError in DirectoryCollection
[ https://issues.apache.org/jira/browse/MAPREDUCE-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-4194: --- Target Version/s: 0.23.3, 2.0.0 (was: 2.0.0, 0.23.3) Status: Patch Available (was: Open) > ConcurrentModificationError in DirectoryCollection > -- > > Key: MAPREDUCE-4194 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4194 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-4194.patch, MAPREDUCE-4194.patch > > > As found as part of work on MAPREDUCE-4169, it is possible for a > ConcurrentModificationException to be thrown upon disk failure. > DirectoryCollection hands out its internal list structure that is accessed > across multiple threads. Upon disk failure its internal list is modified, > invalidating all current iterators to that structure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4194) ConcurrentModificationError in DirectoryCollection
[ https://issues.apache.org/jira/browse/MAPREDUCE-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261951#comment-13261951 ] Jonathan Eagles commented on MAPREDUCE-4194: Thanks for the review, Bobby. I have changed the test to put temp files in target//* and that dir is now cleaned up after the test class is run. Also I have changed the return type to an unmodifiableList, keeping the CopyOnWriteArrayList backing store. This prevents the ConcurrentModificationException while protecting the internal data structure from being accidentally being changed. Note the test is in a new file. > ConcurrentModificationError in DirectoryCollection > -- > > Key: MAPREDUCE-4194 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4194 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-4194.patch, MAPREDUCE-4194.patch > > > As found as part of work on MAPREDUCE-4169, it is possible for a > ConcurrentModificationException to be thrown upon disk failure. > DirectoryCollection hands out its internal list structure that is accessed > across multiple threads. Upon disk failure its internal list is modified, > invalidating all current iterators to that structure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4194) ConcurrentModificationError in DirectoryCollection
[ https://issues.apache.org/jira/browse/MAPREDUCE-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated MAPREDUCE-4194: --- Attachment: MAPREDUCE-4194.patch > ConcurrentModificationError in DirectoryCollection > -- > > Key: MAPREDUCE-4194 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4194 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-4194.patch, MAPREDUCE-4194.patch > > > As found as part of work on MAPREDUCE-4169, it is possible for a > ConcurrentModificationException to be thrown upon disk failure. > DirectoryCollection hands out its internal list structure that is accessed > across multiple threads. Upon disk failure its internal list is modified, > invalidating all current iterators to that structure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3613) web service calls header contains 2 content types
[ https://issues.apache.org/jira/browse/MAPREDUCE-3613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-3613: - Status: Patch Available (was: Open) No way to easily unit test this as we would have to start up the entire httpserver with the yarn web app framework and all, so no tests were added. The existing webservice tests will verify nothing was broken. I manually tested all the webservice calls via curl to make sure only the xml/json content type came out. > web service calls header contains 2 content types > - > > Key: MAPREDUCE-3613 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3613 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Attachments: MAPREDUCE-3613.patch > > > when doing requesting info from the web services rest API, curl seems to > return content-type of both text and json or xml: > > Accept: application/xml > > > < HTTP/1.1 200 OK > < Content-Type: text/plain; charset=utf-8 > < Content-Type: application/xml -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3613) web service calls header contains 2 content types
[ https://issues.apache.org/jira/browse/MAPREDUCE-3613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-3613: - Target Version/s: 0.23.3 (was: 0.23.0) > web service calls header contains 2 content types > - > > Key: MAPREDUCE-3613 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3613 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Attachments: MAPREDUCE-3613.patch > > > when doing requesting info from the web services rest API, curl seems to > return content-type of both text and json or xml: > > Accept: application/xml > > > < HTTP/1.1 200 OK > < Content-Type: text/plain; charset=utf-8 > < Content-Type: application/xml -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3613) web service calls header contains 2 content types
[ https://issues.apache.org/jira/browse/MAPREDUCE-3613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-3613: - Attachment: MAPREDUCE-3613.patch gets rid of the extra content-type header added by the generic HttpServer. > web service calls header contains 2 content types > - > > Key: MAPREDUCE-3613 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3613 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Critical > Attachments: MAPREDUCE-3613.patch > > > when doing requesting info from the web services rest API, curl seems to > return content-type of both text and json or xml: > > Accept: application/xml > > > < HTTP/1.1 200 OK > < Content-Type: text/plain; charset=utf-8 > < Content-Type: application/xml -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4124) Job Priority is not changing
[ https://issues.apache.org/jira/browse/MAPREDUCE-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261930#comment-13261930 ] Robert Joseph Evans commented on MAPREDUCE-4124: I am fine with making sure that the priority information is passed up to the scheduler, and if you want to update the FIFO scheduler to go off of priority as well I don't see much of an issue with that either. But for any other scheduler I would like to see that work done as a separate JIRA because it is likely to be a lot of work, and may be somewhat controversial. > Job Priority is not changing > - > > Key: MAPREDUCE-4124 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4124 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 3.0.0 >Reporter: Nishan Shetty >Priority: Critical > > 1.Submit job > 2.Change the job priority using setPriority() or CLI command ./mapred > job-set-priority > Observe that Job priority is not changed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4138) Reduce memory usage of counters due to non-static nested classes
[ https://issues.apache.org/jira/browse/MAPREDUCE-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261923#comment-13261923 ] Robert Joseph Evans commented on MAPREDUCE-4138: s/Husdon/Jenkins/g > Reduce memory usage of counters due to non-static nested classes > > > Key: MAPREDUCE-4138 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4138 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Tom White >Assignee: Tom White > Attachments: MAPREDUCE-4138.patch, MAPREDUCE-4138.patch > > > FrameworkCounter is a non-static nested class of FrameworkCounterGroup which > means it retains a reference to the outer class, which isn't really needed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4138) Reduce memory usage of counters due to non-static nested classes
[ https://issues.apache.org/jira/browse/MAPREDUCE-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261922#comment-13261922 ] Robert Joseph Evans commented on MAPREDUCE-4138: Assuming Hudson comes back OK I am a +1 on this. > Reduce memory usage of counters due to non-static nested classes > > > Key: MAPREDUCE-4138 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4138 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Tom White >Assignee: Tom White > Attachments: MAPREDUCE-4138.patch, MAPREDUCE-4138.patch > > > FrameworkCounter is a non-static nested class of FrameworkCounterGroup which > means it retains a reference to the outer class, which isn't really needed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4101) nodemanager depends on /bin/bash
[ https://issues.apache.org/jira/browse/MAPREDUCE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261914#comment-13261914 ] Radim Kolar commented on MAPREDUCE-4101: You want configurable shell? I think it is overkill. > nodemanager depends on /bin/bash > > > Key: MAPREDUCE-4101 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: nodemanager >Affects Versions: 0.23.1, trunk > Environment: FreeBSD 8.2 / 64 bit >Reporter: Radim Kolar > Labels: patch > Attachments: bash-replace-by-sh.txt > > > Currently nodemanager depends on bash shell. It should be well documented for > system not having bash installed by default such as FreeBSD. Because only > basic functionality of bash is used, probably changing bash to /bin/sh would > work enough. > i found 2 cases: > 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in > writeLocalWrapperScript. (this needs bash in /bin) > 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 > INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: > launchContainer: [bash, -c, > /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_12/default_container_executor.sh] > this created script is also launched by bash - bash anywhere in path works - > in freebsd it is /usr/local/bin/bash -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3940) ContainerTokens should have an expiry interval
[ https://issues.apache.org/jira/browse/MAPREDUCE-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3940: --- Status: Patch Available (was: Open) > ContainerTokens should have an expiry interval > -- > > Key: MAPREDUCE-3940 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3940 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3940-20120308.txt, > MAPREDUCE-3940-20120416.txt, MAPREDUCE-3940-20120425.txt > > > - RM should generate the expiry time for a container > - A ContainerToken should have its expire time encoded > - NMs should reject containers with expired tokens. > - Expiry interval for a ContainerToken is same as the expiry interval for a > container. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3940) ContainerTokens should have an expiry interval
[ https://issues.apache.org/jira/browse/MAPREDUCE-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated MAPREDUCE-3940: --- Attachment: MAPREDUCE-3940-20120425.txt bq. The MR AM will end up seing some harmless exceptions when invoking stopContainer Ths issue of using ContainerTokens with stopContainer() and getContainerStatus() is known, but I did miss the MR AM getting exceptions. We will need to fix circumventing the verification logic of ContainerTokens for the stopContainer() and getContainerStatus() APIs, will open a separate ticket. Short term, we can get this into trunk and merge it into 23 branches after the followup ticket is fixed. Updating the patch to be applicable against later trunk. > ContainerTokens should have an expiry interval > -- > > Key: MAPREDUCE-3940 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3940 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3940-20120308.txt, > MAPREDUCE-3940-20120416.txt, MAPREDUCE-3940-20120425.txt > > > - RM should generate the expiry time for a container > - A ContainerToken should have its expire time encoded > - NMs should reject containers with expired tokens. > - Expiry interval for a ContainerToken is same as the expiry interval for a > container. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4138) Reduce memory usage of counters due to non-static nested classes
[ https://issues.apache.org/jira/browse/MAPREDUCE-4138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-4138: - Attachment: MAPREDUCE-4138.patch Alejandro, Robert - thanks for taking a look. Here's a new patch which addresses the javac warning. > Reduce memory usage of counters due to non-static nested classes > > > Key: MAPREDUCE-4138 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4138 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Tom White >Assignee: Tom White > Attachments: MAPREDUCE-4138.patch, MAPREDUCE-4138.patch > > > FrameworkCounter is a non-static nested class of FrameworkCounterGroup which > means it retains a reference to the outer class, which isn't really needed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3540) saveVersion.sh script fails in windows/cygwin (hadoop-yarn-common)
[
https://issues.apache.org/jira/browse/MAPREDUCE-3540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261852#comment-13261852
]
Uma Maheswara Rao G commented on MAPREDUCE-3540:
Hi Alejandro, Could you please take a look for committ?
In windows environment, always it fails due to this problem.
> saveVersion.sh script fails in windows/cygwin (hadoop-yarn-common)
> --
>
> Key: MAPREDUCE-3540
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3540
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build
>Affects Versions: 0.24.0
>Reporter: Alejandro Abdelnur
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3540.patch
>
>
> {code}
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec
> (generate-version) on project hadoop-yarn-common: Comman
> d execution failed. Cannot run program "scripts\saveVersion.sh" (in directory
> "C:\cygwin\home\tucu\src\hadoop\hadoop-mapreduce-proje
> ct\hadoop-yarn\hadoop-yarn-common"): CreateProcess error=2, The system cannot
> find the file specified -> [Help 1]
> [ERROR]
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3921) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy
[ https://issues.apache.org/jira/browse/MAPREDUCE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261843#comment-13261843 ] Bikas Saha commented on MAPREDUCE-3921: --- Here is how I read TaskAttemptCompletedEventTransition in JobImpl Task T has a new attempt completion event T2. At this point T1 is a successful attempt that has been recorded in successAttemptCompletionEventNoMap. So it is still a valid successful event. T is removed from successAttemptCompletionEventNoMap and its T1 taskAttemptCompletionEvents entry is marked obsolete. Now T2 status is checked and if successful, T is added to successAttemptCompletionEventNoMap. This means while retry T2 was running, T was considered successful because it was in successAttemptCompletionEventNoMap. 1) So if we want to *not* leave a task as successful when being retried, then T should already have been removed from successAttemptCompletionEventNoMap and T1 marked obsolete. Removing T after completion of T2 is not correct. 2) However, if we want T to remain successful until we have another successful attempt, then it should be removed from successAttemptCompletionEventNoMap only when T2 is successful. But currently we remove T from successAttemptCompletionEventNoMap regardless of T2's status. > MR AM should act on the nodes liveliness information when nodes go > up/down/unhealthy > > > Key: MAPREDUCE-3921 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3921 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Bikas Saha > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3921-1.patch, MAPREDUCE-3921-3.patch, > MAPREDUCE-3921-4.patch, MAPREDUCE-3921-5.patch, MAPREDUCE-3921-6.patch, > MAPREDUCE-3921-7.patch, MAPREDUCE-3921-branch-0.23.patch, > MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch, > MAPREDUCE-3921.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4193) broken doc link for yarn-default.xml in site.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-4193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261835#comment-13261835 ] Hudson commented on MAPREDUCE-4193: --- Integrated in Hadoop-Mapreduce-trunk-Commit #2145 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2145/]) MAPREDUCE-4193. broken doc link for yarn-default.xml in site.xml. Contributed by Patrick Hunt. (Revision 1330427) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330427 Files : * /hadoop/common/trunk/BUILDING.txt * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/xsl * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/xsl/configuration.xsl * /hadoop/common/trunk/hadoop-project/src/site/site.xml > broken doc link for yarn-default.xml in site.xml > > > Key: MAPREDUCE-4193 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4193 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Fix For: 2.0.0 > > Attachments: MAPREDUCE-4193.patch > > > the link to yarn-default.xml in site.xml is incorrect, generated docs link is > broken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4157) ResourceManager should not kill apps that are well behaved
[ https://issues.apache.org/jira/browse/MAPREDUCE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261828#comment-13261828 ] Ravi Prakash commented on MAPREDUCE-4157: - Hi Jason! I was thinking one of those parameters was the time the NM should wait before sending the SIGTERM. Alas, I was wrong ContainerLaunch.java:337 sends the SIGTERM pretty much instantly. That is messing me up in MAPREDUCE-4135. I'm still thinking having a small time delay before sending even the SIGTERM should be good. If we can do it on the NM and not on the RM, I think we might be able to conserve RM resources. > ResourceManager should not kill apps that are well behaved > -- > > Key: MAPREDUCE-4157 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4157 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 2.0.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: MAPREDUCE-4157.patch > > > Currently when the ApplicationMaster unregisters with the ResourceManager, > the RM kills (via the NMs) all the active containers for an application. > This introduces a race where the AM may be trying to clean up and may not > finish before it is killed. The RM should give the AM a chance to exit > cleanly on its own rather than always race with a pending kill on shutdown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4193) broken doc link for yarn-default.xml in site.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-4193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261824#comment-13261824 ] Hudson commented on MAPREDUCE-4193: --- Integrated in Hadoop-Common-trunk-Commit #2129 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2129/]) MAPREDUCE-4193. broken doc link for yarn-default.xml in site.xml. Contributed by Patrick Hunt. (Revision 1330427) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330427 Files : * /hadoop/common/trunk/BUILDING.txt * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/xsl * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/xsl/configuration.xsl * /hadoop/common/trunk/hadoop-project/src/site/site.xml > broken doc link for yarn-default.xml in site.xml > > > Key: MAPREDUCE-4193 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4193 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Fix For: 2.0.0 > > Attachments: MAPREDUCE-4193.patch > > > the link to yarn-default.xml in site.xml is incorrect, generated docs link is > broken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4135) MRAppMaster throws IllegalStateException while shutting down
[
https://issues.apache.org/jira/browse/MAPREDUCE-4135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261823#comment-13261823
]
Ravi Prakash commented on MAPREDUCE-4135:
-
I'm sorry I was wrong about my earlier statement:
bq. This is causing the subsequent statement (FileSystem.closeAll() ) to be
skipped. So to fix this JIRA, I'll see what I can do to fix the
InterruptedException.
The InterruptedException is thrown and then caught. So its not that. Its
MAPREDUCE-4157. The SIGTERM is sent without any delay on
ContainerLaunch.java:337.
> MRAppMaster throws IllegalStateException while shutting down
>
>
> Key: MAPREDUCE-4135
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4135
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 2.0.0, 3.0.0
>Reporter: Devaraj K
>Assignee: Ravi Prakash
> Attachments: MAPREDUCE-4135.branch-0.23.patch, MAPREDUCE-4135.patch,
> MAPREDUCE-4135.patch
>
>
> Always MRAppMaster throws IllegalStateException in the stderr while shutting
> down. It doesn't look good having this exception in the stderr file of
> MRAppMaster container.
> {code:xml}
> WARNING: org.apache.hadoop.metrics.jvm.EventCounter is deprecated. Please use
> org.apache.hadoop.log.metrics.EventCounter in all the log4j.properties files.
> Exception in thread "Thread-1" java.lang.IllegalStateException: Shutdown in
> progress
> at
> java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:55)
> at java.lang.Runtime.removeShutdownHook(Runtime.java:220)
> at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2148)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2180)
> at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2157)
> at org.apache.hadoop.fs.FileSystem.closeAll(FileSystem.java:361)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1014)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4193) broken doc link for yarn-default.xml in site.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-4193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261820#comment-13261820 ] Hudson commented on MAPREDUCE-4193: --- Integrated in Hadoop-Hdfs-trunk-Commit #2203 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2203/]) MAPREDUCE-4193. broken doc link for yarn-default.xml in site.xml. Contributed by Patrick Hunt. (Revision 1330427) Result = SUCCESS tomwhite : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330427 Files : * /hadoop/common/trunk/BUILDING.txt * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/xsl * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/xsl/configuration.xsl * /hadoop/common/trunk/hadoop-project/src/site/site.xml > broken doc link for yarn-default.xml in site.xml > > > Key: MAPREDUCE-4193 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4193 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Fix For: 2.0.0 > > Attachments: MAPREDUCE-4193.patch > > > the link to yarn-default.xml in site.xml is incorrect, generated docs link is > broken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4193) broken doc link for yarn-default.xml in site.xml
[ https://issues.apache.org/jira/browse/MAPREDUCE-4193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated MAPREDUCE-4193: - Resolution: Fixed Fix Version/s: 2.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) +1 I just committed this, after testing it (added some instructions to BUILDING.txt on how to build a staging site) and also adding an XSL file so the default file is rendered in the same way as the Common/HDFS/MapReduce ones. Thanks Patrick! > broken doc link for yarn-default.xml in site.xml > > > Key: MAPREDUCE-4193 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4193 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Fix For: 2.0.0 > > Attachments: MAPREDUCE-4193.patch > > > the link to yarn-default.xml in site.xml is incorrect, generated docs link is > broken. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3921) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy
[ https://issues.apache.org/jira/browse/MAPREDUCE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261811#comment-13261811 ] Siddharth Seth commented on MAPREDUCE-3921: --- Thanks for the updated patch Bikas. Will take a look. Still waiting for input from the MR veterans on some of the previous comments - how things were handled in 20 - specifically for killing map/reduce tasks on unhealthy nodes, and treating 'node unhealthy' similar to 'fetch failure' (State Killed / Failed as well as counting towards max_attempts). bq. About the OBSOLETE part. I get how it is used. What I dont get is why we are marking a previously successful task as obsolete and invalid upon the completion of a new task without first checking if the new task was itself successful or not. Are you considering leaving the task in SUCCESSFUL state, even if it's being retried, so that the Reduce *may* be able to pull data - before there's a new SUCCESSFUL attempt ? Otherwise, marking the attempt as OBSOLETE and removing the task from successAttemptCompletionEventNoMap (tracks only SUCCESSUL attempts) seems like the correct thing to do. > MR AM should act on the nodes liveliness information when nodes go > up/down/unhealthy > > > Key: MAPREDUCE-3921 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3921 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mr-am, mrv2 >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Bikas Saha > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3921-1.patch, MAPREDUCE-3921-3.patch, > MAPREDUCE-3921-4.patch, MAPREDUCE-3921-5.patch, MAPREDUCE-3921-6.patch, > MAPREDUCE-3921-7.patch, MAPREDUCE-3921-branch-0.23.patch, > MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch, > MAPREDUCE-3921.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3943) RM-NM secret-keys should be randomly generated and rolled every so often
[ https://issues.apache.org/jira/browse/MAPREDUCE-3943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261805#comment-13261805 ] Siddharth Seth commented on MAPREDUCE-3943: --- Looked at the patch. Needs some fixes. - There's a possibility of an AM getting a container token using the new master key - before the NM receives the updated key. We have to make sure the NMs receive the updated key before they are used. - The NM updates it's master key each time it receives one on a heartbeat. The key is being sent on each heartbeat - irrespective of whether it's been updated or not - so the NM will end up losing track of the older keys. - Minor: ContainerTokenIdentifier doesn't need it's methods to be synchronized. > RM-NM secret-keys should be randomly generated and rolled every so often > > > Key: MAPREDUCE-3943 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3943 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, security >Affects Versions: 0.23.0 >Reporter: Vinod Kumar Vavilapalli >Assignee: Vinod Kumar Vavilapalli > Fix For: 0.23.2 > > Attachments: MAPREDUCE-3943-20120416.txt > > > - RM should generate the master-key randomly > - The master-key should roll every so often > - NM should remember old expired keys so that already doled out > container-requests can be satisfied. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3540) saveVersion.sh script fails in windows/cygwin (hadoop-yarn-common)
[
https://issues.apache.org/jira/browse/MAPREDUCE-3540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261771#comment-13261771
]
xieguiming commented on MAPREDUCE-3540:
---
This patch also works my windows env.
Thanks for the patch.
> saveVersion.sh script fails in windows/cygwin (hadoop-yarn-common)
> --
>
> Key: MAPREDUCE-3540
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3540
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build
>Affects Versions: 0.24.0
>Reporter: Alejandro Abdelnur
> Fix For: 0.24.0
>
> Attachments: MAPREDUCE-3540.patch
>
>
> {code}
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.2:exec
> (generate-version) on project hadoop-yarn-common: Comman
> d execution failed. Cannot run program "scripts\saveVersion.sh" (in directory
> "C:\cygwin\home\tucu\src\hadoop\hadoop-mapreduce-proje
> ct\hadoop-yarn\hadoop-yarn-common"): CreateProcess error=2, The system cannot
> find the file specified -> [Help 1]
> [ERROR]
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4189) TestContainerManagerSecurity is failing
[
https://issues.apache.org/jira/browse/MAPREDUCE-4189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261727#comment-13261727
]
Daryn Sharp commented on MAPREDUCE-4189:
Somebody *really* needs to kill the lingering java processes on the hadoop3
build server and end the Russian roulette -- tests only fail on hadoop3 from
port in use exceptions. Please, it's been doing this for at least a week.
> TestContainerManagerSecurity is failing
> ---
>
> Key: MAPREDUCE-4189
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4189
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: trunk
>Reporter: Devaraj K
>Assignee: Devaraj K
>Priority: Critical
> Attachments: MAPREDUCE-4189.patch, MAPREDUCE-4189.patch
>
>
> {code:xml}
> ---
> T E S T S
> ---
> Running org.apache.hadoop.yarn.server.TestDiskFailures
> Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 26.519 sec
> Running org.apache.hadoop.yarn.server.TestContainerManagerSecurity
> Tests run: 3, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 10.673 sec
> <<< FAILURE!
> Results :
> Tests in error:
>
> testAuthenticatedUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
> testMaliceUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
>
> testUnauthorizedUser(org.apache.hadoop.yarn.server.TestContainerManagerSecurity)
> Tests run: 5, Failures: 0, Errors: 3, Skipped: 0
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4194) ConcurrentModificationError in DirectoryCollection
[ https://issues.apache.org/jira/browse/MAPREDUCE-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4194: --- Target Version/s: 0.23.3, 2.0.0 (was: 2.0.0, 0.23.3) Status: Open (was: Patch Available) > ConcurrentModificationError in DirectoryCollection > -- > > Key: MAPREDUCE-4194 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4194 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-4194.patch > > > As found as part of work on MAPREDUCE-4169, it is possible for a > ConcurrentModificationException to be thrown upon disk failure. > DirectoryCollection hands out its internal list structure that is accessed > across multiple threads. Upon disk failure its internal list is modified, > invalidating all current iterators to that structure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4194) ConcurrentModificationError in DirectoryCollection
[ https://issues.apache.org/jira/browse/MAPREDUCE-4194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261692#comment-13261692 ] Robert Joseph Evans commented on MAPREDUCE-4194: The test will delete the ./target dir, which is not really something that I want removed after running tests because it may have all of my compiled .class files in it, along with who knows what else. I would rather see something like target// as the testDir. Also getGoodDirs() and getFailedDirs() are still returning internal data structures. This is not really what this bug was trying to address because this is just to avoid a concurrent modification exception, but I is still bad practice, because I can delete an entry from the list returned and that will be reflected for all people with the list. I would like to see the Lists returned to be immutable, or if they are changed they will have no impact on DirectoryCollection. That can be done on a separate JIRA > ConcurrentModificationError in DirectoryCollection > -- > > Key: MAPREDUCE-4194 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4194 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.3, 2.0.0 >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: MAPREDUCE-4194.patch > > > As found as part of work on MAPREDUCE-4169, it is possible for a > ConcurrentModificationException to be thrown upon disk failure. > DirectoryCollection hands out its internal list structure that is accessed > across multiple threads. Upon disk failure its internal list is modified, > invalidating all current iterators to that structure. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3863) 0.22 branch mvn deploy is not publishing hadoop-streaming JAR
[ https://issues.apache.org/jira/browse/MAPREDUCE-3863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261656#comment-13261656 ] Hudson commented on MAPREDUCE-3863: --- Integrated in Hadoop-Mapreduce-22-branch #102 (See [https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/102/]) MAPREDUCE-3863. 0.22 branch mvn deploy is not publishing hadoop-streaming JAR. Contributed by Benoy Antony (Revision 1330126) Result = SUCCESS cos : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1330126 Files : * /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt * /hadoop/common/branches/branch-0.22/mapreduce/build.xml * /hadoop/common/branches/branch-0.22/mapreduce/src/contrib/streaming/build.xml > 0.22 branch mvn deploy is not publishing hadoop-streaming JAR > - > > Key: MAPREDUCE-3863 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3863 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 0.22.0, 0.22.1 >Reporter: Alejandro Abdelnur >Assignee: Konstantin Boudnik >Priority: Critical > Fix For: 0.22.1 > > Attachments: MAPREDUCE-3863.patch, MAPREDUCE-3863.patch, > MAPREDUCE-3863.patch > > > Without this JAR Oozie cannot be built/tested against 0.22 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4157) ResourceManager should not kill apps that are well behaved
[ https://issues.apache.org/jira/browse/MAPREDUCE-4157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261646#comment-13261646 ] Jason Lowe commented on MAPREDUCE-4157: --- Those properties are unrelated to this scenario. They are properties for the nodemanager to control how long to wait for a process to generate its pid file and how long to wait after sending SIGTERM before we send SIGKILL, respectively. In this case, we're not waiting for the process to generate a pid file nor waiting for it to die after sending SIGTERM. We're waiting for the process to exit normally on its own accord. If it doesn't exit normally, then the SIGTERM/SIGKILL approach and sleep-delay-before-sigkill would be relevant. > ResourceManager should not kill apps that are well behaved > -- > > Key: MAPREDUCE-4157 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4157 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mrv2 >Affects Versions: 2.0.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: MAPREDUCE-4157.patch > > > Currently when the ApplicationMaster unregisters with the ResourceManager, > the RM kills (via the NMs) all the active containers for an application. > This introduces a race where the AM may be trying to clean up and may not > finish before it is killed. The RM should give the AM a chance to exit > cleanly on its own rather than always race with a pending kill on shutdown. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4079) Allow MR AppMaster to limit ephemeral port range.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261621#comment-13261621 ] Hudson commented on MAPREDUCE-4079: --- Integrated in Hadoop-Mapreduce-trunk #1060 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1060/]) MAPREDUCE-4079. Allow MR AppMaster to limit ephemeral port range.(bobby via tgraves) (Revision 1329694) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329694 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factories/RpcServerFactory.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factories/impl/pb/RpcServerFactoryPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/HadoopYarnProtoRPC.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/YarnRPC.java > Allow MR AppMaster to limit ephemeral port range. > - > > Key: MAPREDUCE-4079 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4079 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am, mrv2 >Affects Versions: 0.23.2, 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.3 > > Attachments: MR-4079-full-branch-0.23.txt, > MR-4079-full-branch-0.23.txt, MR-4079-full-branch-0.23.txt, > MR-4079-full-branch-0.23.txt, MR-4079-trunk.txt, MR-4079-trunk.txt, > MR-4079-trunk.txt, MR-4079-trunk.txt, MR-4079-trunk.txt, MR-4079-trunk.txt > > > Having the MapReduce Application Masters bind to any ephemeral port makes it > very difficult to setup ACLs. mapreduce.job.am-access-disabled from > MAPREDUCE-3251 is not a practical permanent solution for all jobs. > Especially for tools like pig where they are not aware of > mapreduce.job.am-access-disabled and may deal with it properly. > We should add in a config option that would allow someone to restrict the > range of ports that the MR-AM can bind to. It will slow down startup in some > cases because we will have to probe for open ports instead of just asking the > OS to find one for us. But we can make that conditional on this config so > users who do not set this config do not see any performance degradation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4079) Allow MR AppMaster to limit ephemeral port range.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261607#comment-13261607 ] Hudson commented on MAPREDUCE-4079: --- Integrated in Hadoop-Hdfs-trunk #1025 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1025/]) MAPREDUCE-4079. Allow MR AppMaster to limit ephemeral port range.(bobby via tgraves) (Revision 1329694) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329694 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factories/RpcServerFactory.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factories/impl/pb/RpcServerFactoryPBImpl.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/HadoopYarnProtoRPC.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/YarnRPC.java > Allow MR AppMaster to limit ephemeral port range. > - > > Key: MAPREDUCE-4079 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4079 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am, mrv2 >Affects Versions: 0.23.2, 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.3 > > Attachments: MR-4079-full-branch-0.23.txt, > MR-4079-full-branch-0.23.txt, MR-4079-full-branch-0.23.txt, > MR-4079-full-branch-0.23.txt, MR-4079-trunk.txt, MR-4079-trunk.txt, > MR-4079-trunk.txt, MR-4079-trunk.txt, MR-4079-trunk.txt, MR-4079-trunk.txt > > > Having the MapReduce Application Masters bind to any ephemeral port makes it > very difficult to setup ACLs. mapreduce.job.am-access-disabled from > MAPREDUCE-3251 is not a practical permanent solution for all jobs. > Especially for tools like pig where they are not aware of > mapreduce.job.am-access-disabled and may deal with it properly. > We should add in a config option that would allow someone to restrict the > range of ports that the MR-AM can bind to. It will slow down startup in some > cases because we will have to probe for open ports instead of just asking the > OS to find one for us. But we can make that conditional on this config so > users who do not set this config do not see any performance degradation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4079) Allow MR AppMaster to limit ephemeral port range.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261600#comment-13261600 ] Hudson commented on MAPREDUCE-4079: --- Integrated in Hadoop-Hdfs-0.23-Build #238 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/238/]) MAPREDUCE-4079. Allow MR AppMaster to limit ephemeral port range.(bobby via tgraves) (Revision 1329732) MAPREDUCE-4079. Allow MR AppMaster to limit ephemeral port range.(bobby via tgraves) (Revision 1329696) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329732 Files : * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestServer.java tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329696 Files : * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/AvroRpcEngine.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RPC.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/RpcEngine.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/WritableRpcEngine.java * /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestConfiguration.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/client/MRClientService.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factories/RpcServerFactory.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/factories/impl/pb/RpcServerFactoryPBImpl.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/HadoopYarnProtoRPC.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/ProtoOverHadoopRpcEngine.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/ipc/YarnRPC.java > Allow MR AppMaster to limit ephemeral port range. > - > > Key: MAPREDUCE-4079 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4079 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: mr-am, mrv2 >Affects Versions: 0.23.2, 2.0.0 >Reporter: Robert Joseph Evans >Assignee: Robert Joseph Evans >Priority: Blocker > Fix For: 0.23.3 > > Attachments: MR-4079-full-branch-0.23.txt, > MR-4079-full-branch-0.23.txt, MR-4079-full-branch-0.23.txt, > MR-4079-full-branch-0.23.txt, MR-4079-trunk.txt, MR-4079-trunk.txt, > MR-4079-trunk.txt, MR-4079-trunk.txt, MR-4079-trunk.txt, MR-4079-trunk.txt > > > Having the MapReduce Application Masters bind to any ephemeral port makes it > very difficult to setup ACLs. mapreduce.job.am-access-disabled from > MAPREDUCE-3251 is not a practical permanent solution for all jobs. > Especially for tools like pig where they are not aware of > mapreduce.job.am-access-disabled and may deal with it properly. > We should add in a config option that would allow someone to restrict the > range of ports that the MR-AM can bind to. It will slow down startup in some > cases because we will have to probe for open ports instead of just asking the > OS to find one for us. But we can make that conditional on this config so > users who do not set this config do not see any performance degradation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information
[jira] [Commented] (MAPREDUCE-3451) Port Fair Scheduler to MR2
[ https://issues.apache.org/jira/browse/MAPREDUCE-3451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13261472#comment-13261472 ] Luke Lu commented on MAPREDUCE-3451: bq. It seems like it’s not easy to trigger a 404 from within a view, rather than at the URL router. You can trigger a 404 or any http response code anywhere (preferably in controller code as view is not guaranteed to be fully buffered for long responses by design) by calling Controller#setStatus (via context().setStatus in views (if it's not yet flushed) and then throw an WebAppException (do not use generic RuntimeException). You can bind ErrorPage.class in a webapp to any view class if you want a custom error page. > Port Fair Scheduler to MR2 > -- > > Key: MAPREDUCE-3451 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3451 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: mrv2, scheduler >Reporter: Patrick Wendell >Assignee: Patrick Wendell > Attachments: MAPREDUCE-3451.v1.patch.txt, > MAPREDUCE-3451.v2.patch.txt, MAPREDUCE-3451.v3.patch.txt > > > The Fair Scheduler is in widespread use today in MR1 clusters, but not yet > ported to MR2. This is to track the porting of the Fair Scheduler to MR2 and > will be updated to include design considerations and progress. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
