[jira] [Assigned] (MAPREDUCE-3664) HDFS Federation Documentation has incorrect configuration example
[ https://issues.apache.org/jira/browse/MAPREDUCE-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE reassigned MAPREDUCE-3664: - Assignee: Brandon Li Should we merge this to 0.23? > HDFS Federation Documentation has incorrect configuration example > - > > Key: MAPREDUCE-3664 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3664 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Affects Versions: 0.23.0, 0.24.0 >Reporter: praveen sripati >Assignee: Brandon Li >Priority: Minor > Attachments: HDFS-2778.txt, HDFS-2778.txt > > > HDFS Federation documentation example (1) has the following > > dfs.namenode.rpc-address.ns1 > hdfs://nn-host1:rpc-port > > dfs.namenode.rpc-address.* should be set to hostname:port, hdfs:// should not > be there. > (1) - > http://hadoop.apache.org/common/docs/r0.23.0/hadoop-yarn/hadoop-yarn-site/Federation.html -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3674) If invoked with no queueName request param, jobqueue_details.jsp injects a null queue name into schedulers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186683#comment-13186683
]
Hadoop QA commented on MAPREDUCE-3674:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12510664/MAPREDUCE-3674.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
-1 patch. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1614//console
This message is automatically generated.
> If invoked with no queueName request param, jobqueue_details.jsp injects a
> null queue name into schedulers.
> ---
>
> Key: MAPREDUCE-3674
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3674
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Affects Versions: 1.0.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Critical
> Attachments: MAPREDUCE-3674.patch
>
>
> When you access /jobqueue_details.jsp manually, instead of via a link, it has
> queueName set to null internally and this goes for a lookup into the
> scheduling info maps as well.
> As a result, if using FairScheduler, a Pool with String name = null gets
> created and this brings the scheduler down. I have not tested what happens to
> the CapacityScheduler, but ideally if no queueName is set in that jsp, it
> should fall back to 'default'. Otherwise, this brings down the JobTracker
> completely.
> FairScheduler must also add a check to not create a pool with 'null' name.
> The following is the strace that ensues:
> {code}
> ERROR org.mortbay.log: /jobqueue_details.jsp
> java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.jobqueue_005fdetails_jsp._jspService(jobqueue_005fdetails_jsp.java:71)
>
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>
> at
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:829)
>
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9001, call
> heartbeat from XYZ:MNOP: error: java.io.IOException:
> java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:95)
>
> at
> org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:68)
>
> at java.util.Arrays.mergeSort(Unknown Source)
> at java.util.Arrays.sort(Unknown Source)
> at java.util.Collections.sort(Unknown Source)
> at org.apache.hadoop.mapred.FairScheduler.assignTasks(FairScheduler.java:435)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3226)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown So
[jira] [Updated] (MAPREDUCE-3674) If invoked with no queueName request param, jobqueue_details.jsp injects a null queue name into schedulers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated MAPREDUCE-3674:
---
Attachment: MAPREDUCE-3674.patch
Manually tested patch to verify it does not inject null queuename pools into
FairScheduler.
> If invoked with no queueName request param, jobqueue_details.jsp injects a
> null queue name into schedulers.
> ---
>
> Key: MAPREDUCE-3674
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3674
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Affects Versions: 1.0.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Critical
> Attachments: MAPREDUCE-3674.patch
>
>
> When you access /jobqueue_details.jsp manually, instead of via a link, it has
> queueName set to null internally and this goes for a lookup into the
> scheduling info maps as well.
> As a result, if using FairScheduler, a Pool with String name = null gets
> created and this brings the scheduler down. I have not tested what happens to
> the CapacityScheduler, but ideally if no queueName is set in that jsp, it
> should fall back to 'default'. Otherwise, this brings down the JobTracker
> completely.
> FairScheduler must also add a check to not create a pool with 'null' name.
> The following is the strace that ensues:
> {code}
> ERROR org.mortbay.log: /jobqueue_details.jsp
> java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.jobqueue_005fdetails_jsp._jspService(jobqueue_005fdetails_jsp.java:71)
>
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>
> at
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:829)
>
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9001, call
> heartbeat from XYZ:MNOP: error: java.io.IOException:
> java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:95)
>
> at
> org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:68)
>
> at java.util.Arrays.mergeSort(Unknown Source)
> at java.util.Arrays.sort(Unknown Source)
> at java.util.Collections.sort(Unknown Source)
> at org.apache.hadoop.mapred.FairScheduler.assignTasks(FairScheduler.java:435)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3226)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrec
[jira] [Updated] (MAPREDUCE-3674) If invoked with no queueName request param, jobqueue_details.jsp injects a null queue name into schedulers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated MAPREDUCE-3674:
---
Status: Patch Available (was: Open)
> If invoked with no queueName request param, jobqueue_details.jsp injects a
> null queue name into schedulers.
> ---
>
> Key: MAPREDUCE-3674
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3674
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Affects Versions: 1.0.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Critical
> Attachments: MAPREDUCE-3674.patch
>
>
> When you access /jobqueue_details.jsp manually, instead of via a link, it has
> queueName set to null internally and this goes for a lookup into the
> scheduling info maps as well.
> As a result, if using FairScheduler, a Pool with String name = null gets
> created and this brings the scheduler down. I have not tested what happens to
> the CapacityScheduler, but ideally if no queueName is set in that jsp, it
> should fall back to 'default'. Otherwise, this brings down the JobTracker
> completely.
> FairScheduler must also add a check to not create a pool with 'null' name.
> The following is the strace that ensues:
> {code}
> ERROR org.mortbay.log: /jobqueue_details.jsp
> java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.jobqueue_005fdetails_jsp._jspService(jobqueue_005fdetails_jsp.java:71)
>
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>
> at
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:829)
>
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9001, call
> heartbeat from XYZ:MNOP: error: java.io.IOException:
> java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:95)
>
> at
> org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:68)
>
> at java.util.Arrays.mergeSort(Unknown Source)
> at java.util.Arrays.sort(Unknown Source)
> at java.util.Collections.sort(Unknown Source)
> at org.apache.hadoop.mapred.FairScheduler.assignTasks(FairScheduler.java:435)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3226)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministra
[jira] [Commented] (MAPREDUCE-3674) If invoked with no queueName request param, jobqueue_details.jsp injects a null queue name into schedulers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186680#comment-13186680
]
Harsh J commented on MAPREDUCE-3674:
Or rather, since problem appears to be in FairScheduler, I'll write a patch
accordingly for that.
I do not see CapacityScheduler doing dynamic queue names so it is safe.
> If invoked with no queueName request param, jobqueue_details.jsp injects a
> null queue name into schedulers.
> ---
>
> Key: MAPREDUCE-3674
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3674
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobtracker
>Affects Versions: 1.0.0
>Reporter: Harsh J
>Assignee: Harsh J
>Priority: Critical
>
> When you access /jobqueue_details.jsp manually, instead of via a link, it has
> queueName set to null internally and this goes for a lookup into the
> scheduling info maps as well.
> As a result, if using FairScheduler, a Pool with String name = null gets
> created and this brings the scheduler down. I have not tested what happens to
> the CapacityScheduler, but ideally if no queueName is set in that jsp, it
> should fall back to 'default'. Otherwise, this brings down the JobTracker
> completely.
> FairScheduler must also add a check to not create a pool with 'null' name.
> The following is the strace that ensues:
> {code}
> ERROR org.mortbay.log: /jobqueue_details.jsp
> java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.jobqueue_005fdetails_jsp._jspService(jobqueue_005fdetails_jsp.java:71)
>
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
>
> at
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:829)
>
> at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
>
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
> at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:326)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
> at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
>
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
> at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
> at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
> INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9001, call
> heartbeat from XYZ:MNOP: error: java.io.IOException:
> java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:95)
>
> at
> org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:68)
>
> at java.util.Arrays.mergeSort(Unknown Source)
> at java.util.Arrays.sort(Unknown Source)
> at java.util.Collections.sort(Unknown Source)
> at org.apache.hadoop.mapred.FairScheduler.assignTasks(FairScheduler.java:435)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3226)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Unknown Source)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
>
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)
> {code}
--
This message is automatically
[jira] [Created] (MAPREDUCE-3674) If invoked with no queueName request param, jobqueue_details.jsp injects a null queue name into schedulers.
If invoked with no queueName request param, jobqueue_details.jsp injects a null
queue name into schedulers.
---
Key: MAPREDUCE-3674
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3674
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 1.0.0
Reporter: Harsh J
Assignee: Harsh J
Priority: Critical
When you access /jobqueue_details.jsp manually, instead of via a link, it has
queueName set to null internally and this goes for a lookup into the scheduling
info maps as well.
As a result, if using FairScheduler, a Pool with String name = null gets
created and this brings the scheduler down. I have not tested what happens to
the CapacityScheduler, but ideally if no queueName is set in that jsp, it
should fall back to 'default'. Otherwise, this brings down the JobTracker
completely.
FairScheduler must also add a check to not create a pool with 'null' name.
The following is the strace that ensues:
{code}
ERROR org.mortbay.log: /jobqueue_details.jsp
java.lang.NullPointerException
at
org.apache.hadoop.mapred.jobqueue_005fdetails_jsp._jspService(jobqueue_005fdetails_jsp.java:71)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:829)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9001, call heartbeat
from XYZ:MNOP: error: java.io.IOException: java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
at
org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:95)
at
org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:68)
at java.util.Arrays.mergeSort(Unknown Source)
at java.util.Arrays.sort(Unknown Source)
at java.util.Collections.sort(Unknown Source)
at org.apache.hadoop.mapred.FairScheduler.assignTasks(FairScheduler.java:435)
at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3226)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3669) Getting a lot of PriviledgedActionException / SaslException when running a job
[ https://issues.apache.org/jira/browse/MAPREDUCE-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186584#comment-13186584 ] Hadoop QA commented on MAPREDUCE-3669: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12510638/MAPREDUCE-3669.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1613//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1613//console This message is automatically generated. > Getting a lot of PriviledgedActionException / SaslException when running a job > -- > > Key: MAPREDUCE-3669 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3669 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Mahadev konar >Priority: Blocker > Attachments: MAPREDUCE-3669.patch > > > On a secure cluster, when running a job we are seeing a lot of > PriviledgedActionException / SaslExceptions. The job runs fine, its just the > jobclient can't connect to the AM to get the progress information. > Its in a very tight loop retrying while getting the exceptions. > snip of the client log is: > 12/01/13 15:33:45 INFO security.SecurityUtil: Acquired token Ident: 00 1c 68 > 61 64 6f 6f 70 71 61 40 44 45 56 2e 59 47 > 52 49 44 2e 59 41 48 4f 4f 2e 43 4f 4d 08 6d 61 70 72 65 64 71 61 00 8a 01 34 > d7 b3 ff f5 8a 01 34 fb c0 83 f5 08 02, > Kind: HDFS_DELEGATION_TOKEN, Service: 10.10.10.10:8020 > 12/01/13 15:33:45 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 8 > for user1 on 10.10.10.10:8020 > 12/01/13 15:33:45 INFO security.TokenCache: Got dt for > hdfs://host1.domain.com:8020;uri=10.10.10.10:8020;t.service=10.10.10.10:8020 > 12/01/13 15:33:45 WARN conf.Configuration: mapred.used.genericoptionsparser > is deprecated. Instead, use > mapreduce.client.genericoptionsparser.used > 12/01/13 15:33:45 INFO mapreduce.JobSubmitter: number of splits:2 > 12/01/13 15:33:45 INFO mapred.ResourceMgrDelegate: Submitted application > application_1326410042859_0008 to > ResourceManager at rmhost.domain/10.10.10.11:8040 > 12/01/13 15:33:45 INFO mapreduce.Job: Running job: job_1326410042859_0008 > 12/01/13 15:33:52 INFO mapred.ClientServiceDelegate: The url to track the job: > rmhost.domain:8088/proxy/application_1326410042859_0008/ > 12/01/13 15:33:52 ERROR security.UserGroupInformation: > PriviledgedActionException as:us...@dev.ygrid.yahoo.com > (auth:SIMPLE) cause:javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: Fail > ed to find any > Kerberos tgt)] > 12/01/13 15:33:52 WARN ipc.Client: Exception encountered while connecting to > the server : > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided > (Mechanism level: Failed to find any Kerberos tgt)] > 12/01/13 15:33:52 ERROR security.UserGroupInformation: > PriviledgedActionException as:us...@dev.ygrid.yahoo.com > (auth:SIMPLE) cause:java.io.IOException: javax.security.sasl.SaslException: > GSS initiate failed [Caused by GSSException: No valid credentials provided ( > Mechanism level: > Failed to find any Kerberos tgt)] > 12/01/13 15:33:52 INFO mapred.ClientServiceDelegate: The url to track the job: > rmhost.domain:8088/proxy/application_1326410042859_0008/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3669) Getting a lot of PriviledgedActionException / SaslException when running a job
[ https://issues.apache.org/jira/browse/MAPREDUCE-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3669: - Status: Patch Available (was: Open) @Thomas, Can you give this a try? I cant reproduce the issue on my setup. > Getting a lot of PriviledgedActionException / SaslException when running a job > -- > > Key: MAPREDUCE-3669 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3669 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Mahadev konar >Priority: Blocker > Attachments: MAPREDUCE-3669.patch > > > On a secure cluster, when running a job we are seeing a lot of > PriviledgedActionException / SaslExceptions. The job runs fine, its just the > jobclient can't connect to the AM to get the progress information. > Its in a very tight loop retrying while getting the exceptions. > snip of the client log is: > 12/01/13 15:33:45 INFO security.SecurityUtil: Acquired token Ident: 00 1c 68 > 61 64 6f 6f 70 71 61 40 44 45 56 2e 59 47 > 52 49 44 2e 59 41 48 4f 4f 2e 43 4f 4d 08 6d 61 70 72 65 64 71 61 00 8a 01 34 > d7 b3 ff f5 8a 01 34 fb c0 83 f5 08 02, > Kind: HDFS_DELEGATION_TOKEN, Service: 10.10.10.10:8020 > 12/01/13 15:33:45 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 8 > for user1 on 10.10.10.10:8020 > 12/01/13 15:33:45 INFO security.TokenCache: Got dt for > hdfs://host1.domain.com:8020;uri=10.10.10.10:8020;t.service=10.10.10.10:8020 > 12/01/13 15:33:45 WARN conf.Configuration: mapred.used.genericoptionsparser > is deprecated. Instead, use > mapreduce.client.genericoptionsparser.used > 12/01/13 15:33:45 INFO mapreduce.JobSubmitter: number of splits:2 > 12/01/13 15:33:45 INFO mapred.ResourceMgrDelegate: Submitted application > application_1326410042859_0008 to > ResourceManager at rmhost.domain/10.10.10.11:8040 > 12/01/13 15:33:45 INFO mapreduce.Job: Running job: job_1326410042859_0008 > 12/01/13 15:33:52 INFO mapred.ClientServiceDelegate: The url to track the job: > rmhost.domain:8088/proxy/application_1326410042859_0008/ > 12/01/13 15:33:52 ERROR security.UserGroupInformation: > PriviledgedActionException as:us...@dev.ygrid.yahoo.com > (auth:SIMPLE) cause:javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: Fail > ed to find any > Kerberos tgt)] > 12/01/13 15:33:52 WARN ipc.Client: Exception encountered while connecting to > the server : > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided > (Mechanism level: Failed to find any Kerberos tgt)] > 12/01/13 15:33:52 ERROR security.UserGroupInformation: > PriviledgedActionException as:us...@dev.ygrid.yahoo.com > (auth:SIMPLE) cause:java.io.IOException: javax.security.sasl.SaslException: > GSS initiate failed [Caused by GSSException: No valid credentials provided ( > Mechanism level: > Failed to find any Kerberos tgt)] > 12/01/13 15:33:52 INFO mapred.ClientServiceDelegate: The url to track the job: > rmhost.domain:8088/proxy/application_1326410042859_0008/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3669) Getting a lot of PriviledgedActionException / SaslException when running a job
[ https://issues.apache.org/jira/browse/MAPREDUCE-3669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated MAPREDUCE-3669: - Attachment: MAPREDUCE-3669.patch This patch should fix the issue. I think a unit test should be feasible. Will update the patch with one. > Getting a lot of PriviledgedActionException / SaslException when running a job > -- > > Key: MAPREDUCE-3669 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3669 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 0.23.0 >Reporter: Thomas Graves >Assignee: Mahadev konar >Priority: Blocker > Attachments: MAPREDUCE-3669.patch > > > On a secure cluster, when running a job we are seeing a lot of > PriviledgedActionException / SaslExceptions. The job runs fine, its just the > jobclient can't connect to the AM to get the progress information. > Its in a very tight loop retrying while getting the exceptions. > snip of the client log is: > 12/01/13 15:33:45 INFO security.SecurityUtil: Acquired token Ident: 00 1c 68 > 61 64 6f 6f 70 71 61 40 44 45 56 2e 59 47 > 52 49 44 2e 59 41 48 4f 4f 2e 43 4f 4d 08 6d 61 70 72 65 64 71 61 00 8a 01 34 > d7 b3 ff f5 8a 01 34 fb c0 83 f5 08 02, > Kind: HDFS_DELEGATION_TOKEN, Service: 10.10.10.10:8020 > 12/01/13 15:33:45 INFO hdfs.DFSClient: Created HDFS_DELEGATION_TOKEN token 8 > for user1 on 10.10.10.10:8020 > 12/01/13 15:33:45 INFO security.TokenCache: Got dt for > hdfs://host1.domain.com:8020;uri=10.10.10.10:8020;t.service=10.10.10.10:8020 > 12/01/13 15:33:45 WARN conf.Configuration: mapred.used.genericoptionsparser > is deprecated. Instead, use > mapreduce.client.genericoptionsparser.used > 12/01/13 15:33:45 INFO mapreduce.JobSubmitter: number of splits:2 > 12/01/13 15:33:45 INFO mapred.ResourceMgrDelegate: Submitted application > application_1326410042859_0008 to > ResourceManager at rmhost.domain/10.10.10.11:8040 > 12/01/13 15:33:45 INFO mapreduce.Job: Running job: job_1326410042859_0008 > 12/01/13 15:33:52 INFO mapred.ClientServiceDelegate: The url to track the job: > rmhost.domain:8088/proxy/application_1326410042859_0008/ > 12/01/13 15:33:52 ERROR security.UserGroupInformation: > PriviledgedActionException as:us...@dev.ygrid.yahoo.com > (auth:SIMPLE) cause:javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism level: Fail > ed to find any > Kerberos tgt)] > 12/01/13 15:33:52 WARN ipc.Client: Exception encountered while connecting to > the server : > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided > (Mechanism level: Failed to find any Kerberos tgt)] > 12/01/13 15:33:52 ERROR security.UserGroupInformation: > PriviledgedActionException as:us...@dev.ygrid.yahoo.com > (auth:SIMPLE) cause:java.io.IOException: javax.security.sasl.SaslException: > GSS initiate failed [Caused by GSSException: No valid credentials provided ( > Mechanism level: > Failed to find any Kerberos tgt)] > 12/01/13 15:33:52 INFO mapred.ClientServiceDelegate: The url to track the job: > rmhost.domain:8088/proxy/application_1326410042859_0008/ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3641) CapacityScheduler should be more conservative assigning off-switch requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-3641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186479#comment-13186479 ] Hadoop QA commented on MAPREDUCE-3641: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12510620/MAPREDUCE-3641.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1612//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1612//console This message is automatically generated. > CapacityScheduler should be more conservative assigning off-switch requests > --- > > Key: MAPREDUCE-3641 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3641 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, scheduler >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Attachments: MAPREDUCE-3641.patch, MAPREDUCE-3641.patch, > MAPREDUCE-3641.patch, MAPREDUCE-3641.patch > > > In hadoop-1, the CS is very conservative handing out off-switch assignments, > we need to do the same in YARN. > We noticed performance regressions due to this, particularly for reduces. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3641) CapacityScheduler should be more conservative assigning off-switch requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-3641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3641: - Status: Patch Available (was: Open) > CapacityScheduler should be more conservative assigning off-switch requests > --- > > Key: MAPREDUCE-3641 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3641 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, scheduler >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Attachments: MAPREDUCE-3641.patch, MAPREDUCE-3641.patch, > MAPREDUCE-3641.patch, MAPREDUCE-3641.patch > > > In hadoop-1, the CS is very conservative handing out off-switch assignments, > we need to do the same in YARN. > We noticed performance regressions due to this, particularly for reduces. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3641) CapacityScheduler should be more conservative assigning off-switch requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-3641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3641: - Status: Open (was: Patch Available) > CapacityScheduler should be more conservative assigning off-switch requests > --- > > Key: MAPREDUCE-3641 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3641 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, scheduler >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Attachments: MAPREDUCE-3641.patch, MAPREDUCE-3641.patch, > MAPREDUCE-3641.patch, MAPREDUCE-3641.patch > > > In hadoop-1, the CS is very conservative handing out off-switch assignments, > we need to do the same in YARN. > We noticed performance regressions due to this, particularly for reduces. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3641) CapacityScheduler should be more conservative assigning off-switch requests
[ https://issues.apache.org/jira/browse/MAPREDUCE-3641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun C Murthy updated MAPREDUCE-3641: - Attachment: MAPREDUCE-3641.patch Added unit tests, ready for review/commit. > CapacityScheduler should be more conservative assigning off-switch requests > --- > > Key: MAPREDUCE-3641 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3641 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: mrv2, scheduler >Affects Versions: 0.23.0 >Reporter: Arun C Murthy >Assignee: Arun C Murthy >Priority: Blocker > Attachments: MAPREDUCE-3641.patch, MAPREDUCE-3641.patch, > MAPREDUCE-3641.patch, MAPREDUCE-3641.patch > > > In hadoop-1, the CS is very conservative handing out off-switch assignments, > we need to do the same in YARN. > We noticed performance regressions due to this, particularly for reduces. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
