[jira] [Commented] (MAPREDUCE-2364) Shouldn't hold lock on rjob while localizing resources.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040755#comment-13040755 ] Binglin Chang commented on MAPREDUCE-2364: -- We encounter the same problem, when TaskTracker download unJar a very big job.jar in localizeJob(), it stops sending heartbeat and web service hangs too. Our solution for this issue is to add a new lock in RunningJob class called localizing. Instead of holding the whole rjob lock, rjob.localizing is locked. Shouldn't hold lock on rjob while localizing resources. --- Key: MAPREDUCE-2364 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2364 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.20.203.0 Reporter: Owen O'Malley Assignee: Devaraj Das Fix For: 0.20.203.0 There is a deadlock while localizing resources on the TaskTracker. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2364) Shouldn't hold lock on rjob while localizing resources.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Binglin Chang updated MAPREDUCE-2364: - Attachment: MAPREDUCE-2364.patch trunk patch Shouldn't hold lock on rjob while localizing resources. --- Key: MAPREDUCE-2364 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2364 Project: Hadoop Map/Reduce Issue Type: Bug Components: tasktracker Affects Versions: 0.20.203.0 Reporter: Owen O'Malley Assignee: Devaraj Das Fix For: 0.20.203.0 Attachments: MAPREDUCE-2364.patch There is a deadlock while localizing resources on the TaskTracker. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2469) Task counters should also report the total heap usage of the task
[ https://issues.apache.org/jira/browse/MAPREDUCE-2469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040770#comment-13040770 ] Hadoop QA commented on MAPREDUCE-2469: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480766/mapreduce-2469-v1.1.patch against trunk revision 1128394. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.cli.TestMRCLI org.apache.hadoop.tools.TestHadoopArchives org.apache.hadoop.tools.TestHarFileSystem -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/320//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/320//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/320//console This message is automatically generated. Task counters should also report the total heap usage of the task - Key: MAPREDUCE-2469 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2469 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task Affects Versions: 0.23.0 Reporter: Amar Kamat Assignee: Amar Kamat Labels: mapreduce Attachments: 2469.v0.1.patch, mapreduce-2469-v1.1.patch Currently, the task counters report VSS and RSS usage of the task. The task counter should also report the total heap usage of the task also. The task might be configured with a max heap size of M but the task's total heap usage might only be H, where H M. In such a case, knowing only M doesn't provide a complete picture of the task's memory usage. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-1270) Hadoop C++ Extention
[ https://issues.apache.org/jira/browse/MAPREDUCE-1270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040800#comment-13040800 ] Binglin Chang commented on MAPREDUCE-1270: -- Koth, In HCE socket is only used for passing control messages(not like c++ pipes), which has little impact on performance, as for data processing, such as input/map/mid-output/reduce/output, since everything is implemented in C++, JNI is not needed, except reading input from HDFS and writing output to HDFS, HCE uses libhdfs, which is JNI based. I think JNI based C++ extension for MR have the advantage of non-intrusive, and has better compatibility. In current HCE design, we need to reimplement many features already exists in Java, some of those get performance benefit(sort, spill), some of those are purely duplicate work. In current HCE design, if you wan't performance benefits in HCE, the only way is to use HCE interface, my thought is to extract the high performance part(sort, spill, compression in MapOutputCollector), wrap it using JNI as native lib like compress codecs, a jobconf item is used to enable/disable native optimization, so the code is compatible and java based jobs can also get performance benefits. Hadoop C++ Extention Key: MAPREDUCE-1270 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1270 Project: Hadoop Map/Reduce Issue Type: Improvement Components: task Affects Versions: 0.20.1 Environment: hadoop linux Reporter: Wang Shouyan Attachments: HADOOP-HCE-1.0.0.patch, HCE InstallMenu.pdf, HCE Performance Report.pdf, HCE Tutorial.pdf, Overall Design of Hadoop C++ Extension.doc Hadoop C++ extension is an internal project in baidu, We start it for these reasons: 1 To provide C++ API. We mostly use Streaming before, and we also try to use PIPES, but we do not find PIPES is more efficient than Streaming. So we think a new C++ extention is needed for us. 2 Even using PIPES or Streaming, it is hard to control memory of hadoop map/reduce Child JVM. 3 It costs so much to read/write/sort TB/PB data by Java. When using PIPES or Streaming, pipe or socket is not efficient to carry so huge data. What we want to do: 1 We do not use map/reduce Child JVM to do any data processing, which just prepares environment, starts C++ mapper, tells mapper which split it should deal with, and reads report from mapper until that finished. The mapper will read record, ivoke user defined map, to do partition, write spill, combine and merge into file.out. We think these operations can be done by C++ code. 2 Reducer is similar to mapper, it was started after sort finished, it read from sorted files, ivoke user difined reduce, and write to user defined record writer. 3 We also intend to rewrite shuffle and sort with C++, for efficience and memory control. at first, 1 and 2, then 3. What's the difference with PIPES: 1 Yes, We will reuse most PIPES code. 2 And, We should do it more completely, nothing changed in scheduling and management, but everything in execution. *UPDATE:* Now you can get a test version of HCE from this link http://docs.google.com/leaf?id=0B5xhnqH1558YZjcxZmI0NzEtODczMy00NmZiLWFkNjAtZGM1MjZkMmNkNWFkhl=zh_CNpli=1 This is a full package with all hadoop source code. Following document HCE InstallMenu.pdf in attachment, you will build and deploy it in your cluster. Attachment HCE Tutorial.pdf will lead you to write the first HCE program and give other specifications of the interface. Attachment HCE Performance Report.pdf gives a performance report of HCE compared to Java MapRed and Pipes. Any comments are welcomed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2521) Mapreduce RPM integration project
[ https://issues.apache.org/jira/browse/MAPREDUCE-2521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040811#comment-13040811 ] Hudson commented on MAPREDUCE-2521: --- Integrated in Hadoop-Mapreduce-trunk #694 (See [https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/694/]) Mapreduce RPM integration project - Key: MAPREDUCE-2521 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2521 Project: Hadoop Map/Reduce Issue Type: New Feature Components: build Environment: Java 6, RHEL 5.5 Reporter: Eric Yang Assignee: Eric Yang Fix For: 0.23.0 Attachments: MAPREDUCE-2521-1.patch, MAPREDUCE-2521-2.patch, MAPREDUCE-2521-3.patch, MAPREDUCE-2521-4.patch, MAPREDUCE-2521-5.patch, MAPREDUCE-2521-6.patch, MAPREDUCE-2521-7.patch, MAPREDUCE-2521.patch This jira is corresponding to HADOOP-6255 and associated directory layout change. The patch for creating Mapreduce rpm packaging should be posted here for patch test build to verify against mapreduce svn trunk. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2528) NullPointerException in the job tracker UI, when we perform kill or change the priority of jobs without selecting the any job.
[
https://issues.apache.org/jira/browse/MAPREDUCE-2528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated MAPREDUCE-2528:
---
Status: Open (was: Patch Available)
NullPointerException in the job tracker UI, when we perform kill or change
the priority of jobs without selecting the any job.
--
Key: MAPREDUCE-2528
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2528
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 0.23.0
Reporter: Devaraj K
Assignee: Devaraj K
Fix For: 0.23.0
Attachments: MAPREDUCE-2528.patch
If we click on Kill Selected Jobs or Change button without selecting any job,
it is giving the below exception in the UI.
{code}
java.lang.NullPointerException
at
org.apache.hadoop.http.HttpServer$QuotingInputFilter$RequestQuoter.getParameterValues(HttpServer.java:798)
at org.apache.hadoop.mapred.JSPUtil.processButtons(JSPUtil.java:209)
at
org.apache.hadoop.mapred.jobtracker_jsp._jspService(jobtracker_jsp.java:146)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1124)
at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:871)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1115)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:361)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:879)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:741)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:213)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2528) NullPointerException in the job tracker UI, when we perform kill or change the priority of jobs without selecting the any job.
[
https://issues.apache.org/jira/browse/MAPREDUCE-2528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13040985#comment-13040985
]
Todd Lipcon commented on MAPREDUCE-2528:
This should be fixed server side, not with JS validation.
NullPointerException in the job tracker UI, when we perform kill or change
the priority of jobs without selecting the any job.
--
Key: MAPREDUCE-2528
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2528
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 0.23.0
Reporter: Devaraj K
Assignee: Devaraj K
Fix For: 0.23.0
Attachments: MAPREDUCE-2528.patch
If we click on Kill Selected Jobs or Change button without selecting any job,
it is giving the below exception in the UI.
{code}
java.lang.NullPointerException
at
org.apache.hadoop.http.HttpServer$QuotingInputFilter$RequestQuoter.getParameterValues(HttpServer.java:798)
at org.apache.hadoop.mapred.JSPUtil.processButtons(JSPUtil.java:209)
at
org.apache.hadoop.mapred.jobtracker_jsp._jspService(jobtracker_jsp.java:146)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1124)
at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:871)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1115)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:361)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at
org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:879)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:741)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:213)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
