[jira] [Updated] (MAPREDUCE-3315) Master-Worker Application on YARN
[ https://issues.apache.org/jira/browse/MAPREDUCE-3315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nikhil S. Ketkar updated MAPREDUCE-3315: Attachment: MAPREDUCE-3315-3.patch New patch with all previous changes, plus the change suggested by Robert Joseph Evans on finding an unused port for the Master. Master-Worker Application on YARN - Key: MAPREDUCE-3315 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3315 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Sharad Agarwal Assignee: Sharad Agarwal Fix For: 0.24.0 Attachments: MAPREDUCE-3315-1.patch, MAPREDUCE-3315-2.patch, MAPREDUCE-3315-3.patch, MAPREDUCE-3315.patch Currently master worker scenarios are forced fit into Map-Reduce. Now with YARN, these can be first class and would benefit real/near realtime workloads and be more effective in using the cluster resources. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4108) Correct org.apache.hadoop.util.TestRunJar
[
https://issues.apache.org/jira/browse/MAPREDUCE-4108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247062#comment-13247062
]
Hadoop QA commented on MAPREDUCE-4108:
--
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12521462/MAPREDUCE-4108.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 10 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2152//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2152//console
This message is automatically generated.
Correct org.apache.hadoop.util.TestRunJar
-
Key: MAPREDUCE-4108
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4108
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4108.patch
Project : *hadoop-mapreduce-client-jobclient*
{code:xml}
Running org.apache.hadoop.util.TestRunJar
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.031 sec
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4107) Correct org.apache.hadoop.ipc.TestSocketFactory
[
https://issues.apache.org/jira/browse/MAPREDUCE-4107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247063#comment-13247063
]
Hadoop QA commented on MAPREDUCE-4107:
--
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12521461/MAPREDUCE-4107.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2153//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2153//console
This message is automatically generated.
Correct org.apache.hadoop.ipc.TestSocketFactory
---
Key: MAPREDUCE-4107
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4107
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4107.patch
Project : *hadoop-mapreduce-client-jobclient*
{code:xml}
Running org.apache.hadoop.ipc.TestSocketFactory
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.031 sec
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3674) If invoked with no queueName request param, jobqueue_details.jsp injects a null queue name into schedulers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated MAPREDUCE-3674:
---
Attachment: hadoop-findbugs-report.html
MAPREDUCE-3674.patch
New patch with one test added.
{code}
-1 overall.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
-1 findbugs. The patch appears to introduce 196 new Findbugs (version
2.0.0) warnings.
{code}
None of the 196 findbugs are me. Report attached.
If invoked with no queueName request param, jobqueue_details.jsp injects a
null queue name into schedulers.
---
Key: MAPREDUCE-3674
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3674
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 1.0.0
Reporter: Harsh J
Assignee: Harsh J
Priority: Critical
Attachments: MAPREDUCE-3674.patch, MAPREDUCE-3674.patch,
MAPREDUCE-3674.patch, hadoop-findbugs-report.html
When you access /jobqueue_details.jsp manually, instead of via a link, it has
queueName set to null internally and this goes for a lookup into the
scheduling info maps as well.
As a result, if using FairScheduler, a Pool with String name = null gets
created and this brings the scheduler down. I have not tested what happens to
the CapacityScheduler, but ideally if no queueName is set in that jsp, it
should fall back to 'default'. Otherwise, this brings down the JobTracker
completely.
FairScheduler must also add a check to not create a pool with 'null' name.
The following is the strace that ensues:
{code}
ERROR org.mortbay.log: /jobqueue_details.jsp
java.lang.NullPointerException
at
org.apache.hadoop.mapred.jobqueue_005fdetails_jsp._jspService(jobqueue_005fdetails_jsp.java:71)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:829)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9001, call
heartbeat from XYZ:MNOP: error: java.io.IOException:
java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
at
org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:95)
at
org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:68)
at java.util.Arrays.mergeSort(Unknown Source)
at java.util.Arrays.sort(Unknown Source)
at java.util.Collections.sort(Unknown Source)
at org.apache.hadoop.mapred.FairScheduler.assignTasks(FairScheduler.java:435)
at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3226)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
at
[jira] [Updated] (MAPREDUCE-3674) If invoked with no queueName request param, jobqueue_details.jsp injects a null queue name into schedulers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J updated MAPREDUCE-3674:
---
Attachment: MAPREDUCE-3674.patch
If invoked with no queueName request param, jobqueue_details.jsp injects a
null queue name into schedulers.
---
Key: MAPREDUCE-3674
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3674
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 1.0.0
Reporter: Harsh J
Assignee: Harsh J
Priority: Critical
Attachments: MAPREDUCE-3674.patch, MAPREDUCE-3674.patch,
MAPREDUCE-3674.patch, MAPREDUCE-3674.patch, hadoop-findbugs-report.html
When you access /jobqueue_details.jsp manually, instead of via a link, it has
queueName set to null internally and this goes for a lookup into the
scheduling info maps as well.
As a result, if using FairScheduler, a Pool with String name = null gets
created and this brings the scheduler down. I have not tested what happens to
the CapacityScheduler, but ideally if no queueName is set in that jsp, it
should fall back to 'default'. Otherwise, this brings down the JobTracker
completely.
FairScheduler must also add a check to not create a pool with 'null' name.
The following is the strace that ensues:
{code}
ERROR org.mortbay.log: /jobqueue_details.jsp
java.lang.NullPointerException
at
org.apache.hadoop.mapred.jobqueue_005fdetails_jsp._jspService(jobqueue_005fdetails_jsp.java:71)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:829)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9001, call
heartbeat from XYZ:MNOP: error: java.io.IOException:
java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
at
org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:95)
at
org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:68)
at java.util.Arrays.mergeSort(Unknown Source)
at java.util.Arrays.sort(Unknown Source)
at java.util.Collections.sort(Unknown Source)
at org.apache.hadoop.mapred.FairScheduler.assignTasks(FairScheduler.java:435)
at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3226)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1434)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1430)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1428)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
[jira] [Commented] (MAPREDUCE-3674) If invoked with no queueName request param, jobqueue_details.jsp injects a null queue name into schedulers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247074#comment-13247074
]
Hadoop QA commented on MAPREDUCE-3674:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12521466/MAPREDUCE-3674.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2154//console
This message is automatically generated.
If invoked with no queueName request param, jobqueue_details.jsp injects a
null queue name into schedulers.
---
Key: MAPREDUCE-3674
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3674
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 1.0.0
Reporter: Harsh J
Assignee: Harsh J
Priority: Critical
Attachments: MAPREDUCE-3674.patch, MAPREDUCE-3674.patch,
MAPREDUCE-3674.patch, MAPREDUCE-3674.patch, hadoop-findbugs-report.html
When you access /jobqueue_details.jsp manually, instead of via a link, it has
queueName set to null internally and this goes for a lookup into the
scheduling info maps as well.
As a result, if using FairScheduler, a Pool with String name = null gets
created and this brings the scheduler down. I have not tested what happens to
the CapacityScheduler, but ideally if no queueName is set in that jsp, it
should fall back to 'default'. Otherwise, this brings down the JobTracker
completely.
FairScheduler must also add a check to not create a pool with 'null' name.
The following is the strace that ensues:
{code}
ERROR org.mortbay.log: /jobqueue_details.jsp
java.lang.NullPointerException
at
org.apache.hadoop.mapred.jobqueue_005fdetails_jsp._jspService(jobqueue_005fdetails_jsp.java:71)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:829)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9001, call
heartbeat from XYZ:MNOP: error: java.io.IOException:
java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
at
org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:95)
at
org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:68)
at java.util.Arrays.mergeSort(Unknown Source)
at java.util.Arrays.sort(Unknown Source)
at java.util.Collections.sort(Unknown Source)
at org.apache.hadoop.mapred.FairScheduler.assignTasks(FairScheduler.java:435)
at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3226)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
at
[jira] [Commented] (MAPREDUCE-3674) If invoked with no queueName request param, jobqueue_details.jsp injects a null queue name into schedulers.
[
https://issues.apache.org/jira/browse/MAPREDUCE-3674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247077#comment-13247077
]
Hadoop QA commented on MAPREDUCE-3674:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12521468/MAPREDUCE-3674.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2155//console
This message is automatically generated.
If invoked with no queueName request param, jobqueue_details.jsp injects a
null queue name into schedulers.
---
Key: MAPREDUCE-3674
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3674
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 1.0.0
Reporter: Harsh J
Assignee: Harsh J
Priority: Critical
Attachments: MAPREDUCE-3674.patch, MAPREDUCE-3674.patch,
MAPREDUCE-3674.patch, MAPREDUCE-3674.patch, hadoop-findbugs-report.html
When you access /jobqueue_details.jsp manually, instead of via a link, it has
queueName set to null internally and this goes for a lookup into the
scheduling info maps as well.
As a result, if using FairScheduler, a Pool with String name = null gets
created and this brings the scheduler down. I have not tested what happens to
the CapacityScheduler, but ideally if no queueName is set in that jsp, it
should fall back to 'default'. Otherwise, this brings down the JobTracker
completely.
FairScheduler must also add a check to not create a pool with 'null' name.
The following is the strace that ensues:
{code}
ERROR org.mortbay.log: /jobqueue_details.jsp
java.lang.NullPointerException
at
org.apache.hadoop.mapred.jobqueue_005fdetails_jsp._jspService(jobqueue_005fdetails_jsp.java:71)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
at
org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.java:829)
at
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
at
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
at
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at
org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
at
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9001, call
heartbeat from XYZ:MNOP: error: java.io.IOException:
java.lang.NullPointerException
java.io.IOException: java.lang.NullPointerException
at
org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:95)
at
org.apache.hadoop.mapred.SchedulingAlgorithms$FairShareComparator.compare(SchedulingAlgorithms.java:68)
at java.util.Arrays.mergeSort(Unknown Source)
at java.util.Arrays.sort(Unknown Source)
at java.util.Collections.sort(Unknown Source)
at org.apache.hadoop.mapred.FairScheduler.assignTasks(FairScheduler.java:435)
at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3226)
at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:557)
at
[jira] [Created] (MAPREDUCE-4109) availability of a job info in HS should be atomic
availability of a job info in HS should be atomic
-
Key: MAPREDUCE-4109
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4109
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: applicationmaster, jobhistoryserver, mrv2
Affects Versions: 2.0.0
Reporter: Alejandro Abdelnur
Priority: Blocker
Fix For: 2.0.0
It seems that the HS starts serving info about a job before it has all the info
available.
In the trace below, a RunningJob throws a NPE when trying to access the
counters.
This is happening on off, thus I assume it is related to either the AM not
flushing all job info to HDFS before notifying HS or the HS not loading all the
job info from HDFS before start serving it.
In case it helps to diagnose the issue, this is happening in a secure cluster.
This makes Oozie to mark jobs as failed.
{code}
java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$MRClientProtocolHandler.getCounters(HistoryClientService.java:214)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:149)
at
org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:206)
at
org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Server.call(ProtoOverHadoopRpcEngine.java:355)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1660)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1656)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1654)
at LocalTrace:
org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
at
org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:163)
at $Proxy31.getCounters(Unknown Source)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getCounters(MRClientProtocolPBClientImpl.java:162)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:296)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobCounters(ClientServiceDelegate.java:325)
at
org.apache.hadoop.mapred.YARNRunner.getJobCounters(YARNRunner.java:472)
at org.apache.hadoop.mapreduce.Job$8.run(Job.java:714)
at org.apache.hadoop.mapreduce.Job$8.run(Job.java:711)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:416)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.mapreduce.Job.getCounters(Job.java:711)
at
org.apache.hadoop.mapred.JobClient$NetworkedJob.getCounters(JobClient.java:396)
at
org.apache.oozie.action.hadoop.LauncherMapper.hasIdSwap(LauncherMapper.java:296)
at
org.apache.oozie.action.hadoop.JavaActionExecutor.check(JavaActionExecutor.java:886)
at
org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:162)
at
org.apache.oozie.command.wf.ActionCheckXCommand.execute(ActionCheckXCommand.java:51)
at org.apache.oozie.command.XCommand.call(XCommand.java:260)
at
org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4110) Fix tests in org.apache.hadoop.mapred.TestMiniMRClasspath org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
Fix tests in org.apache.hadoop.mapred.TestMiniMRClasspath
org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
---
Key: MAPREDUCE-4110
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4110
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2, test
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Sub Project : *hadoop-mapreduce-client-jobclient*
{code:xml}
Running org.apache.hadoop.mapred.TestMiniMRClasspath
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.032 sec
{code}
{code:xml}
Running org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.031 sec
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4111) Fix tests in org.apache.hadoop.mapred.TestJobName
Fix tests in org.apache.hadoop.mapred.TestJobName
-
Key: MAPREDUCE-4111
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4111
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2, test
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Sub Project : *hadoop-mapreduce-client-jobclient*
{code:xml}
Running org.apache.hadoop.mapred.TestJobName
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.031 sec
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4112) Fix tests org.apache.hadoop.mapred.TestClusterMapReduceTestCase
Fix tests org.apache.hadoop.mapred.TestClusterMapReduceTestCase
---
Key: MAPREDUCE-4112
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4112
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2, test
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Sub Project : *hadoop-mapreduce-client-jobclient*
{code:xml}
Running org.apache.hadoop.mapred.TestClusterMapReduceTestCase
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.032 sec
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4110) Fix tests in org.apache.hadoop.mapred.TestMiniMRClasspath org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
[
https://issues.apache.org/jira/browse/MAPREDUCE-4110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-4110:
-
Attachment: MAPREDUCE-4110.patch
Fix tests in org.apache.hadoop.mapred.TestMiniMRClasspath
org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
---
Key: MAPREDUCE-4110
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4110
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2, test
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4110.patch
Sub Project : *hadoop-mapreduce-client-jobclient*
{code:xml}
Running org.apache.hadoop.mapred.TestMiniMRClasspath
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.032 sec
{code}
{code:xml}
Running org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.031 sec
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4110) Fix tests in org.apache.hadoop.mapred.TestMiniMRClasspath org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
[
https://issues.apache.org/jira/browse/MAPREDUCE-4110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-4110:
-
Status: Patch Available (was: Open)
Attached patch to fix this.
Fix tests in org.apache.hadoop.mapred.TestMiniMRClasspath
org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
---
Key: MAPREDUCE-4110
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4110
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2, test
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4110.patch
Sub Project : *hadoop-mapreduce-client-jobclient*
{code:xml}
Running org.apache.hadoop.mapred.TestMiniMRClasspath
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.032 sec
{code}
{code:xml}
Running org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.031 sec
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4111) Fix tests in org.apache.hadoop.mapred.TestJobName
[
https://issues.apache.org/jira/browse/MAPREDUCE-4111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-4111:
-
Attachment: MAPREDUCE-4111.patch
Fix tests in org.apache.hadoop.mapred.TestJobName
-
Key: MAPREDUCE-4111
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4111
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2, test
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4111.patch
Sub Project : *hadoop-mapreduce-client-jobclient*
{code:xml}
Running org.apache.hadoop.mapred.TestJobName
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.031 sec
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4111) Fix tests in org.apache.hadoop.mapred.TestJobName
[
https://issues.apache.org/jira/browse/MAPREDUCE-4111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-4111:
-
Status: Patch Available (was: Open)
Attached patch to fix this.
Fix tests in org.apache.hadoop.mapred.TestJobName
-
Key: MAPREDUCE-4111
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4111
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2, test
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4111.patch
Sub Project : *hadoop-mapreduce-client-jobclient*
{code:xml}
Running org.apache.hadoop.mapred.TestJobName
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.031 sec
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4112) Fix tests org.apache.hadoop.mapred.TestClusterMapReduceTestCase
[
https://issues.apache.org/jira/browse/MAPREDUCE-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-4112:
-
Attachment: MAPREDUCE-4112.patch
Fix tests org.apache.hadoop.mapred.TestClusterMapReduceTestCase
---
Key: MAPREDUCE-4112
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4112
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2, test
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4112.patch
Sub Project : *hadoop-mapreduce-client-jobclient*
{code:xml}
Running org.apache.hadoop.mapred.TestClusterMapReduceTestCase
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.032 sec
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4113) Fix tests org.apache.hadoop.mapred.TestClusterMRNotification
[
https://issues.apache.org/jira/browse/MAPREDUCE-4113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-4113:
-
Status: Patch Available (was: Open)
Attached patch to fix this.
Fix tests org.apache.hadoop.mapred.TestClusterMRNotification
Key: MAPREDUCE-4113
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4113
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2, test
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4113.patch
Sub Project : *hadoop-mapreduce-client-jobclient*
{code:xml}
Running org.apache.hadoop.mapred.TestClusterMRNotification
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.031 sec
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4113) Fix tests org.apache.hadoop.mapred.TestClusterMRNotification
[
https://issues.apache.org/jira/browse/MAPREDUCE-4113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-4113:
-
Attachment: MAPREDUCE-4113.patch
Fix tests org.apache.hadoop.mapred.TestClusterMRNotification
Key: MAPREDUCE-4113
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4113
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2, test
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4113.patch
Sub Project : *hadoop-mapreduce-client-jobclient*
{code:xml}
Running org.apache.hadoop.mapred.TestClusterMRNotification
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.031 sec
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4112) Fix tests org.apache.hadoop.mapred.TestClusterMapReduceTestCase
[
https://issues.apache.org/jira/browse/MAPREDUCE-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Devaraj K updated MAPREDUCE-4112:
-
Status: Patch Available (was: Open)
Attached patch to fix this.
Fix tests org.apache.hadoop.mapred.TestClusterMapReduceTestCase
---
Key: MAPREDUCE-4112
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4112
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2, test
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4112.patch
Sub Project : *hadoop-mapreduce-client-jobclient*
{code:xml}
Running org.apache.hadoop.mapred.TestClusterMapReduceTestCase
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.032 sec
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4071) NPE while executing MRAppMaster shutdown hook
[
https://issues.apache.org/jira/browse/MAPREDUCE-4071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bhallamudi Venkata Siva Kamesh reassigned MAPREDUCE-4071:
-
Assignee: (was: Bhallamudi Venkata Siva Kamesh)
NPE while executing MRAppMaster shutdown hook
-
Key: MAPREDUCE-4071
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4071
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am, mrv2
Affects Versions: 0.23.3, 2.0.0
Reporter: Bhallamudi Venkata Siva Kamesh
Attachments: MAPREDUCE-4071-1.patch, MAPREDUCE-4071.patch
While running the shutdown hook of MRAppMaster, hit NPE
{noformat}
Exception in thread Thread-1 java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.setSignalled(MRAppMaster.java:668)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1004)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4071) NPE while executing MRAppMaster shutdown hook
[
https://issues.apache.org/jira/browse/MAPREDUCE-4071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bhallamudi Venkata Siva Kamesh updated MAPREDUCE-4071:
--
Attachment: MAPREDUCE-4071-1.patch
Hi Robert,
Thanks for looking into the patch. I totally ignored the unregister method in
RMCommunicator. I modified the patch and most of the changes are re-factoring
like moving object instantiation to init method etc... If you feel, this is
not the right way to fix this, please look into this issue.
NPE while executing MRAppMaster shutdown hook
-
Key: MAPREDUCE-4071
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4071
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am, mrv2
Affects Versions: 0.23.3, 2.0.0
Reporter: Bhallamudi Venkata Siva Kamesh
Assignee: Bhallamudi Venkata Siva Kamesh
Attachments: MAPREDUCE-4071-1.patch, MAPREDUCE-4071.patch
While running the shutdown hook of MRAppMaster, hit NPE
{noformat}
Exception in thread Thread-1 java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.setSignalled(MRAppMaster.java:668)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1004)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3621) TestDBJob and TestDataDrivenDBInputFormat ant tests fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-3621: - Status: Patch Available (was: Open) TestDBJob and TestDataDrivenDBInputFormat ant tests fail Key: MAPREDUCE-3621 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3621 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Ravi Prakash Attachments: MAPREDUCE-3621.patch, MAPREDUCE-3621.patch, MAPREDUCE-3621.patch The following mapred ant tests fail and have been failing for a very long time: [junit] Running org.apache.hadoop.mapreduce.lib.db.TestDBJob [junit] Running org.apache.hadoop.mapreduce.lib.db.TestDataDrivenDBInputFormat -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4071) NPE while executing MRAppMaster shutdown hook
[
https://issues.apache.org/jira/browse/MAPREDUCE-4071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-4071:
---
Status: Patch Available (was: Open)
Kicking Jenkins again
NPE while executing MRAppMaster shutdown hook
-
Key: MAPREDUCE-4071
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4071
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am, mrv2
Affects Versions: 0.23.3, 2.0.0
Reporter: Bhallamudi Venkata Siva Kamesh
Attachments: MAPREDUCE-4071-1.patch, MAPREDUCE-4071.patch
While running the shutdown hook of MRAppMaster, hit NPE
{noformat}
Exception in thread Thread-1 java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.setSignalled(MRAppMaster.java:668)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1004)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4071) NPE while executing MRAppMaster shutdown hook
[
https://issues.apache.org/jira/browse/MAPREDUCE-4071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247224#comment-13247224
]
Hadoop QA commented on MAPREDUCE-4071:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12521491/MAPREDUCE-4071-1.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The patch appears to cause tar ant target to fail.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2158//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2158//console
This message is automatically generated.
NPE while executing MRAppMaster shutdown hook
-
Key: MAPREDUCE-4071
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4071
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am, mrv2
Affects Versions: 0.23.3, 2.0.0
Reporter: Bhallamudi Venkata Siva Kamesh
Attachments: MAPREDUCE-4071-1.patch, MAPREDUCE-4071.patch
While running the shutdown hook of MRAppMaster, hit NPE
{noformat}
Exception in thread Thread-1 java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.setSignalled(MRAppMaster.java:668)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1004)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4071) NPE while executing MRAppMaster shutdown hook
[
https://issues.apache.org/jira/browse/MAPREDUCE-4071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247230#comment-13247230
]
Robert Joseph Evans commented on MAPREDUCE-4071:
The patch does not appear to compile against trunk.
{code}
/home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java:[1217,4]
method does not override or implement a method from a supertype
{code}
It could be a hickup in Jenkins, but could you please try to upmerge and verify
that everything still builds and runs correctly.
NPE while executing MRAppMaster shutdown hook
-
Key: MAPREDUCE-4071
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4071
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am, mrv2
Affects Versions: 0.23.3, 2.0.0
Reporter: Bhallamudi Venkata Siva Kamesh
Attachments: MAPREDUCE-4071-1.patch, MAPREDUCE-4071.patch
While running the shutdown hook of MRAppMaster, hit NPE
{noformat}
Exception in thread Thread-1 java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.setSignalled(MRAppMaster.java:668)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1004)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3729) Commit build failing TestJobClientGetJob, TestMRWithDistributedCache, TestLocalModeWithNewApis
[ https://issues.apache.org/jira/browse/MAPREDUCE-3729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved MAPREDUCE-3729. Resolution: Cannot Reproduce It looks like there is really no way to make code changes unless we can get this to happen again. And even then it is questionable what kind of changes we could make to prevent this in the future. Commit build failing TestJobClientGetJob, TestMRWithDistributedCache, TestLocalModeWithNewApis -- Key: MAPREDUCE-3729 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3729 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, test Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Robert Joseph Evans Priority: Critical Fix For: 0.23.1 See https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1671//testReport/. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4113) Fix tests org.apache.hadoop.mapred.TestClusterMRNotification
[
https://issues.apache.org/jira/browse/MAPREDUCE-4113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247249#comment-13247249
]
Hadoop QA commented on MAPREDUCE-4113:
--
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12521484/MAPREDUCE-4113.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2156//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2156//console
This message is automatically generated.
Fix tests org.apache.hadoop.mapred.TestClusterMRNotification
Key: MAPREDUCE-4113
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4113
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2, test
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4113.patch
Sub Project : *hadoop-mapreduce-client-jobclient*
{code:xml}
Running org.apache.hadoop.mapred.TestClusterMRNotification
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.031 sec
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4112) Fix tests org.apache.hadoop.mapred.TestClusterMapReduceTestCase
[
https://issues.apache.org/jira/browse/MAPREDUCE-4112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247251#comment-13247251
]
Hadoop QA commented on MAPREDUCE-4112:
--
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12521483/MAPREDUCE-4112.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2157//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2157//console
This message is automatically generated.
Fix tests org.apache.hadoop.mapred.TestClusterMapReduceTestCase
---
Key: MAPREDUCE-4112
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4112
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2, test
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4112.patch
Sub Project : *hadoop-mapreduce-client-jobclient*
{code:xml}
Running org.apache.hadoop.mapred.TestClusterMapReduceTestCase
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.032 sec
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4111) Fix tests in org.apache.hadoop.mapred.TestJobName
[
https://issues.apache.org/jira/browse/MAPREDUCE-4111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247250#comment-13247250
]
Hadoop QA commented on MAPREDUCE-4111:
--
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12521482/MAPREDUCE-4111.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2159//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2159//console
This message is automatically generated.
Fix tests in org.apache.hadoop.mapred.TestJobName
-
Key: MAPREDUCE-4111
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4111
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2, test
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4111.patch
Sub Project : *hadoop-mapreduce-client-jobclient*
{code:xml}
Running org.apache.hadoop.mapred.TestJobName
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.031 sec
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4110) Fix tests in org.apache.hadoop.mapred.TestMiniMRClasspath org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
[
https://issues.apache.org/jira/browse/MAPREDUCE-4110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247264#comment-13247264
]
Hadoop QA commented on MAPREDUCE-4110:
--
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12521481/MAPREDUCE-4110.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 14 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2160//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2160//console
This message is automatically generated.
Fix tests in org.apache.hadoop.mapred.TestMiniMRClasspath
org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
---
Key: MAPREDUCE-4110
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4110
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Components: mrv2, test
Affects Versions: 2.0.0, 3.0.0
Reporter: Devaraj K
Assignee: Devaraj K
Attachments: MAPREDUCE-4110.patch
Sub Project : *hadoop-mapreduce-client-jobclient*
{code:xml}
Running org.apache.hadoop.mapred.TestMiniMRClasspath
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.032 sec
{code}
{code:xml}
Running org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers
Tests run: 1, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 0.031 sec
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avner BenHanoch updated MAPREDUCE-4049: --- Attachment: test.diff src.tgz mapred-site.xml mapred.diff Attached is fix for supporting shuffle consumer shuffle provider plugins in Hadoop. The fix is intended for review. Two comments are already known to me: 1. I changed several members of ReduceTask to be public for allowing easy access by ShuffleConsumerPlugin(s). I will enhance that in future commits. 2. ShuffleProviderPlugin has currently very few members. Hence, currently, it doesn't support plugins that support security. I will enhance that in future commits. Technical info: 1. The fix contains 2 new source files (that were splitted from the BIG ReduceTask.java file) and changes in few existing files (One of them is a unit test file). 2. The fix is based on Hadoop 1.0.0. After your comments I'll be glad to provide enhanced fix for future 1.x.y versions (as well as for 2.x) 3. The fix is provided in two formats: a) as 2 diff files (src test), and b) as source files in src.tgz plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 0.23.1, 1.0.1 Reporter: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Attachments: Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, mapred-site.xml, mapred.diff, src.tgz, test.diff Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avner BenHanoch updated MAPREDUCE-4049: --- Fix Version/s: 1.0.0 Target Version/s: 0.24.0, 1.1.0, 0.23.2, 1.0.2, 0.23.3, 1.0.3 (was: 1.0.3, 0.23.3, 1.0.2, 0.23.2, 1.1.0, 0.24.0) Status: Patch Available (was: Open) plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 1.0.1, 0.23.1 Reporter: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Fix For: 1.0.0 Attachments: Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, mapred-site.xml, mapred.diff, src.tgz, test.diff Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247317#comment-13247317 ] Hadoop QA commented on MAPREDUCE-4049: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521522/test.diff against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2161//console This message is automatically generated. plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 0.23.1, 1.0.1 Reporter: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Fix For: 1.0.0 Attachments: Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, mapred-site.xml, mapred.diff, src.tgz, test.diff Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avner BenHanoch updated MAPREDUCE-4049: --- Attachment: HADOOP-1.0.2.patch This is same patch as before - this time it should be in correct format for automation (and not just methodological as before) Notice that this time it is intended for hadoop-1.0.2 plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 0.23.1, 1.0.1 Reporter: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Fix For: 1.0.0 Attachments: HADOOP-1.0.2.patch, Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, mapred-site.xml, mapred.diff, src.tgz, test.diff Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avner BenHanoch updated MAPREDUCE-4049: --- Target Version/s: 0.24.0, 1.1.0, 0.23.2, 1.0.2, 0.23.3, 1.0.3 (was: 1.0.3, 0.23.3, 1.0.2, 0.23.2, 1.1.0, 0.24.0) Status: Open (was: Patch Available) this was only methodological patch and not a formal one, based on svn paths plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 1.0.1, 0.23.1 Reporter: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Fix For: 1.0.0 Attachments: HADOOP-1.0.2.patch, Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, mapred-site.xml, mapred.diff, src.tgz, test.diff Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avner BenHanoch updated MAPREDUCE-4049: --- Fix Version/s: (was: 1.0.0) 1.0.2 Target Version/s: 0.24.0, 1.1.0, 0.23.2, 1.0.2, 0.23.3, 1.0.3 (was: 1.0.3, 0.23.3, 1.0.2, 0.23.2, 1.1.0, 0.24.0) Status: Patch Available (was: Open) HADOOP-1.0.2.patch plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 1.0.1, 0.23.1 Reporter: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Fix For: 1.0.2 Attachments: HADOOP-1.0.2.patch, Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, mapred-site.xml, mapred.diff, src.tgz, test.diff Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247374#comment-13247374 ] Hadoop QA commented on MAPREDUCE-4049: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521533/HADOOP-1.0.2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2162//console This message is automatically generated. plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 0.23.1, 1.0.1 Reporter: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Fix For: 1.0.2 Attachments: HADOOP-1.0.2.patch, Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, mapred-site.xml, mapred.diff, src.tgz, test.diff Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avner BenHanoch updated MAPREDUCE-4049: --- Target Version/s: 0.24.0, 1.1.0, 0.23.2, 1.0.2, 0.23.3, 1.0.3 (was: 1.0.3, 0.23.3, 1.0.2, 0.23.2, 1.1.0, 0.24.0) Status: Open (was: Patch Available) plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 1.0.1, 0.23.1 Reporter: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Fix For: 1.0.2 Attachments: HADOOP-1.0.2.patch, Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, mapred-site.xml, mapred.diff, src.tgz, test.diff Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avner BenHanoch updated MAPREDUCE-4049: --- Attachment: HADOOP-1.0.x.patch This time the patch is after svn add for new files plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 0.23.1, 1.0.1 Reporter: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Fix For: 1.0.2 Attachments: HADOOP-1.0.2.patch, HADOOP-1.0.x.patch, Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, mapred-site.xml, mapred.diff, src.tgz, test.diff Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avner BenHanoch updated MAPREDUCE-4049: --- Target Version/s: 0.24.0, 1.1.0, 0.23.2, 1.0.2, 0.23.3, 1.0.3 (was: 1.0.3, 0.23.3, 1.0.2, 0.23.2, 1.1.0, 0.24.0) Status: Patch Available (was: Open) patch for 1.0.2 (includes svn add for 3 new files) plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 1.0.1, 0.23.1 Reporter: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Fix For: 1.0.2 Attachments: HADOOP-1.0.2.patch, HADOOP-1.0.x.patch, Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, mapred-site.xml, mapred.diff, src.tgz, test.diff Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247393#comment-13247393 ] Hadoop QA commented on MAPREDUCE-4049: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521534/HADOOP-1.0.x.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2163//console This message is automatically generated. plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 0.23.1, 1.0.1 Reporter: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Fix For: 1.0.2 Attachments: HADOOP-1.0.2.patch, HADOOP-1.0.x.patch, Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, mapred-site.xml, mapred.diff, src.tgz, test.diff Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avner BenHanoch updated MAPREDUCE-4049: --- Attachment: MAPREDUCE-4049-branch-1.0.2.patch same patch with different name and with ASF license plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 0.23.1, 1.0.1 Reporter: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Fix For: 1.0.2 Attachments: HADOOP-1.0.2.patch, HADOOP-1.0.x.patch, Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, MAPREDUCE-4049-branch-1.0.2.patch, mapred-site.xml, mapred.diff, src.tgz, test.diff Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avner BenHanoch updated MAPREDUCE-4049: --- Target Version/s: 0.24.0, 1.1.0, 0.23.2, 1.0.2, 0.23.3, 1.0.3 (was: 1.0.3, 0.23.3, 1.0.2, 0.23.2, 1.1.0, 0.24.0) Status: Open (was: Patch Available) plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 1.0.1, 0.23.1 Reporter: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Fix For: 1.0.2 Attachments: HADOOP-1.0.2.patch, HADOOP-1.0.x.patch, Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, MAPREDUCE-4049-branch-1.0.2.patch, mapred-site.xml, mapred.diff, src.tgz, test.diff Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Avner BenHanoch updated MAPREDUCE-4049: --- Target Version/s: 0.24.0, 1.1.0, 0.23.2, 1.0.2, 0.23.3, 1.0.3 (was: 1.0.3, 0.23.3, 1.0.2, 0.23.2, 1.1.0, 0.24.0) Status: Patch Available (was: Open) same patch with different meta data plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 1.0.1, 0.23.1 Reporter: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Fix For: 1.0.2 Attachments: HADOOP-1.0.2.patch, HADOOP-1.0.x.patch, Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, MAPREDUCE-4049-branch-1.0.2.patch, mapred-site.xml, mapred.diff, src.tgz, test.diff Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247439#comment-13247439 ] Aaron T. Myers commented on MAPREDUCE-4049: --- Hey Avner, apologies if you already know this, but the automated test-patch isn't going to work for any patch made against branch-1. test-patch always blindly tries to apply the patch to trunk, regardless of what branch it's intended for. The source layout is dramatically different between branch-1 and trunk, so it's unlikely that any patch will apply cleanly. You should probably run test-patch manually on your box and post the results here. plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 0.23.1, 1.0.1 Reporter: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Fix For: 1.0.2 Attachments: HADOOP-1.0.2.patch, HADOOP-1.0.x.patch, Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, MAPREDUCE-4049-branch-1.0.2.patch, mapred-site.xml, mapred.diff, src.tgz, test.diff Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247449#comment-13247449 ] Hadoop QA commented on MAPREDUCE-4049: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521536/MAPREDUCE-4049-branch-1.0.2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2164//console This message is automatically generated. plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 0.23.1, 1.0.1 Reporter: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Fix For: 1.0.2 Attachments: HADOOP-1.0.2.patch, HADOOP-1.0.x.patch, Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, MAPREDUCE-4049-branch-1.0.2.patch, mapred-site.xml, mapred.diff, src.tgz, test.diff Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247461#comment-13247461 ] Avner BenHanoch commented on MAPREDUCE-4049: Thanks Aaron, I was not aware of that and it seems strange to me (also the error message is not helping). The patch is for 1.0.2, not for the trunk. It was created using: *svn diff src/mapred/ src/test/*. It runs smoothly in my env. I am able to apply it on clean 1.0.2 tree using *patch -p0 patch-file* I hope it can be reviewed based on the attached source/patch. Otherwise, I'll try test-patch after Passover :) plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 0.23.1, 1.0.1 Reporter: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Fix For: 1.0.2 Attachments: HADOOP-1.0.2.patch, HADOOP-1.0.x.patch, Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, MAPREDUCE-4049-branch-1.0.2.patch, mapred-site.xml, mapred.diff, src.tgz, test.diff Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247464#comment-13247464 ] Aaron T. Myers commented on MAPREDUCE-4049: --- Yea, it's an unfortunate known deficiency of test-patch: HADOOP-7435 plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 0.23.1, 1.0.1 Reporter: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Fix For: 1.0.2 Attachments: HADOOP-1.0.2.patch, HADOOP-1.0.x.patch, Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, MAPREDUCE-4049-branch-1.0.2.patch, mapred-site.xml, mapred.diff, src.tgz, test.diff Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247467#comment-13247467 ] Avner BenHanoch commented on MAPREDUCE-4049: thanks. Meanwhile, I'll ignore those automatic QA messages. plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 0.23.1, 1.0.1 Reporter: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Fix For: 1.0.2 Attachments: HADOOP-1.0.2.patch, HADOOP-1.0.x.patch, Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, MAPREDUCE-4049-branch-1.0.2.patch, mapred-site.xml, mapred.diff, src.tgz, test.diff Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated MAPREDUCE-4049: -- Target Version/s: 0.24.0, 1.1.0, 0.23.2, 0.23.3, 1.0.3 (was: 1.0.3, 0.23.3, 1.0.2, 0.23.2, 1.1.0, 0.24.0) Fix Version/s: (was: 1.0.2) This patch was not committed to Hadoop-1.0.2. Corrected the Fix Versions and Target Versions fields. plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 0.23.1, 1.0.1 Reporter: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Attachments: HADOOP-1.0.2.patch, HADOOP-1.0.x.patch, Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, MAPREDUCE-4049-branch-1.0.2.patch, mapred-site.xml, mapred.diff, src.tgz, test.diff Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4088) Task stuck in JobLocalizer prevented other tasks on the same node from committing
[ https://issues.apache.org/jira/browse/MAPREDUCE-4088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated MAPREDUCE-4088: -- Target Version/s: 1.0.3 (was: 1.0.2) Task stuck in JobLocalizer prevented other tasks on the same node from committing - Key: MAPREDUCE-4088 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4088 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1 Affects Versions: 0.20.205.0 Reporter: Ravi Prakash Priority: Critical We saw that as a result of HADOOP-6963, one task was stuck in this Thread 23668: (state = IN_NATIVE) - java.io.UnixFileSystem.getBooleanAttributes0(java.io.File) @bci=0 (Compiled frame; information may be imprecise) - java.io.UnixFileSystem.getBooleanAttributes(java.io.File) @bci=2, line=228 (Compiled frame) - java.io.File.exists() @bci=20, line=733 (Compiled frame) - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=3, line=446 (Compiled frame) - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 (Compiled frame) - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 (Compiled frame) TONS MORE OF THIS SAME LINE - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 (Compiled frame) . . - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 (Compiled frame) - org.apache.hadoop.fs.FileUtil.getDU(java.io.File) @bci=52, line=455 (Interpreted frame) ne=451 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer.downloadPrivateCacheObjects(org.apache.hadoop.conf.Configuration, java.net.URI[], org.apache.hadoop.fs.Path[], long[], boolean[], boolean) @bci=150, line=324 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer.downloadPrivateCache(org.apache.hadoop.conf.Configuration) @bci=40, line=349 (Interpreted frame) 51, line=383 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer.runSetup(java.lang.String, java.lang.String, org.apache.hadoop.fs.Path, org.apache.hadoop.mapred.TaskUmbilicalProtocol) @bci=46, line=477 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer$3.run() @bci=20, line=534 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer$3.run() @bci=1, line=531 (Interpreted frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction, java.security.AccessControlContext) @bci=0 (Interpreted frame) - javax.security.auth.Subject.doAs(javax.security.auth.Subject, java.security.PrivilegedExceptionAction) @bci=42, line=396 (Interpreted frame) - org.apache.hadoop.security.UserGroupInformation.doAs(java.security.PrivilegedExceptionAction) @bci=14, line=1082 (Interpreted frame) - org.apache.hadoop.mapred.JobLocalizer.main(java.lang.String[]) @bci=266, line=530 (Interpreted frame) While all other tasks on the same node were stuck in Thread 32141: (state = BLOCKED) - java.lang.Thread.sleep(long) @bci=0 (Interpreted frame) - org.apache.hadoop.mapred.Task.commit(org.apache.hadoop.mapred.TaskUmbilicalProtocol, org.apache.hadoop.mapred.Task$TaskReporter, org.apache.hadoop.mapreduce.OutputCommitter) @bci=24, line=980 (Compiled frame) - org.apache.hadoop.mapred.Task.done(org.apache.hadoop.mapred.TaskUmbilicalProtocol, org.apache.hadoop.mapred.Task$TaskReporter) @bci=146, line=871 (Interpreted frame) - org.apache.hadoop.mapred.ReduceTask.run(org.apache.hadoop.mapred.JobConf, org.apache.hadoop.mapred.TaskUmbilicalProtocol) @bci=470, line=423 (Interpreted frame) - org.apache.hadoop.mapred.Child$4.run() @bci=29, line=255 (Interpreted frame) - java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction, java.security.AccessControlContext) @bci=0 (Interpreted frame) - javax.security.auth.Subject.doAs(javax.security.auth.Subject, java.security.PrivilegedExceptionAction) @bci=42, line=396 (Interpreted frame) - org.apache.hadoop.security.UserGroupInformation.doAs(java.security.PrivilegedExceptionAction) @bci=14, line=1082 (Interpreted frame) - org.apache.hadoop.mapred.Child.main(java.lang.String[]) @bci=738, line=249 (Interpreted frame) This should never happen. A stuck task should never prevent other tasks from different jobs on the same node from committing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4101) nodemanager depends on /bin/bash
[
https://issues.apache.org/jira/browse/MAPREDUCE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247526#comment-13247526
]
Bikas Saha commented on MAPREDUCE-4101:
---
Could you check if the test failure is related to your changes?
Comments
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/TaskLog.java
@@ -379,7 +379,7 @@ public class TaskLog {
}
}
- private static final String bashCommand = bash;
+ private static final String bashCommand = sh;
Now you might want to rename this to shellCommand.
+++
b/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
@@ -169,7 +169,7 @@ public class DefaultContainerExecutor extends
ContainerExecutor {
ContainerExecutor.TASK_LAUNCH_SCRIPT_PERMISSION);
// Setup command to run
- String[] command = {bash, -c,
+ String[] command = {/bin/sh, -c,
Why the extra /bin/ ?
nodemanager depends on /bin/bash
Key: MAPREDUCE-4101
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: nodemanager
Affects Versions: 0.23.1, trunk
Environment: FreeBSD 8.2 / 64 bit
Reporter: Radim Kolar
Labels: patch
Attachments: bash-replace-by-sh.txt
Currently nodemanager depends on bash shell. It should be well documented for
system not having bash installed by default such as FreeBSD. Because only
basic functionality of bash is used, probably changing bash to /bin/sh would
work enough.
i found 2 cases:
1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in
writeLocalWrapperScript. (this needs bash in /bin)
2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798
INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor:
launchContainer: [bash, -c,
/tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_12/default_container_executor.sh]
this created script is also launched by bash - bash anywhere in path works -
in freebsd it is /usr/local/bin/bash
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4114) saveVersion.sh fails if build directory contains space
saveVersion.sh fails if build directory contains space --- Key: MAPREDUCE-4114 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4114 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.2 Environment: FreeBSD 8.2, 64bit Reporter: Radim Kolar if you rename build directory to something without space like /tmp/hadoop then it works [INFO] [INFO] [INFO] Building hadoop-yarn-common 0.23.3-SNAPSHOT [INFO] [INFO] [INFO] --- maven-antrun-plugin:1.6:run (create-testdirs) @ hadoop-yarn-common --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- maven-antrun-plugin:1.6:run (create-protobuf-generated-sources-directory) @ hadoop-yarn-common --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- exec-maven-plugin:1.2:exec (generate-sources) @ hadoop-yarn-common --- [INFO] [INFO] --- exec-maven-plugin:1.2:exec (generate-version) @ hadoop-yarn-common --- scripts/saveVersion.sh: cannot create /usr/local/jboss/.jenkins/jobs/Hadoop 0.23 branch/workspace/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/target/generated-sources/version/org/apache/hadoop/yarn/package-info.java: No such file or directory [JENKINS] Archiving /usr/local/jboss/.jenkins/jobs/Hadoop 0.23 branch/workspace/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/pom.xml to /usr/local/jboss/.jenkins/jobs/Hadoop 0.23 branch/modules/org.apache.hadoop$hadoop-yarn-common/builds/2012-04-05_19-44-16/archive/org.apache.hadoop/hadoop-yarn-common/0.23.3-SNAPSHOT/hadoop-yarn-common-0.23.3-SNAPSHOT.pom [INFO] [INFO] Reactor Summary: [INFO] [INFO] Apache Hadoop Main SUCCESS [5.124s] [INFO] Apache Hadoop Project POM . SUCCESS [1.692s] [INFO] Apache Hadoop Annotations . SUCCESS [1.672s] [INFO] Apache Hadoop Project Dist POM SUCCESS [1.823s] [INFO] Apache Hadoop Assemblies .. SUCCESS [0.796s] [INFO] Apache Hadoop Auth SUCCESS [2.456s] [INFO] Apache Hadoop Auth Examples ... SUCCESS [1.093s] [INFO] Apache Hadoop Common .. SUCCESS [23.648s] [INFO] Apache Hadoop Common Project .. SUCCESS [0.434s] [INFO] Apache Hadoop HDFS SUCCESS [22.124s] [INFO] Apache Hadoop HttpFS .. SUCCESS [3.251s] [INFO] Apache Hadoop HDFS Project SUCCESS [0.443s] [INFO] hadoop-yarn ... SUCCESS [1.175s] [INFO] hadoop-yarn-api ... SUCCESS [7.049s] [INFO] hadoop-yarn-common FAILURE [5.565s] [INFO] hadoop-yarn-server SKIPPED [INFO] hadoop-yarn-server-common . SKIPPED [INFO] hadoop-yarn-server-nodemanager SKIPPED [INFO] hadoop-yarn-server-web-proxy .. SKIPPED [INFO] hadoop-yarn-server-resourcemanager SKIPPED [INFO] hadoop-yarn-server-tests .. SKIPPED [INFO] hadoop-mapreduce-client ... SKIPPED [INFO] hadoop-mapreduce-client-core .. SKIPPED [INFO] hadoop-yarn-applications .. SKIPPED [INFO] hadoop-yarn-applications-distributedshell . SKIPPED [INFO] hadoop-yarn-site .. SKIPPED [INFO] hadoop-mapreduce-client-common SKIPPED [INFO] hadoop-mapreduce-client-shuffle ... SKIPPED [INFO] hadoop-mapreduce-client-app ... SKIPPED [INFO] hadoop-mapreduce-client-hs SKIPPED [INFO] hadoop-mapreduce-client-jobclient . SKIPPED [INFO] Apache Hadoop MapReduce Examples .. SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] Apache Hadoop MapReduce Streaming . SKIPPED [INFO] Apache Hadoop Distributed Copy SKIPPED [INFO] Apache Hadoop Archives SKIPPED [INFO] Apache Hadoop Rumen ... SKIPPED [INFO] Apache Hadoop Extras .. SKIPPED [INFO] Apache Hadoop Tools Dist .. SKIPPED [INFO] Apache Hadoop Tools ... SKIPPED [INFO] Apache Hadoop Distribution SKIPPED [INFO] Apache Hadoop Client .. SKIPPED
[jira] [Updated] (MAPREDUCE-4017) Add jobname to jobsummary log
[ https://issues.apache.org/jira/browse/MAPREDUCE-4017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-4017: - Component/s: jobtracker jobhistoryserver Target Version/s: 1.1.0, 0.23.3 Affects Version/s: 0.20.205.0 0.23.0 Assignee: Thomas Graves Add jobname to jobsummary log - Key: MAPREDUCE-4017 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4017 Project: Hadoop Map/Reduce Issue Type: Improvement Components: jobhistoryserver, jobtracker Affects Versions: 0.20.205.0, 0.23.0 Reporter: Koji Noguchi Assignee: Thomas Graves Priority: Trivial Attachments: MAPREDUCE-4017-v0.20-1.patch We occasionally use jobsummary from the JobTracker to collect users' slot usage on our clusters. It would be useful if the jobname was part of this jobsummary so that I don't need to join with other logs. Same jobsummary. 2012-03-15 16:05:55,919 INFO mapred.JobInProgress$JobSummary: jobId=job_201202160624_1089972,submitTime=1331827523632,launchTime=1331827528197,firstMapTaskLaunchTime=1331827536917,firstReduceTaskLaunchTime=1331827541251,firstJobSetupTaskLaunchTime=1331827528200,firstJobCleanupTaskLaunchTime=1331827551655,finishTime=1331827555919,numMaps=1,numSlotsPerMap=1,numReduces=1,numSlotsPerReduce=1,user=tortuga,queue=queue1,status=SUCCEEDED,mapSlotSeconds=13,reduceSlotsSeconds=10,clusterMapCapacity=,clusterReduceCapacity= I'd like to see jobName added to the end. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3621) TestDBJob and TestDataDrivenDBInputFormat ant tests fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated MAPREDUCE-3621: - Resolution: Fixed Fix Version/s: 0.23.3 Status: Resolved (was: Patch Available) +1. Thanks Ravi - and thanks for moving those tests over to mrv2 directories! I've committed this to trunk, branch-2, and branch-0.23 TestDBJob and TestDataDrivenDBInputFormat ant tests fail Key: MAPREDUCE-3621 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3621 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Ravi Prakash Fix For: 0.23.3 Attachments: MAPREDUCE-3621.patch, MAPREDUCE-3621.patch, MAPREDUCE-3621.patch The following mapred ant tests fail and have been failing for a very long time: [junit] Running org.apache.hadoop.mapreduce.lib.db.TestDBJob [junit] Running org.apache.hadoop.mapreduce.lib.db.TestDataDrivenDBInputFormat -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4049) plugin for generic shuffle service
[ https://issues.apache.org/jira/browse/MAPREDUCE-4049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247624#comment-13247624 ] Avner BenHanoch commented on MAPREDUCE-4049: please advise If there is anything else I need to do. thanks. plugin for generic shuffle service -- Key: MAPREDUCE-4049 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4049 Project: Hadoop Map/Reduce Issue Type: Improvement Components: performance, task, tasktracker Affects Versions: 0.23.1, 1.0.1 Reporter: Avner BenHanoch Labels: merge, plugin, rdma, shuffle Attachments: HADOOP-1.0.2.patch, HADOOP-1.0.x.patch, Hadoop Shuffle Consumer Plugin TLD.rtf, Hadoop Shuffle Provider Plugin TLD.rtf, MAPREDUCE-4049-branch-1.0.2.patch, mapred-site.xml, mapred.diff, src.tgz, test.diff Support generic shuffle service as set of two plugins: ShuffleProvider ShuffleConsumer. This will satisfy the following needs: # Better shuffle and merge performance. For example: we are working on shuffle plugin that performs shuffle over RDMA in fast networks (10gE, 40gE, or Infiniband) instead of using the current HTTP shuffle. Based on the fast RDMA shuffle, the plugin can also utilize a suitable merge approach during the intermediate merges. Hence, getting much better performance. # Satisfy MAPREDUCE-3060 - generic shuffle service for avoiding hidden dependency of NodeManager with a specific version of mapreduce shuffle (currently targeted to 0.24.0). References: # Hadoop Acceleration through Network Levitated Merging, by Prof. Weikuan Yu from Auburn University with others, [http://pasl.eng.auburn.edu/pubs/sc11-netlev.pdf] # I am attaching 2 documents with suggested Top Level Design for both plugins (currently, based on 1.0 branch) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3621) TestDBJob and TestDataDrivenDBInputFormat ant tests fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247626#comment-13247626 ] Hudson commented on MAPREDUCE-3621: --- Integrated in Hadoop-Common-trunk-Commit #2007 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2007/]) MAPREDUCE-3621. TestDBJob and TestDataDrivenDBInputFormat ant tests fail (Ravi Prakash via tgraves) (Revision 1310043) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1310043 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/db/TestDataDrivenDBInputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/DBCountPageView.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/test/java/org/apache/hadoop/mapreduce * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/test/java/org/apache/hadoop/mapreduce/lib * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/test/java/org/apache/hadoop/mapreduce/lib/db * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/test/java/org/apache/hadoop/mapreduce/lib/db/TestDBJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/db/TestDBJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/db/TestDataDrivenDBInputFormat.java TestDBJob and TestDataDrivenDBInputFormat ant tests fail Key: MAPREDUCE-3621 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3621 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Ravi Prakash Fix For: 0.23.3 Attachments: MAPREDUCE-3621.patch, MAPREDUCE-3621.patch, MAPREDUCE-3621.patch The following mapred ant tests fail and have been failing for a very long time: [junit] Running org.apache.hadoop.mapreduce.lib.db.TestDBJob [junit] Running org.apache.hadoop.mapreduce.lib.db.TestDataDrivenDBInputFormat -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3621) TestDBJob and TestDataDrivenDBInputFormat ant tests fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247635#comment-13247635 ] Hudson commented on MAPREDUCE-3621: --- Integrated in Hadoop-Hdfs-trunk-Commit #2082 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2082/]) MAPREDUCE-3621. TestDBJob and TestDataDrivenDBInputFormat ant tests fail (Ravi Prakash via tgraves) (Revision 1310043) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1310043 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/db/TestDataDrivenDBInputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/DBCountPageView.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/test/java/org/apache/hadoop/mapreduce * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/test/java/org/apache/hadoop/mapreduce/lib * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/test/java/org/apache/hadoop/mapreduce/lib/db * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/test/java/org/apache/hadoop/mapreduce/lib/db/TestDBJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/db/TestDBJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/db/TestDataDrivenDBInputFormat.java TestDBJob and TestDataDrivenDBInputFormat ant tests fail Key: MAPREDUCE-3621 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3621 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Ravi Prakash Fix For: 0.23.3 Attachments: MAPREDUCE-3621.patch, MAPREDUCE-3621.patch, MAPREDUCE-3621.patch The following mapred ant tests fail and have been failing for a very long time: [junit] Running org.apache.hadoop.mapreduce.lib.db.TestDBJob [junit] Running org.apache.hadoop.mapreduce.lib.db.TestDataDrivenDBInputFormat -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4073) CS assigns multiple off-switch containers when using multi-level-queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-4073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans updated MAPREDUCE-4073: --- Resolution: Fixed Fix Version/s: 2.0.0 0.23.3 Status: Resolved (was: Patch Available) Thanks Sid, I just put this into branch-0.23, branch-2, and trunk CS assigns multiple off-switch containers when using multi-level-queues --- Key: MAPREDUCE-4073 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4073 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, scheduler Affects Versions: 0.23.1 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Critical Fix For: 0.23.3, 2.0.0 Attachments: MAPREDUCE-4073.patch, MR4073.txt, MR4073.txt CS is supposed to be allocating a single off-switch container per node heartbeat (MAPREDUCE-3641). This works for queues directly under root, but not in the case of multi-level queues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4073) CS assigns multiple off-switch containers when using multi-level-queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-4073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247642#comment-13247642 ] Hudson commented on MAPREDUCE-4073: --- Integrated in Hadoop-Hdfs-trunk-Commit #2083 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2083/]) MAPREDUCE-4073. CS assigns multiple off-switch containers when using multi-level-queues (Siddharth Seth via bobby) (Revision 1310056) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1310056 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSAssignment.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java CS assigns multiple off-switch containers when using multi-level-queues --- Key: MAPREDUCE-4073 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4073 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, scheduler Affects Versions: 0.23.1 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Critical Fix For: 0.23.3, 2.0.0 Attachments: MAPREDUCE-4073.patch, MR4073.txt, MR4073.txt CS is supposed to be allocating a single off-switch container per node heartbeat (MAPREDUCE-3641). This works for queues directly under root, but not in the case of multi-level queues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4073) CS assigns multiple off-switch containers when using multi-level-queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-4073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247643#comment-13247643 ] Hudson commented on MAPREDUCE-4073: --- Integrated in Hadoop-Common-trunk-Commit #2008 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2008/]) MAPREDUCE-4073. CS assigns multiple off-switch containers when using multi-level-queues (Siddharth Seth via bobby) (Revision 1310056) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1310056 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSAssignment.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java CS assigns multiple off-switch containers when using multi-level-queues --- Key: MAPREDUCE-4073 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4073 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, scheduler Affects Versions: 0.23.1 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Critical Fix For: 0.23.3, 2.0.0 Attachments: MAPREDUCE-4073.patch, MR4073.txt, MR4073.txt CS is supposed to be allocating a single off-switch container per node heartbeat (MAPREDUCE-3641). This works for queues directly under root, but not in the case of multi-level queues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3621) TestDBJob and TestDataDrivenDBInputFormat ant tests fail
[ https://issues.apache.org/jira/browse/MAPREDUCE-3621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247645#comment-13247645 ] Hudson commented on MAPREDUCE-3621: --- Integrated in Hadoop-Mapreduce-trunk-Commit #2020 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2020/]) MAPREDUCE-3621. TestDBJob and TestDataDrivenDBInputFormat ant tests fail (Ravi Prakash via tgraves) (Revision 1310043) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1310043 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/db/TestDataDrivenDBInputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/DBCountPageView.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/test/java/org/apache/hadoop/mapreduce * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/test/java/org/apache/hadoop/mapreduce/lib * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/test/java/org/apache/hadoop/mapreduce/lib/db * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/test/java/org/apache/hadoop/mapreduce/lib/db/TestDBJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/db/TestDBJob.java * /hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/lib/db/TestDataDrivenDBInputFormat.java TestDBJob and TestDataDrivenDBInputFormat ant tests fail Key: MAPREDUCE-3621 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3621 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2 Affects Versions: 0.23.0 Reporter: Thomas Graves Assignee: Ravi Prakash Fix For: 0.23.3 Attachments: MAPREDUCE-3621.patch, MAPREDUCE-3621.patch, MAPREDUCE-3621.patch The following mapred ant tests fail and have been failing for a very long time: [junit] Running org.apache.hadoop.mapreduce.lib.db.TestDBJob [junit] Running org.apache.hadoop.mapreduce.lib.db.TestDataDrivenDBInputFormat -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4101) nodemanager depends on /bin/bash
[ https://issues.apache.org/jira/browse/MAPREDUCE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247655#comment-13247655 ] Radim Kolar commented on MAPREDUCE-4101: /bin/sh is always better then plain sh for security purposes nodemanager depends on /bin/bash Key: MAPREDUCE-4101 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101 Project: Hadoop Map/Reduce Issue Type: Bug Components: nodemanager Affects Versions: 0.23.1, trunk Environment: FreeBSD 8.2 / 64 bit Reporter: Radim Kolar Labels: patch Attachments: bash-replace-by-sh.txt Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough. i found 2 cases: 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin) 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_12/default_container_executor.sh] this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4115) hadoop-project-dist/pom.xml is invalid
hadoop-project-dist/pom.xml is invalid -- Key: MAPREDUCE-4115 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4115 Project: Hadoop Map/Reduce Issue Type: Bug Components: build Affects Versions: 0.23.1 Reporter: Radim Kolar Mentioned POM is invalid. It fails XML validation and can not be deployed into repository with validating repository manager, such as Artifactory. Problematic are inside antrun-plugin configuration. It should be gt; Line 342 and 379 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4101) nodemanager depends on /bin/bash
[ https://issues.apache.org/jira/browse/MAPREDUCE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247672#comment-13247672 ] Bikas Saha commented on MAPREDUCE-4101: --- Security is a beast by itself :P Leaving it as sh lets users setup their installations as long as required stuff is in the PATH. Making it /bin/sh forces installation node to have /bin etc. Some of these things may be difficult, say on Windows. nodemanager depends on /bin/bash Key: MAPREDUCE-4101 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101 Project: Hadoop Map/Reduce Issue Type: Bug Components: nodemanager Affects Versions: 0.23.1, trunk Environment: FreeBSD 8.2 / 64 bit Reporter: Radim Kolar Labels: patch Attachments: bash-replace-by-sh.txt Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough. i found 2 cases: 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin) 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_12/default_container_executor.sh] this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4071) NPE while executing MRAppMaster shutdown hook
[
https://issues.apache.org/jira/browse/MAPREDUCE-4071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Joseph Evans updated MAPREDUCE-4071:
---
Status: Open (was: Patch Available)
NPE while executing MRAppMaster shutdown hook
-
Key: MAPREDUCE-4071
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4071
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mr-am, mrv2
Affects Versions: 0.23.3, 2.0.0
Reporter: Bhallamudi Venkata Siva Kamesh
Attachments: MAPREDUCE-4071-1.patch, MAPREDUCE-4071.patch
While running the shutdown hook of MRAppMaster, hit NPE
{noformat}
Exception in thread Thread-1 java.lang.NullPointerException
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.setSignalled(MRAppMaster.java:668)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$MRAppMasterShutdownHook.run(MRAppMaster.java:1004)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4073) CS assigns multiple off-switch containers when using multi-level-queues
[ https://issues.apache.org/jira/browse/MAPREDUCE-4073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247687#comment-13247687 ] Hudson commented on MAPREDUCE-4073: --- Integrated in Hadoop-Mapreduce-trunk-Commit #2021 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2021/]) MAPREDUCE-4073. CS assigns multiple off-switch containers when using multi-level-queues (Siddharth Seth via bobby) (Revision 1310056) Result = ABORTED bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1310056 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSAssignment.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java CS assigns multiple off-switch containers when using multi-level-queues --- Key: MAPREDUCE-4073 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4073 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv2, scheduler Affects Versions: 0.23.1 Reporter: Siddharth Seth Assignee: Siddharth Seth Priority: Critical Fix For: 0.23.3, 2.0.0 Attachments: MAPREDUCE-4073.patch, MR4073.txt, MR4073.txt CS is supposed to be allocating a single off-switch container per node heartbeat (MAPREDUCE-3641). This works for queues directly under root, but not in the case of multi-level queues. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4101) nodemanager depends on /bin/bash
[ https://issues.apache.org/jira/browse/MAPREDUCE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247705#comment-13247705 ] Radim Kolar commented on MAPREDUCE-4101: Nodemanager is creating script with #!/bin/bash, so every node has to have /bin nodemanager depends on /bin/bash Key: MAPREDUCE-4101 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101 Project: Hadoop Map/Reduce Issue Type: Bug Components: nodemanager Affects Versions: 0.23.1, trunk Environment: FreeBSD 8.2 / 64 bit Reporter: Radim Kolar Labels: patch Attachments: bash-replace-by-sh.txt Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough. i found 2 cases: 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin) 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_12/default_container_executor.sh] this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4101) nodemanager depends on /bin/bash
[ https://issues.apache.org/jira/browse/MAPREDUCE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247739#comment-13247739 ] Bikas Saha commented on MAPREDUCE-4101: --- so we should work on fixing it and not adding new ones, right? nodemanager depends on /bin/bash Key: MAPREDUCE-4101 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101 Project: Hadoop Map/Reduce Issue Type: Bug Components: nodemanager Affects Versions: 0.23.1, trunk Environment: FreeBSD 8.2 / 64 bit Reporter: Radim Kolar Labels: patch Attachments: bash-replace-by-sh.txt Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough. i found 2 cases: 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin) 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_12/default_container_executor.sh] this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4101) nodemanager depends on /bin/bash
[ https://issues.apache.org/jira/browse/MAPREDUCE-4101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247903#comment-13247903 ] Radim Kolar commented on MAPREDUCE-4101: I am not sure if shell script can be launched without #! /abs path. nodemanager depends on /bin/bash Key: MAPREDUCE-4101 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4101 Project: Hadoop Map/Reduce Issue Type: Bug Components: nodemanager Affects Versions: 0.23.1, trunk Environment: FreeBSD 8.2 / 64 bit Reporter: Radim Kolar Labels: patch Attachments: bash-replace-by-sh.txt Currently nodemanager depends on bash shell. It should be well documented for system not having bash installed by default such as FreeBSD. Because only basic functionality of bash is used, probably changing bash to /bin/sh would work enough. i found 2 cases: 1. DefaultContainerExecutor.java creates file with /bin/bash hardcoded in writeLocalWrapperScript. (this needs bash in /bin) 2. yarn-hduser-nodemanager-ponto.amerinoc.com.log:2012-04-03 19:50:10,798 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [bash, -c, /tmp/nm-local-dir/usercache/hduser/appcache/application_1333474251533_0002/container_1333474251533_0002_01_12/default_container_executor.sh] this created script is also launched by bash - bash anywhere in path works - in freebsd it is /usr/local/bin/bash -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3921) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy
[ https://issues.apache.org/jira/browse/MAPREDUCE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-3921: -- Status: Open (was: Patch Available) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy Key: MAPREDUCE-3921 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3921 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Bikas Saha Fix For: 0.23.2 Attachments: MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3921) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy
[ https://issues.apache.org/jira/browse/MAPREDUCE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-3921: -- Attachment: MAPREDUCE-3921.patch Some cleanup and creating new diff wrt trunk since the previous one was for 0.23 MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy Key: MAPREDUCE-3921 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3921 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Bikas Saha Fix For: 0.23.2 Attachments: MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3921) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy
[ https://issues.apache.org/jira/browse/MAPREDUCE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated MAPREDUCE-3921: -- Status: Patch Available (was: Open) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy Key: MAPREDUCE-3921 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3921 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Bikas Saha Fix For: 0.23.2 Attachments: MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3921) MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy
[ https://issues.apache.org/jira/browse/MAPREDUCE-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247943#comment-13247943 ] Hadoop QA commented on MAPREDUCE-3921: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521616/MAPREDUCE-3921.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 509 javac compiler warnings (more than the trunk's current 507 warnings). +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2165//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2165//console This message is automatically generated. MR AM should act on the nodes liveliness information when nodes go up/down/unhealthy Key: MAPREDUCE-3921 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3921 Project: Hadoop Map/Reduce Issue Type: Bug Components: mr-am, mrv2 Affects Versions: 0.23.0 Reporter: Vinod Kumar Vavilapalli Assignee: Bikas Saha Fix For: 0.23.2 Attachments: MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921-branch-0.23.patch, MAPREDUCE-3921.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
