[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13224312#comment-13224312
]
Hudson commented on MAPREDUCE-3944:
---
Integrated in Hadoop-Mapreduce-trunk #1012 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1012/])
MAPREDUCE-3944. Change the history jobs/ webservice to return partial job
info for a significant performance improvement. (Contributed by Robert Joseph
Evans) (Revision 1297672)
Result = FAILURE
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297672
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/JobInfo.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/MockHistoryJobs.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobs.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobsQuery.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/VerifyJobsUtils.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Fix For: 0.23.2
>
> Attachments: MR-3944.txt, MR-3944.txt
>
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13224291#comment-13224291
]
Hudson commented on MAPREDUCE-3944:
---
Integrated in Hadoop-Mapreduce-0.23-Build #218 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/218/])
merge MAPREDUCE-3944 from trunk (Revision 1297673)
Result = FAILURE
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297673
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/JobInfo.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/MockHistoryJobs.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobs.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobsQuery.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/VerifyJobsUtils.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Fix For: 0.23.2
>
> Attachments: MR-3944.txt, MR-3944.txt
>
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13224273#comment-13224273
]
Hudson commented on MAPREDUCE-3944:
---
Integrated in Hadoop-Hdfs-trunk #977 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/977/])
MAPREDUCE-3944. Change the history jobs/ webservice to return partial job
info for a significant performance improvement. (Contributed by Robert Joseph
Evans) (Revision 1297672)
Result = FAILURE
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297672
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/JobInfo.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/MockHistoryJobs.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobs.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobsQuery.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/VerifyJobsUtils.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Fix For: 0.23.2
>
> Attachments: MR-3944.txt, MR-3944.txt
>
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13224250#comment-13224250
]
Hudson commented on MAPREDUCE-3944:
---
Integrated in Hadoop-Hdfs-0.23-Build #190 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/190/])
merge MAPREDUCE-3944 from trunk (Revision 1297673)
Result = SUCCESS
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297673
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/JobInfo.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/MockHistoryJobs.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobs.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobsQuery.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/VerifyJobsUtils.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Fix For: 0.23.2
>
> Attachments: MR-3944.txt, MR-3944.txt
>
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223640#comment-13223640
]
Hudson commented on MAPREDUCE-3944:
---
Integrated in Hadoop-Mapreduce-0.23-Commit #648 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/648/])
merge MAPREDUCE-3944 from trunk (Revision 1297673)
Result = SUCCESS
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297673
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/JobInfo.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/MockHistoryJobs.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobs.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobsQuery.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/VerifyJobsUtils.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Fix For: 0.23.2
>
> Attachments: MR-3944.txt, MR-3944.txt
>
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223639#comment-13223639
]
Hudson commented on MAPREDUCE-3944:
---
Integrated in Hadoop-Mapreduce-trunk-Commit #1850 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1850/])
MAPREDUCE-3944. Change the history jobs/ webservice to return partial job
info for a significant performance improvement. (Contributed by Robert Joseph
Evans) (Revision 1297672)
Result = SUCCESS
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297672
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/JobInfo.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/MockHistoryJobs.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobs.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobsQuery.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/VerifyJobsUtils.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Fix For: 0.23.2
>
> Attachments: MR-3944.txt, MR-3944.txt
>
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223630#comment-13223630
]
Hudson commented on MAPREDUCE-3944:
---
Integrated in Hadoop-Hdfs-0.23-Commit #637 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/637/])
merge MAPREDUCE-3944 from trunk (Revision 1297673)
Result = SUCCESS
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297673
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/JobInfo.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/MockHistoryJobs.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobs.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobsQuery.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/VerifyJobsUtils.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Fix For: 0.23.2
>
> Attachments: MR-3944.txt, MR-3944.txt
>
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223626#comment-13223626
]
Hudson commented on MAPREDUCE-3944:
---
Integrated in Hadoop-Common-trunk-Commit #1843 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1843/])
MAPREDUCE-3944. Change the history jobs/ webservice to return partial job
info for a significant performance improvement. (Contributed by Robert Joseph
Evans) (Revision 1297672)
Result = SUCCESS
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297672
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/JobInfo.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/MockHistoryJobs.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobs.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobsQuery.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/VerifyJobsUtils.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Fix For: 0.23.2
>
> Attachments: MR-3944.txt, MR-3944.txt
>
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223625#comment-13223625
]
Hudson commented on MAPREDUCE-3944:
---
Integrated in Hadoop-Common-0.23-Commit #647 (See
[https://builds.apache.org/job/Hadoop-Common-0.23-Commit/647/])
merge MAPREDUCE-3944 from trunk (Revision 1297673)
Result = SUCCESS
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297673
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/JobInfo.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/MockHistoryJobs.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobs.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobsQuery.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/VerifyJobsUtils.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Fix For: 0.23.2
>
> Attachments: MR-3944.txt, MR-3944.txt
>
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223623#comment-13223623
]
Hudson commented on MAPREDUCE-3944:
---
Integrated in Hadoop-Hdfs-trunk-Commit #1917 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1917/])
MAPREDUCE-3944. Change the history jobs/ webservice to return partial job
info for a significant performance improvement. (Contributed by Robert Joseph
Evans) (Revision 1297672)
Result = SUCCESS
sseth : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1297672
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/HsWebServices.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/webapp/dao/JobInfo.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/MockHistoryJobs.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobs.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/TestHsWebServicesJobsQuery.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/java/org/apache/hadoop/mapreduce/v2/hs/webapp/VerifyJobsUtils.java
*
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/HistoryServerRest.apt.vm
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Fix For: 0.23.2
>
> Attachments: MR-3944.txt, MR-3944.txt
>
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223596#comment-13223596
]
Siddharth Seth commented on MAPREDUCE-3944:
---
Patch looks good. +1. Thanks Bobby
PartialJob.getTotalMaps/getTotalReduces isn't correct - which affects the jobs
webservice as well as the history UI. Creating a separate jira for that.
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Attachments: MR-3944.txt, MR-3944.txt
>
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223561#comment-13223561
]
Robert Joseph Evans commented on MAPREDUCE-3944:
The failed tests are unrelated and have been failing for several other people
already. MAPREDUCE-3729 was reopened to address them.
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Attachments: MR-3944.txt, MR-3944.txt
>
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13223508#comment-13223508
]
Hadoop QA commented on MAPREDUCE-3944:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12517259/MR-3944.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 11 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.mapred.TestJobClientGetJob
org.apache.hadoop.mapred.TestMRWithDistributedCache
org.apache.hadoop.mapred.TestLocalModeWithNewApis
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2012//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2012//console
This message is automatically generated.
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Attachments: MR-3944.txt, MR-3944.txt
>
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222644#comment-13222644
]
Siddharth Seth commented on MAPREDUCE-3944:
---
Bobby, looks like you missed including MockHistoryJobs in the patch.
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Attachments: MR-3944.txt
>
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222544#comment-13222544
]
Hadoop QA commented on MAPREDUCE-3944:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12517109/MR-3944.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The patch appears to cause tar ant target to fail.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed the unit tests build
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2001//testReport/
Console output:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2001//console
This message is automatically generated.
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Attachments: MR-3944.txt
>
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222535#comment-13222535
]
Robert Joseph Evans commented on MAPREDUCE-3944:
I filed MAPREDUCE-3971 for limiting the number of times that the web services
returns, and MAPREDUCE-3972 to fix the locking in the server.
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
> Attachments: MR-3944.txt
>
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222518#comment-13222518
]
Robert Joseph Evans commented on MAPREDUCE-3944:
I forgot to address Vinod's question about where the LRU is happening.
loadedJobCache, which holds a mapping between JobId and the CompletedJob is an
instance of LinkedHashMap which I believe removes the least recently used entry
when expungeEldest returns true. I could be wrong about that, it may just
remove the entry that has been in the list the longest.
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222445#comment-13222445
]
Robert Joseph Evans commented on MAPREDUCE-3944:
Yes, I have been looking at the code, and there are a lot of problems with
locking in the code, but it is a very large file, 1100 lines, and I am not
going to have time to really fix all of the locking issues by 0.23.2. So I
will file a follow on JIRA for that work.
I should have a patch shortly that addresses most of the issues by using the
partial job for the web service. After that in a separate JIRA we can look at
doing things with cursor positions and the like.
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222108#comment-13222108
]
Amol Kekre commented on MAPREDUCE-3944:
---
Vinod, we have MAPREDUCE-3901. What is the eta on MAPREDUCE-3966?
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13221752#comment-13221752
]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3944:
One question: Do you have MAPREDUCE-3901 part of your install? IIUC, that patch
will alleviate the problem A LOT.
Course of action IMO
(1) If the issue still exists after MAPREDUCE-3901, I agree that first next
thing to do is to use PartialJobs instead of completedJobs.
(2) MAPREDUCE-3966 also should alleviate the issue, once we increase the
cache-size for jobs without tasks loaded.
(3) To avoid the too-many-jobs-to-return case, the correct solution is a
simple cursor with hard limits: The user can for example ask jobs from
startTimeA to startTimeB and pass a running cursor-position. We can do this in
a separate ticket.
@Robert
bq. If we are using LRU to remove entries the current code will loop through
all entries.
Where is this?
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220478#comment-13220478
]
Robert Joseph Evans commented on MAPREDUCE-3944:
I have not thought about rate limiting at this point. It would be difficult to
split up the web service from the UI. But RPC thread count and I think the web
server thread count are already configurable. I could go beyond that but I am
hoping it is not necessary.
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220441#comment-13220441
]
Siddharth Seth commented on MAPREDUCE-3944:
---
bq. I think it would be better for the webservices to return the partial job
instead of just the job id. From my understanding, if you are returning the ids
you have all the partial job information also. That will be equivalent to the
job history web page and could atleast give a user some useful information. If
they need the complete then can do the next query of the specific job.
That works well. Instead of adding another webservice to return jobIds, the
current one returns a smaller set of fields for now, and can go back to
returning what it does rightnow once CompleteJobStatusStore is implemented or
there's a more efficient way of getting additional job info.
Bobby, the stacktrace you had posted earlier - that's from multiple parallel
calls right ? Are you planning some kind of rate limiting as well, or
restricting the number of worker threads based on the source of the request -
UI / webservice / RPC.
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220378#comment-13220378
]
Robert Joseph Evans commented on MAPREDUCE-3944:
I agree with Tom on this. I would also like to be able to limit the total
number of entries that can be returned, but we have the issue with ordering
here like Sid said. I am going to try to set up a stress test for this so I
can try out several fixes to see what happens. There are really two different
goals to this JIRA. The most important one is that even if someone is hitting
the web service a lot, the history server should still be usable. The second
part is to optimize the web service calls so that they reduce the load on the
system. I think that the locking is the primary thing for fixing the first
part, and I will put in the filtering changes too. Once I have those working I
will do some profiling and see what else looks like hot spots.
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220361#comment-13220361
]
Thomas Graves commented on MAPREDUCE-3944:
--
I think it would be better for the webservices to return the partial job
instead of just the job id. From my understanding, if you are returning the ids
you have all the partial job information also. That will be equivalent to the
job history web page and could atleast give a user some useful information. If
they need the complete then can do the next query of the specific job.
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220342#comment-13220342
]
Siddharth Seth commented on MAPREDUCE-3944:
---
bq. There are two things that I can think of that can help make this faster.
The first one is to move as much filtering for the web service up to use
PartialJob where possible. This will allow us to not load as many full jobs.
The next one is that we need to fix the locking on getJob, so that we can be
loading more then one job at a time. we need to some how lock on the JobID and
not on the HistoryServer.
+1 for these two changes. JobHistory used to have a method which did filtering
on some parameters. That was removed a while ago - since it was never used. You
may want to pull out the history for JobHistory.java.
Considering the webservice ends up building CompletedJob objects for each job
that it has to return - it can grow much bigger than the configured
CompletedJobCache size and can cause the JobHistory server to go OOM.
MAPREDUCE-3755 will help towards avoiding this, and also making this webservice
call faster.
For now, I'd propose having a hard limit on the number of entries this
webservice can return.
Also, I'd prefer having a webservice to return jobIds (based on the specified
filters), instead of returning completed job info. The jobs/{jobId} webservice
can then be used to pull job info for each job.
The order of results returned can be problematic as well. The webservice
doesn't have an order parameter - and ends up depending on whatever order the
history server returns, which can change in the future.
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3944) JobHistory web services are slower then the UI and can easly overload the JH
[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13219627#comment-13219627
]
Robert Joseph Evans commented on MAPREDUCE-3944:
There are two things that I can think of that can help make this faster. The
first one is to move as much filtering for the web service up to use PartialJob
where possible. This will allow us to not load as many full jobs. The next
one is that we need to fix the locking on getJob, so that we can be loading
more then one job at a time. we need to some how lock on the JobID and not on
the HistoryServer.
We probably also want to investigate how we are doing caching. If we are using
LRU to remove entries the current code will loop through all entries. This is
the worst case for LRU possible as if the cache is small it will force us load
all entries unless we are lucky and the ones at the start of the list are in
the cache. It would be preferable to have something that would allow us to set
what we are doing and expunge them randomly instead of LRU in this one case.
We probably also want to warn people about bad configuration settings when
using the web service.
> JobHistory web services are slower then the UI and can easly overload the JH
>
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
>Affects Versions: 0.23.1, 0.23.2
>Reporter: Robert Joseph Evans
>Assignee: Robert Joseph Evans
>Priority: Blocker
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
>java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x2aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
