[
https://issues.apache.org/jira/browse/MAPREDUCE-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13220441#comment-13220441
]
Siddharth Seth commented on MAPREDUCE-3944:
-------------------------------------------
bq. I think it would be better for the webservices to return the partial job
instead of just the job id. From my understanding, if you are returning the ids
you have all the partial job information also. That will be equivalent to the
job history web page and could atleast give a user some useful information. If
they need the complete then can do the next query of the specific job.
That works well. Instead of adding another webservice to return jobIds, the
current one returns a smaller set of fields for now, and can go back to
returning what it does rightnow once CompleteJobStatusStore is implemented or
there's a more efficient way of getting additional job info.
Bobby, the stacktrace you had posted earlier - that's from multiple parallel
calls right ? Are you planning some kind of rate limiting as well, or
restricting the number of worker threads based on the source of the request -
UI / webservice / RPC.
> JobHistory web services are slower then the UI and can easly overload the JH
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-3944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3944
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1, 0.23.2
> Reporter: Robert Joseph Evans
> Assignee: Robert Joseph Evans
> Priority: Blocker
>
> When our first customer started using the Job History web services today the
> History Server ground to a halt. We found 250 Jetty threads stuck on the
> following stack trace.
> {noformat}
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.getJob(JobHistory.java:898)
> - waiting to lock <0x00002aaab364ba60> (a
> org.apache.hadoop.mapreduce.v2.hs.JobHistory)
> at
> org.apache.hadoop.mapreduce.v2.hs.webapp.HsWebServices.getJobs(HsWebServices.java:188)
> {noformat}
> HsWebServices.java:188 corresponds to the /mapreduce/jobs service.
> Looking at the code there are a number of optimizations that need to be done
> to improve its performance.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
