[ https://issues.apache.org/jira/browse/AMBARI-16131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hurley updated AMBARI-16131: ------------------------------------- Fix Version/s: 2.2-next > Prevent Views From Causing a Loss of Service For Ambari > ------------------------------------------------------- > > Key: AMBARI-16131 > URL: https://issues.apache.org/jira/browse/AMBARI-16131 > Project: Ambari > Issue Type: Bug > Components: ambari-views > Affects Versions: 2.0.0 > Reporter: Jonathan Hurley > Assignee: Jonathan Hurley > Priority: Critical > Fix For: 2.4.0, 2.2-next > > Attachments: AMBARI-16131.patch > > > The underlying problem is that views are accessed off of the REST endpoint > ({{/api/v1/views}}). This means that the Ambari REST API connector is going > to handle the request from its own threadpool. There is no way to configure > Jetty to use a different threadpool for the same connector. As a result, if a > request to load a view holds the Jetty thread hostage, eventually we will see > thread starvation and loss of service. > An example of this situation is a view which makes an innocent request to a > remote resource. If the view's request has a timeout of 60 seconds, then the > Jetty thread is going to be held for that amount of time. With concurrent > users and multiple instances of that view deployed, the Jetty threadpool can > becomes exhausted quickly. > Although there are more graceful ways of handling this situation, they mostly > involve substantial re-architecture and design: > - The use of a new connector and threadpool would require binding to another > port for view requests. This will cause problems with "local" views and their > assumption that if they run on the Ambari server they can share the same > session. > - The use of a > [Continuation|https://wiki.eclipse.org/Jetty/Feature/Continuations] in Jetty > which can suspend the incoming request. We would need the ability for views > to signal that they have completed their work in order to proceed with the > suspended request. > A quicker and far less invasive fix would be to create a filter which > intercepts requests for views. It will determine how many executing view > requests exist and decide if it will allow the new request through. For > example, if configured to allow a maximum of 10 concurrent view requests, > then the 11th request would be denied with an {{HTTP 503 - Service > Unavailable}}. Although the thread is temporarily used while the filter is > processing, it's quickly returned to the Jetty pool when it's determined > there are too many other running view requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)