[ https://issues.apache.org/jira/browse/AMBARI-16131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15260843#comment-15260843 ]
Hudson commented on AMBARI-16131: --------------------------------- SUCCESS: Integrated in Ambari-branch-2.2 #675 (See [https://builds.apache.org/job/Ambari-branch-2.2/675/]) AMBARI-16131 - Prevent Views From Causing a Loss of Service For Ambari (jhurley: [http://git-wip-us.apache.org/repos/asf?p=ambari.git&a=commit&h=84e1d84d1f72ca9a4799f5c592a59479c6f7ab42]) * ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java * ambari-server/src/main/java/org/apache/ambari/server/view/ViewThrottleFilter.java * ambari-server/src/main/java/org/apache/ambari/server/utils/VersionUtils.java * ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariServer.java * ambari-server/src/test/java/org/apache/ambari/server/view/ViewThrottleFilterTest.java * ambari-server/src/main/java/org/apache/ambari/server/view/ViewRegistry.java > Prevent Views From Causing a Loss of Service For Ambari > ------------------------------------------------------- > > Key: AMBARI-16131 > URL: https://issues.apache.org/jira/browse/AMBARI-16131 > Project: Ambari > Issue Type: Bug > Components: ambari-views > Affects Versions: 2.0.0 > Reporter: Jonathan Hurley > Assignee: Jonathan Hurley > Priority: Critical > Fix For: 2.4.0, 2.2-next > > Attachments: AMBARI-16131.patch > > > The underlying problem is that views are accessed off of the REST endpoint > ({{/api/v1/views}}). This means that the Ambari REST API connector is going > to handle the request from its own threadpool. There is no way to configure > Jetty to use a different threadpool for the same connector. As a result, if a > request to load a view holds the Jetty thread hostage, eventually we will see > thread starvation and loss of service. > An example of this situation is a view which makes an innocent request to a > remote resource. If the view's request has a timeout of 60 seconds, then the > Jetty thread is going to be held for that amount of time. With concurrent > users and multiple instances of that view deployed, the Jetty threadpool can > becomes exhausted quickly. > Although there are more graceful ways of handling this situation, they mostly > involve substantial re-architecture and design: > - The use of a new connector and threadpool would require binding to another > port for view requests. This will cause problems with "local" views and their > assumption that if they run on the Ambari server they can share the same > session. > - The use of a > [Continuation|https://wiki.eclipse.org/Jetty/Feature/Continuations] in Jetty > which can suspend the incoming request. We would need the ability for views > to signal that they have completed their work in order to proceed with the > suspended request. > A quicker and far less invasive fix would be to create a filter which > intercepts requests for views. It will determine how many executing view > requests exist and decide if it will allow the new request through. For > example, if configured to allow a maximum of 10 concurrent view requests, > then the 11th request would be denied with an {{HTTP 503 - Service > Unavailable}}. Although the thread is temporarily used while the filter is > processing, it's quickly returned to the Jetty pool when it's determined > there are too many other running view requests. -- This message was sent by Atlassian JIRA (v6.3.4#6332)