Re: [openstack-dev] [Neutron] API list operations are not fast as they could because they're dumb
On 17 April 2014 04:02, Aaron Rosen wrote: > Hi, > > Comments inline: > > > On Tue, Apr 8, 2014 at 3:16 PM, Salvatore Orlando wrote: > >> I have been recently investigating reports of slowness for list responses >> in the Neutron API. >> This was first reported in [1], and then recently was observed with both >> the ML2 and the NSX plugins. >> The root cause of this issues is that a policy engine check is performed >> for every attribute of every resource returned in a response. >> When tenant grow to a lot of ports, or when the API is executed with >> admin credentials without filters, this might become a non-negligible scale >> issue. >> This issue is mostly due to three factors: >> 1) A log statement printing a line in the log for every attribute for >> which no policy criterion is defined; this has been treated with [2] >> 2) The fact that for every check neutron currently checks whether cached >> policy rules are still valid [3] >> 3) The fact that anyway Neutron perform really a lot of policy checks >> whether it should not >> >> Despite the improvements [2] and [3] (mostly [2]), neutron for a list >> operation still spends for post-plugin operations (ie: policy checks) abotu >> 50% of the time it spends in the plugin. >> Solving this problem is not difficult, but it might require changes which >> are worth of a discussion on the mailing list. >> Up to the Havana release policy checks were performed in the plugin; this >> basically made responses dependent on plugin implementation and was >> terrible for API compatibility and portability; we took care of that with >> [4], which moved all policy checks to the API layer. However for one fix >> that we fixed, another thing was broken (*) >> >> The API layer for list responses puts every item through policy checks to >> see which should not be visible to the user at all, which is fine. >> However it also puts every attribute through a policy check to exclude >> those which should not be visible to the user, such as provider attributes >> for regular users. >> Doing this for every resource might make sense if an attribute should be >> visible or not according to the data in the resource itself. >> For instance a policy that shows port binding attributes could be defined >> for all the ports whose name is "ernest". >> This might appear as great flexibility, but does it make any sense at all? >> Does it make sense that an API list operation return a set of attributes >> for some items and a different one for others? >> I think not. >> >> For this reason I am thinking we should what technically is a simple >> change: use policy cghecks determine the list of attributes to show only >> once for list response, and then re-use that list for the whole response. >> The limitation here is that we should not have 'attribute-level' policies >> (**) which rely on the resource value. >> I think this limitation is fair. If you like the approach I have some >> code here: http://paste.openstack.org/show/75371/ >> > > I think this makes sense. In theory the first element of the list should > have all the same columns as the next set of elements so inspecting the > first one should be fine. > Correct. It would be important however to not have attribute-level checks which depend on resource data. I reckon this assumption is valid. > >> And this leads me to the second part of the discussion I'd like to start. >> The policy engine currently allows me to start a neutron server where, >> for instance, port binding are visible by admins only, and another neutron >> server where any user can see them. >> This kind of makes the API not really portable, as people programming >> against the neutron API might encounter unexpected behaviours. >> > > > This doesn't seem like a neutron specific issue to me. If I understand you > correctly what you're saying is if an admin changes the policy.json file to > exclude some data from the response that now the users of the api might > have to change their code? ..port-binding-extension... X.o > Neutron is the only project which extends policy checks into attributes, meaning that to a "regular user" a resource might look different from one deployment to another. If one is writing an application or script which uses the Openstack APIs, it is necessary to perform extra checks to verify whether some attributes are available or not. They might not be available either because some extensions are not available or because policy checks are stripping them off. The former is a consequence of the fact neutron has a very small core and everything else is an extensions, and this is surely not just a neutron problem; the latter issue instead is a consequence of enabling flexible authorisation policies and extending them into attribute visibility. However, I have no plans at the moment to make changes in this area - mostly because hardly any change can be made without causing backward compatibility issue. > > > >> To this aim, one solution would
Re: [openstack-dev] [Neutron] API list operations are not fast as they could because they're dumb
Hi, Comments inline: On Tue, Apr 8, 2014 at 3:16 PM, Salvatore Orlando wrote: > I have been recently investigating reports of slowness for list responses > in the Neutron API. > This was first reported in [1], and then recently was observed with both > the ML2 and the NSX plugins. > The root cause of this issues is that a policy engine check is performed > for every attribute of every resource returned in a response. > When tenant grow to a lot of ports, or when the API is executed with admin > credentials without filters, this might become a non-negligible scale issue. > This issue is mostly due to three factors: > 1) A log statement printing a line in the log for every attribute for > which no policy criterion is defined; this has been treated with [2] > 2) The fact that for every check neutron currently checks whether cached > policy rules are still valid [3] > 3) The fact that anyway Neutron perform really a lot of policy checks > whether it should not > > Despite the improvements [2] and [3] (mostly [2]), neutron for a list > operation still spends for post-plugin operations (ie: policy checks) abotu > 50% of the time it spends in the plugin. > Solving this problem is not difficult, but it might require changes which > are worth of a discussion on the mailing list. > Up to the Havana release policy checks were performed in the plugin; this > basically made responses dependent on plugin implementation and was > terrible for API compatibility and portability; we took care of that with > [4], which moved all policy checks to the API layer. However for one fix > that we fixed, another thing was broken (*) > > The API layer for list responses puts every item through policy checks to > see which should not be visible to the user at all, which is fine. > However it also puts every attribute through a policy check to exclude > those which should not be visible to the user, such as provider attributes > for regular users. > Doing this for every resource might make sense if an attribute should be > visible or not according to the data in the resource itself. > For instance a policy that shows port binding attributes could be defined > for all the ports whose name is "ernest". > This might appear as great flexibility, but does it make any sense at all? > Does it make sense that an API list operation return a set of attributes > for some items and a different one for others? > I think not. > > For this reason I am thinking we should what technically is a simple > change: use policy cghecks determine the list of attributes to show only > once for list response, and then re-use that list for the whole response. > The limitation here is that we should not have 'attribute-level' policies > (**) which rely on the resource value. > I think this limitation is fair. If you like the approach I have some code > here: http://paste.openstack.org/show/75371/ > I think this makes sense. In theory the first element of the list should have all the same columns as the next set of elements so inspecting the first one should be fine. > > And this leads me to the second part of the discussion I'd like to start. > The policy engine currently allows me to start a neutron server where, for > instance, port binding are visible by admins only, and another neutron > server where any user can see them. > This kind of makes the API not really portable, as people programming > against the neutron API might encounter unexpected behaviours. > This doesn't seem like a neutron specific issue to me. If I understand you correctly what you're saying is if an admin changes the policy.json file to exclude some data from the response that now the users of the api might have to change their code? ..port-binding-extension... X.o > To this aim, one solution would be to 'hardcode' attributes' access rights > into extensions definition. This way port bindings will always be > admin_only regardless of which neutron endpoint one is accessing. > However, there are two drawbacks in this approach: > 1 - This could break existing deployment which tweaked default policy, so > the upgrade should be carefully planned > 2 - This will make API definitions dependent on entries in policy.json. If > a policy definition states that an attribute is admin_only, one will also > have to ensure such policy is defined in policy.json. > > Thanks for reading this email, > Salvatore > > [1] https://bugs.launchpad.net/neutron/+bug/1236704 > [2] https://bugs.launchpad.net/neutron/+bug/1302467 > [3] https://bugs.launchpad.net/neutron/+bug/1302611 > [4] https://wiki.openstack.org/wiki/Neutron/Make-authz-orthogonal > (*) typical behaviour of software 'fixed' by Salvatore > (**) policies such as get_network:provider:network_type. > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenSta
Re: [openstack-dev] [Neutron] API list operations are not fast as they could because they're dumb
Checking whether a policy used for visibility depends on an attribute value is possible. I just don't think it's right. For instance, one could define the following policies: "the_funny_one": "field:networks:name=Funny" "get_network:provider:physical_network": "rule:admin_only or rule:the_funny_one" And cause provider network details to be shown to everyone if the name of the network was "Funny". While this is clearly a dumb example, the problem I have with it is that it will produce list results which basically are not tables, meaning that each row might potentially have a different set of columns. I wanted to go to further and just remove all attribute visibility policies from policy.json. Attribute access level would have been defined in the attribute map. However this might have broken some existing deployments, so I think unfortunately it's not the case to do this change. Finally, since I did not see any opposition to the proposed approach, I'm going to smooth out the patch I linked and propose it to gerrit. Salvatore On 13 April 2014 21:49, Eugene Nikanorov wrote: > Hi Salvatore, > > > For this reason I am thinking we should what technically is a simple > change: use policy checks determine the list of attributes to show only > once for list response, and then re-use that list for the whole response. > > I'm +1 to this idea. > Also, would it be too difficult to have some insight into set of policy > rules and see, if there are any rules that apply per attribute value of > given resource? If no - use simplified approach, if yes - fallback to > existing slow one. > Wouldn't it both speedup most of operations while preserving existing > deployments? > > Thanks, > Eugene. > > > On Wed, Apr 9, 2014 at 2:16 AM, Salvatore Orlando wrote: > >> I have been recently investigating reports of slowness for list responses >> in the Neutron API. >> This was first reported in [1], and then recently was observed with both >> the ML2 and the NSX plugins. >> The root cause of this issues is that a policy engine check is performed >> for every attribute of every resource returned in a response. >> When tenant grow to a lot of ports, or when the API is executed with >> admin credentials without filters, this might become a non-negligible scale >> issue. >> This issue is mostly due to three factors: >> 1) A log statement printing a line in the log for every attribute for >> which no policy criterion is defined; this has been treated with [2] >> 2) The fact that for every check neutron currently checks whether cached >> policy rules are still valid [3] >> 3) The fact that anyway Neutron perform really a lot of policy checks >> whether it should not >> >> Despite the improvements [2] and [3] (mostly [2]), neutron for a list >> operation still spends for post-plugin operations (ie: policy checks) abotu >> 50% of the time it spends in the plugin. >> Solving this problem is not difficult, but it might require changes which >> are worth of a discussion on the mailing list. >> Up to the Havana release policy checks were performed in the plugin; this >> basically made responses dependent on plugin implementation and was >> terrible for API compatibility and portability; we took care of that with >> [4], which moved all policy checks to the API layer. However for one fix >> that we fixed, another thing was broken (*) >> >> The API layer for list responses puts every item through policy checks to >> see which should not be visible to the user at all, which is fine. >> However it also puts every attribute through a policy check to exclude >> those which should not be visible to the user, such as provider attributes >> for regular users. >> Doing this for every resource might make sense if an attribute should be >> visible or not according to the data in the resource itself. >> For instance a policy that shows port binding attributes could be defined >> for all the ports whose name is "ernest". >> This might appear as great flexibility, but does it make any sense at all? >> Does it make sense that an API list operation return a set of attributes >> for some items and a different one for others? >> I think not. >> >> For this reason I am thinking we should what technically is a simple >> change: use policy cghecks determine the list of attributes to show only >> once for list response, and then re-use that list for the whole response. >> The limitation here is that we should not have 'attribute-level' policies >> (**) which rely on the resource value. >> I think this limitation is fair. If you like the approach I have some >> code here: http://paste.openstack.org/show/75371/ >> >> And this leads me to the second part of the discussion I'd like to start. >> The policy engine currently allows me to start a neutron server where, >> for instance, port binding are visible by admins only, and another neutron >> server where any user can see them. >> This kind of makes the API not really portable, as people programming >> against th
Re: [openstack-dev] [Neutron] API list operations are not fast as they could because they're dumb
Hi Salvatore, > For this reason I am thinking we should what technically is a simple change: use policy checks determine the list of attributes to show only once for list response, and then re-use that list for the whole response. I'm +1 to this idea. Also, would it be too difficult to have some insight into set of policy rules and see, if there are any rules that apply per attribute value of given resource? If no - use simplified approach, if yes - fallback to existing slow one. Wouldn't it both speedup most of operations while preserving existing deployments? Thanks, Eugene. On Wed, Apr 9, 2014 at 2:16 AM, Salvatore Orlando wrote: > I have been recently investigating reports of slowness for list responses > in the Neutron API. > This was first reported in [1], and then recently was observed with both > the ML2 and the NSX plugins. > The root cause of this issues is that a policy engine check is performed > for every attribute of every resource returned in a response. > When tenant grow to a lot of ports, or when the API is executed with admin > credentials without filters, this might become a non-negligible scale issue. > This issue is mostly due to three factors: > 1) A log statement printing a line in the log for every attribute for > which no policy criterion is defined; this has been treated with [2] > 2) The fact that for every check neutron currently checks whether cached > policy rules are still valid [3] > 3) The fact that anyway Neutron perform really a lot of policy checks > whether it should not > > Despite the improvements [2] and [3] (mostly [2]), neutron for a list > operation still spends for post-plugin operations (ie: policy checks) abotu > 50% of the time it spends in the plugin. > Solving this problem is not difficult, but it might require changes which > are worth of a discussion on the mailing list. > Up to the Havana release policy checks were performed in the plugin; this > basically made responses dependent on plugin implementation and was > terrible for API compatibility and portability; we took care of that with > [4], which moved all policy checks to the API layer. However for one fix > that we fixed, another thing was broken (*) > > The API layer for list responses puts every item through policy checks to > see which should not be visible to the user at all, which is fine. > However it also puts every attribute through a policy check to exclude > those which should not be visible to the user, such as provider attributes > for regular users. > Doing this for every resource might make sense if an attribute should be > visible or not according to the data in the resource itself. > For instance a policy that shows port binding attributes could be defined > for all the ports whose name is "ernest". > This might appear as great flexibility, but does it make any sense at all? > Does it make sense that an API list operation return a set of attributes > for some items and a different one for others? > I think not. > > For this reason I am thinking we should what technically is a simple > change: use policy cghecks determine the list of attributes to show only > once for list response, and then re-use that list for the whole response. > The limitation here is that we should not have 'attribute-level' policies > (**) which rely on the resource value. > I think this limitation is fair. If you like the approach I have some code > here: http://paste.openstack.org/show/75371/ > > And this leads me to the second part of the discussion I'd like to start. > The policy engine currently allows me to start a neutron server where, for > instance, port binding are visible by admins only, and another neutron > server where any user can see them. > This kind of makes the API not really portable, as people programming > against the neutron API might encounter unexpected behaviours. > To this aim, one solution would be to 'hardcode' attributes' access rights > into extensions definition. This way port bindings will always be > admin_only regardless of which neutron endpoint one is accessing. > However, there are two drawbacks in this approach: > 1 - This could break existing deployment which tweaked default policy, so > the upgrade should be carefully planned > 2 - This will make API definitions dependent on entries in policy.json. If > a policy definition states that an attribute is admin_only, one will also > have to ensure such policy is defined in policy.json. > > Thanks for reading this email, > Salvatore > > [1] https://bugs.launchpad.net/neutron/+bug/1236704 > [2] https://bugs.launchpad.net/neutron/+bug/1302467 > [3] https://bugs.launchpad.net/neutron/+bug/1302611 > [4] https://wiki.openstack.org/wiki/Neutron/Make-authz-orthogonal > (*) typical behaviour of software 'fixed' by Salvatore > (**) policies such as get_network:provider:network_type. > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/c
[openstack-dev] [Neutron] API list operations are not fast as they could because they're dumb
I have been recently investigating reports of slowness for list responses in the Neutron API. This was first reported in [1], and then recently was observed with both the ML2 and the NSX plugins. The root cause of this issues is that a policy engine check is performed for every attribute of every resource returned in a response. When tenant grow to a lot of ports, or when the API is executed with admin credentials without filters, this might become a non-negligible scale issue. This issue is mostly due to three factors: 1) A log statement printing a line in the log for every attribute for which no policy criterion is defined; this has been treated with [2] 2) The fact that for every check neutron currently checks whether cached policy rules are still valid [3] 3) The fact that anyway Neutron perform really a lot of policy checks whether it should not Despite the improvements [2] and [3] (mostly [2]), neutron for a list operation still spends for post-plugin operations (ie: policy checks) abotu 50% of the time it spends in the plugin. Solving this problem is not difficult, but it might require changes which are worth of a discussion on the mailing list. Up to the Havana release policy checks were performed in the plugin; this basically made responses dependent on plugin implementation and was terrible for API compatibility and portability; we took care of that with [4], which moved all policy checks to the API layer. However for one fix that we fixed, another thing was broken (*) The API layer for list responses puts every item through policy checks to see which should not be visible to the user at all, which is fine. However it also puts every attribute through a policy check to exclude those which should not be visible to the user, such as provider attributes for regular users. Doing this for every resource might make sense if an attribute should be visible or not according to the data in the resource itself. For instance a policy that shows port binding attributes could be defined for all the ports whose name is "ernest". This might appear as great flexibility, but does it make any sense at all? Does it make sense that an API list operation return a set of attributes for some items and a different one for others? I think not. For this reason I am thinking we should what technically is a simple change: use policy cghecks determine the list of attributes to show only once for list response, and then re-use that list for the whole response. The limitation here is that we should not have 'attribute-level' policies (**) which rely on the resource value. I think this limitation is fair. If you like the approach I have some code here: http://paste.openstack.org/show/75371/ And this leads me to the second part of the discussion I'd like to start. The policy engine currently allows me to start a neutron server where, for instance, port binding are visible by admins only, and another neutron server where any user can see them. This kind of makes the API not really portable, as people programming against the neutron API might encounter unexpected behaviours. To this aim, one solution would be to 'hardcode' attributes' access rights into extensions definition. This way port bindings will always be admin_only regardless of which neutron endpoint one is accessing. However, there are two drawbacks in this approach: 1 - This could break existing deployment which tweaked default policy, so the upgrade should be carefully planned 2 - This will make API definitions dependent on entries in policy.json. If a policy definition states that an attribute is admin_only, one will also have to ensure such policy is defined in policy.json. Thanks for reading this email, Salvatore [1] https://bugs.launchpad.net/neutron/+bug/1236704 [2] https://bugs.launchpad.net/neutron/+bug/1302467 [3] https://bugs.launchpad.net/neutron/+bug/1302611 [4] https://wiki.openstack.org/wiki/Neutron/Make-authz-orthogonal (*) typical behaviour of software 'fixed' by Salvatore (**) policies such as get_network:provider:network_type. ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
