[Bug 2092217] Re: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope
This bug was fixed in the package python-neutron-lib - 2.20.0-0ubuntu1.1~cloud0 --- python-neutron-lib (2.20.0-0ubuntu1.1~cloud0) focal-yoga; urgency=medium . * New update for the Ubuntu Cloud Archive. . python-neutron-lib (2.20.0-0ubuntu1.1) jammy; urgency=medium . [ Corey Bryant ] * d/gbp.conf: Create stable/yoga branch. . [ Bryan Fraschetti ] * d/p/rbac-group-by-optimization.patch: Introduce missing "GROUP BY" clause on RBAC queries to optimize performance (LP: #2092217). * d/p/deprecated-tenant-validation.patch: Remove the test that validates the tenant property as this was removed in oslo.context. ** Changed in: cloud-archive/yoga Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2092217 Title: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2092217/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2092217] Re: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope
This bug was fixed in the package python-neutron-lib - 3.4.0-0ubuntu1.1~cloud0 --- python-neutron-lib (3.4.0-0ubuntu1.1~cloud0) jammy-antelope; urgency=medium . [ Corey Bryant ] * d/gbp.conf: Create stable/2023.1 branch. . [ Bryan Fraschetti ] * d/p/rbac-group-by-optimization.patch: Introduce missing "GROUP BY" clause on RBAC queries to optimize performance (LP: #2092217) ** Changed in: cloud-archive/antelope Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2092217 Title: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2092217/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2092217] Re: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope
Testing for Antelope using the same procedure as in the previous comment: Deployed a fresh openstack cloud and ran the timing performance script: Performance in virtually empty fresh cloud Time taken for network list: 1528 ms Time taken for security group list: 1334 ms Time taken for network rbac list: 1243 ms Time taken for network show: 1296 ms Used some scripts to create 10 networks, 10 security groups, 1000 projects, and 2 RBAC rules sharing everything Performance of unpatched with 2 RBAC rules Time taken for network list: 4735 ms Time taken for security group list: 3955 ms Time taken for network rbac list: 18564 ms Time taken for network show: 1367 ms Enabled Proposed: Performance of proposed with 2 RBAC rules Time taken for network list: 2189 ms Time taken for security group list: 1605 ms Time taken for network rbac list: 13678 ms Time taken for network show: 1498 ms --- Testing on Yoga In a fresh openstack cloud Performance in virtually empty fresh cloud Time taken for network list: 1711 ms Time taken for security group list: 1718 ms Time taken for network rbac list: 1296 ms Time taken for network show: 1274 ms Created 10x Networks, 10x Security Groups, 1000x Projects, and created the rbac policies Performance of unpatched with 2 RBAC rules Time taken for network list: 5223 ms Time taken for security group list: 5744 ms Time taken for network rbac list: 18511 ms Time taken for network show: 1782 ms Performance of proposed with 2 RBAC rules Time taken for network list: 2228 ms Time taken for security group list: 2860 ms Time taken for network rbac list: 14086 ms Time taken for network show: 1645 ms ** Tags removed: verification-antelope-needed verification-yoga-needed ** Tags added: verification-antelope-done verification-yoga-done -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2092217 Title: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2092217/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2092217] Re: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope
This bug was fixed in the package python-neutron-lib - 2.20.0-0ubuntu1.1 --- python-neutron-lib (2.20.0-0ubuntu1.1) jammy; urgency=medium [ Corey Bryant ] * d/gbp.conf: Create stable/yoga branch. [ Bryan Fraschetti ] * d/p/rbac-group-by-optimization.patch: Introduce missing "GROUP BY" clause on RBAC queries to optimize performance (LP: #2092217). * d/p/deprecated-tenant-validation.patch: Remove the test that validates the tenant property as this was removed in oslo.context. -- Corey Bryant Mon, 25 Apr 2022 10:04:26 -0400 ** Changed in: python-neutron-lib (Ubuntu Jammy) Status: Fix Committed => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2092217 Title: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2092217/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2092217] Re: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope
Hello, I've performed the SRU verification and all looks good on my end. Firstly, after deploying a fresh openstack cloud I ran the attached test script, which benchmarks performance and response times from the neutron api service. The following results show the performance with the default ext_net and private networks, admin security group, and no rbac policies as a baseline Performance in virtually empty fresh cloud Time taken for network list: 1496 ms Time taken for security group list: 1274 ms Time taken for network rbac list: 1194 ms Time taken for network show: 1408 ms Then I used a script to create 1000 projects, 10 networks, and 10 security groups, and share each network and security group with each project. Effectively creating 2 RBAC rules, which is slightly less than the ~3 in the original bug report. At this point I ran the same test script, which produced the following performance: Performance of unpatched with 2 RBAC rules Time taken for network list: 9330 ms Time taken for security group list: 7824 ms Time taken for network rbac list: 20576 ms Time taken for network show: 1544 ms Here I enabled proposed and ran the same test Performance of proposed with 2 RBAC rules Time taken for network list: 2400 ms Time taken for security group list: 1952 ms Time taken for network rbac list: 13727 ms Time taken for network show: 1341 ms As expected, listing the networks and security groups requires significantly less time due to the lower cardinality of the returned set by the /network API. In the original report this endpoint was noted as being particularly affected. Thanks! ** Attachment added: "test.sh" https://bugs.launchpad.net/ubuntu/+source/python-neutron-lib/+bug/2092217/+attachment/5867728/+files/test.sh ** Tags removed: verification-needed verification-needed-jammy ** Tags added: verification-done verification-done-jammy -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2092217 Title: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2092217/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2092217] Re: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope
Hello Bryan, or anyone else affected, Accepted python-neutron-lib into jammy-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/python- neutron-lib/2.20.0-0ubuntu1.1 in a few hours, and then in the -proposed repository. Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users. If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed- jammy to verification-done-jammy. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification- failed-jammy. In either case, without details of your testing we will not be able to proceed. Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping! N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days. ** Changed in: python-neutron-lib (Ubuntu Jammy) Status: Confirmed => Fix Committed ** Tags added: verification-needed verification-needed-jammy -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2092217 Title: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2092217/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2092217] Re: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope
@tjaalton as noted in my comment (missing SRU information); I don't usually sponsor without that in place but on this occasion missed that it was not present. I see it is now - thanks @bryanfrashetti -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2092217 Title: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2092217/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2092217] Re: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope
** Description changed: [ Impact ] * There is a missing performance optimization in the Yoga (2.20.0-0ubuntu1) and Antelope (3.4.0-0ubuntu1) releases of neutron-lib caused by a missing database query optimization at [1], which is present upstream. In particular, the commit at [1] minimizes the number of objects returned from the database (grouping by resource id to prevent returning duplicated rows). The commit is a follow up to the commit at [2] in neutron. Both commits are needed to get the performance benefit. Fortunately said neutron commit [2] is present in Yoga (2:20.5.0-0ubuntu2) and Antelope (2:22.0.2-0ubuntu1) * To summarize, commit [1] is present in Ubuntu's releases of Zed (3.1.2-0ubuntu1), Bobcat (3.8.0-0ubuntu1), or Caracal (3.11.0-0ubuntu1) but not Yoga (2.20.0-0ubuntu1) or Antelope (3.4.0-0ubuntu1) * This proposed SRU patch is a cherry-pick backport of the upstream commit [1] [ Test Plan ] * Deploy openstack yoga/antelope on jammy and create as many RBAC rules as possible (the original report had 15000 to 3 on various clouds). A simple and programmatic way to generate thousands of rules would be to generate thousands of projects, a few admin security groups and networks and share them with all projects * Measure the time elapsed when executing: openstack network rbac list openstack security group list + openstack network list + openstack network show * The expected behaviour is that without the patch the performance will degrade with added RBAC groups while there will be no significant performance change with the patch. [ Where problems could occur ] * While it is not expected to introduce a regression as the Ubuntu releases surrounding Yoga and Antelope contain the patch, the patch inherently affects performance. There may be edge cases with a reduction in performance. For example, this patch reduces the number of returned values of the query to the set of rows (eliminating duplicates) - if there are few RBAC rules, this may be unnecessary overhead as the query may not have produced many duplicates results to begin with. * This affects the aggregation of sql queries, if there is an integration that doesn't expect this grouping (for some reason expects duplicates), it may be affected [ Other Info ] * There may be questions regarding whether there are accompanying tests to prove that there is no regression. Those tests are present in the associated neutron patch (as per the commit message [3]). * Note that I have included an additional patch in the Yoga debdiff. This additional patch removes a failing test related to tenant_id. It's not actually related to the neutron patch but I incidentally discovered it when addressing the missing optimization that is the primary concern of this SRU. If one clones the source of the Yoga release and then tries to debuild it, the build fails with a single error associated with a tenant_id test. This test was removed upstream in March 2022 [4], before the Yoga release, shortly after the request context in oslo deprecated the same field in January 2022 [5], which manifested the test failure. - * The original report of the poor scaling with RBAC rules that lead to + * The original report of the poor scaling with RBAC rules that lead to the development of the change is located at the LP bug [6], which may provide some additional context. Neutron RBAC performance optimization: [1] https://opendev.org/openstack/neutron-lib/commit/829e97024c2b73dd67bfd8a04c65f03be556eec8 [2] https://opendev.org/openstack/neutron/commit/4c654dc553fa8c52e4459c7ac781dca536dba05b [3] https://review.opendev.org/c/openstack/neutron/+/884877 [6] https://bugs.launchpad.net/neutron/+bug/1918145 Removal of tenant_id test: [4] https://opendev.org/openstack/neutron-lib/commit/a8abe9d592da5bcf065af40e0ba1cd3599ede1e7 [5] https://review.opendev.org/c/openstack/oslo.context/+/815938 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2092217 Title: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2092217/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2092217] Re: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope
** Description changed: [ Impact ] * There is a missing performance optimization in the Yoga (2.20.0-0ubuntu1) and Antelope (3.4.0-0ubuntu1) releases of neutron-lib caused by a missing database query optimization at [1], which is present upstream. In particular, the commit at [1] minimizes the number of objects returned from the database (grouping by resource id to prevent returning duplicated rows). The commit is a follow up to the commit at [2] in neutron. Both commits are needed to get the performance benefit. Fortunately said neutron commit [2] is present in Yoga (2:20.5.0-0ubuntu2) and Antelope (2:22.0.2-0ubuntu1) * To summarize, commit [1] is present in Ubuntu's releases of Zed (3.1.2-0ubuntu1), Bobcat (3.8.0-0ubuntu1), or Caracal (3.11.0-0ubuntu1) but not Yoga (2.20.0-0ubuntu1) or Antelope (3.4.0-0ubuntu1) * This proposed SRU patch is a cherry-pick backport of the upstream commit [1] [ Test Plan ] * Deploy openstack yoga/antelope on jammy and create as many RBAC - rules as possible (the original report had 28000). A simple and - programmatic way to generate thousands of rules would be to generate - thousands of projects, a few admin security groups and networks and - share them with all projects + rules as possible (the original report had 15000 to 3 on various + clouds). A simple and programmatic way to generate thousands of rules + would be to generate thousands of projects, a few admin security groups + and networks and share them with all projects * Measure the time elapsed when executing: openstack network rbac list openstack security group list * The expected behaviour is that without the patch the performance will degrade with added RBAC groups while there will be no significant performance change with the patch. [ Where problems could occur ] * While it is not expected to introduce a regression as the Ubuntu releases surrounding Yoga and Antelope contain the patch, the patch inherently affects performance. There may be edge cases with a reduction in performance. For example, this patch reduces the number of returned values of the query to the set of rows (eliminating duplicates) - if there are few RBAC rules, this may be unnecessary overhead as the query may not have produced many duplicates results to begin with. * This affects the aggregation of sql queries, if there is an integration that doesn't expect this grouping (for some reason expects duplicates), it may be affected [ Other Info ] * There may be questions regarding whether there are accompanying tests to prove that there is no regression. Those tests are present in the associated neutron patch (as per the commit message [3]). * Note that I have included an additional patch in the Yoga debdiff. This additional patch removes a failing test related to tenant_id. It's not actually related to the neutron patch but I incidentally discovered it when addressing the missing optimization that is the primary concern of this SRU. If one clones the source of the Yoga release and then tries to debuild it, the build fails with a single error associated with a tenant_id test. This test was removed upstream in March 2022 [4], before the Yoga release, shortly after the request context in oslo deprecated the same field in January 2022 [5], which manifested the test failure. + * The original report of the poor scaling with RBAC rules that lead to + the development of the change is located at the LP bug [6], which may + provide some additional context. + Neutron RBAC performance optimization: [1] https://opendev.org/openstack/neutron-lib/commit/829e97024c2b73dd67bfd8a04c65f03be556eec8 [2] https://opendev.org/openstack/neutron/commit/4c654dc553fa8c52e4459c7ac781dca536dba05b [3] https://review.opendev.org/c/openstack/neutron/+/884877 + [6] https://bugs.launchpad.net/neutron/+bug/1918145 Removal of tenant_id test: [4] https://opendev.org/openstack/neutron-lib/commit/a8abe9d592da5bcf065af40e0ba1cd3599ede1e7 [5] https://review.opendev.org/c/openstack/oslo.context/+/815938 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2092217 Title: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2092217/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2092217] Re: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope
** Description changed: [ Impact ] - * There is a missing performance optimization in the Yoga + * There is a missing performance optimization in the Yoga (2.20.0-0ubuntu1) and Antelope (3.4.0-0ubuntu1) releases of neutron-lib caused by a missing database query optimization at [1], which is present upstream. In particular, the commit at [1] minimizes the number of objects returned from the database (grouping by resource id to prevent returning duplicated rows). The commit is a follow up to the commit at [2] in neutron. Both commits are needed to get the performance benefit. Fortunately said neutron commit [2] is present in Yoga (2:20.5.0-0ubuntu2) and Antelope (2:22.0.2-0ubuntu1) - * To summarize, commit [1] is present in Ubuntu's releases of Zed + * To summarize, commit [1] is present in Ubuntu's releases of Zed (3.1.2-0ubuntu1), Bobcat (3.8.0-0ubuntu1), or Caracal (3.11.0-0ubuntu1) but not Yoga (2.20.0-0ubuntu1) or Antelope (3.4.0-0ubuntu1) - * This proposed SRU patch is a cherry-pick backport of the upstream + * This proposed SRU patch is a cherry-pick backport of the upstream commit [1] [ Test Plan ] - * Deploy openstack yoga/antelope on jammy and create as many RBAC + * Deploy openstack yoga/antelope on jammy and create as many RBAC rules as possible (the original report had 28000). A simple and programmatic way to generate thousands of rules would be to generate thousands of projects, a few admin security groups and networks and share them with all projects - * Measure the time elapsed when executing: - openstack network rbac list - openstack security group list + * Measure the time elapsed when executing: + openstack network rbac list + openstack security group list - * The expected behaviour is that without the patch the performance + * The expected behaviour is that without the patch the performance will degrade with added RBAC groups while there will be no significant performance change with the patch. - [ Where problems could occur ] + [ Where problems could occur ] - * While it is not expected to introduce a regression as the Ubuntu + * While it is not expected to introduce a regression as the Ubuntu releases surrounding Yoga and Antelope contain the patch, the patch inherently affects performance. There may be edge cases with a reduction in performance. For example, this patch reduces the number of returned values of the query to the set of rows (eliminating duplicates) - if there are few RBAC rules, this may be unnecessary overhead as the query may not have produced many duplicates results to begin with. - * This affects the aggregation of sql queries, if there is an + * This affects the aggregation of sql queries, if there is an integration that doesn't expect this grouping (for some reason expects duplicates), it may be affected [ Other Info ] - * There may be questions regarding whether there are accompanying + * There may be questions regarding whether there are accompanying tests to prove that there is no regression. Those tests are present in the associated neutron patch (as per the commit message [3]). - * Note that I have included an additional patch in the Yoga debdiff. + * Note that I have included an additional patch in the Yoga debdiff. This additional patch removes a failing test related to tenant_id. It's not actually related to the neutron patch but I incidentally discovered it when addressing the missing optimization that is the primary concern of this SRU. If one clones the source of the Yoga release and then tries - to debuild it (without making any modifications to source), the build - fails with a single error associated with a tenant_id test. This test - was removed upstream in March 2022 [4], before the Yoga release, shortly - after the request context in oslo deprecated the same field in January - 2022 [5], which manifested the test failure. + to debuild it, the build fails with a single error associated with a + tenant_id test. This test was removed upstream in March 2022 [4], before + the Yoga release, shortly after the request context in oslo deprecated + the same field in January 2022 [5], which manifested the test failure. Neutron RBAC performance optimization: [1] https://opendev.org/openstack/neutron-lib/commit/829e97024c2b73dd67bfd8a04c65f03be556eec8 [2] https://opendev.org/openstack/neutron/commit/4c654dc553fa8c52e4459c7ac781dca536dba05b [3] https://review.opendev.org/c/openstack/neutron/+/884877 Removal of tenant_id test: [4] https://opendev.org/openstack/neutron-lib/commit/a8abe9d592da5bcf065af40e0ba1cd3599ede1e7 [5] https://review.opendev.org/c/openstack/oslo.context/+/815938 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2092217 Title: [SRU] Missing Group-By Optimization
[Bug 2092217] Re: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope
** Description changed: - This SRU is related to bug #1918145, which reported slow RBAC queries in - Neutron. The performance issue was addressed upstream through modifying - the SQL queries [1] and using a GROUP BY clause in the Neutron lib [2]. + [ Impact ] - Looking at the Ubuntu Neutron lib package [3], this group by - optimization clause [2] is missing in stable/yoga and stable/antelope, - although it present in Zed, Bobcat, and Caracal + * There is a missing performance optimization in the Yoga + (2.20.0-0ubuntu1) and Antelope (3.4.0-0ubuntu1) releases of neutron-lib + caused by a missing database query optimization at [1], which is present + upstream. In particular, the commit at [1] minimizes the number of + objects returned from the database (grouping by resource id to prevent + returning duplicated rows). The commit is a follow up to the commit at + [2] in neutron. Both commits are needed to get the performance benefit. + Fortunately said neutron commit [2] is present in Yoga + (2:20.5.0-0ubuntu2) and Antelope (2:22.0.2-0ubuntu1) - Resources: - [1] https://review.opendev.org/c/openstack/neutron/+/884877 (Upstream neutron change alongside the library optimization) + * To summarize, commit [1] is present in Ubuntu's releases of Zed + (3.1.2-0ubuntu1), Bobcat (3.8.0-0ubuntu1), or Caracal (3.11.0-0ubuntu1) + but not Yoga (2.20.0-0ubuntu1) or Antelope (3.4.0-0ubuntu1) - [2] https://opendev.org/openstack/neutron- - lib/commit/829e97024c2b73dd67bfd8a04c65f03be556eec8 (Neutron lib group - by optimization change that is missing from yoga and antelope) + * This proposed SRU patch is a cherry-pick backport of the upstream + commit [1] - [3] https://code.launchpad.net/~ubuntu-openstack- - dev/ubuntu/+source/python-neutron-lib/+git/python-neutron-lib (Ubuntu - Neutron lib in the UCA) + [ Test Plan ] + + * Deploy openstack yoga/antelope on jammy and create as many RBAC + rules as possible (the original report had 28000). A simple and + programmatic way to generate thousands of rules would be to generate + thousands of projects, a few admin security groups and networks and + share them with all projects + + * Measure the time elapsed when executing: + openstack network rbac list + openstack security group list + + * The expected behaviour is that without the patch the performance + will degrade with added RBAC groups while there will be no significant + performance change with the patch. + + [ Where problems could occur ] + + * While it is not expected to introduce a regression as the Ubuntu + releases surrounding Yoga and Antelope contain the patch, the patch + inherently affects performance. There may be edge cases with a reduction + in performance. For example, this patch reduces the number of returned + values of the query to the set of rows (eliminating duplicates) - if + there are few RBAC rules, this may be unnecessary overhead as the query + may not have produced many duplicates results to begin with. + + * This affects the aggregation of sql queries, if there is an + integration that doesn't expect this grouping (for some reason expects + duplicates), it may be affected + + [ Other Info ] + + * There may be questions regarding whether there are accompanying + tests to prove that there is no regression. Those tests are present in + the associated neutron patch (as per the commit message [3]). + + * Note that I have included an additional patch in the Yoga debdiff. + This additional patch removes a failing test related to tenant_id. It's + not actually related to the neutron patch but I incidentally discovered + it when addressing the missing optimization that is the primary concern + of this SRU. If one clones the source of the Yoga release and then tries + to debuild it (without making any modifications to source), the build + fails with a single error associated with a tenant_id test. This test + was removed upstream in March 2022 [4], before the Yoga release, shortly + after the request context in oslo deprecated the same field in January + 2022 [5], which manifested the test failure. + + Neutron RBAC performance optimization: + [1] https://opendev.org/openstack/neutron-lib/commit/829e97024c2b73dd67bfd8a04c65f03be556eec8 + [2] https://opendev.org/openstack/neutron/commit/4c654dc553fa8c52e4459c7ac781dca536dba05b + [3] https://review.opendev.org/c/openstack/neutron/+/884877 + + Removal of tenant_id test: + [4] https://opendev.org/openstack/neutron-lib/commit/a8abe9d592da5bcf065af40e0ba1cd3599ede1e7 + [5] https://review.opendev.org/c/openstack/oslo.context/+/815938 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2092217 Title: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2092217/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@l
[Bug 2092217] Re: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope
missing sru information please, don't sponsor uploads without it! -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2092217 Title: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2092217/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2092217] Re: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope
Also for the jammy upload you will need to detail why the additional patch for oslo.context behavioural changes is required - I'm guessing without it you see a test failure during the package build? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2092217 Title: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2092217/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2092217] Re: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope
Thanks Bryan I'm happy with the updates and have sponsored them into the selected targets HOWEVER this bug will need full SRU documentation before we subscribe the ubuntu-sru team and accept into proposed pockets for testing. ** Changed in: python-neutron-lib (Ubuntu) Status: Invalid => Fix Released ** Changed in: cloud-archive Status: Invalid => Fix Released ** Changed in: cloud-archive/zed Status: Fix Released => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2092217 Title: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2092217/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2092217] Re: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope
** Changed in: python-neutron-lib (Ubuntu) Status: Confirmed => Invalid ** Changed in: cloud-archive Status: New => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2092217 Title: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2092217/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2092217] Re: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope
** Changed in: python-neutron-lib (Ubuntu) Assignee: (unassigned) => Bryan Fraschetti (bryanfraschetti) ** Changed in: python-neutron-lib (Ubuntu) Status: New => Confirmed ** Changed in: python-neutron-lib (Ubuntu Jammy) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2092217 Title: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2092217/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2092217] Re: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope
** No longer affects: cloud-archive/antelope ** Also affects: cloud-archive/antelope Importance: Undecided Status: New ** Also affects: python-neutron-lib (Ubuntu Jammy) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2092217 Title: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2092217/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2092217] Re: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope
Surely ubuntu-sponsors isn't the right team to sponsor OpenStack uploads? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2092217 Title: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2092217/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2092217] Re: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope
The attachment "lp2092217_antelope.debdiff" seems to be a debdiff. The ubuntu-sponsors team has been subscribed to the bug report so that they can review and hopefully sponsor the debdiff. If the attachment isn't a patch, please remove the "patch" flag from the attachment, remove the "patch" tag, and if you are member of the ~ubuntu-sponsors, unsubscribe the team. [This is an automated message performed by a Launchpad user owned by ~brian-murray, for any issue please contact him.] ** Tags added: patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2092217 Title: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2092217/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2092217] Re: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope
** Also affects: cloud-archive Importance: Undecided Status: New ** Also affects: cloud-archive/caracal Importance: Undecided Status: New ** Also affects: cloud-archive/yoga Importance: Undecided Status: New ** Also affects: cloud-archive/zed Importance: Undecided Status: New ** Also affects: cloud-archive/bobcat Importance: Undecided Status: New ** Also affects: cloud-archive/antelope Importance: Undecided Status: New ** Changed in: cloud-archive/zed Status: New => Fix Released ** Changed in: cloud-archive/caracal Status: New => Fix Released ** Changed in: cloud-archive/bobcat Status: New => Fix Released -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2092217 Title: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-archive/+bug/2092217/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2092217] Re: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope
This debdiff contains the upstream neutron lib change as a patch in debian/patches/rbac-group-by-optimization.patch for the yoga release. Note that in this debdiff, accompanying the above change is a patch for an unrelated test change: debian/patches/deprecated-tenant- validation.patch. This patch addresses a test in neutron_lib/tests/unit/test_context.py which checks for a field that was deprecated in oslo. In particular, the change removes said test. This patch was merged upstream right around the time that Yoga was released: https://opendev.org/openstack/neutron- lib/commit/a8abe9d592da5bcf065af40e0ba1cd3599ede1e7. As debuild was failing on the test that this change removes, I chose to include the commit in the diff ** Patch added: "lp2092217_yoga.debdiff" https://bugs.launchpad.net/ubuntu/+source/python-neutron-lib/+bug/2092217/+attachment/5846269/+files/lp2092217_yoga.debdiff ** Description changed: This SRU is related to bug #1918145, which reported slow RBAC queries in Neutron. The performance issue was addressed upstream through modifying the SQL queries [1] and using a GROUP BY clause in the Neutron lib [2]. Looking at the Ubuntu Neutron lib package [3], this group by - optimization clause is missing in stable/yoga and stable/antelope, + optimization clause [2] is missing in stable/yoga and stable/antelope, although it present in Zed, Bobcat, and Caracal Resources: - [1] https://review.opendev.org/c/openstack/neutron/+/884877 (Upstream neutron change in preparation of the library optimization) + [1] https://review.opendev.org/c/openstack/neutron/+/884877 (Upstream neutron change alongside the library optimization) [2] https://opendev.org/openstack/neutron- lib/commit/829e97024c2b73dd67bfd8a04c65f03be556eec8 (Neutron lib group - by optimization change) + by optimization change that is missing from yoga and antelope) [3] https://code.launchpad.net/~ubuntu-openstack- dev/ubuntu/+source/python-neutron-lib/+git/python-neutron-lib (Ubuntu Neutron lib in the UCA) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2092217 Title: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-neutron-lib/+bug/2092217/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2092217] Re: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope
This debdiff contains the upstream neutron lib change as a patch in debian/patches/rbac-group-by-optimization.patch ** Patch added: "lp2092217_antelope.debdiff" https://bugs.launchpad.net/ubuntu/+source/python-neutron-lib/+bug/2092217/+attachment/5846243/+files/lp2092217_antelope.debdiff -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2092217 Title: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-neutron-lib/+bug/2092217/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2092217] Re: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope
** Description changed: This SRU is related to bug #1918145, which reported slow RBAC queries in Neutron. The performance issue was addressed upstream through modifying the SQL queries [1] and using a GROUP BY clause in the Neutron lib [2]. Looking at the Ubuntu Neutron lib package [3], this group by optimization clause is missing in stable/yoga and stable/antelope, although it present in Zed, Bobcat, and Caracal - Resources: [1] https://review.opendev.org/c/openstack/neutron/+/884877 (Upstream neutron change in preparation of the library optimization) - [2] https://github.com/openstack/neutron- + [2] https://opendev.org/openstack/neutron- lib/commit/829e97024c2b73dd67bfd8a04c65f03be556eec8 (Neutron lib group by optimization change) [3] https://code.launchpad.net/~ubuntu-openstack- dev/ubuntu/+source/python-neutron-lib/+git/python-neutron-lib (Ubuntu Neutron lib in the UCA) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2092217 Title: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-neutron-lib/+bug/2092217/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 2092217] Re: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2092217 Title: [SRU] Missing Group-By Optimization Backport in Yoga and Antelope To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/python-neutron-lib/+bug/2092217/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
