Public bug reported: Short Summary: In order to rectify negative user experience caused by timeouts associated with loading a large amount of objects from the database and displaying them in horizon, emulation of pagination within horizon is proposed.
Summary: When running OpenStack at scale with circa 6000 different projects, horizon experiences a timeout when it tries to load project resources (accessing through the API works fine). It can be assumed that the issue will hold for users and domain alike since the mechanism is the same. The issue is reproducible with a large number of users, domains or projects. We believe that pagination support in keystone API would solve the aforementioned issues. Unfortunately, pagination was removed from keystone in the v3 api and as far as we're aware it was never re-added. https://docs.openstack.org/api-ref/identity/v3/?expanded=get-available- project-scopes-detail,list-projects-detail,list-domains-detail#list- projects https://docs.openstack.org/api-ref/identity/v3/?expanded=get-available- project-scopes-detail,list-projects-detail,list-domains-detail#list- domains The decision to remove pagination was a conscious and was done around 2015. https://lists.openstack.org/pipermail/openstack- dev/2015-August/thread.html#72082 Furthermore, a look at the code confirms the aforementioned deletion of pagination support. This seems like a fundamental performance oriented feature because the overhead in data conversion can be quite a bit when you have a large number of any objects being returned from a database. https://github.com/openstack/keystone/blame/d7b1d57cae738183f8d85413e942402a8a4efb31/keystone/server/flask/common.py#L675 Currently and effort is under way to get a discussion going in order to reimplement pagination support within keystone API for domains and projects, however it is currently not possible to implement the same feature for users. Long story short, aforementioned changes to the keystone API lead to a negative user experience when using Horizon. In order to rectify the said negative experience in the short run, it is believe possible to emulate pagination and load users, projects and domains in chunks. In the long run changes to the keystone API should follow with further subsequent changes to Horizon for projects and domains, and possibly in the very long run, users. ** Affects: horizon Importance: Undecided Status: New ** Summary changed: - Horizon times out when loading a large amount of projects (>6000) + Horizon times out when loading a large amount of resources to display -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Dashboard (Horizon). https://bugs.launchpad.net/bugs/1982966 Title: Horizon times out when loading a large amount of resources to display Status in OpenStack Dashboard (Horizon): New Bug description: Short Summary: In order to rectify negative user experience caused by timeouts associated with loading a large amount of objects from the database and displaying them in horizon, emulation of pagination within horizon is proposed. Summary: When running OpenStack at scale with circa 6000 different projects, horizon experiences a timeout when it tries to load project resources (accessing through the API works fine). It can be assumed that the issue will hold for users and domain alike since the mechanism is the same. The issue is reproducible with a large number of users, domains or projects. We believe that pagination support in keystone API would solve the aforementioned issues. Unfortunately, pagination was removed from keystone in the v3 api and as far as we're aware it was never re-added. https://docs.openstack.org/api-ref/identity/v3/?expanded=get- available-project-scopes-detail,list-projects-detail,list-domains- detail#list-projects https://docs.openstack.org/api-ref/identity/v3/?expanded=get- available-project-scopes-detail,list-projects-detail,list-domains- detail#list-domains The decision to remove pagination was a conscious and was done around 2015. https://lists.openstack.org/pipermail/openstack- dev/2015-August/thread.html#72082 Furthermore, a look at the code confirms the aforementioned deletion of pagination support. This seems like a fundamental performance oriented feature because the overhead in data conversion can be quite a bit when you have a large number of any objects being returned from a database. https://github.com/openstack/keystone/blame/d7b1d57cae738183f8d85413e942402a8a4efb31/keystone/server/flask/common.py#L675 Currently and effort is under way to get a discussion going in order to reimplement pagination support within keystone API for domains and projects, however it is currently not possible to implement the same feature for users. Long story short, aforementioned changes to the keystone API lead to a negative user experience when using Horizon. In order to rectify the said negative experience in the short run, it is believe possible to emulate pagination and load users, projects and domains in chunks. In the long run changes to the keystone API should follow with further subsequent changes to Horizon for projects and domains, and possibly in the very long run, users. To manage notifications about this bug go to: https://bugs.launchpad.net/horizon/+bug/1982966/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp