On Tue, Aug 13, 2013 at 3:10 AM, Henry Nash <hen...@linux.vnet.ibm.com>wrote:
> Hi > > So few comebacks to the various comments: > > 1) While I understand the idea that a client would follow the next/prev > links returned in collections, I wasn't aware that we considered > 'page'/'per-page' as not standardized. We list these explicitly throughout > the identity API spec (look in each List 'entity' example). > They were essentially relics from a very early draft of the spec that were thoughtlessly copy/pasted around (I'm guilty of this myself)... they were recently cleaned up and removed from the spec. > How I imagined it would work would be: > > a) If a client did not include 'page' in the url we would not paginate > Make that a deployment option? per_page could simply default to a very high value. > b) Once we are paginating, a client can either build the next/prevs urls > themselves if they want (by incrementing/decrementing the page number), or > just follow the next/prev links (which come with the appropriate 'page=x' > in them) returned in the collection which saves them having to do this. > I'm obviously very opposed to this because it unreasonably forces a single approach to pagination across all drivers. > c) Regarding implementation, the controller would continue to be able to > paginate on behalf of drivers that couldn't, but those paginate-aware > drivers would take over that capability (and indicate this to the > controller the state of the pagination so that it can build the correct > next/prev links) > > 2) On the subject of huge enumerates, options are: > a) Support a backend manager scoped (i.e. identity/assignent/token) limit > in the conf file which would be honored by drivers. Assuming that you set > this larger than your pagination limit, this would make sense whether your > driver is paginating or not in terms of minimizing the delay in responding > data as well as not messing up pagination. In the non-paginated case when > we hit the limit, should we indicate this to the client? Maybe a 206 > return code? Although i) not quite sure that meets http standards, and ii) > would we break a bunch of clients by doing this? > I'm not clear on what kind of limit you're referring to? A 206 sounds unexpected for this use case though. > b) We scrap the whole idea of pagination, and just set a conf limit as in > 2a). To make this work of course, we must implement any defined filters in > the backend (otherwise we still end up with today's performance problems - > remember that today, in general, filtering is done in the controller on a > full enumeration of the entities in question). I was planning to implement > this backend filtering anyway as part of (or on top of) my change, since we > are holding (at least one of) our hands behind our backs right now by not > doing so. And our filters need to be powerful, do we support wildcards for > example, e.g. GET /users?name = fred* ? > There were some discussions on this topic from about a year ago that I'd love to continue. I don't want to invent a new "language," but we do need to settle on an approach that we can apply across a wide variety of backends. That probably means keeping it very simple (like your example). Asterisks need to be URL encoded, though. One suggestion I particularly liked (which happens to avoid claiming perfectly valid characters - asterisks - as special characters) was to adopt the syntax used in the django ORM's filter function: ?name__startswith=Fred ?name__istartswith=fred ?name__endswith=Fred ?name__iendswith=fred ?name__contains=Fred ?name__icontains=fred This probably represents the immediately useful subset of parameters for us, but for more: https://docs.djangoproject.com/en/dev/topics/db/queries/ > Henry > > On 13 Aug 2013, at 04:40, Adam Young wrote: > > On 08/12/2013 09:22 PM, Miller, Mark M (EB SW Cloud - R&D - Corvallis) > wrote: > > The main reason I use user lists (i.e. keystone user-list) is to get the > list of usernames/IDs for other keystone commands. I do not see the value > of showing all of the users in an LDAP server when they are not part of the > keystone database (i.e. do not have roles assigned to them). Performing a > “keystone user-list” command against the HP Enterprise Directory locks up > keystone for about 1 ½ hours in that it will not perform any other commands > until it is done. If it is decided that user lists are necessary, then at > a minimum they need to be paged to return control back to keystone for > another command. > > > We need a way to tell HP ED to limit the number of rows, and to do > filtering. > > We have a bug for the second part. I'll open one for the limit. > > **** > > ** ** > > Mark**** > > ** ** > > *From:* Adam Young [mailto:ayo...@redhat.com <ayo...@redhat.com>] > *Sent:* Monday, August 12, 2013 5:27 PM > *To:* openstack-dev@lists.openstack.org > *Subject:* Re: [openstack-dev] [keystone] Pagination**** > > ** ** > > On 08/12/2013 05:34 PM, Henry Nash wrote:**** > > Hi **** > > ** ** > > I'm working on extending the pagination into the backends. Right now, we > handle the pagination in the v3 controller class....and in fact it is > disabled right now and we return the whole list irrespective of whether > page/per-page is set in the query string, e.g.:**** > > Pagination is a broken concept. We should not be returning lists so long > that we need to paginate. Instead, we should have query limits, and > filters to refine the queries. > > Some people are doing full user lists against LDAP. I don't need to tell > you how broken that is. Why do we allow user-list at the Domain (or > unscoped level)? > > I'd argue that we should drop enumeration of objects in general, and > certainly limit the number of results that come back. Pagination in LDAP > requires cursors, and thus continuos connections from Keystone to > LDAP...this is not a scalable solution. > > Do we really need this? > > > > **** > > ** ** > > def *paginate*(cls, context, refs):**** > > *"""Paginates a list of references by page & per_page query > strings."""***** > > # FIXME(*dolph*): client needs to support pagination first**** > > return refs**** > > ** ** > > page = context[*'query_string'*].get(*'page'*, 1)**** > > per_page = context[*'query_string'*].get(*'per_page'*, 30)**** > > return refs[per_page * (page - 1):per_page * page]**** > > ** ** > > I wonder both for the V3 controller (which still needs to handle > pagination for backends that do not support it) and the backends that > do....whether we could use wether 'page' is defined in the query-string as > an indicator as to whether we should paginate or not? That way clients who > can handle it can ask for it, those that don'twill just get everything. * > *** > > ** ** > > Henry**** > > ** ** > > > > > **** > > _______________________________________________**** > > OpenStack-dev mailing list**** > > OpenStack-dev@lists.openstack.org**** > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev**** > > ** ** > > > _______________________________________________ > OpenStack-dev mailing > listOpenStack-dev@lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > -- -Dolph
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev