Re: [openstack-dev] [keystone] Pagination

[Dolph Mathews]

The way paginated links are defined by the v3 API (via `next` and
`previous` links), it can be completely up to the driver as to what the
query parameters look like. So, the client shouldn't have (nor require) any
knowledge of how to build query parameters for pagination. It just needs to
follow the links it's given.

'page' and 'per_page' are trivial for the controller to implement (as it's
just slicing into an list... as shown)... so that's a reasonable default
behavior (for when a driver does not support pagination). However, if the
underlying driver DOES support pagination, it should provide a way for the
controller to ask for the query parameters required to specify the
next/previous links (so, one driver could return `marker` and `limit`
parameters while another only exposes the `page` number, but not quantity
`per_page`).


On Mon, Aug 12, 2013 at 4:34 PM, Henry Nash wrote:

> Hi
>
> I'm working on extending the pagination into the backends.  Right now, we
> handle the pagination in the v3 controller classand in fact it is
> disabled right now and we return the whole list irrespective of whether
> page/per-page is set in the query string, e.g.:
>
> def *paginate*(cls, context, refs):
> *"""Paginates a list of references by page & per_page query
> strings."""*
> # FIXME(dolph): client needs to support pagination first
> return refs
>
> page = context[*'query_string'*].get(*'page'*, 1)
> per_page = context[*'query_string'*].get(*'per_page'*, 30)
> return refs[per_page * (page - 1):per_page * page]
>
> I wonder both for the V3 controller (which still needs to handle
> pagination for backends that do not support it) and the backends that
> dowhether we could use wether 'page' is defined in the query-string as
> an indicator as to whether we should paginate or not?  That way clients who
> can handle it can ask for it, those that don'twill just get everything.
>
> Henry
>
>


-- 

-Dolph
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] Pagination

[Adam Young]

On 08/12/2013 05:34 PM, Henry Nash wrote:

Hi

I'm working on extending the pagination into the backends.  Right now, 
we handle the pagination in the v3 controller classand in fact it 
is disabled right now and we return the whole list irrespective of 
whether page/per-page is set in the query string, e.g.:
Pagination is a broken concept. We should not be returning lists so long 
that we need to paginate.  Instead, we should have query limits, and 
filters to refine the queries.


Some people are doing full user lists against LDAP.  I don't need to 
tell you how broken that is.  Why do we allow user-list at the Domain 
(or unscoped level)?


I'd argue that we should drop enumeration of objects in general, and 
certainly limit the number of results that come back.  Pagination in 
LDAP requires cursors, and thus continuos connections from Keystone to 
LDAP...this is not a scalable solution.


Do we really need this?




def *paginate*(cls, context, refs):
/"""Paginates a list of references by page & per_page query strings."""/
# FIXME(dolph): client needs to support pagination first
return refs

page = context[/'query_string'/].get(/'page'/, 1)
per_page = context[/'query_string'/].get(/'per_page'/, 30)
return refs[per_page * (page - 1):per_page * page]

I wonder both for the V3 controller (which still needs to handle 
pagination for backends that do not support it) and the backends that 
dowhether we could use wether 'page' is defined in the 
query-string as an indicator as to whether we should paginate or not? 
 That way clients who can handle it can ask for it, those that 
don'twill just get everything.


Henry



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] Pagination

[Jay Pipes]

On 08/12/2013 08:27 PM, Adam Young wrote:

On 08/12/2013 05:34 PM, Henry Nash wrote:

Hi

I'm working on extending the pagination into the backends.  Right now,
we handle the pagination in the v3 controller classand in fact it
is disabled right now and we return the whole list irrespective of
whether page/per-page is set in the query string, e.g.:

Pagination is a broken concept. We should not be returning lists so long
that we need to paginate.  Instead, we should have query limits, and
filters to refine the queries.

Some people are doing full user lists against LDAP.  I don't need to
tell you how broken that is.  Why do we allow user-list at the Domain
(or unscoped level)?

I'd argue that we should drop enumeration of objects in general, and
certainly limit the number of results that come back.  Pagination in
LDAP requires cursors, and thus continuos connections from Keystone to
LDAP...this is not a scalable solution.

Do we really need this?


Yes. It is very painful for operators right now to do any sort of 
administration of identity information when using the SQL backend. In 
Horizon, the users admin page takes forever and a day to load hundreds 
or thousands of user records (same for tenants). The CLI is similarly 
painful in production environments with thousands of user/tenants.


Best,
-jay


def *paginate*(cls, context, refs):
/"""Paginates a list of references by page & per_page query strings."""/
# FIXME(dolph): client needs to support pagination first
return refs

page = context[/'query_string'/].get(/'page'/, 1)
per_page = context[/'query_string'/].get(/'per_page'/, 30)
return refs[per_page * (page - 1):per_page * page]

I wonder both for the V3 controller (which still needs to handle
pagination for backends that do not support it) and the backends that
dowhether we could use wether 'page' is defined in the
query-string as an indicator as to whether we should paginate or not?
 That way clients who can handle it can ask for it, those that
don'twill just get everything.

Henry



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] Pagination

[Adam Young]

On 08/12/2013 08:33 PM, Jay Pipes wrote:

On 08/12/2013 08:27 PM, Adam Young wrote:

On 08/12/2013 05:34 PM, Henry Nash wrote:

Hi

I'm working on extending the pagination into the backends. Right now,
we handle the pagination in the v3 controller classand in fact it
is disabled right now and we return the whole list irrespective of
whether page/per-page is set in the query string, e.g.:

Pagination is a broken concept. We should not be returning lists so long
that we need to paginate.  Instead, we should have query limits, and
filters to refine the queries.

Some people are doing full user lists against LDAP.  I don't need to
tell you how broken that is.  Why do we allow user-list at the Domain
(or unscoped level)?

I'd argue that we should drop enumeration of objects in general, and
certainly limit the number of results that come back. Pagination in
LDAP requires cursors, and thus continuos connections from Keystone to
LDAP...this is not a scalable solution.

Do we really need this?


Yes. It is very painful for operators right now to do any sort of 
administration of identity information when using the SQL backend. In 
Horizon, the users admin page takes forever and a day to load hundreds 
or thousands of user records (same for tenants). The CLI is similarly 
painful in production environments with thousands of user/tenants.
Not arguing that it is not broken.   I would argue that there is 
something broken with out workflows. Pagination is not the answer. Not 
asking for the entire user list is the answer.
Honestly, if the list is 100K long, you are not going to page through 
it.  You need a better search filter.  Lets limit the number of results 
shown, and figure out how to correctly filter results.




Best,
-jay


def *paginate*(cls, context, refs):
/"""Paginates a list of references by page & per_page query 
strings."""/

# FIXME(dolph): client needs to support pagination first
return refs

page = context[/'query_string'/].get(/'page'/, 1)
per_page = context[/'query_string'/].get(/'per_page'/, 30)
return refs[per_page * (page - 1):per_page * page]

I wonder both for the V3 controller (which still needs to handle
pagination for backends that do not support it) and the backends that
dowhether we could use wether 'page' is defined in the
query-string as an indicator as to whether we should paginate or not?
 That way clients who can handle it can ask for it, those that
don'twill just get everything.

Henry



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] Pagination

[Jamie Lennox]

I'm not sure where it would make sense within the API to return the name
of the page/per_page variables to the client that doesn't involve having
already issued the call (ie returning the names within the links box
means you've already issued the query). If we standardize on the
page/per_page combination then this can be handled at the controller
level then the driver has permission to simply ignore it - or have the
controller do the slicing after the driver has returned.

To weigh in on the other question i think it should be checked that page
is an integer, unless per_page is specified in which case default to 1.

For example: 

GET /v3/users?page=

I would expect to return all users as page is not set. However: 

GET /v3/users?per_page=30

As per_page is useless without a page i think we can default to page=1.

As an aside are we indexing from 1?

On Mon, 2013-08-12 at 19:05 -0500, Dolph Mathews wrote:
> The way paginated links are defined by the v3 API (via `next` and
> `previous` links), it can be completely up to the driver as to what
> the query parameters look like. So, the client shouldn't have (nor
> require) any knowledge of how to build query parameters for
> pagination. It just needs to follow the links it's given.
> 
> 
> 'page' and 'per_page' are trivial for the controller to implement (as
> it's just slicing into an list... as shown)... so that's a reasonable
> default behavior (for when a driver does not support pagination).
> However, if the underlying driver DOES support pagination, it should
> provide a way for the controller to ask for the query parameters
> required to specify the next/previous links (so, one driver could
> return `marker` and `limit` parameters while another only exposes the
> `page` number, but not quantity `per_page`).
> 
> 
> On Mon, Aug 12, 2013 at 4:34 PM, Henry Nash
>  wrote:
> Hi
> 
> 
> I'm working on extending the pagination into the backends.
>  Right now, we handle the pagination in the v3 controller
> classand in fact it is disabled right now and we return
> the whole list irrespective of whether page/per-page is set in
> the query string, e.g.:
> 
> 
> def paginate(cls, context, refs):
> """Paginates a list of references by page & per_page
> query strings."""
> # FIXME(dolph): client needs to support pagination
> first
> return refs
> 
> 
> page = context['query_string'].get('page', 1)
> per_page = context['query_string'].get('per_page', 30)
> return refs[per_page * (page - 1):per_page * page]
> 
> 
> I wonder both for the V3 controller (which still needs to
> handle pagination for backends that do not support it) and the
> backends that dowhether we could use wether 'page' is
> defined in the query-string as an indicator as to whether we
> should paginate or not?  That way clients who can handle it
> can ask for it, those that don'twill just get everything.  
> 
> 
> Henry
> 
> 
> 
> 
> 
> 
> -- 
> 
> 
> -Dolph
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] Pagination

[Miller, Mark M (EB SW Cloud - R&D - Corvallis)]

The main reason I use user lists (i.e. keystone user-list) is to get the list 
of usernames/IDs for other keystone commands. I do not see the value of showing 
all of the users in an LDAP server when they are not part of the keystone 
database (i.e. do not have roles assigned to them). Performing a "keystone 
user-list" command against the HP Enterprise Directory locks up keystone for 
about 1 ½ hours in that it will not perform any other commands until it is 
done.  If it is decided that user lists are necessary, then at a minimum they 
need to be paged to return control back to keystone for another command.

Mark

From: Adam Young [mailto:ayo...@redhat.com]
Sent: Monday, August 12, 2013 5:27 PM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [keystone] Pagination

On 08/12/2013 05:34 PM, Henry Nash wrote:
Hi

I'm working on extending the pagination into the backends.  Right now, we 
handle the pagination in the v3 controller classand in fact it is disabled 
right now and we return the whole list irrespective of whether page/per-page is 
set in the query string, e.g.:
Pagination is a broken concept. We should not be returning lists so long that 
we need to paginate.  Instead, we should have query limits, and filters to 
refine the queries.

Some people are doing full user lists against LDAP.  I don't need to tell you 
how broken that is.  Why do we allow user-list at the Domain (or unscoped 
level)?

I'd argue that we should drop enumeration of objects in general, and certainly 
limit the number of results that come back.  Pagination in LDAP requires 
cursors, and thus continuos connections from Keystone to LDAP...this is not a 
scalable solution.

Do we really need this?




def paginate(cls, context, refs):
"""Paginates a list of references by page & per_page query strings."""
# FIXME(dolph): client needs to support pagination first
return refs

page = context['query_string'].get('page', 1)
per_page = context['query_string'].get('per_page', 30)
return refs[per_page * (page - 1):per_page * page]

I wonder both for the V3 controller (which still needs to handle pagination for 
backends that do not support it) and the backends that dowhether we could 
use wether 'page' is defined in the query-string as an indicator as to whether 
we should paginate or not?  That way clients who can handle it can ask for it, 
those that don'twill just get everything.

Henry





___

OpenStack-dev mailing list

OpenStack-dev@lists.openstack.org<mailto:OpenStack-dev@lists.openstack.org>

http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] Pagination

[Dolph Mathews]

On Mon, Aug 12, 2013 at 7:51 PM, Jamie Lennox  wrote:

> I'm not sure where it would make sense within the API to return the name
> of the page/per_page variables to the client that doesn't involve having
> already issued the call (ie returning the names within the links box
> means you've already issued the query).


I think you're missing the point (and you're right: that wouldn't make
sense at all). The API client follows links. The controller builds links.
The driver defines it's own pagination interface to build related links.

If the client is forced to understand the pagination interface then the
abstraction is broken.


> If we standardize on the
> page/per_page combination


There doesn't need to be a "standard."


> then this can be handled at the controller
> level then the driver has permission to simply ignore it - or have the
> controller do the slicing after the driver has returned.
>

Correct. This sort of "default" pagination can be implemented by the
manager, and overridden by a specific driver.


>
> To weigh in on the other question i think it should be checked that page
> is an integer, unless per_page is specified in which case default to 1.
>
> For example:
>
> GET /v3/users?page=
>
> I would expect to return all users as page is not set. However:
>
> GET /v3/users?per_page=30
>
> As per_page is useless without a page i think we can default to page=1.
>
> As an aside are we indexing from 1?
>

Rhetorical: why not index from -1 and count in base 64? This is all
arbitrary and can vary by driver.


>
> On Mon, 2013-08-12 at 19:05 -0500, Dolph Mathews wrote:
> > The way paginated links are defined by the v3 API (via `next` and
> > `previous` links), it can be completely up to the driver as to what
> > the query parameters look like. So, the client shouldn't have (nor
> > require) any knowledge of how to build query parameters for
> > pagination. It just needs to follow the links it's given.
> >
> >
> > 'page' and 'per_page' are trivial for the controller to implement (as
> > it's just slicing into an list... as shown)... so that's a reasonable
> > default behavior (for when a driver does not support pagination).
> > However, if the underlying driver DOES support pagination, it should
> > provide a way for the controller to ask for the query parameters
> > required to specify the next/previous links (so, one driver could
> > return `marker` and `limit` parameters while another only exposes the
> > `page` number, but not quantity `per_page`).
> >
> >
> > On Mon, Aug 12, 2013 at 4:34 PM, Henry Nash
> >  wrote:
> > Hi
> >
> >
> > I'm working on extending the pagination into the backends.
> >  Right now, we handle the pagination in the v3 controller
> > classand in fact it is disabled right now and we return
> > the whole list irrespective of whether page/per-page is set in
> > the query string, e.g.:
> >
> >
> > def paginate(cls, context, refs):
> > """Paginates a list of references by page & per_page
> > query strings."""
> > # FIXME(dolph): client needs to support pagination
> > first
> > return refs
> >
> >
> > page = context['query_string'].get('page', 1)
> > per_page = context['query_string'].get('per_page', 30)
> > return refs[per_page * (page - 1):per_page * page]
> >
> >
> > I wonder both for the V3 controller (which still needs to
> > handle pagination for backends that do not support it) and the
> > backends that dowhether we could use wether 'page' is
> > defined in the query-string as an indicator as to whether we
> > should paginate or not?  That way clients who can handle it
> > can ask for it, those that don'twill just get everything.
> >
> >
> > Henry
> >
> >
> >
> >
> >
> >
> > --
> >
> >
> > -Dolph
> > ___
> > OpenStack-dev mailing list
> > OpenStack-dev@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 

-Dolph
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] Pagination

[Adam Young]

On 08/12/2013 09:22 PM, Miller, Mark M (EB SW Cloud - R&D - Corvallis) 
wrote:


The main reason I use user lists (i.e. keystone user-list) is to get 
the list of usernames/IDs for other keystone commands. I do not see 
the value of showing all of the users in an LDAP server when they are 
not part of the keystone database (i.e. do not have roles assigned to 
them). Performing a "keystone user-list" command against the HP 
Enterprise Directory locks up keystone for about 1 ½ hours in that it 
will not perform any other commands until it is done.  If it is 
decided that user lists are necessary, then at a minimum they need to 
be paged to return control back to keystone for another command.




We need a way to tell HP ED to limit the number of rows, and to do 
filtering.


We have a bug for the second part.  I'll open one for the limit.


Mark

*From:*Adam Young [mailto:ayo...@redhat.com]
*Sent:* Monday, August 12, 2013 5:27 PM
*To:* openstack-dev@lists.openstack.org
*Subject:* Re: [openstack-dev] [keystone] Pagination

On 08/12/2013 05:34 PM, Henry Nash wrote:

Hi

I'm working on extending the pagination into the backends.  Right
now, we handle the pagination in the v3 controller classand in
fact it is disabled right now and we return the whole list
irrespective of whether page/per-page is set in the query string,
e.g.:

Pagination is a broken concept. We should not be returning lists so 
long that we need to paginate. Instead, we should have query limits, 
and filters to refine the queries.


Some people are doing full user lists against LDAP.  I don't need to 
tell you how broken that is.  Why do we allow user-list at the Domain 
(or unscoped level)?


I'd argue that we should drop enumeration of objects in general, and 
certainly limit the number of results that come back.  Pagination in 
LDAP requires cursors, and thus continuos connections from Keystone to 
LDAP...this is not a scalable solution.


Do we really need this?



def*paginate*(cls, context, refs):

/"""Paginates a list of references by page & per_page query strings."""/

# FIXME(_dolph_): client needs to support pagination first

returnrefs

  page = context[/'query_string'/].get(/'page'/, 1)

  per_page = context[/'query_string'/].get(/'per_page'/, 30)

returnrefs[per_page * (page - 1):per_page * page]

I wonder both for the V3 controller (which still needs to handle 
pagination for backends that do not support it) and the backends that 
dowhether we could use wether 'page' is defined in the 
query-string as an indicator as to whether we should paginate or not? 
 That way clients who can handle it can ask for it, those that 
don'twill just get everything.


Henry




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org  <mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] Pagination

[Yee, Guang]

Passing the query parameters, whatever they are, into the driver if the
given driver supports pagination and allowing the driver to override the
manager default pagination functionality seem reasonable to me.

 

 

Guang

 

 

From: Dolph Mathews [mailto:dolph.math...@gmail.com] 
Sent: Monday, August 12, 2013 8:22 PM
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] [keystone] Pagination

 

 

On Mon, Aug 12, 2013 at 7:51 PM, Jamie Lennox  wrote:

I'm not sure where it would make sense within the API to return the name
of the page/per_page variables to the client that doesn't involve having
already issued the call (ie returning the names within the links box
means you've already issued the query).

 

I think you're missing the point (and you're right: that wouldn't make sense
at all). The API client follows links. The controller builds links. The
driver defines it's own pagination interface to build related links.

 

If the client is forced to understand the pagination interface then the
abstraction is broken.

 

If we standardize on the
page/per_page combination

 

There doesn't need to be a "standard."

 

then this can be handled at the controller
level then the driver has permission to simply ignore it - or have the
controller do the slicing after the driver has returned.

 

Correct. This sort of "default" pagination can be implemented by the
manager, and overridden by a specific driver.

 


To weigh in on the other question i think it should be checked that page
is an integer, unless per_page is specified in which case default to 1.

For example:

GET /v3/users?page=

I would expect to return all users as page is not set. However:

GET /v3/users?per_page=30

As per_page is useless without a page i think we can default to page=1.

As an aside are we indexing from 1?

 

Rhetorical: why not index from -1 and count in base 64? This is all
arbitrary and can vary by driver.

 


On Mon, 2013-08-12 at 19:05 -0500, Dolph Mathews wrote:
> The way paginated links are defined by the v3 API (via `next` and
> `previous` links), it can be completely up to the driver as to what
> the query parameters look like. So, the client shouldn't have (nor
> require) any knowledge of how to build query parameters for
> pagination. It just needs to follow the links it's given.
>
>
> 'page' and 'per_page' are trivial for the controller to implement (as
> it's just slicing into an list... as shown)... so that's a reasonable
> default behavior (for when a driver does not support pagination).
> However, if the underlying driver DOES support pagination, it should
> provide a way for the controller to ask for the query parameters
> required to specify the next/previous links (so, one driver could
> return `marker` and `limit` parameters while another only exposes the
> `page` number, but not quantity `per_page`).
>
>
> On Mon, Aug 12, 2013 at 4:34 PM, Henry Nash
>  wrote:
> Hi
>
>
> I'm working on extending the pagination into the backends.
>  Right now, we handle the pagination in the v3 controller
> classand in fact it is disabled right now and we return
> the whole list irrespective of whether page/per-page is set in
> the query string, e.g.:
>
>
> def paginate(cls, context, refs):
> """Paginates a list of references by page & per_page
> query strings."""
> # FIXME(dolph): client needs to support pagination
> first
> return refs
>
>
> page = context['query_string'].get('page', 1)
> per_page = context['query_string'].get('per_page', 30)
> return refs[per_page * (page - 1):per_page * page]
>
>
> I wonder both for the V3 controller (which still needs to
> handle pagination for backends that do not support it) and the
> backends that dowhether we could use wether 'page' is
> defined in the query-string as an indicator as to whether we
> should paginate or not?  That way clients who can handle it
> can ask for it, those that don'twill just get everything.
>
>
> Henry
>
>
>
>
>
>
> --
>
>
> -Dolph

> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev





 

-- 

 

-Dolph 



smime.p7s
Description: S/MIME cryptographic signature
___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] Pagination

[Henry Nash]

Hi

So few comebacks to the various comments:

1) While I understand the idea that a client would follow the next/prev links 
returned in collections, I wasn't aware that we considered 'page'/'per-page' as 
not standardized.   We list these explicitly throughout the identity API spec 
(look in each List 'entity' example).  How I imagined it would work would be:

a) If a client did not include 'page' in the url we would not paginate
b) Once we are paginating, a client can either build the next/prevs urls 
themselves if they want (by incrementing/decrementing the page number), or just 
follow the next/prev links (which come with the appropriate 'page=x' in them) 
returned in the collection which saves them having to do this.
c) Regarding implementation, the controller would continue to be able to 
paginate on behalf of drivers that couldn't, but those paginate-aware drivers 
would take over that capability (and indicate this to the controller the state 
of the pagination so that it can build the correct next/prev links)

2) On the subject of huge enumerates, options are:
a) Support a backend manager scoped (i.e. identity/assignent/token) limit in 
the conf file which would be honored by drivers.  Assuming that you set this 
larger than your pagination limit, this would make sense whether your driver is 
paginating or not in terms of minimizing the delay in responding data as well 
as not messing up pagination.  In the non-paginated case when we hit the limit, 
should we indicate this to the client?  Maybe a 206 return code?  Although i) 
not quite sure that meets http standards, and ii) would we break a bunch of 
clients by doing this?
b) We scrap the whole idea of pagination, and just set a conf limit as in 2a).  
To make this work of course, we must implement any defined filters in the 
backend (otherwise we still end up with today's performance problems - remember 
that today, in general,  filtering is done in the controller on a full 
enumeration of the entities in question).  I was planning to implement this 
backend filtering anyway as part of (or on top of) my change, since we are 
holding (at least one of) our hands behind our backs right now by not doing so. 
 And our filters need to be powerful, do we support wildcards for example, e.g. 
GET /users?name = fred*  ?
 
Henry

On 13 Aug 2013, at 04:40, Adam Young wrote:

> On 08/12/2013 09:22 PM, Miller, Mark M (EB SW Cloud - R&D - Corvallis) wrote:
>> The main reason I use user lists (i.e. keystone user-list) is to get the 
>> list of usernames/IDs for other keystone commands. I do not see the value of 
>> showing all of the users in an LDAP server when they are not part of the 
>> keystone database (i.e. do not have roles assigned to them). Performing a 
>> “keystone user-list” command against the HP Enterprise Directory locks up 
>> keystone for about 1 ½ hours in that it will not perform any other commands 
>> until it is done.  If it is decided that user lists are necessary, then at a 
>> minimum they need to be paged to return control back to keystone for another 
>> command.
> 
> We need a way to tell HP ED to limit the number of rows, and to do filtering.
> 
> We have a bug for the second part.  I'll open one for the limit.
> 
>>  
>> Mark
>>  
>> From: Adam Young [mailto:ayo...@redhat.com] 
>> Sent: Monday, August 12, 2013 5:27 PM
>> To: openstack-dev@lists.openstack.org
>> Subject: Re: [openstack-dev] [keystone] Pagination
>>  
>> On 08/12/2013 05:34 PM, Henry Nash wrote:
>> Hi
>>  
>> I'm working on extending the pagination into the backends.  Right now, we 
>> handle the pagination in the v3 controller classand in fact it is 
>> disabled right now and we return the whole list irrespective of whether 
>> page/per-page is set in the query string, e.g.:
>> Pagination is a broken concept. We should not be returning lists so long 
>> that we need to paginate.  Instead, we should have query limits, and filters 
>> to refine the queries.
>> 
>> Some people are doing full user lists against LDAP.  I don't need to tell 
>> you how broken that is.  Why do we allow user-list at the Domain (or 
>> unscoped level)?  
>> 
>> I'd argue that we should drop enumeration of objects in general, and 
>> certainly limit the number of results that come back.  Pagination in LDAP 
>> requires cursors, and thus continuos connections from Keystone to 
>> LDAP...this is not a scalable solution.
>> 
>> Do we really need this?
>> 
>> 
>> 
>>  
>> def paginate(cls, context, refs):
>> """Paginates a list of references by page & per_page query 
>> strings.""

Re: [openstack-dev] [keystone] Pagination

[Dolph Mathews]

On Tue, Aug 13, 2013 at 3:10 AM, Henry Nash wrote:

> Hi
>
> So few comebacks to the various comments:
>
> 1) While I understand the idea that a client would follow the next/prev
> links returned in collections, I wasn't aware that we considered
> 'page'/'per-page' as not standardized. We list these explicitly throughout
> the identity API spec (look in each List 'entity' example).
>

They were essentially relics from a very early draft of the spec that were
thoughtlessly copy/pasted around (I'm guilty of this myself)... they were
recently cleaned up and removed from the spec.


> How I imagined it would work would be:
>
> a) If a client did not include 'page' in the url we would not paginate
>

Make that a deployment option? per_page could simply default to a very high
value.


> b) Once we are paginating, a client can either build the next/prevs urls
> themselves if they want (by incrementing/decrementing the page number), or
> just follow the next/prev links (which come with the appropriate 'page=x'
> in them) returned in the collection which saves them having to do this.
>

I'm obviously very opposed to this because it unreasonably forces a single
approach to pagination across all drivers.


> c) Regarding implementation, the controller would continue to be able to
> paginate on behalf of drivers that couldn't, but those paginate-aware
> drivers would take over that capability (and indicate this to the
> controller the state of the pagination so that it can build the correct
> next/prev links)
>
> 2) On the subject of huge enumerates, options are:
> a) Support a backend manager scoped (i.e. identity/assignent/token) limit
> in the conf file which would be honored by drivers.  Assuming that you set
> this larger than your pagination limit, this would make sense whether your
> driver is paginating or not in terms of minimizing the delay in responding
> data as well as not messing up pagination.  In the non-paginated case when
> we hit the limit, should we indicate this to the client?  Maybe a 206
> return code?  Although i) not quite sure that meets http standards, and ii)
> would we break a bunch of clients by doing this?
>

I'm not clear on what kind of limit you're referring to? A 206 sounds
unexpected for this use case though.


> b) We scrap the whole idea of pagination, and just set a conf limit as in
> 2a).  To make this work of course, we must implement any defined filters in
> the backend (otherwise we still end up with today's performance problems -
> remember that today, in general,  filtering is done in the controller on a
> full enumeration of the entities in question).  I was planning to implement
> this backend filtering anyway as part of (or on top of) my change, since we
> are holding (at least one of) our hands behind our backs right now by not
> doing so.  And our filters need to be powerful, do we support wildcards for
> example, e.g. GET /users?name = fred*  ?
>

There were some discussions on this topic from about a year ago that I'd
love to continue. I don't want to invent a new "language," but we do need
to settle on an approach that we can apply across a wide variety of
backends. That probably means keeping it very simple (like your example).
Asterisks need to be URL encoded, though. One suggestion I particularly
liked (which happens to avoid claiming perfectly valid characters -
asterisks - as special characters) was to adopt the syntax used in the
django ORM's filter function:

  ?name__startswith=Fred
  ?name__istartswith=fred
  ?name__endswith=Fred
  ?name__iendswith=fred
  ?name__contains=Fred
  ?name__icontains=fred

This probably represents the immediately useful subset of parameters for
us, but for more:

  https://docs.djangoproject.com/en/dev/topics/db/queries/


> Henry
>
> On 13 Aug 2013, at 04:40, Adam Young wrote:
>
>  On 08/12/2013 09:22 PM, Miller, Mark M (EB SW Cloud - R&D - Corvallis)
> wrote:
>
> The main reason I use user lists (i.e. keystone user-list) is to get the
> list of usernames/IDs for other keystone commands. I do not see the value
> of showing all of the users in an LDAP server when they are not part of the
> keystone database (i.e. do not have roles assigned to them). Performing a
> “keystone user-list” command against the HP Enterprise Directory locks up
> keystone for about 1 ½ hours in that it will not perform any other commands
> until it is done.  If it is decided that user lists are necessary, then at
> a minimum they need to be paged to return control back to keystone for
> another command.
>
>
> We need a way to tell HP ED to limit the number of rows, and to do
> filtering.
>
> We have a bug for the second part.  I'll open one f

Re: [openstack-dev] [keystone] Pagination

[Henry Nash]

red
>   ?name__istartswith=fred
>   ?name__endswith=Fred
>   ?name__iendswith=fred
>   ?name__contains=Fred
>   ?name__icontains=fred
> 
> This probably represents the immediately useful subset of parameters for us, 
> but for more:
> 
>   https://docs.djangoproject.com/en/dev/topics/db/queries/
> 
>  
> Henry
> 
> On 13 Aug 2013, at 04:40, Adam Young wrote:
> 
>> On 08/12/2013 09:22 PM, Miller, Mark M (EB SW Cloud - R&D - Corvallis) wrote:
>>> The main reason I use user lists (i.e. keystone user-list) is to get the 
>>> list of usernames/IDs for other keystone commands. I do not see the value 
>>> of showing all of the users in an LDAP server when they are not part of the 
>>> keystone database (i.e. do not have roles assigned to them). Performing a 
>>> “keystone user-list” command against the HP Enterprise Directory locks up 
>>> keystone for about 1 ½ hours in that it will not perform any other commands 
>>> until it is done.  If it is decided that user lists are necessary, then at 
>>> a minimum they need to be paged to return control back to keystone for 
>>> another command.
>>> 
>> 
>> We need a way to tell HP ED to limit the number of rows, and to do filtering.
>> 
>> We have a bug for the second part.  I'll open one for the limit.
>> 
>>>  
>>> 
>>> Mark
>>> 
>>>  
>>> 
>>> From: Adam Young [mailto:ayo...@redhat.com] 
>>> Sent: Monday, August 12, 2013 5:27 PM
>>> To: openstack-dev@lists.openstack.org
>>> Subject: Re: [openstack-dev] [keystone] Pagination
>>> 
>>>  
>>> 
>>> On 08/12/2013 05:34 PM, Henry Nash wrote:
>>> 
>>> Hi
>>> 
>>>  
>>> 
>>> I'm working on extending the pagination into the backends.  Right now, we 
>>> handle the pagination in the v3 controller classand in fact it is 
>>> disabled right now and we return the whole list irrespective of whether 
>>> page/per-page is set in the query string, e.g.:
>>> 
>>> Pagination is a broken concept. We should not be returning lists so long 
>>> that we need to paginate.  Instead, we should have query limits, and 
>>> filters to refine the queries.
>>> 
>>> Some people are doing full user lists against LDAP.  I don't need to tell 
>>> you how broken that is.  Why do we allow user-list at the Domain (or 
>>> unscoped level)?  
>>> 
>>> I'd argue that we should drop enumeration of objects in general, and 
>>> certainly limit the number of results that come back.  Pagination in LDAP 
>>> requires cursors, and thus continuos connections from Keystone to 
>>> LDAP...this is not a scalable solution.
>>> 
>>> Do we really need this?
>>> 
>>> 
>>> 
>>> 
>>>  
>>> 
>>> def paginate(cls, context, refs):
>>> 
>>> """Paginates a list of references by page & per_page query 
>>> strings."""
>>> 
>>> # FIXME(dolph): client needs to support pagination first
>>> 
>>> return refs
>>> 
>>>  
>>> 
>>> page = context['query_string'].get('page', 1)
>>> 
>>> per_page = context['query_string'].get('per_page', 30)
>>> 
>>> return refs[per_page * (page - 1):per_page * page]
>>> 
>>>  
>>> 
>>> I wonder both for the V3 controller (which still needs to handle pagination 
>>> for backends that do not support it) and the backends that dowhether we 
>>> could use wether 'page' is defined in the query-string as an indicator as 
>>> to whether we should paginate or not?  That way clients who can handle it 
>>> can ask for it, those that don'twill just get everything.  
>>> 
>>>  
>>> 
>>> Henry
>>> 
>>>  
>>> 
>>> 
>>> 
>>> 
>>> 
>>> ___
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>  
>>> 
>>> 
>>> 
>>> ___
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> 
> -- 
> 
> -Dolph
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] Pagination

[Jay Pipes]

On 08/13/2013 03:05 AM, Yee, Guang wrote:

Passing the query parameters, whatever they are, into the driver if the
given driver supports pagination and allowing the driver to override the
manager default pagination functionality seem reasonable to me.


Please do use the standards that are supported in other OpenStack 
services already: limit, marker, sort_key and sort_dir.


Pagination is meaningless without a sort key and direction, so picking a 
sensible default for user/project records is good. I'd go with either 
created_at (what Glance/Nova/Cinder use..) or with the user/project UUID.


The Glance DB API pagination is well-documented and clean [1]. I highly 
recommend it as a starting point.


Nova uses the same marker/limit/sort_key/sort_dir options for queries 
that it allows pagination on. An example is the 
instance_get_all_by_filters() call [2].


Cinder uses the same marker/limit/sort_key/sort_dir options for query 
pagination as well. [3]


Finally, I'd consider supporting the standard change-since parameter for 
listing operations. Both Nova [4] and Glance [5] support the parameter, 
which is useful for tools that poll the APIs for "new" events/records.


In short, go with what is already a standard in the other projects...

Best,
-jay

[1] 
https://github.com/openstack/glance/blob/master/glance/db/sqlalchemy/api.py#L429
[2] 
https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L1709
[3] 
https://github.com/openstack/cinder/blob/master/cinder/common/sqlalchemyutils.py#L33
[4] 
https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L1766
[5] 
https://github.com/openstack/glance/blob/master/glance/db/sqlalchemy/api.py#L618





___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] Pagination

[Lyle, David (Cloud Services)]

The marker/limit pagination scheme is inferior.  The use of page/page_size 
allows access to arbitrary pages, whereas limit/marker only allows forward 
progress.   In Horizon's use case, with page/page_size we can provide the user 
access to any page they have already visited, rather than just the previous 
page (using prev/next links returned in the response).  

-David

On 08/13/2013 10:29 AM, Pipes, Jay wrote:

>On 08/13/2013 03:05 AM, Yee, Guang wrote:
>> Passing the query parameters, whatever they are, into the driver if 
>> the given driver supports pagination and allowing the driver to 
>> override the manager default pagination functionality seem reasonable to me.

>Please do use the standards that are supported in other OpenStack services 
>already: limit, marker, sort_key and sort_dir.

>Pagination is meaningless without a sort key and direction, so picking a 
>sensible default for user/project records is good. I'd go with either 
>created_at (what Glance/Nova/Cinder use..) or with the user/project >UUID.

>The Glance DB API pagination is well-documented and clean [1]. I highly 
>recommend it as a starting point.

>Nova uses the same marker/limit/sort_key/sort_dir options for queries that it 
>allows pagination on. An example is the
>instance_get_all_by_filters() call [2].

>Cinder uses the same marker/limit/sort_key/sort_dir options for query 
>pagination as well. [3]

>Finally, I'd consider supporting the standard change-since parameter for 
>listing operations. Both Nova [4] and Glance [5] support the parameter, which 
>is useful for tools that poll the APIs for "new" >events/records.

>In short, go with what is already a standard in the other projects...

>Best,
>-jay

>[1]
>https://github.com/openstack/glance/blob/master/glance/db/sqlalchemy/api.py#L429
>[2]
>https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L1709
>[3]
>https://github.com/openstack/cinder/blob/master/cinder/common/sqlalchemyutils.py#L33
>[4]
>https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L1766
>[5]
>https://github.com/openstack/glance/blob/master/glance/db/sqlalchemy/api.py#L618




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] Pagination

[Jay Pipes]

On 08/13/2013 12:55 PM, Lyle, David (Cloud Services) wrote:

The marker/limit pagination scheme is inferior.


A bold statement that flies in the face of experience and the work 
already done in all the other projects.


>The use of page/page_size allows access to arbitrary pages, whereas 
limit/marker only allows forward progress.


I don't see this as a particularly compelling use case considering the 
performance manifestations of using LIMIT OFFSET pagination.


>In Horizon's use case, with page/page_size we can provide the user 
access to any page they have already visited, rather than just the 
previous page (using prev/next links returned in the response).


I don't see this as a particularly useful thing, but in any case, you 
could still do this by keeping the markers for previous pages on the 
client (Horizon) side.


The point of marker/limit is to eliminate poor performance of LIMIT 
OFFSET queries and to force proper index usage in the listing queries.


You can see the original discussion about this from more than two years 
and even see where I was originally arguing for a LIMIT OFFSET strategy 
and was brought around to the current limit/marker strategy by the 
responses of Justin Santa Barbara and Greg Holt:


https://lists.launchpad.net/openstack/msg02548.html

Best,
-jay


-David

On 08/13/2013 10:29 AM, Pipes, Jay wrote:


On 08/13/2013 03:05 AM, Yee, Guang wrote:

Passing the query parameters, whatever they are, into the driver if
the given driver supports pagination and allowing the driver to
override the manager default pagination functionality seem reasonable to me.



Please do use the standards that are supported in other OpenStack services 
already: limit, marker, sort_key and sort_dir.



Pagination is meaningless without a sort key and direction, so picking a sensible 
default for user/project records is good. I'd go with either created_at (what 
Glance/Nova/Cinder use..) or with the user/project >UUID.



The Glance DB API pagination is well-documented and clean [1]. I highly 
recommend it as a starting point.



Nova uses the same marker/limit/sort_key/sort_dir options for queries that it 
allows pagination on. An example is the
instance_get_all_by_filters() call [2].



Cinder uses the same marker/limit/sort_key/sort_dir options for query 
pagination as well. [3]



Finally, I'd consider supporting the standard change-since parameter for listing operations. 
Both Nova [4] and Glance [5] support the parameter, which is useful for tools that poll the 
APIs for "new" >events/records.



In short, go with what is already a standard in the other projects...



Best,
-jay



[1]
https://github.com/openstack/glance/blob/master/glance/db/sqlalchemy/api.py#L429
[2]
https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L1709
[3]
https://github.com/openstack/cinder/blob/master/cinder/common/sqlalchemyutils.py#L33
[4]
https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L1766
[5]
https://github.com/openstack/glance/blob/master/glance/db/sqlalchemy/api.py#L618





___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] Pagination

[Henry Nash]

Jay,

Thanks for all the various links - most useful.

To map this into keystone context, if we were to follow this logic we would:

1) Support 'limit' and 'marker' (as opposed to 'page', 'page_szie', or anything 
else).  These would be standard, independent of what backing store keystone was 
using.  If neither are included in the url, then we return the first N entires, 
where N is defined by the cloud provider.  This ensures that for at least 
smaller deployments, non-pagination aware clients still work.  If either 
'limit' or 'marker' are specified, then we paginate, passing them down into the 
driver layer wherever possible to ensure efficiency (some drivers may not be 
able to support pagination, hence we will do this, inefficiently, at a higher 
layer)
2) If we are paginating at the driver level, we must, by definition, be doing 
all the filtering down there as well (otherwise it all gets mucked)
3) We should look at supporting the other standard options (sort order etc.), 
but irrespective of that, by definition, we must ensure that we any driver that 
is paginating must be getting is entries back in a consistent order (otherwise, 
again, pagination doesn't work reliably)

Henry
On 13 Aug 2013, at 18:10, Jay Pipes wrote:

> On 08/13/2013 12:55 PM, Lyle, David (Cloud Services) wrote:
>> The marker/limit pagination scheme is inferior.
> 
> A bold statement that flies in the face of experience and the work already 
> done in all the other projects.
> 
> >The use of page/page_size allows access to arbitrary pages, whereas 
> >limit/marker only allows forward progress.
> 
> I don't see this as a particularly compelling use case considering the 
> performance manifestations of using LIMIT OFFSET pagination.
> 
> >In Horizon's use case, with page/page_size we can provide the user access to 
> >any page they have already visited, rather than just the previous page 
> >(using prev/next links returned in the response).
> 
> I don't see this as a particularly useful thing, but in any case, you could 
> still do this by keeping the markers for previous pages on the client 
> (Horizon) side.
> 
> The point of marker/limit is to eliminate poor performance of LIMIT OFFSET 
> queries and to force proper index usage in the listing queries.
> 
> You can see the original discussion about this from more than two years and 
> even see where I was originally arguing for a LIMIT OFFSET strategy and was 
> brought around to the current limit/marker strategy by the responses of 
> Justin Santa Barbara and Greg Holt:
> 
> https://lists.launchpad.net/openstack/msg02548.html
> 
> Best,
> -jay
> 
>> -David
>> 
>> On 08/13/2013 10:29 AM, Pipes, Jay wrote:
>> 
>>> On 08/13/2013 03:05 AM, Yee, Guang wrote:
 Passing the query parameters, whatever they are, into the driver if
 the given driver supports pagination and allowing the driver to
 override the manager default pagination functionality seem reasonable to 
 me.
>> 
>>> Please do use the standards that are supported in other OpenStack services 
>>> already: limit, marker, sort_key and sort_dir.
>> 
>>> Pagination is meaningless without a sort key and direction, so picking a 
>>> sensible default for user/project records is good. I'd go with either 
>>> created_at (what Glance/Nova/Cinder use..) or with the user/project >UUID.
>> 
>>> The Glance DB API pagination is well-documented and clean [1]. I highly 
>>> recommend it as a starting point.
>> 
>>> Nova uses the same marker/limit/sort_key/sort_dir options for queries that 
>>> it allows pagination on. An example is the
>>> instance_get_all_by_filters() call [2].
>> 
>>> Cinder uses the same marker/limit/sort_key/sort_dir options for query 
>>> pagination as well. [3]
>> 
>>> Finally, I'd consider supporting the standard change-since parameter for 
>>> listing operations. Both Nova [4] and Glance [5] support the parameter, 
>>> which is useful for tools that poll the APIs for "new" >events/records.
>> 
>>> In short, go with what is already a standard in the other projects...
>> 
>>> Best,
>>> -jay
>> 
>>> [1]
>>> https://github.com/openstack/glance/blob/master/glance/db/sqlalchemy/api.py#L429
>>> [2]
>>> https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L1709
>>> [3]
>>> https://github.com/openstack/cinder/blob/master/cinder/common/sqlalchemyutils.py#L33
>>> [4]
>>> https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L1766
>>> [5]
>>> https://github.com/openstack/glance/blob/master/glance/db/sqlalchemy/api.py#L618
>> 
>> 
>> 
>> 
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
> 
> 
> ___
> OpenSt

Re: [openstack-dev] [keystone] Pagination

[Miller, Mark M (EB SW Cloud - R&D - Corvallis)]

I have been following this exchange of ideas on how to solve/implement 
pagination. I would ask you to keep in mind that a solution needs to take into 
account a split LDAP/SQL backend (you are not always dealing with a single 
Keystone SQL database). Having a split backend means that the query information 
is divided between both backends and that you may not have as much flexibility 
with the LDAP backend

Mark.

-Original Message-
From: Jay Pipes [mailto:jaypi...@gmail.com] 
Sent: Tuesday, August 13, 2013 10:10 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [keystone] Pagination

On 08/13/2013 12:55 PM, Lyle, David (Cloud Services) wrote:
> The marker/limit pagination scheme is inferior.

A bold statement that flies in the face of experience and the work already done 
in all the other projects.

 >The use of page/page_size allows access to arbitrary pages, whereas 
 >limit/marker only allows forward progress.

I don't see this as a particularly compelling use case considering the 
performance manifestations of using LIMIT OFFSET pagination.

 >In Horizon's use case, with page/page_size we can provide the user access to 
 >any page they have already visited, rather than just the previous page (using 
 >prev/next links returned in the response).

I don't see this as a particularly useful thing, but in any case, you could 
still do this by keeping the markers for previous pages on the client (Horizon) 
side.

The point of marker/limit is to eliminate poor performance of LIMIT OFFSET 
queries and to force proper index usage in the listing queries.

You can see the original discussion about this from more than two years and 
even see where I was originally arguing for a LIMIT OFFSET strategy and was 
brought around to the current limit/marker strategy by the responses of Justin 
Santa Barbara and Greg Holt:

https://lists.launchpad.net/openstack/msg02548.html

Best,
-jay

> -David
>
> On 08/13/2013 10:29 AM, Pipes, Jay wrote:
>
>> On 08/13/2013 03:05 AM, Yee, Guang wrote:
>>> Passing the query parameters, whatever they are, into the driver if 
>>> the given driver supports pagination and allowing the driver to 
>>> override the manager default pagination functionality seem reasonable to me.
>
>> Please do use the standards that are supported in other OpenStack services 
>> already: limit, marker, sort_key and sort_dir.
>
>> Pagination is meaningless without a sort key and direction, so picking a 
>> sensible default for user/project records is good. I'd go with either 
>> created_at (what Glance/Nova/Cinder use..) or with the user/project >UUID.
>
>> The Glance DB API pagination is well-documented and clean [1]. I highly 
>> recommend it as a starting point.
>
>> Nova uses the same marker/limit/sort_key/sort_dir options for queries 
>> that it allows pagination on. An example is the
>> instance_get_all_by_filters() call [2].
>
>> Cinder uses the same marker/limit/sort_key/sort_dir options for query 
>> pagination as well. [3]
>
>> Finally, I'd consider supporting the standard change-since parameter for 
>> listing operations. Both Nova [4] and Glance [5] support the parameter, 
>> which is useful for tools that poll the APIs for "new" >events/records.
>
>> In short, go with what is already a standard in the other projects...
>
>> Best,
>> -jay
>
>> [1]
>> https://github.com/openstack/glance/blob/master/glance/db/sqlalchemy/
>> api.py#L429
>> [2]
>> https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.
>> py#L1709
>> [3]
>> https://github.com/openstack/cinder/blob/master/cinder/common/sqlalch
>> emyutils.py#L33
>> [4]
>> https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.
>> py#L1766
>> [5]
>> https://github.com/openstack/glance/blob/master/glance/db/sqlalchemy/
>> api.py#L618
>
>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] Pagination

[Jay Pipes]

On 08/13/2013 01:51 PM, Miller, Mark M (EB SW Cloud - R&D - Corvallis) 
wrote:

I have been following this exchange of ideas on how to solve/implement 
pagination. I would ask you to keep in mind that a solution needs to take into 
account a split LDAP/SQL backend (you are not always dealing with a single 
Keystone SQL database). Having a split backend means that the query information 
is divided between both backends and that you may not have as much flexibility 
with the LDAP backend


Yes, absolutely understood and a good point.

In the case of engines that don't support filtering, ordering, or other 
DB-like operations, then a pagination implementation in the controller 
would have to be provided. Not efficient, but better than nothing.


-jay


Mark.

-Original Message-
From: Jay Pipes [mailto:jaypi...@gmail.com]
Sent: Tuesday, August 13, 2013 10:10 AM
To: openstack-dev@lists.openstack.org
Subject: Re: [openstack-dev] [keystone] Pagination

On 08/13/2013 12:55 PM, Lyle, David (Cloud Services) wrote:

The marker/limit pagination scheme is inferior.


A bold statement that flies in the face of experience and the work already done 
in all the other projects.

  >The use of page/page_size allows access to arbitrary pages, whereas 
limit/marker only allows forward progress.

I don't see this as a particularly compelling use case considering the 
performance manifestations of using LIMIT OFFSET pagination.

  >In Horizon's use case, with page/page_size we can provide the user access to 
any page they have already visited, rather than just the previous page (using 
prev/next links returned in the response).

I don't see this as a particularly useful thing, but in any case, you could 
still do this by keeping the markers for previous pages on the client (Horizon) 
side.

The point of marker/limit is to eliminate poor performance of LIMIT OFFSET 
queries and to force proper index usage in the listing queries.

You can see the original discussion about this from more than two years and 
even see where I was originally arguing for a LIMIT OFFSET strategy and was 
brought around to the current limit/marker strategy by the responses of Justin 
Santa Barbara and Greg Holt:

https://lists.launchpad.net/openstack/msg02548.html

Best,
-jay


-David

On 08/13/2013 10:29 AM, Pipes, Jay wrote:


On 08/13/2013 03:05 AM, Yee, Guang wrote:

Passing the query parameters, whatever they are, into the driver if
the given driver supports pagination and allowing the driver to
override the manager default pagination functionality seem reasonable to me.



Please do use the standards that are supported in other OpenStack services 
already: limit, marker, sort_key and sort_dir.



Pagination is meaningless without a sort key and direction, so picking a sensible 
default for user/project records is good. I'd go with either created_at (what 
Glance/Nova/Cinder use..) or with the user/project >UUID.



The Glance DB API pagination is well-documented and clean [1]. I highly 
recommend it as a starting point.



Nova uses the same marker/limit/sort_key/sort_dir options for queries
that it allows pagination on. An example is the
instance_get_all_by_filters() call [2].



Cinder uses the same marker/limit/sort_key/sort_dir options for query
pagination as well. [3]



Finally, I'd consider supporting the standard change-since parameter for listing operations. 
Both Nova [4] and Glance [5] support the parameter, which is useful for tools that poll the 
APIs for "new" >events/records.



In short, go with what is already a standard in the other projects...



Best,
-jay



[1]
https://github.com/openstack/glance/blob/master/glance/db/sqlalchemy/
api.py#L429
[2]
https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.
py#L1709
[3]
https://github.com/openstack/cinder/blob/master/cinder/common/sqlalch
emyutils.py#L33
[4]
https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.
py#L1766
[5]
https://github.com/openstack/glance/blob/master/glance/db/sqlalchemy/
api.py#L618





___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] Pagination

[Jay Pipes]

On 08/13/2013 01:43 PM, Henry Nash wrote:

Jay,

Thanks for all the various links - most useful.


No problem. It's an old problem in the OpenStack world that's been 
discussed many times before :) Might as well take advantage of prior 
art/discussions... I personally had a change of heart on the issue a 
couple years ago after the initial discussions.



To map this into keystone context, if we were to follow this logic we would:

1) Support 'limit' and 'marker' (as opposed to 'page', 'page_szie', or anything 
else).  These would be standard, independent of what backing store keystone was 
using.  If neither are included in the url, then we return the first N entires, 
where N is defined by the cloud provider.  This ensures that for at least 
smaller deployments, non-pagination aware clients still work.  If either 
'limit' or 'marker' are specified, then we paginate, passing them down into the 
driver layer wherever possible to ensure efficiency (some drivers may not be 
able to support pagination, hence we will do this, inefficiently, at a higher 
layer)
2) If we are paginating at the driver level, we must, by definition, be doing 
all the filtering down there as well (otherwise it all gets mucked)
3) We should look at supporting the other standard options (sort order etc.), 
but irrespective of that, by definition, we must ensure that we any driver that 
is paginating must be getting is entries back in a consistent order (otherwise, 
again, pagination doesn't work reliably)


Yup, all of the above meets matches my understanding. Filter support 
should come first, followed by paging-parameter pushdown to the engines.


Best,
-jay


On 13 Aug 2013, at 18:10, Jay Pipes wrote:


On 08/13/2013 12:55 PM, Lyle, David (Cloud Services) wrote:

The marker/limit pagination scheme is inferior.


A bold statement that flies in the face of experience and the work already done 
in all the other projects.


The use of page/page_size allows access to arbitrary pages, whereas 
limit/marker only allows forward progress.


I don't see this as a particularly compelling use case considering the 
performance manifestations of using LIMIT OFFSET pagination.


In Horizon's use case, with page/page_size we can provide the user access to 
any page they have already visited, rather than just the previous page (using 
prev/next links returned in the response).


I don't see this as a particularly useful thing, but in any case, you could 
still do this by keeping the markers for previous pages on the client (Horizon) 
side.

The point of marker/limit is to eliminate poor performance of LIMIT OFFSET 
queries and to force proper index usage in the listing queries.

You can see the original discussion about this from more than two years and 
even see where I was originally arguing for a LIMIT OFFSET strategy and was 
brought around to the current limit/marker strategy by the responses of Justin 
Santa Barbara and Greg Holt:

https://lists.launchpad.net/openstack/msg02548.html

Best,
-jay


-David

On 08/13/2013 10:29 AM, Pipes, Jay wrote:


On 08/13/2013 03:05 AM, Yee, Guang wrote:

Passing the query parameters, whatever they are, into the driver if
the given driver supports pagination and allowing the driver to
override the manager default pagination functionality seem reasonable to me.



Please do use the standards that are supported in other OpenStack services 
already: limit, marker, sort_key and sort_dir.



Pagination is meaningless without a sort key and direction, so picking a sensible 
default for user/project records is good. I'd go with either created_at (what 
Glance/Nova/Cinder use..) or with the user/project >UUID.



The Glance DB API pagination is well-documented and clean [1]. I highly 
recommend it as a starting point.



Nova uses the same marker/limit/sort_key/sort_dir options for queries that it 
allows pagination on. An example is the
instance_get_all_by_filters() call [2].



Cinder uses the same marker/limit/sort_key/sort_dir options for query 
pagination as well. [3]



Finally, I'd consider supporting the standard change-since parameter for listing operations. 
Both Nova [4] and Glance [5] support the parameter, which is useful for tools that poll the 
APIs for "new" >events/records.



In short, go with what is already a standard in the other projects...



Best,
-jay



[1]
https://github.com/openstack/glance/blob/master/glance/db/sqlalchemy/api.py#L429
[2]
https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L1709
[3]
https://github.com/openstack/cinder/blob/master/cinder/common/sqlalchemyutils.py#L33
[4]
https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api.py#L1766
[5]
https://github.com/openstack/glance/blob/master/glance/db/sqlalchemy/api.py#L618





___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] Pagination

[Gabriel Hurley]

I have been one of the earliest, loudest, and most consistent PITA's about 
pagination, so I probably oughta speak up. I would like to state three facts:

1. Marker + limit (e.g. forward-only) pagination is horrific for building a 
user interface.
2. Pagination doesn't scale.
3. OpenStack's APIs have historically had useless filtering capabilities.

In a world where pagination is a "must-have" feature we need to have page 
number + limit pagination in order to build a reasonable UI. Ironically though, 
I'm in favor of ditching pagination altogether. It's the lowest-common 
denominator, used because we as a community haven't buckled down and built 
meaningful ways for our users to get to the data they really want.

Filtering is great, but it's only 1/3 of the solution. Let me break it down 
with problems and high level "solutions":

Problem 1: I know what I want and I need to find it.
Solution: filtering/search systems.

Problem 2: I don't know what I want, and it may or may not exist.
Solution: tailored discovery mechanisms.

Problem 3: I need to know something about *all* the data in my system.
Solution: reporting systems.

We've got the better part of none of that. But I'd like to solve these issues. 
I have lots of thoughts on all of those, and I think the UX and design 
communities can offer a lot in terms of the usability of the solutions we come 
up with. Even more, I think this would be an awesome working group session at 
the next summit to talk about nothing other than "how can we get rid of 
pagination?"

As a parting thought, what percentage of the time do you click to the second 
page of results in Google?

All the best,

- Gabriel


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] Pagination

[Jay Pipes]

On 08/13/2013 05:04 PM, Gabriel Hurley wrote:

I have been one of the earliest, loudest, and most consistent PITA's about 
pagination, so I probably oughta speak up. I would like to state three facts:

1. Marker + limit (e.g. forward-only) pagination is horrific for building a 
user interface.
2. Pagination doesn't scale.
3. OpenStack's APIs have historically had useless filtering capabilities.

In a world where pagination is a "must-have" feature we need to have page 
number + limit pagination in order to build a reasonable UI. Ironically though, I'm in 
favor of ditching pagination altogether. It's the lowest-common denominator, used because 
we as a community haven't buckled down and built meaningful ways for our users to get to 
the data they really want.

Filtering is great, but it's only 1/3 of the solution. Let me break it down with problems 
and high level "solutions":

Problem 1: I know what I want and I need to find it.
Solution: filtering/search systems.


This is a good place to start. Glance has excellent filtering/search 
capabilities -- built in to the API from early on in the Essex 
timeframe, and only expanded over the last few releases.


Pagination solutions should build on a solid filtering/search 
functionality in the API, where there is a consistent sort key and 
direction (either hard-coded or user-determined, doesn't matter).


Limit/offset pagination solutions (forward and backwards paging, random 
skip-to-a-page) are inefficient from a SQL query perspective and should 
be a last resort, IMO, compared to limit/marker. With some smart 
session-storage of a page's markers, backwards paging with limit/marker 
APIs is certainly possible -- just store the previous page's marker.



Problem 2: I don't know what I want, and it may or may not exist.
Solution: tailored discovery mechanisms.


This should not be a use case that we spend much time on. Frankly, this 
use case can be summarized as "the window shopper scenario". Providing a 
quality search/filtering mechanism, including the *API* itself providing 
REST-ful discovery of the filters and search criteria the API supports, 
is way more important...



Problem 3: I need to know something about *all* the data in my system.
Solution: reporting systems.


Sure, no disagreement there.


We've got the better part of none of that.


I disagree. Some of the APIs have support for a good bit of 
search/filtering. We just need to bring all the projects up to search 
speed, Captain.


Best,
-jay

p.s. I very often go to the second and third pages of Google searches. 
:) But I never skip to the 127th page of results.


> But I'd like to solve these issues. I have lots of thoughts on all of 
those, and I think the UX and design communities can offer a lot in 
terms of the usability of the solutions we come up with. Even more, I 
think this would be an awesome working group session at the next summit 
to talk about nothing other than "how can we get rid of pagination?"


As a parting thought, what percentage of the time do you click to the second 
page of results in Google?

All the best,

 - Gabriel


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] Pagination

[Kieran Spear]

On 14/08/2013, at 7:40 AM, Jay Pipes  wrote:

> On 08/13/2013 05:04 PM, Gabriel Hurley wrote:
>> I have been one of the earliest, loudest, and most consistent PITA's about 
>> pagination, so I probably oughta speak up. I would like to state three facts:
>> 
>> 1. Marker + limit (e.g. forward-only) pagination is horrific for building a 
>> user interface.
>> 2. Pagination doesn't scale.
>> 3. OpenStack's APIs have historically had useless filtering capabilities.
>> 
>> In a world where pagination is a "must-have" feature we need to have page 
>> number + limit pagination in order to build a reasonable UI. Ironically 
>> though, I'm in favor of ditching pagination altogether. It's the 
>> lowest-common denominator, used because we as a community haven't buckled 
>> down and built meaningful ways for our users to get to the data they really 
>> want.
>> 
>> Filtering is great, but it's only 1/3 of the solution. Let me break it down 
>> with problems and high level "solutions":
>> 
>> Problem 1: I know what I want and I need to find it.
>> Solution: filtering/search systems.
> 
> This is a good place to start. Glance has excellent filtering/search 
> capabilities -- built in to the API from early on in the Essex timeframe, and 
> only expanded over the last few releases.
> 
> Pagination solutions should build on a solid filtering/search functionality 
> in the API, where there is a consistent sort key and direction (either 
> hard-coded or user-determined, doesn't matter).
> 
> Limit/offset pagination solutions (forward and backwards paging, random 
> skip-to-a-page) are inefficient from a SQL query perspective and should be a 
> last resort, IMO, compared to limit/marker. With some smart session-storage 
> of a page's markers, backwards paging with limit/marker APIs is certainly 
> possible -- just store the previous page's marker.

Not just the previous page's marker, but the marker of every previous page 
since we would like to be able to click the previous button more than once. Any 
previous markers we store are also likely to become stale pretty quickly. And 
all this is based on the assumption that the user's session even started at the 
first 'page' - it could be they followed a link from elsewhere in Horizon or 
the greater internet.

I completely agree with Dolph that this is something the client shouldn't need 
to care about at all. The next/prev links returned with each page of results 
should hide all of this. next/prev links also make it trivial for the client to 
discover whether there's even a next page at all, since we don't want to make a 
user click a link to go to an empty page.

Having said that, I think we can improve the current marker/limit system 
without hurting performance if we split the marker into 'before' and 'after' 
parameters. That way all the information needed to go forward or backwards is 
included in the results for the current page. Supporting 'before' should be as 
simple as reversing the sort order and then flipping the order of the results.


Kieran


> 
>> Problem 2: I don't know what I want, and it may or may not exist.
>> Solution: tailored discovery mechanisms.
> 
> This should not be a use case that we spend much time on. Frankly, this use 
> case can be summarized as "the window shopper scenario". Providing a quality 
> search/filtering mechanism, including the *API* itself providing REST-ful 
> discovery of the filters and search criteria the API supports, is way more 
> important...
> 
>> Problem 3: I need to know something about *all* the data in my system.
>> Solution: reporting systems.
> 
> Sure, no disagreement there.
> 
>> We've got the better part of none of that.
> 
> I disagree. Some of the APIs have support for a good bit of search/filtering. 
> We just need to bring all the projects up to search speed, Captain.
> 
> Best,
> -jay
> 
> p.s. I very often go to the second and third pages of Google searches. :) But 
> I never skip to the 127th page of results.
> 
> > But I'd like to solve these issues. I have lots of thoughts on all of 
> > those, and I think the UX and design communities can offer a lot in terms 
> > of the usability of the solutions we come up with. Even more, I think this 
> > would be an awesome working group session at the next summit to talk about 
> > nothing other than "how can we get rid of pagination?"
>> 
>> As a parting thought, what percentage of the time do you click to the second 
>> page of results in Google?
>> 
>> All the best,
>> 
>> - Gabriel
>> 
>> 
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


___
OpenStack-dev mailing list
OpenStack-dev@l

Re: [openstack-dev] [keystone] Pagination

[Yufang Zhang]

2013/8/14 Kieran Spear 

>
> On 14/08/2013, at 7:40 AM, Jay Pipes  wrote:
>
> > On 08/13/2013 05:04 PM, Gabriel Hurley wrote:
> >> I have been one of the earliest, loudest, and most consistent PITA's
> about pagination, so I probably oughta speak up. I would like to state
> three facts:
> >>
> >> 1. Marker + limit (e.g. forward-only) pagination is horrific for
> building a user interface.
> >> 2. Pagination doesn't scale.
> >> 3. OpenStack's APIs have historically had useless filtering
> capabilities.
> >>
> >> In a world where pagination is a "must-have" feature we need to have
> page number + limit pagination in order to build a reasonable UI.
> Ironically though, I'm in favor of ditching pagination altogether. It's the
> lowest-common denominator, used because we as a community haven't buckled
> down and built meaningful ways for our users to get to the data they really
> want.
> >>
> >> Filtering is great, but it's only 1/3 of the solution. Let me break it
> down with problems and high level "solutions":
> >>
> >> Problem 1: I know what I want and I need to find it.
> >> Solution: filtering/search systems.
> >
> > This is a good place to start. Glance has excellent filtering/search
> capabilities -- built in to the API from early on in the Essex timeframe,
> and only expanded over the last few releases.
> >
> > Pagination solutions should build on a solid filtering/search
> functionality in the API, where there is a consistent sort key and
> direction (either hard-coded or user-determined, doesn't matter).
> >
> > Limit/offset pagination solutions (forward and backwards paging, random
> skip-to-a-page) are inefficient from a SQL query perspective and should be
> a last resort, IMO, compared to limit/marker. With some smart
> session-storage of a page's markers, backwards paging with limit/marker
> APIs is certainly possible -- just store the previous page's marker.
>
> Not just the previous page's marker, but the marker of every previous page
> since we would like to be able to click the previous button more than once.
> Any previous markers we store are also likely to become stale pretty
> quickly. And all this is based on the assumption that the user's session
> even started at the first 'page' - it could be they followed a link from
> elsewhere in Horizon or the greater internet.
>
> I think we don't have to store previous pages' markers. Just storing
current page's markers is OK. Nova or Glance has supported 'sort_dir' for
long, if you want to go to the previous page, just use the first entry of
current page as marker and set sort_dir as 'asc'; Otherwise, use the last
entry of current page as marker and set sort_dir as ‘desc’.



> I completely agree with Dolph that this is something the client shouldn't
> need to care about at all. The next/prev links returned with each page of
> results should hide all of this. next/prev links also make it trivial for
> the client to discover whether there's even a next page at all, since we
> don't want to make a user click a link to go to an empty page.
>
> Having said that, I think we can improve the current marker/limit system
> without hurting performance if we split the marker into 'before' and
> 'after' parameters. That way all the information needed to go forward or
> backwards is included in the results for the current page. Supporting
> 'before' should be as simple as reversing the sort order and then flipping
> the order of the results.
>
>
> Kieran
>
>
> >
> >> Problem 2: I don't know what I want, and it may or may not exist.
> >> Solution: tailored discovery mechanisms.
> >
> > This should not be a use case that we spend much time on. Frankly, this
> use case can be summarized as "the window shopper scenario". Providing a
> quality search/filtering mechanism, including the *API* itself providing
> REST-ful discovery of the filters and search criteria the API supports, is
> way more important...
> >
> >> Problem 3: I need to know something about *all* the data in my system.
> >> Solution: reporting systems.
> >
> > Sure, no disagreement there.
> >
> >> We've got the better part of none of that.
> >
> > I disagree. Some of the APIs have support for a good bit of
> search/filtering. We just need to bring all the projects up to search
> speed, Captain.
> >
> > Best,
> > -jay
> >
> > p.s. I very often go to the second and third pages of Google searches.
> :) But I never skip to the 127th page of results.
> >
> > > But I'd like to solve these issues. I have lots of thoughts on all of
> those, and I think the UX and design communities can offer a lot in terms
> of the usability of the solutions we come up with. Even more, I think this
> would be an awesome working group session at the next summit to talk about
> nothing other than "how can we get rid of pagination?"
> >>
> >> As a parting thought, what percentage of the time do you click to the
> second page of results in Google?
> >>
> >> All the best,
> >>
> >> - Gabriel
> >>
> >>
> >> ___

Re: [openstack-dev] [keystone] Pagination

[Mac Innes, Kiall]

So, Are we saying that UIs built on OpenStack APIs shouldn't be able to 
show traditional pagination controls? Or am I missing how this should 
work with marker/limit?

e.g. for 11 pages of content, something like: 1 2 3 .. 10 11

Thanks,
Kiall

On 13/08/13 22:45, Jay Pipes wrote:
> On 08/13/2013 05:04 PM, Gabriel Hurley wrote:
>> I have been one of the earliest, loudest, and most consistent PITA's about 
>> pagination, so I probably oughta speak up. I would like to state three facts:
>>
>> 1. Marker + limit (e.g. forward-only) pagination is horrific for building a 
>> user interface.
>> 2. Pagination doesn't scale.
>> 3. OpenStack's APIs have historically had useless filtering capabilities.
>>
>> In a world where pagination is a "must-have" feature we need to have page 
>> number + limit pagination in order to build a reasonable UI. Ironically 
>> though, I'm in favor of ditching pagination altogether. It's the 
>> lowest-common denominator, used because we as a community haven't buckled 
>> down and built meaningful ways for our users to get to the data they really 
>> want.
>>
>> Filtering is great, but it's only 1/3 of the solution. Let me break it down 
>> with problems and high level "solutions":
>>
>> Problem 1: I know what I want and I need to find it.
>> Solution: filtering/search systems.
>
> This is a good place to start. Glance has excellent filtering/search
> capabilities -- built in to the API from early on in the Essex
> timeframe, and only expanded over the last few releases.
>
> Pagination solutions should build on a solid filtering/search
> functionality in the API, where there is a consistent sort key and
> direction (either hard-coded or user-determined, doesn't matter).
>
> Limit/offset pagination solutions (forward and backwards paging, random
> skip-to-a-page) are inefficient from a SQL query perspective and should
> be a last resort, IMO, compared to limit/marker. With some smart
> session-storage of a page's markers, backwards paging with limit/marker
> APIs is certainly possible -- just store the previous page's marker.
>
>> Problem 2: I don't know what I want, and it may or may not exist.
>> Solution: tailored discovery mechanisms.
>
> This should not be a use case that we spend much time on. Frankly, this
> use case can be summarized as "the window shopper scenario". Providing a
> quality search/filtering mechanism, including the *API* itself providing
> REST-ful discovery of the filters and search criteria the API supports,
> is way more important...
>
>> Problem 3: I need to know something about *all* the data in my system.
>> Solution: reporting systems.
>
> Sure, no disagreement there.
>
>> We've got the better part of none of that.
>
> I disagree. Some of the APIs have support for a good bit of
> search/filtering. We just need to bring all the projects up to search
> speed, Captain.
>
> Best,
> -jay
>
> p.s. I very often go to the second and third pages of Google searches.
> :) But I never skip to the 127th page of results.
>
>   > But I'd like to solve these issues. I have lots of thoughts on all of
> those, and I think the UX and design communities can offer a lot in
> terms of the usability of the solutions we come up with. Even more, I
> think this would be an awesome working group session at the next summit
> to talk about nothing other than "how can we get rid of pagination?"
>>
>> As a parting thought, what percentage of the time do you click to the second 
>> page of results in Google?
>>
>> All the best,
>>
>>   - Gabriel
>>
>>
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] Pagination

[Henry Nash]

So I have started an etherpad to summarize the views expressed over this email 
chain: https://etherpad.openstack.org/pagination

Feel free to add to (and correct) this.

Henry
On 14 Aug 2013, at 17:25, Mac Innes, Kiall wrote:

> So, Are we saying that UIs built on OpenStack APIs shouldn't be able to 
> show traditional pagination controls? Or am I missing how this should 
> work with marker/limit?
> 
> e.g. for 11 pages of content, something like: 1 2 3 .. 10 11
> 
> Thanks,
> Kiall
> 
> On 13/08/13 22:45, Jay Pipes wrote:
>> On 08/13/2013 05:04 PM, Gabriel Hurley wrote:
>>> I have been one of the earliest, loudest, and most consistent PITA's about 
>>> pagination, so I probably oughta speak up. I would like to state three 
>>> facts:
>>> 
>>> 1. Marker + limit (e.g. forward-only) pagination is horrific for building a 
>>> user interface.
>>> 2. Pagination doesn't scale.
>>> 3. OpenStack's APIs have historically had useless filtering capabilities.
>>> 
>>> In a world where pagination is a "must-have" feature we need to have page 
>>> number + limit pagination in order to build a reasonable UI. Ironically 
>>> though, I'm in favor of ditching pagination altogether. It's the 
>>> lowest-common denominator, used because we as a community haven't buckled 
>>> down and built meaningful ways for our users to get to the data they really 
>>> want.
>>> 
>>> Filtering is great, but it's only 1/3 of the solution. Let me break it down 
>>> with problems and high level "solutions":
>>> 
>>> Problem 1: I know what I want and I need to find it.
>>> Solution: filtering/search systems.
>> 
>> This is a good place to start. Glance has excellent filtering/search
>> capabilities -- built in to the API from early on in the Essex
>> timeframe, and only expanded over the last few releases.
>> 
>> Pagination solutions should build on a solid filtering/search
>> functionality in the API, where there is a consistent sort key and
>> direction (either hard-coded or user-determined, doesn't matter).
>> 
>> Limit/offset pagination solutions (forward and backwards paging, random
>> skip-to-a-page) are inefficient from a SQL query perspective and should
>> be a last resort, IMO, compared to limit/marker. With some smart
>> session-storage of a page's markers, backwards paging with limit/marker
>> APIs is certainly possible -- just store the previous page's marker.
>> 
>>> Problem 2: I don't know what I want, and it may or may not exist.
>>> Solution: tailored discovery mechanisms.
>> 
>> This should not be a use case that we spend much time on. Frankly, this
>> use case can be summarized as "the window shopper scenario". Providing a
>> quality search/filtering mechanism, including the *API* itself providing
>> REST-ful discovery of the filters and search criteria the API supports,
>> is way more important...
>> 
>>> Problem 3: I need to know something about *all* the data in my system.
>>> Solution: reporting systems.
>> 
>> Sure, no disagreement there.
>> 
>>> We've got the better part of none of that.
>> 
>> I disagree. Some of the APIs have support for a good bit of
>> search/filtering. We just need to bring all the projects up to search
>> speed, Captain.
>> 
>> Best,
>> -jay
>> 
>> p.s. I very often go to the second and third pages of Google searches.
>> :) But I never skip to the 127th page of results.
>> 
>>> But I'd like to solve these issues. I have lots of thoughts on all of
>> those, and I think the UX and design communities can offer a lot in
>> terms of the usability of the solutions we come up with. Even more, I
>> think this would be an awesome working group session at the next summit
>> to talk about nothing other than "how can we get rid of pagination?"
>>> 
>>> As a parting thought, what percentage of the time do you click to the 
>>> second page of results in Google?
>>> 
>>> All the best,
>>> 
>>>  - Gabriel
>>> 
>>> 
>>> ___
>>> OpenStack-dev mailing list
>>> OpenStack-dev@lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> 
>> 
>> 
>> ___
>> OpenStack-dev mailing list
>> OpenStack-dev@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
> 
> ___
> OpenStack-dev mailing list
> OpenStack-dev@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [keystone] Pagination

[Jay Pipes]

On 08/14/2013 12:25 PM, Mac Innes, Kiall wrote:

So, Are we saying that UIs built on OpenStack APIs shouldn't be able to
show traditional pagination controls? Or am I missing how this should
work with marker/limit?


No, not quite what I'm saying. The operation to get the total number of 
pages -- or more explicitly, the operation to get the *exact* number of 
pages in a list result -- is expensive, and in order to be reasonably 
efficient, some level of caching is almost always needed.


However, being able to page forwards and backwards is absolutely 
possible with limit/marker solutions. It simply requires the paging 
client (in this case, Horizon), to store the list of previously seen 
page links returned in listing results (there is a next and prev link in 
the returned list images results, for example).



e.g. for 11 pages of content, something like: 1 2 3 .. 10 11


Yeah, that's not an efficient operation unless you have some sort of 
caching in place. You can use things like MySQL's SQL_CALC_FOUND_ROWS, 
but that is not efficient since instead of stopping the query after 
LIMIT rows, you end up executing the entire query to determine the 
number of rows that *would* have been returned if no LIMIT was applied. 
In order to make such a thing efficient, you'd want to cache the value 
of SQL_CALC_FOUND_ROWS in the session and use that to calculate the 
number of pages.


It's something that can be done, but isn't, IMHO, worth it to get the 
traditional UI you describe. Instead, a good filter/search UI would be 
better, with just next/prev links.


Best,
-jay


Thanks,
Kiall

On 13/08/13 22:45, Jay Pipes wrote:

On 08/13/2013 05:04 PM, Gabriel Hurley wrote:

I have been one of the earliest, loudest, and most consistent PITA's about 
pagination, so I probably oughta speak up. I would like to state three facts:

1. Marker + limit (e.g. forward-only) pagination is horrific for building a 
user interface.
2. Pagination doesn't scale.
3. OpenStack's APIs have historically had useless filtering capabilities.

In a world where pagination is a "must-have" feature we need to have page 
number + limit pagination in order to build a reasonable UI. Ironically though, I'm in 
favor of ditching pagination altogether. It's the lowest-common denominator, used because 
we as a community haven't buckled down and built meaningful ways for our users to get to 
the data they really want.

Filtering is great, but it's only 1/3 of the solution. Let me break it down with problems 
and high level "solutions":

Problem 1: I know what I want and I need to find it.
Solution: filtering/search systems.


This is a good place to start. Glance has excellent filtering/search
capabilities -- built in to the API from early on in the Essex
timeframe, and only expanded over the last few releases.

Pagination solutions should build on a solid filtering/search
functionality in the API, where there is a consistent sort key and
direction (either hard-coded or user-determined, doesn't matter).

Limit/offset pagination solutions (forward and backwards paging, random
skip-to-a-page) are inefficient from a SQL query perspective and should
be a last resort, IMO, compared to limit/marker. With some smart
session-storage of a page's markers, backwards paging with limit/marker
APIs is certainly possible -- just store the previous page's marker.


Problem 2: I don't know what I want, and it may or may not exist.
Solution: tailored discovery mechanisms.


This should not be a use case that we spend much time on. Frankly, this
use case can be summarized as "the window shopper scenario". Providing a
quality search/filtering mechanism, including the *API* itself providing
REST-ful discovery of the filters and search criteria the API supports,
is way more important...


Problem 3: I need to know something about *all* the data in my system.
Solution: reporting systems.


Sure, no disagreement there.


We've got the better part of none of that.


I disagree. Some of the APIs have support for a good bit of
search/filtering. We just need to bring all the projects up to search
speed, Captain.

Best,
-jay

p.s. I very often go to the second and third pages of Google searches.
:) But I never skip to the 127th page of results.

   > But I'd like to solve these issues. I have lots of thoughts on all of
those, and I think the UX and design communities can offer a lot in
terms of the usability of the solutions we come up with. Even more, I
think this would be an awesome working group session at the next summit
to talk about nothing other than "how can we get rid of pagination?"


As a parting thought, what percentage of the time do you click to the second 
page of results in Google?

All the best,

   - Gabriel


___
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




___
OpenStack-dev mailing list

Re: [openstack-dev] [keystone] Pagination

[Kieran Spear]

On 15/08/2013, at 3:02 AM, Jay Pipes  wrote:

> On 08/14/2013 12:25 PM, Mac Innes, Kiall wrote:
>> So, Are we saying that UIs built on OpenStack APIs shouldn't be able to
>> show traditional pagination controls? Or am I missing how this should
>> work with marker/limit?
> 
> No, not quite what I'm saying. The operation to get the total number of pages 
> -- or more explicitly, the operation to get the *exact* number of pages in a 
> list result -- is expensive, and in order to be reasonably efficient, some 
> level of caching is almost always needed.
> 
> However, being able to page forwards and backwards is absolutely possible 
> with limit/marker solutions. It simply requires the paging client (in this 
> case, Horizon), to store the list of previously seen page links returned in 
> listing results (there is a next and prev link in the returned list images 
> results, for example).

Is there a prev link? It's optional in Compute v2 and Images v1 and Nova/Glance 
don't seem to be returning it. I suspect Nova v3 isn't either but I haven't 
tested it. There's no mention of it in Images v2 at all and no prev returned 
there either.

If there is a next and prev link in the returned results then Horizon wouldn't 
need to store anything - the links can be rendered straight into the page.

Anyway, you're right, this is all secondary to the need for decent filtering.

> 
>> e.g. for 11 pages of content, something like: 1 2 3 .. 10 11
> 
> Yeah, that's not an efficient operation unless you have some sort of caching 
> in place. You can use things like MySQL's SQL_CALC_FOUND_ROWS, but that is 
> not efficient since instead of stopping the query after LIMIT rows, you end 
> up executing the entire query to determine the number of rows that *would* 
> have been returned if no LIMIT was applied. In order to make such a thing 
> efficient, you'd want to cache the value of SQL_CALC_FOUND_ROWS in the 
> session and use that to calculate the number of pages.
> 
> It's something that can be done, but isn't, IMHO, worth it to get the 
> traditional UI you describe. Instead, a good filter/search UI would be 
> better, with just next/prev links.
> 
> Best,
> -jay
> 
>> Thanks,
>> Kiall
>> 
>> On 13/08/13 22:45, Jay Pipes wrote:
>>> On 08/13/2013 05:04 PM, Gabriel Hurley wrote:
 I have been one of the earliest, loudest, and most consistent PITA's about 
 pagination, so I probably oughta speak up. I would like to state three 
 facts:
 
 1. Marker + limit (e.g. forward-only) pagination is horrific for building 
 a user interface.
 2. Pagination doesn't scale.
 3. OpenStack's APIs have historically had useless filtering capabilities.
 
 In a world where pagination is a "must-have" feature we need to have page 
 number + limit pagination in order to build a reasonable UI. Ironically 
 though, I'm in favor of ditching pagination altogether. It's the 
 lowest-common denominator, used because we as a community haven't buckled 
 down and built meaningful ways for our users to get to the data they 
 really want.
 
 Filtering is great, but it's only 1/3 of the solution. Let me break it 
 down with problems and high level "solutions":
 
 Problem 1: I know what I want and I need to find it.
 Solution: filtering/search systems.
>>> 
>>> This is a good place to start. Glance has excellent filtering/search
>>> capabilities -- built in to the API from early on in the Essex
>>> timeframe, and only expanded over the last few releases.
>>> 
>>> Pagination solutions should build on a solid filtering/search
>>> functionality in the API, where there is a consistent sort key and
>>> direction (either hard-coded or user-determined, doesn't matter).
>>> 
>>> Limit/offset pagination solutions (forward and backwards paging, random
>>> skip-to-a-page) are inefficient from a SQL query perspective and should
>>> be a last resort, IMO, compared to limit/marker. With some smart
>>> session-storage of a page's markers, backwards paging with limit/marker
>>> APIs is certainly possible -- just store the previous page's marker.
>>> 
 Problem 2: I don't know what I want, and it may or may not exist.
 Solution: tailored discovery mechanisms.
>>> 
>>> This should not be a use case that we spend much time on. Frankly, this
>>> use case can be summarized as "the window shopper scenario". Providing a
>>> quality search/filtering mechanism, including the *API* itself providing
>>> REST-ful discovery of the filters and search criteria the API supports,
>>> is way more important...
>>> 
 Problem 3: I need to know something about *all* the data in my system.
 Solution: reporting systems.
>>> 
>>> Sure, no disagreement there.
>>> 
 We've got the better part of none of that.
>>> 
>>> I disagree. Some of the APIs have support for a good bit of
>>> search/filtering. We just need to bring all the projects up to search
>>> speed, Captain.
>>> 
>>> Best,
>>> -jay
>>> 
>>> p.s