Re: [Freeipa-devel] Paging in Web UI
On 8/31/2012 9:54 AM, Petr Vobornik wrote: How about this, when you open the search page for the first time it will do a regular search, no paging. Instead of showing the paging controls (Prev Next buttons) we show See more results... If you click that it will rerun the search with paging (which will give us the total number of entries) and show the paging control. I think we are in an agreement. It's basically what I wrote in different words. I just mentioned current size limit. Let me try to summarize. We are dealing with 3 different aspects: In the UI we will use 2 levels of search pages. In the initial page we will do a quick search to get a partial result. If you want to see the full result you have to click 'more'. The full result can displayed using multiple pages (like the current implementation) or single continuous page (the one I'm suggesting as an alternative solution). The multi-page solution will have paging controls (prev next buttons, page links), the single-page won't. Between the UI and the server the we can use 2-step operation (UI does the paging, it gets the pkey list first then gets the visible entries, this is the current implementation) or 1-step operation (server does the paging, UI gets the entries one page at a time). Any of these options can be used to implement either multi-page or single-page solution above. In the server we can retrieve the LDAP data with a simple search (default size limit is 100, can be increased up to 2000), with SPR (unlimited size, see below), or with VLV (unlimited size, but requires predefined filter). Any of these options can be used with either 1-step or 2-step operations to get the pkey list or the entries. User workflow when searching for record (the same thing reworded): 1) open search page: VLV or normal search with empty filter 1a) when VLV: user would see paging ui 1b) when normal search: not sure if to show 'see more results' link for switching to paging. For smaller setups (2000) it might be there, for larger rather not - pkey list would be truncated. For the initial page if we use the simple search (1b) the UI will look like this: http://edewata.fedorapeople.org/images/paging1.png Here the UI will get up to 100 entries (default size limit), then it displays them in a single page. There will be a scroll bar to scroll through these 100 entries, but it will not pull more entries. To see other entries you click 'more', it will go to full result (3) below. The simple search doesn't return the total number of entries, so we can't show the paging controls. If we use SPR or VLV in the initial page we will know the total number of entries, so we don't really need an initial page, we can use the same page as the full result (3) below. Not sure about pros cons of using SPR/VLV in the initial page, probably they will consume more server resources compared to simple search. 2) search for record: enter filter to textbox, normal search would be used This is the same as (1b) except that in (1b) the filter is empty. Similarly, if the entry you're looking for is not there, you can click the 'more' button to see the full result (3). Here we can also use SPR, but not VLV because VLV filter is fixed in the VLV configuration. I'm not sure if we can apply LDAP filter on top of VLV filter. 3) record not listed: switch to paging by clicking on 'see more results' Paging should be OK because most likely we would get less than 2000 records. Otherwise filter should be improved. I guess we can't go around this unless we increase the limit or use other mean of paging. According to this page we can use the nsslapd-pagedsizelimit to remove the global size limit on SPR: https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Directory_Server/9.0/html/Administration_Guide/User_Account_Management-Setting_Resource_Limits_Based_on_the_Bind_DN.html#Setting_Resource_Limits_Based_on_the_Bind_DN-Setting_Resource_Limits_Using_the_Command_Line So with SPR the server we can get the full result, one page at a time, no more 2000 entry limitation. If we use the multi-page solution it will look like this: http://edewata.fedorapeople.org/images/paging2.png Currently we use 20 entries per page. We can change that to 100, same as the default search limit, so the transition from the initial page to the full result will be seamless. Both SPR and VLV will return the total number of entries, so we can show the paging controls. We can use simple search too (like the current implementation) but it will be limited to 2000 entries. 4) record still not listed: page to find the record using current paging UI or some improved There's no need for (4) because we can get all entries with SPR or VLV. Regarding the continuous paging: I would extend the page by clicking at a bar at the end of the list and/or by mouse-scrolling at the end of the list? Ideally the scroll bar size should correspond to the visible part of
Re: [Freeipa-devel] Paging in Web UI
On 08/29/2012 06:50 PM, Endi Sukma Dewata wrote: On 8/29/2012 8:49 AM, Petr Vobornik wrote: Another possibility is to use VLV, but it seems to require open connection too and only works with a predefined filter. Apart from the possible connection problem I think it is very usable for default views of search pages. User can freely page. I don't understand from the wiki page if it can reasonably obtain total record count. Correction, VLV doesn't seem to require a continuous connection. The server will return a 'contentCount' in the VLV response: http://www.ietf.org/proceedings/55/I-D/draft-ietf-ldapext-ldapv3-vlv-09.txt IMO a modification of a hybrid solution may be best: When user opens page a VLV result would be shown. User can page and see all the data. Currently we don't support custom sorting so disability of custom sorting isn't an issue. With predefined sorts we can even improved these possibilities. If there is many records and user don't want to go through pages he will use search. If the search string is good he should fit to search limit so I wouldn't use paging at this moment. If not the case and he really don't know how to improve the filter he should have option to enable paging (current implementation) and go through the records by hand. Such record set should fit to limit of 2000 records. If not the filter is really bad. The thing is the filter is only used on certain attributes only. For example I have a user with full name Test User. Searching the Test User in the filter doesn't return anything because the server doesn't search the cn or displayName attributes. In that case I'd have to search with first name or last name only, which may return too many results. If we don't want to implement VLV or until that time I would set not-paging as default setting. How about this, when you open the search page for the first time it will do a regular search, no paging. Instead of showing the paging controls (Prev Next buttons) we show See more results... If you click that it will rerun the search with paging (which will give us the total number of entries) and show the paging control. I think we are in an agreement. It's basically what I wrote in different words. I just mentioned current size limit. User workflow when searching for record (the same thing reworded): 1) open search page: VLV or normal search with empty filter 1a) when VLV: user would see paging ui 1b) when normal search: not sure if to show 'see more results' link for switching to paging. For smaller setups (2000) it might be there, for larger rather not - pkey list would be truncated. 2) search for record: enter filter to textbox, normal search would be used 3) record not listed: switch to paging by clicking on 'see more results' Paging should be OK because most likely we would get less than 2000 records. Otherwise filter should be improved. I guess we can't go around this unless we increase the limit or use other mean of paging. 4) record still not listed: page to find the record using current paging UI or some improved Agreed, but if we implement the single continuous page I described in the earlier email the page size won't be much of an issue anymore. It may matter. For example when testing Web UI permissions. It often requires to navigate to the last page. I can enter the number directly and press enter or extend it for times (hypothetically) but I would rather see those 90 permissions right away because I can be at the list end in 0.5s. If we use a continuous page we can use a bigger default page size, say 100 entries. We probably could optimize it such that if you scroll directly to the bottom it will load just the last page. Comments below. We probably could also combine it with initial non-paging search like above, so you'll need to click See more results..., then the page will extend and you can scroll to the last entry. Yes as I wrote above. We can also improve pager to offer some subset of pages. For example: First Prev 22 23 *24* 25 26 Next Last Page [ ] where 24 is current page. Yes, that will work too. Regarding the continuous paging: I would extend the page by clicking at a bar at the end of the list and/or by mouse-scrolling at the end of the list? Ideally the scroll bar size should correspond to the visible part of the entire page. So you should be able to go anywhere in the entire page by moving the scroll bar around (either by keyboard, mouse click, mouse drag, or mouse scroll). In this case no need to extend the page. The thing is that we might have to create a table with many empty rows, which could be big, and populate them if they become visible. Maybe there's a way to avoid creating empty rows, we'll need to investigate. Did you meant by 'continuous paging' this solution or the one when you click on 'see more results' (or scroll to last 10% of a scrollbar) and it would show you next page? I originally understood it as the latter but now I'm
Re: [Freeipa-devel] Paging in Web UI
On 08/28/2012 08:31 PM, Endi Sukma Dewata wrote: On 8/28/2012 11:23 AM, Petr Vobornik wrote: Your possible solution does not address how many results are fetched (unless I misunderstood). If paging is enabled it doesn't, but it expects, that admin will disable it for larger setups.For smaller setups it isn't of much an issue. If paging is disabled, the limit is server 'search size limit' or --sizelimit option supplied by Web UI. I'm not sure how per-user preferences are handled in browsers, but don't forget we now have session support in the server. Server session data is available for use. I was thinking about using browser local storage (basically key-value DB). It has a benefit over session, that it survives a browser restart but it should contain only non-sensitive data (other users may see it). You mean browser cookie? No. https://developer.mozilla.org/en-US/docs/DOM/Storage Petr Spacek also suggested to have an attribute in user object to store IPA specific data as JSON. Therefore changing browser wouldn't be an issue. If you don't want to fetch every record to implement paging smartly in conjunction with it's performance issues why not do the query on the server, cache the results in the session, and have the RPC return the total number of results plus a subset of the result. Another RPC could retrieve the next/previous subset of results from the cached result in the session. I think most software do paging like this. I don't know the reasons for not doing it that way first time. My solution was counting with that we still don't want to do it. Endi do you know the reasons? Earlier sessions didn't exists, but it is doable without them too. Yes, at the time the sessions didn't exist and we needed a quick/temporary solution. I agree ideally this should be done on the server side, but how would the server maintain the paging information? The server can keep the search result (either just the pkey list or the entire entries) in memcached, but the result might be large and there might be multiple users accessing multiple search pages, so the total memory requirement could be large. I can imagine that if used only by admins it may be OK but if such functionality would be used by other applications it may become problem. If we don't want the server to keep the search result, the server could rerun the search and return only the requested page and discard most of the results each time the user change page. This may affect server performance. We can also use Simple Paged Results, but if I understood correctly it requires the httpd to maintain an open connection to the LDAP server for each user and for each page. I'm not sure memcached can be used to move the connection object among forked httpd processes. Also Simple Paged Results can only go forward, so no Prev button unless somebody keeps the results. Seems unusable. Another possibility is to use VLV, but it seems to require open connection too and only works with a predefined filter. Apart from the possible connection problem I think it is very usable for default views of search pages. User can freely page. I don't understand from the wiki page if it can reasonably obtain total record count. Here's the page I wrote about this: http://freeipa.org/page/IPAv3_Directory_Browsing Currently we're using Option #2 but I'm not sure if we use SPR between httpd and the LDAP server. But even with SPR the result is still bound by some limits. It looks you have to connect as Directory Manager to get the complete list of pkey list/entries. See the table in this page: This is one of the main reasons why I say it is broken. http://directory.fedoraproject.org/wiki/Simple_Paged_Results_Design The current implementation (keeping the pkey list in the UI) is conceptually similar to the front-end approach described in the above page. IMO a modification of a hybrid solution may be best: When user opens page a VLV result would be shown. User can page and see all the data. Currently we don't support custom sorting so disability of custom sorting isn't an issue. With predefined sorts we can even improved these possibilities. If there is many records and user don't want to go through pages he will use search. If the search string is good he should fit to search limit so I wouldn't use paging at this moment. If not the case and he really don't know how to improve the filter he should have option to enable paging (current implementation) and go through the records by hand. Such record set should fit to limit of 2000 records. If not the filter is really bad. If we don't want to implement VLV or until that time I would set not-paging as default setting. I don't think there any need in JSON formatted data for pretty printing with indentation. Is it an accident or oversight we're pretty printing the JSON data in an RPC. For large data sets compression would be a win, but most of our RPC communication is not large.
Re: [Freeipa-devel] Paging in Web UI
On 08/29/2012 07:34 AM, John Dennis wrote: On 08/28/2012 02:31 PM, Endi Sukma Dewata wrote: The server can keep the search result (either just the pkey list or the entire entries) in memcached, but the result might be large and there might be multiple users accessing multiple search pages, so the total memory requirement could be large. The default max size of an entry in memcached is 1MB. It can be increased to an upper limit of 128MB (but the memcached implementors do not recommend this due to degraded performance and the impact on the system). The session data is stored in a dict. You would be sharing the session data with other parts of the system. Currently that only includes the authentication data which is relatively small. I believe there is also some minor bookkeeping overhead that detracts from the per item total. If we need to exceed the upper bound for paged data I suppose we could implement caching within the cache. Almost 1MB of data is a lot of paging (and that limit can be increased), it would take a fair amount of paging to consume all that data. But the cached query could be broken up into cached chunks to limit the impact on memcached and to accommodate truly unlimited paging. In most instance you would fetch the next/prev page from the cache but if you walked off either end of the cached query you could query again and cache that result. In fact two levels of caching might be an actual implementation requirement to handle all cases. We can also use Simple Paged Results, but if I understood correctly it requires the httpd to maintain an open connection to the LDAP server for each user and for each page. Not for each user. In 389-ds-base-1.2.11 you can have multiple simple paged result searches on a single connection - see https://fedorahosted.org/389/ticket/260 This is the problem that VLV and Simple Paged Results are trying to solve - how to allow users to scroll/page through very large result sets. I'm not sure memcached can be used to move the connection object among forked httpd processes. Also Simple Paged Results can only go forward, so no Prev button unless somebody keeps the results. No, the connection object cannot be moved between processes via memcached because sockets are a property of the process that created it. ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] Paging in Web UI
On 08/29/2012 10:35 AM, Rich Megginson wrote: On 08/29/2012 07:34 AM, John Dennis wrote: On 08/28/2012 02:31 PM, Endi Sukma Dewata wrote: We can also use Simple Paged Results, but if I understood correctly it requires the httpd to maintain an open connection to the LDAP server for each user and for each page. Not for each user. In 389-ds-base-1.2.11 you can have multiple simple paged result searches on a single connection - see https://fedorahosted.org/389/ticket/260 Well this is the crux of the problem. We do not maintain a connection per user. LDAP connections exist for the duration of a single IPA RPC call. Those RPC calls may be multiplexed across multiple IPA server instances, each of which is it's own process. Our LDAP connections are very short lived and are scattered across processes. This is the problem that VLV and Simple Paged Results are trying to solve - how to allow users to scroll/page through very large result sets. -- John Dennis jden...@redhat.com Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] Paging in Web UI
On 8/29/2012 9:49 AM, Rich Megginson wrote: We can also use Simple Paged Results, but if I understood correctly it requires the httpd to maintain an open connection to the LDAP server foreach user and for each page. Not for each user. In 389-ds-base-1.2.11 you can have multiple simple paged result searches on a single connection - see https://fedorahosted.org/389/ticket/260 Well this is the crux of the problem. We do not maintain a connection per user. LDAP connections exist for the duration of a single IPA RPC call. Those RPC calls may be multiplexed across multiple IPA server instances, each of which is it's own process. Our LDAP connections are very short lived and are scattered across processes. So it sounds like, in order to be useful to IPA, we need to extend simple paged results: 1) ability to have the cookie (i.e. the results list and current position in that list) live outside of a connection 2) ability to go backwards in a list Is this correct? If so, please file 389 RFE's for these. For (1) how does the httpd send the information that it wants to use the result list from a previous connection? Is it going to use a new LDAP control? Or would there be a session ID? If we implement (2) does it mean the pages still need to be accessed sequentially, or can we jump to any random page? Also if I understood correctly the LDAP connections are made using user credentials, not Directory Manager, so things like nsslapd-sizelimit will apply. Does it mean a non-admin cannot browse the entire directory? -- Endi S. Dewata ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] Paging in Web UI
On 08/29/2012 09:16 AM, Endi Sukma Dewata wrote: On 8/29/2012 9:49 AM, Rich Megginson wrote: We can also use Simple Paged Results, but if I understood correctly it requires the httpd to maintain an open connection to the LDAP server foreach user and for each page. Not for each user. In 389-ds-base-1.2.11 you can have multiple simple paged result searches on a single connection - see https://fedorahosted.org/389/ticket/260 Well this is the crux of the problem. We do not maintain a connection per user. LDAP connections exist for the duration of a single IPA RPC call. Those RPC calls may be multiplexed across multiple IPA server instances, each of which is it's own process. Our LDAP connections are very short lived and are scattered across processes. So it sounds like, in order to be useful to IPA, we need to extend simple paged results: 1) ability to have the cookie (i.e. the results list and current position in that list) live outside of a connection 2) ability to go backwards in a list Is this correct? If so, please file 389 RFE's for these. For (1) how does the httpd send the information that it wants to use the result list from a previous connection? Is it going to use a new LDAP control? Not sure. Might be able to use the existing simple paged result control. Or would there be a session ID? If we implement (2) does it mean the pages still need to be accessed sequentially, or can we jump to any random page? We should be able support random page access. But I think we could support the ability to go backwards from the current page without random access support. Also if I understood correctly the LDAP connections are made using user credentials, not Directory Manager, so things like nsslapd-sizelimit will apply. Does it mean a non-admin cannot browse the entire directory? in 1.2.10 we have different limits for paged result searches: https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Directory_Server/9.0/html/Administration_Guide/User_Account_Management-Setting_Resource_Limits_Based_on_the_Bind_DN.html ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] Paging in Web UI
On 8/29/2012 8:49 AM, Petr Vobornik wrote: Another possibility is to use VLV, but it seems to require open connection too and only works with a predefined filter. Apart from the possible connection problem I think it is very usable for default views of search pages. User can freely page. I don't understand from the wiki page if it can reasonably obtain total record count. Correction, VLV doesn't seem to require a continuous connection. The server will return a 'contentCount' in the VLV response: http://www.ietf.org/proceedings/55/I-D/draft-ietf-ldapext-ldapv3-vlv-09.txt IMO a modification of a hybrid solution may be best: When user opens page a VLV result would be shown. User can page and see all the data. Currently we don't support custom sorting so disability of custom sorting isn't an issue. With predefined sorts we can even improved these possibilities. If there is many records and user don't want to go through pages he will use search. If the search string is good he should fit to search limit so I wouldn't use paging at this moment. If not the case and he really don't know how to improve the filter he should have option to enable paging (current implementation) and go through the records by hand. Such record set should fit to limit of 2000 records. If not the filter is really bad. The thing is the filter is only used on certain attributes only. For example I have a user with full name Test User. Searching the Test User in the filter doesn't return anything because the server doesn't search the cn or displayName attributes. In that case I'd have to search with first name or last name only, which may return too many results. If we don't want to implement VLV or until that time I would set not-paging as default setting. How about this, when you open the search page for the first time it will do a regular search, no paging. Instead of showing the paging controls (Prev Next buttons) we show See more results... If you click that it will rerun the search with paging (which will give us the total number of entries) and show the paging control. Agreed, but if we implement the single continuous page I described in the earlier email the page size won't be much of an issue anymore. It may matter. For example when testing Web UI permissions. It often requires to navigate to the last page. I can enter the number directly and press enter or extend it for times (hypothetically) but I would rather see those 90 permissions right away because I can be at the list end in 0.5s. If we use a continuous page we can use a bigger default page size, say 100 entries. We probably could optimize it such that if you scroll directly to the bottom it will load just the last page. We probably could also combine it with initial non-paging search like above, so you'll need to click See more results..., then the page will extend and you can scroll to the last entry. We can also improve pager to offer some subset of pages. For example: First Prev 22 23 *24* 25 26 Next Last Page [ ] where 24 is current page. Yes, that will work too. Regarding the continuous paging: I would extend the page by clicking at a bar at the end of the list and/or by mouse-scrolling at the end of the list? Ideally the scroll bar size should correspond to the visible part of the entire page. So you should be able to go anywhere in the entire page by moving the scroll bar around (either by keyboard, mouse click, mouse drag, or mouse scroll). In this case no need to extend the page. The thing is that we might have to create a table with many empty rows, which could be big, and populate them if they become visible. Maybe there's a way to avoid creating empty rows, we'll need to investigate. Alternatively we could do like image search on bing.com. If you bring the scroll bar say within 10% from the bottom it will load the next page. The thing is it will load the page sequentially, and if you continue to the bottom you'll end up with a very big table. Another possibility, we can use the top 10% of the scroll bar as a Prev button, and the bottom %10 as the Next button, and we remove the pages that aren't visible anymore. But this is not much different than the current implementation, only different presentation. -- Endi S. Dewata ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] Paging in Web UI
On 08/28/2012 09:30 AM, Petr Vobornik wrote: I would like to point out a problem in Web UI related to paging and suggest a possible solution: Current implementation of paging in Web UI is not serving it purpose which should be to do operations faster and be able to find all users. How current implementation should work: 1) get all primary keys (pkeys) of certain object type (users, ...) 2) load first 20 of them 3) from 1) we know record count so we can choose other page (different subset of 20) How it actually works: * get up to 2000 keys, with more items a LDAP limit is reached and result is truncated * we don't know how many are there, paging doesn't work as it should Problem: Let's say that paging is working (no LDAP limit) and consider this: Our page size is 20 records. With 400 users it is 20 pages. With 2000 users (current search limit) it is 200 pages and so on. Personally I wouldn't like to go through 20 pages. 200 or more pages sounds like a punishment. There is also a question if we even need to do it. I don't see a benefit of paging here. For smaller setup an increase of page size may help admin to see more users on a page more quickly (rolling mouse wheel is faster than click on next page button). For large setups, size of the pkey list must also be taken on mind. For each record there is a pkey and dn (dn is not required but it would need some hacking in python code to remove it). At the moment the list size raises drastically because JSON response includes indenting by spaces and is not compressed (easy to fix, but still there). With 10 000 or more users this pkey and dn list is pretty big (slow load on slower networks -vpns). This list is also loaded on each page change (record-wise, can be improved - cached). IMO with hundreds or thousands of users the most common use case is search by login or name or something. Paging is not required this case. It may be required if the result count is greater than size limit. In such case an option to temporary increase size limit or enable paging would be beneficial. Apart from this case, paging should be off. It will speed up page load because it executes only one request. Possible solution: 1) I suggest to introduce configuration options for paging. They can be global (default for all search pages) or individual for pages or Web UI's users. Per-user configuration can be stored in browser (to not pollute LDAP with application stuff). Global configuration on server in config plugin (minor polluting). Those options should be (per page and global): * enable paging * page size Note: when paging is disabled page size is basically --sizelimit option. 2) On search pages we should have controls to enable/disable paging and to change sizelimit for this particular moment. 3) Compress JSON responses (on httpd level) This way admin can configure default's for his deployment and user's can adjust for certain situations. Btw always including member_xx attributes in find or show commands is not good for search pages either but that's another topic. Comments are welcome. Your possible solution does not address how many results are fetched (unless I misunderstood). I'm not sure how per-user preferences are handled in browsers, but don't forget we now have session support in the server. Server session data is available for use. If you don't want to fetch every record to implement paging smartly in conjunction with it's performance issues why not do the query on the server, cache the results in the session, and have the RPC return the total number of results plus a subset of the result. Another RPC could retrieve the next/previous subset of results from the cached result in the session. I don't think there any need in JSON formatted data for pretty printing with indentation. Is it an accident or oversight we're pretty printing the JSON data in an RPC. For large data sets compression would be a win, but most of our RPC communication is not large. Compression has an overhead. With small data you'll use more cycles compressing and uncompressing than you would sending verbose but small data blobs. Compression should be an RPC option specified by either side of the connection and the receiver should be prepared to conditionally uncompress based on a flag value in the RPC. If you use server session data to support paging you may not need to introduce compression since you would only be passing one page of data at a time. User specified page size (without limitation) is an absolute necessity. I am frequently annoyed by web sites which either do not allow me to specify page size or constrain it to ridiculous hard coded limits such as 10, 20 or 30. -- John Dennis jden...@redhat.com Looking to carve out IT costs? www.redhat.com/carveoutcosts/ ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
Re: [Freeipa-devel] Paging in Web UI
On 08/28/2012 04:17 PM, John Dennis wrote: On 08/28/2012 09:30 AM, Petr Vobornik wrote: I would like to point out a problem in Web UI related to paging and suggest a possible solution: Current implementation of paging in Web UI is not serving it purpose which should be to do operations faster and be able to find all users. How current implementation should work: 1) get all primary keys (pkeys) of certain object type (users, ...) 2) load first 20 of them 3) from 1) we know record count so we can choose other page (different subset of 20) How it actually works: * get up to 2000 keys, with more items a LDAP limit is reached and result is truncated * we don't know how many are there, paging doesn't work as it should Problem: Let's say that paging is working (no LDAP limit) and consider this: Our page size is 20 records. With 400 users it is 20 pages. With 2000 users (current search limit) it is 200 pages and so on. Personally I wouldn't like to go through 20 pages. 200 or more pages sounds like a punishment. There is also a question if we even need to do it. I don't see a benefit of paging here. For smaller setup an increase of page size may help admin to see more users on a page more quickly (rolling mouse wheel is faster than click on next page button). For large setups, size of the pkey list must also be taken on mind. For each record there is a pkey and dn (dn is not required but it would need some hacking in python code to remove it). At the moment the list size raises drastically because JSON response includes indenting by spaces and is not compressed (easy to fix, but still there). With 10 000 or more users this pkey and dn list is pretty big (slow load on slower networks -vpns). This list is also loaded on each page change (record-wise, can be improved - cached). IMO with hundreds or thousands of users the most common use case is search by login or name or something. Paging is not required this case. It may be required if the result count is greater than size limit. In such case an option to temporary increase size limit or enable paging would be beneficial. Apart from this case, paging should be off. It will speed up page load because it executes only one request. Possible solution: 1) I suggest to introduce configuration options for paging. They can be global (default for all search pages) or individual for pages or Web UI's users. Per-user configuration can be stored in browser (to not pollute LDAP with application stuff). Global configuration on server in config plugin (minor polluting). Those options should be (per page and global): * enable paging * page size Note: when paging is disabled page size is basically --sizelimit option. 2) On search pages we should have controls to enable/disable paging and to change sizelimit for this particular moment. 3) Compress JSON responses (on httpd level) This way admin can configure default's for his deployment and user's can adjust for certain situations. Btw always including member_xx attributes in find or show commands is not good for search pages either but that's another topic. Comments are welcome. Your possible solution does not address how many results are fetched (unless I misunderstood). If paging is enabled it doesn't, but it expects, that admin will disable it for larger setups. For smaller setups it isn't of much an issue. If paging is disabled, the limit is server 'search size limit' or --sizelimit option supplied by Web UI. I'm not sure how per-user preferences are handled in browsers, but don't forget we now have session support in the server. Server session data is available for use. I was thinking about using browser local storage (basically key-value DB). It has a benefit over session, that it survives a browser restart but it should contain only non-sensitive data (other users may see it). If you don't want to fetch every record to implement paging smartly in conjunction with it's performance issues why not do the query on the server, cache the results in the session, and have the RPC return the total number of results plus a subset of the result. Another RPC could retrieve the next/previous subset of results from the cached result in the session. I think most software do paging like this. I don't know the reasons for not doing it that way first time. My solution was counting with that we still don't want to do it. Endi do you know the reasons? Earlier sessions didn't exists, but it is doable without them too. I don't think there any need in JSON formatted data for pretty printing with indentation. Is it an accident or oversight we're pretty printing the JSON data in an RPC. For large data sets compression would be a win, but most of our RPC communication is not large. Compression has an overhead. With small data you'll use more cycles compressing and uncompressing than you would sending verbose but small data blobs. Compression should be an RPC option specified by either side of
Re: [Freeipa-devel] Paging in Web UI
On 8/28/2012 8:30 AM, Petr Vobornik wrote: I would like to point out a problem in Web UI related to paging and suggest a possible solution: Current implementation of paging in Web UI is not serving it purpose which should be to do operations faster and be able to find all users. How current implementation should work: 1) get all primary keys (pkeys) of certain object type (users, ...) 2) load first 20 of them 3) from 1) we know record count so we can choose other page (different subset of 20) How it actually works: * get up to 2000 keys, with more items a LDAP limit is reached and result is truncated * we don't know how many are there, paging doesn't work as it should We have a 'search limit' which can be specified by the user as a search parameter. It has a default value of 100 which can configured in IPA Server - Configuration. There's also something like 'max result size' (I don't remember the actual name) of 2000 entries which is a global limit imposed by the LDAP backend. I'm not sure if the 'max result size' is configurable, but if I remember correctly if we want to get the entire entries in LDAP we'd need to use Simple Paged Results or VLV. Problem: Let's say that paging is working (no LDAP limit) and consider this: Our page size is 20 records. With 400 users it is 20 pages. With 2000 users (current search limit) it is 200 pages and so on. Personally I wouldn't like to go through 20 pages. 200 or more pages sounds like a punishment. In the current UI we could jump to a particular page directly, so it's not necessary to go through all pages. There is also a question if we even need to do it. I don't see a benefit of paging here. For smaller setup an increase of page size may help admin to see more users on a page more quickly (rolling mouse wheel is faster than click on next page button). We probably can make it look like a single continuous page, but we would still use paging to retrieve the entries. So initially it will show the entries of the first page, when you scroll down it will get the next page, but it will show the entries one-by-one by scrolling up the page. If none of the entries in the previous page are visible anymore the page can be removed from cache, but we might want to keep several pages in the cache. If we implement this, the interface will look the same regardless of the page size or whether we enable/disable paging. For large setups, size of the pkey list must also be taken on mind. For each record there is a pkey and dn (dn is not required but it would need some hacking in python code to remove it). At the moment the list size raises drastically because JSON response includes indenting by spaces and is not compressed (easy to fix, but still there). With 10 000 or more users this pkey and dn list is pretty big (slow load on slower networks -vpns). This list is also loaded on each page change (record-wise, can be improved - cached). Yes, ideally the pkey list should be cached and reloaded on new search or cache expiration. Or the server can do the query as John described, so the UI/CLI will just need to specify the page number. IMO with hundreds or thousands of users the most common use case is search by login or name or something. Paging is not required this case. It may be required if the result count is greater than size limit. In such case an option to temporary increase size limit or enable paging would be beneficial. For large setup we might need to use simple paged results or VLV, but they have limitations and require server support. Here's a page I wrote sometime ago: http://freeipa.org/page/IPAv3_Directory_Browsing Apart from this case, paging should be off. It will speed up page load because it executes only one request. Possible solution: 1) I suggest to introduce configuration options for paging. They can be global (default for all search pages) or individual for pages or Web UI's users. Per-user configuration can be stored in browser (to not pollute LDAP with application stuff). Global configuration on server in config plugin (minor polluting). Those options should be (per page and global): * enable paging * page size Note: when paging is disabled page size is basically --sizelimit option. Agreed. The actual number of entries would be different for each deployment and for each entity too, so it makes sense to use different settings to optimize the performance. If we implement the single continuous page above this setting will not affect the visual, but it will affect the performance. 2) On search pages we should have controls to enable/disable paging and to change sizelimit for this particular moment. If we keep the current UI users might want to save the setting (either on the server or on the client as you described) so they don't have to reconfigure it every time they log in. But if we implement the single continuous page above this might not be necessary anymore because users
Re: [Freeipa-devel] Paging in Web UI
On 8/28/2012 11:23 AM, Petr Vobornik wrote: Your possible solution does not address how many results are fetched (unless I misunderstood). If paging is enabled it doesn't, but it expects, that admin will disable it for larger setups.For smaller setups it isn't of much an issue. If paging is disabled, the limit is server 'search size limit' or --sizelimit option supplied by Web UI. I'm not sure how per-user preferences are handled in browsers, but don't forget we now have session support in the server. Server session data is available for use. I was thinking about using browser local storage (basically key-value DB). It has a benefit over session, that it survives a browser restart but it should contain only non-sensitive data (other users may see it). You mean browser cookie? If you don't want to fetch every record to implement paging smartly in conjunction with it's performance issues why not do the query on the server, cache the results in the session, and have the RPC return the total number of results plus a subset of the result. Another RPC could retrieve the next/previous subset of results from the cached result in the session. I think most software do paging like this. I don't know the reasons for not doing it that way first time. My solution was counting with that we still don't want to do it. Endi do you know the reasons? Earlier sessions didn't exists, but it is doable without them too. Yes, at the time the sessions didn't exist and we needed a quick/temporary solution. I agree ideally this should be done on the server side, but how would the server maintain the paging information? The server can keep the search result (either just the pkey list or the entire entries) in memcached, but the result might be large and there might be multiple users accessing multiple search pages, so the total memory requirement could be large. If we don't want the server to keep the search result, the server could rerun the search and return only the requested page and discard most of the results each time the user change page. This may affect server performance. We can also use Simple Paged Results, but if I understood correctly it requires the httpd to maintain an open connection to the LDAP server for each user and for each page. I'm not sure memcached can be used to move the connection object among forked httpd processes. Also Simple Paged Results can only go forward, so no Prev button unless somebody keeps the results. Another possibility is to use VLV, but it seems to require open connection too and only works with a predefined filter. Here's the page I wrote about this: http://freeipa.org/page/IPAv3_Directory_Browsing Currently we're using Option #2 but I'm not sure if we use SPR between httpd and the LDAP server. But even with SPR the result is still bound by some limits. It looks you have to connect as Directory Manager to get the complete list of pkey list/entries. See the table in this page: http://directory.fedoraproject.org/wiki/Simple_Paged_Results_Design The current implementation (keeping the pkey list in the UI) is conceptually similar to the front-end approach described in the above page. I don't think there any need in JSON formatted data for pretty printing with indentation. Is it an accident or oversight we're pretty printing the JSON data in an RPC. For large data sets compression would be a win, but most of our RPC communication is not large. Compression has an overhead. With small data you'll use more cycles compressing and uncompressing than you would sending verbose but small data blobs. Compression should be an RPC option specified by either side of the connection and the receiver should be prepared to conditionally uncompress based on a flag value in the RPC. If you use server session data to support paging you may not need to introduce compression since you would only be passing one page of data at a time. User specified page size (without limitation) is an absolute necessity. I am frequently annoyed by web sites which either do not allow me to specify page size or constrain it to ridiculous hard coded limits such as 10, 20 or 30. +1 Agreed, but if we implement the single continuous page I described in the earlier email the page size won't be much of an issue anymore. -- Endi S. Dewata ___ Freeipa-devel mailing list Freeipa-devel@redhat.com https://www.redhat.com/mailman/listinfo/freeipa-devel
