Re: [Zope3-Users] zc.tables integration with Zope3
Jim Washington wrote: This is probably a bit premature, but I have been doing some thinking about reordering/batching of large sets. My current (not-quite-ready-for-prime-time) solution involves factoradics, and I have done a bit of a write-up on my blog, http://blog.hill-street.net/?p=5 . I don't have time to read your article right now, but I will do so later. I'm quite interested in making the presentation of tabular data scale better. Right now, zc.table excepts to have access to the whole list in order to sort and batch it, but with huge amounts of data this may not scale so well. Relational databases have limit queries that seem to help here, though I'm not sure how much optimization is happening on the backend level. Anyway, just wanted to register my interest in this topic, if indeed I understand you correctly. Regards, Martijn ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] zc.tables integration with Zope3
Martijn Faassen wrote: Jim Washington wrote: This is probably a bit premature, but I have been doing some thinking about reordering/batching of large sets. My current (not-quite-ready-for-prime-time) solution involves factoradics, and I have done a bit of a write-up on my blog, http://blog.hill-street.net/?p=5 . I don't have time to read your article right now, but I will do so later. I'm quite interested in making the presentation of tabular data scale better. Right now, zc.table excepts to have access to the whole list in order to sort and batch it, but with huge amounts of data this may not scale so well. Relational databases have limit queries that seem to help here, though I'm not sure how much optimization is happening on the backend level. Anyway, just wanted to register my interest in this topic, if indeed I understand you correctly. Hi, Martijn Good to hear that there may be some interest. As usual, soon after publishing, I came up with some ideas for improving the implementation algorithms. It's on my to-do list. BTW, I have recently been having some fun generating Zope 3 pages using lxml. I plan to do another article about that, soon. -Jim Washington ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] zc.tables integration with Zope3
Hi Jim. This approach is completely new to me. It is interesting and I'll keep an eye on your blog. How expensive is updating indexes with data modification. You are talking about sqlite, does this mean you figure the indexes would be a significant drag on the ZODB due their size or is this idea a performance consideration. Many thanks. Regards, David Jim Washington wrote: This is probably a bit premature, but I have been doing some thinking about reordering/batching of large sets. My current (not-quite-ready-for-prime-time) solution involves factoradics, and I have done a bit of a write-up on my blog, http://blog.hill-street.net/?p=5 . The idea is that you can have indexes to several permutations (first-name-sort, last-name sort, last-modified-date-sort, etc) of a set, and refer to those permutations by name, which refers to an integer index. The factoradic representation of that integer can be used in a quick algorithm to make a list of the ids in the set in that specific order, and simply slicing that list would give easy batching capability without needing to access the objects at all. There would, however, be a need to update all of the permutation indexes when the data change. One thing I have not dealt with yet to make this really useful would be a utility that memoizes the factorials and factoradics, which are expensive to calculate, probably using an sqlite database. The big trade-off is doing a sort each time the data is accessed vs. doing all of the named sorts each time the data is changed. I am not sure at the moment whether this solves a real need yet, but I thought I would bring it up since we are discussing batching. I think it is just at the interesting idea stage for right now. -Jim Washington ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] zc.tables integration with Zope3
David Pratt wrote: Hi Jim. This approach is completely new to me. It is interesting and I'll keep an eye on your blog. How expensive is updating indexes with data modification. You are talking about sqlite, does this mean you figure the indexes would be a significant drag on the ZODB due their size or is this idea a performance consideration. Many thanks. The expense is in recalculating the new factoradic for each stored index, which involves a lot of division and factorial operations. But this might be done in a smart way. Maybe instead of on-change, we could recalculate an index on first access to that index after change. Sqlite would be mainly for performance, ATM. A lookup is less expensive than calculating a factoradic of a very large number. Size may also be an issue if the index is stored in the zodb. An integer (long as it might be) is the most compact representation of the factoradic. I'm pretty sure that longs end up as base ten strings in a pickle, but that is still smaller than the pickle of the factoradic list representation. I still need to experiment more, but it's not out of the question that the indexes could end up as sqlite-backed annotations to the container/data source, particularly when sqlite may already be in the mix for memoizing factorial and/or factoradic calculations. It may still turn out that this is all not worth the effort in practice. -Jim Washington ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] zc.tables integration with Zope3
Hi Stephan. Could a Zope3 branch be set up in the zope repository so that your enhanced tables code can be added to the branch (so contents view refactored using it)? The contents view class could be the model for using the tables package in zope3. In fact, it could be become the base class with overrides to columns for custom views to provide sorting and batching (with the capability of displaying DC data). I like the idea of using divs to create generic presentation structures with batching and sorting capability. I had similar thoughts of modifying zc.tables to do something similar (after reading the code). Has anyone an opinion about Boston vs Rotterdam as default skin and the need for a generic batching utility? I think at some point, it is time to say out with the old and in with the new - particularly since viewlets factor largely in skinning (and skinning flexibility). I think if this is what is advocated for skinning, Zope3 ought to be presented with a default skin that uses this approach. Many thanks. Regards, David Stephan Richter wrote: On Monday 21 August 2006 15:14, David Pratt wrote: Hi. I am wondering if zc.tables could not be incorporated into Zope3 trunk. My rational for its inclusion is so that the contents views may be refactored to use it. The current contents view makes little sense without batching and sorting. The ZMI could really use this basic face lift to bring it in line with the fundamental views provided in its Zope2 counterpart. I'd also suggest that the Boston skin be given preference over rotterdam as the default skin for zope. zc.table still needs an iteration or two before it is ready for the trunk. I have started a branch for another iteration, but it will take me a while to complete. My goal is to cleanly include some of my enhancements (already checked in for demonstration), to generalize the code and make a div based alternative. Regards, Stephan ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] zc.tables integration with Zope3
On Tuesday 22 August 2006 09:27, David Pratt wrote: Hi Stephan. Could a Zope3 branch be set up in the zope repository so that your enhanced tables code can be added to the branch (so contents view refactored using it)? The contents view class could be the model for using the tables package in zope3. In fact, it could be become the base class with overrides to columns for custom views to provide sorting and batching (with the capability of displaying DC data). Note that I am -1 for including zc.table at this stage. I feel that it needs some more thought and real-life usage. Note: I use zc.table *a lot*! Has anyone an opinion about Boston vs Rotterdam as default skin and the need for a generic batching utility? I think at some point, it is time to say out with the old and in with the new - particularly since viewlets factor largely in skinning (and skinning flexibility). I think if this is what is advocated for skinning, Zope3 ought to be presented with a default skin that uses this approach. Many thanks. I personally do not care about the ZMI at all. For the customer I never base any code on those skins, since they are insanely complex. Of course, Boston is mainly so complex, because of all the slots that are required from Rotterdam. :-( As for batching, it would be nice to have a utility, but I would like to see some out-of-core implementation first and see it evolve a couple iterations. Regards, Stephan -- Stephan Richter CBU Physics Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] zc.tables integration with Zope3
Benji York wrote: Stephan Richter wrote: Note that I am -1 for including zc.table at this stage. I feel that it needs some more thought and real-life usage. Note: I use zc.table *a lot*! I'm +1 on your -1. It's a bit early yet. Plus the eggification of 3.4 will hopefully reduce the need to put things in the core. I'll note here that I've eggified zc.table before and it's in the Zope eggs directory here: http://download.zope.org/distribution/ If you don't want to use it in the Zope 3 core (which I understand the original poster wants to do, so this won't help him), but just want to use it in your project, having an egg makes it easier. Regards, Martijn ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
[Zope3-Users] zc.tables integration with Zope3
Hi. I am wondering if zc.tables could not be incorporated into Zope3 trunk. My rational for its inclusion is so that the contents views may be refactored to use it. The current contents view makes little sense without batching and sorting. The ZMI could really use this basic face lift to bring it in line with the fundamental views provided in its Zope2 counterpart. I'd also suggest that the Boston skin be given preference over rotterdam as the default skin for zope. Beyond this, a batching utility package is really needed in the way that ZTUtils provided this support in Zope2. While schooltool has this support, its GPL status does not make it accessible to non GPL developers (that might anticipate this generic utility in the code base). There are some really complex packages out there but this basic utility is still not available in our toolset. Many thanks. Regards, David ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] zc.tables integration with Zope3
On Monday 21 August 2006 15:14, David Pratt wrote: Hi. I am wondering if zc.tables could not be incorporated into Zope3 trunk. My rational for its inclusion is so that the contents views may be refactored to use it. The current contents view makes little sense without batching and sorting. The ZMI could really use this basic face lift to bring it in line with the fundamental views provided in its Zope2 counterpart. I'd also suggest that the Boston skin be given preference over rotterdam as the default skin for zope. zc.table still needs an iteration or two before it is ready for the trunk. I have started a branch for another iteration, but it will take me a while to complete. My goal is to cleanly include some of my enhancements (already checked in for demonstration), to generalize the code and make a div based alternative. Regards, Stephan -- Stephan Richter CBU Physics Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] zc.tables integration with Zope3
On Monday 21 August 2006 17:07, Fred Drake wrote: Huh? Whatever for? Are you planning to use zc.table for something other than tabular data? (X)HTML tables are entirely reasonable for tabular data, which is the only use case I'm aware of. That depends on the designer you talk to. :-) I agree with you, but I have this requirement. Regards, Stephan -- Stephan Richter CBU Physics Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] zc.tables integration with Zope3
Stephan Richter wrote: On Monday 21 August 2006 17:07, Fred Drake wrote: Huh? Whatever for? Are you planning to use zc.table for something other than tabular data? (X)HTML tables are entirely reasonable for tabular data, which is the only use case I'm aware of. That depends on the designer you talk to. :-) I agree with you, but I have this requirement. I don't think I'd be very happy with a div-based zc.table. -- Benji York Senior Software Engineer Zope Corporation ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] zc.tables integration with Zope3
On Monday 21 August 2006 17:34, Benji York wrote: I don't think I'd be very happy with a div-based zc.table. Then don't use it. It will be an alternative, of course. I will not break the existing API and output. Regards, Stephan -- Stephan Richter CBU Physics Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
Re: [Zope3-Users] zc.tables integration with Zope3
Stephan Richter wrote: On Monday 21 August 2006 17:34, Benji York wrote: I don't think I'd be very happy with a div-based zc.table. Then don't use it. It will be an alternative, of course. I will not break the existing API and output. Oh! I misunderstood. I couldn't imagine that the current architecture could be coerced into allowing both styles of output. I look forward to the improvements! -- Benji York Senior Software Engineer Zope Corporation ___ Zope3-users mailing list Zope3-users@zope.org http://mail.zope.org/mailman/listinfo/zope3-users
