Re: [Zope3-dev] Re: Community opinion about search+filter
Hello Martijn, I think that would give a great GSOC project. The subject is well contained, knownledge might be ported from RDB and/or AdvancedQuery. Thursday, March 15, 2007, 7:04:03 PM, you wrote: MF Hello, MF Adam Groszer wrote: I'd like to ask your opinion, your experiences about searching and filtering in quite large object DBs. We need to add search and filter functions to our current app, where the user might be able to create quite _sophisticated_ filter criterias. (The app is a pure Z3 app, subject is document management) Currently we're looking at something based on catalog/indexes. As I checked the most comfortable solution would be based on hurry.query. Some questions arose: - Is it necessary/worth adding indexes on all attributes? - How does the index perform on modification and retrieval? The biggest problem is that this will be our first try, so we're missing experiences and are a bit puzzled about the right solution. Certain is that moving to RDB is not an option. MF I think one of the main limitations of the current catalog (and MF hurry.query) is efficient support for sorting and batching the query MF results. The Zope 3 catalog returns all matching results, which can then MF be sorted and batched. This will stop being scalable for large MF collections. A relational database is able to do this internally, and is MF potentially able to use optimizations there. MF It would be very nice if someone could look into expanding hurry.query MF and/or the catalog to support these cases. It would be interesting to MF look at what Dieter Maurer has done with AdvancedQuery in Zope 2 in this MF regard as well. MF Regards, MF Martijn MF ___ MF Zope3-dev mailing list MF Zope3-dev@zope.org MF Unsub: MF http://mail.zope.org/mailman/options/zope3-dev/agroszer%40gmail.com -- Best regards, Groszer Adam -- Quote of the day: Evil is only good perverted. - Henry Wadsworth Longfellow ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] 3.2 branch tests failing
It looks like this has been going on for a while: http://buildbot.zope.org/Zope3%20branches%203.2%202.4%20Windows%202000%20zc-bbwin/builds/163/test_2/0 Apparently someone changed the capitalization of the HTTP auth relm without running the tests. -- Benji York Senior Software Engineer Zope Corporation ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] buildbot failure in Zope3 branches 3.2 2.4 Linux remy
The Buildbot has detected a failed build of Zope3 branches 3.2 2.4 Linux remy. Buildbot URL: http://buildbot.zope.org/ Build Reason: The web-page 'force build' button was pressed by 'Benji': see if tests are still failing Build Source Stamp: None Blamelist: BUILD FAILED: failed test_2 sincerely, -The Buildbot ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] buildbot failure in Zope3 trunk 2.4 Linux zc-buildbot
The Buildbot has detected a failed build of Zope3 trunk 2.4 Linux zc-buildbot. Buildbot URL: http://buildbot.zope.org/ Build Reason: changes Build Source Stamp: 406 Blamelist: andreasjung,dobe,fdrake BUILD FAILED: failed test sincerely, -The Buildbot ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] buildbot failure in Zope3 trunk 2.4 Linux zc-buildbot
The Buildbot has detected a failed build of Zope3 trunk 2.4 Linux zc-buildbot. Buildbot URL: http://buildbot.zope.org/ Build Reason: changes Build Source Stamp: 425 Blamelist: andreasjung,dobe,fdrake BUILD FAILED: failed test sincerely, -The Buildbot ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [ZODB-Dev] Re: [Zope3-dev] Re: Community opinion about search+filter
On Mar 25, 2007, at 3:01 AM, Adam Groszer wrote: MF I think one of the main limitations of the current catalog (and MF hurry.query) is efficient support for sorting and batching the query MF results. The Zope 3 catalog returns all matching results, which can then MF be sorted and batched. This will stop being scalable for large MF collections. A relational database is able to do this internally, and is MF potentially able to use optimizations there. What evidence to you have to support this assertion? We did some literature search on this a few years ago and found no special trick to avoid sorting costs. I know of 2 approaches to reducing sort cost: 1. Sort your results based on the primary key and therefore, pick your primary key to match your sort results. In terms of the Zope catalog framework, the primary keys are the document IDs, which are traditionally chosen randomly. You can pick your primary keys based on a desired sort order instead. A variation on this theme is to use multiple sets of document ids, storing multiple sets of ids in each index. Of course, this approach doesn't help with something like relevance ranks. 2. Use an N-best algorithm. If N is the size of the batch and M is the corpus size, then this is O(M*ln(N)) rather than O(M*ln(M)) which is a significant improvement if N M, but still quite expensive. I don't think relational databases have any magic bullet to get around sorting costs. Sorting is expensive. In many ways, I think the sorting support in the catalog gave people a false sense of security. Jim -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [ZODB-Dev] Re: [Zope3-dev] Re: Community opinion about search+filter
On 3/25/07, Jim Fulton [EMAIL PROTECTED] wrote: On Mar 25, 2007, at 3:01 AM, Adam Groszer wrote: MF I think one of the main limitations of the current catalog (and MF hurry.query) is efficient support for sorting and batching the query MF results. The Zope 3 catalog returns all matching results, which can then MF be sorted and batched. This will stop being scalable for large MF collections. A relational database is able to do this internally, and is MF potentially able to use optimizations there. What evidence to you have to support this assertion? We did some literature search on this a few years ago and found no special trick to avoid sorting costs. I know of 2 approaches to reducing sort cost: 1. Sort your results based on the primary key and therefore, pick your primary key to match your sort results. In terms of the Zope catalog framework, the primary keys are the document IDs, which are traditionally chosen randomly. You can pick your primary keys based on a desired sort order instead. A variation on this theme is to use multiple sets of document ids, storing multiple sets of ids in each index. Of course, this approach doesn't help with something like relevance ranks. 2. Use an N-best algorithm. If N is the size of the batch and M is the corpus size, then this is O(M*ln(N)) rather than O(M*ln(M)) which is a significant improvement if N M, but still quite expensive. I don't think relational databases have any magic bullet to get around sorting costs. Sorting is expensive. In many ways, I think the sorting support in the catalog gave people a false sense of security. I don't know if relational databases typically does this internally (I don't think so). However, some search engines do it, like Lucene. And supposedly also Dieters IncrementalSearch (haven't used it yet). -- Lennart Regebro: Zope and Plone consulting. http://www.colliberty.com/ +33 661 58 14 64 ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [ZODB-Dev] Re: [Zope3-dev] Re: Community opinion about search+filter
On Mar 25, 2007, at 11:08 AM, Lennart Regebro wrote: ... 2. Use an N-best algorithm. If N is the size of the batch and M is the corpus size, then this is O(M*ln(N)) rather than O(M*ln(M)) which is a significant improvement if N M, but still quite expensive. I don't think relational databases have any magic bullet to get around sorting costs. Sorting is expensive. In many ways, I think the sorting support in the catalog gave people a false sense of security. I don't know if relational databases typically does this internally (I don't think so). However, some search engines do it, like Lucene. And supposedly also Dieters IncrementalSearch (haven't used it yet). Our catalog framework also has N-best support. JIm -- Jim Fulton mailto:[EMAIL PROTECTED]Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporationhttp://www.zope.com http://www.zope.org ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: SVN: Zope3/branches/3.2/src/zope/app/ftests/doctest.txt Fix of broken test by my change at rev 71681
Adam Groszer wrote: Log message for revision 71926: Fix of broken test by my change at rev 71681 Hi Adam, apparently that fix breaks tests on Windows: http://buildbot.zope.org/Zope3%20branches%203.2%202.4%20Windows%202000%20zc-bbwin/builds/163/test_2/0 Changed: U Zope3/branches/3.2/src/zope/app/ftests/doctest.txt -=- Modified: Zope3/branches/3.2/src/zope/app/ftests/doctest.txt === --- Zope3/branches/3.2/src/zope/app/ftests/doctest.txt 2007-01-11 17:11:51 UTC (rev 71925) +++ Zope3/branches/3.2/src/zope/app/ftests/doctest.txt 2007-01-11 17:12:29 UTC (rev 71926) @@ -21,9 +21,12 @@ ... GET /@@contents.html HTTP/1.1 ... ) HTTP/1.1 401 Unauthorized + Cache-Control: no-store, no-cache, must-revalidate Content-Length: ... Content-Type: text/html;charset=utf-8 - WWW-Authenticate: basic realm=zope + Expires: Mon, 26 Jul 1997 05:00:00 GMT + Pragma: no-cache + WWW-Authenticate: basic realm=Zope BLANKLINE !DOCTYPE html PUBLIC ... @@ -32,6 +35,7 @@ - A 404 response, - A WWW-Authenticate header, and - An html body with an error message + - Some technical headers to keep squid happy Note that we used ellipeses to indicate ininteresting details. -- http://worldcookery.com -- Professional Zope documentation and training ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] Re: Zope3 Standalone Page Templates
On 3/13/07, Tres Seaver [EMAIL PROTECTED] wrote: Have a look at the source distributions / eggs available at: http://download.zope.org/distribution/ Also, a checkout of the following might be instructive: svn+ssh://svn.zope.org/zope.pagetemplate/trunk For instance, this should work if you have setuptools / easyinstall already in hand for your python:: $ bin/easy_install -f http://download.zope.org/distribution \ zope.pagetemplate I still get a crazy amount of packages installed when I do this (see below). Either I'm doing something wrong (any ideas?) or I've got a very different definition of stand-alone ;-) When I ran the easy_install command above, the following were installed: Best match: zope.pagetemplate 3.3dev Best match: zope.schema 3.4dev-r73183 Best match: zope.component 3.4dev-r72903 Best match: pytz 2005r Best match: zope.thread 3.3dev-r72587 Best match: zope.proxy 3.4.0a1 Best match: zope.exceptions 3.3dev-r72546 Best match: zope.event 3.3dev-r72545 Best match: zope.deferredimport 3.3dev-r72527 Best match: zope.app 3.4dev-r73509 Best match: zope.traversing 3.4dev-r72940 Best match: zope.testing 3.4dev-r73174 Best match: docutils 0.4 Best match: RestrictedPython 3.2.0 Best match: zope.viewlet 3.4dev-r73054 Best match: zope.documenttemplate 3.3dev-r72533 Best match: zope.structuredtext 3.3dev-r72637 Best match: zope.testbrowser 3.4dev-r73268 Best match: zope.size 3.4dev-r72579 Best match: zope.server 3.4dev-r72578 Best match: zope.rdb 3.4dev-r72882 Best match: zope.publisher 3.3dev-r72907 Best match: zope.modulealias 0.1dev-r72559 Best match: zope.lifecycleevent 3.4dev-r72552 Best match: zope.index 3.4dev-r72728 Best match: zope.hookable 3.4dev-r72561 Best match: zope.formlib 3.4dev-r73110 Best match: zope.filerepresentation 3.4dev-r72616 Best match: zope.dublincore 3.4dev-r73114 Best match: zope.dottedname 0.1dev-r72536 Best match: zope.decorator 0.1dev-r72650 Best match: zope.datetime 3.4dev-r72522 Best match: zope.contenttype 3.4dev-r72520 Best match: zope.copypastemove 3.4dev-r72841 Best match: zope.cachedescriptors 3.4dev-r72515 Best match: zope.annotation 3.4dev-r72925 Best match: zodbcode 0.1dev-r72731 Best match: zdaemon 2.0a6 Best match: ZODB3 3.8.0a1.dev-r73128 Best match: zope.contentprovider 3.4dev-r72519 d ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
