Gary Poster wrote:
On Aug 30, 2005, at 1:57 PM, Martijn Faassen wrote:
[snip]
It would be helpful if someone could explain the motivations behind
the extent catalog, by the way -- this information seems to be
missing in zc.catalog. Am I at all on the right track with my
thinking on it?
It should be pointed out initially that the son-of-queued-catalog
code doesn't have anything to do with extents. I think Jim wants
that factored out when we have time so that can be a mix-and-match
capability. I think you are asking about extents themselves, though.
Okay, I didn't realize yet glancing at this that this is *also*
son-of-queued catalog. Interesting. I'll glance at it some more. :)
We had three use cases that led us to extents.
First, we wanted several catalogs that only indexed certain different
things. This could have been done by subscribers, so this wasn't
terribly compelling by itself.
Okay, this is clear. It's not that clear to me how to efficiently make a
subscriber only handle one object type (I've been using the "is this a
IFoo? If not, return" pattern at the start of subscribers), but that's
another discussion.
Second, we wanted to transparently support queries that merged
results across catalog-like data structures. The catalog defined the
items we wanted to search through, while some of the other data
structures kept track of a larger set of objects (subsuming the set
that the catalog cared about). Sometimes, users could perform a
query that didn't actually use any of the catalog's data structures,
but that should be filtered by the set of the catalog's objects--its
extent.
I'm not sure I comprehend the motivations behind this one. Could you
elaborate?
Third, we wanted to let our indexes data be usable for NOT queries.
In order to do that, we needed an IFBTree structure that describes
the complete set for a given catalog, so that a contained index can
simply (and reasonably efficiently) subtract the query result from
the full set. The indexes in zc.catalog also use extents for some
other similar tricks.
This one's also clear.
An extent that accepts all objects would effectively be the data
structure you want, as I understand it.
I'm not sure -- 'not' is indeed context dependent, so which extent is in
use to determine the results of a 'not' operation depends on the query.
I think it's okay to ask the users to explicitly specify the extent when
they're doing the query, as long as there's an easy way to construct it
for the simple cases.
It is actually (at least typically for us) different than the intid
mapping because there are several classes of things that have intids
that are not cataloged.
If more than one catalog all index the same
objects, I'd first wonder why the indexes were not all in the same
catalog;
Good question. I think one example of such a scenario is if you wrote a
codebase, and I extended this codebase with some adapters which carry
around information I also want indexed. I may decide not to introduce
new indexes into your catalog but instead produce my own catalog to have
the concerns separate from each other. In this case I'd want to do
queries over multiple catalogs which index the same objects.
I'd second say that yes, they probably could share a
filter-less extent.
Why filter-less? I mean, wouldn't you want to filter on object type?
If we want any of zc.catalog in the Zope core, each component will
certainly need a proposal, by the way: we're offering this as code
that has helped us out and that we think might help others, either
directly or as ideas, so we are not duplicating effort. We're not
proclaiming it to necessarily be our next core step.
Understood. I'm giving you feedback. :) We (Infrae) are going to put
some code online eventually that we produced in the project we're
working on, for the same reasons.
Regards,
Martijn
_______________________________________________
Zope3-dev mailing list
Zope3-dev@zope.org
Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
