[Zope3-dev] KeywordIndex
I'm working on a simple application which is the first time I get to use the catalog in Zope 3. I'm writing against Zope 3.1b1. I was dismayed not to see KeywordIndex in the main catalog set, but then I found it in zope.index.keyword. But it seems to be a bit behind. I have it somewhat working through subclassing, etc, but it's been purely guess work on my part to get things this far. In my product package, I have the following: -- from zope.app.catalog.interfaces import IAttributeIndex, ICatalogIndex from zope.app.catalog.attribute import AttributeIndex from zope.app.container.contained import Contained from zope.index.keyword.index import KeywordIndex as KeywordIndexBase from zope.proxy import removeAllProxies from BTrees.IFBTree import IFTreeSet, IFSet, multiunion class IKeywordIndex(IAttributeIndex, ICatalogIndex): """ Interface-based catalog keyword index. """ class KeywordIndex(AttributeIndex, KeywordIndexBase, Contained): implements(IKeywordIndex) def index_doc(self, docid, value): # All security proxies need to be removed from the value. value = removeAllProxies(value) return super(KeywordIndex, self).index_doc(docid, value) def apply(self, query): # Keyword index doesn't implement apply(query) either. return self.search(removeAllProxies(query)) def _insert_forward(self, docid, words): """insert a sequence of words into the forward index """ # Replaces parent _insert_forward because apply() claims to want IFSets idx = self._fwd_index has_key = idx.has_key for word in words: if not has_key(word): idx[word] = IFSet() idx[word].insert(docid) -- I first overrode index_doc because the base KeywordIndex does an isinstance(value, (ListType, TupleType)), which failed on a security proxy guarded value. Then I added 'apply()' when I noticed that the base KeywordIndex didn't implement apply. Looking at the other supported indexes and at the index interfaces in zope.index, I noticed that IFSets were what was desired as the output of apply(), and that's when I replaced _insert_forward with a near identical copy that uses IFSet. This works... so long as I only search for one keyword. If I search for more than one through the catalog interface (and I imagine I would get the same result manually), I get the following traceback: -- File "/Users/jshell/Documents/Programming/kbase/lib/python/br/kbase/browser/search.py", line 22, in search results = catalog.searchResults(tags=query) File "/Library/ZopeX3/3.1/lib/python/zope/app/catalog/catalog.py", line 105, in searchResults results = self.apply(searchterms) File "/Library/ZopeX3/3.1/lib/python/zope/app/catalog/catalog.py", line 84, in apply r = index.apply(index_query) File "/Users/jshell/Documents/Programming/kbase/lib/python/br/kbase/catalog.py", line 36, in apply return self.search(removeAllProxies(query)) File "/Library/ZopeX3/3.1/lib/python/zope/index/keyword/index.py", line 139, in search rs = f(rs, docids) TypeError: invalid argument -- 'f' is IISet.intersection() The implementation of search() in the base KeywordIndex uses IISets for default values. I don't know if this is conflicting with the IFSets I set up in my subclass. I tried quickly editing zope.index.keyword.index to use IFSets instead, but I got the same traceback and then quickly reverted back to leaving the code untouched. It's been *years* since I've even touched simple indexing code, so I don't really know what's going on here or what's required. I would really like to have Keyword Index. In fact, such an index is the core of my application. I can throw together my own, I'm sure, that's a bit more brute force for my own purposes if necessary. I don't claim to have a solid understanding of how indexes and the catalog work (although it's been much easier to figure out in Zope 3, thanks!) Is there any reason why KeywordIndex seems half-abandoned? I guess it's not exposed to zope.app because of this. What would it take to make it catch up to FieldIndex? Thanks -- Jeff Shell ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] KeywordIndex
On Jul 18, 2005, at 11:14 AM, Jeff Shell wrote: I'm working on a simple application which is the first time I get to use the catalog in Zope 3. I'm writing against Zope 3.1b1. I was dismayed not to see KeywordIndex in the main catalog set, but then I found it in zope.index.keyword. But it seems to be a bit behind. Hi. Yes, we needed it too. Here's another thing we want to open source. Look at the attached .txt file; if you want it then tell me and I'll make it available in a sandbox. We'll move it over into the Zope repo (probably with a new name, or rearranged on the appropriate locations (zope.index and zope.app.catalog, etc.) RSN. Downsides: - Note that some functionality requires that you use an extent catalog, another goodie in the package. - We have some refactoring of this that we want to do. We'll have legacy issues ourselves, then. Additional upside: - This package also includes a replacement for the field index (called a value index) and customizations of the value and set indexes specific to timezone-aware datetimes, as well as a few other things. Gary The setindex is an index similar to, but more general than a traditional keyword index. The values indexed are expected to be iterables; the index allows searches for documents that contain any of a set of values; all of a set of values; or between a set of values. Additionally, the index supports an interface that allows examination of the indexed values. It is as policy-free as possible, and is intended to be the engine for indexes with more policy, as well as being useful itself. On creation, the index has no wordCount, no documentCount, and is, as expected, fairly empty. >>> from zc.catalog.index import SetIndex >>> index = SetIndex() >>> index.documentCount() 0 >>> index.wordCount() 0 >>> index.maxValue() # doctest: +ELLIPSIS Traceback (most recent call last): ... ValueError:... >>> index.minValue() # doctest: +ELLIPSIS Traceback (most recent call last): ... ValueError:... >>> list(index.values()) [] >>> len(index.apply({'any_of': (5,)})) 0 The index supports indexing any value. All values within a given index must sort consistently across Python versions. In our example, we hope that strings and integers will sort consistently; this may not be a reasonable hope. >>> data = {1: ['a', 1], ... 2: ['b', 'a', 3, 4, 7], ... 3: [1], ... 4: [1, 4, 'c'], ... 5: [7], ... 6: [5, 6, 7], ... 7: ['c'], ... 8: [1, 6], ... 9: ['a', 'c', 2, 3, 4, 6,], ... } >>> for k, v in data.items(): ... index.index_doc(k, v) ... After indexing, the statistics and values match the newly entered content. >>> list(index.values()) [1, 2, 3, 4, 5, 6, 7, 'a', 'b', 'c'] >>> index.documentCount() 9 >>> index.wordCount() 10 >>> index.maxValue() 'c' >>> index.minValue() 1 >>> list(index.ids()) [1, 2, 3, 4, 5, 6, 7, 8, 9] The index supports five types of query. The first is 'any_of'. It takes an iterable of values, and returns an iterable of document ids that contain any of the values. The results are weighted. >>> list(index.apply({'any_of':('b', 1, 5)})) [1, 2, 3, 4, 6, 8] >>> list(index.apply({'any_of': ('b', 1, 5)})) [1, 2, 3, 4, 6, 8] >>> list(index.apply({'any_of':(42,)})) [] >>> index.apply({'any_of': ('a', 3, 7)}) BTrees._IFBTree.IFBucket([(1, 1.0), (2, 3.0), (5, 1.0), (6, 1.0), (9, 2.0)]) Another query is 'qny', If the key is None, all indexed document ids with any values are returned. If the key is an extent, the intersection of the extent and all document ids with any values is returned. >>> list(index.apply({'any': None})) [1, 2, 3, 4, 5, 6, 7, 8, 9] >>> from zc.catalog.extentcatalog import FilterExtent >>> extent = FilterExtent(lambda extent, uid, obj: True) >>> for i in range(15): ... extent.add(i, i) ... >>> list(index.apply({'any': extent})) [1, 2, 3, 4, 5, 6, 7, 8, 9] >>> limited_extent = FilterExtent(lambda extent, uid, obj: True) >>> for i in range(5): ... limited_extent.add(i, i) ... >>> list(index.apply({'any': limited_extent})) [1, 2, 3, 4] The 'contains_all' argument also takes an iterable of values, but returns an iterable of document ids that contains all of the values. The results are not weighted. >>> list(index.apply({'all_of': ('a',)})) [1, 2, 9] >>> list(index.apply({'all_of': (3, 4)})) [2, 9] The 'between' argument takes from 1 to four values. The first is the minimum, and defaults to None, indicating no minimum; the second is the maximum, and defaults to None, indicating no maximum; the next is a boolean for whether the minimum value should be excluded, and defaults to False; and the last is a boolean for whether the maximum value sh
Re: [Zope3-dev] ServerControlForm.html / Restart, Shutdown does not work
[Adam Groszer] >> If I click "Shutdown" on the ServerControlForm.html it does not >> shutdown Zope, but responds with a simple "You shut down the server." >> message in the browser and Zope is still working. >> >> I think the problem lies in the modified ThreadedAsync. Darn. I'm sure I broke this, when removing layers of apparently unused cruft from LoopCallback.py. Apparently there was voodoo control flow intended here, wherein Zope stuffs a module global into LoopCallback and expects ... well, _something_ ... to happen as a result. I guess it expects "the server to shut down" . [Stephan Richter] > Oh, that seems strange. However, I propose not to fix the bug, since we will > switch to Twisted in 3.2 and there it should definitely work. I don't think that will help (this is apparently aimed at ZEO, at least in part, and I have no plan to switch ZEO to Twisted). Looks like you added the "voodoo control flow" in rev 3432 of LoopCallback.py (March 2004), so we should figure out what you wanted and how to get there now. The signature of asyncore.loop changed in Python 2.4, and it was hugely evil regardless that LoopCallback monkey-patched asyncore.loop with its own loop function, never invoking Python's asyncore.loop (I shudder to imagine how many days of debugging time were lost by various people wondering why breakpoints and debugging prints in asyncore.loop never triggered). So in ZODB 3.4.1a1, I changed LoopCallback.py's loop() to invoke Python's asyncore.loop instead That way it reproduces the correct loop() signature for the version of Python in use, and stopped magically rendering debug code in Python's asyncore.loop useless. Now I want to stop ZEO from sharing its asyncore socket map with other subsystems using asyncore, and for several reasons. When that happens, Zope will need a different way to tell ZEO to stop. We should think about a sane API for that now, although it looks like a hack is needed to get over the immediate problem. This also seems to affect the Zope 2.8 line (hard to tell for sure -- I don't understand the code intent here), and if so waiting for Zope 3.2 isn't good enough on that count either. Question: was the value of LoopCallback.exit_status used by anything? I've been unable to find any code (in any version of ZODB, Zope, or zdaemon) that *looked* at LoopCallback.exit_status, apart from the internal reference in LoopCallback.loop(). There it only cared whether or not the value was None. Did the distinction between a value of 1 and a value of 2 have any meaning? I'm trying to dream up a sane API here, and the first step is to figure out what the code thought it was doing (before I broke it). Another: did this actually work before? The only place I've found that set LoopCallback.exit_status is zope/app/server/servercontrol.py, and both methods there have "TODO: ... does not work yet" comments (in both Zope and Zope3 SVN trunks). ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Re: ServerControlForm.html / Restart, Shutdown does not work
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Stephan Richter wrote: > On Saturday 16 July 2005 03:48, Adam Groszer wrote: > >>If I click "Shutdown" on the ServerControlForm.html it does not >>shutdown Zope, but responds with a simple "You shut down the server." >>message in the browser and Zope is still working. >> >>I think the problem lies in the modified ThreadedAsync. > > > Oh, that seems strange. However, I propose not to fix the bug, since we will > switch to Twisted in 3.2 and there it should definitely work. I don't think this is acceptable, given that 3.1 isn't out yet, and that it will be months before 3.2 is out. AFAIK, the choice to replace zasync with Twisted might not even *land* in 3.2, assuming we stick to our time-based release schedule. Tres. - -- === Tres Seaver +1 202-558-7113 [EMAIL PROTECTED] Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFC3BDf+gerLs4ltQ4RAjtaAJ9cQ3icYdgmHbnPZ9lDo5MMAwK5xwCguvZl 4we5olCySZzNm2Y3NlP9o00= =ROh7 -END PGP SIGNATURE- ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] KeywordIndex
On Mon, 2005-07-18 at 12:00 -0400, [EMAIL PROTECTED] wrote: > Date: Mon, 18 Jul 2005 09:14:16 -0600 > From: Jeff Shell <[EMAIL PROTECTED]> > Subject: [Zope3-dev] KeywordIndex > To: zope3-dev@zope.org > Message-ID: <[EMAIL PROTECTED]> > Content-Type: text/plain; charset=ISO-8859-1 > > I'm working on a simple application which is the first time I get to > use the catalog in Zope 3. I'm writing against Zope 3.1b1. I was > dismayed not to see KeywordIndex in the main catalog set, but then I > found it in zope.index.keyword. But it seems to be a bit behind. I > have it somewhat working through subclassing, etc, but it's been > purely guess work on my part to get things this far. In my product > package, I have the following: 1. I'm unable to help you directly with your problem, although Gary's post about the SetIndex looks very promising and I would like to see that code as well. As you said, something like a keyword index is exactly what your application is designed around, but if I could digress from the topic a little I'd like to suggest another solution, rdflib. rdflib covers some of the same use cases as a keyword index. Your objects (content, whatever) and your keywords would be assigned unique identifiers. You then add relations to an rdflib.Graph that associate the keyword with your objects: >>> dc = rdflib.Namespace('http://purl.org/dc/elements/1.1/') >>> blue = rdflib.BNode() # creates a URI for you >>> g = rdflib.Graph() >>> g.add((object_uri, dc.keywords, blue)) and then query the data back out with either a low-level g.triples((s, p, o)) pattern or a sparql query. For example, print a list of all object URIs that have the blue_uri keyword: >>> print [s for (s, p, o) in g.triples((rdflib.Any, dc.keywords, blue))] or sparql: >>> sg = rdflib.sparqlGraph(g) >>> select = ("?object_uri",) >>> where = rdflib.GraphPattern([("?object_uri", dc.keywords, blue)]) >>> for object_uri in sg.query(select, where): print object_uri is the same query, but longer. With sparql you can do more complex sql like queries however against other relations. Hope this helps, -Michel ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] zope.security.interfaces.ForbiddenAttribute: ('utcoffset', )
This might be something on my end, but I figure I'd throw it out in case it's related to any changes related to naïve/non-naïve time zones. Here's the relevant part of the traceback: File "/opt/aktari/zope/src/zope/i18n/format.py", line 175, in format info = buildDateTimeInfo(obj, self.calendar, bin_pattern) File "/opt/aktari/zope/src/zope/i18n/format.py", line 603, in buildDateTimeInfo tz_secs = tzinfo.utcoffset(dt).seconds I'll dig further, but at a casual glance I couldn't turn up any security declarations for UTC objects. Could this be a case where datetime values are suddenly spouting proxied tzinfos? -- Garrett ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Z-I with GCC 4
I tried installing Zope Interface on FC4 using GCC 4, and I received the following error: running install running build running build_py running build_ext building 'zope.interface._zope_interface_coptimizations' extension gcc -pthread -fno-strict-aliasing -DNDEBUG -O2 -g -pipe -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -m32 -march=i386 -mtune=pentium4 -fasynchronous-unwind-tables -D_GNU_SOURCE -fPIC -fPIC -IDependencies/zope.interface-ZopeInterface-3.0.0/zope.interface -I/usr/include/python2.4 -c Dependencies/zope.interface-ZopeInterface-3.0.0/zope.interface/_zope_interface_coptimizations.c -o build/temp.linux-i686-2.4/Dependencies/zope.interface-ZopeInterface-3.0.0/zope.interface/_zope_interface_coptimizations.o Dependencies/zope.interface-ZopeInterface-3.0.0/zope.interface/_zope_interface_coptimizations.c:339: error: static declaration of ‘SpecType’ follows non-static declaration Dependencies/zope.interface-ZopeInterface-3.0.0/zope.interface/_zope_interface_coptimizations.c:73: error: previous declaration of ‘SpecType’ was here error: command 'gcc' failed with exit status 1 Does Zope not yet work with GCC 4? Is there anything I can do to work around this? Thanks, Chris Spencer ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
[Zope3-dev] Formatting dates
Now that dates have UTC time zones associated with them, will we be adjusting how they're displayed in various views? Somehow it doesn't seem appropriate to display UTC by default. I'd assume Zope would use the server's timezone offset. -- Garrett ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com
Re: [Zope3-dev] KeywordIndex
Hi Gary! I'd be very interested in this. It's not critical for me right now, so there's no need to rush making something available. I have an inefficient but fun solution for my system that can be replaced when this comes along. I primarily wanted to know the state of the indexes. Is what's there right now going to be what ships with Zope 3.1 final? On 7/18/05, Gary Poster <[EMAIL PROTECTED]> wrote: > > On Jul 18, 2005, at 11:14 AM, Jeff Shell wrote: > > > I'm working on a simple application which is the first time I get to > > use the catalog in Zope 3. I'm writing against Zope 3.1b1. I was > > dismayed not to see KeywordIndex in the main catalog set, but then I > > found it in zope.index.keyword. But it seems to be a bit behind. > > Hi. Yes, we needed it too. Here's another thing we want to open > source. Look at the attached .txt file; if you want it then tell me > and I'll make it available in a sandbox. We'll move it over into the > Zope repo (probably with a new name, or rearranged on the appropriate > locations (zope.index and zope.app.catalog, etc.) RSN. > > Downsides: > > - Note that some functionality requires that you use an extent > catalog, another goodie in the package. > > - We have some refactoring of this that we want to do. We'll have > legacy issues ourselves, then. > > Additional upside: > > - This package also includes a replacement for the field index > (called a value index) and customizations of the value and set > indexes specific to timezone-aware datetimes, as well as a few other > things. ___ Zope3-dev mailing list Zope3-dev@zope.org Unsub: http://mail.zope.org/mailman/options/zope3-dev/archive%40mail-archive.com