Re: [Zope-dev] KeywordIndex errors
2002-11-27T16:16:26 ERROR(200) KeywordIndex unindex_object could not remove documentId -869933785 from index companies. This should not happen. exceptions.KeyError: -869933785 This usually happens when there are duplicates in the indexed keywords and the record isn't updated before unindexing. Shouldn't cause any problems, though it isn't nice. This can be easily fixed by changing the KeywordIndex' index_object method to make keywords unique before indexing. Hmm, recall that PathIndex error I reported a few days ago? That was also an unindex error, also exercised by migrating to 2.6.0. I wonder why the process of *migrating* should cause dups in the index...? seb Hi, started seeing quite a few of these showing up in one of our server's logs: -- 2002-11-27T16:16:26 ERROR(200) KeywordIndex unindex_object could not remove documentId -869933785 from index companies. This should not happen. exceptions.KeyError: -869933785 This is Zope 2.6.0, hence the total lack of traceback :-( Anyone got any idea what's going on here? cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] KeywordIndex errors
seb bacon wrote: 2002-11-27T16:16:26 ERROR(200) KeywordIndex unindex_object could not remove documentId -869933785 from index companies. This should not happen. exceptions.KeyError: -869933785 This usually happens when there are duplicates in the indexed keywords and the record isn't updated before unindexing. Shouldn't cause any problems, though it isn't nice. This can be easily fixed by changing the KeywordIndex' index_object method to make keywords unique before indexing. Hmm, recall that PathIndex error I reported a few days ago? That was also an unindex error, also exercised by migrating to 2.6.0. I wonder why the process of *migrating* should cause dups in the index...? The errors were reported when I was deleting object created in Zope 2.6.0 already, but that instance of Plone probably migrated 2.5 - 2.6. We are migrating Plone, not Zope right now... m. -- Myroslav Opyr zope.net.ua http://zope.net.ua/ ° Ukrainian Zope Hosting e-mail: [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] KeywordIndex errors
- Original Message - From: seb bacon [EMAIL PROTECTED] Sent: Thursday, November 28, 2002 11:22 AM Subject: Re: [Zope-dev] KeywordIndex errors 2002-11-27T16:16:26 ERROR(200) KeywordIndex unindex_object could not remove documentId -869933785 from index companies. This should not happen. exceptions.KeyError: -869933785 This usually happens when there are duplicates in the indexed keywords and the record isn't updated before unindexing. Shouldn't cause any problems, though it isn't nice. This can be easily fixed by changing the KeywordIndex' index_object method to make keywords unique before indexing. Hmm, recall that PathIndex error I reported a few days ago? That was also an unindex error, also exercised by migrating to 2.6.0. I wonder why the process of *migrating* should cause dups in the index...? It shouldn't cause them, they were probably already there. The PathIndex errors resulted from a change in the api, I don't think they are related to the errors Chris was seeing. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] KeywordIndex errors
Hi, started seeing quite a few of these showing up in one of our server's logs: -- 2002-11-27T16:16:26 ERROR(200) KeywordIndex unindex_object could not remove documentId -869933785 from index companies. This should not happen. exceptions.KeyError: -869933785 This is Zope 2.6.0, hence the total lack of traceback :-( Anyone got any idea what's going on here? cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] KeywordIndex errors
--On Mittwoch, 27. November 2002 18:08 + Chris Withers [EMAIL PROTECTED] wrote: Hi, started seeing quite a few of these showing up in one of our server's logs: -- 2002-11-27T16:16:26 ERROR(200) KeywordIndex unindex_object could not remove documentId -869933785 from index companies. This should not happen. exceptions.KeyError: -869933785 This looks like another i-will-never-go-away-bug. In fact I encountered this problem several times but it does not seem to be reproducable and I have no glue about the reason. :-( -aj ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] KeywordIndex errors
Andreas Jung wrote: --On Mittwoch, 27. November 2002 18:08 + Chris Withers [EMAIL PROTECTED] wrote: Hi, started seeing quite a few of these showing up in one of our server's logs: -- 2002-11-27T16:16:26 ERROR(200) KeywordIndex unindex_object could not remove documentId -869933785 from index companies. This should not happen. exceptions.KeyError: -869933785 This looks like another i-will-never-go-away-bug. In fact I encountered this problem several times but it does not seem to be reproducable and I have no glue about the reason. :-( We've seen that too and in a big amounts. We have experimaental Data.fs for Plone migration purposes (branch-copy of production one). For ease of testing I was deleting almost all content to speed up migration processes under inspection. Documents have KeywordIndex applied also. And I've seen something like several hundreds such errors when deleting 3000 documents. Thus it appear to happen more or less each 10th unindex_object. I'd say rather reproducible case. m. -- Myroslav Opyr zope.net.ua http://zope.net.ua/ ° Ukrainian Zope Hosting e-mail: [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] KeywordIndex errors
This usually happens when there are duplicates in the indexed keywords and the record isn't updated before unindexing. Shouldn't cause any problems, though it isn't nice. This can be easily fixed by changing the KeywordIndex' index_object method to make keywords unique before indexing. - Original Message - Sent: Wednesday, November 27, 2002 7:08 PM Subject: [Zope-dev] KeywordIndex errors Hi, started seeing quite a few of these showing up in one of our server's logs: -- 2002-11-27T16:16:26 ERROR(200) KeywordIndex unindex_object could not remove documentId -869933785 from index companies. This should not happen. exceptions.KeyError: -869933785 This is Zope 2.6.0, hence the total lack of traceback :-( Anyone got any idea what's going on here? cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] KeywordIndex errors
--On Donnerstag, 28. November 2002 01:48 +0100 Wolfram Kerber [EMAIL PROTECTED] wrote: This usually happens when there are duplicates in the indexed keywords and the record isn't updated before unindexing. Shouldn't cause any problems, though it isn't nice. This can be easily fixed by changing the KeywordIndex' index_object method to make keywords unique before indexing. Can you submit a patch? -aj ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] KeywordIndex errors: This should not happen.
On Mon, 30 Sep 2002 09:50:56 -0400 Casey Duncan [EMAIL PROTECTED] wrote: On Monday 30 September 2002 08:36 am, Roché Compaan wrote: Don't know if this will help but these errors specifically occur with 'lines' properties with Keyword Indexes on them. What is also significant is that there are legitimate duplicates in the lines properties and that they are duplicated on the index data as well. This doesn't feel right. I would have thought that only unique values should be stored in the index data. If you feel this is a bug, you should file a collector issue for it. I'm not sure yet ... -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] KeywordIndex errors: This should not happen.
Have you tried it in 2.6? Yeah, this'd be my suggestion having suffered more terminally from exactly these bugs :-S Hmm! That's interesting. I was worried that we might be causing it by tickling the Catalog the wrong way .. Did you ever get to the bottom of the bugs? Do they have collector issues, perhaps? -- Jean Jordaan Upfront Systems http://www.upfrontsystems.co.za ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] KeywordIndex errors: This should not happen.
On Tue, Oct 01, 2002 at 08:37:16AM +0200, Jean Jordaan wrote: Have you tried it in 2.6? Yeah, this'd be my suggestion having suffered more terminally from exactly these bugs :-S Hmm! That's interesting. I was worried that we might be causing it by tickling the Catalog the wrong way .. Did you ever get to the bottom of the bugs? Do they have collector issues, perhaps? It would be fine if we could get a reproducable testcase for this problem. I have seen this problem multiple times in the past I have not been able to track it down to a simple testcase. Andreas ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] KeywordIndex errors: This should not happen.
It would be fine if we could get a reproducable testcase for this problem. Absolutely. When the error we see occurs, the RID that is looked for has already gone missing. It looks like the real error happens silently sometime before, and not in all cases: the majority of the instances of the affected class are cataloged OK. -- Jean Jordaan Upfront Systems http://www.upfrontsystems.co.za ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] KeywordIndex errors: This should not happen.
Jean Jordaan wrote: Hmm! That's interesting. I was worried that we might be causing it by tickling the Catalog the wrong way .. Did you ever get to the bottom of the bugs? Do they have collector issues, perhaps? They were bugs in Zope's BTree implementation which were fixed for 2.6. cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] KeywordIndex errors: This should not happen.
Hi all
Our app has been hitting these errors for months:
2002-09-30T11:06:27 ERROR(200) KeywordIndex unindex_object could not remove
documentId -444153566 from index ComponentVersions. This should not happen.
Traceback (innermost last):
File
/usr/local/zope/2-5-1/lib/python/Products/PluginIndexes/common/UnIndex.py, line
168, in removeForwardIndexEntry
(Object: ComponentVersions)
KeyError: -444153566
Here is the relevant piece Zope code::
def removeForwardIndexEntry(self, entry, documentId):
Take the entry provided and remove any reference to documentId
in its entry in the index.
global _marker
indexRow = self._index.get(entry, _marker)
if indexRow is not _marker:
try:
indexRow.remove(documentId)
if not indexRow:
del self._index[entry]
try: self.__len__.change(-1)
except AttributeError: pass # pre-BTrees-module instance
except AttributeError:
# index row is an int
del self._index[entry]
try: self.__len__.change(-1)
except AttributeError: pass # pre-BTrees-module instance
except:
import pdb; pdb.set_trace() #DBG
I inserted that 'set_trace' just before the error is logged. At this
point, the app state looks like this::
(Pdb) self
KeywordIndex instance at 9544940
(Pdb) p self.id
'ComponentVersions'
(Pdb) entry
'44.131'
(Pdb) p indexRow
IITreeSet object at 0x1355a3d0
(Pdb) len(indexRow.keys())
566
(Pdb) indexRow.remove(documentId)
*** KeyError: -444153566
(Pdb) [self._index.keys()[n] for n in range(0,69)]
['', ' 2.13', '1', '1.14', '1.22', '1.25', '1.3', '1.6', '1.8.7',
'1.9.8', '17.9', '2.0.6', '2.1', '2.15', '2.2.1', '2.2.3m', '2.2.7',
'2.2.7m', '2.3.4m', '2.4', '2.4.2m', '2.4.3m', '2.4.4m', '2.4.5',
'2.5', '2.6.6m', '2.7.7m', '2.8.5m', '2.8.6m', '2.8.7m', '2.9',
'20.13-37', '22.13-42', '22.42-47', '22.5-42', '24.12-49',
'24.13-49', '24.19-49', '4.4', '4.9', '40.54e', '41.184', '42.95',
'43.11', '43.96', '43.96 ', '44.120', '44.131', '45.11', '7.3',
'7.7', '8.0', '8.1', '8.2', '8.4', '8.4.1', '8.5.9', '8.7.2m',
'8.7.6m', '8.8.9m', '8.9.6m', '8.9.7m', '9.0', '9.12', '9.2', '9.3',
'9.4 ', '9.8', 'x']
(Pdb) [indexRow.keys()[n] for n in range(0, len(indexRow.keys()))]
[-1948802264, -1018656130, -444153568, -111852827, -32258423,
-32258421, -32258398, -32258397, -32258396, 226891955, (and so on)
(Pdb) catalog = self.restrictedTraverse('Catalog')
(Pdb) catalog.getpath(-444153566)
'/Licenses/LicenseDocs/License_377'
(Pdb) catalog.getIndexDataForRID(-444153566)['ComponentVersions']
['2.4', '8.9.7m', '44.131', '44.131', '2.4.3m', '4.9', '9.12',
'24.13-49', '1.25', '2.15', '9.3', '2.8.6m']
Rephrasing that, I'd say: we're trying to remove the RID -444153566
from the list returned by: indexRow.keys()['44.131']
That RID is absent from that list, which raises a KeyError that
should not happen. There *is* an entry in the Catalog for that RID,
which *has* '44.131' in the list under the 'ComponentVersions' key in
the dict returned by getIndexDataForRID.
Did the RID get removed from the KeywordIndex list of RIDs prematurely?
Or did it never get inserted? Can anyone shed any enlightenment? Where
else should I be looking?
--
Jean Jordaan
Upfront Systems http://www.upfrontsystems.co.za
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] KeywordIndex errors: This should not happen.
Don't know if this will help but these errors specifically occur with 'lines' properties with Keyword Indexes on them. What is also significant is that there are legitimate duplicates in the lines properties and that they are duplicated on the index data as well. This doesn't feel right. I would have thought that only unique values should be stored in the index data. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za On Mon, 30 Sep 2002 14:17:17 +0200 Jean Jordaan [EMAIL PROTECTED] wrote: Hi all Our app has been hitting these errors for months: 2002-09-30T11:06:27 ERROR(200) KeywordIndex unindex_object could not remove documentId -444153566 from index ComponentVersions. This should not happen. Traceback (innermost last): File /usr/local/zope/2-5-1/lib/python/Products/PluginIndexes/common/UnIndex.py, line 168, in removeForwardIndexEntry (Object: ComponentVersions) KeyError: -444153566 ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] KeywordIndex errors: This should not happen.
On Monday 30 Sep 2002 1:17 pm, Jean Jordaan wrote: Hi all Our app has been hitting these errors for months: Have you tried it in 2.6? A couple of bug in this area have been fixed since 2.5.x ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] KeywordIndex errors: This should not happen.
On Monday 30 September 2002 08:36 am, Roché Compaan wrote: Don't know if this will help but these errors specifically occur with 'lines' properties with Keyword Indexes on them. What is also significant is that there are legitimate duplicates in the lines properties and that they are duplicated on the index data as well. This doesn't feel right. I would have thought that only unique values should be stored in the index data. If you feel this is a bug, you should file a collector issue for it. -Casey ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] KeywordIndex errors: This should not happen.
If you feel this is a bug, you should file a collector issue for it. We don't have a deep enough understanding of the Catalog to diagnose :/ We're not sure if it's a Catalog issue, or something we're doing wrong; i.e. we don't know how the Catalog gets into that state. -- Jean Jordaan Upfront Systems http://www.upfrontsystems.co.za ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] KeywordIndex errors: This should not happen.
Toby Dickenson wrote: Our app has been hitting these errors for months: Have you tried it in 2.6? A couple of bug in this area have been fixed since 2.5.x Yeah, this'd be my suggestion having suffered more terminally from exactly these bugs :-S cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
