Re: [Tutor] perplexing error with shelve REVISED
On 10/31/07, Orest Kozyar [EMAIL PROTECTED] wrote: Please post the entire traceback (omitting duplicate lines). Sorry, I should have included the traceback. I've revised the sample script so that it generates the traceback when run. The sample script is at the very bottom of this email. ### SCRIPT OUTPUT ### [kozyar]:~$ python example.py Successfully retrieved and parsed XML document with ID 16842423 Successfully shelved XML document with ID 16842423 Successfully retrieved and parsed XML document with ID 16842422 Traceback (most recent call last): File example.py, line 30, in module data[badkey] = doc File /usr/lib64/python2.5/shelve.py, line 123, in __setitem__ p.dump(value) RuntimeError: maximum recursion depth exceeded Exception exceptions.RuntimeError: 'maximum recursion depth exceeded' in bound method DbfilenameShelf.__del__ of {'16842423': xml.dom.minidom.Document instance at 0x96f290} ignored ### START SCRIPT ### import urllib, shelve from xml.dom import minidom baseurl = 'http://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?' params = { 'db': 'pubmed', 'retmode': 'xml', 'rettype': 'medline' } badkey = '16842422' goodkey = '16842423' #or just about any other ID data = shelve.open('data.tmp', writeback=True) params['id'] = goodkey url = baseurl + urllib.urlencode(params) doc = minidom.parseString(urllib.urlopen(url).read()) print 'Successfully retrieved and parsed XML document with ID %s' % goodkey data[goodkey] = doc print 'Successfully shelved XML document with ID %s' % goodkey params['id'] = badkey url = baseurl + urllib.urlencode(params) doc = minidom.parseString(urllib.urlopen(url).read()) print 'Successfully retrieved and parsed XML document with ID %s' % badkey data[badkey] = doc #Will fail on the above line print 'Successfully shelved XML document with ID %s' % badkey The program worked fine for me (python 2.5.1 - stackless on Mac OSX). Can you provide details about your platform (python version, OS, etc.) ? -- Aditya ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] perplexing error with shelve REVISED
Orest Kozyar wrote: Please post the entire traceback (omitting duplicate lines). Sorry, I should have included the traceback. I've revised the sample script so that it generates the traceback when run. The sample script is at the very bottom of this email. ### SCRIPT OUTPUT ### [kozyar]:~$ python example.py Successfully retrieved and parsed XML document with ID 16842423 Successfully shelved XML document with ID 16842423 Successfully retrieved and parsed XML document with ID 16842422 Traceback (most recent call last): File example.py, line 30, in module data[badkey] = doc File /usr/lib64/python2.5/shelve.py, line 123, in __setitem__ p.dump(value) RuntimeError: maximum recursion depth exceeded Exception exceptions.RuntimeError: 'maximum recursion depth exceeded' in bound method DbfilenameShelf.__del__ of {'16842423': xml.dom.minidom.Document instance at 0x96f290} ignored ### START SCRIPT ### import urllib, shelve from xml.dom import minidom baseurl = 'http://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?' params = { 'db': 'pubmed', 'retmode': 'xml', 'rettype': 'medline' } badkey = '16842422' goodkey = '16842423' #or just about any other ID data = shelve.open('data.tmp', writeback=True) params['id'] = goodkey url = baseurl + urllib.urlencode(params) doc = minidom.parseString(urllib.urlopen(url).read()) print 'Successfully retrieved and parsed XML document with ID %s' % goodkey data[goodkey] = doc print 'Successfully shelved XML document with ID %s' % goodkey params['id'] = badkey url = baseurl + urllib.urlencode(params) doc = minidom.parseString(urllib.urlopen(url).read()) print 'Successfully retrieved and parsed XML document with ID %s' % badkey The problem seems to be with pickling the doc object. try adding: import pickle print pickle.dumps(doc) right here in your code. Are you sure the XML is well formed? If nothing else, you could try shelving the XML rather than the doc. data[badkey] = doc #Will fail on the above line print 'Successfully shelved XML document with ID %s' % badkey ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] perplexing error with shelve REVISED
Orest Kozyar wrote: Please post the entire traceback (omitting duplicate lines). Sorry, I should have included the traceback. I've revised the sample script so that it generates the traceback when run. The sample script is at the very bottom of this email. It appears you have a cyclic reference in your doc object. Try adding doc.unlink() before you add it to your shelf. ### SCRIPT OUTPUT ### [kozyar]:~$ python example.py Successfully retrieved and parsed XML document with ID 16842423 Successfully shelved XML document with ID 16842423 Successfully retrieved and parsed XML document with ID 16842422 Traceback (most recent call last): File example.py, line 30, in module data[badkey] = doc File /usr/lib64/python2.5/shelve.py, line 123, in __setitem__ p.dump(value) RuntimeError: maximum recursion depth exceeded Exception exceptions.RuntimeError: 'maximum recursion depth exceeded' in bound method DbfilenameShelf.__del__ of {'16842423': xml.dom.minidom.Document instance at 0x96f290} ignored ### START SCRIPT ### import urllib, shelve from xml.dom import minidom baseurl = 'http://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?' params = { 'db': 'pubmed', 'retmode': 'xml', 'rettype': 'medline' } badkey = '16842422' goodkey = '16842423' #or just about any other ID data = shelve.open('data.tmp', writeback=True) params['id'] = goodkey url = baseurl + urllib.urlencode(params) doc = minidom.parseString(urllib.urlopen(url).read()) print 'Successfully retrieved and parsed XML document with ID %s' % goodkey data[goodkey] = doc print 'Successfully shelved XML document with ID %s' % goodkey params['id'] = badkey url = baseurl + urllib.urlencode(params) doc = minidom.parseString(urllib.urlopen(url).read()) print 'Successfully retrieved and parsed XML document with ID %s' % badkey data[badkey] = doc #Will fail on the above line print 'Successfully shelved XML document with ID %s' % badkey ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] perplexing error with shelve REVISED
Orest Kozyar wrote: Please post the entire traceback (omitting duplicate lines). Sorry, I should have included the traceback. I've revised the sample script so that it generates the traceback when run. The sample script is at the very bottom of this email. I've poked at this a little. The problem is in the pickling of the minidom object. It fails for me using Python 2.5.1 (r251:54869, Apr 18 2007, 22:08:04) [GCC 4.0.1 (Apple Computer, Inc. build 5367)] on darwin If you use pickle instead of cPickle the stack trace at least shows the recursive calls. Here is a slightly shorter program that demonstrates the problem a little better: import urllib from pickle import Pickler from cStringIO import StringIO from xml.dom import minidom baseurl = 'http://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?' params = { 'db': 'pubmed', 'retmode': 'xml', 'rettype': 'medline', } badkey = '16842422' params['id'] = badkey url = baseurl + urllib.urlencode(params) doc = minidom.parseString(urllib.urlopen(url).read()) print 'Successfully retrieved and parsed XML document with ID %s' % badkey f = StringIO() p = Pickler(f, 0) p.dump(doc) #Will fail on the above line print 'Successfully shelved XML document with ID %s' % badkey Here is the top of the stack trace: File BadShelve.py, line 35, in module p.dump(doc) File /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py, line 224, in dump self.save(obj) File /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py, line 286, in save f(self, obj) # Call unbound method with explicit self File /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py, line 725, in save_inst save(stuff) File /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py, line 286, in save f(self, obj) # Call unbound method with explicit self File /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py, line 649, in save_dict self._batch_setitems(obj.iteritems()) File /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py, line 663, in _batch_setitems save(v) File /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py, line 286, in save f(self, obj) # Call unbound method with explicit self File /Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/pickle.py, line 725, in save_inst save(stuff) You might want to take this to comp.lang.python or perhaps xml-sig. Kent ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] perplexing error with shelve REVISED
It appears you have a cyclic reference in your doc object. Try adding doc.unlink() before you add it to your shelf. That fixed the problem. I did not realize that XML documents could have cyclic references. Thanks for all your help! Orest ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] perplexing error with shelve REVISED
Eric Brunson wrote: Orest Kozyar wrote: Please post the entire traceback (omitting duplicate lines). Sorry, I should have included the traceback. I've revised the sample script so that it generates the traceback when run. The sample script is at the very bottom of this email. It appears you have a cyclic reference in your doc object. Try adding doc.unlink() before you add it to your shelf. I thought pickle was supposed to deal with cyclic references? The docs say, The pickle module keeps track of the objects it has already serialized, so that later references to the same object won't be serialized again. marshal doesn't do this. This has implications both for recursive objects and object sharing. Recursive objects are objects that contain references to themselves. These are not handled by marshal, and in fact, attempting to marshal recursive objects will crash your Python interpreter. Object sharing happens when there are multiple references to the same object in different places in the object hierarchy being serialized. pickle stores such objects only once, and ensures that all other references point to the master copy. Shared objects remain shared, which can be very important for mutable objects. Kent ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] perplexing error with shelve REVISED
It appears you have a cyclic reference in your doc object. Try adding doc.unlink() before you add it to your shelf. Actually, I just realized that doc.unlink() seems to delete the entire XML content, which kind of defeats the purpose of caching it. I'll check on xml-sig and the comp.lang.python group to see if there are any suggestions. Thanks again for your help! Orest ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] perplexing error with shelve REVISED
Orest Kozyar wrote: I have a program which queries an online database (Medline) for XML data. It caches all data using shelve to avoid hitting the database too many times. For some reason, I keep getting a RuntimeError: maximum recursion depth exceeded when attempting to add a certain record. Please post the entire traceback (omitting duplicate lines). Else we can't (or won't try to) help. ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] perplexing error with shelve REVISED
Please post the entire traceback (omitting duplicate lines). Sorry, I should have included the traceback. I've revised the sample script so that it generates the traceback when run. The sample script is at the very bottom of this email. ### SCRIPT OUTPUT ### [kozyar]:~$ python example.py Successfully retrieved and parsed XML document with ID 16842423 Successfully shelved XML document with ID 16842423 Successfully retrieved and parsed XML document with ID 16842422 Traceback (most recent call last): File example.py, line 30, in module data[badkey] = doc File /usr/lib64/python2.5/shelve.py, line 123, in __setitem__ p.dump(value) RuntimeError: maximum recursion depth exceeded Exception exceptions.RuntimeError: 'maximum recursion depth exceeded' in bound method DbfilenameShelf.__del__ of {'16842423': xml.dom.minidom.Document instance at 0x96f290} ignored ### START SCRIPT ### import urllib, shelve from xml.dom import minidom baseurl = 'http://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi?' params = { 'db': 'pubmed', 'retmode': 'xml', 'rettype': 'medline' } badkey = '16842422' goodkey = '16842423' #or just about any other ID data = shelve.open('data.tmp', writeback=True) params['id'] = goodkey url = baseurl + urllib.urlencode(params) doc = minidom.parseString(urllib.urlopen(url).read()) print 'Successfully retrieved and parsed XML document with ID %s' % goodkey data[goodkey] = doc print 'Successfully shelved XML document with ID %s' % goodkey params['id'] = badkey url = baseurl + urllib.urlencode(params) doc = minidom.parseString(urllib.urlopen(url).read()) print 'Successfully retrieved and parsed XML document with ID %s' % badkey data[badkey] = doc #Will fail on the above line print 'Successfully shelved XML document with ID %s' % badkey ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor