Re: [ZODB-Dev] problem with _p_mtime
Hi, Thanks for your help, everyone. Some of my replies seemed to go off-list, so just updating this in case anyone else has the same issue. I managed to solve the problem by exporting the full folder tree as a .zexp, and re-importing it. This fixed up the _p_mtime values to the date it was imported. This worked for us because the data was not that huge. Before this, I had a temporary workaround that caught the exception in App.PersistentExtra when trying to read _p_mtime, and replaced it with today's date. It also set _p_changed to write the data back. However this just caused performance problems, and the data never seemed to get written back. fsrefs still returned errors on the new data, as it did on the old. In particular, it reported it could not find objects that I could load from the "zopectl debug" prompt. The data works fine and does not cause errors, so it looks like an issue in fsrefs. I have not investigated this further. Thanks again, Miles Miles wrote: > Hi, > > I've moved a FileStorage from one (old) machine to another (new) > machine, but when I mount it on the new machine I get a lot of time errrors: > > Traceback (innermost last): >Module ZPublisher.Publish, line 115, in publish >Module ZPublisher.mapply, line 88, in mapply >Module ZPublisher.Publish, line 41, in call_object >Module Shared.DC.Scripts.Bindings, line 311, in __call__ >Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec >Module App.special_dtml, line 176, in _exec >Module DocumentTemplate.DT_Let, line 76, in render >Module DocumentTemplate.DT_In, line 703, in renderwob >Module DocumentTemplate.DT_With, line 76, in render >Module DocumentTemplate.DT_Var, line 214, in render >Module App.PersistentExtra, line 43, in bobobase_modification_time >Module DateTime.DateTime, line 509, in __init__ >Module DateTime.DateTime, line 760, in _parse_args >Module DateTime.DateTime, line 437, in safelocaltime > TimeError: The time 98040302366.810165 is beyond the range of this > Python implementation. > > I googled around on this problem and didn't find much, but I did find > that as a temporary fix, I've put a try/except in App.PersistentExtra, > so if it generates an exception, it returns the current date and time. > This keeps things working. > > Out of interst, can anyone tell me why there is a difference? On both > machines, python was compiled from source with the same options, so I'm > not sure what this is down to. > > More importantly, what's the most straightforward way to set all the > dates to usable values? From my POV, it's fine to set them all to the > current date, but I don't know how to go about doing this exhaustively. > > Thanks for your help, > > Miles > > > ___ > For more information about ZODB, see the ZODB Wiki: > http://www.zope.org/Wikis/ZODB/ > > ZODB-Dev mailing list - ZODB-Dev@zope.org > http://mail.zope.org/mailman/listinfo/zodb-dev > ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] problem with _p_mtime
Miles Waller wrote at 2008-12-4 19:42 +: >fstest - no problems >checkbtrees - no problems > >fsrefs - returns errors about invalid objects (and reports all objects >as last updated: 5076-10-09 17:19:26.809896!), and finally fails with a >KeyError > >Traceback (most recent call last): > File "/usr/local/Zope-2.9.8/bin/fsrefs.py", line 157, in ? >main(path) > File "/usr/local/Zope-2.9.8/bin/fsrefs.py", line 130, in main >refs = get_refs(data) > File "/usr/local/Zope-2.9.8/lib/python/ZODB/serialize.py", line 687, >in get_refs >data = oid_klass_loaders[reference_type](*args) >KeyError: 'n' This indicates that "fsrefs" does not understand the data. There are several possible causes: * "fsrefs" does not have the correct version * "fsrefs" has a bug * your storage is damaged. As you have reported that the storage content could be successfully exported, a damage is not that likely (the export should have the same problem in this case). > >I think I can see some corruption in the oids of the referenced objects >as they show as: >\x00\x00\x00\x00\x00\x11'@ >\x00\x00\x00\x00\x00#\xd4\xa9 >\x00\x00\x00\x00\x00\x11'* >etc... - i wasn't expecting to see [EMAIL PROTECTED] and friends. This does not indicate any corruption: the oids are treated as 8 byte binary strings. If a byte has a printable representation, this one is used on printing, otherwise its hex representation. >For example, fsrefs reports not being able to find >'\x00\x00\x00\x00\x00#\xd4"'. However, I can load the database at the >zopectl prompt and load objects, and get ob._p_oid to report >'\x00\x00\x00\x00\x00#\xd4"'. Looks like an "fsrefs" bug. If you can load an object from the storage, "fsrefs" should not report it as missing. >I also wondered if the first few bytes of the database could have been >cut off This is unlikely. The first bytes contain a magic number (identifying the storage format). I think, a "FileStorage" would not open when the magic number were unrecognizable. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] problem with _p_mtime
Alan Runyan wrote: > We run consistency checks on the ZODB - the same as we do on SVN -- nightly. > > http://svn.zope.org/ZODB/trunk/src/ZODB/scripts > - fstest.py > - checkbtrees.py > - fsrefs.py When reading documentation of fsrefs.py, I see it is not recommended to run it again a live FileStorage. Do you stop your servers nightly to do it ? > > We have never had corruption in years. > But I just wanted to ask what people are doing to find any problems > with consistency; so the problem can be solved ASAP. > > Miles, have you run any of these scripts across your production ZODB? > > cheers > alan -- Godefroid Chapelle (aka __gotcha) http://bubblenet.be ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] problem with _p_mtime
We run consistency checks on the ZODB - the same as we do on SVN -- nightly. http://svn.zope.org/ZODB/trunk/src/ZODB/scripts - fstest.py - checkbtrees.py - fsrefs.py We have never had corruption in years. But I just wanted to ask what people are doing to find any problems with consistency; so the problem can be solved ASAP. Miles, have you run any of these scripts across your production ZODB? cheers alan On Thu, Dec 4, 2008 at 12:45 PM, Dieter Maurer <[EMAIL PROTECTED]> wrote: > Miles wrote at 2008-12-4 13:39 +: >>I've moved a FileStorage from one (old) machine to another (new) >>machine, but when I mount it on the new machine I get a lot of time errrors: >> >>Traceback (innermost last): >> Module ZPublisher.Publish, line 115, in publish >> Module ZPublisher.mapply, line 88, in mapply >> Module ZPublisher.Publish, line 41, in call_object >> Module Shared.DC.Scripts.Bindings, line 311, in __call__ >> Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec >> Module App.special_dtml, line 176, in _exec >> Module DocumentTemplate.DT_Let, line 76, in render >> Module DocumentTemplate.DT_In, line 703, in renderwob >> Module DocumentTemplate.DT_With, line 76, in render >> Module DocumentTemplate.DT_Var, line 214, in render >> Module App.PersistentExtra, line 43, in bobobase_modification_time >> Module DateTime.DateTime, line 509, in __init__ >> Module DateTime.DateTime, line 760, in _parse_args >> Module DateTime.DateTime, line 437, in safelocaltime >>TimeError: The time 98040302366.810165 is beyond the range of this >>Python implementation. > > I expect that your storage is damanged and contains some garbage > at a place where a serial should be. > > This could probably be fixed -- but other parts of your storage > might be damanged, too. > > > > -- > Dieter > ___ > For more information about ZODB, see the ZODB Wiki: > http://www.zope.org/Wikis/ZODB/ > > ZODB-Dev mailing list - ZODB-Dev@zope.org > http://mail.zope.org/mailman/listinfo/zodb-dev > -- Alan Runyan Enfold Systems, Inc. http://www.enfoldsystems.com/ phone: +1.713.942.2377x111 fax: +1.832.201.8856 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] problem with _p_mtime
Miles wrote at 2008-12-4 13:39 +: >I've moved a FileStorage from one (old) machine to another (new) >machine, but when I mount it on the new machine I get a lot of time errrors: > >Traceback (innermost last): > Module ZPublisher.Publish, line 115, in publish > Module ZPublisher.mapply, line 88, in mapply > Module ZPublisher.Publish, line 41, in call_object > Module Shared.DC.Scripts.Bindings, line 311, in __call__ > Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec > Module App.special_dtml, line 176, in _exec > Module DocumentTemplate.DT_Let, line 76, in render > Module DocumentTemplate.DT_In, line 703, in renderwob > Module DocumentTemplate.DT_With, line 76, in render > Module DocumentTemplate.DT_Var, line 214, in render > Module App.PersistentExtra, line 43, in bobobase_modification_time > Module DateTime.DateTime, line 509, in __init__ > Module DateTime.DateTime, line 760, in _parse_args > Module DateTime.DateTime, line 437, in safelocaltime >TimeError: The time 98040302366.810165 is beyond the range of this >Python implementation. I expect that your storage is damanged and contains some garbage at a place where a serial should be. This could probably be fixed -- but other parts of your storage might be damanged, too. -- Dieter ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] problem with _p_mtime
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Miles Waller wrote: > Tres Seaver wrote: >> -BEGIN PGP SIGNED MESSAGE- >> Hash: SHA1 >> >> Miles wrote: >> >>> Hi, >>> >>> I've moved a FileStorage from one (old) machine to another (new) >>> machine, but when I mount it on the new machine I get a lot of time errrors: >>> >>> Traceback (innermost last): >>>Module ZPublisher.Publish, line 115, in publish >>>Module ZPublisher.mapply, line 88, in mapply >>>Module ZPublisher.Publish, line 41, in call_object >>>Module Shared.DC.Scripts.Bindings, line 311, in __call__ >>>Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec >>>Module App.special_dtml, line 176, in _exec >>>Module DocumentTemplate.DT_Let, line 76, in render >>>Module DocumentTemplate.DT_In, line 703, in renderwob >>>Module DocumentTemplate.DT_With, line 76, in render >>>Module DocumentTemplate.DT_Var, line 214, in render >>>Module App.PersistentExtra, line 43, in bobobase_modification_time >>>Module DateTime.DateTime, line 509, in __init__ >>>Module DateTime.DateTime, line 760, in _parse_args >>>Module DateTime.DateTime, line 437, in safelocaltime >>> TimeError: The time 98040302366.810165 is beyond the range of this >>> Python implementation. >>> >>> I googled around on this problem and didn't find much, but I did find >>> that as a temporary fix, I've put a try/except in App.PersistentExtra, >>> so if it generates an exception, it returns the current date and time. >>> This keeps things working. >>> >>> Out of interst, can anyone tell me why there is a difference? On both >>> machines, python was compiled from source with the same options, so I'm >>> not sure what this is down to. >>> >>> More importantly, what's the most straightforward way to set all the >>> dates to usable values? From my POV, it's fine to set them all to the >>> current date, but I don't know how to go about doing this exhaustively. >>> >>> Thanks for your help, >>> >> For each machine: what version of Python and Zope? what OS? >> > Old machine: Python 2.3.5, Zope 2.8.6, OS is Red Hat 8.0 (!) > > New machine: Python 2.4.3, Zope 2.9.8, OS is Ubuntu 7.10 > > I had discounted python/zope version differences as other databases had > upgraded fine. (CC'ing the newsgroup) Zope is defending against a change in Python's time module, which in turn is due to a change in the platform's time_t: $ bin/python2.3 Python 2.3.7 (#1, Aug 20 2008, 12:39:04) [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> int(98040302366.81016) 98040302366L >>> from time import gmtime, localtime >>> gmtime(int(98040302366.81016)) (1901, 12, 13, 20, 45, 52, 4, 347, 0) >>> localtime(int(98040302366.81016)) (1901, 12, 13, 15, 45, 52, 4, 347, 0) $ bin/python2.4 Python 2.4.4 (#2, Nov 14 2007, 13:58:47) [GCC 4.1.2 (Ubuntu 4.1.2-0ubuntu4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> int(98040302366.81016) 98040302366L >>> from time import gmtime, localtime >>> gmtime(int(98040302366.81016)) Traceback (most recent call last): File "", line 1, in ? ValueError: timestamp out of range for platform time_t >>> localtime(int(98040302366.81016)) Traceback (most recent call last): File "", line 1, in ? ValueError: timestamp out of range for platform time_t As you can see, the Python 2.3 version has a wraparound problem. I have no idea how you got such an odd value stored for your '_p_mtime', unless you have been using Guido's time machine to run your appserver in 2039. ;) Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJOAcs+gerLs4ltQ4RAooLAKCbzF9wBkwhtnhISmIGqIt22y7+rACfThhx IyjRAFhHuPE+LAVSj4Ct3Mk= =iq+a -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] problem with _p_mtime
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Miles wrote: > Hi, > > I've moved a FileStorage from one (old) machine to another (new) > machine, but when I mount it on the new machine I get a lot of time errrors: > > Traceback (innermost last): >Module ZPublisher.Publish, line 115, in publish >Module ZPublisher.mapply, line 88, in mapply >Module ZPublisher.Publish, line 41, in call_object >Module Shared.DC.Scripts.Bindings, line 311, in __call__ >Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec >Module App.special_dtml, line 176, in _exec >Module DocumentTemplate.DT_Let, line 76, in render >Module DocumentTemplate.DT_In, line 703, in renderwob >Module DocumentTemplate.DT_With, line 76, in render >Module DocumentTemplate.DT_Var, line 214, in render >Module App.PersistentExtra, line 43, in bobobase_modification_time >Module DateTime.DateTime, line 509, in __init__ >Module DateTime.DateTime, line 760, in _parse_args >Module DateTime.DateTime, line 437, in safelocaltime > TimeError: The time 98040302366.810165 is beyond the range of this > Python implementation. > > I googled around on this problem and didn't find much, but I did find > that as a temporary fix, I've put a try/except in App.PersistentExtra, > so if it generates an exception, it returns the current date and time. > This keeps things working. > > Out of interst, can anyone tell me why there is a difference? On both > machines, python was compiled from source with the same options, so I'm > not sure what this is down to. > > More importantly, what's the most straightforward way to set all the > dates to usable values? From my POV, it's fine to set them all to the > current date, but I don't know how to go about doing this exhaustively. > > Thanks for your help, For each machine: what version of Python and Zope? what OS? Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJN/BX+gerLs4ltQ4RAkcrAJ41ZnKq8GMFiSWLf9bk5/3DH855rgCgs7sX zVNGq01iU9Bo4pI+MQ/ATTI= =Mkwi -END PGP SIGNATURE- ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] problem with _p_mtime
Hi, I've moved a FileStorage from one (old) machine to another (new) machine, but when I mount it on the new machine I get a lot of time errrors: Traceback (innermost last): Module ZPublisher.Publish, line 115, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 41, in call_object Module Shared.DC.Scripts.Bindings, line 311, in __call__ Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec Module App.special_dtml, line 176, in _exec Module DocumentTemplate.DT_Let, line 76, in render Module DocumentTemplate.DT_In, line 703, in renderwob Module DocumentTemplate.DT_With, line 76, in render Module DocumentTemplate.DT_Var, line 214, in render Module App.PersistentExtra, line 43, in bobobase_modification_time Module DateTime.DateTime, line 509, in __init__ Module DateTime.DateTime, line 760, in _parse_args Module DateTime.DateTime, line 437, in safelocaltime TimeError: The time 98040302366.810165 is beyond the range of this Python implementation. I googled around on this problem and didn't find much, but I did find that as a temporary fix, I've put a try/except in App.PersistentExtra, so if it generates an exception, it returns the current date and time. This keeps things working. Out of interst, can anyone tell me why there is a difference? On both machines, python was compiled from source with the same options, so I'm not sure what this is down to. More importantly, what's the most straightforward way to set all the dates to usable values? From my POV, it's fine to set them all to the current date, but I don't know how to go about doing this exhaustively. Thanks for your help, Miles ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
