Re: [ZODB-Dev] Relstorage pack problems

[Shane Hathaway]

Alan Runyan wrote:
> 
> 
> On Fri, Jan 23, 2009 at 1:45 PM, Shane Hathaway  > wrote:
> => select zoid from current_object where tid != (select max(tid) from
> object_state where object_state.zoid = current_object.zoid);
>  zoid
> --
> 7
>10
>12
>11
> 9
> 8
> (6 rows)
> 
> 
> Wow.  All of that with some SQL.

Cool, huh?  One caveat: While PostgreSQL executes this query in seconds, 
MySQL 5.0 would likely mis-optimize the subquery and take hours. :-/ 
For MySQL I'd have to use join syntax instead.

> Out of curiosity what would be the process if you were using FileStorage 
> to diagnose the same problem?  Do the same thing by iterating over the 
> entire storage and filling in some structures and doing the comparison 
> manually?

Probably, though ZODB has some pretty good tools for low-level 
FileStorage debugging.

> Sounds like it would take a few hours, with SQL -- it seems 
> pretty straightforward.

I think so too, although with FileStorage it would have been possible to 
guess at the time when the databases were accidentally merged, by 
looking at the position of the bad transactions in the file. 
FileStorage also would have reacted differently to the merged databases, 
though I have no idea how.

> The sooner this gets into the mainline of ZODB -- the better.

Thanks.  Still, FileStorage should always be the default. It's simpler 
and easier to set up than RelStorage.

Shane
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Relstorage pack problems

[Alan Runyan]

On Fri, Jan 23, 2009 at 1:45 PM, Shane Hathaway wrote:

> Shane Hathaway wrote:
> > Assuming your bad script caused your problem, it is likely that packing
> > will still mess up your database, since you still probably have mixed-up
> > object_state rows.  Don't pack until I've had a chance to look again.
>
> Here is some more analysis.  Now that I understand you accidentally
> merged two databases into one by forcing copyTransactionsFrom() to run
> when it shouldn't, I looked for the transactions you merged.  First I
> looked for the OIDs with a confused transaction ID.
>
>
> => select zoid from current_object where tid != (select max(tid) from
> object_state where object_state.zoid = current_object.zoid);
>  zoid
> --
> 7
>10
>12
>11
> 9
> 8
> (6 rows)


Wow.  All of that with some SQL.

Out of curiosity what would be the process if you were using FileStorage to
diagnose the same problem?  Do the same thing by iterating over the entire
storage and filling in some structures and doing the comparison manually?
Sounds like it would take a few hours, with SQL -- it seems pretty
straightforward.

I'm impressed.

The sooner this gets into the mainline of ZODB -- the better.
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Relstorage pack problems

[Shane Hathaway]

Shane Hathaway wrote:
> Assuming your bad script caused your problem, it is likely that packing 
> will still mess up your database, since you still probably have mixed-up 
> object_state rows.  Don't pack until I've had a chance to look again.

Here is some more analysis.  Now that I understand you accidentally 
merged two databases into one by forcing copyTransactionsFrom() to run 
when it shouldn't, I looked for the transactions you merged.  First I 
looked for the OIDs with a confused transaction ID.


=> select zoid from current_object where tid != (select max(tid) from 
object_state where object_state.zoid = current_object.zoid);
  zoid
--
 7
10
12
11
 9
 8
(6 rows)


Then I listed all non-current transaction IDs for those objects.


=>  select zoid, tid from object_state where zoid in (7,8,9,10,11,12) 
and tid != (select tid from current_object where current_object.zoid = 
object_state.zoid);
  zoid |tid
--+
 8 | 250499913748614178
 9 | 250499913748614178
10 | 250499913748614178
11 | 250499913748614178
12 | 250499913748614178
 7 | 250499913748614178
(6 rows)


Based on this information and the information in my last email, I can 
deduce that you fortunately merged only two transactions from another 
database and that while the merge caused conflicts, these objects 
haven't been otherwise modified.  Note that the bad database merge could 
have happened at any time, not necessarily November 17 when these 
transactions were created.  Anyone with access to your database and your 
broken script could cause this problem again.  Fix the script quickly.

Here are the two bad transactions:

250499913441768123 | initial database creation
250499913748614178 | /manage_main\012\012Created Zope Application

You need to delete all traces of these two transactions from your 
database.  Before you do, please ensure nothing is actually using them. 
  The query below should not return any rows.


select * from current_object where tid in (250499913441768123, 
250499913748614178);


Assuming that query returns no rows, here is how you can remove the bad 
transactions:


update object_state set prev_tid = 0 where prev_tid in 
(250499913441768123, 250499913748614178);
delete from object_state where tid in (250499913441768123, 
250499913748614178);
delete from object_ref where tid in (250499913441768123, 
250499913748614178);
delete from object_refs_added where tid in (250499913441768123, 
250499913748614178);
delete from transaction where tid in (250499913441768123, 
250499913748614178);
commit;


Once you've done that, you should see no more anomalies in current_object:


=>  select zoid from current_object where tid != (select max(tid) from 
object_state where object_state.zoid = current_object.zoid);
  zoid
--
(0 rows)


I used several shortcuts for this solution, particularly the statement 
that sets prev_tid to 0.  If you had merged a more complex database, I 
wouldn't have been able to use shortcuts.

I'm glad to know RelStorage didn't do anything wrong after all.  Perhaps 
the copyTransactionsFrom() method could work harder to prevent a mishap 
like this, but that method is part of the ZODB API, not RelStorage, so I 
don't have as much control over it.

However, I still don't want you to pack yet because my experiments with 
packing your database has revealed some unexpected behavior.  I'm going 
to look into it.

Shane

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Relstorage in comparison of Object Relational Mapping

[Shane Hathaway]

Andreas Jung wrote:
> On 23.01.2009 17:41 Uhr, Sandra wrote:
>> Hi,
>> I am try to understand, why Relstorage was been create, Could you help
>> me Shane, what was your motivations? Have you any documentation about
>> Relstorage?
>> and in oder side could we said that, the technic of relstorage is like
>> Object Relational Mapping?
> 
> Relstorage has nothing to do with ORM. Like Filestorage, Relstorage deal
> with Python pickles only. It's basically like ZEO+FileStorage but
> implemented on top a  RDBMS storage layer.

Correct.  You can read more here, Sandra:

http://wiki.zope.org/ZODB/RelStorage

Shane

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Relstorage pack problems

[Shane Hathaway]

Santi Camps wrote:
> I attach the script we use to do the conversion.   Be free to include
> in Relstorage if you think it's useful and it is well done (as I said,
> I really don't know much about ZODB, I just mix zodbconvert.py with
> some DirectoryStorage code)

Ah-ha, you removed from zodbconvert.py the code that checks whether the 
destination already has data.  That was unnecessary and a big mistake. 
By removing that error condition, you made the script blind to the mess 
it was making.

Assuming your bad script caused your problem, it is likely that packing 
will still mess up your database, since you still probably have mixed-up 
object_state rows.  Don't pack until I've had a chance to look again.

Put the error condition back.  Here is what it looks like:

 if storage_has_data(destination):
 msg = "Error: the destination storage has data.  Try --clear."
 sys.exit(msg)

It comes just before the call to copyTransactionsFrom().  Also, your 
current storage_has_data() function simply returns True, which is 
insane.  Here is the correct method:

def storage_has_data(storage):
 i = storage.iterator()
 try:
 i[0]
 except IndexError:
 return False
 return True

It's possible that function has a bug due to recent changes in the 
storage iterator API.  If it does, I'll fix it.

Shane

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Relstorage in comparison of Object Relational Mapping

[Andreas Jung]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 23.01.2009 17:41 Uhr, Sandra wrote:
> Hi,
> I am try to understand, why Relstorage was been create, Could you help
> me Shane, what was your motivations? Have you any documentation about
> Relstorage?
> and in oder side could we said that, the technic of relstorage is like
> Object Relational Mapping?

Relstorage has nothing to do with ORM. Like Filestorage, Relstorage deal
with Python pickles only. It's basically like ZEO+FileStorage but
implemented on top a  RDBMS storage layer.

- -aj
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkl59F4ACgkQCJIWIbr9KYwBIACeOaz6w0bwZ5WjH2colRUQUfwC
CpsAoJIsn9dXYd0Km+5soRPzmnmyRhGi
=IhU/
-END PGP SIGNATURE-
begin:vcard
fn:Andreas Jung
n:Jung;Andreas
org:ZOPYX Ltd. & Co. KG
adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany
email;internet:i...@zopyx.com
title:CEO
tel;work:+49-7071-793376
tel;fax:+49-7071-7936840
tel;home:+49-7071-793257
x-mozilla-html:FALSE
url:www.zopyx.com
version:2.1
end:vcard

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

[ZODB-Dev] Relstorage in comparison of Object Relational Mapping

[Sandra]

Hi, 
I am try to understand, why Relstorage was been create, Could you help me 
Shane, what was your motivations? Have you any documentation about Relstorage?
and in oder side could we said that, the technic of relstorage is like Object 
Relational Mapping?
Thanks,
 
Sandrine 

--- En date de : Ven 23.1.09, Shane Hathaway  a écrit :

De: Shane Hathaway 
Objet: Re: [ZODB-Dev] How to turn off 'GC' when packing on ZODB3.6.2
À: "eastxing" 
Cc: zodb-dev@zope.org
Date: Vendredi 23 Janvier 2009, 2h50

eastxing wrote:
> For choice 3, 'RelStorage' provide a 'ZODBConverter', but
I do not to 
> know if it can used with ZODB3.6.2?

I think RelStorage could support ZODB 3.6 without much effort, but I 
haven't tested it.  I'd go with Jim's suggestion of using a new ZEO
server.

Shane
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev



  ___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Relstorage pack problems

[Santi Camps]

On Fri, Jan 23, 2009 at 2:38 AM, Shane Hathaway  wrote:
> Santi,
>
> I hope you don't mind me discussing your database in public.  I'm not going
> to talk about anything that looks like it could be private.  Other
> RelStorage users might benefit from the analysis.
>

Hi Shane

That's right, of course.  Furthermore, thanks a lot for this analysis.

> Looking at your database, I see that something bad happened just before
> transaction 250499913441768123.  That number is an encoded time stamp:
>
>  >>> from ZODB.TimeStamp import TimeStamp
>  >>> from ZODB.utils import p64
>  >>> str(TimeStamp(p64(250499913441768123)))
>  '2008-11-17 19:36:04.913130'
>
> The transaction log entry says "initial database creation", which means that
> the database had no root object (OID 0), so ZODB created one and started a
> brand new database.  Strange!  This happened about an hour after a
> transaction labeled:
>
>  /asp_ekartek/kmkey_iso/portal_setup/manage_doUpgrades
>
> I'm guessing that an upgrade script did something horribly wrong that day.
>

I've been revising what happens this day, and I think we are near to
get the guilty.This day the database was migrated from
DirectoryStorage to Relstorage.AFAIK, the upgrade should be done
in DirectoryStorage, before the conversion to Relstorage.   I don't
think the upgrade can corrupt the database, all operations ara high
level ones, and we never change ZODB objects by hand.   So, the
problem should be in the conversion process.

I attach the script we use to do the conversion.   Be free to include
in Relstorage if you think it's useful and it is well done (as I said,
I really don't know much about ZODB, I just mix zodbconvert.py with
some DirectoryStorage code)

> Furthermore, the entry for OID 0 in the current_object table points to an
> old transaction rather than the most recent transaction that modified OID 0.
>  That's not supposed to happen, even when you undo.  I hope RelStorage
> didn't do that!
>
> Did you or someone on your team change current_object by hand?  I can
> understand why you would, since a simple modification to current_object
> would be a nice quick fix for the broken upgrade.  The fix would not be
> complete, though, because now the object_state table and the current_object
> table disagree on the current state of OID 0.
>
> According to object_state, even now, the current state of OID 0 still points
> to the small object graph that was accidentally created on November 17.  The
> pack code relies more on object_state than on current_object, so the pack
> code sees only a handful of objects that are reachable.  Packing with
> garbage collection removes everything that is not reachable.
>
> The current_object table is really just a cache of object_state.  If the
> schema were fully normalized, there would not be a current_object table.  In
> theory, the current_object table makes it possible to load ZODB objects
> quickly.  But if the current_object table results in problems like this, I
> need to consider alternatives.
>

We've used the attached script to convert a lot of others databases,
that are packing successfully, so something special should occurs in
this case.The only explanation I can found is that the conversion
would be done without unmounting the DirectoryStorage database from
its Zope mount point, and that caused the problem.   The right way we
use to convert databases from DS to RS is, first of all, detach them
from Zope, then convert, and then mount the resultant RS, but It's
possible that were a human mistake that day.

> In any case, I believe you can get out of this mess pretty easily.  You need
> to delete the extra object states for OID 0 created on November 17.  I tried
> this in my copy of your database:
>
> delete from object_state where zoid = 0 and tid in (
>  250499913441768123, 250499913748614178);
>
> After that, "select count(1) from object_state where zoid = 0;" should tell
> you there is only one state in the database for OID 0.  Packing should work
> fine then.  It seemed to do the right thing on my copy, but I don't have
> your application code to check it.
>

Thank you very much for that information.  I really could not be able
to found it myself.We will try it as soon as possible and let you
know the results

Regards
-- 
Santi Camps (Earcon S.L.)
http://www.earcon.com
http://www.kmkey.com
#!/usr/bin/env python
##
#
# Copyright (c) 2008 Zope Corporation and Contributors.
# All Rights Reserved.
#
# This software is subject to the provisions of the Zope Public License,
# Version 2.1 (ZPL).  A copy of the ZPL should accompany this distribution.
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY AND ALL EXPRESS OR IMPLIED
# WARRANTIES ARE DISCLAIMED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
# WARRANTIES OF TITLE, MERCHANTABILITY, AGAINST INFRINGEMENT, AND FITNESS
# FOR A PARTICULAR PURPOSE.
#
###