Re: [ZODB-Dev] Implementing Storage Decorators

[Jim Fulton]

On May 4, 2007, at 3:52 PM, Tino Wildenhain wrote:


Dieter Maurer schrieb:

Jim Fulton wrote at 2007-5-4 14:40 -0400:

On May 4, 2007, at 2:33 PM, Dieter Maurer wrote:


Jim Fulton wrote at 2007-5-2 11:52 -0400:

...
I think I still rather like explicit, but I'm on the fence about
which approach is best.  What do other people think?
From your description, I would use a subclassing (and forget about

proxy and copying).

That would be a nightmare, on multiple levels:

- All of the separate implementations would become tightly  
coupled,  which is what happens with inheritance.


- Either someone would have to create classes for the various   
permutations of features, or consumers would have to mix and  
match  multiple classes to get what they want and sort out the  
variate  internal implementation incompatibilities.

Your decorators would become mixin classes
and the final classes would list the features they like -- simpler
than ZCML binding together...
Of course, some features may not play well with one another.
But, that will make problems also with proxies or copying...


Even this could be sorted out with special cleanup or
secure mixins which can do dirty tricks like accessing
self.__class__.__bases__ to fixup ;) (Much like combining
the copy with the subclass approach)


Uh huh.


I'm for mixin with strong guidelines on implementation
(e.g. how to call private attributes to not pollute the
 namespace)


Forget it. Inheritance is not an option.

Jim

--
Jim Fulton  mailto:[EMAIL PROTECTED]Python 
Powered!
CTO (540) 361-1714  
http://www.python.org
Zope Corporationhttp://www.zope.com http://www.zope.org



___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Implementing Storage Decorators

[Jim Fulton]

On May 4, 2007, at 3:14 PM, Dieter Maurer wrote:


Jim Fulton wrote at 2007-5-4 14:40 -0400:


On May 4, 2007, at 2:33 PM, Dieter Maurer wrote:


Jim Fulton wrote at 2007-5-2 11:52 -0400:

...
I think I still rather like explicit, but I'm on the fence about
which approach is best.  What do other people think?



From your description, I would use a subclassing (and forget about

proxy and copying).


That would be a nightmare, on multiple levels:

- All of the separate implementations would become tightly coupled,
which is what happens with inheritance.

- Either someone would have to create classes for the various
permutations of features, or consumers would have to mix and match
multiple classes to get what they want and sort out the variate
internal implementation incompatibilities.


Your decorators would become mixin classes
and the final classes would list the features they like


So the second case above.  As I said, inheritance causes tight  
coupling.  Mix in classes can work when the features provides are  
truly independent.  That would certainly not be the case here.   
Trying to combine multiple storage implementations in a single class  
would be a real mess.



-- simpler
than ZCML binding together...


There's no ZCML involved here, although there would likely be ZConfig  
files.  People would want to be able to combine storages in their  
configuration files.



Of course, some features may not play well with one another.
But, that will make problems also with proxies or copying...


All of the examples I mentioned can be handled very well with a  
decorator model.


Jim

--
Jim Fulton  mailto:[EMAIL PROTECTED]Python 
Powered!
CTO (540) 361-1714  
http://www.python.org
Zope Corporationhttp://www.zope.com http://www.zope.org



___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Supporting a DataManager without Two Phase Commit

[Andreas Jung]

--On 5. Mai 2007 02:46:58 +0200 Andreas Jung <[EMAIL PROTECTED]> wrote:



If commit fails or another DataManager fails, data is not commited to the
database.



However I am not sure if this is completely true. Although this approach 
ensures that all SQL statements were execute properly it does not take into 
account that a commit operation itself might fail even at that point for 
some reason. Even the Python DB API itself does not support the TPC and 
most don't driver don't support TPC through an API. So that's basically a 
flaw in the Python DB-API level. Even up2date drivers like psycopg2 for 
Postgres don't support something like PREPARE TRANSACTION or COMMIT 
PREPARED.


See also



Looking back at your hack..it seems to introduce some more safety for a
particular usecase (perhaps general for people dealing only with one 
database at a time).


But thanks for pointing out this problem. I'll change my code in 
z3c.sqlalchemy in a similar way to gain at least a bit more safety.


Andreas

pgpDNDiDzggFQ.pgp
Description: PGP signature
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Supporting a DataManager without Two Phase Commit

[Andreas Jung]

--On 4. Mai 2007 22:33:05 +0100 Laurence Rowe <[EMAIL PROTECTED]> wrote:

Following a discussion with several of the sqlalchemy integration authors
on #plone today we came up with the following hack to implement this:
http://dev.plone.org/collective/browser/collective.lead/trunk/collective/
lead/tx.py



I think z3c.zalchemy is doing it right:

"""
Changed how the two phase commit works.

This is how the two phase commit is used in zope.

   1. tpc_begin(txn)
   2. commit(txn)
   3. tpc_vote(txn)
   4. tpc_finish(txn)

What zalchemy is doing:

 - commit does a session.flush() which actually executes all sql 
statements.

 - tpc_finish() does a transaction.commit() in the sqlalchemy transaction
 - tpc_abort() does a transaction.rollback() in the sqlalchemy transaction

If commit fails or another DataManager fails, data is not commited to the
database.

Thanks to Michael Bayer (the author of sqlalchemy) who gave me the hint to
integrate sqlalchemy correctly into the two phase commit.

"""

Andreas

pgpTh1fCqZm2X.pgp
Description: PGP signature
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Supporting a DataManager without Two Phase Commit

[Andreas Jung]

--On 4. Mai 2007 22:33:05 +0100 Laurence Rowe <[EMAIL PROTECTED]> wrote:


Following a discussion with several of the sqlalchemy integration authors
on #plone today we came up with the following hack to implement this:
http://dev.plone.org/collective/browser/collective.lead/trunk/collective/
lead/tx.py

The DataManager is given a high sortKey to ensure that it is considered
last, and commits in tpc_vote, before the other (2PC) DataManagers'
tpc_finish methods are called.

The hack obviously relies on only one DataManager making use of the
trick. It would be nice to make this was supported directly so that an a
error could be thrown when more than one 1PC DataManager joined a
transaction.


Well, the hack is really only a hack. As soon as more than one data manager 
is involved you'll run into the same problems. I have some apps with 
multiple SQL connections so this hack would give me nothing.




This could be implemented by changing the signature of
transaction._transaction.Transaction.join to have an optional
single_phase argument (default would be False). The 1PC resource would
then be registered seperately to the 2PC resources and _commitResources
would call commit on the 1PC resource between tpc_vote and tpc_finish.

If you think this would be helpful I'll try and supply a patch (need to
look into the detail of how failed transactions are cleaned up).


You could resolve this issue only if all subsystems participating in 
transaction could rollback a commit happened with this transaction in case 
of a data manager failing during its commit.


Andreas


pgp2PTDD5Or7O.pgp
Description: PGP signature
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

[ZODB-Dev] Supporting a DataManager without Two Phase Commit

[Laurence Rowe]

Hi,

Several people have made SQLalchemy integrations recently. SQLAlchemy 
does not support Two Phase Commit (2PC) so correctly tying it in with 
zope's transactions is tricky. With multiple One Phase Commit (1PC) 
DataManagers the problem is of course intractable, but given the 
popularity of mappers like SQLAlchemy I think Zope should support a 
single 1PC DataManager.


This websphere document describes a method to integrate a single 1PC 
resource with 2PC resources:

http://publib.boulder.ibm.com/infocenter/wasinfo/v6r0/index.jsp?topic=/com.ibm.websphere.express.doc/info/exp/lao/tasks/tla_ep.html

Following a discussion with several of the sqlalchemy integration 
authors on #plone today we came up with the following hack to implement 
this:

http://dev.plone.org/collective/browser/collective.lead/trunk/collective/lead/tx.py

The DataManager is given a high sortKey to ensure that it is considered 
last, and commits in tpc_vote, before the other (2PC) DataManagers' 
tpc_finish methods are called.


The hack obviously relies on only one DataManager making use of the 
trick. It would be nice to make this was supported directly so that an a 
 error could be thrown when more than one 1PC DataManager joined a 
transaction.


This could be implemented by changing the signature of 
transaction._transaction.Transaction.join to have an optional 
single_phase argument (default would be False). The 1PC resource would 
then be registered seperately to the 2PC resources and _commitResources 
would call commit on the 1PC resource between tpc_vote and tpc_finish.


If you think this would be helpful I'll try and supply a patch (need to 
look into the detail of how failed transactions are cleaned up).


Laurence

___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

[ZODB-Dev] Re: checking what refers to an object in zodb

[Laurence Rowe]

Chris,

I think you're looking at forward references when you want to look at 
back references.


This might help: http://plone.org/documentation/how-to/debug-zodb-bloat

(you might have to change the refmap to be in a zodb with that much data 
though)


Laurence

Chris Withers wrote:

Hi All,

We have a big(ish) zodb, which is about 29GB in size.
Thanks to the laughable difficulty of getting larger disks in big 
corporates, we've been looking into what's taking up that 29GB and were 
a bit surprised by the results.


Using space.py from the ZODBTools in Zope 2.9.4, it turns out that we 
have a lot of PersistentMapping's:


990,35913,555,382,871Persistence.mapping.PersistentMapping

So, that's almost half of the 29GB!

AT's default storage is a PersistentMapping called _md so this isn't too 
surprising. However, when looking into it, it turns out that half of the 
PersistentMapping's actually appear to be workflow_history's from 
DCWorkflow.


To try and find out which objects were referencing all these workflow 
histories, we tried the following starting with one of the oid of these 
histories:


from ZODB.FileStorage import FileStorage
from ZODB.serialize import referencesf

fs = FileStorage(path, read_only=1)
data, serialno = fs.load(oid, '')
refs = referencesf(data)

To our surprise, all of the workflow histories returned an empty list 
for refs.


What does this mean? Is there a bug that means these objects are hanging 
around even though there are no references? Are we using the wrong 
method to find references to these objects?


(if it helps, we pack to 1 day and each pack removes between 0.5GB and 
1GB from the overall size)


If there's any more info that would be helpful here, please ask away...

cheers,

Chris



___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Implementing Storage Decorators

[Tino Wildenhain]

Dieter Maurer schrieb:

Jim Fulton wrote at 2007-5-4 14:40 -0400:

On May 4, 2007, at 2:33 PM, Dieter Maurer wrote:


Jim Fulton wrote at 2007-5-2 11:52 -0400:

...
I think I still rather like explicit, but I'm on the fence about
which approach is best.  What do other people think?
From your description, I would use a subclassing (and forget about

proxy and copying).

That would be a nightmare, on multiple levels:

- All of the separate implementations would become tightly coupled,  
which is what happens with inheritance.


- Either someone would have to create classes for the various  
permutations of features, or consumers would have to mix and match  
multiple classes to get what they want and sort out the variate  
internal implementation incompatibilities.


Your decorators would become mixin classes
and the final classes would list the features they like -- simpler
than ZCML binding together...

Of course, some features may not play well with one another.
But, that will make problems also with proxies or copying...


Even this could be sorted out with special cleanup or
secure mixins which can do dirty tricks like accessing
self.__class__.__bases__ to fixup ;) (Much like combining
the copy with the subclass approach)

I'm for mixin with strong guidelines on implementation
(e.g. how to call private attributes to not pollute the
 namespace)

Regards
Tino
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] checking what refers to an object in zodb

[Dieter Maurer]

Chris Withers wrote at 2007-5-4 18:53 +0100:
>To try and find out which objects were referencing all these workflow 
>histories, we tried the following starting with one of the oid of these 
>histories:
>
>from ZODB.FileStorage import FileStorage
>from ZODB.serialize import referencesf
>
>fs = FileStorage(path, read_only=1)
>data, serialno = fs.load(oid, '')
>refs = referencesf(data)
>
>To our surprise, all of the workflow histories returned an empty list 
>for refs.

Isn't that very natural?

"referencesf(data)" determines which objects are referenced by
"data" and not which objects do reference the object the state of
which is given by "data".

Usually, the workflow history does not reference persistent objects.



-- 
Dieter
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Same transaction object (re)-used for subsequent requests?

[Andreas Jung]

--On 4. Mai 2007 21:22:49 +0200 Dieter Maurer <[EMAIL PROTECTED]> wrote:


Andreas Jung wrote at 2007-5-4 21:13 +0200:



--On 4. Mai 2007 21:05:00 +0200 Dieter Maurer <[EMAIL PROTECTED]>
wrote:


But, the transactions are not concurrent in your original description!
Instead, one transaction has been committed and (only!) then you
see a transaction with the same id again.


What are you trying to tell? The issue would also happen with a
concurrent  setup. That's why I presented a case where I would see the
error in a  non-concurrent environment. Got it?


No. Not at all.

Lets look at the subject which fortunately remained:

  You assume that some transaction objects are reused for subsequent
  requests. Transactions from subsequent requests are *NEVER* concurrent.

Tim explained to you that it is natural that transactions in
subsequent requests can easily have the same id.
But, *CONCURRENT* transaction can *NEVER* have the same id.

Thus, you can easily use your current caching algorithm:
if you see the same transaction id, then either it is indeed the
same transaction and you must associate the same connection
or it is a different transaction. In this case, it is definitely
nonconcurrent and can therefore, too, get the same connection.



As I said, I have a working solution meanwhile :-)

Andreas

pgpFDifJT5DHg.pgp
Description: PGP signature
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Same transaction object (re)-used for subsequent requests?

[Dieter Maurer]

Andreas Jung wrote at 2007-5-4 21:13 +0200:
>
>
>--On 4. Mai 2007 21:05:00 +0200 Dieter Maurer <[EMAIL PROTECTED]> wrote:
>
>> But, the transactions are not concurrent in your original description!
>> Instead, one transaction has been committed and (only!) then you
>> see a transaction with the same id again.
>
>What are you trying to tell? The issue would also happen with a concurrent 
>setup. That's why I presented a case where I would see the error in a 
>non-concurrent environment. Got it?

No. Not at all.

Lets look at the subject which fortunately remained:

  You assume that some transaction objects are reused for subsequent
  requests. Transactions from subsequent requests are *NEVER* concurrent.

Tim explained to you that it is natural that transactions in
subsequent requests can easily have the same id.
But, *CONCURRENT* transaction can *NEVER* have the same id.

Thus, you can easily use your current caching algorithm:
if you see the same transaction id, then either it is indeed the
same transaction and you must associate the same connection
or it is a different transaction. In this case, it is definitely nonconcurrent
and can therefore, too, get the same connection.

>> And if you read carefully you see "provided their lifespans do not
>> overlap". Obviously, transactions with non overlapping lifespans are not
>> concurrent...
>
>The transactions were not overlapping. As Tim wrote: the transactions were 
>distinct but they used the address.

Yes, therefore, you do not need to distinguish the two
transactions -- it is sufficient to distinguish the transaction ids.



-- 
Dieter
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Implementing Storage Decorators

[Dieter Maurer]

Jim Fulton wrote at 2007-5-4 14:40 -0400:
>
>On May 4, 2007, at 2:33 PM, Dieter Maurer wrote:
>
>> Jim Fulton wrote at 2007-5-2 11:52 -0400:
>>> ...
>>> I think I still rather like explicit, but I'm on the fence about
>>> which approach is best.  What do other people think?
>>
>>> From your description, I would use a subclassing (and forget about
>> proxy and copying).
>
>That would be a nightmare, on multiple levels:
>
>- All of the separate implementations would become tightly coupled,  
>which is what happens with inheritance.
>
>- Either someone would have to create classes for the various  
>permutations of features, or consumers would have to mix and match  
>multiple classes to get what they want and sort out the variate  
>internal implementation incompatibilities.

Your decorators would become mixin classes
and the final classes would list the features they like -- simpler
than ZCML binding together...

Of course, some features may not play well with one another.
But, that will make problems also with proxies or copying...


-- 
Dieter
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Same transaction object (re)-used for subsequent requests?

[Andreas Jung]

--On 4. Mai 2007 21:05:00 +0200 Dieter Maurer <[EMAIL PROTECTED]> wrote:


But, the transactions are not concurrent in your original description!
Instead, one transaction has been committed and (only!) then you
see a transaction with the same id again.


What are you trying to tell? The issue would also happen with a concurrent 
setup. That's why I presented a case where I would see the error in a 
non-concurrent environment. Got it?





And if you read carefully you see "provided their lifespans do not
overlap". Obviously, transactions with non overlapping lifespans are not
concurrent...


The transactions were not overlapping. As Tim wrote: the transactions were 
distinct but they used the address.



Problem solved, end-of-thred,
Andreas

pgpZwrh1NiYwT.pgp
Description: PGP signature
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Same transaction object (re)-used for subsequent requests?

[Dieter Maurer]

Andreas Jung wrote at 2007-5-4 20:04 +0200:
>
>
>--On 4. Mai 2007 19:58:47 +0200 Dieter Maurer <[EMAIL PROTECTED]> wrote:
>
>> Andreas Jung wrote at 2007-5-1 11:23 +0200:
>>> ...
>>> I think you are right (as always). Then let me rephrase the question:
>>> how  can one distinguish if two transaction objects represent the same or
>>> different transactions in such case where memory address is identical?
>>
>> Why are you interested in such a distinction?
>>
>>   While you must deliver the same connection in the same transaction,
>>   there is no harm to deliver a given connection to different transactions
>>   (provided their lifespans do not overlap).
>
>You can't use one connection for concurrent transactions in different 
>threads. Starting a transaction over the same connection would result in 
>nested transactions instead of concurrent transactions.

But, the transactions are not concurrent in your original description!
Instead, one transaction has been committed and (only!) then you
see a transaction with the same id again.

And if you read carefully you see "provided their lifespans do not overlap".
Obviously, transactions with non overlapping lifespans are not
concurrent...



-- 
Dieter
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Implementing Storage Decorators

[Jim Fulton]

On May 4, 2007, at 2:33 PM, Dieter Maurer wrote:


Jim Fulton wrote at 2007-5-2 11:52 -0400:

...
I think I still rather like explicit, but I'm on the fence about
which approach is best.  What do other people think?



From your description, I would use a subclassing (and forget about

proxy and copying).


That would be a nightmare, on multiple levels:

- All of the separate implementations would become tightly coupled,  
which is what happens with inheritance.


- Either someone would have to create classes for the various  
permutations of features, or consumers would have to mix and match  
multiple classes to get what they want and sort out the variate  
internal implementation incompatibilities.


Jim

--
Jim Fulton  mailto:[EMAIL PROTECTED]Python 
Powered!
CTO (540) 361-1714  
http://www.python.org
Zope Corporationhttp://www.zope.com http://www.zope.org



___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Implementing Storage Decorators

[Dieter Maurer]

Jim Fulton wrote at 2007-5-2 11:52 -0400:
> ...
>I think I still rather like explicit, but I'm on the fence about  
>which approach is best.  What do other people think?

>From your description, I would use a subclassing (and forget about
proxy and copying).



-- 
Dieter
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Same transaction object (re)-used for subsequent requests?

[Andreas Jung]

--On 4. Mai 2007 19:58:47 +0200 Dieter Maurer <[EMAIL PROTECTED]> wrote:


Andreas Jung wrote at 2007-5-1 11:23 +0200:

...
I think you are right (as always). Then let me rephrase the question:
how  can one distinguish if two transaction objects represent the same or
different transactions in such case where memory address is identical?


Why are you interested in such a distinction?

  While you must deliver the same connection in the same transaction,
  there is no harm to deliver a given connection to different transactions
  (provided their lifespans do not overlap).


You can't use one connection for concurrent transactions in different 
threads. Starting a transaction over the same connection would result in 
nested transactions instead of concurrent transactions.


-aj



pgpA3zA8VZfG3.pgp
Description: PGP signature
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Same transaction object (re)-used for subsequent requests?

[Dieter Maurer]

Andreas Jung wrote at 2007-5-1 11:23 +0200:
> ...
>I think you are right (as always). Then let me rephrase the question: how 
>can one distinguish if two transaction objects represent the same or
>different transactions in such case where memory address is identical?

Why are you interested in such a distinction?

  While you must deliver the same connection in the same transaction,
  there is no harm to deliver a given connection to different transactions
  (provided their lifespans do not overlap).



-- 
Dieter
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

[ZODB-Dev] checking what refers to an object in zodb

[Chris Withers]

Hi All,

We have a big(ish) zodb, which is about 29GB in size.
Thanks to the laughable difficulty of getting larger disks in big 
corporates, we've been looking into what's taking up that 29GB and were 
a bit surprised by the results.


Using space.py from the ZODBTools in Zope 2.9.4, it turns out that we 
have a lot of PersistentMapping's:


990,359 13,555,382,871  Persistence.mapping.PersistentMapping

So, that's almost half of the 29GB!

AT's default storage is a PersistentMapping called _md so this isn't too 
surprising. However, when looking into it, it turns out that half of the 
PersistentMapping's actually appear to be workflow_history's from 
DCWorkflow.


To try and find out which objects were referencing all these workflow 
histories, we tried the following starting with one of the oid of these 
histories:


from ZODB.FileStorage import FileStorage
from ZODB.serialize import referencesf

fs = FileStorage(path, read_only=1)
data, serialno = fs.load(oid, '')
refs = referencesf(data)

To our surprise, all of the workflow histories returned an empty list 
for refs.


What does this mean? Is there a bug that means these objects are hanging 
around even though there are no references? Are we using the wrong 
method to find references to these objects?


(if it helps, we pack to 1 day and each pack removes between 0.5GB and 
1GB from the overall size)


If there's any more info that would be helpful here, please ask away...

cheers,

Chris

--
Simplistix - Content Management, Zope & Python Consulting
   - http://www.simplistix.co.uk
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] DisconnectedError

[Jim Fulton]

On May 4, 2007, at 12:36 PM, José Pedro Ferreira wrote:


Yes, the DB did not exist.


OK


And yes, it's not a hack,


It looks like the hack was to open and close the storage with  
FileStorage, rather than ZEO.


But with Python 2.4, you were able to create the database using ZEO.   
Strange.
Presumably I'll be able to reproduce the problem you had with Python  
2.5.


Jim

--
Jim Fulton  mailto:[EMAIL PROTECTED]Python 
Powered!
CTO (540) 361-1714  
http://www.python.org
Zope Corporationhttp://www.zope.com http://www.zope.org



___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] DisconnectedError

[José Pedro Ferreira]

Yes, the DB did not exist. And yes, it's not a hack, it's only opening 
and closing the database... but it unexpectedly works after doing it.


Cheers,

Pedro


On May 4, 2007, at 11:25 AM, José Pedro Ferreira wrote:


Jim Fulton wrote:


I don't know why the client is being disconnected.  As others have 
hinted, you should try this with Python 2.4 -- and please report 
back if that changes the outcome.


OK, I've tested it with Python 2.4.4, and it worked OK. For Python 
2.5, I had to use the hack I posted in the last mail (weird...).


I don't really see a hack.  All you did was open the database as 
usual.  Before you ran your script with the "hack", did you make sure 
the database did not exist?


Jim

--
Jim Fultonmailto:[EMAIL PROTECTED]Python Powered!
CTO (540) 361-1714http://www.python.org
Zope Corporationhttp://www.zope.comhttp://www.zope.org





___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] DisconnectedError

[Jim Fulton]

On May 4, 2007, at 11:25 AM, José Pedro Ferreira wrote:


Jim Fulton wrote:


I don't know why the client is being disconnected.  As others have  
hinted, you should try this with Python 2.4 -- and please report  
back if that changes the outcome.


OK, I've tested it with Python 2.4.4, and it worked OK. For Python  
2.5, I had to use the hack I posted in the last mail (weird...).


I don't really see a hack.  All you did was open the database as  
usual.  Before you ran your script with the "hack", did you make sure  
the database did not exist?


Jim

--
Jim Fulton  mailto:[EMAIL PROTECTED]Python 
Powered!
CTO (540) 361-1714  
http://www.python.org
Zope Corporationhttp://www.zope.com http://www.zope.org



___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] DisconnectedError

[José Pedro Ferreira]

Jim Fulton wrote:


I don't know why the client is being disconnected.  As others have 
hinted, you should try this with Python 2.4 -- and please report back 
if that changes the outcome.


OK, I've tested it with Python 2.4.4, and it worked OK. For Python 2.5, 
I had to use the hack I posted in the last mail (weird...).


Best regards,

Pedro Ferreira
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] DisconnectedError

[José Pedro Ferreira]

Gary Poster wrote:

Hi. ZODB does not currently work with Py2.5.

Gary

OK, thanks.
However, I was able to make it work, running this simple script, and 
then running the application again.




#!/usr/bin/python

import ZODB
from ZODB import FileStorage, DB
import transaction

storage = FileStorage.FileStorage("zeo/data.fs")
db = DB(storage)
connection = db.open()
dbroot = connection.root()


transaction.commit()

connection.close()
db.close()
storage.close()



And now it works perfectly :).
Is there any possible explanation for this?

Thanks again,

Pedro
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] DisconnectedError

[Jim Fulton]

On May 4, 2007, at 10:39 AM, José Pedro Ferreira wrote:


Hello.
I am experiencing a problem with ZODB-3.6.0 / Python 2.5, under Linux.
Each time I initialize a database, i get a DisconnectedError  
exception in the client, with the following traceback:




 File "/home/pferreir/workspace/indico/code/code/MaKaC/common/ 
db.py", line 50, in __init__

   self._db=MaKaCDB(self._storage)

 File "/usr/local/lib/python2.5/site-packages/ZODB/DB.py", line  
238, in __init__

   storage.load(z64,'')

 File "/usr/local/lib/python2.5/site-packages/ZEO/ 
ClientStorage.py", line 746, in load

   return self.loadEx(oid, version)[:2]

 File "/usr/local/lib/python2.5/site-packages/ZEO/ 
ClientStorage.py", line 769, in loadEx

   data, tid, ver = self._server.loadEx(oid, version)

 File "/usr/local/lib/python2.5/site-packages/ZEO/ServerStub.py",  
line 192, in loadEx

   return self.rpc.call("loadEx", oid, version)

 File "/usr/local/lib/python2.5/site-packages/ZEO/zrpc/ 
connection.py", line 531, in call

   r_flags, r_args = self.wait(msgid)

 File "/usr/local/lib/python2.5/site-packages/ZEO/zrpc/ 
connection.py", line 621, in wait

   raise DisconnectedError()

DisconnectedError



In the server side, I get:



2007-05-04T16:20:36 (11926) new connection ('127.0.0.1', 38126):  


2007-05-04T16:20:36 (127.0.0.1:38126) received handshake 'Z303'
2007-05-04T16:20:36 (127.0.0.1:38126) loadEx() raised exception: 0x00
Traceback (most recent call last):
 File "/usr/local/lib/python2.5/site-packages/ZEO/zrpc/ 
connection.py", line 421, in handle_request

   ret = meth(*args)
 File "/usr/local/lib/python2.5/site-packages/ZEO/ 
StorageServer.py", line 248, in loadEx

   return self.storage.loadEx(oid, version)
 File "/usr/local/lib/python2.5/site-packages/ZODB/FileStorage/ 
FileStorage.py", line 523, in loadEx

   pos = self._lookup_pos(oid)
 File "/usr/local/lib/python2.5/site-packages/ZODB/FileStorage/ 
FileStorage.py", line 514, in _lookup_pos

   raise POSKeyError(oid)
POSKeyError: 0x00
2007-05-04T16:20:36 (11926/127.0.0.1:38126) disconnected
2007-05-04T16:20:36 (11926) new connection ('127.0.0.1', 38127):  


2007-05-04T16:20:36 (127.0.0.1:38127) received handshake 'Z303'



What confuses me is that I though that a root object for the  
database would be created by the client, and a DisconnectedError is  
raised before that happens. What can be happening?


The root object is only created if it doesn't exist.  The database is  
trying to determine if it exists or not.


I don't know why the client is being disconnected.  As others have  
hinted, you should try this with Python 2.4 -- and please report back  
if that changes the outcome.


Jim

--
Jim Fulton  mailto:[EMAIL PROTECTED]Python 
Powered!
CTO (540) 361-1714  
http://www.python.org
Zope Corporationhttp://www.zope.com http://www.zope.org



___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] DisconnectedError

[Jim Fulton]

On May 4, 2007, at 10:45 AM, Gary Poster wrote:


Hi.  ZODB does not currently work with Py2.5.


I doubt that this has anything to do with his problem. He should try  
with Python 2.4 though.  I'll be interested and depressed if this  
solves his problem .


Jim

--
Jim Fulton  mailto:[EMAIL PROTECTED]Python 
Powered!
CTO (540) 361-1714  
http://www.python.org
Zope Corporationhttp://www.zope.com http://www.zope.org



___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] DisconnectedError

[Gary Poster]

Hi.  ZODB does not currently work with Py2.5.

Gary
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] DisconnectedError

[Andreas Jung]

--On 4. Mai 2007 16:39:30 +0200 José Pedro Ferreira 
<[EMAIL PROTECTED]> wrote:



Hello.
I am experiencing a problem with ZODB-3.6.0 / Python 2.5, under Linux.


None of the ZODB versions is certified for Python 2.5 so far, perhaps
it's related to the Python version.

-aj

pgpdY7EBYo2U6.pgp
Description: PGP signature
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

[ZODB-Dev] DisconnectedError

[José Pedro Ferreira]

Hello.
I am experiencing a problem with ZODB-3.6.0 / Python 2.5, under Linux.
Each time I initialize a database, i get a DisconnectedError exception 
in the client, with the following traceback:




 File "/home/pferreir/workspace/indico/code/code/MaKaC/common/db.py", 
line 50, in __init__

   self._db=MaKaCDB(self._storage)

 File "/usr/local/lib/python2.5/site-packages/ZODB/DB.py", line 238, in 
__init__

   storage.load(z64,'')

 File "/usr/local/lib/python2.5/site-packages/ZEO/ClientStorage.py", 
line 746, in load

   return self.loadEx(oid, version)[:2]

 File "/usr/local/lib/python2.5/site-packages/ZEO/ClientStorage.py", 
line 769, in loadEx

   data, tid, ver = self._server.loadEx(oid, version)

 File "/usr/local/lib/python2.5/site-packages/ZEO/ServerStub.py", line 
192, in loadEx

   return self.rpc.call("loadEx", oid, version)

 File "/usr/local/lib/python2.5/site-packages/ZEO/zrpc/connection.py", 
line 531, in call

   r_flags, r_args = self.wait(msgid)

 File "/usr/local/lib/python2.5/site-packages/ZEO/zrpc/connection.py", 
line 621, in wait

   raise DisconnectedError()

DisconnectedError



In the server side, I get:



2007-05-04T16:20:36 (11926) new connection ('127.0.0.1', 38126): 


2007-05-04T16:20:36 (127.0.0.1:38126) received handshake 'Z303'
2007-05-04T16:20:36 (127.0.0.1:38126) loadEx() raised exception: 0x00
Traceback (most recent call last):
 File "/usr/local/lib/python2.5/site-packages/ZEO/zrpc/connection.py", 
line 421, in handle_request

   ret = meth(*args)
 File "/usr/local/lib/python2.5/site-packages/ZEO/StorageServer.py", 
line 248, in loadEx

   return self.storage.loadEx(oid, version)
 File 
"/usr/local/lib/python2.5/site-packages/ZODB/FileStorage/FileStorage.py", 
line 523, in loadEx

   pos = self._lookup_pos(oid)
 File 
"/usr/local/lib/python2.5/site-packages/ZODB/FileStorage/FileStorage.py", 
line 514, in _lookup_pos

   raise POSKeyError(oid)
POSKeyError: 0x00
2007-05-04T16:20:36 (11926/127.0.0.1:38126) disconnected
2007-05-04T16:20:36 (11926) new connection ('127.0.0.1', 38127): 


2007-05-04T16:20:36 (127.0.0.1:38127) received handshake 'Z303'



What confuses me is that I though that a root object for the database 
would be created by the client, and a DisconnectedError is raised before 
that happens. What can be happening?


Thanks in advance,

Pedro
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev