On 9/25/11 8:17 AM, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 09/24/2011 08:02 PM, David Glick wrote:
Alan Runyan asked me to post my notes from my attempt to get ZODB
running on PyPy. It's very much an experimental work in progress
(that I got distracted from), but hopefully this is at least useful
to anyone else who wants to attempt the same thing.
I first tried building ZODB trunk (with C extensions). Ran into the
following issues: * PyPy is missing an implementation of the
_Py_ForgetReference macro which is used by ZODB's
persistent/cPickleCache.c * The check_argument_cmp check in
BTrees/objectkeymacros.h fails for the root object of the database on
PyPy * I got a PyPy exception re wrap_objobjargspec which appeared to
be some issue with using the __setitem__ slot of an extension type
At this point I gave up on trying to build the C extensions, and
instead did the following: 1. Started with Jim's python-btrees branch
(http://svn.zope.org/repos/main/ZODB/branches/jim-python-btrees) 2.
Replaced the 'persistent' package with a checkout of Tres'
pure-Python persistent implementation
(http://svn.zope.org/repos/main/persistent/trunk/persistent) 3. Set
exts = [] in setup.py to disable all C extensions when installing.
Next I worked around some issues in the pure-Python branches: *
BTrees.___BTree.fsBucket class was missing toString and fromString
methods, which I implemented. * persistent.picklecache.PickleCache
class was missing update_object_size_estimation method (and indeed
the whole limit-cache-size-by-bytes feature). I added it as a no-op
stub. * persistent.picklecache.PickleCache.__setitem__ was raising a
KeyError for a duplicate oid even when trying to set the same object
already stored under that key * persistent.TimeStamp did not give the
Python implementation, which I fixed by an import of
persistent.timestamp + module alias if persistent.TimeStamp (the C
module) is missing * persistent.pyPersistence.Persistent's __new__
needs to accept *args and **kw
At this point I was able to start up the Pyramid ZODB scaffold and
add an object to the DB root.
Next I tried to get zodbshootout running as a benchmark. I didn't get
this working, as I got stuck on an issue with the pure-Python
PersistentMapping getting committed without its object state
(appeared to be something to do with failing to get unghosted before
it gets committed). Along the way, I noticed a couple
incompatibilities with PyPy because PyPy implements the cPickle
module as an alias to the pure-Python pickle module. This leads to
some problems due to slight semantic differences between cPickle and
pickle. In particular, cPickle.Pickler accepts a lone protocol
argument; pickle.Pickler does not (leading to an error in
ZEO.zrpc.marshal.encode). And pickle.Unpickler does not have a noload
method, which ZODB.serialize.referencesf uses to obtain the
referenced oids from a pickle without the overhead of unpickling.
That's as far as I got. Hope this is helpful to the next guy. :)
Cool, great work. Are the changes you needed to make available as
patchers, or checked in?
Most were temporary hacks and the rest have no tests yet, so not checked
in anywhere. But here's (attached) the diff from my working directory.
----------
David Glick
Web Developer
davidgl...@groundwire.org
206.286.1235x32
Online tools and strategies for the environmental movement.
Sign up for our newsletter: http://www.groundwire.org/email-capture
Index: src/persistent/picklecache.py
===================================================================
--- src/persistent/picklecache.py (revision 122718)
+++ src/persistent/picklecache.py (working copy)
@@ -65,7 +65,8 @@
raise ValueError('OID must be string: %s' % oid)
# XXX
if oid in self.persistent_classes or oid in self.data:
- raise KeyError('Duplicate OID: %s' % oid)
+ if self.data[oid] is not value:
+ raise KeyError('Duplicate OID: %s' % oid)
if type(value) is type:
self.persistent_classes[oid] = value
else:
@@ -255,3 +256,5 @@
elif oid in self.persistent_classes:
del self.persistent_classes[oid]
+ def update_object_size_estimation(self, oid, new_size):
+ pass
Index: src/persistent/pyPersistence.py
===================================================================
--- src/persistent/pyPersistence.py (revision 122718)
+++ src/persistent/pyPersistence.py (working copy)
@@ -80,7 +80,7 @@
__slots__ = ('__jar', '__oid', '__serial', '__flags', '__size')
implements(IPersistent)
- def __new__(cls):
+ def __new__(cls, *args, **kw):
inst = super(Persistent, cls).__new__(cls)
inst.__jar = None
inst.__oid = None
Index: src/persistent/__init__.py
===================================================================
--- src/persistent/__init__.py (revision 122718)
+++ src/persistent/__init__.py (working copy)
@@ -39,6 +39,13 @@
except ImportError:
from picklecache import PickleCache
+try:
+ import TimeStamp
+except ImportError:
+ import timestamp as TimeStamp
+ import sys
+ sys.modules['persistent.TimeStamp'] = sys.modules['persistent.timestamp']
+
if _HAVE_CPERSISTENCE:
# Make an interface declaration for Persistent, if zope.interface
# is available. XXX that the pyPersistent version already does this?
Index: src/BTrees/___BTree.py
===================================================================
--- src/BTrees/___BTree.py (revision 122717)
+++ src/BTrees/___BTree.py (working copy)
@@ -477,7 +477,19 @@
for i in range(0, len(state), 2):
keys.append(state[i])
values.append(state[i+1])
+
+ def toString(self):
+ return ''.join(self.keys() + self.values())
+
+ def fromString(self, s):
+ r = len(s) // 8
+ for i in range(r):
+ k = s[i*2:i*2+2]
+ v = s[r*2+i*6:r*2+i*6+6]
+ self[k] = v
+ return self
+
class Set(_SetBase, _BucketBase):
def __getstate__(self):
Index: src/BTrees/objectkeymacros.h
===================================================================
--- src/BTrees/objectkeymacros.h (revision 122717)
+++ src/BTrees/objectkeymacros.h (working copy)
@@ -9,6 +9,7 @@
static int
check_argument_cmp(PyObject *arg)
{
+ return 1;
/* printf("check cmp %p %p %p %p\n", */
/* arg->ob_type->tp_richcompare, */
/* ((PyTypeObject *)object_)->ob_type->tp_richcompare, */
Index: src/ZEO/zrpc/marshal.py
===================================================================
--- src/ZEO/zrpc/marshal.py (revision 122717)
+++ src/ZEO/zrpc/marshal.py (working copy)
@@ -33,7 +33,7 @@
# pickle.py does not.
pickler = Pickler(1)
pickler.fast = 1
- return pickler.dump(args, 1)
+ return pickler.dump(args)
@apply
Index: src/ZODB/serialize.py
===================================================================
--- src/ZODB/serialize.py (revision 122717)
+++ src/ZODB/serialize.py (working copy)
@@ -625,9 +625,9 @@
refs = []
u = cPickle.Unpickler(cStringIO.StringIO(p))
- u.persistent_load = refs
- u.noload()
- u.noload()
+ u.persistent_load = refs.append
+ u.load()
+ u.load()
# Now we have a list of referencs. Need to convert to list of
# oids:
Index: setup.py
===================================================================
--- setup.py (revision 122717)
+++ setup.py (working copy)
@@ -121,6 +121,7 @@
cPickleCache,
TimeStamp,
]
+exts = []
def _modname(path, base, name=''):
if path == base:
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
