[ZODB-Dev] Relstorage zodbconvert's --incremental

2014-12-18 Thread Sylvain Viollon 
Hello,

 I found out that this option did not work at all and seemed to be half 
implemented.

 I believe I fixed it: 
https://github.com/minddistrict/relstorage/compare/zodb:master...master

 If you have any comment or thing I could have missed.

 I am testing it right now on a sizable database, interrupting the process 
and starting over
after and so on. I will do a pull request if everything goes ok.

 Sylvain,

— 
Sylvain Viollon
Minddistrict Development
http://minddistrict.com

___
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Relstorage zodbconvert's --incremental

2014-12-18 Thread Sylvain Viollon 

 Op 18 dec. 2014, om 11:40 heeft Mauro Amico mauro.am...@gmail.com het 
 volgende geschreven:
 
 There is also another PR for same, could you try or compare with your 
 implementation?
 
 https://github.com/zodb/relstorage/pull/19
 

   Yes, except you cannot change the signature of copyTransactionsFrom, since 
if you convert
from relstorage to a regular storage, the ZODB implementation is called instead 
of the one in
relstorage. I made this proxy object in order to have it work with all backends.

Sylvain,

— 
Sylvain Viollon
Minddistrict Development
http://minddistrict.com

___
For more information about ZODB, see http://zodb.org/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

[ZODB-Dev] Bug during pack with Relstorage 1.5.0

2011-08-30 Thread Sylvain Viollon 


  Hello,

I have a customer using RelStorage 1.5.0 in production, and he cannot pack 
anymore his Data.fs. When he tries he have the following error:

 2011-08-29 15:43:03,459 [zodbpack] INFO Opening storage (RelStorageFactory)...
 2011-08-29 15:43:03,966 [zodbpack] INFO Packing storage (RelStorageFactory).
 2011-08-29 15:43:03,974 [relstorage] INFO pack: analyzing transactions 
 committed Fri Aug 26 18:04:56 2011 or before
 2011-08-29 15:43:03,977 [relstorage.adapters.packundo] INFO pre_pack: start 
 with gc enabled
 2011-08-29 15:43:04,784 [relstorage.adapters.packundo] INFO analyzing 
 references from objects in 436309 new transaction(s)
 /var/www/env_ucpa_silva/ucpa_silva/eggs/RelStorage-1.5.0-py2.6.egg/relstorage/adapters/scriptrunner.py:52:
  Warning: Field 'zoid' doesn't have a default value
 cursor.execute(stmt, generic_params)
 2011-08-29 15:43:04,799 [relstorage.adapters.scriptrunner] WARNING script 
 statement failed: '\nINSERT INTO object_refs_added (tid)\n VALUES 
 (%(tid)s)\n'; parameters: {'tid': 247097224207245124L}
 2011-08-29 15:43:04,799 [relstorage.adapters.packundo] ERROR pre_pack: failed
 Traceback (most recent call last):
 File 
 /var/www/env_ucpa_silva/ucpa_silva/eggs/RelStorage-1.5.0-py2.6.egg/relstorage/adapters/packundo.py,
  line 486, in pre_pack
   conn, cursor, pack_tid, get_references)
 File 
 /var/www/env_ucpa_silva/ucpa_silva/eggs/RelStorage-1.5.0-py2.6.egg/relstorage/adapters/packundo.py,
  line 580, in _pre_pack_with_gc
   self.fill_object_refs(conn, cursor, get_references)
 File 
 /var/www/env_ucpa_silva/ucpa_silva/eggs/RelStorage-1.5.0-py2.6.egg/relstorage/adapters/packundo.py,
  line 387, in fill_object_refs
   self._add_refs_for_tid(cursor, tid, get_references)
 File 
 /var/www/env_ucpa_silva/ucpa_silva/eggs/RelStorage-1.5.0-py2.6.egg/relstorage/adapters/packundo.py,
  line 459, in _add_refs_for_tid
   self.runner.run_script_stmt(cursor, stmt, {'tid': tid})
 File 
 /var/www/env_ucpa_silva/ucpa_silva/eggs/RelStorage-1.5.0-py2.6.egg/relstorage/adapters/scriptrunner.py,
  line 52, in run_script_stmt
   cursor.execute(stmt, generic_params)
 File 
 /var/www/env_ucpa_silva/ucpa_silva/eggs/MySQL_python-1.2.3-py2.6-linux-x86_64.egg/MySQLdb/cursors.py,
  line 174, in execute
   self.errorhandler(self, exc, value)
 File 
 /var/www/env_ucpa_silva/ucpa_silva/eggs/MySQL_python-1.2.3-py2.6-linux-x86_64.egg/MySQLdb/connections.py,
  line 36, in defaulterrorhandler
   raise errorclass, errorvalue
 IntegrityError: (1062, Duplicate entry '0' for key 'PRIMARY')

  He tried to cleanup the tables object_refs_added and object_refs but the 
error persist and seems to stay the same.

  Before 1.5.0 they had 1.4.1 (the database have been created using that 
version), and when this problem appear, I recommended as 1.5.0 contains various 
fixes for pack on large database. 

  Do you have any ideas on how to fix this pack ?

  Regards,

  Sylvain,


-- 
Sylvain Viollon -- Infrae
t +31 10 243 7051 -- http://infrae.com
Hoevestraat 10 3033GC Rotterdam -- The Netherlands



___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] zc.zodbdgc bug - undefined variable

2011-08-08 Thread Sylvain Viollon 


  Hello,

Op 5 aug 2011, om 22:12 heeft Jim Fulton het volgende geschreven:

 On Fri, Aug 5, 2011 at 1:55 PM, Hanno Schlichting ha...@hannosch.eu wrote:
 
 
 Indeed, in current trunk oldtid is used on line 398 and 400. Looking
 at the code I cannot tell if it's safe to just remove that code.
 
 It is not. The bug is on line 395.  It should be:
 
oldtid = f.read(8)
 
 Of course, this needs to be verified with a test.
 
 This seems to be a (obviously untested) case where
 there are multiple records of a garbage object.  I'll have to
 ponder how this could actually happen.
 

  That's a very old Data.fs, from 2002, where we always upgraded the 
application without creating
  a new Data.fs. I think you have all the different format and case of pickling 
that the ZODB uses to
  encode references in it, I already got trouble because the Python pickle 
module have been cleaned
  in Python 2.6 of a specific ZODB hack. I fixed it with zodbupdate.

  I think that Data.fs is a pretty good test for any ZODB tool.

  Regards,

  Sylvain,

-- 
Sylvain Viollon -- Infrae
t +31 10 243 7051 -- http://infrae.com
Hoevestraat 10 3033GC Rotterdam -- The Netherlands



___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] My zc.zodbdgc docs (incl. approach to fixing poskey errors)

2011-08-01 Thread Sylvain Viollon 

Op 24 jul 2011, om 16:59 heeft Hanno Schlichting het volgende geschreven:

 Hi.
 

  Hello,

 I polished my zc.zodbdgc a bit and published them at
 http://zodb.org/documentation/articles/multi-zodb-gc.html
 
 Those include a description on using the reference database to find
 and fix PosKeyErrors (tested by Andreas).
 

  I had a look at it today.

  However, if I run multi-check-gc, I get a Python error:

Traceback (most recent call last):
  File ./bin/multi-zodb-gc, line 210, in module
zc.zodbdgc.gc_command()
  File 
/Users/sylvain/Documents/debug-eggs/zc.zodbdgc-0.6.0-py2.7.egg/zc/zodbdgc/__init__.py,
 line 99, in gc_command
untransform=untransform, ptid=ptid)
  File 
/Users/sylvain/Documents/debug-eggs/zc.zodbdgc-0.6.0-py2.7.egg/zc/zodbdgc/__init__.py,
 line 105, in gc
return gc_(close, conf, days, ignore, conf2, fs, untransform, ptid)
  File 
/Users/sylvain/Documents/debug-eggs/zc.zodbdgc-0.6.0-py2.7.egg/zc/zodbdgc/__init__.py,
 line 220, in gc_
getrefs(data, name, ignore))
  File 
/Users/sylvain/Documents/debug-eggs/zc.zodbdgc-0.6.0-py2.7.egg/zc/zodbdgc/__init__.py,
 line 398, in insert
tid = max(tid, oldtid)
NameError: global name 'oldtid' is not defined

  Which is true, oldtid is not defined anywhere in this Python module.

  I used zc.zodbdgc 0.6 and ZODB 2.10.3 (Zope 2 2.13.8) with Python 2.7.

  (I just run the script with a database that do have POSKeyError, using only a 
configuration file).

  Regards,

  Sylvain,

-- 
Sylvain Viollon -- Infrae
t +31 10 243 7051 -- http://infrae.com
Hoevestraat 10 3033GC Rotterdam -- The Netherlands



___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

[ZODB-Dev] Zeo server error message

2011-01-31 Thread Sylvain Viollon 
Hello,

  I have a customer who run a ZEO setup with ZODB 3.8.4. They report
  this error:

2011-01-31T12:02:24 Unexpected error
Traceback (most recent call last):
  File
/opt/zope/newbuildout/parts/zope2/lib/python/ZODB/ConflictResolution.py,
line 207, in tryToResolveConflict
resolved = resolve(old, committed, newstate)
  File
/opt/zope/newbuildout/parts/zope2/lib/python/ZODB/ConflictResolution.py,
line 141, in __cmp__
raise ValueError(
ValueError: can't reliably compare against different
PersistentReferences

  Does anyone know to what this error relate ?

  They have been running their ZEO setup for quite a while, and never
  experienced any issue. That doesn't seems to 'break' the database either.

  I am not really sure what to say about this error.

  Regards,

  Sylvain,

-- 
Sylvain Viollon -- Infrae
t +31 10 243 7051 -- http://infrae.com
Hoevestraat 10 3033GC Rotterdam -- The Netherlands
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Traceback with RelStorage 1.4.1 + Zope 2.12.12 + python_memcached 1.4.5

2010-10-25 Thread Sylvain Viollon 
On Fri, 22 Oct 2010 19:05:18 +0200
Anton Stonor an...@headnet.dk wrote:

 2010/10/22 Sylvain Viollon sylv...@infrae.com wrote:
 
  I have a Zope 2, using RelStorage and python_memcached. I use
   python_memcached because I use it in some other extension.
 
   I got the following traceback on a commit (..)
 
 TypeError: 'int' object is unsubscriptable
 
 
 
 Shane answered that one last month:
 
 https://mail.zope.org/pipermail/zodb-dev/2010-September/013651.html
 

  Sorry, I should have looked a bit more.

  Sylvain,


-- 
Sylvain Viollon -- Infrae
t +31 10 243 7051 -- http://infrae.com
Hoevestraat 10 3033GC Rotterdam -- The Netherlands
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

[ZODB-Dev] Traceback with RelStorage 1.4.1 + Zope 2.12.12 + python_memcached 1.4.5

2010-10-22 Thread Sylvain Viollon 
Hello,

  I have a Zope 2, using RelStorage and python_memcached. I use
  python_memcached because I use it in some other extension.

  I got the following traceback on a commit (some objects did
  change, and some where added to the database during the
  transaction):

Traceback (most recent call last):
  File .../eggs/repoze.who-2.0a3-py2.6.egg/repoze/who/middleware.py, line 88, 
in __call__
app_iter = app(environ, wrapper.wrap_start_response)
  File .../eggs/infrae.wsgi-1.1-py2.6.egg/infrae/wsgi/publisher.py, line 364, 
in __call__
return publication()
  File .../eggs/infrae.wsgi-1.1-py2.6.egg/infrae/wsgi/publisher.py, line 302, 
in __call__
data = self.publish_and_retry()
  File .../eggs/infrae.wsgi-1.1-py2.6.egg/infrae/wsgi/publisher.py, line 273, 
in publish_and_retry
data = self.publish()
  File .../eggs/infrae.wsgi-1.1-py2.6.egg/infrae/wsgi/publisher.py, line 266, 
in publish
return self.result()
  File .../eggs/infrae.wsgi-1.1-py2.6.egg/infrae/wsgi/publisher.py, line 165, 
in result
self.finish()
  File .../eggs/infrae.wsgi-1.1-py2.6.egg/infrae/wsgi/publisher.py, line 151, 
in finish
self.commit()
  File .../eggs/infrae.wsgi-1.1-py2.6.egg/infrae/wsgi/publisher.py, line 130, 
in commit
self.app.transaction.commit()
  File 
.../eggs/Zope2-2.12.12-py2.6-freebsd-8.0-RELEASE-amd64.egg/Zope2/App/startup.py,
 line 291, in commit
transaction.commit()
  File .../eggs/transaction-1.0.0-py2.6.egg/transaction/_manager.py, line 93, 
in commit
return self.get().commit()
  File .../eggs/transaction-1.0.0-py2.6.egg/transaction/_transaction.py, line 
322, in commit
self._commitResources()
  File .../eggs/transaction-1.0.0-py2.6.egg/transaction/_transaction.py, line 
424, in _commitResources
rm.tpc_finish(self)
  File 
.../eggs/ZODB3-3.9.7-py2.6-freebsd-8.0-RELEASE-amd64.egg/ZODB/Connection.py, 
line 798, in tpc_finish
self._storage.tpc_finish(transaction, callback)
  File .../eggs/RelStorage-1.4.1-py2.6.egg/relstorage/storage.py, line 860, 
in tpc_finish
self._finish(self._tid, u, d, e)
  File .../eggs/RelStorage-1.4.1-py2.6.egg/relstorage/storage.py, line 880, 
in _finish
self._cache.after_tpc_finish(self._tid)
  File .../eggs/RelStorage-1.4.1-py2.6.egg/relstorage/cache.py, line 385, in 
after_tpc_finish
self.send_queue(tid)
  File .../eggs/RelStorage-1.4.1-py2.6.egg/relstorage/cache.py, line 332, in 
send_queue
client.set_multi(to_send)
  File .../eggs/python_memcached-1.45-py2.6.egg/memcache.py, line 643, in 
set_multi
write(set %s %d %d %d\\r\\n%s\\r\\n % (key, store_info[0], time, 
store_info[1], store_info[2]))
TypeError: 'int' object is unsubscriptable

  I could not reproduce this error, but still it is worth to report I guess.

  Regards,

  Sylvain,

-- 
Sylvain Viollon -- Infrae
t +31 10 243 7051 -- http://infrae.com
Hoevestraat 10 3033GC Rotterdam -- The Netherlands
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

[ZODB-Dev] Status of gocept.zeoraid

2010-10-20 Thread Sylvain Viollon 
Hello,

  One of my customers would like to use gocept.zeoraid in production.
  So him, and I have some questions:

  - Is it stable enough to be used in production ?

  - Does it bring speed improvements over regular ZEO setup (I don't
think so) ?

  - Does it work with blobs ? I will have a blob directory shared
between different servers. Can I use it to access and edit directly
the blob in that directory ?

  Regards,

  Sylvain,

-- 
Sylvain Viollon -- Infrae
t +31 10 243 7051 -- http://infrae.com
Hoevestraat 10 3033GC Rotterdam -- The Netherlands
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] Testing gocept.zeoraid

2010-03-22 Thread Sylvain Viollon 
On Mon, 22 Mar 2010 10:37:05 -0400
Tres Seaver tsea...@palladion.com wrote:

  Hello,

 Sylvain Viollon wrote:
 
 I am currently testing gocept.zeoraid, and this seems to work
  fine.
  
 However, I have few questions:
  
 - On the pypi the last released version is 1.0b7. In the Zope
  SVN it seems to be only 1.0b1.
 
 Hmm, I see tags for 1.0a1, b1-b7::
 
  $ svn ls $ZSVN/gocept.zeoraid/tags
  1.0a1/
  1.0b1/
  1.0b2/
  1.0b3/
  1.0b4/
  1.0b5/
  1.0b6/
  1.0b7/
 


  Ha, sorry. I got owned again by viewvc which add a ?rev= to the URL.
  I hate that stuff.

  Regards,

  Sylvain,

-- 
Sylvain Viollon -- Infrae
t +31 10 243 7051 -- http://infrae.com
Hoevestraat 10 3033GC Rotterdam -- The Netherlands
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] zodbupdate branch sylvain-persistent-load

2010-02-02 Thread Sylvain Viollon 
On Tuesday 02 February 2010 09:08:42 Christian Theune wrote:
 Hi,
 
 

  Hello,

 I think I'm fine with your approach in general by now. I don't have the
 time to review anything, though. The old version is happily there in SVN.
 
 IIRC you hadn't updated all unit tests last time, did you catch up with
 that yet?
 

  I fixed the current tests yesterday. I removed a bunch of them a while ago, 
all the one which was testing the pickle modification code, since I removed 
the corresponding code.

  I plan to add a few more today,

 Otherwise, feel free to merge and release.
 

  Thank you, I think I will have look at the trunk, but if there is no changes 
in it since my branch, I will move it as a branch called pickle-version, and 
rename my branch as trunk. I will tag it, but if you want me to do the 
release, you need to give me rights on pypi. My username is thefunny42.

  Regards,

  Sylvain,


-- 
Sylvain Viollon -- Infrae
t +31 10 243 7051 -- http://infrae.com
Hoevestraat 10 3033GC Rotterdam -- The Netherlands
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] zodbupdate branch sylvain-persistent-load

2010-02-02 Thread Sylvain Viollon 
On Tue, 02 Feb 2010 11:30:41 +0100
Christian Theune c...@gocept.com wrote:


  Hello,

 Meh. That looks quirky in the history. If there are no changes, then
 the merge will be straight-forward.
 

  I merged my branch into the trunk, it's ready for a release.

  Regards,

  Sylvain,

-- 
Sylvain Viollon -- Infrae
t +31 10 243 7051 -- http://infrae.com
Hoevestraat 10 3033GC Rotterdam -- The Netherlands
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

Re: [ZODB-Dev] zodbupdate branch sylvain-persistent-load

2010-02-02 Thread Sylvain Viollon 
On Tue, 02 Feb 2010 16:44:19 +0100
Christian Theune c...@gocept.com wrote:

 On 02/02/2010 03:58 PM, Sylvain Viollon wrote:
  On Tue, 02 Feb 2010 11:30:41 +0100
  Christian Theune c...@gocept.com wrote:
  
  
Hello,
  
  Meh. That looks quirky in the history. If there are no changes,
  then the merge will be straight-forward.
 
  
I merged my branch into the trunk, it's ready for a release.
 
 What's your pypi ID? I'll give you release rights then.
 

  thefunny42, I already mentioned it in one of my previous mail.

  Sylvain,



-- 
Sylvain Viollon -- Infrae
t +31 10 243 7051 -- http://infrae.com
Hoevestraat 10 3033GC Rotterdam -- The Netherlands
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev

[ZODB-Dev] zodbupdate branch sylvain-persistent-load

2010-02-01 Thread Sylvain Viollon 
Hello,

   I did few month ago a branch of zodbupdate, which works completely 
differently than the trunk. Instead of changing the pickle code of the record, 
it unpickle it, changing references at that point, and repickle if there is 
changes.

  I did this in order to support the ZODB references that are implemented 
using persistent ID, in which a reference to a class can be expressed as a 
tuple (module, classname) instead of the class pickle opcode. That tuple 
(module, classname) can appear in the class meta pickle of a record as well. 
You can have more details about this by reading the comments of the 
serialize.py file in the ZODB code.

  If I understand correctly, those are legacy formats, that the ZODB doesn't 
use anymore to write data, but is still able to read data stored using those 
formats for backward compatibility support.

  The issue is that I want to use zodbupdate on database that have been 
created a long time ago, and I do need zodbupdate to be able to read those 
formats and correct class references in those records as well. That's the 
motivation of my changes.

  My version of zodbupdate is feature alike with the old one: 

- it is able to update records which have backward compatibility imports in 
the code, and report them at the end, even if they are not described in the 
renaming rules,

-  it is able to ignore missing modules and classes (it use ZODB.broken),

- command line invocation stays the same,

- performance-wise, it seems even to be faster (I use cPickle),

  As well, it's able to work on 'broken data.fs' which trigger POSKey errors.

  I did my testing using the infrae.com website database, which have been 
created around 2002 I think, and contain a lot of interesting cases.

  So I would like to release my work now, since I need to use it in production 
environment. Since there is lot of changes, I can release it as a fork as 
well, if it's not possible to merge it with the trunk.

  Regards,

  Sylvain, 

-- 
Sylvain Viollon -- Infrae
t +31 10 243 7051 -- http://infrae.com
Hoevestraat 10 3033GC Rotterdam -- The Netherlands
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev