Re: [ZODB-Dev] RelStorage and PosKey errors - is this a risky hotfix?
Hi Shane, Thanks for pursuing this. I have lots of other ideas now, but I don't know which to pursue. I need a lot more information. It would be helpful if you sent me your database to analyze. Some possible causes: - Have you looked for filesystem-level corruption yet? I asked this before and I am waiting for an answer. Yep, I've checked for file system consistency and Mysql consistency without any error reported. - Although there is a pack lock, that lock unfortunately gets released automatically if MySQL disconnects prematurely. Therefore, it is possible to force RelStorage to run multiple pack operations in parallel, which would have unpredictable effects. Is there any possibility that you accidentally ran multiple pack operations in parallel? For example, maybe you have a cron job, or you were setting up a cron job at the time, and you started a pack while the cron job was running. (Normally, any attempt to start parallel pack operations will just generate an error, but if MySQL disconnects in just the right way, you'll get a mess.) That's not unlikely! I've actually seen traces of packing invoked TTW, however the cron job uses zodbpack. I will try to figure out if the PosKeys starts to surface right after that. - Every SQL database has nasty surprises. Oracle, for example, has a nice read only mode, but it turns out that mode works differently in RAC environments, leading to silent corruption. As a result, we never use that feature of Oracle anymore. Maybe MySQL has some nasty surprises I haven't yet discovered; maybe the MySQL-specific delete using statement doesn't work as expected. That could also be the case. In fact we have also seen Mysql locking up longer than expected, but that's another story. - Applications can accidentally cause POSKeyErrors in a variety of ways. For example, persistent objects cached globally can cause POSKeyErrors. Maybe Plone 4 or some add-on uses ZODB incorrectly. I was not aware of that. Next step here would probably be to inspect log files further and grab a copy of the dabase before PosKeys started to appear and see if it is possible to recreate the incident. Again, thanks. Anton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] RelStorage and PosKey errors - is this a risky hotfix?
Hi there,
We have recently experienced a couple of PosKey errors with a Plone 4 site
running RelStorage 1.4.1 and Mysql 5.1.
After digging down we found that the objects that were throwing
PosKeyErrors actually existed in the object_state table with pickles etc,
however not in the current_object table.
After inserting the missing pointers into the current_object table,
everything worked fine:
mysql SELECT zoid, tid FROM object_state WHERE zoid=561701;
+++
| zoid | tid|
+++
| 561701 | 255267099158685832 |
+++
mysql INSERT INTO current_object(zoid, tid) VALUES('561701',
'255267099158685832');
Looks like it works -- but is this a safe way to fix PosKeyErrors?
Now, I wonder why these pointers were deleted from the current_object table
in the first place. My money is on packing -- and it might fit with the fact
that we recently ran a pack that removed an unusual large amount of
transactions in a single pack (100.000+ transactions).
But I don't know how to investigate the root cause further. Ideas?
/Anton
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] RelStorage: Write on master, read from slave(s)?
Hi, In order to scale an application using RelStorage I was thinking about seperating reads and writes accross databases. Writes would go to a Mysql master and the app would read from one or more Mysql slaves. It looks like RelStorage already keeps seperate connections for read and write (even though they are not exposed through configuration), so it is probably easy to set up. Would it work or is there a risk of getting inconsistent data? Or is that entirely up to the ability of doing synchronous replication of the underlying database? /Anton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage: Write on master, read from slave(s)?
Hi Shane, In order to scale an application using RelStorage I was thinking about seperating reads and writes accross databases. Writes would go to a Mysql master and the app would read from one or more Mysql slaves. If you mean that you intend to set up some clients to write to a master database, while other clients only read from asynchronous replicas, then yes, you'll be in good shape. You should be able to get amazing scalability that way. That is what I want. Sounds great. My assumption was that if a client was reading from a relica before fresh data was replicated I would get old data. That was why I was talking about synchronous replication. /Anton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Traceback with RelStorage 1.4.1 + Zope 2.12.12 + python_memcached 1.4.5
2010/10/22 Sylvain Viollon sylv...@infrae.com wrote: I have a Zope 2, using RelStorage and python_memcached. I use python_memcached because I use it in some other extension. I got the following traceback on a commit (..) TypeError: 'int' object is unsubscriptable Shane answered that one last month: https://mail.zope.org/pipermail/zodb-dev/2010-September/013651.html /Anton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage: Memcache clients throws errors on commit now and then
Hi Shane, 2010/9/15 Shane Hathaway sh...@hathawaymix.org This is a bug in python-memcached. Under some circumstances, _val_to_store_info() returns 0 (see line 713), yet set_multi(), line 643, is not prepared to handle that return value. You would be doing a good service if you reported this to the authors. I always use pylibmc. It's slightly harder to set up, but in my experience, it's fast and reliable. If you use it, be Thanks alot! Will move to pylibmc. I actually thought I was choosing stability over speed by going with python-memcached instead of a wrapper around a C-library (fear of potential issues with concurrency and memory leaks). /Anton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage: Memcache clients throws errors on commit now and then
2010/9/14 Alan Runyan runy...@gmail.com you didnt specify version of relstorage or zodb you are using, nor version of memcahe or python memcached lib version Good point. python-memcached 1.45 ZODB 3.9.5 Zope 2.12.10 Relstorage 1.4.0c3 /Anton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: Note for the docs: RelStorage requires at least MySQLdb 1.2.2 to work with Mysql
Hi Shane, Well, it's already in the docs. Look for 1.2.1 in README.txt. Doh - I was too fast. I'll accept suggestions on how to reorganize the documentation to make this more noticeable. Well, it's actually quite clear as is, I realize. But a trouble shooting section possible including the error message would be icing on the cake. My guess is you're using Debian Etch. I wonder if it would be possible to get the Debian package maintainer to upgrade Etch's version of python-mysqldb, since the version that's there now is awfully buggy. Right. /Anton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: Relstorage
Hi there, Ben Mason wrote: We've come across the following problem: When trying to revert to an old version of a page, we get a unicode error: Module Products.CMFEditions.ZVCStorageTool, line 505, in _encodeMetadata UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 0: ordinal not in range(128) All that has been done here is... Click Edit on the front page of a clean Plone site, click Save. Go to the History tab and click on 'Revert to this revision' This doesn't happen if we're using the traditional Data.fs way of storing data. Anyone know what is going on? I just had a look at this and am capable of reproducing it (on Plone 3.1.1) -- but also with filestorage! So it doesn't really look like a RelStorage issue to me. Are you sure you are using the same Plone and CMFEditions versions when you are comparing the storages? The reason for this error is that Products.CMFEditions.ZVCStorageTool,_encodeMetadata expects comment to be a string, but with the newest Plone it gets an unicode object. Probably due to changes in CMFEdtions skinscript revertversion.py. /Anton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: ZEO Client deadlocking in asyncore.poll - how to I debug
Thanks for you suggestions, Alan, Roché and Dieter, I'll switch the zeo server logging to debug level even though the amount of data is scary -- and try to find a way to reduce the load on the ZEO server (Roché). I think you (Alan and Dieter) might be right that there could be a network issue that gets triggered during high load. We don't have any apparent package filtering rules. Maybe having a closer look with tcpdump/wireshark could reveal something. I'll keep you posted. While we are working on getting to the root of this, isn't there a way to set a timeout on the client side, so it wont wait forever for a response that are lost in the mail? Thanks again, Anton ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
