Re: [ZODB-Dev] RelStorage: Which tables to replicate?
Hi, Shane Hathaway wrote: Stefan H. Holek wrote: We have put up two boxes in a MySQL master-master replication setup [1]. As long as we only write to one of the masters all is fine. Writing to both masters (and expecting MySQL to sort it out) is giving us trouble. We see things like: Last_Errno: 1062 Last_Error: Error 'Duplicate entry '41860' for key 1' on query. Default database: 'prod_zodb'. Query: 'INSERT INTO new_oid VALUES ()' Perhaps we can solve this by making each server allocate a different set of OIDs. For example, one server might allocate odd-numbered OIDs while the other allocates even-numbered OIDs. Even better would be some kind of standard MySQL solution, but I haven't found any yet. the standard approach for master-master replicated data is to use an UUID-Datatype (see http://en.wikipedia.org/wiki/UUID ) Last_Errno: 1050 Last_Error: Error 'Table 'temp_store' already exists' on query. Default database: 'prod_zodb'. Query: 'CREATE TEMPORARY TABLE temp_store ( zoidBIGINT NOT NULL PRIMARY KEY, prev_tidBIGINT NOT NULL, md5 CHAR(32), state LONGBLOB ) ENGINE MyISAM' Uh-oh. The commit lock should have prevented this, so this error suggests that each master has a completely independent set of locks! surprise ;) That is a serious issue that will lead to database corruption. The commit and pack locks need to be cluster-wide. Does MySQL have a way to do that? Well, synchronous replication is a hard beast, master-master synchronous even harder (and you have to ask what problem you really want to solve with it, since the trade-offs are massive) Tino smime.p7s Description: S/MIME Cryptographic Signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage: Which tables to replicate?
Stefan H. Holek wrote: I am playing with database replication (MySQL) and have found the following caveats: - don't mix engine types (MyISAM, InnoDB) - don't use temporary tables Now, RelStorage does both and so I was wondering how to proceed with replication. Would it be prudent to assume that all I ever need to Maybe use postgres and slony instead? The mysql shortcomings on replication and integrity should be well known. It should also be possible to replicate on ZODB layer instead. At least I read something in the ML beside ZRE of course :-) Cheers Tino smime.p7s Description: S/MIME Cryptographic Signature ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] adding mount points
Christian Theune wrote: Hi, Am Dienstag, den 18.12.2007, 01:42 -0800 schrieb Tarek Ziadé: Hi, I am working on a project that needs to do mass hosting of zope sites. I would like to have several site per zope instance, and to make a zodb mount point for each new site created to ease their maintenance. Is there a way to dynamically load new mount points on a ZODB to avoid relaunching the zope instance everytime a website is added ? (then reflecting the changes in zope.conf) My guess is to override getConfiguration() in ZODBMountpoint.MountedObject but maybe i miss something ? On ZODB-level this is not an issue with the multi-database API. I'm not sure about the specifics of the ZODBMountpoint product though. Just restarting a twin instance usually causes the mountpoint to appear so with a little wiring this should be doable within one instance as well :-) Regards Tino ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: AW: [ZODB-Dev] diploma thesis: ZODB Indexing
Christian Theune schrieb: Am Mittwoch, den 05.09.2007, 09:24 -0400 schrieb Jim Fulton: I'd like to see a generic framework for defining collections and indexes in Python and querying them efficiently. No ZODB expertise should be needed, I have the feeling you already pondered this a bit and have some more specific ideas ... :) I also have the feeling that our goal for ad-hoc querying would be incompatible with your envisioned framework for defining collections and indexes. My impression is many people have thought about this problem. Now someone stood up and starts actual working, this is good :-) I think ad-hoc queries are not per se incompatible, they would just act like sequential scan in relational databases - therefore work but not too efficient. Maybe the api can generate a warning if desired so the application developer can add indexing. Consequently thinking about the whole scope, sometime at the end we will even need further abstraction regarding authorization and access to objects and attributes. This would dramatically change the way zope works with ZODB but open a lot more uses of ZODB independently of the Zope world. Just my 1e-21 cents :-) Tino ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: AW: [ZODB-Dev] diploma thesis: ZODB Indexing
Sebastian Wehrmann schrieb: Am 4. September 2007, 16:17:27 Uhr schrieb Jim Fulton: I would very much like to see an open indexing+querying framework for Python objects. I'm thinking of something *like* an SQL engine that allowed one to plug in relation and index implementations and that took queries in some form, optimized them and executing them using the given index and relations. We plan to realize three things important to us: - We don't want indexing on application-level (e.g. application-specific) - We want ad-hoc queries - We don't want to rely on transforming the data into a relational model very well! Greets Tino ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Implementing Storage Decorators
Dieter Maurer schrieb: Jim Fulton wrote at 2007-5-4 14:40 -0400: On May 4, 2007, at 2:33 PM, Dieter Maurer wrote: Jim Fulton wrote at 2007-5-2 11:52 -0400: ... I think I still rather like explicit, but I'm on the fence about which approach is best. What do other people think? From your description, I would use a subclassing (and forget about proxy and copying). That would be a nightmare, on multiple levels: - All of the separate implementations would become tightly coupled, which is what happens with inheritance. - Either someone would have to create classes for the various permutations of features, or consumers would have to mix and match multiple classes to get what they want and sort out the variate internal implementation incompatibilities. Your decorators would become mixin classes and the final classes would list the features they like -- simpler than ZCML binding together... Of course, some features may not play well with one another. But, that will make problems also with proxies or copying... Even this could be sorted out with special cleanup or secure mixins which can do dirty tricks like accessing self.__class__.__bases__ to fixup ;) (Much like combining the copy with the subclass approach) I'm for mixin with strong guidelines on implementation (e.g. how to call private attributes to not pollute the namespace) Regards Tino ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] History-less FileStorage?
Stefan H. Holek schrieb: Do we have a history-less (i.e. no-grow) FileStorage? How do you think it should work? I mean, technically in theory? How would you implement MVCC? Regards Tino ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Commit Progress Indictor
Jim Fulton schrieb: ... I can put a fake progress indicator anytime that advances every second a bit and never reaches 100%. Good idea. :) I fear this idea is already patented by Microso~1 ;) Regards Tino ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
ot, was: Re: [ZODB-Dev] Re: What makes the ZODB slow?
Dieter Maurer wrote: ... > If for example 10 Zope objects are modified and > this cause the full text indexes to be updated > then this can cause more modifications than > the update of hundreds of Postgres rows > (as such rows cannot contain mass data -- due to the restriction > to simple types). > ... as a side note, postgres makes a bad counter-example because its not absolutely restricted to simple types :-) Regards Tino ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: [Zope-dev] tempstorage to ZEO?
Sidnei da Silva schrieb: I was looking, and trying to understand why 'tempstorage' is not included in the ZODB. Since some people might want to share sessions between ZEO clients, and that 'tempstorage' is what is mounted as '/temp_folder/session_data', why not make that easy to mount via ZEO? Does anyone see a problem with this? actually it is easy mountable via ZEO. I'm doing this already... Do you mean including the schema for zeo.conf per default? Regards Tino ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Before-abort hook?
Stefan H. Holek schrieb: Hi all, The ExtFile product I maintain keeps its payload on the filesystem. All operations are performed using temporary files which are then saved or deleted on transaction commit or abort respectively. Do you remember, when you use the tempfile module, you get a filehandle with a "nameless" file which automatically vanishes if you close the handle. Maybe this solves the problem for you? Regards Tino ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Re: [Zope-dev] New mailing list: [EMAIL PROTECTED] for all checkin messages
Jim Fulton schrieb: > > I've set up a new mailing list that gets all checkins for the > zope.org CVS and Subversion repositories: > > http://mail.zope.org/mailman/listinfo/checkins > > Note to contributors: > > Only members can send messages to this list. This means that you will > get messages for a while when you check in saying your messages are > being held for moderation. I am moderating the list. When I see > legitimate held messages, I'll add the sender to the list of valid > non-member senders. So if you don't want to subscribe, you don't have > to. :) I seen this with the cvs list. Cant the checkin messages not automatically accepted? I mean the sending server is known... Regards Tino ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] how to let java applet using mysql database work in zope/plone?
LonLon schrieb: > Thanks for your suggestion jens, but my java applet is working outside > zope, it has no problem with mySQL database. Only when I integrate my > applet into zope, it can not take mySQL data anymore. That's why I > request suggestions here. I am still hoping somebody knows how to fix > this problem. :-( Nothing to do with ZODB (since you are using mySQL?) and even not a Zope problem. Applets in a browser can only connect back to the same host. If you start your applet locally, it is allowed to connect to any host it wants. Read about it in your java documentation. Solutions: put the database and your web site on the same host (or use firewall/proxy tricks to let it appear like this) - or better access your database thru zope - via XMLRPC and friend - and this way get on topic :-) --Tino ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RE: lost "implementer"
Chris Withers schrieb: ... One of the non-cool things though, is that Firefox doesn't download links on pages without you clicking on them... bummer when you're working offline ;-) apt-get install wwwoffle :-) Regards Tino ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB with a Relational Database
José Carlos Senciales schrieb: thanks my phyton console return "0L", so i have have large file support. So if you still experience problems with files >2GB, you might move to another filesystem (if moving to another OS is not an option ;) ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB with a Relational Database
José Carlos Senciales schrieb:
thanks
How can i know if i have my Python configured with large file support ?
My version is:
Zope Version (Zope 2.8.4-final, python 2.3.5, win32)
Python Version 2.3.5 (#62, Feb 8 2005, 16:23:02) [MSC v.1200 32 bit
(Intel)]
System Platform win32
double-klick on the python.exe or start it in a console
window and try:
>>> f=open("somefile.txt","w")
>>> f.tell()
0L
>>>
if its 0L, you probably have large file support.
If its just 0, you probably dont.
Not sure if on windows it also depends on the filesystem
you use. Try with NTFS if in doubt.
Regards
Tino
___
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Query Regrading ZODB FileStorage(.fs file)
tav schrieb: >> Why is it significantly easier to protect the key[s] >> used for the encryption than the storage itself? > > > one could always passphrase-protect the key, i.e. use symmetric encryption. > > admittedly, this could potentially be brute-forced, but ... should be > good enough for most purposes? > And how does your Application (Zope) access the storage? Exactly. It needs the key - if it has the key - the "attacker" can just read the data thru the application. In the end this does not buy you anything but overhead. If you want to encrypt, just use a crypted filesystem as DM already suggested. Best performance, best transparency and well tested. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Fixing Broken Objects?
Am Donnerstag, den 13.10.2005, 18:09 -0400 schrieb Jim Fulton: > Tino Wildenhain wrote: > > Am Donnerstag, den 13.10.2005, 08:03 +0200 schrieb Andreas Jung: > > > >>--On 13. Oktober 2005 01:45:30 -0400 Chris Spencer > >><[EMAIL PROTECTED]> wrote: > >> > >> > >>>I noticed that if a ZEOServer client doesn't have the definition for an > >>>object's class, it loads it as a broken object. Is there any way to fix > >>>the broken object by obtaining the class source from the server? > >>> > >> > >>Your sources on the client side must be complete. Otherwise the object is > >>broken. The server has nothing to do with problem. Fix your client. > > > > > > While it would certainly be nice if the ZEO server could deliver > > codeobjects too. > > > I don't think it would be useful to deliver ordinary Python modules > via the DB. Why not? This way one could keep all code changes in a central repository, w/o the need of either manually syncing or some special (flaky) network filesystem. > It would be useful to support persistent modules, which would have > somewhat different semantics than standard Python modules. > There is limited experimental support for this in the Zope 3 repository. Another usecase and valid too - this way one could do an entire timetravel with code, data and objects in the ZODB. SVN made right, so to say :) > > > In Theory and with much much bad hackery (overwriting > > __import__ and friends) it could be possible. Jim? > > For some definition of "it", sure. Btw, (hijacking the thread because a bit related) how can I serialize the state of a generator object? (e.g. for caching it between requests) Pickle does not work for it (whyever) Regards Tino ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Fixing Broken Objects?
Am Donnerstag, den 13.10.2005, 08:03 +0200 schrieb Andreas Jung: > > --On 13. Oktober 2005 01:45:30 -0400 Chris Spencer > <[EMAIL PROTECTED]> wrote: > > > I noticed that if a ZEOServer client doesn't have the definition for an > > object's class, it loads it as a broken object. Is there any way to fix > > the broken object by obtaining the class source from the server? > > > > Your sources on the client side must be complete. Otherwise the object is > broken. The server has nothing to do with problem. Fix your client. While it would certainly be nice if the ZEO server could deliver codeobjects too. In Theory and with much much bad hackery (overwriting __import__ and friends) it could be possible. Jim? This would especially be nice in a non Zope-situation where you have ZEO deliver to many sites and code synchronization would be a mere nightmare. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] [Proposal] Size controlled ZODB cache
Am Donnerstag, den 02.06.2005, 19:28 +0200 schrieb Dieter Maurer: > Currently, the ZODB cache can only be controlled via the maximal number > of objects. This makes configuration complex as the actual limiting > factor is the amount of available RAM and it is very difficult to > estimate the size of the objects in the cache. > > I therefore propose the implementation of cache replacement policies > based on the estimated size of its objects. > > I propose to use the pickle size as the size estimate. > The connection could store the pickle size in the object > as "_p_size" (and may call a hook function "_p_estimateSize", > if it is defined -- but I do not think, we need this). > I am aware that the actual size of an object may significantly > differ from its pickle size, but usually, they will at least > be in the same order. > > As additional limiting parameters, I propose "MAX_OBJECT_SIZE" and > "MAX_TOTAL_SIZE". > > Objects with size >= "MAX_OBJECT_SIZE" are invalidated at the next > possible time (at a transaction boundary) before other potential > invalidations are considered. > The purpose of the limit it to prevent a single (or few) large objects > to flush large amounts of small objects. Such large objects > are managed in a special (doubly linked) list in order to quickly locate them. > > After large objects are flushed, the replacement policy works > as it does now. However, beside the number of objects, their > total estimated size is accumulated. As soon as > either the "MAX_OBJECT_NUMBER" or "MAX_TOTAL_SIZE" is reached, > the remaining objects are invalidated (as far as possible). > Would be nice to optionally have the posibility to define finer grained cache policy (e.g. reserve 80% of cache mem to objects <5k and 20% for all the others - or something the like - much like altq does for ip-traffic shaping). E.g. some clean hooks to cache store management and cache policy management. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] ZODB memory problems (was: processing a Very Large file)
Am Sonntag, den 29.05.2005, 09:51 +0200 schrieb Andreas Jung: > > --On 29. Mai 2005 11:29:06 +0200 Christian Theune <[EMAIL PROTECTED]> wrote: > > > Am Samstag, den 21.05.2005, 17:38 +0200 schrieb Christian Heimes: > >> Grab the Zope2 sources and read lib/python/OFS/Image.py. Zope's > >> OFS.Image.Image class (and also Zope3's implementation) is using a so > >> called possible large data class (Pdata) that is a subclass of > >> Persistent. > >> > >> Pdata is using a simple and genious approach to minimize the memory > >> usage when storing large binary data in ZODB. The data is read from a > >> [...] > > > > Actually Pdata has some drawbacks. When the blobsupport branch gets > > declared stable (I think it's not gonna happen in 3.4, but nobody told > > me otherwise) we'll have really good blob support without this black > > magic. Especially the ZEO handling of blobs could be improved IIRC. > > The Pdata approach in general is not bad. I have implemented a CVS-like file > repository lately where we store binary content using a pdata like > structure. > Our largest files are around (100MB) and the performance and efficiency is > not bad > although it could be better. The bottleneck is either the ZEO communication > or just the network. > I reach about 3.5 MB/second while reading such a large file from the ZEO > server. Thats not bad given that this might at least saturate most customers downstream :) Would a multi thread ZEO server improve anything here? Especially with concurrent access? ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
