Re: [ZODB-Dev] ZODB Book
It's a great idea, Alan. Also put me down for USD100, and tell me where/when to send it. A check would be simple. --r On Jun 16, 2010, at 1:05 PM, Roel Bruggink wrote: > On Wed, Jun 16, 2010 at 9:11 PM, Alan Runyan wrote: > > Greate idea! Just, what happens if the goal isn't met? > > Plus, I'd really like to see docs on DirectoryStorage. > > If I can not get enough contributions for an initial > advance for Carlos to begin work -- the project > will be aborted. > > As we start getting contributions there will be more > activity on the outline / material expected to be > covered. > > alan > > Aight, I'll personally donate 150 euro to the project. Let me know the paypal > account :) > > -- > Roel Bruggink > http://www.fourdigits.nl/mensen/roel-bruggink > > Four Digits BV > http://www.fourdigits.nl > Willemsplein 44, 6811 KD, Arnhem > tel: +31(0)26 4422700 fax: +31(0)84 2206117 > KVK 09162137 BTW 8161.22.234.B01 > ___ > For more information about ZODB, see the ZODB Wiki: > http://www.zope.org/Wikis/ZODB/ > > ZODB-Dev mailing list - ZODB-Dev@zope.org > https://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org https://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] Minimal changes for Plone 3.1.6 running on ZODB 3.8.1 ?
Hi all, In a warty old application, I'm hitting memory constraints that I think the 3.8.1 changes in ZODB might fix. Is someone using Plone on 3.8.1 already? If so, what should I change in my highly customised buildout to give it a try? --r. Russ Ferriday - Topia Systems - Open Source content management with Plone and Zope [EMAIL PROTECTED] - office: +44 2076 1777588 - mobile: +44 7789 338868 - skype: ferriday ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] SVN woes.
On 11 Nov 2008, at 18:15, Jim Fulton wrote: > > The repository is back now. Much thanks to Christian Theune for > providing a replication image. > > Jim > > -- > Jim Fulton > Zope Corporation > And many thanks to you both for the work involved in setting up these servers and keeping them running. I've been using Andreas Jung's server these past few days. Thanks to Andreas, too. Best wishes, --r. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage: Which tables to replicate?
Stefan! This is like holding all your eyelids apart and shouting out for somebody with a sharp stick! ;) Watcha gonna do when you add a third server? er.. lemme see... Multiply all OIDs by three and allocate new OIDs modulo three? Eeek! Why not make the OID a composite of server ID and sequence? Even if these were both packed in the same 64 bits, just so you keep it a *little bit nasty*, and then maybe just get one eye poked out! ;) (You could get away with a byte for the server ID, leaving masses of bits for the item ID.) Best wishes! --r On 22 May 2008, at 14:49, Stefan H. Holek wrote: Hi Russ, This is correct. I should have mentioned that the two servers are configured so that server A creates odd, server B even sequence numbers. This makes the duplicate id error pretty mysterious, IMO ;-) Stefan On 22.05.2008, at 13:15, Russ Ferriday wrote: I have not read the reference, but it seems that this scheme can never work without keys (OIDs) being qualified by which master they originated on. Otherwise one of the mySQLs would need to be responsible for handing out unique oids. You need an OID origination namespace for each master. -- Stefan H. Holek [EMAIL PROTECTED] Russ Ferriday - Topia Systems - Open Source content management with Plone and Zope [EMAIL PROTECTED] - office: +44 2076 1777588 - mobile: +44 7789 338868 - skype: ferriday ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] RelStorage: Which tables to replicate?
Stefan, Thanks for opening this thread here. I'm not actually doing anything like this, but it's great to see you pushing the envelop. I have done a lot of work on replicated data, over the years, that's where the following comment comes from. On 22 May 2008, at 10:36, Stefan H. Holek wrote: We have put up two boxes in a MySQL master-master replication setup [1]. As long as we only write to one of the masters all is fine. Writing to both masters (and expecting MySQL to sort it out) is giving us trouble. We see things like: Last_Errno: 1062 Last_Error: Error 'Duplicate entry '41860' for key 1' on query. Default database: 'prod_zodb'. Query: 'INSERT INTO new_oid VALUES ()' I have not read the reference, but it seems that this scheme can never work without keys (OIDs) being qualified by which master they originated on. Otherwise one of the mySQLs would need to be responsible for handing out unique oids. You need an OID origination namespace for each master. On the other issue, no idea. Best, --r. and Last_Errno: 1050 Last_Error: Error 'Table 'temp_store' already exists' on query. Default database: 'prod_zodb'. Query: 'CREATE TEMPORARY TABLE temp_store ( zoidBIGINT NOT NULL PRIMARY KEY, prev_tidBIGINT NOT NULL, md5 CHAR(32), state LONGBLOB ) ENGINE MyISAM' Stefan [1] http://www.howtoforge.com/mysql_master_master_replication On 20.05.2008, at 18:36, Shane Hathaway wrote: That is mostly correct. You need to replicate transaction, object_state, current_object, and do something with new_oid. There is no requirement to replicate the other tables, since they are used only for packing, although replicating object_ref and object_refs_added may save some work. The new_oid table is special to the MySQL adapter. RelStorage uses the new_oid table as a kind of sequence. Like any sequence, it is intentionally non-transactional. Perhaps there would be no ill effects in switching it to the InnoDB engine. Which MySQL replication method are you looking into? -- Stefan H. Holek [EMAIL PROTECTED] ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev Russ Ferriday - Topia Systems - Open Source content management with Plone and Zope [EMAIL PROTECTED] - office: +44 2076 1777588 - mobile: +44 7789 338868 - skype: ferriday ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
This is the 'batch' or 'distribute' pattern that crops up in many fields. The best path is normally to understand what the conflicts are, and where the time is spent. If in, this case, much time is spent in the preamble, and the actual inserts are quick, then diving down one time through the security layers and stuffing in 10 items is clearly better than 10 preambles, one for each insert. The other truism is that all optimisation is for a single case. There may be different answers for different cases. Ideally a single parameter would be enough to tune the system for different cases. Good luck Roche, with the outcome, I'm excited to see some figures. --r. On 2 Nov 2007, at 15:24, David Binger wrote: On Nov 2, 2007, at 10:58 AM, Lennart Regebro wrote: It seems to me having one thread doing a background consolidation one transaction at a time seems a better way to go, Maybe, but maybe that just causes big buckets to get invalidated in all of the clients over and over again, when we could accomplish the same objective in one invalidation by waiting longer and executing a bigger consolidation. although certainly the best thing would be to test all kinds of solutions and see. No doubt about that. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev Russ Ferriday [EMAIL PROTECTED] office: +44 118 3217026 mobile: +44 7789 338868 skype: ferriday ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: ZODB Benchmarks
Quick note... Smaller buckets, fewer conflicts, more overhead on reading and writing. Larger buckets, more conflicts, less overhead on reading and writing. One bucket ... constant conflicts. I'd bet that the additional tree with tiny buckets would be best. Transfer them into the normal tree once the overhead starts rising. How to transfer them? You would not want a single transaction to take the hit for the whole transfer, so have a low and high water mark. When hitting HWM, transfer only until LWM is reached. Or, just focus on transferring some items out of a single tree, to avoid the cost of tree rebalancing on the additional tree at the same time as rebalancing on the main tree. Sounds like a fin project, --r. On 1 Nov 2007, at 21:00, David Binger wrote: On Nov 1, 2007, at 4:25 PM, Matt Hamilton wrote: David Binger mems-exchange.org> writes: On Nov 1, 2007, at 7:05 AM, Matt Hamilton wrote: Ie we perhaps look at a catalog data structure in which writes are initially done to some kind of queue then moved to the BTrees at a later point. A suggestion: use a pair of BTrees, one with a high branching factor (bucket size) and one with a very low branching factor. Force all writes into the tree with little buckets. Make every search look in both trees. Consolidate occasionally. An interesting idea. Surely we need the opposite though, and that is an additional BTree with a very large bucket size, as we want to minimize the chance of a bucket split when inserting? Then we occasionally consolidate and move the items in the original BTree with the regular bucket size/ branch factor. You may be right about that. Conflict resolution makes it harder for me to predict which way is better. If you don't have conflict resolution for insertions, then I think the smaller buckets are definitely better for avoiding conflicts. In either case, smaller buckets reduce the size and serialization time of the insertion transactions, and that alone *might* be a reason to favor them. I think I'd still bet on smaller buckets, but tests would expose the trade-offs. ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev Russ Ferriday [EMAIL PROTECTED] office: +44 118 3217026 mobile: +44 7789 338868 skype: ferriday ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] more lockup information / zope2.9.6+zodb 3.6.2
On 10 Apr 2007, at 21:39, Chris Withers wrote: I'd look at the switches and maybe even the nics and cables :-S And if you are doing that, also check your network to make sure all IP addresses are unique. (..but I'm not a sysadmin.) And neither am I! --r Russ Ferriday - Topia Systems - Open Source content management with Plone and Zope [EMAIL PROTECTED] - office: +44 2076 1777588 - mobile: +44 7789 338868 - skype: ferriday a member of Zea Partners ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] We need to pay down debt
Do you have some favorite documents that you would point me to as most accurate or up to date? Decent clues will be enough to get started. - Doc in test or source files - web pages, - forums chats, - email threadsWhat should one not bother with (out of date, incorrect, etc.)Anyone else feel free to chime in, I'll collect links and see what comes together.--rOn 5 Oct 2006, at 13:50, Jim Fulton wrote:Obviously, you should be guided, at least in part, by existing non-executable documentation. Russ Ferriday - Topia Systems - multilingual content management contact: [EMAIL PROTECTED] - (+44) (0)2076 1777588 - skype: ferriday a member of the evenios group___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] We need to pay down debt
On 5 Oct 2006, at 14:37, Jim Fulton wrote:Note that I suggest we want new documentation that is executable. Perhaps we can reuse some of this other material, but I'm mainly pointing out that this other content might give some hints about scope or organization. got it.--r Russ Ferriday - Topia Systems - multilingual content management contact: [EMAIL PROTECTED] - (+44) (0)2076 1777588 - skype: ferriday a member of the evenios group___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] We need to pay down debt
I'll provide some time for this, Jim.I'm no expert on the ZODB, which might be spun as an advantage, and I'm prepared to play a supporting role cleaning up doctests, or helping with doc organization. This means I don't mind gathering spippets from those who have them and pulling them together.--rOn 5 Oct 2006, at 13:32, Jim Fulton wrote:I'm happy that there is interest in improving ZODB. I think, however,that the most pressing need is for documentation.When I originally developed the ZODB, I created a UML model: http://www.zope.org/Documentation/Developer/Models/ZODBThis provided a fairly thorough and clear documentation ofthe ZODB architecture at the time. It still contains useful information.Unfortunately, the UML model became corrupted by the tool I was usingand could not be updated. Given advances in our own technologiessince then, I don't think I would use UML today, except perhapsas a tool for making pictures. (Diagrams can be very useful andtheir benefits should not be underestimated.)We desperately need current and correct documentation of theZODB architecture and APIs. I think that the lack ofclear documentation and specifications severely hampers ourability to make progress. (It certainly hampers mine.)ZODB authors were aggressive in creating automated tests for ZODB.They should be commended for this. These tests, however,were (mostly) created before we learned how to employ docteststo create executable documentation and understandable tests.As a result, the existing tests don't do much to document the system andmake it difficult to deal with tests failures, especially whenrefactoring, as it is often difficult to tell what the intent offailing tests was.Here's what I'd really like to see soon. I'd like to see usclearly document our architecture and APIs through:1. A complete and accurate set of interfaces. I made a start at this a few months ago on the jim-dev branch, but didn't finish the effort due to uncertainty and other priorities.2. A collection of executable documentation (doctests) focused on how to use the APIs, including both static and dynamic behavior. The focus of these should be documentation, but the documentation needs to be in the form of doctests so we can be fairly sure that the documentation is and remains correct.I think this is an area where a lot of volunteers could makecontributions. Perhaps we could even schedule some ZODB "Doc days",similar bug days, but with a different emphasis.I won't insist that new work should wait for this effort, althoughI'd like to. :) Certainly, I've refrained from pursuing someideas of mine in large part because I think we need some foundation workfirst.Thoughts? Is anyone willing to help out? Anybody interested in aZODB Doc Day?Jim-- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered!CTO (540) 361-1714 http://www.python.orgZope Corporation http://www.zope.com http://www.zope.org___For more information about ZODB, see the ZODB Wiki:http://www.zope.org/Wikis/ZODB/ZODB-Dev mailing list - ZODB-Dev@zope.orghttp://mail.zope.org/mailman/listinfo/zodb-dev Russ Ferriday - Topia Systems - multilingual content management contact: [EMAIL PROTECTED] - (+44) (0)2076 1777588 - skype: ferriday a member of the evenios group___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] HOWTO recover from Data.fs.tmp
But for an app to be writing into thin air without complaining - THAT's a bug! Even having a backup of the Data.fs would not help! Not picking on zope, but if that's what's happening it has to be called a bug. --r On 2 Oct 2006, at 19:00, Benji York wrote: Juan Pablo Giménez wrote: El lun, 02-10-2006 a las 19:41 +0200, Dieter Maurer escribió: Juan Pablo Giménez wrote at 2006-9-29 18:23 -0300: >I have, > >Plone 2.1.2, >Zope (Zope 2.8.6-final, python 2.3.5, linux2), >Python 2.3.5 (#1, Apr 11 2006, 16:55:30) [GCC 3.3.5 (Debian >1:3.3.5-13)], >> >For some reason zope stop writing at Data.fs about a week ago. Now i >have a Data.fs with 1 week older data and a Data.fs.tmp with, I hope, >the new data... I fear you are out of luck... "Data.fs.tmp" usually only contains the last transaction (although, it may not free unused space at the end). but... that's a bug... why zope keep writing in .tmp file if we can't recover nothing from there?!?!... Why is it a bug? A .tmp file is just that, temporary. It's not a backup. -- Benji York Senior Software Engineer Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Commit Progress Indictor
What's wanted is a progress indicator.What's feasible is a 'sign of life', with no start or end.A spinner fits the bill better than a fake progress bar.--rOn 31 Aug 2006, at 16:08, Lennart Regebro wrote:realize you could have made two coffees or taken a small nap. ;)A progress bar like that is worse than no progress bar at all.Just my 2 centimes.This reminds me of a scene from Office Space : http://imdb.com/title/tt0151804/___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Re: micro patch to OFS
On 23 Jul 2006, at 17:46, Tres Seaver wrote: It looks fine to me. http://www.zope.org/Collectors/Zope/ as that will perpetuate the motivation and give us something to point at in the changelog. Done. Thanks Tres. http://www.zope.org/Collectors/Zope/2157 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] micro patch to OFS
Hello all!I've spent a frustrating hour tracing the repercussions of a module name change in a Product on Zope 2.9.3. The problem was visible only on uninstall.No need to go into all the details, but the tiny piece of additional information (in this case, the class name) given by this patch gave me just the clue I needed. To make things easier for the next guy, I offer it for inclusion, with no reservations.Would one of you Zope committers like to vet and apply it? In zope/lib/python/OFS:diff -u Uninstalled.py.orig Uninstalled.py--- Uninstalled.py.orig 2006-07-23 17:06:43.0 +0100+++ Uninstalled.py 2006-07-23 16:38:38.0 +0100@@ -38,7 +38,8 @@ raise SystemError, ( """This object was originally created by a product that is no longer installed. It cannot be updated.- """ )+ %s+ """ % repr(self) ) def __getattr__(self, name): if name[:3]=='_p_':--r ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] Undo differences between Z2 and Z3
And undo is prominently exposed in Plone, and its use is not discouraged.Do you think, Jim, that we should reconsider that?--rOn 17 May 2006, at 08:18, Chris Withers wrote:Hi Jim,Jim Fulton wrote: BTW, I strongly discourage use of Undo except in emergencies.Sadly, except when undoing the last (non-undo) transactions ina database can lead to inconsistency. What sort of inconsistencies are you referring to?Undo can be a very attractive feature although usually hamstrung by catalog tie-in with Zope 2. I was hoping to look at ways of getting back some of this functionality when it came to Zope 3...cheers,Chris-- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk___For more information about ZODB, see the ZODB Wiki:http://www.zope.org/Wikis/ZODB/ZODB-Dev mailing list - ZODB-Dev@zope.orghttp://mail.zope.org/mailman/listinfo/zodb-dev —Russ FerridayTopia Systemstel: (+44) (0) 2076 177758mobile: (+44) (0) 7789 338868web: http://www.topia.comskype: ferriday ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
