Re: [Zope-dev] zopectl does not terminate
My apologies. I sent this to the wrong list. H ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] zopectl does not terminate
Hi all I run my script foo.zctl with zopectl run foo.ctl param1 param2. This script operates on a large ZODB and catches ConflictErrors accordingly. It iterates over a set, updates data and commits the transaction every 100 iterations. But I've noticed two things: 1. ConflictErrors are never fully caught. The show up in the console (this is acceptable I suppose), but my script stops executing on the conflict and does not continue. The zope process stays alive. 2. In the event of no conflict errors my script executes its last line (print 'done') but the process does not always terminate. If I instruct my script to not update the ZODB at all it terminates without problems. I'm running it on a live site with 7 ZEO clients. I've stopped a client (say client 2) so it is not accessed concurrently and run my script with client2/zopectl. It is in fact a Plone site but that should be irrelevant. Thanks for any help Hedley ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Next Step to Bug Resolution???
Hi Tim I'm more involved with Plone but can provide a slightly more digestible answer :) Chris mentioned unit tests. You do not have to write a new unit test. What is required is to have a look at the tests and then identify one that is relevant to your problematic method. This test should be very easy to read. The hard part is for you to expand this test to demonstrate the exception. This will typically require a few lines of code but since you already fixed a bug I assume it won't be hard to do. Then you have to run all the other tests to check that you haven't broken anything else by doing ./bin/instance test -s zope.something. Finally create a patch file and attach it to your original report on the tracker. Mmm, after writing all this I can see why it is a pain. I do however understand the need for unit tests so I don't think you are going to be able to get past that bit. Perhaps a kind Zope dev will pick up on this. Hedley ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
The usual Plone catalogs (portal_catalog, uid_catalog, reference_catalog and membrane_tool) all run above 90% hit rate if the server is up to it. portal_catalog is invalidated the most so it fluctuates the most. If the server is severely underpowered then catalogcache is much less effective. portal_catalog hit rates will degrade over time. This is the situation I'm currently facing with on one site, but more servers will fix that. It's quite easy to benchmark / load test with funkload. What I've found is that memcached is very light on CPU, but if the Zope processes are constantly using all CPU it is starved and runs into trouble. As long as you avoid that case (which would be fatal without catalogcache in any case) then everything works perfectly. Run a few tests and let me know please. Hedley ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
Have you measures the time needs for some standard ZCatalog queries used with a Plone site with the communication overhead with memcached? Generally spoken: I think the ZCatalog is in general fast. Queries using a fulltext index are known to be more expensive or if you have to deal with large resultsets or complex queries. No I haven't. Roche Compaan has done extensive benchmarking using funkload testing plain catalog vs module level cache vs memcached, but the tests are more about page serving than catalog query time. I'll ask him to comment more on that. As for standard queries on a Plone site the typical folder contents query is a good example. The query will be fast unless it sorts on sortable_title (a ZCTextIndex) right? Not sure right now. Since memcached is distributed only a single Zope client needs to perform that query and the result is available to all other Zope clients. And the cache is persistent as long as memcached runs, so you can merrily restart Zope instances and have a warm cache. I didn't even realise this until Roche pointed it out to me. To answer the question: I believe catalogcache will win every time since the return time of a cached query is not dependent on the complexity of the query. We should get a few benchmarks running at query level. I'll have a bit of time next week. Hedley ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
I'd love if this wouldn't be a monkey patch. So would I, but I couldn't find another way in this case. Also, there is nothing that makes this integrate correctly with transactions. Your cache will happily deliver never-committed data and also it will not isolate transactions from each other. I patched 4 methods - clear, search, catalogObject, uncatalogObject. Method clear is the simplest one - I simply flush the cache. Methods catalogObject and uncatalogObject both invalidate the cache. Should the transaction fail later the only drawback is that you threw a few things out of the cache. They'll soon be re-entered by subsequent searches. Method search just inspects queries and stores results to memcache. Can you give me an example where the cache would deliver non-committed data? ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
In addition, you need to include a serial in your cache keys to avoid dirty reads. The cache invalidation code actively removes items from the cache. Am I understanding you correctly? H ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
On Sat, Oct 25, 2008 at 2:57 PM, Andreas Jung [EMAIL PROTECTED] wrote: On 25.10.2008 14:53 Uhr, Hedley Roos wrote: I'd love if this wouldn't be a monkey patch. So would I, but I couldn't find another way in this case. Also, there is nothing that makes this integrate correctly with transactions. Your cache will happily deliver never-committed data and also it will not isolate transactions from each other. I patched 4 methods - clear, search, catalogObject, uncatalogObject. Method clear is the simplest one - I simply flush the cache. Methods catalogObject and uncatalogObject both invalidate the cache. Should the transaction fail later the only drawback is that you threw a few things out of the cache. They'll soon be re-entered by subsequent searches. Using a DataManager is likely the better and more safe choice. Andreas Thanks Andreas. I'll have a look at your code when available. Christian, I do have a mistake in my reasoning. If an object is added to the catalog in a transaction and I cache that object as result of a query in that same transaction, and then the transaction fails I'll have a bad cache. H ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
If you catalog an object, then search for it and then abort the transaction, your cache will have data in it that isn't committed. Kind of like how I came to the same conclusion in parallel to you and stuffed up this thread :) Additionally when another transaction is already running in parallel, it will see cache inserts from other transactions. So this is the area I have to focus on right now. H ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] ZCatalog caching with memcached
Additionally when another transaction is already running in parallel, it will see cache inserts from other transactions. A possible solution is to keep a module level cache which can be committed to the memcache on transaction boundaries. That way I'll incur no performance penalty. H ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] ZCatalog caching with memcached
Hi all The past few weeks I've been optimizing a busy Plone site and so collective.catalogcache was born. It uses memcached as a distributed ZCatalog cache. It is currently in production and seems to be holding just fine. The site went from being unusable to serving quite a bit of data. I'll modify it in future to not be too tied to just memcached, but that's still some way off. BTW, if you do not have memcached available the catalog functions as usual. It's only been tested for Zope 2.9.6 - 2.10.6. The repo is at http://dev.plone.org/collective/browser/collective.catalogcache. The URL may contain the word Plone but it's all Zope :) The readme contains more info and instructions. The product is a monkey patch to Catalog.py. I'd love some feedback and suggestions. Hedley ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
