Re: [Zope-CMF] CMF write performance as poor as Plone?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ross Patterson wrote: > Tres Seaver <[EMAIL PROTECTED]> writes: > >> Ross Patterson wrote: >>> Tres Seaver <[EMAIL PROTECTED]> writes: >>> Andreas Jung wrote: > On 23.11.2008 11:57 Uhr, Charlie Clark wrote: >> Am 23.11.2008 um 09:24 schrieb Andreas Jung: >> >>> This issue is independent of the client-side. ab2 and cmf/plone were >>> running on the same (fast) machine. >> Is this really content that is suited for the ZODB? > I am talking of the standard content-types that are available in CMF and > Plone like Document, News etc. Are you trying to proceess feed content, one document per request? You should be batching up that content, to get better amortization of the indexing overhead. You should also be checking for conflict errors: they would result in a *large* performance hit. >>> Quick question. How much of the write concurrency problem comes from >>> the fact that the ZPublisher retries the entire *request* on >>> WriteConflict instead of just trying to committing the transaction >>> again? >> It can't retry the commit: the point is that the conflict can't be >> resolved using the values read from the database at the start of the >> transaction. We have to redo the whole request, re-reading any objects >> in the context of a new transaction; otherwise, we would be back to >> "last write wins", which is a lose. > > Not really what I meant, sorry for being unclear. I meant how much of > the write performance problem comes from the overhead of redoing the > whole request, rendering templates and such? I'm just wondering if > anyone has quantified that. I doubt anyone has done so: the results would likely be highly dependent on the specific application. Back in the day, the render-time penalty was one of the reasons I disliked self-posting forms (for updates): I preferred the "update-and-redirect" model, both to avoid rendering during / after the update, and also to avoid screwing up the browser cache with POST requests. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJKbJZ+gerLs4ltQ4RAr3fAKCbL7zElLcQ+ay8J3vl3EvUEwX0wQCfaEPY y12fwCwskKLPZy/a0hVrbvU= =gBMJ -END PGP SIGNATURE- ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests
Re: [Zope-CMF] CMF write performance as poor as Plone?
Tres Seaver <[EMAIL PROTECTED]> writes: > Ross Patterson wrote: >> Tres Seaver <[EMAIL PROTECTED]> writes: >> >>> Andreas Jung wrote: On 23.11.2008 11:57 Uhr, Charlie Clark wrote: > Am 23.11.2008 um 09:24 schrieb Andreas Jung: > >> This issue is independent of the client-side. ab2 and cmf/plone were >> running on the same (fast) machine. > Is this really content that is suited for the ZODB? I am talking of the standard content-types that are available in CMF and Plone like Document, News etc. >>> Are you trying to proceess feed content, one document per request? You >>> should be batching up that content, to get better amortization of the >>> indexing overhead. >>> >>> You should also be checking for conflict errors: they would result in a >>> *large* performance hit. >> >> Quick question. How much of the write concurrency problem comes from >> the fact that the ZPublisher retries the entire *request* on >> WriteConflict instead of just trying to committing the transaction >> again? > > It can't retry the commit: the point is that the conflict can't be > resolved using the values read from the database at the start of the > transaction. We have to redo the whole request, re-reading any objects > in the context of a new transaction; otherwise, we would be back to > "last write wins", which is a lose. Not really what I meant, sorry for being unclear. I meant how much of the write performance problem comes from the overhead of redoing the whole request, rendering templates and such? I'm just wondering if anyone has quantified that. Ross ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests
Re: [Zope-CMF] CMF write performance as poor as Plone?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Ross Patterson wrote: > Tres Seaver <[EMAIL PROTECTED]> writes: > >> Andreas Jung wrote: >>> On 23.11.2008 11:57 Uhr, Charlie Clark wrote: Am 23.11.2008 um 09:24 schrieb Andreas Jung: > This issue is independent of the client-side. ab2 and cmf/plone were > running on the same (fast) machine. Is this really content that is suited for the ZODB? >>> >>> I am talking of the standard content-types that are available in CMF and >>> Plone like Document, News etc. >> Are you trying to proceess feed content, one document per request? You >> should be batching up that content, to get better amortization of the >> indexing overhead. >> >> You should also be checking for conflict errors: they would result in a >> *large* performance hit. > > Quick question. How much of the write concurrency problem comes from > the fact that the ZPublisher retries the entire *request* on > WriteConflict instead of just trying to committing the transaction > again? It can't retry the commit: the point is that the conflict can't be resolved using the values read from the database at the start of the transaction. We have to redo the whole request, re-reading any objects in the context of a new transaction; otherwise, we would be back to "last write wins", which is a lose. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJKaxt+gerLs4ltQ4RApTVAKDHRJBiEBOqedIh8bh27d7VJMMozQCfacFs jhNWzVm+L5XWHuK2XxNYlR4= =Xgr3 -END PGP SIGNATURE- ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests
Re: [Zope-CMF] CMF write performance as poor as Plone?
Tres Seaver <[EMAIL PROTECTED]> writes: > Andreas Jung wrote: >> On 23.11.2008 11:57 Uhr, Charlie Clark wrote: >>> Am 23.11.2008 um 09:24 schrieb Andreas Jung: >>> This issue is independent of the client-side. ab2 and cmf/plone were running on the same (fast) machine. >>> >>> Is this really content that is suited for the ZODB? >> >> >> I am talking of the standard content-types that are available in CMF and >> Plone like Document, News etc. > > Are you trying to proceess feed content, one document per request? You > should be batching up that content, to get better amortization of the > indexing overhead. > > You should also be checking for conflict errors: they would result in a > *large* performance hit. Quick question. How much of the write concurrency problem comes from the fact that the ZPublisher retries the entire *request* on WriteConflict instead of just trying to committing the transaction again? Ross ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests
Re: [Zope-CMF] CMF write performance as poor as Plone?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Andreas Jung wrote: > On 23.11.2008 11:57 Uhr, Charlie Clark wrote: >> Am 23.11.2008 um 09:24 schrieb Andreas Jung: >> >>> This issue is independent of the client-side. ab2 and cmf/plone were >>> running on the same (fast) machine. >> >> Is this really content that is suited for the ZODB? > > > I am talking of the standard content-types that are available in CMF and > Plone like Document, News etc. Are you trying to proceess feed content, one document per request? You should be batching up that content, to get better amortization of the indexing overhead. You should also be checking for conflict errors: they would result in a *large* performance hit. Tres. - -- === Tres Seaver +1 540-429-0999 [EMAIL PROTECTED] Palladion Software "Excellence by Design"http://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJKVZL+gerLs4ltQ4RAtfYAJ4y11dcMlEFLJYKxcRCvI+qoWM3zgCg0Sh2 0CvciY+vn6F+XLC8eedAirs= =bmyq -END PGP SIGNATURE- ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests
[Zope-CMF] CMF Tests: 6 OK
Summary of messages to the cmf-tests list. Period Sat Nov 22 12:00:00 2008 UTC to Sun Nov 23 12:00:00 2008 UTC. There were 6 messages: 6 from CMF Tests. Tests passed OK --- Subject: OK : CMF-2.1 Zope-2.10 Python-2.4.5 : Linux From: CMF Tests Date: Sat Nov 22 21:02:00 EST 2008 URL: http://mail.zope.org/pipermail/cmf-tests/2008-November/010429.html Subject: OK : CMF-2.1 Zope-2.11 Python-2.4.5 : Linux From: CMF Tests Date: Sat Nov 22 21:03:30 EST 2008 URL: http://mail.zope.org/pipermail/cmf-tests/2008-November/010430.html Subject: OK : CMF-trunk Zope-2.10 Python-2.4.5 : Linux From: CMF Tests Date: Sat Nov 22 21:05:00 EST 2008 URL: http://mail.zope.org/pipermail/cmf-tests/2008-November/010431.html Subject: OK : CMF-trunk Zope-2.11 Python-2.4.5 : Linux From: CMF Tests Date: Sat Nov 22 21:06:30 EST 2008 URL: http://mail.zope.org/pipermail/cmf-tests/2008-November/010432.html Subject: OK : CMF-trunk Zope-trunk Python-2.4.5 : Linux From: CMF Tests Date: Sat Nov 22 21:08:00 EST 2008 URL: http://mail.zope.org/pipermail/cmf-tests/2008-November/010433.html Subject: OK : CMF-trunk Zope-trunk Python-2.5.2 : Linux From: CMF Tests Date: Sat Nov 22 21:09:31 EST 2008 URL: http://mail.zope.org/pipermail/cmf-tests/2008-November/010434.html ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests
Re: [Zope-CMF] CMF write performance as poor as Plone?
On 23.11.2008 11:57 Uhr, Charlie Clark wrote: Am 23.11.2008 um 09:24 schrieb Andreas Jung: This issue is independent of the client-side. ab2 and cmf/plone were running on the same (fast) machine. Is this really content that is suited for the ZODB? I am talking of the standard content-types that are available in CMF and Plone like Document, News etc. I'm just thinking of an environment with lots of concurrent writes and content management doesn't spring directly to mind. Write performance specially in Plone in always a topic in large installations. Read performance is doable on large installations with caching etc. But we know of several Plone projects that failed at some point because you can't get a reasonable performance with lots of editors working the same time with the system. If that isn't possible then you will probably have to look at using an RDBMS and even then you might need server-side optimisations for performance? I just benchmarked that. I just wrote a a simple RDBMS-based Document implementation (using SQLAlchemy and table inheritance for sharing the dublin core table structure among different content-types). Against a Postgres 7.4 database I could reach a performance of roughly 350 new documents per second with a pure python implementation. I am now going to test the performance of the implementation with a CMF integration. Andreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. & Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:[EMAIL PROTECTED] title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests
Re: [Zope-CMF] CMF write performance as poor as Plone?
Am 23.11.2008 um 09:24 schrieb Andreas Jung: > This issue is independent of the client-side. ab2 and cmf/plone were > running on the same (fast) machine. Is this really content that is suited for the ZODB? I'm just thinking of an environment with lots of concurrent writes and content management doesn't spring directly to mind. Have you looked at performance with a standalone ZODB? If that isn't possible then you will probably have to look at using an RDBMS and even then you might need server-side optimisations for performance? Charlie -- Charlie Clark Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-938-5360 GSM: +49-178-782-6226 ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests
Re: [Zope-CMF] CMF write performance as poor as Plone?
Andreas Jung wrote at 2008-11-23 09:24 +0100: > ... >> Have you profiled an individual request to learn what the time is spent for? >> >> If the time is dominated by client side activity, you need client side >> skaling to increase the throughput significantly. > >> >> If, on the other hand, the time is heavily dominated by server side activity, >> you would need backend skaling (which is currently quite difficult >> to get) or get the backend significantly faster (I assume that >> the bandwidth is not the limiting factor). >> > >This issue is independent of the client-side. ab2 and cmf/plone were >running on the same (fast) machine. The other responses you have gotten indicate that client side activity (especially indexing) can significantly influence the observed "write performance" (going down by one order of magnitude). That means: it might be worth to check explicitely that this is not the case in your setup. If indexing is suppressed/delayed (as indicated in the other responses), the number of "store" operations also decrease. Each "store" operation may account for a few ms (I have observed a general ZEO overhead of about 3 ms -- however some years ago and therefore on an older hardware geneation; things may be faster nowadays). Looking at the number of your "store"s (and the time they take), you may check whether this may account for the pure write performance you observe. In this case, optimizations would be possible. In principle, the "store"s could be batched rather than transmitted individually (of course, this would require ZODB modifications). -- Dieter ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests
Re: [Zope-CMF] CMF write performance as poor as Plone?
On 23.11.2008 9:17 Uhr, Dieter Maurer wrote: Andreas Jung wrote at 2008-11-21 07:31 +0100: ... write performance ... However the transaction size does not seem to have any impact on the number of simulataneous writes. Have you profiled an individual request to learn what the time is spent for? If the time is dominated by client side activity, you need client side skaling to increase the throughput significantly. If, on the other hand, the time is heavily dominated by server side activity, you would need backend skaling (which is currently quite difficult to get) or get the backend significantly faster (I assume that the bandwidth is not the limiting factor). This issue is independent of the client-side. ab2 and cmf/plone were running on the same (fast) machine. Andreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. & Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:[EMAIL PROTECTED] title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests
Re: [Zope-CMF] CMF write performance as poor as Plone?
Andreas Jung wrote at 2008-11-21 07:31 +0100: > ... write performance ... >However the transaction size does not >seem to have any impact on the number of simulataneous writes. Have you profiled an individual request to learn what the time is spent for? If the time is dominated by client side activity, you need client side skaling to increase the throughput significantly. If, on the other hand, the time is heavily dominated by server side activity, you would need backend skaling (which is currently quite difficult to get) or get the backend significantly faster (I assume that the bandwidth is not the limiting factor). -- Dieter ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests
Re: [Zope-CMF] CMF write performance as poor as Plone?
On 22.11.2008 15:39 Uhr, Charlie Clark wrote: Hi Andreas, a very interesting situation. I've never thought of object creation when choosing "pure" CMF over Plone as this is largely a ZODB issue. As Roché points out it is less likely to be the transactions and more likely to be the cataloguing and any other event subscribers that are limiting factors here. Speed comparions between CMF and Plone only make sense for serving content where I find pure CMF to be at least 10 times as fast as Plone - I think the speed difference is largely down to the sheer size of Archetypes and the overloading of getattr(). The catalog is of course a hotspot. There is collective.indexing and the catalog queue that ease the pain a bit and bring some improvements to the overall performance (for both Plone and CMF) - however not as satisfying as I was thinking of. Going with a RDBMS as backend is likely the only option when it comes scalablity on a system with lots of concurrent write. Andreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. & Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:[EMAIL PROTECTED] title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests
Re: [Zope-CMF] [dev] Should portal_setup be registered as utility?
Charlie Clark wrote at 2008-11-20 20:33 +0100: > ... >Agreed. If third party tools have problems, then they should provide >the solutions. The Plone people are much more open to integrate third party solutions (a good thing in principle). But, they have only limited control over third party solutions. Therefore, they changed their CMF version to give third parties more time to adapt to a more Zope 3 style of doing things (e.g. accessing request only on code paths specifically designed for request processing). Note, that not everybody finds a more Zope 3 style of doing a worth in itself -- with corresponding reluctance to change something just for this purpose. -- Dieter ___ Zope-CMF maillist - Zope-CMF@lists.zope.org http://mail.zope.org/mailman/listinfo/zope-cmf See https://bugs.launchpad.net/zope-cmf/ for bug reports and feature requests
