[Zope-dev] z3 server+publication refactor for z2
Hi, Sidnei has been working on the Zope 2 publication-refactor branch where he's allowed the use of the Zope 3 Twisted-based server, and of a Zope 3 based publication process. I'd like to see this merge branched in Zope 2 trunk because I'd like Zope 2.10 to be Twisted-based. What's missing from the branch preventing this? What problems have been encountered? (This query is a reaction to my diving in to current asyncore+medusa +ThreadedAsync+PubCore that gives me nightmares when I realize I'll have to implement new server types or new stuff more akin to the ZEO storage server.) Thanks, Florent PS: what do people think of changing ZEO so that it integrates with Twisted properly instead of relying on a private event loop hack [please move to zodb-dev if you answer this] -- Florent Guillaume, Nuxeo (Paris, France) Director of RD +33 1 40 33 71 59 http://nuxeo.com [EMAIL PROTECTED] ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] z3 server+publication refactor for z2
--On 13. April 2006 11:46:20 +0200 Florent Guillaume [EMAIL PROTECTED] wrote: Hi, Sidnei has been working on the Zope 2 publication-refactor branch where he's allowed the use of the Zope 3 Twisted-based server, and of a Zope 3 based publication process. I'd like to see this merge branched in Zope 2 trunk because I'd like Zope 2.10 to be Twisted-based. What's missing from the branch preventing this? What problems have been encountered? The question is: how complete and stable is this stuff? Does it replace the current implementation or is it an optional feature as in Zope 3.2? If the implementation is half-backed then it will be a show-stopper, otherwise we need some confidence that it works as it should with breaking something. Andreas -- ZOPYX Ltd. Co. KG - Charlottenstr. 37/1 - 72070 Tübingen - Germany Web: www.zopyx.com - Email: [EMAIL PROTECTED] - Phone +49 - 7071 - 793376 E-Publishing, Python, Zope Plone development and consulting pgpJb3A5FRF8H.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] z3 server+publication refactor for z2
On 13.04.2006, at 11:55, Andreas Jung wrote: --On 13. April 2006 11:46:20 +0200 Florent Guillaume [EMAIL PROTECTED] wrote: Hi, Sidnei has been working on the Zope 2 publication-refactor branch where he's allowed the use of the Zope 3 Twisted-based server, and of a Zope 3 based publication process. I'd like to see this merge branched in Zope 2 trunk because I'd like Zope 2.10 to be Twisted-based. What's missing from the branch preventing this? What problems have been encountered? The question is: how complete and stable is this stuff? Does it replace the current implementation or is it an optional feature as in Zope 3.2? twisted is the standard server in zope 3.2 - zserver is optional If the implementation is half-backed then it will be a show- stopper, otherwise we need some confidence that it works as it should with breaking something. Andreas -- ZOPYX Ltd. Co. KG - Charlottenstr. 37/1 - 72070 Tübingen - Germany Web: www.zopyx.com - Email: [EMAIL PROTECTED] - Phone +49 - 7071 - 793376 E-Publishing, Python, Zope Plone development and consulting ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: z3 server+publication refactor for z2
On Thu, Apr 13, 2006 at 11:46:20AM +0200, Florent Guillaume wrote: | Hi, | | Sidnei has been working on the Zope 2 publication-refactor branch | where he's allowed the use of the Zope 3 Twisted-based server, and of | a Zope 3 based publication process. | | I'd like to see this merge branched in Zope 2 trunk because I'd like | Zope 2.10 to be Twisted-based. What's missing from the branch | preventing this? What problems have been encountered? Well, the biggest is making an adapter for the Zope 3 request so that it implements the same interface as the Zope 2 request. Other than that, it was pretty much working. Oh, and now I recall, we don't have 'streaming', 'chunked', and 'gzipping' yet, though the latter two would be easily implemented as wsgi middleware (could even use the implementation from django/turbogears). That's the second biggest :) | (This query is a reaction to my diving in to current asyncore+medusa | +ThreadedAsync+PubCore that gives me nightmares when I realize I'll | have to implement new server types or new stuff more akin to the ZEO | storage server.) It's not that bad :) | Thanks, | Florent | | PS: what do people think of changing ZEO so that it integrates with | Twisted properly instead of relying on a private event loop hack | [please move to zodb-dev if you answer this] -- Sidnei da Silva Enfold Systems, Inc. http://enfoldsystems.com ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: z3 server+publication refactor for z2
Sidnei da Silva wrote: On Thu, Apr 13, 2006 at 11:46:20AM +0200, Florent Guillaume wrote: | Hi, | | Sidnei has been working on the Zope 2 publication-refactor branch | where he's allowed the use of the Zope 3 Twisted-based server, and of | a Zope 3 based publication process. | | I'd like to see this merge branched in Zope 2 trunk because I'd like | Zope 2.10 to be Twisted-based. What's missing from the branch | preventing this? What problems have been encountered? Well, the biggest is making an adapter for the Zope 3 request so that it implements the same interface as the Zope 2 request. Other than that, it was pretty much working. We might not need a special request type. We could try to continue to use ZPublisher's request implementation, at least for now. In WSGI, the application gets the env and streams. In Zope 3, this is the WSGIPublisherApplication (see zope.app.wsgi). It is responsible for creating the request, but it chooses to delegate to an HTTP request factory. This factory decides, based on certian rules, whether we're dealing with a plain HTTP/WebDAV request or XML-RPC or browser etc. In Zope 2, this factory (or the WSGIPublisherApplciation itself) could simply create a ZPublisher request object. After creating the request, the WSGIPublisherApplication would turn the request over to ZPublisher.Publish.publish and sends it on its normal path. This is the small solution in which we provide a WSGI-capable frontend to the ZPublisher. The big solution would be to replace ZPublisher with zope.publisher and a custom Zope2-oriented publication + appropriate traversers. In this case I wouldn't advise for adapting the Zope 3 request to a Zope 2 request. I would rather introduce a new request type, IZope2Request, based on zope.publisher's IBrowserRequest, that provides all the additional Zope 2 API. I think the big solution would take a considerable effort. It's less than three weeks before feature freeze. That is very little time even for the small solution. Oh, and now I recall, we don't have 'streaming', 'chunked', We have something along those lines. Jim can say more. and 'gzipping' yet, though the latter two would be easily implemented as wsgi middleware (could even use the implementation from django/turbogears). Yeah. Good idea. Perhaps we can see if we can create a common wsgzip package or something that is open to all WSGI-capable Python frameworks. Philipp ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: z3 server+publication refactor for z2
--On 13. April 2006 15:39:06 +0200 Philipp von Weitershausen [EMAIL PROTECTED] wrote: This is the small solution in which we provide a WSGI-capable frontend to the ZPublisher. The big solution would be to replace ZPublisher with zope.publisher and a custom Zope2-oriented publication + appropriate traversers. In this case I wouldn't advise for adapting the Zope 3 request to a Zope 2 request. I would rather introduce a new request type, IZope2Request, based on zope.publisher's IBrowserRequest, that provides all the additional Zope 2 API. I think the big solution would take a considerable effort. It's less than three weeks before feature freeze. That is very little time even for the small solution. Big or small? Would this be an optional and configurable feature or replacement of the current infrastructure? Andreas -- ZOPYX Ltd. Co. KG - Charlottenstr. 37/1 - 72070 Tübingen - Germany Web: www.zopyx.com - Email: [EMAIL PROTECTED] - Phone +49 - 7071 - 793376 E-Publishing, Python, Zope Plone development, Consulting pgpe6PAKRkeFX.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: z3 server+publication refactor for z2
Philipp von Weitershausen wrote: I think the big solution would take a considerable effort. It's less than three weeks before feature freeze. That is very little time even for the small solution. Actually, I *think* I was wrong. The feature freeze will be June 1st, not May 1st. Perhaps the release manager can clear that up? Philipp ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: z3 server+publication refactor for z2
From: Andreas Jung [EMAIL PROTECTED] To: Philipp von Weitershausen [EMAIL PROTECTED], Sidnei da Silva [EMAIL PROTECTED], Andreas Jung [EMAIL PROTECTED], Florent Guillaume [EMAIL PROTECTED] cc: List Zope-dev zope-dev@zope.org Subject: Re: z3 server+publication refactor for z2 Date-Sent: 13. April 2006 15:56:17 --On 13. April 2006 15:53:33 +0200 Philipp von Weitershausen [EMAIL PROTECTED] wrote: Philipp von Weitershausen wrote: I think the big solution would take a considerable effort. It's less than three weeks before feature freeze. That is very little time even for the small solution. Actually, I *think* I was wrong. The feature freeze will be June 1st, not May 1st. Perhaps the release manager can clear that up? Bad question :-) I thought for this yr: 1.7 and 1.12 for the releases (freeze one month earlier) and starting next yr: 1.6 and 1.12...but can find find Jim's posting anymore. -aj pgpOy7lSAVqU4.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: z3 server+publication refactor for z2
--On 13. April 2006 15:58:44 +0200 Philipp von Weitershausen [EMAIL PROTECTED] wrote: I think the big solution would take a considerable effort. It's less than three weeks before feature freeze. That is very little time even for the small solution. Big or small? Would this be an optional and configurable feature or replacement of the current infrastructure? I think it'd be technically possible to have both solutions coexist. After all, that's what we're doing in Zope 3. zope.app.twisted and zope.app.server can coexist easily, I don't see why it shouldn't be possible in Zope 2. They must coexist in any case. We can not get rid or replace a major component - neither without breaking compatibility nor without deprecation. -aj -- ZOPYX Ltd. Co. KG - Charlottenstr. 37/1 - 72070 Tübingen - Germany Web: www.zopyx.com - Email: [EMAIL PROTECTED] - Phone +49 - 7071 - 793376 E-Publishing, Python, Zope Plone development, Consulting pgpuHLbQm0kve.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: z3 server+publication refactor for z2
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Florent Guillaume wrote: Hi, Sidnei has been working on the Zope 2 publication-refactor branch where he's allowed the use of the Zope 3 Twisted-based server, and of a Zope 3 based publication process. I'd like to see this merge branched in Zope 2 trunk because I'd like Zope 2.10 to be Twisted-based. What's missing from the branch preventing this? What problems have been encountered? - -1 to using Twisted by default in 2.10 -- it is still much slower than ZServer, AFAIK. I don't think we have time to land this and get it tested before 2.10, frankely, although I wouldn't mind it, assuming that one had to explicitly configure the server to use Twisted. (This query is a reaction to my diving in to current asyncore+medusa +ThreadedAsync+PubCore that gives me nightmares when I realize I'll have to implement new server types or new stuff more akin to the ZEO storage server.) Thanks, Florent PS: what do people think of changing ZEO so that it integrates with Twisted properly instead of relying on a private event loop hack [please move to zodb-dev if you answer this] Tres. - -- === Tres Seaver +1 202-558-7113 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFEPnB9+gerLs4ltQ4RAjgfAJ9nPGp87OdimICcrnnOkRUX+0ueRgCg1P5n yNyOReyQQhhXznRpgFYWgbI= =SbAn -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: z3 server+publication refactor for z2
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Bernd Dorn wrote: On 13.04.2006, at 11:55, Andreas Jung wrote: --On 13. April 2006 11:46:20 +0200 Florent Guillaume [EMAIL PROTECTED] wrote: Hi, Sidnei has been working on the Zope 2 publication-refactor branch where he's allowed the use of the Zope 3 Twisted-based server, and of a Zope 3 based publication process. I'd like to see this merge branched in Zope 2 trunk because I'd like Zope 2.10 to be Twisted-based. What's missing from the branch preventing this? What problems have been encountered? The question is: how complete and stable is this stuff? Does it replace the current implementation or is it an optional feature as in Zope 3.2? twisted is the standard server in zope 3.2 - zserver is optional That is really an accident: it is still in experimental status, and is known to have a *big* performance loss compared to ZServer. Tres. - -- === Tres Seaver +1 202-558-7113 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFEPnC6+gerLs4ltQ4RAscoAJ9XikAbVs8VnG607zEgN2gLrBXXowCgye40 4XOHEXBeGlTGvFicMxRXXpE= =rPsA -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: z3 server+publication refactor for z2
--On 13. April 2006 11:38:38 -0400 Tres Seaver [EMAIL PROTECTED] wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Florent Guillaume wrote: Hi, Sidnei has been working on the Zope 2 publication-refactor branch where he's allowed the use of the Zope 3 Twisted-based server, and of a Zope 3 based publication process. I'd like to see this merge branched in Zope 2 trunk because I'd like Zope 2.10 to be Twisted-based. What's missing from the branch preventing this? What problems have been encountered? - -1 to using Twisted by default in 2.10 -- it is still much slower than ZServer, AFAIK. I don't think we have time to land this and get it tested before 2.10, frankely, although I wouldn't mind it, assuming that one had to explicitly configure the server to use Twisted. Twisted as default really is not acceptable. If it is stable and does not inter with the rest of Zope we could include but it really depends on the stability and compatibility. Including it just for the sake having it in is not acceptable. -aj -- ZOPYX Ltd. Co. KG - Charlottenstr. 37/1 - 72070 Tübingen - Germany Web: www.zopyx.com - Email: [EMAIL PROTECTED] - Phone +49 - 7071 - 793376 E-Publishing, Python, Zope Plone development, Consulting pgp1wThnLyVUS.pgp Description: PGP signature ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] 64-bit BTrees
I have a need for 64-bit BTrees (at least for IOBTree and OIBTree), and I'm not the first. I've created a feature development branch for this, and checked in my initial implementation. I've modified the existing code to use PY_LONG_LONG instead of int for the key and/or value type; there's no longer a 32-bit version in the modified code. Any Python int or long that can fit in 64 bits is accepted; ValueError is raised for values that require 65 bits (or more). Keys and values that can be reported as Python ints are, and longs are only returned when the value cannot be converted to a Python int. This can have a substantial effect on memory consumption, since keys and/or values now take twice the space. There may be performance issues as well, but those have not been tested. There are new unit tests, but more are likely needed. If you're interested in getting the code from Subversion, it's available at: svn://svn.zope.org/repos/main/ZODB/branches/fdrake-64bits/ Ideally, this or some variation on this could be folded back into the main development for ZODB. If this is objectionable, making 64-bit btrees available would require introducing new versions of the btrees (possibly named LLBTree, LOBTree, and OLBTree). I welcome comments. -Fred -- Fred L. Drake, Jr.fdrake at gmail.com Don't let schooling interfere with your education. -- Mark Twain ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
[Zope-dev] Re: 64-bit BTrees
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Fred Drake wrote: I have a need for 64-bit BTrees (at least for IOBTree and OIBTree), and I'm not the first. I've created a feature development branch for this, and checked in my initial implementation. I've modified the existing code to use PY_LONG_LONG instead of int for the key and/or value type; there's no longer a 32-bit version in the modified code. Any Python int or long that can fit in 64 bits is accepted; ValueError is raised for values that require 65 bits (or more). Keys and values that can be reported as Python ints are, and longs are only returned when the value cannot be converted to a Python int. This can have a substantial effect on memory consumption, since keys and/or values now take twice the space. There may be performance issues as well, but those have not been tested. There are new unit tests, but more are likely needed. If you're interested in getting the code from Subversion, it's available at: svn://svn.zope.org/repos/main/ZODB/branches/fdrake-64bits/ Ideally, this or some variation on this could be folded back into the main development for ZODB. If this is objectionable, making 64-bit btrees available would require introducing new versions of the btrees (possibly named LLBTree, LOBTree, and OLBTree). I think coming up with new types is the only reasonable thing to do, given the prevalence of persistent BTrees out in the wild. Changing the runtime behavior (footprint, performance) of those objects is probably not something which most users are going to want, at least not without carefully considering the implications. Tres. - -- === Tres Seaver +1 202-558-7113 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFEPpyu+gerLs4ltQ4RAmh1AJ9/dLigNMrQgIFNASKWbpvboapywwCePV22 /3d8kFGTjipAVCsy5fnuLa4= =xe6v -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: 64-bit BTrees
Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Fred Drake wrote: I have a need for 64-bit BTrees (at least for IOBTree and OIBTree), and I'm not the first. I've created a feature development branch for this, and checked in my initial implementation. I've modified the existing code to use PY_LONG_LONG instead of int for the key and/or value type; there's no longer a 32-bit version in the modified code. Any Python int or long that can fit in 64 bits is accepted; ValueError is raised for values that require 65 bits (or more). Keys and values that can be reported as Python ints are, and longs are only returned when the value cannot be converted to a Python int. This can have a substantial effect on memory consumption, since keys and/or values now take twice the space. There may be performance issues as well, but those have not been tested. There are new unit tests, but more are likely needed. If you're interested in getting the code from Subversion, it's available at: svn://svn.zope.org/repos/main/ZODB/branches/fdrake-64bits/ Ideally, this or some variation on this could be folded back into the main development for ZODB. If this is objectionable, making 64-bit btrees available would require introducing new versions of the btrees (possibly named LLBTree, LOBTree, and OLBTree). I think coming up with new types is the only reasonable thing to do, given the prevalence of persistent BTrees out in the wild. Changing the runtime behavior (footprint, performance) of those objects is probably not something which most users are going to want, at least not without carefully considering the implications. It really depends on what the impact is. It would be nice to get a feel for whether this really impacts memory or performance for real applications. This adds 4-bytes per key or value. That isn't much, especially in a typical Zope application. Similarly, it's hard to say what the difference in C integer operations will be. I can easily imagine it being negligible (or being significant :). OTOH, adding a new type could be a huge PITA. We'd like to use these with existing catalog and index code, all of which uses IIBTrees. If the performance impacts are modest, I'd much rather declare IIBTrees to use 64-bit rather than 32-bit integers. I suppose an alternative would be to add a mechanism to configure IIBTrees to use either 32-bit or 64-bit integers at run-time. Jim -- Jim Fulton mailto:[EMAIL PROTECTED] Python Powered! CTO (540) 361-1714http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Re: 64-bit BTrees
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jim Fulton wrote: Tres Seaver wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Fred Drake wrote: I have a need for 64-bit BTrees (at least for IOBTree and OIBTree), and I'm not the first. I've created a feature development branch for this, and checked in my initial implementation. I've modified the existing code to use PY_LONG_LONG instead of int for the key and/or value type; there's no longer a 32-bit version in the modified code. Any Python int or long that can fit in 64 bits is accepted; ValueError is raised for values that require 65 bits (or more). Keys and values that can be reported as Python ints are, and longs are only returned when the value cannot be converted to a Python int. This can have a substantial effect on memory consumption, since keys and/or values now take twice the space. There may be performance issues as well, but those have not been tested. There are new unit tests, but more are likely needed. If you're interested in getting the code from Subversion, it's available at: svn://svn.zope.org/repos/main/ZODB/branches/fdrake-64bits/ Ideally, this or some variation on this could be folded back into the main development for ZODB. If this is objectionable, making 64-bit btrees available would require introducing new versions of the btrees (possibly named LLBTree, LOBTree, and OLBTree). I think coming up with new types is the only reasonable thing to do, given the prevalence of persistent BTrees out in the wild. Changing the runtime behavior (footprint, performance) of those objects is probably not something which most users are going to want, at least not without carefully considering the implications. It really depends on what the impact is. It would be nice to get a feel for whether this really impacts memory or performance for real applications. This adds 4-bytes per key or value. That isn't much, especially in a typical Zope application. Similarly, it's hard to say what the difference in C integer operations will be. I can easily imagine it being negligible (or being significant :). OTOH, adding a new type could be a huge PITA. We'd like to use these with existing catalog and index code, all of which uses IIBTrees. If the performance impacts are modest, I'd much rather declare IIBTrees to use 64-bit rather than 32-bit integers. I suppose an alternative would be to add a mechanism to configure IIBTrees to use either 32-bit or 64-bit integers at run-time. Who uses IOBTree / OIBTree / IIBTree? - Catalogs map RIDs to UIDs as IOBTrees (one record per indexed object) - Most indexes (those derived from Unindex) map RID to indexed value as an IOBTree (one record per object with a value meaningful to that index) and map values to RIDs as OOBTrees (where the second O is usually an IITreeSet). - ZCTextIndex uses IIBTrees to map word IDs to RIDs, in various ways, and make use of IOBTrees as wel.. - Relationship indexes (typically not stored within catalogs) usually have an IIBTree which is the mapping of the edges as pairs of internal node IDs (one per explicit relationship), with OIBTrees to map the user-supplied node value to a node ID. I would guess that if you could do a census of all the OIDs in all the Datas.fs in the world, a significant majority of them would be instances of classes declared in IOBTree / IIBTree (certainly the bulk of *transaction* records are going to be tied up with them). Tres. - -- === Tres Seaver +1 202-558-7113 [EMAIL PROTECTED] Palladion Software Excellence by Designhttp://palladion.com -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFEPtfe+gerLs4ltQ4RAkDoAJ9998Bj5yMqVpKoQOn/s3Hf5GZkBwCcC4uY kXTqmBsu6vMYx4fzAOWF5uo= =yVVq -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope3-dev] Re: [Zope-dev] Re: 64-bit BTrees
[Tres Seaver] ... I would guess that if you could do a census of all the OIDs in all the Datas.fs in the world, a significant majority of them would be instances of classes declared in IOBTree / IIBTree (certainly the bulk of *transaction* records are going to be tied up with them). Provided it still works, people can use ZODB's analyze.py to figure that out. But supposing I flavors of BTrees are the only objects that exist, what follows from that? It's not clear. I can guarantee that multiunion() will run slower, because its workhorse radix sort will need 8 (instead of 4) passes. Beyond that, it requires someone to try it. I'm reminded that when the MEMS Exchange wrote Durus (a kind of ZODB lite ;-): http://www.mems-exchange.org/software/durus/ ) they left their entire BTree implementation coded in Python -- it was fast enough that way. The difference between ZODB's BTree C code pushing 4 or 8 bytes around at a time may well be insignificant overall. If done carefully, pickle sizes probably won't change: cPickle has a large number of ways to pickle integers, and picks the smallest one needed to hold an integer's actual value. Provided the internal getstate() functions are careful to avoid Python longs when possible, bigger pickles won't happen unless more than 32 bits are actually needed to hold an integer. There's also that ZODB's current I trees are badly broken on 64-bit boxes (except for Win64) in at least this way: http://collector.zope.org/Zope/1592 That problem would go away by magic. looks-like-a-case-of-measure-twice-cut-once-ly y'rs - tim ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope] getting error in export and import on zope2.9
--On 13. April 2006 02:46:44 -0700 shivayogi kumbar [EMAIL PROTECTED] wrote: I am trying to import file from zope2.9 that has been exported in xml format from the same.I am getting error as follows. Site Error An error was encountered while publishing this resource. Error Type: UnicodeDecodeError Error Value: 'ascii' codec can't decode byte 0x83 in position 10: ordinal not in range(128) -- You asked the same question an hour ago on the Five list and you got the answer that the XML export is buggy and the hint to use the .zexp format. -aj -- ZOPYX Ltd. Co. KG - Charlottenstr. 37/1 - 72070 Tübingen - Germany Web: www.zopyx.com - Email: [EMAIL PROTECTED] - Phone +49 - 7071 - 793376 E-Publishing, Python, Zope Plone development and consulting pgpOu461iYdh1.pgp Description: PGP signature ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] creating users in zope. Login error
JulianRead wrote: Hi i have created a plone site and now i want to create a user using zope and give them ownership rights to plone. Sounds like a Plohn problem. Go ask on their lists. Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] TAL page whitespace removal
Robert (Jamie) Munro wrote: gzip will add enormous processing overhead to the server. Striping spaces will add negligible overhead, likely less overhead than it saves. I hope you've got a full set of tests that prove these sweeping statements you're making ;-) I have written TAL that produces very large dumps of XML data in the past, even whole sites. It's a really nice way to dump data from a database (SQL or Zope DB), but Zope has to build the whole output in RAM before sending any of it, so it can cause the site to crash. Then write your code better. While it's easiest to do this all in memory, it doesn't scale, as you're explaining... I would hope in this kind of case that the TAL is the major user of RAM on the site, Actually, unless you're careful, you're likely to be dragging all the zope objects into memory too, and that'll be what's really killing it ;-) _p_deactivate and zodb object cache minimisation are your friends ;-) so any saving would be really good, but it all cases (except pre tags, which I would never use) it seems like a possibly significant gain. I'm afraid I think you're mistaken. Speed-wise, I'll bet the security architecture will cost you much more than gzip _and_ space stripping combined ;-) For memory, doing the whole thing in one go simply won't scale, so you'll have to re-think... cheers, Chris -- Simplistix - Content Management, Zope Python Consulting - http://www.simplistix.co.uk ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] getting error in export and import on zope2.9
--On 13. April 2006 11:49:38 +0200 Andreas Jung [EMAIL PROTECTED] wrote: --On 13. April 2006 02:46:44 -0700 shivayogi kumbar [EMAIL PROTECTED] wrote: I am trying to import file from zope2.9 that has been exported in xml format from the same.I am getting error as follows. Site Error An error was encountered while publishing this resource. Error Type: UnicodeDecodeError Error Value: 'ascii' codec can't decode byte 0x83 in position 10: ordinal not in range(128) -- Also you did not tell us about the exact Zope 2.9 version. According to the logs there are some changes in 2.9.2 concerning XML export. -aj -- ZOPYX Ltd. Co. KG - Charlottenstr. 37/1 - 72070 Tübingen - Germany Web: www.zopyx.com - Email: [EMAIL PROTECTED] - Phone +49 - 7071 - 793376 E-Publishing, Python, Zope Plone development and consulting pgpmM6bvm4HlV.pgp Description: PGP signature ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] TAL page whitespace removal
On Wed, Apr 12, 2006 at 01:56:58PM -0500, Floyd May wrote: One solution I've found is to buffer the writes to REQUEST.RESPONSE by using a python script which the calls granular page templates rather than a single monolithic template, and outputting the results 25k at a time or so; it gives the rest of the server some time to catch up. Note that this doesn't buy you any improved responsiveness if you're running behind e.g. apache, because apache has to read the entire response from Zope before it starts sending it back to the client. -- Paul Winkler http://www.slinkp.com ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] TAL page whitespace removal
Paul Winkler wrote: On Wed, Apr 12, 2006 at 01:56:58PM -0500, Floyd May wrote: One solution I've found is to buffer the writes to REQUEST.RESPONSE by using a python script which the calls granular page templates rather than a single monolithic template, and outputting the results 25k at a time or so; it gives the rest of the server some time to catch up. Note that this doesn't buy you any improved responsiveness if you're running behind e.g. apache, because apache has to read the entire response from Zope before it starts sending it back to the client. Wasn't aware of that, but I've tested it from behind Squid, and it works like a charm. -- Floyd May Senior Systems Analyst CTLN - CareerTech Learning Network [EMAIL PROTECTED] ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] TAL page whitespace removal
On Thu, Apr 13, 2006 at 09:04:38AM -0500, Floyd May wrote: Paul Winkler wrote: On Wed, Apr 12, 2006 at 01:56:58PM -0500, Floyd May wrote: One solution I've found is to buffer the writes to REQUEST.RESPONSE by using a python script which the calls granular page templates rather than a single monolithic template, and outputting the results 25k at a time or so; it gives the rest of the server some time to catch up. Note that this doesn't buy you any improved responsiveness if you're running behind e.g. apache, because apache has to read the entire response from Zope before it starts sending it back to the client. Wasn't aware of that, but I've tested it from behind Squid, and it works like a charm. Actually I really should qualify that; it depends what you're trying to do. The only problem I have with streaming behind mod_proxy / mod_rewrite is that it does some buffering, and AFAIK there's no way to turn that off on a per-request basis. Even on a global basis it looks like the ProxyReceiveBufferSize can't be set to less than 512 bytes. (Which would probably make performance suck for everything else anyway.) So if you're trying to do some quick-and-dirty pre-AJAX-style status information, where you're streaming small bits of text to the browser, as I did in the ZSyncer UI, then you're out of luck. It's trivial to verify this with a particular reverse proxy setup by visiting a script something like: # Assuming you've made time importable... import time response = context.REQUEST.RESPONSE msgs = 'br/hello\n' * 20 response.setHeader('content-type', 'text/html') response.setHeader('content-length', str(len(msgs))) for line in msgs.split(): response.write(line + '\n') time.sleep(0.5) If I view this directly at the zope server, I see each hello appear after a short delay. If I view it via apache with mod_rewrite, I see nothing for 10 seconds, then the whole page at once. (Note you can simply leave out the content-length header to get response.write() to use http 1.1-style chunking, which is convenient if you can't pre-calculate an accurate size. This has the same buffering issue behind Apache, and additionally requires the client to be using http 1.1.) OTOH, if you're streaming large blobs in chunks of e.g. 64kb, streaming through apache seems to work just fine. This is probably a more common case. -- Paul Winkler http://www.slinkp.com ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] TAL page whitespace removal
One more correction for the archives: On Thu, Apr 13, 2006 at 11:45:41AM -0400, Paul Winkler wrote: On Thu, Apr 13, 2006 at 09:04:38AM -0500, Floyd May wrote: Paul Winkler wrote: Note that this doesn't buy you any improved responsiveness if you're running behind e.g. apache, because apache has to read the entire response from Zope before it starts sending it back to the client. That sentence is flat wrong. It only looks that way if your data size is smaller than apache's buffer size. -- Paul Winkler http://www.slinkp.com ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Generic SQL insert
Whenever I'm using SQL databases in zope, I always seem to have to make a ZSQL instance for inserting into every table in my database, and they are all nearly the same - they just have a list of all the fields in the database in the parameters, then they say: insert into [table] ([list of fields]) values ([list of dtml-sqlvars]) I'd much rather have a dictionary of fields and values, and just throw it at the DB, not having to make those queries for every table. I have acheived it like so: mydict = {field1:value1 , field2:value2 ,...} (fields,values)=zip(*myDict.items()) context.genericInsert(table='table name',fields=fields,values=values) Where generic insert is the following ZSQL method: insert into dtml-var table (dtml-in expr=fieldsdtml-var sequence-itemdtml-if sequence-enddtml-else,/dtml-if/dtml-in) values (dtml-in expr=valuesdtml-sqlvar sequence-item type=stringdtml-if sequence-enddtml-else,/dtml-if/dtml-in); with parameters: * table - table name * fields - list of fieldnames * values - list of values in the same order What do other people think of this? Is it a really bad idea? Robert Munro ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] creating users in zope. Login error
On 4/12/06, JulianRead [EMAIL PROTECTED] wrote: Hi i have created a plone site and now i want to create a user using zope and give them ownership rights to plone. At the moment i have created a user using the acl users button on the left menu bar and given them ownership rights Which acl_users folder? The site root's, or the Plone site's? Once i have created the user i am unable to log into zope using them. You mean log into the Zope management screens (ZMI)? I believe that giving your new user the Owner role is not enough to allow them to login to the ZMI of the site root. Manager role would achieve that. Does anyone know how to giv my newly created user the abillity to login. What are you trying to achieve with this user? Logging in as an application manager with full access to the full Zope instance, or with management rights only to the Plone site, or as a Plone user/member, or what? Regards, Rob ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Generic SQL insert
Great idea. Not to be recommended in general. This works because every field is textual, and you are sql-quoting by using type=string. Here are the problems: 1) if someone reads this and does not use the type=string tag, or equivalent, they will be wide open to sql injection. 2) OR, they can pass a list of type with each variable. 3) If you have to handle casts, then you will have to pass a list of cast-types, as well. So, you have essentially moved the problem from making at least one insertion call per table to a single insertion method that requires the creation of two, three, or four lists. This does not self-evidently require less work. You can no longer inspect the method to see if it is correct. You have to look to each call-point to determine what is actually being used. Just as bad, your application goes happily on its way if you are missing (non-key) variables. Keep zsql methods a simple as possible. Use as few tricks as possible. Your goal is self-evident correctness, not the minimization of typing. jim penny [EMAIL PROTECTED] wrote on 04/13/2006 02:23:22 PM: Whenever I'm using SQL databases in zope, I always seem to have to make a ZSQL instance for inserting into every table in my database, and they are all nearly the same - they just have a list of all the fields in the database in the parameters, then they say: insert into [table] ([list of fields]) values ([list of dtml-sqlvars]) I'd much rather have a dictionary of fields and values, and just throw it at the DB, not having to make those queries for every table. I have acheived it like so: mydict = {field1:value1 , field2:value2 ,...} (fields,values)=zip(*myDict.items()) context.genericInsert(table='table name',fields=fields,values=values) Where generic insert is the following ZSQL method: insert into dtml-var table (dtml-in expr=fieldsdtml-var sequence-itemdtml-if sequence-enddtml-else,/dtml-if/dtml-in) values (dtml-in expr=valuesdtml-sqlvar sequence-item type=stringdtml-if sequence-enddtml-else,/dtml-if/dtml-in); with parameters: * table - table name * fields - list of fieldnames * values - list of values in the same order What do other people think of this? Is it a really bad idea? Robert Munro ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
RE: [Zope] ExtFile and TextIndexNG issues
Stefan, Thanks very very much for your help. I just installed ExtFile 1.5 updated with some extra properties I had added to my ExtFile 1.4.4. It works nicely. All of my ExtFile objects are getting catalogged along with some custom content types I created. I just added a txng_get method to those. Very easy. Looks like calls to index_object and reindex_object work fine too. I was getting nervous that I wouldn't be able to do full-text indexing on pdfs, word, etc. but now it looks like everything is all good. Thanks, Tom Palermo -Original Message- From: Stefan H. Holek [mailto:[EMAIL PROTECTED] Sent: Wednesday, April 12, 2006 4:06 AM To: Palermo, Tom Cc: 'zope@zope.org' Subject: Re: [Zope] ExtFile and TextIndexNG issues This however is *exactly* the problem. Please use 1.5. ExtFile 1.4.4's index_html writes to the request regardless of how it was called (by publisher or script). Stefan On 11. Apr 2006, at 19:24, Palermo, Tom wrote: I haven't tried upgrading to ExtFile 1.5 from the svn trunk yet but I really don't think that's the problem. I'm going to try that next (upgrading to ExtFile 1.5). Does anyone have any ideas about what would cause the ExtFile to launch instead of just get catalogged? -- Anything that happens, happens. --Douglas Adams ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: zope 2.8.6 on Mac Intel
manuel spuhler wrote at 2006-4-12 21:40 +0200: ... File /opt/Zope2.8/lib/python/ZODB/__init__.py, line 21, in ? from persistent import TimeStamp File /opt/Zope2.8/lib/python/persistent/__init__.py, line 19, in ? from cPersistence import Persistent, GHOST, UPTODATE, CHANGED, STICKY ImportError: Inappropriate file type for dynamic loading Something is wrong with your cPersistent.so file. It should be a dynamically loaded shared object but somehow it is not. I cannot tell you why... -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Strategies for testing generated sql?
Paul Winkler wrote at 2006-4-12 12:32 -0400: ... Functionally, we are missing some important testables: ... * we have no way to verify that these queries behave as expected against a sample data set. If you want to test this, then do it. It will also cover all other test requirements (e.g. that the SQL is well formed). Of course, you will need an RDBMS to perform the tests against. How do people test this sort of thing? Do you go whole-hog and fire up MySQL or whatever? Or use gadfly? sqlite? or what? We do something like this for Postgres. The test setup assumes that a Postgres installation is running and available for the test. The test is executed against a special test database, copied on test start from a template. After the test, the test database is deleted again. Drawbacks: * there can not be more than a single test running in parallel. Parallel tests would cause Postgres to complain about already existing test databases * deleting the test database was a nightmare. Ever and ever again, Postgres complained about a deletion trial while there were still some uses of the database: sometimes, some transactions have not been committed/aborted; sometimes Postgres connections were still open; sometimes, Postgres did not behave synchronously (the connection was closed, but some part of Postgres was not yet informed about the fact). -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] newbie needs advice on where to use zope
Sells, Fred wrote at 2006-4-12 16:32 -0400: ... So after all that rambling, Is Zope a viable tool or am I killing a gnat with a cannon? After your somewhat vague description, I would say that your task is not a standard Zope use case. Zope is Web application server. Your task does not seem to be a web application. If you want to manage your contracts via a Web interface, then Zope may help you with this part of your task. Zope by itself will not help you to generate layout faithful PDF. There are tools around, like reportlab, for such tasks that can be used together with Zope. You can also automate OpenOffice from Zope (but see in [EMAIL PROTECTED] my notes of warning concerning this use case). -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: zope 2.8.6 on Mac Intel
Dieter Maurer wrote: manuel spuhler wrote at 2006-4-12 21:40 +0200: ... File "/opt/Zope2.8/lib/python/ZODB/__init__.py", line 21, in ? from persistent import TimeStamp File "/opt/Zope2.8/lib/python/persistent/__init__.py", line 19, in ? from cPersistence import Persistent, GHOST, UPTODATE, CHANGED, STICKY ImportError: Inappropriate file type for dynamic loading Something is wrong with your "cPersistent.so" file. It should be a dynamically loaded shared object but somehow it is not. I cannot tell you why... Manuel, Last time I saw cPersistent.so mentioned as a problem it had to do with conflicting python versions. No idea either, otherwise. Did you mention your zope version *and* the python versions you are using? (check your control panel) Didnt see the start of this thread. David ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )