Re: [Zope-dev] Adding gzip compression to HTTPResponse.py
On Wed, 6 Feb 2002 11:22:23 -0500, "Brad Clements" <[EMAIL PROTECTED]> wrote: >> >Also, RESPONSE.setBody really should have access to REQUEST.headers. >> >What's the clean way to do that? Just pass the request object to response >> >object's init method? >> >> RESPONSE objects have a REQUEST attribute > >Are you sure? I know that request objects have a response object. But looking at >publish.py doesn't look like it goes the other way. D'Oh, you are right. A better recommendation is to look at make_request in ZServer/HTTPResponse. It probably does make sense for the request headers to be stored in the response object, and I suspect this would be the place to do it. >I am using Apache with mod_rewrite. Sure, it'd be great to compression there, but >Apache doesn't cache, you need squid for that, right? I know it can be done with apache, but I prefer squid. >I agree, Transfer Encoding is the way to go, but based on remarks at: > >http://www.iol.ie/~alank/python/httpcomp.html#encoding > >I stuck with the simpler to understand content-encoding. I think those remarks are misleading, and I dont find rfc2616 at all unclear in *this* area. Content-Encoding is a property of the entity being transferred. It is an end-to-end property (that is, from origin server to browser) and intermediate proxies or caches are not permitted to tamper with it. Transfer-Encoding is a property of the HTTP conversation (that is the conversation from browser to cache, from cache to another cache, from cache to front-end-proxy, and front-end-proxy to back-end-zope). If both ends of a conversation implement the same transfer-encoding (gzip, rsync, etc) then they could transparently apply the encoding as the document travels over the wire. My conclusions: 1. Adding Transfer-Encoding support to Zope is unnecessary if you are using a front-end proxy (there are many reasons why most people should, but some important exceptions where you need not). 2. Adding Content-Encoding support to Zope is a good thing only if your front-end proxy does not support on-the-fly Transfer-Encoding. I think apache does, but squid does not. Toby Dickenson [EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Adding gzip compression to HTTPResponse.py
On 6 Feb 2002 at 10:40, Toby Dickenson wrote: > > I think you also need to check the accept-encoding header, to allow > for clients that do not know how to gunzip. That also means you should set > caching headers to prevent the compressed and uncompressed responses > getting delivered to the wrong clients by a cache. Right, but I couldn't figure out how to see the request headers from response.setBody > > >Also, RESPONSE.setBody really should have access to REQUEST.headers. > >What's the clean way to do that? Just pass the request object to response > >object's init method? > > RESPONSE objects have a REQUEST attribute Are you sure? I know that request objects have a response object. But looking at publish.py doesn't look like it goes the other way. > At the time, this type of auto-compressing proxies looked like they > were just coming of age (http://rproxy.samba.org looked good at the > time too). Unfortunately nothing has changed since. Today I think only > Apache can do this (and has done for ages). Support in squid has stalled > (http://devel.squid-cache.org/projects.html#te). Although I still think > this is the way of the future, I suspect the short-term advantage of > content-encoding the way you implemented it may be an advanatge for longer > than I originally thought. I am using Apache with mod_rewrite. Sure, it'd be great to compression there, but Apache doesn't cache, you need squid for that, right? For non-xmlrpc responses I'd want the stuff cached. I agree, Transfer Encoding is the way to go, but based on remarks at: http://www.iol.ie/~alank/python/httpcomp.html#encoding I stuck with the simpler to understand content-encoding. -- Shouldn't downstream caching proxies ungzip a response if they get a connection from a client that doesn't support gzip? Or will they only do this if its Transfer-Encoded? Brad Clements,[EMAIL PROTECTED] (315)268-1000 http://www.murkworks.com (315)268-9812 Fax netmeeting: ils://ils.murkworks.com AOL-IM: BKClements ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Adding gzip compression to HTTPResponse.py
On 6 Feb 2002 at 10:02, seb bacon wrote: > I don't have much useful to add - I just wanted to mention that I know > there are people out there who have succesfully used mod_gzip with Zope; > and that I *like* the name dogzip :-) "That's my dog, zip!" Brad Clements,[EMAIL PROTECTED] (315)268-1000 http://www.murkworks.com (315)268-9812 Fax netmeeting: ils://ils.murkworks.com AOL-IM: BKClements ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Adding gzip compression to HTTPResponse.py
On Tue, 5 Feb 2002 17:34:26 -0500, "Brad Clements" <[EMAIL PROTECTED]> wrote: >I hacked the attached code into HTTPResponse, at the end of setBody. It works for >xml-rpc responses and I suppose any text output, so long as the response object has a >header named "dogzip" set. I think you also need to check the accept-encoding header, to allow for clients that do not know how to gunzip. That also means you should set caching headers to prevent the compressed and uncompressed responses getting delivered to the wrong clients by a cache. >Also, RESPONSE.setBody really should have access to REQUEST.headers. What's >the clean way to do that? Just pass the request object to response object's init >method? RESPONSE objects have a REQUEST attribute >Anyone think this is worthwhile? I looked at this roughly 18 months ago and came to the conclusion that (at the time) adding content-encoding support in Zope was the wrong way to do it. It you are using Zope behind a front-end proxy and you really should be. then it seems like a better idea to deliver the message to that proxy in an uncompressed form, and let it negotiate a transfer-encoding on its own. (Note that is transfer-, not content-encoding) The advantages of this scheme come from the fact that transfer-encoding is a hop-by-hop property. Two downstream caches can negotiate the best compression for that hop. Pushing everything downstream takes load away from zope, and making it a local choice means that the choice is often a better one. At the time, this type of auto-compressing proxies looked like they were just coming of age (http://rproxy.samba.org looked good at the time too). Unfortunately nothing has changed since. Today I think only Apache can do this (and has done for ages). Support in squid has stalled (http://devel.squid-cache.org/projects.html#te). Although I still think this is the way of the future, I suspect the short-term advantage of content-encoding the way you implemented it may be an advanatge for longer than I originally thought. I hope this helps, Toby Dickenson [EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Adding gzip compression to HTTPResponse.py
I don't have much useful to add - I just wanted to mention that I know there are people out there who have succesfully used mod_gzip with Zope; and that I *like* the name dogzip :-) seb On Tue, 2002-02-05 at 22:34, Brad Clements wrote: > I'm looking for architectural suggestions for adding gzip compression to > HTTPResponse for text types. > > First, I just wanted to compress xml-rpc output, since I'm returing lots of table >data as > XML text (not objects), then loading that text/xml into a DOM for XSLT processing. > > I hacked the attached code into HTTPResponse, at the end of setBody. It works for > xml-rpc responses and I suppose any text output, so long as the response object has >a > header named "dogzip" set. > > I know "dogzip" is a stupid name, but this is just a testing thing. > > Representative compressions: > > compress oldlen 150366 new len 11926 > compress oldlen 204382 new len 14170 > compress oldlen 12746 new len 1364 > > As you can see, very useful compressions for xml-rpc output. > > But for HTML output, what's really needed is I think a special kind of Cache Object. > One that combines HTTP Caching with Ram caching to keep gzip compressed objects > "in memory". > > Some HTML pages are really quite large, and gzip compression can make a noticable > difference. Just the javascript code sizes themselves are .. really big. > > For xml-rpc, obviously every response must be compressed if it's "worth it", and I >can > see that having to set a response property on a per request basis is appropriate for > xml-rpc. > > But for text file objects, Page Templates and stuff.. How does setBody work with Ram > Cache objects? I have some ideas... > > Anyone think this is worthwhile? > > Also, RESPONSE.setBody really should have access to REQUEST.headers. What's > the clean way to do that? Just pass the request object to response object's init > method? > > Here's quick gzip compression hack-in, based on code posted by Neil Schemenauer > > Thanks Neil. > > Added about line 265 in HTTPResponse.py in Zope 2.5 B3 > > try: > dogzip = self.headers['dogzip'] > del self.headers['dogzip'] > if dogzip and split(content_type,'/')[0] == 'text': > body = self.body > startlen = len(body) > import zlib, struct > _gzip_header = ("\037\213" # magic > "\010" # compression method > "\000" # flags > "\000\000\000\000" # time > "\002" > "\377") > co = zlib.compressobj(6,zlib.DEFLATED,-zlib.MAX_WBITS, > zlib.DEF_MEM_LEVEL,0) > chunks = [_gzip_header, co.compress(body), > co.flush(),struct.pack(" z = join(chunks,"") > newlen = len(z) > print "compress oldlen ",startlen,"new len",newlen > if newlen < startlen: > self.body = z > self.setHeader('content-length', newlen) > self.setHeader('content-encoding','gzip') > except: > pass > > > > Brad Clements,[EMAIL PROTECTED] (315)268-1000 > http://www.murkworks.com (315)268-9812 Fax > netmeeting: ils://ils.murkworks.com AOL-IM: BKClements > > > ___ > Zope-Dev maillist - [EMAIL PROTECTED] > http://lists.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Adding gzip compression to HTTPResponse.py
I have been working on a compression patch as a component of my PatchKit product for some time now, but I have been hitting a few problems, and had put it to one side while I worked on a couple of other products. If people still want to see something like this, I can bring the code out of moth-balls and start working on the problems. Adrian... -- The difficulty of tactical maneuvering consists in turning the devious into the direct, and misfortune into gain. - Sun Tzu - Original Message - From: "Brad Clements" <[EMAIL PROTECTED]> To: <[EMAIL PROTECTED]> Sent: Tuesday, February 05, 2002 10:34 PM Subject: [Zope-dev] Adding gzip compression to HTTPResponse.py > I'm looking for architectural suggestions for adding gzip compression to > HTTPResponse for text types. > > First, I just wanted to compress xml-rpc output, since I'm returing lots of table data as > XML text (not objects), then loading that text/xml into a DOM for XSLT processing. > > I hacked the attached code into HTTPResponse, at the end of setBody. It works for > xml-rpc responses and I suppose any text output, so long as the response object has a > header named "dogzip" set. > > I know "dogzip" is a stupid name, but this is just a testing thing. > > Representative compressions: > > compress oldlen 150366 new len 11926 > compress oldlen 204382 new len 14170 > compress oldlen 12746 new len 1364 > > As you can see, very useful compressions for xml-rpc output. > > But for HTML output, what's really needed is I think a special kind of Cache Object. > One that combines HTTP Caching with Ram caching to keep gzip compressed objects > "in memory". > > Some HTML pages are really quite large, and gzip compression can make a noticable > difference. Just the javascript code sizes themselves are .. really big. > > For xml-rpc, obviously every response must be compressed if it's "worth it", and I can > see that having to set a response property on a per request basis is appropriate for > xml-rpc. > > But for text file objects, Page Templates and stuff.. How does setBody work with Ram > Cache objects? I have some ideas... > > Anyone think this is worthwhile? > > Also, RESPONSE.setBody really should have access to REQUEST.headers. What's > the clean way to do that? Just pass the request object to response object's init > method? > > Here's quick gzip compression hack-in, based on code posted by Neil Schemenauer > > Thanks Neil. > > Added about line 265 in HTTPResponse.py in Zope 2.5 B3 > > try: > dogzip = self.headers['dogzip'] > del self.headers['dogzip'] > if dogzip and split(content_type,'/')[0] == 'text': > body = self.body > startlen = len(body) > import zlib, struct > _gzip_header = ("\037\213" # magic > "\010" # compression method > "\000" # flags > "\000\000\000\000" # time > "\002" > "\377") > co = zlib.compressobj(6,zlib.DEFLATED,-zlib.MAX_WBITS, > zlib.DEF_MEM_LEVEL,0) > chunks = [_gzip_header, co.compress(body), > co.flush(),struct.pack(" z = join(chunks,"") > newlen = len(z) > print "compress oldlen ",startlen,"new len",newlen > if newlen < startlen: > self.body = z > self.setHeader('content-length', newlen) > self.setHeader('content-encoding','gzip') > except: > pass > > > > Brad Clements,[EMAIL PROTECTED] (315)268-1000 > http://www.murkworks.com (315)268-9812 Fax > netmeeting: ils://ils.murkworks.com AOL-IM: BKClements > > > ___ > Zope-Dev maillist - [EMAIL PROTECTED] > http://lists.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope ) > ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Adding gzip compression to HTTPResponse.py
I'm looking for architectural suggestions for adding gzip compression to HTTPResponse for text types. First, I just wanted to compress xml-rpc output, since I'm returing lots of table data as XML text (not objects), then loading that text/xml into a DOM for XSLT processing. I hacked the attached code into HTTPResponse, at the end of setBody. It works for xml-rpc responses and I suppose any text output, so long as the response object has a header named "dogzip" set. I know "dogzip" is a stupid name, but this is just a testing thing. Representative compressions: compress oldlen 150366 new len 11926 compress oldlen 204382 new len 14170 compress oldlen 12746 new len 1364 As you can see, very useful compressions for xml-rpc output. But for HTML output, what's really needed is I think a special kind of Cache Object. One that combines HTTP Caching with Ram caching to keep gzip compressed objects "in memory". Some HTML pages are really quite large, and gzip compression can make a noticable difference. Just the javascript code sizes themselves are .. really big. For xml-rpc, obviously every response must be compressed if it's "worth it", and I can see that having to set a response property on a per request basis is appropriate for xml-rpc. But for text file objects, Page Templates and stuff.. How does setBody work with Ram Cache objects? I have some ideas... Anyone think this is worthwhile? Also, RESPONSE.setBody really should have access to REQUEST.headers. What's the clean way to do that? Just pass the request object to response object's init method? Here's quick gzip compression hack-in, based on code posted by Neil Schemenauer Thanks Neil. Added about line 265 in HTTPResponse.py in Zope 2.5 B3 try: dogzip = self.headers['dogzip'] del self.headers['dogzip'] if dogzip and split(content_type,'/')[0] == 'text': body = self.body startlen = len(body) import zlib, struct _gzip_header = ("\037\213" # magic "\010" # compression method "\000" # flags "\000\000\000\000" # time "\002" "\377") co = zlib.compressobj(6,zlib.DEFLATED,-zlib.MAX_WBITS, zlib.DEF_MEM_LEVEL,0) chunks = [_gzip_header, co.compress(body), co.flush(),struct.pack("http://www.murkworks.com (315)268-9812 Fax netmeeting: ils://ils.murkworks.com AOL-IM: BKClements ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )