Re: reverse proxy in depth tutorial?
On Thu, 24 Jul 2003, Garrett Goebel wrote: Most everything I'm stumbled upon has been short on detail and examples. For instance, I never found an example of how to just reverse proxy everything for a given backend server. All the examples I saw showed how to proxy something like http://foo/bar but not http://foo. Eventually I came up with the following: [Reverse Proxy] ... Listen 192.168.1.1:80 RewriteEngine on RewriteRule ^/?(.*) http://127.0.0.1:80/$1 [P] ProxyPassReverse / http://127.0.0.1:80/ ... [Backend Server] ... Listen 127.0.0.1:80 ... Is this kosher? Is there a better way to do this? A way to do it without mod_rewrite using only mod_proxy directives? It can be done easy without mod_rewrite: ProxyPass / http://127.0.0.1:80/ ProxyPassReverse / http://127.0.0.1:80/ Are there any strong arguments against using a mod_proxy/mod_rewrite httpd accelerator? Or preferred alternatives? I can not say about Apache2's mod_proxy (I do not use Apache2) but Apache 1.3's one has several drawbacks when it serves the big responses for the slow clients. It reads the backend response and synchronously sends it to client. The maximum size of the data that can be read from backend without delay is ProxyIOBufferSize + frontend kernel TCP send buffer + frontend kernel TCP recieve buffer + backend kernel TCP send buffer. Kernel TCP buffers are usually 16K-64K but can be made bigger. To eliminate many mod_proxy drawbacks I had written more than two years ago mod_accel module to do the reverse-proxing. mod_accel allows to configure read buffer in memory but if the response is bigger then it saves it to the temporary file and frees the backend as soon as possible. mod_accel has directives to configure the caching backend responses, to limit the connections to some backend to avoid the starvation of other hosts and many other features. mod_accel is clean module and is used for two years on several Russian loaded sites. One of them serves 100 requests per seconds without any segfaults. The main mod_accel drawback for non-Russian users is that there's no complete English documentation. Here is very incomplete documentaion http://dapi.chaz.ru/articles/plain/en/mod_accel.xml Also there're some English links on http://sysoev.ru/en/ Using the loopback for the backend has the security advantage of completely isolating the backend from any direct communication with external hosts. How do I keep the backend on 127.0.0.1 _and_ handle name based virtual hosts? Using mod_proxy you can set up backend listening on several loopback addresses: 127.0.0.1, 127.0.0.2, etc. Of course, you need to configure these additional addresses in OS before the use. Also backend can listen on several ports and the single address: 127.0.0.1:8000, 127.0.0.1:8001, etc. Here is example for two virtual hosts on several addresses, note that 1) the frontend servers are name based while the backend ones are IP based; 2) ServerName of the virtual server pairs are the same. [Reverse Proxy] NameVirtualHost frontend VirtualHost frontend ServerName name1 ProxyPass / http://127.0.0.1:80/ ProxyPassReverse / http://127.0.0.1:80/ ... /VirtualHost VirtualHost frontend ServerName name2 ProxyPass / http://127.0.0.2:80/ ProxyPassReverse / http://127.0.0.2:80/ ... /VirtualHost [Backend Server] UseCanonicalName on VirtualHost 127.0.0.1 ServerName name1 ... /VirtualHost VirtualHost 127.0.0.2 ServerName name2 ... /VirtualHost Using mod_accel you can use the same scheme and also there's another way: [PH] flag of the AccelPass directive preserves Host header. [Reverse Proxy] AccelPass / http://127.0.0.1/ [PH] [Backend Server] UseCanonicalName on NameVirtualHost 127.0.0.1 VirtualHost 127.0.0.1 ServerName name1 ... /VirtualHost VirtualHost 127.0.0.1 ServerName name2 ... /VirtualHost What are the issues with regard to virtual hosting and ssl? All your SSL-enabled sites have to be IP-based. Any tips on keeping the config files maintainable? For instance if I'm doing a mason site with co-branding through multiple component roots... What would minimal configurations for proxy and backend servers look like in order to redirect an externally exposed ip address on the proxy to a backend on 127.0.0.1 and still preserve the name based virtual hosts? It that possible? Yes it's possible, see previous examples. What are the variations, tradeoffs, and issues scaling it out to multiple physical servers? etc. mod_accel supports the primitive load balancing and fault tolerance using DNS. mod_proxy in Apache 1.3 also has code that allows to connect to the next server returned by DNS but this code is broken - it uses the same socket to connect to the next backend and this always fails. Igor Sysoev http://sysoev.ru/en/
Re: Content compression FAQ
On Mon, 21 Jul 2003, Slava Bizyayev wrote: I will patch FAQ with this shortly. Well but what does HTTP/1.1 support mean ? As far as I know the compressing has not any HTTP/1.1 specific features. mod_deflate compresses content using 8K buffer. When the buffer is filled mod_deflate passes the compressed data to Apache. Sorry, I had mixed up with gzipping module of my upcoming lightweight http and reverse-proxy server. mod_deflate does not wait to fill up a whole 8K buffer, it passes the compressed data to Apache as soon as it is available. And also it supports flushing the compressed data as described below. If an upstream module calls ap_bflush() to flush a pending data then mod_deflate asks zlib to flush a compressed stream and then passes a partially filled buffer to Apache. Since flushing a compressed stream descreases a compression ratio so mod_deflate has DeflateIgnoreFlush to ignore the ap_bflush() calls of an upstream module. It can be used for example with Chili!Soft ASP that calls ap_bflush() after any output even after an output of several bytes. Igor Sysoev http://sysoev.ru/en/
Re: Content compression FAQ
On Sat, 19 Jul 2003, Slava Bizyayev wrote: Here is small correction to Web Content Compression FAQ. What does HTTP/1.0 only support mean in this answer: Q: Are there any content compression solutions for vanilla Apache 1.3.X? A: Yes, There are two compression modules written in C that are available for vanilla Apache 1.3.X: * mod_deflate an Apache handler written in C by Igor Sysoev (Russia). * mod_gzip an Apache handler written in C. Original author: Kevin Kiley, Remote Communications, Inc. (U.S.) Both of these modules support HTTP/1.0 only. mod_deflate does not care about chunks at all - Apache does it. If Apache's responses is HTTP/1.1 then Apache send it in chunks because mod_deflate always deletes Content-Length header. mod_deflate compresses content using 8K buffer. When the buffer is filled mod_deflate passes the compressed data to Apache. If an upstream module calls ap_bflush() to flush a pending data then mod_deflate asks zlib to flush a compressed stream and then passes a partially filled buffer to Apache. Since flushing a compressed stream descreases a compression ratio so mod_deflate has DeflateIgnoreFlush to ignore the ap_bflush() calls of an upstream module. It can be used for example with Chili!Soft ASP that calls ap_bflush() after any output even after an output of several bytes. So mod_deflate as well as Apache::Dynagzip are the only handlers to date that begins transmission of compressed data as soon as the initial uncompressed pieces of data arrive from their source, at a time when the source process may not even have completed generating the full document it is sending. Transmission can therefore be taking place concurrent with creation of later document content. Igor Sysoev http://sysoev.ru/en/
Re: Load balancers
On Mon, 13 Jan 2003, John Siracusa wrote: Thanks for all the info, and please feel free to send me more, especially if there's some gem of a software load balancer out there somewhere... :) mod_accel can do primitive load balancing (via DNS) and primitive fault tolerance. It connect to next backend 1) if connect() to current backend failed; 2) if it can not get HTTP header from backend (timeout or error); 3) if it received 5XX response from backend (it can be disabled). Nevertheless it does not remember any faults to future use so I called it primitive. You can find it at http://sysoev.ru/en/ If you use DNS-based load balancing then you should install local caching DNS server on the same computer to speed up DNS queries. Igor Sysoev http://sysoev.ru/en/
Re: Apache 2?
On Tue, 26 Nov 2002, Philip Mak wrote: On Tue, Nov 26, 2002 at 11:40:00AM -0800, Grant Cooper wrote: What do yo mean a modem will tie up the Server? I've never heard this before. Let's say you have a mod_perl page that returns a 100k document, and a 28.8k modem downloads that document. The mod_perl process that is serving that document will be tied up until that modem finishes downloading the document, which is inefficient since the mod_perl processes take up a lot of memory. A lightweight front-end proxy that loads the data from the mod_perl process all at once and then feeds it to the modem would save memory. Things are bit complicated. Kernel can buffer some part of document and there is Apache feature named lingering close. For example, client's speed is 3K/s. If you have 64K kernel TCP send buffer then mod_perl will write 64K very soon but it will wait about 12 seconds to write other 36K. And after mod_perl will have written these 36K it will wait 2 second in lingering close. Even you increase kernel TCP send buffer (it's not always good idea) to 100K anyway you still 2 second lingering close delay for slow clients. Note that you never see lingering close state in /server-status (at least in Apache 1.3) - such processes are marked as idle but they are not really idle - they can not handle requests. Also lingering close time is not accounted in %T directive in the log. You can only estimate the count of Apache processes that is in linegring close state using netstat utulity and looking for FIN_WAIT2 socket states. Igor Sysoev http://sysoev.ru/en/
Re: Apache::Clean, Apache::Compress, mod_gzip/deflate, cross sitescripting and more.
On Sun, 27 Oct 2002, Richard Clarke wrote: Before I embark on a day exploring the pros and cons of today's cleaning/compression tools, I wondered if any of you could give me some feedback about your own experiences within the context of medium/large scale web sites/applications (E-Toys etc). Is it too presumtious to expect that many users now have high speed connections and tools that clean and/or compress html are of a small benefit anymore? That is not to mention the proliferation of pretty websites with 90% graphics. We'are using mod_deflate on rambler.ru about 1.5 year. It's one of the biggest Russian portals and search engines. We use gzipping on caching and accelerating mod_accel frontends but not on mod_perl backends. We use conservative mod_deflate settings i.e. we do not compress responses to requests that go through any proxy servers. These settings allow us to save about 5-10% of all bandwidth. Of course benefit of compressing depends to clients and content. Many Russian users have slow connections and even they have fast enough links these links usually shared so their resulting bandwith is low. Graphics is usually good cacheble but texts is not always. If it's your case then gzipping allows you to save bandwidth. I see 3 some trades off of compression using. 1. Perfomance. mod_deflate has capabilty to check system idle time (on FreeBSD only) and to disable gzipping if idle time would be less then specified. Anyway I never saw less then 30% idle time on out frontends. It seems that modern CPUs can easy gzip several small enough (30K) responses. 2. Memory. zlib uses about 300K for compressing. It seems to me more important resource then CPU. Additional 300K is big enough to lightweight frontend so mod_deflate has directive to disable gzipping if there are more Apache childs then specified - this allows to avoid intensive swapping when number of Apache childs increases for some reason. 3. Browser bugs. But it seems that modern browsers have not serious gzipping bugs. Igor Sysoev http://sysoev.ru
Re: SSL - mod_gzip - mod_perl = mod_proxy error
On Fri, 18 Oct 2002, Nigel Hamilton wrote: Even better ... is there a way to do SSL compression in mod_perl with only one server? mod_deflate. http://sysoev.ru/mod_deflate/mod_deflate-1.0.15.tar.gz Documentation is in Russian only but feel free to ask me directly. There is also Babelfish translation at: http://pflanze.mine.nu/~chris/mod_deflate/mod_deflate_readme_EN.html mod_deflate compresses any module output (except pre-1.3.24 mod_proxy, however I did not test it with 1.3.24+ mod_proxy). It works with mod_ssl correctly. ProxyRequests On This directive is not needed for reverse proxying. It makes yet another public proxy. Igor Sysoev http://sysoev.ru
Re: Reverse Proxy Setup
On Thu, 26 Sep 2002, Scott Alexander wrote: I have two experimental servers frontend and mod_perl. mod_perl is apache with mod_perl frontend is apache lightweight My config only passes *.pl thru to mod_perl When a user uploads a file they use a mod_perl script on mod_perl and the file is saved on mod_perl. If you use mod_proxy thne mod_perl would be busy for all upload time. Although there's patch for mod_proxy to buffer upload in temp file. mod_accel always buffers client's upload. But I want the file to be on frontend as it makes sense for when they download the file ... they only need a lightweight apache server to do that. So far I can think of doing 1. Putting a cgi script on frontend to handle file uploads and save the file on frontend. 2. Use network file share between mod_perl and frontend. You can send file from mod_perl and cache it on frontend. If you use mod_accel then it buffers whole reponse from backend in temp file and releases mod_perl as soon as possible. You even do not need to cache it. Igor Sysoev http://sysoev.ru
Re: performance regarding mod_perl vs mod_c with embedded perl
On Wed, 18 Sep 2002, Peter Bi wrote: Problem in authentication: if mod_perl returns cached header and the document is proxy cached in the plain Apache, the backend authentication handler (in the mod_perl server) will not be able to protect it. If you use HTTP Basic authentication then response to request with Authorization header will never be cached by mod_proxy or mod_accel. However mod_accel can ignore Authorization header by per location/files basis. Igor Sysoev http://sysoev.ru
Re: Redirecting through proxy
On Thu, 29 Aug 2002, Abd El-Hamid Mohammed wrote: Anybody can tell me how to redirect http://www.mydomain.com/abc To http://abc.mydomain.com:8080 I had used the following RewriteEngine on RewriteRule ^/abc(.*) http://abc.use-trade.com:8080$1 [P] ProxyPass /abc/ http://abc.use-trade.com:8080/ ProxyPassReverse /abc/ http://abc.use-trade.com:8080/ and it works great for redirecting http://www.mydomain.com/abc/ but it fails with http://www.mydomain.com/abc without the trailing slash as the first page is the only page that displays correctly, Any link in it is prefixed with http://www.mydomain.com/ which is wrong as it should be http://www.mydomain.com/abc/ Can anyone tells me how to solve it. You can add RewriteRule ^/abc$ http://abc.use-trade.com/abc/ [R] to send redirect to browser from /abc to /abc/ as mod_dir does for any directory without trailing slash. BTW, mod_accel send such redirection automatically, i.e if you have AccelPass /one/http://backend/two/ then if browser ask /one it will get redirect to /one/. Igor Sysoev http://sysoev.ru
Re: apache 1.3.26 reverse proxy
On Mon, 1 Jul 2002, E Kolve wrote: I was watching the apache scoreboard file and it appeared the the mod_perl process was not being immediately freed by the proxy. Normally there will be 3 or 4 mod_perl procs in mode W Sending Reply, but after around 20 - 30 seconds on 1.3.26 as the proxy all 30 (that is my MaxClients for mod_perl) were in W. This seems like a problem with either the IOBufferSize that was recently added or possibly something to do with the proxy not releasing the mod_perl process once it has gotten the last byte... You can try mod_accel. Incomplete English documentation is available here: http://dapi.chaz.ru/articles/mod_accel.xml Also feel free to ask me directly. Igor Sysoev http://sysoev.ru
Re: Apache Web Server vulnerability
On Thu, 20 Jun 2002, Lupe Christoph wrote: On Thursday, 2002-06-20 at 18:22:10 +0400, Igor Sysoev wrote: On Thu, 20 Jun 2002, Lupe Christoph wrote: and the mod_perl module seems to prevent the crash: telnet proxy.customer.de 80 Trying 213.155.64.138... Connected to proxy.customer.de. Escape character is '^]'. POST /x.html HTTP/1.1 Host: proxy.customer.de Transfer-Encoding: chunked 8000 Rapid 7 0 ^] telnet Connection closed. Does mod_perl do it's own de-chunking? So mod_perl does not return any response ? There are two ways to prevent crush with particular URI: 1. return 411 error if client send chunked body (standard mod_proxy, mod_cgi and mod_isapi do it); 2. ignore body at all. I suspect second in your case. Sorry that is not the answer to my question - the question is if my code gets a chance to do *anything*, or will the httpd code just crash at a later time? It does not crash like a non-mod_perl httpd. I think if you Apache does not send any response then it vulnerable with this particular URI. I've tried you request with plain Apache - one process starting to eat CPU without faults. Igor Sysoev http://sysoev.ru
Re: Apache Web Server vulnerability
On Fri, 21 Jun 2002, Richard [utf-8] Čepas wrote: On Wed Jun 19 17:54:02 2002 +0400 Igor Sysoev wrote: On 19 Jun 2002, Ilya Martynov wrote: If you still do not know about it: http://httpd.apache.org/info/security_bulletin_20020617.txt Now mod_perl question. mod_perl servers often are used as backend servers. I.e. they are not accessed directly but they are accessed either via fronted Apache or via proxy (like squid or oops) in accelerated mode. As I understand advisory in this case backend mod_perl server is not vulnerable since attacker do not have direct access to it. Can anybody confirm it? If your backend is proxied via mod_proxy or mod_accel then backend is not vulnerable because both modules do not accept client's chunked body at all. I can not say anything about Squid and Oops. They have in the changelog for 1.3.26: * A large number of fixes in mod_proxy including: adding support for dechunking chunked responses, correcting a timeout problem ... Does this change anything? I.e. backend is still safe? In 1.3.24 mod_proxy try to support chunked responses that go from servers. It never supports client's chunked body. As far as I know now there are no browsers that send chunked body. Igor Sysoev http://sysoev.ru
Re: Apache Web Server vulnerability
On Fri, 21 Jun 2002, Ask Bjoern Hansen wrote: On Thu, 20 Jun 2002, Lupe Christoph wrote: [...] Sorry that is not the answer to my question - the question is if my code gets a chance to do *anything*, or will the httpd code just crash at a later time? It does not crash like a non-mod_perl httpd. I believe it only crashes when using the default handler. Don't count on it though; there is plenty of obscure ways this issues has been biting us already. I think not only default handler. If you return 404 or some another error and keepalive is enabled then Apache calls ap_discard_request_body() and start to read chunked body. Second way is to send wrong 'Expect' header - Apache again calls ap_discard_request_body(). Igor Sysoev http://sysoev.ru
Re: tutorials (was: Re: rfc Apache::Dynagzip)
On Tue, 18 Jun 2002, Slava Bizyayev wrote: Igor seems upset with the current (not still final) results of this discussion, which HE tried to turn selfishly to a wrong way. Impolite style of discussions accomplished with attempts to hide some information and to provide some fantasies instead of real results of professional research does not work on this professional mailing list. What you call impolite style is simlpy poor English. It's very hard to me to express my thoughts in correct and polite English. That's right, instead of the main discussion we've spent a lot of time to learn a lesson, how the impolite selfish guy playing fool with professionals is punished finally, but wasn't it a fun? Ok, let me consider the lesson is over, and we may now go back to the main subject of this discussion. Yes, Beavis, it was a fun ! (I bag pardon to all subscribers, I can not resist, it's last personal email) Igor Sysoev http://sysoev.ru
Re: tutorials (was: Re: rfc Apache::Dynagzip)
On Mon, 17 Jun 2002, Slava Bizyayev wrote: Yes, absolutely. And no one on the earth may restrict your rights to log anything you feel important in your own handler. Just every overhead decrements the number of your prospective users. And when you try to dictate your very own needs (or fantasies) to be accepted as a standard, you are incorrect. It was my reply to Valerio_Valdez. He had asked me why I need logging. I had answered him. You said me once again about another ways of logging. I know them. Thank you. I do not try to dictate anyone my needs or fantasies. Relax and do anything in fixup handler and tutorial - I would try not disturb you in any way and hope that is the last my email on this subject. Igor Sysoev http://sysoev.ru
Re: tutorials (was: Re: rfc Apache::Dynagzip)
On Fri, 14 Jun 2002, Slava Bizyayev wrote:
Does it make sense? For me it means that every fact which is coming from
Igor Sysoev should be double-checked independently prior to practical usage.
OK. It's your right.
I guess some significant changes in the next version of the tutorial... On
other hand, I feel better, knowing that guys from Squid are caring about our
clients, and life goes on.
I did not check how Squid work with Vary header because any value in
this header simply disables caching in MSIE. I prefer client caching
to compression.
4. You should not unset Accept-Encoding. Better way is to set
$r-note('disable_gzip').
Sometimes it seems like Igor does not really understand what he is speaking
about. No comments.
I mean that that you should not change any incoming header.
Igor Sysoev
http://sysoev.ru
Re: tutorials (was: Re: rfc Apache::Dynagzip)
On Fri, 14 Jun 2002, Slava Bizyayev wrote:
I did not check how Squid work with Vary header because any value in
this header simply disables caching in MSIE. I prefer client caching
to compression.
It's not the truth again. I'm using Vary accomplished with Expires to
control MSIE local cache about half a year. Works fine.
I have just checked 3 MSIE:
Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)
Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 5.0; T312461)
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Q312461)
All of them had recevied responses like this:
HTTP/1.1 200 OK
Date: Fri, 14 Jun 2002 09:51:59 GMT
Server: Apache/1.3.22 (Unix)
Vary: Content-Encoding
Cache-Control: max-age=86400
Expires: Sat, 15 Jun 2002 09:51:59 GMT
Last-Modified: Tue, 09 Apr 2002 14:15:31 GMT
ETag: 1d3a9-65d-3cb2f783
Accept-Ranges: bytes
Content-Length: 1629
Connection: close
Content-Type: text/html; charset=koi8-r
All of MSIE do not cache responses. They even do not
send If-Modified-Since header.
Can you show me URL with Vary and Expires that MSIE would cache.
4. You should not unset Accept-Encoding. Better way is to set
$r-note('disable_gzip').
Sometimes it seems like Igor does not really understand what he is
speaking
about. No comments.
I mean that that you should not change any incoming header.
?! No comments.
How can I log a real Accept-Encoding header if you unset it ?
Igor Sysoev
http://sysoev.ru
Re: tutorials (was: Re: rfc Apache::Dynagzip)
On Fri, 14 Jun 2002, Slava Bizyayev wrote:
From: Igor Sysoev [EMAIL PROTECTED]
Can you show me URL with Vary and Expires that MSIE would cache.
You have this combination when you access my preview with your MSIE by
HTTP/1.1 with no proxy (it's still old version of
Yes, your response is really cached at least in MSIE 5.5.
I have just investigate this.
Responses with Vary: Accept-Encoding and Content-Encoding: gzip
are cached by MSIE. The Expires header is not needed.
Responses with Vary: Accept-Encoding but without Content-Encoding: gzip
are not cached by MSIE.
Furthermore, responses with Vary: Any,dummy,words and
Content-Encoding: gzip are also cached by MSIE.
And as I said before responses with Vary: Any,dummy,words and
without Content-Encoding: gzip are not cached by MSIE.
All these was tested with MSIE 5.5 only.
4. You should not unset Accept-Encoding. Better way is to set
$r-note('disable_gzip').
Sometimes it seems like Igor does not really understand what he is
speaking
about. No comments.
I mean that that you should not change any incoming header.
?! No comments.
How can I log a real Accept-Encoding header if you unset it ?
There is more than one way to do this, using mod_perl.
I mean that handler can do following:
if ($r-headers_in(Accept-Encoding) =~ /gzip/
and not $r-note(disable_gzip))
{
do gzipping
}
Igor Sysoev
http://sysoev.ru
Re: tutorials (was: Re: rfc Apache::Dynagzip)
On Wed, 12 Jun 2002, Slava Bizyayev wrote:
Since now the draft tutorial Features of Content Compression for Different
Web Clients (for Part IV: Client side facts and bugs) is available for
preview and discussion at
http://devl4.outlook.net/devdoc/Dynagzip/ContentCompressionClients.html .
Here is first part of criticism.
1. You should not mix proxies and browsers.
2. As I said you MS Proxy has not mask at all. ^1\.1 is not a mask.
^Squid/ is incorrect mask.
Here is example of Via header of HTTP/1.1 request that goes
though Squid, MS Proxy and Oops:
1.1 proxy1.domain1.com:3128 (Squid/2.4.STABLE2), 1.0 PROXY2,
proxy3.domain3.com:3128 (Oops 1.5.22f1)
3. Proxy masks will not help. About 70% of all proxied request are going
through Squid and about 15% are going through MS Proxy.
So much safe way is to disable proxy requests at all instead of tring
to look mask.
Besides I suspect that most proxies handle compressed content incorrectly.
I had checked only Squid, MS Proxy and Oops but there are many another
proxies. So I think you should disable gzip for all proxied request
or enable it for all proxied request if you do not care about old broswer.
mod_deflate by default disable it.
4. You should not unset Accept-Encoding. Better way is to set
$r-note('disable_gzip').
5. There are two unrelated mod_deflate. First is my mod_deflate for Apache 1.3:
http://sysoev.ru/mod_deflate/
It'd been tested for one year on loaded sites.
Second is expiremental module for Apache2 by Apache tream.
Next part will be about browsers bugs.
Igor Sysoev
http://sysoev.ru
Re: tutorials (was: Re: rfc Apache::Dynagzip)
On Wed, 12 Jun 2002, Slava Bizyayev wrote:
Since now the draft tutorial Features of Content Compression for Different
Web Clients (for Part IV: Client side facts and bugs) is available for
preview and discussion at
http://devl4.outlook.net/devdoc/Dynagzip/ContentCompressionClients.html .
About browsers.
First we should not mention any browser that does not send 'Accept-Encoding'.
So remove Opera 3.5 from list.
Netscape 4.x
All NN 4.x does not understand chunked content. And they should not.
Chunked encoding is part of HTTP/1.1 and NN4.x support HTTP/1.0 only.
It's server bug if server sends chunked response to HTTP/1.0 request.
And your devl4.outlook.net has this bug (I suspect 1.3.24 mod_proxy).
So remove this bug from list.
Netscape 4.0-4.05 understand gzip and x-gzip. They did not
send 'Accept-Encoding'.
MSIE 4.x
Code
$r-header_in('Range') 0
or
length($r-uri) 253
is wrong.
1. Range. We should send full (and probaly gzipped) response
to MSIE 4.x if it asks byte-range and our response can be gzipped.
We should disable byte-range but not gzipping.
2. $r-uri 253. Not only $r-uri but also $r-args, server name, port
and user name and password. In short - all without 'http://' prefix.
URI can be - http://name:password@host:port/uri?args.
So in mod_deflate I simply check r-unparsed_uri 200
MSIE 6.0
I do not understand this:
A: Unfortunately, IE6 (and perhaps earlier versions?) appears
to have a bug with gzip over SSL where the first 2048
characters are not included in the HTML rendering of the page
when refresh is pressed. It only seems to happen on longish
pages, and not when the page is first loaded. In fact, sometimes
it doesn't happen at all. The only current solution is to put
a 2048 character comment at the start of your longish pages
of all spaces (which compresses pretty well, fortunately).
If bug with first 2048 characters are NOT included in HTML rendering
then why to put 2048 character comment ? Comments do not included
in rendering.
FlashPlayer 4.x-5.x
Plug-in. Fails to decompress data received from the browser.
flash recieves all data from browser. But it can play gzipped swf
because all browsers (except NN 4.x for Windows) uncompressed them.
All flash players can not handle gzipped data received via loadVariables()
and XML.Load() (5.x).
You had mentioned Galeon and SkipStone but had not mentioned
Mozilla 0.9.1 and Netscape6.1b1 that have bug. Yes, this browsers
already old and rare but Galeon and Skipstone now include own version string.
It's much simply to translate from Russian my information than to try
to interpret it.
Igor Sysoev
http://sysoev.ru
Re: rfc Apache::Dynagzip
On Thu, 6 Jun 2002, Slava Bizyayev wrote: I mean that Apache fixup handler is not right place to decide should we use gzip encoding or not. I'd prefer to address it the point where the web server administrator has to fix the Accept-Encoding HTTP header, if one is incorrectly issued by buggy web client. Any early stage of the request proccessing is good for that as long as the following stages do not alter this header. The FixUp is the most appropriate place for such correction in terms of Apache architecture. I don not think that change or deleting of incoming headers is good idea. How can I log a real Accept-Encoding value ? Igor Sysoev http://sysoev.ru
Re: rfc Apache::Dynagzip
On Tue, 4 Jun 2002, Slava Bizyayev wrote:
Dear Valerio, Alvar, and Igor,
Thank you very much for your attention to Apache::Dynagzip.
It is on CPAN now:
The uploaded file Apache-Dynagzip-0.04.tar.gz has entered CPAN as
file: $CPAN/authors/id/S/SL/SLAVA/Apache-Dynagzip-0.04.tar.gz
size: 24939 bytes
md5: 586610ffdd35cd1ceceaba4796fd6212
I've just fixed a few typos and updated README for this version.
Now I'm back in our discussion.
I believe the questions, which we began to discuss yesterday with Valerio
privately should go public being helpful for others too. Especially, seeing
that they are similar to Alvar's interests.
I feel this discussion very important, but this time it is not about the
Dynagzip itself. Indeed, it is about the implementation of gzip compression
in most common view.
You know, there are 6 open source modules/packages for the web content
compression available in the World (in alphabetic order):
·Apache::Compress
·Apache::Dynagzip
·Apache::Gzip
·Apache::GzipChain
·mod_deflate
I should note that there are two unrelated mod_deflate modules - my one for
Apache 1.3 and Apache2's one.
·mod_gzip
On other hand, you know of-course, that the compression is used rarely in
practice to date. My own observations show that there are only few content
providers, who compress their traffic. I would mention Oracle as the leader,
which covers a significant percent of its on-line documentation with gzip
compression (looks like a static approach). On Yahoo there are few pages
gzipped only. Even the US governmental sites do not use gzip. Why?
Because it is not that simple to implement the web compression to date. The
success of the compression implementation depends on quality of client side
software. I would not consider the efforts of browser vendors sufficient
these days. We know many bugs. Others are still hidden. We can't fix them.
To adjust a web server appropriately is a common problem for any of
mentioned compression modules.
mod_deflate had been used for one year on several popular Russian sites
without problems so I think the main browsers have good gzip support.
That's why I believe it is better to have one simple common rule in all
compression modules (like an Accept-Encoding header), and a flexible FixUp
handler, which should control the $r-header_in('Accept-Encoding') prior to
compression handler.
Fixup handler is too early for this.
I don't know should it be a kitchen of every system administrator, or
somebody could volunteer to serve the public web site about the current
conditions of different web clients and recommended masks?.. I can't host it
on my devl4, because it is a development machine. That would be better to
find some stable place, like mod_perl, or apache project ;-).
I can host it on my site. I already have list of browser gzip/deflate
capabilites and bugs in mod_deflate Russian documentation.
I need simply translate it to English.
Igor Sysoev
http://sysoev.ru
Re: rfc Apache::Dynagzip
On Mon, 3 Jun 2002, Slava Bizyayev wrote: Finally, I'm going to upload the Apache::Dynagzip 0.03 to CPAN by the evening of June 4-th. I'm not sure about the quality of my documentation yet. It's available now at http://devl4.outlook.net/devdoc/Dynagzip/Dynagzip.html for preview. I would appreciate any comments in order to fix something important before the first upload. I can not see http://devl4.outlook.net/devdoc/Dynagzip/Dynagzip.html in NN 4.78. It show message box A commmunication error occured. I think this is due to Transfer-Encoding: chunked and Content-Encoding: gzip. NN 4.78 does not understand chunked response but can show it (with strange numbers of course) if there were no gzip encoding. Igor Sysoev http://sysoev.ru
Re: rfc Apache::Dynagzip
On Tue, 4 Jun 2002, Alvar Freude wrote:
The last years I experimented a lot with gzipped output and wrote my own
gzipping output function (but without chunked encoding :-) ). I realized a
lot of bugs with some different browsers under some environments or
situations. I found no remarks according this in your documentation, so
perhaps you missed something?
(if I missed something, feel free to write ;-) )
So, e.g. I created a website with a big server generated flash movie (with
Flash generator, mod_perl, Postgres; yep, Flash sucks!), and Netscape
doesn't decompress the output, but claims to understand gzip. AARRGH!
NN4 for Windows does not decompress gzipped Flash. NN4 for Unix does.
But no one browser decompress gzipped response that is fetched via
Flash's loadVariables() and XML.load() functions.
Also compressed output does not work on IE, if the request was POSTed (only
tested with older version, I gess 4.0, of IE).
I did not know about IE4 and POSTs, at least I did not received mod_deflate
bug reports about it. I know two IE4 gzip bugs. They described in
http://sysoev.ru/mod_deflate/readme.html#browsers (sorry, Russian only).
gzip compression is enabled when:
- browser acceps gzip
- or browser is an Bugscape 4.x on X11 or Mac
All NN4.5+ say Accept-Encoding: gzip. NN4.0x for Unix and PPC Mac do
not say but understand it but they too rare now so you can ignore them.
but gzip is DISABLED when:
- method is not GET:
at least on some versions of IE (4.x), compressed pages didn't work
together with POST!
I did not know this bug. I will check.
- every image/* content type needs no gzip (at least GIF and JPEG ...)
- browser is not compatible and Content-Type is not text/html,
even if browser said that he understand gzip
At least Bugscape 4.x didn't decompress embedded objects like
flash movies
NN4 also does not undresatnd gzipped application/x-javascript and ext/css:
http://sysoev.ru/mod_deflate/readme.html#types
# Important: we change output
# according to this headers
$r-header_out('Vary', 'Accept-Encoding, User-Agent');
It's not good idea - it disables caching in MSIE.
# don't forget Content-Length!
# it enables keepalive and disables
# some Netscape bugs
$r-header_out('Content-Length', length(${ $out-{document} }));
I'm not sure that sending Content-Length with gzipped content is good
idea. MSIE understands chunked transfer and can keep alive connection.
For NN4 we should close connection.
Igor Sysoev
http://sysoev.ru
Re: rfc Apache::Dynagzip
On Tue, 4 Jun 2002, Alvar Freude wrote:
Also compressed output does not work on IE, if the request was POSTed
(only tested with older version, I gess 4.0, of IE).
I did not know about IE4 and POSTs, at least I did not received
mod_deflate bug reports about it.
I realized this on http://www.a-blast.org/ when posting some texts. As I
remember this was very strange, because sometimes it works but mostly not.
I know two IE4 gzip bugs. They
described in http://sysoev.ru/mod_deflate/readme.html#browsers (sorry,
Russian only).
hmm, Don't understand Russion ;) -- perhaps it's a siede effect of one of
this bugs?
I think these bugs unrelated.
1. MSIE 4.x can not handle gzipped response if its URL without http://;
prefix is bigger then 253 bytes.
2. MSIE 4.x has bug with gzipped response and byte-range request (I found
this bug while testing mod_deflate and it should never occured under usual
conditions).
# Important: we change output
# according to this headers
$r-header_out('Vary', 'Accept-Encoding, User-Agent');
It's not good idea - it disables caching in MSIE.
really?
hrmpf!
I tested it now with MSIE 5.5; sometimes it fetched the file, sometimes it
came from cache. I also remember, that at another project i set for testing
purpose the expire time +100 seconds. And forgot it to remove after
testing. AARRGH! == MSIE cached the page even if there was a hard reload,
the old page remains in cache. It was a hard to find bug ;-)
But Vary there was also set, so it seems that at least under some
circumstances MSIE caches data with vary.
In my tests MSIE did not cache responses with any Vary header.
I included it because a proxy server may get an request from a browser
which understands gzip; and some seconds later from one who doesn't
understand it (i.e. MSIE Mac). So, if the proxy forwards the compressed
data to the second browser this will fail.
What experiences do you have with this?
By default mod_deflate does not gzip responses to proxied requests (that
have Via header) and HTTP/1.0 requests. It can be overridden.
# don't forget Content-Length!
# it enables keepalive and
# disables some Netscape bugs
$r-header_out('Content-Length', length(${ $out-{document} }));
I'm not sure that sending Content-Length with gzipped content is good
idea. MSIE understands chunked transfer and can keep alive connection.
For NN4 we should close connection.
i mostly closed connection , because keep alive blocked a bug Apache (on
http://www.a-blast.org/ each httpd is about 38-42 MB, but mostly shared);
but now i moved to frontend and backend apache, so there are not too much
memory consumption thrown away for static requests or requests to my other
Vhosts.
I had several really *hard* problems without Content-Length and NN4:
If in the page is some document.write JavaScript and there is no
Content-Length, NN4 writes some of the document.write-Code into a random
part of the HTML file. :-( This often destroys the hole page, e.g. if it
overwrites closing table-Tags.
Several weeks i searched for this ugly bug, because sometimes it destoys
the page and sometimes not. If destoying, Netscape does also not realise
that the connection is already closed and the stars are moving, indicating
further loading. But when the content-length is correctly set, it works.
I guess that NN4 gets confused with the end of the page and when he should
start with running of the JavaScript.
I usually disable javascript in my NN4 (it's so buggy) so I did not see
this problem.
Igor Sysoev
http://sysoev.ru
Re: Problem: Internal redirection of ErrorDocument to mod_perlhandler
On Sat, 1 Jun 2002, Richard Clarke wrote: Before I present a detailed description of the problem I thought I'd just bounce of an intro to see if its a common fault. I have found, using apache 1.x latest and mod_perl 1.x latest that if I use internal redirection to a mod_perl handler for ErrorDocument 500 (server_error) then even though I return the perfectly correct data (html headers etc) to the browser, every browser works fine. Except that is, Internet Explorer 6 (maybe earlier too but I have no test bed). Internet Explorer chooses to display the default Microsoft internal server error page and not the html which is actually returned. If I use a network monitor like Iris I can see that the data is returned perfectly...but Internet Explorer refuses to display it. This problem my apply to other error codes. The only fix I found is to use a fully qualified host redirect for the ErrorDocument.. but then I lose fancy stuff like previous request analysis. Any ideas? You error response size should be more then 512 bytes: http://support.microsoft.com/support/kb/articles/Q294/8/07.ASP Igor Sysoev http://sysoev.ru
Re: Proxied mod_perl server problem
On Mon, 13 May 2002, Ian Macdonald wrote: I'm setting up a new server and wish to run two Apache configurations, one vanilla and one mod_perl. I've configured pretty much according to Stas' excellent guide, using the rewrite module to push the perl requests to the second (ie mod_perl) server. Versions; HP-UX 10.20, Apache 1.3.24, mod_perl 1.26. It's _nearly_ working, but I'm getting one weird piece of behaviour which looks to me like the sort of problem someone with more experience will recognise very swiftly; when I request a perl script via the mod_perl-enabled locations, the content is preceded by a hex number, which corresponds very closely to the size of the content, and then a blank line, as shown below: 4e9 PID = 3282 DOCUMENT_ROOT=/opt/web_root/html GATEWAY_INTERFACE=CGI-Perl/1.1 etc. Your backend send chunked response - I think you use 1.3.24 mod_proxy and it probably send HTTP/1.1 request. You should force mod_perl to HTTP/1.0 Another way is to use mod_accel. It always uses HTTP/1.0 with backend and more advanced then mod_proxy. Igor Sysoev http://sysoev.ru
Re: Proxy authentication against a mod_perl backend - how?
On Mon, 1 Apr 2002, Fury wrote: I've recently reconfigured a web server to use the front-end proxy/back-end mod_perl configuration. One application runs happily on the mod_perl server, with static content being served from the proxy, and dynamic requests forwarded to the backend. However, I have a request to insert a bunch of static content onto the proxy server, which is fine. As an added requirement, the content is to be protected. Now, I could just use the standard Apache access files, but they're ugly, and get slow as they grow. Ideally, I'd like to use the backend database to hold the user information used for authentication. How can I configure the proxy (which doesn't have mod_perl) to authenticate against the back end? My first thoughts were to use mod_proxy to forward requests for /protected/login to the backend, where the authentication will be done. Then, just redirect the request to another URL behind /protected. The authentication information should be passed as part of the request, should it not? I'd also reverse proxy the URL's - I don't think this would work if I didn't. mod_accel can do such authorization with directive AccelRevalidateUser. Authorization can be done via HTTP Basic authorization or cookie. If directive is active the following happens: 1. if there is no cached response then it request is passed to backend; 2 if backend reponses with 200 code then reponse is cached; 3. if there is cached reponse then request is passed to backend with If-Modified-Since header that set to Last-Modified header of cached reponse; 4. if backend reponses with code 304 or 200 then cached or received reponse send to client; 5. if backend reponses with any other codes then reponse send to client but cached reponse is not deleted. But this feature is usefull only if backend authorization and 304 (Not Modified) generating is cheaper then full reponse (200) generating. Igor Sysoev igor sysoev.ru
Re: [OT] Replacing reverse squid with mod_proxy
On Mon, 25 Mar 2002, Hans Juergen von Lengerke wrote: We are currently using squid set up as a reverse proxy to accelerate several heavy backends (mod_perl, etc) and to protect them from slow client connections. I am looking into replacing the squid with apache+mod_proxy. Why? Because ultimately I'd like to be able to cluster the frontend using mod_backhand + wackamole. The primary reason for clustering isn't for load-balancing (yet) but for failover handling. So, ideally, one machine should be enough to serve the whole server load. Speaking of load, the squid (2.3.STABLE1) is currently doing up to 80 requests per second at a cache hit ratio of around 72%. This is on one box, a Dual 500MHz Pentium III with 1GB RAM. Average object size is 6KB. 200/304 ratio is around 5/3. Now, I've tried to replace the squid with apache+mod_proxy (1.3.11) and the frontend very quickly came to a standstill. I set MaxClients to 256 but all slots filled up fast. I upped MaxClients to 512 (recompiled with patched httpd.h) but the result was the same. All slots were occupied in no time and the server ground to a halt. Now I'm left with two choices: give up or try harder :-) Before I decide for one of them I thought I'd ask on the lists (sorry for the x-post) to see if the above numbers (80 Hits/Second) are in fact feasible with apache/mod_proxy on one box. Are there any benchmarks around? Does someone have a similar setup and how many requests can they serve? Should I up MaxClients any further? Will I get better results using a newer version of apache? Or should I give up and stick with squid? Apache can easy handle 80r/s on Ethernet or localhost connection. But if connections are slow (i.e. modem users) and responses are big enough then you can reach MaxClients. As far as I know 80r/s rate requires about 400-600 Apache with 20-40K responses. If your responses are bigger then more Apaches you need. You can increase MaxClients until your swap is free. On 1G you can run about 1000-2000 plain Apaches. But if you plan to use mod_proxy to proxy backend I think you should use mod_accel. Igor Sysoev
Re: Creating a proxy using mod_perl
On Fri, 15 Mar 2002, Marius Kjeldahl wrote: Any other ways of accomplishing the same without the added overhead of my perl module? You can use 1. mod_proxy: ProxyPass /images/http://image.site/image/ 2. mod_accel: AccelPass /images/http://image.site/image/ 3. default-handler - images must be on the same host: Location /images/ SetHandler default-handler /Location Igor Sysoev
Re: Creating a proxy using mod_perl
On Fri, 15 Mar 2002, Marius Kjeldahl wrote: I guess these all suffer from the fact that the parameters have to be specified in httpd.conf, which makes it impossible to pass a url to fetch from in a parameter, right? So mod_rewite with mod_proxy or mod_accel: RewriteRule /proxy_url=http://(.+)$http://$1 [L,P] Note that 'proxy?url=' is changed to 'proxy_url='. Igor Sysoev
Re: Creating a proxy using mod_perl
On Fri, 15 Mar 2002, Bill Moseley wrote: At 05:11 PM 3/15/2002 +0300, Igor Sysoev wrote: On Fri, 15 Mar 2002, Marius Kjeldahl wrote: I guess these all suffer from the fact that the parameters have to be specified in httpd.conf, which makes it impossible to pass a url to fetch from in a parameter, right? So mod_rewite with mod_proxy or mod_accel: RewriteRule /proxy_url=http://(.+)$http://$1 [L,P] Note that 'proxy?url=' is changed to 'proxy_url='. Any concern about being an open proxy there? I'd want to only proxy the sites I'm working with. I'd rather cache the images locally, just in case you are working with a slow site or if they do something silly like check referer on requests. My prefrence is using static parameters in httpd.conf: AccelPass /mercant1/http://mercant1/umages/ AccelPass /mercant2/http://mercant2/umages/ ... AccelPass /mercant3/http://mercant3/umages/ And of course proxied images can be cached. Igor Sysoev
Re: how to pass data in internal redirects?
On Tue, 26 Feb 2002, F. Xavier Noria wrote: I suppose that controllers would use internal redirects to call the views, is there a way to pass Perl data this way? For example, in the hangman game in O'Reilly's book a controller would load a session from the cookie, process user's guest, modify the state and redirect the request internally to the view. Ideally the view shouldn't read the data to display from the database again... could it be passed somehow by the first content handler? As far as I know r-notes() do not persist across internal redirections. You can try r-err_header_out() but clean up it in second handler before content output. Igor Sysoev
Re: Streaming compression of output from mod_perl handler?
On Wed, 20 Feb 2002, [iso-8859-1] Nicholas Oxhøj wrote: I'm not sure that lynx can handle compressed response on the fly - it uses gzip in pipe. The best way to test it using netcat. Well, lynx didn't decompress it, it just output the gzip compressed content to stdout. As I didn't have netcat readily available on the machine, I instead put an strace on lynx, to be absolutely sure, that it didn't receive any output until the very end - and it didn't :-( I you like to test I can make patch for mod_deflate to flush Apache. But if major browsers can not handle compressed content on the fly it's not valuable. That would be an interesting patch, but with approx 450KB of uncompressed HTML, I would expect mod_deflate to receive compressible input, regardless if the content producer specifically flushes or not. But I might have misunderstood something. Regarding the browsers ability to handle compressed content on the fly, we probably won't know until I find a module that is able to produce such output. Sorry again. I've just checked sources and found that if mod_deflate received flush then it flushes both zlib and Apache. You can try to set autoflush in perl module with $|=1; or call $r-rflush; when you like to flush output. Igor Sysoev
Re: Streaming compression of output from mod_perl handler?
On Wed, 20 Feb 2002, [iso-8859-1] Nicholas Oxhøj wrote: Sorry again. I've just checked sources and found that if mod_deflate received flush then it flushes both zlib and Apache. You can try to set autoflush in perl module with $|=1; or call $r-rflush; when you like to flush output. I just tried using $r-rflush in my handler and it works perfectly :-) The output gets rendered on the fly and it barely hurts the compression ratio. The 430 KB gets compressed to 26 KB instead of 24.5 KB :-) :-) What browsers did you test ? But wouldn't it be nice to have some mod_deflate option where you could specify that mod_deflate should flush and send the currently compressed output every time it had received a certain amount of input or every time it had generated a certain amount of output. It's complicates things. Besides Apache never flushes output on timeout or amount of content - it flushes only if you ask it. We are, for instance, using a template module to generate the output. We just give the template module a data structure and a template, and it then goes away and fills in the template, outputting HTML. This means that we don't have any easy way of calling flush at certain So we don't have any easy way of calling rflush once in a while. Of course we might just modify or substitute the template module, but it seems like a more automatic kind of streaming deflating (like described above) would be nice to have. You can set $|=1 as Eagle book says: This method [r-flush()] is also called automatically after each print() if the Perl global variable $| is nonzero. Igor Sysoev
Re: mod_perl, mod_gzip, incredible suckage
On Tue, 19 Feb 2002, Eric Cholet wrote: --On vendredi 15 février 2002 17:19 +0300 Igor Sysoev [EMAIL PROTECTED] wrote: You can try ftp://ftp.lexa.ru/pub/apache-rus/contrib/mod_deflate-1.0.11.tar.gz It compresses content on the fly without any temporary files as mod_gzip does. It has workarounds with buggy browsers. mod_gzip has not. Drawbacks: It patches Apache and can not be loaded as DSO. It compresses static files every time. mod_gzip can use cached compressed file. It has Russian documentation only but you can ask me if you have any problems. Yeah, can you translate that little paragraph about FreeBSD? There are 3 times when FreeBSD is referred: in /.configure: --with-idle-check - check CPU idle state. This parameter is available on FreeBSD 3.x and above only. DeflateIdleCheck: Syntax: DeflateIdleCheck number Default: DeflateIdleCheck 1 Context: server config Set interval for CPU idle checking in seconds. This directive is available on FreeBSD 3.x and above only if you set --with-idle-check configure parameter. DeflateMinIdle: Syntax: DeflateMinIdle 0 .. 100 Default: DeflateMinIdle 0 Context: server config Set minumal CPU idle level in percent when compressing is enabled. This directive is available on FreeBSD 3.x and above only if you set --with-idle-check configure parameter. Small comment. We are using idle checking on www.rambler.ru but it seems it is not usefull on small files (50K) and modern CPU. Igor Sysoev
Re: Streaming compression of output from mod_perl handler?
On Tue, 19 Feb 2002, [iso-8859-1] Nicholas Oxhøj wrote: I am looking for an Apache module which will allow me to compress the output of my mod_perl handler (a native handler, i.e. not running under Apache::Registry). But since my handler can potentially take a long time to finish, the output has to be compressed in a streaming fashion (or in small blocks) so that the browser will start receiving data before my handler has completely finished. I have been experimenting with all the different Apache compression modules I have been able to find, but have not been able to get the desired result. I have tried Apache::GzipChain, Apache::Compress, mod_gzip and mod_deflate, with different results. One I cannot get to work at all. Most work, but seem to collect all the output before compressing it and sending it to the browser. There also seems to be an issue about the new HTTP/1.1 chunked transfer-encoding. For instance, mod_gzip will not compress chunked output, unless you allow it to dechunk it by collecting all the output and compressing it as one big block. So I am basically looking for anyone who has had any success in achieving this kind of streaming compression, who could direct me at an appropriate Apache module. What mod_deflate did you try ? My or Apache 2.0 ? I can comment my mod_deflate. First, mod_deflate did not collect all output before compressing - it compress it on the fly. But it emits compressed content in 8K block. It's Apache's HUGE_STRING_LEN #define and it can be changed in sources. Besides if some module flushes output then mod_deflate would flushed it too. mod_deflate did not have problems with chunked transfer-encoding because it compress content before Apache start to make chunks. mod_deflate remove Content-Length header so compressed content would be sent to client chunked (HTTP/1.1) or not (HTTP/1.0). Igor Sysoev
Re: Streaming compression of output from mod_perl handler?
On Tue, 19 Feb 2002, [iso-8859-1] Nicholas Oxhøj wrote: What mod_deflate did you try ? My or Apache 2.0 ? Yours I can comment my mod_deflate. First, mod_deflate did not collect all output before compressing - it compress it on the fly. But it emits compressed content in 8K block. It's Apache's HUGE_STRING_LEN #define and it can be changed in sources. Besides if some module flushes output then mod_deflate would flushed it too. mod_deflate did not have problems with chunked transfer-encoding because it compress content before Apache start to make chunks. mod_deflate remove Content-Length header so compressed content would be sent to client chunked (HTTP/1.1) or not (HTTP/1.0). At the moment I have made my mod_perl handler extra slow, so that it is easier to see the effects of the different compression modules. The handler returns approx. 430KB of uncompressed HTML. With no compression module, the HTML slowly trickles to the browser, and the data gets displayed/rendered as it arrives. With mod_deflate enabled, nothing gets displayed in the browser until all 24KB of the gzip compressed HTML has arrived at the browser. Unfortunately I don't really have the tools to see when data are received at the client side, I can only judge by the time they are displayed/rendered in the browser. So I can't really tell if the problem is, that all the data are received in one big batch or if they are actually received in 8K blocks, but some bug-like feature in IE 5.5 makes it unable to decode the gzipped data in a streaming fashion as they arrive. Sorry, if mod_deflate will receive flush request it will flush deflate encoding and will write compressed content to Apache buffer. But it does not flush Apache. Anyway the max block is 8K, Apache buffer is 4K and OS usually should send data if in OS's buffer there is more then 2K. So it's probably MSIE feature as well as NC4 can not render tables until it receive '/table'. Igor Sysoev
Re: Streaming compression of output from mod_perl handler?
On Tue, 19 Feb 2002, [iso-8859-1] Nicholas Oxhøj wrote: if mod_deflate will receive flush request it will flush deflate encoding and will write compressed content to Apache buffer. But it does not flush Apache. Anyway the max block is 8K, Apache buffer is 4K and OS usually should send data if in OS's buffer there is more then 2K. So it's probably MSIE feature as well as NC4 can not render tables until it receive '/table'. If it was a bug in MSIE, it must be something specifically related to receiving compressed content, since the same data sent uncompressed, gets rendered as they arrive. Anyway, I just tried getting the same data using lynx, and this made it evident that *without* mod_deflate, the data gets sent by Apache as they are ready, whereas *with* mod_deflate, all the compressed data are sent as one big block at the end. I'm not sure that lynx can handle compressed response on the fly - it uses gzip in pipe. The best way to test it using netcat. So it seems that I am still unable to get the functionality I am looking for. I you like to test I can make patch for mod_deflate to flush Apache. But if major browsers can not handle compressed content on the fly it's not valueable. Igor Sysoev
Re: PerlPassEnv and Proxies
On Mon, 18 Feb 2002, Andrew Green wrote: I'm using a typical dual-server combination for mod_perl content on a selection of sites. Requests go to the vanilla Apache and a mod_rewrite/mod_proxy combo forward them on to the mod_perl Apache. As part of this, I'm trying to allow a prefix to the URL path to set an environment variable I can then use in my mod_perl programs to determine databases, templates etc. The vanilla httpd.conf features a line thus: RewriteRule ^/(test|access|ewok)/(.*)$ http://spare.article7.co.uk:8080/page/$2 [P,E=A7_FRAMEWORK:$1,L] (ignoring the wrapping). I also set PerlPassEnv A7_FRAMEWORK in my mod_perl httpd.conf -- but that environment variable always remains undef. The order of the P,E,L flags doesn't seem to make a difference. I appreciate that this is perhaps more of a mod_proxy issue than a mod_perl one, but I suspect I'm not alone in having tried something similar. I haven't found anything in the guide that deals with this specifically (I really don't want to append the information as a query string), but if I've overlooked anything, I'd be very happy to be corrected. You set A7_FRAMEWORK in vanilla Apache so mod_perl Apache does not receive it. You can try my mod_accel module. It send original URL to backend in X-URL header. Also you do not need to use mod_rewrite if you want to exclude some URL from proxing. Igor Sysoev
Re: mod_deflate problem with chunked encoding
On Fri, 18 Jan 2002, Jeremy Howard wrote: Geoffrey Young wrote: Philip Mak wrote: The following webpage on a mod_deflate enabled server is not working correctly in some browsers: ... a few of us have been wondering where all the traffic on these modules has been coming from, is all - I thought it might genuinely be some mis-documentation or something... I originally introduced these modules to this list in an earlier post titled Report on mod_accel and mod_deflate in which I promoted these modules as an excellent tool for creating a front-end accelerator to mod_perl web applications. It included brief documentation for installing and configuring the modules--previously there had only been Russian documentation. There's been no other location for English language discussion of these modules. So the discussion has ended up here. I take full responsibility for any OT pollution as a result. ;-) Given that these modules are new, don't have an active discussion home in English yet, and are very relevent to mod_perl programmers, I'd like to see discussion remain here, or at least have Igor post the occassional update on new releases etc. As to mod_deflate I think mod_perl list is not right place to discuss it. But mod_accel discussion is more relevent to this list as well as lingerd, mod_proxy, Squid and other reverse proxing or accelerating technology. If that bothers anyone then I'm happy to set up a mailing list elsewhere, of course. I think it would be nice. Igor Sysoev
Re: mod_deflate problem with chunked encoding
On Thu, 17 Jan 2002, Geoffrey Young wrote: Philip Mak wrote: On Thu, Jan 17, 2002 at 07:38:17PM -0500, Geoffrey Young wrote: Philip Mak wrote: The following webpage on a mod_deflate enabled server is not working correctly in some browsers: [OT stuff snipped] did you email the right list? I figured that the mod_perl mailing list was the best place to post about these problems, since mod_deflate/mod_accel doesn't have its own mailing list and there's a lot of people on this list who use it... If you have mod_accel/mod_deflate problem feel free contact me directly. Igor Sysoev
Re: mod_deflate problem with chunked encoding
On Thu, 17 Jan 2002, Philip Mak wrote: The following webpage on a mod_deflate enabled server is not working correctly in some browsers: http://www.aaanime.net/pmak/sylphiel/ If I telnet www.aaanime.net 80 and send the following commands: GET /pmak/sylphiel/ HTTP/1.1 Host: www.aaanime.net Accept-Encoding: gzip then the data it sends back is partially gzip, and partially plain text! It is sent with Transfer-Encoding: chunked and the first 1 or 2 chunks are gzipped, but the rest are not. The source code for the page is an .shtml file. Inside that .shtml file, I have the directive !--#exec cgi=navbar.cgi--. Everything until after that directive is gzipped, then the rest of it is not. navbar.cgi is a simple perl script that does: print Content-type: text/html\n\n; followed by a few more print statements. Any idea how to fix this problem? Do I need to provide any additional information in particular? Thanks. The problem really exists. I suspect '!--#exec' although '!--#include' works fine on our several sites. I'll try to repeat it in my local enviroment. Igor Sysoev
Re: DECLINED unless 'text/html' but images never make it
On Mon, 14 Jan 2002, Jon Robison wrote: How about trying: return DECLINED unless $r-is_initial_req; Image calls are not initial requests, they are sub requests. No. Requests for inline images are not subrequests. --Jon Robison R.Munden wrote: I've a script (controlled by a Location directive) that wraps a standard header and footer around an HTML page I've this at the top of my script: my $r = shift; return DECLINED unless ($r-content_type() eq 'text/html'); but any images that may be inline never make it to the browser (also, if I explicitly call the image in question it never makes it to the browser). Apache gives a 200 status code for these requests in the access log but Netscape 6.2 just sits there and IE returns a 'Cannot find server...' error. Igor Sysoev
Re: Unsetting standard response headers?
On Sun, 13 Jan 2002, Perrin Harkins wrote: I have noticed that Yahoo uses Location: header only for redirect responses and thought it may be good to save half of the bandwidth and do the same, as my particular script/server is serving redirects mostly. So my question is how to unset Date:, Server: and Content-Type: response headers? Who is setting them in the first place? If they are generated by your script and you don't set them, Apache will not add them. You may be seeing them added for redirects that Apache does for you, like sending http://yoursite to http://yoursite/. You can handle those yourself instead if you want to. Apache core always sets 'Server' and 'Date' headers. You can not simply overwrite them - you need patch Apache or use low-level Apache API. Igor Sysoev
mod_accel English documentation
Danil Pismenny had begun to translate into English mod_accel documentation: http://dapi.chaz.ru/articles/mod_accel.xml?lang=en Please send him ([EMAIL PROTECTED]) corrections. If you don't understand some translation at all ask me. Igor Sysoev
Re: Fixed (Re: HTTP file uploads with mod_accel)
On Sun, 6 Jan 2002, Philip Mak wrote: Never mind, I'm an idiot. I just took a look at the error_log of my frontend and the problem became clear. [Sun Jan 6 09:42:04 2002] [error] [client 206.173.36.189] (13)Permission denied: accel: can't create tempfile /usr/local/apache/cache/tmpFtYxlf My fault. I've just fixed it and in next release mod_accel would return 500 in this case. Igor Sysoev
Re: Fixed (Re: HTTP file uploads with mod_accel)
On Sun, 6 Jan 2002, Issac Goldstand wrote: DOES mod_accel buffer the uploads as they come through? That feature would be critical for compatibility with libapreq's UPLOAD_HOOK, which I'm finding lots of nice uses for... Yes, mod_accel buffers completly uploads and starting from 1.0.8 broken uploads even do not go to backend. The are two reasons: 1. Backend should not wait slow client upload. 2. To implement simple fault-tolerance I need to collect whole upload. Although second reason does not require to collect whole upload before sending it to backend. There is workaround to use UPLOAD_HOOK on backend - you can use mod_proxy with mod_accel: ProxyPass /upload/ http://backend/upload/ ProxyPassReverse /upload/ http://backend/upload/ AccelPass / http://backend/ AccelNoPass/upload/ Igor Sysoev
Re: Strange Apache 2.0 rewrite/proxy issue
On Thu, 3 Jan 2002, John Armstrong wrote: This 'seems' to be a modperl issue. My configuration. I needed a 1.1 compliant reverse proxy in order to support Chunked encoding for an xml gateway. Why do you need chunked encoding from backend ? Since only Apache 2.0 has a 1.1 compliant reverse proxy I replaced my Apache 1.3.14 standard apache wth an Apache 2.0.28 with the proxy support compiled in. My modperl server on the backend is still running as 1.3.14. The 2.0.28 proxy uses mod_rewrite. When it rewrites url's internally to go to a static apache server all works great! However, when it rewrites to my modperl 1.3.14 server apache 2.0.28 children start segfaulting left and right with a code 11. I can not find any reference to this on the 2.0 mailing lists/groups. Does anyone know why Apache 2.0 running as a rewrite reverse proxy would die only when it talks to my modperl servers? This is destroying my development time and about to kill me on some significant deadlines :( I'm not sure that mod_proxy in Apache 2 is stable. Igor Sysoev
Re: Strange Apache 2.0 rewrite/proxy issue
On Thu, 3 Jan 2002, John Armstrong wrote: Because the front end reverse proxy needs to connect to one of 3 different servers. 1) Static html server. 2) Mod Perl dynamic content server 3) Windows based xml servers that need to use 1.1 to communicate. So if one will make request to xml server with HTTP/1.0 then he will receive response with error code ? Can not it work in HTTP/1.0 at all ? So for 3 we need chunked or the Content-Length, either way, we need 1.1. compatibility. John- On Thursday, January 3, 2002, at 12:33 PM, Igor Sysoev wrote: On Thu, 3 Jan 2002, John Armstrong wrote: This 'seems' to be a modperl issue. My configuration. I needed a 1.1 compliant reverse proxy in order to support Chunked encoding for an xml gateway. Why do you need chunked encoding from backend ? Igor Sysoev
Re: Strange Apache 2.0 rewrite/proxy issue
On Thu, 3 Jan 2002, John Armstrong wrote: Correct, with 1.0 we lose persistency and things slow down significantly. I guess I should have just said 'Persistency' in the first place, sorry about that :) OK. Where do you need persistent connection - between frontend and xml backend or between client and frontend ? On Thursday, January 3, 2002, at 01:44 PM, Igor Sysoev wrote: On Thu, 3 Jan 2002, John Armstrong wrote: Because the front end reverse proxy needs to connect to one of 3 different servers. 1) Static html server. 2) Mod Perl dynamic content server 3) Windows based xml servers that need to use 1.1 to communicate. So if one will make request to xml server with HTTP/1.0 then he will receive response with error code ? Can not it work in HTTP/1.0 at all ? So for 3 we need chunked or the Content-Length, either way, we need 1.1. compatibility. John- On Thursday, January 3, 2002, at 12:33 PM, Igor Sysoev wrote: On Thu, 3 Jan 2002, John Armstrong wrote: This 'seems' to be a modperl issue. My configuration. I needed a 1.1 compliant reverse proxy in order to support Chunked encoding for an xml gateway. Why do you need chunked encoding from backend ? Igor Sysoev
Re: Strange Apache 2.0 rewrite/proxy issue
On Thu, 3 Jan 2002, John Armstrong wrote: When the reverse proxy connects to the windows boxes it needs to maintain a persistent connection since the client is an appliance, not a browser. So if you will have persistent connection between client and frontend and have non-persistent connection between frontend and xml backend will it slow down your things significantly ? Igor Sysoev This works fine with 2.0 when 2.0 is not segfaulting :) Think I'll just get out of apache land and let the F5 handle it for now, we won't hit the 100 rule limit for a few more months and hopefully apache 2.0 is more stable by then and can take over the URL rewriting. John- On Thursday, January 3, 2002, at 01:58 PM, Igor Sysoev wrote: On Thu, 3 Jan 2002, John Armstrong wrote: Correct, with 1.0 we lose persistency and things slow down significantly. I guess I should have just said 'Persistency' in the first place, sorry about that :) OK. Where do you need persistent connection - between frontend and xml backend or between client and frontend ? On Thursday, January 3, 2002, at 01:44 PM, Igor Sysoev wrote: On Thu, 3 Jan 2002, John Armstrong wrote: Because the front end reverse proxy needs to connect to one of 3 different servers. 1) Static html server. 2) Mod Perl dynamic content server 3) Windows based xml servers that need to use 1.1 to communicate. So if one will make request to xml server with HTTP/1.0 then he will receive response with error code ? Can not it work in HTTP/1.0 at all ? So for 3 we need chunked or the Content-Length, either way, we need 1.1. compatibility. John- On Thursday, January 3, 2002, at 12:33 PM, Igor Sysoev wrote: On Thu, 3 Jan 2002, John Armstrong wrote: This 'seems' to be a modperl issue. My configuration. I needed a 1.1 compliant reverse proxy in order to support Chunked encoding for an xml gateway. Why do you need chunked encoding from backend ? Igor Sysoev
Re: DirectoryAccelNoPass in mod_accel
On Mon, 31 Dec 2001, Philip Mak wrote: Is there a way to specify an AccelNoPass directive (from mod_accel) that only affects a certain directory? For example, consider the following scenario: AccelPass /~user1/ http://127.0.0.1:8001/ AccelNoPass ~*\.gif$ ~*\.jpg$ AccelPass /~user2/ http://127.0.0.1:8002/ AccelNoPass ~*\.gif$ Someone might want to specify separate AccelNoPass settings for those two directories. It doesn't seem to work when I put it in Directory though; I get AccelNoPass not allowed here error. (I don't actually need this functionality at this point and I think it's an obscure case, but I felt it was worth pointing out.) No. Both AccelPass and AccelNoPass run in translation phase and sets or does not set 'accel-handler'. So if AccelNoPass could run in Location or Directory then it means that mod_accel needs to skip 'accel-handler' and found another one instead - mod_accel needs to run subrequest. I think it complicates processing and is not needed in many cases. Besides in your example case you can use such regexps: AccelNoPass ~*\.gif$ ~*^/~user1/.*\.jpg$ Igor Sysoev
mod_accel-1.0.10 and mod_deflate-1.0.10
ftp://ftp.lexa.ru/pub/apache-rus/contrib/mod_accel-1.0.10.tar.gz Changes with mod_accel 1.0.1028 Dec 2001 *) Feature: AccelReverse directive was added. AccelReverse is similar to ProxyPassReverse directive but it's needed to use with mod_rewrite only. ftp://ftp.lexa.ru/pub/apache-rus/contrib/mod_deflate-1.0.10.tar.gz Changes with mod_deflate 1.0.10 28 Dec 2001 *) Feature: --with-zlib and --with-patch directives were added in configure. Igor Sysoev
Re: mod_accel-1.0.10 and mod_deflate-1.0.10
On Sat, 29 Dec 2001, John Siracusa wrote: On 12/29/01 8:23 AM, Igor Sysoev wrote: ftp://ftp.lexa.ru/pub/apache-rus/contrib/mod_accel-1.0.10.tar.gz Is there any timeline on a release of mod_accel with English documentation? I hope it will be ready in January. At least Danil Pismenny promised it. Igor Sysoev
Re: mod_deflate warning
On Sat, 29 Dec 2001, Jeremy Howard wrote: I posted a couple of weeks ago about how well the mod_deflate/mod_accel combination works on a front-end HTTP accelerator. I've just discovered a problem that I thought I'd mention here for anyone who's trying this out. It appears that Mozilla-based browsers fail when using gzip encoding with POST requests. I've added DeflateDisableRange MSIE 4. Mozilla to my httpd.conf as a workaround for this bug. The bug appears intermittently, and I haven't yet worked out exactly how to replicate it. However, it's been reported by many of my users and I have experienced it myself, so I'd suggest being wary of this combination. First, DeflateDisableRange doesn't disable gzipping - it disables partial responses (byte-range) if this request can be gzipped. If you want to disable gzipping for some browser you need to set enviroment variable: BrowserMatch Gecko no_gzip Second, don't use Mozilla string - this string is in all Netscape 1-6, and MSIE 1-6. Use Gecko instead - all Mozilla-derived browsers has this string. And third - can you send me all bug reports with Mozilla ? Igor Sysoev
Re: mod_accel reverse proxying?
On Sat, 29 Dec 2001, Jeremy Howard wrote: Philip Mak wrote: On Fri, 28 Dec 2001, Igor Sysoev wrote: I think it should have reverse syntax: AccelReverse http://127.0.0.1:8001/ / Or not ? Of course it complicates porting from mod_proxy to mod_accel but I think it's clearer then ProxyPassReverse syntax. I don't think either order is more clearer than the other, but since ProxyPassReverse has it like / http://127.0.0.1:8001/, my personal opinion is that AccelPassReverse should have it in the same order too to avoid confusion. On the other hand, I find mod_proxy's syntax weird and support Igor's proposal. Still, either one would be very handy! RewriteRule ^/$ http://backend/$1 AccelReverse / http://backend/ Today there will be mod_accel-1.0.10 and mod_deflate-1.0.10 tarballs. I'll post message. Igor Sysoev
Re: mod_accel reverse proxying?
On Thu, 27 Dec 2001, Philip Mak wrote: Does mod_accel have a reverse proxying directive (similar to the ProxyPassReverse directive in mod_proxy) in order to make redirects work? Yes, AccelPass automatically handles reverse proxying in Location and Refresh headers. I believe the AccelPass directive automatically handles reverse proxying, but what if I used RewriteRule instead: RewriteRule ^(.*)\.asp$ http://127.0.0.1:8001/$1.asp [L,P] That does not setup reverse proxying for me... Yes, it doesn't. It's difficult to figure proxied URL parts in mod_rewrite so I have to make explicit directive to specify reverse rewrite. I will make it today or tomorrow. I think it should have reverse syntax: AccelReverse http://127.0.0.1:8001/ / Or not ? Of course it complicates porting from mod_proxy to mod_accel but I think it's clearer then ProxyPassReverse syntax. Igor Sysoev
Re: File upload error
On Fri, 21 Dec 2001, Alexei Danchenkov wrote: This might be a bit offtopic, but while uploading a file to a server, all zeros (0x0) are replaced with spaces (0x20). What is wrong with the following code then or what's wrong with the server configuration? I am using HTML::Mason with Apache server 1.3.22 under RedHat Linux 6.2 When I am doing the same on the Win32 (local devel machine, no problem occurs). I think you are using Russian Apache. http://apache.lexa.ru/configure.html#charsetrecodemultipartforms Igor Sysoev
Re: Report on mod_accel and mod_deflate
On Thu, 20 Dec 2001, Philip Mak wrote: On Thu, 20 Dec 2001, Jeremy Howard wrote: Note that mod_accel can also be called by utilising the mod_rewrite [P] directive, just like with mod_proxy. Even more. You can use mod_accel/mod_rewrite/mod_include: RewriteRule ^/one.html$ http://backend/$1 [P] !--#include virtual=/one.html?arg=some -- If I put [P] in a RewriteRule, how does Apache know whether I want it to use mod_proxy or mod_accel? mod_proxy and mod_accel can work together expect one case - mod_rewrite [P]. You can disable mod_accel's [P] with --without-mod_rewrite mod_accel configure parameter. AccelSet* adds X-* headers to the request to the backend. This is useful to know what the original request details were. In ftp://ftp.lexa.ru/pub/apache-rus/contrib/ (where I have been told to download mod_accel/mod_deflate from before), I see another file called mod_realip-1.0.tar.gz just released one week ago. From looking at the keywords in the documentation, it looks like a module to be installed on the backend httpd that will parse these X-* headers to retrieve the original IP address. Yes, it can set IP using X-Real-IP header (default) or X-Forwarde-For header. Also mod_realip set IP in one of three phases - postread, header and fixups. By default only text/html is compressed. I think it's safe to compress text/plain by default, too; I've never seen any browser problems with compressed text/plain (only text/js and text/css). There is no text/js type - application/x-javascript instead. Macromedia FlashPlayer 4.x-5.x doesn't understand compressed text files received via loadVariables() function. These files can have any type but usually they have text/plain. If you site doesn't use such flash movies - you can safely compress text/plain: DeflateTypes text/plain Igor Sysoev
Re: Report on mod_accel and mod_deflate
On Thu, 20 Dec 2001, Jeremy Howard wrote: Thank you, Jeremy. Here is some comments. OK, enough evangelism. On to practical matters. Many of the following snippets and comments were provided to me by Igor in private email, so thank him, not me. Firstly, here's how to compile everything (including mod_ssl on the front-end) and install it into /usr/local/apache_accel: tar -zxvf mod_accel-1.0.7.tar.gz tar -zxvf mod_deflate-1.0.9.tar.gz tar -zxvf mm-1.1.3.tar.gz cd mm-1.1.3 ./configure --disable-shared make cd ../mod_ssl-2.8.5-1.3.22 ./configure --with-apache=../apache_1.3.22_accel/ \ --with-crt=/INSERT_APPROPRIATE_PATH.crt \ --with-key=/INSERT_APPROPRIATE_PATH.key cd ../mod_accel-1.0.7 ./configure --with-apache=../apache_1.3.22_accel/ \ --with-eapi=../mod_ssl-2.8.5-1.3.22/pkg.eapi/ make You don't need to use mod_ssl but you need EAPI distributed with mod_ssl. So if you don't use mod_ssl you need untar it only and set path to EAPI: cd ../mod_accel-1.0.7 ./configure --with-apache=../apache_1.3.22_accel/ \ --with-eapi=../mod_ssl-2.8.5-1.3.22/pkg.eapi/ make But if use mod_ssl then you can not to specify EAPI path - mod_ssl's configure set it on Apache source itself: cd ../mod_accel-1.0.7 ./configure --with-apache=../apache_1.3.22_accel/ make cd ../mod_deflate-1.0.9 ./configure --with-apache=../apache_1.3.22_accel/ make cd ../apache_1.3.22_accel/ EAPI_MM=../mm-1.1.3 SSLBASE=SYSTEM ./configure --enable-rule=EAPI \ --prefix=/usr/local/apache_accel --disable-rule=EXPAT \ --enable-module=most \ --enable-module=rewrite \ --activate-module=src/modules/accel/libaccel.a \ --activate-module=src/modules/extra/mod_deflate.o make make install mkdir /var/accelcache chown nobody:nobody /var/accelcache/ /usr/local/apache_accel/bin/apachectl startssl Now, in /usr/local/apache_accel/conf/httpd.conf, append the following: AccelCacheRoot /var/accelcache 1 AccelNoCacheon AccelPass/mail/http://127.0.0.1:8080/mail/ AccelWaitBeforeBodyRead 100 AccelUnlinkNoCached off AccelSetXHost on AccelSetXRealIP on AccelSetXURL on This creates an HTTP accelerator without any caching. I don't know much about the caching directives and haven't tried them out, so I can't provide any guidance there. The AccelCacheRoot directive sets the number of levels of subdirectories in the cache. 16 first level subdirectories named 0 .. f and 256 second level subdirectories named 00 .. ff are used by default (i.e. AccelCacheRoot path 1 2). Since I'm not caching, my config above does not specify any directory hashing (i.e. AccelCacheRoot path 1). AccelCacheRoot allow to set up to 3 levels of two type subdirectories - named 0 .. f and 00 .. ff. Also AccelCacheRoot has parameter 'noauto' that specify do not create automaticaly all nested subdirectories. They shoud be created before with create_cache script. Here's the config for mod_deflate: DeflateEnable On DeflateMinLength 3000 DeflateCompLevel 1 DeflateProxied Off This is default settings of DeflateCompLevel and DeflateProxied. DeflateHTTP 1.0 DeflateDisableRange MSIE 4. DeflateProxied Off means that anything with a 'Via' header is not proxied--Igor says that some proxies are broken so this is best to be safe. is not compressed And MSIE 4 apparently is a bit broken too. By default only text/html is compressed. DeflateCompLevel 1 provides adequate compression for text. Igor Sysoev
Re: Tips tricks needed :)
On Thu, 20 Dec 2001, Mark Fowler wrote: (sorry to break threading but I'm getting this from multiple lists) that IE 6 (beta at the time) considered my cookies to be third party because I used frame-based domain redirection and by default would not accept them. You need to include a P3P header in your HTTP header that contains a Compact Policy (CP) - a geek code of what your P3P xml privacy document contains. See http://www.w3c.org/P3P/. Some research I did seems to indicate that current implementations of IE6 will accept cookies no matter what CP you use (rather than checking it against your security settings and deciding if the CP represents a privacy policy that violates your chosen level of disclosure.) I'd really appreciate it other people could check this and confirm that IE6 is not offering any actual privacy level protection and is just discriminated against people that don't have P3P headers. I found that IE6 require P3P header with medium and higher security settings but CP content doesn't matter - it need simply P3P: CP='anything'. Igor Sysoev
Re: Report on mod_accel and mod_deflate
On Thu, 20 Dec 2001, Thomas Eibner wrote: On Thu, Dec 20, 2001 at 07:04:50AM -0500, Philip Mak wrote: AccelSet* adds X-* headers to the request to the backend. This is useful to know what the original request details were. In ftp://ftp.lexa.ru/pub/apache-rus/contrib/ (where I have been told to download mod_accel/mod_deflate from before), I see another file called mod_realip-1.0.tar.gz just released one week ago. From looking at the keywords in the documentation, it looks like a module to be installed on the backend httpd that will parse these X-* headers to retrieve the original IP address. I basically wrote something similar for the users of mod_proxy_add_forward 2-3 months ago, it's called mod_rpaf and sets r-connection-remote_ip and moved the X-Host: headers into the incoming Host: header and then updates the vhost structure so virtualhosting via mod_proxy_add_forward works. The module can be found at: http://stderr.net/apache/rpaf/ There is one drawback in setting r-connection-remote_addr. If you want to disable direct access to backend then you can't do it with mod_access - you need to configure firewall. Igor Sysoev
Re: Report on mod_accel and mod_deflate
On Thu, 20 Dec 2001, Igor Sysoev wrote: On Thu, 20 Dec 2001, Thomas Eibner wrote: On Thu, Dec 20, 2001 at 07:04:50AM -0500, Philip Mak wrote: AccelSet* adds X-* headers to the request to the backend. This is useful to know what the original request details were. In ftp://ftp.lexa.ru/pub/apache-rus/contrib/ (where I have been told to download mod_accel/mod_deflate from before), I see another file called mod_realip-1.0.tar.gz just released one week ago. From looking at the keywords in the documentation, it looks like a module to be installed on the backend httpd that will parse these X-* headers to retrieve the original IP address. I basically wrote something similar for the users of mod_proxy_add_forward 2-3 months ago, it's called mod_rpaf and sets r-connection-remote_ip and moved the X-Host: headers into the incoming Host: header and then updates the vhost structure so virtualhosting via mod_proxy_add_forward works. The module can be found at: http://stderr.net/apache/rpaf/ There is one drawback in setting r-connection-remote_addr. If you want to disable direct access to backend then you can't do it with mod_access - you need to configure firewall. Sorry, I did not notice RPAFproxy_ips. Igor Sysoev
Re: Report on mod_accel and mod_deflate
On Thu, 20 Dec 2001, Stas Bekman wrote: Jeremy Howard wrote: Igor Sysoev mentioned recently on this list that he has written a module called 'mod_accel' that provides a caching HTTP accelerator, as well as a mod_gzip replacement called 'mod_deflate'. These modules are both used on Kaspersky labs' busy sites, as well as at the popular portal http://www.rambler.ru. mod_deflate is used at around 700 sites. Igor, sorry to tell you but httpd-2.0 has introduced mod_deflate as a replacement for mod_gzip, so you are going to have a lot of confused users when httpd 2.0 hits the streets. I suggest that you raise this issue on the httpd dev list before it's too late. May be it's too late already, since there was already a long discussion about why it should be called deflate and not gzip (check the archives). Thank you. I'd overlooked it. Last time I saw discussion about mod_gzip and mod_gz in new-httpd list was September. By the way do you speak Russian ? Igor Sysoev
Re: Apache::SizeLimit Exit is Delayed
On Wed, 12 Dec 2001, Perrin Harkins wrote: what about $r-headers_out-add(Connection = 'close'); Good idea! I'll put that into a future release. Some bug report about Apache::SizeLimit diagnostic: I don't know about Linux and Solaris but under FreeBSD shared memory shows some incredible numbers: Apache::SizeLimit httpd process too big, exiting at SIZE=25092 KB SHARE=1823412 KB REQUESTS=79 LIFETIME=293 seconds I think that it due to rusage.ru_ixrss is integral value and it should be divided (but I don't know where to find divisor). And some recomendation - I'm using Apache::SizeLimit as PerlCleanupHandler - so Apache would exit after request is completed. mod_perl usually grows at response phase so exiting in fixup is not very usefull. I'm using mod_class for mod_perl size hard control during reponse phase. Igor Sysoev
Re: Apache::SizeLimit Exit is Delayed
On Thu, 13 Dec 2001, Perrin Harkins wrote: And some recomendation - I'm using Apache::SizeLimit as PerlCleanupHandler - so Apache would exit after request is completed. You should use it in an early phase, like PerlFixupHandler. It pushes a cleanup handler if it needs to exit. It will not exit until after the request is done. I didn't know it. I think you should document it. But any way I think it's better to check size in cleanup. I'm using mod_class for mod_perl size hard control during reponse phase. I've never heard of mod_class. Do you have a link for it? It's BSD specific module. It allow to set login class for process to limit memory or time usage. mod_class: ftp://ftp.lexa.ru/pub/apache-rus/contrib/mod_class.c There is more early module - mod_setclass: http://www.apache.org/dist/httpd/contrib/modules/1.3/mod_setclass.c The main difference - mod_setclass sets login class for each Apache child while mod_class sets class for main Apache process. mod_setclass is better to set time restriction - with mod_class you need to restart main Apache every day (it's common practice to rotate logs) otherwise main Apache can exhaust time limit. My official recommendation is to set Apache::SizeLimit up with low enough numbers that you can handle it if it grows a little more during the final request, and use Apache::Resource as a hard limit to prevent runaway processes from eating up all your RAM. Apache::Resource will kill your process even if it's in the middle of a response, so you don't want to use it for normal size control. mod_class and mod_setclass make the same on BSD. Igor Sysoev
Re: form upload limit
On Thu, 13 Dec 2001, Stas Bekman wrote: Viljo Marrandi wrote: Hello, I didn't find anywhere in Net how much is browser form upload limit (with POST) and how much is Apache's default form data access(input) limit. If anyone knows where I can find this data i'd be grateful. There is no such a limit in Apache and probably most browsers. The limit can be the amount of available RAM (if the file is aggregated in the memory) or the filesystem limits (not enough disk space, if the uploaded file is created on the filesystem). You can limit it within your code, though. About one third requests going through proxies. And about 70% of proxies are Squids. And nearly all Squids has default upload limit of 1M. Igor Sysoev
Re: Apache::SizeLimit Exit is Delayed
On Thu, 13 Dec 2001, Perrin Harkins wrote: You should use it in an early phase, like PerlFixupHandler. It pushes a cleanup handler if it needs to exit. It will not exit until after the request is done. I didn't know it. I think you should document it. But any way I think it's better to check size in cleanup. I agree and I plan to change this. It's BSD specific module. It allow to set login class for process to limit memory or time usage. How about sending Stas a patch for the guide with information on this? It might be useful to other BSD'ers. My English is too poor for it. Igor Sysoev
Re: [RFC] Apache::CacheContent - Caching PerlFixupHandler
On Thu, 6 Dec 2001, Paul Lindner wrote: On Thu, Dec 06, 2001 at 10:04:26AM -0800, Bill Moseley wrote: At 08:19 AM 12/06/01 -0800, Paul Lindner wrote: Ok, hit me over the head. Why wouldn't you want to use a caching proxy? Apache::CacheContent gives you more control over the caching process and keeps the expiration headers from leaking to the browser. Or maybe you want to dynamically control the TTL? You can use mod_accel to cache flexible backend: ftp://ftp.lexa.ru/pub/apache-rus/contrib/mod_accel-1.0.7.tar.gz mod_accel understands standard Expires and Cache-Control headers and special X-Accel-Expires header (it is not sent to client). Besides it allows to ignore Expires and Cache-Control headers from backend and set expiration by AccelDefaultExpire directive. Comparing to mod_proxy mod_accel reads backend response and closes connection to backend as soon as possible. There is no 2-second backend lingering close timeout with big answers and slow clients. Big answer means bigger then frontend kernel TCP-send buffer - 16K in FreeBSD and 64K in Linux by default. Besides mod_accel read whole POST body before connecting to backend. mod_accel can ignore client's Pragma: no-cache, Cache-Control: no-cache and even Authorization headers. mod_accel allow to not pass to backend some URLs. mod_accel allow to tune various buffer size and timeouts. mod_accel can cache responses with cookie-depended content. mod_accel can use busy locks and can limit number of connection to backend. mod_accel allows simple fault-tolerance with DNS-balanced backends. mod_accel logs various information about request processing. mod_accel can invalidate cache on per-URL basis. mod_accel has two drawbacks only - too much memory per connection (inherited Apache drawback) and Russian only documentation. Igor Sysoev
Re: ProxyPass and DirectoryIndex
On Fri, 9 Nov 2001, Philip Mak wrote: On port 80, I'm running a non-mod_perl httpd. On port 8001, I'm running a mod_perl httpd. Port 80 is ProxyPassing to port 8001 like this: RewriteRule ^/(.+)\.asp$ http://127.0.0.1:8001/$1.asp [p] The httpds have different DocumentRoots however, so if I visit http://mysite.com/ it will return a directory index rather than calling the index.asp file. My current solution is to touch index.asp in the port 80 DocumentRoot and have DirectoryIndex index.asp so that it knows to ProxyPass those requests. I'd have to touch index.asp manually for every directory, though. Is there a better way around this? Why do you use RewriteRule instead of ProxyPass ? ProxyPass/http://127.0.0.1:8081/ Or do you have static files that you don't want to pass to mod_perl ? You can try with my mod_accel: ftp://ftp.lexa.ru/pub/apache-rus/contrib/mod_accel-1.0.6.tar.gz It's replacement of mod_proxy. Documentation in Russian only. But you configuration is simple and can be following: AccelCacheRoot cache AccelNoCacheon AccelPass / http://127.0.0.1:8081/ AccelNoPass ~*\.jpg$ ~*\.gif$ It's Unix only. It can probably run on Win32 under CygWin - but I didn't try it. Igor Sysoev
Re: ProxyPass and DirectoryIndex
On Fri, 9 Nov 2001, Balazs Rauznitz wrote: On Fri, 9 Nov 2001, Philip Mak wrote: On port 80, I'm running a non-mod_perl httpd. On port 8001, I'm running a mod_perl httpd. Port 80 is ProxyPassing to port 8001 like this: RewriteRule ^/(.+)\.asp$ http://127.0.0.1:8001/$1.asp [p] The httpds have different DocumentRoots however, so if I visit http://mysite.com/ it will return a directory index rather than calling the index.asp file. My current solution is to touch index.asp in the port 80 DocumentRoot and have DirectoryIndex index.asp so that it knows to ProxyPass those requests. I'd have to touch index.asp manually for every directory, though. Is there a better way around this? How about: RewriteRule ^(.*)/$ http://127.0.0.1:8001$1/index.asp [p] In any case you need to double directory hierachy to allow mod_dir handle redirects from /dir to /dir/. Even without touching index.asp. Igor Sysoev
Re: ProxyPass and DirectoryIndex
On Fri, 9 Nov 2001, Philip Mak wrote: You can try with my mod_accel: ftp://ftp.lexa.ru/pub/apache-rus/contrib/mod_accel-1.0.6.tar.gz AccelCacheRoot cache AccelNoCacheon AccelPass / http://127.0.0.1:8081/ AccelNoPass ~*\.jpg$ ~*\.gif$ Hmm, so that would pass any URL that doesn't end in .jpg or .gif. While not semantically equivalent to what I'm doing, I think it would actually work for my case (I just have to specify what extensions NOT to pass). You can specify not only extension but any Apache regexp or location: AccelNoPass /some_static_location AccelNoPass /download ~*\.jpg$ ~*\.gif$ '*' after '~' means case-insensitive regexp. '~\.jpg$' is case-sensitive regexp. Also with mod_accel you will have other benefits - it allows to make a real backend buffering, to use busy locks and to limit connections to backend. Igor Sysoev
Apache::Compress - any caveats?
I had developed Apache module mod_deflate that allow to gzip or deflate content: ftp://ftp.lexa.ru/pub/apache-rus/contrib/mod_deflate-1.0.7.tar.gz This module tested at one of the loaded Russian sites - www.rambler.ru and several other small Russian sites. Documentation in Russion only but basic configuration is very simple: DeflateEnable on mod_deflate encodes 'text/*' content type. By default mod_deflate allows encoding if request is HTTP/1.1 and request is not proxied (it can be changed). So Netscape 4 requests is not encoded by default. During testing I found some browser bugs with gzip: MSIE 4.x can't handle gzipped content if length of URL string without 'http://' prefix is bigger then 253 characters. Mozilla 0.9.1 and Netscape 6.1b1 don't understand encoding if header line has tralling spaces - Content-Encoding: gzip . Macromedia FlashPlayer 4.x-5.x doesn't understand gzipped content recieved by browser via getVariables() function. Igor Sysoev
