Re: apache & log files
On Friday 05 November 2004 19:47, "Francesco P. Lovergine" <[EMAIL PROTECTED]> wrote: > On Fri, Nov 05, 2004 at 01:35:28AM +1100, Russell Coker wrote: > > My clftools package allows you to split and mangle the log files if you > > have Apache configured for a single log file... > > Uhm, not found in current sid archive Sorry, it's logtools. It's been so long since I've worked on it that I'd forgotten the name. It still works well though, I've got it processing all the web stats on the server that hosts my domain. -- http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/Postal SMTP/POP benchmark http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: apache & log files
On Fri, Nov 05, 2004 at 09:40:28AM +0100, Francesco P. Lovergine wrote: > On Fri, Nov 05, 2004 at 09:09:16AM +1100, Craig Sanders wrote: > > > For ErrorLog you can pipe to a suitable program which does the same. > > > > but this doesn't. unless apache has added this feature since i last looked > > into this (about six months ago) the "suitable program" has no way of > > separating the error logs for each virtual host, because it's just STDERR with > > no vhost prefix on each line. > > > > ErrorLog "| mytrickyprog www.mydomain.com" > > where mytrickyprog simply echos stdin on the right per-domain file or > the same log file with the right prefix for each line. Of course you > need a different directive for each vhost. which means one open file handle per virtual host per apache process. which is exactly what we were trying to avoid. there's no benefit in doing this...in fact, you're much worse off than just specifying the ErrorLog filename because you not only have num_vhost * num_apache_children file handles, you also have the same number of "mytrickyprog" instances running. each of which takes up memory and CPU time, and has at least 4 file handles open itself (stdin, stdout, stderr, and the error log file) the whole point of this thread was how to reduce the number of file handles open, per apache process and on the entire system. craig -- craig sanders <[EMAIL PROTECTED]> (part time cyborg) -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: apache & log files
On Fri, Nov 05, 2004 at 09:40:28AM +0100, Francesco P. Lovergine wrote: > On Fri, Nov 05, 2004 at 09:09:16AM +1100, Craig Sanders wrote: > > > For ErrorLog you can pipe to a suitable program which does the same. > > > > but this doesn't. unless apache has added this feature since i last looked > > into this (about six months ago) the "suitable program" has no way of > > separating the error logs for each virtual host, because it's just STDERR with > > no vhost prefix on each line. > > > > ErrorLog "| mytrickyprog www.mydomain.com" Then what's the use? The point was to avoid having ErrorLog directives per vhost, since that increases the number of open files. -- EARTH smog | bricks AIR -- mud -- FIRE soda water | tequila WATER -- with thanks to fortune -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: apache & log files
On Fri, Nov 05, 2004 at 09:54:18AM +0100, Wouter Verhelst,,, wrote: > > > > ErrorLog "| mytrickyprog www.mydomain.com" > > Then what's the use? The point was to avoid having ErrorLog directives > per vhost, since that increases the number of open files. > I'm not an expert of apache internals, but I think it does not open a permanent pipe file for each directive, doesn't it? -- Francesco P. Lovergine -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: apache & log files
On Fri, Nov 05, 2004 at 01:35:28AM +1100, Russell Coker wrote: > My clftools package allows you to split and mangle the log files if you have > Apache configured for a single log file... > Uhm, not found in current sid archive -- Francesco P. Lovergine -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: apache & log files
On Fri, Nov 05, 2004 at 09:09:16AM +1100, Craig Sanders wrote: > > For ErrorLog you can pipe to a suitable program which does the same. > > but this doesn't. unless apache has added this feature since i last looked > into this (about six months ago) the "suitable program" has no way of > separating the error logs for each virtual host, because it's just STDERR with > no vhost prefix on each line. > ErrorLog "| mytrickyprog www.mydomain.com" where mytrickyprog simply echos stdin on the right per-domain file or the same log file with the right prefix for each line. Of course you need a different directive for each vhost. A silly implementation would use a perl/sh script to do the thing; a more efficient solution would use a nice tiny C program to do the same. Both are left as exercise to the reader :) -- Francesco P. Lovergine -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: apache & log files
On 04.11.2004 23:09, Craig Sanders wrote: For ErrorLog you can pipe to a suitable program which does the same. but this doesn't. unless apache has added this feature since i last looked into this (about six months ago) the "suitable program" has no way of separating the error logs for each virtual host, because it's just STDERR with no vhost prefix on each line. does somebody know if this thing was changed in apache 2 ? greets KoS -- Martin Kos +41-76-384-93-33 http://kos.liSay NO to HTML in mail ICQ# 13556143 Proudly running Debian GNU/Linux -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: apache & log files
On Thu, Nov 04, 2004 at 11:19:22AM +0100, Francesco P. Lovergine wrote: > I personally prefer a single CustomLog file with a suitable domain > prefix for every domain. That allows a nice grepping to extract > information and avoid resources wasting. yes, this works. > For ErrorLog you can pipe to a suitable program which does the same. but this doesn't. unless apache has added this feature since i last looked into this (about six months ago) the "suitable program" has no way of separating the error logs for each virtual host, because it's just STDERR with no vhost prefix on each line. craig -- craig sanders <[EMAIL PROTECTED]> (part time cyborg) -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: apache & log files
On Thursday 04 November 2004 09:11, Marek Podmaka <[EMAIL PROTECTED]> wrote: > I have apache 1.3 webserver hosting about 150 domains (more than 400 > virtual hosts). Now I have separate error log for each domain My clftools package allows you to split and mangle the log files if you have Apache configured for a single log file... -- http://www.coker.com.au/selinux/ My NSA Security Enhanced Linux packages http://www.coker.com.au/bonnie++/ Bonnie++ hard drive benchmark http://www.coker.com.au/postal/Postal SMTP/POP benchmark http://www.coker.com.au/~russell/ My home page -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: apache & log files
Instead of setting ulimit in an init script, you can set it in the file /etc/security/limits.conf like this: www-data hardnofile 4096 www-data softnofile 8192 I'm not sure what hard and soft really mean, they're discribed as: # can have the two values: #- "soft" for enforcing the soft limits #- "hard" for enforcing hard limits but I don't know what kind of a difference they make. /proc/sys/fs/file-max is 26214 on that system, I'm not sure if it's influenced by the contents of limits.conf ... On another system it's even higher though I've not changed anything. Anybody knows what changes this value? Sönke Am Mittwoch, 3. November 2004 23:42 schrieb Craig Sanders: > On Wed, Nov 03, 2004 at 11:11:13PM +0100, Marek Podmaka wrote: > > I have apache 1.3 webserver hosting about 150 domains (more than 400 > > virtual hosts). Now I have separate error log for each domain > > (something.sk) and separate combined log for each virtual host (for > > example www.abcq.sk and new.abcq.sk). This has many positives for > > me: easy to find some data related to each virtual host and that I > > can make seaparate statistics for each virtual host. I use awstats. > > And now the bad side - the number of open files in each apache > > process is more than 500 just for these log files. It's no problem > > for now, but with more domains in future it will hit the 1024 per > > process limit of open files. > > > > And now the questions :) > > 1) Where does that 1024 open files limit come from? Is it somewhere > > configurable? > > edit /etc/init.d/apache and add a line like "ulimit -n 4096" > > "-n" is the maximum number of open file descriptors per process. use a > value that's about twice as much as you think you'll need. > > you also need to set /proc/sys/fs/file-max to some suitably high value > (again, calculate how many you think you'll need and double it). this can > be set in /etc/sysctl.conf > > > Or do you think it's totally bad idea to have such number of log > > files? > > until recently, i ran a web server with about 600 virtual hosts on it, each > with its own access.log and error.log files. > > with 200 max apache children, that worked out as up to about 240,000 (600 x > 200 x 2) file handles opened by apache processes for logging at any given > time. this was on a dual p3-866 with 512MB RAM. it worked. > > it bothered me a little that it wasn't really scalable and that eventually > i'd have to do something about logging. i had some ideas on what to do, > but was limited by the fact that i wanted to have separate error.log files > for each virtual host. overall, though, my attitude was "it aint broke, so > don't fix it". > > this wouldn't be a problem if apache could be configured to prefix each > error.log line with the virtual host's domain name..then you could have > a single pipe to an error logging script which wrote the error lines to the > right file, same as you can do for the access.log. > > but apache can't be configured to do that, and i never bothered looking at > the source to see how easy it would be to hack it in, so that means you > either have a shared error.log for all vhosts or you put up with having > lots of open file handles. i chose the latter, and occasionally increased > both "ulimit -n" and /proc/sys/fs/file-max as requiredi never did run > into any limit. > > > craig > > -- > craig sanders <[EMAIL PROTECTED]> (part time cyborg) -- Sönke von Stamm System Administration [EMAIL PROTECTED] Tel.: +49 511 - 35 77 19 - 29 Fax.: +49 511 - 35 77 19 - 19 Imedia Gesellschaft für neue Medien mbH Limmerstr. 46 | 30451 Hannover
Re: apache & log files
On Wed, Nov 03, 2004 at 11:11:13PM +0100, Marek Podmaka wrote: > Hello, > > I have apache 1.3 webserver hosting about 150 domains (more than 400 > virtual hosts). Now I have separate error log for each domain > (something.sk) and separate combined log for each virtual host (for > example www.abcq.sk and new.abcq.sk). This has many positives for > me: easy to find some data related to each virtual host and that I > can make seaparate statistics for each virtual host. I use awstats. > And now the bad side - the number of open files in each apache > process is more than 500 just for these log files. It's no problem > for now, but with more domains in future it will hit the 1024 per > process limit of open files. > > 3) Or you maybe use some other solution to this problem? > I personally prefer a single CustomLog file with a suitable domain prefix for every domain. That allows a nice grepping to extract information and avoid resources wasting. For ErrorLog you can pipe to a suitable program which does the same. -- Francesco P. Lovergine -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: apache & log files
On Wed, Nov 03, 2004 at 11:11:13PM +0100, Marek Podmaka wrote: > I have apache 1.3 webserver hosting about 150 domains (more than 400 > virtual hosts). Now I have separate error log for each domain > (something.sk) and separate combined log for each virtual host (for > example www.abcq.sk and new.abcq.sk). This has many positives for > me: easy to find some data related to each virtual host and that I > can make seaparate statistics for each virtual host. I use awstats. > And now the bad side - the number of open files in each apache > process is more than 500 just for these log files. It's no problem > for now, but with more domains in future it will hit the 1024 per > process limit of open files. > > And now the questions :) > 1) Where does that 1024 open files limit come from? Is it somewhere > configurable? edit /etc/init.d/apache and add a line like "ulimit -n 4096" "-n" is the maximum number of open file descriptors per process. use a value that's about twice as much as you think you'll need. you also need to set /proc/sys/fs/file-max to some suitably high value (again, calculate how many you think you'll need and double it). this can be set in /etc/sysctl.conf > Or do you think it's totally bad idea to have such number of log > files? until recently, i ran a web server with about 600 virtual hosts on it, each with its own access.log and error.log files. with 200 max apache children, that worked out as up to about 240,000 (600 x 200 x 2) file handles opened by apache processes for logging at any given time. this was on a dual p3-866 with 512MB RAM. it worked. it bothered me a little that it wasn't really scalable and that eventually i'd have to do something about logging. i had some ideas on what to do, but was limited by the fact that i wanted to have separate error.log files for each virtual host. overall, though, my attitude was "it aint broke, so don't fix it". this wouldn't be a problem if apache could be configured to prefix each error.log line with the virtual host's domain name..then you could have a single pipe to an error logging script which wrote the error lines to the right file, same as you can do for the access.log. but apache can't be configured to do that, and i never bothered looking at the source to see how easy it would be to hack it in, so that means you either have a shared error.log for all vhosts or you put up with having lots of open file handles. i chose the latter, and occasionally increased both "ulimit -n" and /proc/sys/fs/file-max as requiredi never did run into any limit. craig -- craig sanders <[EMAIL PROTECTED]> (part time cyborg) -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Re: apache & log files
Marek Podmaka wrote: Hello, I have apache 1.3 webserver hosting about 150 domains (more than 400 virtual hosts). Now I have separate error log for each domain (something.sk) and separate combined log for each virtual host (for example www.abcq.sk and new.abcq.sk). This has many positives for me: easy to find some data related to each virtual host and that I can make seaparate statistics for each virtual host. I use awstats. And now the bad side - the number of open files in each apache process is more than 500 just for these log files. It's no problem for now, but with more domains in future it will hit the 1024 per process limit of open files. And now the questions :) 1) Where does that 1024 open files limit come from? Is it somewhere configurable? Or do you think it's totally bad idea to have such number of log files? 2) Is it possible to have separate statistics for each virtual host with awstats and only one log file? It would be nice if certain virtual hosts could have different configuration of awstats. Now I use my script which checks all virtual hosts in mysql (from where also the apache config is refreshed every 15 mins) and creates config file for awstats if none exists. Then it runs awstats for each config (one after another of course). 3) Or you maybe use some other solution to this problem? Check out Cronolog. Piping your logs to that would at least get them into another process! Apache 2 config would be: CustomLog "| /path/to/cronolog /path/to/logfile_%Y_%m.log" combined Would that help you? Dunno if it scales to the levels you need - just an idea. Regards, Upayavira -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
apache & log files
Hello, I have apache 1.3 webserver hosting about 150 domains (more than 400 virtual hosts). Now I have separate error log for each domain (something.sk) and separate combined log for each virtual host (for example www.abcq.sk and new.abcq.sk). This has many positives for me: easy to find some data related to each virtual host and that I can make seaparate statistics for each virtual host. I use awstats. And now the bad side - the number of open files in each apache process is more than 500 just for these log files. It's no problem for now, but with more domains in future it will hit the 1024 per process limit of open files. And now the questions :) 1) Where does that 1024 open files limit come from? Is it somewhere configurable? Or do you think it's totally bad idea to have such number of log files? 2) Is it possible to have separate statistics for each virtual host with awstats and only one log file? It would be nice if certain virtual hosts could have different configuration of awstats. Now I use my script which checks all virtual hosts in mysql (from where also the apache config is refreshed every 15 mins) and creates config file for awstats if none exists. Then it runs awstats for each config (one after another of course). 3) Or you maybe use some other solution to this problem? thanks -- bYE, Marki -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
