apache process size
Hi folks, I'm trying to make my apache processes as small as possible, and one thing I find lacking in optimisation guides is actual real world process sizes. So... how big is yours? ;) Mine start at around 60MB, jump to 80MB when my app is initialised, and can peak at 120MB during their lifetime. For reference I tried making the tiniest possible one with everything stripped out, and that was still 40MB - is this normal? cheers John
Re: apache process size
On Fri, Feb 22, 2008 at 09:52:30AM +, John ORourke wrote: > Hi folks, > > I'm trying to make my apache processes as small as possible, and one > thing I find lacking in optimisation guides is actual real world process > sizes. > > So... how big is yours? ;) > > Mine start at around 60MB, jump to 80MB when my app is initialised, and > can peak at 120MB during their lifetime. For reference I tried making > the tiniest possible one with everything stripped out, and that was > still 40MB - is this normal? > Your question does not contain enough information. What is your basis the 60MB, 80MB, 120MB, etc, numbers? Regards, -Roberto -- Roberto C. Sánchez http://people.connexer.com/~roberto http://www.connexer.com signature.asc Description: Digital signature
A question about wiring up filehandles for direct dump to HTTP pipe
Is it possible? Hi all, Here is my situation. I would like a mod-perl script (a Mason script actually, but from the mod_perl API POV it's the same thing) to act as a gatekeeper for file access -- regulate access by application-specific ACLs, do the accounting on download stats, etc; but then, once the housekeeping is done, I would like it to take the file (quite likely a very large video file) and dump it directly to the HTTP pipe. The obvious solution -- a sysread/syswrite loop -- will probably be quite a bit slower than simply having Apache read the filehandle directly. Given that I expect these files to be large and many, I would really prefer to avoid the sysread/syswrite loop solution. Is there any way, to, uhhh, tell Apache programatically to simply slurp up the file from an open filehandle, or to grab a given file from disk? Doing a redirect to the real location is less than ideal precisely because that will open a door to the circumvention of this accounting script, a door which will be rather hard to close. Any ideas or alternative solutions are welcome. Thanks in advance. -- Victor Danilchenko Senior Software Engineer, AskOnline.net [EMAIL PROTECTED] - 617-273-0119
Re: A question about wiring up filehandles for direct dump to HTTP pipe
Victor Danilchenko wrote: > Is there any way, to, uhhh, tell Apache programatically to simply > slurp up the file from an open filehandle I could be wrong, but I doubt you can pass a Perl file handle to a C program that is not Perl aware (like Apache). > or to grab a given file from > disk? Doing a redirect to the real location is less than ideal precisely > because that will open a door to the circumvention of this accounting > script, a door which will be rather hard to close. What about an internal redirect? It won't expose where the door is and you can make it such that access to that URL is forbidden to any request not coming from your machine. But the best approach seems to convert your code into an Apache Authen or Authz handler to do all the book keeping. Then simply let Apache continue with the processing after that. You'd probably have to ditch Mason for that piece of the code, but it's pretty straight forward. -- Michael Peters Plus Three, LP
Re: A question about wiring up filehandles for direct dump to HTTP pipe
On Fri 22 Feb 2008, Victor Danilchenko wrote: > or to grab a given file from disk? Perhaps $r->sendfile($filename)? Defined in Apache2::RequestIO. Torsten
Re: A question about wiring up filehandles for direct dump to HTTP pipe
Michael Peters wrote: Victor Danilchenko wrote: Is there any way, to, uhhh, tell Apache programatically to simply slurp up the file from an open filehandle I could be wrong, but I doubt you can pass a Perl file handle to a C program that is not Perl aware (like Apache). I am hoping you are wrong, but I am not expecting it. To be honest, I kinda figured that there would be no easy and perfect way around it. or to grab a given file from disk? Doing a redirect to the real location is less than ideal precisely because that will open a door to the circumvention of this accounting script, a door which will be rather hard to close. What about an internal redirect? It won't expose where the door is and you can make it such that access to that URL is forbidden to any request not coming from your machine. Huh, I didn't even realize this was possible. $r->internal-redirect, eh? That sounds just like what I need -- coupled with some form of hotlink protection, it should do the job. Thanks a lot. But the best approach seems to convert your code into an Apache Authen or Authz handler to do all the book keeping. Then simply let Apache continue with the processing after that. You'd probably have to ditch Mason for that piece of the code, but it's pretty straight forward. If you are familiar with Apache internals, I am sure it is. i am not. Unfortunately, this happens to be one of those "the roof is on fire" deals -- I have to not only solve this problem, but solve it *fast*. Many thanks for your immensely helpful advice. I think the internal redirect is close enough to what I need to be useful. once again, thanks. -- Victor Danilchenko Senior Software Engineer, AskOnline.net [EMAIL PROTECTED] - 617-273-0119
Re: A question about wiring up filehandles for direct dump to HTTP pipe
Torsten Foertsch wrote: On Fri 22 Feb 2008, Victor Danilchenko wrote: or to grab a given file from disk? Perhaps $r->sendfile($filename)? Defined in Apache2::RequestIO. There is precious little documentation on it, so I will have to experiment with it a little, but this sounds like it might do the trick. Thanks. -- Victor Danilchenko Senior Software Engineer, AskOnline.net [EMAIL PROTECTED] - 617-273-0119
Re: apache process size
On Fri, 2008-02-22 at 09:52 +, John ORourke wrote: > Hi folks, > > I'm trying to make my apache processes as small as possible, and one > thing I find lacking in optimisation guides is actual real world process > sizes. > > So... how big is yours? ;) I'm running on 32-bit linux 2.6, using apache 2.2.4, mod_perl 2.03, running this site : http://announce.jpress.co.uk A typical httpd process looks like this (as output by the smem.pl script at http://www.contrib.andrew.cmu.edu/~bmaurer/memory/smem.pl ) VMSIZE: 136832 kb RSS: 42236 kb total 25796 kb shared 64 kb private clean 16376 kb private dirty > > Mine start at around 60MB, jump to 80MB when my app is initialised, and > can peak at 120MB during their lifetime. For reference I tried making > the tiniest possible one with everything stripped out, and that was > still 40MB - is this normal? Depends what you're measuring - it sounds like you're including VM in there as well. Clint
Re: apache process size
Roberto C. Sánchez wrote: On Fri, Feb 22, 2008 at 09:52:30AM +, John ORourke wrote: Mine start at around 60MB, jump to 80MB when my app is initialised, and can peak at 120MB during their lifetime. For reference I tried making the tiniest possible one with everything stripped out, and that was still 40MB - is this normal? Your question does not contain enough information. What is your basis the 60MB, 80MB, 120MB, etc, numbers? Might be showing off my lack of unix admin knowledge here but using 'top', it's the value of the 'virt' column - ie. the biggest possible process size if the whole thing was resident. I've been using it to work out how many clients I should limit apache to, but looking at it the 'res' figure never seems to get anywhere near the 'virt' figure. Time to rewrite my assumptions I think! John
Re: A question about wiring up filehandles for direct dump to HTTP pipe
This solution ended up being perfect, and SIMPLE. I just had to sprinkle a little Mime::FileInfo::Magic over the request, to get the content-type to work right, and POOF! it worked. Just like that. My entire wrapper code (without the accounting, ACL, and somesuch stuff) is under 10 lines, and half of it is constructing the 404 responder. Torsten Foertsch wrote: On Fri 22 Feb 2008, Victor Danilchenko wrote: or to grab a given file from disk? Perhaps $r->sendfile($filename)? Defined in Apache2::RequestIO. -- Victor Danilchenko Senior Software Engineer, AskOnline.net [EMAIL PROTECTED] - 617-273-0119
Amazon
I've heard from a few reputable sources that Amazon is looking to drop mod_perl, and push into another technology ( which I've also head is likely to be Java ). They have a HUGE deployment on mp, and have been my prime "Um, not enterprise? Hello, AMAZON." repsonse. I know that people 'in the know' can't comment on record... however I'm wondering if anyone with second-hand intel has heard , and can share : a- what the bottleneck / scaling issues were b- were these due to apache/mod_perl, or because of the framework implementation... and this is just an opportunity to switch technologies while they switch frameworks ( ie, is MP to blame, or Template Toolkit... and MP is taking a fall since its going to be a PITA to ditch TT ) c- what the hell the financial projections were on doing this. are they looking to save on hardware scaling, developer scaling, is the codebase just unmanageable? this would be a costly transition // Jonathan Vanasco w. http://findmeon.com/user/jvanasco e. [EMAIL PROTECTED] | Founder/CEO - FindMeOn, Inc. | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - | FindMeOn.com - The cure for Multiple Web Personality Disorder | Privacy Minded Web Identity Management and 3D Social Networking | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Re: Amazon
Doesn't sound right to me that they would jettison the existing deployment unless it was really dysfunctional, which doesn't seem to be the case...I'm curious what the real story is. d Jonathan Vanasco wrote: I've heard from a few reputable sources that Amazon is looking to drop mod_perl, and push into another technology ( which I've also head is likely to be Java ). They have a HUGE deployment on mp, and have been my prime "Um, not enterprise? Hello, AMAZON." repsonse. I know that people 'in the know' can't comment on record... however I'm wondering if anyone with second-hand intel has heard , and can share : a- what the bottleneck / scaling issues were b- were these due to apache/mod_perl, or because of the framework implementation... and this is just an opportunity to switch technologies while they switch frameworks ( ie, is MP to blame, or Template Toolkit... and MP is taking a fall since its going to be a PITA to ditch TT ) c- what the hell the financial projections were on doing this. are they looking to save on hardware scaling, developer scaling, is the codebase just unmanageable? this would be a costly transition // Jonathan Vanasco w. http://findmeon.com/user/jvanasco e. [EMAIL PROTECTED] | Founder/CEO - FindMeOn, Inc. | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - | FindMeOn.com - The cure for Multiple Web Personality Disorder | Privacy Minded Web Identity Management and 3D Social Networking | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Re: apache process size
John ORourke wrote: Might be showing off my lack of unix admin knowledge here but using 'top', it's the value of the 'virt' column - ie. the biggest possible process size if the whole thing was resident. I've been using it to work out how many clients I should limit apache to, but looking at it the 'res' figure never seems to get anywhere near the 'virt' figure. Time to rewrite my assumptions I think! I get about 90 MB "virt" (70 MB "res") per process, running some pretty heavy custom mod_perl templating software with some AJAX stuff that uses (gasp) Python as a back-end via mod_python, along with DBI (DBD::Pg). Sometimes, the load average hits 6.0, but everything always seems to load quickly enough. :) Linux 2.6.17.13, Debian, 2 x Pentium IV 3.0 GHz. Colin
Re: Amazon
Can be anything really, though I admit I'm not in the know. Sometimes it's simply a business decision: perhaps moving development off-shore to companies that are full of Java/ .NET people? Can't find enough competent mp developers? Jonathan Vanasco wrote: I've heard from a few reputable sources that Amazon is looking to drop mod_perl, and push into another technology ( which I've also head is likely to be Java ). They have a HUGE deployment on mp, and have been my prime "Um, not enterprise? Hello, AMAZON." repsonse. I know that people 'in the know' can't comment on record... however I'm wondering if anyone with second-hand intel has heard , and can share : a- what the bottleneck / scaling issues were b- were these due to apache/mod_perl, or because of the framework implementation... and this is just an opportunity to switch technologies while they switch frameworks ( ie, is MP to blame, or Template Toolkit... and MP is taking a fall since its going to be a PITA to ditch TT ) c- what the hell the financial projections were on doing this. are they looking to save on hardware scaling, developer scaling, is the codebase just unmanageable? this would be a costly transition // Jonathan Vanasco w. http://findmeon.com/user/jvanasco e. [EMAIL PROTECTED] | Founder/CEO - FindMeOn, Inc. | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - | FindMeOn.com - The cure for Multiple Web Personality Disorder | Privacy Minded Web Identity Management and 3D Social Networking | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Re: Amazon
On 23.02.2008 02:59 Jonathan Vanasco wrote: I've heard from a few reputable sources that Amazon is looking to drop mod_perl, and push into another technology ( which I've also head is likely to be Java ). They have a HUGE deployment on mp, and have been my prime "Um, not enterprise? Hello, AMAZON." repsonse. In a recent issue of the German iX magazine there was a report about a similar migration (www.mobile.de I think). There the reasons were, if I remember correctly: - scaling issues. The underlying design was such that not every component of the site was easily scalable. - some of the original developers left, so it was difficult to maintain the original codebase - JAVA community with lots of help and high quality tools and libraries. - the new solution used less resources (they saved a few hundred servers) and still ran faster. For me this article showed some important points - often Perl projects start as quick and dirty solutions that end in a maintainability nightmare. Of course it is possible to write code in Perl that meets the best standards, but often Perl is used when this is not the main focus but when instant results are needed. - Perl usage is declining. I read some statistics from O'Reilly and they showed that Perl book sales are going down. A few years ago the 'P' in LAMP clearly was 'Perl', now it is 'PHP' in most cases. Developers tend to go (even if slowly) where the money is. - Even the great CPAN repository doesn't seem to match what is available for JAVA. I could see this myself when I needed to write a Webservice client against a JAVA server with SOAP-Lite or when I tried to connect to a CORBA based application. Just my 0.02 Euro -Michael I know that people 'in the know' can't comment on record... however I'm wondering if anyone with second-hand intel has heard , and can share : a- what the bottleneck / scaling issues were b- were these due to apache/mod_perl, or because of the framework implementation... and this is just an opportunity to switch technologies while they switch frameworks ( ie, is MP to blame, or Template Toolkit... and MP is taking a fall since its going to be a PITA to ditch TT ) c- what the hell the financial projections were on doing this. are they looking to save on hardware scaling, developer scaling, is the codebase just unmanageable? this would be a costly transition // Jonathan Vanasco w. http://findmeon.com/user/jvanasco e. [EMAIL PROTECTED] | Founder/CEO - FindMeOn, Inc. | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - | FindMeOn.com - The cure for Multiple Web Personality Disorder | Privacy Minded Web Identity Management and 3D Social Networking | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Re: Amazon
Hi there, I normally don't borge into a thread like this since I am not an active user of mod_perl. But have you noticed how cheap memory is these days? You can set up a dual 64 bit processers server with 4Gig bytes of memory and set up 10 VM on the same machine. It is extremely fast and efficient, and much easy to manage. All these VM virtualization technology is driving the cost of ownership down tremendously. And then each application runs faster because each has its own VM and a fix partition of memory. Why make software scalable while you can scale using cheap hardware and virtualization? Then you can deploy simply and primitive systems built by outsourced global development centers on these new hardware platforms. The playground may be changing very rapidly for developing scalable software. I also used `curl -I` to look at what webserver amazon is running on. It's neither IIS nor Apache because the response didn't provide any specific server ID. They may be tomcat. Can someone tell what system components are they using? Oh well. - Clement -Original Message- >From: Jonathan Vanasco <[EMAIL PROTECTED]> >Sent: Feb 22, 2008 5:59 PM >To: modperl >Subject: Amazon > >I've heard from a few reputable sources that Amazon is looking to >drop mod_perl, and push into another technology ( which I've also >head is likely to be Java ). > >They have a HUGE deployment on mp, and have been my prime "Um, not >enterprise? Hello, AMAZON." repsonse. > >I know that people 'in the know' can't comment on record... however >I'm wondering if anyone with second-hand intel has heard , and can >share : > > a- what the bottleneck / scaling issues were > b- were these due to apache/mod_perl, or because of the framework >implementation... and this is just an opportunity to switch >technologies while they switch frameworks ( ie, is MP to blame, or >Template Toolkit... and MP is taking a fall since its going to be a >PITA to ditch TT ) > c- what the hell the financial projections were on doing this. are >they looking to save on hardware scaling, developer scaling, is the >codebase just unmanageable? this would be a costly transition > > > > >// Jonathan Vanasco > >w. http://findmeon.com/user/jvanasco >e. [EMAIL PROTECTED] > >| Founder/CEO - FindMeOn, Inc. >| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - >- - - - - - - - - - - - - - - - - - - >| FindMeOn.com - The cure for Multiple Web Personality Disorder >| Privacy Minded Web Identity Management and 3D Social Networking >| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - >- - - - - - - - - - - - - - - - - - - >
Re: Amazon
On Sat, 23 Feb 2008, Foo JH wrote: > Can be anything really, though I admit I'm not in the know. Sometimes > it's simply a business decision: perhaps moving development off-shore to > companies that are full of Java/ .NET people? Can't find enough > competent mp developers? It is certainly hard to find good mp developers. However, mod_perl has always been for the tech people; unless in a technocracy, java would always be the choice for the class of management. Choosing java for better performance would certainly be a joke. If a java solution fails, it would be an industry-standard failure, backside covered. :-) Cheers, Jie
Re: Amazon
On Fri, 22 Feb 2008, c chan wrote: > Date: Fri, 22 Feb 2008 23:21:34 -0800 (GMT-08:00) > From: c chan <[EMAIL PROTECTED]> > To: Jonathan Vanasco <[EMAIL PROTECTED]>, > modperl > Subject: Re: Amazon > > Hi there, > > I normally don't borge into a thread like this since I am not an active user > of mod_perl. > > But have you noticed how cheap memory is these days? You can set up a dual 64 > bit processers server with 4Gig bytes of memory and set up 10 VM on the same > machine. It is extremely fast and efficient, and much easy to manage. > > All these VM virtualization technology is driving the cost of ownership down > tremendously. And then each application runs faster because each has its own > VM and a fix partition of memory. Why make software scalable while you can > scale using cheap hardware and virtualization? Then you can deploy simply and > primitive systems built by outsourced global development centers on these new > hardware platforms. I don't really want to start an argument that is off-topic anyway, but with all the overhead and quirks of virtualisation, performance is actually weird (a new way of describing performance). Regards, Jie
Re: Amazon
Jie Gao wrote: and quirks of virtualisation, performance is actually weird (a new way of describing performance). I tend to achieve an average 0.6 wierds, although in the summer it can reach 0.72 wierds. John
