Re: [Catalyst] Re: memory usage of mod_perl process
Just a couple of comments on this topic. If you're using apache2.2 then mod_cache is available. This can be used to cache the result of a request either in memory or disk. Ideal for the situations where you want to cache the front page of your site every minute. http://httpd.apache.org/docs/2.2/mod/mod_cache.html Also if you want to offload all your static resources to a non-Cat server then the simplest way of doing that is to simply stick a full url in the HTML. img src=http://static.example.com/images/logo.png; / Use a config option so you can vary it: img src=[% c.config.static_path %]/images/logo.png / Using ConfigLoader you can have different devel/live values for static_path too. Carl ___ List: Catalyst@lists.rawmode.org Listinfo: http://lists.rawmode.org/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.rawmode.org/ Dev site: http://dev.catalyst.perl.org/
RE: [Catalyst] Re: memory usage of mod_perl process
Jeffrey Ng wrote: so are you saying in general catalyst is pretty heavy, and its not for site with many concurrent users?... No, not at all. It all depends on what your application does, how many web screens, how many peak concurrent users and equivalent requests/second, how many discrete users, how many data feeds in/out, data volumes, network transfer volumes. Your limiting factor might be CPU, database I/O or network bandwidth. Different architectural models are best for different cases. There is a cost equation between using a powerful framework that increases robustness, maintainability and speed of development versus how many servers you need. Say a server costs you $100 per month to rent and each Catalyst server maxes at 30 requests per second but you get a peak load of 90 reqs/sec. You would need to load balance your app across 3 servers. There would be an initial configuration cost and then you'd be paying $200 p.m. = $2400 p.a. more for the two extra servers. Let's say a developer, including overhead, costs you $1000 per day. If you move away from Catalyst to a lighter framework and it costs you more than 2.5 developer days extra then you're losing money. Catalyst has a lot of tried and tested pluggable modules and if you end up re-writing any of those you're almost certainly spending money. what other light weight approach would you go for? First thing, set up proper caching (either Squid or a lightweight Apache) and use a tool like Fiddler to examine the HTTP request Cache headers to make sure it's working properly. That's a huge saving. Then do profiling and benchmarking to determine exactly what the bottlenecks are before making any changes. If you're doing a lot of dynamic Ajax stuff with complex SQL, there might be a case for hand-tuning that part of the app. Look at whether you can pre-calculate and cache expensive queries or change the app design to allow this. If necessary, you could write a separate lightweight app to serve those requests. I didn't find a suitable lightweight mod_perl framework so wrote one using CGI::Session, DBIx::Class, Template toolkit. It's not that difficult. If you would like some details please feel free to email off-list or call me. However, that was an atypical case and as soon as you start needing internationalization, RSS feeds etc. I think you're better off with Catalyst. You said you need to serve JS from the same server as the heavyweight app. Assuming the JS pages are static or can be precalculated and stored in a static dir, and you have the JS below /js and your app below /app in the URI namespace: On the same server run two Apaches (or one Apache + one Fastcgi) 1) Lightweight proxy. Build it yourself from source, turn off all unnecessary options and shared libraries except mod_proxy/mod_rewrite. Use rewrite rules to proxy /app/. on to heavy server, e.g. based on an Apache 1 setup I have that's live: RewriteEngine On RewriteRule^/app/(.*$) http://localhost:2000/app/$1 [P] ProxyPassReverse / http://localhost:2000/ RewriteRule^proxy:.* - [F] The /js and /images get served by the proxy server (which has a 1MB memory footprint per child) 2) Heavy mod_perl server listening at localhost:2000. Maps /app on to your handler (40-100MB memory per child). Say you had to handle the Ajax part separately. With this model you can add another proxy entry that routes, say, /ajax/ on to another Apache server on the same host (e.g. localhost:2001) or on another host altogether (e.g. http://ajax1.zorpia.com http://ajax1.zorpia.com/ ). If you need to scale more, use DNS round robin to a set of front end proxies like this and route requests to a farm of backend servers using a shared database server. I'd be surprised if you couldn't get decent performance out of a well-designed Catalyst app. There are businesses listed on the Catalyst pages that offer consultancy help if you need further pointers. As a business I used Shadowcat Systems last year and received good technical design advice. Regards, Peter Edwards www.dragonstaff.com http://www.dragonstaff.com/ ~ Business IT Consultancy ___ List: Catalyst@lists.rawmode.org Listinfo: http://lists.rawmode.org/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.rawmode.org/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Re: memory usage of mod_perl process
On 2/9/07, Peter Edwards [EMAIL PROTECTED] wrote: Are you hitting the database very hard? Complex joins? Are your pages very complex? No, it's a fairly simple backend processor that mostly handles XML transactions for a Flash frontend. There are some XHTML admin pages generated with Template::Toolkit (and so are slower) but they are rarely hit. For the database a typical run hits a session table, user table, writes a log table and reads data from a couple of other tables. No complex joins. The app handles user registration, some league tables, polls, competition entries and users submitting images that are approved and shown on the site. In this case I knew there was the potential for a huge number of users so went for a lightweight approach that could easily be tuned further. For a more complex app with fewer peak users I'd have used Catalyst. what other light weight approach would you go for? so are you saying in general catalyst is pretty heavy, and its not for site with many concurrent users?... Regards, Peter www.dragonstaff.com ___ List: Catalyst@lists.rawmode.org Listinfo: http://lists.rawmode.org/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.rawmode.org/ Dev site: http://dev.catalyst.perl.org/ -- Jeffrey Ng CEO, Zorpia.com ___ List: Catalyst@lists.rawmode.org Listinfo: http://lists.rawmode.org/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.rawmode.org/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Re: memory usage of mod_perl process
On 2/9/07, Oliver Gorwits [EMAIL PROTECTED] wrote: Jeffrey Ng wrote: I am the coworker of Fayland who posted the topic memory usage of mod_perl process. Our company has invested quite some time on migrating our perl code to catalyst (more than half year of time by 7 programmers). However, I am starting to worry that moving to catalyst is causing too much overhead for our processes and thus severely slowing down our site. Take a good, hard look at what's coming out of your web servers. Install a Squid farm in front of the servers and let them cache anything suitable... yes we are thinking about that now. however, we are not sure how to update the cache we we want to refresh the page. i guess we will adjust the http header when we want to flush it? how does squid know if our page is refreshed when it has already cached our page though?... letting Apache fork just to serve an image can be really damaging to performance. Double check the HTTP headers on all outgoing pages include proper caching instructions for Squid so it'll work properly. yah, i know that. our catalyst apache servers are only for serving dynamic pages. however, there are some ajax javascript files we have to host on the exact same subdomain to work. i still feel that those javascript files are wasting our apache resources. for ajax javascript like that, how should i redirect the load to other servers while maintaining the same subdomain in the location bar? i tried to use mod_rewrite, and seems like its even slower than serving the file itself! Even whole store-front pages from Catalyst sites can usually be cached well, because users don't have state at that point, and perhaps most hits are to your homepage (so remember to let Squid know your homepage is cacheable, in the headers). Remember mod_perl can be quite leaky as well, so kill off the child processes after they've served a few requests. Oh, and switch off hyperthreading if you have dual P4/Xeon boxes, and use PreFork MPM. There's more info in the slides from this LPW talk: http://london.pm.org/lpw/talks/2006/dave_hodgkinson-mod_perl_server_farm_architecture_done_right.pdf HTH, oliver. ___ List: Catalyst@lists.rawmode.org Listinfo: http://lists.rawmode.org/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.rawmode.org/ Dev site: http://dev.catalyst.perl.org/ -- Jeffrey Ng CEO, Zorpia.com ___ List: Catalyst@lists.rawmode.org Listinfo: http://lists.rawmode.org/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.rawmode.org/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Re: memory usage of mod_perl process
On 2/9/07, Jeffrey Ng [EMAIL PROTECTED] wrote: here's the result on a live server: total used free sharedbuffers cached Mem: 415097226540161496956 0 275481313344 -/+ buffers/cache:13131242837848 Swap: 8385912 69208378992 [...] hi how can you tell our apache is just using 1.3 GB? what king of files does the rest of it cache?... Look at the second line of output from free (or free -m, for megabytes). It's telling you that you're only using 1313124 bytes for applications, and the rest is buffers and cache. For more info on buffers and cache, try googling it: http://www.google.com/search?hl=ensafe=offq=linux+free+buffers+cachebtnG=Search The first hit looks pretty good: http://gentoo-wiki.com/FAQ_Linux_Memory_Management can we run devel::DProf with mod_perl catatlyst? or do we have to use apache::Dprof? Apache::DProf should work, but I think people often profile catalyst apps on the test server using Devel::DProf. There's probably something on the catalyst site about this if you search for it. - Perrin ___ List: Catalyst@lists.rawmode.org Listinfo: http://lists.rawmode.org/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.rawmode.org/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Re: memory usage of mod_perl process
On Fri February 9 2007 12:53 pm, Jeffrey Ng wrote: On 2/9/07, Oliver Gorwits [EMAIL PROTECTED] wrote: Jeffrey Ng wrote: I am the coworker of Fayland who posted the topic memory usage of mod_perl process. Our company has invested quite some time on migrating our perl code to catalyst (more than half year of time by 7 programmers). However, I am starting to worry that moving to catalyst is causing too much overhead for our processes and thus severely slowing down our site. Take a good, hard look at what's coming out of your web servers. Install a Squid farm in front of the servers and let them cache anything suitable... yes we are thinking about that now. however, we are not sure how to update the cache we we want to refresh the page. i guess we will adjust the http header when we want to flush it? how does squid know if our page is refreshed when it has already cached our page though?... You can call the PURGE command on squid. You must allow this in the conf and put in place the proper acl to allow it. I believe you can call squid with the following options to force a purge of the cache of a given page. -m PURGE http://mywebsite.com/index.html We are not using catalyst in production yet so we don't have squid running in front of any catalyst apps. We do however have it running in front of some python/zope/plone sites. There are plugins that handle the purging directly for us. We just provide the url to purge to a method that handles the purge when needed. letting Apache fork just to serve an image can be really damaging to performance. Double check the HTTP headers on all outgoing pages include proper caching instructions for Squid so it'll work properly. yah, i know that. our catalyst apache servers are only for serving dynamic pages. however, there are some ajax javascript files we have to host on the exact same subdomain to work. i still feel that those javascript files are wasting our apache resources. for ajax javascript like that, how should i redirect the load to other servers while maintaining the same subdomain in the location bar? i tried to use mod_rewrite, and seems like its even slower than serving the file itself! Even whole store-front pages from Catalyst sites can usually be cached well, because users don't have state at that point, and perhaps most hits are to your homepage (so remember to let Squid know your homepage is cacheable, in the headers). Remember mod_perl can be quite leaky as well, so kill off the child processes after they've served a few requests. Oh, and switch off hyperthreading if you have dual P4/Xeon boxes, and use PreFork MPM. There's more info in the slides from this LPW talk: http://london.pm.org/lpw/talks/2006/dave_hodgkinson-mod_perl_server_farm_ architecture_done_right.pdf HTH, oliver. ___ List: Catalyst@lists.rawmode.org Listinfo: http://lists.rawmode.org/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.rawmode.org/ Dev site: http://dev.catalyst.perl.org/ ___ List: Catalyst@lists.rawmode.org Listinfo: http://lists.rawmode.org/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.rawmode.org/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Re: memory usage of mod_perl process
Tony Losey [EMAIL PROTECTED] wrote on 02/09/2007 04:05:17 PM: On Fri February 9 2007 12:53 pm, Jeffrey Ng wrote: On 2/9/07, Oliver Gorwits [EMAIL PROTECTED] wrote: Jeffrey Ng wrote: I am the coworker of Fayland who posted the topic memory usage of mod_perl process. Our company has invested quite some time on migrating our perl code to catalyst (more than half year of time by 7 programmers). However, I am starting to worry that moving to catalyst is causing too much overhead for our processes and thus severely slowing down our site. Take a good, hard look at what's coming out of your web servers. Install a Squid farm in front of the servers and let them cache anything suitable... yes we are thinking about that now. however, we are not sure how to update the cache we we want to refresh the page. i guess we will adjust the http header when we want to flush it? how does squid know if our page is refreshed when it has already cached our page though?... You can call the PURGE command on squid. You must allow this in the conf and put in place the proper acl to allow it. I believe you can call squid with the following options to force a purge of the cache of a given page. -m PURGE http://mywebsite.com/index.html We are not using catalyst in production yet so we don't have squid running in front of any catalyst apps. We do however have it running in front of some python/zope/plone sites. There are plugins that handle the purging directly for us. We just provide the url to purge to a method that handles the purge when needed. Or set the Time to live on the page headers to a reasonable amount of time -- lets say 1 minute. This basically serves cache for 99.9% of the hits and only goes to the app server once per minute for a refresh. No perge needed, just tune the cache time to be the maximum you feel comfortable with. Hitting the app server from your frontend every once in a while for static or cacheable pages is no huge deal -- don't sacrifice administrative costs for one or two hits a minute. ___ List: Catalyst@lists.rawmode.org Listinfo: http://lists.rawmode.org/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.rawmode.org/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Re: memory usage of mod_perl process
On 2/10/07, Tony Losey [EMAIL PROTECTED] wrote: On Fri February 9 2007 12:53 pm, Jeffrey Ng wrote: On 2/9/07, Oliver Gorwits [EMAIL PROTECTED] wrote: Jeffrey Ng wrote: I am the coworker of Fayland who posted the topic memory usage of mod_perl process. Our company has invested quite some time on migrating our perl code to catalyst (more than half year of time by 7 programmers). However, I am starting to worry that moving to catalyst is causing too much overhead for our processes and thus severely slowing down our site. Take a good, hard look at what's coming out of your web servers. Install a Squid farm in front of the servers and let them cache anything suitable... yes we are thinking about that now. however, we are not sure how to update the cache we we want to refresh the page. i guess we will adjust the http header when we want to flush it? how does squid know if our page is refreshed when it has already cached our page though?... You can call the PURGE command on squid. You must allow this in the conf and put in place the proper acl to allow it. I believe you can call squid with the following options to force a purge of the cache of a given page. -m PURGE http://mywebsite.com/index.html We are not using catalyst in production yet so we don't have squid running in front of any catalyst apps. We do however have it running in front of some python/zope/plone sites. There are plugins that handle the purging directly for us. We just provide the url to purge to a method that handles the purge when needed. right! i just googled and found http://search.cpan.org/~sock/Net-Squid-Purge-0.1/ that cpan module should be able to purge the cached. letting Apache fork just to serve an image can be really damaging to performance. Double check the HTTP headers on all outgoing pages include proper caching instructions for Squid so it'll work properly. yah, i know that. our catalyst apache servers are only for serving dynamic pages. however, there are some ajax javascript files we have to host on the exact same subdomain to work. i still feel that those javascript files are wasting our apache resources. for ajax javascript like that, how should i redirect the load to other servers while maintaining the same subdomain in the location bar? i tried to use mod_rewrite, and seems like its even slower than serving the file itself! Even whole store-front pages from Catalyst sites can usually be cached well, because users don't have state at that point, and perhaps most hits are to your homepage (so remember to let Squid know your homepage is cacheable, in the headers). Remember mod_perl can be quite leaky as well, so kill off the child processes after they've served a few requests. Oh, and switch off hyperthreading if you have dual P4/Xeon boxes, and use PreFork MPM. There's more info in the slides from this LPW talk: http://london.pm.org/lpw/talks/2006/dave_hodgkinson-mod_perl_server_farm_ architecture_done_right.pdf HTH, oliver. ___ List: Catalyst@lists.rawmode.org Listinfo: http://lists.rawmode.org/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.rawmode.org/ Dev site: http://dev.catalyst.perl.org/ ___ List: Catalyst@lists.rawmode.org Listinfo: http://lists.rawmode.org/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.rawmode.org/ Dev site: http://dev.catalyst.perl.org/ -- Jeffrey Ng CEO, Zorpia.com ___ List: Catalyst@lists.rawmode.org Listinfo: http://lists.rawmode.org/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.rawmode.org/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Re: memory usage of mod_perl process
On 2/8/07, Jeffrey Ng [EMAIL PROTECTED] wrote: I have read practical mod_perl. I tried to preload many of our modules in startup.pl. But the shared memory value doesnt change at all. Also, doesnt 5.7M shared memory usage sound too small comparing to the 92.6M total size? You're not looking at the right thing there. This is not actual shared memory, but rather copy-on-write sharing. It will never show in your SHARE section. On versions of Linux with a 2.6 kernel, you can use the techniques in Apache::SizeLimit to get a sesne of how much sharing is going. This doesn't work with older kernels. A reliable way to tell if your sharing is improved is to start up apache without preloading and look at the available memory from free, and then put in the preloading and restart apache and check it again. You should have more free memory after preloading, allowing you to run more processes. - Perrin ___ List: Catalyst@lists.rawmode.org Listinfo: http://lists.rawmode.org/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.rawmode.org/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Re: memory usage of mod_perl process
Jeffrey Ng wrote: I am the coworker of Fayland who posted the topic memory usage of mod_perl process. Our company has invested quite some time on migrating our perl code to catalyst (more than half year of time by 7 programmers). However, I am starting to worry that moving to catalyst is causing too much overhead for our processes and thus severely slowing down our site. I remember the memory usage of each process before catalyst was around 25Mb. And now the new code is using around 90MB as fayland said. During peak hour, our servers were able to handle 30+ requests per sec before catalyst upgrade. Now we are only able to handle around 10 requests per sec. As a result, the response time of the site is much slower after the upgrade. it takes at least 2 sec for the browser to start responding during non-peak hours. Here's the memory stats for a typical process: Memory Usage (in bytes): Size : 97107968 (92.6M) Share :6012928 ( 5.7M) VSize : 97107968 (92.6M) RSS: 85131264 (81.2M) Memory Segments Usage (in bytes): Text : 253952 ( 248K) Shlib : 0 ( 0K) Data : 78696448 (75.1M) Stack : 0 ( 0K) I have read practical mod_perl. I tried to preload many of our modules in startup.pl. But the shared memory value doesnt change at all. Also, doesnt 5.7M shared memory usage sound too small comparing to the 92.6M total size? I wonder how much is your shared memory? Is the memory overhead of catalyst huge? Is catalyst inherently slow? I too have found that my Catalyst apps are huge compared to previous mod_perl or FastCGI apps. I had been using Class::DBI and a hand-rolled framework, but now use Catalyst and DBIx::Class. The difference is at least a factor of 2. It's not been a problem for me at my current traffic levels, and I'm more than happy to trade some RAM for the joy the design and coding has become compared to my previous situation. But I do wonder where all the RAM is going. I especially concur that not much is being shared which is curious given that the modules are preloaded by mod_perl. I had done some profiling to measure how much RAM a dummy Cat app took vs a dummy DBIC schema, maybe I'll dig that up and send it to the list. Best, Brian ___ List: Catalyst@lists.rawmode.org Listinfo: http://lists.rawmode.org/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.rawmode.org/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Re: memory usage of mod_perl process
On 2/9/07, Perrin Harkins [EMAIL PROTECTED] wrote: On 2/8/07, Jeffrey Ng [EMAIL PROTECTED] wrote: I have read practical mod_perl. I tried to preload many of our modules in startup.pl. But the shared memory value doesnt change at all. Also, doesnt 5.7M shared memory usage sound too small comparing to the 92.6M total size? You're not looking at the right thing there. This is not actual shared memory, but rather copy-on-write sharing. It will never show in your SHARE section. On versions of Linux with a 2.6 kernel, you can use the techniques in Apache::SizeLimit to get a sesne of how much sharing is going. This doesn't work with older kernels. A reliable way to tell if your sharing is improved is to start up apache without preloading and look at the available memory from free, and then put in the preloading and restart apache and check it again. You should have more free memory after preloading, allowing you to run more processes. when i run this free test, should i run it on the live server, or on a test server with single process apache mode? here's the result on a live server: total used free sharedbuffers cached Mem: 415097226540161496956 0 275481313344 -/+ buffers/cache:13131242837848 Swap: 8385912 69208378992 and here's the top results on one of the server: top - 14:21:55 up 144 days, 9:30, 0 users, load average: 19.30, 18.65, 12.27 Tasks: 87 total, 1 running, 86 sleeping, 0 stopped, 0 zombie Cpu(s): 69.6% us, 1.3% sy, 0.0% ni, 27.7% id, 0.0% wa, 0.2% hi, 1.2% si Mem: 4150972k total, 2623968k used, 1527004k free,27380k buffers Swap: 8385912k total, 6920k used, 8378992k free, 1312832k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 26343 apache15 0 97196 83m 6236 S 10.0 2.1 0:08.69 httpd 26340 apache15 0 94740 81m 5896 S 7.8 2.0 0:07.62 httpd 26315 apache15 0 95552 81m 5896 S 7.4 2.0 0:08.31 httpd 26326 apache15 0 94836 81m 5868 S 7.4 2.0 0:08.17 httpd 26371 apache15 0 95496 81m 5872 S 7.4 2.0 0:08.21 httpd I set the maxserver as 60. Our CPU load is extremely high. But the actual RAM usage is not high at all. we are using only 260 gig out of 400 gig. From our top results, does it look like the memory usage of our processes is the bottleneck? - Perrin ___ List: Catalyst@lists.rawmode.org Listinfo: http://lists.rawmode.org/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.rawmode.org/ Dev site: http://dev.catalyst.perl.org/ -- Jeffrey Ng CEO, Zorpia.com ___ List: Catalyst@lists.rawmode.org Listinfo: http://lists.rawmode.org/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.rawmode.org/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Re: memory usage of mod_perl process
Jeffrey Ng wrote: I am the coworker of Fayland who posted the topic memory usage of mod_perl process. Our company has invested quite some time on migrating our perl code to catalyst (more than half year of time by 7 programmers). However, I am starting to worry that moving to catalyst is causing too much overhead for our processes and thus severely slowing down our site. Take a good, hard look at what's coming out of your web servers. Install a Squid farm in front of the servers and let them cache anything suitable... letting Apache fork just to serve an image can be really damaging to performance. Double check the HTTP headers on all outgoing pages include proper caching instructions for Squid so it'll work properly. Even whole store-front pages from Catalyst sites can usually be cached well, because users don't have state at that point, and perhaps most hits are to your homepage (so remember to let Squid know your homepage is cacheable, in the headers). Remember mod_perl can be quite leaky as well, so kill off the child processes after they've served a few requests. Oh, and switch off hyperthreading if you have dual P4/Xeon boxes, and use PreFork MPM. There's more info in the slides from this LPW talk: http://london.pm.org/lpw/talks/2006/dave_hodgkinson-mod_perl_server_farm_architecture_done_right.pdf HTH, oliver. ___ List: Catalyst@lists.rawmode.org Listinfo: http://lists.rawmode.org/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.rawmode.org/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Re: memory usage of mod_perl process
On 2/8/07, Jeffrey Ng [EMAIL PROTECTED] wrote: when i run this free test, should i run it on the live server, or on a test server with single process apache mode? I wouldn't mess with things on your live server, but you want to run in normal mode, not single-process. here's the result on a live server: total used free sharedbuffers cached Mem: 415097226540161496956 0 275481313344 -/+ buffers/cache:13131242837848 Swap: 8385912 69208378992 [...] Our CPU load is extremely high. But the actual RAM usage is not high at all. we are using only 260 gig out of 400 gig. Not even that much. You're using about 1.3 GB and the rest of that 2.6 is just being used to cache files. You should be able to run a lot more apache processes on this machine, or maybe give some RAM to whatever else needs it on there, like a database. From our top results, does it look like the memory usage of our processes is the bottleneck? No, you have tons of free RAM. It sounds like something else is slowing you down. You have a lot of CPU being used on there (about 60%) which might mean you need to profile your code for CPU bottlenecks. You may also be waiting on database queries, which you can find out with Devel::DProf or the DBI profiler. - Perrin ___ List: Catalyst@lists.rawmode.org Listinfo: http://lists.rawmode.org/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.rawmode.org/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Re: memory usage of mod_perl process
On Thu, Feb 08, 2007 at 10:37:24PM -, Peter Edwards wrote: For comparison, I'm using DBIx::Class across 15 tables and a really simple hand-rolled MVC (not Cat) for a medium volume site. It gives 'top' lines like: PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 21495 apache15 0 203m 43m 6188 S 0.0 4.5 0:01.46 httpd This can handle over 300 requests/sec with 1GB memory on an x86_64 single processor. 300/sec from the application server? That's impressive. I'm happy when I see a see a tenth of that. Are you hitting the database very hard? Complex joins? Are your pages very complex? -- Bill Moseley [EMAIL PROTECTED] ___ List: Catalyst@lists.rawmode.org Listinfo: http://lists.rawmode.org/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.rawmode.org/ Dev site: http://dev.catalyst.perl.org/
