Re: how would you scale a cake app?
On 12/27/06, bgmill <[EMAIL PROTECTED]> wrote: Thanks for your reply Firstly, I don't claim to be a hardware guru, and my estimations are purely based on brief conversations with the techs who run the servers. So with that in mind... all the machines are dual xeon 2.8GHz with 2Gb of memory. With this setup, I would again speculate that you would would get much more than 1.2 mill and still remain much below the load average of 3 (I keep 3 as the alarm point) May Santa give your site get many more hits so that we can test out this hypothesis :) The decision to use 3+1 straight off the bat was mainly due to the need for handling rapid expansion -- the site only launched 2 weeks ago. Add to the mix the need for video transcoding and an element of the unknown in this respect meant it was easier to start with more than less. How is the MySQL setup on this? is it like what I speculated in a message later on 1 Master + 3 Slaves ? Cheers Tarique -- = PHP Applications for E-Biz: http://www.sanisoft.com Cheesecake-Photoblog: http://cheesecake-photoblog.org = --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Cake PHP" group. To post to this group, send email to cake-php@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/cake-php?hl=en -~--~~~~--~~--~--~---
Re: how would you scale a cake app?
On 12/27/06, Langdon Stevenson <[EMAIL PROTECTED]> wrote: Thanks for that info, much appreciated. One more question though, when you say "3+1" do you mean a cluster of 3 database servers and one web server? I think he said 3 web servers + 1 MySQL, MySQL server is probably a master and handles all the writes and then replicated back on the 3 web servers which are used for subsequent reads. but I am just speculating here Cheers Tarique -- = PHP Applications for E-Biz: http://www.sanisoft.com Cheesecake-Photoblog: http://cheesecake-photoblog.org = --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Cake PHP" group. To post to this group, send email to cake-php@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/cake-php?hl=en -~--~~~~--~~--~--~---
Re: how would you scale a cake app?
Hi bgmill The decision to use 3+1 straight off the bat was mainly due to the need for handling rapid expansion Thanks for that info, much appreciated. One more question though, when you say "3+1" do you mean a cluster of 3 database servers and one web server? Regards, Langdon --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Cake PHP" group. To post to this group, send email to cake-php@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/cake-php?hl=en -~--~~~~--~~--~--~---
Re: how would you scale a cake app?
Dr. Tarique Sani wrote: Not to sound like a prick but 400,000 page views are not much for a typical 3+1 configuration. Given your estimate of higher performance you say you will reach about 1.2 million page views on the same setup which again makes me want to know your hardware configuration. Firstly, I don't claim to be a hardware guru, and my estimations are purely based on brief conversations with the techs who run the servers. So with that in mind... all the machines are dual xeon 2.8GHz with 2Gb of memory. But like I said, the machines have not flinched even at peak traffic so far (I've rarely seen them above 0.1), whether we could handle 1.2mil or 10mil page views a day on the current hardware was just speculation on my end. The decision to use 3+1 straight off the bat was mainly due to the need for handling rapid expansion -- the site only launched 2 weeks ago. Add to the mix the need for video transcoding and an element of the unknown in this respect meant it was easier to start with more than less. To put things from my perspective - we have conventional php application which manages 12 to 15 million page views on a 4+2 cluster. Yes the hardware on this is awesome! Sounds like a nice setup you have there :) Regards. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Cake PHP" group. To post to this group, send email to cake-php@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/cake-php?hl=en -~--~~~~--~~--~--~---
Re: how would you scale a cake app?
That's a dream! http://www.mytechsky.com On 12 24 , 3 10 , "keymaster" <[EMAIL PROTECTED]> wrote: .. just dreaming, but.. If one wanted to use cake to build an app which is scalable to the extreme, ie. flickr.com or yourtube.com or myspace.com, how would you do it? Would you just separate out the db server and replicate the rest on identical nodes in a cluster? Would you break up the app into view servers, controller servers, and model servers? Imagine you were tasked with building one of these systems, what system architecture would you have? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Cake PHP" group. To post to this group, send email to cake-php@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/cake-php?hl=en -~--~~~~--~~--~--~---
Re: how would you scale a cake app?
Hi Woodsman Caching, and lots of it, and DB clustering/replication are definitely the quickest fixes. Caching (both on the server and the client side) are certainly looking good for my current project. I haven't used Cake's view caching yet, but am definitely going to look into it. I can well imagine the benefits One minor point is to avoid using Cake's multi join SQL, which (I believe) can literally add thousands of queries when a single one would do. I have spent some time optimising queries just with unbinding and have seen substantial improvements in DB performance (like half or one third of the processing time), so I can well believe the potential for improvement there. And remember, hardware is less expensive than development time, so don't sacrifice code maintainability and clarity for tiny performance increases, as you'll pay a lot more in the long run. This has become one of my main mantra. Yes, hardware is very cheap, especially compared to coding labor. And there is also nothing stopping you optimising code over time as well to take full advantage of that hardware. Regards, Langdon --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Cake PHP" group. To post to this group, send email to cake-php@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/cake-php?hl=en -~--~~~~--~~--~--~---
Re: how would you scale a cake app?
On 12/24/06, bgmill <[EMAIL PROTECTED]> wrote: I'm running a cake app on 3 webservers (round robin dns) and 1 mysql server right now, traffic yesterday was around 200,000 uniques and 400,000 page views. What kind of hardware are you using? Not to sound like a prick but 400,000 page views are not much for a typical 3+1 configuration. Given your estimate of higher performance you say you will reach about 1.2 million page views on the same setup which again makes me want to know your hardware configuration. To put things from my perspective - we have conventional php application which manages 12 to 15 million page views on a 4+2 cluster. Yes the hardware on this is awesome! However this is a conventional php application which started life about 4 years ago, now maintaining it is an daunting task - adding functionality to it gives best of my programmers shivers :) and that is where a framework comes in... Looking forward to hear from you Thanks and merry Christmas Tarique -- = PHP Applications for E-Biz: http://www.sanisoft.com Coppermine Picture Gallery: http://coppermine.sf.net = --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Cake PHP" group. To post to this group, send email to cake-php@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/cake-php?hl=en -~--~~~~--~~--~--~---
Re: how would you scale a cake app?
Also, check if Cake doesn't get more than you need, and consider dynamic unbinding where you don't need as much info as you get using default models' options. -- Olexandr Melnyk, http://omelnyk.net/ --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Cake PHP" group. To post to this group, send email to cake-php@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/cake-php?hl=en -~--~~~~--~~--~--~---
Re: how would you scale a cake app?
Caching, and lots of it, and DB clustering/replication are definitely the quickest fixes. One minor point is to avoid using Cake's multi join SQL, which (I believe) can literally add thousands of queries when a single one would do. In such cases, don't be afraid to use query() instead of findAll - although examining your DB for low performance or often repeated queries is great for deciding if the benefits are relly worth the effort. And remember, hardware is less expensive than development time, so don't sacrifice code maintainability and clarity for tiny performance increases, as you'll pay a lot more in the long run. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Cake PHP" group. To post to this group, send email to cake-php@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/cake-php?hl=en -~--~~~~--~~--~--~---
Re: how would you scale a cake app?
Hi bgmill Sounds like you have a good system/site there. Can you tell me what sort of hardware it is running on? Regards, Langdon I'm running a cake app on 3 webservers (round robin dns) and 1 mysql server right now, traffic yesterday was around 200,000 uniques and 400,000 page views. Caching is being used on some of the heavier hit controllers but all in all, load is minimal across the board -- we could easily handle 2 or 3 times the traffic on the current setup. Scaling up in terms of adding webservers and db servers when required will be easy to accomplish. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Cake PHP" group. To post to this group, send email to cake-php@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/cake-php?hl=en -~--~~~~--~~--~--~---
Re: how would you scale a cake app?
I may well write an article for the bakery after the holiday season - are there any other high traffic sites known to be using cake? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Cake PHP" group. To post to this group, send email to cake-php@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/cake-php?hl=en -~--~~~~--~~--~--~---
RE: how would you scale a cake app?
I am talking about what the CakePHP framework offers you in terms of scalability, things like: 1. View caching: you can cache views as a whole or even specify which parts of a view should not be cached: http://manual.cakephp.org/chapter/view_cache 2. Model/Object caching: cache model queries or even complex objects returned by components, http://rossoft.wordpress.com/2006/03/15/cache-objects-component/ 3. Database optimization: it is easy, if you need it, to have a DB engine running your updates and another DB engine (that replicates previous record updates) to be used only for querying: http://groups-beta.google.com/group/cake-php/browse_thread/thread/58ea010f93 0fab6c MVC helps maintain your code, but a powerful framework can also help you boost performance by making code optimizations that are transparent to your application logic. That is one of the things CakePHP is great at. -MI --- Remember, smart coders answer ten questions for every question they ask. So be smart, be cool, and share your knowledge. BAKE ON! -Mensaje original- De: cake-php@googlegroups.com [mailto:[EMAIL PROTECTED] En nombre de keymaster Enviado el: Domingo, 24 de Diciembre de 2006 01:21 p.m. Para: Cake PHP Asunto: Re: how would you scale a cake app? I understand how the MVC separation and strict coding conventions make maintainability easier as the application gets larger (in terms of lines of code). This I know cake is outstanding at. But you seem to be alluding to a feature of cake I have not yet digested (pardon the pun), which is an aspect of cake which "allows you to easily develop your application thinking about scalability all the way". If you were referring to scalability from a traffic perspective, together with the associated underlying system architectural changes needed, can you please elaborate for us what you meant, Mariano ? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Cake PHP" group. To post to this group, send email to cake-php@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/cake-php?hl=en -~--~~~~--~~--~--~---
RE: how would you scale a cake app?
Your situation sounds like a great source for a potential bakery article, mainly for: 1. Understanding how CakePHP is behaving on a high-traffic/clustered server. 2. Giving a short tutorial on how you managed to add new servers and how it impacted on your application. Just a thought :) -MI --- Remember, smart coders answer ten questions for every question they ask. So be smart, be cool, and share your knowledge. BAKE ON! -Mensaje original- De: cake-php@googlegroups.com [mailto:[EMAIL PROTECTED] En nombre de bgmill Enviado el: Domingo, 24 de Diciembre de 2006 01:29 p.m. Para: Cake PHP Asunto: Re: how would you scale a cake app? I'm running a cake app on 3 webservers (round robin dns) and 1 mysql server right now, traffic yesterday was around 200,000 uniques and 400,000 page views. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Cake PHP" group. To post to this group, send email to cake-php@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/cake-php?hl=en -~--~~~~--~~--~--~---
Re: how would you scale a cake app?
On Dec 24, 4:01 pm, "Mariano Iglesias" <[EMAIL PROTECTED]> wrote: CakePHP is a framework that allows you to easily develop your application thinking about scalability all the way, so take advantage of that. I understand how the MVC separation and strict coding conventions make maintainability easier as the application gets larger (in terms of lines of code). This I know cake is outstanding at. But you seem to be alluding to a feature of cake I have not yet digested (pardon the pun), which is an aspect of cake which "allows you to easily develop your application thinking about scalability all the way". If you were referring to scalability from a traffic perspective, together with the associated underlying system architectural changes needed, can you please elaborate for us what you meant, Mariano ? Thanks. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Cake PHP" group. To post to this group, send email to cake-php@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/cake-php?hl=en -~--~~~~--~~--~--~---
Re: how would you scale a cake app?
I'm running a cake app on 3 webservers (round robin dns) and 1 mysql server right now, traffic yesterday was around 200,000 uniques and 400,000 page views. Caching is being used on some of the heavier hit controllers but all in all, load is minimal across the board -- we could easily handle 2 or 3 times the traffic on the current setup. Scaling up in terms of adding webservers and db servers when required will be easy to accomplish. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Cake PHP" group. To post to this group, send email to cake-php@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/cake-php?hl=en -~--~~~~--~~--~--~---
Re: how would you scale a cake app?
A simple way to let a webapp scale is to moved session's and db's data to cookies as much as possible. olivvv Nimrod A. Abing wrote: On 12/24/06, keymaster <[EMAIL PROTECTED]> wrote: .. just dreaming, but.. If one wanted to use cake to build an app which is scalable to the extreme, ie. flickr.com or yourtube.com or myspace.com, how would you do it? http://www.onjava.com/pub/a/onjava/2003/10/15/php_scalability.html http://www.oreillynet.com/onlamp/blog/2006/04/digg_phps_scalability_and_perf.html Would you just separate out the db server and replicate the rest on identical nodes in a cluster? Would you break up the app into view servers, controller servers, and model servers? That depends. Scalability is a difficult issue to handle if it is not considered during the initial application design stage. If you want to be able to scale up your application, you might as well decide on a strategy *now*. --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Cake PHP" group. To post to this group, send email to cake-php@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/cake-php?hl=en -~--~~~~--~~--~--~---
RE: how would you scale a cake app?
Check this out, it's not that high of an explanation but provides some useful tips on how to improve PHP performance: http://talks.php.net/show/hpp For some reason the creators of that presentation decided that they don't support IE so you'll need to use FF or something else to view it. In regards to your question I also think that scalability should be on the table right from the get-go. CakePHP is a framework that allows you to easily develop your application thinking about scalability all the way, so take advantage of that. I believe that in time, as your traffic hits high numbers, the first optimization you'll need is to allow the DB to be on its own clustered servers, so will your webserver, having giga connection amongst the two. Also if you are using MySQL as your backend server move CakePHP towards using mysqli extension instead of mysql, as it is faster. Also don't neglect the connectivity your server has to the outside world, some people improve their application performance and forget about the backbones where their application is running. Separating Models / Controllers: oh well I don't think you'll need such a thing. I would have DB on one side, CakePHP application on another side, work my way to PHP optimization on the CakePHP side and enable caching to places where it can be used. -MI --- Remember, smart coders answer ten questions for every question they ask. So be smart, be cool, and share your knowledge. BAKE ON! -Mensaje original- De: cake-php@googlegroups.com [mailto:[EMAIL PROTECTED] En nombre de Langdon Stevenson Enviado el: Domingo, 24 de Diciembre de 2006 05:33 a.m. Para: cake-php@googlegroups.com Asunto: Re: how would you scale a cake app? I would be very interested to hear anyone else's thoughts on this too. Does the development team have any recommendations about the best approach to scaling up a site? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Cake PHP" group. To post to this group, send email to cake-php@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/cake-php?hl=en -~--~~~~--~~--~--~---
Re: how would you scale a cake app?
Langdon Stevenson <[EMAIL PROTECTED]> wrote: > I think that I would likely just replicate the entire application > across multiple web servers, and hand out the requests in a round > robin to the web servers. Or maybe FastCGI's load balancing? > I would be very interested to hear anyone else's thoughts on this > too. Good use of caching would help prolong the time before having to scale up, as well as reducing hardware requirements. Cake's own caching is really neat, as are things like lighttpd's mod_cml or traditional options like squid. Remember that web servers are *extremely* good at serving static content! I should really add the general warning against premature optimisation here, but I'm sure you've heard it before. Best wishes, -- Chris Lamb, Cambridgeshire, UKGPG: 0x634F9A20 signature.asc Description: PGP signature
Re: how would you scale a cake app?
On 12/24/06, keymaster <[EMAIL PROTECTED]> wrote: .. just dreaming, but.. If one wanted to use cake to build an app which is scalable to the extreme, ie. flickr.com or yourtube.com or myspace.com, how would you do it? http://www.onjava.com/pub/a/onjava/2003/10/15/php_scalability.html http://www.oreillynet.com/onlamp/blog/2006/04/digg_phps_scalability_and_perf.html Would you just separate out the db server and replicate the rest on identical nodes in a cluster? Would you break up the app into view servers, controller servers, and model servers? That depends. Scalability is a difficult issue to handle if it is not considered during the initial application design stage. If you want to be able to scale up your application, you might as well decide on a strategy *now*. -- _nimrod_a_abing_ [?] http://abing.gotdns.com --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Cake PHP" group. To post to this group, send email to cake-php@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/cake-php?hl=en -~--~~~~--~~--~--~---
Re: how would you scale a cake app?
It's an interesting question. I am working on an application that (if it works out for the client) may grow to support thousands of users with requirements for high availability. If it ever grows that far, then I expect that we will need to redevelop at least some parts of the application . For me, the first step (to a small cluster) would be to hive off the database onto a cluster of three to five servers. DB access on this system is heavy, so that will be the likely first bottleneck. After that, I am not sure. I think that I would likely just replicate the entire application across multiple web servers, and hand out the requests in a round robin to the web servers. I would be very interested to hear anyone else's thoughts on this too. Does the development team have any recommendations about the best approach to scaling up a site? Regards, Langdon keymaster wrote: .. just dreaming, but.. If one wanted to use cake to build an app which is scalable to the extreme, ie. flickr.com or yourtube.com or myspace.com, how would you do it? Would you just separate out the db server and replicate the rest on identical nodes in a cluster? Would you break up the app into view servers, controller servers, and model servers? Imagine you were tasked with building one of these systems, what system architecture would you have? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Cake PHP" group. To post to this group, send email to cake-php@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/cake-php?hl=en -~--~~~~--~~--~--~---
how would you scale a cake app?
.. just dreaming, but.. If one wanted to use cake to build an app which is scalable to the extreme, ie. flickr.com or yourtube.com or myspace.com, how would you do it? Would you just separate out the db server and replicate the rest on identical nodes in a cluster? Would you break up the app into view servers, controller servers, and model servers? Imagine you were tasked with building one of these systems, what system architecture would you have? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups "Cake PHP" group. To post to this group, send email to cake-php@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/cake-php?hl=en -~--~~~~--~~--~--~---
