Re: [web2py] Re: Discussion: Why is web2py bad for large deployments?
Thanks mariano, it's very usefull. 2010/11/15 Mariano Reingart reing...@gmail.com Django is not faster than web2py for real world applications, this is my benchmark that proves it (in this particular case, YMMV): django: Pycon Argentina 2009 site: (based on pycon-tech, pycon 2010 us talk propossal/ site) ab -n 100 http://pycon.sistemasagiles.com.ar/2009/about/ Requests per second:23.13 [#/sec] (mean) (it is the original server, then it has been moved to ar.pycon.org) web2py: PyDay Buenos Aires 2010 (based on web2conf, pycon 2010 us registration site) ab -n 100 http://www.pyday.com.ar/buenosbench/default/index Requests per second:24.53 [#/sec] (mean) web2py app it is a striped down version to include only pycon-tech about page exposed features (wiki, with menu and sponsor logos). web2py app doesn't uses cache and is not optimized (django one is, and it is very very very very too much complicated than the web2py app) exec vs import is not the most important issue (and I think it isn't relevant at all in a full cost vs performance analysis, ROI as you say in english). The biggest performance penality of web2py I've found is the filesystem access (looking for multiples .py files slow downs requests/sec). In this case (in my tests), each new db_*.py model has a performance lost of ~ 5% to 10% . Each directory scan has a 1% of performance loss. I'm working in some optimizations for compiled apps (avoiding directory scanning and saving codeobjects in memory improves a lot, but has some drawbacks as all cache systems). Maybe when we compile the app can make a sort of dispatcher that already knows where the files are, so having to list directories is not needed. Anyway, we shouldn't have this kind of discussions on web2py-developers list? When I have some time I'll fully publish this benchmark, just to demythologize this issue ;-) (the same for the table definition order, that is not really needed -there are easy workarounds-) Best regards, Mariano Reingart http://www.sistemasagiles.com.ar http://reingart.blogspot.com -- My blog: http://martin.tecnodoc.com.ar My portfolio *spanish*: http://www.tecnodoc.com.ar Checkout my last proyect instant-press: http://www.instant2press.com
Re: [web2py] Re: Discussion: Why is web2py bad for large deployments?
On 15 November 2010 01:38, Thadeus Burgess thade...@thadeusb.com wrote: Doesn't work with code completion, and will be a maintenance nightmare! I spent weeks trying different code organisation patterns with web2py including what you mentioned, everything I attempted just caused more problems and made the code less maintainable. Like I keep saying, it all comes down to programmers preference. I started programming with Java, therefore I like object oriented programming that Flask+SQLAlchemy uses. Do you use SQLAlchemy with web2py? How? Regards Johann -- May grace and peace be yours in abundance through the full knowledge of God and of Jesus our Lord! His divine power has given us everything we need for life and godliness through the full knowledge of the one who called us by his own glory and excellence. 2 Pet. 1:2b,3a
Re: [web2py] Re: Discussion: Why is web2py bad for large deployments?
On Mon, Nov 15, 2010 at 9:13 AM, Johann Spies johann.sp...@gmail.com wrote: Do you use SQLAlchemy with web2py? How? I think he meant Flask[1] + SQLAlchemy, not web2py + SQLAlchemy. [1] http://flask.pocoo.org/ -- Branko Vukelić bg.bra...@gmail.com stu...@brankovukelic.com Check out my blog: http://www.brankovukelic.com/ Check out my portfolio: http://www.flickr.com/photos/foxbunny/ Registered Linux user #438078 (http://counter.li.org/) I hang out on identi.ca: http://identi.ca/foxbunny Gimp Brushmakers Guild http://bit.ly/gbg-group
Re: [web2py] Re: Discussion: Why is web2py bad for large deployments?
Django is not faster than web2py for real world applications, this is my benchmark that proves it (in this particular case, YMMV): django: Pycon Argentina 2009 site: (based on pycon-tech, pycon 2010 us talk propossal/ site) ab -n 100 http://pycon.sistemasagiles.com.ar/2009/about/ Requests per second:23.13 [#/sec] (mean) (it is the original server, then it has been moved to ar.pycon.org) web2py: PyDay Buenos Aires 2010 (based on web2conf, pycon 2010 us registration site) ab -n 100 http://www.pyday.com.ar/buenosbench/default/index Requests per second:24.53 [#/sec] (mean) web2py app it is a striped down version to include only pycon-tech about page exposed features (wiki, with menu and sponsor logos). web2py app doesn't uses cache and is not optimized (django one is, and it is very very very very too much complicated than the web2py app) exec vs import is not the most important issue (and I think it isn't relevant at all in a full cost vs performance analysis, ROI as you say in english). The biggest performance penality of web2py I've found is the filesystem access (looking for multiples .py files slow downs requests/sec). In this case (in my tests), each new db_*.py model has a performance lost of ~ 5% to 10% . Each directory scan has a 1% of performance loss. I'm working in some optimizations for compiled apps (avoiding directory scanning and saving codeobjects in memory improves a lot, but has some drawbacks as all cache systems). Maybe when we compile the app can make a sort of dispatcher that already knows where the files are, so having to list directories is not needed. Anyway, we shouldn't have this kind of discussions on web2py-developers list? When I have some time I'll fully publish this benchmark, just to demythologize this issue ;-) (the same for the table definition order, that is not really needed -there are easy workarounds-) Best regards, Mariano Reingart http://www.sistemasagiles.com.ar http://reingart.blogspot.com
Re: [web2py] Re: Discussion: Why is web2py bad for large deployments?
On Sun, Nov 14, 2010 at 1:44 AM, b00m_chef r...@devshell.org wrote: Let me just say that the chance that your app will have the traffic of facebook or twitter or even livejournal (yes it is still alive) is so small, that you really don't need to worry about it too much at such an early stage. True. -- Branko Vukelić bg.bra...@gmail.com stu...@brankovukelic.com Check out my blog: http://www.brankovukelic.com/ Check out my portfolio: http://www.flickr.com/photos/foxbunny/ Registered Linux user #438078 (http://counter.li.org/) I hang out on identi.ca: http://identi.ca/foxbunny Gimp Brushmakers Guild http://bit.ly/gbg-group
[web2py] Re: Discussion: Why is web2py bad for large deployments?
+1 Totally agree with that. Speed is a luxary problem - concentrate on making an awesome app, not a fast one. On Nov 14, 1:44 am, b00m_chef r...@devshell.org wrote: Let me just say that the chance that your app will have the traffic of facebook or twitter or even livejournal (yes it is still alive) is so small, that you really don't need to worry about it too much at such an early stage. On Nov 13, 3:11 pm, Branko Vukelic bg.bra...@gmail.com wrote: On Sat, Nov 13, 2010 at 9:49 PM, Jason Brower encomp...@gmail.com wrote: These are encouraging. In large deployment I mean it in the most external sense of it. Facebook is big, myspace is big, those kinds of object were what I was aiming for. And For that kind of large deployment, you'll probably hit the limitation of the Python programming language sooner or later, too. Erlang seems to have become quite popular lately and even Facebook uses it for some backend stuff. The main reason is out-of-box concurrency, and great reliability under load which comes from supervisor-worker setup that is built-in. A good example of what this means is Yaws http server which is virtually impossible to kill even if you bombard it with a huge (as in MUCH more than http server X) number of concurrent requests. -- Branko Vukelić bg.bra...@gmail.com stu...@brankovukelic.com Check out my blog:http://www.brankovukelic.com/ Check out my portfolio:http://www.flickr.com/photos/foxbunny/ Registered Linux user #438078 (http://counter.li.org/) I hang out on identi.ca:http://identi.ca/foxbunny Gimp Brushmakers Guildhttp://bit.ly/gbg-group
Re: [web2py] Re: Discussion: Why is web2py bad for large deployments?
And I wonder.. in large large large deployments: I suppose we should use distributed deployment as GAE, EC2.. or run on our servers APPScale, typhoonae, eucalyptus. I know we are not talking about the same, but there's related with that. On Sun, Nov 14, 2010 at 11:38 AM, Branko Vukelic bg.bra...@gmail.comwrote: On Sun, Nov 14, 2010 at 1:44 AM, b00m_chef r...@devshell.org wrote: Let me just say that the chance that your app will have the traffic of facebook or twitter or even livejournal (yes it is still alive) is so small, that you really don't need to worry about it too much at such an early stage. True. -- Branko Vukelić bg.bra...@gmail.com stu...@brankovukelic.com Check out my blog: http://www.brankovukelic.com/ Check out my portfolio: http://www.flickr.com/photos/foxbunny/ Registered Linux user #438078 (http://counter.li.org/) I hang out on identi.ca: http://identi.ca/foxbunny Gimp Brushmakers Guild http://bit.ly/gbg-group
Re: [web2py] Re: Discussion: Why is web2py bad for large deployments?
A good advise, but, not always possible. I am considering to migrate an App in the range of 70-100 tables to web2py and there are tables that are like pillars, belong to several groups (customers, employees, projects,...) How we could manage this shared tables? Also sometimes its hard to determine an order for the tables to avoid referencing a still undefined table. There is another type of large applications, enterprise level with hundreds or thousands of tables. The approach of multiples apps sharing same database is useful, but how could we solve the division of tables in apps, is it better to define the model in one place and import the required tables on the apps? Jose. 2010/11/14 mdipierro mdipie...@cs.depaul.edu I agree that there are things we can improve in web2py. Yet, if a complex application have many pieces that are untangled, the import mechanism does not necessarily help you. The solution is trying to design components that are relatively independent. If you can do this, web2py does not bet in the way. For example, you have 100 tables functionally grouped in ~10 and all they need to share is authentication, you may want to separate the functions in 10 different web2py applications (all sharing one database, each defining only the tables it needs, and sharing one session cookie). This modular design solves lots of problems. I have heard of people who have developed complex applications in Django (which uses the import you suggest) and run into problems because they were using more ram than needed since all modules were always imported. In web2py controllers are executed only when called and if you make many small controllers you save lots of memory. Controllers than then import the modules they need and only those. Massimo On Nov 13, 7:28 pm, Thadeus Burgess thade...@thadeusb.com wrote: The biggest contributing factor is that web2py is executed and not import. The coder must always be sure to align objects and database in the correct order, including any auxiliary functions, class mappings, queries. He/she has to worry about the order of everything. Sometimes you can't have such order for complex systems. I think an import based system solves this in that the import statements declare the order instead of the physical location of the code doing that. -- Thadeus On Sat, Nov 13, 2010 at 2:49 PM, Jason Brower encomp...@gmail.com wrote: These are encouraging. In large deployment I mean it in the most external sense of it. Facebook is big, myspace is big, those kinds of object were what I was aiming for. And with Massimo's comment about the database size. I think your right, the framework does become less relavent. In what ways in a large codebase making web2py code messier and is there a solution to it in another framework? BR, J On Sat, 2010-11-13 at 11:48 -0600, Thadeus Burgess wrote: What is large deployment? Is it a large codebase that you must manage for an internal dashbaord, or just alot of users/database io that needs to scale out for worldwide access? If its the first case, web2py can get really complicated in dealing with lots of models and difficult to manage in an efficient manner. The larger your codebase the messier web2py apps will become. In the end, this would ultimately be up to the preferences of you and your team and what your willing to put up with. In the second case, framework hardly matters at that point. Disqus uses django, facebook uses php, reddit uses pylons, myspace uses coldfusion, microsoft uses asp, oracle uses java. Its always the database that becomes an issue regardless of programming language or web framework. -- Thadeus On Sat, Nov 13, 2010 at 10:53 AM, mdipierro mdipie...@cs.depaul.edu wrote: I agree with Villas. The larger the development the more the database becomes the bottleneck and the framework irrelevant. Massimo On Nov 13, 8:35 am, villas villa...@gmail.com wrote: Hi Jason I guess you have to define 'large deployment' first of all. Number of records and size of DB? Number of concurrent users? Large data model or number of forms etc? Number of servers -- or replication? Global coverage? In principle I don't think there's any reason why Web2py would be worse than other frameworks. Usually it is much better! As an example, I think deploying to the Google App Engine should be able to scale sufficiently for everything but extreme cases :) If you specify more about what you wish to achieve this group may be able to give more specific advice how best to organise your project. -D On Nov 13, 7:12 am, Jason Brower encomp...@gmail.com wrote: I love web2py and it's the only framework i feel i am fully capable to do or learn to do quickly. However, I remember see that this framework is intended for
Re: [web2py] Re: Discussion: Why is web2py bad for large deployments?
2010/11/14 José Ignacio Hurtado jose.i.hurt...@gmail.com: A good advise, but, not always possible. I am considering to migrate an App in the range of 70-100 tables to web2py and there are tables that are like I thought it would become obvious by now that you cannot _keep your existing architecture_ if your app hits the limits of scalability of _that_ architecture... -- Branko Vukelić bg.bra...@gmail.com stu...@brankovukelic.com Check out my blog: http://www.brankovukelic.com/ Check out my portfolio: http://www.flickr.com/photos/foxbunny/ Registered Linux user #438078 (http://counter.li.org/) I hang out on identi.ca: http://identi.ca/foxbunny Gimp Brushmakers Guild http://bit.ly/gbg-group
Re: [web2py] Re: Discussion: Why is web2py bad for large deployments?
On Sun, Nov 14, 2010 at 1:51 PM, Branko Vukelic bg.bra...@gmail.com wrote: 2010/11/14 José Ignacio Hurtado jose.i.hurt...@gmail.com: A good advise, but, not always possible. I am considering to migrate an App in the range of 70-100 tables to web2py and there are tables that are like I thought it would become obvious by now that you cannot _keep your existing architecture_ if your app hits the limits of scalability of _that_ architecture... Substitute architecture with design (or whatever you find more fitting) if my hobbyist terminology is not correct. -- Branko Vukelić bg.bra...@gmail.com stu...@brankovukelic.com Check out my blog: http://www.brankovukelic.com/ Check out my portfolio: http://www.flickr.com/photos/foxbunny/ Registered Linux user #438078 (http://counter.li.org/) I hang out on identi.ca: http://identi.ca/foxbunny Gimp Brushmakers Guild http://bit.ly/gbg-group
Re: [web2py] Re: Discussion: Why is web2py bad for large deployments?
I was trying to make a point and talk about boundaries of web2py and large applications, or better, applications with large models. I see the limits of scalability way far ;) 2010/11/14 Branko Vukelic bg.bra...@gmail.com On Sun, Nov 14, 2010 at 1:51 PM, Branko Vukelic bg.bra...@gmail.com wrote: 2010/11/14 José Ignacio Hurtado jose.i.hurt...@gmail.com: A good advise, but, not always possible. I am considering to migrate an App in the range of 70-100 tables to web2py and there are tables that are like I thought it would become obvious by now that you cannot _keep your existing architecture_ if your app hits the limits of scalability of _that_ architecture... Substitute architecture with design (or whatever you find more fitting) if my hobbyist terminology is not correct. -- Branko Vukelić bg.bra...@gmail.com stu...@brankovukelic.com Check out my blog: http://www.brankovukelic.com/ Check out my portfolio: http://www.flickr.com/photos/foxbunny/ Registered Linux user #438078 (http://counter.li.org/) I hang out on identi.ca: http://identi.ca/foxbunny Gimp Brushmakers Guild http://bit.ly/gbg-group -- Un saludo, José Ignacio Hurtado López
Re: [web2py] Re: Discussion: Why is web2py bad for large deployments?
A good advise, but, not always possible. I am considering to migrate an App in the range of 70-100 tables to web2py and there are tables that are like pillars, belong to several groups (customers, employees, projects,...) How we could manage this shared tables? Also sometimes its hard to determine an order for the tables to avoid referencing a still undefined table. There is another type of large applications, enterprise level with hundreds or thousands of tables. The approach of multiples apps sharing same database is useful, but how could we solve the division of tables in apps, is it better to define the model in one place and import the required tables on the apps? This is why I prefer Flask + SQLAlchemy. -- Thadeus 2010/11/14 José Ignacio Hurtado jose.i.hurt...@gmail.com A good advise, but, not always possible. I am considering to migrate an App in the range of 70-100 tables to web2py and there are tables that are like pillars, belong to several groups (customers, employees, projects,...) How we could manage this shared tables? Also sometimes its hard to determine an order for the tables to avoid referencing a still undefined table. There is another type of large applications, enterprise level with hundreds or thousands of tables. The approach of multiples apps sharing same database is useful, but how could we solve the division of tables in apps, is it better to define the model in one place and import the required tables on the apps? Jose. 2010/11/14 mdipierro mdipie...@cs.depaul.edu I agree that there are things we can improve in web2py. Yet, if a complex application have many pieces that are untangled, the import mechanism does not necessarily help you. The solution is trying to design components that are relatively independent. If you can do this, web2py does not bet in the way. For example, you have 100 tables functionally grouped in ~10 and all they need to share is authentication, you may want to separate the functions in 10 different web2py applications (all sharing one database, each defining only the tables it needs, and sharing one session cookie). This modular design solves lots of problems. I have heard of people who have developed complex applications in Django (which uses the import you suggest) and run into problems because they were using more ram than needed since all modules were always imported. In web2py controllers are executed only when called and if you make many small controllers you save lots of memory. Controllers than then import the modules they need and only those. Massimo On Nov 13, 7:28 pm, Thadeus Burgess thade...@thadeusb.com wrote: The biggest contributing factor is that web2py is executed and not import. The coder must always be sure to align objects and database in the correct order, including any auxiliary functions, class mappings, queries. He/she has to worry about the order of everything. Sometimes you can't have such order for complex systems. I think an import based system solves this in that the import statements declare the order instead of the physical location of the code doing that. -- Thadeus On Sat, Nov 13, 2010 at 2:49 PM, Jason Brower encomp...@gmail.com wrote: These are encouraging. In large deployment I mean it in the most external sense of it. Facebook is big, myspace is big, those kinds of object were what I was aiming for. And with Massimo's comment about the database size. I think your right, the framework does become less relavent. In what ways in a large codebase making web2py code messier and is there a solution to it in another framework? BR, J On Sat, 2010-11-13 at 11:48 -0600, Thadeus Burgess wrote: What is large deployment? Is it a large codebase that you must manage for an internal dashbaord, or just alot of users/database io that needs to scale out for worldwide access? If its the first case, web2py can get really complicated in dealing with lots of models and difficult to manage in an efficient manner. The larger your codebase the messier web2py apps will become. In the end, this would ultimately be up to the preferences of you and your team and what your willing to put up with. In the second case, framework hardly matters at that point. Disqus uses django, facebook uses php, reddit uses pylons, myspace uses coldfusion, microsoft uses asp, oracle uses java. Its always the database that becomes an issue regardless of programming language or web framework. -- Thadeus On Sat, Nov 13, 2010 at 10:53 AM, mdipierro mdipie...@cs.depaul.edu wrote: I agree with Villas. The larger the development the more the database becomes the bottleneck and the framework irrelevant. Massimo On Nov 13, 8:35 am, villas villa...@gmail.com wrote: Hi Jason I guess you have to define 'large deployment' first of all. Number of records and size
[web2py] Re: Discussion: Why is web2py bad for large deployments?
Nothing prevents you in web2py to create a modules/models/mymodel.py then import applications.yourapp/modules/models.mymodel.py and then call a function in there that defines the tables you need. And you can have more than one mymodel. This anyway will be easy if we add support for lazy tables and thread local globals. On Nov 14, 10:36 am, Thadeus Burgess thade...@thadeusb.com wrote: A good advise, but, not always possible. I am considering to migrate an App in the range of 70-100 tables to web2py and there are tables that are like pillars, belong to several groups (customers, employees, projects,...) How we could manage this shared tables? Also sometimes its hard to determine an order for the tables to avoid referencing a still undefined table. There is another type of large applications, enterprise level with hundreds or thousands of tables. The approach of multiples apps sharing same database is useful, but how could we solve the division of tables in apps, is it better to define the model in one place and import the required tables on the apps? This is why I prefer Flask + SQLAlchemy. -- Thadeus 2010/11/14 José Ignacio Hurtado jose.i.hurt...@gmail.com A good advise, but, not always possible. I am considering to migrate an App in the range of 70-100 tables to web2py and there are tables that are like pillars, belong to several groups (customers, employees, projects,...) How we could manage this shared tables? Also sometimes its hard to determine an order for the tables to avoid referencing a still undefined table. There is another type of large applications, enterprise level with hundreds or thousands of tables. The approach of multiples apps sharing same database is useful, but how could we solve the division of tables in apps, is it better to define the model in one place and import the required tables on the apps? Jose. 2010/11/14 mdipierro mdipie...@cs.depaul.edu I agree that there are things we can improve in web2py. Yet, if a complex application have many pieces that are untangled, the import mechanism does not necessarily help you. The solution is trying to design components that are relatively independent. If you can do this, web2py does not bet in the way. For example, you have 100 tables functionally grouped in ~10 and all they need to share is authentication, you may want to separate the functions in 10 different web2py applications (all sharing one database, each defining only the tables it needs, and sharing one session cookie). This modular design solves lots of problems. I have heard of people who have developed complex applications in Django (which uses the import you suggest) and run into problems because they were using more ram than needed since all modules were always imported. In web2py controllers are executed only when called and if you make many small controllers you save lots of memory. Controllers than then import the modules they need and only those. Massimo On Nov 13, 7:28 pm, Thadeus Burgess thade...@thadeusb.com wrote: The biggest contributing factor is that web2py is executed and not import. The coder must always be sure to align objects and database in the correct order, including any auxiliary functions, class mappings, queries. He/she has to worry about the order of everything. Sometimes you can't have such order for complex systems. I think an import based system solves this in that the import statements declare the order instead of the physical location of the code doing that. -- Thadeus On Sat, Nov 13, 2010 at 2:49 PM, Jason Brower encomp...@gmail.com wrote: These are encouraging. In large deployment I mean it in the most external sense of it. Facebook is big, myspace is big, those kinds of object were what I was aiming for. And with Massimo's comment about the database size. I think your right, the framework does become less relavent. In what ways in a large codebase making web2py code messier and is there a solution to it in another framework? BR, J On Sat, 2010-11-13 at 11:48 -0600, Thadeus Burgess wrote: What is large deployment? Is it a large codebase that you must manage for an internal dashbaord, or just alot of users/database io that needs to scale out for worldwide access? If its the first case, web2py can get really complicated in dealing with lots of models and difficult to manage in an efficient manner. The larger your codebase the messier web2py apps will become. In the end, this would ultimately be up to the preferences of you and your team and what your willing to put up with. In the second case, framework hardly matters at that point. Disqus uses django, facebook uses php, reddit uses pylons, myspace uses coldfusion, microsoft uses asp, oracle uses java. Its always the database that becomes
Re: [web2py] Re: Discussion: Why is web2py bad for large deployments?
Doesn't work with code completion, and will be a maintenance nightmare! I spent weeks trying different code organisation patterns with web2py including what you mentioned, everything I attempted just caused more problems and made the code less maintainable. Like I keep saying, it all comes down to programmers preference. I started programming with Java, therefore I like object oriented programming that Flask+SQLAlchemy uses. -- Thadeus On Sun, Nov 14, 2010 at 11:23 AM, mdipierro mdipie...@cs.depaul.edu wrote: Nothing prevents you in web2py to create a modules/models/mymodel.py then import applications.yourapp/modules/models.mymodel.py and then call a function in there that defines the tables you need. And you can have more than one mymodel. This anyway will be easy if we add support for lazy tables and thread local globals. On Nov 14, 10:36 am, Thadeus Burgess thade...@thadeusb.com wrote: A good advise, but, not always possible. I am considering to migrate an App in the range of 70-100 tables to web2py and there are tables that are like pillars, belong to several groups (customers, employees, projects,...) How we could manage this shared tables? Also sometimes its hard to determine an order for the tables to avoid referencing a still undefined table. There is another type of large applications, enterprise level with hundreds or thousands of tables. The approach of multiples apps sharing same database is useful, but how could we solve the division of tables in apps, is it better to define the model in one place and import the required tables on the apps? This is why I prefer Flask + SQLAlchemy. -- Thadeus 2010/11/14 José Ignacio Hurtado jose.i.hurt...@gmail.com A good advise, but, not always possible. I am considering to migrate an App in the range of 70-100 tables to web2py and there are tables that are like pillars, belong to several groups (customers, employees, projects,...) How we could manage this shared tables? Also sometimes its hard to determine an order for the tables to avoid referencing a still undefined table. There is another type of large applications, enterprise level with hundreds or thousands of tables. The approach of multiples apps sharing same database is useful, but how could we solve the division of tables in apps, is it better to define the model in one place and import the required tables on the apps? Jose. 2010/11/14 mdipierro mdipie...@cs.depaul.edu I agree that there are things we can improve in web2py. Yet, if a complex application have many pieces that are untangled, the import mechanism does not necessarily help you. The solution is trying to design components that are relatively independent. If you can do this, web2py does not bet in the way. For example, you have 100 tables functionally grouped in ~10 and all they need to share is authentication, you may want to separate the functions in 10 different web2py applications (all sharing one database, each defining only the tables it needs, and sharing one session cookie). This modular design solves lots of problems. I have heard of people who have developed complex applications in Django (which uses the import you suggest) and run into problems because they were using more ram than needed since all modules were always imported. In web2py controllers are executed only when called and if you make many small controllers you save lots of memory. Controllers than then import the modules they need and only those. Massimo On Nov 13, 7:28 pm, Thadeus Burgess thade...@thadeusb.com wrote: The biggest contributing factor is that web2py is executed and not import. The coder must always be sure to align objects and database in the correct order, including any auxiliary functions, class mappings, queries. He/she has to worry about the order of everything. Sometimes you can't have such order for complex systems. I think an import based system solves this in that the import statements declare the order instead of the physical location of the code doing that. -- Thadeus On Sat, Nov 13, 2010 at 2:49 PM, Jason Brower encomp...@gmail.com wrote: These are encouraging. In large deployment I mean it in the most external sense of it. Facebook is big, myspace is big, those kinds of object were what I was aiming for. And with Massimo's comment about the database size. I think your right, the framework does become less relavent. In what ways in a large codebase making web2py code messier and is there a solution to it in another framework? BR, J On Sat, 2010-11-13 at 11:48 -0600, Thadeus Burgess wrote: What is large deployment? Is it a large codebase that you must manage for an internal dashbaord, or just alot
[web2py] Re: Discussion: Why is web2py bad for large deployments?
Hi Jason I guess you have to define 'large deployment' first of all. Number of records and size of DB? Number of concurrent users? Large data model or number of forms etc? Number of servers -- or replication? Global coverage? In principle I don't think there's any reason why Web2py would be worse than other frameworks. Usually it is much better! As an example, I think deploying to the Google App Engine should be able to scale sufficiently for everything but extreme cases :) If you specify more about what you wish to achieve this group may be able to give more specific advice how best to organise your project. -D On Nov 13, 7:12 am, Jason Brower encomp...@gmail.com wrote: I love web2py and it's the only framework i feel i am fully capable to do or learn to do quickly. However, I remember see that this framework is intended for small to medium sized deployments. Is this true? What is it that stops us from larger deployment? Should i pickup django because i may need it? Regards, jb
[web2py] Re: Discussion: Why is web2py bad for large deployments?
I agree with Villas. The larger the development the more the database becomes the bottleneck and the framework irrelevant. Massimo On Nov 13, 8:35 am, villas villa...@gmail.com wrote: Hi Jason I guess you have to define 'large deployment' first of all. Number of records and size of DB? Number of concurrent users? Large data model or number of forms etc? Number of servers -- or replication? Global coverage? In principle I don't think there's any reason why Web2py would be worse than other frameworks. Usually it is much better! As an example, I think deploying to the Google App Engine should be able to scale sufficiently for everything but extreme cases :) If you specify more about what you wish to achieve this group may be able to give more specific advice how best to organise your project. -D On Nov 13, 7:12 am, Jason Brower encomp...@gmail.com wrote: I love web2py and it's the only framework i feel i am fully capable to do or learn to do quickly. However, I remember see that this framework is intended for small to medium sized deployments. Is this true? What is it that stops us from larger deployment? Should i pickup django because i may need it? Regards, jb
Re: [web2py] Re: Discussion: Why is web2py bad for large deployments?
What is large deployment? Is it a large codebase that you must manage for an internal dashbaord, or just alot of users/database io that needs to scale out for worldwide access? If its the first case, web2py can get really complicated in dealing with lots of models and difficult to manage in an efficient manner. The larger your codebase the messier web2py apps will become. In the end, this would ultimately be up to the preferences of you and your team and what your willing to put up with. In the second case, framework hardly matters at that point. Disqus uses django, facebook uses php, reddit uses pylons, myspace uses coldfusion, microsoft uses asp, oracle uses java. Its always the database that becomes an issue regardless of programming language or web framework. -- Thadeus On Sat, Nov 13, 2010 at 10:53 AM, mdipierro mdipie...@cs.depaul.edu wrote: I agree with Villas. The larger the development the more the database becomes the bottleneck and the framework irrelevant. Massimo On Nov 13, 8:35 am, villas villa...@gmail.com wrote: Hi Jason I guess you have to define 'large deployment' first of all. Number of records and size of DB? Number of concurrent users? Large data model or number of forms etc? Number of servers -- or replication? Global coverage? In principle I don't think there's any reason why Web2py would be worse than other frameworks. Usually it is much better! As an example, I think deploying to the Google App Engine should be able to scale sufficiently for everything but extreme cases :) If you specify more about what you wish to achieve this group may be able to give more specific advice how best to organise your project. -D On Nov 13, 7:12 am, Jason Brower encomp...@gmail.com wrote: I love web2py and it's the only framework i feel i am fully capable to do or learn to do quickly. However, I remember see that this framework is intended for small to medium sized deployments. Is this true? What is it that stops us from larger deployment? Should i pickup django because i may need it? Regards, jb
[web2py] Re: Discussion: Why is web2py bad for large deployments?
We are considering making this even easier. Two non-exclusive proposals on the table are: 1) make conditional models that are executed only for some controllers 2) promote the global objects (request,session,response,T,cache) to be thread local (as in Flask) so that they can be imported by modules and they do not have to be passed around. (Auth,Crud,Service could also be written). The problem is that we do not know if this will affect performance and it may potentially break something. We'll keep experimenting. Massimo On Nov 13, 1:48 pm, b00m_chef r...@devshell.org wrote: I disagree that web2py would become messy wrt the code-base. The solution I found was to simply put all my actual logic code in modules, then, I simply call up the specific object method in the controller function linked to the particular html file (view). I therefore don't keep any code in the Controller except object method calls to the module where the actual code resides. Within the module file, I can organize my data anyway I want. Django is no more organized, and no less organized. Most important thing to remember is that when you get to a size that web2py is no longer suitable, you probably have enough funding to pay a team of coders to write your app in pylons from scratch. Web2py can get you to that point though. On Nov 13, 9:48 am, Thadeus Burgess thade...@thadeusb.com wrote: What is large deployment? Is it a large codebase that you must manage for an internal dashbaord, or just alot of users/database io that needs to scale out for worldwide access? If its the first case, web2py can get really complicated in dealing with lots of models and difficult to manage in an efficient manner. The larger your codebase the messier web2py apps will become. In the end, this would ultimately be up to the preferences of you and your team and what your willing to put up with. In the second case, framework hardly matters at that point. Disqus uses django, facebook uses php, reddit uses pylons, myspace uses coldfusion, microsoft uses asp, oracle uses java. Its always the database that becomes an issue regardless of programming language or web framework. -- Thadeus On Sat, Nov 13, 2010 at 10:53 AM, mdipierro mdipie...@cs.depaul.edu wrote: I agree with Villas. The larger the development the more the database becomes the bottleneck and the framework irrelevant. Massimo On Nov 13, 8:35 am, villas villa...@gmail.com wrote: Hi Jason I guess you have to define 'large deployment' first of all. Number of records and size of DB? Number of concurrent users? Large data model or number of forms etc? Number of servers -- or replication? Global coverage? In principle I don't think there's any reason why Web2py would be worse than other frameworks. Usually it is much better! As an example, I think deploying to the Google App Engine should be able to scale sufficiently for everything but extreme cases :) If you specify more about what you wish to achieve this group may be able to give more specific advice how best to organise your project. -D On Nov 13, 7:12 am, Jason Brower encomp...@gmail.com wrote: I love web2py and it's the only framework i feel i am fully capable to do or learn to do quickly. However, I remember see that this framework is intended for small to medium sized deployments. Is this true? What is it that stops us from larger deployment? Should i pickup django because i may need it? Regards, jb
Re: [web2py] Re: Discussion: Why is web2py bad for large deployments?
These are encouraging. In large deployment I mean it in the most external sense of it. Facebook is big, myspace is big, those kinds of object were what I was aiming for. And with Massimo's comment about the database size. I think your right, the framework does become less relavent. In what ways in a large codebase making web2py code messier and is there a solution to it in another framework? BR, J On Sat, 2010-11-13 at 11:48 -0600, Thadeus Burgess wrote: What is large deployment? Is it a large codebase that you must manage for an internal dashbaord, or just alot of users/database io that needs to scale out for worldwide access? If its the first case, web2py can get really complicated in dealing with lots of models and difficult to manage in an efficient manner. The larger your codebase the messier web2py apps will become. In the end, this would ultimately be up to the preferences of you and your team and what your willing to put up with. In the second case, framework hardly matters at that point. Disqus uses django, facebook uses php, reddit uses pylons, myspace uses coldfusion, microsoft uses asp, oracle uses java. Its always the database that becomes an issue regardless of programming language or web framework. -- Thadeus On Sat, Nov 13, 2010 at 10:53 AM, mdipierro mdipie...@cs.depaul.edu wrote: I agree with Villas. The larger the development the more the database becomes the bottleneck and the framework irrelevant. Massimo On Nov 13, 8:35 am, villas villa...@gmail.com wrote: Hi Jason I guess you have to define 'large deployment' first of all. Number of records and size of DB? Number of concurrent users? Large data model or number of forms etc? Number of servers -- or replication? Global coverage? In principle I don't think there's any reason why Web2py would be worse than other frameworks. Usually it is much better! As an example, I think deploying to the Google App Engine should be able to scale sufficiently for everything but extreme cases :) If you specify more about what you wish to achieve this group may be able to give more specific advice how best to organise your project. -D On Nov 13, 7:12 am, Jason Brower encomp...@gmail.com wrote: I love web2py and it's the only framework i feel i am fully capable to do or learn to do quickly. However, I remember see that this framework is intended for small to medium sized deployments. Is this true? What is it that stops us from larger deployment? Should i pickup django because i may need it? Regards, jb
Re: [web2py] Re: Discussion: Why is web2py bad for large deployments?
On Sat, Nov 13, 2010 at 9:49 PM, Jason Brower encomp...@gmail.com wrote: These are encouraging. In large deployment I mean it in the most external sense of it. Facebook is big, myspace is big, those kinds of object were what I was aiming for. And For that kind of large deployment, you'll probably hit the limitation of the Python programming language sooner or later, too. Erlang seems to have become quite popular lately and even Facebook uses it for some backend stuff. The main reason is out-of-box concurrency, and great reliability under load which comes from supervisor-worker setup that is built-in. A good example of what this means is Yaws http server which is virtually impossible to kill even if you bombard it with a huge (as in MUCH more than http server X) number of concurrent requests. -- Branko Vukelić bg.bra...@gmail.com stu...@brankovukelic.com Check out my blog: http://www.brankovukelic.com/ Check out my portfolio: http://www.flickr.com/photos/foxbunny/ Registered Linux user #438078 (http://counter.li.org/) I hang out on identi.ca: http://identi.ca/foxbunny Gimp Brushmakers Guild http://bit.ly/gbg-group
[web2py] Re: Discussion: Why is web2py bad for large deployments?
Let me just say that the chance that your app will have the traffic of facebook or twitter or even livejournal (yes it is still alive) is so small, that you really don't need to worry about it too much at such an early stage. On Nov 13, 3:11 pm, Branko Vukelic bg.bra...@gmail.com wrote: On Sat, Nov 13, 2010 at 9:49 PM, Jason Brower encomp...@gmail.com wrote: These are encouraging. In large deployment I mean it in the most external sense of it. Facebook is big, myspace is big, those kinds of object were what I was aiming for. And For that kind of large deployment, you'll probably hit the limitation of the Python programming language sooner or later, too. Erlang seems to have become quite popular lately and even Facebook uses it for some backend stuff. The main reason is out-of-box concurrency, and great reliability under load which comes from supervisor-worker setup that is built-in. A good example of what this means is Yaws http server which is virtually impossible to kill even if you bombard it with a huge (as in MUCH more than http server X) number of concurrent requests. -- Branko Vukelić bg.bra...@gmail.com stu...@brankovukelic.com Check out my blog:http://www.brankovukelic.com/ Check out my portfolio:http://www.flickr.com/photos/foxbunny/ Registered Linux user #438078 (http://counter.li.org/) I hang out on identi.ca:http://identi.ca/foxbunny Gimp Brushmakers Guildhttp://bit.ly/gbg-group
Re: [web2py] Re: Discussion: Why is web2py bad for large deployments?
The biggest contributing factor is that web2py is executed and not import. The coder must always be sure to align objects and database in the correct order, including any auxiliary functions, class mappings, queries. He/she has to worry about the order of everything. Sometimes you can't have such order for complex systems. I think an import based system solves this in that the import statements declare the order instead of the physical location of the code doing that. -- Thadeus On Sat, Nov 13, 2010 at 2:49 PM, Jason Brower encomp...@gmail.com wrote: These are encouraging. In large deployment I mean it in the most external sense of it. Facebook is big, myspace is big, those kinds of object were what I was aiming for. And with Massimo's comment about the database size. I think your right, the framework does become less relavent. In what ways in a large codebase making web2py code messier and is there a solution to it in another framework? BR, J On Sat, 2010-11-13 at 11:48 -0600, Thadeus Burgess wrote: What is large deployment? Is it a large codebase that you must manage for an internal dashbaord, or just alot of users/database io that needs to scale out for worldwide access? If its the first case, web2py can get really complicated in dealing with lots of models and difficult to manage in an efficient manner. The larger your codebase the messier web2py apps will become. In the end, this would ultimately be up to the preferences of you and your team and what your willing to put up with. In the second case, framework hardly matters at that point. Disqus uses django, facebook uses php, reddit uses pylons, myspace uses coldfusion, microsoft uses asp, oracle uses java. Its always the database that becomes an issue regardless of programming language or web framework. -- Thadeus On Sat, Nov 13, 2010 at 10:53 AM, mdipierro mdipie...@cs.depaul.edu wrote: I agree with Villas. The larger the development the more the database becomes the bottleneck and the framework irrelevant. Massimo On Nov 13, 8:35 am, villas villa...@gmail.com wrote: Hi Jason I guess you have to define 'large deployment' first of all. Number of records and size of DB? Number of concurrent users? Large data model or number of forms etc? Number of servers -- or replication? Global coverage? In principle I don't think there's any reason why Web2py would be worse than other frameworks. Usually it is much better! As an example, I think deploying to the Google App Engine should be able to scale sufficiently for everything but extreme cases :) If you specify more about what you wish to achieve this group may be able to give more specific advice how best to organise your project. -D On Nov 13, 7:12 am, Jason Brower encomp...@gmail.com wrote: I love web2py and it's the only framework i feel i am fully capable to do or learn to do quickly. However, I remember see that this framework is intended for small to medium sized deployments. Is this true? What is it that stops us from larger deployment? Should i pickup django because i may need it? Regards, jb
[web2py] Re: Discussion: Why is web2py bad for large deployments?
I agree that there are things we can improve in web2py. Yet, if a complex application have many pieces that are untangled, the import mechanism does not necessarily help you. The solution is trying to design components that are relatively independent. If you can do this, web2py does not bet in the way. For example, you have 100 tables functionally grouped in ~10 and all they need to share is authentication, you may want to separate the functions in 10 different web2py applications (all sharing one database, each defining only the tables it needs, and sharing one session cookie). This modular design solves lots of problems. I have heard of people who have developed complex applications in Django (which uses the import you suggest) and run into problems because they were using more ram than needed since all modules were always imported. In web2py controllers are executed only when called and if you make many small controllers you save lots of memory. Controllers than then import the modules they need and only those. Massimo On Nov 13, 7:28 pm, Thadeus Burgess thade...@thadeusb.com wrote: The biggest contributing factor is that web2py is executed and not import. The coder must always be sure to align objects and database in the correct order, including any auxiliary functions, class mappings, queries. He/she has to worry about the order of everything. Sometimes you can't have such order for complex systems. I think an import based system solves this in that the import statements declare the order instead of the physical location of the code doing that. -- Thadeus On Sat, Nov 13, 2010 at 2:49 PM, Jason Brower encomp...@gmail.com wrote: These are encouraging. In large deployment I mean it in the most external sense of it. Facebook is big, myspace is big, those kinds of object were what I was aiming for. And with Massimo's comment about the database size. I think your right, the framework does become less relavent. In what ways in a large codebase making web2py code messier and is there a solution to it in another framework? BR, J On Sat, 2010-11-13 at 11:48 -0600, Thadeus Burgess wrote: What is large deployment? Is it a large codebase that you must manage for an internal dashbaord, or just alot of users/database io that needs to scale out for worldwide access? If its the first case, web2py can get really complicated in dealing with lots of models and difficult to manage in an efficient manner. The larger your codebase the messier web2py apps will become. In the end, this would ultimately be up to the preferences of you and your team and what your willing to put up with. In the second case, framework hardly matters at that point. Disqus uses django, facebook uses php, reddit uses pylons, myspace uses coldfusion, microsoft uses asp, oracle uses java. Its always the database that becomes an issue regardless of programming language or web framework. -- Thadeus On Sat, Nov 13, 2010 at 10:53 AM, mdipierro mdipie...@cs.depaul.edu wrote: I agree with Villas. The larger the development the more the database becomes the bottleneck and the framework irrelevant. Massimo On Nov 13, 8:35 am, villas villa...@gmail.com wrote: Hi Jason I guess you have to define 'large deployment' first of all. Number of records and size of DB? Number of concurrent users? Large data model or number of forms etc? Number of servers -- or replication? Global coverage? In principle I don't think there's any reason why Web2py would be worse than other frameworks. Usually it is much better! As an example, I think deploying to the Google App Engine should be able to scale sufficiently for everything but extreme cases :) If you specify more about what you wish to achieve this group may be able to give more specific advice how best to organise your project. -D On Nov 13, 7:12 am, Jason Brower encomp...@gmail.com wrote: I love web2py and it's the only framework i feel i am fully capable to do or learn to do quickly. However, I remember see that this framework is intended for small to medium sized deployments. Is this true? What is it that stops us from larger deployment? Should i pickup django because i may need it? Regards, jb
[web2py] Re: Discussion: Why is web2py bad for large deployments?
I think that as long as you develop good code, choose a good interpreter, use assynchronous requests as adequate and good caching, you shouldn't have many constraints other than the database. There, I think there is no work-around. You have to make specific/manual DB handling for each DBMS to assure good scalibility. Out of the box, there aren't many things like this that do much better. On Nov 14, 3:41 am, mdipierro mdipie...@cs.depaul.edu wrote: I agree that there are things we can improve in web2py. Yet, if a complex application have many pieces that are untangled, the import mechanism does not necessarily help you. The solution is trying to design components that are relatively independent. If you can do this, web2py does not bet in the way. For example, you have 100 tables functionally grouped in ~10 and all they need to share is authentication, you may want to separate the functions in 10 different web2py applications (all sharing one database, each defining only the tables it needs, and sharing one session cookie). This modular design solves lots of problems. I have heard of people who have developed complex applications in Django (which uses the import you suggest) and run into problems because they were using more ram than needed since all modules were always imported. In web2py controllers are executed only when called and if you make many small controllers you save lots of memory. Controllers than then import the modules they need and only those. Massimo On Nov 13, 7:28 pm, Thadeus Burgess thade...@thadeusb.com wrote: The biggest contributing factor is that web2py is executed and not import. The coder must always be sure to align objects and database in the correct order, including any auxiliary functions, class mappings, queries. He/she has to worry about the order of everything. Sometimes you can't have such order for complex systems. I think an import based system solves this in that the import statements declare the order instead of the physical location of the code doing that. -- Thadeus On Sat, Nov 13, 2010 at 2:49 PM, Jason Brower encomp...@gmail.com wrote: These are encouraging. In large deployment I mean it in the most external sense of it. Facebook is big, myspace is big, those kinds of object were what I was aiming for. And with Massimo's comment about the database size. I think your right, the framework does become less relavent. In what ways in a large codebase making web2py code messier and is there a solution to it in another framework? BR, J On Sat, 2010-11-13 at 11:48 -0600, Thadeus Burgess wrote: What is large deployment? Is it a large codebase that you must manage for an internal dashbaord, or just alot of users/database io that needs to scale out for worldwide access? If its the first case, web2py can get really complicated in dealing with lots of models and difficult to manage in an efficient manner. The larger your codebase the messier web2py apps will become. In the end, this would ultimately be up to the preferences of you and your team and what your willing to put up with. In the second case, framework hardly matters at that point. Disqus uses django, facebook uses php, reddit uses pylons, myspace uses coldfusion, microsoft uses asp, oracle uses java. Its always the database that becomes an issue regardless of programming language or web framework. -- Thadeus On Sat, Nov 13, 2010 at 10:53 AM, mdipierro mdipie...@cs.depaul.edu wrote: I agree with Villas. The larger the development the more the database becomes the bottleneck and the framework irrelevant. Massimo On Nov 13, 8:35 am, villas villa...@gmail.com wrote: Hi Jason I guess you have to define 'large deployment' first of all. Number of records and size of DB? Number of concurrent users? Large data model or number of forms etc? Number of servers -- or replication? Global coverage? In principle I don't think there's any reason why Web2py would be worse than other frameworks. Usually it is much better! As an example, I think deploying to the Google App Engine should be able to scale sufficiently for everything but extreme cases :) If you specify more about what you wish to achieve this group may be able to give more specific advice how best to organise your project. -D On Nov 13, 7:12 am, Jason Brower encomp...@gmail.com wrote: I love web2py and it's the only framework i feel i am fully capable to do or learn to do quickly. However, I remember see that this framework is intended for small to medium sized deployments. Is this true? What is it that stops us from larger deployment? Should i pickup django because i may need it? Regards, jb
