Re: [web2py] Web2py in large web scenarios
On 30 January 2012 15:51, Anthony wrote: > >> There's an example at the end of the section on caching: > http://web2py.com/books/default/chapter/29/4#cache. response.render() is > briefly mentioned here: > http://web2py.com/books/default/chapter/29/4#response. Note, if you leave > out the filename, it will assume the current response.view. No changes are > necessary in the view code. > > > Thanks. Johann -- Because experiencing your loyal love is better than life itself, my lips will praise you. (Psalm 63:3)
Re: [web2py] Web2py in large web scenarios
> > I also noted best performance directly rendering the templates and >> caching views where it can be cached. >> >> just replacing >> >> return dict() >> >> with >> >> return response.render(filename, context) >> > Is there documentation about this? How do you code the view differently > in such a case? > There's an example at the end of the section on caching: http://web2py.com/books/default/chapter/29/4#cache. response.render() is briefly mentioned here: http://web2py.com/books/default/chapter/29/4#response. Note, if you leave out the filename, it will assume the current response.view. No changes are necessary in the view code. Anthony
Re: [web2py] Web2py in large web scenarios
On 28 January 2012 01:06, Michele Comitini wrote: > About db bottlenecks: > > I think that using executesql gives a performance gain of 2 orders of > magnitude over DAL on datasets with more than 1000 records. The DAL > is a huge bottleneck if used inappropriately. > I agree. I often find it easier to use executesql - many times just because formulating a complex query for me is not that easy using DAL. Also DAL cannot handle all the query types that one can do in Postgresql. It is a pity that SQLFORM.grid seems to depend on DAL queries. The drawback is that one cannot use it in direct combination with executesql - or can one? I would like to know whether this is possible. Regards Johann -- Because experiencing your loyal love is better than life itself, my lips will praise you. (Psalm 63:3)
Re: [web2py] Web2py in large web scenarios
On 27 January 2012 20:24, Bruno Rocha wrote: > I also noted best performance directly rendering the templates and > caching views where it can be cached. > > just replacing > > return dict() > > with > > return response.render(filename, context) > Is there documentation about this? How do you code the view differently in such a case? > the bottleneck is always server and database, so it is better to use pure > Python to sort, find, filter Rows objects than using a lot of database > requests. > > DAL provides .sort .find .exclude and Python has a lot of good things like > map, reduce, filter. With one db request you can fetch records put them in > cache and use Python in some cases to avoid more sql queries. Even > paginations can be done without the need to go to db again. > This might be true for smaller queries, but when working with thousands of records Postgresql is much more efficient than Python. Regards Johann -- Because experiencing your loyal love is better than life itself, my lips will praise you. (Psalm 63:3)
Re: [web2py] Web2py in large web scenarios
submodels are good, but you cant import models. So you cant reuse code. you end with too much repetition. in modules you can have a DRY structure. https://github.com/rochacbruno/web2py_model_less_app http://zerp.ly/rochacbruno Em 28/01/2012 23:00, "Vinicius Assef" escreveu: > I'd use folders inside models. > I think I'd use web2py infrastructure that's already available to me. > > And, AFAIK, /modules were not planned for that. Except for pluggable apps. > > -- > Vinicius Assef. > > > > On Fri, Jan 27, 2012 at 10:35 PM, Bruno Rocha wrote: > >> I am using /models just to define small global functions and to set some >> response and request keys. >> >> All my code including datamodels I am putting into /modules >> >> In my mind I changed the /models to /scripts or /batches so it is more >> easy to understand why avoiding it. >> >> my current structure is: >> >> /modules/datamodels/someentity.py - Database definitions >> >> /modules/handlers/someentity.py - my code logic and template rendering >> >> /modules/helpers/* - miscelanious >> >> /modules/myappname.py - My.custom Auth, db, Mail, Service etc.. >> >> /controllers/someentytity.py - it will just be a point of entry, here I >> do selective imports, intantiate the entities and call the template >> rendering. >> >> views files can be stored anywhere filesystem or database. >> >> http://zerp.ly/rochacbruno >> Em 27/01/2012 17:48, "Magnitus" escreveu: >> >> >This is very good advice. I have moved many of my plugins and apps from >>> using models to modules because of the performance gain. There is nothing >>> wrong with the models implementation, but it's really meant to define >>> tables and that's it. Functionality that doesn't belong in a controller >>> should go to modules.< >>> >>> Interesting, so to clarify: are you minimizing the amount of code into >>> the model (limiting it to table definitions) or you avoid using a model at >>> all? >>> >> >
Re: [web2py] Web2py in large web scenarios
I'd use folders inside models. I think I'd use web2py infrastructure that's already available to me. And, AFAIK, /modules were not planned for that. Except for pluggable apps. -- Vinicius Assef. On Fri, Jan 27, 2012 at 10:35 PM, Bruno Rocha wrote: > I am using /models just to define small global functions and to set some > response and request keys. > > All my code including datamodels I am putting into /modules > > In my mind I changed the /models to /scripts or /batches so it is more > easy to understand why avoiding it. > > my current structure is: > > /modules/datamodels/someentity.py - Database definitions > > /modules/handlers/someentity.py - my code logic and template rendering > > /modules/helpers/* - miscelanious > > /modules/myappname.py - My.custom Auth, db, Mail, Service etc.. > > /controllers/someentytity.py - it will just be a point of entry, here I do > selective imports, intantiate the entities and call the template rendering. > > views files can be stored anywhere filesystem or database. > > http://zerp.ly/rochacbruno > Em 27/01/2012 17:48, "Magnitus" escreveu: > > >This is very good advice. I have moved many of my plugins and apps from >> using models to modules because of the performance gain. There is nothing >> wrong with the models implementation, but it's really meant to define >> tables and that's it. Functionality that doesn't belong in a controller >> should go to modules.< >> >> Interesting, so to clarify: are you minimizing the amount of code into >> the model (limiting it to table definitions) or you avoid using a model at >> all? >> >
Re: [web2py] Web2py in large web scenarios
Hi, Nice Bruno, But those approachs how many traffic deals? By the way, is the first time that i notice about use the /modules approach to gain performance, are you measure the performance por models inside /model and /modules? To all: Mores experiences? Problem resolutions? Maybe put all of this inside a document will be great for help to many users when scalability is their minds. Saludos, Alfonso de la Guarda Centro Open Source(COS) http://www.cos-la.net http://alfonsodg.net Twitter: @alfonsodg Redes sociales: alfonsodg Telef. 991935157 1024D/B23B24A4 5469 ED92 75A3 BBDB FD6B 58A5 54A1 851D B23B 24A4 On Fri, Jan 27, 2012 at 19:35, Bruno Rocha wrote: > I am using /models just to define small global functions and to set some > response and request keys. > > All my code including datamodels I am putting into /modules > > In my mind I changed the /models to /scripts or /batches so it is more easy > to understand why avoiding it. > > my current structure is: > > /modules/datamodels/someentity.py - Database definitions > > /modules/handlers/someentity.py - my code logic and template rendering > > /modules/helpers/* - miscelanious > > /modules/myappname.py - My.custom Auth, db, Mail, Service etc.. > > /controllers/someentytity.py - it will just be a point of entry, here I do > selective imports, intantiate the entities and call the template rendering. > > views files can be stored anywhere filesystem or database. > > http://zerp.ly/rochacbruno > > Em 27/01/2012 17:48, "Magnitus" escreveu: > >> >This is very good advice. I have moved many of my plugins and apps from >> > using models to modules because of the performance gain. There is nothing >> > wrong with the models implementation, but it's really meant to define >> > tables >> > and that's it. Functionality that doesn't belong in a controller should go >> > to modules.< >> >> Interesting, so to clarify: are you minimizing the amount of code into the >> model (limiting it to table definitions) or you avoid using a model at all?
Re: [web2py] Web2py in large web scenarios
I wonder if DAL should have an ability to return JSON data sets that are then manipulated with JavaScript? Or something like that?
Re: [web2py] Web2py in large web scenarios
What confuses me is what modules have access to and what needs to be imported (in modules and controllers).
Re: [web2py] Web2py in large web scenarios
I am using /models just to define small global functions and to set some response and request keys. All my code including datamodels I am putting into /modules In my mind I changed the /models to /scripts or /batches so it is more easy to understand why avoiding it. my current structure is: /modules/datamodels/someentity.py - Database definitions /modules/handlers/someentity.py - my code logic and template rendering /modules/helpers/* - miscelanious /modules/myappname.py - My.custom Auth, db, Mail, Service etc.. /controllers/someentytity.py - it will just be a point of entry, here I do selective imports, intantiate the entities and call the template rendering. views files can be stored anywhere filesystem or database. http://zerp.ly/rochacbruno Em 27/01/2012 17:48, "Magnitus" escreveu: > >This is very good advice. I have moved many of my plugins and apps from > using models to modules because of the performance gain. There is nothing > wrong with the models implementation, but it's really meant to define > tables and that's it. Functionality that doesn't belong in a controller > should go to modules.< > > Interesting, so to clarify: are you minimizing the amount of code into the > model (limiting it to table definitions) or you avoid using a model at all? >
Re: [web2py] Web2py in large web scenarios
About db bottlenecks: I think that using executesql gives a performance gain of 2 orders of magnitude over DAL on datasets with more than 1000 records. The DAL is a huge bottleneck if used inappropriately. I suggest using posgtresql with proper indexing. How? Just take your query and run it inside an EXPLAIN statement, see where indexes are needed and *create those indexes*. Speed improvements are *dramatic*. Pass to the DAL only small datasets: even if python has the best sorting and filtering algorithms around (no kidding), postgresql having those same algorithims is simply faster, because of the way objects are handled in memory. When Massimo says the db is the bottleneck, he does not mean that relational DBs are slow in general, IMHO he means that _your_ DB is slow, badly designed and/or badly queried. I also suggest to pay attention to one more thing. When you write the application and you do some test you do it in a quasi-singlethreaded setup. Well everything works fine. Then you go in production and after you reach 20 hits/sec your db becomes a bottleneck whatever hardware you have. If you are using postgresql use a concurrency model based on procesess, *avoid threads*. i.e. configure web2py to work as uwsgi, scgi or fcgi service. mic 2012/1/27 Vinicius Assef : > It really depends. > > Querying and returning large datasets can become a huge bottleneck. > More than letting db server work on its own relations. If you use a > real RDBMS, it's designed to work on it. > > Web2py doesn't create indexes. You have to make it manually. Making > so, RDBMS thanks you a lot and can be your close friend. > > -- > Vinicius Assef. > > > > On Fri, Jan 27, 2012 at 4:24 PM, Bruno Rocha wrote: >> I also noted best performance directly rendering the templates and caching >> views where it can be cached. >> >> just replacing >> >> return dict() >> >> with >> >> return response.render(filename, context) >> >> in conttrollers and also usins @cached controllers and cached queries. >> >> the bottleneck is always server and database, so it is better to use pure >> Python to sort, find, filter Rows objects than using a lot of database >> requests. >> >> DAL provides .sort .find .exclude and Python has a lot of good things like >> map, reduce, filter. With one db request you can fetch records put them in >> cache and use Python in some cases to avoid more sql queries. Even >> paginations can be done without the need to go to db again. >> >> Redis is being a good cache solution. >> >> http://zerp.ly/rochacbruno >> >> Em 27/01/2012 13:40, "Alfonso de la Guarda" escreveu: >> >>> Hi, >>> >>> I'm about to start a major project development and although I am a >>> regular user of web2py in my development, they have been mostly in >>> intranets, what worries me is if anyone has had previous experience of >>> putting an application based on web2py with several thousand users, in >>> which case I would like your job with this. >>> >>> Previously, I have dealt with a project based on django super (50 >>> unique users month) and problems of performance, scalability and >>> others were so serious that they chose to pass it to php, I would not >>> do the same with this application in web2py. >>> >>> >>> Thanks >>> >>> Saludos, >>> >>> >>> Alfonso de la Guarda >>> Centro Open Source(COS) >>> http://www.cos-la.net >>> http://alfonsodg.net >>> Twitter: @alfonsodg >>> Redes sociales: alfonsodg >>> Telef. 991935157 >>> 1024D/B23B24A4 >>> 5469 ED92 75A3 BBDB FD6B 58A5 54A1 851D B23B 24A4
Re: [web2py] Web2py in large web scenarios
Are there any good instructions for how to do this?
Re: [web2py] Web2py in large web scenarios
It really depends. Querying and returning large datasets can become a huge bottleneck. More than letting db server work on its own relations. If you use a real RDBMS, it's designed to work on it. Web2py doesn't create indexes. You have to make it manually. Making so, RDBMS thanks you a lot and can be your close friend. -- Vinicius Assef. On Fri, Jan 27, 2012 at 4:24 PM, Bruno Rocha wrote: > I also noted best performance directly rendering the templates and caching > views where it can be cached. > > just replacing > > return dict() > > with > > return response.render(filename, context) > > in conttrollers and also usins @cached controllers and cached queries. > > the bottleneck is always server and database, so it is better to use pure > Python to sort, find, filter Rows objects than using a lot of database > requests. > > DAL provides .sort .find .exclude and Python has a lot of good things like > map, reduce, filter. With one db request you can fetch records put them in > cache and use Python in some cases to avoid more sql queries. Even > paginations can be done without the need to go to db again. > > Redis is being a good cache solution. > > http://zerp.ly/rochacbruno > > Em 27/01/2012 13:40, "Alfonso de la Guarda" escreveu: > >> Hi, >> >> I'm about to start a major project development and although I am a >> regular user of web2py in my development, they have been mostly in >> intranets, what worries me is if anyone has had previous experience of >> putting an application based on web2py with several thousand users, in >> which case I would like your job with this. >> >> Previously, I have dealt with a project based on django super (50 >> unique users month) and problems of performance, scalability and >> others were so serious that they chose to pass it to php, I would not >> do the same with this application in web2py. >> >> >> Thanks >> >> Saludos, >> >> >> Alfonso de la Guarda >> Centro Open Source(COS) >> http://www.cos-la.net >> http://alfonsodg.net >> Twitter: @alfonsodg >> Redes sociales: alfonsodg >> Telef. 991935157 >> 1024D/B23B24A4 >> 5469 ED92 75A3 BBDB FD6B 58A5 54A1 851D B23B 24A4
Re: [web2py] Web2py in large web scenarios
>This is very good advice. I have moved many of my plugins and apps from using models to modules because of the performance gain. There is nothing wrong with the models implementation, but it's really meant to define tables and that's it. Functionality that doesn't belong in a controller should go to modules.< Interesting, so to clarify: are you minimizing the amount of code into the model (limiting it to table definitions) or you avoid using a model at all?
Re: [web2py] Web2py in large web scenarios
I also noted best performance directly rendering the templates and caching views where it can be cached. just replacing return dict() with return response.render(filename, context) in conttrollers and also usins @cached controllers and cached queries. the bottleneck is always server and database, so it is better to use pure Python to sort, find, filter Rows objects than using a lot of database requests. DAL provides .sort .find .exclude and Python has a lot of good things like map, reduce, filter. With one db request you can fetch records put them in cache and use Python in some cases to avoid more sql queries. Even paginations can be done without the need to go to db again. Redis is being a good cache solution. http://zerp.ly/rochacbruno Em 27/01/2012 13:40, "Alfonso de la Guarda" escreveu: > Hi, > > I'm about to start a major project development and although I am a > regular user of web2py in my development, they have been mostly in > intranets, what worries me is if anyone has had previous experience of > putting an application based on web2py with several thousand users, in > which case I would like your job with this. > > Previously, I have dealt with a project based on django super (50 > unique users month) and problems of performance, scalability and > others were so serious that they chose to pass it to php, I would not > do the same with this application in web2py. > > > Thanks > > Saludos, > > > Alfonso de la Guarda > Centro Open Source(COS) > http://www.cos-la.net > http://alfonsodg.net > Twitter: @alfonsodg > Redes sociales: alfonsodg >Telef. 991935157 > 1024D/B23B24A4 > 5469 ED92 75A3 BBDB FD6B 58A5 54A1 851D B23B 24A4 >
Re: [web2py] Web2py in large web scenarios
One other thing I noticed after reading your question a second time: Python (Django and web2py) will run much faster than PHP in almost every case. If you needed to start passing things to PHP in order to speed things up, then you must have been using a highly-tuned PHP configuration and a very untuned Python configuration.
Re: [web2py] Web2py in large web scenarios
This is very good advice. I have moved many of my plugins and apps from using models to modules because of the performance gain. There is nothing wrong with the models implementation, but it's really meant to define tables and that's it. Functionality that doesn't belong in a controller should go to modules. Some other advice that I have been gathering for my own deployments is to use Nginx instead of Apache. You will see an incredible increase in the number of requests per second that Nginx can handle over Apache. Also setting up an Nginx front-facing server that is proxying requests to another server running Nginx + web2py might be a good idea for lots of traffic. Have the front-facing server handle the SSL traffic, and maybe even enable caching on the front-end as well. When you need to add another server to handle increased traffic, this front-facing Nginx server can be configured to load balance so that you can scale as much as you need.
Re: [web2py] Web2py in large web scenarios
My first tip is, avoid /models have your code in /modules import what you need only when and where you need! On Fri, Jan 27, 2012 at 1:39 PM, Alfonso de la Guarda wrote: > Hi, > > I'm about to start a major project development and although I am a > regular user of web2py in my development, they have been mostly in > intranets, what worries me is if anyone has had previous experience of > putting an application based on web2py with several thousand users, in > which case I would like your job with this. > > Previously, I have dealt with a project based on django super (50 > unique users month) and problems of performance, scalability and > others were so serious that they chose to pass it to php, I would not > do the same with this application in web2py. > > > Thanks > > Saludos, > > > Alfonso de la Guarda > Centro Open Source(COS) > http://www.cos-la.net > http://alfonsodg.net > Twitter: @alfonsodg > Redes sociales: alfonsodg >Telef. 991935157 > 1024D/B23B24A4 > 5469 ED92 75A3 BBDB FD6B 58A5 54A1 851D B23B 24A4 > -- Bruno Rocha [http://rochacbruno.com.br]
[web2py] Web2py in large web scenarios
Hi, I'm about to start a major project development and although I am a regular user of web2py in my development, they have been mostly in intranets, what worries me is if anyone has had previous experience of putting an application based on web2py with several thousand users, in which case I would like your job with this. Previously, I have dealt with a project based on django super (50 unique users month) and problems of performance, scalability and others were so serious that they chose to pass it to php, I would not do the same with this application in web2py. Thanks Saludos, Alfonso de la Guarda Centro Open Source(COS) http://www.cos-la.net http://alfonsodg.net Twitter: @alfonsodg Redes sociales: alfonsodg Telef. 991935157 1024D/B23B24A4 5469 ED92 75A3 BBDB FD6B 58A5 54A1 851D B23B 24A4
