This also explains why the problem occurred with Apache ab but not for (usually) people using web2py in production. In production you do not have thousand of session files because many of the requests from from the same few users.
There are exceptions. Apps that have many semi-static pages or are probed by ab or are subject to DoD attacks. those apps may have suffered from the problem that is now fixed. Massimo On Mar 14, 11:12 pm, Massimo Di Pierro <massimo.dipie...@gmail.com> wrote: > I think also address the scalability problem that has occasionally > been reported using ab tests. > > Apache ab sends many uncorrelated requests and each of them results > (resulted) in a new session file saved. Eventually the more tests you > run the more files you have. The more files in the same folder you > have, the slower the file system becomes, the longer it takes web2py > to save a session, the longer the thread runs, the less requests/ > second you can process, the sooner you reach the max number of threads > and requests will be dropped. > > The latest fix should address the issue since only REAL sessions will > not result in file system access. > > It should also considerably speed up GAE since less datastore write > access. > > There may be room for further improvement but I believe this is a big > improvement already. > > Please report any data you may have. > > Massimo > > On Mar 14, 10:40 pm, Massimo Di Pierro <massimo.dipie...@gmail.com> > wrote: > > > > > > > > > At PyCon some people have suggested that they would like the ability > > to switch off some web2py functionality to make it faster. I never > > thought of this being an issue because I do not think web2py does much > > outside models/views and controllers. > > > I run a lot of tests today using a barebone > > > def index(): return 'hello world' > > > controller (no view, no model) and I discovered, to my surprise that > > even in this simple case, web2py spends huge amount of time saving > > sessions even if not used (~40ms/request on my laptop). This is also a > > problem because if you have many passive visitors web2py creates a lot > > of empty session files that fills the sessions folder and slow down > > the file system even more. > > > I made some changes so that sessions are no longer saved if not > > modified or empty. For my simple code, the running time went down to > > 4ms/request. That is 10x than before. That is only 5x the bare Rocket > > or Tornado speed (considering it is still checking for session, doing > > routing, parsing input and performing a lot of checks, and a lot of > > other stuff). > > > I do not think it should break anything and you should feel it > > snappier immediately. Give it a try and let me know. > > > Try run ab -n 1000 with both web2py 1.93.1 and trunk for your app. Let > > me know what you get. > > > Enjoy! > > > Massimo
