[web2py] Re: rocket performance issues

2010-07-29 Thread mdipierro
Mike, I modified the expire_sessions.py in trunk and added a try:...execept... If the problem was that a file was locked and the cron task got stuck, it may be solved now. Can you test it? Massimo On Jul 28, 4:10 pm, MikeEllis michael.f.el...@gmail.com wrote: Update: (Summarizes a couple of

Re: [web2py] Re: rocket performance issues

2010-07-29 Thread Michael Ellis
Testing under way... (Iceberg, I'm running OS X 10.6.4) Cheers, Mike On Thu, Jul 29, 2010 at 2:57 AM, mdipierro mdipie...@cs.depaul.edu wrote: Mike, I modified the expire_sessions.py in trunk and added a try:...execept... If the problem was that a file was locked and the cron task got

Re: [web2py] Re: rocket performance issues

2010-07-29 Thread Michael Ellis
Has anything changed at tip in the view parser? Looks like there may be a problem now with colons at the end of comment lines. I have a view with the following that was working fine before I updated this morning. {{ ## On entry, vars returned by controller are: ## problem, c, s, sc, wsc,

[web2py] Re: rocket performance issues

2010-07-29 Thread mdipierro
we have a new template parser. My guess is that it is not ignoring the : at the end of {{# comments. On Jul 29, 7:55 am, Michael Ellis michael.f.el...@gmail.com wrote: Has anything changed at tip in the view parser? Looks like there may be a problem now with colons at the end of comment lines.

Re: [web2py] Re: rocket performance issues

2010-07-29 Thread Michael Ellis
No process leakage in instances I've been running since this morning. Open file count appears stable, too. Cheers, Mike On Thu, Jul 29, 2010 at 9:22 AM, mdipierro mdipie...@cs.depaul.edu wrote: we have a new template parser. My guess is that it is not ignoring the : at the end of {{#

[web2py] Re: rocket performance issues

2010-07-29 Thread mdipierro
with or without -N On Jul 29, 10:27 am, Michael Ellis michael.f.el...@gmail.com wrote: No process leakage in instances I've been running since this morning.  Open file count appears stable, too. Cheers, Mike On Thu, Jul 29, 2010 at 9:22 AM, mdipierro mdipie...@cs.depaul.edu wrote: we have

Re: [web2py] Re: rocket performance issues

2010-07-29 Thread Michael Ellis
without On Thu, Jul 29, 2010 at 11:38 AM, mdipierro mdipie...@cs.depaul.edu wrote: with or without -N On Jul 29, 10:27 am, Michael Ellis michael.f.el...@gmail.com wrote: No process leakage in instances I've been running since this morning. Open file count appears stable, too.

Re: [web2py] Re: rocket performance issues

2010-07-29 Thread Jonathan Lundell
On Jul 28, 2010, at 11:57 PM, mdipierro wrote: I modified the expire_sessions.py in trunk and added a try:...execept... If the problem was that a file was locked and the cron task got stuck, it may be solved now. Can you test it? In this patch: @@ -6,8 +6,14 @@ now=time.time() for file in

[web2py] Re: rocket performance issues

2010-07-29 Thread mdipierro
good point. This runs as a daemon therefore there is no stdout by definition. I will change it into logging. On Jul 29, 11:19 am, Jonathan Lundell jlund...@pobox.com wrote: On Jul 28, 2010, at 11:57 PM, mdipierro wrote: I modified the expire_sessions.py in trunk and added a

[web2py] Re: rocket performance issues

2010-07-29 Thread mdipierro
Can you check if this issue is solved in trunk? Massimo On Jul 29, 7:55 am, Michael Ellis michael.f.el...@gmail.com wrote: Has anything changed at tip in the view parser? Looks like there may be a problem now with colons at the end of comment lines. I have a view with the following that was

Re: [web2py] Re: rocket performance issues

2010-07-29 Thread Michael Ellis
Seems to be. I updated, restarted, and restored the colon after the comment. No errors when I load the page. Thanks for fixing it so quickly! On Thu, Jul 29, 2010 at 5:21 PM, mdipierro mdipie...@cs.depaul.edu wrote: Can you check if this issue is solved in trunk? Massimo On Jul 29,

[web2py] Re: rocket performance issues

2010-07-29 Thread mdipierro
It is easy to fix things when problems are clearly reported. ;-) On Jul 29, 4:38 pm, Michael Ellis michael.f.el...@gmail.com wrote: Seems to be.  I updated, restarted, and restored the colon after the comment.  No errors when I load the page.  Thanks for fixing it so quickly! On Thu, Jul 29,

[web2py] Re: rocket performance issues

2010-07-28 Thread mdipierro
There clearly is a problem with the cron processes. Massimo On Jul 28, 4:10 pm, MikeEllis michael.f.el...@gmail.com wrote: Update: (Summarizes a couple of emails exchanged with Massimo) Yesterday I found that my instance of web2py appeared be leaking processes at about 1 per hour.  Massimo

[web2py] Re: rocket performance issues

2010-07-28 Thread Iceberg
The one leak process per hour coincident is coincidently matching the one run per hour of admin/cron/expire_sessions.py That really raises my concern, cause I just gonna use cron as a trick to touch my app's cache frequently. :-/ By the way, which os is Mike Ellis using? I did not observe same

[web2py] Re: rocket performance issues

2010-07-27 Thread Rahul
Thanks everyone, for getting this issue resolved... Web2py rocks! Cheers, Rahul On Jul 24, 6:25 pm, Phyo Arkar phyo.arkarl...@gmail.com wrote: Yes , Since from start of web2py , Massimo and Us never recommend to use built-in web2py server at production. They are mainly for development purpose.

Re: [web2py] Re: rocket performance issues

2010-07-27 Thread Michael Ellis
Not sure if this is related to Rocket or whether a new topic is needed. This morning I found several OSError reports about Too many open files in a web2py development server that's been running locally for several days. The app code isn't doing any explicit file i/o so I don't know what's going

[web2py] Re: rocket performance issues

2010-07-27 Thread mdipierro
Are you opening any files explicitely in your app? Which web2py version and which Python version are you using? On Jul 27, 7:33 am, Michael Ellis michael.f.el...@gmail.com wrote: Not sure if this is related to Rocket or whether a new topic is needed. This morning I found several OSError

[web2py] Re: rocket performance issues

2010-07-27 Thread mdipierro
Ignore my previous email I see you are using 2.6 and the problem is with cron. I think the problem is a cron process that does not end and keeps restarting. Are you using cron? On Jul 27, 7:33 am, Michael Ellis michael.f.el...@gmail.com wrote: Not sure if this is related to Rocket or whether

[web2py] Re: rocket performance issues

2010-07-27 Thread mdipierro
Yet is being called. It is on by default. Try set cron=False at the bottom of web2py.py On Jul 27, 7:49 am, Michael Ellis michael.f.el...@gmail.com wrote: Not using cron. On Tue, Jul 27, 2010 at 8:43 AM, mdipierro mdipie...@cs.depaul.edu wrote: Ignore my previous email I see you are

Re: [web2py] Re: rocket performance issues

2010-07-27 Thread Michael Ellis
ok, I'll give that a shot in a few minutes if I don't get a fork break by then. BTW, I verified that cron/crontab is empty in my app. However it's not empty in admin and examples: $ cat admin/cron/crontab 10 * * * * root **applications/admin/cron/expire_sessions.py $ cat examples/cron/crontab

Re: [web2py] Re: rocket performance issues

2010-07-27 Thread Jonathan Lundell
On Jul 27, 2010, at 5:33 AM, Michael Ellis wrote: This morning I found several OSError reports about Too many open files in a web2py development server that's been running locally for several days. The app code isn't doing any explicit file i/o so I don't know what's going on, but here

Re: [web2py] Re: rocket performance issues

2010-07-27 Thread Michael Ellis
Thanks, Jonathan. That's a good suggestion. I last restarted the app a couple of hours ago. ps reports 4 processes that match 'web2py'. Grepping for each pid in lsof shows nothing extraordinary. Each process has 80-90 files open. The vast majority are libraries (.so). There are a few chr

[web2py] Re: rocket performance issues

2010-07-27 Thread mdipierro
The picture is too small. Anything interesting in there? On Jul 27, 11:01 am, Michael Ellis michael.f.el...@gmail.com wrote: Thanks for the info about sloth. I may try it later.  It was pretty easy to get what I needed with ipython: procs = !ps ax | grep web2py pids = [r[0] for r in

Re: [web2py] Re: rocket performance issues

2010-07-27 Thread Michael Ellis
Sorry, what I sent was very large (~750K each). Didn't realize GoogleGroups would reduce it to a thumbnail. I'll try to forward the original message directly to you ... On Tue, Jul 27, 2010 at 12:07 PM, mdipierro mdipie...@cs.depaul.edu wrote: The picture is too small. Anything interesting

Re: [web2py] Re: rocket performance issues

2010-07-27 Thread Jonathan Lundell
On Jul 27, 2010, at 9:01 AM, Michael Ellis wrote: Thanks for the info about sloth. I may try it later. It was pretty easy to get what I needed with ipython: procs = !ps ax | grep web2py pids = [r[0] for r in procs.fields()[0:-2]] for p in pids: print p !lsof | grep $p | wc

[web2py] Re: rocket performance issues

2010-07-24 Thread Scott
Please allow me to preface my comments: I have nothing against Rocket; my opinions come from years of experience with Java EE deployments. I think raising the max_threads to 1024 is a good idea. However, my opinion is that Rocket alone should not be used for a production deployment; much as I

[web2py] Re: rocket performance issues

2010-07-23 Thread mdipierro
This is now in 1.81.5 On Jul 22, 11:18 pm, Phyo Arkar phyo.arkarl...@gmail.com wrote: Thats very good news. I will be trying out soon! On 7/22/10, Timbo tfarr...@owassobible.org wrote: OK, so it's a thread-starvation issue.  Let this be the interim fix for web2py.  I've got several busy

[web2py] Re: rocket performance issues

2010-07-23 Thread mdipierro
I am not convinced this is a GIL problem. Why the lock after the 400 errors only? On Jul 23, 9:18 am, Michele Comitini michele.comit...@gmail.com wrote: Because of the GIL threads on CPython are problematic more than one could think.  Googling GIL python gives an idea... Not to start the war

[web2py] Re: rocket performance issues

2010-07-23 Thread mdipierro
On a second thought this open the door to more sever denial of service attacks than caused by the original problem. How about, until there is a better under understanding and solution, we just increase max_threads from the original 128 to 1024. On Jul 22, 11:27 am, Timbo tfarr...@owassobible.org

Re: [web2py] Re: rocket performance issues

2010-07-22 Thread Jonathan Lundell
On Jul 22, 2010, at 6:55 AM, MikeEllis wrote: Doesn't fix the symptom Ah. Well, negative results are still results. On Jul 21, 8:54 pm, Jonathan Lundell jlund...@pobox.com wrote: On Jul 21, 2010, at 2:03 PM, mdipierro wrote: The fact that you are having the problem with static files is

[web2py] Re: rocket performance issues

2010-07-22 Thread mdipierro
I can reproduce the problem. I did on localhost with two different browsers. Using firebug I can see it takes 25seconds to download base.css (the problem is not always with the same file). While I did the test, I also monitored httpserver.log and I find that it NEVER takes more than 1.2ms serve

[web2py] Re: rocket performance issues

2010-07-22 Thread Timbo
Try one quick change for me please...rocket is constructed around line 655 in main.py Add a parameter to the constructor call(s): max_threads=0 Please let me know if that affects the problem. -tim On Jul 22, 10:34 am, mdipierro mdipie...@cs.depaul.edu wrote: I can reproduce the problem. I

Re: [web2py] Re: rocket performance issues

2010-07-22 Thread Michael Ellis
Much improved with suggested change. Latencies for .js/.css files reduced to 100-150 msec. Thanks Tim! Cheers, Mike On Thu, Jul 22, 2010 at 12:27 PM, Timbo tfarr...@owassobible.org wrote: Try one quick change for me please...rocket is constructed around line 655 in main.py Add a

[web2py] Re: rocket performance issues

2010-07-22 Thread Timbo
OK, so it's a thread-starvation issue. Let this be the interim fix for web2py. I've got several busy days ahead of me so I'll put out a real fix for Rocket in the coming days. -tim On Jul 22, 11:39 am, Michael Ellis michael.f.el...@gmail.com wrote: Much improved with suggested change.  

Re: [web2py] Re: rocket performance issues

2010-07-22 Thread Phyo Arkar
Thats very good news. I will be trying out soon! On 7/22/10, Timbo tfarr...@owassobible.org wrote: OK, so it's a thread-starvation issue. Let this be the interim fix for web2py. I've got several busy days ahead of me so I'll put out a real fix for Rocket in the coming days. -tim On Jul

[web2py] Re: rocket performance issues

2010-07-21 Thread mdipierro
Kuba, please try set cron=False in web2py.py. Does it make things better? Massimo On Jul 21, 10:33 am, Kuba Kucharski kuba.kuchar...@gmail.com wrote: sometimes, once every few hits, page loads significantly slower weirdest thing is when you re-click the link it loads instantly, when you left

Re: [web2py] Re: rocket performance issues

2010-07-21 Thread Kuba Kucharski
yes, but I am on the GPRS connection now, I am not able to test it so the results be reliable, I will try to do that asap

[web2py] Re: rocket performance issues

2010-07-21 Thread MikeEllis
I tried changing to cron=False in web2py.py. No impact on problem. Same two-laptop setup as described previous. Happens even on pages that contain very little. Chrome DevTools shows same problem as before: One or more the .js/.css/.png files are delayed by 20 to 25 seconds latency. The

[web2py] Re: rocket performance issues

2010-07-21 Thread MikeEllis
One more detail. As you would hope, it's generally not the case that both browsers stall at the same time. While stalled on either laptop I can keep reloading on the other. On Jul 21, 3:03 pm, MikeEllis michael.f.el...@gmail.com wrote: I tried changing to cron=False in web2py.py.  No impact on

[web2py] Re: rocket performance issues

2010-07-21 Thread mdipierro
The fact that you are having the problem with static files is giving me an idea for a test. Can you please try the web2py I just posted in trunk? Massimo On Jul 21, 2:08 pm, MikeEllis michael.f.el...@gmail.com wrote: One more detail.  As you would hope, it's generally not the case that both

[web2py] Re: rocket performance issues

2010-07-21 Thread MikeEllis
Alas, no change with Version 1.81.4 (2010-07-17 16:14:44) One more little observation: In general, I can load a page from either, but not both, of the browsers repeatedly and not get the problem. It happens with alternation, as discussed previously, but I now notice that I seem to be able to

Re: [web2py] Re: rocket performance issues

2010-07-21 Thread Jonathan Lundell
On Jul 21, 2010, at 2:03 PM, mdipierro wrote: The fact that you are having the problem with static files is giving me an idea for a test. Can you please try the web2py I just posted in trunk? That seems like a nice optimization, but (assuming it fixes the symptom) does it explain the

Re: [web2py] Re: rocket performance

2010-04-22 Thread Thadeus Burgess
I know I have been talking about x-sendfile alot lately Is there any reason that rocket should support it? -- Thadeus On Tue, Apr 20, 2010 at 4:06 PM, mdipierro mdipie...@cs.depaul.edu wrote: ok. will fix this soon. On Apr 20, 4:03 pm, Timothy Farrell tfarr...@swgen.com wrote: Ton's

Re: [web2py] Re: rocket performance

2010-04-22 Thread Timothy Farrell
This is the first I'd heard of x-sendfile so I looked it up. It seems to work to off-load work from the application server to the web-server for serving large files. In theory, there would be no real benefit from this in Rocket since Rocket and web2py run in the same process. Serving a file

[web2py] Re: rocket performance

2010-04-20 Thread mdipierro
I made a mistake. Left references to cherry in some of the pages in applications/example. Could you help me locate them and suggest changes? Massimo On Apr 20, 8:11 am, Timothy Farrell tfarr...@swgen.com wrote: Thank you.  It's always good to see the fruits of one's labor come out. On

Re: [web2py] Re: rocket performance

2010-04-20 Thread Timothy Farrell
Ton's of references in epydoc. Those will be there as long as wsgiserver.py is in the distro. Otherwise, run epydoc and sphinx. Other than that, theres: \examples\views\default\index.html:line 35 - change link to http://launchpad.net/rocket \examples\views\default\who.html:line 77 - remove

[web2py] Re: rocket performance

2010-04-20 Thread mdipierro
ok. will fix this soon. On Apr 20, 4:03 pm, Timothy Farrell tfarr...@swgen.com wrote: Ton's of references in epydoc.  Those will be there as long as wsgiserver.py is in the distro.  Otherwise, run epydoc and sphinx. Other than that, theres: \examples\views\default\index.html:line 35 -

[web2py] Re: rocket performance

2010-04-19 Thread mdipierro
:-) On Apr 19, 3:57 pm, elffikk elff...@gmail.com wrote:  a simple test loading one by one the same link, leaving concurrency and advanced testing to Tim and others :) just wanted to feel the improvement, and I have to say Tim did his job very well web2py 1.76.5  vs 1.77.1 load test 10