[web2py] Re: Send Errors To Email
If you want a realtime solution, you can trigger an email on each error.
Here's a sketch:
1. Add an error handler to routes.py:
routes_onerror = [
('appname/*', '/appname/default/show_error')
]
2. and in default.py:
def show_error():
Scheduler.insert(function_name='send_self_email',
vars=sj.dumps({ticket=request.vars.ticket}))
return 'Server Error. Super sorry about that! We be lookin into it.'
3. Now in a models file add the scheduler email function:
def send_self_email(ticket):
message = 'http://%s:%s/admin/default/ticket/%s' % (server,port,ticket)
SENDMAIL = "/usr/sbin/sendmail" # sendmail location
import os
p = os.popen("%s -t" % SENDMAIL, "w")
p.write("To: " + email_address + "\n")
p.write("Subject: " + subject + "\n")
p.write("\n") # blank line separating headers from body
p.write(message)
p.write("\n")
status = p.close()
Scheduler(db)
4. Run the scheduler with the server with python web2py.py -K appname -X
On Thursday, September 27, 2012 5:32:36 AM UTC-7, Hassan Alnatour wrote:
>
> Dear ALL ,
>
> Is there a way to make all the error be sent to my email when it happens ?
>
> Best regards,
>
--
[web2py] Re: Usage example for 'executesql(...,fields=,columns=) allows parsing of results in Rows'
This is awesome! Thanks for the example!
On Thursday, August 30, 2012 1:56:09 PM UTC-7, Anthony wrote:
>
>
>
> db.define_table('person', Field('name'), Field('email'))
> db.define_table('dog', Field('name'), Field('owner', 'reference person'))
> db.executesql([SQL code returning person.name and dog.name fields], fields
> =[db.person.name, db.dog.name])
> db.executesql([SQL code returning all fields from db.person], fields=db.
> person)
> db.executesql([SQL code returning all fields from both tables], fields=[db
> .person, db.dog])
> db.executesql([SQL code returning person.name and all db.dog fields],fields
> =[db.person.name, db.dog])
>
>
--
[web2py] Re: web2py 2.0.2 is out
I just discovered this sweet hidden improvement: >> db(db.mytable.id>1).select() The Rows object now prints out the number of rows in the repr() function! That's so useful! Thanks everyone! --
[web2py] Re: web2py 2.0.2 is out
Oh, I see, these are scheduler.py options! -b: sets the heartbeat time -L: sets the logging level -E: sets the max empty runs On Wednesday, August 29, 2012 10:23:29 PM UTC-7, Michael Toomim wrote: > > What do -E -b -L do? I don't see them in --help or in the widget.py code. --
[web2py] Re: Changing DB passwords, and database/*.table files
This makes sense to me too! The simple way would break backwards compatibility. But this could be avoided if hash function first checks to see if a schema file exists WITH the password, and returns that, else returns a hash w/o the password. On Tuesday, August 28, 2012 10:17:02 AM UTC-7, Chris wrote: > > > (2) I wonder if a change to the current behavior would be better -- > change the logic to build a hash using all of the URI except the password > part? Changing server or DB name feels like a real change to me; and in > general changing just the user ID is too since the user may have different > permissions, views etc. in the same database. But changing just the > password? Should not change the underlying identity of the database > connection or database object definitions. (In my view of the world.) > What do you think? > --
Re: [web2py] Re: web2py 2.0 stats
That is true! It makes me think, perhaps it would be worthwhile at some point to pause, take stock of all the features, which ones might be better than which other ones, and write up a set of best practices to put into the book. On Wednesday, August 29, 2012 12:49:29 PM UTC-7, Richard wrote: > > With 2.0 I am sometimes thinking of recoding my app start almost 3 years > ago from scratch, since there is so much new features since that time... > > And it's very difficult to know sometimes what the best practices or which > experimental feature will stay in the future or not. > > But I love web2py > > :) > > Richard > > On Wed, Aug 29, 2012 at 3:45 PM, vinic...@gmail.com < > vinic...@gmail.com > wrote: > >> +1. >> >> That's a big weakness in our beloved framework. >> But, unfortunately, I cannot help with that. I've tried in the past, but >> it's difficult to follow pitch of new features. >> >> -- >> Vinicius Assef >> >> >> >> On 08/29/2012 02:32 PM, apps in tables wrote: >> >>> Examples and Self explanatory documentation will save you a lot >>> questions, specially from me:) >>> >>> Ashraf >>> >>> -- >>> >>> >>> >>> >> -- >> >> >> >> > --
[web2py] Re: A new and easier way to test your apps
Sometimes you write things that are just really exciting. --
[web2py] Re: web2py 2.0.2 is out
I'm really excited about the new scheduler -X option. What do -E -b -L do? I don't see them in --help or in the widget.py code. On Wednesday, August 29, 2012 10:17:48 PM UTC-7, Michael Toomim wrote: > > Wow, this is cool! > > But I'm hitting a bug in rewrite_on_error: > http://code.google.com/p/web2py/issues/detail?id=964 > --
[web2py] Re: web2py 2.0.2 is out
Wow, this is cool! But I'm hitting a bug in rewrite_on_error: http://code.google.com/p/web2py/issues/detail?id=964 --
Re: [web2py] Re: possible scheduler bug
Thanks for the great work on the scheduler niphlod! On Wednesday, August 1, 2012 1:19:48 PM UTC-7, Niphlod wrote: > > The consideration behind that is that if your function doesn't return > anything, you don't need the results. Backward compatibility is quite > broken in that sense (but scheduler is still marked as experimental). > If the function returns something, the scheduler_run record is preserved > (unless the discard_results parameter is set to True). > We thought that if for some strange reason a user still needs the > scheduler_run record, he could easily add a "return 1" at the end and the > behaviour of the "old" scheduler is preserved. > Traceback are always saved (no matter what). > > > On Wednesday, August 1, 2012 9:56:55 PM UTC+2, Vincenzo Ampolo wrote: >> >> On 08/01/2012 12:49 PM, Niphlod wrote: >> > Happy to see someone noticing actual improvements :P >> > If you need any further explanations / tips&tricks, just ask. >> > >> > PS: another italian added to web2py-users ;) >> >> Thanks :) >> >> just some considerations though. >> >> I noticed that now in scheduler_run all the COMPLETED tasks get >> cancelled. >> Will it happen even if the completed one have an output? >> >> If there is a FAILED am I going to see the traceback from the >> scheduler_task -> scheduler_run relation like I was doing before? >> >> btw went up to 14 workers. only 3.00 load :) woot woot (that means that >> the overall scheduling system is *really* lighweight) >> >> -- >> Vincenzo Ampolo >> http://vincenzo-ampolo.net >> http://goshawknest.wordpress.com >> > --
Re: [web2py] Re: Broken admin in trunk?
Ah, so I was wrong, great, thank you! On Jul 10, 2012, at 3:20 PM, Massimo Di Pierro wrote: > The normal behavior is, as Dave indicated, that you must be over https or > from localhost. The change in trunk is that the condition is false, the login > form is not even displayed to prevent you from accidentally submitting > credentials over insecure channel. > > You have hacked admin to bypass those conditions. Now you have one more place > to hack it, in the admin/default/index.html view.
Re: [web2py] Re: Broken admin in trunk?
That's not the problem. I've already hacked it. That displays "admin disabled because insecure channel." On Jul 10, 2012, at 3:00 PM, Dave wrote: > the /admin app only works either over https or via 127.0.0.1 / ::1 > > You can hack the app to change that, but it is default behavior > > On Tuesday, July 10, 2012 5:04:43 PM UTC-4, Michael Toomim wrote: > I just upgraded to the trunk. I'm trying to log into the admin, but there's > no password entry box. > > What's wrong? How can I debug this? > >
[web2py] Re: please help us test web2py
On Friday, July 6, 2012 6:35:43 PM UTC-7, Massimo Di Pierro wrote: > > 2. Remove "Share" link from welcome app >> I think we agreed to remove "Share" link because it's not used very much. >> > > I think we agreed to remove the link to addtoany. Do you really want to > remove the share tab at the bottom? I think it can be useful. > +1 on removing it... The internet is covered with too many meaningless share buttons already. It looks bad. There's nothing to share in the welcome app.
[web2py] Re: Best practice using scheduler as a task queue?
This is a nice solution, and clever, thanks!
The upside (compared to postgres locks, as discussed above) is this works
for any database. The downside is it creates a whole new table.
On Thursday, July 5, 2012 2:49:36 PM UTC-7, nick name wrote:
>
> This might have been solved in this week, but in case it wasn't:
>
> You're tackling a general database problem, not a specific task queue or
> web2py problem. So you need to solve it with the database: set up another
> table to refer to you the task table, such as:
>
> db.define_table('tasks_that_were_set_up',
>Field('name', 'string', unique=true),
>Field('scheduler', db.scheduler_task, notnull=True, required=True,
> unique=True, ondelete='CASCADE'),
> )
>
> Make your insert code insert a task to this table as well:
>
> try:
> rid = db.scheduler_task.insert(name='task7',)
> db.tasks_that_were_set_up.insert(name='task7', scheduler=rid)
> print 'Task was just added to db'
> except db.integrity_error_class():
> print 'Tasks were already in db */
>
> Now, if the task gets removed from the scheduler_task table, because of
> the cascading on_delete (which is the default - I just put it there for
> emphasis), the record in tasks_that_were_set_up will be removed as well.
> And otherwise, because of the unique constraint, the insert to
> tasks_that_were_set_up can only succeed once -- and thanks to the
> transaction nature -- therefore so does scheduler_task.insert.
>
>
Re: [web2py] Installing scheduler on linux
Maybe this should go into the docs somewhere. Maybe the scheduler docstring
next to the upstart script? Maybe post an issue on google code to update the
docs? http://code.google.com/p/web2py/issues/list
On Jun 28, 2012, at 5:41 AM, Tyrone wrote:
> Hi Guys,
>
> Although this script works great when upstart is available, when upstart is
> not available you can't do it like this. I struggled to find a solution on
> here or anywhere online, so I have made an alternative script which will work
> on sysvinit (/etc/init.d)
>
> #!/bin/sh
>
> DAEMON=/usr/local/bin/python
> PARAMETERS="/var/www/html/web2py/web2py.py -K init"
> LOGFILE=/var/log/web2py-scheduler.log
>
> start() {
> echo -n "starting up $DAEMON"
> RUN=`$DAEMON $PARAMETERS > $LOGFILE 2>&1`
> if [ "$?" -eq 0 ]; then
> echo " Done."
> else
> echo " FAILED."
> fi
> }
> stop() {
> killall $DAEMON
> }
> status() {
> killall -0 $DAEMON
> if [ "$?" -eq 0 ]; then
> echo "Running."
> else
> echo "Not Running."
> fi
> }
> case "$1" in
> start)
> start
> ;;
> restart)
> stop
> sleep 2
> start
> ;;
> stop)
> stop
> ;;
> status)
> status
> ;;
> *)
> echo "usage : $0 start|restart|stop|status"
> ;;
> esac
> exit 0
>
> I installed this on CentOS release 5.5, however I needed to follow these
> instructions to install:
>
> 1.) # vi web2py-scheduler
> 2.) Paste in the above
> 3.) Add in the following 2 lines in the web2py-scheduler file (required for
> centos i believe):
>
> # chkconfig: 2345 90 10
> # description: web2py-scheduler
>
> 4.) make it executable # chmod 755 web2py-scheduler
> 5.) add it to startup # chkconfig --add web2py-scheduler
[web2py] Re: Best practice using scheduler as a task queue?
On Wednesday, June 27, 2012 5:02:26 PM UTC-7, ptressel wrote: > > This won't solve your installation / setup issue, but I wonder if it would > help with the overrun and timeout problems... Instead of scheduling a > periodic task, what about having the task reschedule itself? When it's > done with the queue, schedule itself for later. Remove the time limit so > it can take whatever time it needs to finish the queue. Or maybe launch a > process on startup outside of the scheduler -- when it exhausts the queue, > have it sleep and either wake periodically to check the queue, or have it > waked when something is inserted. > I don't see why we'd do this instead of just setting stop_time=infinity and repeats=0. > Is the transaction processing issue you encountered with PostgreSQL > preventing you from setting up your queue as a real producer consumer > queue, where you could have multiple workers? > No, I only want one worker. The scheduler itself works great as a producer/consumer queue. I may have mislead you with the title of this thread—I'm trying to set up a general repeating background process, not a task queue in particular. Processing a task queue was just one use of the repeating background process function.
Re: [web2py] Re: database locking web2py vs. "external" access...
This is all a great unearthing of the Mystery of Transactions. Thanks for the investigation, Doug. This was difficult for me to learn when I got into web2py as well. Perhaps we could write up all this knowledge somewhere, now that you're figuring it out? Can we have a section on Transactions in the book, or somewhere?
Re: [web2py] Quoting reserved words in DAL
Ok, here's one step closer to a fix! This is what had to be manually changed in order to quote the names of tables/fields to load a postgres database into MySQL. This won't work for other databases. And this only works for importing a database—table creation, with foreign keys, and insert statements. Nothing else (e.g. selects, updates) was tested. And we had to guide it manually through the table creation, to get foreign key constraints satisfied in the right order. But this will show an interested developer exactly where to begin fixing the code for this bug... that seems to be biting a few of us. On Jun 20, 2012, at 1:19 PM, Rene Dohmen wrote: > I'm having the same problem: > https://groups.google.com/d/msg/web2py/hCsxVaDLfT4/K6UMbG5p5uAJ > > > On Mon, Jun 18, 2012 at 9:30 AM, Michael Toomim wrote: > I just got bit by the reserved-word problem: > https://groups.google.com/d/msg/web2py/aSPtD_mGXdM/c7et_2l_54wJ > > I am trying to port a postgres database to a friend's mysql database, but we > are stuck because the DAL does not quote identifiers. > > This problem has been discussed a fair amount: > https://groups.google.com/d/msg/web2py/QKktETHk3yo/Mwm3D-JhNmAJ > > ...and it seems all we need to do is add some quoting mechanisms to the DAL. > In postgresql you surround names with "quotes" and in mysql you use > `backticks`. > > Does anyone have ideas for what to do? > > -- On Jun 20, 2012, at 1:19 PM, Rene Dohmen wrote:I'm having the same problem:https://groups.google.com/d/msg/web2py/hCsxVaDLfT4/K6UMbG5p5uAJ On Mon, Jun 18, 2012 at 9:30 AM, Michael Toomim <too...@gmail.com> wrote: I just got bit by the reserved-word problem:https://groups.google.com/d/msg/web2py/aSPtD_mGXdM/c7et_2l_54wJ I am trying to port a postgres database to a friend's mysql database, but we are stuck because the DAL does not quote identifiers.This problem has been discussed a fair amount: https://groups.google.com/d/msg/web2py/QKktETHk3yo/Mwm3D-JhNmAJ...and it seems all we need to do is add some quoting mechanisms to the DAL. In postgresql you surround names with "quotes" and in mysql you use `backticks`. Does anyone have ideas for what to do? -- dal.py.changes Description: Binary data
Re: [web2py] Re: Best practice using scheduler as a task queue?
:) Because I'm a perfectionist, and I want other developers to be able to install my system by just unzipping the code, running ./serve, and have it just work. So I want to use the built-in webserver and scheduler. There's no reason they shouldn't be able to manage these race conditions correctly. I'm super excited that the Ctrl-C bug was fixed! Your idea of putting the initializer in the @cron reboot is very appealing! I will think about this and see if I can come up with a nice solution with it. Ideally I could re-use this "daemon_task" setup for other projects as well, as I find it to be a quite common scenario. I understand you do not find it to be common. I am not sure why we have different experiences. Would portalocker be a good thing to use for this situation? I would like to be cross-platform instead of relying on postgres locks. On Wednesday, June 27, 2012 12:37:27 PM UTC-7, Niphlod wrote: > > uhm. why not having them started with systemd or upstart or > supervisord ? > > Scheduler is "by design" allowed to run with multiple instances (to > process a longer queue you may want to start more of them), but if you're > really loosing money why didn't you rely on that services to be sure that > there's only one instance running? > There are a lot of nice implementations out there and the one I > mentioned are pretty much "state-of-the-art" :D (while contributing to fix > current issues) > > BTW: - "responding to ctrl+c" fixed in trunk recently >- "os messed up maybe" require you to check the os, python > programs can't be omniscient :D >- "messy developers", no easy fix for that too > > > On Wednesday, June 27, 2012 9:18:06 PM UTC+2, Michael Toomim wrote: >> >> The problem with terminating the processes is: >> • sometimes they don't respond to control-c, and need a kill -9 >> • or sometimes that doesn't work, maybe the os is messed up >> • or sometimes the developer might run two instances simultaneously, >> forgetting that one was already running >> >> You're right that usually I can shut them both down with control-c, but I >> need a safeguard. My application spends money on mechanical turk and I'll >> spend erroneous money and upset my users if it goes wrong by accident. >> >> On Wednesday, June 27, 2012 12:56:52 AM UTC-7, Niphlod wrote: >>> >>> BTW: I'm pretty sure that when you say "scheduler should be terminated >>> alongside web2py" you're not perfectly grasping how webdevelopment in >>> production works. If you're using "standalone" versions, i.e. not mounted >>> on a webserver, you can start your instances as web2py -a mypassword & >>> web2py -K myapp and I'm pretty sure when hitting ctrl+c both will shutdown. >>> >>
Re: [web2py] Re: Best practice using scheduler as a task queue?
The problem with terminating the processes is: • sometimes they don't respond to control-c, and need a kill -9 • or sometimes that doesn't work, maybe the os is messed up • or sometimes the developer might run two instances simultaneously, forgetting that one was already running You're right that usually I can shut them both down with control-c, but I need a safeguard. My application spends money on mechanical turk and I'll spend erroneous money and upset my users if it goes wrong by accident. On Wednesday, June 27, 2012 12:56:52 AM UTC-7, Niphlod wrote: > > BTW: I'm pretty sure that when you say "scheduler should be terminated > alongside web2py" you're not perfectly grasping how webdevelopment in > production works. If you're using "standalone" versions, i.e. not mounted > on a webserver, you can start your instances as web2py -a mypassword & > web2py -K myapp and I'm pretty sure when hitting ctrl+c both will shutdown. >
Re: [web2py] Re: Best practice using scheduler as a task queue?
I'm totally interested in solutions! It's a big problem I need to solve. The recurring maintenance task does not fix the initialization problem—because now you need to initialize the recurring maintenance task. This results in the same race condition. It does fine with the 40,000 records problem. But it's just a lot of complexity we're introducing to solve a simple problem (looping tasks) with a complex solution (scheduler). I'd still love to find a clean way to do this. Maybe we should extend the scheduler like this: • Add a daemon_tasks parameter when you call it from models "Scheduler(db, daemon_tasks=[func1, func2])" • When scheduler boots up, it handles locks and everything and ensures there are two tasks that just call these functions • Then it dispatches the workers processes as usual ...ah, shoot, looking in widget.py, it looks like the code that starts schedulers doesn't have access to the parameters passed to Scheduler() because models haven't been run yet. Hmph. On Wednesday, June 27, 2012 12:56:52 AM UTC-7, Niphlod wrote: > > I don't know if continuing to give you fixes and alternative > implementations is to be considered as harassment at this point, stop me if > you're not interested into those. > > There is a very biiig problem in your statements: if your vision is > > Woohoo, this scheduler will *automatically handle locks*—so I don't need > to worry about stray background processes running in parallel > automatically, and it will *automatically start/stop the processes* with > the web2py server with -K, which makes it much easier to deploy the code! > > then the scheduler is the right tool for you. it's your app that doesn't > handle locks, because of your initialization code put into models. > > At least 2 of your problems (initialization and 40,000 scheduler_run > records) could be fixed by a "recurring maintenance" task that will do > check_daemon() without advisory locks and prune the scheduler_run table. > > BTW: I'm pretty sure that when you say "scheduler should be terminated > alongside web2py" you're not perfectly grasping how webdevelopment in > production works. If you're using "standalone" versions, i.e. not mounted > on a webserver, you can start your instances as web2py -a mypassword & > web2py -K myapp and I'm pretty sure when hitting ctrl+c both will shutdown. >
Re: [web2py] Re: Best practice using scheduler as a task queue?
In case it is useful to someone, here is the full code I used with locking,
using postgresql advisory locks. The benefit of using postgresql's locks
are that:
• It locks on the database—works across multiple clients
• The locks are automatically released if a client disconnects from the db
• I think it's fast
def check_daemon(task_name, period=None):
period = period or 4
tasks_query = ((db.scheduler_task.function_name == task_name)
& db.scheduler_task.status.belongs(('QUEUED',
'ASSIGNED',
'RUNNING',
'ACTIVE')))
# Launch a launch_queue task if there isn't one already
tasks = db(tasks_query).select()
if len(tasks) > 1: # Check for error
raise Exception('Too many open %s tasks!!! N, there are %s'
% (task_name, len(tasks)))
if len(tasks) < 1:
if not db.executesql('select pg_try_advisory_lock(1);')[0][0]:
debug('Tasks table is already locked.')
return
# Check again now that we're locked
if db(tasks_query).count() >= 1:
debug('Caught a race condition! Glad we got outa there!')
db.executesql('select pg_advisory_unlock(1);')
return
debug('Adding a %s task!', task_name)
db.scheduler_task.insert(function_name=task_name,
application_name='utility/utiliscope',
task_name=task_name,
stop_time = now +
timedelta(days=9),
repeats=0, period=period)
db.commit()
db.executesql('select pg_advisory_unlock(1);')
elif tasks[0].period != period:
debug('Updating period for task %s', task_name)
tasks[0].update_record(period=period)
db.commit()
check_daemon('process_launch_queue_task')
check_daemon('refresh_hit_status')
check_daemon('process_bonus_queue')
On Tuesday, June 26, 2012 7:57:25 PM UTC-7, Michael Toomim wrote:
>
> All, thank you for the excellent discussion!
>
> I should explain why I posted that recommendation. The "vision" of using
> the scheduler for background tasks was:
>
> "Woohoo, this scheduler will *automatically handle locks*—so I don't need
> to worry about stray background processes running in parallel
> automatically, and it will *automatically start/stop the processes* with
> the web2py server with -K, which makes it much easier to deploy the code!"
>
>
> It turned out:
> • Setting up scheduler tasks was complicated in itself.
> • 3 static tasks had to be inserted into every new db.
>This requires new installations of my software to run a setup
> routine. Yuck.
> • When I made that automatic in models/, it required locks to avoid
> db race condition.
>(I used postgresql advisory locks. Not cross-platform, but I dunno
> a better solution.)
> • The goal was to avoid locks in the first place!
> • When things go wrong, it's harder to debug.
> • The scheduler adds a new layer of complexity.
> • Because now I have to make sure my tasks are there properly.
> • And then look for the scheduler_run instances to see how they went.
>
> I must admit that this second problem would probably go away if we fixed
> all the scheduler's bugs! But it still leaves me uneasy. And I don't like
> having 40,000 scheduler_run instances build up over time.
>
> At this point, I realized that what I really want is a new feature in
> web2py that:
> • Runs a function in models (akin to scheduler's executor function) in a
> subprocess repeatedly
> • Ensures, with locks etc., that:
> • Only one is running at a time
> • That it dies if the parent web2py process dies
>
> And it seems better to just implement this as a web2py feature, than to
> stranglehold the scheduler into a different design.
>
> Cron's @reboot is very close to this. I used to use it. The problems:
> • I still had to implement my own locks and kills. (what I was trying to
> avoid)
> • It spawns 2 python subprocesses for each cron task (ugly, but not
> horrible)
> • It was really buggy. @reboot didn't work. I think massimo fixed this.
> • Syntax is gross.
> I basically just got scared of cron.
> Now I guess I'm scared of everything. :/
>
> Hopefully this detailed report of my experience will be of help to
> somebody.
Re: [web2py] Re: Best practice using scheduler as a task queue?
All, thank you for the excellent discussion!
I should explain why I posted that recommendation. The "vision" of using
the scheduler for background tasks was:
"Woohoo, this scheduler will *automatically handle locks*—so I don't need
to worry about stray background processes running in parallel
automatically, and it will *automatically start/stop the processes* with
the web2py server with -K, which makes it much easier to deploy the code!"
It turned out:
• Setting up scheduler tasks was complicated in itself.
• 3 static tasks had to be inserted into every new db.
This requires new installations of my software to run a setup
routine. Yuck.
• When I made that automatic in models/, it required locks to avoid db
race condition.
(I used postgresql advisory locks. Not cross-platform, but I dunno a
better solution.)
• The goal was to avoid locks in the first place!
• When things go wrong, it's harder to debug.
• The scheduler adds a new layer of complexity.
• Because now I have to make sure my tasks are there properly.
• And then look for the scheduler_run instances to see how they went.
I must admit that this second problem would probably go away if we fixed
all the scheduler's bugs! But it still leaves me uneasy. And I don't like
having 40,000 scheduler_run instances build up over time.
At this point, I realized that what I really want is a new feature in
web2py that:
• Runs a function in models (akin to scheduler's executor function) in a
subprocess repeatedly
• Ensures, with locks etc., that:
• Only one is running at a time
• That it dies if the parent web2py process dies
And it seems better to just implement this as a web2py feature, than to
stranglehold the scheduler into a different design.
Cron's @reboot is very close to this. I used to use it. The problems:
• I still had to implement my own locks and kills. (what I was trying to
avoid)
• It spawns 2 python subprocesses for each cron task (ugly, but not
horrible)
• It was really buggy. @reboot didn't work. I think massimo fixed this.
• Syntax is gross.
I basically just got scared of cron.
Now I guess I'm scared of everything. :/
Hopefully this detailed report of my experience will be of help to
somebody. I'm sure that fixing the bugs will make things 5x better. I will
try your new scheduler.py Niphlod!
On Tuesday, June 26, 2012 12:13:32 PM UTC-7, Niphlod wrote:
>
> problem here started as "I can't ensure my app to insert only one task per
> function", that is not a scheduler problem "per se": it's a common database
> problem. Would have been the same if someone created a
> db.define_table('mytable',
> Field('name'),
> Field('uniquecostraint')
> )
> and have to ensure, without specifying Field('uniquecostraint',
> unique=True) that there are no records with the same value into the column
> uniquecostraint.
>
> From there to "now I have tasks stuck in RUNNING status, please avoid
> using the scheduler" without any further details, the leap is quite
> "undocumented".
>
> And please do note that scheduler in trunk has gone under some changes:
> there was a point in time where abnormally killed schedulers (as kill
> -SIGKILL the process) left tasks in RUNNING status, that would not be
> picked up by subsequent scheduler processes.
>
> That was a design issue: if a task is RUNNING and you kill scheduler while
> the task was processed, you had no absolutely way to tell what the function
> did (say, send a batch of 500 emails) before it was actually killed.
> If the task was not planned properly it could send e.g. 359 mails, be
> killed, and if it was picked up again by another scheduler after the "first
> killed round" 359 of your recipients would get 2 identical mails.
> It has been decided to requeue RUNNING tasks without any active worker
> doing that (i.e. leave to the function the eventual check of what has been
> done), so now RUNNING tasks with a dead worker assigned get requeued.
>
> With other changes (soon in trunk, the previously attached file) you're
> able to stop workers, so they may be killed "ungracefully" being sure that
> they're not processing tasks.
>
> If you need more details, as always, I'm happy to help, and other
> developers too, I'm sure :D
>
>
--
[web2py] Re: Best practice using scheduler as a task queue?
Er, let me rephrase: I don't recommend using the scheduler for *infinitely
looping background tasks*.
On Monday, June 25, 2012 4:54:30 PM UTC-7, Michael Toomim wrote:
>
> This scenario is working out worse and worse.
>
> Now I'm getting tasks stuck in the 'RUNNING' state... even when there
> aren't any scheduler processes running behind them running! I'm guessing
> the server got killed mid-process, and now it doesn't know how to recover.
> Looks like a bug in the scheduler.
>
> I don't recommend using the scheduler as a task queue to anybody.
>
> On Tuesday, June 12, 2012 10:24:15 PM UTC-7, Michael Toomim wrote:
>>
>> Here's a common scenario. I'm looking for the best implementation using
>> the scheduler.
>>
>> I want to support a set of background tasks (task1, task2...), where each
>> task:
>> • processes a queue of items
>> • waits a few seconds
>>
>> It's safe to have task1 and task2 running in parallel, but I cannot have
>> two task1s running in parallel. They will duplicately process the same
>> queue of items.
>>
>> I found the scheduler supports this nicely with parameters like:
>>
>> db.scheduler_task.insert(function_name='task1',
>> task_name='task1',
>> stop_time = now + timedelta(days=9),
>> repeats=0,
>> period=10)
>>
>> I can launch 3 workers, and they coordinate amongst themselves to make
>> sure that only one will run the task at a time. Great! This task will last
>> forever...
>>
>> ...but now we encounter my problem...
>>
>> What happens if it crashes, or passes stop_time? Then the task will turn
>> off, and the queue is no longer processed. Or what happens if I reset the
>> database, or install this code on a new server? It isn't nice if I have to
>> re-run the insert function by hand.
>>
>> So how can I ensure there is always EXACTLY ONE of each task in the
>> database?
>>
>> I tried putting this code into models:
>>
>> def initialize_task_queue(task_name):
>> num_tasks = db((db.scheduler_task.function_name == task_name)
>>& ((db.scheduler_task.status == 'QUEUED')
>> | (db.scheduler_task.status == 'ASSIGNED')
>> | (db.scheduler_task.status == 'RUNNING')
>> | (db.scheduler_task.status == 'ACTIVE'))).count()
>>
>> # Add a task if there isn't one already
>> if num_tasks < 1:
>> db.scheduler_task.insert(function_name=task_name,
>> task_name=task_name,
>> stop_time = now + timedelta(days=9),
>> repeats=0,
>> period=period)
>> db.commit()
>>
>> initialize_task_queue('task1')
>> initialize_task_queue('task2')
>> initialize_task_queue('task3')
>>
>> This worked, except it introduces a race condition! If you start three
>> web2py processes simultaneously (e.g., for three scheduler processes), they
>> will insert duplicate tasks:
>> process 1: count number of 'task1' tasks
>> process 2: count number of 'task1' tasks
>> process 1: there are less than 1, insert a 'task1' task
>> process 2: there are less than 1, insert a 'task1' task
>>
>> I was counting on postgresql's MVCC transaction support to make each of
>> these atomic. Unfortunately, that's not how it works. I do not understand
>> why. As a workaround, I'm currently wrapping the code inside
>> "initialize_task_queue" with postgresql advisory lock:
>>
>> if not db.executesql('select pg_try_advisory_lock(1);')[0][0]:
>> return
>>
>> ... count tasks, add one if needed ...
>>
>> db.executesql('select pg_advisory_unlock(1);')
>>
>> But this sucks.
>> What's a better way to ensure there is always 1 infinite-repeat task in
>> the scheduler? Or... am I using the wrong design entirely?
>>
>
--
[web2py] Re: Best practice using scheduler as a task queue?
This scenario is working out worse and worse.
Now I'm getting tasks stuck in the 'RUNNING' state... even when there
aren't any scheduler processes running behind them running! I'm guessing
the server got killed mid-process, and now it doesn't know how to recover.
Looks like a bug in the scheduler.
I don't recommend using the scheduler as a task queue to anybody.
On Tuesday, June 12, 2012 10:24:15 PM UTC-7, Michael Toomim wrote:
>
> Here's a common scenario. I'm looking for the best implementation using
> the scheduler.
>
> I want to support a set of background tasks (task1, task2...), where each
> task:
> • processes a queue of items
> • waits a few seconds
>
> It's safe to have task1 and task2 running in parallel, but I cannot have
> two task1s running in parallel. They will duplicately process the same
> queue of items.
>
> I found the scheduler supports this nicely with parameters like:
>
> db.scheduler_task.insert(function_name='task1',
> task_name='task1',
> stop_time = now + timedelta(days=9),
> repeats=0,
> period=10)
>
> I can launch 3 workers, and they coordinate amongst themselves to make
> sure that only one will run the task at a time. Great! This task will last
> forever...
>
> ...but now we encounter my problem...
>
> What happens if it crashes, or passes stop_time? Then the task will turn
> off, and the queue is no longer processed. Or what happens if I reset the
> database, or install this code on a new server? It isn't nice if I have to
> re-run the insert function by hand.
>
> So how can I ensure there is always EXACTLY ONE of each task in the
> database?
>
> I tried putting this code into models:
>
> def initialize_task_queue(task_name):
> num_tasks = db((db.scheduler_task.function_name == task_name)
>& ((db.scheduler_task.status == 'QUEUED')
> | (db.scheduler_task.status == 'ASSIGNED')
> | (db.scheduler_task.status == 'RUNNING')
> | (db.scheduler_task.status == 'ACTIVE'))).count()
>
> # Add a task if there isn't one already
> if num_tasks < 1:
> db.scheduler_task.insert(function_name=task_name,
> task_name=task_name,
> stop_time = now + timedelta(days=9),
> repeats=0,
> period=period)
> db.commit()
>
> initialize_task_queue('task1')
> initialize_task_queue('task2')
> initialize_task_queue('task3')
>
> This worked, except it introduces a race condition! If you start three
> web2py processes simultaneously (e.g., for three scheduler processes), they
> will insert duplicate tasks:
> process 1: count number of 'task1' tasks
> process 2: count number of 'task1' tasks
> process 1: there are less than 1, insert a 'task1' task
> process 2: there are less than 1, insert a 'task1' task
>
> I was counting on postgresql's MVCC transaction support to make each of
> these atomic. Unfortunately, that's not how it works. I do not understand
> why. As a workaround, I'm currently wrapping the code inside
> "initialize_task_queue" with postgresql advisory lock:
>
> if not db.executesql('select pg_try_advisory_lock(1);')[0][0]:
> return
>
> ... count tasks, add one if needed ...
>
> db.executesql('select pg_advisory_unlock(1);')
>
> But this sucks.
> What's a better way to ensure there is always 1 infinite-repeat task in
> the scheduler? Or... am I using the wrong design entirely?
>
--
[web2py] Quoting reserved words in DAL
I just got bit by the reserved-word problem: https://groups.google.com/d/msg/web2py/aSPtD_mGXdM/c7et_2l_54wJ I am trying to port a postgres database to a friend's mysql database, but we are stuck because the DAL does not quote identifiers. This problem has been discussed a fair amount: https://groups.google.com/d/msg/web2py/QKktETHk3yo/Mwm3D-JhNmAJ ...and it seems all we need to do is add some quoting mechanisms to the DAL. In postgresql you surround names with "quotes" and in mysql you use `backticks`. Does anyone have ideas for what to do?
[web2py] Re: Best practice using scheduler as a task queue?
To respond to your last two points: You're right that models only runs on every request... I figured if my website isn't getting any usage then the tasks don't matter anyway. :P Yes, I think there are design issues here, but I haven't found a better solution. I'm very interested in hearing better overall solutions! The obvious alternative is to write a standalone script that loops forever, and launch it separately using something like "python web2py.py -S app/controller -M -N -R background_work.py -A foo". But this requires solving the following problems that are *already solved* by the scheduler: • During development, restarting & reloading models as I change code • Killing these background processes when I quit the server • Ensuring that no more than one background process runs at a time On Wednesday, June 13, 2012 7:16:56 AM UTC-7, Niphlod wrote: > > Maybe I didn't get exactly what you need , but .. you have 3 tasks, > that needs to be unique. > Also, you want to be sure that if a task crashes doesn't remain "hanged". > > This should never happen with the scheduler the worst situation is > that if a worker crashes (here "crashes" is it disconnects from the > database) leaves the task status as running, but as soon as another > scheduler checks if that one sends heartbeats, he removes the dead worker > and requeue that task. > If your task goes into timeout and it's a repeating task the best practice > should be to raise the timeout. > > Assured this, you need to initialize the database if someone truncates the > scheduler_task table, inserting the 3 records in one transaction. > > If you need to be sure, why all the hassle when you can "prepare" the > task_name column as a unique value and then do > db.update_or_insert(task_name==myuniquetaskname, **task_record) ? > > PS: code in models get executed every request. What if you have no users > accessing the site and in the need to call initialize_task_queue ? Isn't it > better to insert the values and then start the workers ? > > BTW: a task that needs to be running "forever" but can't be "launched" in > two instances seems to suffer some design issues but hey, everyone needs to > be able to do what he wants ;-) >
[web2py] Re: Best practice using scheduler as a task queue?
Thanks for the response, niphlod! Let me explain: The task can be marked FAILED or EXPIRED if: • The code in the task throws an exception • A run of the task exceeds the timeout • The system clock goes past stop_time And it will just not plain exist if: • You have just set up the code • You install the code on a new database In order to implement a perpetually processing task queue, we need to ensure that there is always an active "process queue" task ready in the scheduler. So what's the best way to do this without creating race conditions? This method creates a race condition: 1. Check if task exists 2. Insert if not ...if multiple processes are checking and inserting at the same time. The only solution I've found is to wrap that code within a postgresql database advisory lock, but this only works on postgres. (And the update_or_insert() function you mentioned does the same thing as this internally, so it still suffers from a race condition.) On Wednesday, June 13, 2012 7:16:56 AM UTC-7, Niphlod wrote: > > Maybe I didn't get exactly what you need , but .. you have 3 tasks, > that needs to be unique. > Also, you want to be sure that if a task crashes doesn't remain "hanged". > > This should never happen with the scheduler the worst situation is > that if a worker crashes (here "crashes" is it disconnects from the > database) leaves the task status as running, but as soon as another > scheduler checks if that one sends heartbeats, he removes the dead worker > and requeue that task. > If your task goes into timeout and it's a repeating task the best practice > should be to raise the timeout. > > Assured this, you need to initialize the database if someone truncates the > scheduler_task table, inserting the 3 records in one transaction. > > If you need to be sure, why all the hassle when you can "prepare" the > task_name column as a unique value and then do > db.update_or_insert(task_name==myuniquetaskname, **task_record) ? > > PS: code in models get executed every request. What if you have no users > accessing the site and in the need to call initialize_task_queue ? Isn't it > better to insert the values and then start the workers ? > > BTW: a task that needs to be running "forever" but can't be "launched" in > two instances seems to suffer some design issues but hey, everyone needs to > be able to do what he wants ;-) >
[web2py] Best practice using scheduler as a task queue?
Here's a common scenario. I'm looking for the best implementation using the
scheduler.
I want to support a set of background tasks (task1, task2...), where each
task:
• processes a queue of items
• waits a few seconds
It's safe to have task1 and task2 running in parallel, but I cannot have
two task1s running in parallel. They will duplicately process the same
queue of items.
I found the scheduler supports this nicely with parameters like:
db.scheduler_task.insert(function_name='task1',
task_name='task1',
stop_time = now + timedelta(days=9),
repeats=0,
period=10)
I can launch 3 workers, and they coordinate amongst themselves to make sure
that only one will run the task at a time. Great! This task will last
forever...
...but now we encounter my problem...
What happens if it crashes, or passes stop_time? Then the task will turn
off, and the queue is no longer processed. Or what happens if I reset the
database, or install this code on a new server? It isn't nice if I have to
re-run the insert function by hand.
So how can I ensure there is always EXACTLY ONE of each task in the
database?
I tried putting this code into models:
def initialize_task_queue(task_name):
num_tasks = db((db.scheduler_task.function_name == task_name)
& ((db.scheduler_task.status == 'QUEUED')
| (db.scheduler_task.status == 'ASSIGNED')
| (db.scheduler_task.status == 'RUNNING')
| (db.scheduler_task.status == 'ACTIVE'))).count()
# Add a task if there isn't one already
if num_tasks < 1:
db.scheduler_task.insert(function_name=task_name,
task_name=task_name,
stop_time = now + timedelta(days=9),
repeats=0,
period=period)
db.commit()
initialize_task_queue('task1')
initialize_task_queue('task2')
initialize_task_queue('task3')
This worked, except it introduces a race condition! If you start three
web2py processes simultaneously (e.g., for three scheduler processes), they
will insert duplicate tasks:
process 1: count number of 'task1' tasks
process 2: count number of 'task1' tasks
process 1: there are less than 1, insert a 'task1' task
process 2: there are less than 1, insert a 'task1' task
I was counting on postgresql's MVCC transaction support to make each of
these atomic. Unfortunately, that's not how it works. I do not understand
why. As a workaround, I'm currently wrapping the code inside
"initialize_task_queue" with postgresql advisory lock:
if not db.executesql('select pg_try_advisory_lock(1);')[0][0]:
return
... count tasks, add one if needed ...
db.executesql('select pg_advisory_unlock(1);')
But this sucks.
What's a better way to ensure there is always 1 infinite-repeat task in the
scheduler? Or... am I using the wrong design entirely?
[web2py] Cron problems
I'm finding multiple problems getting cron to start the scheduler. Here's
the cron line:
@reboot dummyuser python web2py.py -K utility
...but it does not work without modifying web2py source.
First, let's get an easy bug out of the way. The web2py book gives this
example for @reboot:
@reboot * * * * root *mycontroller/myfunction
But those asterisks shouldn't be there for @reboot tasks. Can we remove
them from the book?
Now, when I put that line into my crontab and run web2py, it gives me this
error:
web2py Web Framework
Created by Massimo Di Pierro, Copyright 2007-2011
Version 1.99.7 (2012-03-04 22:12:08) stable
Database drivers available: SQLite3, pymysql, psycopg2, pg8000, CouchDB,
IMAP
Starting hardcron...
please visit:
http://192.168.56.101:8000
use "kill -SIGTERM 10818" to shutdown the web2py server
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python2.6/threading.py", line 532, in __bootstrap_inner
self.run()
File "/home/toomim/projects/utility/web2py/gluon/newcron.py", line 234,
in run
shell=self.shell)
File "/usr/lib/python2.6/subprocess.py", line 633, in __init__
errread, errwrite)
File "/usr/lib/python2.6/subprocess.py", line 1139, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
This is an error in subprocess.Popen. I inserted some print statements and
found that it's calling it like this:
subprocess.Popen('python web2py.py -K utility')
This is incorrect, it should be:
subprocess.Popen(['python', 'web2py.py' '-K' 'utility'])
I was able to make it work by adding a call to split(), as you can see here
(in newcron.py: cronlauncher.run()):
def run(self):
import subprocess
proc = subprocess.Popen(self.cmd.split(),
But I do not understand how anybody could have made this work before,
without adding a split() call? And what confuses me further is that there
is an explicit join() call in the __init__() method that runs immediately
beforehand, as if we really did NOT want to have lists:
elif isinstance(cmd,list):
cmd = ' '.join(cmd)
So does cron @reboot work for anybody running a script? It seems impossible
for it to work right now. Is this a bug?
Finally, it would be great if we did not have to pass in a dummy user to
each cron line that does nothing...
[web2py] Re: Changing the controller on the fly
This is working great! It's exactly what I needed, and makes my code much
simpler.
Thank you very much! I love it!
On Friday, May 11, 2012 5:03:51 AM UTC-7, Anthony wrote:
>
> Or to avoid a redirect, you can change the function and controller in a
> model file:
>
> db = DAL(...)
>
> if request.function == 'dispatch':
> request.controller, request.function = [fetch from db]
> response.view = '%s/%s.%s' % (request.controller, request.function,request
> .extension)
> response.generic_patterns = ['html'] # to enable the generic.html
> view if needed
>
> Anthony
>
> On Friday, May 11, 2012 6:07:56 AM UTC-4, simon wrote:
>>
>> You can do:
>>
>> def dispatch():
>> controller,function = ... load these from the database ...
>> redirect(URL(c=controller, f=function, vars=request.vars,
>> args=request.args))
>>
>>
>> On Friday, 11 May 2012 10:17:19 UTC+1, Michael Toomim wrote:
>>>
>>> I need to be able to dispatch to a different controller based on a
>>> database lookup. So a user will go to a url (say '/dispatch'), and we'll
>>> look up in the database some information on that user, choose a new
>>> controller and function, and call that controller and function with its
>>> view.
>>>
>>> I've almost got this working below, but the models are not being loaded
>>> into the new controller. Is there a way to fix that?
>>>
>>> In default.py:
>>> def dispatch():
>>> controller,function = ... load these from the database ...
>>> response.view = '%s/%s.html' % (controller,
>>> function)
>>>
>>> if not os.path.exists(request.folder + '/views/' + response.view):
>>> response.view = 'generic.html'
>>>
>>> from gluon.shell import exec_environment
>>> controller = exec_environment('%s/controllers/%s.py'
>>> % (request.folder,
>>> controller),
>>> request=request,
>>> response=response,
>>> session=session)
>>> return controller[request.task_function]()
>>>
>>> Unfortunately, the controller being called has access to request,
>>> response, and session, but none of the global variables defined in my
>>> models. Is there a way to get exec_environment() to run a function in
>>> another controller WITHOUT losing all the model definitions?
>>>
>>> Or is there a better way to do this?
>>>
>>
[web2py] Changing the controller on the fly
I need to be able to dispatch to a different controller based on a database
lookup. So a user will go to a url (say '/dispatch'), and we'll look up in
the database some information on that user, choose a new controller and
function, and call that controller and function with its view.
I've almost got this working below, but the models are not being loaded
into the new controller. Is there a way to fix that?
In default.py:
def dispatch():
controller,function = ... load these from the database ...
response.view = '%s/%s.html' % (controller,
function)
if not os.path.exists(request.folder + '/views/' + response.view):
response.view = 'generic.html'
from gluon.shell import exec_environment
controller = exec_environment('%s/controllers/%s.py'
% (request.folder,
controller),
request=request,
response=response,
session=session)
return controller[request.task_function]()
Unfortunately, the controller being called has access to request, response,
and session, but none of the global variables defined in my models. Is
there a way to get exec_environment() to run a function in another
controller WITHOUT losing all the model definitions?
Or is there a better way to do this?
[web2py] Re: Installing scheduler on linux
Also: 1. replace "friendbo" with the name of your app. 2. To start/stop the scheduler, use "sudo start web2py-scheduler" "sudo stop web2py-scheduler" "sudo status web2py-scheduler" ...etc. On Saturday, May 5, 2012 6:47:33 PM UTC-7, Michael Toomim wrote: > > Here's a solution I wrote. Is there a good place (like the web2py book) to > share this recipe? > > Put this in /etc/init/web2py-scheduler.conf: > > description "web2py task scheduler" > > start on (local-filesystems and net-device-up IFACE=eth0) > stop on shutdown > > # Give up if restart occurs 8 times in 60 seconds. > > respawn limit 8 60 > > exec sudo -u python /home//web2py/web2py.py -K friendbo > respawn > > This assumes your web2py is in 's home directory, running with > permission . Replace with the right user id and change eth0 if > your server uses a different network interface. > > On Thursday, May 3, 2012 1:22:25 PM UTC-7, Michael Toomim wrote: >> >> Anyone have a recipe to make the scheduler run on boot? I'm using ubuntu. >> Web2py is run in apache (using the recipe in the book), so I can't just use >> the cron @reboot line. >> >> This is the line that needs to be run when my system boots: >>python /home/web2py/web2py/web2py.py -K >> >> It seems ubuntu uses Upstart instead of sysvinit. And it might be >> possible to tell Upstart to launch the scheduler WITH apache, using a line >> like "start on started apache2" but I don't know how it works, and >> apache2 to use upstart. >> >> It would be nice to have a recipe for the scheduler that we could add >> into the book, to synchronize it with apache. >> >
[web2py] Re: Installing scheduler on linux
Here's a solution I wrote. Is there a good place (like the web2py book) to share this recipe? Put this in /etc/init/web2py-scheduler.conf: description "web2py task scheduler" start on (local-filesystems and net-device-up IFACE=eth0) stop on shutdown # Give up if restart occurs 8 times in 60 seconds. respawn limit 8 60 exec sudo -u python /home//web2py/web2py.py -K friendbo respawn This assumes your web2py is in 's home directory, running with permission . Replace with the right user id and change eth0 if your server uses a different network interface. On Thursday, May 3, 2012 1:22:25 PM UTC-7, Michael Toomim wrote: > > Anyone have a recipe to make the scheduler run on boot? I'm using ubuntu. > Web2py is run in apache (using the recipe in the book), so I can't just use > the cron @reboot line. > > This is the line that needs to be run when my system boots: >python /home/web2py/web2py/web2py.py -K > > It seems ubuntu uses Upstart instead of sysvinit. And it might be possible > to tell Upstart to launch the scheduler WITH apache, using a line like "start > on started apache2" but I don't know how it works, and apache2 to use > upstart. > > It would be nice to have a recipe for the scheduler that we could add into > the book, to synchronize it with apache. >
[web2py] Re: Why Bottle and Web2py?
I think the best combination of web2py and bottle would be, as you suggested—importing the web2py DAL into bottle. The DAL is the most important thing that bottle lacks, and the web2py DAL is great to plug into other projects. I use it a lot for that. That said, in my experience, you will quickly want a templating language to use with bottle as well. Because you will be making HTML. And then you will want a way to connect views to controllers... and soon I imagine you will just want to be using web2py. Web2py is very simple. I suggest you spend just a little bit more time getting to know how web2py works, and then decide whether to use bottle+web2py or web2py. On Wednesday, May 2, 2012 3:56:24 PM UTC-7, David Johnston wrote: > > > Can someone explain to me why someone would want to use web2py with > Bottle? I am considering the combination. > > Is it for when you want to keep the app very simple but need some of the > more advanced functionality of Web2py such as the DAL? I am trying to learn > web programming and wonder if this might be a good place to start. I find > the full-blown web2py a little intimidating especially since I can't find > what I am looking for in the documentation. I find Bottle the opposite. Too > sparse. For example, I don't mind making forms but don't really want to > write SQL queries. I am coming from a (short) background in Wordpress so I > am looking for something like that but in Python and better for web app > development. Bottle feels like building a webapp from the bottle up. Web2py > feels like building it from the top down, (i.e. take the example apps and > reverse engineer them to what you want). I guess I would like something in > the middle. Wordpress wasn't THAT bad but I want easier CRUD and would > rather ditch PHP. > > Web2py is supposed to be easier than Django but Django has so much better > documentation, books etc that I might find this the opposite. Suggestions? > > Dave >
[web2py] Installing scheduler on linux
Anyone have a recipe to make the scheduler run on boot? I'm using ubuntu. Web2py is run in apache (using the recipe in the book), so I can't just use the cron @reboot line. This is the line that needs to be run when my system boots: python /home/web2py/web2py/web2py.py -K It seems ubuntu uses Upstart instead of sysvinit. And it might be possible to tell Upstart to launch the scheduler WITH apache, using a line like "start on started apache2" but I don't know how it works, and apache2 to use upstart. It would be nice to have a recipe for the scheduler that we could add into the book, to synchronize it with apache.
[web2py] Re: Web2py templates for HamlPY
This is cool! how do we use it?
On Sunday, January 9, 2011 5:07:28 PM UTC-8, Dane wrote:
>
> Hey all, thought you might be interested to know that I just patched a
> project HamlPy, a library for converting a pythonic haml-like syntax
> to django templates/html, to work with web2py templates.
>
> It allows for a less crufty, indentation-based syntax. Because it's
> indentation-based, {{ pass }} is no longer needed to close flow-
> control statements, and blocks are self-closed. Overall I think it's a
> much cleaner and quicker way of creating templates. And if you do want
> to use traditional {{ }} syntax, you can also do that in the .hamlpy
> files and they'll be converted as-is.
>
> https://github.com/danenania/HamlPy
>
> Hope someone gets some use out of this. My first real open source
> effort!
[web2py] Create indices from DAL
Here's an improved way to create indices in the DAL. Works only with
postgresql and sqlite.
def create_indices(*fields):
'''
Creates a set of indices if they do not exist
Use like:
create_indices(db.posts.created_at,
db.users.first_name,
etc...)
'''
for field in fields:
table = field.tablename
column = field.name
db = field.db
if db._uri.startswith('sqlite:'):
db.executesql('create index if not exists %s_%s_index on
%s (%s);'
% (table, column, table, column))
elif db._uri.startswith('postgres:'):
# Our indexes end with "_index", but web2py autogenerates
# one named "_key" for fields created with unique=True.
# So let's check to see if one exists of either form.
index_exists = \
db.executesql("select count(*) from pg_class where
relname='%s_%s_index' or relname='%s_%s_key';"
% (table, column, table, column))[0][0]
== 1
if not index_exists:
db.executesql('create index %s_%s_index on %s (%s);'
% (table, column, table, column))
db.commit()
This improves on this one I posted a while back:
http://groups.google.com/group/web2py/browse_thread/thread/8f6179915a6df8ee/cb58f509ae0a478d?lnk=gst&q=create+index#cb58f509ae0a478d
[web2py] Re: Help with OAuth20 facebook infinite redirect
Well, I don't need to debug this anymore. I switched to a different
facebook app, and I'm no longer having the problem.
On Dec 21, 7:55 pm, Michael Toomim wrote:
> I just upgraded from a modified 1.98.2 to 1.99.4 and now I'm getting
> an infinite redirect when logging in with OAuth20 and facebook.
>
> I'm having trouble debugging. Can someone help?
>
> What happens:
> User goes to /user/login
> This calls this code in tools.py:
>
> # we need to pass through login again before going on
> next = self.url('user',args='login')
> redirect(cas.login_url(next))
>
> which calls this in contrib/login_methods/oauth20_account.py:
>
> def login_url(self, next="/"):
> self.__oauth_login(next)
> return next
>
> and __oauth_login(next) will eventually redirect the user to this
> Facebook url to authenticate:
>
> https://graph.facebook.com/oauth/authorize?scope=email&redirect_uri=m...
>
> ...the user then logs in at facebook, and facebook returns back a code
> to us at /user/login?code=
>
> Ok! Now we're at /user/login again. This calls the same functions as
> above (cas.login_url(next), which again calls __oath_login(next)), but
> this time the "code" variable is set, so we get an access token
> created. Great!
>
> BUT then __oath_login() returns to login_url() which returns "/user/
> login" to the redirect function I pasted earlier:
>
> # we need to pass through login again before going on
> next = self.url('user',args='login')
> redirect(cas.login_url(next))
>
> ...And the whole thing redirects BACK to /user/login. And then the
> whole cycle repeats itself from scratch! The login function redirects
> us to facebook, facebook gives us a code, sends us back to login,
> login creates an access_token, and then this all returns to tools.py
> which redirects us back to /user/login.
>
> Where is this supposed to stop cycling and go to a normal url instead
> of /user/login?
[web2py] Help with OAuth20 facebook infinite redirect
I just upgraded from a modified 1.98.2 to 1.99.4 and now I'm getting
an infinite redirect when logging in with OAuth20 and facebook.
I'm having trouble debugging. Can someone help?
What happens:
User goes to /user/login
This calls this code in tools.py:
# we need to pass through login again before going on
next = self.url('user',args='login')
redirect(cas.login_url(next))
which calls this in contrib/login_methods/oauth20_account.py:
def login_url(self, next="/"):
self.__oauth_login(next)
return next
and __oauth_login(next) will eventually redirect the user to this
Facebook url to authenticate:
https://graph.facebook.com/oauth/authorize?scope=email&redirect_uri=myapp.com%2Fuser%2Flogin&response_type=code&client_id=181047918589726
...the user then logs in at facebook, and facebook returns back a code
to us at /user/login?code=
Ok! Now we're at /user/login again. This calls the same functions as
above (cas.login_url(next), which again calls __oath_login(next)), but
this time the "code" variable is set, so we get an access token
created. Great!
BUT then __oath_login() returns to login_url() which returns "/user/
login" to the redirect function I pasted earlier:
# we need to pass through login again before going on
next = self.url('user',args='login')
redirect(cas.login_url(next))
...And the whole thing redirects BACK to /user/login. And then the
whole cycle repeats itself from scratch! The login function redirects
us to facebook, facebook gives us a code, sends us back to login,
login creates an access_token, and then this all returns to tools.py
which redirects us back to /user/login.
Where is this supposed to stop cycling and go to a normal url instead
of /user/login?
[web2py] Re: Can't use custom classes in cache
So let me tease these problems apart. 1. Some objects are not pickleable. These cannot be cached to disk. 2. If the object's class is not defined in the scope available to gluon/cache.py, then the object cannot be unpickled. Both of these problems can be avoided by using cache.ram. (That's what I'm doing, and probably why Bruno's works.) Another workaround would be for the user to pickle and unpickle objects himself, in his own code, and then pass the string to the cache. We could also eliminate problem #2 by making the set of additional class definitions available to gluon/cache.py. Perhaps with something like "cache.set_custom_class_definitions([Blah])". This solution seems like the shortest path... but adding new APIs like this feels distasteful. :P (Massimo, I'm not sure what the problem is you're referring to with creating an instance before I cache.) On Nov 14, 7:41 pm, Massimo Di Pierro wrote: > Even if they are pickable. it is possible that they get pickled but > web2py cannot unpickle them. > > On Nov 14, 9:10 pm, Bruno Rocha wrote: > > > > > > > > > I notice that if you are planning to run on GAE, your cached object needs > > to be Pickable, in my case, the config object instance is a king of > > Storage, which acts like a dict. > > > So follow what Massimo said, and store only dict like objects in GAE, > > otherwise you will heve this issue: > > > PicklingError: Can't pickle the object
[web2py] Can't use custom classes in cache
Hi all, it appears I can't use any of my own classes in the cache:
class Blah:
pass
b = blah()
cache.disk('blah', lambda: b)
This results in:
AttributeError: 'module' object has no attribute 'Blah'
I think this is because the things I'm defining (e.g. in models/)
isn't accessible from cache.py in gluon/. Is there a way around this?
[web2py] Status of new VirtualFields design?
What's the status of the new VirtualFields design? The latest
I've seen is:
db.item.adder=Field.lazy(lambda self:self.item.a + self.item.b)
I have two design proposals:
(1) Instead of calling them "VirtualFields" and "Lazy VirtualFields,"
I think they should be "VirtualFields" and "Methods." Two
completely different things.
The primary usage of lazy virtual fields seems to be adding helper
methods to rows, so that one can call them as row.func() instead
of func_on_table(row). This is a "method," not a "virtual field."
(2) Put this API into db.define_table() instead of a separate set of
statements.
This puts all "objects you can access on a row" (Field,
VirtualField, ComputedField, and Method) in one place in your
code. And compared to separate class definitions, it reduces the
lines of code necessary.
Here's an example:
db.define_table('item',
Field('a', 'double'),
Field('b', 'integer'),
VirtualField('added',
lambda self: self.item.a + self.item.b),
Method('notify',
lambda self: send_email(self.item.a)))
[web2py] Re: RFC: New design for {{include file.html}} in views
Yes this is a much better design pattern. I will just make a file of
these helpers and include them all at once.
Thanks massimo!
On Sep 9, 10:32 pm, Massimo Di Pierro
wrote:
> Why not do:
>
> in friend_selector.html
>
> {{def selector(id=None):}}
>
>
> name="access_input"
> class="access_input" size="30" type="text" value="" />
>
> {{return}}
>
> and in the extending view {{include 'friend_selector.html'}}
> {{selector(id='main-access')}}
>
> This is already supported and allows to define more than one function
> in include. And it is more pythonic.
>
> On Sep 9, 10:12 pm, Michael Toomim wrote:
>
>
>
>
>
>
>
> > I frequently write short snippets of HTML that I want to replicate in
> > many places, like:
>
> >
> >
> > > class="access_input" size="30" type="text" value="" />
> >
>
> > So I can put this into a .html file "friend_selector.html", and then
> > include it whenever I need this. But I often need to parameterize it.
> > For instance, I might want to set a different id each time.
>
> > So I do it like this:
>
> > {{vars = {id : 'main_access_input'} }}
> > {{include 'friend_selector.html'}}
> > {{vars = {} }}
>
> > And then parameterize my html like this:
>
> >
> >
> > > vars['id']:}}id="{{=vars['id']}}"{{pass}}
> > name="access_input" class="access_input" size="30" type="text"
> > value="" />
> >
>
> > Basically, I'm re-inventing a function call via the {{include}}
> > feature. Wouldn't it be awesome if this just happened automatically???
> > Like this:
>
> > {{include 'friend_selector.html' (id='main_access_input')}}
>
> > Would you like this feature? Does this sound hard to implement?
>
> > Appendix:
> > 1. You can also do this I think with template inheritance, but that's
> > just complicated. Or you could define and call parameterized functions
> > in python, but editing embedded html in python strings is gross.
> > 2. The (id='main_access_input') part would ideally be a full python-
> > style parameter list, supporting e.g. (a=1, b=3). This is simpler to
> > implement. But to support positional args, like (1,3,5, a=1), the html
> > blocks would need to declare their supported parameters. They could do
> > so with by including a "declare" snippet like this:
>
> > {{ declare (a, b, c, id=None): }}
> >
> >
> > > name="access_input"
> > class="access_input" size="30" type="text" value="" />
> >
[web2py] RFC: New design for {{include file.html}} in views
I frequently write short snippets of HTML that I want to replicate in
many places, like:
So I can put this into a .html file "friend_selector.html", and then
include it whenever I need this. But I often need to parameterize it.
For instance, I might want to set a different id each time.
So I do it like this:
{{vars = {id : 'main_access_input'} }}
{{include 'friend_selector.html'}}
{{vars = {} }}
And then parameterize my html like this:
Basically, I'm re-inventing a function call via the {{include}}
feature. Wouldn't it be awesome if this just happened automatically???
Like this:
{{include 'friend_selector.html' (id='main_access_input')}}
Would you like this feature? Does this sound hard to implement?
Appendix:
1. You can also do this I think with template inheritance, but that's
just complicated. Or you could define and call parameterized functions
in python, but editing embedded html in python strings is gross.
2. The (id='main_access_input') part would ideally be a full python-
style parameter list, supporting e.g. (a=1, b=3). This is simpler to
implement. But to support positional args, like (1,3,5, a=1), the html
blocks would need to declare their supported parameters. They could do
so with by including a "declare" snippet like this:
{{ declare (a, b, c, id=None): }}
[web2py] Re: problems changing database field type
I think we need more tools for fixing broken migrations!
When I have something broken, sometimes I go into the sql console,
edit the database manually, and then use these functions to tell
web2py that I've changed the table in sql. (However, I haven't had to
use these for at least a year... maybe some dal bugs have been fixed
and migrations don't break as frequently anymore?)
def db_hash():
import cPickle, hashlib
return hashlib.md5(database).hexdigest()
def get_migrate_status(table_name):
import cPickle, hashlib
f = open('applications/init/databases/%s_%s.table'
% (hashlib.md5(database).hexdigest(),
table_name),
'r')
result = cPickle.load(f)
f.close()
return result
def save_migrate_status(table_name, status):
import cPickle, hashlib
f = open('applications/init/databases/%s_%s.table'
% (hashlib.md5(database).hexdigest(),
table_name),
'w')
cPickle.dump(status, f)
f.close()
print 'saved'
def del_migrate_column(table_name, column_name):
a = get_migrate_status(table_name)
del a[column_name]
save_migrate_status(table_name, a)
On Sep 9, 11:17 am, pbreit wrote:
> I think we need some more documentation on migrations, especially around
> fixing broken migrations, how to modify schema manually, how the sql.log and
> .table files work and how/when to recreate, etc.
[web2py] Re: Lazy virtual fields - strange result!
After some thought, I'm really liking this design for virtual
fields... what if lazy/virtual fields were declared directly in
db.define_table()? Like so:
db.define_table('item',
Field('unit_price','double'),
Field('quantity','integer'),
VirtualField('total_price',
lambda self:
self.item.unit_price*self.item.quantity,
lazy=True))
It's so simple.
I still kinda feel like we might find better names than lazy/virtual
though. So here's a design I like even more:
db.define_table('item',
Field('unit_price','double'),
Field('quantity','integer'),
Method('total_price',
lambda self:
self.item.unit_price*self.item.quantity,
precompute=False))
`precompute' means "not lazy" and would default to false.
On Aug 25, 6:14 am, Massimo Di Pierro
wrote:
> We are moving away from this because of many problems. Try this
> instead. It is still experimental but may go into stable soon.
>
> def vfields():
> db.define_table('item',
> Field('unit_price','double'),
> Field('quantity','integer'))
> db(db.item.id>0).delete()
>
> db.item.lazy_total_price=Field.lazy(lambda
> self:self.item.unit_price*self.item.quantity)
>
> db.item.bulk_insert([{'unit_price':12.00, 'quantity': 15},
> {'unit_price':10.00, 'quantity': 99},
> {'unit_price':120.00, 'quantity': 2},])
> res = []
> for r in db(db.item.id>0).select():
> res.append([r.unit_price, r.quantity, r.lazy_total_price()])
> return dict(res=res)
>
[web2py] Re: Lazy virtual fields - strange result!
I'm sorry, that was a doofus comment. Of course lambdas allow side-
effects! I wish mailing lists supported "delete."
On Aug 27, 1:08 pm, Michael Toomim wrote:
> Interesting approach to use lambdas. Since lambdas don't do side-
> effects, I checked out my virtualfields to see if my uses have side
> effects.
>
> In my app I have:
> 12 methods total across 3 database tables
> 10 of those methods have no side-effects
> 2 have side-effects
>
> The two methods with side-effects are:
> 1. row.delete() ... deletes the relevant rows in related tables
> 2. row.toggle_x() ... toggles the boolean field of x and updates
> cached data on other tables and sends email notifications
>
> Perhaps this information is useful in designing the next lazy
> virtualfields.
>
> On Aug 25, 9:14 pm, Massimo Di Pierro
> wrote:
>
>
>
>
>
>
>
> > We are moving away from this because of many problems. Try this
> > instead. It is still experimental but may go into stable soon.
>
> > def vfields():
> > db.define_table('item',
> > Field('unit_price','double'),
> > Field('quantity','integer'))
> > db(db.item.id>0).delete()
>
> > db.item.lazy_total_price=Field.lazy(lambda
> > self:self.item.unit_price*self.item.quantity)
>
> > db.item.bulk_insert([{'unit_price':12.00, 'quantity': 15},
> > {'unit_price':10.00, 'quantity': 99},
> > {'unit_price':120.00, 'quantity': 2},])
> > res = []
> > for r in db(db.item.id>0).select():
> > res.append([r.unit_price, r.quantity, r.lazy_total_price()])
> > return dict(res=res)
>
> > On Aug 25, 7:50 am, Martin Weissenboeck wrote:
>
> > > I wanted to learn more about lazy virtual fields and therefore I have
> > > repeated the example from the book:
>
> > > def vfields():
> > > db.define_table('item',
> > > Field('unit_price','double'),
> > > Field('quantity','integer'))
>
> > > db(db.item.id>0).delete()
>
> > > class MyVirtualFields:
> > > def lazy_total_price(self):
> > > return lambda self=self: self.item.unit_price*self.item.quantity
>
> > > db.item.virtualfields.append (MyVirtualFields())
>
> > > db.item.bulk_insert([{'unit_price':12.00, 'quantity': 15},
> > > {'unit_price':10.00, 'quantity': 99},
> > > {'unit_price':120.00, 'quantity': 2},])
>
> > > res = []
> > > for r in db(db.item.id>0).select():
> > > res.append([r.unit_price, r.quantity, r.lazy_total_price()])
> > > return dict(res=res)
>
> > > The expected output is:
> > > [[12.0, 15, 180.0], [10.0, 99, 990.0], [120.0, 2, 240.0]]
>
> > > But I got
> > > * [[12.0, 15, *240.0]*, [10.0, 99, *240.0*], [120.0, 2, 240.0]]*
> > > *
> > > *
> > > *Three times the same result.
> > > *
> > > I have read the book and my program over and over again - but I cannot see
> > > any error.*
> > > *
>
> > > Does somebody have an idea?
> > > Martin
[web2py] Re: Lazy virtual fields - strange result!
Interesting approach to use lambdas. Since lambdas don't do side-
effects, I checked out my virtualfields to see if my uses have side
effects.
In my app I have:
12 methods total across 3 database tables
10 of those methods have no side-effects
2 have side-effects
The two methods with side-effects are:
1. row.delete() ... deletes the relevant rows in related tables
2. row.toggle_x() ... toggles the boolean field of x and updates
cached data on other tables and sends email notifications
Perhaps this information is useful in designing the next lazy
virtualfields.
On Aug 25, 9:14 pm, Massimo Di Pierro
wrote:
> We are moving away from this because of many problems. Try this
> instead. It is still experimental but may go into stable soon.
>
> def vfields():
> db.define_table('item',
> Field('unit_price','double'),
> Field('quantity','integer'))
> db(db.item.id>0).delete()
>
> db.item.lazy_total_price=Field.lazy(lambda
> self:self.item.unit_price*self.item.quantity)
>
> db.item.bulk_insert([{'unit_price':12.00, 'quantity': 15},
> {'unit_price':10.00, 'quantity': 99},
> {'unit_price':120.00, 'quantity': 2},])
> res = []
> for r in db(db.item.id>0).select():
> res.append([r.unit_price, r.quantity, r.lazy_total_price()])
> return dict(res=res)
>
> On Aug 25, 7:50 am, Martin Weissenboeck wrote:
>
>
>
>
>
>
>
> > I wanted to learn more about lazy virtual fields and therefore I have
> > repeated the example from the book:
>
> > def vfields():
> > db.define_table('item',
> > Field('unit_price','double'),
> > Field('quantity','integer'))
>
> > db(db.item.id>0).delete()
>
> > class MyVirtualFields:
> > def lazy_total_price(self):
> > return lambda self=self: self.item.unit_price*self.item.quantity
>
> > db.item.virtualfields.append (MyVirtualFields())
>
> > db.item.bulk_insert([{'unit_price':12.00, 'quantity': 15},
> > {'unit_price':10.00, 'quantity': 99},
> > {'unit_price':120.00, 'quantity': 2},])
>
> > res = []
> > for r in db(db.item.id>0).select():
> > res.append([r.unit_price, r.quantity, r.lazy_total_price()])
> > return dict(res=res)
>
> > The expected output is:
> > [[12.0, 15, 180.0], [10.0, 99, 990.0], [120.0, 2, 240.0]]
>
> > But I got
> > * [[12.0, 15, *240.0]*, [10.0, 99, *240.0*], [120.0, 2, 240.0]]*
> > *
> > *
> > *Three times the same result.
> > *
> > I have read the book and my program over and over again - but I cannot see
> > any error.*
> > *
>
> > Does somebody have an idea?
> > Martin
[web2py] Re: Virtual Fields. Strange behavior when saving db result sets Build in types
I don't have a direct solution, but FYI I added this info to a bug
report on a related topic. http://code.google.com/p/web2py/issues/detail?id=374
Thanks for pointing out the problem.
On Aug 14, 8:57 am, Santiago Gilabert
wrote:
> anyone?
>
> I found that someone else asked about the same issue 5 month ago but there
> are no comments about it.
>
> http://groups.google.com/group/web2py/browse_thread/thread/845e6cdef5...
>
> Thanks
> Santiago
>
> On Sat, Aug 13, 2011 at 7:07 PM, Santiago wrote:
>
>
>
>
>
>
>
> > Hello,
>
> > I have the following definitions in db.py
>
> > class ElectionVirtualField(object):
> > ...
> > def is_presidential(self):
> > def lazy(self=self):
> > return self.election.category == 'P'
> > return lazy
> > ...
>
> > db.define_table('election',
> > ...
> > Field('category', length=1, label=T('Category')
> > ...
>
> > The problem I have is that when I add a bunch of elections in dict()
> > (key = election.id, value = election row) and then traverse the dict,
> > I get wrong values from is_presidential()
>
> > Example:
>
> > elections = dict()
> > for election in db(db.election).select():
> > elections[election.id] = election
>
> > for (k, v) in elections.items():
> > print k , ' ', v.category, ' ', v.is_presidential()
>
> > Output:
> > 81 D True
> > 79 P True
>
> > As you can see, it returns True for both, but for the first one, it
> > should return False.
>
> > If I change the code to reload the election from the database, the
> > output is different:
>
> > Example:
>
> > elections = dict()
> > for election in db(db.election).select():
> > elections[election.id] = election
>
> > for (k, v) in elections.items():
> > reloaded_election = db.election(k)
> > print k , ' ', v.category, ' ', v.is_presidential()
>
> > Output:
>
> > 81 D False
> > 79 P True
>
> > Does this mean that we can't save rows from DB on Build in types ?
>
> > Thanks in advance,
> > Santiago
[web2py] Re: RFC about issue
I agree not a big deal: http://www.quora.com/Should-buttons-in-web-apps-be-capitalized On Aug 11, 3:24 am, Massimo Di Pierro wrote: > What do people think? > > http://code.google.com/p/web2py/issues/detail?id=370 > > I do not have a strong opinion.
[web2py] Here's a helper for db.table.first() and last()
Often I'm at the shell and want to quickly pull up the most recent entry in a table. I wrote a couple of helpers for this. For instance, in a blog app: db.posts.last() ...will get the most recent post. By putting this code at the bottom of db.py, it'll automatically create a first() and last() method for each table in your app. for table in db.tables: def first(self): return db(self.id>0).select(orderby=self.id, limitby=(0,1)).first() def last(self): return db(self.id>0).select(orderby=~self.id, limitby=(0,1)).first() t = db[table] t.first = types.MethodType(first, t) t.last = types.MethodType(last, t)
[web2py] Re: Bug in virtualfields w/ session
Ok, it's here http://code.google.com/p/web2py/issues/detail?id=374
Thank you for looking into this Massimo! I do not know the best way to
do this... my code is just a first reaction to making something
faster.
On Aug 11, 2:55 am, Massimo Di Pierro
wrote:
> This is really interesting. Please give me some time to study it,
> meanwhile, so that I do not forget, please open an issue and post the
> code there.
>
> Massimo
>
> On Aug 10, 7:11 pm, MichaelToomim wrote:
>
>
>
>
>
>
>
> > Ok. The basic idea is to allow you to define helpers methods on rows,
> > sort of like the Models of rails/django.
>
> > You use it like this... I put this in models/db_methods.py:
>
> > @extra_db_methods
> > class Users():
> > def name(self):
> > return '%s %s' % ((self.first_name or ''),
> > (self.last_name or ''))
> > def fb_name(self):
> > p = self.person()
> > return (p and p.name) or 'Unknown dude'
> > def person(self):
> > return db.people(db.people.fb_id == self.fb_id)
> > def friends(self):
> > return [Storage(name=f[0], id=f[1])
> > for f in sj.loads(self.friends_cache)]
>
> > @extra_db_methods
> > class People():
> > ... etc
>
> > These are for tables db.users and db.people. It looks up the table
> > name from the class name. For each table that you want to extend, you
> > make a class and put @extra_db_methods on top.
>
> > It's implemented with the following @extra_db_methods decorator and a
> > patch to dal.py. The decorator just traverses the class, pulls out all
> > methods, and throws them into a "methods" variable on the appropriate
> > table in dal. Then the dal's parse() routine adds these methods each
> > row object, using the python type.MethodType() routine for
> > retargetting a method from one class to another object.
>
> > The downside is extending dal with yet ANOTHER way of adding methods
> > to objects. That makes 3 apis to maintain for similar things
> > (virtualfields, computedfields, and this). And I'm not sure about the
> > names (like "extra_db_methods") for these things yet. Also I think we
> > might be able to get it even faster by being more clever with python
> > inheritance in the Row class. Right now it has roughly 10% overhead on
> > selects in my tests (uncompiled code).
>
> > At the bottom of this message is the decorator that implements the
> > same functionality using the existing virtualfields mechanism and your
> > "lazy" decorator. Its downside is a 2x to 3x overhead on selects and
> > instead of self.field you have to say self..field in the
> > method bodies.
>
> > def extra_db_methods(clss):
> > tablename = clss.__name__.lower()
> > if not tablename in db:
> > raise Error('There is no `%s\' table to put virtual methods in'
> > % tablename)
>
> > for k in clss.__dict__.keys():
> > method = clss.__dict__[k]
> > if type(method).__name__ == 'function' or type(method).__name__
> > == 'instancemethod':
> > db[tablename].methods.update({method.__name__ : method})
>
> > return clss
>
> > --- k/web2py/gluon/dal.py 2011-08-03 16:46:39.0 -0700
> > +++ web2py/gluon/dal.py 2011-08-10 17:04:48.344795251 -0700
> > @@ -1459,6 +1459,7 @@
> > new_rows.append(new_row)
> > rowsobj = Rows(db, new_rows, colnames, rawrows=rows)
> > for tablename in virtualtables:
> > + rowsobj.setmethods(tablename, db[tablename].methods)
> > for item in db[tablename].virtualfields:
> > try:
> > rowsobj =
> > rowsobj.setvirtualfields(**{tablename:item})
> > @@ -4559,6 +4560,7 @@
> > tablename = tablename
> > self.fields = SQLCallableList()
> > self.virtualfields = []
> > + self.methods = {}
> > fields = list(fields)
>
> > if db and self._db._adapter.uploads_in_blob==True:
> > @@ -5574,6 +5576,14 @@
> > self.compact = compact
> > self.response = rawrows
>
> > + def setmethods(self, tablename, methods):
> > + if len(methods) < 0: return
> > + for row in self.records:
> > + if tablename not in row: break # Abort on this and all
> > rows. For efficiency.
> > + for (k,v) in methods.items():
> > + r = row[tablename]
> > + r.__dict__[k] = types.MethodType(v, r)
> > + return self
> > def setvirtualfields(self,**keyed_virtualfields):
> > if not keyed_virtualfields:
> > return self
>
> > ---
> > And Here's the implementation using virtualfields:
>
> > def lazy(f):
> > def g(self,f=f):
> > import copy
> > self=copy.copy(self)
> > return lambda *a,**b: f(self,*a,**b)
> > return g
>
> > def extra_db_methods_vf(clss):
> > ''' This decorator clears virtualfields on the table and replaces
> > them with the methods on this class.
> > '''
> > # First let's make the methods lazy
> >
[web2py] Re: Bug in virtualfields w/ session
Ok. The basic idea is to allow you to define helpers methods on rows,
sort of like the Models of rails/django.
You use it like this... I put this in models/db_methods.py:
@extra_db_methods
class Users():
def name(self):
return '%s %s' % ((self.first_name or ''),
(self.last_name or ''))
def fb_name(self):
p = self.person()
return (p and p.name) or 'Unknown dude'
def person(self):
return db.people(db.people.fb_id == self.fb_id)
def friends(self):
return [Storage(name=f[0], id=f[1])
for f in sj.loads(self.friends_cache)]
@extra_db_methods
class People():
... etc
These are for tables db.users and db.people. It looks up the table
name from the class name. For each table that you want to extend, you
make a class and put @extra_db_methods on top.
It's implemented with the following @extra_db_methods decorator and a
patch to dal.py. The decorator just traverses the class, pulls out all
methods, and throws them into a "methods" variable on the appropriate
table in dal. Then the dal's parse() routine adds these methods each
row object, using the python type.MethodType() routine for
retargetting a method from one class to another object.
The downside is extending dal with yet ANOTHER way of adding methods
to objects. That makes 3 apis to maintain for similar things
(virtualfields, computedfields, and this). And I'm not sure about the
names (like "extra_db_methods") for these things yet. Also I think we
might be able to get it even faster by being more clever with python
inheritance in the Row class. Right now it has roughly 10% overhead on
selects in my tests (uncompiled code).
At the bottom of this message is the decorator that implements the
same functionality using the existing virtualfields mechanism and your
"lazy" decorator. Its downside is a 2x to 3x overhead on selects and
instead of self.field you have to say self..field in the
method bodies.
def extra_db_methods(clss):
tablename = clss.__name__.lower()
if not tablename in db:
raise Error('There is no `%s\' table to put virtual methods in'
% tablename)
for k in clss.__dict__.keys():
method = clss.__dict__[k]
if type(method).__name__ == 'function' or type(method).__name__
== 'instancemethod':
db[tablename].methods.update({method.__name__ : method})
return clss
--- k/web2py/gluon/dal.py 2011-08-03 16:46:39.0 -0700
+++ web2py/gluon/dal.py 2011-08-10 17:04:48.344795251 -0700
@@ -1459,6 +1459,7 @@
new_rows.append(new_row)
rowsobj = Rows(db, new_rows, colnames, rawrows=rows)
for tablename in virtualtables:
+rowsobj.setmethods(tablename, db[tablename].methods)
for item in db[tablename].virtualfields:
try:
rowsobj =
rowsobj.setvirtualfields(**{tablename:item})
@@ -4559,6 +4560,7 @@
tablename = tablename
self.fields = SQLCallableList()
self.virtualfields = []
+self.methods = {}
fields = list(fields)
if db and self._db._adapter.uploads_in_blob==True:
@@ -5574,6 +5576,14 @@
self.compact = compact
self.response = rawrows
+def setmethods(self, tablename, methods):
+if len(methods) < 0: return
+for row in self.records:
+if tablename not in row: break # Abort on this and all
rows. For efficiency.
+for (k,v) in methods.items():
+r = row[tablename]
+r.__dict__[k] = types.MethodType(v, r)
+return self
def setvirtualfields(self,**keyed_virtualfields):
if not keyed_virtualfields:
return self
---
And Here's the implementation using virtualfields:
def lazy(f):
def g(self,f=f):
import copy
self=copy.copy(self)
return lambda *a,**b: f(self,*a,**b)
return g
def extra_db_methods_vf(clss):
''' This decorator clears virtualfields on the table and replaces
them with the methods on this class.
'''
# First let's make the methods lazy
for k in clss.__dict__.keys():
if type(getattr(clss, k)).__name__ == 'instancemethod':
setattr(clss, k, lazy(getattr(clss, k)))
tablename = clss.__name__.lower()
if not tablename in db:
raise Error('There is no `%s\' table to put virtual methods in'
% tablename)
del db[tablename].virtualfields[:] # We clear virtualfields each
time
db[tablename].virtualfields.append(clss())
return clss
You use this just like before but with @extra_db_methods_vf instead of
@extra_db_methods, and append to each use of "self".
On Aug 9, 11:16 pm, Massimo Di Pierro
wrote:
> let us see it!
>
> On Aug 9, 9:36 pm, MichaelToomim wrote:
>
>
>
>
>
>
>
> > Result: Fixed by upgrading. I was seeing this
> > bug:http://code.google.com/p/web2py/issues/detail?id=345
>
> > However, virtualfields still take more time than they should. My
> > selects take 2-3x longer wit
[web2py] Re: Bug in virtualfields w/ session
Result: Fixed by upgrading. I was seeing this bug:
http://code.google.com/p/web2py/issues/detail?id=345
However, virtualfields still take more time than they should. My
selects take 2-3x longer with virtualfields enabled than without. I
implemented a little hack in the dal that adds methods to rows with
only a 10% overhead (instead of 200-300%) and can share that if
anyone's interested.
On Aug 8, 8:38 pm, Michael Toomim wrote:
> It turns out the speed problem is REALLY bad. I have a table with
> virtualfields of 14,000 rows. When I run raw sql:
>
> a = db.executesql('select * from people;')
>
> ...the query returns in 121ms. But when I run it through the DAL on
> only a subset of the data:
>
> a = db(db.people.id > 0).select(limitby=(0,1000))
>
> ...it returns in 141096.431ms. That's... 141 seconds. So 1000x longer
> on .1 of the database.
>
> My virtualfields are all lazy functions. I'm looking into what's
> causing it and will report back when I find out. It seems it might
> have something to do with the lazy decorator func because when I hit C-
> c the code is often stuck there... inside import copy or something.
>
> def lazy(f):
> def g(self,f=f):
> import copy
> self=copy.copy(self)
> return lambda *a,**b: f(self,*a,**b)
> return g
>
> Anyway, I'll send an update when I have more info.
>
> On Aug 2, 3:03 pm, MichaelToomim wrote:
>
>
>
>
>
>
>
> > That's way better syntax! Great idea!
>
> > On Aug 2, 2011, at 2:31 AM, Massimo Di Pierro wrote:
>
> > > We need to work on the speed. This can perhaps help the syntax:
>
> > > db=DAL()
> > > db.define_table('a',Field('b','integer'))
> > > for i in range(10):
> > > db.a.insert(b=i)
>
> > > def lazy(f):
> > > def g(self,f=f):
> > > import copy
> > > self=copy.copy(self)
> > > return lambda *a,**b: f(self,*a,**b)
> > > return g
>
> > > class Scale:
> > > @lazy
> > > def c(self,scale=1):
> > > return self.a.b*scale
>
> > > db.a.virtualfields.append(Scale())
> > > for row in db(db.a).select():
> > > print row.b, row.c(1), row.c(2), row.c(3)
>
> > > On Aug 1, 3:10 pm, MichaelToomim wrote:
> > >> Maybe it helps for me to explain my use-case. I mainly use virtual
> > >> fields as lazy methods, to help traverse related tables. I was actually
> > >> surprised that lazy evaluation wasn't the default. I noticed a few
> > >> implications of this:
> > >> - Large queries are slowed byvirtualfields, even if they won't be
> > >> needed, esp if they query db
> > >> - My definitions forvirtualfieldsaren't as clean as they could be,
> > >> because I have many nested "lazy" funcs in the class definition
> > >> - We can't serialize all objects intosessionvariables
>
> > >> So really I'm just using this because it's a nicer notation to call
> > >> row.otherthing() instead of getotherthing(row). Maybe I really want some
> > >> different feature here?
>
> > >> On Aug 1, 2011, at 5:40 AM, Anthony Bastardi wrote:
>
> > >>> Note, after looking at this some more, Massimo recalled that the reason
> > >>> auth_user virtual fields were excluded from auth.user (and therefore
> > >>> from saving in thesession) is because some virtual fields are objects
> > >>> that cannot be pickled and therefore cannot be serialized to store in
> > >>> thesession. So, we're thinking of either creating an option to store
> > >>> auth_user virutual fields in auth.user, or maybe testing to make sure
> > >>> the virtual fields can be pickled, and excluding them if not.
>
> > >>> Anthony
>
> > >>> On Mon, Aug 1, 2011 at 5:30 AM, MichaelToomim
> > >>> wrote:
> > >>> Awesome! I did not know there was an issue submission system.
>
> > >>> On Jul 30, 2011, at 7:02 AM, Anthony wrote:
>
> > >>>> An issue has been submitted, and this should be corrected soon.
>
> > >>>> Anthony
>
> > >>>> On Friday, July 29, 2011 9:57:30 PM UTC-4, Anthony wrote:
> > >>>> auth.user is Storage(table_user._filter_fields(user, id=True)). The
> > >>>> _filter_fields method of the auth_user table only selects actual table
> > >>>> fields, not virt
[web2py] Re: Bug in virtualfields w/ session
Mid-status note: it would be great if the profiler worked with the
web2py shell!
Then I could run commands at the command prompt in isolation and see
how long they take.
On Aug 8, 8:38 pm, Michael Toomim wrote:
> It turns out the speed problem is REALLY bad. I have a table
> withvirtualfieldsof 14,000 rows. When I run raw sql:
>
> a = db.executesql('select * from people;')
>
> ...the query returns in 121ms. But when I run it through the DAL on
> only a subset of the data:
>
> a = db(db.people.id > 0).select(limitby=(0,1000))
>
> ...it returns in 141096.431ms. That's... 141 seconds. So 1000x longer
> on .1 of the database.
>
> Myvirtualfieldsare all lazy functions. I'm looking into what's
> causing it and will report back when I find out. It seems it might
> have something to do with the lazy decorator func because when I hit C-
> c the code is often stuck there... inside import copy or something.
>
> def lazy(f):
> def g(self,f=f):
> import copy
> self=copy.copy(self)
> return lambda *a,**b: f(self,*a,**b)
> return g
>
> Anyway, I'll send an update when I have more info.
>
> On Aug 2, 3:03 pm, Michael Toomim wrote:
>
>
>
>
>
>
>
> > That's way better syntax! Great idea!
>
> > On Aug 2, 2011, at 2:31 AM, Massimo Di Pierro wrote:
>
> > > We need to work on the speed. This can perhaps help the syntax:
>
> > > db=DAL()
> > > db.define_table('a',Field('b','integer'))
> > > for i in range(10):
> > > db.a.insert(b=i)
>
> > > def lazy(f):
> > > def g(self,f=f):
> > > import copy
> > > self=copy.copy(self)
> > > return lambda *a,**b: f(self,*a,**b)
> > > return g
>
> > > class Scale:
> > > @lazy
> > > def c(self,scale=1):
> > > return self.a.b*scale
>
> > > db.a.virtualfields.append(Scale())
> > > for row in db(db.a).select():
> > > print row.b, row.c(1), row.c(2), row.c(3)
>
> > > On Aug 1, 3:10 pm, Michael Toomim wrote:
> > >> Maybe it helps for me to explain my use-case. I mainly use virtual
> > >> fields as lazy methods, to help traverse related tables. I was actually
> > >> surprised that lazy evaluation wasn't the default. I noticed a few
> > >> implications of this:
> > >> - Large queries are slowed byvirtualfields, even if they won't be
> > >> needed, esp if they query db
> > >> - My definitions forvirtualfieldsaren't as clean as they could be,
> > >> because I have many nested "lazy" funcs in the class definition
> > >> - We can't serialize all objects intosessionvariables
>
> > >> So really I'm just using this because it's a nicer notation to call
> > >> row.otherthing() instead of getotherthing(row). Maybe I really want some
> > >> different feature here?
>
> > >> On Aug 1, 2011, at 5:40 AM, Anthony Bastardi wrote:
>
> > >>> Note, after looking at this some more, Massimo recalled that the reason
> > >>> auth_user virtual fields were excluded from auth.user (and therefore
> > >>> from saving in thesession) is because some virtual fields are objects
> > >>> that cannot be pickled and therefore cannot be serialized to store in
> > >>> thesession. So, we're thinking of either creating an option to store
> > >>> auth_user virutual fields in auth.user, or maybe testing to make sure
> > >>> the virtual fields can be pickled, and excluding them if not.
>
> > >>> Anthony
>
> > >>> On Mon, Aug 1, 2011 at 5:30 AM, Michael Toomim
> > >>> wrote:
> > >>> Awesome! I did not know there was an issue submission system.
>
> > >>> On Jul 30, 2011, at 7:02 AM, Anthony wrote:
>
> > >>>> An issue has been submitted, and this should be corrected soon.
>
> > >>>> Anthony
>
> > >>>> On Friday, July 29, 2011 9:57:30 PM UTC-4, Anthony wrote:
> > >>>> auth.user is Storage(table_user._filter_fields(user, id=True)). The
> > >>>> _filter_fields method of the auth_user table only selects actual table
> > >>>> fields, not virtual fields, so auth.user will not include any virtual
> > >>>> fields. Perhaps this should be changed.
>
> > >>>> Anthony
>
> > >>>> On Friday, July 29, 2011 9:05:39 PM UTC-4, Michael Toomim wrote:
> > >>>> I think I found a bug invirtualfields. I have the following
> > >>>> controller:
>
> > >>>> def posts():
> > >>>> user =session.auth.user
> > >>>> n = user.name # returns None
>
> > >>>> Where "person" is defined as a virtualfield on user:
>
> > >>>> class Users():
> > >>>> def name(self):
> > >>>> return self.users.first_name + ' ' + self.users.last_name
> > >>>> db.users.virtualfields.append(Users())
>
> > >>>> The problem is that user.name returns None, because apparently the
> > >>>> virtualfield isn't loaded into thesessionvariable of user.
>
> > >>>> I made this work with the following modification to the controller:
>
> > >>>> def posts():
> > >>>> user = db.users[session.auth.user.id]
> > >>>> n = user.name # returns the user name correctly!
>
> > >>>> I just had to refetch the user from the database.
[web2py] Re: Bug in virtualfields w/ session
It turns out the speed problem is REALLY bad. I have a table with
virtualfields of 14,000 rows. When I run raw sql:
a = db.executesql('select * from people;')
...the query returns in 121ms. But when I run it through the DAL on
only a subset of the data:
a = db(db.people.id > 0).select(limitby=(0,1000))
...it returns in 141096.431ms. That's... 141 seconds. So 1000x longer
on .1 of the database.
My virtualfields are all lazy functions. I'm looking into what's
causing it and will report back when I find out. It seems it might
have something to do with the lazy decorator func because when I hit C-
c the code is often stuck there... inside import copy or something.
def lazy(f):
def g(self,f=f):
import copy
self=copy.copy(self)
return lambda *a,**b: f(self,*a,**b)
return g
Anyway, I'll send an update when I have more info.
On Aug 2, 3:03 pm, Michael Toomim wrote:
> That's way better syntax! Great idea!
>
> On Aug 2, 2011, at 2:31 AM, Massimo Di Pierro wrote:
>
>
>
>
>
>
>
> > We need to work on the speed. This can perhaps help the syntax:
>
> > db=DAL()
> > db.define_table('a',Field('b','integer'))
> > for i in range(10):
> > db.a.insert(b=i)
>
> > def lazy(f):
> > def g(self,f=f):
> > import copy
> > self=copy.copy(self)
> > return lambda *a,**b: f(self,*a,**b)
> > return g
>
> > class Scale:
> > @lazy
> > def c(self,scale=1):
> > return self.a.b*scale
>
> > db.a.virtualfields.append(Scale())
> > for row in db(db.a).select():
> > print row.b, row.c(1), row.c(2), row.c(3)
>
> > On Aug 1, 3:10 pm, Michael Toomim wrote:
> >> Maybe it helps for me to explain my use-case. I mainly use virtual fields
> >> as lazy methods, to help traverse related tables. I was actually surprised
> >> that lazy evaluation wasn't the default. I noticed a few implications of
> >> this:
> >> - Large queries are slowed byvirtualfields, even if they won't be
> >> needed, esp if they query db
> >> - My definitions forvirtualfieldsaren't as clean as they could be,
> >> because I have many nested "lazy" funcs in the class definition
> >> - We can't serialize all objects intosessionvariables
>
> >> So really I'm just using this because it's a nicer notation to call
> >> row.otherthing() instead of getotherthing(row). Maybe I really want some
> >> different feature here?
>
> >> On Aug 1, 2011, at 5:40 AM, Anthony Bastardi wrote:
>
> >>> Note, after looking at this some more, Massimo recalled that the reason
> >>> auth_user virtual fields were excluded from auth.user (and therefore from
> >>> saving in thesession) is because some virtual fields are objects that
> >>> cannot be pickled and therefore cannot be serialized to store in
> >>> thesession. So, we're thinking of either creating an option to store
> >>> auth_user virutual fields in auth.user, or maybe testing to make sure the
> >>> virtual fields can be pickled, and excluding them if not.
>
> >>> Anthony
>
> >>> On Mon, Aug 1, 2011 at 5:30 AM, Michael Toomim
> >>> wrote:
> >>> Awesome! I did not know there was an issue submission system.
>
> >>> On Jul 30, 2011, at 7:02 AM, Anthony wrote:
>
> >>>> An issue has been submitted, and this should be corrected soon.
>
> >>>> Anthony
>
> >>>> On Friday, July 29, 2011 9:57:30 PM UTC-4, Anthony wrote:
> >>>> auth.user is Storage(table_user._filter_fields(user, id=True)). The
> >>>> _filter_fields method of the auth_user table only selects actual table
> >>>> fields, not virtual fields, so auth.user will not include any virtual
> >>>> fields. Perhaps this should be changed.
>
> >>>> Anthony
>
> >>>> On Friday, July 29, 2011 9:05:39 PM UTC-4, Michael Toomim wrote:
> >>>> I think I found a bug invirtualfields. I have the following
> >>>> controller:
>
> >>>> def posts():
> >>>> user =session.auth.user
> >>>> n = user.name # returns None
>
> >>>> Where "person" is defined as a virtualfield on user:
>
> >>>> class Users():
> >>>> def name(self):
> >>>> return self.users.first_name + ' ' + self.users.last_name
> >>>> db.users.virtualfields.append(Users())
>
> >>>> The problem is that user.name returns None, because apparently the
> >>>> virtualfield isn't loaded into thesessionvariable of user.
>
> >>>> I made this work with the following modification to the controller:
>
> >>>> def posts():
> >>>> user = db.users[session.auth.user.id]
> >>>> n = user.name # returns the user name correctly!
>
> >>>> I just had to refetch the user from the database.
Re: [web2py] Re: Bug in virtualfields w/ session
That's way better syntax! Great idea!
On Aug 2, 2011, at 2:31 AM, Massimo Di Pierro wrote:
> We need to work on the speed. This can perhaps help the syntax:
>
> db=DAL()
> db.define_table('a',Field('b','integer'))
> for i in range(10):
>db.a.insert(b=i)
>
> def lazy(f):
>def g(self,f=f):
>import copy
>self=copy.copy(self)
>return lambda *a,**b: f(self,*a,**b)
>return g
>
> class Scale:
>@lazy
>def c(self,scale=1):
>return self.a.b*scale
>
> db.a.virtualfields.append(Scale())
> for row in db(db.a).select():
>print row.b, row.c(1), row.c(2), row.c(3)
>
>
>
> On Aug 1, 3:10 pm, Michael Toomim wrote:
>> Maybe it helps for me to explain my use-case. I mainly use virtual fields as
>> lazy methods, to help traverse related tables. I was actually surprised that
>> lazy evaluation wasn't the default. I noticed a few implications of this:
>> - Large queries are slowed by virtualfields, even if they won't be needed,
>> esp if they query db
>> - My definitions for virtualfields aren't as clean as they could be,
>> because I have many nested "lazy" funcs in the class definition
>> - We can't serialize all objects into session variables
>>
>> So really I'm just using this because it's a nicer notation to call
>> row.otherthing() instead of getotherthing(row). Maybe I really want some
>> different feature here?
>>
>> On Aug 1, 2011, at 5:40 AM, Anthony Bastardi wrote:
>>
>>
>>
>>
>>
>>
>>
>>> Note, after looking at this some more, Massimo recalled that the reason
>>> auth_user virtual fields were excluded from auth.user (and therefore from
>>> saving in the session) is because some virtual fields are objects that
>>> cannot be pickled and therefore cannot be serialized to store in the
>>> session. So, we're thinking of either creating an option to store auth_user
>>> virutual fields in auth.user, or maybe testing to make sure the virtual
>>> fields can be pickled, and excluding them if not.
>>
>>> Anthony
>>
>>> On Mon, Aug 1, 2011 at 5:30 AM, Michael Toomim
>>> wrote:
>>> Awesome! I did not know there was an issue submission system.
>>
>>> On Jul 30, 2011, at 7:02 AM, Anthony wrote:
>>
>>>> An issue has been submitted, and this should be corrected soon.
>>
>>>> Anthony
>>
>>>> On Friday, July 29, 2011 9:57:30 PM UTC-4, Anthony wrote:
>>>> auth.user is Storage(table_user._filter_fields(user, id=True)). The
>>>> _filter_fields method of the auth_user table only selects actual table
>>>> fields, not virtual fields, so auth.user will not include any virtual
>>>> fields. Perhaps this should be changed.
>>
>>>> Anthony
>>
>>>> On Friday, July 29, 2011 9:05:39 PM UTC-4, Michael Toomim wrote:
>>>> I think I found a bug in virtualfields. I have the following
>>>> controller:
>>
>>>> def posts():
>>>> user = session.auth.user
>>>> n = user.name # returns None
>>
>>>> Where "person" is defined as a virtualfield on user:
>>
>>>> class Users():
>>>> def name(self):
>>>> return self.users.first_name + ' ' + self.users.last_name
>>>> db.users.virtualfields.append(Users())
>>
>>>> The problem is that user.name returns None, because apparently the
>>>> virtualfield isn't loaded into the session variable of user.
>>
>>>> I made this work with the following modification to the controller:
>>
>>>> def posts():
>>>> user = db.users[session.auth.user.id]
>>>> n = user.name # returns the user name correctly!
>>
>>>> I just had to refetch the user from the database.
Re: [web2py] Bug in virtualfields w/ session
Maybe it helps for me to explain my use-case. I mainly use virtual fields as lazy methods, to help traverse related tables. I was actually surprised that lazy evaluation wasn't the default. I noticed a few implications of this: - Large queries are slowed by virtualfields, even if they won't be needed, esp if they query db - My definitions for virtualfields aren't as clean as they could be, because I have many nested "lazy" funcs in the class definition - We can't serialize all objects into session variables So really I'm just using this because it's a nicer notation to call row.otherthing() instead of getotherthing(row). Maybe I really want some different feature here? On Aug 1, 2011, at 5:40 AM, Anthony Bastardi wrote: > Note, after looking at this some more, Massimo recalled that the reason > auth_user virtual fields were excluded from auth.user (and therefore from > saving in the session) is because some virtual fields are objects that cannot > be pickled and therefore cannot be serialized to store in the session. So, > we're thinking of either creating an option to store auth_user virutual > fields in auth.user, or maybe testing to make sure the virtual fields can be > pickled, and excluding them if not. > > Anthony > > On Mon, Aug 1, 2011 at 5:30 AM, Michael Toomim > wrote: > Awesome! I did not know there was an issue submission system. > > On Jul 30, 2011, at 7:02 AM, Anthony wrote: > >> An issue has been submitted, and this should be corrected soon. >> >> Anthony >> >> On Friday, July 29, 2011 9:57:30 PM UTC-4, Anthony wrote: >> auth.user is Storage(table_user._filter_fields(user, id=True)). The >> _filter_fields method of the auth_user table only selects actual table >> fields, not virtual fields, so auth.user will not include any virtual >> fields. Perhaps this should be changed. >> >> Anthony >> >> On Friday, July 29, 2011 9:05:39 PM UTC-4, Michael Toomim wrote: >> I think I found a bug in virtualfields. I have the following >> controller: >> >> def posts(): >> user = session.auth.user >> n = user.name # returns None >> >> Where "person" is defined as a virtualfield on user: >> >> class Users(): >> def name(self): >> return self.users.first_name + ' ' + self.users.last_name >> db.users.virtualfields.append(Users()) >> >> The problem is that user.name returns None, because apparently the >> virtualfield isn't loaded into the session variable of user. >> >> I made this work with the following modification to the controller: >> >> def posts(): >> user = db.users[session.auth.user.id] >> n = user.name # returns the user name correctly! >> >> I just had to refetch the user from the database. > >
[web2py] Bug in virtualfields w/ session
I think I found a bug in virtualfields. I have the following controller: def posts(): user = session.auth.user n = user.name # returns None Where "person" is defined as a virtualfield on user: class Users(): def name(self): return self.users.first_name + ' ' + self.users.last_name db.users.virtualfields.append(Users()) The problem is that user.name returns None, because apparently the virtualfield isn't loaded into the session variable of user. I made this work with the following modification to the controller: def posts(): user = db.users[session.auth.user.id] n = user.name # returns the user name correctly! I just had to refetch the user from the database.
[web2py] Re: Apache, Wsgi problem
Great!!
I also had "threads=25" and changed this to "threads=1 processes=5",
so it makes sense that I was encountering the same problem. It sounds
like something in web2py might not be thread-safe. The next time I
run a production test I will report if this fixes the problem.
On Feb 10, 2:38 pm, VP wrote:
> Alright people short answer: I think I figured this out (at least
> with my configuration)
>
> After testing various configurations, here's the result with: ab -kc
> 100 -t 20https://domain.com/imageblog/default/index/ (same
> imageblog app, 100 connections, 20 seconds stress test).
>
> Two things you will notice with this result.
>
> 1. ZERO failed request. No morewsgipremature script error
> 2. Complete requests is 1234, 61 requests/second on average.
> Compare to prior configuration: 588 total requests, 29 requests/
> sec on average. Not to mention 15 failed requests due towsgi
> premature script errors!!!
>
> This is insane!!!
>
> So how did I configure this? here it is:
>
> WSGIDaemonProcess web2py user=username group=username \
> display-name=%{GROUP} processes=5 threads=1
>
> The important option being 5 processes, 1 thread.
>
> With this configuration, my real app also did not getwsgipremature
> script errors anymore. And guess what... the requests/sec
> triples
>
> I am still curious about this. While my real app can possibly be not
> thread-safe, but the imageblog app should be thread safe (the index
> was simply a listing of images, i.e. read only). Why would there be
> aproblemwith more than 1 thread?
>
>
>
> Document Path: /imageblog/default/index
> Document Length: 13083 bytes
>
> Concurrency Level: 100
> Time taken for tests: 20.008 seconds
> Complete requests: 1234
> Failed requests: 0
> Write errors: 0
> Keep-Alive requests: 1234
> Total transferred: 16827432 bytes
> HTML transferred: 16171262 bytes
> Requests per second: 61.68 [#/sec] (mean)
> Time per request: 1621.377 [ms] (mean)
> Time per request: 16.214 [ms] (mean, across all concurrent
> requests)
> Transfer rate: 821.33 [Kbytes/sec] received
>
> Connection Times (ms)
> min mean[+/-sd] median max
> Connect: 0 1 9.4 0 73
> Processing: 82 481 890.5 317 5475
> Waiting: 76 443 878.7 274 5393
> Total: 82 483 894.7 317 5503
>
> Percentage of the requests served within a certain time (ms)
> 50% 317
> 66% 342
> 75% 360
> 80% 372
> 90% 416
> 95% 489
> 98% 5351
> 99% 5397
> 100% 5503 (longest request)
>
[web2py] Re: Django vs web2py pain points
The biggest django pain points to me: - Templating system is a PAIN. You have to learn a new language, and in the end it's not as powerful as python. - Database ORM can be a pain. Same reasons. You have to learn a big special-purpose API in addition to SQL, and learn how it translates between the two. In general, django tries to build too many special-purpose abstractions for everything. This makes it more complicated. You have to learn more. And they get in your way when they aren't powerful enough to do what you need. Then you have to hack them. more details on django pain I've encountered: - template system requires learning a new language - that new language is limited, e.g. difficult to define a variable, makes simple things hard - because it uses ORM instead of DAL, you need to learn new object language for selecting and updating database, and learn how it maps to SQL, because eventually you need to understand the raw tables too - you have to type things at the command line more often to make changes. So you shut down the server, type a command, restart the server. some of these things are automatic in web2py. You just edit the code, and everything reloads automatically. - database migrations require using "south", an external plugin system - errors in your use of django will often crash deep within django. you'll be looking at a stack trace to some internal django file and not realize it's because you added a '/' to the end of a path in settings.py - more complicated to set up, mucking around in settings.py - constrained in how you pass variables to views from controllers. e.g. can't set global variables, that apply across multiple views, like a breadcrumb path. have to specify it manually in each controller and view. - have to type arcane things sometimes like "return render_to_response(blah)" at the end of a controller instead of just "return blah"
[web2py] Re: Apache, Wsgi problem
Yes, this echos my experiences exactly! Using apache ab benchmark alone would NOT trigger the error. I had plenty of RAM available. Seems to be a concurrency bug. On Jan 19, 10:53 am, VP wrote: > What is curious is that RAM is still available, with this error. > Monitoring CPU (using top) shows CPU % is about 70% for apache2 (70% > is probably capped by my VPS host, given to each VPS slice). > > And this occurs for a very simple app. I hope Massimo or someone else > can reproduce this error with this app. Note that while testing with > ab, you might have to browse the site, clicking on images, etc. > Otherwise, ab just it the same domain name repeatedly.
[web2py] Re: Apache, Wsgi problem
1.74.5. I will upgrade when I can reproduce the problem locally.
On Jan 17, 5:13 pm, Massimo Di Pierro
wrote:
> How old web2py? We have had bugs in the past that may cause your
> problem.
> You should try upgrade.
>
> Massimo
>
> On Jan 17, 6:58 pm, Michael Toomim wrote:
>
>
>
> > The problem for me is that this occurs on a webapp used by mechanical
> > turk, and it fails when I have hundreds of mechanical turkers using my
> > app... which only happens when I pay them hundreds of dollars. So
> > it's hard to reproduce right now without hundreds of dollars.
>
> > I am excited to try using VP's ab benchmark to try to reproduce the
> > error more cheaply. When I can do that, and get a minimal system
> > exhibiting the error, I will let you know.
>
> > I'm also currently using an older version of web2py. One pre-DAL.
>
> > On Jan 17, 12:20 pm, Massimo Di Pierro
> > wrote:
>
> > > You should really have
>
> > > db = DAL('postgres://name:password@localhost:5432/db',pool_size=20)
>
> > > The reason is that client-server databases may set a max to number of
> > > open connections and it takes time to perform the 3-way handshake to
> > > establish a new connection at every http request. Without pooling you
> > > may hit the max sooner than you should.
>
> > > This will also make your app faster.
>
> > > Massimo
>
> > > On Jan 17, 1:39 pm, VP wrote:
>
> > > > Here it is with private information replaced with generic information.
>
> > > > if request.env.web2py_runtime_gae: # if running on Google
> > > > App Engine
> > > > db = DAL('gae') # connect to Google
> > > > BigTable
> > > > session.connect(request, response, db = db) # and store sessions
> > > > and tickets there
> > > > else: # else use a normal
> > > > relational database
> > > > db = DAL('postgres://name:password@localhost:5432/db')
> > > > # db = DAL('sqlite://storage.sqlite')
>
> > > > On Jan 17, 1:19 pm, Massimo Di Pierro
> > > > wrote:
>
> > > > > Are you using connection pooling? Can I see the line db = DAL(...)?
>
> > > > > On Jan 16, 2:24 pm, VP wrote:
>
> > > > > > Kenneth (see post above) had the same problem with MySQL.
>
> > > > > > On Jan 16, 3:55 am, ron_m wrote:
>
> > > > > > > Is there any chance to try another server based database such as
> > > > > > > MySQL.
> > > > > > > Since SQLite did not exhibit a problem that is a valid data point
> > > > > > > but it is
> > > > > > > an embedded database so called directly by the web2py server code
> > > > > > > instead of
> > > > > > > using a driver. There is a possibility there is a problem with
> > > > > > > the driver on
> > > > > > > PostgreSQL if the connections are being reused in a pool. This is
> > > > > > > very
> > > > > > > difficult to find without some further logging. I gather the page
> > > > > > > request
> > > > > > > you see failing as a user simply never completes. I saw this once
> > > > > > > while
> > > > > > > testing the application but the loading is only myself running
> > > > > > > tests. I use
> > > > > > > PostgreSQL in this app but I also have a version that runs on
> > > > > > > MySQL which
> > > > > > > has only a connection string difference plus the content of the
> > > > > > > databases
> > > > > > > directory. Maybe I can try this tomorrow, it is almost 2 am here
> > > > > > > so I don't
> > > > > > > want to start something new right now. I just need to learn how
> > > > > > > to use ab
> > > > > > > which I take it isn't going to be much to figure out.
[web2py] Re: Apache, Wsgi problem
The problem for me is that this occurs on a webapp used by mechanical
turk, and it fails when I have hundreds of mechanical turkers using my
app... which only happens when I pay them hundreds of dollars. So
it's hard to reproduce right now without hundreds of dollars.
I am excited to try using VP's ab benchmark to try to reproduce the
error more cheaply. When I can do that, and get a minimal system
exhibiting the error, I will let you know.
I'm also currently using an older version of web2py. One pre-DAL.
On Jan 17, 12:20 pm, Massimo Di Pierro
wrote:
> You should really have
>
> db = DAL('postgres://name:password@localhost:5432/db',pool_size=20)
>
> The reason is that client-server databases may set a max to number of
> open connections and it takes time to perform the 3-way handshake to
> establish a new connection at every http request. Without pooling you
> may hit the max sooner than you should.
>
> This will also make your app faster.
>
> Massimo
>
> On Jan 17, 1:39 pm, VP wrote:
>
>
>
> > Here it is with private information replaced with generic information.
>
> > if request.env.web2py_runtime_gae: # if running on Google
> > App Engine
> > db = DAL('gae') # connect to Google
> > BigTable
> > session.connect(request, response, db = db) # and store sessions
> > and tickets there
> > else: # else use a normal
> > relational database
> > db = DAL('postgres://name:password@localhost:5432/db')
> > # db = DAL('sqlite://storage.sqlite')
>
> > On Jan 17, 1:19 pm, Massimo Di Pierro
> > wrote:
>
> > > Are you using connection pooling? Can I see the line db = DAL(...)?
>
> > > On Jan 16, 2:24 pm, VP wrote:
>
> > > > Kenneth (see post above) had the same problem with MySQL.
>
> > > > On Jan 16, 3:55 am, ron_m wrote:
>
> > > > > Is there any chance to try another server based database such as
> > > > > MySQL.
> > > > > Since SQLite did not exhibit a problem that is a valid data point but
> > > > > it is
> > > > > an embedded database so called directly by the web2py server code
> > > > > instead of
> > > > > using a driver. There is a possibility there is a problem with the
> > > > > driver on
> > > > > PostgreSQL if the connections are being reused in a pool. This is very
> > > > > difficult to find without some further logging. I gather the page
> > > > > request
> > > > > you see failing as a user simply never completes. I saw this once
> > > > > while
> > > > > testing the application but the loading is only myself running tests.
> > > > > I use
> > > > > PostgreSQL in this app but I also have a version that runs on MySQL
> > > > > which
> > > > > has only a connection string difference plus the content of the
> > > > > databases
> > > > > directory. Maybe I can try this tomorrow, it is almost 2 am here so I
> > > > > don't
> > > > > want to start something new right now. I just need to learn how to
> > > > > use ab
> > > > > which I take it isn't going to be much to figure out.
[web2py] Re: Apache, Wsgi problem
I have pool_size=100, and get the error.
On Jan 17, 12:20 pm, Massimo Di Pierro
wrote:
> You should really have
>
> db = DAL('postgres://name:password@localhost:5432/db',pool_size=20)
>
> The reason is that client-server databases may set a max to number of
> open connections and it takes time to perform the 3-way handshake to
> establish a new connection at every http request. Without pooling you
> may hit the max sooner than you should.
>
> This will also make your app faster.
>
> Massimo
>
> On Jan 17, 1:39 pm, VP wrote:
>
>
>
> > Here it is with private information replaced with generic information.
>
> > if request.env.web2py_runtime_gae: # if running on Google
> > App Engine
> > db = DAL('gae') # connect to Google
> > BigTable
> > session.connect(request, response, db = db) # and store sessions
> > and tickets there
> > else: # else use a normal
> > relational database
> > db = DAL('postgres://name:password@localhost:5432/db')
> > # db = DAL('sqlite://storage.sqlite')
>
> > On Jan 17, 1:19 pm, Massimo Di Pierro
> > wrote:
>
> > > Are you using connection pooling? Can I see the line db = DAL(...)?
>
> > > On Jan 16, 2:24 pm, VP wrote:
>
> > > > Kenneth (see post above) had the same problem with MySQL.
>
> > > > On Jan 16, 3:55 am, ron_m wrote:
>
> > > > > Is there any chance to try another server based database such as
> > > > > MySQL.
> > > > > Since SQLite did not exhibit a problem that is a valid data point but
> > > > > it is
> > > > > an embedded database so called directly by the web2py server code
> > > > > instead of
> > > > > using a driver. There is a possibility there is a problem with the
> > > > > driver on
> > > > > PostgreSQL if the connections are being reused in a pool. This is very
> > > > > difficult to find without some further logging. I gather the page
> > > > > request
> > > > > you see failing as a user simply never completes. I saw this once
> > > > > while
> > > > > testing the application but the loading is only myself running tests.
> > > > > I use
> > > > > PostgreSQL in this app but I also have a version that runs on MySQL
> > > > > which
> > > > > has only a connection string difference plus the content of the
> > > > > databases
> > > > > directory. Maybe I can try this tomorrow, it is almost 2 am here so I
> > > > > don't
> > > > > want to start something new right now. I just need to learn how to
> > > > > use ab
> > > > > which I take it isn't going to be much to figure out.
[web2py] Re: Output of sum(), simplifying the JSON
I find it easiest and cleanest to reformat data structures in python,
using list comprehensions. Javascript sucks for loops. So instead of
jsonifying the raw database output, fix it first:
export_optimizer_records = [{'FreezeTime': r.panel_1hrs.FreezeTime,
'StringID': r.panel_1hrs.StringID, 'Po_avg_sum':
r._extra['sum(panel_1hrs.Po_avg)']} for r in export_optimizer_records]
Basically, just add this line to your controller.
On Jan 14, 5:58 pm, Lorin Rivers wrote:
> Controller:
> export_optimizer_records =
> dbset(db.table.FreezeTime,db.table.StringID,db.table.Po_avg.sum(),groupby=.
> .FreezeTime|..StringID).as_list()
>
> View:
> var optimizerdata = {{response.write(json(export_optimizer_records),
> escape=False)}};
>
> The JSON looks like this:
>
> [{
> "panel_1hrs": {
> "FreezeTime": "2010-12-12 19:00:00",
> "StringID": "S0001"
> },
> "_extra": {
> "sum(panel_1hrs.Po_avg)": 519.912549612443
> }},
>
> {
> "panel_1hrs": {
> "FreezeTime": "2010-12-12 19:00:00",
> "StringID": "S0002"
> },
> "_extra": {
> "sum(panel_1hrs.Po_avg)": 532.390706326218
> }
>
> }]
>
> What I want is this:
>
> [{
> "FreezeTime": "2010-12-12 19:00:00",
> "StringID": "S0001",
> "Po_avg_sum": 519.912549612443},
>
> {
> "FreezeTime": "2010-12-12 19:00:00",
> "StringID": "S0002",
> "Po_avg_sum": 532.390706326218
>
> }]
>
> What's the easiest way to get that?
> --
> Lorin Rivers
> Mosasaur: Killer Technical Marketing
>
> 512/203.3198 (m)
[web2py] Re: Apache, Wsgi problem
Thanks, I just investigated this, but it looks like it did not fix the problem. In 8.4.6 Postgres changed the default "wal_sync_method" to "fdatasync," because the old default "open_datasync" failed on ext4. I use ext3 (on ubuntu 9.10), but I tried changing this option in my postgres database anyway. I changed it, restarted postgres and apache, but still get the error.
[web2py] Re: Apache, Wsgi problem
I'm still having this problem too (previous posts linked below). I
would love to find a solution. I'm not sure how to debug.
VP: Can you provide instructions for reproducing this bug using ab? I
had trouble using ab in the past. I am also on a VPS.
Since my last post (linked below), I have tried the following:
1) setting migrate=False
2) compiling the app
These did not fix the problem.
On Jan 9, 9:30 am, VP wrote:
> It is indeed the case that there was a segmentation fault, reported in
> apache error log.
>
> Perhaps, it's not clear, but this problem occurs under posgres under
> debian lenny, not sqlite. I am not running web2py as a CGIscript. I
> am using the web2py deploymentscript(for setting up apache and
> web2py):
>
>
> WSGIDaemonProcess web2py user=myusername group=myusername \
> display-name=%{GROUP}
> WSGIProcessGroup web2py
> WSGIScriptAlias / /home/myusername/web2py/wsgihandler.py
>
>
>
> On Jan 8, 9:59 pm, Graham Dumpleton
> wrote:
>
>
>
> > You were possibly using a an old version of sqlite which isn't safe to use
> > in a multithreaded configuration.
>
> > The MPM settings are not going to help in this case as that error could only
> > come about because you are using mod_wsgi daemon mode and so application is
> > running in distinct process and not those affected by the MPM or its
> > settings.
>
> > The only other way you could get that error is that you are actually running
> > web2py as a CGIscript.
>
> > Overall, that specific error message means your daemon mode process that is
> > running web2py crashed. You would likely find that there is a segmentation
> > fault messages in main Apache error log as well at that time.
>
> > Crashing could be because of sqlite thread problems, but could also be
> > because you are forcing web2py to run in main interpreter of daemon
> > processes and at the same time are using a third party C extension module
> > for Python that is not safe for use in sub interpreters.
>
> > So, ensure sqlite is up to date. And ensure that you have:
>
> > WSGIApplicationGroup %{GLOBAL}
>
> > in configuration to force use of main interpreter.
>
> > Graham
>
> > On Sunday, January 9, 2011 6:44:14 AM UTC+11, VP wrote:
>
> > > We occasionally got an Apache error so the page didn't get displayed.
> > > So I decided to stress test using Apache Bench (ab). It seems the
> > > site suffered failure up to 50-100 concurrent connection.
>
> > > Apache error log showed this error:
>
> > > >>Prematureendofscriptheaders: wsgihandler.py
>
> > > After digging around, I found similar discussions and change
> > > apache2.conf like this:
>
> > > # prefork MPM
> > > StartServers 5
> > > MinSpareServers 5
> > > MaxSpareServers 10
> > > MaxClients 256
> > > MaxRequestsPerChild 500
> > > ServerLimit 256
>
> > > Didn't seem to help.
>
> > >
>
> > > A few notes:
>
> > > + It appears when I switched to sqlite instead of posgres, I didn't
> > > have the problem. (Sqlite had other problems, such as occasional
> > > database locking, which is more serious)
>
> > > + I am on a VPS with 768MB with 1GB burstable. While I'm doing the
> > > stress test with Apache Bench (ab), using free on the server revealed
> > > memory usage was about 450MB. (Which is a lot, but is still under
> > > limit).
>
> > > =
>
> > > In summary, memory was available. But we got this wsgi error in
> > > Apache with multiple requests.
>
> > > Any idea please?
>
> > > Thanks.
[web2py] How to create indexes on postgres if not exists
I wanted the equivalent of sqlite's "create index if not exists" on
postgresql. Here's a solution for web2py. It is useful whenever you
set up a new database, or migrate new tables to an existing database
after a code update and want to ensure the right indexes are set up.
def create_indices_on_postgres():
'''Creates a set of indices if they do not exist'''
## Edit this list of table columns to index
## The format is [('table', 'column')...]
indices = [('actions', 'study'),
('actions', 'assid'),
('actions', 'hitid'),
('actions', 'time'),
('actions', 'workerid'),
('countries', 'code'),
('continents', 'code'),
('ips', 'from_ip'),
('ips', 'to_ip')]
for table, column in indices:
index_exists = db.executesql("select count(*) from pg_class
where relname='%s_%s_idx';"
% (table, column))[0][0] == 1
if not index_exists:
db.executesql('create index %s_%s_idx on %s (%s);'
% (table, column, table, column))
db.commit()
[web2py] Re: Error in wsgi/apache
Great! I want to understand it too!
Your breakdown helps me think about how to look into this. I will do
some more analysis, but for now:
- I'm not using forms
- I had migrate=True and the app was not compiled earlier. now it's
compiled and migrate=False and I have a bigger machine, things are
running ok, but I changed quite a few variables. Would migrate=True
create locking issues? The problems seemed similar to locking issues.
- My machine's ram slowly filled up from ~700mb/1500mb to 1400mb/
1500mb over 5 hours. But I haven't looked to see which process grew
yet, I will investigate.
On Jul 21, 1:13 am, mdipierro wrote:
> I still want to understand why you are having this problem. I see the
> following possibility:
>
> 1) There is a threshold in requests/seconds that depends on memory
> available and it is different for different frameworks. web2py does
> more than Flask (which is a microframework by definitions) and this
> threshold may be lower.
> If this is the case the problem should go away with more ram.
>
> 2) There is a bug in web2py or one of its modules that causes a memory
> leak (did you experience a memory leak?) or a locking issues (for
> example a process crashes before a locked resource is released).
>
> I remember you mentioning have this problem exclusively with actions
> using SQLFORM under heavy load. Is that correct? This could help us
> narrow down the problem.
>
> Massimo
>
> On Jul 20, 7:17 pm, Thadeus Burgess wrote:
>
>
>
> > The solution: I switched to Flask.
>
> > And the problems dissipated completely, without modifying any configuration
> > of the web server.
>
> > I would not, and will not use web2py for any application that is mission
> > critical. For personal sites, or quick projects that I know won't receive
> > that much attention, web2py is fine. For quickly prototyping something,
> > web2py excels. For stability, reliability, and scalability, use Flask or
> > Django.
>
> > The DAL is great though, nothing quite like it, thats why I am working on a
> > Flask-DAL extension, and I am working to re-write parts of the DAL and strip
> > out the web2py cohesion (such as SQLFORM validators).
>
> > Using the DAL inside of flask works fine, and I do not run into these
> > errors. This means that the DAL is not the cause of these errors, but web2py
> > core code. Most likely in dealing with pages that have forms is where these
> > errors arise. Web2py core is messy, and it ignores the wsgi specification
> > for the most part. I am sure that these errors arise from the fact that
> > web2py uses execfile in many places over and over again, which is a
> > discouraged practice among the python community, and you see why now.
>
> > --
> > Thadeus
>
> > On Tue, Jul 20, 2010 at 4:17 PM, Michael Toomim wrote:
> > > Thank you for the clarification.
>
> > > My wsgi.conf has default values, so I have not set maximum-requests.
> > > Perhaps there are settings there I should look into?
>
> > > I still have free memory, so perhaps there is not a memory leak
> > > issue. I'm also not sure how one would get memory leaks in web2py,
> > > since isn't the environment wiped clean with each request?
>
> > > This looks similar to the issue here:
>
> > >http://groups.google.com/group/web2py/browse_thread/thread/49a7ecabf4...
> > > Was there any resolution?
>
> > > I use logging by having the following file in models/0_log.py:
>
> > > import logging
> > > def get_log():
> > > try:
> > > f = open(logging.handlers[0].baseFilename, 'r')
> > > c = f.readlines()
> > > f.close()
> > > return {'log':TABLE(*[TR(str(item)) for item in c])}
> > > except:
> > > return ()
>
> > > # This model file defines some magic to implement app_wide_log.
> > > def _init_log(): # Does not work on GAE
> > > import os,logging,logging.handlers
> > > logger = logging.getLogger(request.application)
> > > logger.setLevel(logging.DEBUG)
> > > handler = logging.handlers.RotatingFileHandler(
> > > os.path.join( # so that it can be served as http://
> > > .../yourapp/static/applog.txt
> > > request.folder,'static','applog.txt'),'a',1024*1024,1)
> > > handler.setLevel(logging.DEBUG)
> > > handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)s %
> > > (filename)s:%(lineno)d %(funcName)s(): %(message)s"))
> > > logger.
[web2py] Re: Error in wsgi/apache
Great! I want to understand it too!
Your breakdown helps me think about how to look into this. I will do
some more analysis, but for now:
- I'm not using forms
- I had migrate=True and the app was not compiled earlier. now it's
compiled and migrate=False and I have a bigger machine, things are
running ok, but I changed quite a few variables. Would migrate=True
create locking issues? The problems seemed similar to locking issues.
- My machine's ram slowly filled up from ~700mb/1500mb to 1400mb/
1500mb over 5 hours. But I haven't looked to see which process grew
yet, I will investigate.
On Jul 21, 1:13 am, mdipierro wrote:
> I still want to understand why you are having this problem. I see the
> following possibility:
>
> 1) There is a threshold in requests/seconds that depends on memory
> available and it is different for different frameworks. web2py does
> more than Flask (which is a microframework by definitions) and this
> threshold may be lower.
> If this is the case the problem should go away with more ram.
>
> 2) There is a bug in web2py or one of its modules that causes a memory
> leak (did you experience a memory leak?) or a locking issues (for
> example a process crashes before a locked resource is released).
>
> I remember you mentioning have this problem exclusively with actions
> using SQLFORM under heavy load. Is that correct? This could help us
> narrow down the problem.
>
> Massimo
>
> On Jul 20, 7:17 pm, Thadeus Burgess wrote:
>
>
>
> > The solution: I switched to Flask.
>
> > And the problems dissipated completely, without modifying any configuration
> > of the web server.
>
> > I would not, and will not use web2py for any application that is mission
> > critical. For personal sites, or quick projects that I know won't receive
> > that much attention, web2py is fine. For quickly prototyping something,
> > web2py excels. For stability, reliability, and scalability, use Flask or
> > Django.
>
> > The DAL is great though, nothing quite like it, thats why I am working on a
> > Flask-DAL extension, and I am working to re-write parts of the DAL and strip
> > out the web2py cohesion (such as SQLFORM validators).
>
> > Using the DAL inside of flask works fine, and I do not run into these
> > errors. This means that the DAL is not the cause of these errors, but web2py
> > core code. Most likely in dealing with pages that have forms is where these
> > errors arise. Web2py core is messy, and it ignores the wsgi specification
> > for the most part. I am sure that these errors arise from the fact that
> > web2py uses execfile in many places over and over again, which is a
> > discouraged practice among the python community, and you see why now.
>
> > --
> > Thadeus
>
> > On Tue, Jul 20, 2010 at 4:17 PM, Michael Toomim wrote:
> > > Thank you for the clarification.
>
> > > My wsgi.conf has default values, so I have not set maximum-requests.
> > > Perhaps there are settings there I should look into?
>
> > > I still have free memory, so perhaps there is not a memory leak
> > > issue. I'm also not sure how one would get memory leaks in web2py,
> > > since isn't the environment wiped clean with each request?
>
> > > This looks similar to the issue here:
>
> > >http://groups.google.com/group/web2py/browse_thread/thread/49a7ecabf4...
> > > Was there any resolution?
>
> > > I use logging by having the following file in models/0_log.py:
>
> > > import logging
> > > def get_log():
> > > try:
> > > f = open(logging.handlers[0].baseFilename, 'r')
> > > c = f.readlines()
> > > f.close()
> > > return {'log':TABLE(*[TR(str(item)) for item in c])}
> > > except:
> > > return ()
>
> > > # This model file defines some magic to implement app_wide_log.
> > > def _init_log(): # Does not work on GAE
> > > import os,logging,logging.handlers
> > > logger = logging.getLogger(request.application)
> > > logger.setLevel(logging.DEBUG)
> > > handler = logging.handlers.RotatingFileHandler(
> > > os.path.join( # so that it can be served as http://
> > > .../yourapp/static/applog.txt
> > > request.folder,'static','applog.txt'),'a',1024*1024,1)
> > > handler.setLevel(logging.DEBUG)
> > > handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)s %
> > > (filename)s:%(lineno)d %(funcName)s(): %(message)s"))
> > > logger.
[web2py] Re: Error in wsgi/apache
Ah, preventing multithreading is a good idea to try too. It wasn't a file descriptor problem either, I had Files used: 1376 out of 75556 On Jul 20, 9:14 pm, Graham Dumpleton wrote: > On Jul 21, 1:41 pm, Michael Toomim wrote: > > > I'm using daemon mode... I didn't realize that the directive won't > > matter in daemon mode. > > > Yes, I think I probably will run into the problem again when I get > > more usage. However, I'm still not convinced it's a memory problem, > > because I had 30mb free on my 740mb machine when I was having the > > problem, with 0 swap usage. > > Well, as I explained before, it perhaps is a resource leakage such as > file descriptors. You can exhaust kernel file descriptors and still > have lots of memory available. > > I have seen various cases before for different peoples applications on > different frameworks where file objects weren't being explicitly > closed, or database connection pools not being managed properly, such > that the number of open file descriptors went up and up and eventually > they ran out. This can cause all sorts of weird errors to manifest > when it occurs, including it impacting other applications if is > exhausted system wide. For example, in a shell, not being able to > execute commands to even debug the problem. > > I would suggest you become familiar with some of the basic monitoring > commands such as 'lsof' or 'ofiles', depending on what your system > provides. You can then use these to monitor file descriptor usage by > your processes. > > Also be aware that such problems may only arise when multithreading > kicks in and concurrent requests run. In other words, due to code > which isn't thread safe. If you don't get the concurrency, you may > well see your application run quite happily. > > Thus, one suggestion is to not use multiple threads for daemon mode > processes and instead use something like 'processes=5 threads=1'. This > will avoid the potential of it being caused by multithreading issues > at least. > > Graham > > > > > I don't know what I'll do if I this happens again. My code just does > > simple database lookups and updates, it doesn't create circular > > references nor store anything in global variables, so if there's a > > memory leak I worry it's somewhere further up the stack. I don't know > > any ways to investigate memory consumption to see where it's being > > used. > > > On Jul 20, 8:23 pm, Graham Dumpleton > > wrote: > > > > On Jul 21, 1:03 pm, Michael Toomim wrote: > > > > > THANK YOU ALL SO MUCH for your help! > > > > > I just learned a LOT. It looks like resource consumption was the > > > > problem, because things are doing better on the bigger machine and > > > > scaled down code. I've also added the MaxRequestsPerChild directive. > > > > Are you using mod_wsgi embedded mode or daemon mode? That directive > > > should not be required if you are using daemon mode of mod_wsgi. > > > > It is generally a bad idea to make arbitrary changes without > > > understanding whether they are necessary. Changing to a larger machine > > > without understanding why your application is using lots of memory in > > > the first place is also questionable. All you have done is increased > > > your head room but potentially not solved the original underlying > > > problem. You may well just find that it all just blows up again when > > > you get hit with a larger amount of traffic or a request which > > > generates a lot of data. What are you going to do then, get an even > > > bigger machine? > > > > Graham > > > > > I am soo happy to have this web server working, and very pleased > > > > to know what to do when I hit a such a scaling wall again! > > > > > And flask looks interesting, but I must say I really really like how > > > > web2py's execfile puts things into global scope from the controllers > > > > and automatically reloads code with each request. > > > > > On Jul 20, 5:02 pm, Graham Dumpleton > > > > wrote: > > > > > > On Jul 21, 8:18 am, mdipierro wrote: > > > > > > > Can you comment on memory usage? > > > > > > > I have see this once: "after a while web serving slows" > > > > > > it appeared to be due to a memory leak somewhere (did not experience > > > > > > it with web2py+Rocket but only in web2py+mod_wsgi+apache). > > >
[web2py] Re: Error in wsgi/apache
I'm using daemon mode... I didn't realize that the directive won't matter in daemon mode. Yes, I think I probably will run into the problem again when I get more usage. However, I'm still not convinced it's a memory problem, because I had 30mb free on my 740mb machine when I was having the problem, with 0 swap usage. I don't know what I'll do if I this happens again. My code just does simple database lookups and updates, it doesn't create circular references nor store anything in global variables, so if there's a memory leak I worry it's somewhere further up the stack. I don't know any ways to investigate memory consumption to see where it's being used. On Jul 20, 8:23 pm, Graham Dumpleton wrote: > On Jul 21, 1:03 pm, Michael Toomim wrote: > > > THANK YOU ALL SO MUCH for your help! > > > I just learned a LOT. It looks like resource consumption was the > > problem, because things are doing better on the bigger machine and > > scaled down code. I've also added the MaxRequestsPerChild directive. > > Are you using mod_wsgi embedded mode or daemon mode? That directive > should not be required if you are using daemon mode of mod_wsgi. > > It is generally a bad idea to make arbitrary changes without > understanding whether they are necessary. Changing to a larger machine > without understanding why your application is using lots of memory in > the first place is also questionable. All you have done is increased > your head room but potentially not solved the original underlying > problem. You may well just find that it all just blows up again when > you get hit with a larger amount of traffic or a request which > generates a lot of data. What are you going to do then, get an even > bigger machine? > > Graham > > > > > I am soo happy to have this web server working, and very pleased > > to know what to do when I hit a such a scaling wall again! > > > And flask looks interesting, but I must say I really really like how > > web2py's execfile puts things into global scope from the controllers > > and automatically reloads code with each request. > > > On Jul 20, 5:02 pm, Graham Dumpleton > > wrote: > > > > On Jul 21, 8:18 am, mdipierro wrote: > > > > > Can you comment on memory usage? > > > > > I have see this once: "after a while web serving slows" > > > > it appeared to be due to a memory leak somewhere (did not experience > > > > it with web2py+Rocket but only in web2py+mod_wsgi+apache). > > > > > I googled it and I found Django was having the same problem on some > > > > hosts: > > > > Not sure how you can draw a parallel to that as it is a completely > > > different framework and just because another framework, or more > > > specifically one persons code, has issues, doesn't imply there is an > > > issue with underlying web hosting. > > > > These sorts of problems are isolated cases. If there was an issue with > > > memory leakage in the hosting mechanism it would be affecting everyone > > > and there have been no such reports of mod_wsgi itself leaking memory. > > > That said, ensure you read: > > > > http://blog.dscpl.com.au/2009/11/save-on-memory-with-modwsgi-30.html > > > > This describes how Python itself leaks memory. For mod_wsgi 2.X and > > > older, or if you are still loading mod_python into your Apache server, > > > then you can be affected by this, but not if using mod_wsgi 3.X. > > > > That post also explains how to completely disable initialisation of > > > Python in Apache server child processes, ie., embedded, if you aren't > > > using it. > > > > >http://stackoverflow.com/questions/229/django-memory-usage-going-.. > > > > > I followed the advice from a comment in the last post to limit the > > > > number of requests served by each process: > > > > Which is actually a useless thing to do if you are using daemon mode > > > which I understood you were, as MaxRequestsPerChild directive only > > > affects Apache server child process, ie., those use for embedded mode, > > > and not daemon mode processes. > > > > If using that directive helped and you were using daemon mode, then > > > you likely have a memory leak in some other Apache module. > > > > What you should have ensured you were doing was using display-name > > > option to WSGIDaemonProcess to name the process. That way in 'ps' you > > > can easily distinguish the mod_wsgi daemon mode processes from the > > > A
[web2py] Re: Error in wsgi/apache
THANK YOU ALL SO MUCH for your help! I just learned a LOT. It looks like resource consumption was the problem, because things are doing better on the bigger machine and scaled down code. I've also added the MaxRequestsPerChild directive. I am soo happy to have this web server working, and very pleased to know what to do when I hit a such a scaling wall again! And flask looks interesting, but I must say I really really like how web2py's execfile puts things into global scope from the controllers and automatically reloads code with each request. On Jul 20, 5:02 pm, Graham Dumpleton wrote: > On Jul 21, 8:18 am, mdipierro wrote: > > > Can you comment on memory usage? > > > I have see this once: "after a while web serving slows" > > it appeared to be due to a memory leak somewhere (did not experience > > it with web2py+Rocket but only in web2py+mod_wsgi+apache). > > > I googled it and I found Django was having the same problem on some > > hosts: > > Not sure how you can draw a parallel to that as it is a completely > different framework and just because another framework, or more > specifically one persons code, has issues, doesn't imply there is an > issue with underlying web hosting. > > These sorts of problems are isolated cases. If there was an issue with > memory leakage in the hosting mechanism it would be affecting everyone > and there have been no such reports of mod_wsgi itself leaking memory. > That said, ensure you read: > > http://blog.dscpl.com.au/2009/11/save-on-memory-with-modwsgi-30.html > > This describes how Python itself leaks memory. For mod_wsgi 2.X and > older, or if you are still loading mod_python into your Apache server, > then you can be affected by this, but not if using mod_wsgi 3.X. > > That post also explains how to completely disable initialisation of > Python in Apache server child processes, ie., embedded, if you aren't > using it. > > >http://stackoverflow.com/questions/229/django-memory-usage-going-.. > > > I followed the advice from a comment in the last post to limit the > > number of requests served by each process: > > Which is actually a useless thing to do if you are using daemon mode > which I understood you were, as MaxRequestsPerChild directive only > affects Apache server child process, ie., those use for embedded mode, > and not daemon mode processes. > > If using that directive helped and you were using daemon mode, then > you likely have a memory leak in some other Apache module. > > What you should have ensured you were doing was using display-name > option to WSGIDaemonProcess to name the process. That way in 'ps' you > can easily distinguish the mod_wsgi daemon mode processes from the > Apache processes and work out which is leaking memory. If it is the > daemon processes, it is likely to be a Python web application issue. > If the Apache parent process is getting fatter and you perform a lot > of Apache restart/reloads, then it could be that you are still using > mod_wsgi 2.X or mod_python is loaded at same time, and you are using a > version of Python that has lots of memory leaks on restarts. > > If your daemon processes are not getting fat and the Apache server > child processes are, then you may through incorrect configuration not > even be running Python web application in daemon mode. This is where > WSGIRestrictEmbedded as described in my post is good, as it will > highlight when the configuration is screwed up. > > > > > > > # prefork MPM > > StartServers 5 > > MinSpareServers 5 > > MaxSpareServers 10 > > MaxClients 256 > > MaxRequestsPerChild 500 > > ServerLimit 30 > > > instead of the default: > > > # prefork MPM > > StartServers 5 > > MinSpareServers 5 > > MaxSpareServers 10 > > MaxClients 256 > > > The problem disappeared. The exact values that fix the problem may > > depend on the ram available. > > The other difference with above is that I think by setting ServerLimit > to 30, you have effectively overridden MaxClients down to 30 even > though set to 256. You have thus in part limited the exact problems > described in: > > http://blog.dscpl.com.au/2009/03/load-spikes-and-excessive-memory-usa... > > if it so happens you were using embedded mode and not daemon mode. > > Graham > > > > > Massimo > > > On Jul 20, 4:30 pm, Michael Toomim wrote: > > > > Let me also summarize the issues so far. > > > > Originally: > > > - I got three types of error messages in apache logs > > > - Logging messages were often duplicated 2, 3, 5 times > > > - I got the IOError ticket a
[web2py] Re: Error in wsgi/apache
Let me also summarize the issues so far. Originally: - I got three types of error messages in apache logs - Logging messages were often duplicated 2, 3, 5 times - I got the IOError ticket a few times - After a while the web serving slowed (some requests took up to a minute) and then quit completely After rebooting: - I get one type of error message in apache logs, in big batches - I get the IOError ticket once or twice - After a while web serving slows (sometimes 150s per request) and stops So I haven't been seeing the duplicate log messages anymore. I upgraded to a bigger machine and am changing my code to remove ajax (will reduce load by 60x by decreasing functionality). I don't know what else to do. On Jul 20, 2:03 am, mdipierro wrote: > Thanks for the clarification. > > @Michael, do you use the logging module? How? > > On Jul 20, 4:00 am, Graham Dumpleton > wrote: > > > > > On Jul 20, 5:17 pm, mdipierro wrote: > > > > The problem with IOError, I can understand. As Graham says, if the > > > client closes the connection before the server responds or if the > > > server timesout the socket is closed and apache logs the IOError. > > > That isn't what I said. If you see that message when using daemon > > mode, the Apache server process that is proxying to the daemon process > > is crashing. This is different to the HTTP client closing the > > connection. You would only see that message if HTTP client closed > > connection if using embedded mode. > > > I know they are using daemon mode as that is the only situation where > > they could also see the message about premature end of script headers. > > > > What I really do not understand is why some requests are handled by > > > multiple threads. web2py is agnostic to this (unless you use Rocket > > > which you do not). web2py only provides a wsgi application which is > > > executed - per thread - by the web server. It is the web server (in > > > your case apache) that spans the thread, maps requests to threads, > > > calls the web2py wsgi application for each of them. > > > > If this is happening it is a problem with apache or with mod_wsgi. > > > More likely the problem is that they are registering the logging > > module from multiple places and that is why logging is displayed more > > than once. They should log the thread ID as well as that would confirm > > whether actually from the same thread where logging module handler has > > been registered multiple times. > > > Multiple registrations of logging handler could occur if it isn't done > > in a thread safe why, ie., so as to avoid multiple threads doing it at > > the same time. > > > Graham > > > > Can > > > you tell us more about the version of ubuntu, apache and mod_wsgi that > > > you are using? Any additional information will be very useful. > > > > Massimo > > > > On Jul 19, 9:01 pm, Michael Toomim wrote: > > > > > I'm getting errors like these in my apache error logs: > > > > > [Mon Jul 19 18:55:20 2010] [error] [client 65.35.93.74] Premature end > > > > of script headers: wsgihandler.py, > > > > referer:http://yuno.us/init/hits/hit?assignmentId=1A7KADKCHTB1IJS3Z5CR16OZM4V... > > > > [Mon Jul 19 18:55:20 2010] [error] [client 143.166.226.43] Premature > > > > end of script headers: wsgihandler.py, > > > > referer:http://yuno.us/init/hits/hit?assignmentId=1A9FV5YBGVV54NALMIRILFKHPT1... > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.204.99.178] mod_wsgi > > > > (pid=7730): Exception occurred processing WSGI script '/home/toomim/ > > > > projects/utility/web2py/wsgihandler.py'. > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi > > > > (pid=7730): Exception occurred processing WSGI script '/home/toomim/ > > > > projects/utility/web2py/wsgihandler.py'. > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi > > > > (pid=7730): Exception occurred processing WSGI script '/home/toomim/ > > > > projects/utility/web2py/wsgihandler.py'. > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError: > > > > failed to write data > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi > > > > (pid=7730): Exception occurred processing WSGI script '/home/toomim/ > > > > projects/utility/web2py/wsgihandler.py'. > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError: &
[web2py] Re: Error in wsgi/apache
Thank you for the clarification.
My wsgi.conf has default values, so I have not set maximum-requests.
Perhaps there are settings there I should look into?
I still have free memory, so perhaps there is not a memory leak
issue. I'm also not sure how one would get memory leaks in web2py,
since isn't the environment wiped clean with each request?
This looks similar to the issue here:
http://groups.google.com/group/web2py/browse_thread/thread/49a7ecabf4910bcc/b6fac1806ffebcd1?lnk=gst&q=ioerror#b6fac1806ffebcd1
Was there any resolution?
I use logging by having the following file in models/0_log.py:
import logging
def get_log():
try:
f = open(logging.handlers[0].baseFilename, 'r')
c = f.readlines()
f.close()
return {'log':TABLE(*[TR(str(item)) for item in c])}
except:
return ()
# This model file defines some magic to implement app_wide_log.
def _init_log(): # Does not work on GAE
import os,logging,logging.handlers
logger = logging.getLogger(request.application)
logger.setLevel(logging.DEBUG)
handler = logging.handlers.RotatingFileHandler(
os.path.join( # so that it can be served as
http://.../yourapp/static/applog.txt
request.folder,'static','applog.txt'),'a',1024*1024,1)
handler.setLevel(logging.DEBUG)
handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)s %
(filename)s:%(lineno)d %(funcName)s(): %(message)s"))
logger.addHandler(handler)
return logger
logging =
cache.ram('app_wide_log',lambda:_init_log(),time_expire=None)
On Jul 20, 2:03 am, mdipierro wrote:
> Thanks for the clarification.
>
> @Michael, do you use the logging module? How?
>
> On Jul 20, 4:00 am, Graham Dumpleton
> wrote:
>
>
>
> > On Jul 20, 5:17 pm, mdipierro wrote:
>
> > > The problem with IOError, I can understand. As Graham says, if the
> > > client closes the connection before the server responds or if the
> > > server timesout the socket is closed and apache logs the IOError.
>
> > That isn't what I said. If you see that message when using daemon
> > mode, the Apache server process that is proxying to the daemon process
> > is crashing. This is different to the HTTP client closing the
> > connection. You would only see that message if HTTP client closed
> > connection if using embedded mode.
>
> > I know they are using daemon mode as that is the only situation where
> > they could also see the message about premature end of script headers.
>
> > > What I really do not understand is why some requests are handled by
> > > multiple threads. web2py is agnostic to this (unless you use Rocket
> > > which you do not). web2py only provides a wsgi application which is
> > > executed - per thread - by the web server. It is the web server (in
> > > your case apache) that spans the thread, maps requests to threads,
> > > calls the web2py wsgi application for each of them.
>
> > > If this is happening it is a problem with apache or with mod_wsgi.
>
> > More likely the problem is that they are registering the logging
> > module from multiple places and that is why logging is displayed more
> > than once. They should log the thread ID as well as that would confirm
> > whether actually from the same thread where logging module handler has
> > been registered multiple times.
>
> > Multiple registrations of logging handler could occur if it isn't done
> > in a thread safe why, ie., so as to avoid multiple threads doing it at
> > the same time.
>
> > Graham
>
> > > Can
> > > you tell us more about the version of ubuntu, apache and mod_wsgi that
> > > you are using? Any additional information will be very useful.
>
> > > Massimo
>
> > > On Jul 19, 9:01 pm, Michael Toomim wrote:
>
> > > > I'm getting errors like these in my apache error logs:
>
> > > > [Mon Jul 19 18:55:20 2010] [error] [client 65.35.93.74] Premature end
> > > > of script headers: wsgihandler.py,
> > > > referer:http://yuno.us/init/hits/hit?assignmentId=1A7KADKCHTB1IJS3Z5CR16OZM4V...
> > > > [Mon Jul 19 18:55:20 2010] [error] [client 143.166.226.43] Premature
> > > > end of script headers: wsgihandler.py,
> > > > referer:http://yuno.us/init/hits/hit?assignmentId=1A9FV5YBGVV54NALMIRILFKHPT1...
> > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.204.99.178] mod_wsgi
> > > > (pid=7730): Exception occurred processing WSGI script '/home/toomim/
> > > > projects/utility/web2py/wsgihandler.py'.
> > > > [Mon Jul 19 18:55:50 2010] [
[web2py] Re: Error in wsgi/apache
Thanks! I tried rebooting the OS. Now my resources seem ok (but I didn't check before the reboot): Files used: 1376 out of 75556 Mem used: 580mb out of 796mb Swap used: 0 CPU: 88-99% idle And I know longer see the "Exception occurred" or "IOError" messages, however I DO still see "Premature end of script headers". These errors come in batches, every 10-20 seconds or so I get a continuous block of 10-20 "Premature end of script headers" errors from different clients. These are followed by errors notifying me that clients' ajax requests failed. I also found three of these in my web2py tickets: Traceback (most recent call last): File "gluon/main.py", line 337, in wsgibase parse_get_post_vars(request, environ) File "gluon/main.py", line 222, in parse_get_post_vars request.body = copystream_progress(request) ### stores request body File "gluon/main.py", line 95, in copystream_progress copystream(source, dest, size, chunk_size) File "gluon/fileutils.py", line 301, in copystream data = src.read(size) IOError: request data read error However, I've gotten around 3000 "premature end of script" errors, and only 3 of these IOErrors. Is there a way to identify what is causing the "Premature end of script" errors? On Jul 19, 7:50 pm, Graham Dumpleton wrote: > On Jul 20, 12:01 pm, Michael Toomim wrote: > > > I'm getting errors like these in my apache error logs: > > > [Mon Jul 19 18:55:20 2010] [error] [client 65.35.93.74] Premature end > > of script headers: wsgihandler.py, > > referer:http://yuno.us/init/hits/hit?assignmentId=1A7KADKCHTB1IJS3Z5CR16OZM4V... > > [Mon Jul 19 18:55:20 2010] [error] [client 143.166.226.43] Premature > > end of script headers: wsgihandler.py, > > referer:http://yuno.us/init/hits/hit?assignmentId=1A9FV5YBGVV54NALMIRILFKHPT1... > > The above is because the daemon process you are running web2py in > crashed. > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.204.99.178] mod_wsgi > > (pid=7730): Exception occurred processing WSGI script '/home/toomim/ > > projects/utility/web2py/wsgihandler.py'. > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi > > (pid=7730): Exception occurred processing WSGI script '/home/toomim/ > > projects/utility/web2py/wsgihandler.py'. > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi > > (pid=7730): Exception occurred processing WSGI script '/home/toomim/ > > projects/utility/web2py/wsgihandler.py'. > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError: > > failed to write data > > In the case of daemon mode being used, this is because the Apache > server child process crashed. > > > > > > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi > > (pid=7730): Exception occurred processing WSGI script '/home/toomim/ > > projects/utility/web2py/wsgihandler.py'. > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError: > > failed to write data > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi > > (pid=7730): Exception occurred processing WSGI script '/home/toomim/ > > projects/utility/web2py/wsgihandler.py'. > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError: > > failed to write data > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi > > (pid=7730): Exception occurred processing WSGI script '/home/toomim/ > > projects/utility/web2py/wsgihandler.py'. > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError: > > failed to write data > > > My web app gets about 7 requests per second. At first, things work > > fine. Then after a while it seems like every request gets handled by > > MULTIPLE threads, because my logging.debug() statements print multiple > > copies of each message and it seems my database gets multiple entries. > > And I get these errors in the apache logs (with LogLevel debug). > > > Any idea what to do? Where to look? I'm on ubuntu. > > Look at your systems resource usage, ie., memory, open files etc. The > above are symptomatic of your operating system running out of > resources and processes not coping too well with that. > > Graham
[web2py] Re: Error in wsgi/apache
And after a while apache completely freezes. On Jul 19, 7:05 pm, Michael Toomim wrote: > This message about "bucket brigade" is also appearing in the apache > error log: > > [Mon Jul 19 19:01:53 2010] [error] [client 183.87.223.111] (9)Bad file > descriptor: mod_wsgi (pid=7940): Unable to get bucket brigade for > request., > referer:http://yuno.us/init/hits/hit?assignmentId=1WL68USPJR0HY1ENS50GN6IJ33Z... > > On Jul 19, 7:01 pm, Michael Toomim wrote: > > > > > I'm getting errors like these in my apache error logs: > > > [Mon Jul 19 18:55:20 2010] [error] [client 65.35.93.74] Premature end > > of script headers: wsgihandler.py, > > referer:http://yuno.us/init/hits/hit?assignmentId=1A7KADKCHTB1IJS3Z5CR16OZM4V... > > [Mon Jul 19 18:55:20 2010] [error] [client 143.166.226.43] Premature > > end of script headers: wsgihandler.py, > > referer:http://yuno.us/init/hits/hit?assignmentId=1A9FV5YBGVV54NALMIRILFKHPT1... > > [Mon Jul 19 18:55:50 2010] [error] [client 117.204.99.178] mod_wsgi > > (pid=7730): Exception occurred processing WSGI script '/home/toomim/ > > projects/utility/web2py/wsgihandler.py'. > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi > > (pid=7730): Exception occurred processing WSGI script '/home/toomim/ > > projects/utility/web2py/wsgihandler.py'. > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi > > (pid=7730): Exception occurred processing WSGI script '/home/toomim/ > > projects/utility/web2py/wsgihandler.py'. > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError: > > failed to write data > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi > > (pid=7730): Exception occurred processing WSGI script '/home/toomim/ > > projects/utility/web2py/wsgihandler.py'. > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError: > > failed to write data > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi > > (pid=7730): Exception occurred processing WSGI script '/home/toomim/ > > projects/utility/web2py/wsgihandler.py'. > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError: > > failed to write data > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi > > (pid=7730): Exception occurred processing WSGI script '/home/toomim/ > > projects/utility/web2py/wsgihandler.py'. > > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError: > > failed to write data > > > My web app gets about 7 requests per second. At first, things work > > fine. Then after a while it seems like every request gets handled by > > MULTIPLE threads, because my logging.debug() statements print multiple > > copies of each message and it seems my database gets multiple entries. > > And I get these errors in the apache logs (with LogLevel debug). > > > Any idea what to do? Where to look? I'm on ubuntu.
[web2py] Re: Error in wsgi/apache
This message about "bucket brigade" is also appearing in the apache error log: [Mon Jul 19 19:01:53 2010] [error] [client 183.87.223.111] (9)Bad file descriptor: mod_wsgi (pid=7940): Unable to get bucket brigade for request., referer: http://yuno.us/init/hits/hit?assignmentId=1WL68USPJR0HY1ENS50GN6IJ33ZY32&hitId=1TK6NH2ZSBU3RCI3F8FK7JE1YXMG96&workerId=AJ8R357DF74FF&turkSubmitTo=https%3A%2F%2Fwww.mturk.com On Jul 19, 7:01 pm, Michael Toomim wrote: > I'm getting errors like these in my apache error logs: > > [Mon Jul 19 18:55:20 2010] [error] [client 65.35.93.74] Premature end > of script headers: wsgihandler.py, > referer:http://yuno.us/init/hits/hit?assignmentId=1A7KADKCHTB1IJS3Z5CR16OZM4V... > [Mon Jul 19 18:55:20 2010] [error] [client 143.166.226.43] Premature > end of script headers: wsgihandler.py, > referer:http://yuno.us/init/hits/hit?assignmentId=1A9FV5YBGVV54NALMIRILFKHPT1... > [Mon Jul 19 18:55:50 2010] [error] [client 117.204.99.178] mod_wsgi > (pid=7730): Exception occurred processing WSGI script '/home/toomim/ > projects/utility/web2py/wsgihandler.py'. > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi > (pid=7730): Exception occurred processing WSGI script '/home/toomim/ > projects/utility/web2py/wsgihandler.py'. > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi > (pid=7730): Exception occurred processing WSGI script '/home/toomim/ > projects/utility/web2py/wsgihandler.py'. > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError: > failed to write data > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi > (pid=7730): Exception occurred processing WSGI script '/home/toomim/ > projects/utility/web2py/wsgihandler.py'. > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError: > failed to write data > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi > (pid=7730): Exception occurred processing WSGI script '/home/toomim/ > projects/utility/web2py/wsgihandler.py'. > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError: > failed to write data > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi > (pid=7730): Exception occurred processing WSGI script '/home/toomim/ > projects/utility/web2py/wsgihandler.py'. > [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError: > failed to write data > > My web app gets about 7 requests per second. At first, things work > fine. Then after a while it seems like every request gets handled by > MULTIPLE threads, because my logging.debug() statements print multiple > copies of each message and it seems my database gets multiple entries. > And I get these errors in the apache logs (with LogLevel debug). > > Any idea what to do? Where to look? I'm on ubuntu.
[web2py] Error in wsgi/apache
I'm getting errors like these in my apache error logs: [Mon Jul 19 18:55:20 2010] [error] [client 65.35.93.74] Premature end of script headers: wsgihandler.py, referer: http://yuno.us/init/hits/hit?assignmentId=1A7KADKCHTB1IJS3Z5CR16OZM4VLSQ&hitId=1NAV09D0NWNU2X87QR3I6RXXG0ER8N&workerId=A37YC0D90LZF2M&turkSubmitTo=https%3A%2F%2Fwww.mturk.com [Mon Jul 19 18:55:20 2010] [error] [client 143.166.226.43] Premature end of script headers: wsgihandler.py, referer: http://yuno.us/init/hits/hit?assignmentId=1A9FV5YBGVV54NALMIRILFKHPT1O3I&hitId=1G15BSUI1DBLMZPV54KGZFTE6JM0Z3&workerId=A3I5DLZHYT46GS&turkSubmitTo=https%3A%2F%2Fwww.mturk.com [Mon Jul 19 18:55:50 2010] [error] [client 117.204.99.178] mod_wsgi (pid=7730): Exception occurred processing WSGI script '/home/toomim/ projects/utility/web2py/wsgihandler.py'. [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi (pid=7730): Exception occurred processing WSGI script '/home/toomim/ projects/utility/web2py/wsgihandler.py'. [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi (pid=7730): Exception occurred processing WSGI script '/home/toomim/ projects/utility/web2py/wsgihandler.py'. [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError: failed to write data [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi (pid=7730): Exception occurred processing WSGI script '/home/toomim/ projects/utility/web2py/wsgihandler.py'. [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError: failed to write data [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi (pid=7730): Exception occurred processing WSGI script '/home/toomim/ projects/utility/web2py/wsgihandler.py'. [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError: failed to write data [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] mod_wsgi (pid=7730): Exception occurred processing WSGI script '/home/toomim/ projects/utility/web2py/wsgihandler.py'. [Mon Jul 19 18:55:50 2010] [error] [client 117.201.42.84] IOError: failed to write data My web app gets about 7 requests per second. At first, things work fine. Then after a while it seems like every request gets handled by MULTIPLE threads, because my logging.debug() statements print multiple copies of each message and it seems my database gets multiple entries. And I get these errors in the apache logs (with LogLevel debug). Any idea what to do? Where to look? I'm on ubuntu.
[web2py] Re: Best practice for logging w/ wsgi?
Oops I'm sorry, when I upgraded the log file moved and I was looking at the wrong one. It works! On Apr 7, 11:44 pm, Michael Toomim wrote: > Now that I'm on apache, I find that the logging library iceberg wrote > no longer > works:http://groups.google.com/group/web2py/browse_thread/thread/ae37920ce0... > > I suspect this is because of the stdout/stderr problem with wsgi, but > I thought that would only affect print statements... which is the > reason for using logging.debug(). But my logging.debug() doesn't work > on apache. > > Is there a way to fix this? How do you guys debug on apache? -- You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web...@googlegroups.com. To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/web2py?hl=en.
[web2py] Best practice for logging w/ wsgi?
Now that I'm on apache, I find that the logging library iceberg wrote no longer works: http://groups.google.com/group/web2py/browse_thread/thread/ae37920ce03ba165/6e5d746f6222f70a I suspect this is because of the stdout/stderr problem with wsgi, but I thought that would only affect print statements... which is the reason for using logging.debug(). But my logging.debug() doesn't work on apache. Is there a way to fix this? How do you guys debug on apache? -- You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web...@googlegroups.com. To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/web2py?hl=en.
[web2py] Re: webserver slow, misreported
and I'm using postgres not sqlite.
On Apr 5, 12:44 pm, Michael Toomim wrote:
> Thanks guys. Each time I run a test, though, it costs me money
> because I'm paying people on mechanical turk. And if it's slow, it
> gives me a bad reputation. So I don't want to run more slow tests
> unless we have good request time logging in place and a good
> hypothesis to test.
>
> Wouldn't cron only make it slow every minute or something?
>
> On Apr 5, 5:49 am, Timothy Farrell wrote:
>
>
>
> > You are right, I was going to add that feature and then forgot about
> > it. Someone reported a PyPI bug over the weekend (it would not affect
> > web2py). I'll see if I can make the logging a bit more flexible and
> > release a 1.1 in the next few days.
>
> > In the meantime, look into the cron thing.
>
> > -tim
>
> > On 4/4/2010 6:44 PM, Michael Toomim wrote:
>
> > > I see, thank you. I want to measure the web server's response time
> > > when I deploy this on turk... Unfortunately the rocket log does not
> > > report time to serve a request. Do you think it is easy to get that
> > > information from rocket? Do you store the start and stop times for
> > > each request? I see start times stored in "connections", but I'm not
> > > sure that's the right object.
>
> > > On Mar 30, 6:09 am, Timothy Farrell wrote:
>
> > >> I don't think upgrading will help much since Cherrypy was also slow.
> > >> However, doing so would help cover all your bases.
>
> > >> If you want to use the http log from Rocket you can do this. I'm
> > >> assuming you invoke web2py.py from a bash script or just run it
> > >> manually. Paste the following code into the top of web2py.py
>
> > >> import logging
> > >> import logging.handlers
> > >> log = logging.getLogger('Rocket.Requests')
> > >> log.setLevel(logging.INFO)
> > >> log.addHandler(logging.handlers.FileHandler('rocket.log')
>
> > >> I, like Yarko, do think this has more to do with something else. At one
> > >> point web2py had a profiler built-in. That could be a good tool for
> > >> finding slow spots.
>
> > >> -tim
>
> > >> On 3/29/2010 7:59 PM, MichaelToomimwrote:
>
> > >>> Yes, this is on linux! Do you recommend upgrading and trying again?
>
> > >>> mturk doesn't affect anything, I am just serving webpages that appear
> > >>> in iframes on the mturk website. From our perspective, I'm serving
> > >>> webpages.
>
> > >>> Do you have a method of logging how much time it takes to serve a page
> > >>> with rocket? Something that I can use instead of httpserver.log? It
> > >>> seems important for me to measure real-world performance, which ab
> > >>> does not do.
>
> > >>> My server has 768MB ram, and the only thing it does is run this web2py
> > >>> server. I assumed ram was not full, but did not check. I will check
> > >>> next time.
>
> > >>> On Mar 29, 12:10 pm, Timothy Farrell wrote:
>
> > >>
--
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To post to this group, send email to web...@googlegroups.com.
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en.
[web2py] Re: webserver slow, misreported
Thanks guys. Each time I run a test, though, it costs me money
because I'm paying people on mechanical turk. And if it's slow, it
gives me a bad reputation. So I don't want to run more slow tests
unless we have good request time logging in place and a good
hypothesis to test.
Wouldn't cron only make it slow every minute or something?
On Apr 5, 5:49 am, Timothy Farrell wrote:
> You are right, I was going to add that feature and then forgot about
> it. Someone reported a PyPI bug over the weekend (it would not affect
> web2py). I'll see if I can make the logging a bit more flexible and
> release a 1.1 in the next few days.
>
> In the meantime, look into the cron thing.
>
> -tim
>
> On 4/4/2010 6:44 PM, Michael Toomim wrote:
>
>
>
> > I see, thank you. I want to measure the web server's response time
> > when I deploy this on turk... Unfortunately the rocket log does not
> > report time to serve a request. Do you think it is easy to get that
> > information from rocket? Do you store the start and stop times for
> > each request? I see start times stored in "connections", but I'm not
> > sure that's the right object.
>
> > On Mar 30, 6:09 am, Timothy Farrell wrote:
>
> >> I don't think upgrading will help much since Cherrypy was also slow.
> >> However, doing so would help cover all your bases.
>
> >> If you want to use the http log from Rocket you can do this. I'm
> >> assuming you invoke web2py.py from a bash script or just run it
> >> manually. Paste the following code into the top of web2py.py
>
> >> import logging
> >> import logging.handlers
> >> log = logging.getLogger('Rocket.Requests')
> >> log.setLevel(logging.INFO)
> >> log.addHandler(logging.handlers.FileHandler('rocket.log')
>
> >> I, like Yarko, do think this has more to do with something else. At one
> >> point web2py had a profiler built-in. That could be a good tool for
> >> finding slow spots.
>
> >> -tim
>
> >> On 3/29/2010 7:59 PM, MichaelToomimwrote:
>
> >>> Yes, this is on linux! Do you recommend upgrading and trying again?
>
> >>> mturk doesn't affect anything, I am just serving webpages that appear
> >>> in iframes on the mturk website. From our perspective, I'm serving
> >>> webpages.
>
> >>> Do you have a method of logging how much time it takes to serve a page
> >>> with rocket? Something that I can use instead of httpserver.log? It
> >>> seems important for me to measure real-world performance, which ab
> >>> does not do.
>
> >>> My server has 768MB ram, and the only thing it does is run this web2py
> >>> server. I assumed ram was not full, but did not check. I will check
> >>> next time.
>
> >>> On Mar 29, 12:10 pm, Timothy Farrell wrote:
>
> >>
--
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To post to this group, send email to web...@googlegroups.com.
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en.
[web2py] Re: webserver slow, misreported
I started using apache with mod_wsgi, and now it's fast! So this
indicates it's a problem that only occurs when using rocket or
cherrypy, but again I'm only measuring it with firebug in my browser.
I have 768MB of ram, ~500MB free. I use cron for @reboot only.
I only run it on a remote machine (on slicehost VPS). I do not think
it is network problems because it is fast in my web browser until I
get hit by a bunch of users. Of course I'd like to eliminate the
network from my measurements, but for that I need webserver processing-
time logs.
On Apr 4, 9:08 pm, mdipierro wrote:
> Some more questions:
>
> how much ram?
> can you check memory usage? A memory leak may cause slowness.
> are you using cron? when cron starts it may spike memory usage.
> are you experience the slowness from localhost or from remote
> machines?
>
> On Apr 4, 6:46 pm, Michael Toomim wrote:
>
>
>
> > You are both right that I do not know where the slowness is coming
> > from. My goal is to measure it so that I can narrow in on the
> > problem. So far I know that it is external to web2py because it does
> > not show up in httpserver.log, so my reasoning is to look at rocket
> > which wraps the web2py part.
>
> > On Apr 4, 4:44 pm, Michael Toomim wrote:
>
> > > I see, thank you. I want to measure the web server's response time
> > > when I deploy this on turk... Unfortunately the rocket log does not
> > > report time to serve a request. Do you think it is easy to get that
> > > information from rocket? Do you store the start and stop times for
> > > each request? I see start times stored in "connections", but I'm not
> > > sure that's the right object.
>
> > > On Mar 30, 6:09 am, Timothy Farrell wrote:
>
> > > > I don't think upgrading will help much since Cherrypy was also slow.
> > > > However, doing so would help cover all your bases.
>
> > > > If you want to use the http log from Rocket you can do this. I'm
> > > > assuming you invoke web2py.py from a bash script or just run it
> > > > manually. Paste the following code into the top of web2py.py
>
> > > > import logging
> > > > import logging.handlers
> > > > log = logging.getLogger('Rocket.Requests')
> > > > log.setLevel(logging.INFO)
> > > > log.addHandler(logging.handlers.FileHandler('rocket.log')
>
> > > > I, like Yarko, do think this has more to do with something else. At one
> > > > point web2py had a profiler built-in. That could be a good tool for
> > > > finding slow spots.
>
> > > > -tim
>
> > > > On 3/29/2010 7:59 PM, MichaelToomimwrote:
>
> > > > > Yes, this is on linux! Do you recommend upgrading and trying again?
>
> > > > > mturk doesn't affect anything, I am just serving webpages that appear
> > > > > in iframes on the mturk website. From our perspective, I'm serving
> > > > > webpages.
>
> > > > > Do you have a method of logging how much time it takes to serve a page
> > > > > with rocket? Something that I can use instead of httpserver.log? It
> > > > > seems important for me to measure real-world performance, which ab
> > > > > does not do.
>
> > > > > My server has 768MB ram, and the only thing it does is run this web2py
> > > > > server. I assumed ram was not full, but did not check. I will check
> > > > > next time.
>
> > > > > On Mar 29, 12:10 pm, Timothy Farrell wrote:
>
> > > >
--
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To post to this group, send email to web...@googlegroups.com.
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en.
[web2py] Re: webserver slow, misreported
You are both right that I do not know where the slowness is coming
from. My goal is to measure it so that I can narrow in on the
problem. So far I know that it is external to web2py because it does
not show up in httpserver.log, so my reasoning is to look at rocket
which wraps the web2py part.
On Apr 4, 4:44 pm, Michael Toomim wrote:
> I see, thank you. I want to measure the web server's response time
> when I deploy this on turk... Unfortunately the rocket log does not
> report time to serve a request. Do you think it is easy to get that
> information from rocket? Do you store the start and stop times for
> each request? I see start times stored in "connections", but I'm not
> sure that's the right object.
>
> On Mar 30, 6:09 am, Timothy Farrell wrote:
>
>
>
> > I don't think upgrading will help much since Cherrypy was also slow.
> > However, doing so would help cover all your bases.
>
> > If you want to use the http log from Rocket you can do this. I'm
> > assuming you invoke web2py.py from a bash script or just run it
> > manually. Paste the following code into the top of web2py.py
>
> > import logging
> > import logging.handlers
> > log = logging.getLogger('Rocket.Requests')
> > log.setLevel(logging.INFO)
> > log.addHandler(logging.handlers.FileHandler('rocket.log')
>
> > I, like Yarko, do think this has more to do with something else. At one
> > point web2py had a profiler built-in. That could be a good tool for
> > finding slow spots.
>
> > -tim
>
> > On 3/29/2010 7:59 PM, MichaelToomimwrote:
>
> > > Yes, this is on linux! Do you recommend upgrading and trying again?
>
> > > mturk doesn't affect anything, I am just serving webpages that appear
> > > in iframes on the mturk website. From our perspective, I'm serving
> > > webpages.
>
> > > Do you have a method of logging how much time it takes to serve a page
> > > with rocket? Something that I can use instead of httpserver.log? It
> > > seems important for me to measure real-world performance, which ab
> > > does not do.
>
> > > My server has 768MB ram, and the only thing it does is run this web2py
> > > server. I assumed ram was not full, but did not check. I will check
> > > next time.
>
> > > On Mar 29, 12:10 pm, Timothy Farrell wrote:
>
> >
--
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To post to this group, send email to web...@googlegroups.com.
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en.
[web2py] Re: webserver slow, misreported
I see, thank you. I want to measure the web server's response time
when I deploy this on turk... Unfortunately the rocket log does not
report time to serve a request. Do you think it is easy to get that
information from rocket? Do you store the start and stop times for
each request? I see start times stored in "connections", but I'm not
sure that's the right object.
On Mar 30, 6:09 am, Timothy Farrell wrote:
> I don't think upgrading will help much since Cherrypy was also slow.
> However, doing so would help cover all your bases.
>
> If you want to use the http log from Rocket you can do this. I'm
> assuming you invoke web2py.py from a bash script or just run it
> manually. Paste the following code into the top of web2py.py
>
> import logging
> import logging.handlers
> log = logging.getLogger('Rocket.Requests')
> log.setLevel(logging.INFO)
> log.addHandler(logging.handlers.FileHandler('rocket.log')
>
> I, like Yarko, do think this has more to do with something else. At one
> point web2py had a profiler built-in. That could be a good tool for
> finding slow spots.
>
> -tim
>
> On 3/29/2010 7:59 PM, MichaelToomimwrote:
>
>
>
> > Yes, this is on linux! Do you recommend upgrading and trying again?
>
> > mturk doesn't affect anything, I am just serving webpages that appear
> > in iframes on the mturk website. From our perspective, I'm serving
> > webpages.
>
> > Do you have a method of logging how much time it takes to serve a page
> > with rocket? Something that I can use instead of httpserver.log? It
> > seems important for me to measure real-world performance, which ab
> > does not do.
>
> > My server has 768MB ram, and the only thing it does is run this web2py
> > server. I assumed ram was not full, but did not check. I will check
> > next time.
>
> > On Mar 29, 12:10 pm, Timothy Farrell wrote:
>
>
--
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To post to this group, send email to web...@googlegroups.com.
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en.
[web2py] Re: webserver slow, misreported
Yes, this is on linux! Do you recommend upgrading and trying again? mturk doesn't affect anything, I am just serving webpages that appear in iframes on the mturk website. From our perspective, I'm serving webpages. Do you have a method of logging how much time it takes to serve a page with rocket? Something that I can use instead of httpserver.log? It seems important for me to measure real-world performance, which ab does not do. My server has 768MB ram, and the only thing it does is run this web2py server. I assumed ram was not full, but did not check. I will check next time. On Mar 29, 12:10 pm, Timothy Farrell wrote: > On 3/29/2010 1:39 PM, Michael Toomim wrote: > > > I was having slowness problems with cherrypy too! That's why I > > switched to rocket. So perhaps it's something common to cherrypy and > > rocket, or perhaps they are both slow in their own ways? > > > This is using web2py from march 16th, so it's not the latest rocket. > > Do you think something important changed for concurrency? > > I'm the author of Rocket. I _know_ something important changed on March > 18th. =) But that important change only really affects the *nix > platform. You haven't said what you're running on. > > I'm not familiar with MTurk very well. Is it directly connected to your > web2py setup? Does it run on Windows/Linux? > > You said that you were having trouble with Cherrypy too. Is Rocket > better or worse than Cherrypy? > > The one hang-up that I can see here is if you're server is > memory-limited then multiple concurrent connections will cause thrashing > due to swapping. This situation would be fast with one but slow with > multiple connections. > > We need some more information before we can help you further. But if > Cherrypy wasn't cutting it then perhaps you should look into some of the > native code solutions such as Apache. This sounds like something wider > than just the webserver. > > -tim > > > > > On Mar 29, 5:56 am, Timothy Farrell wrote: > > >> Perhaps a simpler set of questions: > > >> Did you have this working with Cherrypy beforehand? > >> If so, is Rocket the only thing to have changed? > > >> The latest changes to Rocket were committed to the Mercurial web2py repo > >> on March 18th. I'm assuming you've run a checkout since then. > > >> -tim > > >> On 3/28/2010 4:23 PM, mdipierro wrote: > > >>> One more thing. You ask > > >>>> But a single process doing complex joins should not slow down > >>>> all other simple selects and inserts, right? > > >>> no, except for sqlite. sqlite serializes all requests because locks > >>> the db. That could explain the 0.20s if you have lots of queries per > >>> request, but not the 54s for the server. > > >>> On Mar 28, 4:22 pm, mdipierro wrote: > > >>>> On Mar 28, 3:46 pm, Michael Toomim wrote: > > >>>>> Any idea why there is a discrepancy between Firebug and > >>>>> httpserver.log? > > >>>> httpserver.log logs the time spend in web2py, not including the time > >>>> for sending and receiving the http request/response. > >>>> firebug logs the the total time, including time spend by the web > >>>> server for communication. > > >>>>> I am using postgresql. What would indicate "model complexity"? I > >>>>> have around 9 tables, but most of the requests just do single-object > >>>>> selects and inserts. No complex joins are in public-facing pages, but > >>>>> myself as an administrator periodically load a page that does big > >>>>> joins. But a single process doing complex joins should not slow down > >>>>> all other simple selects and inserts, right? > > >>>> In your case there are two problems (and I do not know what causes > >>>> them): > >>>> 1) web2py is taking 0.20seconds to process a response. That is more > >>>> than 10 times what it should be. > >>>> 2) the communication between the web server and the browser takes very > >>>> very long time. > > >>>> Is the server on localhost? If not this could be a network issue. > > >>>>> On Mar 27, 6:48 am, mdipierro wrote: > > >>>>>> Mind that if you use sqlite there is no concurrency. Still these > >>>>>> numbers are very low. > >>>>>> Are your models very complex? > > >>>
[web2py] Re: webserver slow, misreported
I was having slowness problems with cherrypy too! That's why I switched to rocket. So perhaps it's something common to cherrypy and rocket, or perhaps they are both slow in their own ways? This is using web2py from march 16th, so it's not the latest rocket. Do you think something important changed for concurrency? On Mar 29, 5:56 am, Timothy Farrell wrote: > Perhaps a simpler set of questions: > > Did you have this working with Cherrypy beforehand? > If so, is Rocket the only thing to have changed? > > The latest changes to Rocket were committed to the Mercurial web2py repo > on March 18th. I'm assuming you've run a checkout since then. > > -tim > > On 3/28/2010 4:23 PM, mdipierro wrote: > > > > > One more thing. You ask > > >> But a single process doing complex joins should not slow down > >> all other simple selects and inserts, right? > > > no, except for sqlite. sqlite serializes all requests because locks > > the db. That could explain the 0.20s if you have lots of queries per > > request, but not the 54s for the server. > > > On Mar 28, 4:22 pm, mdipierro wrote: > > >> On Mar 28, 3:46 pm, Michael Toomim wrote: > > >>> Any idea why there is a discrepancy between Firebug and > >>> httpserver.log? > > >> httpserver.log logs the time spend in web2py, not including the time > >> for sending and receiving the http request/response. > >> firebug logs the the total time, including time spend by the web > >> server for communication. > > >>> I am using postgresql. What would indicate "model complexity"? I > >>> have around 9 tables, but most of the requests just do single-object > >>> selects and inserts. No complex joins are in public-facing pages, but > >>> myself as an administrator periodically load a page that does big > >>> joins. But a single process doing complex joins should not slow down > >>> all other simple selects and inserts, right? > > >> In your case there are two problems (and I do not know what causes > >> them): > >> 1) web2py is taking 0.20seconds to process a response. That is more > >> than 10 times what it should be. > >> 2) the communication between the web server and the browser takes very > >> very long time. > > >> Is the server on localhost? If not this could be a network issue. > > >>> On Mar 27, 6:48 am, mdipierro wrote: > > >>>> Mind that if you use sqlite there is no concurrency. Still these > >>>> numbers are very low. > >>>> Are your models very complex? > > >>>> On 27 Mar, 00:06, Michael Toomim wrote: > > >>>>> I'm using web2py+rocket to serve jobs on mechanical turk. The server > >>>>> probably gets a hit per second or so by workers on mechanical turk > >>>>> using it. > > >>>>> When I have no users, everything is fast. But in active use, I notice > >>>>> that web pages often load reay slow in my web browser, but the > >>>>> httpserver.log file reports only small times. > > >>>>> For instance, I just loaded a page that httpserver.log said took > >>>>> 0.20 seconds, but Firebug said took 54.21 seconds. That's a big > >>>>> difference. Any idea what's going on? I guess I'll have to try apache? -- You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web...@googlegroups.com. To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/web2py?hl=en.
[web2py] Re: webserver slow, misreported
Any idea why there is a discrepancy between Firebug and httpserver.log? I am using postgresql. What would indicate "model complexity"? I have around 9 tables, but most of the requests just do single-object selects and inserts. No complex joins are in public-facing pages, but myself as an administrator periodically load a page that does big joins. But a single process doing complex joins should not slow down all other simple selects and inserts, right? On Mar 27, 6:48 am, mdipierro wrote: > Mind that if you use sqlite there is no concurrency. Still these > numbers are very low. > Are your models very complex? > > On 27 Mar, 00:06, Michael Toomim wrote: > > > > > I'm using web2py+rocket to serve jobs on mechanical turk. The server > > probably gets a hit per second or so by workers on mechanical turk > > using it. > > > When I have no users, everything is fast. But in active use, I notice > > that web pages often load reay slow in my web browser, but the > > httpserver.log file reports only small times. > > > For instance, I just loaded a page that httpserver.log said took > > 0.20 seconds, but Firebug said took 54.21 seconds. That's a big > > difference. Any idea what's going on? I guess I'll have to try apache? -- You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web...@googlegroups.com. To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/web2py?hl=en.
[web2py] Re: webserver slow, misreported
Actually it's handling about 5 requests per second, so there is def some concurrency. On Mar 26, 10:06 pm, Michael Toomim wrote: > I'm using web2py+rocket to serve jobs on mechanical turk. The server > probably gets a hit per second or so by workers on mechanical turk > using it. > > When I have no users, everything is fast. But in active use, I notice > that web pages often load reay slow in my web browser, but the > httpserver.log file reports only small times. > > For instance, I just loaded a page that httpserver.log said took > 0.20 seconds, but Firebug said took 54.21 seconds. That's a big > difference. Any idea what's going on? I guess I'll have to try apache? -- You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web...@googlegroups.com. To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/web2py?hl=en.
[web2py] webserver slow, misreported
I'm using web2py+rocket to serve jobs on mechanical turk. The server probably gets a hit per second or so by workers on mechanical turk using it. When I have no users, everything is fast. But in active use, I notice that web pages often load reay slow in my web browser, but the httpserver.log file reports only small times. For instance, I just loaded a page that httpserver.log said took 0.20 seconds, but Firebug said took 54.21 seconds. That's a big difference. Any idea what's going on? I guess I'll have to try apache? -- You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web...@googlegroups.com. To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/web2py?hl=en.
[web2py] create index on postgres
I can't create an index on postgresql using executesql. Here's what
happens:
>> db.executesql('create index bq_index on bonus_queue (hitid);')
...but the index does not show up in psql. It does not return
anything. It seems like the command might be blocking psql, because if
I run another index command on psql:
=# create index bq_index2 on bonus_queue (reason);
...it will block (not return) until I exit the web2py process that I
ran the executesql command from. The postgresql docs says that this
command will lock the table from writes until it completes.
BUT if I just run the command from psql without trying the
db.executesql(), it does what it should, returning immediately with
the message "CREATE INDEX" and shows me an index in psql:
=# \d bonus_queue
...
Indexes:
"bq_index2" btree (reason)
Any idea how to get db.executesql() to work?
--
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To post to this group, send email to web...@googlegroups.com.
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en.
[web2py] Re: no more cherrypy wsgiserver
Did you do anything special to use apachebench on the cherrypy
server? When I run "ab http://localhost/init/"; I get a
"apr_socket_recv: Connection refused (111)" error from apachebench.
If I do the same command when running the latest hg tip of web2py
(with rocket), the benchmark works.
I'm trying to see if rocket will speed up my website.
On Mar 12, 9:13 am, Timothy Farrell wrote:
> The benchmarks are in. As you can see from the attached PDF, there is a
> strong case for Rocket.
>
> How I conducted these benchmarks:
>
> CPU: Athlon 4050e 2.1 GHz
> RAM: 3GB
> OS: Windows 7 Ultimate
> Python 2.6.1
> Rocket 0.3.1
> Cherrypy 3.1.2
>
> I used ApacheBench to run the numbers you see.
>
> The wsgi app used was as basic as it gets:
>
> def test_app(env, start_response):
> start_response('200 OK',
> [('Content-Type', 'text/plain')])
> return ["True"]
>
> Apache (and mod_wsgi) were not particularly tuned but were included to
> show generally where it would end up on scales. Don't take this as a
> definitive look at Apache or mod_wsgi's performance (back you
> nginx/cherokee/lighty trolls! ;-). This is about a server that can be
> included in web2py.
>
> You'll notice some blank entries in the numbers...here's why:
>
> My original intervals were 1,2,5,10,25,50,100,250,500,1000. However, I
> added in 6,7,8 after seeing Cherrypy's performance hit a wall. I wanted
> to show where that happened. I didn't see it necessary to include
> Rocket or mod_wsgi in those iterations since they saw no such wall.
> mod_wsgi does not include numbers for 500 or 1000 concurrent connections
> because at that point Apache started rejecting connections. This would
> not be an issue on a properly configured Apache. Once again, the main
> comparison here is between Rocket and Cherrypy's wsgiserver.
>
> If you would like the full spreadsheet, email me privately.
>
> -tim
>
> On 3/11/2010 10:19 AM, Timothy Farrell wrote:
>
>
>
> > The code has changed since version 0.1, Let me re-run some
> > benchmarks. I'll have time to tomorrow.
>
> > For those curious, the basic difference is that Rocket handles a few
> > concurrent connections as fast as wsgiserver and many concurrent
> > connections much much faster. It's also smaller, with cleaner code.
>
> > -tim
>
> > On 3/11/2010 10:08 AM, mdipierro wrote:
> >> We moved from cherrypy wsgiserver to Rocket, by Timothy Farrell.
>
> >> I included an older version, need to include the latest one.
>
> >> It needs to be tested but let's wait I post the latest version before
> >> we do so.
>
> >> Why?
> >> @Tim, you made a very convincing case to me some time ago. Can you
> >> share your benchmark with the rest of the users?
>
> >> Massimo
>
> Rocket Benchmarks.pdf
> 10KViewDownload
--
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To post to this group, send email to web...@googlegroups.com.
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en.
[web2py] Helper functions to "get one" from database
Hi guys, I've found the following functions to be commonly useful in practice. Has anyone else written anything similar? Is there a better idiom here, or better names or interfaces for these? def get_one(query): result = db(query).select() assert len(result) <= 1, "GAH get_one called when there's MORE than one!" return result[0] if len(result) == 1 else None def get_or_make_one(query, table, default_values): result = get_one(query) if not result: table.insert(**default_values) return get_one(query) -- You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web...@googlegroups.com. To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/web2py?hl=en.
[web2py] Re: no more cherrypy wsgiserver
I'm so excited! I was about to try moving to rocket myself, because I need the scalability and it is very useful for my app to run without apache. THANKS GUYS! On Mar 11, 8:08 am, mdipierro wrote: > We moved from cherrypy wsgiserver toRocket, by Timothy Farrell. > > I included an older version, need to include the latest one. > > It needs to be tested but let's wait I post the latest version before > we do so. > > Why? > @Tim, you made a very convincing case to me some time ago. Can you > share your benchmark with the rest of the users? > > Massimo -- You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web...@googlegroups.com. To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/web2py?hl=en.
[web2py] How does None and Null work in dal?
How is a database "Null" entry represented in python when using the DAL? How can you query for null rows? How can you set them? Is this the same as None? And if you create a database row without setting a value for a column, this is set to Null=None, right? Thank you! -- You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web...@googlegroups.com. To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/web2py?hl=en.
[web2py] Re: persistent background process
Oh yeah, and that database lock line only works with postgresql. You
might try file locking if you're not using it.
On Feb 26, 2:05 pm, Michael Toomim wrote:
> I got this to work by using @reboot cron. It works great, and super
> simple. Cross-platform, needs no external daemons.
>
> The trick was that web2py wasn't shutting down when my script was
> running, because I had a try-catch that was catching the
> KeyboardInterrupt exception. So here's what I did:
>
> In cron/crontab:
>
> �...@reboot toomim *applications/init/cron/background_work.py
>
> (Thanks to achipa and massimo for fixing @reboot recently!)
> In cron/background_work.py:
>
> import time, sys, commands, datetime
>
> # Lock the database so we don't accidentally run two background
> # processes at the same time, e.g. if we run web2py twice by
> accident
> if not db.executesql('select pg_try_advisory_lock(1);')[0][0]:
> logging.debug('CRON: FAILED to get background process lock')
> raise Exception('Somebody else is running background
> processes')
> logging.debug('CRON: got lock')
>
> count = 0
> while True:
> count += 1
> logging.debug('Sleeping for the %dth time' % count)
> time.sleep(10)
> try:
> logging.debug('Processing launch queue')
> process_launch_queue()
> logging.debug('Processing bonus queue')
> process_bonus_queue()
> except KeyboardInterrupt as e:
> logging.debug('Someone killed us.')
> raise
> except:
> logging.debug("Error in background process: %s" %
> sys.exc_info()[0])
>
> Note that I explicitly catch the KeyboardInterrupt and re-raise it so
> the script dies. If you kill web2py with some other type of exception
> you might need to put it in there too.
>
> The other trick is that print statements and exceptions aren't
> displayed at the console because this is run as a separate python
> process. So you need to use logging.debug from log.py. I don't
> remember where I got that file...
>
> On Jan 3, 1:57 pm, Richard wrote:
>
>
>
> > I was doing that but sometimes the process crashed or was killed by my
> > hosting provider, so I need a way to continue it automatically. Also I
> > may want to execute a number of background tasks in parallel. And it
> > uses extra resources that aren't needed, which is a problem on my
> > hosting where I am reaching the limits.
>
> > Michael, are you looking for a particular platform solution or cross
> > platform?
>
> > On Jan 4, 5:23 am, mdipierro wrote:
>
> > > I normally just start a separate web2py process.
>
> > > python web2py.py -S app -M -S thescript
>
> > > On Jan 3, 11:58 am, Michael Toomim wrote:
>
> > > > Great! I am also trying to implement this. Richard and Auden, have
> > > > you gotten anything working yet? How about we share solutions?
>
> > > > My current difficulty is figuring out where to create thepersistent
> > > > background thread.
> > > > - If I spawn it in a controller or model file will it be limited to
> > > > 10 seconds?
> > > > - If I run it in a periodic cron there will be a delay and I will
> > > > need to handle the case of multiple threads running simultaneously
> > > > - If I use @reboot cron, it prevents web2py from shutting down (it
> > > > seems to wait for my thread to quit, which does not know to quit)
>
> > > > Any ideas on these problems? It would be great to have a general task
> > > > queue in web2py, like the one on GAE. Although I've never learned how
> > > > to use the GAE one, perhaps what we're making could be easier to use.
>
> > > > On Jan 3, 8:06 am, mdipierro wrote:
>
> > > > > This is what I would do:
>
> > > > > db.define_table('queue',Field('status'),Field('descr'))
>
> > > > > In actions
>
> > > > > db.queue.insert(status='PENDING',descr='whatever')
>
> > > > > In your own scrips that you can run via CRON or via background process
>
> > > > > while True:
> > > > > tasks =
> > > > > db(db.queue.status=='PENDING').select(limitby=(0,10)).first
> > > > > ()
> > > > > if task:
> > > > > task.update_record(status='PROCESSIN
[web2py] Re: persistent background process
I got this to work by using @reboot cron. It works great, and super
simple. Cross-platform, needs no external daemons.
The trick was that web2py wasn't shutting down when my script was
running, because I had a try-catch that was catching the
KeyboardInterrupt exception. So here's what I did:
In cron/crontab:
@reboot toomim *applications/init/cron/background_work.py
(Thanks to achipa and massimo for fixing @reboot recently!)
In cron/background_work.py:
import time, sys, commands, datetime
# Lock the database so we don't accidentally run two background
# processes at the same time, e.g. if we run web2py twice by
accident
if not db.executesql('select pg_try_advisory_lock(1);')[0][0]:
logging.debug('CRON: FAILED to get background process lock')
raise Exception('Somebody else is running background
processes')
logging.debug('CRON: got lock')
count = 0
while True:
count += 1
logging.debug('Sleeping for the %dth time' % count)
time.sleep(10)
try:
logging.debug('Processing launch queue')
process_launch_queue()
logging.debug('Processing bonus queue')
process_bonus_queue()
except KeyboardInterrupt as e:
logging.debug('Someone killed us.')
raise
except:
logging.debug("Error in background process: %s" %
sys.exc_info()[0])
Note that I explicitly catch the KeyboardInterrupt and re-raise it so
the script dies. If you kill web2py with some other type of exception
you might need to put it in there too.
The other trick is that print statements and exceptions aren't
displayed at the console because this is run as a separate python
process. So you need to use logging.debug from log.py. I don't
remember where I got that file...
On Jan 3, 1:57 pm, Richard wrote:
> I was doing that but sometimes the process crashed or was killed by my
> hosting provider, so I need a way to continue it automatically. Also I
> may want to execute a number of background tasks in parallel. And it
> uses extra resources that aren't needed, which is a problem on my
> hosting where I am reaching the limits.
>
> Michael, are you looking for a particular platform solution or cross
> platform?
>
> On Jan 4, 5:23 am, mdipierro wrote:
>
>
>
> > I normally just start a separate web2py process.
>
> > python web2py.py -S app -M -S thescript
>
> > On Jan 3, 11:58 am, Michael Toomim wrote:
>
> > > Great! I am also trying to implement this. Richard and Auden, have
> > > you gotten anything working yet? How about we share solutions?
>
> > > My current difficulty is figuring out where to create thepersistent
> > > background thread.
> > > - If I spawn it in a controller or model file will it be limited to
> > > 10 seconds?
> > > - If I run it in a periodic cron there will be a delay and I will
> > > need to handle the case of multiple threads running simultaneously
> > > - If I use @reboot cron, it prevents web2py from shutting down (it
> > > seems to wait for my thread to quit, which does not know to quit)
>
> > > Any ideas on these problems? It would be great to have a general task
> > > queue in web2py, like the one on GAE. Although I've never learned how
> > > to use the GAE one, perhaps what we're making could be easier to use.
>
> > > On Jan 3, 8:06 am, mdipierro wrote:
>
> > > > This is what I would do:
>
> > > > db.define_table('queue',Field('status'),Field('descr'))
>
> > > > In actions
>
> > > > db.queue.insert(status='PENDING',descr='whatever')
>
> > > > In your own scrips that you can run via CRON or via background process
>
> > > > while True:
> > > > tasks = db(db.queue.status=='PENDING').select(limitby=(0,10)).first
> > > > ()
> > > > if task:
> > > > task.update_record(status='PROCESSING')
> > > > db.commit()
> > > > try:
> > > > #process task
> > > > task.update_record(status='COMPLETED')
> > > > except:
> > > > task.update_record(status='FAILED')
> > > > db.commit()
> > > > else:
> > > > sleep(10) # retry in 10 seconds
>
> > > > On Jan 3, 7:28 am, Richard wrote:
>
> > > > > I haven't implemented this yet, but my plan on Linux to keep a
> > >
