[web2py] Re: DAL issues on production server (xen VM)
Issue resolved. The source of all trouble was that the Scheduler uses the same db connection as the task it executes. Thus output updates (print '!clear!%(progress)s' % progress) changes the connection status, randomly, hence the strange exceptions. Suggestion: explicitly mention this as a gotcha in the scheduler section of the w2p book. Cheers, Yassen -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[web2py] Re: web2py ala static-site-generator
How about generating most of the pages and serving them as static content (e.g. templates that have nothing to render, just plain HTML); then having some templates that really have stuff that needs to be rendered and processed (e.g. a contact form + a POST handler) ? -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[web2py] DAL issues on production server (xen VM)
Hello guys, Recently updated from w2p 2.2.1 to 2.8.2 as I was advised; now random errors on database inserts started to raise for stuff that worked on the old version (2.2.1). Tracebacks of the two errors: This is the most common one: File /home/www-data/web2py/gluon/scheduler.py, line 238, in executor result = dumps(_function(*args, **vars)) File applications/socialjack/modules/contactManager.py, line 97, in importSocialMediaContacts if db.contacts.insert(contact_name=contact_name, url=url, network=network, user_id=user_id, ring=4): File /home/www-data/web2py/gluon/dal.py, line 9118, in insert ret = self._db._adapter.insert(self, self._listify(fields)) File /home/www-data/web2py/gluon/dal.py, line 1347, in insert id = self.lastrowid(table) File /home/www-data/web2py/gluon/dal.py, line 2724, in lastrowid return int(self.cursor.fetchone()[0]) TypeError: 'NoneType' object has no attribute '__getitem__' And I just noticed one more (more rare): File /home/www-data/web2py/gluon/scheduler.py, line 238, in executor result = dumps(_function(*args, **vars)) File applications/socialjack/modules/contactManager.py, line 97, in importSocialMediaContacts if not db((db.contacts.url == url) (db.contacts.user_id == user_id)).select(): File /home/www-data/web2py/gluon/dal.py, line 10471, in select return adapter.select(self.query,fields,attributes) File /home/www-data/web2py/gluon/dal.py, line 1862, in select return self._select_aux(sql,fields,attributes) File /home/www-data/web2py/gluon/dal.py, line 1843, in _select_aux return processor(rows,fields,self._colnames,cacheable=cacheable) File /home/www-data/web2py/gluon/dal.py, line 2227, in parse value = row[j] IndexError: tuple index out of range These appear during long-running scheduler task that inserts records into a db table. The exact insert code is: if not db((db.contacts.url == url) (db.contacts.user_id == user_id)).select(): if db.contacts.insert(contact_name=contact_name, url=url, network=network, user_id=user_id, ring=4): import_count += 1 Any suggestions why this happens? The app runs on an ubuntu 12.04 xen VM; w2p version: 2.8.2-stable+timestamp.2013.12.14.20.02.42 mysql-server version: 5.5.29-0ubuntu0.12.04.2 Any help is very much appreciated, thanks! Yassen -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[web2py] Re: DAL issues on production server (xen VM)
On Monday, December 16, 2013 2:46:00 AM UTC+2, Massimo Di Pierro wrote: Do you user virtual fields? Are you doing migrations while the scheduler tasks are running. This is strange because it is a very fundamental piece of web2py code that has not changed in long time: Massimo, thanks for your reply! No migration is done at all. No virtual fields as well. What I forgot to tell is that when it used to work, it was NOT run as a scheduled task, it was run within a controller. I also find this very very strange -- the return int(self.cursor.fetchone()[0]) should always work once the record has been inserted. Also, It breaks at a random record -- it is different each time. Seems like the MySQL server gets confused somehow ... Can it be a memory problem? Or the VM system is not syncing properly? Just shooting in the dark. Any clue anyone? Thanks! Yassen -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[web2py] Scheduler-related database tables created on sqLite but NOT on MySQL
Hello all,
I follow a simple video-tutorial (http://vimeo.com/27478796); so I create a
model scheduler.py with this content:
def f():
t = time.ctime()
open('/tmp/tasks', 'w').write(t + '\n')
return f
from gluon.scheduler import Scheduler
Scheduler(db, dict(our_func=f))
After saving it, I go to the database administration of that same
application and can see db.scheduler_task, db.scheduler_run,
db.scheduler_worker links; when I click on db.scheduler_task, the form is
there and I can fill in the data, however, on saving, it spits this error:
Traceback (most recent call last):
File /home/www-data/web2py/gluon/restricted.py, line 217, in restricted
exec ccode in environment
File
/home/www-data/web2py-2.8.2/applications/socialjack/controllers/contacts.py
https://apps.socialjack.com/admin/default/edit/socialjack/controllers/contacts.py,
line 236, in module
File /home/www-data/web2py/gluon/globals.py, line 372, in lambda
self._caller = lambda f: f()
File /home/www-data/web2py/gluon/tools.py, line 3239, in f
return action(*a, **b)
File
/home/www-data/web2py-2.8.2/applications/socialjack/controllers/contacts.py
https://apps.socialjack.com/admin/default/edit/socialjack/controllers/contacts.py,
line 54, in importcontacts
scheduler.queue_task(testfunc, pargs=[request.vars.sna], kwargs=task_kwargs)
File /home/www-data/web2py/gluon/scheduler.py, line 983, in queue_task
**kwargs)
File /home/www-data/web2py/gluon/dal.py, line 9114, in validate_and_insert
value,error = self[key].validate(value)
File /home/www-data/web2py/gluon/dal.py, line 10036, in validate
(value, error) = validator(value)
File /home/www-data/web2py/gluon/validators.py, line 668, in __call__
row = subset.select(table._id, field, limitby=(0, 1),
orderby_on_limitby=False).first()
File /home/www-data/web2py/gluon/dal.py, line 10450, in select
return adapter.select(self.query,fields,attributes)
File /home/www-data/web2py/gluon/dal.py, line 1861, in select
return self._select_aux(sql,fields,attributes)
File /home/www-data/web2py/gluon/dal.py, line 1826, in _select_aux
self.execute(sql)
File /home/www-data/web2py/gluon/dal.py, line 1948, in execute
return self.log_execute(*a, **b)
File /home/www-data/web2py/gluon/dal.py, line 1942, in log_execute
ret = self.cursor.execute(command, *a[1:], **b)
File /usr/lib/python2.7/dist-packages/MySQLdb/cursors.py, line 174, in
execute
self.errorhandler(self, exc, value)
File /usr/lib/python2.7/dist-packages/MySQLdb/connections.py, line 36, in
defaulterrorhandler
raise errorclass, errorvalue
ProgrammingError: (1146, Table 'socialjack.scheduler_task' doesn't exist)
If I do the same for the welcome application, it all works as expected.
Can anyone tell what is wrong here? The tables are indeed non-existent when
inspecting the MySQL database. This is web2py 2.8.2., updated from
mercurial repo, VERSION: 2.8.2-stable+timestamp.2013.12.04.19.34.45.
Thanks,
YD
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
web2py-users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
[web2py] Re: Spawning a thread or a process for a long-running task [urgent!]
Niphlod, yes, that was it !! It was not in the db.py but there was a 0.py with 'settings.migrate = True' (this is not my app, I was just asked to hack there to fix an issue). An easy workaround might be to also set fake_migrations to True (fake_migrate_all=True in DAL() creation) and copy and execute the proper statements from databases/sql.log. That's what I did because just turning that settings.migrate into True brought me issues with tables already existing. I was digging in this half a day. An idea: showing the migration status of the app somewhere in the header of the database admin page. Huge thanks again for your great help! Yassen On Thursday, December 5, 2013 11:53:43 PM UTC+2, Niphlod wrote: check for db = DAL(migrate_enabled=False) either this prevented the creation of the table, or some another migrate-related glitch. To create scheduler's tables, you just need to do (as you did) Scheduler(db, ...) -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[web2py] Re: Spawning a thread or a process for a long-running task [urgent!]
Leonel, thanks so much for the advice! On Wednesday, December 4, 2013 10:28:43 PM UTC+2, Leonel Câmara wrote: Instead of rerunning the task with period, schedule new tasks every time you need to go get the user's friends and have many scheduler workers if there's a chance you will have to do many of these in parallel. Of course, this seems great for my use case! You should be able to lower the interval as much as you need, you could even reduce the scheduler heartbeat from the default 3 seconds. Can you elaborate here? Is this a config setting somewhere? (Forgive my ignorance.) Y. -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[web2py] Re: Spawning a thread or a process for a long-running task [urgent!]
On Wednesday, December 4, 2013 10:28:43 PM UTC+2, Leonel Câmara wrote: Instead of rerunning the task with period, schedule new tasks every time you need to go get the user's friends I guess I have to create a record into the scheduler_task table, with some reasonable values, and then the first available worker will pick that up, correct? -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[web2py] Re: Spawning a thread or a process for a long-running task [urgent!]
On Thursday, December 5, 2013 8:40:28 AM UTC+2, Yassen D. wrote:
On Wednesday, December 4, 2013 10:28:43 PM UTC+2, Leonel Câmara wrote:
Instead of rerunning the task with period, schedule new tasks every time
you need to go get the user's friends
I guess I have to create a record into the scheduler_task table, with
some reasonable values, and then the first available worker will pick that
up, correct?
What I do currently: create a scheduler in a dedicated model:
from gluon.scheduler import Scheduler
from jobs import testfunc
scheduler = Scheduler(db, dict(testfunc_task=testfunc))
Then in a controller, I do:
task_kwargs = { 'immediate': True, 'task_name': 'ImportContacts-' +
str(time.time())[:-4], }
from jobs import testfunc
scheduler.queue_task(testfunc, pargs=[request.vars.sna],
kwargs=task_kwargs)
and I get an exception saying that table socialjack.scheduler_task does not
exist (traceback below). But the table IS there, I can see it in the db
admin interface and on the MySQL console ... Please help! Thanks!
Traceback (most recent call last):
File /home/www-data/web2py/gluon/restricted.py, line 217, in restricted
exec ccode in environment
File
/home/www-data/web2py-2.8.2/applications/socialjack/controllers/contacts.py
https://apps.socialjack.com/admin/default/edit/socialjack/controllers/contacts.py,
line 236, in module
File /home/www-data/web2py/gluon/globals.py, line 372, in lambda
self._caller = lambda f: f()
File /home/www-data/web2py/gluon/tools.py, line 3239, in f
return action(*a, **b)
File
/home/www-data/web2py-2.8.2/applications/socialjack/controllers/contacts.py
https://apps.socialjack.com/admin/default/edit/socialjack/controllers/contacts.py,
line 54, in importcontacts
scheduler.queue_task(testfunc, pargs=[request.vars.sna], kwargs=task_kwargs)
File /home/www-data/web2py/gluon/scheduler.py, line 983, in queue_task
**kwargs)
File /home/www-data/web2py/gluon/dal.py, line 9114, in validate_and_insert
value,error = self[key].validate(value)
File /home/www-data/web2py/gluon/dal.py, line 10036, in validate
(value, error) = validator(value)
File /home/www-data/web2py/gluon/validators.py, line 668, in __call__
row = subset.select(table._id, field, limitby=(0, 1),
orderby_on_limitby=False).first()
File /home/www-data/web2py/gluon/dal.py, line 10450, in select
return adapter.select(self.query,fields,attributes)
File /home/www-data/web2py/gluon/dal.py, line 1861, in select
return self._select_aux(sql,fields,attributes)
File /home/www-data/web2py/gluon/dal.py, line 1826, in _select_aux
self.execute(sql)
File /home/www-data/web2py/gluon/dal.py, line 1948, in execute
return self.log_execute(*a, **b)
File /home/www-data/web2py/gluon/dal.py, line 1942, in log_execute
ret = self.cursor.execute(command, *a[1:], **b)
File /usr/lib/python2.7/dist-packages/MySQLdb/cursors.py, line 174, in
execute
self.errorhandler(self, exc, value)
File /usr/lib/python2.7/dist-packages/MySQLdb/connections.py, line 36, in
defaulterrorhandler
raise errorclass, errorvalue
ProgrammingError: (1146, Table 'socialjack.scheduler_task' doesn't exist)
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
web2py-users group.
To unsubscribe from this group and stop receiving emails from it, send an email
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
[web2py] Re: Spawning a thread or a process for a long-running task [urgent!]
On Thursday, December 5, 2013 9:33:15 AM UTC+2, Yassen D. wrote: ... I get an exception saying that table socialjack.scheduler_task does not exist (traceback below). But the table IS there, I can see it in the db admin interface and on the MySQL console ... Not really, the table is shown in the database interface but is NOT present on the SQL server, unfortunately. What should I do to create it? -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[web2py] Re: Spawning a thread or a process for a long-running task [urgent!]
On Monday, December 2, 2013 6:20:54 PM UTC+2, Massimo Di Pierro wrote: The scheduler should do what you need. You can start many workers to manage the load Thanks, Massimo! I upgraded to 2.8.2 with minor issues (MySQL date fields seem to have been mapped to strings and now to datetime objects). I now experiment with the scheduler but the lowest interval I get between task re-runs is about 15 seconds which is not ideal for this use case. Is it possible to lower it even further, to about 5 seconds? Thanks again! YD -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[web2py] Re: Spawning a thread or a process for a long-running task [urgent!]
Massimo, thanks a lot for your help! I will upgrade right away then. (I have very limited time, so I was afraid to break things.) My comments: (a) We need to get the import process run instantly on user UI action (link click). Couple of seconds later is okay, but half a minute later is not okay. (b) We may have several users doing that at the same time, thus n such tasks need to be able to run in parallel. Will the scheduler allow me to implement an acceptable solution? Thanks! YD On Monday, December 2, 2013 9:41:40 AM UTC+2, Massimo Di Pierro wrote: The first recommendation would be upgrade to 2.8.2 (nothing should break) and use the scheduler. Do not use threads because they may still be killed by the web server. You can run a simple background process that every minute checks if there is something to do. Then use an auxiliary table to queue the fb credentials of those users that need to be processed. -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[web2py] Spawning a thread or a process for a long-running task [urgent!]
Hello guys, I am trying to help with a web2py application (web2py v. 2.2.1, ancient, yes) that needs an urgent patch as follows: There is a controller function that fetches all facebook friends of the user and stores them in the db (mysql). Currently this is done plainly in that function, during the request-response cycle, so the browser waits for the contacts import to happen and then gets the response. For people with many friends (thousands) this takes minutes, so uwsgi kills the python process. I tried to resolve this in two different ways, neither worked so far: (1) the controller function spawns a separate thread for the import and then let it run and return response. This kinda worked but importing does strange pauses every couple of seconds, (pausing for a random period, typically 2-6 seconds, sometimes longer). Don't know why that happens, but it makes the whole process unbearably slow. (2) tried to spawn a separate process using multiprocessing, but import multiprocessing fails. (3) Using the scheduler is another option, but I hesitate to go for it because of the ancient version of web2py. (4) Another approach that comes to my mind is to somehow send the response but keep the thread of that response and do the job in that thread before letting it go. What would you recommend me to do? This is an urgent issue, so any prompt help is much appreciated! Thanks! Yassen -- Resources: - http://web2py.com - http://web2py.com/book (Documentation) - http://github.com/web2py/web2py (Source code) - https://code.google.com/p/web2py/issues/list (Report Issues) --- You received this message because you are subscribed to the Google Groups web2py-users group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
