[web2py] Import error for guppy
My app uses web2py as a background task (as described in Ch. 4 of the
web2py docs). I have a slow memory leak in the background task. I'm trying
to run guppy periodically to find it but I am getting a error when I try to
use it.
The relevant part of my code looks like this:
import guppy
heapy = guppy.hpy()
while True:
...
logging.info ("%s" % heapy.heap())
When the last statement executes, I get an ImportError exception. The end
of the stack trace looks like this:
File "/Users/davidp/dev/python/ssk/gluon/custom_import.py", line 81, in
custom_importer
raise ImportError, 'Cannot import module %s' % str(e)
ImportError: Cannot import module 'guppy'
This code executes fine in the interpreter so I'm guessing the problem has
to do with web2py's import mechanism.
Has anyone else been successful using guppy/heapy with web2py? I see that
this isn't the first time this issue has come up but I don't see any
mention of a solution.
Any help would be gratefully received.
--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
[web2py] Re: Manual upload to database fails to write blob
So I thought the problem might go away if I changed the definition of the
table to:
thumb_field = Field ("thumb", "upload", uploadfield = "thumb_data")
db.define_table ("images",
Field ("image", "upload", uploadfield = "image_data",
requires = IS_IMAGE (extensions = ('bmp', 'gif', 'jpeg', 'jpg', 'png'),
maxsize = (3048, 2400))),
Field ("image_data", "blob"),
thumb_field, #Field ("thumb", "upload", uploadfield = "thumb_data"),
Field ("thumb_data", "blob"),
)
I thought that that way store function code would see that the type of the
field was uploadfield, but that failed, too, showing mostly that I don't
understand the internals of how Field.store and Field.retrieve work for
manually uploaded files that are stored in the database.
Is there something wrong with my original images table definition?
Would I be better served to simply store the image in a blob field and not
use the store/retrieve functions?
Any help would gratefully received.
David
On Friday, October 26, 2012 1:27:09 PM UTC-5, David Phillips wrote:
>
> I am attempting to create and store a thumbnail image in a MySQL database.
> The source image comes from the same record and has already been stored in
> the database. When my code executes, the thumbnail name gets written to the
> "thumb" field, but the "thumb_data" field is null.
>
> My table is defined like this:
>
> db.define_table ("images",
> Field ("image", "upload", uploadfield = "image_data",
> requires = IS_IMAGE (extensions = ('bmp', 'gif', 'jpeg', 'jpg', 'png'),
> maxsize = (3048, 2400))),
> Field ("image_data", "blob"),
> Field ("thumb", "upload", uploadfield = "thumb_data"),
> Field ("thumb_data", "blob"),
> auth.signature,
> )
>
> My code does this:
>
> from cStringIO import StringIO
> from PIL import Image
>
> # Get the image
> images_rec = db (db.images.id == image_id).select().first()
> filename, stream = db.images.image.retrieve (images_rec.image)
>
> # Open the image in PIL and make a thumnail
> thumb_PIL = Image.open (stream)
> thumb_PIL.thumbnail ((thumb_width, thumb_height), Image.ANTIALIAS)
>
> # Create a file-like object and save the thumbnail to it.
> thumb_file = StringIO()
> thumb_PIL.save (thumb_file, "jpeg")
>
> # Store the thumbnail
> thumb_file.seek (0)
> upload_image_rec.update_record (thumb = db.images.thumb.store (thumb_file,
> filename))
>
> I've traced through the store function code in gluon/dal.py and it looks
> to me like the problem is here (line 8461 in version 2.0.8):
>
> if isinstance(self_uploadfield,Field):
> blob_uploadfield_name = self_uploadfield.uploadfield
> keys={self_uploadfield.name: newfilename,
> blob_uploadfield_name: file.read()}
> self_uploadfield.table.insert(**keys)
>
> self_uploadfield is of type "str" with a value of "thumb_data" and the
> test "isinstance(self_uploadfield,Field)" fails. But I've checked
> carefully that the definition of the thumb and thumb_data fields follow the
> examples in the manual.
>
> I am confident that the image field is correct because I can retrieve and
> display the image from it. I also believe that the thumbnail is being
> created properly. At least, thumb_file.getvalue() returns a sensible
> looking block of data.
>
> I am using web2py version 2.0.8 on MacOS 10.8. The database is hosted on
> Amazon RDS.
>
> Thanks for any help you can give me.
>
>
--
[web2py] Manual upload to database fails to write blob
I am attempting to create and store a thumbnail image in a MySQL database.
The source image comes from the same record and has already been stored in
the database. When my code executes, the thumbnail name gets written to the
"thumb" field, but the "thumb_data" field is null.
My table is defined like this:
db.define_table ("images",
Field ("image", "upload", uploadfield = "image_data",
requires = IS_IMAGE (extensions = ('bmp', 'gif', 'jpeg', 'jpg', 'png'),
maxsize = (3048, 2400))),
Field ("image_data", "blob"),
Field ("thumb", "upload", uploadfield = "thumb_data"),
Field ("thumb_data", "blob"),
auth.signature,
)
My code does this:
from cStringIO import StringIO
from PIL import Image
# Get the image
images_rec = db (db.images.id == image_id).select().first()
filename, stream = db.images.image.retrieve (images_rec.image)
# Open the image in PIL and make a thumnail
thumb_PIL = Image.open (stream)
thumb_PIL.thumbnail ((thumb_width, thumb_height), Image.ANTIALIAS)
# Create a file-like object and save the thumbnail to it.
thumb_file = StringIO()
thumb_PIL.save (thumb_file, "jpeg")
# Store the thumbnail
thumb_file.seek (0)
upload_image_rec.update_record (thumb = db.images.thumb.store (thumb_file,
filename))
I've traced through the store function code in gluon/dal.py and it looks to
me like the problem is here (line 8461 in version 2.0.8):
if isinstance(self_uploadfield,Field):
blob_uploadfield_name = self_uploadfield.uploadfield
keys={self_uploadfield.name: newfilename,
blob_uploadfield_name: file.read()}
self_uploadfield.table.insert(**keys)
self_uploadfield is of type "str" with a value of "thumb_data" and the test
"isinstance(self_uploadfield,Field)" fails. But I've checked carefully
that the definition of the thumb and thumb_data fields follow the examples
in the manual.
I am confident that the image field is correct because I can retrieve and
display the image from it. I also believe that the thumbnail is being
created properly. At least, thumb_file.getvalue() returns a sensible
looking block of data.
I am using web2py version 2.0.8 on MacOS 10.8. The database is hosted on
Amazon RDS.
Thanks for any help you can give me.
--
Re: [web2py] HTTPS admin access has stopped working
Thanks for checking, Massimo. I'll start looking into the web server.
David
On Sep 18, 2012, at 1:30 PM, Massimo Di Pierro
wrote:
> This confirms my impression. This is not a web2py 2.0.x issue. Something else
> has changed. env.wsgi_url_scheme and env.https are from the WSGI environment.
> HTTPS is not defined and WSGI_URL_SCHEME is http.
>
> I am suing web2py 2.0.9 with apache and I get
>
> request.env.wsgi_url_scheme: https
> request.env.https: 1
>
> Massimo
>
> On Tuesday, 18 September 2012 10:50:55 UTC-5, David Phillips wrote:
> Sure.
>
> [Tue Sep 18 15:49:25 2012] [error] WARNING:root:request.env.wsgi_url_scheme:
> http
> [Tue Sep 18 15:49:25 2012] [error] WARNING:root:request.env.https: None
>
>
> On Sep 18, 2012, at 10:40 AM, Massimo Di Pierro wrote:
>
>> Can you please print the values of
>>
>> request.env.wsgi_url_scheme and request.env.https
>>
>> It will help me understand what is going on. I do not think we changed the
>> request.is_https behavior. web2py my not be able to detect https if behind a
>> proxy.
>>
>> Massimo
>>
>> On Tuesday, 18 September 2012 10:27:34 UTC-5, David Phillips wrote:
>> Thanks to everyone for their replies. Bruno, that is a neat trick.
>>
>> Unfortunately, it didn't solve my problem. I generated a new password and
>> restarted the apache server, but I am seeing the same behavior --
>> request.is_https is returning false even though I am using https and I
>> cannot use the admin interface.
>>
>> This is harder to solve because https access works locally. It's only on the
>> production apache server (on Elastic Beanstalk) that it fails.
>>
>> Can anyone suggest a strategy to find the cause? A quick survey of the gluon
>> code didn't turn up any obvious place to look.
>>
>> Thanks.
>>
>> David
>>
>>
>>
>> On Monday, September 17, 2012 4:18:34 PM UTC-5, Richard wrote:
>> This is nicer way of restoring it :)
>>
>> Thanks Bruno!
>>
>> Richard
>>
>> On Mon, Sep 17, 2012 at 3:50 PM, Bruno Rocha wrote:
>> assuming your webserver user is www-data and desired password 123456
>>
>> cd path/to/web2py
>>
>> sudo -u www-data python -c "from gluon.main import save_password;
>> save_password('123456',443)"
>>
>> --
>>
>>
>>
>>
>>
>> --
>>
>>
>>
>
>
> --
>
>
>
--
Re: [web2py] HTTPS admin access has stopped working
Sure.
[Tue Sep 18 15:49:25 2012] [error] WARNING:root:request.env.wsgi_url_scheme:
http
[Tue Sep 18 15:49:25 2012] [error] WARNING:root:request.env.https: None
On Sep 18, 2012, at 10:40 AM, Massimo Di Pierro
wrote:
> Can you please print the values of
>
> request.env.wsgi_url_scheme and request.env.https
>
> It will help me understand what is going on. I do not think we changed the
> request.is_https behavior. web2py my not be able to detect https if behind a
> proxy.
>
> Massimo
>
> On Tuesday, 18 September 2012 10:27:34 UTC-5, David Phillips wrote:
> Thanks to everyone for their replies. Bruno, that is a neat trick.
>
> Unfortunately, it didn't solve my problem. I generated a new password and
> restarted the apache server, but I am seeing the same behavior --
> request.is_https is returning false even though I am using https and I cannot
> use the admin interface.
>
> This is harder to solve because https access works locally. It's only on the
> production apache server (on Elastic Beanstalk) that it fails.
>
> Can anyone suggest a strategy to find the cause? A quick survey of the gluon
> code didn't turn up any obvious place to look.
>
> Thanks.
>
> David
>
>
>
> On Monday, September 17, 2012 4:18:34 PM UTC-5, Richard wrote:
> This is nicer way of restoring it :)
>
> Thanks Bruno!
>
> Richard
>
> On Mon, Sep 17, 2012 at 3:50 PM, Bruno Rocha wrote:
> assuming your webserver user is www-data and desired password 123456
>
> cd path/to/web2py
>
> sudo -u www-data python -c "from gluon.main import save_password;
> save_password('123456',443)"
>
> --
>
>
>
>
>
> --
>
>
>
--
Re: [web2py] HTTPS admin access has stopped working
Thanks to everyone for their replies. Bruno, that is a neat trick.
Unfortunately, it didn't solve my problem. I generated a new password and
restarted the apache server, but I am seeing the same behavior --
request.is_https is returning false even though I am using https and I
cannot use the admin interface.
This is harder to solve because https access works locally. It's only on
the production apache server (on Elastic Beanstalk) that it fails.
Can anyone suggest a strategy to find the cause? A quick survey of the
gluon code didn't turn up any obvious place to look.
Thanks.
David
On Monday, September 17, 2012 4:18:34 PM UTC-5, Richard wrote:
>
> This is nicer way of restoring it :)
>
> Thanks Bruno!
>
> Richard
>
> On Mon, Sep 17, 2012 at 3:50 PM, Bruno Rocha
> > wrote:
>
>> assuming your webserver user is www-data and desired password 123456
>>
>> cd path/to/web2py
>>
>> sudo -u www-data python -c "from gluon.main import save_password;
>> save_password('123456',443)"
>>
>> --
>>
>>
>>
>>
>
>
--
[web2py] HTTPS admin access has stopped working
I had a web2py 1.99.7 app running on Elastic Beanstalk. I was able to use the admin interface over https. I then upgraded web2py to 2.0.8 and uploaded it to EB. Now I am getting this message when I try to access the admin interface over HTTPS: ATTENTION: Login requires a secure (HTTPS) connection or running on localhost. Another change I notice is that request.is_https no longer seems to be true. I see that the format of the parameters_*.py file has changed. Is there any reason why I can no longer use the admin interface? Thanks. --
Re: [web2py] MySQL DB reads have stopped working for one table in a background process
I think you have hit on the issue. I added a db.commit() call before I
attempted to read from the table and the read was successful.
Thanks so much.
David
On Jul 24, 2012, at 11:16 PM, Massimo Di Pierro wrote:
> First of all we need to establish there is a problem and this is not the
> expected behavior. I am not sure.
>
> MySQL has 4 different isolation modes and defaults to:
>
> http://dev.mysql.com/doc/refman/5.1/en/set-transaction.html#isolevel_repeatable-read
>
> "All consistent reads within the same transaction read the snapshot
> established by the first read."
>
> I think this means that your background process does not see new updated
> records unless it starts a new transaction. You can check it. Try adding some
> db.commit() to the background process, even if it does not write data.
>
> If this is the problem, you may also be able to change the isolation level
> with SET TRANSACTION.
>
> Massimo
>
>
>
> On Tuesday, 24 July 2012 22:26:26 UTC-5, David Phillips wrote:
> Hello, Massimo.
>
>> What changed? Did you upgrade? What web2py version?
>
> I removed a task in the background process that was periodically calling
> db.commit. In its place, I started using memcache where I had been writing to
> the db.
>
> I haven't upgraded. I've been using 1.99.7 all along.
>
>> When you say the select does not work anymore, dwhat do you mean? Does it
>> lock or do you get a traceback?
>
> I add an account to the account table with account_id == 1 and id == 1
> using appadmin on the web server.
>
> Then I go to the background process and execute these commands both in my
> code and using the debugger:
>
> rows = db (db.account.account_id == 1).select().first()
>
> returns None.
>
> count = db (db.account.id > 0).count()
>
> return zero.
>
> db.executesql("select account_id from account")
>
> returns None.
>
> No error messages, exceptions, lock-ups or the like. select() simply fails to
> find the data in the table which I can see with appadmin and the mysql
> command-line client.
>
> Curiously, if I stop the background process and restart it, I can read the
> account from the account table:
>
> db (db.account.id > 0).count()
>
> returns 1.
>
> It's not a matter of the committing the database after the initial write. I
> added a db.commit() call after writing the account to the db just to make
> sure. It made no difference.
>
>> Which database driver? People have reported problems with pymysql but not
>> with mysqldb.
>
> It's pymysql for both the development platform (Mac OS X 10.7) and the
> production platform (CentOS).
>
>
> Do you have any advice on how to diagnose my problem?
>
> Thanks,
> David
>
>
>
> On Jul 24, 2012, at 8:50 PM, Massimo Di Pierro wrote:
>
>> What changed? Did you upgrade? What web2py version? When you say the select
>> does not work anymore, dwhat do you mean? Does it lock or do you get a
>> traceback? Which database driver? People have reported problems with pymysql
>> but not with mysqldb.
>>
>> On Tuesday, 24 July 2012 19:42:29 UTC-5, David Phillips wrote:
>> On the eve of delivering a project to a client, I've come up against a
>> problem that has me stumped. select() statements on one of my mysql tables
>> have stopped working.
>>
>> My application is a web2py web server and a background process (also called
>> a homemade task queue in the web2py book). They share the database. The web
>> server writes to the table from within an HTTP request, and several seconds
>> later, I attempt to read the record in my background process.
>>
>> Up until yesterday, I didn't have any trouble reading from this or any of
>> the tables. And now, all the others work fine. I can still write and read
>> from the misbehaving table from the web server.
>>
>> I am at a loss. I'm not sure where to look to diagnose the problem. Any
>> pointers would be gratefully received.
>>
>>
>> --
>>
>>
>>
>
>
> --
>
>
>
--
Re: [web2py] Re: MySQL DB reads have stopped working for one table in a background process
Hello, Massimo.
> What changed? Did you upgrade? What web2py version?
I removed a task in the background process that was periodically calling
db.commit. In its place, I started using memcache where I had been writing to
the db.
I haven't upgraded. I've been using 1.99.7 all along.
> When you say the select does not work anymore, dwhat do you mean? Does it
> lock or do you get a traceback?
I add an account to the account table with account_id == 1 and id == 1
using appadmin on the web server.
Then I go to the background process and execute these commands both in my code
and using the debugger:
rows = db (db.account.account_id == 1).select().first()
returns None.
count = db (db.account.id > 0).count()
return zero.
db.executesql("select account_id from account")
returns None.
No error messages, exceptions, lock-ups or the like. select() simply fails to
find the data in the table which I can see with appadmin and the mysql
command-line client.
Curiously, if I stop the background process and restart it, I can read the
account from the account table:
db (db.account.id > 0).count()
returns 1.
It's not a matter of the committing the database after the initial write. I
added a db.commit() call after writing the account to the db just to make sure.
It made no difference.
> Which database driver? People have reported problems with pymysql but not
> with mysqldb.
It's pymysql for both the development platform (Mac OS X 10.7) and the
production platform (CentOS).
Do you have any advice on how to diagnose my problem?
Thanks,
David
On Jul 24, 2012, at 8:50 PM, Massimo Di Pierro wrote:
> What changed? Did you upgrade? What web2py version? When you say the select
> does not work anymore, dwhat do you mean? Does it lock or do you get a
> traceback? Which database driver? People have reported problems with pymysql
> but not with mysqldb.
>
> On Tuesday, 24 July 2012 19:42:29 UTC-5, David Phillips wrote:
> On the eve of delivering a project to a client, I've come up against a
> problem that has me stumped. select() statements on one of my mysql tables
> have stopped working.
>
> My application is a web2py web server and a background process (also called a
> homemade task queue in the web2py book). They share the database. The web
> server writes to the table from within an HTTP request, and several seconds
> later, I attempt to read the record in my background process.
>
> Up until yesterday, I didn't have any trouble reading from this or any of the
> tables. And now, all the others work fine. I can still write and read from
> the misbehaving table from the web server.
>
> I am at a loss. I'm not sure where to look to diagnose the problem. Any
> pointers would be gratefully received.
>
>
> --
>
>
>
--
[web2py] MySQL DB reads have stopped working for one table in a background process
On the eve of delivering a project to a client, I've come up against a problem that has me stumped. select() statements on one of my mysql tables have stopped working. My application is a web2py web server and a background process (also called a homemade task queue in the web2py book). They share the database. The web server writes to the table from within an HTTP request, and several seconds later, I attempt to read the record in my background process. Up until yesterday, I didn't have any trouble reading from this or any of the tables. And now, all the others work fine. I can still write and read from the misbehaving table from the web server. I am at a loss. I'm not sure where to look to diagnose the problem. Any pointers would be gratefully received. --
[web2py] Threads with homemade task queues
I've read hearty admonishments on this forum against creating threads in controller and model files, but what about the case of a background job launched as a separate process – like what chapter four of the book calls homemade task queues and chapter eight calls a background task? My application has a long-running job that make frequent, periodic http requests. After struggling with various approaches, I've finally got a prototype algorithm working efficiently and correctly that makes synchronous, parallel http request in a number of worker threads. I could probably run my code as a separate, non-web2py python script, but it would be convenient to be able to access the model. Do the reasons for not running creating threads inside controllers still apply in the case of a separate background process? Thanks. David
Re: [web2py] Reading web2py tickets on app engine
Thank you, howesc and especially Ricardo. That code snippet worked. I can
now see my tickets on app engine.
On Tuesday, April 24, 2012 8:41:33 PM UTC-5, Ricardo Pedroso wrote:
>
> On Tue, Apr 24, 2012 at 12:57 AM, David Phillips
> > wrote:
> > I don't know how to read the tickets web2py generates on app engine. On
> > devserver, the tickets are shown in their entirety, but on the production
> > server, the datastore viewer only shows the first 1000 or so characters.
> I
> > can't figure out a way to see the entire ticket.
> >
> > The database management (appadmin) page doesn't show the tickets table,
> > although it shows web2py_session_rage.
> >
> > So, I wrote this action in a controller:
> >
> > def download_ticket():
> >
> > id = request.args[0]
> >
> > rec = db (db.web2py_ticket_rage.id == id).select().first()
> >
> > return rec.ticket_data
> >
> >
> > where web2py_ticket_rage is the name of the table where the tickets are
> > stored according to the dashboard. But when I execute, this exception is
> > raised:
> >
> > KeyError: 'web2py_ticket_rage'
> >
>
> You need to db.define_table()
>
> > Fortunately, there is a lot of information in the app engine logs, but I
> > would still like to see the ticket. So, is therea way to view tickets on
> app
> > engine?
>
> I have this in the beginning of appadmin after the imports:
>
> if request.env.web2py_runtime_gae:
> import cPickle
>
> tablename = 'web2py_ticket_'+request.application
> db.define_table(
> tablename,
> Field('ticket_id', length=100),
> Field('ticket_data', 'text'),
> Field('created_datetime', 'datetime'),
> )
>
> def pre_widget(field, value, **attributes):
> return PRE(value)
>
> def traceback_ticket_widget(field, value, **attributes):
> return PRE(cPickle.loads(value)['traceback'],
>_style="color:#ddd;background-color:#000;padding:4px")
>
> db[tablename].ticket_id.widget = pre_widget
> db[tablename].ticket_data.widget = traceback_ticket_widget
> db[tablename].created_datetime.widget = pre_widget
>
>
> and it's working for me.
>
> The definition of the table you can copy from gluon/restricted.py
> around line 75.
>
> Note that if you try insert a new ticket through appadmin it will
> raise an exception,
> but I leave it this way as a way to generate a ticket if I need.
>
> Ricardo
>
>
[web2py] Reading web2py tickets on app engine
I don't know how to read the tickets web2py generates on app engine. On devserver, the tickets are shown in their entirety, but on the production server, the datastore viewer only shows the first 1000 or so characters. I can't figure out a way to see the entire ticket. The database management (appadmin) page doesn't show the tickets table, although it shows web2py_session_rage. So, I wrote this action in a controller: def download_ticket(): id = request.args[0] rec = db (db.web2py_ticket_rage.id == id).select().first() return rec.ticket_data where web2py_ticket_rage is the name of the table where the tickets are stored according to the dashboard. But when I execute, this exception is raised: KeyError: 'web2py_ticket_rage' Fortunately, there is a lot of information in the app engine logs, but I would still like to see the ticket. So, is therea way to view tickets on app engine?
[web2py] Viewing web2py tickets on Google's devserver
How do you view web2py tickets that are generated while running on Google's devserver? When I look at a ticket using the SDK console, it is encoded in a way that I don't recognize. Like this: (dp0 > S'output' > p1 > S" 'NoneType' object is not callable" > p2 > sS'layer' > p3 > S'/Users/davidp/dev/python/rage/applications/rage/controllers/default.py' > p4 > sS'code' > p5 > S'# -*- coding: utf-8 -*-\n'\n\n@auth.requires_login()\ndef > index():\n\tresponse.title = \'Archives\'\n\n\tresponse.files.append What is this format? Is there a viewer for looking at the tickets? I can puzzle out the contents, but it is inconvenient and sometimes time consuming.
[web2py] Curious appengine memcache exception
I am stumped by a problem I am having with memcache on App Engine. It mostly looks like a memcache issue but I am seeing something funny when I step through the code. I am getting an excpetion (TypeError: 'NoneType' object is not callable) when writing an instance to memcache. The instance comes from a simple class that should be pickleable, containing only strings, booleans, an int, and a datetime. The reason I am bringing this to the web2py board is that when I traced down the memcache call chain to a call to pickler.dump() and attempted to step into it, control jumped to code for web2py's _Web2pyImporter object in gluon/custom_import.py. Here is the relevant section of my pdb session. > /Users/davidp/dev/python/google/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/memcache/__init__.py(403)_do_pickle() -> pickler.dump(value) (Pdb) s --Call-- > /Users/davidp/dev/python/rage/gluon/custom_import.py(253)__call__() -> def __call__(self, name, globals=None, locals=None, I stepped through the _Web2pyImporter code, and eventually, when executing the following statement, (line 292-3), control jumped into gluaon/dal.py where it executed the __getattr__() method of the Reference class. return super(_Web2pyImporter, self).__call__(name, globals, locals, fromlist, level) The __getattr__ returns None (it was looking up something called '__getstate__') which causes something to throw an exception. I am, by this time, just scratching my head. Why is appengine's memcache code invoking gluon code? Why is class Reference involved? The object that memcache is pickling contains no DAL references. Why is "custom importing" needed at all? The 'name' parameter to the Web2pyImporter function __call__ has a value of datetime. There is a datetime field in the object I'm trying to write to memcache, but why does the code jump to custom_import.py when the call to pickler.dump() is called? Here is the memcache code that was being executed at the time: def _do_pickle(self, value): """Pickles a provided value.""" pickle_data = cStringIO.StringIO() pickler = self._pickler_factory(pickle_data, protocol=self._pickle_protocol) if self._persistent_id is not None: pickler.persistent_id = self._persistent_id pickler.dump(value)# line 403 return pickle_data.getvalue() Here is how the pdb showed the control flow as I stepped through _do_pickle(). (Pdb) n > /Users/davidp/dev/python/google/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/memcache/__init__.py(403)_do_pickle() -> pickler.dump(value) (Pdb) s --Call-- > /Users/davidp/dev/python/ragearchives/gluon/custom_import.py(253)__call__() -> def __call__(self, name, globals=None, locals=None, The stack trace from the exception is copied below. Why doesn't it show the call into pickler.dump or custom_import? I am most confused. All I want to do is to store my object in memcache and my client is getting impatient. Is this a web2py issue? App Engine's memcache? Pickle? Any help would be appreciated. File "applications/rage/modules/player.py", line 51, in __init__ memcache.set (key, self, Player.MEMCACHE_EXPIRATION) File "/Users/davidp/dev/python/google/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/memcache/__init__.py", line 778, in set namespace=namespace) File "/Users/davidp/dev/python/google/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/memcache/__init__.py", line 883, in _set_with_policy\ntime, \'\', namespace) File "/Users/davidp/dev/python/google/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/memcache/__init__.py", line 962, in _set_multi_async_with_policy stored_value, flags = _validate_encode_value(value, self._do_pickle) File "/Users/davidp/dev/python/google/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/memcache/__init__.py", line 227, in _validate_encode_value stored_value = do_pickle(value) File "/Users/davidp/dev/python/google/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/memcache/__init__.py", line 403, in _do_pickle pickler.dump(value) TypeError: 'NoneType' object is not callable
[web2py] Re: unit testing web2py and app engine
I should have mentioned one other thing. Before running my unit test,
I set the PYTHONPATH environment variable so that I could import the
Google app engine libraries.
export PYTHONPATH=~/dev/python/google/google_appengine_1.6.3_source/
google_appengine
On Mar 21, 11:14 am, David Phillips
wrote:
> Thank you, howesc. I'll take a close look at your file.
>
> I finally cobbled together a file to unit test an object in my project
> that uses the unittest library, Google's memcache and the testbed
> library, and a web2py model. I am hoping it will scale up to test
> controller actions as well as individual modules.
>
> Included below is the relevant portion of that file. Notice the part
> where I had to manually add a path to dev_appserver's yaml library. I
> had to do that to import the testbed module.
>
> Here is the command line to run my unit test.
>
> $ cd web2py
> $ python web2py.py -S myproj -M -R applications/myproj/test/
> test_account.py
>
> --
>
> import sys, unittest
>
> from google.appengine.api import memcache
>
> yaml_path = '~/google_appengine_1.6.3_source/google_appengine/lib/yaml/
> lib/'
> sys.path.append (yaml_path)
>
> from google.appengine.ext import testbed
>
> from account_list import Account_List
> from account import Account
> class Test_Account_List (unittest.TestCase):
> def setUp (self):
> self.testbed = testbed.Testbed()
> self.testbed.activate()
> self.testbed.init_memcache_stub()
>
> def test_account_list (self):
> al1 = Account_List (db)
> al1.clear_list (db) # To start from a known state
> al2 = Account_List (db)
> self.assertIs (al1, al2)
> active_list1 = memcache.get ('account_list')
> active_list2 = al2.active_accounts(db)
> self.assertEqual (active_list1, active_list2)
>
> def tearDown (self):
> self.testbed.deactivate()
>
> suite = unittest.TestSuite()
> suite.addTest (unittest.makeSuite (Test_Account))
> suite.addTest (unittest.makeSuite (Test_Account_List))
> unittest.TextTestRunner (verbosity = 2).run (suite)
>
> On Mar 21, 2:35 am, howesc wrote:
>
>
>
>
>
>
>
> > attached is a controller that i use, adapted from the testing tools in the
> > admin interface. note that i set a request parameter 'test_db' and in
> > models i detect that parameter and connect to a different namespace so that
> > my tests run in a controlled DB environment - remember that if you want to
> > setup some test data to start with. :)
>
> > i have not used this in a live GAE environment so i don't know how well it
> > works. i fear that extensive tests would run past the 60 second timeout
> > for web requests.
>
> > let me know if you have questions.
>
> > On Tuesday, March 20, 2012 1:54:03 PM UTC-7, David Phillips wrote:
>
> > > I am writing a web application using web2py for execution on app
> > > engine. I've deployed other web2py apps there but I've never used any
> > > app engine-specific utilities. In this project I want to use app
> > > engine's taskqueue and run it on a backend instance.
>
> > > I'm developing on app engine's dev_appserver which is working out
> > > okay, but but I'm not seeing how to do unit testing.
>
> > > Is this a problem that has been solved already?
>
> > tests.py
> > 6KViewDownload
[web2py] Re: unit testing web2py and app engine
Thank you, howesc. I'll take a close look at your file.
I finally cobbled together a file to unit test an object in my project
that uses the unittest library, Google's memcache and the testbed
library, and a web2py model. I am hoping it will scale up to test
controller actions as well as individual modules.
Included below is the relevant portion of that file. Notice the part
where I had to manually add a path to dev_appserver's yaml library. I
had to do that to import the testbed module.
Here is the command line to run my unit test.
$ cd web2py
$ python web2py.py -S myproj -M -R applications/myproj/test/
test_account.py
--
import sys, unittest
from google.appengine.api import memcache
yaml_path = '~/google_appengine_1.6.3_source/google_appengine/lib/yaml/
lib/'
sys.path.append (yaml_path)
from google.appengine.ext import testbed
from account_list import Account_List
from account import Account
class Test_Account_List (unittest.TestCase):
def setUp (self):
self.testbed = testbed.Testbed()
self.testbed.activate()
self.testbed.init_memcache_stub()
def test_account_list (self):
al1 = Account_List (db)
al1.clear_list (db)# To start from a known state
al2 = Account_List (db)
self.assertIs (al1, al2)
active_list1 = memcache.get ('account_list')
active_list2 = al2.active_accounts(db)
self.assertEqual (active_list1, active_list2)
def tearDown (self):
self.testbed.deactivate()
suite = unittest.TestSuite()
suite.addTest (unittest.makeSuite (Test_Account))
suite.addTest (unittest.makeSuite (Test_Account_List))
unittest.TextTestRunner (verbosity = 2).run (suite)
On Mar 21, 2:35 am, howesc wrote:
> attached is a controller that i use, adapted from the testing tools in the
> admin interface. note that i set a request parameter 'test_db' and in
> models i detect that parameter and connect to a different namespace so that
> my tests run in a controlled DB environment - remember that if you want to
> setup some test data to start with. :)
>
> i have not used this in a live GAE environment so i don't know how well it
> works. i fear that extensive tests would run past the 60 second timeout
> for web requests.
>
> let me know if you have questions.
>
>
>
>
>
>
>
> On Tuesday, March 20, 2012 1:54:03 PM UTC-7, David Phillips wrote:
>
> > I am writing a web application using web2py for execution on app
> > engine. I've deployed other web2py apps there but I've never used any
> > app engine-specific utilities. In this project I want to use app
> > engine's taskqueue and run it on a backend instance.
>
> > I'm developing on app engine's dev_appserver which is working out
> > okay, but but I'm not seeing how to do unit testing.
>
> > Is this a problem that has been solved already?
>
>
>
> tests.py
> 6KViewDownload
[web2py] unit testing web2py and app engine
I am writing a web application using web2py for execution on app engine. I've deployed other web2py apps there but I've never used any app engine-specific utilities. In this project I want to use app engine's taskqueue and run it on a backend instance. I'm developing on app engine's dev_appserver which is working out okay, but but I'm not seeing how to do unit testing. Is this a problem that has been solved already?
[web2py] Re: Migration on Datastore
Yes, that worked. Thanks for the suggestion, Massima.
On Mar 7, 5:12 pm, Massimo Di Pierro
wrote:
> can you try Field('description', 'text', length = 2048),
>
>
>
>
>
>
>
> On Wednesday, 7 March 2012 15:48:51 UTC-6, David Phillips wrote:
>
> > I have an App Engine web app in production. I defined a field in a
> > database table as a string with no length.
>
> > Field('description'),
>
> > Now I find that the field needs to handle 2,000 characters. That means
> > that the field needs to be changed in the Datastore entity from a
> > ByteString type to a Blob type. I don't,however, want to accidentally
> > delete data, and I'm unsure how to safely convert the field to handle
> > a 2K string.
>
> > I did try one thing. I changed the field to
>
> > Field('description', length = 2048),
>
> > When I deployed the new code, the field remained a string and an
> > attempt to store more than 500 characters caused an exception.
>
> > So is there any way to migrate the existing string field to a blob?
> > And if so, how would one go about it?
>
> > By the way, there was already a bit of a mismatch. The default string
> > field in the DAL is supposed to be 512 bytes. When deployed on
> > appengine, that causes the creation of a ByteString which has a
> > maximum size of 500 characters.
[web2py] Migration on Datastore
I have an App Engine web app in production. I defined a field in a
database table as a string with no length.
Field('description'),
Now I find that the field needs to handle 2,000 characters. That means
that the field needs to be changed in the Datastore entity from a
ByteString type to a Blob type. I don't,however, want to accidentally
delete data, and I'm unsure how to safely convert the field to handle
a 2K string.
I did try one thing. I changed the field to
Field('description', length = 2048),
When I deployed the new code, the field remained a string and an
attempt to store more than 500 characters caused an exception.
So is there any way to migrate the existing string field to a blob?
And if so, how would one go about it?
By the way, there was already a bit of a mismatch. The default string
field in the DAL is supposed to be 512 bytes. When deployed on
appengine, that causes the creation of a ByteString which has a
maximum size of 500 characters.
[web2py] admin over https but not appadmin
I'm confused by the behavior of my web app. It is deployed on a remote server using ubuntu+apache+web2py+mysql with a self-signed certificate and was installed using the setup- web2py-ubuntu.sh script. Everythings seems functional except that I can't access my application's appadmin controller. https://www.example.com/admin/ takes me to the admin application. But: https://www.example.com/app/appadmin takes me to the admin application. I haven't made any changes to the default /etc/apache2/sites-available/default file. Is there a way to access appadmin for my application?
[web2py] Re: web2py and self-signed SSL certificates
Thanks for your help, Anthony. I pulled out the openssl commands and made a private key and an ssl certificate, but I couldn't get them to work with the rocket server. In fact, when I started up the server, it wouldn't serve any pages at all. I used this command: nohup sudo python web2py.py -a '' -i 0.0.0.0 -p 80 -c self_signed.cert -k self_signed.key So I just ran the ubuntu+apache+web2py script and let apache serve my web app. It was painless, and it's probably a better approach for my purposes than using rocket. But still, I wish I knew the answer. There are times when it is easier to run the development server and it would be better to be able to test using both http and https. Am I mis-using the command line arguments? For instance, looking at my command line, I am wondering how to tell the server to respond on both port 80 and 423? Or perhaps this is web2py bug and it isn't able to serve up https. On Mar 1, 8:09 pm, Anthony wrote: > There are scripts for setting up web2py on Ubuntu with Apache > (http://code.google.com/p/web2py/source/browse/scripts/setup-web2py-ub...) > and Nginx/uwsgi > (http://code.google.com/p/web2py/source/browse/scripts/setup-web2py-ng...), > both of which include commands for creating a self-signed certificate using > OpenSSL -- perhaps you can look at them for some guidance. > > Anthony > > > > > > > > On Thursday, March 1, 2012 8:48:08 PM UTC-5, David Phillips wrote: > > > I freely admit that I don't understand how https, SSL, and public key > > infrastructure works. It doesn't seem like it should be hard to use > > but whenever I try, things don't work. > > > For instance, I wanted to access the admin interface for my web2py > > application on a remote host. My thought was that I don't need to buy > > an SSL certificate because I trust myself, for the most part. The > > web2py command line allows the user to specify an SSL certificate (-c) > > and a private key (-k). I figured one of these would work. > > > So I made a self-signed certificate according to some instructions I > > found online: > > > openssl genrsa -des3 -out server.key 1024 > > server.key is a private key > > openssl req -new -key server.key -out server.csr > > server.csr is a certificate signing request > > cp server.key server.key.org > > save off the server key > > openssl rsa -in server.key.org -out server.key > > create a derivative key that doesn't need a passphrase > > openssl rsa -in server.key.org -out server.key > > create the self-signed certificate, server.crt > > > Then I used the certificate to start the rocket server: > > python web2py.py -p 8001 -a '' -c server.crt > > > But when I tried to access the page: > > https://127.0.0.1:8001 > > I get a browser error: > > Error 107 (net::ERR_SSL_PROTOCOL_ERROR): SSL protocol error. > > > Since I don't know what I"m doing, I tried some experiments: > > python web2py.py -p 8001 -a '' -k server.key > > > and > > > python web2py.py -p 8001 -a '' -c server.crt -k > > server.key > > > all to no avail. > > > Thinking that maybe the browser (Chrome) doesn't know to trust the > > certificate, I went to the preferences window -> https/ssl -> manage > > certificates... which launches Keychain Access on my Macintosh. I > > tried to import the new self-signed certificate into Keychain Access > > so that I could mark it as trusted but I got an error (the not very > > helpful: "an error has occurred. Unable to import an item." with > > nothing logged in the console) > > > So I tried it the other way. I created a certificate in Keychain > > access, marked it trusted for SSL and then exported it. I used the > > certificate to start the browser: > > > python web2py.py -p 8001 -a '' -c new.crt > > > Again failure. > > > So what am I doing wrong? > > > As a bonus question, is there a place to go learn about these issues? > > I've looked around and I can't find either a website or a book that > > can explain to me how SSL, CAs, and PKI works. The information must be > > out there, maybe even in a gentle, understandable form.
[web2py] web2py and self-signed SSL certificates
I freely admit that I don't understand how https, SSL, and public key infrastructure works. It doesn't seem like it should be hard to use but whenever I try, things don't work. For instance, I wanted to access the admin interface for my web2py application on a remote host. My thought was that I don't need to buy an SSL certificate because I trust myself, for the most part. The web2py command line allows the user to specify an SSL certificate (-c) and a private key (-k). I figured one of these would work. So I made a self-signed certificate according to some instructions I found online: openssl genrsa -des3 -out server.key 1024 server.key is a private key openssl req -new -key server.key -out server.csr server.csr is a certificate signing request cp server.key server.key.org save off the server key openssl rsa -in server.key.org -out server.key create a derivative key that doesn't need a passphrase openssl rsa -in server.key.org -out server.key create the self-signed certificate, server.crt Then I used the certificate to start the rocket server: python web2py.py -p 8001 -a '' -c server.crt But when I tried to access the page: https://127.0.0.1:8001 I get a browser error: Error 107 (net::ERR_SSL_PROTOCOL_ERROR): SSL protocol error. Since I don't know what I"m doing, I tried some experiments: python web2py.py -p 8001 -a '' -k server.key and python web2py.py -p 8001 -a '' -c server.crt -k server.key all to no avail. Thinking that maybe the browser (Chrome) doesn't know to trust the certificate, I went to the preferences window -> https/ssl -> manage certificates... which launches Keychain Access on my Macintosh. I tried to import the new self-signed certificate into Keychain Access so that I could mark it as trusted but I got an error (the not very helpful: "an error has occurred. Unable to import an item." with nothing logged in the console) So I tried it the other way. I created a certificate in Keychain access, marked it trusted for SSL and then exported it. I used the certificate to start the browser: python web2py.py -p 8001 -a '' -c new.crt Again failure. So what am I doing wrong? As a bonus question, is there a place to go learn about these issues? I've looked around and I can't find either a website or a book that can explain to me how SSL, CAs, and PKI works. The information must be out there, maybe even in a gentle, understandable form.
[web2py] web2py and self-signed SSL certificates
I freely admit that I don't understand how https, SSL, and public key infrastructure works. It doesn't seem like it should be hard to use but whenever I try, things don't work. For instance, I wanted to access the admin interface for my web2py application on a remote host. My thought was that I don't need to buy an SSL certificate because I trust myself, for the most part. The web2py command line allows the user to specify an SSL certificate (-c) and a private key (-k). I figured one of these would work. So I made a self-signed certificate according to some instructions I found online: openssl genrsa -des3 -out server.key 1024 server.key is a private key openssl req -new -key server.key -out server.csr server.csr is a certificate signing request cp server.key server.key.org save off the server key openssl rsa -in server.key.org -out server.key create a derivative key that doesn't need a passphrase openssl rsa -in server.key.org -out server.key create the self-signed certificate, server.crt Then I used the certificate to start the rocket server: python web2py.py -p 8001 -a '' -c server.crt But when I tried to access the page: https://127.0.0.1:8001 I get a browser error: Error 107 (net::ERR_SSL_PROTOCOL_ERROR): SSL protocol error. Since I don't know what I"m doing, I tried some experiments: python web2py.py -p 8001 -a '' -k server.key and python web2py.py -p 8001 -a '' -c server.crt -k server.key all to no avail. Thinking that maybe the browser (Chrome) doesn't know to trust the certificate, I went to the preferences window -> https/ssl -> manage certificates... which launches Keychain Access on my Macintosh. I tried to import the new self-signed certificate into Keychain Access so that I could mark it as trusted but I got an error (the not very helpful: "an error has occurred. Unable to import an item." with nothing logged in the console) So I tried it the other way. I created a certificate in Keychain access, marked it trusted for SSL and then exported it. I used the certificate to start the browser: python web2py.py -p 8001 -a '' -c new.crt Again failure. So what am I doing wrong? As a bonus question, is there a place to go learn about these issues? I've looked around and I can't find either a website or a book that can explain to me how SSL, CAs, and PKI works. The information must be out there, maybe even in a gentle, understandable form.
[web2py] Re: Database deadlocks with new scheduler
Thanks for the advice, Massimo. Yes, I am calling db.scheduler_task.insert() inside a controller. On the theory that it could do harm, I put a db.commit() immediately after the insert and that helped, some. Now I'm getting the scheduler exception in my worker process about a third as often, and when it occurs, I can clear the deadlock easier. Sometimes, I can just restart the worker process. Sometimes I have to user the MySQL client to stop the query (ie., mysql> kill query 24843;) (I don't want it to sound like I really understand how to clear the deadlock. I'm just doing something I read about on stackoverflow.) I'd like to understand better how the deadlocks arise and what I can do to prevent them. Thanks. David On Feb 27, 11:11 pm, Massimo Di Pierro wrote: > Do you do this in a controller? > > db.scheduler_task.insert(...) > > if not. Make sure you db.commit() immediately after. Would is solve > the problem? > > On Feb 27, 9:14 pm, David Phillips > wrote: > > > > > > > > > I am experiencing database deadlocks in a worker processes that is > > being scheduled by web2py's newscheduler. > > > I'm using the latest web2py with Python 2.7.2 and MySQL (a recent > > release) on OS X 10.7.3. > > > The worker does a lengthy calculation that I wanted to take off the > > http thread. Every time I get a certain action, I schedule the thread > > programmatically like this: > > > db.scheduler_task.insert(application_name = 'geo', task_name = > > 'match_task', > > function_name = 'match_listings', vars = > > json.dumps(vars)) > > > In my testing I execute this task about once every two seconds. It has > > failed in two separate tests after about an hour. > > > I'm getting this traceback: > > > File "/Users/davidp/dev/python/web2py/gluon/shell.py", line 214, in > > run > > exec(python_code, _env) > > File "", line 1, in > > File "/Users/davidp/dev/python/web2py/gluon/scheduler.py", line 363, > > in loop > > MetaScheduler.loop(self) > > File "/Users/davidp/dev/python/web2py/gluon/scheduler.py", line 255, > > in loop > > task = self.pop_task() > > File "/Users/davidp/dev/python/web2py/gluon/scheduler.py", line 392, > > in pop_task > > grabbed.update(assigned_worker_name='',status=QUEUED) > > File "/Users/davidp/dev/python/web2py/gluon/dal.py", line 6346, in > > update > > return self.db._adapter.update(tablename,self.query,fields) > > File "/Users/davidp/dev/python/web2py/gluon/dal.py", line 1093, in > > update > > self.execute(sql) > > File "/Users/davidp/dev/python/web2py/gluon/dal.py", line 1359, in > > execute > > return self.log_execute(*a, **b) > > File "/Users/davidp/dev/python/web2py/gluon/dal.py", line 1353, in > > log_execute > > ret = self.cursor.execute(*a, **b) > > File "/Users/davidp/dev/python/web2py/gluon/contrib/pymysql/ > > cursors.py", line 108, in execute > > self.errorhandler(self, exc, value) > > File "/Users/davidp/dev/python/web2py/gluon/contrib/pymysql/ > > connections.py", line 184, in defaulterrorhandler > > raise errorclass, errorvalue > > InternalError: (1213, u'Deadlockfound when trying to get lock; try > > restarting transaction') > > > When executed, the task does this SQL select: > > > expr = '''SELECT *, ( 3959 * acos (cos (radians (%s)) * cos (radians > > (lat)) * > > cos (radians (lon) - radians (%s)) + sin (radians (%s)) * sin > > (radians (lat > > AS distance FROM users HAVING distance < radius;''' % (point[0], > > point[1], point[0]) > > nearby_users = db.executesql(expr, as_dict = True) > > > and then does a read to collect some statistics to track performance: > > > num_listings = db (db.listings.id > 0).count() > > > During this time, I am receiving asynchronous http requests that > > insert, update and delete from this table (and another table, too). > > > My MySQL client responds to a "show engine innodb status;" command > > this way: > > > > > LATEST DETECTEDDEADLOCK > > > > 120227 20:38:16 > > *** (1) TRANSACTION: > > TRANSACTION 27AFF, ACTIVE 0 sec starting index read > > mysql tables in use 1, locked 1 > > LOCK WAIT 3 lock struct(s), heap size 376, 2 ro
[web2py] Database deadlocks with new scheduler
I am experiencing database deadlocks in a worker processes that is
being scheduled by web2py's new scheduler.
I'm using the latest web2py with Python 2.7.2 and MySQL (a recent
release) on OS X 10.7.3.
The worker does a lengthy calculation that I wanted to take off the
http thread. Every time I get a certain action, I schedule the thread
programmatically like this:
db.scheduler_task.insert(application_name = 'geo', task_name =
'match_task',
function_name = 'match_listings', vars =
json.dumps(vars))
In my testing I execute this task about once every two seconds. It has
failed in two separate tests after about an hour.
I'm getting this traceback:
File "/Users/davidp/dev/python/web2py/gluon/shell.py", line 214, in
run
exec(python_code, _env)
File "", line 1, in
File "/Users/davidp/dev/python/web2py/gluon/scheduler.py", line 363,
in loop
MetaScheduler.loop(self)
File "/Users/davidp/dev/python/web2py/gluon/scheduler.py", line 255,
in loop
task = self.pop_task()
File "/Users/davidp/dev/python/web2py/gluon/scheduler.py", line 392,
in pop_task
grabbed.update(assigned_worker_name='',status=QUEUED)
File "/Users/davidp/dev/python/web2py/gluon/dal.py", line 6346, in
update
return self.db._adapter.update(tablename,self.query,fields)
File "/Users/davidp/dev/python/web2py/gluon/dal.py", line 1093, in
update
self.execute(sql)
File "/Users/davidp/dev/python/web2py/gluon/dal.py", line 1359, in
execute
return self.log_execute(*a, **b)
File "/Users/davidp/dev/python/web2py/gluon/dal.py", line 1353, in
log_execute
ret = self.cursor.execute(*a, **b)
File "/Users/davidp/dev/python/web2py/gluon/contrib/pymysql/
cursors.py", line 108, in execute
self.errorhandler(self, exc, value)
File "/Users/davidp/dev/python/web2py/gluon/contrib/pymysql/
connections.py", line 184, in defaulterrorhandler
raise errorclass, errorvalue
InternalError: (1213, u'Deadlock found when trying to get lock; try
restarting transaction')
When executed, the task does this SQL select:
expr = '''SELECT *, ( 3959 * acos (cos (radians (%s)) * cos (radians
(lat)) *
cos (radians (lon) - radians (%s)) + sin (radians (%s)) * sin
(radians (lat
AS distance FROM users HAVING distance < radius;''' % (point[0],
point[1], point[0])
nearby_users = db.executesql(expr, as_dict = True)
and then does a read to collect some statistics to track performance:
num_listings = db (db.listings.id > 0).count()
During this time, I am receiving asynchronous http requests that
insert, update and delete from this table (and another table, too).
My MySQL client responds to a "show engine innodb status;" command
this way:
LATEST DETECTED DEADLOCK
120227 20:38:16
*** (1) TRANSACTION:
TRANSACTION 27AFF, ACTIVE 0 sec starting index read
mysql tables in use 1, locked 1
LOCK WAIT 3 lock struct(s), heap size 376, 2 row lock(s), undo log
entries 1
MySQL thread id 19816, OS thread handle 0x10c5db000, query id 1558959
localhost 127.0.0.1 root Updating
UPDATE scheduler_task SET status='QUEUED',assigned_worker_name=''
WHERE ((scheduler_task.assigned_worker_name = 'Mycroft.local#d600fcf2-
a363-4870-bd24-1ad3694f6a62') AND (scheduler_task.status =
'ASSIGNED'))
*** (1) WAITING FOR THIS LOCK TO BE GRANTED:
RECORD LOCKS space id 0 page no 3031 n bits 120 index `PRIMARY` of
table `geo`.`scheduler_task` trx id 27AFF lock_mode X waiting
Record lock, heap no 2 PHYSICAL RECORD: n_fields 20; compact format;
info bits 0
0: len 4; hex 802c; asc,;;
1: len 6; hex 00024db1; asc M ;;
2: len 7; hex 6a0d4c2f35; asc j L/5;;
3: len 3; hex 67656f; asc geo;;
4: len 10; hex 656d61696c207461736b; asc email task;;
5: len 4; hex 6d61696e; asc main;;
6: len 6; hex 515545554544; asc QUEUED;;
7: len 10; hex 73656e645f656d61696c; asc send_email;;
8: len 2; hex 5b5d; asc [];;
9: len 2; hex 7b7d; asc {};;
10: len 1; hex 54; asc T;;
11: len 8; hex 8000124c9afce86c; ascL l;;
12: len 8; hex 8000124c9afee908; ascL;;
13: len 8; hex 8000124c9b0c193a; ascL :;;
14: len 4; hex 8000; asc ;;
15: len 4; hex 8e10; asc ;;
16: len 4; hex 8dde; asc ;;
17: len 4; hex 800d; asc ;;
18: len 8; hex 8000124c9afec1f8; ascL;;
19: len 0; hex ; asc ;;
*** (2) TRANSACTION:
TRANSACTION 27AFD, ACTIVE 0 sec fetching rows
mysql tables in use 2, locked 2
52 lock struct(s), heap size 6960, 2775 row lock(s), undo log entries
9
MySQL thread id 19817, OS thread handle 0x10c598000, query id 1558955
localhost 127.0.0.1 root Sending data
UPDATE scheduler_task SET status='QUEUED',assigned_worker_name=''
WHERE ((scheduler_task.assigned_worker_name IN (SELECT
scheduler_worker.worker_name FROM scheduler_worker WHERE
(scheduler_worker.last_heartbeat < '2012-02-27 20:38:07'))) AND
(scheduler_task.status IN ('RUNNING','ASSIGNED','QUEUED')))
*** (2) HOLDS THE LOCK(S):
RECORD LOCKS space id
