Re: [web2py] Set up of db.py takes too long time

2013-01-15 Thread Paolo valleri 
Try to add lazy_tables=True to DAL. Like:
db = DAL('sqlite://storage.sqlite', migrate=False, lazy_tables=True)


Paolo

On Tuesday, January 15, 2013 10:06:03 AM UTC+1, Daniel Gonzalez wrote:
>
> No imports in the code-area profiled. Also, I am testing some requests, 
> but the time I am measuring is the time spent in the code-area that I have 
> provided. That is, in the define tables code. The controllers are of course 
> making use of the table definitions later, but that is not what I am 
> measuring.
>
> I have put a "print time.time()" before and after the "define tables" 
> code. So no controller is involved in my measurements.
>
> On Tuesday, January 15, 2013 9:56:57 AM UTC+1, rochacbruno wrote:
>>
>> It will run only when called!
>>
>> But if he is testing a form, the problem maybe there.
>>
>> There are some imports?
>>
>

--

Re: [web2py] Set up of db.py takes too long time

2013-01-15 Thread Daniel Gonzalez 
No imports in the code-area profiled. Also, I am testing some requests, but 
the time I am measuring is the time spent in the code-area that I have 
provided. That is, in the define tables code. The controllers are of course 
making use of the table definitions later, but that is not what I am 
measuring.

I have put a "print time.time()" before and after the "define tables" code. 
So no controller is involved in my measurements.

On Tuesday, January 15, 2013 9:56:57 AM UTC+1, rochacbruno wrote:
>
> It will run only when called!
>
> But if he is testing a form, the problem maybe there.
>
> There are some imports?
>

--

Re: [web2py] Set up of db.py takes too long time

2013-01-15 Thread Bruno Rocha 
It will run only when called!

But if he is testing a form, the problem maybe there.

There are some imports?

--

Re: [web2py] Set up of db.py takes too long time

2013-01-15 Thread David Marko 
It really runs these validators during table definition, and not just in 
time when used? So its better to define this requires= ... in controller 
where this is used?

David

Dne úterý, 15. ledna 2013 9:23:43 UTC+1 rochacbruno napsal(a):
>
>
> On Tue, Jan 15, 2013 at 6:20 AM, Daniel Gonzalez 
> 
> > wrote:
>
>> IS_NOT_IN_DB(db, '%s.email' % (web2py_user_table))
>
>
> If you too many records on user table, the above code will take a long 
> time, because on every request it will "select email from auth_user"
>
>
>
>

--

Re: [web2py] Set up of db.py takes too long time

2013-01-15 Thread Daniel Gonzalez 
Very few records (around 10). I am testing still.

On Tuesday, January 15, 2013 9:23:43 AM UTC+1, rochacbruno wrote:
>
>
> On Tue, Jan 15, 2013 at 6:20 AM, Daniel Gonzalez 
> 
> > wrote:
>
>> IS_NOT_IN_DB(db, '%s.email' % (web2py_user_table))
>
>
> If you too many records on user table, the above code will take a long 
> time, because on every request it will "select email from auth_user"
>
>
>
>

--

Re: [web2py] Set up of db.py takes too long time

2013-01-15 Thread Bruno Rocha 
On Tue, Jan 15, 2013 at 6:20 AM, Daniel Gonzalez  wrote:

> IS_NOT_IN_DB(db, '%s.email' % (web2py_user_table))


If you too many records on user table, the above code will take a long
time, because on every request it will "select email from auth_user"

--

[web2py] Set up of db.py takes too long time

2013-01-15 Thread Daniel Gonzalez 
Hi,

My db.py takes a long time to run (around 40 ms). This means that just the 
overhead of db.py limits my throughput to 25 req/s (in a single process / 
single thread configuration). This seems to me quite low.

I have tried to take a look at where the time is spent. I have seen the 
following runtimes. These are average values:

 8 ms -> session.connect(request,response,db=MEMDB(cache.memcache))
 2 ms -> db = DAL('sqlite://storage.sqlite', migrate=False)
 5 ms -> auth = Auth(db)
20 ms -> define tables

My define tables is like this:

# Use the authorization table for users
web2py_user_table = auth.settings.table_user_name
# web2py_user_table = 'web2py_user'
db.define_table(
web2py_user_table,
Field('org_id',   type='integer', compute=set_org_id),
Field('email',unique=True),
Field('user_doc_id',  length=128, compute=create_new_user),
Field('password', length=512, compute=automatic_password
, type='password', readable=False, label='Password'), # TODO: use CRYPT
# These fields are needed for authorization
Field('registration_key', length=512, writable=False, readable=
False, default=''),
Field('reset_password_key',   length=512, writable=False, readable=
False, default=''),
Field('registration_id',  length=512, writable=False, readable=
False, default=''),
format = '%(email)s')

# Add constraints to the web2py_user table
web2py_user = db[web2py_user_table]
web2py_user.email.requires= [IS_EMAIL(error_message=auth.
messages.invalid_email), IS_NOT_IN_DB(db, '%s.email' % (web2py_user_table))]
# web2py_user.user_doc_id.requires = 
IS_NOT_EMPTY(error_message=auth.messages.is_empty)
# web2py_user.password.requires = [IS_STRONG(min=5, special=0, 
upper=0), CRYPT()]

auth.define_tables()

Those run times look to me quite high. Maybe some of you can give me hints:

   - Is there anything I can do to speed this up?
   - Why is the define tables taking such a long time, even with 
   migrate=False?
   - Would moving to postgres improve things a lot?
   - Is sqlite the bottleneck here?
   
It is difficult for me to believe that sqlite is the cultprit here: I am 
using a single client in my tests, sending requests sequentially (no 
parallel requests).

Thanks,
Daniel

--