Switched to lazy table definitions.
Model init time was cut down to 0.046s.
Some of excess time is eliminated, some (my guess is 30%) is moved into
controller execution. At any rate - this is faster than before.
Next step would be full-scale profiling but not yet.
Here is excerpt from my SQLStorage:
--------------
def __getitem__(self, key):
value = dict.__getitem__(self, str(key))
if not callable(value) or key[0]=='_' or isinstance(value,
SQLCallableList): return value
value.__call__() # That must redefine table in-place
return dict.__getitem__(self, str(key))
------------
and here is excerpt from my db.py:
----------
def define_table_system_participant():
db.define_table('system_participant',
SQLField('firm_id','integer'),
migrate=migrate,
)
db.system_participant=define_table_system_participant
----------
On Saturday 06 June 2009 07:53:18 Alexey Nezhdanov wrote:
> ON Saturday 06 June 2009 00:25:47 mdipierro wrote:
> > One other trick you can try is replace
> >
> > db.define_table('table',SQLField('field'),...)
> > db.table.field.requires=....
> >
> > with
> >
> > db.define_table('table',SQLfield('field',requires=...),...)
> >
> > and so for all the other attributes.
>
> That will make minor difference. I do not have too many 'requires' and mod
> of what I have are set up through function call.
>
> > Did you bytecode compile the app?
> > Does it make a difference?
>
> I just run some automated tests. Here is average time over 100 runs each:
>
> sqlite+nomigrate+py 0.123
> sqlite+nomigrate+pyc 0.122
> mysql+nomigrate+py 0.123
> mysql+nomigrate+pyc 0.123
>
> I think I'll try this approach:
> 1) define each table as a function which yelds a table.
> 2) modify sql.py so that db object will test the type of table.
> if it has __exec__ method - execute it and replace it with return result.
>
> This way my tables will be lazily defined when controller actually needs
> them.
>
> > Massimo
> >
> > On Jun 5, 3:05 pm, Alexey Nezhdanov <snak...@gmail.com> wrote:
> > > On Friday 05 June 2009 21:03:20 mdipierro wrote:> Can you tell us more
> > > about the setup, os, hardward etc. is mysql on
> > >
> > > > the same machine?
> > >
> > > Kubuntu 8.04. Turion64 1.6GHz, 1.6G RAM. MySQL is on the same box. SiS
> > > motherboard w/ nForce chipset. Laptop 3 years old (and it was about 1
> > > year old model when was bought).
> > >
> > > > How much is the the SQLDB() vs the define_tables? Do you have many
> > > > tables? how long?
> > >
> > > 16 tables, 152 SQLFields. single SQLDB (currently MySQL, but I'll
> > > switch it back to SQLite)
> > >
> > > > One trick is to add is statements in the model so that only those
> > > > tables needed are defined, depending on request.controller and
> > > > request.action.
> > >
> > > yes, I thought of that. But that makes it inflexible. That's why I
> > > suggested lazy tables init.
> > >
> > > And regarding 'turion is not very fast'. I don't really have any load
> > > on this box. So 0.5 seconds per GET is VERY slow. 8-years old Celeron
> > > 800 should be behaving something like 0.05 seconds per request (of
> > > course with ad-hoc programming, no DAL).
> > >
> > > This is not the empty complaint. We can't really afford saying 'throw
> > > in more CPU'. If web2py targets GAE - then it absolutely must be
> > > CPU-friendly. GAE can help with adding more nodes but it charges for
> > > processor time anyways. And actually the same goes about dedicated
> > > hosting too. If someone targets only a few visitors per day - it's ok.
> > > But not if we want tens and hundreds pageloads per second.
> > >
> > > > On Jun 5, 11:29 am, Alexey Nezhdanov <snak...@gmail.com> wrote:
> > > > > On Friday 05 June 2009 17:07:55 mdipierro wrote:> In a production
> > > > > environment you would be using mysql or postgresql. In
> > > > >
> > > > > > this case you should be using
> > > > > >
> > > > > > SQLDB(...,pool_size=10)
> > > > > > dn.define_table(....,migrate=False)
> > > > > >
> > > > > > the connection pooling and migrations off make a big difference.
> > > > > > Perhaps you can run some tests and quantify this.
> > > > >
> > > > > migrate=False makes cuts the model init time in half - now I'm
> > > > > getting about 0.15-0.17s each time. Testing MySQL, stand by...
> > > > >
> > > > > Hmmm.
> > > > > 0.21...0.25s with mysql and migrations off... and pool_size=10.
> > > > >
> > > > > > When using sqlite you cannot use pooling and that means web2py
> > > > > > has to open the db every time.
> > > > > >
> > > > > > Massimo
> > > > > >
> > > > > > On Jun 5, 2:58 am, Alexey Nezhdanov <snak...@gmail.com> wrote:
> > > > > > > Hello again.
> > > > > > > Recently I measured the perfomance of web2py regarding to
> > > > > > > 'milliseconds per request'. Got some unexpected results. The
> > > > > > > most slow part of the application is the model. It takes 40-60%
> > > > > > > of total time. Measurement was done simply by putting
> > > > > > > import time;print time.time(),'model start'
> > > > > > > at the beginning of db.py and similar line at the end of it.
> > > > > > > Here is what it produces on my laptop (Turion64, 1.6GHz, 1.5G
> > > > > > > RAM):
> > > > > > >
> > > > > > > 1244187446.32 model start
> > > > > > > 1244187446.62 model stop
> > > > > > > 0.3 second just to set up the model! I can live with 0.05 for
> > > > > > > it, may be even 0.1, but 0.3 for _each_ GET or POST request is
> > > > > > > a bit too much, don't you think?
> > > > > > > That is for not too complex model - 17 tables, averaging 8.6
> > > > > > > SQLFields per one. On another web2py project it takes
> > > > > > > 0.38...0.42 second each time
> > > > > > >
> > > > > > > :(
> > > > > > >
> > > > > > > I tried compiling my app and measuring again:
> > > > > > > 1244187625.31 model start
> > > > > > > 1244187625.69 model stop
> > > > > > > Not any better. In fact, it's even worse, but since results
> > > > > > > vary from run to run I suspect that it is just the same
> > > > > > > perfomance.
> > > > > > >
> > > > > > > Massimo, as I know you've been working on new model for some
> > > > > > > time already. Is there any hope of having a faster model? I
> > > > > > > suspect more lazy evaluation should do the magic, but I didn't
> > > > > > > do any research yet.
> > > > > > >
> > > > > > > Frankly speaking when I first discovered the fact that web2py
> > > > > > > always _executes_ model, controller, view, I thought that it
> > > > > > > may be a perfomance hog. Until I actually did that check I
> > > > > > > thought that it will execute db.py each time it changes on-disk
> > > > > > > and then just keep built structures somewhere around, probably
> > > > > > > pickled. May be it is still possible to use that approach to
> > > > > > > some extent?
> > > > > > >
> > > > > > > Or may be I am just completely missing the point. Please
> > > > > > > comment.
> > > > > > >
> > > > > > > --
> > > > > > > Sincerely yours
> > > > > > > Alexey Nezhdanov
> > > > >
> > > > > --
> > > > > Sincerely yours
> > > > > Alexey Nezhdanov
> > >
> > > --
> > > Sincerely yours
> > > Alexey Nezhdanov
> >
> >
--
Sincerely yours
Alexey Nezhdanov
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"web2py Web Framework" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~----------~----~----~----~------~----~------~--~---
