Hi again. Here are another testing results. First of all, let me remind you about is_integer bug. It doesn't allow negative numbers. Correct code is: is_integer=re.compile('-?[\d]+).match
Now. I've run more timings, more profiling and more code tweaks. This time it was Core2 Duo @2.20GHz with 1G RAM. Ubuntu 8.04. Testing strategy: 20 sets of 1000 GET requests to the main page. Same application as before. For each set I have average GET request time. For a group of 20 sets I get minimum average, global average and maximum average. Also I have average model init time for 20k requests (no splitting into each set). 1) Upstream sql.py model 0.01646 min 0.02979 avg 0.03267 max 0.03597 2) Optimised SQLTable.__init__ and SQLField.__init__ (patches attached, backwards compartible, though you may want to remove my testing marks) model 0.01380 min 0.02697 avg 0.03003 max 0.03189 So - about 8% speedup (by avg time) here. 3) Now lazy tables (patch attached). Since I expect that you will not find any objections to my patches for (2) here are cumulative results, i.e. optimised __init__s + lazy tables ... hmmm. It doesn't improves results significantly anymore. It looks like these patches are interchangeable. I've run my test three times, results are: model 0.01009 min 0.02555 avg 0.0297105 max 0.033360 model 0.01012 min 0.02369 avg 0.02983 max 0.03340 model 0.00966 min 0.02415 avg 0.02850 max 0.03208 So it makes some improvement about 1%-5% but somehow it also makes testing results more disperse. Do not know if it worths including. Massimo, your opinion? 4) Also I've tried to further speedup validator generation w/o losing backwards compartibility, but improvement was about 1% and below the margin of error (even less noticeable than with lazy tables). On Sun, Jun 7, 2009 at 11:45 PM, Alexey Nezhdanov<snak...@gmail.com> wrote: > On Sunday 07 June 2009 18:41:30 mdipierro wrote: >> I implemented a backward compatible speedup of the default validators. >> I also realized that those many calls to is_integer could be avoided. >> I think I fixed it. > 0.0429/0.0844 > Somehow consistency is not kept across reboots :( > ... about a hour later: and actually now I get about 10% noise so it become > very hard to make any good measurements. I'll try tomorrow on better > hardware. In the meanwhile, I'll try to profile it once again. > ... > ok, there are still two perfomance hogs left: SQLTable.__init__ (about 11%) > and SQLField.__init__ (about 6,5%). These percentages are 'self' times - i.e. > it was spent directly in lines 1099-1149, 1693-1752. > Python profiling doesn't allow to attribute CPU consumption to individual > lines so for now I don't have any more precise pointers. But that could be > split into functional blocks and narrowed further down. > > In any case - we collected almost all low-hanging fruits. Any further > optimisation will give gains of range 2-5%, not times anymore. > > Except lazy tables which still may give tens of percents. BTW - Massimo, you > said something about SQLLazyTable class. You should implement SQLLazyField > too then. Or may be just wait a day or two until I do deeper profiling. > >> Please check out the latest trunk. >> >> Massimo >> > > P.S. Here is small fix to is_integer. We forgot about negative numbers: > Also this version is tiny bit faster > is_integer=re.compile('-?[\d]+).match > > -- > Sincerely yours > Alexey Nezhdanov > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "web2py Web Framework" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~----------~----~----~----~------~----~------~--~---
--- /home/snake/Desktop/sql.py 2009-06-08 09:36:27.225210464 +0400 +++ sql.py 2009-06-08 11:32:18.285211851 +0400 @@ -1112,39 +1112,46 @@ @raise SyntaxError: when a supplied field is of incorrect type. """ - new_fields = [] - for field in fields: - if isinstance(field, SQLField): - new_fields.append(field) - elif isinstance(field, SQLTable): - new_fields += [copy.copy(field[f]) for f in - field.fields if f != 'id'] - else: - raise SyntaxError, \ - 'define_table argument is not a SQLField' - fields = new_fields - self._db = db - self._tablename = tablename - self.fields = SQLCallableList() - self._referenced_by = [] - fields = list(fields) - fields.insert(0, SQLField('id', 'id')) - for field in fields: - self.fields.append(field.name) - self[field.name] = field - field._tablename = self._tablename - field._table = self - field._db = self._db - if isinstance(field.type,str) and field.type[:9] == 'reference': - referenced = field.type[10:].strip() - if not referenced: - raise SyntaxError, 'SQLTable: reference to nothing!' - if not referenced in self._db: - raise SyntaxError, 'SQLTable: table does not exist' - referee = self._db[referenced] - if self._tablename in referee.fields: - raise SyntaxError, 'SQLField: table %s has same name as a field in referenced table %s' % (self._tablename, referee._tablename) - referee._referenced_by.append((self._tablename, field.name)) + new_fields = [ SQLField('id', 'id') ] # 1 + for field in fields: # 1 + if isinstance(field, SQLField): # 1 0.11 + new_fields.append(field) # 1 + elif isinstance(field, SQLTable): # 1 + new_fields += [copy.copy(field[f]) for f in # 1 + field.fields if f != 'id'] # 1 + else: # 1 + raise SyntaxError, 'define_table argument is not a SQLField' # 1 +# fields = new_fields # 0 + self._db = db # 0 + self._tablename = tablename # 0 0.11 + fields = self.fields = SQLCallableList() # 0 + self._referenced_by = [] # 0 +# fields = list(fields) # 2 +# fields.insert(0, SQLField('id', 'id')) # 2 0.23 + for field in new_fields: # 2 + field_name=field.name + fields.append(field_name) # 3 0.58 +# self[field_name] = field # 3 + dict.__setitem__(self, field_name, field) + field._tablename = tablename # 3 + field._table = self # 3a 0.43 + field._db = db # 3a + try: + if field.type[:9] != 'reference': + continue + except: +# if isinstance(field.type, str) and field.type[:9] == 'reference': # 3a + referenced = field.type[10:].strip() # 4 + try: + referee = db[referenced] # 4 + except: + if not referenced in db: # 4 + if not referenced: # 4 + raise SyntaxError, 'SQLTable: reference to nothing!' # 4 0.19 + raise SyntaxError, 'SQLTable: table does not exist' # 4 + if tablename in referee.fields: # 4 + raise SyntaxError, 'SQLField: table %s has same name as a field in referenced table %s' % (tablename, referee._tablename) + referee._referenced_by.append((tablename, field_name)) # 4 self.ALL = SQLALL(self)
--- /home/snake/Desktop/sql.py 2009-06-08 09:36:27.225210464 +0400 +++ sql.py 2009-06-08 11:27:23.206210851 +0400 @@ -1718,13 +1725,14 @@ raise SyntaxError, 'SQLField: invalid field name' if isinstance(type, SQLTable): type = 'reference ' + type._tablename - if not length and type == 'string': - type = 'text' - elif not length and type == 'password': - length = 32 - self.type = type # 'string', 'integer' - if type == 'upload': + elif type == 'upload': length = 64 + elif not length: + if type == 'string': + type = 'text' + elif type == 'password': + length = 32 + self.type = type # 'string', 'integer' self.length = length # the length of the string self.default = default # default value for field self.required = required # is this field required @@ -1746,10 +1754,11 @@ self.label = ' '.join([x.capitalize() for x in fieldname.split('_')]) if requires == sqlhtml_validators: - requires = sqlhtml_validators(type, length) + self.requires = sqlhtml_validators(type, length) elif requires is None: - requires = [] - self.requires = requires # list of validators + self.requires = [] + else: + self.requires = requires # list of validators def formatter(self, value): if value is None or not self.requires:
--- /home/snake/Desktop/sql.py 2009-06-08 09:36:27.225210464 +0400 +++ sql.py 2009-06-08 10:26:42.605233069 +0400 @@ -513,13 +513,16 @@ """ def __getitem__(self, key): + value = dict.__getitem__(self, str(key)) + if not callable(value) or key[0]=='_' or not isinstance(value, types.FunctionType): return value +# print 'calling',value + value.__call__() # That must redefine table in-place return dict.__getitem__(self, str(key)) + __getattr__ = __getitem__ def __setitem__(self, key, value): dict.__setitem__(self, str(key), value) - def __getattr__(self, key): - return dict.__getitem__(self,key) def __setattr__(self, key, value): if key in self: @@ -942,14 +945,14 @@ ): for key in args: - if key != 'migrate': + if key not in ('migrate', 'redefine'): raise SyntaxError, 'invalid table attribute: %s' % key migrate = args.get('migrate',True) tablename = cleanup(tablename) - if hasattr(self,tablename) or tablename[0] == '_': - raise SyntaxError, 'invalid table name: %s' % tablename - if tablename in self.tables: + if tablename in self.tables and not args.get('redefine', False): raise SyntaxError, 'table already defined: %s' % tablename + elif tablename[0] == '_' or getattr(SQLDB, tablename, None): + raise SyntaxError, 'invalid table name: %s' % tablename t = self[tablename] = SQLTable(self, tablename, *fields) # db magic