Re: A question on python performance.

2007-09-28 Thread Magnus Lycka
Joe Goldthwaite wrote:
> I didn't know about the getattr function. I tried to search for that
> type of function but not knowing how to word the search request,
> I couldn't find it. 

You should really read through chapter 2 (Built-in Objects) of the
library reference. All that stuff is core Python functionality that
you should be aware of. Reading chapter 3 (Built-in Types) is also
a really good idea. A lot of the rest is good too, but there is
really no excuse for not knowing the contents of those chapters. ;)
It's just a few pages, and very useful to know. Read them now!
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: A question on python performance.

2007-09-28 Thread Bruno Desthuilliers
Joe Goldthwaite a écrit :
(snip)

>  I guess I still think of decorators as the people
> who got the gym ready for the prom.  I've tried getting up to speed on
> decorators but I haven't had much success.

The @decorator syntax is just syntactic sugar for a common use of higher 
order functions. So to understand decorators, you first need understand 
the concepts of higher order functions and closures. Both are documented 
  in many places on the web.

> Bruno Desthuilliers wrote:
> 
>> IOW, direct access to obj.__class__.__dict__ bypasses both inheritence
>> and per-instance overriding.
> 
> Thanks for your response also Bruno. I don't understand the code you
> posted well enough yet to even ask a useful question. There are a number
> of things I haven't seen before.

It's very 101 python code, apart from this line :
   c2.dothis = dothat.__get__(c2, type(c2))

which is a way to dynamically attach a function as method to an 
instance, directly invoking the descriptor protocol (it's documented on 
python.org)

FWIW, it's most often done using the 'new' module, and possibly less 
confusing:

import new
c2.dothis = new.instancemethod(dothat, c2, type(c2))

> Now I really feel like a newbie!

My fault - this example wasn't that necessary, the main point was wrt/ 
direct access to instance.__class__.__dict__ bypassing normal attribute 
lookup rules so inherited attributes are not resolved...

> I'm working on it though so I may have some questions later.

You're welcome.

-- 
http://mail.python.org/mailman/listinfo/python-list


RE: A question on python performance.

2007-09-27 Thread Joe Goldthwaite
[EMAIL PROTECTED] wrote:

>Makes perfect sense to me! Think about it:
>
>method 1: looks up the method directly from the object (fastest)
>method 2: looks up __class__, then looks up __dict__, then gets the
>element from __dict__
>method 3: looks up caller, looks up __class__, looks up __dict__, gets
>element from __dict__
>
>To get the element directly from the object (method 1), Python has to
>internally check __class__.__dict__[element], which shows why method 1
>and method 2 are nearly the same speed. The last version has to look
>up caller in addition to the process described by method 2.
>
>The best way to do what you are doing:
>
>getattr(self, param)(self, *args)

I didn't know about the getattr function. I tried to search for that
type of function but not knowing how to word the search request,
I couldn't find it.  That's a lot cleaner way of doing what I was
trying to do.  I benchmarked it and it's so close in speed to the
hard-coded method as to be identical.

Paul Hankin wrote:

>You're calling a function (getValue) that just calls a method of trend
>(caller), that just calls another method of trend (Ptd or Qtd or ...).
>You can skip all these steps, and just call the method yourself: the
>code that calls getValue(trend, param, per) replace with
>trend.(per) if you're calling getValue with a static value
>for param, or getattr(trend, param)(per) if param is dynamic.

You're right, it makes sense.  Thanks for also pointing out the getattr
function.

Eric Jones wrote:

>What you're describing is a case of mulitple dispatch.  See http://
www.artima.com/weblogs/viewpost.jsp?thread=101605 for a short
>description and (as short) example by Guido.  You can probably fill
>that out and adapt it to your needs.  Alternatively, you could look
>into the multimethods module in the Gnosis Utilities package: http://
>pypi.python.org/pypi/Gnosis_Utils/1.2.1-a

Thanks Eric.  I read that article on multi-methods but I can't say I
really understand it.  I guess I still think of decorators as the people
who got the gym ready for the prom.  I've tried getting up to speed on
decorators but I haven't had much success.  Like I mentioned previously,
with Python, I still feel like a newbie.

Bruno Desthuilliers wrote:

>IOW, direct access to obj.__class__.__dict__ bypasses both inheritence
>and per-instance overriding.

Thanks for your response also Bruno. I don't understand the code you
posted well enough yet to even ask a useful question. There are a number
of things I haven't seen before.  Now I really feel like a newbie!
I'm working on it though so I may have some questions later.

Thanks again for everyone's input. I really appreciate it.






-- 
http://mail.python.org/mailman/listinfo/python-list


Re: A question on python performance.

2007-09-26 Thread Bruno Desthuilliers
Joe Goldthwaite a écrit :
> Hi everyone,
> 
> I'm a developer who's been using python for a couple of years.  I wrote a
> fairly large application using it but I was learning the language at the
> same time so it most of the code kind of sucks.
> 
> I've learned a lot since then and I've been going through my code trying to
> organize it better and make better use of Python's features.  I'm still  not
> an expert by any definition but I'm slowly getting better.
> 
> I've been working on a trend class that takes twelve monthly numbers and
> returns a period to date, quarter to date, year to date and quarterly year
> to date numbers for a specific period. This worked but I ended up with a lot
> of code like this;
> 
> def getValue(trend, param, per):
>if param == 'Ptd':
>  return trend.Ptd(per)
>elif param == 'Qtd':
>   return trend.Qtd(per)
>elif param == 'Ytd':
>   return trend.Ytd(per)
>elif param == 'YtdQ':
>   return trend.YtdQ(per)

The first obvious simplification is to replace this with:

def getValue(trend, param, per):
   meth = getattr(trend, param)
   return meth(per)

The main difference is that it will raise (instead of returning None) if 
param is not the name of a method of trend.

The second simplification is to either get rid of getValue()  (which is 
mostly useless).

> The code gets kind of wordy 

indeed

> so I started trying to figure out how to call
> them dynamically since the param type is the same as the method the
> retrieves it.  I came up with this;
> 
> def getValue(trend, param, per):
>return trend.__class__.__dict__[param](trend, per)

Note that this is not strictly equivalent:
class Parent(object):
 def __init__(self, name):
 self.name = name
 def __repr__(self):
 return "<%s %s>" % (self.__class__.__name__, self.name)

 def dothis(self):
 return "parent.dothis %s" % self

class Child(Parent):
 def dothis(self):
 return "Child.dothis %s" % self

class OtherChild(Parent): pass

def dothat(obj):
 return "dothat %s" % obj

p = Parent('p')
c1 = Child('c1')
c2 = Child('c2')
c2.dothis = dothat.__get__(c2, type(c2))
o1 = OtherChild('o1');
o2 = OtherChild('o2');
o2.dothis = dothat.__get__(o2, type(o2))

for obj in p, c1, c2, o1, o2:
 print "obj : %s" % obj
 print "direct call :"
 print obj.dothis()
 print "via obj.__class__.__dict__ :"
 try:
 print obj.__class__.__dict__["dothis"](obj)
 except KeyError, e:
 print "oops - key error: %s" % e
 print

=>
obj : 
direct call :
parent.dothis 
via obj.__class__.__dict__ :
parent.dothis 

obj : 
direct call :
Child.dothis 
via obj.__class__.__dict__ :
Child.dothis 

obj : 
direct call :
dothat 
via obj.__class__.__dict__ :
Child.dothis 

obj : 
direct call :
parent.dothis 
via obj.__class__.__dict__ :
oops - key error: 'dothis'

obj : 
direct call :
dothat 
via obj.__class__.__dict__ :
oops - key error: 'dothis'


IOW, direct access to obj.__class__.__dict__ bypasses both inheritence 
and per-instance overriding.

> That worked but it seems like the above line would have to do lots more
> object look ups at runtime so I didn't think it would be very efficient.  I
> thought maybe I could add a caller method to the trend class and I came up
> with this;
> 
> class trend:
>...
>...
>...
>def caller(self, param, *args):
>   return self.__class__.__dict__[param](self, *args)
> 
> This simplified the getValue function to this;
> 
> def getValue(trend, param, per):
>   return trend.caller(param, per)

Err... It actually means *more* lookup and function calls - and still 
fails to behave correctly wrt/ polymorphic dispatch.
-- 
http://mail.python.org/mailman/listinfo/python-list


Re: A question on python performance.

2007-09-26 Thread Paul Hankin
On Sep 26, 7:26 pm, "Joe Goldthwaite" <[EMAIL PROTECTED]> wrote:
> The code gets kind of wordy so I started trying to figure out how to call
> them dynamically since the param type is the same as the method the
> retrieves it.  I came up with this;
>
> def getValue(trend, param, per):
>return trend.__class__.__dict__[param](trend, per)
>
> That worked but it seems like the above line would have to do lots more
> object look ups at runtime so I didn't think it would be very efficient.  I
> thought maybe I could add a caller method to the trend class and I came up
> with this;
>
> class trend:
>...
>...
>...
>def caller(self, param, *args):
>   return self.__class__.__dict__[param](self, *args)
>
> This simplified the getValue function to this;
>
> def getValue(trend, param, per):
> return trend.caller(param, per)

You're calling a function (getValue) that just calls a method of trend
(caller), that just calls another method of trend (Ptd or Qtd or ...).
You can skip all these steps, and just call the method yourself: the
code that calls getValue(trend, param, per) replace with
trend.(per) if you're calling getValue with a static value
for param, or getattr(trend, param)(per) if param is dynamic.

--
Paul Hankin

-- 
http://mail.python.org/mailman/listinfo/python-list


Re: A question on python performance.

2007-09-26 Thread Erik Jones
On Sep 26, 2007, at 1:26 PM, Joe Goldthwaite wrote:

> Hi everyone,
>
> I'm a developer who's been using python for a couple of years.  I  
> wrote a
> fairly large application using it but I was learning the language  
> at the
> same time so it most of the code kind of sucks.
>
> I've learned a lot since then and I've been going through my code  
> trying to
> organize it better and make better use of Python's features.  I'm  
> still  not
> an expert by any definition but I'm slowly getting better.
>
> I've been working on a trend class that takes twelve monthly  
> numbers and
> returns a period to date, quarter to date, year to date and  
> quarterly year
> to date numbers for a specific period. This worked but I ended up  
> with a lot
> of code like this;
>
> def getValue(trend, param, per):
>if param == 'Ptd':
>  return trend.Ptd(per)
>elif param == 'Qtd':
>   return trend.Qtd(per)
>elif param == 'Ytd':
>   return trend.Ytd(per)
>elif param == 'YtdQ':
>   return trend.YtdQ(per)
>
> The code gets kind of wordy so I started trying to figure out how  
> to call
> them dynamically since the param type is the same as the method the
> retrieves it.  I came up with this;
>
> def getValue(trend, param, per):
>return trend.__class__.__dict__[param](trend, per)
>
> That worked but it seems like the above line would have to do lots  
> more
> object look ups at runtime so I didn't think it would be very  
> efficient.  I
> thought maybe I could add a caller method to the trend class and I  
> came up
> with this;
>
> class trend:
>...
>...
>...
>def caller(self, param, *args):
>   return self.__class__.__dict__[param](self, *args)
>
> This simplified the getValue function to this;
>
> def getValue(trend, param, per):
>   return trend.caller(param, per)


What you're describing is a case of mulitple dispatch.  See http:// 
www.artima.com/weblogs/viewpost.jsp?thread=101605 for a short  
description and (as short) example by Guido.  You can probably fill  
that out and adapt it to your needs.  Alternatively, you could look  
into the multimethods module in the Gnosis Utilities package: http:// 
pypi.python.org/pypi/Gnosis_Utils/1.2.1-a


Erik Jones

Software Developer | Emma®
[EMAIL PROTECTED]
800.595.4401 or 615.292.5888
615.292.0777 (fax)

Emma helps organizations everywhere communicate & market in style.
Visit us online at http://www.myemma.com


-- 
http://mail.python.org/mailman/listinfo/python-list


Re: A question on python performance.

2007-09-26 Thread chris . monsanto
On Sep 26, 2:26 pm, "Joe Goldthwaite" <[EMAIL PROTECTED]> wrote:
> Hi everyone,
>
> I'm a developer who's been using python for a couple of years.  I wrote a
> fairly large application using it but I was learning the language at the
> same time so it most of the code kind of sucks.
>
> I've learned a lot since then and I've been going through my code trying to
> organize it better and make better use of Python's features.  I'm still  not
> an expert by any definition but I'm slowly getting better.
>
> I've been working on a trend class that takes twelve monthly numbers and
> returns a period to date, quarter to date, year to date and quarterly year
> to date numbers for a specific period. This worked but I ended up with a lot
> of code like this;
>
> def getValue(trend, param, per):
>if param == 'Ptd':
>  return trend.Ptd(per)
>elif param == 'Qtd':
>   return trend.Qtd(per)
>elif param == 'Ytd':
>   return trend.Ytd(per)
>elif param == 'YtdQ':
>   return trend.YtdQ(per)
>
> The code gets kind of wordy so I started trying to figure out how to call
> them dynamically since the param type is the same as the method the
> retrieves it.  I came up with this;
>
> def getValue(trend, param, per):
>return trend.__class__.__dict__[param](trend, per)
>
> That worked but it seems like the above line would have to do lots more
> object look ups at runtime so I didn't think it would be very efficient.  I
> thought maybe I could add a caller method to the trend class and I came up
> with this;
>
> class trend:
>...
>...
>...
>def caller(self, param, *args):
>   return self.__class__.__dict__[param](self, *args)
>
> This simplified the getValue function to this;
>
> def getValue(trend, param, per):
> return trend.caller(param, per)
>
> Out of curiosity, I thought I'd do some benchmarking and see which one
> performs the best. I executed three multiple times;
>
> loop one.  Time=11.71 seconds;
> trend.Ptd(per)
> trend.Qtd(per)
> trend.Ytd(per)
> trend.YtdQ(per)
>
> loop two. 12.107 seconds;
> trend.__class__.__dict__['Ptd'](trend, per)
> trend.__class__.__dict__['Qtd'](trend, per)
> trend.__class__.__dict__['Ytd'](trend, per)
> trend.__class__.__dict__['YtdQ'](trend, per)
>
> loop three. 17.085 seconds;
> trend.caller('Ptd', per)
> trend.caller('Qtd', per)
> trend.caller('Ytd', per)
> trend.caller('YtdQ', per)
>
> The first surprise was how close the first and second loops were.  I would
> have thought the first loop would be much faster.  The second surprise was
> how much slower the third loop was.  I know it has an extra call in there
> but other than that, it's doing basically the same thing as loop two.  Is
> there that much overhead in making a class method call?
>
> Can anyone explain the differences?

Makes perfect sense to me! Think about it:

method 1: looks up the method directly from the object (fastest)
method 2: looks up __class__, then looks up __dict__, then gets the
element from __dict__
method 3: looks up caller, looks up __class__, looks up __dict__, gets
element from __dict__

To get the element directly from the object (method 1), Python has to
internally check __class__.__dict__[element], which shows why method 1
and method 2 are nearly the same speed. The last version has to look
up caller in addition to the process described by method 2.

The best way to do what you are doing:

getattr(self, param)(self, *args)

-- 
http://mail.python.org/mailman/listinfo/python-list


A question on python performance.

2007-09-26 Thread Joe Goldthwaite
Hi everyone,

I'm a developer who's been using python for a couple of years.  I wrote a
fairly large application using it but I was learning the language at the
same time so it most of the code kind of sucks.

I've learned a lot since then and I've been going through my code trying to
organize it better and make better use of Python's features.  I'm still  not
an expert by any definition but I'm slowly getting better.

I've been working on a trend class that takes twelve monthly numbers and
returns a period to date, quarter to date, year to date and quarterly year
to date numbers for a specific period. This worked but I ended up with a lot
of code like this;

def getValue(trend, param, per):
   if param == 'Ptd':
 return trend.Ptd(per)
   elif param == 'Qtd':
  return trend.Qtd(per)
   elif param == 'Ytd':
  return trend.Ytd(per)
   elif param == 'YtdQ':
  return trend.YtdQ(per)

The code gets kind of wordy so I started trying to figure out how to call
them dynamically since the param type is the same as the method the
retrieves it.  I came up with this;

def getValue(trend, param, per):
   return trend.__class__.__dict__[param](trend, per)

That worked but it seems like the above line would have to do lots more
object look ups at runtime so I didn't think it would be very efficient.  I
thought maybe I could add a caller method to the trend class and I came up
with this;

class trend:
   ...
   ...
   ...
   def caller(self, param, *args):
  return self.__class__.__dict__[param](self, *args)

This simplified the getValue function to this;

def getValue(trend, param, per):
return trend.caller(param, per)

Out of curiosity, I thought I'd do some benchmarking and see which one
performs the best. I executed three multiple times;

loop one.  Time=11.71 seconds;
trend.Ptd(per)
trend.Qtd(per)
trend.Ytd(per)
trend.YtdQ(per)

loop two. 12.107 seconds;
trend.__class__.__dict__['Ptd'](trend, per)
trend.__class__.__dict__['Qtd'](trend, per)
trend.__class__.__dict__['Ytd'](trend, per)
trend.__class__.__dict__['YtdQ'](trend, per)

loop three. 17.085 seconds;
trend.caller('Ptd', per)
trend.caller('Qtd', per)
trend.caller('Ytd', per)
trend.caller('YtdQ', per)

The first surprise was how close the first and second loops were.  I would
have thought the first loop would be much faster.  The second surprise was
how much slower the third loop was.  I know it has an extra call in there
but other than that, it's doing basically the same thing as loop two.  Is
there that much overhead in making a class method call?

Can anyone explain the differences?

-- 
http://mail.python.org/mailman/listinfo/python-list