Hi Edward.
I really think for my case, that 25 speed up is a deal breaker !
I have so much data to crunch, that 25 speed is absolutely perfect.
I would only optimise this for CR72, and TSMFK01, since these are the
ones I need now.
And the change of code is only 3-5 lines?
And i was thinking of one thing more.
CR72 always go over loop.
-----------
# Loop over the time points, back calculating the R2eff values.
for i in range(num_points):
# The full eta+/- values.
etapos = etapos_part / cpmg_frqs[i]
etaneg = etaneg_part / cpmg_frqs[i]
# Catch large values of etapos going into the cosh function.
if etapos > 100:
back_calc[i] = 1e100
continue
# The arccosh argument - catch invalid values.
fact = Dpos * cosh(etapos) - Dneg * cos(etaneg)
if fact < 1.0:
back_calc[i] = r20_kex
continue
# The full formula.
back_calc[i] = r20_kex - cpmg_frqs[i] * arccosh(fact)
------------
I would rather do:
etapos = etapos_part / cpmg_frqs
And then check for nan values.
If any of these are there, just return the whole array with 1e100,
instead of single values.
That would replace a loop with a check.
Best
Troels
2014-05-09 14:58 GMT+02:00 Edward d'Auvergne <[email protected]>:
> Hi,
>
> This approach can add a little speed. You really need to stress test
> and have profile timings to understand. You should also try different
> Python versions (2 and 3) because each implementation is different.
> You can sometimes have a speed up in Python 2 which does nothing in
> Python 3 (due to Python 3 being more optimised). There can also be
> huge differences between numpy versions. Anyway, here is a powerful
> test which shows 3 different implementation ideas for the
> back-calculated R2eff data in the dispersion functions:
>
> """
> import cProfile as profile
> from numpy import array, cos, float64, sin, zeros
> import pstats
>
> def in_place(values, bc):
> x = cos(values) * sin(values)
> for i in range(len(bc)):
> bc[i] = x[i]
>
> def really_slow(values, bc):
> for i in range(len(bc)):
> x = cos(values[i]) * sin(values[i])
> bc[i] = x
>
> def return_bc(values):
> return cos(values) * sin(values)
>
> def test_in_place(inc=None, values=None, values2=None, bc=None):
> for i in range(inc):
> in_place(values, bc[0])
> in_place(values2, bc[1])
> print(bc)
>
> def test_really_slow(inc=None, values=None, values2=None, bc=None):
> for i in range(inc):
> really_slow(values, bc[0])
> really_slow(values2, bc[1])
> print(bc)
>
> def test_return_bc(inc=None, values=None, values2=None, bc=None):
> for i in range(inc):
> bc[0] = return_bc(values)
> bc[1] = return_bc(values2)
> print(bc)
>
> def test():
> values = array([1, 3, 0.1], float64)
> values2 = array([0.1, 0.2, 0.3], float64)
> bc = zeros((2, 3), float64)
> inc = 1000000
> test_in_place(inc=inc, values=values, values2=values2, bc=bc)
> test_really_slow(inc=inc, values=values, values2=values2, bc=bc)
> test_return_bc(inc=inc, values=values, values2=values2, bc=bc)
>
> def print_stats(stats, status=0):
> pstats.Stats(stats).sort_stats('time', 'name').print_stats()
> profile.Profile.print_stats = print_stats
> profile.runctx('test()', globals(), locals())
> """"
>
>
> Try running this in Python 2 and 3. If the cProfile import does not
> work on one version, try simply "import profile". You should create
> such scripts for testing out code optimisation ideas. Knowing how to
> profile is essential. For Python 3, I see:
>
> """
> $ python3.4 edward.py
> [[ 0.45464871 -0.13970775 0.09933467]
> [ 0.09933467 0.19470917 0.28232124]]
> [[ 0.45464871 -0.13970775 0.09933467]
> [ 0.09933467 0.19470917 0.28232124]]
> [[ 0.45464871 -0.13970775 0.09933467]
> [ 0.09933467 0.19470917 0.28232124]]
> 10001042 function calls (10001036 primitive calls) in 39.303 seconds
>
> Ordered by: internal time, function name
>
> ncalls tottime percall cumtime percall filename:lineno(function)
> 2000000 19.744 0.000 19.867 0.000 edward.py:10(really_slow)
> 2000000 9.128 0.000 9.286 0.000 edward.py:5(in_place)
> 2000000 5.966 0.000 5.966 0.000 edward.py:15(return_bc)
> 1 1.964 1.964 7.931 7.931 edward.py:30(test_return_bc)
> 1 1.119 1.119 20.987 20.987 edward.py:24(test_really_slow)
> 1 1.099 1.099 10.385 10.385 edward.py:18(test_in_place)
> 4000198 0.281 0.000 0.281 0.000 {built-in method len}
> 27 0.001 0.000 0.001 0.000 {method 'reduce' of
> 'numpy.ufunc' objects}
> """
>
> Here you can see that test_return_bc() is 80% the speed of
> test_in_place(). The 'cumtime' is the important number, this is the
> total amount of time spent in that function. So the speed up is not
> huge. For Python 2:
>
> """
> $ python2.7 edward.py
> [[ 0.45464871 -0.13970775 0.09933467]
> [ 0.09933467 0.19470917 0.28232124]]
> [[ 0.45464871 -0.13970775 0.09933467]
> [ 0.09933467 0.19470917 0.28232124]]
> [[ 0.45464871 -0.13970775 0.09933467]
> [ 0.09933467 0.19470917 0.28232124]]
> 14000972 function calls (14000966 primitive calls) in 38.625 seconds
>
> Ordered by: internal time, function name
>
> ncalls tottime percall cumtime percall filename:lineno(function)
> 2000000 18.373 0.000 19.086 0.000 edward.py:10(really_slow)
> 2000000 8.798 0.000 9.576 0.000 edward.py:5(in_place)
> 2000000 5.937 0.000 5.937 0.000 edward.py:15(return_bc)
> 1 1.839 1.839 7.785 7.785 edward.py:30(test_return_bc)
> 4000021 1.141 0.000 1.141 0.000 {range}
> 1 1.086 1.086 10.675 10.675 edward.py:18(test_in_place)
> 1 1.070 1.070 20.165 20.165 edward.py:24(test_really_slow)
> 4000198 0.379 0.000 0.379 0.000 {len}
> """
>
> Hmmm, Python 2 is faster than Python 3 for this example! See for
> yourself. If you really think that making the code 1.25 times faster,
> as shown in these tests, is worth your time, then this must be done in
> a subversion branch (http://svn.gna.org/viewcvs/relax/branches/).
> That way we can have timing tests between the trunk and the branch.
> As this affects all dispersion models, the changes will be quite
> disruptive. And if the implementation is not faster or if it breaks
> everything, then the branch can be deleted. What ever you do, please
> don't use a git-svn branch.
>
> Regards,
>
> Edward
>
>
>
>
> On 9 May 2014 14:07, Troels Emtekær Linnet <[email protected]> wrote:
>> Hi Edward.
>>
>> How about this script?
>> Here I try to pass the back the r2eff values, and then set them in the
>> back_calculated class object.
>> Will this work ??
>>
>> Or else I found this post about updating values.
>> http://stackoverflow.com/questions/14916284/in-python-class-object-how-to-auto-update-attributes
>> They talk about
>> @property
>> and setter, which I dont get yet. :-)
>>
>> Best
>> Troels
>>
>>
>> ---------------
>>
>>
>> def loop_rep(x, nr):
>> y = [98, 99]
>> for i in range(nr):
>> x[i] = y[i]
>>
>> def not_loop_rep(x, nr):
>> y = [98, 99]
>> x = y
>>
>> def not_loop_rep_new(x, nr):
>> y = [98, 99]
>> x = y
>> return x
>>
>>
>> class MyClass:
>> def __init__(self, x):
>> self.x = x
>> self.nr = len(x)
>>
>> def printc(self):
>> print self.x, self.nr
>>
>> def calc_loop_rep(self, x=None, nr=None):
>> loop_rep(x=self.x, nr=self.nr)
>>
>> def calc_not_loop_rep(self, x=None, nr=None):
>> not_loop_rep(x=self.x, nr=self.nr)
>>
>> def calc_not_loop_rep_new(self, x=None, nr=None):
>> self.x = not_loop_rep_new(x=self.x, nr=self.nr)
>>
>> print("For class where we loop replace ")
>> "Create object of class"
>> t_rep = MyClass([0, 1])
>> "Print object of class"
>> t_rep.printc()
>> "Calc object of class"
>> t_rep.calc_loop_rep()
>> " Then print"
>> t_rep.printc()
>>
>> print("\nFor class where we not loop replace ")
>> " Now try with replace "
>> t = MyClass([3, 4])
>> t.printc()
>> t.calc_not_loop_rep()
>> t.printc()
>>
>> print("\nFor class where we not loop replace ")
>> t_new = MyClass([5, 6])
>> t_new.printc()
>> t_new.calc_not_loop_rep_new()
>> t_new.printc()
>>
>> 2014-05-05 19:07 GMT+02:00 Edward d'Auvergne <[email protected]>:
>>> :) It does slow it down a little, but that's unavoidable. It's also
>>> unavoidable in C, Fortran, Perl, etc. As long as the number of
>>> operations in that loop is minimal, then it's the best you can do. If
>>> it worries you, you could run a test where you call the target
>>> function say 1e6 times, with and without the loop to see the timing
>>> difference. Or simply running in Python 2:
>>>
>>> for i in xrange(1000000):
>>> x = 1
>>>
>>> Then try:
>>>
>>> for i in xrange(100000000):
>>> x = 2
>>>
>>> These two demonstrate the slowness of the Python loop. But the second
>>> case is extreme and you won't encounter that much looping in these
>>> functions. So while it is theoretically slower than C and Fortran
>>> looping, you can probably see that no one would care :) Here is
>>> another test, with Python 2 code:
>>>
>>> """
>>> import cProfile as profile
>>>
>>> def loop_1e6():
>>> for i in xrange(int(1e6)):
>>> x = 1
>>>
>>> def loop_1e8():
>>> for i in xrange(int(1e8)):
>>> x = 1
>>>
>>> def sum_conv():
>>> for i in xrange(100000000):
>>> x = 2 + 2.
>>>
>>> def sum_normal():
>>> for i in xrange(100000000):
>>> x = 2. + 2.
>>>
>>> def test():
>>> loop_1e6()
>>> loop_1e8()
>>> sum_normal()
>>> sum_conv()
>>>
>>> profile.runctx('test()', globals(), locals())
>>> """
>>>
>>> Running this on my system shows:
>>>
>>> """
>>> 7 function calls in 6.707 seconds
>>>
>>> Ordered by: standard name
>>>
>>> ncalls tottime percall cumtime percall filename:lineno(function)
>>> 1 0.000 0.000 6.707 6.707 <string>:1(<module>)
>>> 1 2.228 2.228 2.228 2.228 aaa.py:11(sum_conv)
>>> 1 2.228 2.228 2.228 2.228 aaa.py:15(sum_normal)
>>> 1 0.000 0.000 6.707 6.707 aaa.py:19(test)
>>> 1 0.022 0.022 0.022 0.022 aaa.py:3(loop_1e6)
>>> 1 2.228 2.228 2.228 2.228 aaa.py:7(loop_1e8)
>>> 1 0.000 0.000 0.000 0.000 {method 'disable' of
>>> '_lsprof.Profiler' objects}
>>> """
>>>
>>> That should be self explanatory. The better optimisation targets are
>>> the repeated maths operations and the maths operations that can be
>>> shifted into the target function or the target function
>>> initialisation. Despite the numbers above which prove my int to float
>>> speed argument as utter nonsense, it might still good to remove the
>>> int to float conversions, if only to match the other functions.
>>>
>>> Regards,
>>>
>>> Edward
>>>
>>>
>>>
>>>
>>> On 5 May 2014 18:45, Troels Emtekær Linnet <[email protected]> wrote:
>>>> The reason why I ask, is that I am afraid that this for loop slows
>>>> everything down.
>>>>
>>>> What do you think?
>>>>
>>>> 2014-05-05 18:41 GMT+02:00 Edward d'Auvergne <[email protected]>:
>>>>> This is not Python specific though :) As far as I know, C uses
>>>>> pass-by-value for arguments, unless they are arrays or other funky
>>>>> objects/functions/etc.. This is the same behaviour as Python.
>>>>> Pass-by-reference and pass-by-value is something that needs to be
>>>>> mastered in all languages, whether or not you have pointers to play
>>>>> with.
>>>>>
>>>>> Regards,
>>>>>
>>>>> Edward
>>>>>
>>>>>
>>>>>
>>>>> On 5 May 2014 18:30, Troels Emtekær Linnet <[email protected]> wrote:
>>>>>> This reminds me:
>>>>>>
>>>>>> http://combichem.blogspot.dk/2013/08/you-know-what-really-grinds-my-gears-in.html
>>>>>>
>>>>>> 2014-05-05 17:52 GMT+02:00 Edward d'Auvergne <[email protected]>:
>>>>>>> Hi,
>>>>>>>
>>>>>>> This is an important difference. In the first case (back_calc[i] =
>>>>>>> Minty[i]), what is happening is that your are copying the data into a
>>>>>>> pre-existing structure. In the second case (back_calc = Minty), the
>>>>>>> existing back_calc structure will be overwritten. Therefore the
>>>>>>> back_calc structure in the calling code will be modified in the first
>>>>>>> case but not the second. Here is some demo code:
>>>>>>>
>>>>>>> def mod1(x):
>>>>>>> x[0] = 1
>>>>>>>
>>>>>>> def mod2(x):
>>>>>>> x = [3, 2]
>>>>>>>
>>>>>>> x = [0, 2]
>>>>>>> print(x)
>>>>>>> mod1(x)
>>>>>>> print(x)
>>>>>>> mod2(x)
>>>>>>> print(x)
>>>>>>>
>>>>>>> I don't know of a way around this.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Edward
>>>>>>>
>>>>>>>
>>>>>>> On 5 May 2014 17:42, Troels Emtekær Linnet <[email protected]>
>>>>>>> wrote:
>>>>>>>> Hi Edward.
>>>>>>>>
>>>>>>>> In the library function of b14.py, i am looping over
>>>>>>>> the dispersion points to put in the data.
>>>>>>>>
>>>>>>>> for i in range(num_points):
>>>>>>>>
>>>>>>>> # The full formula.
>>>>>>>> back_calc[i] = Minty[i]
>>>>>>>>
>>>>>>>> Why can I not just set:
>>>>>>>> back_calc = Minty
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> relax (http://www.nmr-relax.com)
>>>>>>>>
>>>>>>>> This is the relax-devel mailing list
>>>>>>>> [email protected]
>>>>>>>>
>>>>>>>> To unsubscribe from this list, get a password
>>>>>>>> reminder, or change your subscription options,
>>>>>>>> visit the list information page at
>>>>>>>> https://mail.gna.org/listinfo/relax-devel
_______________________________________________
relax (http://www.nmr-relax.com)
This is the relax-devel mailing list
[email protected]
To unsubscribe from this list, get a password
reminder, or change your subscription options,
visit the list information page at
https://mail.gna.org/listinfo/relax-devel
