Re: [Tutor] improving speed using and recalling C functions
Gabriele Brambilla wrote:
Ok guys, when I wrote that email I was excited for the apparent speed
increasing (it was jumping the bottleneck for loop for the reason peter
otten outlined).
Now, instead the changes, the speed is not improved (the code still
running from this morning and it's at one forth of the dataset).
What can I do to speed it up?
Not as easy as I had hoped and certainly not as pretty, here's my
modification of the code you sent me. What makes it messy is that
I had to inline your kappa() function; my first attempt with
numpy.vectorize() didn't help much. There is still stuff in the
'for gammar...' loop that doesn't belong there, but I decided it
was time for me to stop ;)
Note that it may still be worthwhile to consult a numpy expert
(which I'm not!).
from scipy import stats
import matplotlib.pyplot as plt
from scipy import optimize
from matplotlib import colors, ticker, cm
import numpy as np
phamin = 0
phamax = 2*pi
obamin = 0
obamax = pi
npha = 100
nobs = 181
stepPHA = (phamax-phamin)/npha
stepOB = (obamax-obamin)/nobs
freq = 10
c = 2.9979*(10**(10))
e = 4.8032*(10**(-10))
hcut = 1.0546*(10**(-27))
eVtoErg = 1.6022*(10**(-12))
from math import *
import numpy as np
from scipy.interpolate import interp1d
kaparg = [
-3.0, -2.0, -1.52287875, -1.22184875, -1.0, -0.69897,
-0.52287875, -0.39794001, -0.30103, -0.22184875,
-0.15490196, 0.0, 0.30103, 0.60205999, 0.69897,
0.77815125, 0.90308999, 1.0]
kapval = [
-0.6716204 , -0.35163999, -0.21183163, -0.13489603,
-0.0872467 , -0.04431225, -0.03432803, -0.04335142,
-0.05998184, -0.08039898, -0.10347378, -0.18641901,
-0.52287875, -1.27572413, -1.66958623, -2.07314329,
-2.88941029, -3.7212464 ]
my_inter = interp1d(kaparg, kapval)
def LEstep(n):
Emin = 10**6
Emax = 5*(10**10)
Lemin = log10(Emin)
Lemax = log10(Emax)
stepE = (Lemax-Lemin)/n
return stepE, n, Lemin, Lemax
def mymain(stepENE, nex, Lemin, Lemax, freq):
eel = np.array(list(range(nex)))
eels = np.logspace(Lemin, Lemax, num=nex, endpoint=False)
rlc = c/(2*pi*freq)
sigmas = [1, 3, 5, 10, 30]
MYMAPS = [
np.zeros([npha, nobs, nex], dtype=float) for _ in sigmas]
alpha = '60_'
ALPHA = (1.732050808/c)*(e**2)
for count, my_line in enumerate(open('datasm0_60_5s.dat')):
myinternet = []
print('reading the line', count, '/599378')
my_parts = np.array(my_line.split(), dtype=float)
phase = my_parts[4]
zobs = my_parts[5]
rho = my_parts[6]
gmils = my_parts[7:12]
i = int((phase-phamin)/stepPHA)
j = int((zobs-obamin)/stepOB)
for gammar, MYMAP in zip(gmils, MYMAPS):
omC = (1.5)*(gammar**3)*c/(rho*rlc)
gig = omC*hcut/eVtoErg
omega = (10**(eel*stepENE+Lemin))*eVtoErg/hcut
x = omega/omC
kap = np.empty(x.shape)
sel = x = 10.0
zsel = x[sel]
kap[sel] = 1.2533 * np.sqrt(zsel)*np.exp(-zsel)
sel = x 0.001
zsel = x[sel]
kap[sel] = (2.1495 * np.exp(0.3 * np.log(zsel))
- 1.8138 * zsel)
sel = ~ ((x = 10.0) | (x 0.001))
zsel = x[sel]
result = my_inter(np.log10(zsel))
kap[sel] = 10**result
Iom = ALPHA*gammar*kap
P = Iom*(c/(rho*rlc))/(2*pi)
phps = P/(hcut*omega)
www = phps/(stepPHA*sin(zobs)*stepOB)
MYMAP[i,j] += www
for sigma, MYMAP in zip(sigmas, MYMAPS):
print(sigma)
filename = _.join(str(p) for p in
[skymap, alpha, sigma, npha, phamin, phamax, nobs,
obamin, obamax, nex, Lemin, Lemax, '.dat']
)
x, y, z = MYMAP.shape
with open(filename, 'ab') as MYfile:
np.savetxt(
MYfile,
MYMAP.reshape(x*y, z, order=F).T,
delimiter=,, fmt=%s, newline=,\n)
if __name__ == __main__:
if len(sys.argv)=1:
stepENE, nex, Lemin, Lemax = LEstep(200)
elif len(sys.argv)=2:
stepENE, nex, Lemin, Lemax = LEstep(int(sys.argv[1]))
else:
stepENE, nex, Lemin, Lemax = LEstep(int(sys.argv[1]))
freq=float(sys.argv[2])
mymain(stepENE, nex, Lemin, Lemax, freq)
For reference here is the original (with the loop over gmlis
instead of gmils):
import sys
from math import *
from scipy import ndimage
from scipy import stats
import matplotlib.pyplot as plt
from scipy import optimize
from matplotlib import colors, ticker, cm
import numpy as np
import cProfile
import pstats
phamin=0
phamax=2*pi
obamin=0
obamax=pi
npha=100
nobs=181
stepPHA=(phamax-phamin)/npha
stepOB=(obamax-obamin)/nobs
freq=10
c=2.9979*(10**(10))
e=4.8032*(10**(-10))
hcut=1.0546*(10**(-27))
eVtoErg=1.6022*(10**(-12))
from math import *
import numpy as np
from scipy.interpolate import interp1d
Re: [Tutor] improving speed using and recalling C functions
Ok guys,
I'm not expert about profile but help me to look at it.
this one is for 715853 elements (to multiply by 5, and for each of this N*5
there is a loop of 200 times)
Sat Apr 12 04:58:50 2014restats
9636507991 function calls in 66809.764 seconds
Ordered by: internal time
List reduced from 47 to 20 due to restriction 20
ncallstottimepercallcumtime percall
filename:lineno(function)
1 13548.507 13548.507 66809.692 66809.692
skymapsI.py:44(mymain)
125800544 13539.337 0.00015998.9250.000
interpolate.py:394(_call_linear)
880603808 5353.3820.000 5353.3820.000
{numpy.core.multiarray.array}
715853000 4998.7400.000 52861.634 0.000
instruments.py:10(kappa)
251601088 4550.9400.000 4550.9400.000{method 'reduce'
of 'numpy.ufunc' objects}
125800544 4312.0780.00010163.6140.000
interpolate.py:454(_check_bounds)
125800544 2944.1260.00014182.9170.000
interpolate.py:330(__init__)
125800544 2846.5770.000 29484.2480.000
interpolate.py:443(_evaluate)
125800544 1665.8520.0006000.6030.000 polyint.py:82(_set_yi)
125800544 1039.4550.000 1039.4550.000 {method 'clip' of
'numpy.ndarray' objects}
251601088 944.8480.000 944.8480.000 {method 'reshape' of
'numpy.ndarray' objects}
251601088 922.9280.000 1651.2180.000
numerictypes.py:735(issubdtype)
503202176 897.0440.000 3434.7680.000
numeric.py:392(asarray)
125800544 816.4010.000 32242.4810.000
polyint.py:37(__call__)
251601088 787.5930.000 5338.5330.000 _methods.py:31(_any)
125800544 689.7790.000 1989.1010.000
polyint.py:74(_reshape_yi)
125800544 638.9460.000 638.9460.000 {method
'searchsorted' of 'numpy.ndarray' objects}
125800544 606.7780.000 2257.9960.000
polyint.py:102(_set_dtype)
125800544 598.0000.0006598.6020.000
polyint.py:30(__init__)
629002720 549.3580.000 549.3580.000 {issubclass}
looking at tottime it seems that skymaps mymain() and interpolate take the
same big amount of time...right?
So it's true that I have to slow down mymain() but interpolate is a problem
too!
do you agree with me?
Now I will read Peter Otten's code and run the new simulation with it
thanks
Gabriele
2014-04-12 6:21 GMT-04:00 Peter Otten __pete...@web.de:
Gabriele Brambilla wrote:
Ok guys, when I wrote that email I was excited for the apparent speed
increasing (it was jumping the bottleneck for loop for the reason peter
otten outlined).
Now, instead the changes, the speed is not improved (the code still
running from this morning and it's at one forth of the dataset).
What can I do to speed it up?
Not as easy as I had hoped and certainly not as pretty, here's my
modification of the code you sent me. What makes it messy is that
I had to inline your kappa() function; my first attempt with
numpy.vectorize() didn't help much. There is still stuff in the
'for gammar...' loop that doesn't belong there, but I decided it
was time for me to stop ;)
Note that it may still be worthwhile to consult a numpy expert
(which I'm not!).
from scipy import stats
import matplotlib.pyplot as plt
from scipy import optimize
from matplotlib import colors, ticker, cm
import numpy as np
phamin = 0
phamax = 2*pi
obamin = 0
obamax = pi
npha = 100
nobs = 181
stepPHA = (phamax-phamin)/npha
stepOB = (obamax-obamin)/nobs
freq = 10
c = 2.9979*(10**(10))
e = 4.8032*(10**(-10))
hcut = 1.0546*(10**(-27))
eVtoErg = 1.6022*(10**(-12))
from math import *
import numpy as np
from scipy.interpolate import interp1d
kaparg = [
-3.0, -2.0, -1.52287875, -1.22184875, -1.0, -0.69897,
-0.52287875, -0.39794001, -0.30103, -0.22184875,
-0.15490196, 0.0, 0.30103, 0.60205999, 0.69897,
0.77815125, 0.90308999, 1.0]
kapval = [
-0.6716204 , -0.35163999, -0.21183163, -0.13489603,
-0.0872467 , -0.04431225, -0.03432803, -0.04335142,
-0.05998184, -0.08039898, -0.10347378, -0.18641901,
-0.52287875, -1.27572413, -1.66958623, -2.07314329,
-2.88941029, -3.7212464 ]
my_inter = interp1d(kaparg, kapval)
def LEstep(n):
Emin = 10**6
Emax = 5*(10**10)
Lemin = log10(Emin)
Lemax = log10(Emax)
stepE = (Lemax-Lemin)/n
return stepE, n, Lemin, Lemax
def mymain(stepENE, nex, Lemin, Lemax, freq):
eel = np.array(list(range(nex)))
eels = np.logspace(Lemin, Lemax, num=nex, endpoint=False)
rlc = c/(2*pi*freq)
sigmas = [1, 3, 5, 10, 30]
MYMAPS = [
np.zeros([npha, nobs, nex], dtype=float) for _ in sigmas]
alpha = '60_'
ALPHA = (1.732050808/c)*(e**2)
for count, my_line in enumerate(open('datasm0_60_5s.dat')):
myinternet = []
print('reading the line', count, '/599378')
Re: [Tutor] improving speed using and recalling C functions
Ok, i just run Peter's code and it seems really faster...I hope to don't
mistake this time!
Thanks
Gabriele
sent from Samsung Mobile
Il giorno 12/apr/2014 08:22, Gabriele Brambilla
gb.gabrielebrambi...@gmail.com ha scritto:
Ok guys,
I'm not expert about profile but help me to look at it.
this one is for 715853 elements (to multiply by 5, and for each of this
N*5 there is a loop of 200 times)
Sat Apr 12 04:58:50 2014restats
9636507991 function calls in 66809.764 seconds
Ordered by: internal time
List reduced from 47 to 20 due to restriction 20
ncallstottimepercallcumtime percall
filename:lineno(function)
1 13548.507 13548.507 66809.692 66809.692
skymapsI.py:44(mymain)
125800544 13539.337 0.00015998.9250.000
interpolate.py:394(_call_linear)
880603808 5353.3820.000 5353.3820.000
{numpy.core.multiarray.array}
715853000 4998.7400.000 52861.634 0.000
instruments.py:10(kappa)
251601088 4550.9400.000 4550.9400.000{method 'reduce'
of 'numpy.ufunc' objects}
125800544 4312.0780.00010163.6140.000
interpolate.py:454(_check_bounds)
125800544 2944.1260.00014182.9170.000
interpolate.py:330(__init__)
125800544 2846.5770.000 29484.2480.000
interpolate.py:443(_evaluate)
125800544 1665.8520.0006000.6030.000 polyint.py:82(_set_yi)
125800544 1039.4550.000 1039.4550.000 {method 'clip' of
'numpy.ndarray' objects}
251601088 944.8480.000 944.8480.000 {method 'reshape'
of 'numpy.ndarray' objects}
251601088 922.9280.000 1651.218
0.000numerictypes.py:735(issubdtype)
503202176 897.0440.000 3434.7680.000
numeric.py:392(asarray)
125800544 816.4010.000 32242.4810.000
polyint.py:37(__call__)
251601088 787.5930.000 5338.5330.000 _methods.py:31(_any)
125800544 689.7790.000 1989.1010.000
polyint.py:74(_reshape_yi)
125800544 638.9460.000 638.9460.000 {method
'searchsorted' of 'numpy.ndarray' objects}
125800544 606.7780.000 2257.9960.000
polyint.py:102(_set_dtype)
125800544 598.0000.0006598.6020.000
polyint.py:30(__init__)
629002720 549.3580.000 549.3580.000 {issubclass}
looking at tottime it seems that skymaps mymain() and interpolate take the
same big amount of time...right?
So it's true that I have to slow down mymain() but interpolate is a
problem too!
do you agree with me?
Now I will read Peter Otten's code and run the new simulation with it
thanks
Gabriele
2014-04-12 6:21 GMT-04:00 Peter Otten __pete...@web.de:
Gabriele Brambilla wrote:
Ok guys, when I wrote that email I was excited for the apparent speed
increasing (it was jumping the bottleneck for loop for the reason peter
otten outlined).
Now, instead the changes, the speed is not improved (the code still
running from this morning and it's at one forth of the dataset).
What can I do to speed it up?
Not as easy as I had hoped and certainly not as pretty, here's my
modification of the code you sent me. What makes it messy is that
I had to inline your kappa() function; my first attempt with
numpy.vectorize() didn't help much. There is still stuff in the
'for gammar...' loop that doesn't belong there, but I decided it
was time for me to stop ;)
Note that it may still be worthwhile to consult a numpy expert
(which I'm not!).
from scipy import stats
import matplotlib.pyplot as plt
from scipy import optimize
from matplotlib import colors, ticker, cm
import numpy as np
phamin = 0
phamax = 2*pi
obamin = 0
obamax = pi
npha = 100
nobs = 181
stepPHA = (phamax-phamin)/npha
stepOB = (obamax-obamin)/nobs
freq = 10
c = 2.9979*(10**(10))
e = 4.8032*(10**(-10))
hcut = 1.0546*(10**(-27))
eVtoErg = 1.6022*(10**(-12))
from math import *
import numpy as np
from scipy.interpolate import interp1d
kaparg = [
-3.0, -2.0, -1.52287875, -1.22184875, -1.0, -0.69897,
-0.52287875, -0.39794001, -0.30103, -0.22184875,
-0.15490196, 0.0, 0.30103, 0.60205999, 0.69897,
0.77815125, 0.90308999, 1.0]
kapval = [
-0.6716204 , -0.35163999, -0.21183163, -0.13489603,
-0.0872467 , -0.04431225, -0.03432803, -0.04335142,
-0.05998184, -0.08039898, -0.10347378, -0.18641901,
-0.52287875, -1.27572413, -1.66958623, -2.07314329,
-2.88941029, -3.7212464 ]
my_inter = interp1d(kaparg, kapval)
def LEstep(n):
Emin = 10**6
Emax = 5*(10**10)
Lemin = log10(Emin)
Lemax = log10(Emax)
stepE = (Lemax-Lemin)/n
return stepE, n, Lemin, Lemax
def mymain(stepENE, nex, Lemin, Lemax, freq):
eel = np.array(list(range(nex)))
eels = np.logspace(Lemin, Lemax, num=nex, endpoint=False)
rlc = c/(2*pi*freq)
sigmas = [1, 3, 5, 10, 30]
Re: [Tutor] improving speed using and recalling C functions
ok Peter Otten code works (very fast),
and this is the profile
Sat Apr 12 11:15:39 2014restats
92834776 function calls in 6218.782 seconds
Ordered by: internal time
List reduced from 41 to 20 due to restriction 20
ncalls tottime percall cumtime percall filename:lineno(function)
1 5301.641 5301.641 6218.763 6218.763 skymapsIO.py:50(mymain)
3489985 380.4690.000 452.4780.000
interpolate.py:394(_call_linear)
3489985 98.1860.000 227.2290.000
interpolate.py:454(_check_bounds)
6979970 96.5670.000 96.5670.000 {method 'reduce' of
'numpy.ufunc'objects}
3489985 44.8530.000 738.1350.000 interpolate.py:443(_evaluate)
7677978 41.0100.000 41.0100.000 {numpy.core.multiarray.array}
5 40.4308.086 40.6218.124 npyio.py:882(savetxt)
3489985 26.9520.000 26.9520.000 {method 'clip' of
'numpy.ndarray'objects}
3489985 24.7490.000 24.7490.000 {method 'searchsorted' of
'numpy.ndarray' objects}
3489985 22.4570.000 828.2380.000 polyint.py:37(__call__)
6979970 19.7200.000 116.2870.000 _methods.py:31(_any)
6979980 15.3300.000 35.0920.000 numeric.py:392(asarray)
3489985 14.8470.000 45.0390.000 polyint.py:57(_prepare_x)
3489990 12.9040.000 12.9040.000 {method 'reshape' of
'numpy.ndarray' objects}
6979970 12.7570.000 129.0440.000 {method 'any' of
'numpy.ndarray' objects}
3489985 11.6240.000 11.6240.000 {method 'astype' of
'numpy.ndarray' objects}
3489985 10.0770.000 10.0770.000 {numpy.core.multiarray.empty}
34899859.9450.000 22.6070.000 polyint.py:63(_finish_y)
34899857.0510.0007.0510.000 {method 'ravel' of
'numpy.ndarray' objects}
6979986.7460.0006.7460.000 {zip}
So I think that in this way it's ok.
Thank you all very much,
Gabriele
p.s: I didn't know this way to write: is there a tutorial for this kind of
operations?
kap = np.empty(x.shape)
sel = x = 10.0
zsel = x[sel]
kap[sel] = 1.2533 * np.sqrt(zsel)*np.exp(-zsel)
sel = x 0.001
zsel = x[sel]
kap[sel] = (2.1495 * np.exp(0.3 * np.log(zsel))
- 1.8138 * zsel)
sel = ~ ((x = 10.0) | (x 0.001))
zsel = x[sel]
result = my_inter(np.log10(zsel))
kap[sel] = 10**result
2014-04-12 9:34 GMT-04:00 Gabriele Brambilla gb.gabrielebrambi...@gmail.com
:
Ok, i just run Peter's code and it seems really faster...I hope to don't
mistake this time!
Thanks
Gabriele
sent from Samsung Mobile
Il giorno 12/apr/2014 08:22, Gabriele Brambilla
gb.gabrielebrambi...@gmail.com ha scritto:
Ok guys,
I'm not expert about profile but help me to look at it.
this one is for 715853 elements (to multiply by 5, and for each of this
N*5 there is a loop of 200 times)
Sat Apr 12 04:58:50 2014restats
9636507991 function calls in 66809.764 seconds
Ordered by: internal time
List reduced from 47 to 20 due to restriction 20
ncallstottimepercallcumtime percall
filename:lineno(function)
1 13548.507 13548.507 66809.692 66809.692
skymapsI.py:44(mymain)
125800544 13539.337 0.00015998.9250.000
interpolate.py:394(_call_linear)
880603808 5353.3820.000 5353.3820.000
{numpy.core.multiarray.array}
715853000 4998.7400.000 52861.634 0.000
instruments.py:10(kappa)
251601088 4550.9400.000 4550.9400.000{method 'reduce'
of 'numpy.ufunc' objects}
125800544 4312.0780.00010163.6140.000
interpolate.py:454(_check_bounds)
125800544 2944.1260.00014182.9170.000
interpolate.py:330(__init__)
125800544 2846.5770.000 29484.2480.000
interpolate.py:443(_evaluate)
125800544 1665.8520.0006000.6030.000
polyint.py:82(_set_yi)
125800544 1039.4550.000 1039.4550.000 {method 'clip' of
'numpy.ndarray' objects}
251601088 944.8480.000 944.8480.000 {method 'reshape'
of 'numpy.ndarray' objects}
251601088 922.9280.000 1651.218
0.000numerictypes.py:735(issubdtype)
503202176 897.0440.000 3434.7680.000
numeric.py:392(asarray)
125800544 816.4010.000 32242.4810.000
polyint.py:37(__call__)
251601088 787.5930.000 5338.5330.000
_methods.py:31(_any)
125800544 689.7790.000 1989.1010.000
polyint.py:74(_reshape_yi)
125800544 638.9460.000 638.9460.000 {method
'searchsorted' of 'numpy.ndarray' objects}
125800544 606.7780.000 2257.9960.000
polyint.py:102(_set_dtype)
125800544 598.0000.0006598.6020.000
polyint.py:30(__init__)
629002720 549.3580.000 549.3580.000
Re: [Tutor] improving speed using and recalling C functions
Gabriele Brambilla wrote: Anyway I would like to try to speed it up using C functions (and maybe comparing the resuts of the two profile in the end) I can't help you on your chosen path, but let me emphasise that the code you posted looks like it has great potential for speed-up by replacing the inner loops with numpy array operations. If you post a small dataset somewhere and a version of the code that can run standalone (no undefined variables or libraries, no commandline arguments) I might even tinker with it myself to demonstrate this potential... ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
From: Gabriele Brambilla gb.gabrielebrambi...@gmail.com
To: Danny Yoo d...@hashcollision.org
Cc: python tutor tutor@python.org
Sent: Friday, April 11, 2014 5:30 AM
Subject: Re: [Tutor] improving speed using and recalling C functions
Hi Danny,
I followed your suggestion.
Tomorrow morning I will run this new version of the code.
Now using a sample of 81 elements (instead of 60) the profile returns:
Thu Apr 10 23:25:59 2014 restats
18101188 function calls in 1218.626 seconds
Ordered by: internal time
List reduced from 13 to 10 due to restriction 10
ncalls tottime percall cumtime percall filename:lineno(function)
1 1015.803 1015.803 1218.334 1218.334 skymaps5.py:44(mymain)
18101000 202.490 0.000 202.490 0.000 {method 'write' of 'file'
objects}
1 0.292 0.292 1218.626 1218.626 string:1(module)
6 0.029 0.005 0.029 0.005 {open}
5 0.010 0.002 0.010 0.002 {method 'close' of 'file'
objects}
81 0.002 0.000 0.002 0.000 {method 'split' of 'str' objects}
82 0.001 0.000 0.001 0.000 {zip}
1 0.000 0.000 0.000 0.000 function_base.py:8(linspace)
1 0.000 0.000 0.000 0.000 function_base.py:93(logspace)
5 0.000 0.000 0.000 0.000 {numpy.core.multiarray.zeros}
Anyway I would like to try to speed it up using C functions (and maybe
comparing the resuts of the two profile in the end)
How can I do it now? Can I use Cython?
If you have a compiler installed already it's just easy_install cython. Writing
Cython is not hard. That is, you easily get speed improvements. It's in a .pyx
file and once you're done you generate the .c and .so/.dll files with a
setup.py like below. Reason why I am posting this snippet is the way to
generate an annotated html file of your cython code. The whiter, the more in C,
the better. Yellow means stuff might still be improved more.
* setup.py
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
import Cython.Compiler.Options
Cython.Compiler.Options.annotate = True # really handy
setup(
cmdclass = {'build_ext': build_ext},
ext_modules = [Extension(myModule, [myModule.pyx])]
)
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
I think I have Cython already installed with Anaconda.
How it works?
Thanks
Gabriele
2014-04-11 8:16 GMT-04:00 Albert-Jan Roskam fo...@yahoo.com:
From: Gabriele Brambilla gb.gabrielebrambi...@gmail.com
To: Danny Yoo d...@hashcollision.org
Cc: python tutor tutor@python.org
Sent: Friday, April 11, 2014 5:30 AM
Subject: Re: [Tutor] improving speed using and recalling C functions
Hi Danny,
I followed your suggestion.
Tomorrow morning I will run this new version of the code.
Now using a sample of 81 elements (instead of 60) the profile returns:
Thu Apr 10 23:25:59 2014restats
18101188 function calls in 1218.626 seconds
Ordered by: internal time
List reduced from 13 to 10 due to restriction 10
ncalls tottime percall cumtime percall filename:lineno(function)
1 1015.803 1015.803 1218.334 1218.334 skymaps5.py:44(mymain)
18101000 202.4900.000 202.4900.000 {method 'write' of 'file'
objects}
10.2920.292 1218.626 1218.626 string:1(module)
60.0290.0050.0290.005 {open}
50.0100.0020.0100.002 {method 'close' of 'file'
objects}
810.0020.0000.0020.000 {method 'split' of 'str'
objects}
820.0010.0000.0010.000 {zip}
10.0000.0000.0000.000 function_base.py:8(linspace)
10.0000.0000.0000.000
function_base.py:93(logspace)
50.0000.0000.0000.000
{numpy.core.multiarray.zeros}
Anyway I would like to try to speed it up using C functions (and maybe
comparing the resuts of the two profile in the end)
How can I do it now? Can I use Cython?
If you have a compiler installed already it's just easy_install cython.
Writing Cython is not hard. That is, you easily get speed improvements.
It's in a .pyx file and once you're done you generate the .c and .so/.dll
files with a setup.py like below. Reason why I am posting this snippet is
the way to generate an annotated html file of your cython code. The whiter,
the more in C, the better. Yellow means stuff might still be improved more.
* setup.py
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
import Cython.Compiler.Options
Cython.Compiler.Options.annotate = True # really handy
setup(
cmdclass = {'build_ext': build_ext},
ext_modules = [Extension(myModule, [myModule.pyx])]
)
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
Hi Danny,
I'm quiet impressed.
the program takes near 30 minutes instead of more than 8 hours!
this is the profile:
Fri Apr 11 09:14:04 2014restats
19532732 function calls in 2105.024 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
1 2087.606 2087.606 2105.006 2105.006 skymapsI.py:44(mymain)
18101000 12.7570.000 12.7570.000 {method 'write' of 'file'
objects}
7158533.4730.0003.4730.000 {method 'split' of 'str'
objects}
7158541.1620.0001.1620.000 {zip}
10.0180.018 2105.024 2105.024 string:1(module)
60.0060.0010.0060.001 {open}
50.0020.0000.0020.000 {method 'close' of 'file'
objects}
10.0000.0000.0000.000 function_base.py:8(linspace)
50.0000.0000.0000.000 {numpy.core.multiarray.zeros}
10.0000.0000.0000.000 function_base.py:93(logspace)
10.0000.0000.0000.000 {numpy.core.multiarray.arange}
30.0000.0000.0000.000 {range}
10.0000.0000.0000.000 {method 'disable' of
'_lsprof.Prof
iler' objects}
I hope to have similar problems in the future to learn better how to do
with them!
but in the profile I don't see any operation regarding reading the file or
the mathematical operations...are them hidden in mymain()?
thanks
Gabriele
2014-04-10 21:38 GMT-04:00 Danny Yoo d...@hashcollision.org:
Comment: You are looping over your sliced eel five times. Do you
need to? I like eel salad a great deal, as well, but, how about:
for k in eel:
MYMAP1[i, j, k] = MYMAP1[i, j, k] + myinternet[oo]
MYMAP2[i, j, k] = MYMAP2[i, j, k] + myinternet[oo]
MYMAP3[i, j, k] = MYMAP3[i, j, k] + myinternet[oo]
MYMAP4[i, j, k] = MYMAP4[i, j, k] + myinternet[oo]
MYMAP5[i, j, k] = MYMAP5[i, j, k] + myinternet[oo]
oo = oo + 1
Hi Gabriele,
Also note that, when Martin looked at this part of the code, he
unfortunately misinterpreted its effect; Martin's proposed rewrite
here does not preserve the meaning of the original code. But rather
than wag my finger at how Martin interpreted the code, I'd rather make
the observation that this is a warning sign that the original code
here was not easy to understand.
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
Hi, I'm sorry but there is a big problem.
the code is producing empty file.dat.
I think it's because of this that previously I have done that strange trick
of myinternet...
So:
for my_line in open('data.dat'):
myinternet = []
gmlis = []
print('reading the line', count, '/599378')
my_parts = [float(i) for i in my_line.split()]
phase = my_parts[4]
zobs = my_parts[5]
rho = my_parts[6]
gmils=[my_parts[7], my_parts[8], my_parts[9], my_parts[10],
my_parts[11]]
i = int((phase-phamin)/stepPHA)
j = int((zobs-obamin)/stepOB)
for gammar, MYMAP in zip(gmlis, MYMAPS):
omC = (1.5)*(gammar**3)*c/(rho*rlc)
gig = omC*hcut/eVtoErg
#check the single emission
for w in eel:
omega =
(10**(w*stepENE+Lemin))*eVtoErg/hcut
x = omega/omC
kap = instruments.kappa(x)
Iom = (1.732050808/c)*(e**2)*gammar*kap
#jackson dI/domega
P = Iom*(c/(rho*rlc))/(2*pi) #jackson P
phps = P/(hcut*omega) #photons per second
www = phps/(stepPHA*sin(zobs)*stepOB)
MYMAP[i,j,w] += www
count = count + 1
when I exit here the MYMAP matrix has all the cells = 0.
Now I will try to fiugre it out why.
Thanks
Gabriele
2014-04-11 9:20 GMT-04:00 Gabriele Brambilla gb.gabrielebrambi...@gmail.com
:
Hi Danny,
I'm quiet impressed.
the program takes near 30 minutes instead of more than 8 hours!
this is the profile:
Fri Apr 11 09:14:04 2014restats
19532732 function calls in 2105.024 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
1 2087.606 2087.606 2105.006 2105.006 skymapsI.py:44(mymain)
18101000 12.7570.000 12.7570.000 {method 'write' of 'file'
objects}
7158533.4730.0003.4730.000 {method 'split' of 'str'
objects}
7158541.1620.0001.1620.000 {zip}
10.0180.018 2105.024 2105.024 string:1(module)
60.0060.0010.0060.001 {open}
50.0020.0000.0020.000 {method 'close' of 'file'
objects}
10.0000.0000.0000.000 function_base.py:8(linspace)
50.0000.0000.0000.000 {numpy.core.multiarray.zeros}
10.0000.0000.0000.000 function_base.py:93(logspace)
10.0000.0000.0000.000
{numpy.core.multiarray.arange}
30.0000.0000.0000.000 {range}
10.0000.0000.0000.000 {method 'disable' of
'_lsprof.Prof
iler' objects}
I hope to have similar problems in the future to learn better how to do
with them!
but in the profile I don't see any operation regarding reading the file or
the mathematical operations...are them hidden in mymain()?
thanks
Gabriele
2014-04-10 21:38 GMT-04:00 Danny Yoo d...@hashcollision.org:
Comment: You are looping over your sliced eel five times. Do you
need to? I like eel salad a great deal, as well, but, how about:
for k in eel:
MYMAP1[i, j, k] = MYMAP1[i, j, k] + myinternet[oo]
MYMAP2[i, j, k] = MYMAP2[i, j, k] + myinternet[oo]
MYMAP3[i, j, k] = MYMAP3[i, j, k] + myinternet[oo]
MYMAP4[i, j, k] = MYMAP4[i, j, k] + myinternet[oo]
MYMAP5[i, j, k] = MYMAP5[i, j, k] + myinternet[oo]
oo = oo + 1
Hi Gabriele,
Also note that, when Martin looked at this part of the code, he
unfortunately misinterpreted its effect; Martin's proposed rewrite
here does not preserve the meaning of the original code. But rather
than wag my finger at how Martin interpreted the code, I'd rather make
the observation that this is a warning sign that the original code
here was not easy to understand.
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
ok, it seems that the code don't enter in this for loop
for gammar, MYMAP in zip(gmlis, MYMAPS):
I don't understand why.
Thanks
Gabriele
2014-04-11 9:56 GMT-04:00 Gabriele Brambilla gb.gabrielebrambi...@gmail.com
:
Hi, I'm sorry but there is a big problem.
the code is producing empty file.dat.
I think it's because of this that previously I have done that strange
trick of myinternet...
So:
for my_line in open('data.dat'):
myinternet = []
gmlis = []
print('reading the line', count, '/599378')
my_parts = [float(i) for i in my_line.split()]
phase = my_parts[4]
zobs = my_parts[5]
rho = my_parts[6]
gmils=[my_parts[7], my_parts[8], my_parts[9],
my_parts[10], my_parts[11]]
i = int((phase-phamin)/stepPHA)
j = int((zobs-obamin)/stepOB)
for gammar, MYMAP in zip(gmlis, MYMAPS):
omC = (1.5)*(gammar**3)*c/(rho*rlc)
gig = omC*hcut/eVtoErg
#check the single emission
for w in eel:
omega =
(10**(w*stepENE+Lemin))*eVtoErg/hcut
x = omega/omC
kap = instruments.kappa(x)
Iom = (1.732050808/c)*(e**2)*gammar*kap
#jackson dI/domega
P = Iom*(c/(rho*rlc))/(2*pi) #jackson P
phps = P/(hcut*omega) #photons per second
www = phps/(stepPHA*sin(zobs)*stepOB)
MYMAP[i,j,w] += www
count = count + 1
when I exit here the MYMAP matrix has all the cells = 0.
Now I will try to fiugre it out why.
Thanks
Gabriele
2014-04-11 9:20 GMT-04:00 Gabriele Brambilla
gb.gabrielebrambi...@gmail.com:
Hi Danny,
I'm quiet impressed.
the program takes near 30 minutes instead of more than 8 hours!
this is the profile:
Fri Apr 11 09:14:04 2014restats
19532732 function calls in 2105.024 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
1 2087.606 2087.606 2105.006 2105.006 skymapsI.py:44(mymain)
18101000 12.7570.000 12.7570.000 {method 'write' of 'file'
objects}
7158533.4730.0003.4730.000 {method 'split' of 'str'
objects}
7158541.1620.0001.1620.000 {zip}
10.0180.018 2105.024 2105.024 string:1(module)
60.0060.0010.0060.001 {open}
50.0020.0000.0020.000 {method 'close' of 'file'
objects}
10.0000.0000.0000.000 function_base.py:8(linspace)
50.0000.0000.0000.000
{numpy.core.multiarray.zeros}
10.0000.0000.0000.000
function_base.py:93(logspace)
10.0000.0000.0000.000
{numpy.core.multiarray.arange}
30.0000.0000.0000.000 {range}
10.0000.0000.0000.000 {method 'disable' of
'_lsprof.Prof
iler' objects}
I hope to have similar problems in the future to learn better how to do
with them!
but in the profile I don't see any operation regarding reading the file
or the mathematical operations...are them hidden in mymain()?
thanks
Gabriele
2014-04-10 21:38 GMT-04:00 Danny Yoo d...@hashcollision.org:
Comment: You are looping over your sliced eel five times. Do you
need to? I like eel salad a great deal, as well, but, how about:
for k in eel:
MYMAP1[i, j, k] = MYMAP1[i, j, k] + myinternet[oo]
MYMAP2[i, j, k] = MYMAP2[i, j, k] + myinternet[oo]
MYMAP3[i, j, k] = MYMAP3[i, j, k] + myinternet[oo]
MYMAP4[i, j, k] = MYMAP4[i, j, k] + myinternet[oo]
MYMAP5[i, j, k] = MYMAP5[i, j, k] + myinternet[oo]
oo = oo + 1
Hi Gabriele,
Also note that, when Martin looked at this part of the code, he
unfortunately misinterpreted its effect; Martin's proposed rewrite
here does not preserve the meaning of the original code. But rather
than wag my finger at how Martin interpreted the code, I'd rather make
the observation that this is a warning sign that the original code
here was not easy to understand.
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
ok
modifying the for in this way (zipping an array of matrix drive it crazy)
it works
dko=0
for gammar in gmils:
omC = (1.5)*(gammar**3)*c/(rho*rlc)
gig = omC*hcut/eVtoErg
#check the single emission
for w in eel:
omega =
(10**(w*stepENE+Lemin))*eVtoErg/hcut
x = omega/omC
kap = instruments.kappa(x)
Iom = (1.732050808/c)*(e**2)*gammar*kap
#jackson dI/domega
P = Iom*(c/(rho*rlc))/(2*pi) #jackson P
phps = P/(hcut*omega) #photons per second
www = phps/(stepPHA*sin(zobs)*stepOB)
MYMAPS[dko][i,j,w] += www
dko += 1
count = count + 1
Now I will tell you how much it takes.
Thanks
Gabriele
2014-04-11 10:05 GMT-04:00 Gabriele Brambilla
gb.gabrielebrambi...@gmail.com:
ok, it seems that the code don't enter in this for loop
for gammar, MYMAP in zip(gmlis, MYMAPS):
I don't understand why.
Thanks
Gabriele
2014-04-11 9:56 GMT-04:00 Gabriele Brambilla
gb.gabrielebrambi...@gmail.com:
Hi, I'm sorry but there is a big problem.
the code is producing empty file.dat.
I think it's because of this that previously I have done that strange
trick of myinternet...
So:
for my_line in open('data.dat'):
myinternet = []
gmlis = []
print('reading the line', count, '/599378')
my_parts = [float(i) for i in my_line.split()]
phase = my_parts[4]
zobs = my_parts[5]
rho = my_parts[6]
gmils=[my_parts[7], my_parts[8], my_parts[9],
my_parts[10], my_parts[11]]
i = int((phase-phamin)/stepPHA)
j = int((zobs-obamin)/stepOB)
for gammar, MYMAP in zip(gmlis, MYMAPS):
omC = (1.5)*(gammar**3)*c/(rho*rlc)
gig = omC*hcut/eVtoErg
#check the single emission
for w in eel:
omega =
(10**(w*stepENE+Lemin))*eVtoErg/hcut
x = omega/omC
kap = instruments.kappa(x)
Iom = (1.732050808/c)*(e**2)*gammar*kap
#jackson dI/domega
P = Iom*(c/(rho*rlc))/(2*pi) #jackson P
phps = P/(hcut*omega) #photons per second
www = phps/(stepPHA*sin(zobs)*stepOB)
MYMAP[i,j,w] += www
count = count + 1
when I exit here the MYMAP matrix has all the cells = 0.
Now I will try to fiugre it out why.
Thanks
Gabriele
2014-04-11 9:20 GMT-04:00 Gabriele Brambilla
gb.gabrielebrambi...@gmail.com:
Hi Danny,
I'm quiet impressed.
the program takes near 30 minutes instead of more than 8 hours!
this is the profile:
Fri Apr 11 09:14:04 2014restats
19532732 function calls in 2105.024 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
1 2087.606 2087.606 2105.006 2105.006 skymapsI.py:44(mymain)
18101000 12.7570.000 12.7570.000 {method 'write' of 'file'
objects}
7158533.4730.0003.4730.000 {method 'split' of 'str'
objects}
7158541.1620.0001.1620.000 {zip}
10.0180.018 2105.024 2105.024 string:1(module)
60.0060.0010.0060.001 {open}
50.0020.0000.0020.000 {method 'close' of 'file'
objects}
10.0000.0000.0000.000
function_base.py:8(linspace)
50.0000.0000.0000.000
{numpy.core.multiarray.zeros}
10.0000.0000.0000.000
function_base.py:93(logspace)
10.0000.0000.0000.000
{numpy.core.multiarray.arange}
30.0000.0000.0000.000 {range}
10.0000.0000.0000.000 {method 'disable' of
'_lsprof.Prof
iler' objects}
I hope to have similar problems in the future to learn better how to do
with them!
but in the profile I don't see any operation regarding reading the file
or the mathematical operations...are them hidden in mymain()?
thanks
Gabriele
2014-04-10 21:38 GMT-04:00 Danny Yoo d...@hashcollision.org:
Comment: You are looping over your sliced eel five times. Do you
need to? I like eel salad a great deal, as well, but, how about:
for k in eel:
MYMAP1[i, j, k] = MYMAP1[i, j, k] + myinternet[oo]
MYMAP2[i, j, k] = MYMAP2[i, j, k] + myinternet[oo]
Re: [Tutor] improving speed using and recalling C functions
this is the profile for a sample of 1000 elements
Fri Apr 11 10:21:21 2014restats
31594963 function calls in 103.708 seconds
Ordered by: internal time
List reduced from 47 to 20 due to restriction 20
ncalls tottime percall cumtime percall filename:lineno(function)
1 57.133 57.133 103.692 103.692 skymapsI.py:44(mymain)
1768329.8980.000 11.7100.000
interpolate.py:394(_call_linear)
181010007.8080.0007.8080.000 {method 'write' of 'file'
objects}
12378243.7940.0003.7940.000 {numpy.core.multiarray.array}
10010003.6100.000 38.3830.000 instruments.py:10(kappa)
3536643.3140.0003.3140.000 {method 'reduce' of
'numpy.ufunc'
objects}
1768323.1570.0007.4280.000
interpolate.py:454(_check_bounds)
1768322.0740.000 10.2130.000 interpolate.py:330(__init__)
1768322.0530.000 21.5220.000 interpolate.py:443(_evaluate)
1768321.2530.0004.4040.000 polyint.py:82(_set_yi)
1768320.7690.0000.7690.000 {method 'clip' of
'numpy.ndarray'
objects}
3536640.7060.0000.7060.000 {method 'reshape' of
'numpy.ndarra
y' objects}
3536640.6670.0001.2050.000
numerictypes.py:735(issubdtype)
7073280.6370.0002.4510.000 numeric.py:392(asarray)
1768320.6010.000 23.5550.000 polyint.py:37(__call__)
3536640.5690.0003.8830.000 _methods.py:31(_any)
1768320.5040.0001.4290.000 polyint.py:74(_reshape_yi)
1768320.4730.0000.4730.000 {method 'searchsorted' of
'numpy.n
darray' objects}
1768320.4400.0001.6450.000 polyint.py:102(_set_dtype)
1768320.4260.0004.8300.000 polyint.py:30(__init__)
thanks
Gabriele
2014-04-11 10:18 GMT-04:00 Gabriele Brambilla
gb.gabrielebrambi...@gmail.com:
ok
modifying the for in this way (zipping an array of matrix drive it crazy)
it works
dko=0
for gammar in gmils:
omC = (1.5)*(gammar**3)*c/(rho*rlc)
gig = omC*hcut/eVtoErg
#check the single emission
for w in eel:
omega =
(10**(w*stepENE+Lemin))*eVtoErg/hcut
x = omega/omC
kap = instruments.kappa(x)
Iom = (1.732050808/c)*(e**2)*gammar*kap
#jackson dI/domega
P = Iom*(c/(rho*rlc))/(2*pi) #jackson P
phps = P/(hcut*omega) #photons per second
www = phps/(stepPHA*sin(zobs)*stepOB)
MYMAPS[dko][i,j,w] += www
dko += 1
count = count + 1
Now I will tell you how much it takes.
Thanks
Gabriele
2014-04-11 10:05 GMT-04:00 Gabriele Brambilla
gb.gabrielebrambi...@gmail.com:
ok, it seems that the code don't enter in this for loop
for gammar, MYMAP in zip(gmlis, MYMAPS):
I don't understand why.
Thanks
Gabriele
2014-04-11 9:56 GMT-04:00 Gabriele Brambilla
gb.gabrielebrambi...@gmail.com:
Hi, I'm sorry but there is a big problem.
the code is producing empty file.dat.
I think it's because of this that previously I have done that strange
trick of myinternet...
So:
for my_line in open('data.dat'):
myinternet = []
gmlis = []
print('reading the line', count, '/599378')
my_parts = [float(i) for i in my_line.split()]
phase = my_parts[4]
zobs = my_parts[5]
rho = my_parts[6]
gmils=[my_parts[7], my_parts[8], my_parts[9],
my_parts[10], my_parts[11]]
i = int((phase-phamin)/stepPHA)
j = int((zobs-obamin)/stepOB)
for gammar, MYMAP in zip(gmlis, MYMAPS):
omC = (1.5)*(gammar**3)*c/(rho*rlc)
gig = omC*hcut/eVtoErg
#check the single emission
for w in eel:
omega =
(10**(w*stepENE+Lemin))*eVtoErg/hcut
x = omega/omC
kap = instruments.kappa(x)
Iom = (1.732050808/c)*(e**2)*gammar*kap
#jackson dI/domega
P = Iom*(c/(rho*rlc))/(2*pi) #jackson P
phps = P/(hcut*omega) #photons per second
www = phps/(stepPHA*sin(zobs)*stepOB)
MYMAP[i,j,w] += www
count = count + 1
when I exit here the MYMAP matrix has all the cells = 0.
Now I will try to fiugre it out why.
Thanks
Gabriele
Re: [Tutor] improving speed using and recalling C functions
On 11/04/14 09:59, Peter Otten wrote: Gabriele Brambilla wrote: Anyway I would like to try to speed it up using C functions ... posted looks like it has great potential for speed-up by replacing the inner loops with numpy array operations. And in case its not obvious much(most?) of numPy consists of C functions. So by using NumPy you are usually using C code not native Python. That's what I alluded to in my first post on this thread: there are other libraries who have trod this route before and done the work for you. HTH -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
Gabriele Brambilla wrote: ok, it seems that the code don't enter in this for loop for gammar, MYMAP in zip(gmlis, MYMAPS): I don't understand why. You have two variables with similar names, gmlis and gmils: gmlis = [] gmils=[my_parts[7], my_parts[8], my_parts[9], my_parts[10], my_parts[11]] for gammar, MYMAP in zip(gmlis, MYMAPS): I assume you wanted for gammar, MYMAP in zip(gmils, MYMAPS): ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
Yes, but I want to make a C extension to run faster a function from scipy.interpolate (interp1d) It woulldn't change anything? thanks Gabriele 2014-04-11 14:47 GMT-04:00 Alan Gauld alan.ga...@btinternet.com: On 11/04/14 09:59, Peter Otten wrote: Gabriele Brambilla wrote: Anyway I would like to try to speed it up using C functions ... posted looks like it has great potential for speed-up by replacing the inner loops with numpy array operations. And in case its not obvious much(most?) of numPy consists of C functions. So by using NumPy you are usually using C code not native Python. That's what I alluded to in my first post on this thread: there are other libraries who have trod this route before and done the work for you. HTH -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
On Fri, Apr 11, 2014 at 1:01 PM, Gabriele Brambilla gb.gabrielebrambi...@gmail.com wrote: Yes, but I want to make a C extension to run faster a function from scipy.interpolate (interp1d) Just to emphasis: I believe your goal should be: I want to make my program fast. Your goal should probably not be: I want to write a C extension. I'm not saying that writing a C extension is necessarily wrong, and it may be that writing a C extension will make your program fast. But this approach may not be the easiest or most maintainable approach to improving your program's performance. Using C is not without its costs and risks. As soon as you are in C territory, the seat belts are off. Just recall the craziness that happened this week with regards to programs written in low-level languages like C. Explicitly: http://heartbleed.com. If you are writing with C, you have to be very, very delicate with your code. Experts get it wrong, with severe consequences. This is why the focus on C extensions to get speed disturbs me so much: it assumes that C is a safe language to use. It's not, especially for beginners. We should strongly discourage low-level languages unless there is some overriding concern. For scientific calculations like the ones you are doing, you should place a premium on getting a right answer, and not just a fast answer. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
On Fri, Apr 11, 2014 at 1:01 PM, Gabriele Brambilla
gb.gabrielebrambi...@gmail.com wrote:
Yes,
but I want to make a C extension to run faster a function from
scipy.interpolate (interp1d)
It woulldn't change anything?
This is precisely why you want to drive your optimization based on
what the profiler is telling you. Look at the profiler's output
again, closely:
---
31594963 function calls in 103.708 seconds
Ordered by: internal time
List reduced from 47 to 20 due to restriction 20
ncalls tottime percall cumtime percall filename:lineno(function)
1 57.133 57.133 103.692 103.692 skymapsI.py:44(mymain)
1768329.8980.000 11.7100.000 interpolate.py:394(_call_linear)
181010007.8080.0007.8080.000 {method 'write' of 'file' objects}
12378243.7940.0003.7940.000 {numpy.core.multiarray.array}
10010003.6100.000 38.3830.000 instruments.py:10(kappa)
3536643.3140.0003.3140.000 {method 'reduce' of 'numpy.ufunc'
[cutting some content]
---
About 8% of the time in your program is being spent in interpolate.py.
But this is in SciPy code, so it is likely difficult to rewrite.
Also note that the amount of time being spent on merely writing the
output is about that much time too! That's what the profile is
saying, qualitatively.
And on the other hand, the code in skymapsI.mymain is still a target
worthy of your attention. Compare how much time it was taking before
we started investigating it. Before, it took 75% of the total runtime
of your program. We improved upon that a lot with a few small
changes. But it's still taking 55% of the total time of your
program's running. If you look at the rest of the profiler's output,
we know that everything else is fairly inconsequential.
That's why we're pushing you to look at the data. Trust the profiler.
Work on the thing that is contributing most to the cost of your
program: continue trying to improve the code in skymapsI.mymain. I am
fairly certain there is still some low-hanging fruit there.
The profile you want to see, eventually, is one where the computation
is being done mostly in numpy code. But as we can see now, the numpy
code is barely contributing to the runtime. That's a situation that
needs improvement.
Peter Otten's offer to help you use NumPy more effectively is one you
should take seriously.
Good luck!
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
Ok guys, when I wrote that email I was excited for the apparent speed increasing (it was jumping the bottleneck for loop for the reason peter otten outlined). Now, instead the changes, the speed is not improved (the code still running from this morning and it's at one forth of the dataset). What can I do to speed it up? Thanks Gabriele sent from Samsung Mobile Il giorno 11/apr/2014 17:00, Danny Yoo d...@hashcollision.org ha scritto: On Fri, Apr 11, 2014 at 1:01 PM, Gabriele Brambilla gb.gabrielebrambi...@gmail.com wrote: Yes, but I want to make a C extension to run faster a function from scipy.interpolate (interp1d) Just to emphasis: I believe your goal should be: I want to make my program fast. Your goal should probably not be: I want to write a C extension. I'm not saying that writing a C extension is necessarily wrong, and it may be that writing a C extension will make your program fast. But this approach may not be the easiest or most maintainable approach to improving your program's performance. Using C is not without its costs and risks. As soon as you are in C territory, the seat belts are off. Just recall the craziness that happened this week with regards to programs written in low-level languages like C. Explicitly: http://heartbleed.com. If you are writing with C, you have to be very, very delicate with your code. Experts get it wrong, with severe consequences. This is why the focus on C extensions to get speed disturbs me so much: it assumes that C is a safe language to use. It's not, especially for beginners. We should strongly discourage low-level languages unless there is some overriding concern. For scientific calculations like the ones you are doing, you should place a premium on getting a right answer, and not just a fast answer. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
Hi there Gabriele, : I have a program that is reading near 60 elements from a : file. For each element it performs 200 times a particular : mathematical operation (a numerical interpolation of a function). : Now these process takes near 8 hours. Sounds fun! Here are some thoughts (though I may not have any solid answers, I hope these pointers are useful): Are you sure (from profiling) that the numerical interpolation is the bottleneck? What is the numerical operation? 1: Is the function implemented in Numpy? [0] 2: How about SciPy? [1] Is there a domain-specific Python library that already does the computation you want to achieve? If so, then you have only the format question--how to put your data into the format required by the library. If not, then on to the next question you ask ... : Creating a C function and calling it from the code could improve : the speed? It could be useful that it contains both the : mathematical operation and the for loop or only the mathematical : operation? : : If it is a reasonable solution to try decreasing the : computational time how can I implement this C function? It is certainly an option! In the event that you cannot find something that is fast enough in Python already and you cannot find any C library that has Python bindings, then, whether you have to write your own, or you have a third party C library, you have several options for writing bindings. You can write your Python bindings using: * ctypes: https://docs.python.org/2/library/ctypes.html * cffi: http://cffi.readthedocs.org/en/release-0.8/ * cython: http://www.cython.org/ * roll your own C! It might be beyond the scope of this list (certainly beyond my capabilities) to provide recommendations on which avenue to take--but if you are asking, you may already have the tools you need to assess for yourself. There is, however, quite a bit of experience loitering around this list, so perhaps somebody else will have a suggestion. -Martin [0] http://www.numpy.org/ [1] http://www.scipy.org/ -- Martin A. Brown http://linux-ip.net/ ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
On 10/04/14 16:58, Gabriele Brambilla wrote: For each element it performs 200 times a particular mathematical operation (a numerical interpolation of a function). Now these process takes near 8 hours. The first thing to do in such cases is check that the time is going where you think it is. Run the cProfile tool on the code (with a smaller data set!) to see where the time is being spent. Creating a C function and calling it from the code could improve the speed? Yes, it could. But it may not be necessary. There are various add on tools that can drastically improve Python efficiency, especially for math type functions so you might want to check that nobody has already written what you need. If it is a reasonable solution to try decreasing the computational time how can I implement this C function? If its already written in Python the easiest way might be to use Cython to convert it into C. I'm no expert in Cython however, but others on this list can help. -- Alan G Author of the Learn to Program web site http://www.alan-g.me.uk/ http://www.flickr.com/photos/alangauldphotos ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
Hi, 2014-04-10 13:05 GMT-04:00 Martin A. Brown mar...@linux-ip.net: Hi there Gabriele, : I have a program that is reading near 60 elements from a : file. For each element it performs 200 times a particular : mathematical operation (a numerical interpolation of a function). : Now these process takes near 8 hours. Sounds fun! Here are some thoughts (though I may not have any solid answers, I hope these pointers are useful): Are you sure (from profiling) that the numerical interpolation is the bottleneck? I see that the time needed by the code increase when I increase the number of these operation. It's the only thing I do in the code. (I'm sorry but I don't know very well what profiling is) What is the numerical operation? 1: Is the function implemented in Numpy? [0] 2: How about SciPy? [1] Is interp1d from scipy.interpolate. I think is the best one Is there a domain-specific Python library that already does the computation you want to achieve? If so, then you have only the format question--how to put your data into the format required by the library. If not, then on to the next question you ask ... No, interp1d do exactly what I want. : Creating a C function and calling it from the code could improve : the speed? It could be useful that it contains both the : mathematical operation and the for loop or only the mathematical : operation? : : If it is a reasonable solution to try decreasing the : computational time how can I implement this C function? It is certainly an option! In the event that you cannot find something that is fast enough in Python already and you cannot find any C library that has Python bindings, then, whether you have to write your own, or you have a third party C library, you have several options for writing bindings. You can write your Python bindings using: * ctypes: https://docs.python.org/2/library/ctypes.html * cffi: http://cffi.readthedocs.org/en/release-0.8/ * cython: http://www.cython.org/ * roll your own C! Do you know if does a C library exist with an already implemented function like interp1d? I use Anaconda, is there an already implemented mode to do it in Anaconda? Cython? It might be beyond the scope of this list (certainly beyond my capabilities) to provide recommendations on which avenue to take--but if you are asking, you may already have the tools you need to assess for yourself. There is, however, quite a bit of experience loitering around this list, so perhaps somebody else will have a suggestion. -Martin [0] http://www.numpy.org/ [1] http://www.scipy.org/ -- Martin A. Brown http://linux-ip.net/ ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
On 10/04/2014 18:29, Gabriele Brambilla wrote: (I'm sorry but I don't know very well what profiling is) Take a look at these for some tips http://www.huyng.com/posts/python-performance-analysis/ and https://wiki.python.org/moin/PythonSpeed/PerformanceTips -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
Hi Gabriele, Have you profiled your program? Please look at: https://docs.python.org/2/library/profile.html If you can, avoid guessing what is causing performance to drop. Rather, use the tools in the profiling libraries to perform measurements. It may be that your program is taking a long time because of something obvious, but perhaps there is some other factor that's contributing. Please do this. More details would be helpful. Are you using any libraries such as Numpy? Just writing something in C doesn't magically make it go faster. CPython is written in C, for example, and yet people do not say that Python itself is very fast. :P It may be the case that writing the computations in C will allow you to specify enough type information so that the computer can effectively run your computations quickly. But if you're using libraries like Numpy to do vector parallel operations, I would not be surprised if that would outperform native non-parallel C code. See: http://technicaldiscovery.blogspot.com/2011/06/speeding-up-python-numpy-cython-and.html which demonstrates that effective use of NumPy may speed up computations by a significant order of magnitude, if you use the library to take advantage of its vectorizing compuations. More domain-specific details may help folks on the tutor list to give good advice here. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
I'm trying to profile it adding this code:
import cProfile
import re
import pstats
cProfile.run('re.compile(foo|bar)', 'restats')
p = pstats.Stats('restats')
p.strip_dirs().sort_stats('name')
p.sort_stats('time').print_stats(10)
but where I have to add this in my code?
because I obtain
Thu Apr 10 15:23:17 2014restats
194 function calls (189 primitive calls) in 0.001 seconds
Ordered by: internal time
List reduced from 34 to 10 due to restriction 10
ncalls tottime percall cumtime percall filename:lineno(function)
3/10.0000.0000.0000.000 sre_compile.py:33(_compile)
10.0000.0000.0000.000
sre_compile.py:208(_optimize_chars
et)
3/10.0000.0000.0000.000 sre_parse.py:141(getwidth)
10.0000.0000.0000.000
sre_compile.py:362(_compile_info)
20.0000.0000.0000.000 sre_parse.py:380(_parse)
10.0000.0000.0000.000 sre_parse.py:302(_parse_sub)
10.0000.0000.0010.001 re.py:226(_compile)
100.0000.0000.0000.000 sre_parse.py:183(__next)
10.0000.0000.0010.001 sre_compile.py:496(compile)
150.0000.0000.0000.000 {isinstance}
but my program take more than 0.001 seconds!
I think it's not working as I want.
Gabriele
2014-04-10 15:09 GMT-04:00 Danny Yoo d...@hashcollision.org:
Hi Gabriele,
Have you profiled your program? Please look at:
https://docs.python.org/2/library/profile.html
If you can, avoid guessing what is causing performance to drop.
Rather, use the tools in the profiling libraries to perform
measurements.
It may be that your program is taking a long time because of something
obvious, but perhaps there is some other factor that's contributing.
Please do this. More details would be helpful. Are you using any
libraries such as Numpy? Just writing something in C doesn't
magically make it go faster. CPython is written in C, for example,
and yet people do not say that Python itself is very fast. :P
It may be the case that writing the computations in C will allow you
to specify enough type information so that the computer can
effectively run your computations quickly. But if you're using
libraries like Numpy to do vector parallel operations, I would not be
surprised if that would outperform native non-parallel C code.
See:
http://technicaldiscovery.blogspot.com/2011/06/speeding-up-python-numpy-cython-and.html
which demonstrates that effective use of NumPy may speed up
computations by a significant order of magnitude, if you use the
library to take advantage of its vectorizing compuations.
More domain-specific details may help folks on the tutor list to give
good advice here.
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
Hi Gabriele,
I should probably have pointed you to:
https://docs.python.org/2/library/profile.html#instant-user-s-manual
instead.
Here is an example that uses the cProfile module. Let's say that I'm
trying to pinpoint where something is going slow in some_program():
#
import cProfile
def slow_string_mul(w, n):
s =
for x in xrange(n):
s = slow_append(s, w)
return s
def slow_append(s, w):
return s + w
def fast_string_mul(w, n):
return w * n
def some_program(w, n):
print slow_string_mul(w, n) == fast_string_mul(w, n)
## Try running the operation, and let cProfile report stats.
cProfile.run('some_program(testing, 5)', None, 'time')
#
We tell cProfile.run to execute some_program(), sorting its
measurements by time. You might save the collected statistics to a
file, but here I have not, and have cProfile.run() just report the
results after the program finishes.
Here's what comes back from cProfile's report:
#
$ python test_profile.py
True
50005 function calls in 1.422 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
51.2250.0001.2250.000 test_profile.py:9(slow_append)
10.1970.1971.4221.422 test_profile.py:3(slow_string_mul)
10.0000.0001.4221.422 test_profile.py:15(some_program)
10.0000.0000.0000.000
test_profile.py:12(fast_string_mul)
10.0000.0001.4221.422 string:1(module)
10.0000.0000.0000.000 {method 'disable' of
'_lsprof.Profiler' objects}
#
This is telling me that slow_append is being called 5 times, which
in hindsight sounds right. It's taking the majority of the time of
this program, which is intuitively true, as I wrote it in a way to do
a very naive string-appending operation, which gets more expensive the
longer the string grows.
The point of examples is to show simple uses of the libraries. But
you will have to adapt the examples to work for your context.
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
Hi,
I get this result:
Thu Apr 10 17:35:53 2014restats
21071736 function calls in 199.883 seconds
Ordered by: internal time
List reduced from 188 to 10 due to restriction 10
ncalls tottime percall cumtime percall filename:lineno(function)
1 149.479 149.479 199.851 199.851 skymaps5.py:16(mymain)
18101000 28.6820.000 28.6820.000 {method 'write' of 'file'
objects}
330445.4700.0006.4440.000
interpolate.py:394(_call_linear)
232.2720.000 21.2790.000 instruments.py:10(kappa)
2313282.1200.0002.1200.000 {numpy.core.multiarray.array}
330441.7190.0003.8360.000
interpolate.py:454(_check_bounds)
660881.6110.0001.6110.000 {method 'reduce' of
'numpy.ufunc'
objects}
330441.1460.000 11.6230.000 interpolate.py:443(_evaluate)
330441.1200.0005.5420.000 interpolate.py:330(__init__)
330440.6590.0002.3290.000 polyint.py:82(_set_yi)
the major time is required by mymain that is the whole program.
the write on file is an operation that I do in the end but if I increase
the number of data it doesn't increase (I tried doubing the sample of
values and I know why it's behaving in this way)
the third are interpolate and kappa: the two functions (one called inside
the other one)
So they are the ones that are taking time.
Now what can I do?
thanks
Gabriele
2014-04-10 17:14 GMT-04:00 Danny Yoo d...@hashcollision.org:
Hi Gabriele,
I should probably have pointed you to:
https://docs.python.org/2/library/profile.html#instant-user-s-manual
instead.
Here is an example that uses the cProfile module. Let's say that I'm
trying to pinpoint where something is going slow in some_program():
#
import cProfile
def slow_string_mul(w, n):
s =
for x in xrange(n):
s = slow_append(s, w)
return s
def slow_append(s, w):
return s + w
def fast_string_mul(w, n):
return w * n
def some_program(w, n):
print slow_string_mul(w, n) == fast_string_mul(w, n)
## Try running the operation, and let cProfile report stats.
cProfile.run('some_program(testing, 5)', None, 'time')
#
We tell cProfile.run to execute some_program(), sorting its
measurements by time. You might save the collected statistics to a
file, but here I have not, and have cProfile.run() just report the
results after the program finishes.
Here's what comes back from cProfile's report:
#
$ python test_profile.py
True
50005 function calls in 1.422 seconds
Ordered by: internal time
ncalls tottime percall cumtime percall filename:lineno(function)
51.2250.0001.2250.000
test_profile.py:9(slow_append)
10.1970.1971.4221.422
test_profile.py:3(slow_string_mul)
10.0000.0001.4221.422
test_profile.py:15(some_program)
10.0000.0000.0000.000
test_profile.py:12(fast_string_mul)
10.0000.0001.4221.422 string:1(module)
10.0000.0000.0000.000 {method 'disable' of
'_lsprof.Profiler' objects}
#
This is telling me that slow_append is being called 5 times, which
in hindsight sounds right. It's taking the majority of the time of
this program, which is intuitively true, as I wrote it in a way to do
a very naive string-appending operation, which gets more expensive the
longer the string grows.
The point of examples is to show simple uses of the libraries. But
you will have to adapt the examples to work for your context.
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
Gabriele,
21071736 function calls in 199.883 seconds
The 21 million function calls isn't really a surprise to me, given
18 million calls to file.write(). Given that the majority of the
time is still spent in skymaps5.py, I think you'll need to
instrument that a bit more to figure out where the hotspot is.
Ordered by: internal time
List reduced from 188 to 10 due to restriction 10
ncalls tottime percall cumtime percall filename:lineno(function)
1 149.479 149.479 199.851 199.851 skymaps5.py:16(mymain)
18101000 28.6820.000 28.6820.000 {method 'write' of 'file'
objects}
the major time is required by mymain that is the whole program.
the write on file is an operation that I do in the end but if I
increase the number of data it doesn't increase (I tried doubing
the sample of values and I know why it's behaving in this way) the
third are interpolate and kappa: the two functions (one called
inside the other one)
This is a good finding, in fact. Now, you know which module
contains the bottleneck. Is your CPU pegged when you run that
skymap5.py code? Can you run the profiler on the skymaps5.py code,
too, to locate the specific hotspot? Great to see that you are
using the profiler, effectively, too!
So they are the ones that are taking time.
Now what can I do?
I think you now need to profile skymaps5.py code? Specifically the
stuff in main().
OK, so I have not myself used scipy.interpolate.interp1d before, but
I went to have a look at it. So, you feed interp1d() an x and a y
(e.g. a plot line on a diagram), and it essentially produces its
best guess of a function which will fit that curve, correct?
Well, this is how it performs with random input and an identity
function:
element count 100, duration 0.000
element count 1000, duration 0.001
element count 1, duration 0.005
element count 10, duration 0.055
element count 100, duration 0.858
element count 1000, duration 30.404
So, with 10 million inputs on an admittedly brain-dead function,
there's not a performance bottleneck. If you can find the parts of
your skymaps5.py code that are the bottleneck, then you could post
it here.
I hadn't thought of using interpolate, myself, as I didn't even know
it existed.
Thanks and good luck,
-Martin
--
Martin A. Brown
http://linux-ip.net/
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
ncalls tottime percall cumtime percall filename:lineno(function)
1 149.479 149.479 199.851 199.851 skymaps5.py:16(mymain)
18101000 28.6820.000 28.6820.000 {method 'write' of 'file'
objects}
330445.4700.0006.4440.000
interpolate.py:394(_call_linear)
232.2720.000 21.2790.000 instruments.py:10(kappa)
2313282.1200.0002.1200.000
{numpy.core.multiarray.array}
330441.7190.0003.8360.000
interpolate.py:454(_check_bounds)
660881.6110.0001.6110.000 {method 'reduce' of
'numpy.ufunc'
objects}
330441.1460.000 11.6230.000
interpolate.py:443(_evaluate)
330441.1200.0005.5420.000 interpolate.py:330(__init__)
330440.6590.0002.3290.000 polyint.py:82(_set_yi)
the major time is required by mymain that is the whole program.
Good! Profiles like this allow us to pinpoint issues.
Wait... ?!
The profiler is saying that the majority of time is in my main, but _not_
in auxiliary functions. That's surprising. Am I misreading the profile?
Can you show what mymain is doing?
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
but main is the program that contains everything.
I used the profile in this way:
import cProfile
import pstats
def mymain():
#all the code
#end of main indentation
cProfile.run('mymain()', 'restats', 'time')
p = pstats.Stats('restats')
p.strip_dirs().sort_stats('name')
p.sort_stats('time').print_stats(10)
So all the function I used are contained in main(), so even all the others
that are appeared.
Gabriele
2014-04-10 19:41 GMT-04:00 Martin A. Brown mar...@linux-ip.net:
Gabriele,
21071736 function calls in 199.883 seconds
The 21 million function calls isn't really a surprise to me, given 18
million calls to file.write(). Given that the majority of the time is
still spent in skymaps5.py, I think you'll need to instrument that a bit
more to figure out where the hotspot is.
Ordered by: internal time
List reduced from 188 to 10 due to restriction 10
ncalls tottime percall cumtime percall filename:lineno(function)
1 149.479 149.479 199.851 199.851 skymaps5.py:16(mymain)
18101000 28.6820.000 28.6820.000 {method 'write' of 'file'
objects}
the major time is required by mymain that is the whole program. the write
on file is an operation that I do in the end but if I increase the number
of data it doesn't increase (I tried doubing the sample of values and I
know why it's behaving in this way) the third are interpolate and kappa:
the two functions (one called inside the other one)
This is a good finding, in fact. Now, you know which module contains the
bottleneck. Is your CPU pegged when you run that skymap5.py code? Can you
run the profiler on the skymaps5.py code, too, to locate the specific
hotspot? Great to see that you are using the profiler, effectively, too!
So they are the ones that are taking time.
Now what can I do?
I think you now need to profile skymaps5.py code? Specifically the stuff
in main().
OK, so I have not myself used scipy.interpolate.interp1d before, but I
went to have a look at it. So, you feed interp1d() an x and a y (e.g. a
plot line on a diagram), and it essentially produces its best guess of a
function which will fit that curve, correct?
Well, this is how it performs with random input and an identity function:
element count 100, duration 0.000
element count 1000, duration 0.001
element count 1, duration 0.005
element count 10, duration 0.055
element count 100, duration 0.858
element count 1000, duration 30.404
So, with 10 million inputs on an admittedly brain-dead function, there's
not a performance bottleneck. If you can find the parts of your
skymaps5.py code that are the bottleneck, then you could post it here.
I hadn't thought of using interpolate, myself, as I didn't even know it
existed.
Thanks and good luck,
-Martin
--
Martin A. Brown
http://linux-ip.net/
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
sure.
def mymain():
def LEstep(n):
Emin=10**6
Emax=5*(10**10)
Lemin=log10(Emin)
Lemax=log10(Emax)
stepE=(Lemax-Lemin)/n
return (stepE, n, Lemin, Lemax)
if __name__ == __main__:
import sys
if len(sys.argv)=1:
stepENE, nex, Lemin, Lemax = LEstep(200)
elif len(sys.argv)=2:
stepENE, nex, Lemin, Lemax =
LEstep(int(sys.argv[1]))
else:
stepENE, nex, Lemin, Lemax =
LEstep(int(sys.argv[1]))
freq=float(sys.argv[2])
eel = list(range(nex))
eels = np.logspace(Lemin, Lemax, num=nex, endpoint=False)
indpha = list(range(npha))
indobs = list(range(nobs))
rlc = c/(2*pi*freq)
MYMAP1 = np.zeros([npha, nobs, nex], dtype=float)
MYMAP2 = np.zeros([npha, nobs, nex], dtype=float)
MYMAP3 = np.zeros([npha, nobs, nex], dtype=float)
MYMAP4 = np.zeros([npha, nobs, nex], dtype=float)
MYMAP5 = np.zeros([npha, nobs, nex], dtype=float)
count=0
omegacliston = []
alpha = '60_'
for my_line in open('datasm0_60_5s.dat'):
myinternet = []
gmlis = []
print('reading the line', count, '/599378')
my_parts = [float(i) for i in my_line.split()]
phase = my_parts[4]
zobs = my_parts[5]
rho = my_parts[6]
gammar1 = my_parts[7]
gammar2 = my_parts[8]
gammar3 = my_parts[9]
gammar4 = my_parts[10]
gammar5 = my_parts[11]
gmlis.append(gammar1)
gmlis.append(gammar2)
gmlis.append(gammar3)
gmlis.append(gammar4)
gmlis.append(gammar5)
i = int((phase-phamin)/stepPHA)
j = int((zobs-obamin)/stepOB)
for gammar in gmlis:
omC = (1.5)*(gammar**3)*c/(rho*rlc)
gig = omC*hcut/eVtoErg
omegacliston.append(omC)
for w in eel[:]:
omega =
(10**(w*stepENE+Lemin))*eVtoErg/hcut
x = omega/omC
kap = instruments.kappa(x)
Iom = (1.732050808/c)*(e**2)*gammar*kap
P = Iom*(c/(rho*rlc))/(2*pi)
phps = P/(hcut*omega )
www = phps/(stepPHA*sin(zobs)*stepOB)
myinternet.append(www)
count = count + 1
oo = 0
for k in eel[:]:
MYMAP1[i, j, k] = MYMAP1[i, j, k] + myinternet[oo]
oo = oo + 1
for k in eel[:]:
MYMAP2[i, j, k] = MYMAP2[i, j, k] + myinternet[oo]
oo = oo + 1
for k in eel[:]:
MYMAP3[i, j, k] = MYMAP3[i, j, k] + myinternet[oo]
oo = oo + 1
for k in eel[:]:
MYMAP4[i, j, k] = MYMAP4[i, j, k] + myinternet[oo]
oo = oo + 1
for k in eel[:]:
MYMAP5[i, j, k] = MYMAP5[i, j, k] + myinternet[oo]
oo = oo + 1
BIGMAPS = [MYMAP1, MYMAP2, MYMAP3, MYMAP4, MYMAP5]
sigmas = [1, 3, 5, 10, 30]
fiq1 = plt.figure()
fiq2 = plt.figure()
fiq3 = plt.figure()
fiq4 = plt.figure()
fiq5 = plt.figure()
fiqqs = [fiq1, fiq2, fiq3, fiq4, fiq5]
multis = zip(sigmas, BIGMAPS, fiqqs)
for sigma, MYMAP, fiq in multis:
filename=alpha+'_'+str(sigma)+'_'+str(npha)+'_'+str(phamin)+'_'+str(phamax)+'_'+str(nobs)+'_'+str(obamin)+'_'+str(obamax)+'_'+str(nex)+'_'+str(Lemin)+'_'+str(Lemax)+'_.dat'
MYfile = open(filename, 'a')
for k in eel[:]:
for j in indobs[:]:
for i in indpha[:]:
A=MYMAP[i, j, k]
stringa = str(A) + ','
MYfile.write(stringa)
accapo = '\n'
MYfile.write(accapo)
MYfile.close()
2014-04-10 19:55 GMT-04:00 Danny Yoo d...@hashcollision.org:
ncalls tottime percall cumtime percall filename:lineno(function)
1 149.479 149.479 199.851 199.851 skymaps5.py:16(mymain)
18101000 28.6820.000 28.6820.000 {method 'write' of 'file'
objects}
Re: [Tutor] improving speed using and recalling C functions
Gabriele,
but main is the program that contains everything.
And, that is precisely the point of profiling the thing that
contains 'everything'. Because the bottleneck is almost always
somewher inside of 'everything'. But, you have to keep digging
until you find it.
I saw that you replied to Danny Yoo with your code, and I have to
say that this is rather domain-specific, so it may be quite
difficult for somebody to glance at it and figure out where the
hotspot is. It is for this reason that we were asking about
profiling.
Some follow-on questions:
Code: for my_line in open('datasm0_60_5s.dat')
Q: How big is datasm0_60_5s.dat? Unless there's a whitespace
pasting issue, it looks like you are reading that file for each
run through mymain(). Has this file changed in size recently?
Code: kap = instruments.kappa(x)
Q: What is instruments? A module? Is the performance hit there?
Code:
for k in eel[:]:
MYMAP1[i, j, k] = MYMAP1[i, j, k] + myinternet[oo]
oo = oo + 1
for k in eel[:]:
MYMAP,[i, j, k] = MYMAP1[i, j, k] + myinternet[oo]
oo = oo + 1
...
Comment: You are looping over your sliced eel five times. Do you
need to? I like eel salad a great deal, as well, but, how about:
for k in eel:
MYMAP1[i, j, k] = MYMAP1[i, j, k] + myinternet[oo]
MYMAP2[i, j, k] = MYMAP2[i, j, k] + myinternet[oo]
MYMAP3[i, j, k] = MYMAP3[i, j, k] + myinternet[oo]
MYMAP4[i, j, k] = MYMAP4[i, j, k] + myinternet[oo]
MYMAP5[i, j, k] = MYMAP5[i, j, k] + myinternet[oo]
oo = oo + 1
That should cut down a bit of looping time. Especially as the eel
grows longer.
Another suggestion, that is more along the lines of how do I figure
out what's broken this time in my code. I almost always add the
logging module to any program larger than a few lines. Why?
Because then, I can simply add logger lines and see what's going on.
Since I'm a perfect programmer and, like you, I don't make mistakes,
I never need this, but I do it anyway to look good around my
colleagues (best practices and all).
In seriousness, using logging [0] is not at all tricky for
standalone scripts. (It does get a bit more involved when you are
importing modules and libraries), but,) Consider the following:
import sys
import logging
logformat='%(asctime)s %(name)s %(levelname)s %(message)s'
logging.basicConfig(format=logformat, stream=sys.stderr, level=logging.INFO)
logger = logging.getLogger({ '__main__': None }.get(__name__, __name__))
With that setup at the top of the program, now you can sprinkle
lines like this throughout your code with impunity.
import os
# -- calling logger.info() will print stuff to STDERR
logger.info(silly example %r, os.environ)
# -- calling logger.debug() will not print to STDERR
#using, above config
logger.debug(debug example %d, 1)
# -- Ok, set it so anything that is set to logging.DEBUG (or
#higher) is shown
logger.setLevel(logging.DEBUG)
logger.debug(debug example %d, 2)
# -- and restore the prior pattern; restting so newer .debug lines
#are not shown
logger.setLevel(logging.INFO)
logger.debug(debug example %d, 3)
OK, so why is this useful? Well, timestamps in log lines is one
reason. Another reason is the typical diagnostic technique
What is the value of variable x, y, z, oo, text_contens
Good luck tracking down your peformance issue!
-Martin
[0] https://docs.python.org/2/library/logging.html
https://docs.python.org/3/library/logging.html
--
Martin A. Brown
http://linux-ip.net/
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
On Thu, Apr 10, 2014 at 11:58:30AM -0400, Gabriele Brambilla wrote: Hi, I have a program that is reading near 60 elements from a file. For each element it performs 200 times a particular mathematical operation (a numerical interpolation of a function). Now these process takes near 8 hours. Why are you repeating each operation 200 times? Surely you don't mean something like this? for element in elements_from_file(): for i in range(200): result = function(element) Before spending your time re-writing the function in C, it may help checking that there are no inefficencies in the code. Calculating the function may not be what is slowing your code down. It might help if you show us your code. -- Steven ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
On Fri, Apr 11, 2014 at 10:59:05AM +1000, Steven D'Aprano wrote: It might help if you show us your code. Oops, never mind, I see you have done so. -- Steven ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
Ok, good. There's a few things you'll want to fix in your mymain() in order for the profiler to work more effectively in pinpointing issues. 1. Move functionality outside of if __name__ == '__main__': At the moment, you've put the entire functionality of your program in the body of that if statement within mymain. That structure is probably not right. I see that this block is computing values for stepENE, nex, Lemin, Lemax, and, conditionally, freq. This should be lifted out into its own function. You should also note that, because 'freq' is computed conditionally, there are certain code paths in which your mymain() will fail. This is most likely a bad thing. Recommendation: have mymain() take in parameters. Move LEstep() toplevel. Restructure to: import sys def mymain(stepENE, nex, Lemin, Lemax, freq): ## everything starting after eel = list(range(nex))... ## if __name__ == '__main__': if len(sys.argv)=1: stepENE, nex, Lemin, Lemax = LEstep(200) elif len(sys.argv) = 2: stepENE, nex, Lemin, Lemax = LEstep(int(sys.argv[1])) else: stepENE, nex, Lemin, Lemax = LEstep(int(sys.argv[1])) freq=float(sys.argv[2]) mymain(stepENE, nex, Lemin, Lemax, freq) 2. Do not slice lists unless you really mean to do so. I see a lot of slices that do not look right. Examples like: for k in eel[:]: # code cut ... Just loop over eel. No copy necessary. This is doing a lot more memory copying over and over again. Instead: for k in eel: # code cut ... Do this everywhere you're creating slices with [:], unless you really need to copy. This is happening in multiple places in the code. 3. Be sure to remove dead variables. There are variables here that are not used. omegacliston is dead code, for example. You're appending to it, but doing nothing with its value. 4. Watch for repetitive code. There's something about the use of MYMAP1, MYMAP2, etc. that looks very suspicious. That is, the block here: oo = 0 for k in eel: MYMAP1[i, j, k] = MYMAP1[i, j, k] + myinternet[oo] oo = oo + 1 for k in eel: MYMAP2[i, j, k] = MYMAP2[i, j, k] + myinternet[oo] oo = oo + 1 for k in eel: MYMAP3[i, j, k] = MYMAP3[i, j, k] + myinternet[oo] oo = oo + 1 for k in eel: MYMAP4[i, j, k] = MYMAP4[i, j, k] + myinternet[oo] oo = oo + 1 for k in eel: MYMAP5[i, j, k] = MYMAP5[i, j, k] + myinternet[oo] oo = oo + 1 feels repetitive and strange. Martin Brown identifies this problem as well, so at least we're on the same page. Solving this problem takes a little bit of work. If you have five maps, with some kind of relationship, represent and use that. Ah. You're already doing this by representing BIGMAPS at reporting time. Then move the definition of an array of maps to the front of the code. Use a container holding those five maps in a single variable. Call it MYMAPS. MYMAPS = [np.zeros([npha, nobs, nex], dtype=float), np.zeros([npha, nobs, nex], dtype=float), np.zeros([npha, nobs, nex], dtype=float), np.zeros([npha, nobs, nex], dtype=float), np.zeros([npha, nobs, nex], dtype=float)] Wen you're tempted to say MYMAP1, use MYMAPS[0]. MYMAP2 -- MYMAPS[1], and so on. This will allow you to dissolve a lot of complexity out of the code. We'll see this in a moment. 5. Change the computational structure. The computation of MYMAPS being done after the processing of gmlis is not right. It's the whole reason why there's this awkward intermediate myinternet structure that's used just to fill in each MYMAP later. Do the processing as part of your earlier loop. We know that gmlis is exactly five elements long. Just say so: gmlis = [my_parts[7], my_parts[8], my_parts[9], my_parts[10], my_parts[11]] Once you have this, and once you have a definition of MYMAPS, then you can kill a lot of the code by doing a zip loop across them both: for gammar, MYMAP in zip(gmlis, MYMAPS):
Re: [Tutor] improving speed using and recalling C functions
Comment: You are looping over your sliced eel five times. Do you need to? I like eel salad a great deal, as well, but, how about: for k in eel: MYMAP1[i, j, k] = MYMAP1[i, j, k] + myinternet[oo] MYMAP2[i, j, k] = MYMAP2[i, j, k] + myinternet[oo] MYMAP3[i, j, k] = MYMAP3[i, j, k] + myinternet[oo] MYMAP4[i, j, k] = MYMAP4[i, j, k] + myinternet[oo] MYMAP5[i, j, k] = MYMAP5[i, j, k] + myinternet[oo] oo = oo + 1 Hi Gabriele, Also note that, when Martin looked at this part of the code, he unfortunately misinterpreted its effect; Martin's proposed rewrite here does not preserve the meaning of the original code. But rather than wag my finger at how Martin interpreted the code, I'd rather make the observation that this is a warning sign that the original code here was not easy to understand. ___ Tutor maillist - Tutor@python.org To unsubscribe or change subscription options: https://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] improving speed using and recalling C functions
Hi Danny,
I followed your suggestion.
Tomorrow morning I will run this new version of the code.
Now using a sample of 81 elements (instead of 60) the profile returns:
Thu Apr 10 23:25:59 2014restats
18101188 function calls in 1218.626 seconds
Ordered by: internal time
List reduced from 13 to 10 due to restriction 10
ncalls tottime percall cumtime percall filename:lineno(function)
1 1015.803 1015.803 1218.334 1218.334 skymaps5.py:44(mymain)
18101000 202.4900.000 202.4900.000 {method 'write' of 'file'
objects}
10.2920.292 1218.626 1218.626 string:1(module)
60.0290.0050.0290.005 {open}
50.0100.0020.0100.002 {method 'close' of 'file'
objects}
810.0020.0000.0020.000 {method 'split' of 'str'
objects}
820.0010.0000.0010.000 {zip}
10.0000.0000.0000.000 function_base.py:8(linspace)
10.0000.0000.0000.000 function_base.py:93(logspace)
50.0000.0000.0000.000 {numpy.core.multiarray.zeros}
Anyway I would like to try to speed it up using C functions (and maybe
comparing the resuts of the two profile in the end)
How can I do it now? Can I use Cython?
Thanks
Gabriele
2014-04-10 21:38 GMT-04:00 Danny Yoo d...@hashcollision.org:
Comment: You are looping over your sliced eel five times. Do you
need to? I like eel salad a great deal, as well, but, how about:
for k in eel:
MYMAP1[i, j, k] = MYMAP1[i, j, k] + myinternet[oo]
MYMAP2[i, j, k] = MYMAP2[i, j, k] + myinternet[oo]
MYMAP3[i, j, k] = MYMAP3[i, j, k] + myinternet[oo]
MYMAP4[i, j, k] = MYMAP4[i, j, k] + myinternet[oo]
MYMAP5[i, j, k] = MYMAP5[i, j, k] + myinternet[oo]
oo = oo + 1
Hi Gabriele,
Also note that, when Martin looked at this part of the code, he
unfortunately misinterpreted its effect; Martin's proposed rewrite
here does not preserve the meaning of the original code. But rather
than wag my finger at how Martin interpreted the code, I'd rather make
the observation that this is a warning sign that the original code
here was not easy to understand.
___
Tutor maillist - Tutor@python.org
To unsubscribe or change subscription options:
https://mail.python.org/mailman/listinfo/tutor
