Re: list comprehension return a list and sum over in loop
On 13/12/2014 03:04, KK Sasa wrote: Sorry, i should say I'm using pythonxy, maybe it imports other things. That is good to know but without any context it's rather difficult to relate it to anything. Some people may have photographic memories and so remember everything that's been said in a thread, that certainly doesn't apply to me, and right now I'm just too lazy to go back and find out what this relates to :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: list comprehension return a list and sum over in loop
KK Sasa genwei...@gmail.com writes: Hi there, The list comprehension is results = [d2(t[k]) for k in xrange(1000)], where d2 is a function returning a list, say [x1,x2,x3,x4] for one example. So results is a list consisting of 1000 lists, each of length four. Here, what I want to get is the sum of 1000 lists, and then the result is a list of length four. Is there any efficient way to do this? Because I found it is slow in my case. I tried sum(d2(t[k]) for k in xrange(1000)), but it returned error: TypeError: unsupported operand type(s) for +: 'int' and 'list'. Thanks. Why didn't you follow Mark Lawrence's advice? In your problem, results is a list of N sublists, each containing exactly four numerical values, Let's try with N=2 In [36]: results = [d2(t[k]) for k in range(2)] In [37]: print results [[1, 2, 3, 4], [5, 6, 7, 8]] Let's try the obvious method to sum In [38]: [sum(el) for el in results] Out[38]: [10, 26] not what you're looking for, but what if we had In [39]: [sum(el) for el in zip(*results)] Out[39]: [6, 8, 10, 12] correct. BTW, as you're using the scientific stack the answer of Christian Gollwitzer is the more appropriate. -- https://mail.python.org/mailman/listinfo/python-list
Re: list comprehension return a list and sum over in loop
Mark Lawrence於 2014年12月12日星期五UTC+8下午3時17分43秒寫道： On 12/12/2014 06:22, KK Sasa wrote: Hi there, The list comprehension is results = [d2(t[k]) for k in xrange(1000)], where d2 is a function returning a list, say [x1,x2,x3,x4] for one example. So results is a list consisting of 1000 lists, each of length four. Here, what I want to get is the sum of 1000 lists, and then the result is a list of length four. Is there any efficient way to do this? Because I found it is slow in my case. I tried sum(d2(t[k]) for k in xrange(1000)), but it returned error: TypeError: unsupported operand type(s) for +: 'int' and 'list'. Thanks. I think you need something like this http://stackoverflow.com/questions/19339/a-transpose-unzip-function-in-python-inverse-of-zip I'll let you add the finishing touches if I'm correct :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence Hi Mark and Yotam, Thanks for kind reply. I think I didn't make my problem clear enough. The slow part is [d2(t[k]) for k in xrange(1000)]. In addition, I don't need to construct a list of 1000 lists inside, but my aim is to get the sum of all d2(t[k]). I wonder if there is any method to sum up efficiently. -- https://mail.python.org/mailman/listinfo/python-list
Re: list comprehension return a list and sum over in loop
Am 12.12.14 09:30, schrieb KK Sasa: Mark Lawrence於 2014年12月12日星期五UTC+8下午3時17分43秒寫道： Hi Mark and Yotam, Thanks for kind reply. I think I didn't make my problem clear enough. The slow part is [d2(t[k]) for k in xrange(1000)]. In addition, I don't need to construct a list of 1000 lists inside, but my aim is to get the sum of all d2(t[k]). I wonder if there is any method to sum up efficiently. Not sure I understand what you need, but it seems that NumPy would be a more efficient method. numpy.sum can sum elements along every dimensions of a higher-dimensional matrix, and in general NumPy stores the elements in native doubles instead of Python lists, which improves both performance and memory usage Christian -- https://mail.python.org/mailman/listinfo/python-list
Re: list comprehension return a list and sum over in loop
KK Sasa wrote: Mark Lawrence於 2014年12月12日星期五UTC+8下午3時17分43秒寫道： On 12/12/2014 06:22, KK Sasa wrote: Hi there, The list comprehension is results = [d2(t[k]) for k in xrange(1000)], where d2 is a function returning a list, say [x1,x2,x3,x4] for one example. So results is a list consisting of 1000 lists, each of length four. Here, what I want to get is the sum of 1000 lists, and then the result is a list of length four. Is there any efficient way to do this? Because I found it is slow in my case. I tried sum(d2(t[k]) for k in xrange(1000)), but it returned error: TypeError: unsupported operand type(s) for +: 'int' and 'list'. Thanks. I think you need something like this http://stackoverflow.com/questions/19339/a-transpose-unzip-function-in-python-inverse-of-zip I'll let you add the finishing touches if I'm correct :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence Hi Mark and Yotam, Thanks for kind reply. I think I didn't make my problem clear enough. The slow part is [d2(t[k]) for k in xrange(1000)]. In addition, I don't need to construct a list of 1000 lists inside, but my aim is to get the sum of all d2(t[k]). I wonder if there is any method to sum up efficiently. If that is slow the culprit is probably the d2() function. If so results = [0] * 4 for k in xrange(1000): for i, v in enumerate(d2(t[k])): results[i] += v won't help. Can you tell us what's inside d2()? -- https://mail.python.org/mailman/listinfo/python-list
Re: list comprehension return a list and sum over in loop
Peter Otten於 2014年12月12日星期五UTC+8下午5時13分58秒寫道： KK Sasa wrote: Mark Lawrence於 2014年12月12日星期五UTC+8下午3時17分43秒寫道： On 12/12/2014 06:22, KK Sasa wrote: Hi there, The list comprehension is results = [d2(t[k]) for k in xrange(1000)], where d2 is a function returning a list, say [x1,x2,x3,x4] for one example. So results is a list consisting of 1000 lists, each of length four. Here, what I want to get is the sum of 1000 lists, and then the result is a list of length four. Is there any efficient way to do this? Because I found it is slow in my case. I tried sum(d2(t[k]) for k in xrange(1000)), but it returned error: TypeError: unsupported operand type(s) for +: 'int' and 'list'. Thanks. I think you need something like this http://stackoverflow.com/questions/19339/a-transpose-unzip-function-in-python-inverse-of-zip I'll let you add the finishing touches if I'm correct :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence Hi Mark and Yotam, Thanks for kind reply. I think I didn't make my problem clear enough. The slow part is [d2(t[k]) for k in xrange(1000)]. In addition, I don't need to construct a list of 1000 lists inside, but my aim is to get the sum of all d2(t[k]). I wonder if there is any method to sum up efficiently. If that is slow the culprit is probably the d2() function. If so results = [0] * 4 for k in xrange(1000): for i, v in enumerate(d2(t[k])): results[i] += v won't help. Can you tell us what's inside d2()? Thanks for reply, Christian and Peter. Actually, the d2() is the Hessian function of a simple function (derived by using ad package, http://pythonhosted.org//ad/). The package hasn't supported Numpy array so far. And I am still not how to take a look inside d2(). Because I have to use d2() in a heavy way for iterative algorithm, the efficiency is a issue in my case. Following is an example. import scipy from scipy import stats import numpy from ad import adnumber from ad.admath import * from ad import jacobian from ad import gh # the gradient and hessian functions generator from ad import * import time people = 1000 range_people = xrange(people) dim0 = 2; mean0 = [0,0]; cov0 = [[1,0],[0,1]] seed([1]) t = stats.multivariate_normal.rvs(mean0,cov0,people) t = t.reshape(people,dim0) t = t.tolist() # back to list x = [0, 0, 1, 2] point = 2 def p(x,t,point,z,obs): d = x[0] tau = [0]+[x[1:point]] a = x[point:len(x)] at = sum(i*j for i, j in zip(a, t)) nu = [exp(z[k]*(at-d)-sum(tau[k])) for k in xrange(point)] de = sum(nu, axis=0) probability = [nu[k]/de for k in xrange(point)] return probability[obs] d1, d2 = gh(p) tStart = time.time() z = range(point) re = [d2(x,t[k],2,z,1) for k in range_people] tEnd = time.time() print It cost %f sec % (tEnd - tStart) -- https://mail.python.org/mailman/listinfo/python-list
Re: list comprehension return a list and sum over in loop
KK Sasa wrote: Hi there, The list comprehension is results = [d2(t[k]) for k in xrange(1000)], where d2 is a function returning a list, say [x1,x2,x3,x4] for one example. So results is a list consisting of 1000 lists, each of length four. Here, what I want to get is the sum of 1000 lists, and then the result is a list of length four. Is there any efficient way to do this? Because I found it is slow in my case. I tried sum(d2(t[k]) for k in xrange(1000)), but it returned error: TypeError: unsupported operand type(s) for +: 'int' and 'list'. Thanks. That's because sum() defaults to adding with a default value of 0: py sum([[1, 2], [3, 4]], 0) Traceback (most recent call last): File stdin, line 1, in module TypeError: unsupported operand type(s) for +: 'int' and 'list' py sum([[1, 2], [3, 4]], []) [1, 2, 3, 4] But don't do that! It will be slow. I don't completely understand your requirements. You should show some typical data, and the expected result. The business with the xrange and t[k] and even d2() is probably irrelevant. The important part is summing the lists. That could mean either of these two things: results = [ [1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]] sum(results) -- gives [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12] The way to do this efficiently is: results = [] for sublist in [d2(t[k]) for k in xrange(1000)]: results.extend(sublist) Or perhaps you mean this: results = [ [1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]] sum(results) -- gives [15, 18, 21, 24] I expect that the fastest, most efficient way to do this will be with the third-party library numpy. But in pure Python, you can do this: def add(alist, blist): if len(blist) != len(alist): raise ValueError for i, x in blist: alist[i] += x results = [0]*4 # Like [0,0,0,0] for sublist in [d2(t[k]) for k in xrange(1000)]: add(results, sublist) -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: list comprehension return a list and sum over in loop
KK Sasa writes: def p(x,t,point,z,obs): d = x[0] tau = [0]+[x[1:point]] a = x[point:len(x)] at = sum(i*j for i, j in zip(a, t)) nu = [exp(z[k]*(at-d)-sum(tau[k])) for k in xrange(point)] de = sum(nu, axis=0) probability = [nu[k]/de for k in xrange(point)] return probability[obs] I must be blind, but this looks like computing a whole probability distribution and then throwing almost all of it away. Can't this just return nu[obs]/de? The expression for tau also seems weird to me. Isn't it equivalent to [0, x[1:point]], a two-element list with the second element a list? How can sum(tau[k]) work at all then, for any k 1? -- https://mail.python.org/mailman/listinfo/python-list
Re: list comprehension return a list and sum over in loop
Jussi Piitulainen wrote: KK Sasa writes: def p(x,t,point,z,obs): d = x[0] tau = [0]+[x[1:point]] a = x[point:len(x)] at = sum(i*j for i, j in zip(a, t)) nu = [exp(z[k]*(at-d)-sum(tau[k])) for k in xrange(point)] de = sum(nu, axis=0) probability = [nu[k]/de for k in xrange(point)] return probability[obs] I must be blind, but this looks like computing a whole probability distribution and then throwing almost all of it away. Can't this just return nu[obs]/de? The expression for tau also seems weird to me. Isn't it equivalent to [0, x[1:point]], a two-element list with the second element a list? How can sum(tau[k]) work at all then, for any k 1? Also, after adding from numpy.random import seed to the code I run into the next problem: Traceback (most recent call last): File hessian.py, line 34, in module re = [d2(x,t[k],2,z,1) for k in range_people] File /home/petto/.local/lib/python2.7/site-packages/ad/__init__.py, line 1114, in hess return func(xa, *args).hessian([xa]) File hessian.py, line 26, in p nu = [exp(z[k]*(at-d)-sum(tau[k])) for k in xrange(point)] TypeError: 'int' object is not iterable OP: Optimizations only make sense when you can start from a known-good base. You should sort out the logic of your problem (we probably can't help you with that), then fix the bugs in your code and only then revisit the speed issue. -- https://mail.python.org/mailman/listinfo/python-list
Re: list comprehension return a list and sum over in loop
Jussi Piitulainen於 2014年12月12日星期五UTC+8下午7時12分39秒寫道： KK Sasa writes: def p(x,t,point,z,obs): d = x[0] tau = [0]+[x[1:point]] a = x[point:len(x)] at = sum(i*j for i, j in zip(a, t)) nu = [exp(z[k]*(at-d)-sum(tau[k])) for k in xrange(point)] de = sum(nu, axis=0) probability = [nu[k]/de for k in xrange(point)] return probability[obs] I must be blind, but this looks like computing a whole probability distribution and then throwing almost all of it away. Can't this just return nu[obs]/de? The expression for tau also seems weird to me. Isn't it equivalent to [0, x[1:point]], a two-element list with the second element a list? How can sum(tau[k]) work at all then, for any k 1? This is just a probability of binary response. Not continuous one. Tau is a parameter and just have two in this case. -- https://mail.python.org/mailman/listinfo/python-list
Re: list comprehension return a list and sum over in loop
Peter Otten於 2014年12月12日星期五UTC+8下午8時32分55秒寫道： Jussi Piitulainen wrote: KK Sasa writes: def p(x,t,point,z,obs): d = x[0] tau = [0]+[x[1:point]] a = x[point:len(x)] at = sum(i*j for i, j in zip(a, t)) nu = [exp(z[k]*(at-d)-sum(tau[k])) for k in xrange(point)] de = sum(nu, axis=0) probability = [nu[k]/de for k in xrange(point)] return probability[obs] I must be blind, but this looks like computing a whole probability distribution and then throwing almost all of it away. Can't this just return nu[obs]/de? The expression for tau also seems weird to me. Isn't it equivalent to [0, x[1:point]], a two-element list with the second element a list? How can sum(tau[k]) work at all then, for any k 1? Also, after adding from numpy.random import seed to the code I run into the next problem: Traceback (most recent call last): File hessian.py, line 34, in module re = [d2(x,t[k],2,z,1) for k in range_people] File /home/petto/.local/lib/python2.7/site-packages/ad/__init__.py, line 1114, in hess return func(xa, *args).hessian([xa]) File hessian.py, line 26, in p nu = [exp(z[k]*(at-d)-sum(tau[k])) for k in xrange(point)] TypeError: 'int' object is not iterable OP: Optimizations only make sense when you can start from a known-good base. You should sort out the logic of your problem (we probably can't help you with that), then fix the bugs in your code and only then revisit the speed issue. I have no idea why you added from numpy.random import seed to this syntax. Even I added that, my python didn't complain any bugs inside. Maybe your ad package was not be installed correctly? -- https://mail.python.org/mailman/listinfo/python-list
Re: list comprehension return a list and sum over in loop
KK Sasa wrote: Peter Otten於 2014年12月12日星期五UTC+8下午8時32分55秒寫道： Jussi Piitulainen wrote: KK Sasa writes: def p(x,t,point,z,obs): d = x[0] tau = [0]+[x[1:point]] a = x[point:len(x)] at = sum(i*j for i, j in zip(a, t)) nu = [exp(z[k]*(at-d)-sum(tau[k])) for k in xrange(point)] de = sum(nu, axis=0) probability = [nu[k]/de for k in xrange(point)] return probability[obs] I must be blind, but this looks like computing a whole probability distribution and then throwing almost all of it away. Can't this just return nu[obs]/de? The expression for tau also seems weird to me. Isn't it equivalent to [0, x[1:point]], a two-element list with the second element a list? How can sum(tau[k]) work at all then, for any k 1? Also, after adding from numpy.random import seed to the code I run into the next problem: Traceback (most recent call last): File hessian.py, line 34, in module re = [d2(x,t[k],2,z,1) for k in range_people] File /home/petto/.local/lib/python2.7/site-packages/ad/__init__.py, line 1114, in hess return func(xa, *args).hessian([xa]) File hessian.py, line 26, in p nu = [exp(z[k]*(at-d)-sum(tau[k])) for k in xrange(point)] TypeError: 'int' object is not iterable OP: Optimizations only make sense when you can start from a known-good base. You should sort out the logic of your problem (we probably can't help you with that), then fix the bugs in your code and only then revisit the speed issue. I have no idea why you added from numpy.random import seed to this syntax. Even I added that, my python didn't complain any bugs inside. Maybe your ad package was not be installed correctly? It may be a version issue. I'm using import ad, numpy ad.__version__ '1.2.2' numpy.__version__ '1.9.1' Or are you using an environment that does some imports automatically? -- https://mail.python.org/mailman/listinfo/python-list
Re: list comprehension return a list and sum over in loop
On 12 December 2014 at 06:22, KK Sasa genwei...@gmail.com wrote: Hi there, The list comprehension is results = [d2(t[k]) for k in xrange(1000)], where d2 is a function returning a list, say [x1,x2,x3,x4] for one example. So results is a list consisting of 1000 lists, each of length four. Here, what I want to get is the sum of 1000 lists, and then the result is a list of length four. Is there any efficient way to do this? Because I found it is slow in my case. I tried sum(d2(t[k]) for k in xrange(1000)), but it returned error: TypeError: unsupported operand type(s) for +: 'int' and 'list'. Thanks. Use numpy.sum: import numpy as np a = [[1, 1, 1, 1], [2, 2, 2, 2], [3, 3, 3, 3]] np.sum(a, 0) array([6, 6, 6, 6]) np.sum(a, 1) array([ 4, 8, 12]) The second argument to numpy.sum is the dimension along which you want to sum e.g. rows or columns. Oscar -- https://mail.python.org/mailman/listinfo/python-list
Re: list comprehension return a list and sum over in loop
Sorry, i should say I'm using pythonxy, maybe it imports other things. -- https://mail.python.org/mailman/listinfo/python-list
list comprehension return a list and sum over in loop
Hi there, The list comprehension is results = [d2(t[k]) for k in xrange(1000)], where d2 is a function returning a list, say [x1,x2,x3,x4] for one example. So results is a list consisting of 1000 lists, each of length four. Here, what I want to get is the sum of 1000 lists, and then the result is a list of length four. Is there any efficient way to do this? Because I found it is slow in my case. I tried sum(d2(t[k]) for k in xrange(1000)), but it returned error: TypeError: unsupported operand type(s) for +: 'int' and 'list'. Thanks. -- https://mail.python.org/mailman/listinfo/python-list
Re: list comprehension return a list and sum over in loop
On 12/12/2014 06:22, KK Sasa wrote: Hi there, The list comprehension is results = [d2(t[k]) for k in xrange(1000)], where d2 is a function returning a list, say [x1,x2,x3,x4] for one example. So results is a list consisting of 1000 lists, each of length four. Here, what I want to get is the sum of 1000 lists, and then the result is a list of length four. Is there any efficient way to do this? Because I found it is slow in my case. I tried sum(d2(t[k]) for k in xrange(1000)), but it returned error: TypeError: unsupported operand type(s) for +: 'int' and 'list'. Thanks. I think you need something like this http://stackoverflow.com/questions/19339/a-transpose-unzip-function-in-python-inverse-of-zip I'll let you add the finishing touches if I'm correct :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
