Re: [Numpy-discussion] A way to compute a 2d histogram
2006/11/14, Xavier Gnata [EMAIL PROTECTED]: Hi,IFAICS these new histograms versions have not yet been merged to svn.Are they problems to be solve before to be able to merge them?How could we help?No, I'm just overloaded with other stuff, I'll submit a patch on Trac today or tomorrow. Up to now, Travis did the merges, but since he looks pretty busy right now, I don't know when it'll show up on svn. DavidXavier Hi, Your histograms functions look fine for me :) As it is a quite usualoperation on an array, I would suggestto put it in numpy as numpy.histogram. IMHO, there is no point to create an numpy.stats only for histograms (or do you have plans to move other stats related function to numpy.stats?) Xavier. Nicolas, thanks for the bug report, I fooled around with argument passing and should have checked every case. You'll find the histogram function that deals with weights on the numpy trac ticket 189, http://projects.scipy.org/scipy/numpy/ticket/189 I'm waiting for some hints as to where the histogram function should reside (numpy.histogram, numpy.stats.histogram, ...) before submitting a patch . Salut, David 2006/10/25, Nicolas Champavert [EMAIL PROTECTED] mailto: [EMAIL PROTECTED]: Hi, it would be great if you could add the weight option in the 1D histogram too. Nicolas David Huard a écrit : Xavier, Here is the patch against svn. Please report any bug. I haven't had the time to test it extensively, something that should be done before commiting the patch to the repo. I'd appreciate your feedback. David 2006/10/24, David Huard [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] mailto: [EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: Hi Xavier, You could tweak histogram2d to do what you want, or you could give me a couple of days and I'll do it and let you know. If you want to help, you could write a test using your particular application and data. David2006/10/24, Xavier Gnata [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED]: Hi, I have a set of 3 1D large arrays. The first 2 one stand for the coordinates of particules and the last one for their masses. I would like to be able to plot this data ie to compute a 2D histogram summing the masses in each bin. I cannot find a way to do that without any loop on the indices resulting too a very slow function. I'm looking for an elegant way to do that with numpy (or scipy??) function. For instance, scipy.histogram2d cannot do the job because it only counts the number of samples in each bin. There is no way to deal with weights. Xavier.-- Xavier Gnata CRAL - Observatoire de Lyon 9, avenue Charles André 69561 Saint Genis Laval cedex Phone: +33 4 78 86 85 28 Fax: +33 4 78 86 83 86 E-mail: [EMAIL PROTECTED] mailto: [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] mailto:[EMAIL PROTECTED] - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net mailto: Numpy-discussion@lists.sourceforge.net mailto:Numpy-discussion@lists.sourceforge.net mailto: Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net mailto: Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid
Re: [Numpy-discussion] A way to compute a 2d histogram
I put the patch on Trac. Ticket 189.2006/11/14, Xavier Gnata [EMAIL PROTECTED]: Hi,IFAICS these new histograms versions have not yet been merged to svn.Are they problems to be solve before to be able to merge them?How could we help?Voice you support on Trac to replace histogram with the upgraded version. Thanks.David - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys - and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
[Numpy-discussion] glibc invalid pointer error
Hi, I have a script that crashes, but only if it runs over 9~10 hours, with the following backtrace from gdb. The script uses PyMC, and repeatedly calls ( 100) likelihood functions written in fortran and wrapped with f2py. Numpy: 1.0.dev3327Python: 2.4.3Does this backtrace give enough info to track the problem or do the gurus need more ?Thanks, David*** glibc detected *** free(): invalid pointer: 0x2c1257e0 *** Program received signal SIGABRT, Aborted.[Switching to Thread 46912504440528 (LWP 25269)]0x2b09011d in raise () from /lib/libc.so.6(gdb) backtrace#0 0x2b09011d in raise () from /lib/libc.so.6 #1 0x2b09184e in abort () from /lib/libc.so.6#2 0x2b0c4e41 in __fsetlocking () from /lib/libc.so.6#3 0x2b0ca90e in malloc_usable_size () from /lib/libc.so.6#4 0x2b0cac56 in free () from /lib/libc.so.6 #5 0x2bff7770 in PyArray_FromArray (arr=0x1569500, newtype=0x2c1257e0, flags=0) at arrayobject.c:7804#6 0x2bfece56 in PyArray_FromAny (op=0x1569500, newtype=0x0, min_depth=0, max_depth=0, flags=0, context=0x0) at arrayobject.c:8257#7 0x2bff40b1 in PyArray_MultiIterNew (n=2) at arrayobject.c:10253#8 0x2bff44bc in _broadcast_cast (out=0x62b5, in=0x6, castfunc=0x2bfbf5a0 DOUBLE_to_FLOAT, iswap=-1, oswap=6) at arrayobject.c:7445#9 0x2bffe301 in PyArray_CastToType (mp=0x156dca0, at=value optimized out, fortran_=0) at arrayobject.c:7344#10 0x2bffe785 in PyArray_FromScalar (scalar=0x1573b30, outcode=0x2c1257e0) at scalartypes.inc.src:219#11 0x2bfecff5 in PyArray_FromAny (op=0x1573b30, newtype=0x2c1257e0, min_depth=0, max_depth=value optimized out, flags=0, context=0x0) at arrayobject.c:8260#12 0x2aaab6038b7b in array_from_pyobj (type_num=11, dims=0x7f8f6200, rank=1, intent=value optimized out, obj=0x1573b30) at build/src.linux-x86_64-2.4/fortranobject.c:653 #13 0x2aaab6034aa9 in f2py_rout_flib_beta ( capi_self=value optimized out, capi_args=value optimized out, capi_keywds=value optimized out, f2py_func=0x2aaab603e830 beta_) at build/src.linux-x86_64-2.4/PyMC/flibmodule.c:2601#14 0x00414490 in PyObject_Call ()#15 0x00475de5 in PyEval_EvalFrame ()#16 0x004bdf69 in PyDescr_NewGetSet ()#17 0x004143eb in PyIter_Next () #18 0x0046ba53 in _PyUnicodeUCS4_IsNumeric ()#19 0x00477ab1 in PyEval_EvalFrame ()#20 0x004783ff in PyEval_EvalCodeEx ()#21 0x0047699b in PyEval_EvalFrame ()#22 0x00476ab6 in PyEval_EvalFrame () #23 0x00476ab6 in PyEval_EvalFrame ()#24 0x004783ff in PyEval_EvalCodeEx ()#25 0x0047699b in PyEval_EvalFrame ()#26 0x004783ff in PyEval_EvalCodeEx ()#27 0x0047699b in PyEval_EvalFrame () #28 0x004783ff in PyEval_EvalCodeEx ()#29 0x0047699b in PyEval_EvalFrame ()#30 0x004783ff in PyEval_EvalCodeEx ()#31 0x00478512 in PyEval_EvalCode ()#32 0x0049c222 in PyRun_FileExFlags () #33 0x0049c4ae in PyRun_SimpleFileExFlags ()#34 0x00410a80 in Py_Main ()#35 0x2b07d49b in __libc_start_main () from /lib/libc.so.6#36 0x0040ffba in _start () - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] glibc invalid pointer error
Ok, I'll update numpy and give it another try tonight. Regards, David2006/10/30, Fernando Perez [EMAIL PROTECTED]: On 10/30/06, David Huard [EMAIL PROTECTED] wrote: Hi, I have a script that crashes, but only if it runs over 9~10 hours, with the following backtrace from gdb. The script uses PyMC, and repeatedly calls ( 100) likelihood functions written in fortran and wrapped with f2py. Numpy: 1.0.dev3327 Python: 2.4.3This sounds awfully reminiscent of the bug I recently mentioned:http://aspn.activestate.com/ASPN/Mail/Message/numpy-discussion/3312099 We left a fresh run over the weekend, but my office mate is currentlyout of the office and his terminal is locked, so I don't know what theresult is.I'll report shortly: we followed Travis' instructions and ran with a fresh SVN build which includes the extra warnings he addedto the dealloc routines.You may want to try the same advice, perhapswith information from both of us the gurus may zero in on the problem, if indeed it is the same.Note that I'm not positive it's the same problem, and our backtracesaren't quite the same.But the rest of the scenario is similar:low-level memory crash from glibc, very long run is needed to fire the bug, potentially millions of calls to both numpy and to f2py-wrappedin-house libraries.Cheers,f-Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easierDownload IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642___Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/numpy-discussion - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] A way to compute a 2d histogram
Hi Xavier,You could tweak histogram2d to do what you want, or you could give me a couple of days and I'll do it and let you know. If you want to help, you could write a test using your particular application and data. David2006/10/24, Xavier Gnata [EMAIL PROTECTED]: Hi,I have a set of 3 1D large arrays.The first 2 one stand for the coordinates of particules and the last onefor their masses.I would like to be able to plot this data ie to compute a 2D histogramsumming the masses in each bin. I cannot find a way to do that without any loop on the indices resultingtoo a very slow function.I'm looking for an elegant way to do that with numpy (or scipy??) function.For instance, scipy.histogram2d cannot do the job because it only countsthe number of samples in each bin.There is no way to deal with weights.Xavier.--Xavier GnataCRAL - Observatoire de Lyon 9, avenue Charles André69561 Saint Genis Laval cedexPhone: +33 4 78 86 85 28Fax: +33 4 78 86 83 86E-mail: [EMAIL PROTECTED] -Using Tomcat but need to do more? Need to support web services, security?Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimohttp://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___Numpy-discussion mailing listNumpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] histogram complete makeover
Thanks for the comments, Here is the code for the new histogram, tests included. I'll wait for comments or suggestions before submitting a patch (numpy / scipy) ?CheersDavid
2006/10/18, Tim Hochberg [EMAIL PROTECTED]:
My $0.02:If histogram is going to get a makeover, particularly one that makes itmore complex than at present, it should probably be moved to SciPy.Failing that, it should be moved to a submodule of numpy with similar
statistical tools. Preferably with consistent interfaces for all of thefunctions.-Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easierDownload IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642___Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/numpy-discussion
# License: Scipy compatible
# Author: David Huard, 2006
from numpy import *
def histogram(a, bins=10, range=None, normed=False, weights=None, axis=None):
histogram(a, bins=10, range=None, normed=False, weights=None, axis=None)
- H, dict
Return the distribution of sample.
Parameters
--
a: Array sample.
bins:Number of bins, or
an array of bin edges, in which case the range is not used.
range: Lower and upper bin edges, default: [min, max].
normed: Boolean, if False, return the number of samples in each bin,
if True, return a frequency distribution.
weights: Sample weights.
axis:Specifies the dimension along which the histogram is computed.
Defaults to None, which aggregates the entire sample array.
Output
--
H:The number of samples in each bin.
If normed is True, H is a frequency distribution.
dict{
'edges': The bin edges, including the rightmost edge.
'upper': Upper outliers.
'lower': Lower outliers.
'bincenters': Center of bins.
}
Examples
x = random.rand(100,10)
H, Dict = histogram(x, bins=10, range=[0,1], normed=True)
H2, Dict = histogram(x, bins=10, range=[0,1], normed=True, axis=0)
See also: histogramnd
a = asarray(a)
if axis is None:
a = atleast_1d(a.ravel())
axis = 0
# Bin edges.
if not iterable(bins):
if range is None:
range = (a.min(), a.max())
mn, mx = [mi+0.0 for mi in range]
if mn == mx:
mn -= 0.5
mx += 0.5
edges = linspace(mn, mx, bins+1, endpoint=True)
else:
edges = asarray(bins, float)
dedges = diff(edges)
decimal = int(-log10(dedges.min())+6)
bincenters = edges[:-1] + dedges/2.
# apply_along_axis accepts only one array input, but we need to pass the
# weights along with the sample. The strategy here is to concatenate the
# weights array along axis, so the passed array contains [sample, weights].
# The array is then split back in __hist1d.
if weights is not None:
aw = concatenate((a, weights), axis)
weighted = True
else:
aw = a
weighted = False
count = apply_along_axis(__hist1d, axis, aw, edges, decimal, weighted)
# Outlier count
upper = count.take(array([-1]), axis)
lower = count.take(array([0]), axis)
# Non-outlier count
core = a.ndim*[slice(None)]
core[axis] = slice(1, -1)
hist = count[core]
if normed:
normalize = lambda x: atleast_1d(x/(x*dedges).sum())
hist = apply_along_axis(normalize, axis, hist)
return hist, {'edges':edges, 'lower':lower, 'upper':upper, \
'bincenters':bincenters}
def __hist1d(aw, edges, decimal, weighted):
Internal routine to compute the 1d histogram.
aw: sample, [weights]
edges: bin edges
decimal: approximation to put values lying on the rightmost edge in the last
bin.
weighted: Means that the weights are appended to array a.
Return the bin count or frequency if normed.
nbin = edges.shape[0]+1
if weighted:
count = zeros(nbin, dtype=float)
a,w = hsplit(aw,2)
w = w/w.mean()
else:
a = aw
count = zeros(nbin, dtype=int)
w = None
binindex = digitize(a, edges)
# Values that fall on an edge are put in the right bin.
# For the rightmost bin, we want values equal to the right
# edge to be counted in the last bin, and not as an outlier.
on_edge = where(around(a,decimal) == around(edges[-1], decimal))[0]
binindex[on_edge] -= 1
# Count the number of identical
[Numpy-discussion] Dimension reduction
Hi, Is there an elegant way to reduce an array but conserve the reduced dimension ?Currently, a = random.random((10,10,10)) a.sum(1).shape(10,10)but i'd like to keep (10,1,10) so I can do a/a.sum(1) directly. Thanks, David - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Dimension reduction
2006/10/19, Robert Kern [EMAIL PROTECTED]: def nonreducing_reducer(reducing_func, arr, axis): reduced = reducing_func(arr, axis=axis) shape = list(reduced.shape) axis = axis % len(arr.shape) shape.insert(axis, 1) reduced.shape = tuple(shape) return reducedCute !Here is another one:1dfunc = lambda x: atleast_1d(reducing_function(x))apply_along_axis(1dfunc, axis, arr)Is it something people often need ? Would it be worth, (for numpy 1.1 ?), to consider complex axis arguments as a wish to preserve the rank of the array? a.shape(n,m,o,p) a.sum(axis=1j) (n,1,o,p) a.sum(axis=1)(n,o,p)Thanks, David - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Moving docstrings from C to Python
I started to do the same with array methods, but before I spend too much time on it, I'd like to be sure I'm doing the right thing.1. In add_newdocs.py, add from numpy.core import ndarray2. then add an entry for each method, eg
add_docstring(ndarray.var, a.var(axis=None, dtype=None) Return the variance, a measure of the spread of a distribution. The variance is the average of the squared deviations from the mean,
i.e. var = mean((x - x.mean())**2). See also: std )3. in arraymethods.c, delete static char doc_var[] = ...remove doc_var in {var, (PyCFunction)array_variance,
METH_VARARGS|METH_KEYWORDS, doc_var},David2006/7/28, Sasha [EMAIL PROTECTED]:
On 7/28/06, Stefan van der Walt [EMAIL PROTECTED] wrote: Would anyone mind if the change was made?If not, where should they go? (numpy/add_newdocs.py or numpy/core/something)
Another +1 for numpy/add_newdocs.py and a suggestion: check forPy_OptimizeFlag 1 in add_newdoc so that docstrings are not loaded ifpython is invoked with -OO option.This willimprove import numpy
time and reduce the memory footprint.I'll make the change if no oneobjects.-Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share youropinions on IT business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV___Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.nethttps://lists.sourceforge.net/lists/listinfo/numpy-discussion
-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Histogram versus histogram2d
I have noticed some that the 1d histogram and 2d histogram.Thehistogram function bins everything between the elements of edges, and then includes everything greater than the last edge element in thelast bin.The histrogram2d function only bins in the range specifiedby edges.Is there a reason these two functions do not operate in the same way?Hi Mikolai, The reason is that I didn't like the way histogram handled outliers so I wrote histogram1d, histogram2d, and histogramdd to handle 1d, 2d and nd data series. I submitted those functions and only histogram2d got included in numpy, hence the clash. Travis suggested that histogram1d and histogramdd could go into scipy, but with the new compatibility paradigm, I suggest that the old histogram is moved into the compatibility module and histogram1d is renamed to histogram and put into the main namespace. histogramdd could indeed go into scipy.stats. I'll submit a new patch if there is some interest. The new function takes an axis argument so you can make an histogram out of a nd array rowwise or columnwise. Ouliers are not counted, and the bin array has length (nbin +1) (+1 for the right hand side edge). The new function will break some code relying on the old behavior, so its inclusion presupposes the agreement of the users. You can find the code at ticket 189.David - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] Moving docstrings from C to Python
2006/7/28, Stefan van der Walt [EMAIL PROTECTED]: Hi,Travis recently advertised the add_docstring and add_newdoc functions,which are used to add documentation to functions written in C.Thisbrings the advantage that you don't have to recompile everytime you change a docstrings, and also that the docstrings are much simpler toformat (no more \n\ tokens all over the place).I took all the docstrings from numpy.core.src.multiarray and formattedthem to be used with add_docstring.I have two questions: Would anyone mind if the change was made?I don't, I wanted to do the same thing. numpy/add_newdocs.py+1David - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy.var and numpy.std
Yes, In SVN, std has a docstring and I submitted one for var in ticket #174. David2006/7/20, Tom Denniston [EMAIL PROTECTED] :Are these functions supposed to compute variance and standarddeviation?They don't have docstrings and I can't find anything about them in Travis' manual.-Take Surveys. Earn Cash. Influence the Future of ITJoin SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cashhttp://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV ___Numpy-discussion mailing listNumpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion - Take Surveys. Earn Cash. Influence the Future of IT Join SourceForge.net's Techsay panel and you'll get the chance to share your opinions on IT business topics through brief surveys -- and earn cash http://www.techsay.com/default.php?page=join.phpp=sourceforgeCID=DEVDEV___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy.vectorize performance
2006/7/14, Nick Fotopoulos [EMAIL PROTECTED]: Any other suggestions?Hi Nick, I had some success by coding the integrand in fortran and wrapping it with f2py. If your probability density function is standard, you may find it in the flib library of the PyMC module of Chris Fonnesbeck ( a library of likelihood functions coded in f77) and save the trouble. Hope this helps, David - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] unique() should return a sorted array
Tim Hochberg wrote: My first question is: why? What's the attraction in returning a sorted answer here? Returning an unsorted array is potentially faster,
depending on the algorithm chosen, and sorting after the fact is trivial. If one was going to spend extra complexity on something, I'd think it would be better spent on preserving the input order.
There is a unique function in matlab that returns a sorted vector. I think a lot of people will expect a numpy and matlab functions with identical names to behave similarly. If we want to preserve the input order, we'd have to choose a convention about whose value's order is retained: do we keep the order of the first value found or the last one ?
Here is the benchmark. Sorry Norbert for not including your code the first time, it turns out that with Alain's suggestion, its the fastest one both for lists and arrays. x = rand(10)*100x = x.astype
('i')l = list(x)For array x:In [166]: timeit unique_alan(x) # with set instead of dict100 loops, best of 3: 8.8 ms per loopIn [167]: timeit unique_norbert(x)100 loops, best of 3: 8.8 ms per loop
In [168]: timeit unique_sasha(x)100 loops, best of 3: 10.8 ms per loopIn [169]: timeit unique(x)10 loops, best of 3: 50.4 ms per loopIn [170]: timeit unique1d(x)10 loops, best of 3: 13.2
ms per loopFor list l:In [196]: timeit unique_norbert(l)10 loops, best of 3: 29 ms per loopIn [197]: timeit unique_alan(l) # with set instead of dict10 loops, best of 3: 14.5 ms per loop
In [193]: timeit unique(l)10 loops, best of 3: 29.6 ms per loopNote:In Norbert function, setting sort=False for flattenable objects returns a sorted array anyway. So I'd suggest to remove the sort keyword, sort if the datatype is sortable, and don't sort if its not.
David
from numpy import *
x = rand(10)*100
x = x.astype('i')
l = list(x)
def unique_sasha(x):
s = sort(x)
r = empty(s.shape, float)
r[:-1] = s[1:]
r[-1] = NaN # Ensures that the last element is always kept, so that unique([1,1,1]) returns [1].
return s[r != s]
def unique_norbert(arr,sort=True):
if hasattr(arr,'flatten'):
tmp = arr.flatten()
tmp.sort()
idx = concatenate(([True],tmp[1:]!=tmp[:-1]))
return tmp[idx]
else: # for compatibility:
set = {}
for item in arr:
set[item] = None
if sort:
return asarray(sorted(set.keys()))
else:
return asarray(set.keys())
def unique_alan(arr,sort=True):
if hasattr(arr,'flatten'):
tmp = arr.flatten()
tmp.sort()
idx = concatenate(([True],tmp[1:]!=tmp[:-1]))
return tmp[idx]
else: #for compatability
items = list(set(arr))
if sort: items.sort()
return asarray(items)
-
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642
___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] suggestions for Matrix-related changes
2006/7/11, JJ [EMAIL PROTECTED]: 1) is it possible to get the function unique() to work with matrices, perhapswith a unique_rows() function to work with matrices of more than one column?The problem is that applying unique to different rows will return vectors with different lengths. So you could not return an array, much less a matrix. You'd have to return a list of arrays or 1D matrices. David - Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642 ___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
[Numpy-discussion] Missing docstrings
Hi, For the beta release, it would be nice if every function had a docstring, even a small one. See Ticket #174Missing docstrings:In [116]: ['issubsctype', 'unicode_', 'string', 'float96', 'pkgload', 'void', 'unicode0', 'void0', 'object0', 'memmap', 'nan_to_num', 'PackageLoader', 'object_', 'dtype', 'unsignedinteger', 'uintc', 'uint0', 'uint8', 'chararray', 'uint64', 'finfo', 'add_newdoc', 'array_repr', 'array_str', 'longlong', 'int16', 'mat', 'uint', 'correlate', 'int64', 'choose', 'complexfloating', 'recarray', 'mean', 'str_', 'ulonglong', 'matrix', 'uint32', 'byte', 'ctypes_load_library', 'signedinteger', 'ndim', 'number', 'bool8', 'msort', 'bool_', 'inexact', 'broadcast', 'short', 'ubyte', 'std', 'double', 'require', 'take', 'issubclass_', 'longfloat', 'deprecate', 'bincount', 'array2string', 'float64', 'ushort', 'float_', 'geterrobj', 'iterable', 'intp', 'flexible', 'sctype2char', 'longdouble', 'flatiter', 'generic', 'show_config', 'i0', 'uintp', 'character', 'uint16', 'float32', 'int32', 'integer', 'get_printoptions', 'seterrobj', 'add_docstring', 'intc', 'var', 'int_', 'histogram', 'issubdtype', 'int0', 'int8', 'record', 'obj2sctype', 'single', 'floating', 'test', 'string0'] In the process of adding docstrings, I noted some slight bugs:bincount returns an error if the input array mixes float and ints, while it accepts lists with mixed floats and ints.histogram does not deal gracefully with the range attribute. All values greater than range[1] are put in the last bin, which is probably not the expected behavior. Cheers, David Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
[Numpy-discussion] Missing docstrings
Hi, For the first release, it would be nice if every function had a docstring, even a small one. There are 279 callable items in the numpy namespace, and 94 of those lack a docstring, albeit most of those probably don't get much usage. To help the process, I filed Ticket #174 and attached a couple of docstrings, feel free to add your own. Missing docstrings: ['issubsctype', 'unicode_', 'string', 'float96', 'pkgload', 'void', 'unicode0', 'void0', 'object0', 'memmap', 'nan_to_num', 'PackageLoader', 'object_', 'dtype', 'unsignedinteger', 'uintc', 'uint0', 'uint8', 'chararray', 'uint64', 'finfo', 'add_newdoc', 'array_repr', 'array_str', 'longlong', 'int16', 'mat', 'uint', 'correlate', 'int64', 'choose', 'complexfloating', 'recarray', 'mean', 'str_', 'ulonglong', 'matrix', 'uint32', 'byte', 'ctypes_load_library', 'signedinteger', 'ndim', 'number', 'bool8', 'msort', 'bool_', 'inexact', 'broadcast', 'short', 'ubyte', 'std', 'double', 'require', 'take', 'issubclass_', 'longfloat', 'deprecate', 'bincount', 'array2string', 'float64', 'ushort', 'float_', 'geterrobj', 'iterable', 'intp', 'flexible', 'sctype2char', 'longdouble', 'flatiter', 'generic', 'show_config', 'i0', 'uintp', 'character', 'uint16', 'float32', 'int32', 'integer', 'get_printoptions', 'seterrobj', 'add_docstring', 'intc', 'var', 'int_', 'histogram', 'issubdtype', 'int0', 'int8', 'record', 'obj2sctype', 'single', 'floating', 'test', 'string0'] Cheers, David Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642___ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Re: [Numpy-discussion] unique() should return a sorted array
Here is a quick benchmark between numpy's unique, unique1d and sasha's unique:x = rand(10)*100x = x.astype('i')%timeit unique(x)10 loops, best of 3: 525 ms per loop%timeit unique_sasha(x)
100 loops, best of 3: 10.7 ms per looptimeit unique1d(x)100 loops, best of 3: 12.6 ms per loopSo I wonder what is the added value of unique? Could unique1d simply become unique ? Cheers,
DavidP.S.
I modified sasha's version to account for the case where all elements are identical, which returned an empty array.
def unique_sasha(x):
s = sort(x)
r = empty(s.shape, float)
r[:-1] = s[1:]
r[-1] = NaN
return s[r != s]2006/7/3, Robert Cimrman [EMAIL PROTECTED]:
Sasha wrote: On 7/2/06, Norbert Nemec [EMAIL PROTECTED] wrote: ... Does anybody know about the internals of the python set? How is
.keys() implemented? I somehow have really doubts about the efficiency of this method. Set implementation (Objects/setobject.c) is a copy and paste job from dictobject with values removed.As a result it is heavily optimized
for the case of string valued keys - a case that is almost irrelevant for numpy. I think something like the following (untested, 1d only) will probably be much faster and sorted:
def unique(x): s = sort(x) r = empty_like(s) r[:-1] = s[1:] r[-1] = s[0] return s[r != s]There are 1d array set operations like this already in numpy
(numpy/lib/arraysetops.py - unique1d, ...)r.Using Tomcat but need to do more? Need to support web services, security?Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimohttp://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642
___Numpy-discussion mailing listNumpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnkkid=120709bid=263057dat=121642___
Numpy-discussion mailing list
Numpy-discussion@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/numpy-discussion
