Re: [Numpy-discussion] Move scipy.org docs to Github?
On Mar 15, 2017 05:47, "Ralf Gommers" wrote: On Wed, Mar 15, 2017 at 3:21 PM, Matthew Brett wrote: > Hi, > > The scipy.org site is down at the moment, and has been for more than 36 > hours: > > https://github.com/numpy/numpy/issues/8779#issue-213781439 > > This has happened before: > > https://github.com/scipy/scipy.org/issues/187#issue-186426408 > > I think it was down for about 24 hours that time. > > From the number of people opening issues or commenting on the > scipy.org website this time, it seems to be causing quite a bit of > disruption. > > It seems to me that we would have a much better chances of avoiding > significant down-time, if we switched to hosting the docs on github > pages. > > What do y'all think? > Once the site is back up we should look at migrating to a better (hosted) infrastructure. I suspect that Github Pages won't work, we'll exceed or be close to exceeding both the 1 GB site size limit and the 100 GB/month bandwidth limit [1]. Rough bandwidth estimate (using page size from http://scipy.github.io/ devdocs/ and Alexa stats): 2 million visits per month, 2.5 page views per visit, 5 kb/page = 25 GB/month (html). Add to that pdf docs, which are ~20 MB in size: if only a small fraction of visitors download those, we'll be at >100 GB. Ralf [1] https://help.github.com/articles/what-is-github-pages/#usage-limits Would it be possible to include the PDF and zipped HTML as sources in GitHub releases and (if desired) link to those on the documentation page? That way only the individual HTML pages would need to be stored in GitHub pages, reducing both the size and bandwidth. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Deprecating matrices.
On Jan 6, 2017 20:28, "Ralf Gommers" wrote: On Sat, Jan 7, 2017 at 2:21 PM, CJ Carey wrote: > > On Fri, Jan 6, 2017 at 6:19 PM, Ralf Gommers > wrote: > >> This sounds like a reasonable idea. Timeline could be something like: >> >> 1. Now: create new package, deprecate np.matrix in docs. >> 2. In say 1.5 years: start issuing visible deprecation warnings in numpy >> 3. After 2020: remove matrix from numpy. >> >> Ralf >> > > I think this sounds reasonable, and reminds me of the deliberate > deprecation process taken for scipy.weave. I guess we'll see how successful > it was when 0.19 is released. > > The major problem I have with removing numpy matrices is the effect on > scipy.sparse, which mostly-consistently mimics numpy.matrix semantics and > often produces numpy.matrix results when densifying. The two are coupled > tightly enough that if numpy matrices go away, all of the existing sparse > matrix classes will have to go at the same time. > > I don't think that would be the end of the world, > Not the end of the world literally, but the impact would be pretty major. I think we're stuck with scipy.sparse, and may at some point will add a new sparse *array* implementation next to it. For scipy we will have to add a dependency on the new npmatrix package or vendor it. Ralf > but it's definitely something that should happen while scipy is still > pre-1.0, if it's ever going to happen. > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion So what about this: 1. Create a sparse array class 2. (optional) Refactor the sparse matrix class to be based on the sparse array class (may not be feasible) 3. Copy the spare matrix class into the matrix package 4. Deprecate the scipy sparse matrix class 5. Remove the scipy sparse matrix class when the numpy matrix class I don't know about the timeline, but this would just need to be done by 2020. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Deprecating matrices.
On Mon, Jan 2, 2017 at 8:36 PM, Charles R Harris wrote: > Hi All, > > Just throwing this click bait out for discussion. Now that the `@` > operator is available and things seem to be moving towards Python 3, > especially in the classroom, we should consider the real possibility of > deprecating the matrix type and later removing it. No doubt there are old > scripts that require them, but older versions of numpy are available for > those who need to run old scripts. > > Thoughts? > > Chuck > > What if the matrix class was split out into its own project, perhaps as a scikit. That way those who really need it can still use it. If there is sufficient desire for it, those who need it can maintain it. If not, it will hopefully it will take long enough for it to bitrot that everyone has transitioned. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Intel random number package
On Thu, Oct 27, 2016 at 12:12 PM, Nathaniel Smith wrote: > Ever notice how Anaconda doesn't provide pyfftw? They can't legally ship > both MKL and pyfftw, and they picked MKL. Anaconda does ship GPL code [1]. They even ship GPL code that depends on numpy, such as cvxcanon and pystan, and there doesn't seem to be anything that prevents me from installing them alongside the MKL version of numpy. So I don't see how it would be any different for pyfftw. [1] https://docs.continuum.io/anaconda/pkg-docs ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Intel random number package
It would still be compatible with SciPy, it would "just" mean that SciPy (and anything else that uses numpy) would be effectively GPL. On Thu, Oct 27, 2016 at 11:42 AM, Robert McLeod wrote: > Releasing NumPy under GPL would make it incompatible with SciPy, which may > be _slightly_ inconvenient to the scientific Python community: > > https://scipy.github.io/old-wiki/pages/License_Compatibility.html > > https://mail.scipy.org/pipermail/scipy-dev/2013-August/019149.html > > Robert > > On Thu, Oct 27, 2016 at 5:14 PM, Julian Taylor < > jtaylor.deb...@googlemail.com> wrote: > >> On 10/27/2016 04:52 PM, Todd wrote: >> >>> On Thu, Oct 27, 2016 at 10:43 AM, Julian Taylor >>> mailto:jtaylor.deb...@googlemail.com>> >>> wrote: >>> >>> On 10/27/2016 04:30 PM, Todd wrote: >>> >>> On Thu, Oct 27, 2016 at 4:25 AM, Ralf Gommers >>> mailto:ralf.gomm...@gmail.com> >>> <mailto:ralf.gomm...@gmail.com <mailto:ralf.gomm...@gmail.com>>> >>> wrote: >>> >>> >>> On Thu, Oct 27, 2016 at 10:25 AM, Pavlyk, Oleksandr >>> >> <mailto:oleksandr.pav...@intel.com> >>> <mailto:oleksandr.pav...@intel.com >>> <mailto:oleksandr.pav...@intel.com>>> wrote: >>> >>> Please see responses inline. >>> >>> >>> >>> *From:*NumPy-Discussion >>> [mailto:numpy-discussion-boun...@scipy.org >>> <mailto:numpy-discussion-boun...@scipy.org> >>> <mailto:numpy-discussion-boun...@scipy.org >>> <mailto:numpy-discussion-boun...@scipy.org>>] *On Behalf Of >>> *Todd >>> *Sent:* Wednesday, October 26, 2016 4:04 PM >>> *To:* Discussion of Numerical Python >>> mailto:numpy-discussion@scipy.org> >>> <mailto:numpy-discussion@scipy.org >>> <mailto:numpy-discussion@scipy.org>>> >>> *Subject:* Re: [Numpy-discussion] Intel random number >>> package >>> >>> >>> >>> >>> On Wed, Oct 26, 2016 at 4:30 PM, Pavlyk, Oleksandr >>> >> <mailto:oleksandr.pav...@intel.com> >>> <mailto:oleksandr.pav...@intel.com >>> >>> <mailto:oleksandr.pav...@intel.com>>> >>> wrote: >>> >>> Another point already raised by Nathaniel is that for >>> numpy's randomness ideally should provide a way to >>> override >>> default algorithm for sampling from a particular >>> distribution. For example RandomState object that >>> implements PCG may rely on default >>> acceptance-rejection >>> algorithm for sampling from Gamma, while the >>> RandomState >>> object that provides interface to MKL might want to >>> call >>> into MKL directly. >>> >>> >>> >>> The approach that pyfftw uses at least for scipy, which >>> may also >>> work here, is that you can monkey-patch the >>> scipy.fftpack module >>> at runtime, replacing it with pyfftw's drop-in >>> replacement. >>> scipy then proceeds to use pyfftw instead of its built-in >>> fftpack implementation. Might such an approach work >>> here? >>> Users can either use this alternative randomstate >>> replacement >>> directly, or they can replace numpy's with it at runtime >>> and >>> numpy will then proceed to use the alternative. >>> >>> >>> The only reason that pyfftw uses monkeypatching is that the >>> better >>> approach is not possible due to license constraints with >>> FFTW (it's >>> GPL). >>> >>> >>> Yes, that is exactly why I brought it up. Better approaches are >>> also >>> not possible with MKL due to license constraints. It is a very >>> similar >>> situatio
Re: [Numpy-discussion] Intel random number package
On Thu, Oct 27, 2016 at 10:43 AM, Julian Taylor < jtaylor.deb...@googlemail.com> wrote: > On 10/27/2016 04:30 PM, Todd wrote: > >> On Thu, Oct 27, 2016 at 4:25 AM, Ralf Gommers > <mailto:ralf.gomm...@gmail.com>> wrote: >> >> >> On Thu, Oct 27, 2016 at 10:25 AM, Pavlyk, Oleksandr >> mailto:oleksandr.pav...@intel.com>> >> wrote: >> >> Please see responses inline. >> >> >> >> *From:*NumPy-Discussion >> [mailto:numpy-discussion-boun...@scipy.org >> <mailto:numpy-discussion-boun...@scipy.org>] *On Behalf Of *Todd >> *Sent:* Wednesday, October 26, 2016 4:04 PM >> *To:* Discussion of Numerical Python > <mailto:numpy-discussion@scipy.org>> >> *Subject:* Re: [Numpy-discussion] Intel random number package >> >> >> >> >> On Wed, Oct 26, 2016 at 4:30 PM, Pavlyk, Oleksandr >> mailto:oleksandr.pav...@intel.com>> >> wrote: >> >> Another point already raised by Nathaniel is that for >> numpy's randomness ideally should provide a way to override >> default algorithm for sampling from a particular >> distribution. For example RandomState object that >> implements PCG may rely on default acceptance-rejection >> algorithm for sampling from Gamma, while the RandomState >> object that provides interface to MKL might want to call >> into MKL directly. >> >> >> >> The approach that pyfftw uses at least for scipy, which may also >> work here, is that you can monkey-patch the scipy.fftpack module >> at runtime, replacing it with pyfftw's drop-in replacement. >> scipy then proceeds to use pyfftw instead of its built-in >> fftpack implementation. Might such an approach work here? >> Users can either use this alternative randomstate replacement >> directly, or they can replace numpy's with it at runtime and >> numpy will then proceed to use the alternative. >> >> >> The only reason that pyfftw uses monkeypatching is that the better >> approach is not possible due to license constraints with FFTW (it's >> GPL). >> >> >> Yes, that is exactly why I brought it up. Better approaches are also >> not possible with MKL due to license constraints. It is a very similar >> situation overall. >> >> > Its not that similar, the better approach is certainly possible with FFTW, > the GPL is compatible with numpys license. It is only a concern users of > binary distributions. Nobody provided the code to use fftw yet, but it > would certainly be accepted. Although it is technically compatible, it would make numpy effectively GPL. Suggestions for this have been explicitly rejected on these grounds [1] [1] https://github.com/numpy/numpy/issues/3485 ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Intel random number package
On Thu, Oct 27, 2016 at 4:25 AM, Ralf Gommers wrote: > > > On Thu, Oct 27, 2016 at 10:25 AM, Pavlyk, Oleksandr < > oleksandr.pav...@intel.com> wrote: > >> Please see responses inline. >> >> >> >> *From:* NumPy-Discussion [mailto:numpy-discussion-boun...@scipy.org] *On >> Behalf Of *Todd >> *Sent:* Wednesday, October 26, 2016 4:04 PM >> *To:* Discussion of Numerical Python >> *Subject:* Re: [Numpy-discussion] Intel random number package >> >> >> >> On Wed, Oct 26, 2016 at 4:30 PM, Pavlyk, Oleksandr < >> oleksandr.pav...@intel.com> wrote: >> >> Another point already raised by Nathaniel is that for numpy's randomness >> ideally should provide a way to override default algorithm for sampling >> from a particular distribution. For example RandomState object that >> implements PCG may rely on default acceptance-rejection algorithm for >> sampling from Gamma, while the RandomState object that provides interface >> to MKL might want to call into MKL directly. >> >> >> >> The approach that pyfftw uses at least for scipy, which may also work >> here, is that you can monkey-patch the scipy.fftpack module at runtime, >> replacing it with pyfftw's drop-in replacement. scipy then proceeds to use >> pyfftw instead of its built-in fftpack implementation. Might such an >> approach work here? Users can either use this alternative randomstate >> replacement directly, or they can replace numpy's with it at runtime and >> numpy will then proceed to use the alternative. >> > > The only reason that pyfftw uses monkeypatching is that the better > approach is not possible due to license constraints with FFTW (it's GPL). > Yes, that is exactly why I brought it up. Better approaches are also not possible with MKL due to license constraints. It is a very similar situation overall. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Intel random number package
On Wed, Oct 26, 2016 at 4:30 PM, Pavlyk, Oleksandr < oleksandr.pav...@intel.com> wrote: > > The module under review, similarly to randomstate package, provides > alternative basic pseudo-random number generators (BRNGs), like MT2203, > MCG31, MRG32K3A, Wichmann-Hill. The scope of support differ, with > randomstate implementing some generators absent in MKL and vice-versa. > > Is there a reason that randomstate shouldn't implement those generators? > Thinking about the possibility of providing the functionality of this > module within the framework of randomstate, I find that randomstate > implements samplers from statistical distributions as functions that take > the state of the underlying BRNG, and produce a single variate, e.g.: > > https://github.com/bashtage/ng-numpy-randomstate/blob/master/randomstate/ > distributions.c#L23-L26 > > This design stands in a way of efficient use of MKL, which generates a > whole vector of variates at a time. This can be done faster than sampling a > variate at a time by using vectorized instructions. So I wrote > mkl_distributions.cpp to provide functions that return a given size vector > of sampled variates from each supported distribution. > I don't know a huge amount about pseudo-random number generators, but this seems superficially to be something that would benefit random number generation as a whole independently of whether MKL is used. Might it be possible to modify the numpy implementation to support this sort of vectorized approach? Another point already raised by Nathaniel is that for numpy's randomness > ideally should provide a way to override default algorithm for sampling > from a particular distribution. For example RandomState object that > implements PCG may rely on default acceptance-rejection algorithm for > sampling from Gamma, while the RandomState object that provides interface > to MKL might want to call into MKL directly. > The approach that pyfftw uses at least for scipy, which may also work here, is that you can monkey-patch the scipy.fftpack module at runtime, replacing it with pyfftw's drop-in replacement. scipy then proceeds to use pyfftw instead of its built-in fftpack implementation. Might such an approach work here? Users can either use this alternative randomstate replacement directly, or they can replace numpy's with it at runtime and numpy will then proceed to use the alternative. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ndarray.T2 for 2D transpose
On Thu, Apr 7, 2016 at 11:35 AM, wrote: > On Thu, Apr 7, 2016 at 11:13 AM, Todd wrote: > > On Wed, Apr 6, 2016 at 5:20 PM, Nathaniel Smith wrote: > >> > >> On Wed, Apr 6, 2016 at 10:43 AM, Todd wrote: > >> > > >> > My intention was to make linear algebra operations easier in numpy. > >> > With > >> > the @ operator available, it is now very easy to do basic linear > algebra > >> > on > >> > arrays without needing the matrix class. But getting an array into a > >> > state > >> > where you can use the @ operator effectively is currently pretty > verbose > >> > and > >> > confusing. I was trying to find a way to make the @ operator more > >> > useful. > >> > >> Can you elaborate on what you're doing that you find verbose and > >> confusing, maybe paste an example? I've never had any trouble like > >> this doing linear algebra with @ or dot (which have similar semantics > >> for 1d arrays), which is probably just because I've had different use > >> cases, but it's much easier to talk about these things with a concrete > >> example in front of us to put everyone on the same page. > >> > > > > Let's say you want to do a simple matrix multiplication example. You > create > > two example arrays like so: > > > >a = np.arange(20) > >b = np.arange(10, 50, 10) > > > > Now you want to do > > > > a.T @ b > > > > First you need to turn a into a 2D array. I can think of 10 ways to do > this > > off the top of my head, and there may be more: > > > > 1a) a[:, None] > > 1b) a[None] > > 1c) a[None, :] > > 2a) a.shape = (1, -1) > > 2b) a.shape = (-1, 1) > > 3a) a.reshape(1, -1) > > 3b) a.reshape(-1, 1) > > 4a) np.reshape(a, (1, -1)) > > 4b) np.reshape(a, (-1, 1)) > > 5) np.atleast_2d(a) > > > > 5 is pretty clear, and will work fine with any number of dimensions, but > is > > also long to type out when trying to do a simple example. The different > > variants of 1, 2, 3, and 4, however, will only work with 1D arrays > (making > > them less useful for functions), are not immediately obvious to me what > the > > result will be (I always need to try it to make sure the result is what I > > expect), and are easy to get mixed up in my opinion. They also require > > people keep a mental list of lots of ways to do what should be a very > simple > > task. > > > > Basically, my argument here is the same as the argument from pep465 for > the > > inclusion of the @ operator: > > > https://www.python.org/dev/peps/pep-0465/#transparent-syntax-is-especially-crucial-for-non-expert-programmers > > > > "A large proportion of scientific code is written by people who are > experts > > in their domain, but are not experts in programming. And there are many > > university courses run each year with titles like "Data analysis for > social > > scientists" which assume no programming background, and teach some > > combination of mathematical techniques, introduction to programming, and > the > > use of programming to implement these mathematical techniques, all > within a > > 10-15 week period. These courses are more and more often being taught in > > Python rather than special-purpose languages like R or Matlab. > > > > For these kinds of users, whose programming knowledge is fragile, the > > existence of a transparent mapping between formulas and code often means > the > > difference between succeeding and failing to write that code at all." > > This doesn't work because of the ambiguity between column and row vector. > > In most cases 1d vectors in statistics/econometrics are column > vectors. Sometime it takes me a long time to figure out whether an > author uses row or column vector for transpose. > > i.e. I often need x.T dot y which works for 1d and 2d to produce > inner product. > but the outer product would require most of the time a column vector > so it's defined as x dot x.T. > > I think keeping around explicitly 2d arrays if necessary is less error > prone and confusing. > > But I wouldn't mind a shortcut for atleast_2d (although more often I > need atleast_2dcol to translate formulas) > > At least from what I have seen, in all cases in numpy where a 1D array is treated as a 2D array, it is always treated as a row vector, the examples I can think of being atleast_2d, hstack, vstack, and dstack. So using this convention would be in line with how it is used elsewhere in numpy. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ndarray.T2 for 2D transpose
On Thu, Apr 7, 2016 at 3:39 AM, Irvin Probst wrote: > On 06/04/2016 04:11, Todd wrote: > > When you try to transpose a 1D array, it does nothing. This is the > correct behavior, since it transposing a 1D array is meaningless. However, > this can often lead to unexpected errors since this is rarely what you > want. You can convert the array to 2D, using `np.atleast_2d` or > `arr[None]`, but this makes simple linear algebra computations more > difficult. > > I propose adding an argument to transpose, perhaps called `expand` or > `expanddim`, which if `True` (it is `False` by default) will force the > array to be at least 2D. A shortcut property, `ndarray.T2`, would be the > same as `ndarray.transpose(True)` > > Hello, > My two cents here, I've seen hundreds of people (literally hundreds) > stumbling on this .T trick with 1D vectors when they were trying to do some > linear algebra with numpy so at first I had the same feeling as you. But > the real issue was that *all* these people were coming from matlab and > expected numpy to behave the same way. Once the logic behind 1D vectors was > explained it made sense to most of them and there were no more problems. > > The problem isn't necessarily understanding, although that is a problem. The bigger problem is having to jump through hoops to do basic matrix math. > And by the way I don't see any way to tell apart a 1D "row vector" from a > 1D "column vector", think of a code mixing a Rn=>R jacobian matrix and some > data supposed to be used as measurements in a linear system, so we have > J=np.array([1,2,3,4]) and B=np.array([5,6,7,8]), what would the output of > J.T2 and B.T2 be ? > > As I said elsewhere, we already have a convention for this established by `np.atleast_2d`. 1D arrays are treated as row vectors. `np.hstack` and `np.vstack` also treat 1D arrays as row vectors. So `arr.T2` will follow this convention, being equivalent to `np.atleast_2d(arr).T`. > I think it's much better to get used to writing > J=np.array([1,2,3,4]).reshape(1,4) and B=np.array([5,6,7,8]).reshape(4,1), > then you can use .T and @ without any verbosity and at least if forces > users (read "my students" here) to think twice before writing some linear > algebra nonsense. > > That works okay when you know beforehand what the shape of the array is (although it may very well be the different between a simple, 1-line piece of code and a 3-line piece of code). But what if you try to turn this into a general-purpose function? Then any function that has linear algebra needs to call `atleast_2d` on every value used in that linear algebra, or use `if` tests. And if you forget, it may not be obvious until much later depending on what you initially use the function for and what you use it for later. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ndarray.T2 for 2D transpose
On Thu, Apr 7, 2016 at 4:59 AM, Joseph Martinot-Lagarde < contreba...@gmail.com> wrote: > Alan Isaac gmail.com> writes: > > > But underlying the proposal is apparently the > > idea that there be an attribute equivalent to > > `atleast_2d`. Then call it `d2p`. > > You can now have `a.d2p.T` which is a lot > > more explicit and general than say `a.T2`, > > while requiring only 3 more keystrokes. > > > How about a.T2d or a .T2D ? > > I thought of that, but I wanted to keep things as short as possible (but not shorter). ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ndarray.T2 for 2D transpose
On Wed, Apr 6, 2016 at 5:20 PM, Nathaniel Smith wrote: > On Wed, Apr 6, 2016 at 10:43 AM, Todd wrote: > > > > My intention was to make linear algebra operations easier in numpy. With > > the @ operator available, it is now very easy to do basic linear algebra > on > > arrays without needing the matrix class. But getting an array into a > state > > where you can use the @ operator effectively is currently pretty verbose > and > > confusing. I was trying to find a way to make the @ operator more > useful. > > Can you elaborate on what you're doing that you find verbose and > confusing, maybe paste an example? I've never had any trouble like > this doing linear algebra with @ or dot (which have similar semantics > for 1d arrays), which is probably just because I've had different use > cases, but it's much easier to talk about these things with a concrete > example in front of us to put everyone on the same page. > > Let's say you want to do a simple matrix multiplication example. You create two example arrays like so: a = np.arange(20) b = np.arange(10, 50, 10) Now you want to do a.T @ b First you need to turn a into a 2D array. I can think of 10 ways to do this off the top of my head, and there may be more: 1a) a[:, None] 1b) a[None] 1c) a[None, :] 2a) a.shape = (1, -1) 2b) a.shape = (-1, 1) 3a) a.reshape(1, -1) 3b) a.reshape(-1, 1) 4a) np.reshape(a, (1, -1)) 4b) np.reshape(a, (-1, 1)) 5) np.atleast_2d(a) 5 is pretty clear, and will work fine with any number of dimensions, but is also long to type out when trying to do a simple example. The different variants of 1, 2, 3, and 4, however, will only work with 1D arrays (making them less useful for functions), are not immediately obvious to me what the result will be (I always need to try it to make sure the result is what I expect), and are easy to get mixed up in my opinion. They also require people keep a mental list of lots of ways to do what should be a very simple task. Basically, my argument here is the same as the argument from pep465 for the inclusion of the @ operator: https://www.python.org/dev/peps/pep-0465/#transparent-syntax-is-especially-crucial-for-non-expert-programmers "A large proportion of scientific code is written by people who are experts in their domain, but are not experts in programming. And there are many university courses run each year with titles like "Data analysis for social scientists" which assume no programming background, and teach some combination of mathematical techniques, introduction to programming, and the use of programming to implement these mathematical techniques, all within a 10-15 week period. These courses are more and more often being taught in Python rather than special-purpose languages like R or Matlab. For these kinds of users, whose programming knowledge is fragile, the existence of a transparent mapping between formulas and code often means the difference between succeeding and failing to write that code at all." ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ndarray.T2 for 2D transpose
On Wed, Apr 6, 2016 at 11:44 AM, Chris Barker - NOAA Federal < chris.bar...@noaa.gov> wrote: > But the truth is that Numpy arrays are arrays, not matrices and vectors. > > The "right" way to do this is to properly extend and support the > matrix object, adding row and column vector objects, and then it would > be clear. But while there has been a lot of discussion about that in > the past, the fact is that no one wants it bad enough to write the > code. > > So I think it's better to keep Numpy arrays "pure", and if you want to > change the rank of an array, you do so explicitly. > I think that cat is already out of the bag. As long as you can do matrix multiplication on arrays using the @ operator, I think they aren't really "pure" anymore. > BTW, if transposing a (N,) array gives you a (N,1) array, what does > transposing a (N,1) array give you? > > (1,N) or (N,) ? > My suggestion is that this explicitly increases the number of dimensions to at least 2. The result will always have at least 2 dimensions. So 0D -> 2D, 1D -> 2D, 2D -> 2D, 3D -> 3D, 4D -> 4D, etc. So this would be equivalent to the existing `atleast_2d` function. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ndarray.T2 for 2D transpose
On Tue, Apr 5, 2016 at 11:14 PM, Nathaniel Smith wrote: > On Tue, Apr 5, 2016 at 7:11 PM, Todd wrote: > > When you try to transpose a 1D array, it does nothing. This is the > correct > > behavior, since it transposing a 1D array is meaningless. However, this > can > > often lead to unexpected errors since this is rarely what you want. You > can > > convert the array to 2D, using `np.atleast_2d` or `arr[None]`, but this > > makes simple linear algebra computations more difficult. > > > > I propose adding an argument to transpose, perhaps called `expand` or > > `expanddim`, which if `True` (it is `False` by default) will force the > array > > to be at least 2D. A shortcut property, `ndarray.T2`, would be the same > as > > `ndarray.transpose(True)`. > > An alternative that was mentioned in the bug tracker > (https://github.com/numpy/numpy/issues/7495), possibly by me, would be > to have arr.T2 act as a stacked-transpose operator, i.e. treat an arr > with shape (..., n, m) as being a (...)-shaped stack of (n, m) > matrices, and transpose each of those matrices, so the output shape is > (..., m, n). And since this operation intrinsically acts on arrays > with shape (..., n, m) then trying to apply it to a 0d or 1d array > would be an error. > > My intention was to make linear algebra operations easier in numpy. With the @ operator available, it is now very easy to do basic linear algebra on arrays without needing the matrix class. But getting an array into a state where you can use the @ operator effectively is currently pretty verbose and confusing. I was trying to find a way to make the @ operator more useful. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ndarray.T2 for 2D transpose
I would make `arr.T2` the same as `np.atleast_2d(arr).T`. So a 1D array would act as a row vector, since that is already the convention for coercing 1D arrays to 2D. On Tue, Apr 5, 2016 at 10:49 PM, Juan Nunez-Iglesias wrote: > Todd, > > Would you consider a 1D array to be a row vector or a column vector for > the purposes of transposition? The "correct" answer is not clear to me. > > Juan. > > On Wed, Apr 6, 2016 at 12:26 PM, Alan Isaac wrote: > >> On 4/5/2016 10:11 PM, Todd wrote: >> >>> When you try to transpose a 1D array, it does nothing. This is the >>> correct behavior, since it transposing a 1D array is meaningless. >>> However, this can often lead to unexpected errors since this is rarely >>> what you want. You can convert the array to 2D, using `np.atleast_2d` >>> or `arr[None]`, but this makes simple linear algebra computations more >>> difficult. >>> >>> I propose adding an argument to transpose, perhaps called `expand` or >>> `expanddim`, which if `True` (it is `False` by default) will force the >>> array to be at least 2D. A shortcut property, `ndarray.T2`, would be >>> the same as `ndarray.transpose(True)`. >>> >> >> >> >> Use `dot`. E.g., >> m.dot(a) >> >> hth, >> Alan Isaac >> >> >> >> ___ >> NumPy-Discussion mailing list >> NumPy-Discussion@scipy.org >> https://mail.scipy.org/mailman/listinfo/numpy-discussion >> > > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > https://mail.scipy.org/mailman/listinfo/numpy-discussion > > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] ndarray.T2 for 2D transpose
When you try to transpose a 1D array, it does nothing. This is the correct behavior, since it transposing a 1D array is meaningless. However, this can often lead to unexpected errors since this is rarely what you want. You can convert the array to 2D, using `np.atleast_2d` or `arr[None]`, but this makes simple linear algebra computations more difficult. I propose adding an argument to transpose, perhaps called `expand` or `expanddim`, which if `True` (it is `False` by default) will force the array to be at least 2D. A shortcut property, `ndarray.T2`, would be the same as `ndarray.transpose(True)`. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposal: stop supporting 'setup.py install'; start requiring 'pip install .' instead
On Oct 29, 2015 00:29, "Sandro Tosi" wrote: > > please, pretty please, do not disable setup.py install or at least > keep providing a way for distribution (Debian in this case) to be able > to build/install numpy in a temporary location for packaging reasons. > pip is not the solution for us What is wrong with "pip install --root" ? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org https://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Question about unaligned access
On Jul 6, 2015 6:21 PM, "Francesc Alted" wrote:
>
> 2015-07-06 18:04 GMT+02:00 Jaime Fernández del Río :
>>
>> On Mon, Jul 6, 2015 at 10:18 AM, Francesc Alted wrote:
>>>
>>> Hi,
>>>
>>> I have stumbled into this:
>>>
>>> In [62]: sa = np.fromiter(((i,i) for i in range(1000*1000)),
dtype=[('f0', np.int64), ('f1', np.int32)])
>>>
>>> In [63]: %timeit sa['f0'].sum()
>>> 100 loops, best of 3: 4.52 ms per loop
>>>
>>> In [64]: sa = np.fromiter(((i,i) for i in range(1000*1000)),
dtype=[('f0', np.int64), ('f1', np.int64)])
>>>
>>> In [65]: %timeit sa['f0'].sum()
>>> 1000 loops, best of 3: 896 µs per loop
>>>
>>> The first structured array is made of 12-byte records, while the second
is made by 16-byte records, but the latter performs 5x faster. Also, using
an structured array that is made of 8-byte records is the fastest
(expected):
>>>
>>> In [66]: sa = np.fromiter(((i,) for i in range(1000*1000)),
dtype=[('f0', np.int64)])
>>>
>>> In [67]: %timeit sa['f0'].sum()
>>> 1000 loops, best of 3: 567 µs per loop
>>>
>>> Now, my laptop has a Ivy Bridge processor (i5-3380M) that should
perform quite well on unaligned data:
>>>
>>>
http://lemire.me/blog/archives/2012/05/31/data-alignment-for-speed-myth-or-reality/
>>>
>>> So, if 4 years-old Intel architectures do not have a penalty for
unaligned access, why I am seeing that in NumPy? That strikes like a quite
strange thing to me.
>>
>>
>> I believe that the way numpy is setup, it never does unaligned access,
regardless of the platform, in case it gets run on one that would go up in
flames if you tried to. So my guess would be that you are seeing chunked
copies into a buffer, as opposed to bulk copying or no copying at all, and
that would explain your timing differences. But Julian or Sebastian can
probably give you a more informed answer.
>
>
> Yes, my guess is that you are right. I suppose that it is possible to
improve the numpy codebase to accelerate this particular access pattern on
Intel platforms, but provided that structured arrays are not that used
(pandas is probably leading this use case by far, and as far as I know,
they are not using structured arrays internally in DataFrames), then maybe
it is not worth to worry about this too much.
>
That may be more of a chicken-and-egg problem. Structured arrays are pretty
complicated to set up, which means they don't get used much, which means
they don't get much attention, which means they remain complicated.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Readings about governance and decision making for F/OSS projects
On Jul 4, 2015 1:47 AM, "Nathaniel Smith" wrote: > > > If you have other links on this topic that you are think are > interesting, please add them to the thread! > The KDE e.v. - represents the KDE community in legal and financial matters. https://ev.kde.org/ ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Verify your sourceforge windows installer downloads
On Mon, Jun 1, 2015 at 3:43 AM, Ralf Gommers wrote: > > > On Fri, May 29, 2015 at 7:28 PM, Benjamin Root wrote: > >> Speaking from the matplotlib project, our binaries are substantial due to >> our suite of test images. Pypi worked with us on relaxing size constraints. >> Also, I think the new cheese shop/warehouse server they are using scales >> better, so size is not nearly the same concern as before. >> >> Ben Root >> On May 29, 2015 1:43 AM, "Todd" wrote: >> >>> On May 28, 2015 7:06 PM, "David Cournapeau" wrote: >>> > On Fri, May 29, 2015 at 2:00 AM, Andrew Collette < >>> andrew.colle...@gmail.com> wrote: >>> >> >>> >> In any case I've always been surprised that NumPy is distributed >>> >> through SourceForge, which has been sketchy for years now. Could it >>> >> simply be hosted on PyPI? >>> > >>> > >>> > They don't accept arbitrary binaries like SF does, and some of our >>> installer formats can't be uploaded there. >>> > >>> > David >>> >>> Is that something that could be fixed? >>> >> > For the current .exe installers that cannot be fixed, because neither pip > nor easy_install can handle those. We actually have to ensure that we don't > link from pypi directly to the sourceforge folder with the latest release, > because then easy_install will follow the link, download the .exe and fail. > > Dmg's were another non-supported format, but we'll stop using those. So > if/when it's SSE2 .exe installers only (make with bdist_wininst and no > NSIS) then PyPi works. Size constraints are not an issue for Numpy I think. > > Ralf > What about adding some mechanism in pypi to flag that certain files should not by downloaded with pip? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Verify your sourceforge windows installer downloads
On May 28, 2015 7:06 PM, "David Cournapeau" wrote: > On Fri, May 29, 2015 at 2:00 AM, Andrew Collette < andrew.colle...@gmail.com> wrote: >> >> In any case I've always been surprised that NumPy is distributed >> through SourceForge, which has been sketchy for years now. Could it >> simply be hosted on PyPI? > > > They don't accept arbitrary binaries like SF does, and some of our installer formats can't be uploaded there. > > David Is that something that could be fixed? Has anyone asked the pypi maintainers whether they could change those rules, either in general or by granting exceptions on a case-by-case basis to projects that have proven track records and importance? It would seem to me that if the rules on pypi are forcing critical projects like numpy to host elsewhere, then the rules are flawed and are preventing pypi from serving is intended purpose. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Advanced indexing: "fancy" vs. orthogonal
On Apr 4, 2015 10:54 AM, "Nathaniel Smith" wrote: > > On Sat, Apr 4, 2015 at 12:17 AM, Ralf Gommers wrote: > > > > > > On Sat, Apr 4, 2015 at 1:54 AM, Nathaniel Smith wrote: > >> > >> > >> But, the real problem here is that we have two different array duck > >> types that force everyone to write their code twice. This is a > >> terrible state of affairs! (And exactly analogous to the problems > >> caused by np.ndarray disagreeing with np.matrix & scipy.sparse about > >> the the proper definition of *, which PEP 465 may eventually > >> alleviate.) IMO we should be solving this indexing problem directly, > >> not applying bandaids to its symptoms, and the way to do that is to > >> come up with some common duck type that everyone can agree on. > >> > >> Unfortunately, AFAICT this means our only options here are to have > >> some kind of backcompat break in numpy, some kind of backcompat break > >> in pandas, or to do nothing and continue indefinitely with the status > >> quo where the same indexing operation might silently return different > >> results depending on the types passed in. All of these options have > >> real costs for users, and it isn't at all clear to me what the > >> relative costs will be when we dig into the details of our various > >> options. > > > > > > I doubt that there is a reasonable way to quantify those costs, especially > > those of breaking backwards compatibility. If someone has a good method, I'd > > be interested though. > > I'm a little nervous about how easily this argument might turn into > "either A or B is better but we can't be 100% *certain* which it is so > instead of doing our best using the data available we should just > choose B." Being a maintainer means accepting uncertainty and doing > our best anyway. I think the burden of proof needs to be on the side proposing a change, and the more invasive the change the higher that burden needs to be. When faced with a situation like this, where the proposed change will cause fundamental alterations to the most basic, high-level operation of numpy, and where the is an alternative approach with no backwards-compatibility issues, I think the burden of proof would necessarily be nearly impossibly large. > But that said I'm still totally on board with erring on the side of > caution (in particular, you can never go back and *un*break > backcompat). An obvious challenge to anyone trying to take this > forward (in any direction!) would definitely be to gather the most > useful data possible. And it's not obviously impossible -- maybe one > could do something useful by scanning ASTs of lots of packages (I have > a copy of pypi if anyone wants it, that I downloaded with the idea of > making some similar arguments for why core python should slightly > break backcompat to allow overloading of a < b < c syntax), or adding > instrumentation to numpy, or running small-scale usability tests, or > surveying people, or ... > > (I was pretty surprised by some of the data gathered during the PEP > 465 process, e.g. on how common dot() calls are relative to existing > built-in operators, and on its associativity in practice.) Surveys like this have the problem of small sample size and selection bias. Usability studies can't measure the effect of the compatibility break, not to mention the effect on numpy's reputation. This is considerably more difficult to scan existing projects for than .dot because it depends on the type being passed (which may not even be defined in the same project). And I am not sure I much like the idea of numpy "phoning home" by default, and an opt-in had the same issues as a survey. So to make a long story short, in this sort of situation I have a hard time imaging ways to get enough reliable, representative data to justify this level of backwards compatibility break. > Core python broke backcompat on a regular basis throughout the python > 2 series, and almost certainly will again -- the bar to doing so is > *very* high, and they use elaborate mechanisms to ease the way > (__future__, etc.), but they do it. A few months ago there was even > some serious consideration given to changing py3 bytestring indexing > to return bytestrings instead of integers. (Consensus was > unsurprisingly that this was a bad idea, but there were core devs > seriously exploring it, and no-one complained about the optics.) There was no break as large as this. In fact I would say this is even a larger change than any individual change we saw in the python 2 to 3 switch. The basic mechanics of indexing are just too fundamental and touch on too many things to make this sort of change feasible. It would be better to have a new language, or in this case anew project. > It's true that numpy has something of a bad reputation in this area, > and I think it's because until ~1.7 or so, we randomly broke stuff by > accident on a pretty regular basis, even in "bug fix" releases. I > think the way to rebuild that trust is to honestly say to
Re: [Numpy-discussion] GSoC'15 - mentors & ideas
I am not able to mentor, but I have some ideas about easier projects. These may be too easy, too hard, or not even desirable so take them or leave them as you please. scipy: Implement a set of circular statistics functions comparable to those in R or MATLAB circular statistics toolbox. Either implement some window functions that only apply to the beginning and end of an array, or implement a wrapper that takes a window function and some parameters and creates a new window that only applies to the beginning and end of an array. numpy: Integrate the bottleneck project optimizations into numpy proper. Integrate as much as possible the matplotlib.mlab functionality into numpy (and, optionally, also scipy). In many places different approaches to the same task have substantially different performance (such as indexing vs. take) and check for one approach being substantially slower. If it is, fix the performance problem if possible (perhaps by using the same implementation), and if not document the difference. Modify ufuncs so their documentation appears in help() in addition to numpy.info(). Hi all, On Fri, Feb 20, 2015 at 10:05 AM, Ralf Gommers wrote: > Hi all, > > It's time to start preparing for this year's Google Summer of Code. There > is actually one urgent thing to be done (before 19.00 UTC today), which is > to get our ideas page in decent shape. It doesn't have to be final, but > there has to be enough on there for the organizers to judge it. This page > is here: https://github.com/scipy/scipy/wiki/GSoC-project-ideas. I'll be > reworking it and linking it from the PSF page today, but if you already > have new ideas please add them there. See > https://wiki.python.org/moin/SummerOfCode/OrgIdeasPageTemplate for this > year's template for adding a new idea. > The ideas page is now in pretty good shape. More ideas are very welcome though, especially easy or easy/intermediate ideas. Numpy right now has zero easy ones and Scipy only one and a half. What we also need is mentors. All ideas already have a potential mentor listed, however some ideas are from last year and I'm not sure that all those mentors really are available this year. And more than one potential mentor per idea is always good. So can everyone please add/remove his or her name on that page? I'm happy to take care of most of the organizational aspects this year, however I'll be offline for two weeks in July and from the end of August onwards, so I'll some help in those periods. Any volunteers? Thanks, Ralf ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Silent Broadcasting considered harmful
On Feb 10, 2015 1:03 AM, "cjw" wrote: > > > On 09-Feb-15 2:34 AM, Stefan Reiterer wrote: >> >> Ok that are indeed some good reasons to keep the status quo, especially since >> performance is crucial for numpy. >> It's a dillemma: Using the matrix class for linear algebra would be the correct >> way for such thing, >> but the matrix API is not that powerful and beautiful as the one of arrays. >> On the other hand arrays are beautiful, but not exactly intended to use for >> linear algebra. >> So maybe the better way would be not to add warnings to braodcasting operations, >> but to overhaul the matrix class >> to make it more attractive for numerical linear algebra(?) > > +1 > I hope that this will be explored. @ could still be used by those who wish remain in the array world. > What about splitting it off into a scikit, or at least some sort of separate package? If there is sufficient interest in it, it can be maintained there. If not, at least people can use it as-is. But there would not be any expectation going forward that the rest of numpy has to work well with it. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] FFTS for numpy's FFTs (was: Re: Choosing between NumPy and SciPy functions)
On Thu, Dec 11, 2014 at 4:55 PM, Robert Kern wrote: > On Thu, Dec 11, 2014 at 3:53 PM, Eric Moore > wrote: > > > > On Thu, Dec 11, 2014 at 10:41 AM, Todd wrote: > > >> I recently became aware of another C-library for doing FFTs (and other > things): > >> > >> https://github.com/arrayfire/arrayfire > >> > >> They claim to have comparable FFT performance to MKL when run on a CPU > (they also support running on the GPU but that is probably outside the > scope of numpy or scipy). It used to be proprietary but now it is under a > BSD-3-Clause license. It seems it supports non-power-of-2 FFT operations > as well (although those are slower). I don't know much beyond that, but it > is probably worth looking in > > > > AFAICT the cpu backend is a FFTW wrapper. > > Indeed. > https://github.com/arrayfire/arrayfire/blob/devel/src/backend/cpu/fft.cpp#L16 > Oh, nevermind then. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] FFTS for numpy's FFTs (was: Re: Choosing between NumPy and SciPy functions)
On Tue, Oct 28, 2014 at 5:28 AM, Nathaniel Smith wrote: > On 28 Oct 2014 04:07, "Matthew Brett" wrote: > > > > Hi, > > > > On Mon, Oct 27, 2014 at 8:07 PM, Sturla Molden > wrote: > > > Sturla Molden wrote: > > > > > >> If we really need a > > >> kick-ass fast FFT we need to go to libraries like FFTW, Intel MKL or > > >> Apple's Accelerate Framework, > > > > > > I should perhaps also mention FFTS here, which claim to be faster than > FFTW > > > and has a BSD licence: > > > > > > http://anthonix.com/ffts/index.html > > > > Nice. And a funny New Zealand name too. > > > > Is this an option for us? Aren't we a little behind the performance > > curve on FFT after we lost FFTW? > > It's definitely attractive. Some potential issues that might need dealing > with, based on a quick skim: > > - seems to have a hard requirement for a processor supporting SSE, AVX, or > NEON. No fallback for old CPUs or other architectures. (I'm not even sure > whether it has x86-32 support.) > > - no runtime CPU detection, e.g. SSE vs AVX appears to be a compile time > decision > > - not sure if it can handle non-power-of-two problems at all, or at all > efficiently. (FFTPACK isn't great here either but major regressions would > be bad.) > > - not sure if it supports all the modes we care about (e.g. rfft) > > This stuff is all probably solveable though, so if someone has a hankering > to make numpy (or scipy) fft dramatically faster then you should get in > touch with the author and see what they think. > > -n > I recently became aware of another C-library for doing FFTs (and other things): https://github.com/arrayfire/arrayfire They claim to have comparable FFT performance to MKL when run on a CPU (they also support running on the GPU but that is probably outside the scope of numpy or scipy). It used to be proprietary but now it is under a BSD-3-Clause license. It seems it supports non-power-of-2 FFT operations as well (although those are slower). I don't know much beyond that, but it is probably worth looking in to. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Setting up a "newcomers" label on the issue tracker ?
KDE calls them"junior jobs". On Nov 27, 2014 2:29 AM, "Benjamin Root" wrote: > FWIW, matplotlib calls it "low hanging fruit". I think it is a better name > than "newcomers". > > On Wed, Nov 26, 2014 at 1:19 PM, Aldcroft, Thomas < > aldcr...@head.cfa.harvard.edu> wrote: > >> >> >> On Wed, Nov 26, 2014 at 8:24 AM, Charles R Harris < >> charlesr.har...@gmail.com> wrote: >> >>> >>> >>> On Wed, Nov 26, 2014 at 2:36 AM, Sebastian Berg < >>> sebast...@sipsolutions.net> wrote: >>> On Mi, 2014-11-26 at 08:44 +, David Cournapeau wrote: > Hi, > > > Would anybody mind if I create a label "newcomers" on GH, and start > labelling simple issues ? We actually have an "easy fix" label, which I think had this in mind. However, I admit that I think some of these issues may not be easy at all (I guess it depends on what you consider easy ;)). In any case, I think just go ahead with creating a new label or reusing the current one. "easy fix" might be a starting point to find some candidate issues. - Sebsatian > > > This is in anticipation to the bloomberg lab event in London this WE. > I will try to give a hand to people interested in numpy/scipy, >>> >>> There is also a documentation label, and about 30 tickets with that >>> label. That should be good for just practicing the mechanics. >>> >> >> FWIW in astropy we settled on two properties, level of effort and level >> of sub-package expertise, with corresponding labels: >> >> - effort-low, effort-medium, and effort-high >> - package-novice, package-intermediate, package-expert >> >> This has been used with reasonable success. >> >> - Tom >> >> >>> >>> Chuck >>> >>> ___ >>> NumPy-Discussion mailing list >>> NumPy-Discussion@scipy.org >>> http://mail.scipy.org/mailman/listinfo/numpy-discussion >>> >>> >> >> ___ >> NumPy-Discussion mailing list >> NumPy-Discussion@scipy.org >> http://mail.scipy.org/mailman/listinfo/numpy-discussion >> >> > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] #2522 numpy.diff fails on unsigned integers
On Thu, Nov 13, 2014 at 8:10 AM, Sebastian wrote: > On 2014-11-04 19:44, Charles R Harris wrote: > > On Tue, Nov 4, 2014 at 11:19 AM, Sebastian wrote: > > > >> On 2014-11-04 15:06, Todd wrote: > >>> On Tue, Nov 4, 2014 at 2:50 PM, Sebastian Wagner >> > >>> <mailto:se...@sebix.at>> wrote: > >>> > >>> Hello, > >>> > >>> I want to bring up Issue #2522 'numpy.diff fails on unsigned > >> integers > >>> (Trac #1929)' [1], as it was resonsible for an error in one > >> of our > >>> programs. Short explanation of the bug: np.diff performs a > >> subtraction > >>> on the input array. If this is of type uint and the data > >> contains > >>> falling data, it results in an artihmetic underflow. > >>> > >>> >>> np.diff(np.array([0,1,0], dtype=np.uint8)) > >>> array([ 1, 255], dtype=uint8) > >>> > >>> @charris proposed either > >>> - a note to the doc string and maybe an example to clarify > >> things > >>> - or raise a warning > >>> but with a discussion on the list. > >>> > >>> I would like to start it now, as it is an error which is not > >> easily > >>> detectable (no errors or warnings are thrown). In our case > >> the > >>> type of a > >>> data sequence, with only zeros and ones, had type f8 as also > >> every > >>> other > >>> one, has been changed to u4. As the programs looked for > >> values ==1 and > >>> ==-1, it broke silently. > >>> In my opinion, a note in the docs is not enough and does not > >> help > >>> if the > >>> type changed or set after the program has been written. > >>> I'd go for automatic upcasting of uints by default and an > >> option > >>> to turn > >>> it off, if this behavior is explicitly wanted. This wouldn't > >> be > >>> correct > >>> from the point of view of a programmer, but as most of the > >> users > >>> have a > >>> scientific background who excpect it 'to work', instead of > >> sth is > >>> theoretically correct but not convenient. (I count myself to > >> the first > >>> group) > >>> > >>> > >>> > >>> When you say "automatic upcasting", that would be, for example > >> uint8 > >>> to int16? What about for uint64? There is no int128. > >> The upcast should go to the next bigger, otherwise it would again > >> result > >> in wrong values. uint64 we can't do that, so it has to stay. > >>> Also, when you say "by default", is this only when an overflow is > >>> detected, or always? > >> I don't know how I could detect an overflow in the diff-function. > >> In > >> subtraction it should be possible, but that's very deep in the > >> numpy-internals. > >>> How would the option to turn it off be implemented? An argument > >> to > >>> np.diff or some sort of global option? > >> I thought of a parameter upcast_int=True for the function. > > > > Could check for non-decreasing sequence in the unsigned case. Note > > that differences of signed integers can also overflow. One way to > > check in general is to determine the expected sign using comparisons. > > I think you mean a decreasing/non-increasing instead of non-decreasing > sequence? > It's also the same check as checking for a sorted sequence. But I > currently don't know how I could do that efficiently without np.diff in > Python, in Cython it should be easily possible. > > > np.gradient has the same problem: > >>> np.random.seed(89) > >>> d = np.random.randint(0,2,size=10).astype(np.uint8); d > array([1, 0, 0, 1, 0, 1, 1, 0, 0, 0], dtype=uint8) > >>> np.diff(d) > array([255, 0, 1, 255, 1, 0, 255, 0, 0], dtype=uint8) > >>> np.gradient(d) > array([ 255. , 127.5,0.5,0. ,0. ,0.5, 127.5, 127.5, > 0. ,0. ]) > > Consider it is generally an error, might it be good to have a general warning built into the int dtypes regarding overflow errors? That warning can then be caught by the diff function. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] #2522 numpy.diff fails on unsigned integers
On Tue, Nov 4, 2014 at 2:50 PM, Sebastian Wagner wrote: > Hello, > > I want to bring up Issue #2522 'numpy.diff fails on unsigned integers > (Trac #1929)' [1], as it was resonsible for an error in one of our > programs. Short explanation of the bug: np.diff performs a subtraction > on the input array. If this is of type uint and the data contains > falling data, it results in an artihmetic underflow. > > >>> np.diff(np.array([0,1,0], dtype=np.uint8)) > array([ 1, 255], dtype=uint8) > > @charris proposed either > - a note to the doc string and maybe an example to clarify things > - or raise a warning > but with a discussion on the list. > > I would like to start it now, as it is an error which is not easily > detectable (no errors or warnings are thrown). In our case the type of a > data sequence, with only zeros and ones, had type f8 as also every other > one, has been changed to u4. As the programs looked for values ==1 and > ==-1, it broke silently. > In my opinion, a note in the docs is not enough and does not help if the > type changed or set after the program has been written. > I'd go for automatic upcasting of uints by default and an option to turn > it off, if this behavior is explicitly wanted. This wouldn't be correct > from the point of view of a programmer, but as most of the users have a > scientific background who excpect it 'to work', instead of sth is > theoretically correct but not convenient. (I count myself to the first > group) > When you say "automatic upcasting", that would be, for example uint8 to int16? What about for uint64? There is no int128. Also, when you say "by default", is this only when an overflow is detected, or always? How would the option to turn it off be implemented? An argument to np.diff or some sort of global option? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Custom dtypes without C -- or, a standard ndarray-like type
On Mon, Sep 22, 2014 at 5:31 AM, Nathaniel Smith wrote: > On Sun, Sep 21, 2014 at 7:50 PM, Stephan Hoyer wrote: > My feeling though is that in most of the cases you mention, > implementing a new array-like type is huge overkill. ndarray's > interface is vast and reimplementing even 90% of it is a huge effort. > For most of the cases that people seem to run into in practice, the > solution is to enhance numpy's dtype interface so that it's possible > for mere mortals to implement new dtypes, e.g. by just subclassing > np.dtype. This is totally doable and would enable a ton of > awesomeness, but it requires someone with the time to sit down and > work on it, and no-one has volunteered yet. Unfortunately it does > require hacking on C code though. > I'm unclear about the last sentence. Do you mean "improving the dtype system will require hacking on C code" or "even if we improve the dtype system dtypes will still have to be written in C"? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] String type again.
On Jul 16, 2014 11:43 AM, "Chris Barker" wrote: > So numpy should have dtypes to match these. We're a bit stuck, however, because 'S' mapped to the py2 string type, which no longer exists in py3. Sorry not running py3 to see what 'S' does now, but I know it's bit broken, and may be too late to change it In py3 a 'S' dtype is converted to a python bytes object. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Fwd: [Python-ideas] PEP pre-draft: Support for indexing with keyword arguments
On Jul 2, 2014 10:49 AM, "Nathaniel Smith" wrote: > > I admit I can't actually think of any features this would enable for us though... Could it be useful for structured arrays? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] big-bangs versus incremental improvements (was: Re: SciPy 2014 BoF NumPy Participation)
On 5 Jun 2014 14:28, "David Cournapeau" wrote: > > > > > On Thu, Jun 5, 2014 at 9:44 AM, Todd wrote: >> >> >> On 5 Jun 2014 02:57, "Nathaniel Smith" wrote: >> > >> > On Wed, Jun 4, 2014 at 7:18 AM, Travis Oliphant wrote: >> > And numpy will be much harder to replace than numeric -- >> > numeric wasn't the most-imported package in the pythonverse ;-). >> >> If numpy is really such a core part of python ecosystem, does it really make sense to keep it as a stand-alone package? Rather than thinking about a numpy 2, might it be better to be focusing on getting ndarray and dtype to a level of quality where acceptance upstream might be plausible? > > > There has been discussions about integrating numpy a long time ago (can't find a reference right now), and the consensus was that this was possible in its current shape nor advisable. The situation has not changed. > > Putting something in the stdlib means it basically cannot change anymore: API compatibility requirements would be stronger than what we provide even now. NumPy is also a large codebase which would need some major clean up to be accepted, etc... > > David I am not suggesting merging all of numpy, only ndarray and dtype (which I know is a huge job itself). And perhaps not even all of what us currently included in those, some methods could be split out to their own functions. And any numpy 2.0 would also imply a major code cleanup. So although ndarray and dtype are certainly not ready for such a thing right now, if you are talking about numpy 2.0 already, perhaps part of that discussion could involve a plan to get the code into a state where such a move might be plausible. Even if the merge doesn't actually happen, having the code at that quality level would still be a good thing. I agree that the relationship between numpy and python has not changed very much in the last few years, but I think the scientific computing landscape is changing. The latter issue is where my primary concern lies. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] big-bangs versus incremental improvements (was: Re: SciPy 2014 BoF NumPy Participation)
On 5 Jun 2014 02:57, "Nathaniel Smith" wrote: > > On Wed, Jun 4, 2014 at 7:18 AM, Travis Oliphant wrote: > And numpy will be much harder to replace than numeric -- > numeric wasn't the most-imported package in the pythonverse ;-). If numpy is really such a core part of python ecosystem, does it really make sense to keep it as a stand-alone package? Rather than thinking about a numpy 2, might it be better to be focusing on getting ndarray and dtype to a level of quality where acceptance upstream might be plausible? Matlab and python are no longer the only games in town for scientific computing anymore. I worry that the lack of a multidimensional array literals, not to mention the lack of built-in multidimensional arrays at all, can only hurt python's attractiveness compared to languages like Julia long-term. For people who already know and love python, this doesn't bother us much if at all. But thinking of attracting new users long-term, I worry that it will be harder to convince outsiders that python is really a first-class scientific computing language when it lacks the key data type for scientific computing. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] How security holes happen
On Mar 3, 2014 3:16 AM, "Charles R Harris" wrote: > > This is from OS X 9 > > if ((err = SSLHashSHA1.update(&hashCtx, &serverRandom)) != 0) > goto fail; > if ((err = SSLHashSHA1.update(&hashCtx, &signedParams)) != 0) > goto fail; > goto fail; > if ((err = SSLHashSHA1.final(&hashCtx, &hashOut)) != 0) > goto fail; > > Heh, maybe there is a reason for braces in even the simplest if statements. > > Chuck Not to mention static code analyzers. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Proposal to make power return float, and other such things.
On Feb 18, 2014 11:55 AM, "Robert Kern" wrote: > > On Tue, Feb 18, 2014 at 9:51 AM, Sturla Molden wrote: > > Charles R Harris wrote: > >> This is apropos issue #899 < >> href="https://github.com/numpy/numpy/issues/899";> https://github.com/numpy/numpy/issues/899>, > >> where it is suggested that power promote integers to float. That sounds > >> reasonable to me, but such a change in behavior makes it a bit iffy. > >> > >> Thoughts? > > > > Numpy should do the same as Python does. > > That's problematic because Python's behavior is value dependent. > > Python 3.3.1 (default, May 16 2013, 17:20:13) > [GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> 2 ** 2 > 4 > >>> 2 ** -2 > 0.25 > > That's fine if you only have one value for each operand. When you have > multiple values for each operand, say an exponent array containing > both positive and negative integers, that becomes a problem. > Generally, we try to make ufuncs return types that are predictable > from the types of the operands, not the values of the operands. > > I am -1 on the proposal to make power(x:int, y:int) always return a > float. It is usually trivial to just make the exponent a float if one > wants a float returned. Or we could introduce an fpow() that always > coerces the inputs to the best inexact type. What about making it floats for int types but int for uint types? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Inheriting from ndarray Was: deprecate numpy.matrix
On Feb 11, 2014 5:01 AM, "Alexander Belopolsky" wrote: > > > On Mon, Feb 10, 2014 at 11:31 AM, Nathaniel Smith wrote: >> >> And in the long run, I >> think the goal is to move people away from inheriting from np.ndarray. > > > This is music to my ears, There are a lot of units (meter, foot, built, etc.) packages that use inheriting from numpy to handle their units. I wouldn't call it a shortcut, it is a common design decision. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] deprecate numpy.matrix
On Feb 11, 2014 3:23 AM, "Alan G Isaac" wrote: > > On 2/10/2014 7:39 PM, Pauli Virtanen wrote: > > The issue here is semantics for basic linear algebra operations, such as > > matrix multiplication, that work for different matrix objects, including > > ndarrays. > > > I'll see if I can restate my suggestion in another way, > because I do not feel you are responding to it. > (I might be wrong.) > > What is a duck? If you ask it to quack, it quacks. > OK, but what is it to quack? > > Here, quacking is behaving like an ndarray (in your view, > as I understand it) when asked. But how do we ask? > Your view (if I understand) is we ask via the operations > supported by ndarrays. But maybe that is the wrong way > for the library to ask this question. > > If so, then scipy libraries could ask an object > to behave like an an ndarray by calling, e.g., > __asarray__ on it. It becomes the responsibility > of the object to return something appropriate > when __asarray__ is called. Objects that know how to do > this will provide __asarray__ and respond > appropriately. Other types can be coerced if > that is the documented behavior (e.g., lists). > The libraries will then be written for a single > type of behavior. What it means to "quack" is > pretty easily documented, and a matrix object > already knows how (e.g., m.A). Presumably in > this scenario __asarray__ would return an object > that behaves like an ndarray and a converter for > turning the final result into the desired object > type (e.g., into a `matrix` if necessary). > > Hope that clearer, even if it proves a terrible idea. > > Alan Isaac I don't currently use the matrix class, but having taken many linear algebra classes I can see the appeal, and if I end up teaching the subject I think I would appreciate having it available. On the other hand, I certainly can see the possibility for confusion, and I don't think it is something that should be used unless someone has a really good reason. So I come out somewhere in the middle here. So, although this may end up being a terrible idea, I would like to purpose what I think is a compromise: instead of just removing matrix, split it out into a scikit. That way, it it's still available for those who need it, but will be less likely to be used accidentally, and won't be interfering with the rest of numpy and scipy development. Specifically, I would split matrix into a scikit, while in the same release deprecate np.matrix. They can then exist in parallel for a few releases to allow code to be ported away from it. However, I would suggest that before the split, all linear algebra routines should be available as functions or methods in numpy proper. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] PEP 208 and upstreaming numpy
But wouldn't there be a benefit from integrating ndarrays directly into the grammar like lists, tuples, and dictionaries? On Jun 18, 2013 11:41 AM, "Robert Kern" wrote: > > On Tue, Jun 18, 2013 at 10:33 AM, Todd wrote: > > I see that the plan to merge old Numeric into the python standard library, > > PEP 208, is listed as withdrawn, although no reasons are given as far as I > > could see. > > > > Considering how mature Numpy has become, and how common it is as a > > dependency for python packages, I was wondering if there were still plans on > > the table to ultimately merge at least the ndarray class into the python > > standard library, and if so where those plans stand. > > No, this will not be happening. Instead, the focus of effort went into > creating the enhanced buffer interface. This allows libraries to > communicate ndarray-like information using that interface instead of > sharing a particular concrete data structure. > > http://www.python.org/dev/peps/pep-3118/ > > -- > Robert Kern > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] PEP 208 and upstreaming numpy
I see that the plan to merge old Numeric into the python standard library, PEP 208, is listed as withdrawn, although no reasons are given as far as I could see. Considering how mature Numpy has become, and how common it is as a dependency for python packages, I was wondering if there were still plans on the table to ultimately merge at least the ndarray class into the python standard library, and if so where those plans stand. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] nanmean(), nanstd() and other "missing" functions for 1.8
On Wed, May 1, 2013 at 5:22 PM, Daπid wrote: > On 1 May 2013 17:13, Todd wrote: > > Speaking of which, I think there should be a function to construct a > complex > > array out of two identically-shaped floating-point arrays, as well as > > perhaps an np.i class that converts a real array to an imaginary one > (using > > __mul__ and such). > > np.i would be exactly the same as array * 1j, or am I missing anything? > > The same goes for constructing a complex, real + imag * 1j > > > it would always produce a numpy array. So array*1j and array*np.i (or np.j if you prefer, perhaps both) would be the same, but list*1j and list*np.i would not. The function version would also probably allow you to specify the dtype, which 1j does not. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] nanmean(), nanstd() and other "missing" functions for 1.8
On Wed, May 1, 2013 at 3:36 AM, Benjamin Root wrote: > > Are there any other functions that others feel are "missing" from numpy > and would like to see for v1.8? Let's discuss them here. > As I mentioned before, I think numpy should have some equations for dealing with n-dimensional vectors (but would also work with complex dtypes). This would include n-dimensional equivalents of np.abs and np.angle, as well as a function to go back to an n-dimensional vector from the length and angle. Considering how critical vector analysis is to signal processing, I am surprised these don't already exist. There aren't even functions that work with 2-dimensional vectors, you have to construct a complex array first (which isn't that easy to do). Speaking of which, I think there should be a function to construct a complex array out of two identically-shaped floating-point arrays, as well as perhaps an np.i class that converts a real array to an imaginary one (using __mul__ and such). ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Finding the same value in List
> > The data type: > x in ndarray and x[ i ]--> int64 > type(f) --> ' list ' > type( f[ 0 ] ) --> ' tuple ' > type( f[ 0][0] ) --> 'ndarray' > type( f[ 0 ][ 0 ][ 0] ) --> 'int64' > > How do you think to avoid diversity if data type in this example? I think > it is not necessary to get diverse dtype as well as more than 1D array.. > That is why I suggested this approach was better ( note the that this is where()[0] instead of just where() as it was in my first example): x,i=numpy.unique(y, return_inverse=True) f=[numpy.where(i==ind)[0] for ind in range(len(x))] type(f) --> list type(f[0]) --> ndarray type(f[0][0]) is meaningless since it is just a single element in an array. It must be an int type of some sort of since indices have to be int types. x will be the same dtype as your input array. You could conceivably change the type of f[0] to a list, but why would you want to? One of the big advantages of python is that usually it doesn't matter what the type is. In this case, a numpy ndarray will work the same as a list in most cases where you would want to use these sorts of indices. It is possibly to change the ndarray to a list, but unless there is a specific reason you need to use lists so then it is better not to. You cannot change the list to an ndarray because the elements of the list are different lengths. ndarray doesn't support that. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Finding the same value in List
On Wed, Apr 17, 2013 at 10:46 AM, Todd wrote: > x,i=numpy.unique(y, return_inverse=True) > f=[numpy.where(i==ind) for ind in range(len(x))] > > > A better version would be (np.where returns tuples, but we don't want tuples): x,i=numpy.unique(y, return_inverse=True) f=[numpy.where(i==ind)[0] for ind in range(len(x))] You can also do it this way, but it is much harder to read IMO: x=numpy.unique(y) f=numpy.split(numpy.argsort(y), numpy.nonzero(numpy.diff(numpy.sort(y)))[0]+1) This version figures out the indexes needed to put the values of y in sorted order (the same order x uses), then splits it into sub-arrays based on value. The principle is simpler but the implementation looks like clear to me. Note that these are only guaranteed to work on 1D arrays, I have not tested them on multidimensional arrays ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Finding the same value in List
x,i=numpy.unique(y, return_inverse=True) f=[numpy.where(i==ind) for ind in range(len(x))] x will give you the list of unique values, and f will give you the indices of each corresponding value in x. So f[0] is the indices of x[0] in y. To explain, unique in this form gives two outputs, a sorted, non-repeating list of values (x), and an array of the same shape as y that gives you the indices of x of each corresponding value of y (i, that is x[i] is the same as y) The second goes through each index of x and finds where that index occurs in i. On Wed, Apr 17, 2013 at 9:44 AM, Happyman wrote: > Hello, > > I have had encountered some problem while I was trying to create the > following code which finds number of the same values and indexes in an > array or list. Here is the code: > > y = [ 1, 12, 3, 3, 5, 1, 1, 34, 0, 0, 1, 5] > OR > y = array( [ 1, 12, 3, 3, 5, 1, 1, 34, 0, 0, 1, 5 ] ) > > b = [ [ item for item in range( len(y) ) if y[ item ] == y[ j ] ] for j > in range(0, len( y ) ) ] > > answer: >[ [ 0, 5, 6, 10], [1], [2, 3], [2, 3], [4, 11], [0, 5, 6, 10], [0, 5, > 6, 10], [7], [8, 9], [8, 9], [0, 5, 6, 10], [4, 11] ] > > The result I want to get is ,not that shown above as an answer, that I > want to calculate the number of the same values and their indexes in not > repeated way as well. For example, > '1' - 4, index: '0, 5, 6, 10' > '12' - 1, index: '1' > '3' - 2, index: '2, 3' > '5' - 2, index: '4, 11' > '34' - 1, index: '7' > '0' - 2, index: '8, 9' > > Any answer would be appreciated.. > > > > > > -- > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-Dev] NumPy/SciPy participation in GSoC 2013
On Tue, Apr 2, 2013 at 8:12 PM, Ralf Gommers wrote: > > > > On Mon, Apr 1, 2013 at 2:27 PM, Todd wrote: > >> >> There were a number of other ideas in this thread: >> >> http://mail.scipy.org/pipermail/numpy-discussion/2013-March/065699.html >> > > Thanks Todd. Your idea 5 is pretty much what Nathaniel just detailed out, > it's on the ideas page now. That should enable idea 1 as well. The other > idea that got some positive feedback was 3: > http://mail.scipy.org/pipermail/numpy-discussion/2013-March/065710.html. > If you or someone else could make that a little more concrete, we can put > that up. > > For 3: With structured arrays, you can access them by name (key) in a manner identical to dictionaries: y = x['f'] It also has a method for accessing the list of names (keys): x.dtype.names This should be maintained for backwards-compatibility, but these methods should also be added: x.keys -- returns a list of field names x.values -- returns a list of views into the array, one for each structure x.items -- returns a list of tuple containing name/structure pairs (the structures being a views) x.iterkeys/itervalues/iteritems -- returns an iterable over the corresponding objects (should not be available in python 3) x.viewkeys/viewvalues/viewitems -- the same as the corresponding item, since they always return views (should not be available in python 3) x.has_key -- tests if a field name is present (should not be available in python 3, should use "key in x.keys()") x.get -- get a field by name, returning a default array (an empty array by default) if not present x.update -- copy values into the matching key from another structured array, a dict, or list of key/value tuples. Unlike dicts this will only work for keys that are already present. Documentation should probably be updated to have these as the default ways of interacting with structured arrays, with the old methods deprecated. "Names" should also probably be renamed "keys" for compatibility with dicts. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] [SciPy-Dev] NumPy/SciPy participation in GSoC 2013
On Mon, Apr 1, 2013 at 1:58 PM, Ralf Gommers wrote: > > > > On Tue, Mar 26, 2013 at 12:27 AM, Ralf Gommers wrote: > >> >> >> >> On Thu, Mar 21, 2013 at 10:20 PM, Ralf Gommers wrote: >> >>> Hi all, >>> >>> It is the time of the year for Google Summer of Code applications. If we >>> want to participate with Numpy and/or Scipy, we need two things: enough >>> mentors and ideas for projects. If we get those, we'll apply under the PSF >>> umbrella. They've outlined the timeline they're working by and guidelines >>> at >>> http://pyfound.blogspot.nl/2013/03/get-ready-for-google-summer-of-code.html. >>> >>> >>> We should be able to come up with some interesting project ideas I'd >>> think, let's put those at >>> http://projects.scipy.org/scipy/wiki/SummerofCodeIdeas. Preferably with >>> enough detail to be understandable for people new to the projects and a >>> proposed mentor. >>> >>> We need at least 3 people willing to mentor a student. Ideally we'd have >>> enough mentors this week, so we can apply to the PSF on time. If you're >>> willing to be a mentor, please send me the following: name, email address, >>> phone nr, and what you're interested in mentoring. If you have time >>> constaints and have doubts about being able to be a primary mentor, being a >>> backup mentor would also be helpful. >>> >> >> So far we've only got one primary mentor (thanks Chuck!), most core devs >> do not seem to have the bandwidth this year. If there are other people >> interested in mentoring please let me know. If not, then it looks like >> we're not participating this year. >> > > Hi all, an update on GSoC'13. We do have enough mentoring power after all; > NumPy/SciPy is now registered as a participating project on the PSF page: > http://wiki.python.org/moin/SummerOfCode/2013 > > Prospective students: please have a look at > http://wiki.python.org/moin/SummerOfCode/Expectations and at > http://projects.scipy.org/scipy/wiki/SummerofCodeIdeas. In particular > note that we require you to make one pull request to NumPy/SciPy which has > to be merged *before* the application deadline (May 3). So please start > thinking about that, and start a discussion on your project idea on this > list. > > Cheers, > Ralf > > > ___ > SciPy-Dev mailing list > scipy-...@scipy.org > http://mail.scipy.org/mailman/listinfo/scipy-dev > > There were a number of other ideas in this thread: http://mail.scipy.org/pipermail/numpy-discussion/2013-March/065699.html ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Polar/spherical coordinates handling
On Fri, Mar 29, 2013 at 4:33 PM, Angus McMorland wrote: > On 29 March 2013 11:15, Todd wrote: > > From what I can see, numpy doesn't have any functions for handling polar > or > > spherical coordinate to/from cartesian coordinate conversion. I think > such > > methods would be pretty useful. I am looking now and it doesn't look > that > > hard to create functions to convert between n-dimensional cartesian and > > n-spherical coordinates. Would anyone be interested in me adding methods > > for this? > > I use these co-ordinate transforms often. I wonder if it wouldn't be > preferable to create a scikit focused on spherical or, more generally, > geometric operations rather than adding to the already hefty number of > functions in numpy. I'd be interested to contribute to such a scikit. > The reason I think these particular functions belong in numpy is that they are closely tied to signal processing and linear algebra, far more than any other coordinate systems. It is really just a generalization of the complex number processing that is already available from numpy. Also, although numpy has methods to convert complex values to magnitude and angle, it doesn't have any methods to go the other way. Again, such a function would just be a special 2-D case of the more general n-dimensional functions. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Polar/spherical coordinates handling
>From what I can see, numpy doesn't have any functions for handling polar or spherical coordinate to/from cartesian coordinate conversion. I think such methods would be pretty useful. I am looking now and it doesn't look that hard to create functions to convert between n-dimensional cartesian and n-spherical coordinates. Would anyone be interested in me adding methods for this? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Execution time difference between 2.7 and 3.2 using numpy
On Sat, Mar 23, 2013 at 12:21 PM, Ralf Gommers wrote: > > That's not the case. The official binaries for NumPy and SciPy are on > SourceForge. The Windows installers on PyPI are there to make easy_install > work, but they're likely slower than the SF installers (no SSE2/SSE3 > instructions). > > Ralf > > Is there a reason why the same binaries can't be used for both? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] unclear output format for numpy.argmax()
On Tue, Mar 12, 2013 at 8:20 AM, soumen ganguly wrote: > Hello, > > There are some doubts that i have regarding the argmax() method of > numpy.As described in reference doc's of numpy,argmax(axis=None,out=None) > returns the indices of the maximum value along the given axis(In this case > 0 is default). > > So, i tried to implement the method to a 2d array with elements > say,[[1,2,3],[4,5,6]] along the axis 1.The output to this code is [2,2] and > when i implement it along the axis 0,it outputs [1,1,1].I dont see the > connection to this output with the scope of argmax method. > > I would appreciate a detailed insight to the argmax method. > > I am not sure I understand the question. For axis 0 (the "outer" dimension in the way it is printed) the things being compared are argmax([1, 4]), argmax(([2, 5]), and argmax([3, 6]).. Amongst those, the second (index 1) is higher in each case, so it returns [1, 1, 1]. With axis 1 (the "inner" dimension in the way it is printed) , the things being compared are argmax([1, 2, 3]) and argmax([4, 5, 6]). In both case the third (index 2) is the highest, so it returns [2, 2]. What is unexpected about this? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] GSOC 2013
On Mar 5, 2013 7:53 PM, "Nathaniel Smith" wrote: > > On 4 Mar 2013 23:21, "Jaime Fernández del Río" wrote: > > > > On Mon, Mar 4, 2013 at 2:29 PM, Todd wrote: > >> > >> > >> 5. Currently dtypes are limited to a set of fixed types, or combinations of these types. You can't have, say, a 48 bit float or a 1-bit bool. This project would be to allow users to create entirely new, non-standard dtypes based on simple rules, such as specifying the length of the sign, length of the exponent, and length of the mantissa for a custom floating-point number. Hopefully this would mostly be used for reading in non-standard data and not used that often, but for some situations it could be useful for storing data too (such as large amounts of boolean data, or genetic code which can be stored in 2 bits and is often very large). > > > > > > I second this general idea. Simply having a pair of packbits/unpackbits functions that could work with 2 and 4 bit uints would make my life easier. If it were possible to have an array of dtype 'uint4' that used half the space of a 'uint8', but could have ufuncs an the like ran on it, it would be pure bliss. Not that I'm complaining, but a man can dream... > > This would be quite difficult, since it would require reworking the guts of the ndarray data structure to store strides and buffer offsets in bits rather than bytes, and probably with endianness handling too. Indexing is all done at the ndarray buffer-of-bytes layer, without any involvement of the dtype. > > Consider: > > a = zeros(10, dtype=uint4) > b = a[1::3] > > Now b is a view onto a discontiguous set of half-bytes within a... > > You could have a dtype that represented several uint4s that together added up to an integral number of bytes, sort of like a structured dtype. Or packbits()/unpackbits(), like you say. > > -n Then perhaps such a project could be a four-stage thing. 1. Allow for the creation of int, unit, float, bool, and complex dtypes with an arbitrary number of bytes 2. Allow for the creation of dtypes which are integer fractions of a byte, (1, 2, or 4 bits), and must be padded to a whole byte. 3. Have an optional internal value in an array that tells it to exclude the last n bits of the last byte. This would be used to hide the padding from step 2. This should be abstracted into a general-purpose method for excluding bits from the byte-to-dtype conversion so it can be used in step 4. 4. Allow for the creation of dtypes that are non-integer fractions of a byte or non-integer multiples of a byte (3, 5, 6, 7, 9, 10, 11, 12, etc, bits). Each element in the array would be stored as a certain number of bytes, with the method from 3 used to cut it down to the right number of bits. So a 3 bit dtype would have two elements per byte with 2 bits excluded. A 5 bit dtype would have 1 element per byte with 3 bits excluded. A 12 bit dtype would have one element in two bytes with with 4 bits excluded from the second byte. This approach would allow for arbitrary numbers of bits without breaking the internal representation, would have each stage building off the previous stage, and we would still have something useful even if not all the stages are completed. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] GSOC 2013
On Mon, Mar 4, 2013 at 9:41 PM, Ralf Gommers wrote: > > > > On Tue, Feb 26, 2013 at 11:17 AM, Todd wrote: > >> Is numpy planning to participate in GSOC this year, either on their own >> or as a part of another group? >> > > If we participate, it should be under the PSF organization. I suspect > participation for NumPy (and SciPy) largely depends on mentors being > available. > > >> If so, should we start trying to get some project suggestions together? >> > > That can't hurt - good project descriptions will be useful not just for > GSOC but also for people new to the project looking for ways to contribute. > I suggest to use the wiki on Github for that. > > Ralf > > > > I have some ideas, but they may not be suitable for GSOC or may just be terrible ideas, so feel free to reject them: 1. A polar dtype. It would be similar to the complex dtype in that it would have two components, but instead of them being real and imaginary, they would be amplitude and angle. Besides the dtype, there should be either functions or methods to convert between complex and polar dtypes, and existing functions should be prepared to handle the new dtype. I it could be made to be able to handle an arbitrary number of dimensions this would be better yet, but I don't know if this is possible not to mention practical. There is a lot of mathematics, including both signal processing and vector analysis, that is often convenient to work with in this format. 2. We discussed this before, but right now subclasses of ndarray don't have any way to preserve their class attributes when using functions that work on multiple ndarrays, such as with concatenate. The current __array_finalize__ method only takes a single array. This project would be to work out a method to handle this sort of situation, perhaps requiring a new method, and making sure numpy methods and functions properly invoke it. 3. Structured arrays are accessed in a manner similar to python dictionaries, using a key. However, they don't support the normal python dictionary methods like keys, values, items, iterkeys, itervalues, iteritems, etc. This project would be to implement as much of the dictionary (and ordereddict) API as possible in structured arrays (making sure that the resulting API presented to the user takes into account whether python 2 or python 3 is being used). 4. The numpy ndarray class stores data in a regular manner in memory. This makes many linear algebra operations easier, but makes changing the number of elements in an array nearly impossible in practice unless you are very careful. There are other data structures that make adding and removing elements easier, but are not as efficient at linear algebra operations. The purpose of this project would be to create such a class in numpy, one that is duck type compatible with ndarray but makes resizing feasible. This would obviously come at a performance penalty for linear algebra related functions. They would still have consistent dtypes and could not be nested, unlike python lists. This could either be based on a new c-based type or be a subclass of list under the hood. 5. Currently dtypes are limited to a set of fixed types, or combinations of these types. You can't have, say, a 48 bit float or a 1-bit bool. This project would be to allow users to create entirely new, non-standard dtypes based on simple rules, such as specifying the length of the sign, length of the exponent, and length of the mantissa for a custom floating-point number. Hopefully this would mostly be used for reading in non-standard data and not used that often, but for some situations it could be useful for storing data too (such as large amounts of boolean data, or genetic code which can be stored in 2 bits and is often very large). ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] GSOC 2013
Is numpy planning to participate in GSOC this year, either on their own or as a part of another group? If so, should we start trying to get some project suggestions together? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Adding .abs() method to the array object
On Tue, Feb 26, 2013 at 10:58 AM, Sebastian Berg wrote:
> On Mon, 2013-02-25 at 22:04 -0500, josef.p...@gmail.com wrote:
> > On Mon, Feb 25, 2013 at 9:58 PM, wrote:
> > > On Mon, Feb 25, 2013 at 9:20 PM, Sebastian Berg
> > > wrote:
> > >> On Mon, 2013-02-25 at 10:50 -0500, Skipper Seabold wrote:
> > >>> On Mon, Feb 25, 2013 at 10:43 AM, Till Stensitzki
> > >>> wrote:
> > >>> >
> > >>> > First, sorry that i didnt search for an old thread, but because i
> > >>> disagree with
> > >>> > conclusion i would at least address my reason:
> > >>> >
>
> > >> Two small things (not sure if it matters much). But first almost all
> of
> > >> these methods are related to the container and not the elements.
> Second
> > >> actually using a method arr.abs() has a tiny pitfall, since abs would
> > >> work on numpy types, but not on python types. This means that:
> > >>
> > >> np.array([1, 2, 3]).max().abs()
> > >>
> > >> works, but
> > >>
> > >> np.array([1, 2, 3], dtype=object).max().abs()
> > >>
> > >> breaks. Python has a safe name for abs already...
> > >
> > (np.array([1, 2, 3], dtype=object)).max()
> > > 3
> > (np.array([1, 2, 3], dtype=object)).__abs__().max()
> > > 3
> > (np.array([1, 2, '3'], dtype=object)).__abs__()
> > > Traceback (most recent call last):
> > > File "", line 1, in
> > > TypeError: bad operand type for abs(): 'str'
> > >
> > map(abs, [1, 2, 3])
> > > [1, 2, 3]
> > map(abs, [1, 2, '3'])
> > > Traceback (most recent call last):
> > > File "", line 1, in
> > > TypeError: bad operand type for abs(): 'str'
> >
> > or maybe more useful
> >
> > >>> from decimal import Decimal
> > >>> d = [Decimal(str(k)) for k in np.linspace(-1, 1, 5)]
> > >>> map(abs, d)
> > [Decimal('1.0'), Decimal('0.5'), Decimal('0.0'), Decimal('0.5'),
> Decimal('1.0')]
> >
> > >>> np.asarray(d).__abs__()
> > array([1.0, 0.5, 0.0, 0.5, 1.0], dtype=object)
> > >>> np.asarray(d).__abs__()[0]
> > Decimal('1.0')
> >
> > Josef
> >
> > >
> > > I don't see a difference.
> > >
> > > (I don't expect to use max abs on anything else than numbers.)
> > >
>
> The difference is about scalars only. And of course __abs__ is fine, but
> if numpy adds an abs method, its scalars would logically have it too.
> But then you diverge from python scalars. That has exactly the downside
> that you may write code that suddenly stops working for python scalars
> without noticing.
>
> I turned around the abs and max order here, so that the abs works on the
> scalar, not useful but just as an example.
>
But doesn't this also apply to many existing methods?
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] What should np.ndarray.__contains__ do
The problem with b is that it breaks down if the two status have the same dimensionality. I think a better approach would be for a in b With a having n dimensions, it returns true if there is any subarray of b that matches a along the last n dimensions. So if a has 3 dimensions and b has 6, a in b is true iff there is any i, j, k, m, n, p such that a=b[i, j, k, m:m+a.shape[0], n:n+a.shape[1], p:p+a.shape[2]] ] This isn't a very clear way to describe it, but I think it is consistent with the concept of a being a subarray of b even when they have the same dimensionality. On Feb 25, 2013 5:34 PM, "Nathaniel Smith" wrote: > On Mon, Feb 25, 2013 at 3:10 PM, Sebastian Berg > wrote: > > Hello all, > > > > currently the `__contains__` method or the `in` operator on arrays, does > > not return what the user would expect when in the operation `a in b` the > > `a` is not a single element (see "In [3]-[4]" below). > > True, I did not expect that! > > > The first solution coming to mind might be checking `all()` for all > > dimensions given in argument `a` (see line "In [5]" for a simplistic > > example). This does not play too well with broadcasting however, but one > > could maybe simply *not* broadcast at all (i.e. a.shape == > > b.shape[b.ndim-a.ndim:]) and raise an error/return False otherwise. > > > > On the other hand one could say broadcasting of `a` onto `b` should be > > "any" along that dimension (see "In [8]"). The other way should maybe > > raise an error though (see "In [9]" to understand what I mean). > > > > I think using broadcasting dimensions where `a` is repeated over `b` as > > the dimensions to use "any" logic on is the most general way for numpy > > to handle this consistently, while the other way around could be handled > > with an `all` but to me makes so little sense that I think it should be > > an error. Of course this is different to a list of lists, which gives > > False in these cases, but arrays are not list of lists... > > > > As a side note, since for loop, etc. use "for item in array", I do not > > think that vectorizing along `a` as np.in1d does is reasonable. `in` > > should return a single boolean. > > Python effectively calls bool() on the return value from __contains__, > so reasonableness doesn't even come into it -- the only possible > behaviours for `in` are to return True, False, or raise an exception. > > I admit that I don't actually really understand any of this discussion > of broadcasting. in's semantics are, "is this scalar in this > container"? (And the scalarness is enforced by Python, as per above.) > So I think we should find some approach where the left argument is > treated as a scalar. > > The two approaches that I can see, and which generalize the behaviour > of simple Python lists in natural ways, are: > > a) the left argument is coerced to a scalar of the appropriate type, > then we check if that value appears anywhere in the array (basically > raveling the right argument). > > b) for an array with shape (n1, n2, n3, ...), the left argument is > treated as an array of shape (n2, n3, ...), and we check if that > subarray (as a whole) appears anywhere in the array. Or in other > words, 'A in B' is true iff there is some i such that > np.array_equals(B[i], A). > > Question 1: are there any other sensible options that aren't on this list? > > Question 2: if not, then which should we choose? (Or we could choose > both, I suppose, depending on what the left argument looks like.) > > Between these two options, I like (a) and don't like (b). The > pretending-to-be-a-list-of-lists special case behaviour for > multidimensional arrays is already weird and confusing, and besides, > I'd expect equality comparison on arrays to use ==, not array_equals. > So (b) feels pretty inconsistent with other numpy conventions to me. > > -n > > > I have opened an issue for it: > > https://github.com/numpy/numpy/issues/3016#issuecomment-14045545 > > > > > > Regards, > > > > Sebastian > > > > In [1]: a = np.array([0, 2]) > > > > In [2]: b = np.arange(10).reshape(5,2) > > > > In [3]: b > > Out[3]: > > array([[0, 1], > >[2, 3], > >[4, 5], > >[6, 7], > >[8, 9]]) > > > > In [4]: a in b > > Out[4]: True > > > > In [5]: (b == a).any() > > Out[5]: True > > > > In [6]: (b == a).all(0).any() # the 0 could be multiple axes > > Out[6]: False > > > > In [7]: a_2d = a[None,:] > > > > In [8]: a_2d in b # broadcast dimension means "any" -> True > > Out[8]: True > > > > In [9]: [0, 1] in b[:,:1] # should not work (or be False, not True) > > Out[9]: True > > > > > > ___ > > NumPy-Discussion mailing list > > NumPy-Discussion@scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___
Re: [Numpy-discussion] another discussion on numpy correlate (and convolution)
We don't actually want remove sensitive data, but this tutorial should
still allow us to remove a file totally and completely from git history. It
doesn't look that hard:
https://help.github.com/articles/remove-sensitive-data
It will require everyone to rebase, so if you want to do this it may be a
good idea to schedule it for a specific day maybe 2-4 weeks in the future
and warn people on the mailing list so nobody tries to commit anything
while the process is underway.
On Feb 22, 2013 5:40 PM, "Matthew Brett" wrote:
> Hi,
>
> On Thu, Feb 21, 2013 at 10:00 AM, Pierre Haessig
> wrote:
> > Hi everybody,
> >
> > (just coming from a discussion on the performance of Matplotlib's
> > (x)corr function which uses np.correlate)
> >
> > There have been already many discussions on how to compute
> > (cross-)correlations of time-series in Python (like
> >
> http://stackoverflow.com/questions/6991471/computing-cross-correlation-function
> ).
> > The discussion is spread between the various stakeholders (just to name
> > some I've in mind : scipy, statsmodel, matplotlib, ...).
> >
> > There are basically 2 implementations : time-domain and frequency-domain
> > (using fft + multiplication). My discussion is only on time-domain
> > implementation. The key usecase which I feel is not well adressed today
> > is when computing the (cross)correlation of two long timeseries on only
> > a few lagpoints.
> >
> > For time-domain, one can either write its own implementation or rely on
> > numpy.correlate. The latter rely on the fast C implementation
> > _pyarray_correlate() in multiarraymodule.c
> > (
> https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/multiarraymodule.c#L1177
> )
> >
> > Now, the signature of this function is meant to accept one of three
> > *computation modes* ('valid', 'same', 'full'). Thoses modes make a lot
> > of sense when using this function in the context of convoluting two
> > signals (say an "input array" and a "impulse response array"). In the
> > context of computing the (cross)correlation of two long timeseries on
> > only a few lagpoints, those computation modes are unadapted and
> > potentially leads to huge computational and memory wastes
> > (for example :
> >
> https://github.com/matplotlib/matplotlib/blob/master/lib/matplotlib/axes.py#L4332
> ).
> >
> >
> >
> > For some time, I was thinking the solution was to write a dedicated
> > function in one of the stakeholder modules (but which ? statsmodel ?)
> > but now I came to think numpy is the best place to put a change and this
> > would quickly benefit to all stackeholders downstream.
> >
> > Indeed, I looked more carefully at the C _pyarray_correlate() function,
> > and I've come to the conclusion that these three computation modes are
> > an unnecessary restriction because the actual computation relies on a
> > triple for loop
> > (
> https://github.com/numpy/numpy/blob/master/numpy/core/src/multiarray/multiarraymodule.c#L1177
> )
> > which boundaries are governed by two integers `n_left` and `n_right`
> > instead of the three modes.
> >
> > Maybe I've misunderstood the inner-working of this function, but I've
> > the feeling that the ability to set these two integers directly instead
> > of just the mode would open up the possibility to compute the
> > correlation on only a few lagpoints.
> >
> > I'm fully aware that changing the signature of such a core numpy is
> > out-of-question but I've got the feeling that a reasonably small some
> > code refactor might lift the longgoing problem of (cross-)correlation
> > computation. The Python np.correlate would require two additional
> > keywords -`n_left` and `n_right`)which would override the `mode`
> > keyword. Only the C function would need some more care to keep good
> > backward compatibility in case it's used externally.
>
> >From complete ignorance, do you think it is an option to allow a
> (n_left, n_right) tuple as a value for 'mode'?
>
> Cheers,
>
> Matthew
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion
>
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Seeking help and support for next-gen math modeling tools using Python
On Feb 20, 2013 12:47 AM, "Rob Clewley" wrote: > > Hi all, and apologies for a little cross-posting: > > First, thanks to those of you who have used and contributed to the > PyDSTool math modeling environment [1]. This project has greatly > benefitted from the underlying platform of numpy / scipy / matplotlib > / ipython. Going forward I have three goals, for which I would like > the urgent input and support of existing or potential users. > > (i) I have ideas for expanding PyDSTool with innovative tools in my > research area, which is essentially for the reverse engineering of > complex mechanisms in multi-scale dynamic systems [2]. These tools > have already been prototyped and show promise, but they need a lot of > work. > (ii) I want to grow and develop the community of users who will help > drive new ideas, provide feedback, and collaborate on writing and > testing code for both the core and application aspects of PyDSTool. > (iii) The first two goals will help me to expand the scientific / > engineering applications and use cases of PyDSTool as well as further > sustain the project in the long-term. > > I am applying for NSF funding to support these software and > application goals over the next few years [3], but the proposal > deadline is in just four weeks! If you are interested in helping in > any way I would greatly appreciate your replies (off list) to either > of the following queries: > > I need to better understand my existing and potential users, many of > whom may not be registered on the sourceforge users list. Please tell > me who you are and what you use PyDSTool for. If you are not using it > yet but you’re interested in this area then please provide feedback > regarding what you would like to see change. > > If you are interested in these future goals, even if you are not an > existing user but may be in the future, please write a brief letter of > support on a letterhead document that I will send in with the proposal > as PDFs. I have sample text that I can send you, as well as my draft > proposal’s introduction and specific aims. These letters can make a > great deal of difference during review. > > Without funding, collaborators, user demand and community support, > these more ambitious goals for PyDSTool will not happen, although I am > committed to a basic level of maintenance. For instance, based on user > feedback I am about to release an Ubuntu-based Live CD [4] that will > allow users to try PyDSTool on any OS without having to install it. > PyDSTool will also acquire an improved setup procedure and will be > added to the NeuroDebian repository [5], among others. I am also > finalizing an integrated interface to CUDA GPUs to perform fast > parallel ODE solving [6]. > > Thanks for your time, > Rob Clewley > > [1] http://pydstool.sourceforge.net > [2] http://www.ni.gsu.edu/~rclewley/Research/index.html, and in > particular http://www.ploscompbiol.org/article/info%3Adoi%2F10.1371%2Fjournal.pcbi.1002628 > [3] NSF Software Infrastructure for Sustained Innovation (SI2-SSE) > program solicitation: > http://www.nsf.gov/pubs/2013/nsf13525/nsf13525.htm > [4] http://help.ubuntu.com/community/LiveCD > [5] http://neuro.debian.net/ > [6] http://www.nvidia.com/object/cuda_home_new.html > I am looking at documentation now, but a couple things from what I seen: Are you particularly tied to sourceforge? It seems a lot of python development is moving to github, and it makes third party contribution much easier. You can still distribute releases through sourceforge even if you use github for revision control. Are you in touch with the neuroensemble mailing list? This seems relevant to it. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Subclassing ndarray with concatenate
On Wed, Jan 30, 2013 at 11:20 AM, Sebastian Berg wrote: > > > > > > > > In my particular case at least, there are clear ways to > > handle corner > > > cases (like being passed a class that lacks these > > attributes), so in > > > principle there no problem handling concatenate in a general > > way, > > > assuming I can get access to the attributes. > > > > > > > > > So is there any way to subclass ndarray in such a way that > > concatenate > > > can be handled properly? > > > > > > > Quite simply, no. If you compare masked arrays, they also > > provide their > > own concatenate for this reason. > > > > I hope that helps a bit... > > > > > > > > Is this something that should be available? For instance a method > > that provides both the new array and the arrays that were used to > > construct it. This would seem to be an extremely common use-case for > > array subclasses, so letting them gracefully handle this would seem to > > be very important. > In any case, yes, it calls __array_finalize__, but as you noticed, it > calls it without the original array. Now it would be very easy and > harmless to change that, however I am not sure if giving only the parent > array is very useful (ie. you only get the one with highest array > priority). > > Another way to get around it would be maybe to call __array_wrap__ like > ufuncs do (with a context, so you get all inputs, but then the non-array > axis argument may not be reasonably placed into the context). > > In any case, if you think it would be helpful to at least get the single > parent array, that would be a very simple change, but I feel the whole > subclassing could use a bit thinking and quite a bit of work probably, > since I am not quite convinced that calling __array_wrap__ with a > complicated context from as many functions as possible is the right > approach for allowing more complex subclasses. > I was more thinking of a new method that is called when more than one input array is used, maybe something like __multi_array_finalize__. This would allow more fine-grained handling of such cases and would not break backwards compatibility with any existing subclasses (if they don't override the method the current behavior will remain). ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Subclassing ndarray with concatenate
On Tue, Jan 22, 2013 at 1:44 PM, Sebastian Berg wrote: > Hey, > > On Tue, 2013-01-22 at 10:21 +0100, Todd wrote: > > > The main exception I have found is concatenate (and hstack/vstack, > > which just wrap concatenate). In this case, __array_finalize__ is > > passed an array that has already been stripped of the additional > > attributes, and I don't see a way to recover this information. > > > There are quite a few functions that simply do not preserve subclasses > (though I think more could/should call `__array_wrap__` probably, even > if the documentation may say that it is about ufuncs, there are some > example of this already). > `np.concatenate` is one of these. It always returns a base array. In any > case it gets a bit difficult if you have multiple input arrays (which > may not matter for you). > I don't think this is right. I tried it and it doesn't return a base array, it returns an instance of the original array subclass. > > > In my particular case at least, there are clear ways to handle corner > > cases (like being passed a class that lacks these attributes), so in > > principle there no problem handling concatenate in a general way, > > assuming I can get access to the attributes. > > > > > > So is there any way to subclass ndarray in such a way that concatenate > > can be handled properly? > > > Quite simply, no. If you compare masked arrays, they also provide their > own concatenate for this reason. > > I hope that helps a bit... > > Is this something that should be available? For instance a method that provides both the new array and the arrays that were used to construct it. This would seem to be an extremely common use-case for array subclasses, so letting them gracefully handle this would seem to be very important. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Subclassing ndarray with concatenate
I am trying to create a subclass of ndarray that has additional attributes. These attributes are maintained with most numpy functions if __array_finalize__ is used. The main exception I have found is concatenate (and hstack/vstack, which just wrap concatenate). In this case, __array_finalize__ is passed an array that has already been stripped of the additional attributes, and I don't see a way to recover this information. In my particular case at least, there are clear ways to handle corner cases (like being passed a class that lacks these attributes), so in principle there no problem handling concatenate in a general way, assuming I can get access to the attributes. So is there any way to subclass ndarray in such a way that concatenate can be handled properly? I have been looking extensively online, but have not been able to find a clear answer on how to do this, or if there even is a way. ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Z-ordering (Morton ordering) for numpy
So, just FYI, my usage of this is for Rubik, where it's a communication latency optimization for the code being mapped to the network. I haven't tested it as an optimization for particular in-core algorithms. However, there was some work on this at LLNL maybe a couple years ago -- I think it was for the solvers. I'll ask around for an example and/or a paper, or maybe Travis has examples. Just from an ease-of-use point of view, though, if you make it simple to do zordering, you might see more people using it :). That's why I wanted to get this into numpy. This brings up another point. This is pure python, so it won't be super-fast. What's the typical way things are integrated and optimized in numpy? Do you contribute something like this in python first then convert to cython/C as necessary? Or would you want it in C to begin with? -Todd On Nov 24, 2012, at 12:17 PM, Aron Ahmadia wrote: > Todd, > > I am optimistic and I think it would be a good idea to put this in. A couple > previous studies [1] haven't found any useful speedups from in-core > applications for Morton-order, and if you have results for real scientific > applications using numpy this would not only be great, but the resulting > paper would have quite a bit of impact. I'm sure you're already connected to > the right people at LLNL, but I can think of a couple other projects which > might be interested in trying this sort of thing out. > > http://www.cs.utexas.edu/~pingali/CS395T/2012sp/papers/co.pdf > > Cheers, > Aron > > > On Sat, Nov 24, 2012 at 8:10 PM, Travis Oliphant wrote: > This is pretty cool.Something like this would be interesting to play > with. There are some algorithms that are faster with z-order arrays.The > code is simple enough and small enough that I could see putting it in NumPy. > What do others think? > > -Travis > > > > On Nov 24, 2012, at 1:03 PM, Gamblin, Todd wrote: > > > Hi all, > > > > In the course of developing a network mapping tool I'm working on, I also > > developed some python code to do arbitrary-dimensional z-order (morton > > order) for ndarrays. The code is here: > > > > https://github.com/tgamblin/rubik/blob/master/rubik/zorder.py > > > > There is a function to put the elements of an array in Z order, and another > > one to enumerate an array's elements in Z order. There is also a ZEncoder > > class that can generate Z-codes for arbitrary dimensions and bit widths. > > > > I figure this is something that would be generally useful. Any interest in > > having this in numpy? If so, what should the interface look like and can > > you point me to a good spot in the code to add it? > > > > I was thinking it might make sense to have a Z-order iterator for ndarrays, > > kind of like ndarray.flat. i.e.: > > > > arr = np.empty([4,4], dtype=int) > > arr.flat = range(arr.size) > > for elt in arr.zorder: > > print elt, > > 0 4 1 5 8 12 9 13 2 6 3 7 10 14 11 15 > > > > Or an equivalent to ndindex: > > > > arr = np.empty(4,4, dtype=int) > > arr.flat = range(arr.size) > > for ix in np.zindex(arr.shape): > > print ix, > > (0, 0) (1, 0) (0, 1) (1, 1) (2, 0) (3, 0) (2, 1) (3, 1) (0, 2) (1, 2) > > (0, 3) (1, 3) (2, 2) (3, 2) (2, 3) (3, 3) > > > > Thoughts? > > > > -Todd > > __ > > Todd Gamblin, tgamb...@llnl.gov, http://people.llnl.gov/gamblin2 > > CASC @ Lawrence Livermore National Laboratory, Livermore, CA, USA > > > > ___ > > NumPy-Discussion mailing list > > NumPy-Discussion@scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion __ Todd Gamblin, tgamb...@llnl.gov, http://people.llnl.gov/gamblin2 CASC @ Lawrence Livermore National Laboratory, Livermore, CA, USA ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Z-ordering (Morton ordering) for numpy
Hi all, In the course of developing a network mapping tool I'm working on, I also developed some python code to do arbitrary-dimensional z-order (morton order) for ndarrays. The code is here: https://github.com/tgamblin/rubik/blob/master/rubik/zorder.py There is a function to put the elements of an array in Z order, and another one to enumerate an array's elements in Z order. There is also a ZEncoder class that can generate Z-codes for arbitrary dimensions and bit widths. I figure this is something that would be generally useful. Any interest in having this in numpy? If so, what should the interface look like and can you point me to a good spot in the code to add it? I was thinking it might make sense to have a Z-order iterator for ndarrays, kind of like ndarray.flat. i.e.: arr = np.empty([4,4], dtype=int) arr.flat = range(arr.size) for elt in arr.zorder: print elt, 0 4 1 5 8 12 9 13 2 6 3 7 10 14 11 15 Or an equivalent to ndindex: arr = np.empty(4,4, dtype=int) arr.flat = range(arr.size) for ix in np.zindex(arr.shape): print ix, (0, 0) (1, 0) (0, 1) (1, 1) (2, 0) (3, 0) (2, 1) (3, 1) (0, 2) (1, 2) (0, 3) (1, 3) (2, 2) (3, 2) (2, 3) (3, 3) Thoughts? -Todd __ Todd Gamblin, tgamb...@llnl.gov, http://people.llnl.gov/gamblin2 CASC @ Lawrence Livermore National Laboratory, Livermore, CA, USA ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Hierarchical vs non-hierarchical ndarray.base and __array_interface__
Hi all, I posted on the change in semantics of ndarray.base here: https://github.com/numpy/numpy/commit/6c0ad59#commitcomment-2153047 And some folks asked me to post my question to the numpy mailing list. I've implemented a tool for mapping processes in parallel applications to nodes in cartesian networks. It uses hierarchies of numpy arrays to represent the domain decomposition of the application, as well as corresponding groups of processes on the network. You can "map" an application to the network using assignment of through views. The tool is here if anyone is curious: https://github.com/tgamblin/rubik. I used numpy to implement this because I wanted to be able to do mappings for arbitrary-dimensional networks. Blue Gene/Q, for example, has a 5-D network. The reason I bring this up is because I rely on the ndarray.base pointer and some of the semantics in __array_interface__ to translate indices within my hierarchy of views. e.g., if a value is at (0,0) in a view I want to know that it's actually at (4,4) in its immediate parent array. After looking over the commit I linked to above, I realized I'm actually relying on a lot of stuff that's not guaranteed by numpy. I rely on .base pointing to its closest parent, and I rely on __array_interface__.data containing the address of the array's memory and its strides. None of these is guaranteed by the API docs: http://docs.scipy.org/doc/numpy/reference/arrays.interface.html So I guess I have a few questions: 1. Is translating indices between base arrays and views something that would be useful to other people? 2. Is there some better way to do this than using ndarray.base and __array_interface__? 3. What's the numpy philosophy on this? Should views know about their parents or not? They obviously have to know a little bit about their memory, but whether or not they know how they were derived from their owning array is a different question. There was some discussion on the vagueness of .base here: http://thread.gmane.org/gmane.comp.python.numeric.general/51688/focus=51703 But it doesn't look like you're deprecating .base in 1.7, only changing its behavior, which I tend to agree is worse than deprecating it. After thinking about all this, I'm not sure what I would like to happen. I can see the value of not keeping extra references around within numpy, and my domain is pretty different from the ways that I imagine people use numpy. I wouldn't have to change my code much to make it work without .base, but I do rely on __array_interface__. If that doesn't include the address and strides, t think I'm screwed as far as translating indices go. Any suggestions? Thanks! -Todd __ Todd Gamblin, tgamb...@llnl.gov, http://people.llnl.gov/gamblin2 CASC @ Lawrence Livermore National Laboratory, Livermore, CA, USA ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] installing numpy in jython (in Netbeans)
On 8/27/2012 9:51 AM, Chris Barker wrote: > On Sun, Aug 26, 2012 at 8:53 PM, Todd Brunhoff wrote: >> Chris, >> winpdb is ok, although it is only a graphic debugger, not an ide, emphasis >> on the 'd'. > yup -- I mentioned, that as you seem to like NB -- and I know I try to > use the same editor for eveything. > > But if you want a nice full-on IDE for Python, there are a lot of > them. I"m an editor_termal guy, so I can't make a recommendation, but > some of the biggies are: > > Eclipse+PyDev > PyCharm > WindIDE > Spyder (particularly nice for numpy/ matplotlib, etc) I had not considered these yet, but they look interesting. I ended up here: http://en.wikipedia.org/wiki/Comparison_of_integrated_development_environments#Python which compares IDEs for every language, and found a free python plugin for VS 2010 which looks excellent. I may also try Spyder since I would expect you atmospheric guys would know where numericals are well integrated. Thanks. Todd > > -Chris > > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] installing numpy in jython (in Netbeans)
Chris, I appreciate the pointers, which appear to confirm that numpy and jython are a ways out. I can see where the c-api support in jython would be required by numpy's implementation. > 1) [NetBeans support for CPython] -- Maybe this: > http://wiki.netbeans.org/Python That seems outdated. I had used http://wiki.netbeans.org/PythonInstall which points at a plugin repository which does pretty good install. But of course it is jython, not python. And the Hudson (Jenkins) build was last done Oct 22, 2010. And interestingly enough, that info points back to the link you found. Oh well. > 2) Python programming can be pretty darn productive with "print" > debugging Works for me. And sometimes there *is* no other way. > 3) There are a lot of good IDEs for CPython -- and stad-alone > debuggers (WinPdb, for example), so you don't need to be married to > the NetBeans environment. winpdb is ok, although it is only a graphic debugger, not an ide, emphasis on the 'd'. As it is, since NB cannot debug python that requires numpy, it means I must rely on vi & Netbeans for editing and winpdb for debugging. I don't think about the ide's as being married to them, but I do like to use the best tools available. NB could be, but is not for python. Thanks again for helping to shorten my search for tools. Todd On 8/26/2012 5:55 PM, Chris Barker wrote: > Todd, > > The short version is: you can't do that. -- Jython uses the JVM, numpy > is very, very tied into the CPython runtime. > > This thread is a bit old, but think still holds: > > http://stackoverflow.com/questions/3097466/using-numpy-and-cpython-with-jython > > There is the junumeric project, but it doesn't look very active: > > https://bitbucket.org/zornslemon/jnumeric-ra/downloads/ > > According to the Jython FAQ > (http://wiki.python.org/jython/JythonFaq/GeneralInfo): > "For the next release of Jython, we plan to support the C Python > Extension API" -- not sure what the timing is on that, but in theory, > you could then use numpy ,but numpy is a pretty complex beast -- I > wouldn't recommend being the first to try it! > > So: other options: > > 1) A couple years ago the NetBeans team was talking about improving > the support for Python (CPython) -- you might want to see if there is > a plugin or something that you could add. Maybe this: > http://wiki.netbeans.org/Python > > 2) Python programming can be pretty darn productive with "print" > debugging -- a lot of us never use a "proper" debugger -- a good > programmers editor and a console is all you need. > > 3) There are a lot of good IDEs for CPython -- and stad-alone > debuggers (WinPdb, for example), so you don't need to be married to > the NetBeans environment. > > In short: It'll probably be a lot easier to find a different > programming (or debugging anyway) )environment than get numpy to work > work with Jython. > > -Chris > > > > > > On Sun, Aug 26, 2012 at 4:36 PM, Todd Brunhoff wrote: >> Being a newbie to this list, I recognize that the answer to this might be >> "why would you do that?". But surely it can't be worse than that. >> >> Briefly put, I want to install numpy and scipy in jython (reasons below). >> Running 'cd; jython setup.py install' runs into errors. But >> installing he same distro under python works fine. So the underlying >> question is: is there a way to install numpy under jython? >> >> Here's how I got here. The reason I would like this under jython is wrapped >> up in Netbeans, Octave, Matlab and Windows 7. I've been using Netbeans under >> linux for years for c++ and a bit of Java, and the editing environment with >> cross referencing and syntax-directed editing is quite good. Netbeans only >> presents python debugging via jython, which makes sense. Most of the work I >> am doing is with matrix algebra and I started with Octave, but while Octave >> is excellent for operating on matrices, it is not good for file format >> manipulations, hence for some operations I'd like to turn to python, and if >> I edit under Netbeans, the debugging requires that I install numpy under >> jython. At work I use a machine running fedora 16 but... I travel a bit >> and my travel machine is a laptop running windows 7. Therefore, Windows 7 + >> Netbeans + numpy + debugging ==> jython + numpy + scipy. >> >> Here's the install problems, which occur under numpy-1.7.0b1 and 1.6.2. The >> first install error is in numpy\distutils\exec_command.py, line 585, where >> it throws an exception because the java exec tests are unlimited. So I >>
[Numpy-discussion] installing numpy in jython (in Netbeans)
Being a newbie to this list, I recognize that the answer to this might be "why would you do that?". But surely it can't be worse than that. Briefly put, I want to install numpy and scipy in jython (reasons below). Running 'cd ; jython setup.py install' runs into errors. But installing he same distro under python works fine. So the underlying question is: is there a way to install numpy under jython? Here's how I got here. The reason I would like this under jython is wrapped up in Netbeans, Octave, Matlab and Windows 7. I've been using Netbeans under linux for years for c++ and a bit of Java, and the editing environment with cross referencing and syntax-directed editing is quite good. Netbeans only presents python debugging via jython, which makes sense. Most of the work I am doing is with matrix algebra and I started with Octave, but while Octave is excellent for operating on matrices, it is not good for file format manipulations, hence for some operations I'd like to turn to python, and if I edit under Netbeans, the debugging requires that I install numpy under jython. At work I use a machine running fedora 16 but... I travel a bit and my travel machine is a laptop running windows 7. Therefore, Windows 7 + Netbeans + numpy + debugging ==> jython + numpy + scipy. Here's the install problems, which occur under numpy-1.7.0b1 and 1.6.2. The first install error is in numpy\distutils\exec_command.py, line 585, where it throws an exception because the java exec tests are unlimited. So I comment out those two lines, and rerun the setup.py. The next errors are notes about features not being available because other packages are also unavailable. I don't think this is really getting in the way, although I could be wrong. The libraries mentioned as missing are: * libraries mkl,vml,guide * libraries ptf77blas,ptcblas,atlas * libraries lapack_atlas So setup gets as far as this: running install running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_src build_src building py_modules sources creating build creating build\src.java1.6.0_33-2.5 creating build\src.java1.6.0_33-2.5\numpy creating build\src.java1.6.0_33-2.5\numpy\distutils building library "npymath" sources No module named jythoncompiler in numpy.distutils; trying from distutils customize GnuFCompiler Could not locate executable g77 Could not locate executable f77 don't know how to compile Fortran code on platform 'java' Some of these messages make it look as if the config wants to use the cygwin gnu compilers, which are ok, but under windows are not nearly as good as mingw gnu compiler or better yet, the visual studio 2010 compiler. I have both, but I don't see a way to steer the numpy setup to use them. The next error is fatal File "...\numpy-1.7.0b1\numpy\distutils\ccompiler.py", line 111, in CCompiler_object_filenames if ext not in self.src_extensions: TypeError: 'NoneType' object is not iterable This one looks as if it does not know what .o or .a or .obj files are. Fixing this one looks like hours of digging through the code. Is there a simpler solution? Thanks in advance, Todd ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numpy issues at startup
All right, here goes... >>> print numpy.__file__ /home/d1/turnertj/Python-2.5.4/lib/python2.5/site-packages/numpy/numpy/__init__.pyc [turne...@daneel numpy]$ ls add_newdocs.py ctypeslib.pyc dual.py _import_tools.py libnumarray setup.py tests coredistutils f2py __init__.py linalg oldnumeric setup.pyc version.py ctypeslib.pydocfft __init__.pyc matlib.py random testingversion.pyc -Original Message- From: numpy-discussion-boun...@scipy.org [mailto:numpy-discussion-boun...@scipy.org] On Behalf Of Robert Kern Sent: Tuesday, June 02, 2009 2:24 PM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] numpy issues at startup On Tue, Jun 2, 2009 at 13:19, Turner, Todd J Civ USAF AFMC AFRL/RXLMP wrote: > I’m having some numpy problems when using the package with Python. My admin > installed numpy for me and we think it’s installed right. When I’m in > python and type ‘import numpy’ I get the “Running numpy from source > directory”. It doesn’t matter which directory I launch python out of, it > always gives me the same errors and then can’t do anything simple like > numpy.ones() or any other routine. Any ideas? Do I have a path variable > set wrong? Please do the following: $ python >>> import numpy >>> print numpy.__file__ That should show you where the numpy on your sys.path is. Double-check that it is in the place you think it is. If so, move to that directory (e.g. /usr/lib/python2.5/site-packages/numpy/) and show us what is in there: $ cd /usr/lib/python2.5/site-packages/numpy/ $ ls -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] numpy issues at startup
I'm having some numpy problems when using the package with Python. My admin installed numpy for me and we think it's installed right. When I'm in python and type 'import numpy' I get the "Running numpy from source directory". It doesn't matter which directory I launch python out of, it always gives me the same errors and then can't do anything simple like numpy.ones() or any other routine. Any ideas? Do I have a path variable set wrong? Thanks for the help. Tj Turner ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] 64-bit numpy questions?
Hi,
I've been looking at a 64-bit numpy problem we were having on Solaris:
>>> a=numpy.zeros(0x18000,dtype='b1')
>>> a.data
Traceback (most recent call last):
File "", line 1, in
ValueError: size must be zero or positive
A working fix seemed to be this:
Index: arrayobject.c
===
--- arrayobject.c(revision 6530)
+++ arrayobject.c(working copy)
@@ -6774,7 +6774,7 @@
static PyObject *
array_data_get(PyArrayObject *self)
{
-intp nbytes;
+Py_ssize_t nbytes;
if (!(PyArray_ISONESEGMENT(self))) {
PyErr_SetString(PyExc_AttributeError, "cannot get single-"\
"segment buffer for discontiguous array");
@@ -6782,10 +6782,10 @@
}
nbytes = PyArray_NBYTES(self);
if PyArray_ISWRITEABLE(self) {
-return PyBuffer_FromReadWriteObject((PyObject *)self, 0, (int)
nbytes);
+return PyBuffer_FromReadWriteObject((PyObject *)self, 0,
(Py_ssize_t) nbytes);
}
else {
-return PyBuffer_FromObject((PyObject *)self, 0, (int) nbytes);
+return PyBuffer_FromObject((PyObject *)self, 0, (Py_ssize_t)
nbytes);
}
}
This fix could be simpler but still illustrates the typical problem:
use of (or cast to) int rather than something "pointer sized".
I can see that a lot of effort has gone into making numpy 64-bit
enabled, but I also see a number of uses of int which look like
problems on LP64 platforms. Is anyone using numpy in 64-bit
environments on a day-to-day basis? Are you using very large arrays,
i.e. over 2G in size?
Cheers,
Todd
___
Numpy-discussion mailing list
Numpy-discussion@scipy.org
http://projects.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] numarray argmax problem
Otto Tronarp wrote: > Hi, > > I have a problem with numarray.argmax, the following code > > import numarray as N > import sys > c = N.zeros((10, ), N.Float64) > while 1: > print 'B: ', sys.getrefcount(None) > l = N.argmax(c) > print 'A: ', sys.getrefcount(None) > print > > Dies with: > Fatal Python error: deallocating None > Aborted > > I'm using numarray 1.5.2 > > Any chance that there is an easy fix for this? I know I should > If you're willing to check out numarray and build it from source yourself, this problem is fixed in numarray CVS here: http://sourceforge.net/cvs/?group_id=1369 You can check it out like this: cvs -d:pserver:[EMAIL PROTECTED]:/cvsroot/numpy login cvs -z3 -d:pserver:[EMAIL PROTECTED]:/cvsroot/numpy co -P /numarray / Regards, Todd > consider switching to numpy, but I don't have the time right now to do > that. > > Otto > > ___ > Numpy-discussion mailing list > Numpy-discussion@scipy.org > http://projects.scipy.org/mailman/listinfo/numpy-discussion > ___ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
