Re: [Numpy-discussion] ANN: HDF5 for Python 2.5.0
Congrats! Also btw, you might want to switch to a new subject line format for these emails -- the mention of Python 2.5 getting hdf5 support made me do a serious double take before I figured out what was going on, and 2.6 and 2.7 will be even worse :-) Ha! Didn't even think of that. For our next release I guess we'll have to go straight to h5py 3.5. Andrew ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Non-meta indexing improvements discussion
On Do, 2015-04-09 at 08:50 +0200, Sebastian Berg wrote: Hi all, Let me take a shot at summing up some suggestions to make the indexing less surprising, and maybe we can gather some more in a more concentrated way now. Did not want to comment on the first mail 1. Implement something like `arr.fancy_index[...]` and `arr.ortho_index[...]` (i.e. Jaimes PR is the start for trying this) I like this, personally. There is not much to be lost and I fully agree with Robert on this. It opens up a lot of possibilities for us and especially also others. 2. Add warnings for non-consecutive advanced indexing (i.e. the original example `arr[0, :, index_array]`). This could be annoying sometimes, but then warnings do not break legacy code, and I think in new code again Robert is right for these cases using arr.fancy_index[...] is more explicit and a nice warning/google help to the confused reader. 3. I do not know if it possible or useful, but I could imagine a module wide switch (similar to __future__ imports) to change the default indexing behaviour. OK, my suggestion But actually I do not know if I like it all that much (nor if it can be done) since 1. and 2. seem to me like enough. But if someone feels strongly about fancy indexing being bad, I wanted to point out that there may be ways to go down the road to switch numpy without actually switching. - Sebastian One more thing, implementing this (especially the new indexing) is non-trivial, so as always help beyond just a discussion is appreciated and in my opinion the best way to push an actual change to happen sooner rather then in some far off future. I do not have time for concentrating much on an implementation for a while myself for a while at least. - Sebastian ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ANN: HDF5 for Python 2.5.0
(Off-list) Congrats! Also btw, you might want to switch to a new subject line format for these emails -- the mention of Python 2.5 getting hdf5 support made me do a serious double take before I figured out what was going on, and 2.6 and 2.7 will be even worse :-) On Apr 9, 2015 2:07 PM, Andrew Collette andrew.colle...@gmail.com wrote: Announcing HDF5 for Python (h5py) 2.5.0 The h5py team is happy to announce the availability of h5py 2.5.0. This release introduces experimental support for the highly-anticipated Single Writer Multiple Reader (SWMR) feature in the upcoming HDF5 1.10 release. SWMR allows sharing of a single HDF5 file between multiple processes without the complexity of MPI or multiprocessing-based solutions. This is an experimental feature that should NOT be used in production code. We are interested in getting feedback from the broader community with respect to performance and the API design. For more details, check out the h5py user guide: http://docs.h5py.org/en/latest/swmr.html SWMR support was contributed by Ulrik Pedersen. What's h5py? The h5py package is a Pythonic interface to the HDF5 binary data format. It lets you store huge amounts of numerical data, and easily manipulate that data from NumPy. For example, you can slice into multi-terabyte datasets stored on disk, as if they were real NumPy arrays. Thousands of datasets can be stored in a single file, categorized and tagged however you want. Documentation is at: http://docs.h5py.org Changes --- * Experimental SWMR support * Group and AttributeManager classes now inherit from the appropriate ABCs * Fixed an issue with 64-bit float VLENS * Cython warning cleanups related to const * Entire code base ported to six; 2to3 removed from setup.py Acknowledgements --- This release incorporates changes from, among others: * Ulrik Pedersen * James Tocknell * Will Parkin * Antony Lee * Peter H. Li * Peter Colberg * Ghislain Antony Vaillant Where to get it --- Downloads, documentation, and more are available at the h5py website: http://www.h5py.org ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: fancy vs. orthogonal)
On Thu, Apr 9, 2015 at 10:11 AM, Alan G Isaac alan.is...@gmail.com wrote: Alan wrote: 3. I admit, my students are NOT using non-boolen fancy indexing on multidimensional arrays. (As far as I know.) Are yours? The only confusing case is mixing slices and integer array indexing for ndim 2. The rest looks unsurprising, AFAIR (AFAICS, my last fancy indexing mailing list discussion is at least 4 years old, with Jonathan Taylor. I don't remember when I discovered the usefulness of the axis argument in take which covers many 3 or higher dimensional indexing use cases.) On 4/9/2015 2:22 AM, Nathaniel Smith wrote: Well, okay, this would explain it, since integer fancy indexing is exactly the confusing case:-) On the plus side, this also means that even if pigs started doing barrel-rolls through hell's winter-vortex-chilled air tomorrow and we simply removed integer fancy indexing, your students would be unaffected:-) Except that they do use statsmodels, which I believe (?) does make use of integer fancy-indexing. And maybe all work would come to a standstill, because every library is using fancy integer indexing. I still don't know what all constitutes fancy indexing. The two most common use cases for me (statsmodels) are indexing for selecting elements like diag_indices, triu_indices and maybe nonzero, and expanding from a unique array like inverse index in np.unique. And there are just a few, AFAIR, orthogonal indexing cases with broadcasting index arrays to select rectangular pieces of an array. Josef Alan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] ANN: HDF5 for Python 2.5.0
Announcing HDF5 for Python (h5py) 2.5.0 The h5py team is happy to announce the availability of h5py 2.5.0. This release introduces experimental support for the highly-anticipated Single Writer Multiple Reader (SWMR) feature in the upcoming HDF5 1.10 release. SWMR allows sharing of a single HDF5 file between multiple processes without the complexity of MPI or multiprocessing-based solutions. This is an experimental feature that should NOT be used in production code. We are interested in getting feedback from the broader community with respect to performance and the API design. For more details, check out the h5py user guide: http://docs.h5py.org/en/latest/swmr.html SWMR support was contributed by Ulrik Pedersen. What's h5py? The h5py package is a Pythonic interface to the HDF5 binary data format. It lets you store huge amounts of numerical data, and easily manipulate that data from NumPy. For example, you can slice into multi-terabyte datasets stored on disk, as if they were real NumPy arrays. Thousands of datasets can be stored in a single file, categorized and tagged however you want. Documentation is at: http://docs.h5py.org Changes --- * Experimental SWMR support * Group and AttributeManager classes now inherit from the appropriate ABCs * Fixed an issue with 64-bit float VLENS * Cython warning cleanups related to const * Entire code base ported to six; 2to3 removed from setup.py Acknowledgements --- This release incorporates changes from, among others: * Ulrik Pedersen * James Tocknell * Will Parkin * Antony Lee * Peter H. Li * Peter Colberg * Ghislain Antony Vaillant Where to get it --- Downloads, documentation, and more are available at the h5py website: http://www.h5py.org ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] Non-meta indexing improvements discussion
On Thu, Apr 9, 2015 at 8:07 AM, Sebastian Berg sebast...@sipsolutions.net wrote: On Do, 2015-04-09 at 08:50 +0200, Sebastian Berg wrote: 3. I do not know if it possible or useful, but I could imagine a module wide switch (similar to __future__ imports) to change the default indexing behaviour. OK, my suggestion But actually I do not know if I like it all that much (nor if it can be done) since 1. and 2. seem to me like enough. But if someone feels strongly about fancy indexing being bad, I wanted to point out that there may be ways to go down the road to switch numpy without actually switching. I can't think of a way to actually make that work (I can list all the ways I thought of that *don't* work if anyone insists, but it's a tedious dead end), but it also seems to me to be a step backward. Assuming that we have both .ortho_ix and .fancy_ix to work with, it seems to me that the explicitness is a good thing. Even if in this module you only want to exploit one of those semantics, your module's readers live in a wider context where both semantics play a role. Moving your marker of this is the non-default semantics I am using to some module or function header away from where the semantics are actually used makes the code harder to read. A newish user trying to read some nontrivial indexing code may come to the list and ask what exactly is this expression doing here? and give us just the line of code with the expression (anecdotally, this is usually how this scenario goes down). If we have to answer it depends; is there an @ortho_indexing decorator at the top of the function?, that's probably a cure worse than the disease. The properties are a good way to provide googleable signposts right where the tricky semantics are being used. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: fancy vs. orthogonal)
On Do, 2015-04-09 at 02:22 -0400, Nathaniel Smith wrote: On Wed, Apr 8, 2015 at 4:02 PM, Alan G Isaac alan.is...@gmail.com wrote: 1. I use numpy in teaching. I have never heard a complaint about its indexing behavior. Have you heard such complaints? Some observations: 1) There's an unrelated thread on numpy-discussion right now in which a user is baffled by the interaction between slicing and integer fancy indexing: http://thread.gmane.org/gmane.comp.python.numeric.general/60321 And one of the three replies AFAICT also doesn't actually make sense, in that its explanation relies on broadcasting two arrays with shape (5,) against each other to produce an array with shape (5, 5). (Which is not how broadcasting works.) To be fair, though, this isn't the poster's fault, because they are quoting the documentation! 2) Again, entirely by coincidence, literally this week a numpy user at Berkeley felt spontaneously moved to send a warning message to the campus py4science list just to warn everyone about the bizarre behaviour they had stumbled on where arr[0, :, idx] produced inexplicable results. They had already found the docs and worked out what was going on, they just felt it was necessary to warn everyone else to be careful out there. 3) I personally regularly get confused by integer fancy indexing. I actually understand it substantially better due to thinking it through while reading these threads, but I'm a bit disturbed that I had that much left to learn. (New key insight: you can think of *scalar* indexing arr[i, j, k] as a function f(i, j, k) - value. If you take that function and make it a ufunc, then you have integer fancy indexing. ...Though there's still an extra pound of explanation needed to describe the mixed slice/fancy cases, it at least captures the basic intuition. Maybe this was already obvious to everyone else, but it helped me.) 4) Even with my New and Improved Explanatory Powers, when this thread came up chatting with Thomas Kluyver today, I attempted to provide a simple, accurate description of how numpy indexing works so that the debate would make sense, and his conclusion was (paraphrased) okay, now I don't understand numpy indexing anymore and never did. I say this not to pick on Thomas, but to make that point that Thomas is a pretty smart guy so maybe this is actually confusing. (Or maybe I'm just terrible at explaining things.) I actually think the evidence is very very strong that numpy's current way of mixing integer fancy indexing and slice-based indexing is a mistake. It's just not clear whether there's anything we can do to mitigate that mistake (or indeed, what would actually be better even if we could start over from scratch). (Which we can't.) I think the best way to think about the mixing is to think about subspaces defined by all of the slices which are taken for each individual fancy indexing element. I.e. each subspaces is something like: new[:, 0, :] = arr[:, fancy1[0], fancy2[0], :] then you iterate the fancy indexes so the subspaces moves ahead: new[:, 1, :] = arr[:, fancy1[1], fancy2[1], :] new[:, 2, :] = arr[:, fancy1[2], fancy2[2], :] and so on. This is also how it is implemented. Plus of course the transposing to the front when the fancy indices are not consecutive and you cannot add the fancy dimensions to where they were. I think you mentioned an error in the docu, I thought I cleared some of them, but proabably that did not make it more understandable sometimes. The whole subspace way of is used, but there is a lot of improvement possible and I would be happy if more feel like stepping up to fill that void, since you do not need to know the implementation details for that. - Sebastian 2. One reason I use numpy in teaching is its indexing behavior. What specific language provides a better indexing model, in your opinion? 3. I admit, my students are NOT using non-boolen fancy indexing on multidimensional arrays. (As far as I know.) Are yours? Well, okay, this would explain it, since integer fancy indexing is exactly the confusing case :-) On the plus side, this also means that even if pigs started doing barrel-rolls through hell's winter-vortex-chilled air tomorrow and we simply removed integer fancy indexing, your students would be unaffected :-) -n signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] SIAM meeting in Snowbird anyone?
Hey, since I am not in the US often and the SIAM conference is pretty large, I was wondering if some more of our community will be at the SIAM conference in Snowbird around May 17th-21st and would like to meet up then. - Sebastian signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] Non-meta indexing improvements discussion
Hi all, Let me take a shot at summing up some suggestions to make the indexing less surprising, and maybe we can gather some more in a more concentrated way now. 1. Implement something like `arr.fancy_index[...]` and `arr.ortho_index[...]` (i.e. Jaimes PR is the start for trying this) 2. Add warnings for non-consecutive advanced indexing (i.e. the original example `arr[0, :, index_array]`). 3. I do not know if it possible or useful, but I could imagine a module wide switch (similar to __future__ imports) to change the default indexing behaviour. One more thing, implementing this (especially the new indexing) is non-trivial, so as always help beyond just a discussion is appreciated and in my opinion the best way to push an actual change to happen sooner rather then in some far off future. I do not have time for concentrating much on an implementation for a while myself for a while at least. - Sebastian signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ANN: HDF5 for Python 2.5.0
On 9 Apr 2015, at 9:41 pm, Andrew Collette andrew.colle...@gmail.com wrote: Congrats! Also btw, you might want to switch to a new subject line format for these emails -- the mention of Python 2.5 getting hdf5 support made me do a serious double take before I figured out what was going on, and 2.6 and 2.7 will be even worse :-) Ha! Didn't even think of that. For our next release I guess we'll have to go straight to h5py 3.5. You may have to hurry though ;-) Monday, March 30, 2015 Python 3.5.0a3 has been released. This is the third alpha release of Python 3.5, which will be the next major release of Python. Python 3.5 is still under heavy development and is far from complete.” 3 alpha releases in 7 weeks… On a more serious note though, h5py 2.5.x in the subject would be perfectly clear enough, I think, and also help to distinguish from pytables releases. Derek ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: fancy vs. orthogonal)
Alan wrote: 3. I admit, my students are NOT using non-boolen fancy indexing on multidimensional arrays. (As far as I know.) Are yours? On 4/9/2015 2:22 AM, Nathaniel Smith wrote: Well, okay, this would explain it, since integer fancy indexing is exactly the confusing case:-) On the plus side, this also means that even if pigs started doing barrel-rolls through hell's winter-vortex-chilled air tomorrow and we simply removed integer fancy indexing, your students would be unaffected:-) Except that they do use statsmodels, which I believe (?) does make use of integer fancy-indexing. Alan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ANN: HDF5 for Python 2.5.0
On Thu, Apr 9, 2015 at 2:41 PM, Nathaniel Smith n...@pobox.com wrote: (Off-list) Congrats! Also btw, you might want to switch to a new subject line format for these emails -- the mention of Python 2.5 getting hdf5 support made me do a serious double take before I figured out what was going on, and 2.6 and 2.7 will be even worse :-) (offlist) I also had to read the subject line and the first paragraph several times to see who is using python 2.5 Josef :} On Apr 9, 2015 2:07 PM, Andrew Collette andrew.colle...@gmail.com wrote: Announcing HDF5 for Python (h5py) 2.5.0 The h5py team is happy to announce the availability of h5py 2.5.0. This release introduces experimental support for the highly-anticipated Single Writer Multiple Reader (SWMR) feature in the upcoming HDF5 1.10 release. SWMR allows sharing of a single HDF5 file between multiple processes without the complexity of MPI or multiprocessing-based solutions. This is an experimental feature that should NOT be used in production code. We are interested in getting feedback from the broader community with respect to performance and the API design. For more details, check out the h5py user guide: http://docs.h5py.org/en/latest/swmr.html SWMR support was contributed by Ulrik Pedersen. What's h5py? The h5py package is a Pythonic interface to the HDF5 binary data format. It lets you store huge amounts of numerical data, and easily manipulate that data from NumPy. For example, you can slice into multi-terabyte datasets stored on disk, as if they were real NumPy arrays. Thousands of datasets can be stored in a single file, categorized and tagged however you want. Documentation is at: http://docs.h5py.org Changes --- * Experimental SWMR support * Group and AttributeManager classes now inherit from the appropriate ABCs * Fixed an issue with 64-bit float VLENS * Cython warning cleanups related to const * Entire code base ported to six; 2to3 removed from setup.py Acknowledgements --- This release incorporates changes from, among others: * Ulrik Pedersen * James Tocknell * Will Parkin * Antony Lee * Peter H. Li * Peter Colberg * Ghislain Antony Vaillant Where to get it --- Downloads, documentation, and more are available at the h5py website: http://www.h5py.org ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: fancy vs. orthogonal)
On 4/9/2015 1:57 AM, Nathaniel Smith wrote: Do you think there's anything we could be doing to reduce this kind of adrenaline reaction while still allowing for relaxed discussion about out-there ideas? numpy3...@scipy.org :-) ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] ANN: HDF5 for Python 2.5.0
On Apr 9, 2015 2:41 PM, Nathaniel Smith n...@pobox.com wrote: (Off-list) Doh, we do reply-to munging, don't we. Oh well. -n ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: fancy vs. orthogonal)
On Wed, Apr 8, 2015 at 4:02 PM, Alan G Isaac alan.is...@gmail.com wrote: 1. I use numpy in teaching. I have never heard a complaint about its indexing behavior. Have you heard such complaints? Some observations: 1) There's an unrelated thread on numpy-discussion right now in which a user is baffled by the interaction between slicing and integer fancy indexing: http://thread.gmane.org/gmane.comp.python.numeric.general/60321 And one of the three replies AFAICT also doesn't actually make sense, in that its explanation relies on broadcasting two arrays with shape (5,) against each other to produce an array with shape (5, 5). (Which is not how broadcasting works.) To be fair, though, this isn't the poster's fault, because they are quoting the documentation! 2) Again, entirely by coincidence, literally this week a numpy user at Berkeley felt spontaneously moved to send a warning message to the campus py4science list just to warn everyone about the bizarre behaviour they had stumbled on where arr[0, :, idx] produced inexplicable results. They had already found the docs and worked out what was going on, they just felt it was necessary to warn everyone else to be careful out there. 3) I personally regularly get confused by integer fancy indexing. I actually understand it substantially better due to thinking it through while reading these threads, but I'm a bit disturbed that I had that much left to learn. (New key insight: you can think of *scalar* indexing arr[i, j, k] as a function f(i, j, k) - value. If you take that function and make it a ufunc, then you have integer fancy indexing. ...Though there's still an extra pound of explanation needed to describe the mixed slice/fancy cases, it at least captures the basic intuition. Maybe this was already obvious to everyone else, but it helped me.) 4) Even with my New and Improved Explanatory Powers, when this thread came up chatting with Thomas Kluyver today, I attempted to provide a simple, accurate description of how numpy indexing works so that the debate would make sense, and his conclusion was (paraphrased) okay, now I don't understand numpy indexing anymore and never did. I say this not to pick on Thomas, but to make that point that Thomas is a pretty smart guy so maybe this is actually confusing. (Or maybe I'm just terrible at explaining things.) I actually think the evidence is very very strong that numpy's current way of mixing integer fancy indexing and slice-based indexing is a mistake. It's just not clear whether there's anything we can do to mitigate that mistake (or indeed, what would actually be better even if we could start over from scratch). (Which we can't.) 2. One reason I use numpy in teaching is its indexing behavior. What specific language provides a better indexing model, in your opinion? 3. I admit, my students are NOT using non-boolen fancy indexing on multidimensional arrays. (As far as I know.) Are yours? Well, okay, this would explain it, since integer fancy indexing is exactly the confusing case :-) On the plus side, this also means that even if pigs started doing barrel-rolls through hell's winter-vortex-chilled air tomorrow and we simply removed integer fancy indexing, your students would be unaffected :-) -n -- Nathaniel J. Smith -- http://vorpus.org ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
[Numpy-discussion] what files to include with compiled fortran extension on Mac
Hello list, I want to send somebody my compiled fortran extension on a Mac (compiled with f2py and gfortran). Problem is that it doesn't work on other Macs unless they also instal xcode (2 GB, yikes!) and gfortran. So apparently there are some additional files missing when I just send the compiled extension. Does anybody know what other files to include or (better) how to compile a fortran extension without needing to send any additional files? Thanks, Mark ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
