Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Han Genuit
There is a way to assign whole masks in the current implementation: a = np.arange(9, maskna=True).reshape((3,3)) a array([[0, 1, 2], [3, 4, 5], [6, 7, 8]]) mask = np.array([[False, False, True], [False, True, False],

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Benjamin Root
It should be possible to remove a mask when copying an array. This was a concession on the part of those pushing for masks. Eventually, I ended up realizing that it resulted in a stronger design. Consider the following: foo(a[4:10]) Should function foo be able to access the rest of array a,

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Han Genuit
Yes, to further iterate on that, you can also create multiple masked views with each its own mask properties. It would be ambiguous to mix a bit-pattern NA together with standard NA's in the same mask, but you can make different specialized masked views on the same data. Also, I like the short

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Stéfan van der Walt
Hi all, On Thu, Oct 27, 2011 at 7:51 PM, Travis Oliphant oliph...@enthought.com wrote: I understand exactly what it would take to add bit-patterns to NumPy.  I also understand what Mark did and agree that it is possible to add Matthew's idea to the current code-base.  I think it is worth

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Gael Varoquaux
On Fri, Oct 28, 2011 at 10:45:09AM +0200, Han Genuit wrote: Also, I like the short and concise abbreviation for 'Not Applicable', NA. It has more common uses than IGNORE. (See also here: http://www.johndcook.com/R_language_for_programmers.html#missing) That's a very R centric point a view:

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Charles R Harris
On Thu, Oct 27, 2011 at 8:51 PM, Travis Oliphant oliph...@enthought.comwrote: As I mentioned. I find the ability to separate an ABSENT idea from an IGNORED idea convincing.In other words, I think distinguishing between masks and bit-patterns is not just an implementation detail, but

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Chris.Barker
On 10/27/11 7:51 PM, Travis Oliphant wrote: As I mentioned. I find the ability to separate an ABSENT idea from an IGNORED idea convincing. In other words, I think distinguishing between masks and bit-patterns is not just an implementation detail, but provides a useful concept for multiple

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Matthew Brett
Hi, On Thu, Oct 27, 2011 at 10:56 PM, Benjamin Root ben.r...@ou.edu wrote: On Thursday, October 27, 2011, Charles R Harris charlesr.har...@gmail.com wrote: On Thu, Oct 27, 2011 at 7:16 PM, Travis Oliphant oliph...@enthought.com wrote: That is a pretty good explanation.   I find myself

[Numpy-discussion] [ANN] SciPy India 2011 Abstracts due November 2nd

2011-10-28 Thread Jarrod Millman
== SciPy 2011 Call for Papers == The third `SciPy India Conference http://scipy.in`_ will be held from December 4th through the 7th at the `Indian Institute of Technology, Bombay (IITB) http://www.iitb.ac.in/`_ in Mumbai, Maharashtra India. At this

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Matthew Brett
Hi, On Fri, Oct 28, 2011 at 9:21 AM, Chris.Barker chris.bar...@noaa.gov wrote: On 10/27/11 7:51 PM, Travis Oliphant wrote: As I mentioned. I find the ability to separate an ABSENT idea from an IGNORED idea convincing. In other words, I think distinguishing between masks and bit-patterns is

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Gary Strangman
I wonder if that might be handled as a scikits-image extension, rather than core numpy? I think Stefan and Nathaniel and Gary Strangman and others are saying we don't want to pay the price of a large memory hike for masking. I suspect that Nathaniel is right, and that a large majority of

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Benjamin Root
On Fri, Oct 28, 2011 at 12:39 PM, Matthew Brett matthew.br...@gmail.comwrote: Hi, On Thu, Oct 27, 2011 at 10:56 PM, Benjamin Root ben.r...@ou.edu wrote: On Thursday, October 27, 2011, Charles R Harris charlesr.har...@gmail.com wrote: On Thu, Oct 27, 2011 at 7:16 PM, Travis

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Benjamin Root
On Fri, Oct 28, 2011 at 12:58 PM, Gary Strangman str...@nmr.mgh.harvard.edu wrote: I wonder if that might be handled as a scikits-image extension, rather than core numpy? I think Stefan and Nathaniel and Gary Strangman and others are saying we don't want to pay the price of a large

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Matthew Brett
Hi, On Fri, Oct 28, 2011 at 11:16 AM, Benjamin Root ben.r...@ou.edu wrote: On Fri, Oct 28, 2011 at 12:39 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Thu, Oct 27, 2011 at 10:56 PM, Benjamin Root ben.r...@ou.edu wrote: On Thursday, October 27, 2011, Charles R Harris

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Lluís
I haven't actually tested the code, but AFAIK the following is a short overview with examples of how the two orthogonal feature axis (ABSENT/IGNORE and PROPAGATE/SKIP) are related and how it all is supposed to work. I have never talked to Mark or anybody else in this list (that is, outside of

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Stéfan van der Walt
On Fri, Oct 28, 2011 at 11:16 AM, Benjamin Root ben.r...@ou.edu wrote: this by making missing data front-and-center.  However, my belief is that Mark's approach is easier to comprehend and is cleaner.  Cleaner features means that it is more likely to be used. Cleaner features may be easier to

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Benjamin Root
2011/10/28 Stéfan van der Walt ste...@sun.ac.za On Fri, Oct 28, 2011 at 11:16 AM, Benjamin Root ben.r...@ou.edu wrote: this by making missing data front-and-center. However, my belief is that Mark's approach is easier to comprehend and is cleaner. Cleaner features means that it is more

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Matthew Brett
Hi, On Fri, Oct 28, 2011 at 12:15 PM, Lluís xscr...@gmx.net wrote: Summarizing: let's forget for a moment that mask has a meaning in english: This is at the core of the problem. You and I know what's really going on - there's a mask over the data. But in what follows we're going to try and

Re: [Numpy-discussion] skip lines at the end of file with loadtxt

2011-10-28 Thread Stéfan van der Walt
On Tue, Oct 25, 2011 at 12:28 PM, Massimo Di Stefano massimodisa...@gmail.com wrote: urllib.urlretrieve('http://www.cdc.noaa.gov/Correlation/amon.us.long.data', 'file.txt') a = np.loadtxt('file.txt', skiprows=1) but it fails becouse of the txt description at the end of the file, It's always

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Stéfan van der Walt
On Fri, Oct 28, 2011 at 12:47 PM, Benjamin Root ben.r...@ou.edu wrote: 2011/10/28 Stéfan van der Walt ste...@sun.ac.za  The implementation as it stands essentially gives us a faster and more integrated version of numpy.ma; but it has become clear from this conversation that such an approach

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Benjamin Root
On Fri, Oct 28, 2011 at 3:02 PM, Matthew Brett matthew.br...@gmail.comwrote: You and I know that I've got an array with values [99, 100, 3] and a mask with values [False, False, True]. So maybe I'd like to see what happens if I take off the mask from the second value. I know that's what I

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Stéfan van der Walt
On Fri, Oct 28, 2011 at 1:14 PM, Benjamin Root ben.r...@ou.edu wrote: Considering that you have admitted before to not regularly using masked arrays, I seriously doubt that you would be able to judge whether this is a significant detriment or not. Let's not be unreasonable; Matthew has a valid

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Matthew Brett
Hi, On Fri, Oct 28, 2011 at 1:14 PM, Benjamin Root ben.r...@ou.edu wrote: On Fri, Oct 28, 2011 at 3:02 PM, Matthew Brett matthew.br...@gmail.com wrote: You and I know that I've got an array with values [99, 100, 3] and a mask with values [False, False, True].  So maybe I'd like to see what

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Charles R Harris
2011/10/28 Stéfan van der Walt ste...@sun.ac.za On Fri, Oct 28, 2011 at 12:47 PM, Benjamin Root ben.r...@ou.edu wrote: 2011/10/28 Stéfan van der Walt ste...@sun.ac.za The implementation as it stands essentially gives us a faster and more integrated version of numpy.ma; but it has

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Benjamin Root
On Fri, Oct 28, 2011 at 3:22 PM, Matthew Brett matthew.br...@gmail.comwrote: Hi, On Fri, Oct 28, 2011 at 1:14 PM, Benjamin Root ben.r...@ou.edu wrote: On Fri, Oct 28, 2011 at 3:02 PM, Matthew Brett matthew.br...@gmail.com wrote: You and I know that I've got an array with values

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Matthew Brett
Hi, On Fri, Oct 28, 2011 at 1:52 PM, Benjamin Root ben.r...@ou.edu wrote: On Fri, Oct 28, 2011 at 3:22 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Fri, Oct 28, 2011 at 1:14 PM, Benjamin Root ben.r...@ou.edu wrote: On Fri, Oct 28, 2011 at 3:02 PM, Matthew Brett

[Numpy-discussion] consensus (was: NA masks in the next numpy release?)

2011-10-28 Thread Nathaniel Smith
On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant oliph...@enthought.com wrote: I think Nathaniel and Matthew provided very specific feedback that was helpful in understanding other perspectives of a difficult problem.     In particular, I really wanted bit-patterns implemented.    However, I

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

2011-10-28 Thread Matthew Brett
Hi, On Fri, Oct 28, 2011 at 2:16 PM, Nathaniel Smith n...@pobox.com wrote: On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant oliph...@enthought.com wrote: I think Nathaniel and Matthew provided very specific feedback that was helpful in understanding other perspectives of a difficult

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

2011-10-28 Thread Matthew Brett
On Fri, Oct 28, 2011 at 2:32 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Fri, Oct 28, 2011 at 2:16 PM, Nathaniel Smith n...@pobox.com wrote: On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant oliph...@enthought.com wrote: I think Nathaniel and Matthew provided very specific

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

2011-10-28 Thread Charles R Harris
On Fri, Oct 28, 2011 at 3:16 PM, Nathaniel Smith n...@pobox.com wrote: On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant oliph...@enthought.com wrote: I think Nathaniel and Matthew provided very specific feedback that was helpful in understanding other perspectives of a difficult problem.

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

2011-10-28 Thread Matthew Brett
Hi, On Fri, Oct 28, 2011 at 2:41 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Fri, Oct 28, 2011 at 3:16 PM, Nathaniel Smith n...@pobox.com wrote: On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant oliph...@enthought.com wrote: I think Nathaniel and Matthew provided very

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

2011-10-28 Thread Matthew Brett
Hi, On Fri, Oct 28, 2011 at 2:43 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Fri, Oct 28, 2011 at 2:41 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Fri, Oct 28, 2011 at 3:16 PM, Nathaniel Smith n...@pobox.com wrote: On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

2011-10-28 Thread Charles R Harris
On Fri, Oct 28, 2011 at 3:56 PM, Matthew Brett matthew.br...@gmail.comwrote: Hi, On Fri, Oct 28, 2011 at 2:43 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Fri, Oct 28, 2011 at 2:41 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Fri, Oct 28, 2011 at 3:16 PM,

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

2011-10-28 Thread Matthew Brett
Hi, On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Fri, Oct 28, 2011 at 3:56 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Fri, Oct 28, 2011 at 2:43 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Fri, Oct 28, 2011 at 2:41

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

2011-10-28 Thread Stéfan van der Walt
On Fri, Oct 28, 2011 at 3:21 PM, Benjamin Root ben.r...@ou.edu wrote: The space issues was never ignored and Mark left room for that to be addressed.  Parameterized dtypes can still be added (and isn't all that different from multi-na). Perhaps I could be convinced of a having np.MA

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

2011-10-28 Thread Charles R Harris
2011/10/28 Stéfan van der Walt ste...@sun.ac.za On Fri, Oct 28, 2011 at 3:21 PM, Benjamin Root ben.r...@ou.edu wrote: The space issues was never ignored and Mark left room for that to be addressed. Parameterized dtypes can still be added (and isn't all that different from multi-na).

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Chris.Barker
On 10/28/11 11:37 AM, Matthew Brett wrote: The main motivation for the alterNEP was our strong feeling that separating ABSENT and IGNORE was easier to comprehend and cleaner. I don't know about easier to comprehend, or cleaner, but it is more feature-full. I see two issues here: 1) being

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

2011-10-28 Thread Matthew Brett
On Fri, Oct 28, 2011 at 3:49 PM, Charles R Harris charlesr.har...@gmail.com wrote: 2011/10/28 Stéfan van der Walt ste...@sun.ac.za On Fri, Oct 28, 2011 at 3:21 PM, Benjamin Root ben.r...@ou.edu wrote: The space issues was never ignored and Mark left room for that to be addressed.  

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Charles R Harris
On Fri, Oct 28, 2011 at 5:05 PM, Chris.Barker chris.bar...@noaa.gov wrote: On 10/28/11 11:37 AM, Matthew Brett wrote: The main motivation for the alterNEP was our strong feeling that separating ABSENT and IGNORE was easier to comprehend and cleaner. I don't know about easier to comprehend,

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

2011-10-28 Thread Charles R Harris
On Fri, Oct 28, 2011 at 5:09 PM, Matthew Brett matthew.br...@gmail.comwrote: On Fri, Oct 28, 2011 at 3:49 PM, Charles R Harris charlesr.har...@gmail.com wrote: 2011/10/28 Stéfan van der Walt ste...@sun.ac.za On Fri, Oct 28, 2011 at 3:21 PM, Benjamin Root ben.r...@ou.edu wrote: The

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

2011-10-28 Thread Ralf Gommers
On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett matthew.br...@gmail.comwrote: Hi, On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Fri, Oct 28, 2011 at 3:56 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Fri, Oct 28, 2011 at 2:43

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Stéfan van der Walt
On Fri, Oct 28, 2011 at 4:19 PM, Charles R Harris charlesr.har...@gmail.com wrote: Memory use is a known problem. One way to start addressing it might be to implement a bit arraytype. It might even be possible to prototype that on top of the existing types. Views make bit arrays a bit more

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

2011-10-28 Thread Matthew Brett
Hi, On Fri, Oct 28, 2011 at 4:21 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Fri, Oct 28, 2011 at 5:09 PM, Matthew Brett matthew.br...@gmail.com wrote: On Fri, Oct 28, 2011 at 3:49 PM, Charles R Harris charlesr.har...@gmail.com wrote: 2011/10/28 Stéfan van der Walt

Re: [Numpy-discussion] NA masks in the next numpy release?

2011-10-28 Thread Charles R Harris
2011/10/28 Stéfan van der Walt ste...@sun.ac.za On Fri, Oct 28, 2011 at 4:19 PM, Charles R Harris charlesr.har...@gmail.com wrote: Memory use is a known problem. One way to start addressing it might be to implement a bit arraytype. It might even be possible to prototype that on top of

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

2011-10-28 Thread Matthew Brett
Hi, On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers ralf.gomm...@googlemail.com wrote: On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Fri, Oct 28, 2011 at 3:56

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

2011-10-28 Thread Benjamin Root
On Friday, October 28, 2011, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers ralf.gomm...@googlemail.com wrote: On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Fri, Oct 28, 2011 at 3:14 PM, Charles R

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

2011-10-28 Thread Wes McKinney
On Fri, Oct 28, 2011 at 7:53 PM, Benjamin Root ben.r...@ou.edu wrote: On Friday, October 28, 2011, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers ralf.gomm...@googlemail.com wrote: On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

2011-10-28 Thread Matthew Brett
Hi, On Fri, Oct 28, 2011 at 4:53 PM, Benjamin Root ben.r...@ou.edu wrote: On Friday, October 28, 2011, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers ralf.gomm...@googlemail.com wrote: On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

2011-10-28 Thread Charles R Harris
On Fri, Oct 28, 2011 at 6:45 PM, Wes McKinney wesmck...@gmail.com wrote: On Fri, Oct 28, 2011 at 7:53 PM, Benjamin Root ben.r...@ou.edu wrote: On Friday, October 28, 2011, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

2011-10-28 Thread Nathaniel Smith
On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris charlesr.har...@gmail.com wrote: Matthew, the problem I have is that it seems that you and Nathaniel won't be satisfied unless things are done *your* way. Hi Charles, I'm sorry if I've given this impression, and I know it's easy to feel this

[Numpy-discussion] consensus (was: NA masks in the next numpy release?)

2011-10-28 Thread Benjamin Root
Matt, On Friday, October 28, 2011, Matthew Brett matthew.br...@gmail.com wrote: Forget about rudeness or decision processes. No, that's a common mistake, which is to assume that any conversation about things which aren't technical, is not important. Nathaniel's point is important.

Re: [Numpy-discussion] consensus

2011-10-28 Thread Jason Grout
On 10/28/11 10:38 PM, Benjamin Root wrote: I might like the idea of a NA to always mean absent (and destroys - even through views), and MA (or some other name) which always means ignore (and has the masking behavior with views). I should point out that if I'm dictating code to someone (e.g.,