Re: [Numpy-discussion] missing data discussion round 2

2011-06-29 Thread Nathaniel Smith
On Wed, Jun 29, 2011 at 2:40 PM, Lluís xscr...@gmx.net wrote: I'm for the option of having a single API when you want to have NA elements, regardless of whether it's using masks or bit patterns. I understand the desire to avoid having two different APIS... [snip] My concern is now about how

Re: [Numpy-discussion] missing data discussion round 2

2011-06-28 Thread Matthew Brett
Hi, On Mon, Jun 27, 2011 at 10:03 PM, Mark Wiebe mwwi...@gmail.com wrote: On Mon, Jun 27, 2011 at 12:18 PM, Matthew Brett matthew.br...@gmail.com ... That seems like a risky strategy to me, as the most likely outcome is that people worried about memory will avoid masked arrays because they

Re: [Numpy-discussion] missing data discussion round 2

2011-06-28 Thread Nathaniel Smith
On Mon, Jun 27, 2011 at 2:03 PM, Mark Wiebe mwwi...@gmail.com wrote: On Mon, Jun 27, 2011 at 12:18 PM, Matthew Brett matthew.br...@gmail.com wrote: You won't get complaints, you'll just lose a group of users, who will, I suspect, stick to NaNs, unsatisfactory as they are. This blade cuts

Re: [Numpy-discussion] missing data discussion round 2

2011-06-28 Thread Charles R Harris
On Tue, Jun 28, 2011 at 9:06 AM, Nathaniel Smith n...@pobox.com wrote: On Mon, Jun 27, 2011 at 2:03 PM, Mark Wiebe mwwi...@gmail.com wrote: On Mon, Jun 27, 2011 at 12:18 PM, Matthew Brett matthew.br...@gmail.com wrote: You won't get complaints, you'll just lose a group of users, who

Re: [Numpy-discussion] missing data discussion round 2

2011-06-28 Thread Nathaniel Smith
On Tue, Jun 28, 2011 at 9:38 AM, Charles R Harris charlesr.har...@gmail.com wrote: Nathaniel, an implementation using masks will look *exactly* like an implementation using na-dtypes from the user's point of view. Except that taking a masked view of an unmasked array allows ignoring values

Re: [Numpy-discussion] missing data discussion round 2

2011-06-28 Thread Eric Firing
On 06/28/2011 07:26 AM, Nathaniel Smith wrote: On Tue, Jun 28, 2011 at 9:38 AM, Charles R Harris charlesr.har...@gmail.com wrote: Nathaniel, an implementation using masks will look *exactly* like an implementation using na-dtypes from the user's point of view. Except that taking a masked

Re: [Numpy-discussion] missing data discussion round 2

2011-06-28 Thread Nathaniel Smith
On Tue, Jun 28, 2011 at 12:41 PM, Eric Firing efir...@hawaii.edu wrote: I think you are exaggerating some of the differences associated with the implementation, and ignoring one *key* difference: for integer types, the masked implementation can handle the full numeric range of the type, while

Re: [Numpy-discussion] missing data discussion round 2

2011-06-28 Thread Pierre GM
On Jun 28, 2011, at 9:41 PM, Eric Firing wrote: One of the real frustrations of the present masked array is that there is no savez/load support. I could roll my own by using a convention like saving the mask of xxx as xxx__mask__, and then reversing the process in a modified load; but I

Re: [Numpy-discussion] missing data discussion round 2

2011-06-28 Thread Pierre GM
All, I'm not sure I understand some aspects of Mark's new proposal, sorry (blame the lack of sleep). I'm pretty excited with the idea of built-in NA like np.dtype(NA['float64']), provided we can come with some shortcuts like np.nafloat64. I think that would really take care of the missing data

Re: [Numpy-discussion] missing data discussion round 2

2011-06-28 Thread Matthew Brett
Hi, On Tue, Jun 28, 2011 at 5:38 PM, Charles R Harris charlesr.har...@gmail.com wrote: Nathaniel, an implementation using masks will look *exactly* like an implementation using na-dtypes from the user's point of view. Except that taking a masked view of an unmasked array allows ignoring values

Re: [Numpy-discussion] missing data discussion round 2

2011-06-28 Thread Matthew Brett
Hi, On Tue, Jun 28, 2011 at 8:41 PM, Eric Firing efir...@hawaii.edu wrote: On 06/28/2011 07:26 AM, Nathaniel Smith wrote: On Tue, Jun 28, 2011 at 9:38 AM, Charles R Harris charlesr.har...@gmail.com  wrote: Nathaniel, an implementation using masks will look *exactly* like an implementation

Re: [Numpy-discussion] missing data discussion round 2

2011-06-28 Thread Matthew Brett
Hi, On Tue, Jun 28, 2011 at 4:06 PM, Nathaniel Smith n...@pobox.com wrote: ... (You might think, what difference does it make if you *can* unmask an item? Us missing data folks could just ignore this feature. But: whatever we end up implementing is something that I will have to explain over

Re: [Numpy-discussion] missing data discussion round 2

2011-06-28 Thread Jason Grout
On 6/28/11 5:20 PM, Matthew Brett wrote: Hi, On Tue, Jun 28, 2011 at 4:06 PM, Nathaniel Smithn...@pobox.com wrote: ... (You might think, what difference does it make if you *can* unmask an item? Us missing data folks could just ignore this feature. But: whatever we end up implementing is

Re: [Numpy-discussion] missing data discussion round 2

2011-06-28 Thread Matthew Brett
Hi, On Tue, Jun 28, 2011 at 11:40 PM, Jason Grout jason-s...@creativetrax.com wrote: On 6/28/11 5:20 PM, Matthew Brett wrote: Hi, On Tue, Jun 28, 2011 at 4:06 PM, Nathaniel Smithn...@pobox.com  wrote: ... (You might think, what difference does it make if you *can* unmask an item? Us

Re: [Numpy-discussion] missing data discussion round 2

2011-06-28 Thread eat
Hi, On Wed, Jun 29, 2011 at 1:40 AM, Jason Grout jason-s...@creativetrax.comwrote: On 6/28/11 5:20 PM, Matthew Brett wrote: Hi, On Tue, Jun 28, 2011 at 4:06 PM, Nathaniel Smithn...@pobox.com wrote: ... (You might think, what difference does it make if you *can* unmask an item? Us

Re: [Numpy-discussion] missing data discussion round 2

2011-06-28 Thread Mark Wiebe
On Tue, Jun 28, 2011 at 10:06 AM, Nathaniel Smith n...@pobox.com wrote: On Mon, Jun 27, 2011 at 2:03 PM, Mark Wiebe mwwi...@gmail.com wrote: On Mon, Jun 27, 2011 at 12:18 PM, Matthew Brett matthew.br...@gmail.com wrote: You won't get complaints, you'll just lose a group of users, who

Re: [Numpy-discussion] missing data discussion round 2

2011-06-28 Thread Mark Wiebe
On Tue, Jun 28, 2011 at 2:41 PM, Eric Firing efir...@hawaii.edu wrote: On 06/28/2011 07:26 AM, Nathaniel Smith wrote: On Tue, Jun 28, 2011 at 9:38 AM, Charles R Harris charlesr.har...@gmail.com wrote: Nathaniel, an implementation using masks will look *exactly* like an implementation

Re: [Numpy-discussion] missing data discussion round 2

2011-06-28 Thread Mark Wiebe
On Tue, Jun 28, 2011 at 3:45 PM, Pierre GM pgmdevl...@gmail.com wrote: All, I'm not sure I understand some aspects of Mark's new proposal, sorry (blame the lack of sleep). I'm pretty excited with the idea of built-in NA like np.dtype(NA['float64']), provided we can come with some shortcuts

Re: [Numpy-discussion] missing data discussion round 2

2011-06-28 Thread Mark Wiebe
On Tue, Jun 28, 2011 at 5:20 PM, Matthew Brett matthew.br...@gmail.comwrote: Hi, On Tue, Jun 28, 2011 at 4:06 PM, Nathaniel Smith n...@pobox.com wrote: ... (You might think, what difference does it make if you *can* unmask an item? Us missing data folks could just ignore this feature.

Re: [Numpy-discussion] missing data discussion round 2

2011-06-28 Thread Mark Wiebe
On Tue, Jun 28, 2011 at 6:00 PM, Matthew Brett matthew.br...@gmail.comwrote: Hi, On Tue, Jun 28, 2011 at 11:40 PM, Jason Grout jason-s...@creativetrax.com wrote: On 6/28/11 5:20 PM, Matthew Brett wrote: Hi, On Tue, Jun 28, 2011 at 4:06 PM, Nathaniel Smithn...@pobox.com wrote: ...

Re: [Numpy-discussion] missing data discussion round 2

2011-06-28 Thread Nathaniel Smith
On Tue, Jun 28, 2011 at 4:37 PM, Mark Wiebe mwwi...@gmail.com wrote: I've nearly finished this parameter, and decided to call it 'where' instead, because it is operating like an SQL where clause. Here if neither a nor b are masked array it will only modify those values of b where the 'where'

Re: [Numpy-discussion] missing data discussion round 2

2011-06-28 Thread Pierre GM
On Jun 29, 2011, at 1:37 AM, Mark Wiebe wrote: On Tue, Jun 28, 2011 at 3:45 PM, Pierre GM pgmdevl...@gmail.com wrote: ... I think that would really take care of the missing data part in a consistent and non-ambiguous way. However, I understand that if a choice would be made, this

Re: [Numpy-discussion] missing data discussion round 2

2011-06-28 Thread Pierre GM
On Jun 29, 2011, at 1:39 AM, Mark Wiebe wrote: On Tue, Jun 28, 2011 at 5:20 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Tue, Jun 28, 2011 at 4:06 PM, Nathaniel Smith n...@pobox.com wrote: ... (You might think, what difference does it make if you *can* unmask an item? Us

Re: [Numpy-discussion] missing data discussion round 2

2011-06-28 Thread Mark Wiebe
On Tue, Jun 28, 2011 at 6:55 PM, Nathaniel Smith n...@pobox.com wrote: On Tue, Jun 28, 2011 at 4:37 PM, Mark Wiebe mwwi...@gmail.com wrote: I've nearly finished this parameter, and decided to call it 'where' instead, because it is operating like an SQL where clause. Here if neither a nor b

Re: [Numpy-discussion] missing data discussion round 2

2011-06-28 Thread Mark Wiebe
On Tue, Jun 28, 2011 at 6:56 PM, Pierre GM pgmdevl...@gmail.com wrote: On Jun 29, 2011, at 1:37 AM, Mark Wiebe wrote: On Tue, Jun 28, 2011 at 3:45 PM, Pierre GM pgmdevl...@gmail.com wrote: ... I think that would really take care of the missing data part in a consistent and

Re: [Numpy-discussion] missing data discussion round 2

2011-06-28 Thread Mark Wiebe
On Tue, Jun 28, 2011 at 6:57 PM, Pierre GM pgmdevl...@gmail.com wrote: On Jun 29, 2011, at 1:39 AM, Mark Wiebe wrote: On Tue, Jun 28, 2011 at 5:20 PM, Matthew Brett matthew.br...@gmail.com wrote: Hi, On Tue, Jun 28, 2011 at 4:06 PM, Nathaniel Smith n...@pobox.com wrote: ... (You

[Numpy-discussion] missing data discussion round 2

2011-06-27 Thread Mark Wiebe
First I'd like to thank everyone for all the feedback you're providing, clearly this is an important topic to many people, and the discussion has helped clarify the ideas for me. I've renamed and updated the NEP, then placed it into the master NumPy repository so it has a more permanent home here:

Re: [Numpy-discussion] missing data discussion round 2

2011-06-27 Thread Charles R Harris
On Mon, Jun 27, 2011 at 9:55 AM, Mark Wiebe mwwi...@gmail.com wrote: First I'd like to thank everyone for all the feedback you're providing, clearly this is an important topic to many people, and the discussion has helped clarify the ideas for me. I've renamed and updated the NEP, then placed

Re: [Numpy-discussion] missing data discussion round 2

2011-06-27 Thread Matthew Brett
Hi, On Mon, Jun 27, 2011 at 5:53 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Mon, Jun 27, 2011 at 9:55 AM, Mark Wiebe mwwi...@gmail.com wrote: First I'd like to thank everyone for all the feedback you're providing, clearly this is an important topic to many people, and the

Re: [Numpy-discussion] missing data discussion round 2

2011-06-27 Thread eat
Hi, On Mon, Jun 27, 2011 at 6:55 PM, Mark Wiebe mwwi...@gmail.com wrote: First I'd like to thank everyone for all the feedback you're providing, clearly this is an important topic to many people, and the discussion has helped clarify the ideas for me. I've renamed and updated the NEP, then

Re: [Numpy-discussion] missing data discussion round 2

2011-06-27 Thread Mark Wiebe
On Mon, Jun 27, 2011 at 12:44 PM, eat e.antero.ta...@gmail.com wrote: Hi, On Mon, Jun 27, 2011 at 6:55 PM, Mark Wiebe mwwi...@gmail.com wrote: First I'd like to thank everyone for all the feedback you're providing, clearly this is an important topic to many people, and the discussion has

Re: [Numpy-discussion] missing data discussion round 2

2011-06-27 Thread eat
On Mon, Jun 27, 2011 at 8:53 PM, Mark Wiebe mwwi...@gmail.com wrote: On Mon, Jun 27, 2011 at 12:44 PM, eat e.antero.ta...@gmail.com wrote: Hi, On Mon, Jun 27, 2011 at 6:55 PM, Mark Wiebe mwwi...@gmail.com wrote: First I'd like to thank everyone for all the feedback you're providing,

Re: [Numpy-discussion] missing data discussion round 2

2011-06-27 Thread josef . pktd
On Mon, Jun 27, 2011 at 2:24 PM, eat e.antero.ta...@gmail.com wrote: On Mon, Jun 27, 2011 at 8:53 PM, Mark Wiebe mwwi...@gmail.com wrote: On Mon, Jun 27, 2011 at 12:44 PM, eat e.antero.ta...@gmail.com wrote: Hi, On Mon, Jun 27, 2011 at 6:55 PM, Mark Wiebe mwwi...@gmail.com wrote: First

Re: [Numpy-discussion] missing data discussion round 2

2011-06-27 Thread Mark Wiebe
On Mon, Jun 27, 2011 at 2:59 PM, josef.p...@gmail.com wrote: On Mon, Jun 27, 2011 at 2:24 PM, eat e.antero.ta...@gmail.com wrote: On Mon, Jun 27, 2011 at 8:53 PM, Mark Wiebe mwwi...@gmail.com wrote: On Mon, Jun 27, 2011 at 12:44 PM, eat e.antero.ta...@gmail.com wrote: Hi, On

Re: [Numpy-discussion] missing data discussion round 2

2011-06-27 Thread Mark Wiebe
On Mon, Jun 27, 2011 at 12:18 PM, Matthew Brett matthew.br...@gmail.comwrote: Hi, On Mon, Jun 27, 2011 at 5:53 PM, Charles R Harris charlesr.har...@gmail.com wrote: On Mon, Jun 27, 2011 at 9:55 AM, Mark Wiebe mwwi...@gmail.com wrote: First I'd like to thank everyone for all the

Re: [Numpy-discussion] missing data discussion round 2

2011-06-27 Thread josef . pktd
On Mon, Jun 27, 2011 at 5:01 PM, Mark Wiebe mwwi...@gmail.com wrote: On Mon, Jun 27, 2011 at 2:59 PM, josef.p...@gmail.com wrote: On Mon, Jun 27, 2011 at 2:24 PM, eat e.antero.ta...@gmail.com wrote: On Mon, Jun 27, 2011 at 8:53 PM, Mark Wiebe mwwi...@gmail.com wrote: On Mon, Jun 27,

Re: [Numpy-discussion] missing data discussion round 2

2011-06-27 Thread Pierre GM
On Jun 27, 2011, at 9:59 PM, josef.p...@gmail.com wrote: Just a question how things would work with the new model. How can you implement the use keyword from R's cov (or cor), with minimal data copying I think the basic masked array version would (or does) just assign 0 to the missing

Re: [Numpy-discussion] missing data discussion round 2

2011-06-27 Thread Mark Wiebe
On Mon, Jun 27, 2011 at 7:07 PM, Keith Goodman kwgood...@gmail.com wrote: On Mon, Jun 27, 2011 at 8:55 AM, Mark Wiebe mwwi...@gmail.com wrote: First I'd like to thank everyone for all the feedback you're providing, clearly this is an important topic to many people, and the discussion has

<    1   2