Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Matthew Brett]

Hi,

On Sun, Oct 30, 2011 at 12:24 PM, Ralf Gommers
 wrote:
>
>
> On Sat, Oct 29, 2011 at 11:55 PM, Matthew Brett 
> wrote:
>>
>> Hi,
>>
>> On Sat, Oct 29, 2011 at 2:48 PM, Ralf Gommers
>>  wrote:
>> >
>> >
>> > On Sat, Oct 29, 2011 at 11:36 PM, Matthew Brett
>> > 
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> On Sat, Oct 29, 2011 at 1:48 PM, Matthew Brett
>> >> 
>> >> wrote:
>> >> > Hi,
>> >> >
>> >> > On Sat, Oct 29, 2011 at 1:44 PM, Ralf Gommers
>> >> >  wrote:
>> >> >>
>> >> >>
>> >> >> On Sat, Oct 29, 2011 at 9:04 PM, Matthew Brett
>> >> >> 
>> >> >> wrote:
>> >> >>>
>> >> >>> Hi,
>> >> >>>
>> >> >>> On Sat, Oct 29, 2011 at 3:26 AM, Ralf Gommers
>> >> >>>  wrote:
>> >> >>> >
>> >> >>> >
>> >> >>> > On Sat, Oct 29, 2011 at 1:37 AM, Matthew Brett
>> >> >>> > 
>> >> >>> > wrote:
>> >> >>> >>
>> >> >>> >> Hi,
>> >> >>> >>
>> >> >>> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
>> >> >>> >>  wrote:
>> >> >>> >> >
>> >> >>> >> >
>> >> >>> >> > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett
>> >> >>> >> > 
>> >> >>> >> > wrote:
>> >> >>> >> >>
>> >> >>> >> >> Hi,
>> >> >>> >> >>
>> >> >>> >> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
>> >> >>> >> >>  wrote:
>> >> >>> >> >> >>
>> >> >>> >> >>
>> >> >>> >> >> No, that's not what Nathaniel and I are saying at all.
>> >> >>> >> >> Nathaniel
>> >> >>> >> >> was
>> >> >>> >> >> pointing to links for projects that care that everyone agrees
>> >> >>> >> >> before
>> >> >>> >> >> they go ahead.
>> >> >>> >> >
>> >> >>> >> > It looked to me like there was a serious intent to come to an
>> >> >>> >> > agreement,
>> >> >>> >> > or
>> >> >>> >> > at least closer together. The discussion in the summer was
>> >> >>> >> > going
>> >> >>> >> > around
>> >> >>> >> > in
>> >> >>> >> > circles though, and was too abstract and complex to follow.
>> >> >>> >> > Therefore
>> >> >>> >> > Mark's
>> >> >>> >> > choice of implementing something and then asking for feedback
>> >> >>> >> > made
>> >> >>> >> > sense
>> >> >>> >> > to
>> >> >>> >> > me.
>> >> >>> >>
>> >> >>> >> I should point out that the implementation hasn't - as far as I
>> >> >>> >> can
>> >> >>> >> see - changed the discussion.  The discussion was about the API.
>> >> >>> >>
>> >> >>> >> Implementations are useful for agreed APIs because they can
>> >> >>> >> point
>> >> >>> >> out
>> >> >>> >> where the API does not make sense or cannot be implemented.  In
>> >> >>> >> this
>> >> >>> >> case, the API Mark said he was going to implement - he did
>> >> >>> >> implement -
>> >> >>> >> at least as far as I can see.  Again, I'm happy to be corrected.
>> >> >>> >
>> >> >>> > Implementations can also help the discussion along, by allowing
>> >> >>> > people
>> >> >>> > to
>> >> >>> > try out some of the proposed changes. It also allows to construct
>> >> >>> > examples
>> >> >>> > that show weaknesses, possibly to be solved by an alternative
>> >> >>> > API.
>> >> >>> > Maybe
>> >> >>> > you
>> >> >>> > can hold the complete history of this topic in your head and
>> >> >>> > comprehend
>> >> >>> > it,
>> >> >>> > but for me it would be very helpful if someone said:
>> >> >>> > - here's my dataset
>> >> >>> > - this is what I want to do with it
>> >> >>> > - this is the best I can do with the current implementation
>> >> >>> > - here's how API X would allow me to solve this better or simpler
>> >> >>> > This can be done much better with actual data and an actual
>> >> >>> > implementation
>> >> >>> > than with a design proposal. You seem to disagree with this
>> >> >>> > statement.
>> >> >>> > That's fine. I would hope though that you recognize that concrete
>> >> >>> > examples
>> >> >>> > help people like me, and construct one or two to help us out.
>> >> >>> That's what use-cases are for in designing APIs.  There are
>> >> >>> examples
>> >> >>> of use in the NEP:
>> >> >>>
>> >> >>>
>> >> >>> https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst
>> >> >>>
>> >> >>> the alterNEP:
>> >> >>>
>> >> >>> https://gist.github.com/1056379
>> >> >>>
>> >> >>> and my longer email to Travis:
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> http://article.gmane.org/gmane.comp.python.numeric.general/46544/match=ignored
>> >> >>>
>> >> >>> Mark has done a nice job of documentation:
>> >> >>>
>> >> >>> http://docs.scipy.org/doc/numpy/reference/arrays.maskna.html
>> >> >>>
>> >> >>> If you want to understand what the alterNEP case is, I'd suggest
>> >> >>> the
>> >> >>> email, just because it's the most recent and I think the
>> >> >>> terminology
>> >> >>> is slightly clearer.
>> >> >>>
>> >> >>> Doing the same examples on a larger array won't make the point
>> >> >>> easier
>> >> >>> to understand.  The discussion is about what the right concepts
>> >> >>> are,
>> >> >>> and you can help by looking at the snippets of code in those
>> >> >>> documents, and deciding for yourself whether you think the current
>> >> >>> masking / NA implementation seems natural and easy to explain, or
>> >> >>> rather forced and d

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Ralf Gommers]

On Sat, Oct 29, 2011 at 11:55 PM, Matthew Brett wrote:

> Hi,
>
> On Sat, Oct 29, 2011 at 2:48 PM, Ralf Gommers
>  wrote:
> >
> >
> > On Sat, Oct 29, 2011 at 11:36 PM, Matthew Brett  >
> > wrote:
> >>
> >> Hi,
> >>
> >> On Sat, Oct 29, 2011 at 1:48 PM, Matthew Brett  >
> >> wrote:
> >> > Hi,
> >> >
> >> > On Sat, Oct 29, 2011 at 1:44 PM, Ralf Gommers
> >> >  wrote:
> >> >>
> >> >>
> >> >> On Sat, Oct 29, 2011 at 9:04 PM, Matthew Brett
> >> >> 
> >> >> wrote:
> >> >>>
> >> >>> Hi,
> >> >>>
> >> >>> On Sat, Oct 29, 2011 at 3:26 AM, Ralf Gommers
> >> >>>  wrote:
> >> >>> >
> >> >>> >
> >> >>> > On Sat, Oct 29, 2011 at 1:37 AM, Matthew Brett
> >> >>> > 
> >> >>> > wrote:
> >> >>> >>
> >> >>> >> Hi,
> >> >>> >>
> >> >>> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
> >> >>> >>  wrote:
> >> >>> >> >
> >> >>> >> >
> >> >>> >> > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett
> >> >>> >> > 
> >> >>> >> > wrote:
> >> >>> >> >>
> >> >>> >> >> Hi,
> >> >>> >> >>
> >> >>> >> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
> >> >>> >> >>  wrote:
> >> >>> >> >> >>
> >> >>> >> >>
> >> >>> >> >> No, that's not what Nathaniel and I are saying at all.
> Nathaniel
> >> >>> >> >> was
> >> >>> >> >> pointing to links for projects that care that everyone agrees
> >> >>> >> >> before
> >> >>> >> >> they go ahead.
> >> >>> >> >
> >> >>> >> > It looked to me like there was a serious intent to come to an
> >> >>> >> > agreement,
> >> >>> >> > or
> >> >>> >> > at least closer together. The discussion in the summer was
> going
> >> >>> >> > around
> >> >>> >> > in
> >> >>> >> > circles though, and was too abstract and complex to follow.
> >> >>> >> > Therefore
> >> >>> >> > Mark's
> >> >>> >> > choice of implementing something and then asking for feedback
> >> >>> >> > made
> >> >>> >> > sense
> >> >>> >> > to
> >> >>> >> > me.
> >> >>> >>
> >> >>> >> I should point out that the implementation hasn't - as far as I
> can
> >> >>> >> see - changed the discussion.  The discussion was about the API.
> >> >>> >>
> >> >>> >> Implementations are useful for agreed APIs because they can point
> >> >>> >> out
> >> >>> >> where the API does not make sense or cannot be implemented.  In
> >> >>> >> this
> >> >>> >> case, the API Mark said he was going to implement - he did
> >> >>> >> implement -
> >> >>> >> at least as far as I can see.  Again, I'm happy to be corrected.
> >> >>> >
> >> >>> > Implementations can also help the discussion along, by allowing
> >> >>> > people
> >> >>> > to
> >> >>> > try out some of the proposed changes. It also allows to construct
> >> >>> > examples
> >> >>> > that show weaknesses, possibly to be solved by an alternative API.
> >> >>> > Maybe
> >> >>> > you
> >> >>> > can hold the complete history of this topic in your head and
> >> >>> > comprehend
> >> >>> > it,
> >> >>> > but for me it would be very helpful if someone said:
> >> >>> > - here's my dataset
> >> >>> > - this is what I want to do with it
> >> >>> > - this is the best I can do with the current implementation
> >> >>> > - here's how API X would allow me to solve this better or simpler
> >> >>> > This can be done much better with actual data and an actual
> >> >>> > implementation
> >> >>> > than with a design proposal. You seem to disagree with this
> >> >>> > statement.
> >> >>> > That's fine. I would hope though that you recognize that concrete
> >> >>> > examples
> >> >>> > help people like me, and construct one or two to help us out.
> >> >>> That's what use-cases are for in designing APIs.  There are examples
> >> >>> of use in the NEP:
> >> >>>
> >> >>>
> https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst
> >> >>>
> >> >>> the alterNEP:
> >> >>>
> >> >>> https://gist.github.com/1056379
> >> >>>
> >> >>> and my longer email to Travis:
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> http://article.gmane.org/gmane.comp.python.numeric.general/46544/match=ignored
> >> >>>
> >> >>> Mark has done a nice job of documentation:
> >> >>>
> >> >>> http://docs.scipy.org/doc/numpy/reference/arrays.maskna.html
> >> >>>
> >> >>> If you want to understand what the alterNEP case is, I'd suggest the
> >> >>> email, just because it's the most recent and I think the terminology
> >> >>> is slightly clearer.
> >> >>>
> >> >>> Doing the same examples on a larger array won't make the point
> easier
> >> >>> to understand.  The discussion is about what the right concepts are,
> >> >>> and you can help by looking at the snippets of code in those
> >> >>> documents, and deciding for yourself whether you think the current
> >> >>> masking / NA implementation seems natural and easy to explain, or
> >> >>> rather forced and difficult to explain, and then email back trying
> to
> >> >>> explain your impression (which is not always easy).
> >> >>
> >> >> If you seriously believe that looking at a few snippets is as helpful
> >> >> and
> >> >> instructive as being able to play around with them in IPython and
> >> >> modify
> >> >> them, then I guess we won't m

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Matthew Brett]

Hi,

On Sat, Oct 29, 2011 at 11:19 PM, Travis Oliphant
 wrote:

Thanks again for your email, I'm sure I'm not the only one who
breathes a deep sigh of relief when I see your posts.

> I appreciate Nathaniel's idea to pull the changes and I can respect his 
> desire to do that.   It seemed like there was a lot more heat than light in 
> the discussion this summer.   The differences seemed to be enflamed by the 
> discussion instead of illuminated by it.  Perhaps, that is why Nathaniel felt 
> like merging Mark's pull request was too strong-armed and not a proper 
> resolution.
>
> However, I did not interpret Matthew or Nathaniel's explanations of their 
> position as manipulative or inappropriate.  Nonetheless, I don't think 
> removing Mark's changes are a productive direction to take at this point.   I 
> agree, it would have been much better to reach a rough consensus before the 
> code was committed.  At least, those who felt like their ideas where not 
> accounted for should have felt like there was some plan to either accommodate 
> them, or some explanation of why that was not a good idea.  The only thing I 
> recall being said was that there was nobody to implement their ideas.   I 
> wish that weren't the case.   I think we can still continue to discuss their 
> concerns and look for ways to reasonably incorporate their use-cases if 
> possible.
>
> I have probably contributed in the past to the idea that "he who writes the 
> code gets the final say".    In early-stage efforts, this is approximately 
> right, but success of anything relies on satisfied users and as projects 
> mature the voice of users becomes more relevant than the voice of 
> contributors in my mind.   I've certainly had to learn that in terms of ABI 
> changes to NumPy.

I think that's right though - that the person who wrote the code has
the final say.  But that's the final say.   The question I wanted to
ask was the one Nathaniel brought up at the beginning of the thread,
which is, before the final say, how hard do we try for consensus?  Is
that - the numpy way?   Here Chuck was saying 'I listen to you in
proportion to your code contribution' (I hope I'm not misrepresenting
him).   I think that's different way of working than the consensus
building that Karl Fogel describes.  But maybe that is just the numpy
way.  I would feel happier to know what that way is.   Then, when we
get into this kind of dispute Chuck can say 'Matthew, change the numpy
constitution or accept the situation because that's how we've agreed
to work'.   And I'll say - 'OK - I don't like it, but I agree those
are the rules'.  And we'll get on with it.  But at the moment, it
feels as if it isn't clear, and, as Ben pointed out, that means we are
having a discussion and a discussion about the discussion at the same
time.

See you,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Matthew Brett]

Hi,

On Sat, Oct 29, 2011 at 10:02 PM, Travis Oliphant
 wrote:
>>>
>>> Here are my needs:
>>>
>>> 1) How NAs are implemented cannot be end user visible. Having to pass
>>> maskna=True is a problem. I suppose a solution is to set the flag to
>>> true on every array inside of pandas so the user never knows (you
>>> mentioned someone else had some other solution, i could go back and
>>> dig it up?)
>>
>> I guess this would be the same with bitpatterns, in that the user
>> would have to specify a custom dtype.
>>
>> Is it possible to add a bitpattern NA (in the NaN values) to the
>> current floating point types, at least in principle?  So that np.float
>> etc would have bitpattern NAs without a custom dtype?
>
> That is an interesting idea.   It's essentially what people like Wes McKinney 
> are doing now.    However, the issue is going to be whether or not you do 
> something special or not with the NA values in the low-level C function the 
> dtype dispatches to.  This is the reason for the special bit-pattern dtype.
>
> I've always thought that requiring NA checks for code that doesn't want to 
> worry about it would slow things down un-necessarily for those use-cases.

Right - now that the caffeine has run through my system adequately, I
have a few glasses of wine to disrupt my logic and / or social skills
but:

Is there any way you could imagine something like this?:

In [3]: a = np.arange(10, dtype=np.float)

In [4]: a.flags
Out[4]:
  C_CONTIGUOUS : True
  F_CONTIGUOUS : True
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False
  MAYBE_NA : False

In [5]: a[0] = np.NA

In [6]: a.flags
Out[6]:
  C_CONTIGUOUS : True
  F_CONTIGUOUS : True
  OWNDATA : True
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False
  MAYBE_NA : True

Obviously extension writers would have to keep the flag maintained...

Sorry if that doesn't make sense, I do not claim to be in full
possession of my faculties,

See you,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Travis Oliphant]

On Oct 29, 2011, at 7:24 PM, Eric Firing wrote:

> On 10/29/2011 12:57 PM, Charles R Harris wrote:
>> 
>> 
>> On Sat, Oct 29, 2011 at 4:47 PM, Eric Firing > > wrote:
>> 
>>On 10/29/2011 12:02 PM, Olivier Delalleau wrote:
>> 
>>> 
>>> I haven't been following the discussion closely, but wouldn't it
>>be instead:
>>> a.mask[0:2] = True?
>> 
>>That would be consistent with numpy.ma  and the
>>opposite of Mark's
>>implementation.
>> 
>>I can live with either, but I much prefer the numpy.ma
>> version because
>>it fits with the use of bit-flags for editing data; set bit 1 if it
>>fails check A, set bit 2 if it fails check B, etc.  So, if it evaluates
>>as True, there is a problem, and the value is masked *out*.
>> 
>>Similarly, in Marks implementation, 7 bits are available for a payload
>>to describe what kind of masking is meant.  This seems more consistent
>>with True as masked (or NA) than with False as masked.
>> 
>> 
>> I wouldn't rely on the 7 bits yet. Mark left them available to keep open
>> possible future use, but didn't implement anything using them yet. If
>> memory use turns out to exclude whole sectors of application we will
>> have to go to bit masks.
> 
> Right; I was only commenting on a subjective sense of internal 
> consistency.  A minor point.
> 
> The larger context of all this is how users end up being able to work 
> with all the different types and specifications of "NA" (in the most 
> general sense) data:
> 
> 1) nans
> 2) numpy.ma
> 3) masks in the core (Mark's new code)
> 4) bit patterns
> 
> Substantial code now in place--including matplotlib--relies on numpy.ma. 
>  It has some rough edges, it can be slow, it is a pain having it as a 
> bolted-on module, it may be more complicated than it needs to be, but it 
> fits a lot of use cases pretty well.  There are many users.  Everyone 
> using matplotlib is using it, whether they know it or not.
> 
> The ideal from my numpy.ma-user's standpoint would an NA-handling 
> implementation in the core that would do two things:
> (1) allow a gradual transition away from numpy.ma, so that the latter 
> would become redundant.
> (2) allow numpy.ma to be reasonably easily modified to use the in-core 
> facilities for greater efficiency during the long transition.  Implicit 
> is the hope that someone (most likely not me, although I might be able 
> to help a bit) would actually perform this modification.
> 
> Mark's mission, paid for by Enthought, was not to please numpy.ma users, 
> but to add NA-handling that would be comfortable for R-users.  He chose 
> to do so with the idea that two possible implementations (masks and 
> bitpatterns) were desirable, each with strengths and weaknesses, and 
> that so as to get *something* done in the very short time he had left, 
> he would start with the mask implementation.  We now have the result, 
> incomplete, but not breaking anything.  Additional development (coding 
> as well as designing) will be needed.
> 
> The main question raised by Matthew and Nathaniel is, I think, whether 
> Mark's code should develop in a direction away from the R-compatibility 
> model, with the idea that the latter would be handled via a bit-pattern 
> implementation, some day, when someone codes it; or whether it should 
> remain as the prototype and first implementation of an API to handle the 
> R-compatible use case, minimizing any divergence from any eventual 
> bit-pattern implementation.
> 
> The answer to this depends on several questions, including:
> 
> 1) Who is available to do how much implementation of any of the 
> possibilities?  My reading of Travis's blog and rare posts to this list 
> suggest that he hopes and expects to be able to free up coding time. 
> Perhaps he will clarify that soon.
> 
> 2) What sorts of changes would actually be needed to make the present 
> implementation good enough for the R use case?  Evolutionary, or 
> revolutionary?
> 
> 3) What sorts of changes would help with the numpy.ma use case? 
> Evolutionary, or revolutionary.
> 
> 4) Given available resources, how can we maximize progress: making numpy 
> more capable, easier to use, etc.
> 
> Unless the answers to questions 2 *and* 3 are "revolutionary", I don't 
> see the point in pulling Mark's changes out of master.  At most, the 
> documentation might be changed to mark the NA API as "experimental" for 
> a release or two.

I appreciate Nathaniel's idea to pull the changes and I can respect his desire 
to do that.   It seemed like there was a lot more heat than light in the 
discussion this summer.   The differences seemed to be enflamed by the 
discussion instead of illuminated by it.  Perhaps, that is why Nathaniel felt 
like merging Mark's pull request was too strong-armed and not a proper 
resolution.   

However, I did not interpret Matthew or Nathaniel's explanations of their 
position as manipulative or inap

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Travis Oliphant]

>> 
>> Here are my needs:
>> 
>> 1) How NAs are implemented cannot be end user visible. Having to pass
>> maskna=True is a problem. I suppose a solution is to set the flag to
>> true on every array inside of pandas so the user never knows (you
>> mentioned someone else had some other solution, i could go back and
>> dig it up?)
> 
> I guess this would be the same with bitpatterns, in that the user
> would have to specify a custom dtype.
> 
> Is it possible to add a bitpattern NA (in the NaN values) to the
> current floating point types, at least in principle?  So that np.float
> etc would have bitpattern NAs without a custom dtype?

That is an interesting idea.   It's essentially what people like Wes McKinney 
are doing now.However, the issue is going to be whether or not you do 
something special or not with the NA values in the low-level C function the 
dtype dispatches to.  This is the reason for the special bit-pattern dtype.

I've always thought that requiring NA checks for code that doesn't want to 
worry about it would slow things down un-necessarily for those use-cases.

But, not dealing with missing data well is a missing NumPy feature. 

-Travis


> See you,
> 
> Matthew
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

---
Travis Oliphant
Enthought, Inc.
oliph...@enthought.com
1-512-536-1057
http://www.enthought.com



___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Matthew Brett]

Hi,

On Sat, Oct 29, 2011 at 7:48 PM, Charles R Harris
 wrote:
>
>
> On Sat, Oct 29, 2011 at 7:47 PM, Matthew Brett 
> wrote:
>>
>> Hi,
>>
>> On Sat, Oct 29, 2011 at 4:11 PM, Matthew Brett 
>> wrote:
>> > Hi,
>> >
>> > On Sat, Oct 29, 2011 at 2:59 PM, Charles R Harris
>> >  wrote:
>> >>
>> >>
>> >> On Sat, Oct 29, 2011 at 3:55 PM, Matthew Brett
>> >> 
>> >> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> On Sat, Oct 29, 2011 at 2:48 PM, Ralf Gommers
>> >>>  wrote:
>> >>> >
>> >>> >
>> >>> > On Sat, Oct 29, 2011 at 11:36 PM, Matthew Brett
>> >>> > 
>> >>> > wrote:
>> >>> >>
>> >>> >> Hi,
>> >>> >>
>> >>> >> On Sat, Oct 29, 2011 at 1:48 PM, Matthew Brett
>> >>> >> 
>> >>> >> wrote:
>> >>> >> > Hi,
>> >>> >> >
>> >>> >> > On Sat, Oct 29, 2011 at 1:44 PM, Ralf Gommers
>> >>> >> >  wrote:
>> >>> >> >>
>> >>> >> >>
>> >>> >> >> On Sat, Oct 29, 2011 at 9:04 PM, Matthew Brett
>> >>> >> >> 
>> >>> >> >> wrote:
>> >>> >> >>>
>> >>> >> >>> Hi,
>> >>> >> >>>
>> >>> >> >>> On Sat, Oct 29, 2011 at 3:26 AM, Ralf Gommers
>> >>> >> >>>  wrote:
>> >>> >> >>> >
>> >>> >> >>> >
>> >>> >> >>> > On Sat, Oct 29, 2011 at 1:37 AM, Matthew Brett
>> >>> >> >>> > 
>> >>> >> >>> > wrote:
>> >>> >> >>> >>
>> >>> >> >>> >> Hi,
>> >>> >> >>> >>
>> >>> >> >>> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
>> >>> >> >>> >>  wrote:
>> >>> >> >>> >> >
>> >>> >> >>> >> >
>> >>> >> >>> >> > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett
>> >>> >> >>> >> > 
>> >>> >> >>> >> > wrote:
>> >>> >> >>> >> >>
>> >>> >> >>> >> >> Hi,
>> >>> >> >>> >> >>
>> >>> >> >>> >> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
>> >>> >> >>> >> >>  wrote:
>> >>> >> >>> >> >> >>
>> >>> >> >>> >> >>
>> >>> >> >>> >> >> No, that's not what Nathaniel and I are saying at all.
>> >>> >> >>> >> >> Nathaniel
>> >>> >> >>> >> >> was
>> >>> >> >>> >> >> pointing to links for projects that care that everyone
>> >>> >> >>> >> >> agrees
>> >>> >> >>> >> >> before
>> >>> >> >>> >> >> they go ahead.
>> >>> >> >>> >> >
>> >>> >> >>> >> > It looked to me like there was a serious intent to come to
>> >>> >> >>> >> > an
>> >>> >> >>> >> > agreement,
>> >>> >> >>> >> > or
>> >>> >> >>> >> > at least closer together. The discussion in the summer was
>> >>> >> >>> >> > going
>> >>> >> >>> >> > around
>> >>> >> >>> >> > in
>> >>> >> >>> >> > circles though, and was too abstract and complex to
>> >>> >> >>> >> > follow.
>> >>> >> >>> >> > Therefore
>> >>> >> >>> >> > Mark's
>> >>> >> >>> >> > choice of implementing something and then asking for
>> >>> >> >>> >> > feedback
>> >>> >> >>> >> > made
>> >>> >> >>> >> > sense
>> >>> >> >>> >> > to
>> >>> >> >>> >> > me.
>> >>> >> >>> >>
>> >>> >> >>> >> I should point out that the implementation hasn't - as far
>> >>> >> >>> >> as I
>> >>> >> >>> >> can
>> >>> >> >>> >> see - changed the discussion.  The discussion was about the
>> >>> >> >>> >> API.
>> >>> >> >>> >>
>> >>> >> >>> >> Implementations are useful for agreed APIs because they can
>> >>> >> >>> >> point
>> >>> >> >>> >> out
>> >>> >> >>> >> where the API does not make sense or cannot be implemented.
>> >>> >> >>> >>  In
>> >>> >> >>> >> this
>> >>> >> >>> >> case, the API Mark said he was going to implement - he did
>> >>> >> >>> >> implement -
>> >>> >> >>> >> at least as far as I can see.  Again, I'm happy to be
>> >>> >> >>> >> corrected.
>> >>> >> >>> >
>> >>> >> >>> > Implementations can also help the discussion along, by
>> >>> >> >>> > allowing
>> >>> >> >>> > people
>> >>> >> >>> > to
>> >>> >> >>> > try out some of the proposed changes. It also allows to
>> >>> >> >>> > construct
>> >>> >> >>> > examples
>> >>> >> >>> > that show weaknesses, possibly to be solved by an alternative
>> >>> >> >>> > API.
>> >>> >> >>> > Maybe
>> >>> >> >>> > you
>> >>> >> >>> > can hold the complete history of this topic in your head and
>> >>> >> >>> > comprehend
>> >>> >> >>> > it,
>> >>> >> >>> > but for me it would be very helpful if someone said:
>> >>> >> >>> > - here's my dataset
>> >>> >> >>> > - this is what I want to do with it
>> >>> >> >>> > - this is the best I can do with the current implementation
>> >>> >> >>> > - here's how API X would allow me to solve this better or
>> >>> >> >>> > simpler
>> >>> >> >>> > This can be done much better with actual data and an actual
>> >>> >> >>> > implementation
>> >>> >> >>> > than with a design proposal. You seem to disagree with this
>> >>> >> >>> > statement.
>> >>> >> >>> > That's fine. I would hope though that you recognize that
>> >>> >> >>> > concrete
>> >>> >> >>> > examples
>> >>> >> >>> > help people like me, and construct one or two to help us out.
>> >>> >> >>> That's what use-cases are for in designing APIs.  There are
>> >>> >> >>> examples
>> >>> >> >>> of use in the NEP:
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>>
>> >>> >> >>> https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst
>> >>> >> >>>
>> >>> >> >>> the alterNEP:
>> >>> >> >>>
>> >>> >> >>> https://gist.github.com/1056379
>> >>> >> >>>
>> >>> >> >>> and

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Charles R Harris]

On Sat, Oct 29, 2011 at 7:47 PM, Matthew Brett wrote:

> Hi,
>
> On Sat, Oct 29, 2011 at 4:11 PM, Matthew Brett 
> wrote:
> > Hi,
> >
> > On Sat, Oct 29, 2011 at 2:59 PM, Charles R Harris
> >  wrote:
> >>
> >>
> >> On Sat, Oct 29, 2011 at 3:55 PM, Matthew Brett  >
> >> wrote:
> >>>
> >>> Hi,
> >>>
> >>> On Sat, Oct 29, 2011 at 2:48 PM, Ralf Gommers
> >>>  wrote:
> >>> >
> >>> >
> >>> > On Sat, Oct 29, 2011 at 11:36 PM, Matthew Brett
> >>> > 
> >>> > wrote:
> >>> >>
> >>> >> Hi,
> >>> >>
> >>> >> On Sat, Oct 29, 2011 at 1:48 PM, Matthew Brett
> >>> >> 
> >>> >> wrote:
> >>> >> > Hi,
> >>> >> >
> >>> >> > On Sat, Oct 29, 2011 at 1:44 PM, Ralf Gommers
> >>> >> >  wrote:
> >>> >> >>
> >>> >> >>
> >>> >> >> On Sat, Oct 29, 2011 at 9:04 PM, Matthew Brett
> >>> >> >> 
> >>> >> >> wrote:
> >>> >> >>>
> >>> >> >>> Hi,
> >>> >> >>>
> >>> >> >>> On Sat, Oct 29, 2011 at 3:26 AM, Ralf Gommers
> >>> >> >>>  wrote:
> >>> >> >>> >
> >>> >> >>> >
> >>> >> >>> > On Sat, Oct 29, 2011 at 1:37 AM, Matthew Brett
> >>> >> >>> > 
> >>> >> >>> > wrote:
> >>> >> >>> >>
> >>> >> >>> >> Hi,
> >>> >> >>> >>
> >>> >> >>> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
> >>> >> >>> >>  wrote:
> >>> >> >>> >> >
> >>> >> >>> >> >
> >>> >> >>> >> > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett
> >>> >> >>> >> > 
> >>> >> >>> >> > wrote:
> >>> >> >>> >> >>
> >>> >> >>> >> >> Hi,
> >>> >> >>> >> >>
> >>> >> >>> >> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
> >>> >> >>> >> >>  wrote:
> >>> >> >>> >> >> >>
> >>> >> >>> >> >>
> >>> >> >>> >> >> No, that's not what Nathaniel and I are saying at all.
> >>> >> >>> >> >> Nathaniel
> >>> >> >>> >> >> was
> >>> >> >>> >> >> pointing to links for projects that care that everyone
> agrees
> >>> >> >>> >> >> before
> >>> >> >>> >> >> they go ahead.
> >>> >> >>> >> >
> >>> >> >>> >> > It looked to me like there was a serious intent to come to
> an
> >>> >> >>> >> > agreement,
> >>> >> >>> >> > or
> >>> >> >>> >> > at least closer together. The discussion in the summer was
> >>> >> >>> >> > going
> >>> >> >>> >> > around
> >>> >> >>> >> > in
> >>> >> >>> >> > circles though, and was too abstract and complex to follow.
> >>> >> >>> >> > Therefore
> >>> >> >>> >> > Mark's
> >>> >> >>> >> > choice of implementing something and then asking for
> feedback
> >>> >> >>> >> > made
> >>> >> >>> >> > sense
> >>> >> >>> >> > to
> >>> >> >>> >> > me.
> >>> >> >>> >>
> >>> >> >>> >> I should point out that the implementation hasn't - as far as
> I
> >>> >> >>> >> can
> >>> >> >>> >> see - changed the discussion.  The discussion was about the
> API.
> >>> >> >>> >>
> >>> >> >>> >> Implementations are useful for agreed APIs because they can
> >>> >> >>> >> point
> >>> >> >>> >> out
> >>> >> >>> >> where the API does not make sense or cannot be implemented.
>  In
> >>> >> >>> >> this
> >>> >> >>> >> case, the API Mark said he was going to implement - he did
> >>> >> >>> >> implement -
> >>> >> >>> >> at least as far as I can see.  Again, I'm happy to be
> corrected.
> >>> >> >>> >
> >>> >> >>> > Implementations can also help the discussion along, by
> allowing
> >>> >> >>> > people
> >>> >> >>> > to
> >>> >> >>> > try out some of the proposed changes. It also allows to
> construct
> >>> >> >>> > examples
> >>> >> >>> > that show weaknesses, possibly to be solved by an alternative
> >>> >> >>> > API.
> >>> >> >>> > Maybe
> >>> >> >>> > you
> >>> >> >>> > can hold the complete history of this topic in your head and
> >>> >> >>> > comprehend
> >>> >> >>> > it,
> >>> >> >>> > but for me it would be very helpful if someone said:
> >>> >> >>> > - here's my dataset
> >>> >> >>> > - this is what I want to do with it
> >>> >> >>> > - this is the best I can do with the current implementation
> >>> >> >>> > - here's how API X would allow me to solve this better or
> simpler
> >>> >> >>> > This can be done much better with actual data and an actual
> >>> >> >>> > implementation
> >>> >> >>> > than with a design proposal. You seem to disagree with this
> >>> >> >>> > statement.
> >>> >> >>> > That's fine. I would hope though that you recognize that
> concrete
> >>> >> >>> > examples
> >>> >> >>> > help people like me, and construct one or two to help us out.
> >>> >> >>> That's what use-cases are for in designing APIs.  There are
> >>> >> >>> examples
> >>> >> >>> of use in the NEP:
> >>> >> >>>
> >>> >> >>>
> >>> >> >>>
> https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst
> >>> >> >>>
> >>> >> >>> the alterNEP:
> >>> >> >>>
> >>> >> >>> https://gist.github.com/1056379
> >>> >> >>>
> >>> >> >>> and my longer email to Travis:
> >>> >> >>>
> >>> >> >>>
> >>> >> >>>
> >>> >> >>>
> >>> >> >>>
> http://article.gmane.org/gmane.comp.python.numeric.general/46544/match=ignored
> >>> >> >>>
> >>> >> >>> Mark has done a nice job of documentation:
> >>> >> >>>
> >>> >> >>> http://docs.scipy.org/doc/numpy/reference/arrays.maskna.html
> >>> >> >>>
> >>> >> >>> If you want to understand what the alterNEP case is, I'd 

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Matthew Brett]

Hi,

On Sat, Oct 29, 2011 at 4:11 PM, Matthew Brett  wrote:
> Hi,
>
> On Sat, Oct 29, 2011 at 2:59 PM, Charles R Harris
>  wrote:
>>
>>
>> On Sat, Oct 29, 2011 at 3:55 PM, Matthew Brett 
>> wrote:
>>>
>>> Hi,
>>>
>>> On Sat, Oct 29, 2011 at 2:48 PM, Ralf Gommers
>>>  wrote:
>>> >
>>> >
>>> > On Sat, Oct 29, 2011 at 11:36 PM, Matthew Brett
>>> > 
>>> > wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> On Sat, Oct 29, 2011 at 1:48 PM, Matthew Brett
>>> >> 
>>> >> wrote:
>>> >> > Hi,
>>> >> >
>>> >> > On Sat, Oct 29, 2011 at 1:44 PM, Ralf Gommers
>>> >> >  wrote:
>>> >> >>
>>> >> >>
>>> >> >> On Sat, Oct 29, 2011 at 9:04 PM, Matthew Brett
>>> >> >> 
>>> >> >> wrote:
>>> >> >>>
>>> >> >>> Hi,
>>> >> >>>
>>> >> >>> On Sat, Oct 29, 2011 at 3:26 AM, Ralf Gommers
>>> >> >>>  wrote:
>>> >> >>> >
>>> >> >>> >
>>> >> >>> > On Sat, Oct 29, 2011 at 1:37 AM, Matthew Brett
>>> >> >>> > 
>>> >> >>> > wrote:
>>> >> >>> >>
>>> >> >>> >> Hi,
>>> >> >>> >>
>>> >> >>> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
>>> >> >>> >>  wrote:
>>> >> >>> >> >
>>> >> >>> >> >
>>> >> >>> >> > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett
>>> >> >>> >> > 
>>> >> >>> >> > wrote:
>>> >> >>> >> >>
>>> >> >>> >> >> Hi,
>>> >> >>> >> >>
>>> >> >>> >> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
>>> >> >>> >> >>  wrote:
>>> >> >>> >> >> >>
>>> >> >>> >> >>
>>> >> >>> >> >> No, that's not what Nathaniel and I are saying at all.
>>> >> >>> >> >> Nathaniel
>>> >> >>> >> >> was
>>> >> >>> >> >> pointing to links for projects that care that everyone agrees
>>> >> >>> >> >> before
>>> >> >>> >> >> they go ahead.
>>> >> >>> >> >
>>> >> >>> >> > It looked to me like there was a serious intent to come to an
>>> >> >>> >> > agreement,
>>> >> >>> >> > or
>>> >> >>> >> > at least closer together. The discussion in the summer was
>>> >> >>> >> > going
>>> >> >>> >> > around
>>> >> >>> >> > in
>>> >> >>> >> > circles though, and was too abstract and complex to follow.
>>> >> >>> >> > Therefore
>>> >> >>> >> > Mark's
>>> >> >>> >> > choice of implementing something and then asking for feedback
>>> >> >>> >> > made
>>> >> >>> >> > sense
>>> >> >>> >> > to
>>> >> >>> >> > me.
>>> >> >>> >>
>>> >> >>> >> I should point out that the implementation hasn't - as far as I
>>> >> >>> >> can
>>> >> >>> >> see - changed the discussion.  The discussion was about the API.
>>> >> >>> >>
>>> >> >>> >> Implementations are useful for agreed APIs because they can
>>> >> >>> >> point
>>> >> >>> >> out
>>> >> >>> >> where the API does not make sense or cannot be implemented.  In
>>> >> >>> >> this
>>> >> >>> >> case, the API Mark said he was going to implement - he did
>>> >> >>> >> implement -
>>> >> >>> >> at least as far as I can see.  Again, I'm happy to be corrected.
>>> >> >>> >
>>> >> >>> > Implementations can also help the discussion along, by allowing
>>> >> >>> > people
>>> >> >>> > to
>>> >> >>> > try out some of the proposed changes. It also allows to construct
>>> >> >>> > examples
>>> >> >>> > that show weaknesses, possibly to be solved by an alternative
>>> >> >>> > API.
>>> >> >>> > Maybe
>>> >> >>> > you
>>> >> >>> > can hold the complete history of this topic in your head and
>>> >> >>> > comprehend
>>> >> >>> > it,
>>> >> >>> > but for me it would be very helpful if someone said:
>>> >> >>> > - here's my dataset
>>> >> >>> > - this is what I want to do with it
>>> >> >>> > - this is the best I can do with the current implementation
>>> >> >>> > - here's how API X would allow me to solve this better or simpler
>>> >> >>> > This can be done much better with actual data and an actual
>>> >> >>> > implementation
>>> >> >>> > than with a design proposal. You seem to disagree with this
>>> >> >>> > statement.
>>> >> >>> > That's fine. I would hope though that you recognize that concrete
>>> >> >>> > examples
>>> >> >>> > help people like me, and construct one or two to help us out.
>>> >> >>> That's what use-cases are for in designing APIs.  There are
>>> >> >>> examples
>>> >> >>> of use in the NEP:
>>> >> >>>
>>> >> >>>
>>> >> >>> https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst
>>> >> >>>
>>> >> >>> the alterNEP:
>>> >> >>>
>>> >> >>> https://gist.github.com/1056379
>>> >> >>>
>>> >> >>> and my longer email to Travis:
>>> >> >>>
>>> >> >>>
>>> >> >>>
>>> >> >>>
>>> >> >>> http://article.gmane.org/gmane.comp.python.numeric.general/46544/match=ignored
>>> >> >>>
>>> >> >>> Mark has done a nice job of documentation:
>>> >> >>>
>>> >> >>> http://docs.scipy.org/doc/numpy/reference/arrays.maskna.html
>>> >> >>>
>>> >> >>> If you want to understand what the alterNEP case is, I'd suggest
>>> >> >>> the
>>> >> >>> email, just because it's the most recent and I think the
>>> >> >>> terminology
>>> >> >>> is slightly clearer.
>>> >> >>>
>>> >> >>> Doing the same examples on a larger array won't make the point
>>> >> >>> easier
>>> >> >>> to understand.  The discussion is about what the right concepts
>>> >> >>> are,
>>> >> >>> and you can help by looki

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Eric Firing]

On 10/29/2011 12:57 PM, Charles R Harris wrote:
>
>
> On Sat, Oct 29, 2011 at 4:47 PM, Eric Firing  > wrote:
>
> On 10/29/2011 12:02 PM, Olivier Delalleau wrote:
>
>  >
>  > I haven't been following the discussion closely, but wouldn't it
> be instead:
>  > a.mask[0:2] = True?
>
> That would be consistent with numpy.ma  and the
> opposite of Mark's
> implementation.
>
> I can live with either, but I much prefer the numpy.ma
>  version because
> it fits with the use of bit-flags for editing data; set bit 1 if it
> fails check A, set bit 2 if it fails check B, etc.  So, if it evaluates
> as True, there is a problem, and the value is masked *out*.
>
> Similarly, in Marks implementation, 7 bits are available for a payload
> to describe what kind of masking is meant.  This seems more consistent
> with True as masked (or NA) than with False as masked.
>
>
> I wouldn't rely on the 7 bits yet. Mark left them available to keep open
> possible future use, but didn't implement anything using them yet. If
> memory use turns out to exclude whole sectors of application we will
> have to go to bit masks.

Right; I was only commenting on a subjective sense of internal 
consistency.  A minor point.

The larger context of all this is how users end up being able to work 
with all the different types and specifications of "NA" (in the most 
general sense) data:

1) nans
2) numpy.ma
3) masks in the core (Mark's new code)
4) bit patterns

Substantial code now in place--including matplotlib--relies on numpy.ma. 
  It has some rough edges, it can be slow, it is a pain having it as a 
bolted-on module, it may be more complicated than it needs to be, but it 
fits a lot of use cases pretty well.  There are many users.  Everyone 
using matplotlib is using it, whether they know it or not.

The ideal from my numpy.ma-user's standpoint would an NA-handling 
implementation in the core that would do two things:
(1) allow a gradual transition away from numpy.ma, so that the latter 
would become redundant.
(2) allow numpy.ma to be reasonably easily modified to use the in-core 
facilities for greater efficiency during the long transition.  Implicit 
is the hope that someone (most likely not me, although I might be able 
to help a bit) would actually perform this modification.

Mark's mission, paid for by Enthought, was not to please numpy.ma users, 
but to add NA-handling that would be comfortable for R-users.  He chose 
to do so with the idea that two possible implementations (masks and 
bitpatterns) were desirable, each with strengths and weaknesses, and 
that so as to get *something* done in the very short time he had left, 
he would start with the mask implementation.  We now have the result, 
incomplete, but not breaking anything.  Additional development (coding 
as well as designing) will be needed.

The main question raised by Matthew and Nathaniel is, I think, whether 
Mark's code should develop in a direction away from the R-compatibility 
model, with the idea that the latter would be handled via a bit-pattern 
implementation, some day, when someone codes it; or whether it should 
remain as the prototype and first implementation of an API to handle the 
R-compatible use case, minimizing any divergence from any eventual 
bit-pattern implementation.

The answer to this depends on several questions, including:

1) Who is available to do how much implementation of any of the 
possibilities?  My reading of Travis's blog and rare posts to this list 
suggest that he hopes and expects to be able to free up coding time. 
Perhaps he will clarify that soon.

2) What sorts of changes would actually be needed to make the present 
implementation good enough for the R use case?  Evolutionary, or 
revolutionary?

3) What sorts of changes would help with the numpy.ma use case? 
Evolutionary, or revolutionary.

4) Given available resources, how can we maximize progress: making numpy 
more capable, easier to use, etc.

Unless the answers to questions 2 *and* 3 are "revolutionary", I don't 
see the point in pulling Mark's changes out of master.  At most, the 
documentation might be changed to mark the NA API as "experimental" for 
a release or two.

Overall, I think that the differences between the R use case and the ma 
use case have been overstated and over-emphasized.

Eric


>
> Chuck
>
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Matthew Brett]

Hi,

On Sat, Oct 29, 2011 at 11:14 AM, Wes McKinney  wrote:
> On Fri, Oct 28, 2011 at 9:32 PM, Charles R Harris
>  wrote:
>>
>>
>> On Fri, Oct 28, 2011 at 6:45 PM, Wes McKinney  wrote:
>>>
>>> On Fri, Oct 28, 2011 at 7:53 PM, Benjamin Root  wrote:
>>> >
>>> >
>>> > On Friday, October 28, 2011, Matthew Brett 
>>> > wrote:
>>> >> Hi,
>>> >>
>>> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
>>> >>  wrote:
>>> >>>
>>> >>>
>>> >>> On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett
>>> >>> 
>>> >>> wrote:
>>> 
>>>  Hi,
>>> 
>>>  On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
>>>   wrote:
>>>  >
>>>  >
>>>  > On Fri, Oct 28, 2011 at 3:56 PM, Matthew Brett
>>>  > 
>>>  > wrote:
>>>  >>
>>>  >> Hi,
>>>  >>
>>>  >> On Fri, Oct 28, 2011 at 2:43 PM, Matthew Brett
>>>  >> 
>>>  >> wrote:
>>>  >> > Hi,
>>>  >> >
>>>  >> > On Fri, Oct 28, 2011 at 2:41 PM, Charles R Harris
>>>  >> >  wrote:
>>>  >> >>
>>>  >> >>
>>>  >> >> On Fri, Oct 28, 2011 at 3:16 PM, Nathaniel Smith
>>>  >> >> 
>>>  >> >> wrote:
>>>  >> >>>
>>>  >> >>> On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant
>>>  >> >>> 
>>>  >> >>> wrote:
>>>  >> >>> > I think Nathaniel and Matthew provided very
>>>  >> >>> > specific feedback that was helpful in understanding other
>>>  >> >>> > perspectives
>>>  >> >>> > of a
>>>  >> >>> > difficult problem.     In particular, I really wanted
>>>  >> >>> > bit-patterns
>>>  >> >>> > implemented.    However, I also understand that Mark did
>>>  >> >>> > quite
>>>  >> >>> > a
>>>  >> >>> > bit
>>>  >> >>> > of
>>>  >> >>> > work
>>>  >> >>> > and altered his original designs quite a bit in response to
>>>  >> >>> > community
>>>  >> >>> > feedback.   I wasn't a major part of the pull request
>>>  >> >>> > discussion,
>>>  >> >>> > nor
>>>  >> >>> > did I
>>>  >> >>> > merge the changes, but I support Charles if he reviewed the
>>>  >> >>> > code
>>>  >> >>> > and
>>>  >> >>> > felt
>>>  >> >>> > like it was the right thing to do.  I likely would have done
>>>  >> >>> > the
>>>  >> >>> > same
>>>  >> >>> > thing
>>>  >> >>> > rather than let Mark Wiebe's work languish.
>>>  >> >>>
>>>  >> >>> My connectivity is spotty this week, so I'll stay out of the
>>>  >> >>> technical
>>>  >> >>> discussion for now, but I want to share a story.
>>>  >> >>>
>>>  >> >>> Maybe a year ago now, Jonathan Taylor and I were debating what
>>>  >> >>> the
>>>  >> >>> best API for describing statistical models would be -- whether
>>>  >> >>> we
>>>  >> >>> wanted something like R's "formulas" (which I supported), or
>>>  >> >>> another
>>>  >> >>> approach based on sympy (his idea). To summarize, I thought
>>>  >> >>> his
>>>  >> >>> API
>>>  >> >>> was confusing, pointlessly complicated, and didn't actually
>>>  >> >>> solve
>>>  >> >>> the
>>>  >> >>> problem; he thought R-style formulas were superficially
>>>  >> >>> simpler
>>>  >> >>> but
>>>  >> >>> hopelessly confused and inconsistent underneath. Now,
>>>  >> >>> obviously,
>>>  >> >>> I
>>>  >> >>> was
>>>  >> >>> right and he was wrong. Well, obvious to me, anyway... ;-) But
>>>  >> >>> it
>>>  >> >>> wasn't like I could just wave a wand and make his arguments go
>>>  >> >>> away,
>>>  >> >>> no I should point out that the implementation hasn't - as far
>>>  >> >>> as
>>>  >> >>> I can
>>> >> see - changed the discussion.  The discussion was about the API.
>>> >> Implementations are useful for agreed APIs because they can point out
>>> >> where the API does not make sense or cannot be implemented.  In this
>>> >> case, the API Mark said he was going to implement - he did implement -
>>> >> at least as far as I can see.  Again, I'm happy to be corrected.
>>> >>
>>>  In saying that we are insisting on our way, you are saying,
>>>  implicitly,
>>>  'I
>>>  am not going to negotiate'.
>>> >>>
>>> >>> That is only your interpretation. The observation that Mark
>>> >>> compromised
>>> >>> quite a bit while you didn't seems largely correct to me.
>>> >>
>>> >> The problem here stems from our inability to work towards agreement,
>>> >> rather than standing on set positions.  I set out what changes I think
>>> >> would make the current implementation OK.  Can we please, please have
>>> >> a discussion about those points instead of trying to argue about who
>>> >> has given more ground.
>>> >>
>>> >>> That commitment would of course be good. However, even if that were
>>> >>> possible
>>> >>> before writing code and everyone agreed that the ideas of you and
>>> >>> Nathaniel
>>> >>> should be implemented in full, it's still not clear that either of you
>>> >>> would
>>> >>> be willing to write any code. Agreement without code still doesn

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Matthew Brett]

Hi,

On Sat, Oct 29, 2011 at 4:28 PM, Han Genuit  wrote:
> To be honest, you have been slandering a lot, also in previous
> discussions, to get what you wanted. This is not a healthy way of
> discussion, nor does it help in any way.

That's a severe accusation.  Please quote something I said that was
false, or unfair.

See you,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Han Genuit]

To be honest, you have been slandering a lot, also in previous
discussions, to get what you wanted. This is not a healthy way of
discussion, nor does it help in any way.

There have been many people willing to listen and agree with you on
points; and this is exactly what discussion is all about, but where
they might agree on some, they might disagree on others. When you
start pulling the - people who won't listen to me are evil - card, it
might have some effect the first time, but the second and third time
they see what's coming.. o/
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Matthew Brett]

Hi,

On Sat, Oct 29, 2011 at 4:24 PM, Benjamin Root  wrote:
>
>
> On Saturday, October 29, 2011, Matthew Brett 
> wrote:
>> Hi,
>>
>> On Sat, Oct 29, 2011 at 2:59 PM, Charles R Harris
>>  wrote:
>>>
>>>
>>> On Sat, Oct 29, 2011 at 3:55 PM, Matthew Brett 
>>> wrote:

 Hi,

 On Sat, Oct 29, 2011 at 2:48 PM, Ralf Gommers
  wrote:
 >
 >
 > On Sat, Oct 29, 2011 at 11:36 PM, Matthew Brett
 > 
 > wrote:
 >>
 >> Hi,
 >>
 >> On Sat, Oct 29, 2011 at 1:48 PM, Matthew Brett
 >> 
 >> wrote:
 >> > Hi,
 >> >
 >> > On Sat, Oct 29, 2011 at 1:44 PM, Ralf Gommers
 >> >  wrote:
 >> >>
 >> >>
 >> >> On Sat, Oct 29, 2011 at 9:04 PM, Matthew Brett
 >> >> 
 >> >> wrote:
 >> >>>
 >> >>> Hi,
 >> >>>
 >> >>> On Sat, Oct 29, 2011 at 3:26 AM, Ralf Gommers
 >> >>>  wrote:
 >> >>> >
 >> >>> >
 >> >>> > On Sat, Oct 29, 2011 at 1:37 AM, Matthew Brett
 >> >>> > 
 >> >>> > wrote:
 >> >>> >>
 >> >>> >> Hi,
 >> >>> >>
 >> >>> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
 >> >>> >>  wrote:
 >> >>> >> >
 >> >>> >> >
 >> >>> >> > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett
 >> >>> >> > 
 >> >>> >> > wrote:
 >> >>> >> >>
 >> >>> >> >> Hi,
 >> >>> >> >>
 >> >>> >> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
 >> >>> >> >>  wrote:
 >> >>> >> >> >>
 >> >>> >> >>
 >> >>> >> >> No, that's not what Nathaniel and I are saying at all.
 >> >>> >> >> Nathaniel
 >> >>> >> >> was
 >> >>> >> >> pointing to links for projects that care that everyone
 >> >>> >> >> agrees
 >> >>> >> >> before
 >> >>> >> >> they go ahead.
 >> >>> >> >
 >> >>> >> > It looked to me like there was a serious intent to come to
 >> >>> >> > an
 >> >>> >> > agreement,
 >> >>> >> > or
 >> >>> >> > at least closer together. The discussion in the summer was
 >> >>> >> > going
 >> >>> >> > around
 >> >>> >> > in
 >> >>> >> > circles though, and was too abstract and complex to follow.
 You are repeating the loaded phrase 'ripping the current code out' and
>> thus making the discussion less sensible and more hostile.
>>
>>>  It isn't like it is (known
>>> to be) buggy, nor has anyone made the case that it isn't a basis on which
>>> build other options. It also smacks of gratuitous violence committed by
>>> someone yet to make a positive contribution to the project.
>>
>> This is cheap, rude, and silly.  All I can see from Nathaniel is a
>> reasonable, fair attempt to discuss the code.  He proposed backing off
>> the code in good faith.   You are emphatically, and, in my view
>> childishly, ignoring the substantial points he is making, and
>> asserting over and over that he deserves no hearing because he has not
>> contributed code.   This is a terribly destructive way to work.  If I
>> was a new developer reading this, I would conclude, that I had better
>> be damn careful which side I'm on, before I express my opinion,
>> otherwise I'm going to be made to feel like I don't exist by the other
>> people on the project.  That is miserable, it is silly, and it's the
>> wrong way to do business.
>>
>> Best,
>>
>> Matthew
>>
>
> /me blows whistle. Personal foul against defense! Personal foul against
> offense! Penalties offset! Repeat first down.

Is that right?  I think I'm calling Charles on giving Nathaniel the
silent treatment.  Am I wrong to do that?   Is that not true?

See you,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Matthew Brett]

Hi,

On Sat, Oct 29, 2011 at 4:18 PM, Charles R Harris
 wrote:
>
>
> On Sat, Oct 29, 2011 at 5:11 PM, Matthew Brett 
> wrote:
>>
>> Hi,
>>
>> On Sat, Oct 29, 2011 at 2:59 PM, Charles R Harris
>>  wrote:
>> >
>> >
>> > On Sat, Oct 29, 2011 at 3:55 PM, Matthew Brett 
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> On Sat, Oct 29, 2011 at 2:48 PM, Ralf Gommers
>> >>  wrote:
>> >> >
>> >> >
>> >> > On Sat, Oct 29, 2011 at 11:36 PM, Matthew Brett
>> >> > 
>> >> > wrote:
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> On Sat, Oct 29, 2011 at 1:48 PM, Matthew Brett
>> >> >> 
>> >> >> wrote:
>> >> >> > Hi,
>> >> >> >
>> >> >> > On Sat, Oct 29, 2011 at 1:44 PM, Ralf Gommers
>> >> >> >  wrote:
>> >> >> >>
>> >> >> >>
>> >> >> >> On Sat, Oct 29, 2011 at 9:04 PM, Matthew Brett
>> >> >> >> 
>> >> >> >> wrote:
>> >> >> >>>
>> >> >> >>> Hi,
>> >> >> >>>
>> >> >> >>> On Sat, Oct 29, 2011 at 3:26 AM, Ralf Gommers
>> >> >> >>>  wrote:
>> >> >> >>> >
>> >> >> >>> >
>> >> >> >>> > On Sat, Oct 29, 2011 at 1:37 AM, Matthew Brett
>> >> >> >>> > 
>> >> >> >>> > wrote:
>> >> >> >>> >>
>> >> >> >>> >> Hi,
>> >> >> >>> >>
>> >> >> >>> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
>> >> >> >>> >>  wrote:
>> >> >> >>> >> >
>> >> >> >>> >> >
>> >> >> >>> >> > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett
>> >> >> >>> >> > 
>> >> >> >>> >> > wrote:
>> >> >> >>> >> >>
>> >> >> >>> >> >> Hi,
>> >> >> >>> >> >>
>> >> >> >>> >> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
>> >> >> >>> >> >>  wrote:
>> >> >> >>> >> >> >>
>> >> >> >>> >> >>
>> >> >> >>> >> >> No, that's not what Nathaniel and I are saying at all.
>> >> >> >>> >> >> Nathaniel
>> >> >> >>> >> >> was
>> >> >> >>> >> >> pointing to links for projects that care that everyone
>> >> >> >>> >> >> agrees
>> >> >> >>> >> >> before
>> >> >> >>> >> >> they go ahead.
>> >> >> >>> >> >
>> >> >> >>> >> > It looked to me like there was a serious intent to come to
>> >> >> >>> >> > an
>> >> >> >>> >> > agreement,
>> >> >> >>> >> > or
>> >> >> >>> >> > at least closer together. The discussion in the summer was
>> >> >> >>> >> > going
>> >> >> >>> >> > around
>> >> >> >>> >> > in
>> >> >> >>> >> > circles though, and was too abstract and complex to follow.
>> >> >> >>> >> > Therefore
>> >> >> >>> >> > Mark's
>> >> >> >>> >> > choice of implementing something and then asking for
>> >> >> >>> >> > feedback
>> >> >> >>> >> > made
>> >> >> >>> >> > sense
>> >> >> >>> >> > to
>> >> >> >>> >> > me.
>> >> >> >>> >>
>> >> >> >>> >> I should point out that the implementation hasn't - as far as
>> >> >> >>> >> I
>> >> >> >>> >> can
>> >> >> >>> >> see - changed the discussion.  The discussion was about the
>> >> >> >>> >> API.
>> >> >> >>> >>
>> >> >> >>> >> Implementations are useful for agreed APIs because they can
>> >> >> >>> >> point
>> >> >> >>> >> out
>> >> >> >>> >> where the API does not make sense or cannot be implemented.
>> >> >> >>> >>  In
>> >> >> >>> >> this
>> >> >> >>> >> case, the API Mark said he was going to implement - he did
>> >> >> >>> >> implement -
>> >> >> >>> >> at least as far as I can see.  Again, I'm happy to be
>> >> >> >>> >> corrected.
>> >> >> >>> >
>> >> >> >>> > Implementations can also help the discussion along, by
>> >> >> >>> > allowing
>> >> >> >>> > people
>> >> >> >>> > to
>> >> >> >>> > try out some of the proposed changes. It also allows to
>> >> >> >>> > construct
>> >> >> >>> > examples
>> >> >> >>> > that show weaknesses, possibly to be solved by an alternative
>> >> >> >>> > API.
>> >> >> >>> > Maybe
>> >> >> >>> > you
>> >> >> >>> > can hold the complete history of this topic in your head and
>> >> >> >>> > comprehend
>> >> >> >>> > it,
>> >> >> >>> > but for me it would be very helpful if someone said:
>> >> >> >>> > - here's my dataset
>> >> >> >>> > - this is what I want to do with it
>> >> >> >>> > - this is the best I can do with the current implementation
>> >> >> >>> > - here's how API X would allow me to solve this better or
>> >> >> >>> > simpler
>> >> >> >>> > This can be done much better with actual data and an actual
>> >> >> >>> > implementation
>> >> >> >>> > than with a design proposal. You seem to disagree with this
>> >> >> >>> > statement.
>> >> >> >>> > That's fine. I would hope though that you recognize that
>> >> >> >>> > concrete
>> >> >> >>> > examples
>> >> >> >>> > help people like me, and construct one or two to help us out.
>> >> >> >>> That's what use-cases are for in designing APIs.  There are
>> >> >> >>> examples
>> >> >> >>> of use in the NEP:
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst
>> >> >> >>>
>> >> >> >>> the alterNEP:
>> >> >> >>>
>> >> >> >>> https://gist.github.com/1056379
>> >> >> >>>
>> >> >> >>> and my longer email to Travis:
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> http://article.gmane.org/gmane.comp.python.numeric.general/46544/match=ignored
>> >> >> >>>
>> >> >> >>> Mark has done a nice job of docum

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Benjamin Root]

On Saturday, October 29, 2011, Matthew Brett 
wrote:
> Hi,
>
> On Sat, Oct 29, 2011 at 2:59 PM, Charles R Harris
>  wrote:
>>
>>
>> On Sat, Oct 29, 2011 at 3:55 PM, Matthew Brett 
>> wrote:
>>>
>>> Hi,
>>>
>>> On Sat, Oct 29, 2011 at 2:48 PM, Ralf Gommers
>>>  wrote:
>>> >
>>> >
>>> > On Sat, Oct 29, 2011 at 11:36 PM, Matthew Brett
>>> > 
>>> > wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> On Sat, Oct 29, 2011 at 1:48 PM, Matthew Brett
>>> >> 
>>> >> wrote:
>>> >> > Hi,
>>> >> >
>>> >> > On Sat, Oct 29, 2011 at 1:44 PM, Ralf Gommers
>>> >> >  wrote:
>>> >> >>
>>> >> >>
>>> >> >> On Sat, Oct 29, 2011 at 9:04 PM, Matthew Brett
>>> >> >> 
>>> >> >> wrote:
>>> >> >>>
>>> >> >>> Hi,
>>> >> >>>
>>> >> >>> On Sat, Oct 29, 2011 at 3:26 AM, Ralf Gommers
>>> >> >>>  wrote:
>>> >> >>> >
>>> >> >>> >
>>> >> >>> > On Sat, Oct 29, 2011 at 1:37 AM, Matthew Brett
>>> >> >>> > 
>>> >> >>> > wrote:
>>> >> >>> >>
>>> >> >>> >> Hi,
>>> >> >>> >>
>>> >> >>> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
>>> >> >>> >>  wrote:
>>> >> >>> >> >
>>> >> >>> >> >
>>> >> >>> >> > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett
>>> >> >>> >> > 
>>> >> >>> >> > wrote:
>>> >> >>> >> >>
>>> >> >>> >> >> Hi,
>>> >> >>> >> >>
>>> >> >>> >> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
>>> >> >>> >> >>  wrote:
>>> >> >>> >> >> >>
>>> >> >>> >> >>
>>> >> >>> >> >> No, that's not what Nathaniel and I are saying at all.
>>> >> >>> >> >> Nathaniel
>>> >> >>> >> >> was
>>> >> >>> >> >> pointing to links for projects that care that everyone
agrees
>>> >> >>> >> >> before
>>> >> >>> >> >> they go ahead.
>>> >> >>> >> >
>>> >> >>> >> > It looked to me like there was a serious intent to come to
an
>>> >> >>> >> > agreement,
>>> >> >>> >> > or
>>> >> >>> >> > at least closer together. The discussion in the summer was
>>> >> >>> >> > going
>>> >> >>> >> > around
>>> >> >>> >> > in
>>> >> >>> >> > circles though, and was too abstract and complex to follow.
>>> You are repeating the loaded phrase 'ripping the current code out' and
> thus making the discussion less sensible and more hostile.
>
>>  It isn't like it is (known
>> to be) buggy, nor has anyone made the case that it isn't a basis on which
>> build other options. It also smacks of gratuitous violence committed by
>> someone yet to make a positive contribution to the project.
>
> This is cheap, rude, and silly.  All I can see from Nathaniel is a
> reasonable, fair attempt to discuss the code.  He proposed backing off
> the code in good faith.   You are emphatically, and, in my view
> childishly, ignoring the substantial points he is making, and
> asserting over and over that he deserves no hearing because he has not
> contributed code.   This is a terribly destructive way to work.  If I
> was a new developer reading this, I would conclude, that I had better
> be damn careful which side I'm on, before I express my opinion,
> otherwise I'm going to be made to feel like I don't exist by the other
> people on the project.  That is miserable, it is silly, and it's the
> wrong way to do business.
>
> Best,
>
> Matthew
>

/me blows whistle. Personal foul against defense! Personal foul against
offense! Penalties offset! Repeat first down.

10 minute rule, please.

Ben Root

P.S. - as a bit of evidence against the idea that chuck doesnt consider
opinions from non-contributors, I haven't felt ignored during this whole
discussion, yet I don't think that anyone had an expectation of me to
produce code.  However, to have an expectation to produce code for
counter-proposals might be a bit unfair because the ones offering counter
proposal may not have the resources available, like we did with mark.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Charles R Harris]

On Sat, Oct 29, 2011 at 5:11 PM, Matthew Brett wrote:

> Hi,
>
> On Sat, Oct 29, 2011 at 2:59 PM, Charles R Harris
>  wrote:
> >
> >
> > On Sat, Oct 29, 2011 at 3:55 PM, Matthew Brett 
> > wrote:
> >>
> >> Hi,
> >>
> >> On Sat, Oct 29, 2011 at 2:48 PM, Ralf Gommers
> >>  wrote:
> >> >
> >> >
> >> > On Sat, Oct 29, 2011 at 11:36 PM, Matthew Brett
> >> > 
> >> > wrote:
> >> >>
> >> >> Hi,
> >> >>
> >> >> On Sat, Oct 29, 2011 at 1:48 PM, Matthew Brett
> >> >> 
> >> >> wrote:
> >> >> > Hi,
> >> >> >
> >> >> > On Sat, Oct 29, 2011 at 1:44 PM, Ralf Gommers
> >> >> >  wrote:
> >> >> >>
> >> >> >>
> >> >> >> On Sat, Oct 29, 2011 at 9:04 PM, Matthew Brett
> >> >> >> 
> >> >> >> wrote:
> >> >> >>>
> >> >> >>> Hi,
> >> >> >>>
> >> >> >>> On Sat, Oct 29, 2011 at 3:26 AM, Ralf Gommers
> >> >> >>>  wrote:
> >> >> >>> >
> >> >> >>> >
> >> >> >>> > On Sat, Oct 29, 2011 at 1:37 AM, Matthew Brett
> >> >> >>> > 
> >> >> >>> > wrote:
> >> >> >>> >>
> >> >> >>> >> Hi,
> >> >> >>> >>
> >> >> >>> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
> >> >> >>> >>  wrote:
> >> >> >>> >> >
> >> >> >>> >> >
> >> >> >>> >> > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett
> >> >> >>> >> > 
> >> >> >>> >> > wrote:
> >> >> >>> >> >>
> >> >> >>> >> >> Hi,
> >> >> >>> >> >>
> >> >> >>> >> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
> >> >> >>> >> >>  wrote:
> >> >> >>> >> >> >>
> >> >> >>> >> >>
> >> >> >>> >> >> No, that's not what Nathaniel and I are saying at all.
> >> >> >>> >> >> Nathaniel
> >> >> >>> >> >> was
> >> >> >>> >> >> pointing to links for projects that care that everyone
> agrees
> >> >> >>> >> >> before
> >> >> >>> >> >> they go ahead.
> >> >> >>> >> >
> >> >> >>> >> > It looked to me like there was a serious intent to come to
> an
> >> >> >>> >> > agreement,
> >> >> >>> >> > or
> >> >> >>> >> > at least closer together. The discussion in the summer was
> >> >> >>> >> > going
> >> >> >>> >> > around
> >> >> >>> >> > in
> >> >> >>> >> > circles though, and was too abstract and complex to follow.
> >> >> >>> >> > Therefore
> >> >> >>> >> > Mark's
> >> >> >>> >> > choice of implementing something and then asking for
> feedback
> >> >> >>> >> > made
> >> >> >>> >> > sense
> >> >> >>> >> > to
> >> >> >>> >> > me.
> >> >> >>> >>
> >> >> >>> >> I should point out that the implementation hasn't - as far as
> I
> >> >> >>> >> can
> >> >> >>> >> see - changed the discussion.  The discussion was about the
> API.
> >> >> >>> >>
> >> >> >>> >> Implementations are useful for agreed APIs because they can
> >> >> >>> >> point
> >> >> >>> >> out
> >> >> >>> >> where the API does not make sense or cannot be implemented.
>  In
> >> >> >>> >> this
> >> >> >>> >> case, the API Mark said he was going to implement - he did
> >> >> >>> >> implement -
> >> >> >>> >> at least as far as I can see.  Again, I'm happy to be
> corrected.
> >> >> >>> >
> >> >> >>> > Implementations can also help the discussion along, by allowing
> >> >> >>> > people
> >> >> >>> > to
> >> >> >>> > try out some of the proposed changes. It also allows to
> construct
> >> >> >>> > examples
> >> >> >>> > that show weaknesses, possibly to be solved by an alternative
> >> >> >>> > API.
> >> >> >>> > Maybe
> >> >> >>> > you
> >> >> >>> > can hold the complete history of this topic in your head and
> >> >> >>> > comprehend
> >> >> >>> > it,
> >> >> >>> > but for me it would be very helpful if someone said:
> >> >> >>> > - here's my dataset
> >> >> >>> > - this is what I want to do with it
> >> >> >>> > - this is the best I can do with the current implementation
> >> >> >>> > - here's how API X would allow me to solve this better or
> simpler
> >> >> >>> > This can be done much better with actual data and an actual
> >> >> >>> > implementation
> >> >> >>> > than with a design proposal. You seem to disagree with this
> >> >> >>> > statement.
> >> >> >>> > That's fine. I would hope though that you recognize that
> concrete
> >> >> >>> > examples
> >> >> >>> > help people like me, and construct one or two to help us out.
> >> >> >>> That's what use-cases are for in designing APIs.  There are
> >> >> >>> examples
> >> >> >>> of use in the NEP:
> >> >> >>>
> >> >> >>>
> >> >> >>>
> https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst
> >> >> >>>
> >> >> >>> the alterNEP:
> >> >> >>>
> >> >> >>> https://gist.github.com/1056379
> >> >> >>>
> >> >> >>> and my longer email to Travis:
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >> >>>
> http://article.gmane.org/gmane.comp.python.numeric.general/46544/match=ignored
> >> >> >>>
> >> >> >>> Mark has done a nice job of documentation:
> >> >> >>>
> >> >> >>> http://docs.scipy.org/doc/numpy/reference/arrays.maskna.html
> >> >> >>>
> >> >> >>> If you want to understand what the alterNEP case is, I'd suggest
> >> >> >>> the
> >> >> >>> email, just because it's the most recent and I think the
> >> >> >>> terminology
> >> >> >>> is slightly clearer.
> >> >> >>>
> >> >> >>> Doing the same examples on a larger array wo

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Charles R Harris]

On Sat, Oct 29, 2011 at 5:11 PM, Matthew Brett wrote:

> Hi,
>
> On Sat, Oct 29, 2011 at 2:59 PM, Charles R Harris
>  wrote:
> >
> >
> > On Sat, Oct 29, 2011 at 3:55 PM, Matthew Brett 
> > wrote:
> >>
> >> Hi,
> >>
> >> On Sat, Oct 29, 2011 at 2:48 PM, Ralf Gommers
> >>  wrote:
> >> >
> >> >
> >> > On Sat, Oct 29, 2011 at 11:36 PM, Matthew Brett
> >> > 
> >> > wrote:
> >> >>
> >> >> Hi,
> >> >>
> >> >> On Sat, Oct 29, 2011 at 1:48 PM, Matthew Brett
> >> >> 
> >> >> wrote:
> >> >> > Hi,
> >> >> >
> >> >> > On Sat, Oct 29, 2011 at 1:44 PM, Ralf Gommers
> >> >> >  wrote:
> >> >> >>
> >> >> >>
> >> >> >> On Sat, Oct 29, 2011 at 9:04 PM, Matthew Brett
> >> >> >> 
> >> >> >> wrote:
> >> >> >>>
> >> >> >>> Hi,
> >> >> >>>
> >> >> >>> On Sat, Oct 29, 2011 at 3:26 AM, Ralf Gommers
> >> >> >>>  wrote:
> >> >> >>> >
> >> >> >>> >
> >> >> >>> > On Sat, Oct 29, 2011 at 1:37 AM, Matthew Brett
> >> >> >>> > 
> >> >> >>> > wrote:
> >> >> >>> >>
> >> >> >>> >> Hi,
> >> >> >>> >>
> >> >> >>> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
> >> >> >>> >>  wrote:
> >> >> >>> >> >
> >> >> >>> >> >
> >> >> >>> >> > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett
> >> >> >>> >> > 
> >> >> >>> >> > wrote:
> >> >> >>> >> >>
> >> >> >>> >> >> Hi,
> >> >> >>> >> >>
> >> >> >>> >> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
> >> >> >>> >> >>  wrote:
> >> >> >>> >> >> >>
> >> >> >>> >> >>
> >> >> >>> >> >> No, that's not what Nathaniel and I are saying at all.
> >> >> >>> >> >> Nathaniel
> >> >> >>> >> >> was
> >> >> >>> >> >> pointing to links for projects that care that everyone
> agrees
> >> >> >>> >> >> before
> >> >> >>> >> >> they go ahead.
> >> >> >>> >> >
> >> >> >>> >> > It looked to me like there was a serious intent to come to
> an
> >> >> >>> >> > agreement,
> >> >> >>> >> > or
> >> >> >>> >> > at least closer together. The discussion in the summer was
> >> >> >>> >> > going
> >> >> >>> >> > around
> >> >> >>> >> > in
> >> >> >>> >> > circles though, and was too abstract and complex to follow.
> >> >> >>> >> > Therefore
> >> >> >>> >> > Mark's
> >> >> >>> >> > choice of implementing something and then asking for
> feedback
> >> >> >>> >> > made
> >> >> >>> >> > sense
> >> >> >>> >> > to
> >> >> >>> >> > me.
> >> >> >>> >>
> >> >> >>> >> I should point out that the implementation hasn't - as far as
> I
> >> >> >>> >> can
> >> >> >>> >> see - changed the discussion.  The discussion was about the
> API.
> >> >> >>> >>
> >> >> >>> >> Implementations are useful for agreed APIs because they can
> >> >> >>> >> point
> >> >> >>> >> out
> >> >> >>> >> where the API does not make sense or cannot be implemented.
>  In
> >> >> >>> >> this
> >> >> >>> >> case, the API Mark said he was going to implement - he did
> >> >> >>> >> implement -
> >> >> >>> >> at least as far as I can see.  Again, I'm happy to be
> corrected.
> >> >> >>> >
> >> >> >>> > Implementations can also help the discussion along, by allowing
> >> >> >>> > people
> >> >> >>> > to
> >> >> >>> > try out some of the proposed changes. It also allows to
> construct
> >> >> >>> > examples
> >> >> >>> > that show weaknesses, possibly to be solved by an alternative
> >> >> >>> > API.
> >> >> >>> > Maybe
> >> >> >>> > you
> >> >> >>> > can hold the complete history of this topic in your head and
> >> >> >>> > comprehend
> >> >> >>> > it,
> >> >> >>> > but for me it would be very helpful if someone said:
> >> >> >>> > - here's my dataset
> >> >> >>> > - this is what I want to do with it
> >> >> >>> > - this is the best I can do with the current implementation
> >> >> >>> > - here's how API X would allow me to solve this better or
> simpler
> >> >> >>> > This can be done much better with actual data and an actual
> >> >> >>> > implementation
> >> >> >>> > than with a design proposal. You seem to disagree with this
> >> >> >>> > statement.
> >> >> >>> > That's fine. I would hope though that you recognize that
> concrete
> >> >> >>> > examples
> >> >> >>> > help people like me, and construct one or two to help us out.
> >> >> >>> That's what use-cases are for in designing APIs.  There are
> >> >> >>> examples
> >> >> >>> of use in the NEP:
> >> >> >>>
> >> >> >>>
> >> >> >>>
> https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst
> >> >> >>>
> >> >> >>> the alterNEP:
> >> >> >>>
> >> >> >>> https://gist.github.com/1056379
> >> >> >>>
> >> >> >>> and my longer email to Travis:
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >> >>>
> >> >> >>>
> http://article.gmane.org/gmane.comp.python.numeric.general/46544/match=ignored
> >> >> >>>
> >> >> >>> Mark has done a nice job of documentation:
> >> >> >>>
> >> >> >>> http://docs.scipy.org/doc/numpy/reference/arrays.maskna.html
> >> >> >>>
> >> >> >>> If you want to understand what the alterNEP case is, I'd suggest
> >> >> >>> the
> >> >> >>> email, just because it's the most recent and I think the
> >> >> >>> terminology
> >> >> >>> is slightly clearer.
> >> >> >>>
> >> >> >>> Doing the same examples on a larger array wo

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Han Genuit]

On Sun, Oct 30, 2011 at 12:47 AM, Eric Firing  wrote:
> On 10/29/2011 12:02 PM, Olivier Delalleau wrote:
>
>>
>> I haven't been following the discussion closely, but wouldn't it be instead:
>> a.mask[0:2] = True?
>
> That would be consistent with numpy.ma and the opposite of Mark's
> implementation.
>
> I can live with either, but I much prefer the numpy.ma version because
> it fits with the use of bit-flags for editing data; set bit 1 if it
> fails check A, set bit 2 if it fails check B, etc.  So, if it evaluates
> as True, there is a problem, and the value is masked *out*.
>

I think in Mark's implementation it works the same:

>>> a = np.arange(3, maskna=True)
>>> a[1] = np.NA
>>> a
array([0, NA, 2])
>>> np.isna(a)
array([False,  True, False], dtype=bool)

This is more consistent than using False to represent an NA mask, I agree.
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Matthew Brett]

Hi,

On Sat, Oct 29, 2011 at 2:59 PM, Charles R Harris
 wrote:
>
>
> On Sat, Oct 29, 2011 at 3:55 PM, Matthew Brett 
> wrote:
>>
>> Hi,
>>
>> On Sat, Oct 29, 2011 at 2:48 PM, Ralf Gommers
>>  wrote:
>> >
>> >
>> > On Sat, Oct 29, 2011 at 11:36 PM, Matthew Brett
>> > 
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> On Sat, Oct 29, 2011 at 1:48 PM, Matthew Brett
>> >> 
>> >> wrote:
>> >> > Hi,
>> >> >
>> >> > On Sat, Oct 29, 2011 at 1:44 PM, Ralf Gommers
>> >> >  wrote:
>> >> >>
>> >> >>
>> >> >> On Sat, Oct 29, 2011 at 9:04 PM, Matthew Brett
>> >> >> 
>> >> >> wrote:
>> >> >>>
>> >> >>> Hi,
>> >> >>>
>> >> >>> On Sat, Oct 29, 2011 at 3:26 AM, Ralf Gommers
>> >> >>>  wrote:
>> >> >>> >
>> >> >>> >
>> >> >>> > On Sat, Oct 29, 2011 at 1:37 AM, Matthew Brett
>> >> >>> > 
>> >> >>> > wrote:
>> >> >>> >>
>> >> >>> >> Hi,
>> >> >>> >>
>> >> >>> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
>> >> >>> >>  wrote:
>> >> >>> >> >
>> >> >>> >> >
>> >> >>> >> > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett
>> >> >>> >> > 
>> >> >>> >> > wrote:
>> >> >>> >> >>
>> >> >>> >> >> Hi,
>> >> >>> >> >>
>> >> >>> >> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
>> >> >>> >> >>  wrote:
>> >> >>> >> >> >>
>> >> >>> >> >>
>> >> >>> >> >> No, that's not what Nathaniel and I are saying at all.
>> >> >>> >> >> Nathaniel
>> >> >>> >> >> was
>> >> >>> >> >> pointing to links for projects that care that everyone agrees
>> >> >>> >> >> before
>> >> >>> >> >> they go ahead.
>> >> >>> >> >
>> >> >>> >> > It looked to me like there was a serious intent to come to an
>> >> >>> >> > agreement,
>> >> >>> >> > or
>> >> >>> >> > at least closer together. The discussion in the summer was
>> >> >>> >> > going
>> >> >>> >> > around
>> >> >>> >> > in
>> >> >>> >> > circles though, and was too abstract and complex to follow.
>> >> >>> >> > Therefore
>> >> >>> >> > Mark's
>> >> >>> >> > choice of implementing something and then asking for feedback
>> >> >>> >> > made
>> >> >>> >> > sense
>> >> >>> >> > to
>> >> >>> >> > me.
>> >> >>> >>
>> >> >>> >> I should point out that the implementation hasn't - as far as I
>> >> >>> >> can
>> >> >>> >> see - changed the discussion.  The discussion was about the API.
>> >> >>> >>
>> >> >>> >> Implementations are useful for agreed APIs because they can
>> >> >>> >> point
>> >> >>> >> out
>> >> >>> >> where the API does not make sense or cannot be implemented.  In
>> >> >>> >> this
>> >> >>> >> case, the API Mark said he was going to implement - he did
>> >> >>> >> implement -
>> >> >>> >> at least as far as I can see.  Again, I'm happy to be corrected.
>> >> >>> >
>> >> >>> > Implementations can also help the discussion along, by allowing
>> >> >>> > people
>> >> >>> > to
>> >> >>> > try out some of the proposed changes. It also allows to construct
>> >> >>> > examples
>> >> >>> > that show weaknesses, possibly to be solved by an alternative
>> >> >>> > API.
>> >> >>> > Maybe
>> >> >>> > you
>> >> >>> > can hold the complete history of this topic in your head and
>> >> >>> > comprehend
>> >> >>> > it,
>> >> >>> > but for me it would be very helpful if someone said:
>> >> >>> > - here's my dataset
>> >> >>> > - this is what I want to do with it
>> >> >>> > - this is the best I can do with the current implementation
>> >> >>> > - here's how API X would allow me to solve this better or simpler
>> >> >>> > This can be done much better with actual data and an actual
>> >> >>> > implementation
>> >> >>> > than with a design proposal. You seem to disagree with this
>> >> >>> > statement.
>> >> >>> > That's fine. I would hope though that you recognize that concrete
>> >> >>> > examples
>> >> >>> > help people like me, and construct one or two to help us out.
>> >> >>> That's what use-cases are for in designing APIs.  There are
>> >> >>> examples
>> >> >>> of use in the NEP:
>> >> >>>
>> >> >>>
>> >> >>> https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst
>> >> >>>
>> >> >>> the alterNEP:
>> >> >>>
>> >> >>> https://gist.github.com/1056379
>> >> >>>
>> >> >>> and my longer email to Travis:
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> http://article.gmane.org/gmane.comp.python.numeric.general/46544/match=ignored
>> >> >>>
>> >> >>> Mark has done a nice job of documentation:
>> >> >>>
>> >> >>> http://docs.scipy.org/doc/numpy/reference/arrays.maskna.html
>> >> >>>
>> >> >>> If you want to understand what the alterNEP case is, I'd suggest
>> >> >>> the
>> >> >>> email, just because it's the most recent and I think the
>> >> >>> terminology
>> >> >>> is slightly clearer.
>> >> >>>
>> >> >>> Doing the same examples on a larger array won't make the point
>> >> >>> easier
>> >> >>> to understand.  The discussion is about what the right concepts
>> >> >>> are,
>> >> >>> and you can help by looking at the snippets of code in those
>> >> >>> documents, and deciding for yourself whether you think the current
>> >> >>> masking / NA implementation seems natural and easy to explain, or
>> >> >>> rather forced and

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Charles R Harris]

On Sat, Oct 29, 2011 at 4:47 PM, Eric Firing  wrote:

> On 10/29/2011 12:02 PM, Olivier Delalleau wrote:
>
> >
> > I haven't been following the discussion closely, but wouldn't it be
> instead:
> > a.mask[0:2] = True?
>
> That would be consistent with numpy.ma and the opposite of Mark's
> implementation.
>
> I can live with either, but I much prefer the numpy.ma version because
> it fits with the use of bit-flags for editing data; set bit 1 if it
> fails check A, set bit 2 if it fails check B, etc.  So, if it evaluates
> as True, there is a problem, and the value is masked *out*.
>
> Similarly, in Marks implementation, 7 bits are available for a payload
> to describe what kind of masking is meant.  This seems more consistent
> with True as masked (or NA) than with False as masked.
>

I wouldn't rely on the 7 bits yet. Mark left them available to keep open
possible future use, but didn't implement anything using them yet. If memory
use turns out to exclude whole sectors of application we will have to go to
bit masks.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Eric Firing]

On 10/29/2011 12:02 PM, Olivier Delalleau wrote:

>
> I haven't been following the discussion closely, but wouldn't it be instead:
> a.mask[0:2] = True?

That would be consistent with numpy.ma and the opposite of Mark's 
implementation.

I can live with either, but I much prefer the numpy.ma version because 
it fits with the use of bit-flags for editing data; set bit 1 if it 
fails check A, set bit 2 if it fails check B, etc.  So, if it evaluates 
as True, there is a problem, and the value is masked *out*.

Similarly, in Marks implementation, 7 bits are available for a payload 
to describe what kind of masking is meant.  This seems more consistent 
with True as masked (or NA) than with False as masked.

Eric

>
> It's something that I actually find a bit difficult to get right in the
> current numpy.ma  implementation: I would find more
> intuitive to have True for "valid" data, and False for invalid / missing
> / ... I realize how the implementation makes sense (and is appropriate
> given that the name is "mask"), but I just thought I'd point this out...
> even if it's just me ;)
>
> -=- Olivier
>
>
>
> ___
> NumPy-Discussion mailing list
> NumPy-Discussion@scipy.org
> http://mail.scipy.org/mailman/listinfo/numpy-discussion

___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Charles R Harris]

On Sat, Oct 29, 2011 at 4:02 PM, Olivier Delalleau  wrote:

> 2011/10/29 Ralf Gommers 
>
>>
>>
>> On Sat, Oct 29, 2011 at 11:36 PM, Matthew Brett 
>> wrote:
>>
>>> Hi,
>>>
>>> On Sat, Oct 29, 2011 at 1:48 PM, Matthew Brett 
>>> wrote:
>>> > Hi,
>>> >
>>> > On Sat, Oct 29, 2011 at 1:44 PM, Ralf Gommers
>>> >  wrote:
>>> >>
>>> >>
>>> >> On Sat, Oct 29, 2011 at 9:04 PM, Matthew Brett <
>>> matthew.br...@gmail.com>
>>> >> wrote:
>>> >>>
>>> >>> Hi,
>>> >>>
>>> >>> On Sat, Oct 29, 2011 at 3:26 AM, Ralf Gommers
>>> >>>  wrote:
>>> >>> >
>>> >>> >
>>> >>> > On Sat, Oct 29, 2011 at 1:37 AM, Matthew Brett <
>>> matthew.br...@gmail.com>
>>> >>> > wrote:
>>> >>> >>
>>> >>> >> Hi,
>>> >>> >>
>>> >>> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
>>> >>> >>  wrote:
>>> >>> >> >
>>> >>> >> >
>>> >>> >> > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett
>>> >>> >> > 
>>> >>> >> > wrote:
>>> >>> >> >>
>>> >>> >> >> Hi,
>>> >>> >> >>
>>> >>> >> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
>>> >>> >> >>  wrote:
>>> >>> >> >> >>
>>> >>> >> >>
>>> >>> >> >> No, that's not what Nathaniel and I are saying at all.
>>> Nathaniel was
>>> >>> >> >> pointing to links for projects that care that everyone agrees
>>> before
>>> >>> >> >> they go ahead.
>>> >>> >> >
>>> >>> >> > It looked to me like there was a serious intent to come to an
>>> >>> >> > agreement,
>>> >>> >> > or
>>> >>> >> > at least closer together. The discussion in the summer was going
>>> >>> >> > around
>>> >>> >> > in
>>> >>> >> > circles though, and was too abstract and complex to follow.
>>> Therefore
>>> >>> >> > Mark's
>>> >>> >> > choice of implementing something and then asking for feedback
>>> made
>>> >>> >> > sense
>>> >>> >> > to
>>> >>> >> > me.
>>> >>> >>
>>> >>> >> I should point out that the implementation hasn't - as far as I
>>> can
>>> >>> >> see - changed the discussion.  The discussion was about the API.
>>> >>> >>
>>> >>> >> Implementations are useful for agreed APIs because they can point
>>> out
>>> >>> >> where the API does not make sense or cannot be implemented.  In
>>> this
>>> >>> >> case, the API Mark said he was going to implement - he did
>>> implement -
>>> >>> >> at least as far as I can see.  Again, I'm happy to be corrected.
>>> >>> >
>>> >>> > Implementations can also help the discussion along, by allowing
>>> people
>>> >>> > to
>>> >>> > try out some of the proposed changes. It also allows to construct
>>> >>> > examples
>>> >>> > that show weaknesses, possibly to be solved by an alternative API.
>>> Maybe
>>> >>> > you
>>> >>> > can hold the complete history of this topic in your head and
>>> comprehend
>>> >>> > it,
>>> >>> > but for me it would be very helpful if someone said:
>>> >>> > - here's my dataset
>>> >>> > - this is what I want to do with it
>>> >>> > - this is the best I can do with the current implementation
>>> >>> > - here's how API X would allow me to solve this better or simpler
>>> >>> > This can be done much better with actual data and an actual
>>> >>> > implementation
>>> >>> > than with a design proposal. You seem to disagree with this
>>> statement.
>>> >>> > That's fine. I would hope though that you recognize that concrete
>>> >>> > examples
>>> >>> > help people like me, and construct one or two to help us out.
>>> >>> That's what use-cases are for in designing APIs.  There are examples
>>> >>> of use in the NEP:
>>> >>>
>>> >>> https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst
>>> >>>
>>> >>> the alterNEP:
>>> >>>
>>> >>> https://gist.github.com/1056379
>>> >>>
>>> >>> and my longer email to Travis:
>>> >>>
>>> >>>
>>> >>>
>>> http://article.gmane.org/gmane.comp.python.numeric.general/46544/match=ignored
>>> >>>
>>> >>> Mark has done a nice job of documentation:
>>> >>>
>>> >>> http://docs.scipy.org/doc/numpy/reference/arrays.maskna.html
>>> >>>
>>> >>> If you want to understand what the alterNEP case is, I'd suggest the
>>> >>> email, just because it's the most recent and I think the terminology
>>> >>> is slightly clearer.
>>> >>>
>>> >>> Doing the same examples on a larger array won't make the point easier
>>> >>> to understand.  The discussion is about what the right concepts are,
>>> >>> and you can help by looking at the snippets of code in those
>>> >>> documents, and deciding for yourself whether you think the current
>>> >>> masking / NA implementation seems natural and easy to explain, or
>>> >>> rather forced and difficult to explain, and then email back trying to
>>> >>> explain your impression (which is not always easy).
>>> >>
>>> >> If you seriously believe that looking at a few snippets is as helpful
>>> and
>>> >> instructive as being able to play around with them in IPython and
>>> modify
>>> >> them, then I guess we won't make progress in this part of the
>>> discussion.
>>> >> You're just telling me to go back and re-read things I'd already read.
>>> >
>>> > The snippets are in ipython or doctest format - aren't they?
>>>
>>> Oops - 10 minute rule.  No

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Olivier Delalleau]

2011/10/29 Ralf Gommers 

>
>
> On Sat, Oct 29, 2011 at 11:36 PM, Matthew Brett 
> wrote:
>
>> Hi,
>>
>> On Sat, Oct 29, 2011 at 1:48 PM, Matthew Brett 
>> wrote:
>> > Hi,
>> >
>> > On Sat, Oct 29, 2011 at 1:44 PM, Ralf Gommers
>> >  wrote:
>> >>
>> >>
>> >> On Sat, Oct 29, 2011 at 9:04 PM, Matthew Brett <
>> matthew.br...@gmail.com>
>> >> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> On Sat, Oct 29, 2011 at 3:26 AM, Ralf Gommers
>> >>>  wrote:
>> >>> >
>> >>> >
>> >>> > On Sat, Oct 29, 2011 at 1:37 AM, Matthew Brett <
>> matthew.br...@gmail.com>
>> >>> > wrote:
>> >>> >>
>> >>> >> Hi,
>> >>> >>
>> >>> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
>> >>> >>  wrote:
>> >>> >> >
>> >>> >> >
>> >>> >> > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett
>> >>> >> > 
>> >>> >> > wrote:
>> >>> >> >>
>> >>> >> >> Hi,
>> >>> >> >>
>> >>> >> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
>> >>> >> >>  wrote:
>> >>> >> >> >>
>> >>> >> >>
>> >>> >> >> No, that's not what Nathaniel and I are saying at all. Nathaniel
>> was
>> >>> >> >> pointing to links for projects that care that everyone agrees
>> before
>> >>> >> >> they go ahead.
>> >>> >> >
>> >>> >> > It looked to me like there was a serious intent to come to an
>> >>> >> > agreement,
>> >>> >> > or
>> >>> >> > at least closer together. The discussion in the summer was going
>> >>> >> > around
>> >>> >> > in
>> >>> >> > circles though, and was too abstract and complex to follow.
>> Therefore
>> >>> >> > Mark's
>> >>> >> > choice of implementing something and then asking for feedback
>> made
>> >>> >> > sense
>> >>> >> > to
>> >>> >> > me.
>> >>> >>
>> >>> >> I should point out that the implementation hasn't - as far as I can
>> >>> >> see - changed the discussion.  The discussion was about the API.
>> >>> >>
>> >>> >> Implementations are useful for agreed APIs because they can point
>> out
>> >>> >> where the API does not make sense or cannot be implemented.  In
>> this
>> >>> >> case, the API Mark said he was going to implement - he did
>> implement -
>> >>> >> at least as far as I can see.  Again, I'm happy to be corrected.
>> >>> >
>> >>> > Implementations can also help the discussion along, by allowing
>> people
>> >>> > to
>> >>> > try out some of the proposed changes. It also allows to construct
>> >>> > examples
>> >>> > that show weaknesses, possibly to be solved by an alternative API.
>> Maybe
>> >>> > you
>> >>> > can hold the complete history of this topic in your head and
>> comprehend
>> >>> > it,
>> >>> > but for me it would be very helpful if someone said:
>> >>> > - here's my dataset
>> >>> > - this is what I want to do with it
>> >>> > - this is the best I can do with the current implementation
>> >>> > - here's how API X would allow me to solve this better or simpler
>> >>> > This can be done much better with actual data and an actual
>> >>> > implementation
>> >>> > than with a design proposal. You seem to disagree with this
>> statement.
>> >>> > That's fine. I would hope though that you recognize that concrete
>> >>> > examples
>> >>> > help people like me, and construct one or two to help us out.
>> >>> That's what use-cases are for in designing APIs.  There are examples
>> >>> of use in the NEP:
>> >>>
>> >>> https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst
>> >>>
>> >>> the alterNEP:
>> >>>
>> >>> https://gist.github.com/1056379
>> >>>
>> >>> and my longer email to Travis:
>> >>>
>> >>>
>> >>>
>> http://article.gmane.org/gmane.comp.python.numeric.general/46544/match=ignored
>> >>>
>> >>> Mark has done a nice job of documentation:
>> >>>
>> >>> http://docs.scipy.org/doc/numpy/reference/arrays.maskna.html
>> >>>
>> >>> If you want to understand what the alterNEP case is, I'd suggest the
>> >>> email, just because it's the most recent and I think the terminology
>> >>> is slightly clearer.
>> >>>
>> >>> Doing the same examples on a larger array won't make the point easier
>> >>> to understand.  The discussion is about what the right concepts are,
>> >>> and you can help by looking at the snippets of code in those
>> >>> documents, and deciding for yourself whether you think the current
>> >>> masking / NA implementation seems natural and easy to explain, or
>> >>> rather forced and difficult to explain, and then email back trying to
>> >>> explain your impression (which is not always easy).
>> >>
>> >> If you seriously believe that looking at a few snippets is as helpful
>> and
>> >> instructive as being able to play around with them in IPython and
>> modify
>> >> them, then I guess we won't make progress in this part of the
>> discussion.
>> >> You're just telling me to go back and re-read things I'd already read.
>> >
>> > The snippets are in ipython or doctest format - aren't they?
>>
>> Oops - 10 minute rule.  Now I see that you mean that you can't
>> experiment with the alternative implementation without working code.
>>
>
> Indeed.
>
>
>> That's true, but I am hoping that the difference between - say:
>>
>> a[0:2] =

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Charles R Harris]

On Sat, Oct 29, 2011 at 3:55 PM, Matthew Brett wrote:

> Hi,
>
> On Sat, Oct 29, 2011 at 2:48 PM, Ralf Gommers
>  wrote:
> >
> >
> > On Sat, Oct 29, 2011 at 11:36 PM, Matthew Brett  >
> > wrote:
> >>
> >> Hi,
> >>
> >> On Sat, Oct 29, 2011 at 1:48 PM, Matthew Brett  >
> >> wrote:
> >> > Hi,
> >> >
> >> > On Sat, Oct 29, 2011 at 1:44 PM, Ralf Gommers
> >> >  wrote:
> >> >>
> >> >>
> >> >> On Sat, Oct 29, 2011 at 9:04 PM, Matthew Brett
> >> >> 
> >> >> wrote:
> >> >>>
> >> >>> Hi,
> >> >>>
> >> >>> On Sat, Oct 29, 2011 at 3:26 AM, Ralf Gommers
> >> >>>  wrote:
> >> >>> >
> >> >>> >
> >> >>> > On Sat, Oct 29, 2011 at 1:37 AM, Matthew Brett
> >> >>> > 
> >> >>> > wrote:
> >> >>> >>
> >> >>> >> Hi,
> >> >>> >>
> >> >>> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
> >> >>> >>  wrote:
> >> >>> >> >
> >> >>> >> >
> >> >>> >> > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett
> >> >>> >> > 
> >> >>> >> > wrote:
> >> >>> >> >>
> >> >>> >> >> Hi,
> >> >>> >> >>
> >> >>> >> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
> >> >>> >> >>  wrote:
> >> >>> >> >> >>
> >> >>> >> >>
> >> >>> >> >> No, that's not what Nathaniel and I are saying at all.
> Nathaniel
> >> >>> >> >> was
> >> >>> >> >> pointing to links for projects that care that everyone agrees
> >> >>> >> >> before
> >> >>> >> >> they go ahead.
> >> >>> >> >
> >> >>> >> > It looked to me like there was a serious intent to come to an
> >> >>> >> > agreement,
> >> >>> >> > or
> >> >>> >> > at least closer together. The discussion in the summer was
> going
> >> >>> >> > around
> >> >>> >> > in
> >> >>> >> > circles though, and was too abstract and complex to follow.
> >> >>> >> > Therefore
> >> >>> >> > Mark's
> >> >>> >> > choice of implementing something and then asking for feedback
> >> >>> >> > made
> >> >>> >> > sense
> >> >>> >> > to
> >> >>> >> > me.
> >> >>> >>
> >> >>> >> I should point out that the implementation hasn't - as far as I
> can
> >> >>> >> see - changed the discussion.  The discussion was about the API.
> >> >>> >>
> >> >>> >> Implementations are useful for agreed APIs because they can point
> >> >>> >> out
> >> >>> >> where the API does not make sense or cannot be implemented.  In
> >> >>> >> this
> >> >>> >> case, the API Mark said he was going to implement - he did
> >> >>> >> implement -
> >> >>> >> at least as far as I can see.  Again, I'm happy to be corrected.
> >> >>> >
> >> >>> > Implementations can also help the discussion along, by allowing
> >> >>> > people
> >> >>> > to
> >> >>> > try out some of the proposed changes. It also allows to construct
> >> >>> > examples
> >> >>> > that show weaknesses, possibly to be solved by an alternative API.
> >> >>> > Maybe
> >> >>> > you
> >> >>> > can hold the complete history of this topic in your head and
> >> >>> > comprehend
> >> >>> > it,
> >> >>> > but for me it would be very helpful if someone said:
> >> >>> > - here's my dataset
> >> >>> > - this is what I want to do with it
> >> >>> > - this is the best I can do with the current implementation
> >> >>> > - here's how API X would allow me to solve this better or simpler
> >> >>> > This can be done much better with actual data and an actual
> >> >>> > implementation
> >> >>> > than with a design proposal. You seem to disagree with this
> >> >>> > statement.
> >> >>> > That's fine. I would hope though that you recognize that concrete
> >> >>> > examples
> >> >>> > help people like me, and construct one or two to help us out.
> >> >>> That's what use-cases are for in designing APIs.  There are examples
> >> >>> of use in the NEP:
> >> >>>
> >> >>>
> https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst
> >> >>>
> >> >>> the alterNEP:
> >> >>>
> >> >>> https://gist.github.com/1056379
> >> >>>
> >> >>> and my longer email to Travis:
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> http://article.gmane.org/gmane.comp.python.numeric.general/46544/match=ignored
> >> >>>
> >> >>> Mark has done a nice job of documentation:
> >> >>>
> >> >>> http://docs.scipy.org/doc/numpy/reference/arrays.maskna.html
> >> >>>
> >> >>> If you want to understand what the alterNEP case is, I'd suggest the
> >> >>> email, just because it's the most recent and I think the terminology
> >> >>> is slightly clearer.
> >> >>>
> >> >>> Doing the same examples on a larger array won't make the point
> easier
> >> >>> to understand.  The discussion is about what the right concepts are,
> >> >>> and you can help by looking at the snippets of code in those
> >> >>> documents, and deciding for yourself whether you think the current
> >> >>> masking / NA implementation seems natural and easy to explain, or
> >> >>> rather forced and difficult to explain, and then email back trying
> to
> >> >>> explain your impression (which is not always easy).
> >> >>
> >> >> If you seriously believe that looking at a few snippets is as helpful
> >> >> and
> >> >> instructive as being able to play around with them in IPython and
> >> >> modify
> >> >> them, then I guess we won't ma

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Matthew Brett]

Hi,

On Sat, Oct 29, 2011 at 2:48 PM, Ralf Gommers
 wrote:
>
>
> On Sat, Oct 29, 2011 at 11:36 PM, Matthew Brett 
> wrote:
>>
>> Hi,
>>
>> On Sat, Oct 29, 2011 at 1:48 PM, Matthew Brett 
>> wrote:
>> > Hi,
>> >
>> > On Sat, Oct 29, 2011 at 1:44 PM, Ralf Gommers
>> >  wrote:
>> >>
>> >>
>> >> On Sat, Oct 29, 2011 at 9:04 PM, Matthew Brett
>> >> 
>> >> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> On Sat, Oct 29, 2011 at 3:26 AM, Ralf Gommers
>> >>>  wrote:
>> >>> >
>> >>> >
>> >>> > On Sat, Oct 29, 2011 at 1:37 AM, Matthew Brett
>> >>> > 
>> >>> > wrote:
>> >>> >>
>> >>> >> Hi,
>> >>> >>
>> >>> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
>> >>> >>  wrote:
>> >>> >> >
>> >>> >> >
>> >>> >> > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett
>> >>> >> > 
>> >>> >> > wrote:
>> >>> >> >>
>> >>> >> >> Hi,
>> >>> >> >>
>> >>> >> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
>> >>> >> >>  wrote:
>> >>> >> >> >>
>> >>> >> >>
>> >>> >> >> No, that's not what Nathaniel and I are saying at all. Nathaniel
>> >>> >> >> was
>> >>> >> >> pointing to links for projects that care that everyone agrees
>> >>> >> >> before
>> >>> >> >> they go ahead.
>> >>> >> >
>> >>> >> > It looked to me like there was a serious intent to come to an
>> >>> >> > agreement,
>> >>> >> > or
>> >>> >> > at least closer together. The discussion in the summer was going
>> >>> >> > around
>> >>> >> > in
>> >>> >> > circles though, and was too abstract and complex to follow.
>> >>> >> > Therefore
>> >>> >> > Mark's
>> >>> >> > choice of implementing something and then asking for feedback
>> >>> >> > made
>> >>> >> > sense
>> >>> >> > to
>> >>> >> > me.
>> >>> >>
>> >>> >> I should point out that the implementation hasn't - as far as I can
>> >>> >> see - changed the discussion.  The discussion was about the API.
>> >>> >>
>> >>> >> Implementations are useful for agreed APIs because they can point
>> >>> >> out
>> >>> >> where the API does not make sense or cannot be implemented.  In
>> >>> >> this
>> >>> >> case, the API Mark said he was going to implement - he did
>> >>> >> implement -
>> >>> >> at least as far as I can see.  Again, I'm happy to be corrected.
>> >>> >
>> >>> > Implementations can also help the discussion along, by allowing
>> >>> > people
>> >>> > to
>> >>> > try out some of the proposed changes. It also allows to construct
>> >>> > examples
>> >>> > that show weaknesses, possibly to be solved by an alternative API.
>> >>> > Maybe
>> >>> > you
>> >>> > can hold the complete history of this topic in your head and
>> >>> > comprehend
>> >>> > it,
>> >>> > but for me it would be very helpful if someone said:
>> >>> > - here's my dataset
>> >>> > - this is what I want to do with it
>> >>> > - this is the best I can do with the current implementation
>> >>> > - here's how API X would allow me to solve this better or simpler
>> >>> > This can be done much better with actual data and an actual
>> >>> > implementation
>> >>> > than with a design proposal. You seem to disagree with this
>> >>> > statement.
>> >>> > That's fine. I would hope though that you recognize that concrete
>> >>> > examples
>> >>> > help people like me, and construct one or two to help us out.
>> >>> That's what use-cases are for in designing APIs.  There are examples
>> >>> of use in the NEP:
>> >>>
>> >>> https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst
>> >>>
>> >>> the alterNEP:
>> >>>
>> >>> https://gist.github.com/1056379
>> >>>
>> >>> and my longer email to Travis:
>> >>>
>> >>>
>> >>>
>> >>> http://article.gmane.org/gmane.comp.python.numeric.general/46544/match=ignored
>> >>>
>> >>> Mark has done a nice job of documentation:
>> >>>
>> >>> http://docs.scipy.org/doc/numpy/reference/arrays.maskna.html
>> >>>
>> >>> If you want to understand what the alterNEP case is, I'd suggest the
>> >>> email, just because it's the most recent and I think the terminology
>> >>> is slightly clearer.
>> >>>
>> >>> Doing the same examples on a larger array won't make the point easier
>> >>> to understand.  The discussion is about what the right concepts are,
>> >>> and you can help by looking at the snippets of code in those
>> >>> documents, and deciding for yourself whether you think the current
>> >>> masking / NA implementation seems natural and easy to explain, or
>> >>> rather forced and difficult to explain, and then email back trying to
>> >>> explain your impression (which is not always easy).
>> >>
>> >> If you seriously believe that looking at a few snippets is as helpful
>> >> and
>> >> instructive as being able to play around with them in IPython and
>> >> modify
>> >> them, then I guess we won't make progress in this part of the
>> >> discussion.
>> >> You're just telling me to go back and re-read things I'd already read.
>> >
>> > The snippets are in ipython or doctest format - aren't they?
>>
>> Oops - 10 minute rule.  Now I see that you mean that you can't
>> experiment with the alternative implementation without working code.
>
> Indeed.
>

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Ralf Gommers]

On Sat, Oct 29, 2011 at 11:36 PM, Matthew Brett wrote:

> Hi,
>
> On Sat, Oct 29, 2011 at 1:48 PM, Matthew Brett 
> wrote:
> > Hi,
> >
> > On Sat, Oct 29, 2011 at 1:44 PM, Ralf Gommers
> >  wrote:
> >>
> >>
> >> On Sat, Oct 29, 2011 at 9:04 PM, Matthew Brett  >
> >> wrote:
> >>>
> >>> Hi,
> >>>
> >>> On Sat, Oct 29, 2011 at 3:26 AM, Ralf Gommers
> >>>  wrote:
> >>> >
> >>> >
> >>> > On Sat, Oct 29, 2011 at 1:37 AM, Matthew Brett <
> matthew.br...@gmail.com>
> >>> > wrote:
> >>> >>
> >>> >> Hi,
> >>> >>
> >>> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
> >>> >>  wrote:
> >>> >> >
> >>> >> >
> >>> >> > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett
> >>> >> > 
> >>> >> > wrote:
> >>> >> >>
> >>> >> >> Hi,
> >>> >> >>
> >>> >> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
> >>> >> >>  wrote:
> >>> >> >> >>
> >>> >> >>
> >>> >> >> No, that's not what Nathaniel and I are saying at all. Nathaniel
> was
> >>> >> >> pointing to links for projects that care that everyone agrees
> before
> >>> >> >> they go ahead.
> >>> >> >
> >>> >> > It looked to me like there was a serious intent to come to an
> >>> >> > agreement,
> >>> >> > or
> >>> >> > at least closer together. The discussion in the summer was going
> >>> >> > around
> >>> >> > in
> >>> >> > circles though, and was too abstract and complex to follow.
> Therefore
> >>> >> > Mark's
> >>> >> > choice of implementing something and then asking for feedback made
> >>> >> > sense
> >>> >> > to
> >>> >> > me.
> >>> >>
> >>> >> I should point out that the implementation hasn't - as far as I can
> >>> >> see - changed the discussion.  The discussion was about the API.
> >>> >>
> >>> >> Implementations are useful for agreed APIs because they can point
> out
> >>> >> where the API does not make sense or cannot be implemented.  In this
> >>> >> case, the API Mark said he was going to implement - he did implement
> -
> >>> >> at least as far as I can see.  Again, I'm happy to be corrected.
> >>> >
> >>> > Implementations can also help the discussion along, by allowing
> people
> >>> > to
> >>> > try out some of the proposed changes. It also allows to construct
> >>> > examples
> >>> > that show weaknesses, possibly to be solved by an alternative API.
> Maybe
> >>> > you
> >>> > can hold the complete history of this topic in your head and
> comprehend
> >>> > it,
> >>> > but for me it would be very helpful if someone said:
> >>> > - here's my dataset
> >>> > - this is what I want to do with it
> >>> > - this is the best I can do with the current implementation
> >>> > - here's how API X would allow me to solve this better or simpler
> >>> > This can be done much better with actual data and an actual
> >>> > implementation
> >>> > than with a design proposal. You seem to disagree with this
> statement.
> >>> > That's fine. I would hope though that you recognize that concrete
> >>> > examples
> >>> > help people like me, and construct one or two to help us out.
> >>> That's what use-cases are for in designing APIs.  There are examples
> >>> of use in the NEP:
> >>>
> >>> https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst
> >>>
> >>> the alterNEP:
> >>>
> >>> https://gist.github.com/1056379
> >>>
> >>> and my longer email to Travis:
> >>>
> >>>
> >>>
> http://article.gmane.org/gmane.comp.python.numeric.general/46544/match=ignored
> >>>
> >>> Mark has done a nice job of documentation:
> >>>
> >>> http://docs.scipy.org/doc/numpy/reference/arrays.maskna.html
> >>>
> >>> If you want to understand what the alterNEP case is, I'd suggest the
> >>> email, just because it's the most recent and I think the terminology
> >>> is slightly clearer.
> >>>
> >>> Doing the same examples on a larger array won't make the point easier
> >>> to understand.  The discussion is about what the right concepts are,
> >>> and you can help by looking at the snippets of code in those
> >>> documents, and deciding for yourself whether you think the current
> >>> masking / NA implementation seems natural and easy to explain, or
> >>> rather forced and difficult to explain, and then email back trying to
> >>> explain your impression (which is not always easy).
> >>
> >> If you seriously believe that looking at a few snippets is as helpful
> and
> >> instructive as being able to play around with them in IPython and modify
> >> them, then I guess we won't make progress in this part of the
> discussion.
> >> You're just telling me to go back and re-read things I'd already read.
> >
> > The snippets are in ipython or doctest format - aren't they?
>
> Oops - 10 minute rule.  Now I see that you mean that you can't
> experiment with the alternative implementation without working code.
>

Indeed.


> That's true, but I am hoping that the difference between - say:
>
> a[0:2] = np.NA
>
> and
>
> a.mask[0:2] = False
>
> would be easy enough to imagine.


It is in this case. I agree the explicit ``a.mask`` is clearer. This is a
quite specific point that could be improved in the current impl

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Matthew Brett]

Hi,

On Sat, Oct 29, 2011 at 1:48 PM, Matthew Brett  wrote:
> Hi,
>
> On Sat, Oct 29, 2011 at 1:44 PM, Ralf Gommers
>  wrote:
>>
>>
>> On Sat, Oct 29, 2011 at 9:04 PM, Matthew Brett 
>> wrote:
>>>
>>> Hi,
>>>
>>> On Sat, Oct 29, 2011 at 3:26 AM, Ralf Gommers
>>>  wrote:
>>> >
>>> >
>>> > On Sat, Oct 29, 2011 at 1:37 AM, Matthew Brett 
>>> > wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
>>> >>  wrote:
>>> >> >
>>> >> >
>>> >> > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett
>>> >> > 
>>> >> > wrote:
>>> >> >>
>>> >> >> Hi,
>>> >> >>
>>> >> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
>>> >> >>  wrote:
>>> >> >> >>
>>> >> >>
>>> >> >> No, that's not what Nathaniel and I are saying at all. Nathaniel was
>>> >> >> pointing to links for projects that care that everyone agrees before
>>> >> >> they go ahead.
>>> >> >
>>> >> > It looked to me like there was a serious intent to come to an
>>> >> > agreement,
>>> >> > or
>>> >> > at least closer together. The discussion in the summer was going
>>> >> > around
>>> >> > in
>>> >> > circles though, and was too abstract and complex to follow. Therefore
>>> >> > Mark's
>>> >> > choice of implementing something and then asking for feedback made
>>> >> > sense
>>> >> > to
>>> >> > me.
>>> >>
>>> >> I should point out that the implementation hasn't - as far as I can
>>> >> see - changed the discussion.  The discussion was about the API.
>>> >>
>>> >> Implementations are useful for agreed APIs because they can point out
>>> >> where the API does not make sense or cannot be implemented.  In this
>>> >> case, the API Mark said he was going to implement - he did implement -
>>> >> at least as far as I can see.  Again, I'm happy to be corrected.
>>> >
>>> > Implementations can also help the discussion along, by allowing people
>>> > to
>>> > try out some of the proposed changes. It also allows to construct
>>> > examples
>>> > that show weaknesses, possibly to be solved by an alternative API. Maybe
>>> > you
>>> > can hold the complete history of this topic in your head and comprehend
>>> > it,
>>> > but for me it would be very helpful if someone said:
>>> > - here's my dataset
>>> > - this is what I want to do with it
>>> > - this is the best I can do with the current implementation
>>> > - here's how API X would allow me to solve this better or simpler
>>> > This can be done much better with actual data and an actual
>>> > implementation
>>> > than with a design proposal. You seem to disagree with this statement.
>>> > That's fine. I would hope though that you recognize that concrete
>>> > examples
>>> > help people like me, and construct one or two to help us out.
>>> That's what use-cases are for in designing APIs.  There are examples
>>> of use in the NEP:
>>>
>>> https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst
>>>
>>> the alterNEP:
>>>
>>> https://gist.github.com/1056379
>>>
>>> and my longer email to Travis:
>>>
>>>
>>> http://article.gmane.org/gmane.comp.python.numeric.general/46544/match=ignored
>>>
>>> Mark has done a nice job of documentation:
>>>
>>> http://docs.scipy.org/doc/numpy/reference/arrays.maskna.html
>>>
>>> If you want to understand what the alterNEP case is, I'd suggest the
>>> email, just because it's the most recent and I think the terminology
>>> is slightly clearer.
>>>
>>> Doing the same examples on a larger array won't make the point easier
>>> to understand.  The discussion is about what the right concepts are,
>>> and you can help by looking at the snippets of code in those
>>> documents, and deciding for yourself whether you think the current
>>> masking / NA implementation seems natural and easy to explain, or
>>> rather forced and difficult to explain, and then email back trying to
>>> explain your impression (which is not always easy).
>>
>> If you seriously believe that looking at a few snippets is as helpful and
>> instructive as being able to play around with them in IPython and modify
>> them, then I guess we won't make progress in this part of the discussion.
>> You're just telling me to go back and re-read things I'd already read.
>
> The snippets are in ipython or doctest format - aren't they?

Oops - 10 minute rule.  Now I see that you mean that you can't
experiment with the alternative implementation without working code.
That's true, but I am hoping that the difference between - say:

a[0:2] = np.NA

and

a.mask[0:2] = False

would be easy enough to imagine.   If it isn't then, let me know,
preferably with something like "I can't see exactly how the following
[code snippet] would work in your conception of the problem" - and
then I can either try and give fake examples, or write a mock up.

Best,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Matthew Brett]

Hi,

On Sat, Oct 29, 2011 at 1:44 PM, Ralf Gommers
 wrote:
>
>
> On Sat, Oct 29, 2011 at 9:04 PM, Matthew Brett 
> wrote:
>>
>> Hi,
>>
>> On Sat, Oct 29, 2011 at 3:26 AM, Ralf Gommers
>>  wrote:
>> >
>> >
>> > On Sat, Oct 29, 2011 at 1:37 AM, Matthew Brett 
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
>> >>  wrote:
>> >> >
>> >> >
>> >> > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett
>> >> > 
>> >> > wrote:
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
>> >> >>  wrote:
>> >> >> >>
>> >> >>
>> >> >> No, that's not what Nathaniel and I are saying at all. Nathaniel was
>> >> >> pointing to links for projects that care that everyone agrees before
>> >> >> they go ahead.
>> >> >
>> >> > It looked to me like there was a serious intent to come to an
>> >> > agreement,
>> >> > or
>> >> > at least closer together. The discussion in the summer was going
>> >> > around
>> >> > in
>> >> > circles though, and was too abstract and complex to follow. Therefore
>> >> > Mark's
>> >> > choice of implementing something and then asking for feedback made
>> >> > sense
>> >> > to
>> >> > me.
>> >>
>> >> I should point out that the implementation hasn't - as far as I can
>> >> see - changed the discussion.  The discussion was about the API.
>> >>
>> >> Implementations are useful for agreed APIs because they can point out
>> >> where the API does not make sense or cannot be implemented.  In this
>> >> case, the API Mark said he was going to implement - he did implement -
>> >> at least as far as I can see.  Again, I'm happy to be corrected.
>> >
>> > Implementations can also help the discussion along, by allowing people
>> > to
>> > try out some of the proposed changes. It also allows to construct
>> > examples
>> > that show weaknesses, possibly to be solved by an alternative API. Maybe
>> > you
>> > can hold the complete history of this topic in your head and comprehend
>> > it,
>> > but for me it would be very helpful if someone said:
>> > - here's my dataset
>> > - this is what I want to do with it
>> > - this is the best I can do with the current implementation
>> > - here's how API X would allow me to solve this better or simpler
>> > This can be done much better with actual data and an actual
>> > implementation
>> > than with a design proposal. You seem to disagree with this statement.
>> > That's fine. I would hope though that you recognize that concrete
>> > examples
>> > help people like me, and construct one or two to help us out.
>> That's what use-cases are for in designing APIs.  There are examples
>> of use in the NEP:
>>
>> https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst
>>
>> the alterNEP:
>>
>> https://gist.github.com/1056379
>>
>> and my longer email to Travis:
>>
>>
>> http://article.gmane.org/gmane.comp.python.numeric.general/46544/match=ignored
>>
>> Mark has done a nice job of documentation:
>>
>> http://docs.scipy.org/doc/numpy/reference/arrays.maskna.html
>>
>> If you want to understand what the alterNEP case is, I'd suggest the
>> email, just because it's the most recent and I think the terminology
>> is slightly clearer.
>>
>> Doing the same examples on a larger array won't make the point easier
>> to understand.  The discussion is about what the right concepts are,
>> and you can help by looking at the snippets of code in those
>> documents, and deciding for yourself whether you think the current
>> masking / NA implementation seems natural and easy to explain, or
>> rather forced and difficult to explain, and then email back trying to
>> explain your impression (which is not always easy).
>
> If you seriously believe that looking at a few snippets is as helpful and
> instructive as being able to play around with them in IPython and modify
> them, then I guess we won't make progress in this part of the discussion.
> You're just telling me to go back and re-read things I'd already read.

The snippets are in ipython or doctest format - aren't they?

> OK, update: I took Ben's 10 minutes to go back and read the reference doc
> and your email again, just in case. The current implementation still seems
> natural to me to explain. It fits my use-cases. Perhaps that's different for
> you because you and I deal with different kinds of data. I don't have to
> explicitly treat absent and ignored data differently; those two are actually
> mixed and indistinguishable already in much of my data. Therefore the
> current implementation works well for me, having to make a distinction would
> be a needless complication.

OK - I'm not sure that contributes much to the discussion, because the
problem is being able to explain to each other in details why one
solution is preferable to another.  To follow your own advice, you'd
post some code snippets showing how you'd see the two ideas playing
out and why one is clearer than the other.

Best,

Matthew
___
NumPy-Discussion 

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Ralf Gommers]

On Sat, Oct 29, 2011 at 9:04 PM, Matthew Brett wrote:

> Hi,
>
> On Sat, Oct 29, 2011 at 3:26 AM, Ralf Gommers
>  wrote:
> >
> >
> > On Sat, Oct 29, 2011 at 1:37 AM, Matthew Brett 
> > wrote:
> >>
> >> Hi,
> >>
> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
> >>  wrote:
> >> >
> >> >
> >> > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett
> >> > 
> >> > wrote:
> >> >>
> >> >> Hi,
> >> >>
> >> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
> >> >>  wrote:
> >> >> >>
> >> >>
> >> >> No, that's not what Nathaniel and I are saying at all. Nathaniel was
> >> >> pointing to links for projects that care that everyone agrees before
> >> >> they go ahead.
> >> >
> >> > It looked to me like there was a serious intent to come to an
> agreement,
> >> > or
> >> > at least closer together. The discussion in the summer was going
> around
> >> > in
> >> > circles though, and was too abstract and complex to follow. Therefore
> >> > Mark's
> >> > choice of implementing something and then asking for feedback made
> sense
> >> > to
> >> > me.
> >>
> >> I should point out that the implementation hasn't - as far as I can
> >> see - changed the discussion.  The discussion was about the API.
> >>
> >> Implementations are useful for agreed APIs because they can point out
> >> where the API does not make sense or cannot be implemented.  In this
> >> case, the API Mark said he was going to implement - he did implement -
> >> at least as far as I can see.  Again, I'm happy to be corrected.
> >
> > Implementations can also help the discussion along, by allowing people to
> > try out some of the proposed changes. It also allows to construct
> examples
> > that show weaknesses, possibly to be solved by an alternative API. Maybe
> you
> > can hold the complete history of this topic in your head and comprehend
> it,
> > but for me it would be very helpful if someone said:
> > - here's my dataset
> > - this is what I want to do with it
> > - this is the best I can do with the current implementation
> > - here's how API X would allow me to solve this better or simpler
> > This can be done much better with actual data and an actual
> implementation
> > than with a design proposal. You seem to disagree with this statement.
> > That's fine. I would hope though that you recognize that concrete
> examples
> > help people like me, and construct one or two to help us out.
> That's what use-cases are for in designing APIs.  There are examples
> of use in the NEP:
>
> https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst
>
> the alterNEP:
>
> https://gist.github.com/1056379
>
> and my longer email to Travis:
>
>
> http://article.gmane.org/gmane.comp.python.numeric.general/46544/match=ignored
>
> Mark has done a nice job of documentation:
>
> http://docs.scipy.org/doc/numpy/reference/arrays.maskna.html
>
> If you want to understand what the alterNEP case is, I'd suggest the
> email, just because it's the most recent and I think the terminology
> is slightly clearer.
>
> Doing the same examples on a larger array won't make the point easier
> to understand.  The discussion is about what the right concepts are,
> and you can help by looking at the snippets of code in those
> documents, and deciding for yourself whether you think the current
> masking / NA implementation seems natural and easy to explain, or
> rather forced and difficult to explain, and then email back trying to
> explain your impression (which is not always easy).
>

If you seriously believe that looking at a few snippets is as helpful and
instructive as being able to play around with them in IPython and modify
them, then I guess we won't make progress in this part of the discussion.
You're just telling me to go back and re-read things I'd already read.

OK, update: I took Ben's 10 minutes to go back and read the reference doc
and your email again, just in case. The current implementation still seems
natural to me to explain. It fits my use-cases. Perhaps that's different for
you because you and I deal with different kinds of data. I don't have to
explicitly treat absent and ignored data differently; those two are actually
mixed and indistinguishable already in much of my data. Therefore the
current implementation works well for me, having to make a distinction would
be a needless complication.

Ralf
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Matthew Brett]

Hi,

On Sat, Oct 29, 2011 at 1:05 PM, Charles R Harris
 wrote:
>
>
> On Sat, Oct 29, 2011 at 1:41 PM, Benjamin Root  wrote:
>>
>>
>> On Saturday, October 29, 2011, Charles R Harris
>>  wrote:
>> >
>> > Who is counted in building a consensus? I tend to pay attention to those
>> > who have made consistent contributions over the years, reviewed code, fixed
>> > bugs, and have generally been active in numpy development. In any group
>> > participation is important, people who just walk in the door and demand
>> > things be done their way aren't going to get a lot of respect. I'll happily
>> > listen to politely expressed feedback, especially if the feedback comes 
>> > from
>> > someone who shows up to work, but that hasn't been my impression of the
>> > disagreements in this case. Heck, Nathaniel wasn't even tracking the Numpy
>> > pull requests or Mark's repository. That doesn't spell "participant" in my
>> > dictionary.
>> >
>> > Chuck
>> >
>>
>> This is a very good point, but I would highly caution against alienating
>> anybody here.  Frankly, I am surprised how much my opinion has been taken
>> here given the very little numpy code I have submitted (I think maybe two or
>> three patches).  The Numpy community is far more than just those who use the
>> core library. There is pandas, bottleneck, mpl, the scikits, and much more.
>>  Numpy would be nearly useless without them, and certainly vice versa.
>>
>
> I was quite impressed by your comments on Mark's work, I thought they were
> excellent. It doesn't really take much to make an impact in a small
> community overburdened by work.
>
>>
>> We are all indebted to each other for our works. We must never lose that
>> perspective.
>>
>> We all seem to have a different set of assumptions of how development
>> should work.  Each project follows its own workflow.  Numpy should be free
>> to adopt their own procedures, and we are free to discuss them.
>>
>> I do agree with chuck that he shouldn't have to make a written invitation
>> to each and every person to review each pull.  However, maybe some work can
>> be done to bring the pull request and issues discussion down to the mailing
>> list. I would like to do something similar with mpl.
>>
>> As for voting rights, let's make that a separate discussion.
>>
>
> With such a small community, I'd rather avoid the whole voting thing if
> possible.

But, if there is one thing worse than voting, it is implicit voting.
Implicit voting is where you ignore people who you don't think should
have a voice.  Unless I'm mistaken, that's what you are suggesting
should be the norm.

Best,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Matthew Brett]

Hi,

On Sat, Oct 29, 2011 at 12:41 PM, Charles R Harris
 wrote:
>
>
> On Sat, Oct 29, 2011 at 1:26 PM, Matthew Brett 
> wrote:
>>
>> Hi,
>>
>> On Sat, Oct 29, 2011 at 12:19 PM, Charles R Harris
>>  wrote:
>> >
>> >
>> > On Sat, Oct 29, 2011 at 1:04 PM, Matthew Brett 
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> On Sat, Oct 29, 2011 at 3:26 AM, Ralf Gommers
>> >>  wrote:
>> >> >
>> >> >
>> >> > On Sat, Oct 29, 2011 at 1:37 AM, Matthew Brett
>> >> > 
>> >> > wrote:
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
>> >> >>  wrote:
>> >> >> >
>> >> >> >
>> >> >> > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett
>> >> >> > 
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> Hi,
>> >> >> >>
>> >> >> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
>> >> >> >>  wrote:
>> >> >> >> >>
>> >> >> >>
>> >> >> >> No, that's not what Nathaniel and I are saying at all. Nathaniel
>> >> >> >> was
>> >> >> >> pointing to links for projects that care that everyone agrees
>> >> >> >> before
>> >> >> >> they go ahead.
>> >> >> >
>> >> >> > It looked to me like there was a serious intent to come to an
>> >> >> > agreement,
>> >> >> > or
>> >> >> > at least closer together. The discussion in the summer was going
>> >> >> > around
>> >> >> > in
>> >> >> > circles though, and was too abstract and complex to follow.
>> >> >> > Therefore
>> >> >> > Mark's
>> >> >> > choice of implementing something and then asking for feedback made
>> >> >> > sense
>> >> >> > to
>> >> >> > me.
>> >> >>
>> >> >> I should point out that the implementation hasn't - as far as I can
>> >> >> see - changed the discussion.  The discussion was about the API.
>> >> >>
>> >> >> Implementations are useful for agreed APIs because they can point
>> >> >> out
>> >> >> where the API does not make sense or cannot be implemented.  In this
>> >> >> case, the API Mark said he was going to implement - he did implement
>> >> >> -
>> >> >> at least as far as I can see.  Again, I'm happy to be corrected.
>> >> >
>> >> > Implementations can also help the discussion along, by allowing
>> >> > people
>> >> > to
>> >> > try out some of the proposed changes. It also allows to construct
>> >> > examples
>> >> > that show weaknesses, possibly to be solved by an alternative API.
>> >> > Maybe
>> >> > you
>> >> > can hold the complete history of this topic in your head and
>> >> > comprehend
>> >> > it,
>> >> > but for me it would be very helpful if someone said:
>> >> > - here's my dataset
>> >> > - this is what I want to do with it
>> >> > - this is the best I can do with the current implementation
>> >> > - here's how API X would allow me to solve this better or simpler
>> >> > This can be done much better with actual data and an actual
>> >> > implementation
>> >> > than with a design proposal. You seem to disagree with this
>> >> > statement.
>> >> > That's fine. I would hope though that you recognize that concrete
>> >> > examples
>> >> > help people like me, and construct one or two to help us out.
>> >> That's what use-cases are for in designing APIs.  There are examples
>> >> of use in the NEP:
>> >>
>> >> https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst
>> >>
>> >> the alterNEP:
>> >>
>> >> https://gist.github.com/1056379
>> >>
>> >> and my longer email to Travis:
>> >>
>> >>
>> >>
>> >> http://article.gmane.org/gmane.comp.python.numeric.general/46544/match=ignored
>> >>
>> >> Mark has done a nice job of documentation:
>> >>
>> >> http://docs.scipy.org/doc/numpy/reference/arrays.maskna.html
>> >>
>> >> If you want to understand what the alterNEP case is, I'd suggest the
>> >> email, just because it's the most recent and I think the terminology
>> >> is slightly clearer.
>> >>
>> >> Doing the same examples on a larger array won't make the point easier
>> >> to understand.  The discussion is about what the right concepts are,
>> >> and you can help by looking at the snippets of code in those
>> >> documents, and deciding for yourself whether you think the current
>> >> masking / NA implementation seems natural and easy to explain, or
>> >> rather forced and difficult to explain, and then email back trying to
>> >> explain your impression (which is not always easy).
>> >>
>> >> >> >> In saying that we are insisting on our way, you are saying,
>> >> >> >> implicitly,
>> >> >> >> 'I
>> >> >> >> am not going to negotiate'.
>> >> >> >
>> >> >> > That is only your interpretation. The observation that Mark
>> >> >> > compromised
>> >> >> > quite a bit while you didn't seems largely correct to me.
>> >> >>
>> >> >> The problem here stems from our inability to work towards agreement,
>> >> >> rather than standing on set positions.  I set out what changes I
>> >> >> think
>> >> >> would make the current implementation OK.  Can we please, please
>> >> >> have
>> >> >> a discussion about those points instead of trying to argue about who
>> >> >> has given more ground.
>> >> >>
>> >> >> > That commitment would of course be good. However, ev

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Charles R Harris]

On Sat, Oct 29, 2011 at 1:41 PM, Benjamin Root  wrote:

>
>
> On Saturday, October 29, 2011, Charles R Harris 
> wrote:
> >
> > Who is counted in building a consensus? I tend to pay attention to those
> who have made consistent contributions over the years, reviewed code, fixed
> bugs, and have generally been active in numpy development. In any group
> participation is important, people who just walk in the door and demand
> things be done their way aren't going to get a lot of respect. I'll happily
> listen to politely expressed feedback, especially if the feedback comes from
> someone who shows up to work, but that hasn't been my impression of the
> disagreements in this case. Heck, Nathaniel wasn't even tracking the Numpy
> pull requests or Mark's repository. That doesn't spell "participant" in my
> dictionary.
> >
> > Chuck
> >
>
> This is a very good point, but I would highly caution against alienating
> anybody here.  Frankly, I am surprised how much my opinion has been taken
> here given the very little numpy code I have submitted (I think maybe two or
> three patches).  The Numpy community is far more than just those who use the
> core library. There is pandas, bottleneck, mpl, the scikits, and much more.
>  Numpy would be nearly useless without them, and certainly vice versa.
>
>
I was quite impressed by your comments on Mark's work, I thought they were
excellent. It doesn't really take much to make an impact in a small
community overburdened by work.


> We are all indebted to each other for our works. We must never lose that
> perspective.
>
> We all seem to have a different set of assumptions of how development
> should work.  Each project follows its own workflow.  Numpy should be free
> to adopt their own procedures, and we are free to discuss them.
>
> I do agree with chuck that he shouldn't have to make a written invitation
> to each and every person to review each pull.  However, maybe some work can
> be done to bring the pull request and issues discussion down to the mailing
> list. I would like to do something similar with mpl.
>
> As for voting rights, let's make that a separate discussion.
>
>
With such a small community, I'd rather avoid the whole voting thing if
possible.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Charles R Harris]

On Sat, Oct 29, 2011 at 1:26 PM, Matthew Brett wrote:

> Hi,
>
> On Sat, Oct 29, 2011 at 12:19 PM, Charles R Harris
>  wrote:
> >
> >
> > On Sat, Oct 29, 2011 at 1:04 PM, Matthew Brett 
> > wrote:
> >>
> >> Hi,
> >>
> >> On Sat, Oct 29, 2011 at 3:26 AM, Ralf Gommers
> >>  wrote:
> >> >
> >> >
> >> > On Sat, Oct 29, 2011 at 1:37 AM, Matthew Brett <
> matthew.br...@gmail.com>
> >> > wrote:
> >> >>
> >> >> Hi,
> >> >>
> >> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
> >> >>  wrote:
> >> >> >
> >> >> >
> >> >> > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett
> >> >> > 
> >> >> > wrote:
> >> >> >>
> >> >> >> Hi,
> >> >> >>
> >> >> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
> >> >> >>  wrote:
> >> >> >> >>
> >> >> >>
> >> >> >> No, that's not what Nathaniel and I are saying at all. Nathaniel
> was
> >> >> >> pointing to links for projects that care that everyone agrees
> before
> >> >> >> they go ahead.
> >> >> >
> >> >> > It looked to me like there was a serious intent to come to an
> >> >> > agreement,
> >> >> > or
> >> >> > at least closer together. The discussion in the summer was going
> >> >> > around
> >> >> > in
> >> >> > circles though, and was too abstract and complex to follow.
> Therefore
> >> >> > Mark's
> >> >> > choice of implementing something and then asking for feedback made
> >> >> > sense
> >> >> > to
> >> >> > me.
> >> >>
> >> >> I should point out that the implementation hasn't - as far as I can
> >> >> see - changed the discussion.  The discussion was about the API.
> >> >>
> >> >> Implementations are useful for agreed APIs because they can point out
> >> >> where the API does not make sense or cannot be implemented.  In this
> >> >> case, the API Mark said he was going to implement - he did implement
> -
> >> >> at least as far as I can see.  Again, I'm happy to be corrected.
> >> >
> >> > Implementations can also help the discussion along, by allowing people
> >> > to
> >> > try out some of the proposed changes. It also allows to construct
> >> > examples
> >> > that show weaknesses, possibly to be solved by an alternative API.
> Maybe
> >> > you
> >> > can hold the complete history of this topic in your head and
> comprehend
> >> > it,
> >> > but for me it would be very helpful if someone said:
> >> > - here's my dataset
> >> > - this is what I want to do with it
> >> > - this is the best I can do with the current implementation
> >> > - here's how API X would allow me to solve this better or simpler
> >> > This can be done much better with actual data and an actual
> >> > implementation
> >> > than with a design proposal. You seem to disagree with this statement.
> >> > That's fine. I would hope though that you recognize that concrete
> >> > examples
> >> > help people like me, and construct one or two to help us out.
> >> That's what use-cases are for in designing APIs.  There are examples
> >> of use in the NEP:
> >>
> >> https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst
> >>
> >> the alterNEP:
> >>
> >> https://gist.github.com/1056379
> >>
> >> and my longer email to Travis:
> >>
> >>
> >>
> http://article.gmane.org/gmane.comp.python.numeric.general/46544/match=ignored
> >>
> >> Mark has done a nice job of documentation:
> >>
> >> http://docs.scipy.org/doc/numpy/reference/arrays.maskna.html
> >>
> >> If you want to understand what the alterNEP case is, I'd suggest the
> >> email, just because it's the most recent and I think the terminology
> >> is slightly clearer.
> >>
> >> Doing the same examples on a larger array won't make the point easier
> >> to understand.  The discussion is about what the right concepts are,
> >> and you can help by looking at the snippets of code in those
> >> documents, and deciding for yourself whether you think the current
> >> masking / NA implementation seems natural and easy to explain, or
> >> rather forced and difficult to explain, and then email back trying to
> >> explain your impression (which is not always easy).
> >>
> >> >> >> In saying that we are insisting on our way, you are saying,
> >> >> >> implicitly,
> >> >> >> 'I
> >> >> >> am not going to negotiate'.
> >> >> >
> >> >> > That is only your interpretation. The observation that Mark
> >> >> > compromised
> >> >> > quite a bit while you didn't seems largely correct to me.
> >> >>
> >> >> The problem here stems from our inability to work towards agreement,
> >> >> rather than standing on set positions.  I set out what changes I
> think
> >> >> would make the current implementation OK.  Can we please, please have
> >> >> a discussion about those points instead of trying to argue about who
> >> >> has given more ground.
> >> >>
> >> >> > That commitment would of course be good. However, even if that were
> >> >> > possible
> >> >> > before writing code and everyone agreed that the ideas of you and
> >> >> > Nathaniel
> >> >> > should be implemented in full, it's still not clear that either of
> >> >> > you
> >> >> > would
> >> >> > be willing to write any 

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Benjamin Root]

On Saturday, October 29, 2011, Charles R Harris 
wrote:
>
> Who is counted in building a consensus? I tend to pay attention to those
who have made consistent contributions over the years, reviewed code, fixed
bugs, and have generally been active in numpy development. In any group
participation is important, people who just walk in the door and demand
things be done their way aren't going to get a lot of respect. I'll happily
listen to politely expressed feedback, especially if the feedback comes from
someone who shows up to work, but that hasn't been my impression of the
disagreements in this case. Heck, Nathaniel wasn't even tracking the Numpy
pull requests or Mark's repository. That doesn't spell "participant" in my
dictionary.
>
> Chuck
>

This is a very good point, but I would highly caution against alienating
anybody here.  Frankly, I am surprised how much my opinion has been taken
here given the very little numpy code I have submitted (I think maybe two or
three patches).  The Numpy community is far more than just those who use the
core library. There is pandas, bottleneck, mpl, the scikits, and much more.
 Numpy would be nearly useless without them, and certainly vice versa.

We are all indebted to each other for our works. We must never lose that
perspective.

We all seem to have a different set of assumptions of how development should
work.  Each project follows its own workflow.  Numpy should be free to adopt
their own procedures, and we are free to discuss them.

I do agree with chuck that he shouldn't have to make a written invitation to
each and every person to review each pull.  However, maybe some work can be
done to bring the pull request and issues discussion down to the mailing
list. I would like to do something similar with mpl.

As for voting rights, let's make that a separate discussion.

Ben Root
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Matthew Brett]

Hi,

On Sat, Oct 29, 2011 at 12:19 PM, Charles R Harris
 wrote:
>
>
> On Sat, Oct 29, 2011 at 1:04 PM, Matthew Brett 
> wrote:
>>
>> Hi,
>>
>> On Sat, Oct 29, 2011 at 3:26 AM, Ralf Gommers
>>  wrote:
>> >
>> >
>> > On Sat, Oct 29, 2011 at 1:37 AM, Matthew Brett 
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
>> >>  wrote:
>> >> >
>> >> >
>> >> > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett
>> >> > 
>> >> > wrote:
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
>> >> >>  wrote:
>> >> >> >>
>> >> >>
>> >> >> No, that's not what Nathaniel and I are saying at all. Nathaniel was
>> >> >> pointing to links for projects that care that everyone agrees before
>> >> >> they go ahead.
>> >> >
>> >> > It looked to me like there was a serious intent to come to an
>> >> > agreement,
>> >> > or
>> >> > at least closer together. The discussion in the summer was going
>> >> > around
>> >> > in
>> >> > circles though, and was too abstract and complex to follow. Therefore
>> >> > Mark's
>> >> > choice of implementing something and then asking for feedback made
>> >> > sense
>> >> > to
>> >> > me.
>> >>
>> >> I should point out that the implementation hasn't - as far as I can
>> >> see - changed the discussion.  The discussion was about the API.
>> >>
>> >> Implementations are useful for agreed APIs because they can point out
>> >> where the API does not make sense or cannot be implemented.  In this
>> >> case, the API Mark said he was going to implement - he did implement -
>> >> at least as far as I can see.  Again, I'm happy to be corrected.
>> >
>> > Implementations can also help the discussion along, by allowing people
>> > to
>> > try out some of the proposed changes. It also allows to construct
>> > examples
>> > that show weaknesses, possibly to be solved by an alternative API. Maybe
>> > you
>> > can hold the complete history of this topic in your head and comprehend
>> > it,
>> > but for me it would be very helpful if someone said:
>> > - here's my dataset
>> > - this is what I want to do with it
>> > - this is the best I can do with the current implementation
>> > - here's how API X would allow me to solve this better or simpler
>> > This can be done much better with actual data and an actual
>> > implementation
>> > than with a design proposal. You seem to disagree with this statement.
>> > That's fine. I would hope though that you recognize that concrete
>> > examples
>> > help people like me, and construct one or two to help us out.
>> That's what use-cases are for in designing APIs.  There are examples
>> of use in the NEP:
>>
>> https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst
>>
>> the alterNEP:
>>
>> https://gist.github.com/1056379
>>
>> and my longer email to Travis:
>>
>>
>> http://article.gmane.org/gmane.comp.python.numeric.general/46544/match=ignored
>>
>> Mark has done a nice job of documentation:
>>
>> http://docs.scipy.org/doc/numpy/reference/arrays.maskna.html
>>
>> If you want to understand what the alterNEP case is, I'd suggest the
>> email, just because it's the most recent and I think the terminology
>> is slightly clearer.
>>
>> Doing the same examples on a larger array won't make the point easier
>> to understand.  The discussion is about what the right concepts are,
>> and you can help by looking at the snippets of code in those
>> documents, and deciding for yourself whether you think the current
>> masking / NA implementation seems natural and easy to explain, or
>> rather forced and difficult to explain, and then email back trying to
>> explain your impression (which is not always easy).
>>
>> >> >> In saying that we are insisting on our way, you are saying,
>> >> >> implicitly,
>> >> >> 'I
>> >> >> am not going to negotiate'.
>> >> >
>> >> > That is only your interpretation. The observation that Mark
>> >> > compromised
>> >> > quite a bit while you didn't seems largely correct to me.
>> >>
>> >> The problem here stems from our inability to work towards agreement,
>> >> rather than standing on set positions.  I set out what changes I think
>> >> would make the current implementation OK.  Can we please, please have
>> >> a discussion about those points instead of trying to argue about who
>> >> has given more ground.
>> >>
>> >> > That commitment would of course be good. However, even if that were
>> >> > possible
>> >> > before writing code and everyone agreed that the ideas of you and
>> >> > Nathaniel
>> >> > should be implemented in full, it's still not clear that either of
>> >> > you
>> >> > would
>> >> > be willing to write any code. Agreement without code still doesn't
>> >> > help
>> >> > us
>> >> > very much.
>> >>
>> >> I'm going to return to Nathaniel's point - it is a highly valuable
>> >> thing to set ourselves the target of resolving substantial discussions
>> >> by consensus.   The route you are endorsing here is 'implementor
>> >> wins'.
>> >
>> > I'm not. All I want to p

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Charles R Harris]

On Sat, Oct 29, 2011 at 1:04 PM, Matthew Brett wrote:

> Hi,
>
> On Sat, Oct 29, 2011 at 3:26 AM, Ralf Gommers
>  wrote:
> >
> >
> > On Sat, Oct 29, 2011 at 1:37 AM, Matthew Brett 
> > wrote:
> >>
> >> Hi,
> >>
> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
> >>  wrote:
> >> >
> >> >
> >> > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett
> >> > 
> >> > wrote:
> >> >>
> >> >> Hi,
> >> >>
> >> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
> >> >>  wrote:
> >> >> >>
> >> >>
> >> >> No, that's not what Nathaniel and I are saying at all. Nathaniel was
> >> >> pointing to links for projects that care that everyone agrees before
> >> >> they go ahead.
> >> >
> >> > It looked to me like there was a serious intent to come to an
> agreement,
> >> > or
> >> > at least closer together. The discussion in the summer was going
> around
> >> > in
> >> > circles though, and was too abstract and complex to follow. Therefore
> >> > Mark's
> >> > choice of implementing something and then asking for feedback made
> sense
> >> > to
> >> > me.
> >>
> >> I should point out that the implementation hasn't - as far as I can
> >> see - changed the discussion.  The discussion was about the API.
> >>
> >> Implementations are useful for agreed APIs because they can point out
> >> where the API does not make sense or cannot be implemented.  In this
> >> case, the API Mark said he was going to implement - he did implement -
> >> at least as far as I can see.  Again, I'm happy to be corrected.
> >
> > Implementations can also help the discussion along, by allowing people to
> > try out some of the proposed changes. It also allows to construct
> examples
> > that show weaknesses, possibly to be solved by an alternative API. Maybe
> you
> > can hold the complete history of this topic in your head and comprehend
> it,
> > but for me it would be very helpful if someone said:
> > - here's my dataset
> > - this is what I want to do with it
> > - this is the best I can do with the current implementation
> > - here's how API X would allow me to solve this better or simpler
> > This can be done much better with actual data and an actual
> implementation
> > than with a design proposal. You seem to disagree with this statement.
> > That's fine. I would hope though that you recognize that concrete
> examples
> > help people like me, and construct one or two to help us out.
> That's what use-cases are for in designing APIs.  There are examples
> of use in the NEP:
>
> https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst
>
> the alterNEP:
>
> https://gist.github.com/1056379
>
> and my longer email to Travis:
>
>
> http://article.gmane.org/gmane.comp.python.numeric.general/46544/match=ignored
>
> Mark has done a nice job of documentation:
>
> http://docs.scipy.org/doc/numpy/reference/arrays.maskna.html
>
> If you want to understand what the alterNEP case is, I'd suggest the
> email, just because it's the most recent and I think the terminology
> is slightly clearer.
>
> Doing the same examples on a larger array won't make the point easier
> to understand.  The discussion is about what the right concepts are,
> and you can help by looking at the snippets of code in those
> documents, and deciding for yourself whether you think the current
> masking / NA implementation seems natural and easy to explain, or
> rather forced and difficult to explain, and then email back trying to
> explain your impression (which is not always easy).
>
> >> >> In saying that we are insisting on our way, you are saying,
> implicitly,
> >> >> 'I
> >> >> am not going to negotiate'.
> >> >
> >> > That is only your interpretation. The observation that Mark
> compromised
> >> > quite a bit while you didn't seems largely correct to me.
> >>
> >> The problem here stems from our inability to work towards agreement,
> >> rather than standing on set positions.  I set out what changes I think
> >> would make the current implementation OK.  Can we please, please have
> >> a discussion about those points instead of trying to argue about who
> >> has given more ground.
> >>
> >> > That commitment would of course be good. However, even if that were
> >> > possible
> >> > before writing code and everyone agreed that the ideas of you and
> >> > Nathaniel
> >> > should be implemented in full, it's still not clear that either of you
> >> > would
> >> > be willing to write any code. Agreement without code still doesn't
> help
> >> > us
> >> > very much.
> >>
> >> I'm going to return to Nathaniel's point - it is a highly valuable
> >> thing to set ourselves the target of resolving substantial discussions
> >> by consensus.   The route you are endorsing here is 'implementor
> >> wins'.
> >
> > I'm not. All I want to point out is is that design and implementation are
> > not completely separated either.
>
> No, they often interact.  I was trying to explain why, in this case,
> the implementation hasn't changed the issues substantially, as far as
> I can see.   If you think

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Matthew Brett]

Hi,

On Sat, Oct 29, 2011 at 3:26 AM, Ralf Gommers
 wrote:
>
>
> On Sat, Oct 29, 2011 at 1:37 AM, Matthew Brett 
> wrote:
>>
>> Hi,
>>
>> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
>>  wrote:
>> >
>> >
>> > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett
>> > 
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
>> >>  wrote:
>> >> >>
>> >>
>> >> No, that's not what Nathaniel and I are saying at all. Nathaniel was
>> >> pointing to links for projects that care that everyone agrees before
>> >> they go ahead.
>> >
>> > It looked to me like there was a serious intent to come to an agreement,
>> > or
>> > at least closer together. The discussion in the summer was going around
>> > in
>> > circles though, and was too abstract and complex to follow. Therefore
>> > Mark's
>> > choice of implementing something and then asking for feedback made sense
>> > to
>> > me.
>>
>> I should point out that the implementation hasn't - as far as I can
>> see - changed the discussion.  The discussion was about the API.
>>
>> Implementations are useful for agreed APIs because they can point out
>> where the API does not make sense or cannot be implemented.  In this
>> case, the API Mark said he was going to implement - he did implement -
>> at least as far as I can see.  Again, I'm happy to be corrected.
>
> Implementations can also help the discussion along, by allowing people to
> try out some of the proposed changes. It also allows to construct examples
> that show weaknesses, possibly to be solved by an alternative API. Maybe you
> can hold the complete history of this topic in your head and comprehend it,
> but for me it would be very helpful if someone said:
> - here's my dataset
> - this is what I want to do with it
> - this is the best I can do with the current implementation
> - here's how API X would allow me to solve this better or simpler
> This can be done much better with actual data and an actual implementation
> than with a design proposal. You seem to disagree with this statement.
> That's fine. I would hope though that you recognize that concrete examples
> help people like me, and construct one or two to help us out.
That's what use-cases are for in designing APIs.  There are examples
of use in the NEP:

https://github.com/numpy/numpy/blob/master/doc/neps/missing-data.rst

the alterNEP:

https://gist.github.com/1056379

and my longer email to Travis:

http://article.gmane.org/gmane.comp.python.numeric.general/46544/match=ignored

Mark has done a nice job of documentation:

http://docs.scipy.org/doc/numpy/reference/arrays.maskna.html

If you want to understand what the alterNEP case is, I'd suggest the
email, just because it's the most recent and I think the terminology
is slightly clearer.

Doing the same examples on a larger array won't make the point easier
to understand.  The discussion is about what the right concepts are,
and you can help by looking at the snippets of code in those
documents, and deciding for yourself whether you think the current
masking / NA implementation seems natural and easy to explain, or
rather forced and difficult to explain, and then email back trying to
explain your impression (which is not always easy).

>> >> In saying that we are insisting on our way, you are saying, implicitly,
>> >> 'I
>> >> am not going to negotiate'.
>> >
>> > That is only your interpretation. The observation that Mark compromised
>> > quite a bit while you didn't seems largely correct to me.
>>
>> The problem here stems from our inability to work towards agreement,
>> rather than standing on set positions.  I set out what changes I think
>> would make the current implementation OK.  Can we please, please have
>> a discussion about those points instead of trying to argue about who
>> has given more ground.
>>
>> > That commitment would of course be good. However, even if that were
>> > possible
>> > before writing code and everyone agreed that the ideas of you and
>> > Nathaniel
>> > should be implemented in full, it's still not clear that either of you
>> > would
>> > be willing to write any code. Agreement without code still doesn't help
>> > us
>> > very much.
>>
>> I'm going to return to Nathaniel's point - it is a highly valuable
>> thing to set ourselves the target of resolving substantial discussions
>> by consensus.   The route you are endorsing here is 'implementor
>> wins'.
>
> I'm not. All I want to point out is is that design and implementation are
> not completely separated either.

No, they often interact.  I was trying to explain why, in this case,
the implementation hasn't changed the issues substantially, as far as
I can see.   If you think otherwise, then that is helpful information,
because you can feed back about where the initial discussion has been
overtaken by the implementation, and so we can strip down the
discussion to its essential parts.

>> We don't need to do it that way.  We're a mature sensible
>> bunch of adults
>
> Agreed:)

Ah - 

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Charles R Harris]

On Sat, Oct 29, 2011 at 12:14 PM, Wes McKinney  wrote:

> On Fri, Oct 28, 2011 at 9:32 PM, Charles R Harris
>  wrote:
> >
> >
> > On Fri, Oct 28, 2011 at 6:45 PM, Wes McKinney 
> wrote:
> >>
> >> On Fri, Oct 28, 2011 at 7:53 PM, Benjamin Root  wrote:
> >> >
> >> >
> >> > On Friday, October 28, 2011, Matthew Brett 
> >> > wrote:
> >> >> Hi,
> >> >>
> >> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
> >> >>  wrote:
> >> >>>
> >> >>>
> >> >>> On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett
> >> >>> 
> >> >>> wrote:
> >> 
> >>  Hi,
> >> 
> >>  On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
> >>   wrote:
> >>  >
> >>  >
> >>  > On Fri, Oct 28, 2011 at 3:56 PM, Matthew Brett
> >>  > 
> >>  > wrote:
> >>  >>
> >>  >> Hi,
> >>  >>
> >>  >> On Fri, Oct 28, 2011 at 2:43 PM, Matthew Brett
> >>  >> 
> >>  >> wrote:
> >>  >> > Hi,
> >>  >> >
> >>  >> > On Fri, Oct 28, 2011 at 2:41 PM, Charles R Harris
> >>  >> >  wrote:
> >>  >> >>
> >>  >> >>
> >>  >> >> On Fri, Oct 28, 2011 at 3:16 PM, Nathaniel Smith
> >>  >> >> 
> >>  >> >> wrote:
> >>  >> >>>
> >>  >> >>> On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant
> >>  >> >>> 
> >>  >> >>> wrote:
> >>  >> >>> > I think Nathaniel and Matthew provided very
> >>  >> >>> > specific feedback that was helpful in understanding other
> >>  >> >>> > perspectives
> >>  >> >>> > of a
> >>  >> >>> > difficult problem. In particular, I really wanted
> >>  >> >>> > bit-patterns
> >>  >> >>> > implemented.However, I also understand that Mark did
> >>  >> >>> > quite
> >>  >> >>> > a
> >>  >> >>> > bit
> >>  >> >>> > of
> >>  >> >>> > work
> >>  >> >>> > and altered his original designs quite a bit in response
> to
> >>  >> >>> > community
> >>  >> >>> > feedback.   I wasn't a major part of the pull request
> >>  >> >>> > discussion,
> >>  >> >>> > nor
> >>  >> >>> > did I
> >>  >> >>> > merge the changes, but I support Charles if he reviewed
> the
> >>  >> >>> > code
> >>  >> >>> > and
> >>  >> >>> > felt
> >>  >> >>> > like it was the right thing to do.  I likely would have
> done
> >>  >> >>> > the
> >>  >> >>> > same
> >>  >> >>> > thing
> >>  >> >>> > rather than let Mark Wiebe's work languish.
> >>  >> >>>
> >>  >> >>> My connectivity is spotty this week, so I'll stay out of the
> >>  >> >>> technical
> >>  >> >>> discussion for now, but I want to share a story.
> >>  >> >>>
> >>  >> >>> Maybe a year ago now, Jonathan Taylor and I were debating
> what
> >>  >> >>> the
> >>  >> >>> best API for describing statistical models would be --
> whether
> >>  >> >>> we
> >>  >> >>> wanted something like R's "formulas" (which I supported), or
> >>  >> >>> another
> >>  >> >>> approach based on sympy (his idea). To summarize, I thought
> >>  >> >>> his
> >>  >> >>> API
> >>  >> >>> was confusing, pointlessly complicated, and didn't actually
> >>  >> >>> solve
> >>  >> >>> the
> >>  >> >>> problem; he thought R-style formulas were superficially
> >>  >> >>> simpler
> >>  >> >>> but
> >>  >> >>> hopelessly confused and inconsistent underneath. Now,
> >>  >> >>> obviously,
> >>  >> >>> I
> >>  >> >>> was
> >>  >> >>> right and he was wrong. Well, obvious to me, anyway... ;-)
> But
> >>  >> >>> it
> >>  >> >>> wasn't like I could just wave a wand and make his arguments
> go
> >>  >> >>> away,
> >>  >> >>> no I should point out that the implementation hasn't - as
> far
> >>  >> >>> as
> >>  >> >>> I can
> >> >> see - changed the discussion.  The discussion was about the API.
> >> >> Implementations are useful for agreed APIs because they can point out
> >> >> where the API does not make sense or cannot be implemented.  In this
> >> >> case, the API Mark said he was going to implement - he did implement
> -
> >> >> at least as far as I can see.  Again, I'm happy to be corrected.
> >> >>
> >>  In saying that we are insisting on our way, you are saying,
> >>  implicitly,
> >>  'I
> >>  am not going to negotiate'.
> >> >>>
> >> >>> That is only your interpretation. The observation that Mark
> >> >>> compromised
> >> >>> quite a bit while you didn't seems largely correct to me.
> >> >>
> >> >> The problem here stems from our inability to work towards agreement,
> >> >> rather than standing on set positions.  I set out what changes I
> think
> >> >> would make the current implementation OK.  Can we please, please have
> >> >> a discussion about those points instead of trying to argue about who
> >> >> has given more ground.
> >> >>
> >> >>> That commitment would of course be good. However, even if that were
> >> >>> possible
> >> >>> before writing code and everyone agreed that the ideas of you and
> >> >>> Nathaniel
> >> >>> should b

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Matthew Brett]

Hi,

On Fri, Oct 28, 2011 at 8:38 PM, Benjamin Root  wrote:
> Matt,
>
> On Friday, October 28, 2011, Matthew Brett  wrote:
>>
>>> Forget about rudeness or decision processes.
>>
>> No, that's a common mistake, which is to assume that any conversation
>> about things which aren't technical, is not important.   Nathaniel's
>> point is important.  Rudeness is important. The reason we've got into
>> this mess is because we clearly don't have an agreed way of making
>> decisions.  That's why countries and open-source projects have
>> constitutions, so this doesn't happen.
>
> Don't get me wrong. In general, you are right.  And maybe we all should
> discuss something to that effect for numpy.  But I would rather do that when
> there isn't such contention and tempers.

That's a reasonable point.

> As for allegations of rudeness, I believe that we are actually very close to
> consensus that I immediately wanted to squelch any sort of
> meta-meta-disagreements about who was being rude to who.  As a quick
> band-aide, anybody who felt slighted by me gets a drink on me at the next
> scipy conference.  From this point on, let's institute a 10 minute rule --
> write your email, wait ten minutes, read it again and edit it.

Good offer.  I make the same one.

>>> I will start by saying that I am willing to separate ignore and absent,
>>> but
>>> only on the write side of things.  On read, I want a single way to
>>> identify
>>> the missing values.  I also want only a single way to perform
>>> calculations
>>> (either skip or propagate).
>>
>> Thank you - that is very helpful.
>>
>> Are you saying that you'd be OK setting missing values like this?
>>
> a.mask[0:2] = False
>>
>
> Probably not that far, because that would be an attribute that may or may
> not exist.  Rather, I might like the idea of a NA to "always" mean absent
> (and destroys - even through views), and MA (or some other name) which
> always means ignore (and has the masking behavior with views). This makes
> specific behaviors tied distinctly to specific objects.

Ah - yes - thank you.  I think you and I at least have somewhere to go
for agreement, but, I don't know how to work towards a numpy-wide
agreement.  Do you have any thoughts?

>> For the read side, do you mean you're OK with this
>>
> a.isna()
>>
>> To identify the missing values, as is currently the case?  Or something
>> else?
>>
>
> Yes.  A missing value is a missing value, regardless of it being absent or
> marked as ignored.  But it is a bit more subtle than that.  I should just be
> able to add two arrays together and the "data should know what to do". When
> the core ufuncs get this right (like min, max, sum, cumsum, diff, etc), then
> I don't have to do much to prepare higher level funcs for missing data.
>
>> If so, then I think we're very close, it's just a discussion about names.
>>
>
> And what does ignore + absent equals. ;-)

ignore + absent == special_value_of_some_sort :)

Just joking,

See you,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Wes McKinney]

On Fri, Oct 28, 2011 at 9:32 PM, Charles R Harris
 wrote:
>
>
> On Fri, Oct 28, 2011 at 6:45 PM, Wes McKinney  wrote:
>>
>> On Fri, Oct 28, 2011 at 7:53 PM, Benjamin Root  wrote:
>> >
>> >
>> > On Friday, October 28, 2011, Matthew Brett 
>> > wrote:
>> >> Hi,
>> >>
>> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
>> >>  wrote:
>> >>>
>> >>>
>> >>> On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett
>> >>> 
>> >>> wrote:
>> 
>>  Hi,
>> 
>>  On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
>>   wrote:
>>  >
>>  >
>>  > On Fri, Oct 28, 2011 at 3:56 PM, Matthew Brett
>>  > 
>>  > wrote:
>>  >>
>>  >> Hi,
>>  >>
>>  >> On Fri, Oct 28, 2011 at 2:43 PM, Matthew Brett
>>  >> 
>>  >> wrote:
>>  >> > Hi,
>>  >> >
>>  >> > On Fri, Oct 28, 2011 at 2:41 PM, Charles R Harris
>>  >> >  wrote:
>>  >> >>
>>  >> >>
>>  >> >> On Fri, Oct 28, 2011 at 3:16 PM, Nathaniel Smith
>>  >> >> 
>>  >> >> wrote:
>>  >> >>>
>>  >> >>> On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant
>>  >> >>> 
>>  >> >>> wrote:
>>  >> >>> > I think Nathaniel and Matthew provided very
>>  >> >>> > specific feedback that was helpful in understanding other
>>  >> >>> > perspectives
>>  >> >>> > of a
>>  >> >>> > difficult problem.     In particular, I really wanted
>>  >> >>> > bit-patterns
>>  >> >>> > implemented.    However, I also understand that Mark did
>>  >> >>> > quite
>>  >> >>> > a
>>  >> >>> > bit
>>  >> >>> > of
>>  >> >>> > work
>>  >> >>> > and altered his original designs quite a bit in response to
>>  >> >>> > community
>>  >> >>> > feedback.   I wasn't a major part of the pull request
>>  >> >>> > discussion,
>>  >> >>> > nor
>>  >> >>> > did I
>>  >> >>> > merge the changes, but I support Charles if he reviewed the
>>  >> >>> > code
>>  >> >>> > and
>>  >> >>> > felt
>>  >> >>> > like it was the right thing to do.  I likely would have done
>>  >> >>> > the
>>  >> >>> > same
>>  >> >>> > thing
>>  >> >>> > rather than let Mark Wiebe's work languish.
>>  >> >>>
>>  >> >>> My connectivity is spotty this week, so I'll stay out of the
>>  >> >>> technical
>>  >> >>> discussion for now, but I want to share a story.
>>  >> >>>
>>  >> >>> Maybe a year ago now, Jonathan Taylor and I were debating what
>>  >> >>> the
>>  >> >>> best API for describing statistical models would be -- whether
>>  >> >>> we
>>  >> >>> wanted something like R's "formulas" (which I supported), or
>>  >> >>> another
>>  >> >>> approach based on sympy (his idea). To summarize, I thought
>>  >> >>> his
>>  >> >>> API
>>  >> >>> was confusing, pointlessly complicated, and didn't actually
>>  >> >>> solve
>>  >> >>> the
>>  >> >>> problem; he thought R-style formulas were superficially
>>  >> >>> simpler
>>  >> >>> but
>>  >> >>> hopelessly confused and inconsistent underneath. Now,
>>  >> >>> obviously,
>>  >> >>> I
>>  >> >>> was
>>  >> >>> right and he was wrong. Well, obvious to me, anyway... ;-) But
>>  >> >>> it
>>  >> >>> wasn't like I could just wave a wand and make his arguments go
>>  >> >>> away,
>>  >> >>> no I should point out that the implementation hasn't - as far
>>  >> >>> as
>>  >> >>> I can
>> >> see - changed the discussion.  The discussion was about the API.
>> >> Implementations are useful for agreed APIs because they can point out
>> >> where the API does not make sense or cannot be implemented.  In this
>> >> case, the API Mark said he was going to implement - he did implement -
>> >> at least as far as I can see.  Again, I'm happy to be corrected.
>> >>
>>  In saying that we are insisting on our way, you are saying,
>>  implicitly,
>>  'I
>>  am not going to negotiate'.
>> >>>
>> >>> That is only your interpretation. The observation that Mark
>> >>> compromised
>> >>> quite a bit while you didn't seems largely correct to me.
>> >>
>> >> The problem here stems from our inability to work towards agreement,
>> >> rather than standing on set positions.  I set out what changes I think
>> >> would make the current implementation OK.  Can we please, please have
>> >> a discussion about those points instead of trying to argue about who
>> >> has given more ground.
>> >>
>> >>> That commitment would of course be good. However, even if that were
>> >>> possible
>> >>> before writing code and everyone agreed that the ideas of you and
>> >>> Nathaniel
>> >>> should be implemented in full, it's still not clear that either of you
>> >>> would
>> >>> be willing to write any code. Agreement without code still doesn't
>> >>> help
>> >>> us
>> >>> very much.
>> >>
>> >> I'm going to return to Nathaniel's point - it is a highly valuable
>> >> thing to set ourselves the target of resolving substantial discussion

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Eric Firing]

On 10/29/2011 12:26 AM, Ralf Gommers wrote:
> The history of this discussion doesn't suggest it straightforward to get
> a design right first time. It's a complex subject.
>
> The second part of your statement, "and then implement", sounds so
> simple. The reality is that there are only a handful of developers who
> have done a significant amount of work on the numpy core in the last two
> years. I haven't seen anyone saying they are planning to implement (part
> of) whatever design the outcome of this discussion will be. I don't
> think it's strange to keep this in mind to some extent.

...including the fact that last summer, Mark had a brief one-time 
opportunity to contribute major NA code.  I expect that even if some 
modifications are made to what he contributed, letting him get on with 
it will turn out to have been the right move.

Apparently Travis hopes to put in a burst of coding in 2012:

http://technicaldiscovery.blogspot.com/2011/10/thoughts-on-porting-numpy-to-pypy.html

Go to the section "NumPy will be evolving rapidly over the coming 
years".  Note that "missing data bit-patterns" is on his list, 
consistent with his most recent messages.

Eric
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Ralf Gommers]

On Sat, Oct 29, 2011 at 1:37 AM, Matthew Brett wrote:

> Hi,
>
> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
>  wrote:
> >
> >
> > On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett  >
> > wrote:
> >>
> >> Hi,
> >>
> >> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
> >>  wrote:
> >> >>
> >>
> >> No, that's not what Nathaniel and I are saying at all. Nathaniel was
> >> pointing to links for projects that care that everyone agrees before
> >> they go ahead.
> >
> > It looked to me like there was a serious intent to come to an agreement,
> or
> > at least closer together. The discussion in the summer was going around
> in
> > circles though, and was too abstract and complex to follow. Therefore
> Mark's
> > choice of implementing something and then asking for feedback made sense
> to
> > me.
>
> I should point out that the implementation hasn't - as far as I can
> see - changed the discussion.  The discussion was about the API.
>
Implementations are useful for agreed APIs because they can point out
> where the API does not make sense or cannot be implemented.  In this
> case, the API Mark said he was going to implement - he did implement -
> at least as far as I can see.  Again, I'm happy to be corrected.
>

Implementations can also help the discussion along, by allowing people to
try out some of the proposed changes. It also allows to construct examples
that show weaknesses, possibly to be solved by an alternative API. Maybe you
can hold the complete history of this topic in your head and comprehend it,
but for me it would be very helpful if someone said:
- here's my dataset
- this is what I want to do with it
- this is the best I can do with the current implementation
- here's how API X would allow me to solve this better or simpler

This can be done much better with actual data and an actual implementation
than with a design proposal. You seem to disagree with this statement.
That's fine. I would hope though that you recognize that concrete examples
help people like me, and construct one or two to help us out.

>
> >> In saying that we are insisting on our way, you are saying, implicitly,
> 'I
> >> am not going to negotiate'.
> >
> > That is only your interpretation. The observation that Mark compromised
> > quite a bit while you didn't seems largely correct to me.
>
> The problem here stems from our inability to work towards agreement,
> rather than standing on set positions.  I set out what changes I think
> would make the current implementation OK.  Can we please, please have
> a discussion about those points instead of trying to argue about who
> has given more ground.
>
> > That commitment would of course be good. However, even if that were
> possible
> > before writing code and everyone agreed that the ideas of you and
> Nathaniel
> > should be implemented in full, it's still not clear that either of you
> would
> > be willing to write any code. Agreement without code still doesn't help
> us
> > very much.
>
> I'm going to return to Nathaniel's point - it is a highly valuable
> thing to set ourselves the target of resolving substantial discussions
> by consensus.   The route you are endorsing here is 'implementor
> wins'.


I'm not. All I want to point out is is that design and implementation are
not completely separated either.


> We don't need to do it that way.  We're a mature sensible
> bunch of adults


Agreed:)


> who can talk out the issues until we agree they are
> ready for implementation, and then implement.


The history of this discussion doesn't suggest it straightforward to get a
design right first time. It's a complex subject.

The second part of your statement, "and then implement", sounds so simple.
The reality is that there are only a handful of developers who have done a
significant amount of work on the numpy core in the last two years. I
haven't seen anyone saying they are planning to implement (part of) whatever
design the outcome of this discussion will be. I don't think it's strange to
keep this in mind to some extent.

Regards,
Ralf
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Ralf Gommers]

On Sat, Oct 29, 2011 at 3:32 AM, Charles R Harris  wrote:

>
>
> On Fri, Oct 28, 2011 at 6:45 PM, Wes McKinney  wrote:
>
>> On Fri, Oct 28, 2011 at 7:53 PM, Benjamin Root  wrote:
>> >
>> >
>> > On Friday, October 28, 2011, Matthew Brett 
>> wrote:
>> >> Hi,
>> >>
>> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
>> >>  wrote:
>> >>>
>> >>>
>> >>> On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett <
>> matthew.br...@gmail.com>
>> >>> wrote:
>> 
>>  Hi,
>> 
>>  On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
>>   wrote:
>>  >
>>  >
>>  > On Fri, Oct 28, 2011 at 3:56 PM, Matthew Brett
>>  > 
>>  > wrote:
>>  >>
>>  >> Hi,
>>  >>
>>  >> On Fri, Oct 28, 2011 at 2:43 PM, Matthew Brett
>>  >> 
>>  >> wrote:
>>  >> > Hi,
>>  >> >
>>  >> > On Fri, Oct 28, 2011 at 2:41 PM, Charles R Harris
>>  >> >  wrote:
>>  >> >>
>>  >> >>
>>  >> >> On Fri, Oct 28, 2011 at 3:16 PM, Nathaniel Smith <
>> n...@pobox.com>
>>  >> >> wrote:
>>  >> >>>
>>  >> >>> On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant
>>  >> >>> 
>>  >> >>> wrote:
>>  >> >>> > I think Nathaniel and Matthew provided very
>>  >> >>> > specific feedback that was helpful in understanding other
>>  >> >>> > perspectives
>>  >> >>> > of a
>>  >> >>> > difficult problem. In particular, I really wanted
>>  >> >>> > bit-patterns
>>  >> >>> > implemented.However, I also understand that Mark did
>> quite
>>  >> >>> > a
>>  >> >>> > bit
>>  >> >>> > of
>>  >> >>> > work
>>  >> >>> > and altered his original designs quite a bit in response to
>>  >> >>> > community
>>  >> >>> > feedback.   I wasn't a major part of the pull request
>>  >> >>> > discussion,
>>  >> >>> > nor
>>  >> >>> > did I
>>  >> >>> > merge the changes, but I support Charles if he reviewed the
>>  >> >>> > code
>>  >> >>> > and
>>  >> >>> > felt
>>  >> >>> > like it was the right thing to do.  I likely would have done
>>  >> >>> > the
>>  >> >>> > same
>>  >> >>> > thing
>>  >> >>> > rather than let Mark Wiebe's work languish.
>>  >> >>>
>>  >> >>> My connectivity is spotty this week, so I'll stay out of the
>>  >> >>> technical
>>  >> >>> discussion for now, but I want to share a story.
>>  >> >>>
>>  >> >>> Maybe a year ago now, Jonathan Taylor and I were debating what
>>  >> >>> the
>>  >> >>> best API for describing statistical models would be -- whether
>> we
>>  >> >>> wanted something like R's "formulas" (which I supported), or
>>  >> >>> another
>>  >> >>> approach based on sympy (his idea). To summarize, I thought
>> his
>>  >> >>> API
>>  >> >>> was confusing, pointlessly complicated, and didn't actually
>> solve
>>  >> >>> the
>>  >> >>> problem; he thought R-style formulas were superficially
>> simpler
>>  >> >>> but
>>  >> >>> hopelessly confused and inconsistent underneath. Now,
>> obviously,
>>  >> >>> I
>>  >> >>> was
>>  >> >>> right and he was wrong. Well, obvious to me, anyway... ;-) But
>> it
>>  >> >>> wasn't like I could just wave a wand and make his arguments go
>>  >> >>> away,
>>  >> >>> no I should point out that the implementation hasn't - as far
>> as
>>  >> >>> I can
>> >> see - changed the discussion.  The discussion was about the API.
>> >> Implementations are useful for agreed APIs because they can point out
>> >> where the API does not make sense or cannot be implemented.  In this
>> >> case, the API Mark said he was going to implement - he did implement -
>> >> at least as far as I can see.  Again, I'm happy to be corrected.
>> >>
>>  In saying that we are insisting on our way, you are saying,
>> implicitly,
>>  'I
>>  am not going to negotiate'.
>> >>>
>> >>> That is only your interpretation. The observation that Mark
>> compromised
>> >>> quite a bit while you didn't seems largely correct to me.
>> >>
>> >> The problem here stems from our inability to work towards agreement,
>> >> rather than standing on set positions.  I set out what changes I think
>> >> would make the current implementation OK.  Can we please, please have
>> >> a discussion about those points instead of trying to argue about who
>> >> has given more ground.
>> >>
>> >>> That commitment would of course be good. However, even if that were
>> >>> possible
>> >>> before writing code and everyone agreed that the ideas of you and
>> >>> Nathaniel
>> >>> should be implemented in full, it's still not clear that either of you
>> >>> would
>> >>> be willing to write any code. Agreement without code still doesn't
>> help
>> >>> us
>> >>> very much.
>> >>
>> >> I'm going to return to Nathaniel's point - it is a highly valuable
>> >> thing to set ourselves the target of resolving substantial discussions
>> >> by consensus.   The route you are endorsing here is 'implementor
>> >> wins'.

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Han Genuit]

Hi, instead of putting up a pull request that reverts all the 25000
lines of code than have been written to support an NA mask, why won't
you set up a pull request that uses the current code base to implement
your own ideas on how it should work?
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Nathaniel Smith]

On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
 wrote:
> Matthew, the problem I have is that it seems that you and Nathaniel won't be
> satisfied unless things are done *your* way.

Hi Charles,

I'm sorry if I've given this impression, and I know it's easy to feel
this way in a contentious discussion. I've even been tempted to
conclude the same about you, based on some of those emails in the last
discussion where you told us that we should only give feedback by
critiquing specific paragraphs of Mark's docs, even though our issues
were with the whole architecture he was suggesting. But I'd like to
believe that that isn't true of you, and in return, I can only point
out the following things:

1) I've actually made a number of different suggestions and attempts
to find ways to compromise (e.g., the "NA concepts" discussion, the
alter-NEP that folded in a design for "ignored" values to try and
satisfy that constituency even though I wouldn't use them myself, and
on the conference call trying to find a subset of features that we
could all agree on to implement first). I don't *want* my proposals
implemented unless everyone else finds them persuasive.

2) This is why in my message I'm *not* advocating that we implement
NAs according to my proposals; I'm advocating that you get just as
much of a veto power on my proposals as I do on yours. Let's be
honest: we both know all else being equal, we'd both rather not deal
with the other right now, and might prefer not to hang out socially.
But if we want this NA stuff to actually work and be used, then we
need to find a way to work together despite that.

Peace?

-- Nathaniel
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Charles R Harris]

On Fri, Oct 28, 2011 at 6:45 PM, Wes McKinney  wrote:

> On Fri, Oct 28, 2011 at 7:53 PM, Benjamin Root  wrote:
> >
> >
> > On Friday, October 28, 2011, Matthew Brett 
> wrote:
> >> Hi,
> >>
> >> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
> >>  wrote:
> >>>
> >>>
> >>> On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett <
> matthew.br...@gmail.com>
> >>> wrote:
> 
>  Hi,
> 
>  On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
>   wrote:
>  >
>  >
>  > On Fri, Oct 28, 2011 at 3:56 PM, Matthew Brett
>  > 
>  > wrote:
>  >>
>  >> Hi,
>  >>
>  >> On Fri, Oct 28, 2011 at 2:43 PM, Matthew Brett
>  >> 
>  >> wrote:
>  >> > Hi,
>  >> >
>  >> > On Fri, Oct 28, 2011 at 2:41 PM, Charles R Harris
>  >> >  wrote:
>  >> >>
>  >> >>
>  >> >> On Fri, Oct 28, 2011 at 3:16 PM, Nathaniel Smith  >
>  >> >> wrote:
>  >> >>>
>  >> >>> On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant
>  >> >>> 
>  >> >>> wrote:
>  >> >>> > I think Nathaniel and Matthew provided very
>  >> >>> > specific feedback that was helpful in understanding other
>  >> >>> > perspectives
>  >> >>> > of a
>  >> >>> > difficult problem. In particular, I really wanted
>  >> >>> > bit-patterns
>  >> >>> > implemented.However, I also understand that Mark did
> quite
>  >> >>> > a
>  >> >>> > bit
>  >> >>> > of
>  >> >>> > work
>  >> >>> > and altered his original designs quite a bit in response to
>  >> >>> > community
>  >> >>> > feedback.   I wasn't a major part of the pull request
>  >> >>> > discussion,
>  >> >>> > nor
>  >> >>> > did I
>  >> >>> > merge the changes, but I support Charles if he reviewed the
>  >> >>> > code
>  >> >>> > and
>  >> >>> > felt
>  >> >>> > like it was the right thing to do.  I likely would have done
>  >> >>> > the
>  >> >>> > same
>  >> >>> > thing
>  >> >>> > rather than let Mark Wiebe's work languish.
>  >> >>>
>  >> >>> My connectivity is spotty this week, so I'll stay out of the
>  >> >>> technical
>  >> >>> discussion for now, but I want to share a story.
>  >> >>>
>  >> >>> Maybe a year ago now, Jonathan Taylor and I were debating what
>  >> >>> the
>  >> >>> best API for describing statistical models would be -- whether
> we
>  >> >>> wanted something like R's "formulas" (which I supported), or
>  >> >>> another
>  >> >>> approach based on sympy (his idea). To summarize, I thought his
>  >> >>> API
>  >> >>> was confusing, pointlessly complicated, and didn't actually
> solve
>  >> >>> the
>  >> >>> problem; he thought R-style formulas were superficially simpler
>  >> >>> but
>  >> >>> hopelessly confused and inconsistent underneath. Now,
> obviously,
>  >> >>> I
>  >> >>> was
>  >> >>> right and he was wrong. Well, obvious to me, anyway... ;-) But
> it
>  >> >>> wasn't like I could just wave a wand and make his arguments go
>  >> >>> away,
>  >> >>> no I should point out that the implementation hasn't - as far
> as
>  >> >>> I can
> >> see - changed the discussion.  The discussion was about the API.
> >> Implementations are useful for agreed APIs because they can point out
> >> where the API does not make sense or cannot be implemented.  In this
> >> case, the API Mark said he was going to implement - he did implement -
> >> at least as far as I can see.  Again, I'm happy to be corrected.
> >>
>  In saying that we are insisting on our way, you are saying,
> implicitly,
>  'I
>  am not going to negotiate'.
> >>>
> >>> That is only your interpretation. The observation that Mark compromised
> >>> quite a bit while you didn't seems largely correct to me.
> >>
> >> The problem here stems from our inability to work towards agreement,
> >> rather than standing on set positions.  I set out what changes I think
> >> would make the current implementation OK.  Can we please, please have
> >> a discussion about those points instead of trying to argue about who
> >> has given more ground.
> >>
> >>> That commitment would of course be good. However, even if that were
> >>> possible
> >>> before writing code and everyone agreed that the ideas of you and
> >>> Nathaniel
> >>> should be implemented in full, it's still not clear that either of you
> >>> would
> >>> be willing to write any code. Agreement without code still doesn't help
> >>> us
> >>> very much.
> >>
> >> I'm going to return to Nathaniel's point - it is a highly valuable
> >> thing to set ourselves the target of resolving substantial discussions
> >> by consensus.   The route you are endorsing here is 'implementor
> >> wins'.   We don't need to do it that way.  We're a mature sensible
> >> bunch of adults who can talk out the issues until we agree they are
> >> ready for implementation, and then implement.  That's all Nathaniel is
> >> saying.  I th

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Matthew Brett]

Hi,

On Fri, Oct 28, 2011 at 4:53 PM, Benjamin Root  wrote:
>
>
> On Friday, October 28, 2011, Matthew Brett  wrote:
>> Hi,
>>
>> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
>>  wrote:
>>>
>>>
>>> On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett 
>>> wrote:

 Hi,

 On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
  wrote:
 >
 >
 > On Fri, Oct 28, 2011 at 3:56 PM, Matthew Brett
 > 
 > wrote:
 >>
 >> Hi,
 >>
 >> On Fri, Oct 28, 2011 at 2:43 PM, Matthew Brett
 >> 
 >> wrote:
 >> > Hi,
 >> >
 >> > On Fri, Oct 28, 2011 at 2:41 PM, Charles R Harris
 >> >  wrote:
 >> >>
 >> >>
 >> >> On Fri, Oct 28, 2011 at 3:16 PM, Nathaniel Smith 
 >> >> wrote:
 >> >>>
 >> >>> On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant
 >> >>> 
 >> >>> wrote:
 >> >>> > I think Nathaniel and Matthew provided very
 >> >>> > specific feedback that was helpful in understanding other
 >> >>> > perspectives
 >> >>> > of a
 >> >>> > difficult problem. In particular, I really wanted
 >> >>> > bit-patterns
 >> >>> > implemented.However, I also understand that Mark did quite
 >> >>> > a
 >> >>> > bit
 >> >>> > of
 >> >>> > work
 >> >>> > and altered his original designs quite a bit in response to
 >> >>> > community
 >> >>> > feedback.   I wasn't a major part of the pull request
 >> >>> > discussion,
 >> >>> > nor
 >> >>> > did I
 >> >>> > merge the changes, but I support Charles if he reviewed the
 >> >>> > code
 >> >>> > and
 >> >>> > felt
 >> >>> > like it was the right thing to do.  I likely would have done
 >> >>> > the
 >> >>> > same
 >> >>> > thing
 >> >>> > rather than let Mark Wiebe's work languish.
 >> >>>
 >> >>> My connectivity is spotty this week, so I'll stay out of the
 >> >>> technical
 >> >>> discussion for now, but I want to share a story.
 >> >>>
 >> >>> Maybe a year ago now, Jonathan Taylor and I were debating what
 >> >>> the
 >> >>> best API for describing statistical models would be -- whether we
 >> >>> wanted something like R's "formulas" (which I supported), or
 >> >>> another
 >> >>> approach based on sympy (his idea). To summarize, I thought his
 >> >>> API
 >> >>> was confusing, pointlessly complicated, and didn't actually solve
 >> >>> the
 >> >>> problem; he thought R-style formulas were superficially simpler
 >> >>> but
 >> >>> hopelessly confused and inconsistent underneath. Now, obviously,
 >> >>> I
 >> >>> was
 >> >>> right and he was wrong. Well, obvious to me, anyway... ;-) But it
 >> >>> wasn't like I could just wave a wand and make his arguments go
 >> >>> away,
 >> >>> no I should point out that the implementation hasn't - as far as
 >> >>> I can
>> see - changed the discussion.  The discussion was about the API.
>> Implementations are useful for agreed APIs because they can point out
>> where the API does not make sense or cannot be implemented.  In this
>> case, the API Mark said he was going to implement - he did implement -
>> at least as far as I can see.  Again, I'm happy to be corrected.
>>
 In saying that we are insisting on our way, you are saying, implicitly,
 'I
 am not going to negotiate'.
>>>
>>> That is only your interpretation. The observation that Mark compromised
>>> quite a bit while you didn't seems largely correct to me.
>>
>> The problem here stems from our inability to work towards agreement,
>> rather than standing on set positions.  I set out what changes I think
>> would make the current implementation OK.  Can we please, please have
>> a discussion about those points instead of trying to argue about who
>> has given more ground.
>>
>>> That commitment would of course be good. However, even if that were
>>> possible
>>> before writing code and everyone agreed that the ideas of you and
>>> Nathaniel
>>> should be implemented in full, it's still not clear that either of you
>>> would
>>> be willing to write any code. Agreement without code still doesn't help
>>> us
>>> very much.
>>
>> I'm going to return to Nathaniel's point - it is a highly valuable
>> thing to set ourselves the target of resolving substantial discussions
>> by consensus.   The route you are endorsing here is 'implementor
>> wins'.   We don't need to do it that way.  We're a mature sensible
>> bunch of adults who can talk out the issues until we agree they are
>> ready for implementation, and then implement.  That's all Nathaniel is
>> saying.  I think he's obviously right, and I'm sad that it isn't as
>> clear to y'all as it is to me.
>>
>> Best,
>>
>> Matthew
>>
>
> Everyone, can we please not do this?! I had enough of adults doing finger
> pointing back over the summer during the whole debt ceiling debate.  I think
> we can all agree that we are better than the US congress?

Yes, please.

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Wes McKinney]

On Fri, Oct 28, 2011 at 7:53 PM, Benjamin Root  wrote:
>
>
> On Friday, October 28, 2011, Matthew Brett  wrote:
>> Hi,
>>
>> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
>>  wrote:
>>>
>>>
>>> On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett 
>>> wrote:

 Hi,

 On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
  wrote:
 >
 >
 > On Fri, Oct 28, 2011 at 3:56 PM, Matthew Brett
 > 
 > wrote:
 >>
 >> Hi,
 >>
 >> On Fri, Oct 28, 2011 at 2:43 PM, Matthew Brett
 >> 
 >> wrote:
 >> > Hi,
 >> >
 >> > On Fri, Oct 28, 2011 at 2:41 PM, Charles R Harris
 >> >  wrote:
 >> >>
 >> >>
 >> >> On Fri, Oct 28, 2011 at 3:16 PM, Nathaniel Smith 
 >> >> wrote:
 >> >>>
 >> >>> On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant
 >> >>> 
 >> >>> wrote:
 >> >>> > I think Nathaniel and Matthew provided very
 >> >>> > specific feedback that was helpful in understanding other
 >> >>> > perspectives
 >> >>> > of a
 >> >>> > difficult problem.     In particular, I really wanted
 >> >>> > bit-patterns
 >> >>> > implemented.    However, I also understand that Mark did quite
 >> >>> > a
 >> >>> > bit
 >> >>> > of
 >> >>> > work
 >> >>> > and altered his original designs quite a bit in response to
 >> >>> > community
 >> >>> > feedback.   I wasn't a major part of the pull request
 >> >>> > discussion,
 >> >>> > nor
 >> >>> > did I
 >> >>> > merge the changes, but I support Charles if he reviewed the
 >> >>> > code
 >> >>> > and
 >> >>> > felt
 >> >>> > like it was the right thing to do.  I likely would have done
 >> >>> > the
 >> >>> > same
 >> >>> > thing
 >> >>> > rather than let Mark Wiebe's work languish.
 >> >>>
 >> >>> My connectivity is spotty this week, so I'll stay out of the
 >> >>> technical
 >> >>> discussion for now, but I want to share a story.
 >> >>>
 >> >>> Maybe a year ago now, Jonathan Taylor and I were debating what
 >> >>> the
 >> >>> best API for describing statistical models would be -- whether we
 >> >>> wanted something like R's "formulas" (which I supported), or
 >> >>> another
 >> >>> approach based on sympy (his idea). To summarize, I thought his
 >> >>> API
 >> >>> was confusing, pointlessly complicated, and didn't actually solve
 >> >>> the
 >> >>> problem; he thought R-style formulas were superficially simpler
 >> >>> but
 >> >>> hopelessly confused and inconsistent underneath. Now, obviously,
 >> >>> I
 >> >>> was
 >> >>> right and he was wrong. Well, obvious to me, anyway... ;-) But it
 >> >>> wasn't like I could just wave a wand and make his arguments go
 >> >>> away,
 >> >>> no I should point out that the implementation hasn't - as far as
 >> >>> I can
>> see - changed the discussion.  The discussion was about the API.
>> Implementations are useful for agreed APIs because they can point out
>> where the API does not make sense or cannot be implemented.  In this
>> case, the API Mark said he was going to implement - he did implement -
>> at least as far as I can see.  Again, I'm happy to be corrected.
>>
 In saying that we are insisting on our way, you are saying, implicitly,
 'I
 am not going to negotiate'.
>>>
>>> That is only your interpretation. The observation that Mark compromised
>>> quite a bit while you didn't seems largely correct to me.
>>
>> The problem here stems from our inability to work towards agreement,
>> rather than standing on set positions.  I set out what changes I think
>> would make the current implementation OK.  Can we please, please have
>> a discussion about those points instead of trying to argue about who
>> has given more ground.
>>
>>> That commitment would of course be good. However, even if that were
>>> possible
>>> before writing code and everyone agreed that the ideas of you and
>>> Nathaniel
>>> should be implemented in full, it's still not clear that either of you
>>> would
>>> be willing to write any code. Agreement without code still doesn't help
>>> us
>>> very much.
>>
>> I'm going to return to Nathaniel's point - it is a highly valuable
>> thing to set ourselves the target of resolving substantial discussions
>> by consensus.   The route you are endorsing here is 'implementor
>> wins'.   We don't need to do it that way.  We're a mature sensible
>> bunch of adults who can talk out the issues until we agree they are
>> ready for implementation, and then implement.  That's all Nathaniel is
>> saying.  I think he's obviously right, and I'm sad that it isn't as
>> clear to y'all as it is to me.
>>
>> Best,
>>
>> Matthew
>>
>
> Everyone, can we please not do this?! I had enough of adults doing finger
> pointing back over the summer during the whole debt ceiling debate.  I think
> we can all agree that we are better than the US congress?
>
> Forget about r

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Benjamin Root]

On Friday, October 28, 2011, Matthew Brett  wrote:
> Hi,
>
> On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
>  wrote:
>>
>>
>> On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett 
>> wrote:
>>>
>>> Hi,
>>>
>>> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
>>>  wrote:
>>> >
>>> >
>>> > On Fri, Oct 28, 2011 at 3:56 PM, Matthew Brett <
matthew.br...@gmail.com>
>>> > wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> On Fri, Oct 28, 2011 at 2:43 PM, Matthew Brett
>>> >> 
>>> >> wrote:
>>> >> > Hi,
>>> >> >
>>> >> > On Fri, Oct 28, 2011 at 2:41 PM, Charles R Harris
>>> >> >  wrote:
>>> >> >>
>>> >> >>
>>> >> >> On Fri, Oct 28, 2011 at 3:16 PM, Nathaniel Smith 
>>> >> >> wrote:
>>> >> >>>
>>> >> >>> On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant
>>> >> >>> 
>>> >> >>> wrote:
>>> >> >>> > I think Nathaniel and Matthew provided very
>>> >> >>> > specific feedback that was helpful in understanding other
>>> >> >>> > perspectives
>>> >> >>> > of a
>>> >> >>> > difficult problem. In particular, I really wanted
>>> >> >>> > bit-patterns
>>> >> >>> > implemented.However, I also understand that Mark did quite
a
>>> >> >>> > bit
>>> >> >>> > of
>>> >> >>> > work
>>> >> >>> > and altered his original designs quite a bit in response to
>>> >> >>> > community
>>> >> >>> > feedback.   I wasn't a major part of the pull request
discussion,
>>> >> >>> > nor
>>> >> >>> > did I
>>> >> >>> > merge the changes, but I support Charles if he reviewed the
code
>>> >> >>> > and
>>> >> >>> > felt
>>> >> >>> > like it was the right thing to do.  I likely would have done
the
>>> >> >>> > same
>>> >> >>> > thing
>>> >> >>> > rather than let Mark Wiebe's work languish.
>>> >> >>>
>>> >> >>> My connectivity is spotty this week, so I'll stay out of the
>>> >> >>> technical
>>> >> >>> discussion for now, but I want to share a story.
>>> >> >>>
>>> >> >>> Maybe a year ago now, Jonathan Taylor and I were debating what
the
>>> >> >>> best API for describing statistical models would be -- whether we
>>> >> >>> wanted something like R's "formulas" (which I supported), or
>>> >> >>> another
>>> >> >>> approach based on sympy (his idea). To summarize, I thought his
API
>>> >> >>> was confusing, pointlessly complicated, and didn't actually solve
>>> >> >>> the
>>> >> >>> problem; he thought R-style formulas were superficially simpler
but
>>> >> >>> hopelessly confused and inconsistent underneath. Now, obviously,
I
>>> >> >>> was
>>> >> >>> right and he was wrong. Well, obvious to me, anyway... ;-) But it
>>> >> >>> wasn't like I could just wave a wand and make his arguments go
>>> >> >>> away,
>>> >> >>> no I should point out that the implementation hasn't - as far as
I can
> see - changed the discussion.  The discussion was about the API.
> Implementations are useful for agreed APIs because they can point out
> where the API does not make sense or cannot be implemented.  In this
> case, the API Mark said he was going to implement - he did implement -
> at least as far as I can see.  Again, I'm happy to be corrected.
>
>>> In saying that we are insisting on our way, you are saying, implicitly,
'I
>>> am not going to negotiate'.
>>
>> That is only your interpretation. The observation that Mark compromised
>> quite a bit while you didn't seems largely correct to me.
>
> The problem here stems from our inability to work towards agreement,
> rather than standing on set positions.  I set out what changes I think
> would make the current implementation OK.  Can we please, please have
> a discussion about those points instead of trying to argue about who
> has given more ground.
>
>> That commitment would of course be good. However, even if that were
possible
>> before writing code and everyone agreed that the ideas of you and
Nathaniel
>> should be implemented in full, it's still not clear that either of you
would
>> be willing to write any code. Agreement without code still doesn't help
us
>> very much.
>
> I'm going to return to Nathaniel's point - it is a highly valuable
> thing to set ourselves the target of resolving substantial discussions
> by consensus.   The route you are endorsing here is 'implementor
> wins'.   We don't need to do it that way.  We're a mature sensible
> bunch of adults who can talk out the issues until we agree they are
> ready for implementation, and then implement.  That's all Nathaniel is
> saying.  I think he's obviously right, and I'm sad that it isn't as
> clear to y'all as it is to me.
>
> Best,
>
> Matthew
>

Everyone, can we please not do this?! I had enough of adults doing finger
pointing back over the summer during the whole debt ceiling debate.  I think
we can all agree that we are better than the US congress?

Forget about rudeness or decision processes.

I will start by saying that I am willing to separate ignore and absent, but
only on the write side of things.  On read, I want a single way to identify
the missing values.  I also want only a single way to perform calculations
(either skip or propagate).

An indicator o

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Matthew Brett]

Hi,

On Fri, Oct 28, 2011 at 4:21 PM, Ralf Gommers
 wrote:
>
>
> On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett 
> wrote:
>>
>> Hi,
>>
>> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
>>  wrote:
>> >
>> >
>> > On Fri, Oct 28, 2011 at 3:56 PM, Matthew Brett 
>> > wrote:
>> >>
>> >> Hi,
>> >>
>> >> On Fri, Oct 28, 2011 at 2:43 PM, Matthew Brett
>> >> 
>> >> wrote:
>> >> > Hi,
>> >> >
>> >> > On Fri, Oct 28, 2011 at 2:41 PM, Charles R Harris
>> >> >  wrote:
>> >> >>
>> >> >>
>> >> >> On Fri, Oct 28, 2011 at 3:16 PM, Nathaniel Smith 
>> >> >> wrote:
>> >> >>>
>> >> >>> On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant
>> >> >>> 
>> >> >>> wrote:
>> >> >>> > I think Nathaniel and Matthew provided very
>> >> >>> > specific feedback that was helpful in understanding other
>> >> >>> > perspectives
>> >> >>> > of a
>> >> >>> > difficult problem.     In particular, I really wanted
>> >> >>> > bit-patterns
>> >> >>> > implemented.    However, I also understand that Mark did quite a
>> >> >>> > bit
>> >> >>> > of
>> >> >>> > work
>> >> >>> > and altered his original designs quite a bit in response to
>> >> >>> > community
>> >> >>> > feedback.   I wasn't a major part of the pull request discussion,
>> >> >>> > nor
>> >> >>> > did I
>> >> >>> > merge the changes, but I support Charles if he reviewed the code
>> >> >>> > and
>> >> >>> > felt
>> >> >>> > like it was the right thing to do.  I likely would have done the
>> >> >>> > same
>> >> >>> > thing
>> >> >>> > rather than let Mark Wiebe's work languish.
>> >> >>>
>> >> >>> My connectivity is spotty this week, so I'll stay out of the
>> >> >>> technical
>> >> >>> discussion for now, but I want to share a story.
>> >> >>>
>> >> >>> Maybe a year ago now, Jonathan Taylor and I were debating what the
>> >> >>> best API for describing statistical models would be -- whether we
>> >> >>> wanted something like R's "formulas" (which I supported), or
>> >> >>> another
>> >> >>> approach based on sympy (his idea). To summarize, I thought his API
>> >> >>> was confusing, pointlessly complicated, and didn't actually solve
>> >> >>> the
>> >> >>> problem; he thought R-style formulas were superficially simpler but
>> >> >>> hopelessly confused and inconsistent underneath. Now, obviously, I
>> >> >>> was
>> >> >>> right and he was wrong. Well, obvious to me, anyway... ;-) But it
>> >> >>> wasn't like I could just wave a wand and make his arguments go
>> >> >>> away,
>> >> >>> no matter how annoying and wrong-headed I thought they were... I
>> >> >>> could
>> >> >>> write all the code I wanted but no-one would use it unless I could
>> >> >>> convince them it's actually the right solution, so I had to engage
>> >> >>> with him, and dig deep into his arguments.
>> >> >>>
>> >> >>> What I discovered was that (as I thought) R-style formulas *do*
>> >> >>> have a
>> >> >>> solid theoretical basis -- but (as he thought) all the existing
>> >> >>> implementations *are* broken and inconsistent! I'm still not sure I
>> >> >>> can actually convince Jonathan to go my way, but, because of his
>> >> >>> stubbornness, I had to invent a better way of handling these
>> >> >>> formulas,
>> >> >>> and so my library[1] is actually the first implementation of these
>> >> >>> things that has a rigorous theory behind it, and in the process it
>> >> >>> avoids two fundamental, decades-old bugs in R. (And I'm not sure
>> >> >>> the R
>> >> >>> folks can fix either of them at this point without breaking a ton
>> >> >>> of
>> >> >>> code, since they both have API consequences.)
>> >> >>>
>> >> >>> --
>> >> >>>
>> >> >>> It's extremely common for healthy FOSS projects to insist on
>> >> >>> consensus
>> >> >>> for almost all decisions, where consensus means something like
>> >> >>> "every
>> >> >>> interested party has a veto"[2]. This seems counterintuitive,
>> >> >>> because
>> >> >>> if everyone's vetoing all the time, how does anything get done? The
>> >> >>> trick is that if anyone *can* veto, then vetoes turn out to
>> >> >>> actually
>> >> >>> be very rare. Everyone knows that they can't just ignore
>> >> >>> alternative
>> >> >>> points of view -- they have to engage with them if they want to get
>> >> >>> anything done. So you get buy-in on features early, and no vetoes
>> >> >>> are
>> >> >>> necessary. And by forcing people to engage with each other, like me
>> >> >>> with Jonathan, you get better designs.
>> >> >>>
>> >> >>> But what about the cost of all that code that doesn't get merged,
>> >> >>> or
>> >> >>> written, because everyone's spending all this time debating
>> >> >>> instead?
>> >> >>> Better designs are nice and all, but how does that justify letting
>> >> >>> working code languish?
>> >> >>>
>> >> >>> The greatest risk for a FOSS project is that people will ignore
>> >> >>> you.
>> >> >>> Projects and features live and die by community buy-in. Consider
>> >> >>> the
>> >> >>> "NA mask" feature right now. It works (at least the parts of it
>> >> >>> that
>> >> >>> are implemented)

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Matthew Brett]

Hi,

On Fri, Oct 28, 2011 at 4:21 PM, Charles R Harris
 wrote:
>
>
> On Fri, Oct 28, 2011 at 5:09 PM, Matthew Brett 
> wrote:
>>
>> On Fri, Oct 28, 2011 at 3:49 PM, Charles R Harris
>>  wrote:
>> >
>> >
>> > 2011/10/28 Stéfan van der Walt 
>> >>
>> >> On Fri, Oct 28, 2011 at 3:21 PM, Benjamin Root  wrote:
>> >> > The space issues was never ignored and Mark left room for that to be
>> >> > addressed.  Parameterized dtypes can still be added (and isn't all
>> >> > that
>> >> > different from multi-na). Perhaps I could be convinced of a having
>> >> > np.MA
>> >> > assignments mean "ignore" and np.NA mean "absent".  How far off are
>> >> > we
>> >> > really from consensus?
>> >>
>> >> Do you know whether Mark is around?  I think his feedback would be
>> >> useful at this point; having written the code, he'll be able to
>> >> evaluate some of the technical suggestions made.
>> >>
>> >
>> > Yes, Mark is around, but I assume he is interested in his school work at
>> > this point. And he might not be inclined to get back into this
>> > particular
>> > discussion. I don't feel he was treated very well by some last time
>> > around.
>>
>> We have not always been good at separating the concept of disagreement
>> from that of rudeness.
>>
>> As I've said before, one form of rudeness (and not disagreement) is
>> ignoring people.
>>
>> We should all be careful to point out - respectfully, and with reasons
>> - when we find our colleagues replies (or non-replies) to be rude,
>> because rudeness is very bad for the spirit of open discussion.
>>
>
> Trying things out in preparation for discussion is also a mark of respect.
> Have you worked with the current implementation?

OK - this seems to me to be rude.  Why?  Because you have presumably
already read what my concerns were, and my discussion of the current
implementation in my reply to Travis.  You haven't made any effort to
point out to me where I may be wrong or failing to understand.  I
infer that you are merely saying 'go away and come back later'.  And
that is rude.

Best,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Ralf Gommers]

On Sat, Oct 29, 2011 at 12:37 AM, Matthew Brett wrote:

> Hi,
>
> On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
>  wrote:
> >
> >
> > On Fri, Oct 28, 2011 at 3:56 PM, Matthew Brett 
> > wrote:
> >>
> >> Hi,
> >>
> >> On Fri, Oct 28, 2011 at 2:43 PM, Matthew Brett  >
> >> wrote:
> >> > Hi,
> >> >
> >> > On Fri, Oct 28, 2011 at 2:41 PM, Charles R Harris
> >> >  wrote:
> >> >>
> >> >>
> >> >> On Fri, Oct 28, 2011 at 3:16 PM, Nathaniel Smith 
> wrote:
> >> >>>
> >> >>> On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant
> >> >>> 
> >> >>> wrote:
> >> >>> > I think Nathaniel and Matthew provided very
> >> >>> > specific feedback that was helpful in understanding other
> >> >>> > perspectives
> >> >>> > of a
> >> >>> > difficult problem. In particular, I really wanted bit-patterns
> >> >>> > implemented.However, I also understand that Mark did quite a
> bit
> >> >>> > of
> >> >>> > work
> >> >>> > and altered his original designs quite a bit in response to
> >> >>> > community
> >> >>> > feedback.   I wasn't a major part of the pull request discussion,
> >> >>> > nor
> >> >>> > did I
> >> >>> > merge the changes, but I support Charles if he reviewed the code
> and
> >> >>> > felt
> >> >>> > like it was the right thing to do.  I likely would have done the
> >> >>> > same
> >> >>> > thing
> >> >>> > rather than let Mark Wiebe's work languish.
> >> >>>
> >> >>> My connectivity is spotty this week, so I'll stay out of the
> technical
> >> >>> discussion for now, but I want to share a story.
> >> >>>
> >> >>> Maybe a year ago now, Jonathan Taylor and I were debating what the
> >> >>> best API for describing statistical models would be -- whether we
> >> >>> wanted something like R's "formulas" (which I supported), or another
> >> >>> approach based on sympy (his idea). To summarize, I thought his API
> >> >>> was confusing, pointlessly complicated, and didn't actually solve
> the
> >> >>> problem; he thought R-style formulas were superficially simpler but
> >> >>> hopelessly confused and inconsistent underneath. Now, obviously, I
> was
> >> >>> right and he was wrong. Well, obvious to me, anyway... ;-) But it
> >> >>> wasn't like I could just wave a wand and make his arguments go away,
> >> >>> no matter how annoying and wrong-headed I thought they were... I
> could
> >> >>> write all the code I wanted but no-one would use it unless I could
> >> >>> convince them it's actually the right solution, so I had to engage
> >> >>> with him, and dig deep into his arguments.
> >> >>>
> >> >>> What I discovered was that (as I thought) R-style formulas *do* have
> a
> >> >>> solid theoretical basis -- but (as he thought) all the existing
> >> >>> implementations *are* broken and inconsistent! I'm still not sure I
> >> >>> can actually convince Jonathan to go my way, but, because of his
> >> >>> stubbornness, I had to invent a better way of handling these
> formulas,
> >> >>> and so my library[1] is actually the first implementation of these
> >> >>> things that has a rigorous theory behind it, and in the process it
> >> >>> avoids two fundamental, decades-old bugs in R. (And I'm not sure the
> R
> >> >>> folks can fix either of them at this point without breaking a ton of
> >> >>> code, since they both have API consequences.)
> >> >>>
> >> >>> --
> >> >>>
> >> >>> It's extremely common for healthy FOSS projects to insist on
> consensus
> >> >>> for almost all decisions, where consensus means something like
> "every
> >> >>> interested party has a veto"[2]. This seems counterintuitive,
> because
> >> >>> if everyone's vetoing all the time, how does anything get done? The
> >> >>> trick is that if anyone *can* veto, then vetoes turn out to actually
> >> >>> be very rare. Everyone knows that they can't just ignore alternative
> >> >>> points of view -- they have to engage with them if they want to get
> >> >>> anything done. So you get buy-in on features early, and no vetoes
> are
> >> >>> necessary. And by forcing people to engage with each other, like me
> >> >>> with Jonathan, you get better designs.
> >> >>>
> >> >>> But what about the cost of all that code that doesn't get merged, or
> >> >>> written, because everyone's spending all this time debating instead?
> >> >>> Better designs are nice and all, but how does that justify letting
> >> >>> working code languish?
> >> >>>
> >> >>> The greatest risk for a FOSS project is that people will ignore you.
> >> >>> Projects and features live and die by community buy-in. Consider the
> >> >>> "NA mask" feature right now. It works (at least the parts of it that
> >> >>> are implemented). It's in mainline. But IIRC, Pierre said last time
> >> >>> that he doesn't think the current design will help him improve or
> >> >>> replace numpy.ma. Up-thread, Wes McKinney is leaning towards
> ignoring
> >> >>> this feature in favor of his library pandas' current hacky NA
> support.
> >> >>> Members of the neuroimaging crowd are saying that the memory
> overhead
> >> >>> is too high and

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Charles R Harris]

On Fri, Oct 28, 2011 at 5:09 PM, Matthew Brett wrote:

> On Fri, Oct 28, 2011 at 3:49 PM, Charles R Harris
>  wrote:
> >
> >
> > 2011/10/28 Stéfan van der Walt 
> >>
> >> On Fri, Oct 28, 2011 at 3:21 PM, Benjamin Root  wrote:
> >> > The space issues was never ignored and Mark left room for that to be
> >> > addressed.  Parameterized dtypes can still be added (and isn't all
> that
> >> > different from multi-na). Perhaps I could be convinced of a having
> np.MA
> >> > assignments mean "ignore" and np.NA mean "absent".  How far off are we
> >> > really from consensus?
> >>
> >> Do you know whether Mark is around?  I think his feedback would be
> >> useful at this point; having written the code, he'll be able to
> >> evaluate some of the technical suggestions made.
> >>
> >
> > Yes, Mark is around, but I assume he is interested in his school work at
> > this point. And he might not be inclined to get back into this particular
> > discussion. I don't feel he was treated very well by some last time
> around.
>
> We have not always been good at separating the concept of disagreement
> from that of rudeness.
>
> As I've said before, one form of rudeness (and not disagreement) is
> ignoring people.
>
> We should all be careful to point out - respectfully, and with reasons
> - when we find our colleagues replies (or non-replies) to be rude,
> because rudeness is very bad for the spirit of open discussion.
>
>
Trying things out in preparation for discussion is also a mark of respect.
Have you worked with the current implementation?

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Matthew Brett]

On Fri, Oct 28, 2011 at 3:49 PM, Charles R Harris
 wrote:
>
>
> 2011/10/28 Stéfan van der Walt 
>>
>> On Fri, Oct 28, 2011 at 3:21 PM, Benjamin Root  wrote:
>> > The space issues was never ignored and Mark left room for that to be
>> > addressed.  Parameterized dtypes can still be added (and isn't all that
>> > different from multi-na). Perhaps I could be convinced of a having np.MA
>> > assignments mean "ignore" and np.NA mean "absent".  How far off are we
>> > really from consensus?
>>
>> Do you know whether Mark is around?  I think his feedback would be
>> useful at this point; having written the code, he'll be able to
>> evaluate some of the technical suggestions made.
>>
>
> Yes, Mark is around, but I assume he is interested in his school work at
> this point. And he might not be inclined to get back into this particular
> discussion. I don't feel he was treated very well by some last time around.

We have not always been good at separating the concept of disagreement
from that of rudeness.

As I've said before, one form of rudeness (and not disagreement) is
ignoring people.

We should all be careful to point out - respectfully, and with reasons
- when we find our colleagues replies (or non-replies) to be rude,
because rudeness is very bad for the spirit of open discussion.

Best,

Matthew
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Charles R Harris]

2011/10/28 Stéfan van der Walt 

> On Fri, Oct 28, 2011 at 3:21 PM, Benjamin Root  wrote:
> > The space issues was never ignored and Mark left room for that to be
> > addressed.  Parameterized dtypes can still be added (and isn't all that
> > different from multi-na). Perhaps I could be convinced of a having np.MA
> > assignments mean "ignore" and np.NA mean "absent".  How far off are we
> > really from consensus?
>
> Do you know whether Mark is around?  I think his feedback would be
> useful at this point; having written the code, he'll be able to
> evaluate some of the technical suggestions made.
>
>
Yes, Mark is around, but I assume he is interested in his school work at
this point. And he might not be inclined to get back into this particular
discussion. I don't feel he was treated very well by some last time around.

Chuck
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Stéfan van der Walt]

On Fri, Oct 28, 2011 at 3:21 PM, Benjamin Root  wrote:
> The space issues was never ignored and Mark left room for that to be
> addressed.  Parameterized dtypes can still be added (and isn't all that
> different from multi-na). Perhaps I could be convinced of a having np.MA
> assignments mean "ignore" and np.NA mean "absent".  How far off are we
> really from consensus?

Do you know whether Mark is around?  I think his feedback would be
useful at this point; having written the code, he'll be able to
evaluate some of the technical suggestions made.

Stéfan
___
NumPy-Discussion mailing list
NumPy-Discussion@scipy.org
http://mail.scipy.org/mailman/listinfo/numpy-discussion

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Matthew Brett]

Hi,

On Fri, Oct 28, 2011 at 3:14 PM, Charles R Harris
 wrote:
>
>
> On Fri, Oct 28, 2011 at 3:56 PM, Matthew Brett 
> wrote:
>>
>> Hi,
>>
>> On Fri, Oct 28, 2011 at 2:43 PM, Matthew Brett 
>> wrote:
>> > Hi,
>> >
>> > On Fri, Oct 28, 2011 at 2:41 PM, Charles R Harris
>> >  wrote:
>> >>
>> >>
>> >> On Fri, Oct 28, 2011 at 3:16 PM, Nathaniel Smith  wrote:
>> >>>
>> >>> On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant
>> >>> 
>> >>> wrote:
>> >>> > I think Nathaniel and Matthew provided very
>> >>> > specific feedback that was helpful in understanding other
>> >>> > perspectives
>> >>> > of a
>> >>> > difficult problem.     In particular, I really wanted bit-patterns
>> >>> > implemented.    However, I also understand that Mark did quite a bit
>> >>> > of
>> >>> > work
>> >>> > and altered his original designs quite a bit in response to
>> >>> > community
>> >>> > feedback.   I wasn't a major part of the pull request discussion,
>> >>> > nor
>> >>> > did I
>> >>> > merge the changes, but I support Charles if he reviewed the code and
>> >>> > felt
>> >>> > like it was the right thing to do.  I likely would have done the
>> >>> > same
>> >>> > thing
>> >>> > rather than let Mark Wiebe's work languish.
>> >>>
>> >>> My connectivity is spotty this week, so I'll stay out of the technical
>> >>> discussion for now, but I want to share a story.
>> >>>
>> >>> Maybe a year ago now, Jonathan Taylor and I were debating what the
>> >>> best API for describing statistical models would be -- whether we
>> >>> wanted something like R's "formulas" (which I supported), or another
>> >>> approach based on sympy (his idea). To summarize, I thought his API
>> >>> was confusing, pointlessly complicated, and didn't actually solve the
>> >>> problem; he thought R-style formulas were superficially simpler but
>> >>> hopelessly confused and inconsistent underneath. Now, obviously, I was
>> >>> right and he was wrong. Well, obvious to me, anyway... ;-) But it
>> >>> wasn't like I could just wave a wand and make his arguments go away,
>> >>> no matter how annoying and wrong-headed I thought they were... I could
>> >>> write all the code I wanted but no-one would use it unless I could
>> >>> convince them it's actually the right solution, so I had to engage
>> >>> with him, and dig deep into his arguments.
>> >>>
>> >>> What I discovered was that (as I thought) R-style formulas *do* have a
>> >>> solid theoretical basis -- but (as he thought) all the existing
>> >>> implementations *are* broken and inconsistent! I'm still not sure I
>> >>> can actually convince Jonathan to go my way, but, because of his
>> >>> stubbornness, I had to invent a better way of handling these formulas,
>> >>> and so my library[1] is actually the first implementation of these
>> >>> things that has a rigorous theory behind it, and in the process it
>> >>> avoids two fundamental, decades-old bugs in R. (And I'm not sure the R
>> >>> folks can fix either of them at this point without breaking a ton of
>> >>> code, since they both have API consequences.)
>> >>>
>> >>> --
>> >>>
>> >>> It's extremely common for healthy FOSS projects to insist on consensus
>> >>> for almost all decisions, where consensus means something like "every
>> >>> interested party has a veto"[2]. This seems counterintuitive, because
>> >>> if everyone's vetoing all the time, how does anything get done? The
>> >>> trick is that if anyone *can* veto, then vetoes turn out to actually
>> >>> be very rare. Everyone knows that they can't just ignore alternative
>> >>> points of view -- they have to engage with them if they want to get
>> >>> anything done. So you get buy-in on features early, and no vetoes are
>> >>> necessary. And by forcing people to engage with each other, like me
>> >>> with Jonathan, you get better designs.
>> >>>
>> >>> But what about the cost of all that code that doesn't get merged, or
>> >>> written, because everyone's spending all this time debating instead?
>> >>> Better designs are nice and all, but how does that justify letting
>> >>> working code languish?
>> >>>
>> >>> The greatest risk for a FOSS project is that people will ignore you.
>> >>> Projects and features live and die by community buy-in. Consider the
>> >>> "NA mask" feature right now. It works (at least the parts of it that
>> >>> are implemented). It's in mainline. But IIRC, Pierre said last time
>> >>> that he doesn't think the current design will help him improve or
>> >>> replace numpy.ma. Up-thread, Wes McKinney is leaning towards ignoring
>> >>> this feature in favor of his library pandas' current hacky NA support.
>> >>> Members of the neuroimaging crowd are saying that the memory overhead
>> >>> is too high and the benefits too marginal, so they'll stick with NaNs.
>> >>> Together these folk a huge proportion of the this feature's target
>> >>> audience. So what have we actually accomplished by merging this to
>> >>> mainline? Are we going to be stuck supporting a feature that only a
>> >>> fra

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Benjamin Root]

On Friday, October 28, 2011, Matthew Brett  wrote:
> Hi,
>
> On Fri, Oct 28, 2011 at 2:43 PM, Matthew Brett 
wrote:
>> Hi,
>>
>> On Fri, Oct 28, 2011 at 2:41 PM, Charles R Harris
>>  wrote:
>>>
>>>
>>> On Fri, Oct 28, 2011 at 3:16 PM, Nathaniel Smith  wrote:

 On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant <
oliph...@enthought.com>
 wrote:
 > I think Nathaniel and Matthew provided very
 > specific feedback that was helpful in understanding other
perspectives
 > of a
 > difficult problem. In particular, I really wanted bit-patterns
 > implemented.However, I also understand that Mark did quite a bit
of
 > work
 > and altered his original designs quite a bit in response to community
 > feedback.   I wasn't a major part of the pull request discussion, nor
 > did I
 > merge the changes, but I support Charles if he reviewed the code and
 > felt
 > like it was the right thing to do.  I likely would have done the same
 > thing
 > rather than let Mark Wiebe's work languish.

 My connectivity is spotty this week, so I'll stay out of the technical
 discussion for now, but I want to share a story.

 Maybe a year ago now, Jonathan Taylor and I were debating what the
 best API for describing statistical models would be -- whether we
 wanted something like R's "formulas" (which I supported), or another
 approach based on sympy (his idea). To summarize, I thought his API
 was confusing, pointlessly complicated, and didn't actually solve the
 problem; he thought R-style formulas were superficially simpler but
 hopelessly confused and inconsistent underneath. Now, obviously, I was
 right and he was wrong. Well, obvious to me, anyway... ;-) But it
 wasn't like I could just wave a wand and make his arguments go away,
 no matter how annoying and wrong-headed I thought they were... I could
 write all the code I wanted but no-one would use it unless I could
 convince them it's actually the right solution, so I had to engage
 with him, and dig deep into his arguments.

 What I discovered was that (as I thought) R-style formulas *do* have a
 solid theoretical basis -- but (as he thought) all the existing
 implementations *are* broken and inconsistent! I'm still not sure I
 can actually convince Jonathan to go my way, but, because of his
 stubbornness, I had to invent a better way of handling these formulas,
 and so my library[1] is actually the first implementation of these
 things that has a rigorous theory behind it, and in the process it
 avoids two fundamental, decades-old bugs in R. (And I'm not sure the R
 folks can fix either of them at this point without breaking a ton of
 code, since they both have API consequences.)

 --

 It's extremely common for healthy FOSS projects to insist on consensus
 for almost all decisions, where consensus means something like "every
 interested party has a veto"[2]. This seems counterintuitive, because
 if everyone's vetoing all the time, how does anything get done? The
 trick is that if anyone *can* veto, then vetoes turn out to actually
 be very rare. Everyone knows that they can't just ignore alternative
 points of view -- they have to engage with them if they want to get
 anything done. So you get buy-in on features early, and no vetoes are
 necessary. And by forcing people to engage with each other, like me
 with Jonathan, you get better designs.

 But what about the cost of all that code that doesn't get merged, or
 written, because everyone's spending all this time debating instead?
 Better designSorry - this was too short and a little rude.  I'm sorry.
>
> I was reacting to what I perceived as intolerance for discussing the
> issues, and I may be wrong in that perception.
>
> I think what Nathaniel is saying, is that it is not in the best
> interests of numpy to push through code where there is not good
> agreement.  In reverting the change, he is, I think, appealing for a
> commitment to that process, for the good of numpy.
>
> I have in the past taken some of your remarks to imply that if someone
> is prepared to write code then that overrides most potential
> disagreement.
>
> The reason I think Nathaniel is the more right, is because most of us,
> I believe, do honestly have the interests of numpy at heart, and, want
> to fully understand the problem, and are prepared to be proven wrong.
> In that situation, in my experience of writing code at least, by far
> the most fruitful way to proceed is by letting all voices be heard.
> On the other hand, if the rule becomes 'unless I see an implementation
> I'm not listening to you' - then we lose the great benefits, to the
> code, of having what is fundamentally a good and strong community.
>
> Best,
>
> Matthew
>

Maybe an alternative implementation isn't really needed.  It seemed t

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Charles R Harris]

On Fri, Oct 28, 2011 at 3:56 PM, Matthew Brett wrote:

> Hi,
>
> On Fri, Oct 28, 2011 at 2:43 PM, Matthew Brett 
> wrote:
> > Hi,
> >
> > On Fri, Oct 28, 2011 at 2:41 PM, Charles R Harris
> >  wrote:
> >>
> >>
> >> On Fri, Oct 28, 2011 at 3:16 PM, Nathaniel Smith  wrote:
> >>>
> >>> On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant <
> oliph...@enthought.com>
> >>> wrote:
> >>> > I think Nathaniel and Matthew provided very
> >>> > specific feedback that was helpful in understanding other
> perspectives
> >>> > of a
> >>> > difficult problem. In particular, I really wanted bit-patterns
> >>> > implemented.However, I also understand that Mark did quite a bit
> of
> >>> > work
> >>> > and altered his original designs quite a bit in response to community
> >>> > feedback.   I wasn't a major part of the pull request discussion, nor
> >>> > did I
> >>> > merge the changes, but I support Charles if he reviewed the code and
> >>> > felt
> >>> > like it was the right thing to do.  I likely would have done the same
> >>> > thing
> >>> > rather than let Mark Wiebe's work languish.
> >>>
> >>> My connectivity is spotty this week, so I'll stay out of the technical
> >>> discussion for now, but I want to share a story.
> >>>
> >>> Maybe a year ago now, Jonathan Taylor and I were debating what the
> >>> best API for describing statistical models would be -- whether we
> >>> wanted something like R's "formulas" (which I supported), or another
> >>> approach based on sympy (his idea). To summarize, I thought his API
> >>> was confusing, pointlessly complicated, and didn't actually solve the
> >>> problem; he thought R-style formulas were superficially simpler but
> >>> hopelessly confused and inconsistent underneath. Now, obviously, I was
> >>> right and he was wrong. Well, obvious to me, anyway... ;-) But it
> >>> wasn't like I could just wave a wand and make his arguments go away,
> >>> no matter how annoying and wrong-headed I thought they were... I could
> >>> write all the code I wanted but no-one would use it unless I could
> >>> convince them it's actually the right solution, so I had to engage
> >>> with him, and dig deep into his arguments.
> >>>
> >>> What I discovered was that (as I thought) R-style formulas *do* have a
> >>> solid theoretical basis -- but (as he thought) all the existing
> >>> implementations *are* broken and inconsistent! I'm still not sure I
> >>> can actually convince Jonathan to go my way, but, because of his
> >>> stubbornness, I had to invent a better way of handling these formulas,
> >>> and so my library[1] is actually the first implementation of these
> >>> things that has a rigorous theory behind it, and in the process it
> >>> avoids two fundamental, decades-old bugs in R. (And I'm not sure the R
> >>> folks can fix either of them at this point without breaking a ton of
> >>> code, since they both have API consequences.)
> >>>
> >>> --
> >>>
> >>> It's extremely common for healthy FOSS projects to insist on consensus
> >>> for almost all decisions, where consensus means something like "every
> >>> interested party has a veto"[2]. This seems counterintuitive, because
> >>> if everyone's vetoing all the time, how does anything get done? The
> >>> trick is that if anyone *can* veto, then vetoes turn out to actually
> >>> be very rare. Everyone knows that they can't just ignore alternative
> >>> points of view -- they have to engage with them if they want to get
> >>> anything done. So you get buy-in on features early, and no vetoes are
> >>> necessary. And by forcing people to engage with each other, like me
> >>> with Jonathan, you get better designs.
> >>>
> >>> But what about the cost of all that code that doesn't get merged, or
> >>> written, because everyone's spending all this time debating instead?
> >>> Better designs are nice and all, but how does that justify letting
> >>> working code languish?
> >>>
> >>> The greatest risk for a FOSS project is that people will ignore you.
> >>> Projects and features live and die by community buy-in. Consider the
> >>> "NA mask" feature right now. It works (at least the parts of it that
> >>> are implemented). It's in mainline. But IIRC, Pierre said last time
> >>> that he doesn't think the current design will help him improve or
> >>> replace numpy.ma. Up-thread, Wes McKinney is leaning towards ignoring
> >>> this feature in favor of his library pandas' current hacky NA support.
> >>> Members of the neuroimaging crowd are saying that the memory overhead
> >>> is too high and the benefits too marginal, so they'll stick with NaNs.
> >>> Together these folk a huge proportion of the this feature's target
> >>> audience. So what have we actually accomplished by merging this to
> >>> mainline? Are we going to be stuck supporting a feature that only a
> >>> fraction of the target audience actually uses? (Maybe they're being
> >>> dumb, but if people are ignoring your code for dumb reasons... they're
> >>> still ignoring your code.)
> >>>
> 

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Matthew Brett]

Hi,

On Fri, Oct 28, 2011 at 2:43 PM, Matthew Brett  wrote:
> Hi,
>
> On Fri, Oct 28, 2011 at 2:41 PM, Charles R Harris
>  wrote:
>>
>>
>> On Fri, Oct 28, 2011 at 3:16 PM, Nathaniel Smith  wrote:
>>>
>>> On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant 
>>> wrote:
>>> > I think Nathaniel and Matthew provided very
>>> > specific feedback that was helpful in understanding other perspectives
>>> > of a
>>> > difficult problem.     In particular, I really wanted bit-patterns
>>> > implemented.    However, I also understand that Mark did quite a bit of
>>> > work
>>> > and altered his original designs quite a bit in response to community
>>> > feedback.   I wasn't a major part of the pull request discussion, nor
>>> > did I
>>> > merge the changes, but I support Charles if he reviewed the code and
>>> > felt
>>> > like it was the right thing to do.  I likely would have done the same
>>> > thing
>>> > rather than let Mark Wiebe's work languish.
>>>
>>> My connectivity is spotty this week, so I'll stay out of the technical
>>> discussion for now, but I want to share a story.
>>>
>>> Maybe a year ago now, Jonathan Taylor and I were debating what the
>>> best API for describing statistical models would be -- whether we
>>> wanted something like R's "formulas" (which I supported), or another
>>> approach based on sympy (his idea). To summarize, I thought his API
>>> was confusing, pointlessly complicated, and didn't actually solve the
>>> problem; he thought R-style formulas were superficially simpler but
>>> hopelessly confused and inconsistent underneath. Now, obviously, I was
>>> right and he was wrong. Well, obvious to me, anyway... ;-) But it
>>> wasn't like I could just wave a wand and make his arguments go away,
>>> no matter how annoying and wrong-headed I thought they were... I could
>>> write all the code I wanted but no-one would use it unless I could
>>> convince them it's actually the right solution, so I had to engage
>>> with him, and dig deep into his arguments.
>>>
>>> What I discovered was that (as I thought) R-style formulas *do* have a
>>> solid theoretical basis -- but (as he thought) all the existing
>>> implementations *are* broken and inconsistent! I'm still not sure I
>>> can actually convince Jonathan to go my way, but, because of his
>>> stubbornness, I had to invent a better way of handling these formulas,
>>> and so my library[1] is actually the first implementation of these
>>> things that has a rigorous theory behind it, and in the process it
>>> avoids two fundamental, decades-old bugs in R. (And I'm not sure the R
>>> folks can fix either of them at this point without breaking a ton of
>>> code, since they both have API consequences.)
>>>
>>> --
>>>
>>> It's extremely common for healthy FOSS projects to insist on consensus
>>> for almost all decisions, where consensus means something like "every
>>> interested party has a veto"[2]. This seems counterintuitive, because
>>> if everyone's vetoing all the time, how does anything get done? The
>>> trick is that if anyone *can* veto, then vetoes turn out to actually
>>> be very rare. Everyone knows that they can't just ignore alternative
>>> points of view -- they have to engage with them if they want to get
>>> anything done. So you get buy-in on features early, and no vetoes are
>>> necessary. And by forcing people to engage with each other, like me
>>> with Jonathan, you get better designs.
>>>
>>> But what about the cost of all that code that doesn't get merged, or
>>> written, because everyone's spending all this time debating instead?
>>> Better designs are nice and all, but how does that justify letting
>>> working code languish?
>>>
>>> The greatest risk for a FOSS project is that people will ignore you.
>>> Projects and features live and die by community buy-in. Consider the
>>> "NA mask" feature right now. It works (at least the parts of it that
>>> are implemented). It's in mainline. But IIRC, Pierre said last time
>>> that he doesn't think the current design will help him improve or
>>> replace numpy.ma. Up-thread, Wes McKinney is leaning towards ignoring
>>> this feature in favor of his library pandas' current hacky NA support.
>>> Members of the neuroimaging crowd are saying that the memory overhead
>>> is too high and the benefits too marginal, so they'll stick with NaNs.
>>> Together these folk a huge proportion of the this feature's target
>>> audience. So what have we actually accomplished by merging this to
>>> mainline? Are we going to be stuck supporting a feature that only a
>>> fraction of the target audience actually uses? (Maybe they're being
>>> dumb, but if people are ignoring your code for dumb reasons... they're
>>> still ignoring your code.)
>>>
>>> The consensus rule forces everyone to do the hardest and riskiest part
>>> -- building buy-in -- up front. Because you *have* to do it sooner or
>>> later, and doing it sooner doesn't just generate better designs. It
>>> drastically reduces the risk of ending up in

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Matthew Brett]

Hi,

On Fri, Oct 28, 2011 at 2:41 PM, Charles R Harris
 wrote:
>
>
> On Fri, Oct 28, 2011 at 3:16 PM, Nathaniel Smith  wrote:
>>
>> On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant 
>> wrote:
>> > I think Nathaniel and Matthew provided very
>> > specific feedback that was helpful in understanding other perspectives
>> > of a
>> > difficult problem.     In particular, I really wanted bit-patterns
>> > implemented.    However, I also understand that Mark did quite a bit of
>> > work
>> > and altered his original designs quite a bit in response to community
>> > feedback.   I wasn't a major part of the pull request discussion, nor
>> > did I
>> > merge the changes, but I support Charles if he reviewed the code and
>> > felt
>> > like it was the right thing to do.  I likely would have done the same
>> > thing
>> > rather than let Mark Wiebe's work languish.
>>
>> My connectivity is spotty this week, so I'll stay out of the technical
>> discussion for now, but I want to share a story.
>>
>> Maybe a year ago now, Jonathan Taylor and I were debating what the
>> best API for describing statistical models would be -- whether we
>> wanted something like R's "formulas" (which I supported), or another
>> approach based on sympy (his idea). To summarize, I thought his API
>> was confusing, pointlessly complicated, and didn't actually solve the
>> problem; he thought R-style formulas were superficially simpler but
>> hopelessly confused and inconsistent underneath. Now, obviously, I was
>> right and he was wrong. Well, obvious to me, anyway... ;-) But it
>> wasn't like I could just wave a wand and make his arguments go away,
>> no matter how annoying and wrong-headed I thought they were... I could
>> write all the code I wanted but no-one would use it unless I could
>> convince them it's actually the right solution, so I had to engage
>> with him, and dig deep into his arguments.
>>
>> What I discovered was that (as I thought) R-style formulas *do* have a
>> solid theoretical basis -- but (as he thought) all the existing
>> implementations *are* broken and inconsistent! I'm still not sure I
>> can actually convince Jonathan to go my way, but, because of his
>> stubbornness, I had to invent a better way of handling these formulas,
>> and so my library[1] is actually the first implementation of these
>> things that has a rigorous theory behind it, and in the process it
>> avoids two fundamental, decades-old bugs in R. (And I'm not sure the R
>> folks can fix either of them at this point without breaking a ton of
>> code, since they both have API consequences.)
>>
>> --
>>
>> It's extremely common for healthy FOSS projects to insist on consensus
>> for almost all decisions, where consensus means something like "every
>> interested party has a veto"[2]. This seems counterintuitive, because
>> if everyone's vetoing all the time, how does anything get done? The
>> trick is that if anyone *can* veto, then vetoes turn out to actually
>> be very rare. Everyone knows that they can't just ignore alternative
>> points of view -- they have to engage with them if they want to get
>> anything done. So you get buy-in on features early, and no vetoes are
>> necessary. And by forcing people to engage with each other, like me
>> with Jonathan, you get better designs.
>>
>> But what about the cost of all that code that doesn't get merged, or
>> written, because everyone's spending all this time debating instead?
>> Better designs are nice and all, but how does that justify letting
>> working code languish?
>>
>> The greatest risk for a FOSS project is that people will ignore you.
>> Projects and features live and die by community buy-in. Consider the
>> "NA mask" feature right now. It works (at least the parts of it that
>> are implemented). It's in mainline. But IIRC, Pierre said last time
>> that he doesn't think the current design will help him improve or
>> replace numpy.ma. Up-thread, Wes McKinney is leaning towards ignoring
>> this feature in favor of his library pandas' current hacky NA support.
>> Members of the neuroimaging crowd are saying that the memory overhead
>> is too high and the benefits too marginal, so they'll stick with NaNs.
>> Together these folk a huge proportion of the this feature's target
>> audience. So what have we actually accomplished by merging this to
>> mainline? Are we going to be stuck supporting a feature that only a
>> fraction of the target audience actually uses? (Maybe they're being
>> dumb, but if people are ignoring your code for dumb reasons... they're
>> still ignoring your code.)
>>
>> The consensus rule forces everyone to do the hardest and riskiest part
>> -- building buy-in -- up front. Because you *have* to do it sooner or
>> later, and doing it sooner doesn't just generate better designs. It
>> drastically reduces the risk of ending up in a huge trainwreck.
>>
>> --
>>
>> In my story at the beginning, I wished I had a magic wand to skip this
>> annoying debate and political stuff. But givi

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Charles R Harris]

On Fri, Oct 28, 2011 at 3:16 PM, Nathaniel Smith  wrote:

> On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant 
> wrote:
> > I think Nathaniel and Matthew provided very
> > specific feedback that was helpful in understanding other perspectives of
> a
> > difficult problem. In particular, I really wanted bit-patterns
> > implemented.However, I also understand that Mark did quite a bit of
> work
> > and altered his original designs quite a bit in response to community
> > feedback.   I wasn't a major part of the pull request discussion, nor did
> I
> > merge the changes, but I support Charles if he reviewed the code and felt
> > like it was the right thing to do.  I likely would have done the same
> thing
> > rather than let Mark Wiebe's work languish.
>
> My connectivity is spotty this week, so I'll stay out of the technical
> discussion for now, but I want to share a story.
>
> Maybe a year ago now, Jonathan Taylor and I were debating what the
> best API for describing statistical models would be -- whether we
> wanted something like R's "formulas" (which I supported), or another
> approach based on sympy (his idea). To summarize, I thought his API
> was confusing, pointlessly complicated, and didn't actually solve the
> problem; he thought R-style formulas were superficially simpler but
> hopelessly confused and inconsistent underneath. Now, obviously, I was
> right and he was wrong. Well, obvious to me, anyway... ;-) But it
> wasn't like I could just wave a wand and make his arguments go away,
> no matter how annoying and wrong-headed I thought they were... I could
> write all the code I wanted but no-one would use it unless I could
> convince them it's actually the right solution, so I had to engage
> with him, and dig deep into his arguments.
>
> What I discovered was that (as I thought) R-style formulas *do* have a
> solid theoretical basis -- but (as he thought) all the existing
> implementations *are* broken and inconsistent! I'm still not sure I
> can actually convince Jonathan to go my way, but, because of his
> stubbornness, I had to invent a better way of handling these formulas,
> and so my library[1] is actually the first implementation of these
> things that has a rigorous theory behind it, and in the process it
> avoids two fundamental, decades-old bugs in R. (And I'm not sure the R
> folks can fix either of them at this point without breaking a ton of
> code, since they both have API consequences.)
>
> --
>
> It's extremely common for healthy FOSS projects to insist on consensus
> for almost all decisions, where consensus means something like "every
> interested party has a veto"[2]. This seems counterintuitive, because
> if everyone's vetoing all the time, how does anything get done? The
> trick is that if anyone *can* veto, then vetoes turn out to actually
> be very rare. Everyone knows that they can't just ignore alternative
> points of view -- they have to engage with them if they want to get
> anything done. So you get buy-in on features early, and no vetoes are
> necessary. And by forcing people to engage with each other, like me
> with Jonathan, you get better designs.
>
> But what about the cost of all that code that doesn't get merged, or
> written, because everyone's spending all this time debating instead?
> Better designs are nice and all, but how does that justify letting
> working code languish?
>
> The greatest risk for a FOSS project is that people will ignore you.
> Projects and features live and die by community buy-in. Consider the
> "NA mask" feature right now. It works (at least the parts of it that
> are implemented). It's in mainline. But IIRC, Pierre said last time
> that he doesn't think the current design will help him improve or
> replace numpy.ma. Up-thread, Wes McKinney is leaning towards ignoring
> this feature in favor of his library pandas' current hacky NA support.
> Members of the neuroimaging crowd are saying that the memory overhead
> is too high and the benefits too marginal, so they'll stick with NaNs.
> Together these folk a huge proportion of the this feature's target
> audience. So what have we actually accomplished by merging this to
> mainline? Are we going to be stuck supporting a feature that only a
> fraction of the target audience actually uses? (Maybe they're being
> dumb, but if people are ignoring your code for dumb reasons... they're
> still ignoring your code.)
>
> The consensus rule forces everyone to do the hardest and riskiest part
> -- building buy-in -- up front. Because you *have* to do it sooner or
> later, and doing it sooner doesn't just generate better designs. It
> drastically reduces the risk of ending up in a huge trainwreck.
>
> --
>
> In my story at the beginning, I wished I had a magic wand to skip this
> annoying debate and political stuff. But giving it to me would have
> been a bad idea. I think that's went wrong with the NA discussion in
> the first place. Mark's an excellent programmer, and he tried his best
> to

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Matthew Brett]

On Fri, Oct 28, 2011 at 2:32 PM, Matthew Brett  wrote:
> Hi,
>
> On Fri, Oct 28, 2011 at 2:16 PM, Nathaniel Smith  wrote:
>> On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant  
>> wrote:
>>> I think Nathaniel and Matthew provided very
>>> specific feedback that was helpful in understanding other perspectives of a
>>> difficult problem.     In particular, I really wanted bit-patterns
>>> implemented.    However, I also understand that Mark did quite a bit of work
>>> and altered his original designs quite a bit in response to community
>>> feedback.   I wasn't a major part of the pull request discussion, nor did I
>>> merge the changes, but I support Charles if he reviewed the code and felt
>>> like it was the right thing to do.  I likely would have done the same thing
>>> rather than let Mark Wiebe's work languish.
>>
>> My connectivity is spotty this week, so I'll stay out of the technical
>> discussion for now, but I want to share a story.
>>
>> Maybe a year ago now, Jonathan Taylor and I were debating what the
>> best API for describing statistical models would be -- whether we
>> wanted something like R's "formulas" (which I supported), or another
>> approach based on sympy (his idea). To summarize, I thought his API
>> was confusing, pointlessly complicated, and didn't actually solve the
>> problem; he thought R-style formulas were superficially simpler but
>> hopelessly confused and inconsistent underneath. Now, obviously, I was
>> right and he was wrong. Well, obvious to me, anyway... ;-) But it
>> wasn't like I could just wave a wand and make his arguments go away,
>> no matter how annoying and wrong-headed I thought they were... I could
>> write all the code I wanted but no-one would use it unless I could
>> convince them it's actually the right solution, so I had to engage
>> with him, and dig deep into his arguments.
>>
>> What I discovered was that (as I thought) R-style formulas *do* have a
>> solid theoretical basis -- but (as he thought) all the existing
>> implementations *are* broken and inconsistent! I'm still not sure I
>> can actually convince Jonathan to go my way, but, because of his
>> stubbornness, I had to invent a better way of handling these formulas,
>> and so my library[1] is actually the first implementation of these
>> things that has a rigorous theory behind it, and in the process it
>> avoids two fundamental, decades-old bugs in R. (And I'm not sure the R
>> folks can fix either of them at this point without breaking a ton of
>> code, since they both have API consequences.)
>>
>> --
>>
>> It's extremely common for healthy FOSS projects to insist on consensus
>> for almost all decisions, where consensus means something like "every
>> interested party has a veto"[2]. This seems counterintuitive, because
>> if everyone's vetoing all the time, how does anything get done? The
>> trick is that if anyone *can* veto, then vetoes turn out to actually
>> be very rare. Everyone knows that they can't just ignore alternative
>> points of view -- they have to engage with them if they want to get
>> anything done. So you get buy-in on features early, and no vetoes are
>> necessary. And by forcing people to engage with each other, like me
>> with Jonathan, you get better designs.
>>
>> But what about the cost of all that code that doesn't get merged, or
>> written, because everyone's spending all this time debating instead?
>> Better designs are nice and all, but how does that justify letting
>> working code languish?
>>
>> The greatest risk for a FOSS project is that people will ignore you.
>> Projects and features live and die by community buy-in. Consider the
>> "NA mask" feature right now. It works (at least the parts of it that
>> are implemented). It's in mainline. But IIRC, Pierre said last time
>> that he doesn't think the current design will help him improve or
>> replace numpy.ma. Up-thread, Wes McKinney is leaning towards ignoring
>> this feature in favor of his library pandas' current hacky NA support.
>> Members of the neuroimaging crowd are saying that the memory overhead
>> is too high and the benefits too marginal, so they'll stick with NaNs.
>> Together these folk a huge proportion of the this feature's target
>> audience. So what have we actually accomplished by merging this to
>> mainline? Are we going to be stuck supporting a feature that only a
>> fraction of the target audience actually uses? (Maybe they're being
>> dumb, but if people are ignoring your code for dumb reasons... they're
>> still ignoring your code.)
>>
>> The consensus rule forces everyone to do the hardest and riskiest part
>> -- building buy-in -- up front. Because you *have* to do it sooner or
>> later, and doing it sooner doesn't just generate better designs. It
>> drastically reduces the risk of ending up in a huge trainwreck.
>>
>> --
>>
>> In my story at the beginning, I wished I had a magic wand to skip this
>> annoying debate and political stuff. But giving it to me would have
>> been a bad ide

Re: [Numpy-discussion] consensus (was: NA masks in the next numpy release?)

[Matthew Brett]

Hi,

On Fri, Oct 28, 2011 at 2:16 PM, Nathaniel Smith  wrote:
> On Tue, Oct 25, 2011 at 2:56 PM, Travis Oliphant  
> wrote:
>> I think Nathaniel and Matthew provided very
>> specific feedback that was helpful in understanding other perspectives of a
>> difficult problem.     In particular, I really wanted bit-patterns
>> implemented.    However, I also understand that Mark did quite a bit of work
>> and altered his original designs quite a bit in response to community
>> feedback.   I wasn't a major part of the pull request discussion, nor did I
>> merge the changes, but I support Charles if he reviewed the code and felt
>> like it was the right thing to do.  I likely would have done the same thing
>> rather than let Mark Wiebe's work languish.
>
> My connectivity is spotty this week, so I'll stay out of the technical
> discussion for now, but I want to share a story.
>
> Maybe a year ago now, Jonathan Taylor and I were debating what the
> best API for describing statistical models would be -- whether we
> wanted something like R's "formulas" (which I supported), or another
> approach based on sympy (his idea). To summarize, I thought his API
> was confusing, pointlessly complicated, and didn't actually solve the
> problem; he thought R-style formulas were superficially simpler but
> hopelessly confused and inconsistent underneath. Now, obviously, I was
> right and he was wrong. Well, obvious to me, anyway... ;-) But it
> wasn't like I could just wave a wand and make his arguments go away,
> no matter how annoying and wrong-headed I thought they were... I could
> write all the code I wanted but no-one would use it unless I could
> convince them it's actually the right solution, so I had to engage
> with him, and dig deep into his arguments.
>
> What I discovered was that (as I thought) R-style formulas *do* have a
> solid theoretical basis -- but (as he thought) all the existing
> implementations *are* broken and inconsistent! I'm still not sure I
> can actually convince Jonathan to go my way, but, because of his
> stubbornness, I had to invent a better way of handling these formulas,
> and so my library[1] is actually the first implementation of these
> things that has a rigorous theory behind it, and in the process it
> avoids two fundamental, decades-old bugs in R. (And I'm not sure the R
> folks can fix either of them at this point without breaking a ton of
> code, since they both have API consequences.)
>
> --
>
> It's extremely common for healthy FOSS projects to insist on consensus
> for almost all decisions, where consensus means something like "every
> interested party has a veto"[2]. This seems counterintuitive, because
> if everyone's vetoing all the time, how does anything get done? The
> trick is that if anyone *can* veto, then vetoes turn out to actually
> be very rare. Everyone knows that they can't just ignore alternative
> points of view -- they have to engage with them if they want to get
> anything done. So you get buy-in on features early, and no vetoes are
> necessary. And by forcing people to engage with each other, like me
> with Jonathan, you get better designs.
>
> But what about the cost of all that code that doesn't get merged, or
> written, because everyone's spending all this time debating instead?
> Better designs are nice and all, but how does that justify letting
> working code languish?
>
> The greatest risk for a FOSS project is that people will ignore you.
> Projects and features live and die by community buy-in. Consider the
> "NA mask" feature right now. It works (at least the parts of it that
> are implemented). It's in mainline. But IIRC, Pierre said last time
> that he doesn't think the current design will help him improve or
> replace numpy.ma. Up-thread, Wes McKinney is leaning towards ignoring
> this feature in favor of his library pandas' current hacky NA support.
> Members of the neuroimaging crowd are saying that the memory overhead
> is too high and the benefits too marginal, so they'll stick with NaNs.
> Together these folk a huge proportion of the this feature's target
> audience. So what have we actually accomplished by merging this to
> mainline? Are we going to be stuck supporting a feature that only a
> fraction of the target audience actually uses? (Maybe they're being
> dumb, but if people are ignoring your code for dumb reasons... they're
> still ignoring your code.)
>
> The consensus rule forces everyone to do the hardest and riskiest part
> -- building buy-in -- up front. Because you *have* to do it sooner or
> later, and doing it sooner doesn't just generate better designs. It
> drastically reduces the risk of ending up in a huge trainwreck.
>
> --
>
> In my story at the beginning, I wished I had a magic wand to skip this
> annoying debate and political stuff. But giving it to me would have
> been a bad idea. I think that's went wrong with the NA discussion in
> the first place. Mark's an excellent programmer, and he tried his best
> to act in the