Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: "fancy" vs. orthogonal)
On Thu, Apr 9, 2015 at 10:11 AM, Alan G Isaac wrote: > > Alan wrote: >>> 3. I admit, my students are NOT using non-boolen fancy indexing on >>> >multidimensional arrays. (As far as I know.) Are yours? The only confusing case is mixing slices and integer array indexing for ndim > 2. The rest looks unsurprising, AFAIR (AFAICS, my last fancy indexing mailing list discussion is at least 4 years old, with Jonathan Taylor. I don't remember when I discovered the usefulness of the axis argument in take which covers many 3 or higher dimensional indexing use cases.) > > > On 4/9/2015 2:22 AM, Nathaniel Smith wrote: >> Well, okay, this would explain it, since integer fancy indexing is >> exactly the confusing case:-) On the plus side, this also means that >> even if pigs started doing barrel-rolls through hell's >> winter-vortex-chilled air tomorrow and we simply removed integer fancy >> indexing, your students would be unaffected:-) > > > Except that they do use statsmodels, which I believe (?) does make use of > integer fancy-indexing. And maybe all work would come to a standstill, because every library is using fancy integer indexing. I still don't know what all constitutes fancy indexing. The two most common use cases for me (statsmodels) are indexing for selecting elements like diag_indices, triu_indices and maybe nonzero, and expanding from a unique array like inverse index in np.unique. And there are just a few, AFAIR, orthogonal indexing cases with broadcasting index arrays to select rectangular pieces of an array. Josef > > Alan > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: "fancy" vs. orthogonal)
> Alan wrote: >> 3. I admit, my students are NOT using non-boolen fancy indexing on >> >multidimensional arrays. (As far as I know.) Are yours? On 4/9/2015 2:22 AM, Nathaniel Smith wrote: > Well, okay, this would explain it, since integer fancy indexing is > exactly the confusing case:-) On the plus side, this also means that > even if pigs started doing barrel-rolls through hell's > winter-vortex-chilled air tomorrow and we simply removed integer fancy > indexing, your students would be unaffected:-) Except that they do use statsmodels, which I believe (?) does make use of integer fancy-indexing. Alan ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: "fancy" vs. orthogonal)
On 4/9/2015 1:57 AM, Nathaniel Smith wrote: > Do you think there's anything we could be > doing to reduce this kind of adrenaline reaction while still allowing > for relaxed discussion about out-there ideas? numpy3...@scipy.org :-) ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: "fancy" vs. orthogonal)
On Do, 2015-04-09 at 02:22 -0400, Nathaniel Smith wrote: > On Wed, Apr 8, 2015 at 4:02 PM, Alan G Isaac wrote: > > 1. I use numpy in teaching. > > I have never heard a complaint about its indexing behavior. > > Have you heard such complaints? > > Some observations: > > 1) There's an unrelated thread on numpy-discussion right now in which > a user is baffled by the interaction between slicing and integer fancy > indexing: > http://thread.gmane.org/gmane.comp.python.numeric.general/60321 > And one of the three replies AFAICT also doesn't actually make sense, > in that its explanation relies on broadcasting two arrays with shape > (5,) against each other to produce an array with shape (5, 5). (Which > is not how broadcasting works.) To be fair, though, this isn't the > poster's fault, because they are quoting the documentation! > > 2) Again, entirely by coincidence, literally this week a numpy user at > Berkeley felt spontaneously moved to send a warning message to the > campus py4science list just to warn everyone about the bizarre > behaviour they had stumbled on where arr[0, :, idx] produced > inexplicable results. They had already found the docs and worked out > what was going on, they just felt it was necessary to warn everyone > else to be careful out there. > > 3) I personally regularly get confused by integer fancy indexing. I > actually understand it substantially better due to thinking it through > while reading these threads, but I'm a bit disturbed that I had that > much left to learn. (New key insight: you can think of *scalar* > indexing arr[i, j, k] as a function f(i, j, k) -> value. If you take > that function and make it a ufunc, then you have integer fancy > indexing. ...Though there's still an extra pound of explanation needed > to describe the mixed slice/fancy cases, it at least captures the > basic intuition. Maybe this was already obvious to everyone else, but > it helped me.) > > 4) Even with my New and Improved Explanatory Powers, when this thread > came up chatting with Thomas Kluyver today, I attempted to provide a > simple, accurate description of how numpy indexing works so that the > debate would make sense, and his conclusion was (paraphrased) "okay, > now I don't understand numpy indexing anymore and never did". I say > this not to pick on Thomas, but to make that point that Thomas is a > pretty smart guy so maybe this is actually confusing. (Or maybe I'm > just terrible at explaining things.) > > I actually think the evidence is very very strong that numpy's current > way of mixing integer fancy indexing and slice-based indexing is a > mistake. It's just not clear whether there's anything we can do to > mitigate that mistake (or indeed, what would actually be better even > if we could start over from scratch). (Which we can't.) > I think the best way to think about the mixing is to think about "subspaces" defined by all of the slices which are taken for each individual fancy indexing "element". I.e. each subspaces is something like: new[:, 0, :] = arr[:, fancy1[0], fancy2[0], :] then you iterate the fancy indexes so the subspaces moves ahead: new[:, 1, :] = arr[:, fancy1[1], fancy2[1], :] new[:, 2, :] = arr[:, fancy1[2], fancy2[2], :] and so on. This is also how it is implemented. Plus of course the transposing to the front when the fancy indices are not consecutive and you cannot add the fancy dimensions to where they were. I think you mentioned an error in the docu, I thought I cleared some of them, but proabably that did not make it more understandable sometimes. The whole subspace way of is used, but there is a lot of improvement possible and I would be happy if more feel like stepping up to fill that void, since you do not need to know the implementation details for that. - Sebastian > > 2. One reason I use numpy in teaching is its indexing behavior. > > What specific language provides a better indexing model, > > in your opinion? > > > > 3. I admit, my students are NOT using non-boolen fancy indexing on > > multidimensional arrays. (As far as I know.) Are yours? > > Well, okay, this would explain it, since integer fancy indexing is > exactly the confusing case :-) On the plus side, this also means that > even if pigs started doing barrel-rolls through hell's > winter-vortex-chilled air tomorrow and we simply removed integer fancy > indexing, your students would be unaffected :-) > > -n > signature.asc Description: This is a digitally signed message part ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: "fancy" vs. orthogonal)
On Wed, Apr 8, 2015 at 4:02 PM, Alan G Isaac wrote: > 1. I use numpy in teaching. > I have never heard a complaint about its indexing behavior. > Have you heard such complaints? Some observations: 1) There's an unrelated thread on numpy-discussion right now in which a user is baffled by the interaction between slicing and integer fancy indexing: http://thread.gmane.org/gmane.comp.python.numeric.general/60321 And one of the three replies AFAICT also doesn't actually make sense, in that its explanation relies on broadcasting two arrays with shape (5,) against each other to produce an array with shape (5, 5). (Which is not how broadcasting works.) To be fair, though, this isn't the poster's fault, because they are quoting the documentation! 2) Again, entirely by coincidence, literally this week a numpy user at Berkeley felt spontaneously moved to send a warning message to the campus py4science list just to warn everyone about the bizarre behaviour they had stumbled on where arr[0, :, idx] produced inexplicable results. They had already found the docs and worked out what was going on, they just felt it was necessary to warn everyone else to be careful out there. 3) I personally regularly get confused by integer fancy indexing. I actually understand it substantially better due to thinking it through while reading these threads, but I'm a bit disturbed that I had that much left to learn. (New key insight: you can think of *scalar* indexing arr[i, j, k] as a function f(i, j, k) -> value. If you take that function and make it a ufunc, then you have integer fancy indexing. ...Though there's still an extra pound of explanation needed to describe the mixed slice/fancy cases, it at least captures the basic intuition. Maybe this was already obvious to everyone else, but it helped me.) 4) Even with my New and Improved Explanatory Powers, when this thread came up chatting with Thomas Kluyver today, I attempted to provide a simple, accurate description of how numpy indexing works so that the debate would make sense, and his conclusion was (paraphrased) "okay, now I don't understand numpy indexing anymore and never did". I say this not to pick on Thomas, but to make that point that Thomas is a pretty smart guy so maybe this is actually confusing. (Or maybe I'm just terrible at explaining things.) I actually think the evidence is very very strong that numpy's current way of mixing integer fancy indexing and slice-based indexing is a mistake. It's just not clear whether there's anything we can do to mitigate that mistake (or indeed, what would actually be better even if we could start over from scratch). (Which we can't.) > 2. One reason I use numpy in teaching is its indexing behavior. > What specific language provides a better indexing model, > in your opinion? > > 3. I admit, my students are NOT using non-boolen fancy indexing on > multidimensional arrays. (As far as I know.) Are yours? Well, okay, this would explain it, since integer fancy indexing is exactly the confusing case :-) On the plus side, this also means that even if pigs started doing barrel-rolls through hell's winter-vortex-chilled air tomorrow and we simply removed integer fancy indexing, your students would be unaffected :-) -n -- Nathaniel J. Smith -- http://vorpus.org ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: "fancy" vs. orthogonal)
On Wed, Apr 8, 2015 at 2:09 PM, Alan G Isaac wrote: > That analogy fails because it suggests a private conversation. This list is > extremely public. > For example, I am just a user, and I am on it. I can tell you that as a > long-time numpy user > my reaction to the proposal to change indexing semantics was (i) OMG YMBFKM > and then > (ii) take a breath; this too will fade away. It is very reasonable to worry > that some users > will start at the same place but them move in a different direction, and that > worry should > affect how such proposals are floated and discussed. I am personally > grateful that the > idea's reception has been so chilly; it's very reassuring. Thanks, this is really useful feedback. I can totally understand that panic flare and it sucks. Do you think there's anything we could be doing to reduce this kind of adrenaline reaction while still allowing for relaxed discussion about out-there ideas? In my mind the "relaxed" part is actually a huge part of the goal: reaching the point where everyone can be confident that their voice will be heard, etc., so that things become less fraught and it becomes easier to focus on ideas. -n -- Nathaniel J. Smith -- http://vorpus.org ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: "fancy" vs. orthogonal)
On Wed, Apr 8, 2015 at 10:20 PM, Robert Kern wrote: > On Wed, Apr 8, 2015 at 8:40 PM, Ralf Gommers > wrote: > > > To address in detail the list of Matthew you mention above: > > > > * implement orthogonal indexing as a method arr.sensible_index[...] > > That's basically Jaime's PR. > > > > * implement the current non-boolean fancy indexing behavior as a > method - arr.crazy_index[...] > > Not that harmful, but only makes sense in combination with the next > steps. > > Well, since we got the peanut butter in our meta, I might as well join in > here. > Yeah, sorry about that. I don't even like peanut butter. > I think this step is useful even without the deprecation steps. > Your arguments make sense. Ralf ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: "fancy" vs. orthogonal)
Trying to bring the meta back into this thread (sorry for Robert's PB :)... The only thing I'd like to add, is that it's perhaps worth messaging that: a PR is just (as the Github folks like to say) "a conversation based on code". It is NOT necessarily something intended explicitly for merging. In IPython, we do sometimes create PRs that we mark explicitly as *Not for merging*, just so that we can play with a concrete implementation of an idea, even when we know in advance it's not going to go in. But it may be a useful way to explore the problem with code everyone can easily grab and run, to have a thread of discussion right there next to the code, to evolve the code together with the discussions as insights arise, and to finally document anything learned, all in one place. So, with a bit of messaging, "encouraging PRs" doesn't need to be seen as "the numpy core devs would like to see every last crazy idea you have in mind to see what we can merge in our next drunken stupor". But rather, "some ideas, even crazy ones, are easier to understand when accompanied by code, could you send a PR for what you have in mind, knowing we're nearly certain we won't merge it, but it will make it easier for us to have a fruitful discussion with you". Back to lurking ;) f -- Fernando Perez (@fperez_org; http://fperez.org) fperez.net-at-gmail: mailing lists only (I ignore this when swamped!) fernando.perez-at-berkeley: contact me here for any direct mail ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: "fancy" vs. orthogonal)
On Wed, Apr 8, 2015 at 8:40 PM, Ralf Gommers wrote: > To address in detail the list of Matthew you mention above: > > * implement orthogonal indexing as a method arr.sensible_index[...] > That's basically Jaime's PR. > > * implement the current non-boolean fancy indexing behavior as a method - arr.crazy_index[...] > Not that harmful, but only makes sense in combination with the next steps. Well, since we got the peanut butter in our meta, I might as well join in here. I think this step is useful even without the deprecation steps. First, allow me to rename things in a less judgy fashion: * arr.ortho_ix is a property that allows for orthogonal indexing, a la Jaime's PR. * arr.fancy_ix is a property that implements the current numpy-standard fancy indexing. Even though arr.fancy_ix "only" replicating the default semantics, having it opens up some possibilities. Other array-like objects can implement both of these with the same names. Thus, to write generic code that exploits the particular behaviors of one of these semantics, you just make sure to use the property that behaves the way you want. Then you don't care what the default syntax does on that object. You don't have to test if an object has arr.__orthogonal_indexing__ and have two different code paths; you just use the property that behaves the way you want. It also allows us to issue warnings when the default indexing syntax is used for some of the behaviors that are weird corner cases, like [1, :, array]. This is one of those corner cases where the behavior is probably not what anyone actually *wants*; it was just the only thing we could do that is consistent with the desired semantics of the rest of the cases. I think it would be reasonable to issue a warning if the default indexing syntax was used with it. It's probably a sign that the user thought that indexing worked like orthogonal indexing. The warning would *not* be issued if the arr.fancy_ix property was used, since that is an explicit signal that the user is specifically requesting a particular set of behaviors. I probably won't want to ever *deprecate* the behavior for the default syntax, but a warning is easy to deal with even with old code that you don't want to modify directly. Lastly, I would appreciate having some signal to tell readers "pay attention; this is nontrivial index manipulation; here is a googleable term so you can look up what this means". I almost always use the default fancy indexing, but I'd use the arr.fancy_ix property for the nontrivial cases just for this alone. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: "fancy" vs. orthogonal)
On 2015/04/08 10:02 AM, Alan G Isaac wrote: > 3. I admit, my students are NOT using non-boolen fancy indexing on > multidimensional arrays. (As far as I know.) Are yours? Yes, one attempted to, essentially by accident. That was in my original message. Please refer back to that. The earlier part of this thread, under its original name, is also relevant to your other questions. I'm not going to discuss this further. The thread is now closed as far as I am concerned. Eric ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: "fancy" vs. orthogonal)
1. I use numpy in teaching. I have never heard a complaint about its indexing behavior. Have you heard such complaints? 2. One reason I use numpy in teaching is its indexing behavior. What specific language provides a better indexing model, in your opinion? 3. I admit, my students are NOT using non-boolen fancy indexing on multidimensional arrays. (As far as I know.) Are yours? Cheers, Alan On 4/8/2015 3:05 PM, Eric Firing wrote: > What sequence of steps might reduce the disconnect between numpy and the > rest of the array-handling world? And make it a little friendlier for > students? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: "fancy" vs. orthogonal)
On 2015/04/08 9:40 AM, Ralf Gommers wrote: > Their proposal is not being discussed; instead that potentially useful > discussion is being completely derailed by insisting on wanting to talk > about changes to numpy's indexing behavior. Good point. That was an unintended consequence of my message. Eric ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: "fancy" vs. orthogonal)
On Wed, Apr 8, 2015 at 9:05 PM, Eric Firing wrote: > On 2015/04/08 8:09 AM, Alan G Isaac wrote: > > That analogy fails because it suggests a private conversation. This list > is extremely public. > > For example, I am just a user, and I am on it. I can tell you that as a > long-time numpy user > > my reaction to the proposal to change indexing semantics was (i) OMG > YMBFKM and then > > (ii) take a breath; this too will fade away. It is very reasonable to > worry that some users > > will start at the same place but them move in a different direction, and > that worry should > > affect how such proposals are floated and discussed. I am personally > grateful that the > > idea's reception has been so chilly; it's very reassuring. > > OK, so I was not sufficiently tactful when I tried to illustrate the > real practical problem associated with a *core* aspect of numpy. My > intent was not to alarm users, and I apologize if I have done so. I'm > glad you have been reassured. I know perfectly well that > back-compatibility and stability are highly important. What I wanted to > do was to stimulate thought about how to handle a serious challenge to > numpy's future--short-term, and long-term. Jaime's PR is a very welcome > response to that challenge, but it might not be the end of the story. > Matthew nicely sketched out one possible scenario, or actually a range > of scenarios. > > Now, can we please get back to consideration of reasonable options? > Well, in many people's definition of reasonable, that's basically Jaime's proposal and maybe the original __orthogonal_indexing__ one. Those both have a chance of actually being implemented, and presumably the original proposers have a use for the latter. Their proposal is not being discussed; instead that potentially useful discussion is being completely derailed by insisting on wanting to talk about changes to numpy's indexing behavior. To address in detail the list of Matthew you mention above: * implement orthogonal indexing as a method arr.sensible_index[...] That's basically Jaime's PR. * implement the current non-boolean fancy indexing behavior as a method - arr.crazy_index[...] Not that harmful, but only makes sense in combination with the next steps. * deprecate non-boolean fancy indexing as standard arr[...] indexing; No, see negative reaction by many people. * wait a long time; * remove non-boolean fancy indexing as standard arr[...] (errors are preferable to change in behavior) Most definitely no. Ralf P.S. random thought: maybe we should have a "numpy-ideas" list, just like python-dev has python-ideas ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: "fancy" vs. orthogonal)
On Wed, Apr 8, 2015 at 8:05 PM, Eric Firing wrote: > Now, can we please get back to consideration of reasonable options? Sure, but I recommend going back to the actually topical thread (or a new one), as this one is meta. -- Robert Kern ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: "fancy" vs. orthogonal)
On 2015/04/08 8:09 AM, Alan G Isaac wrote: > That analogy fails because it suggests a private conversation. This list is > extremely public. > For example, I am just a user, and I am on it. I can tell you that as a > long-time numpy user > my reaction to the proposal to change indexing semantics was (i) OMG YMBFKM > and then > (ii) take a breath; this too will fade away. It is very reasonable to worry > that some users > will start at the same place but them move in a different direction, and that > worry should > affect how such proposals are floated and discussed. I am personally > grateful that the > idea's reception has been so chilly; it's very reassuring. OK, so I was not sufficiently tactful when I tried to illustrate the real practical problem associated with a *core* aspect of numpy. My intent was not to alarm users, and I apologize if I have done so. I'm glad you have been reassured. I know perfectly well that back-compatibility and stability are highly important. What I wanted to do was to stimulate thought about how to handle a serious challenge to numpy's future--short-term, and long-term. Jaime's PR is a very welcome response to that challenge, but it might not be the end of the story. Matthew nicely sketched out one possible scenario, or actually a range of scenarios. Now, can we please get back to consideration of reasonable options? What sequence of steps might reduce the disconnect between numpy and the rest of the array-handling world? And make it a little friendlier for students? Are there *any* changes to indexing, whether by default or as an option, that would help? Consider the example I started with, in which indexing with [1, :, array] gives results that many find surprising and hard to understand. Might it make sense to *slowly* deprecate this? Or are such indexing expressions actually useful? If they are, would it be out of the question to have them *optionally* trigger a warning, so that numpy could be configured to be a little less likely to trip up a non-expert user? Eric > > fwiw, > Alan > > > On 4/7/2015 9:06 PM, Nathaniel Smith wrote: >> If a grad student or junior colleague comes to you with an >> idea where you see some potentially critical flaw, do you >> yell THAT WILL NEVER WORK and kick them out of your >> office? Or, do you maybe ask a few leading questions and >> see where they go? > > ___ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: "fancy" vs. orthogonal)
On Wed, Apr 8, 2015 at 1:38 PM, Robert Kern wrote: > On Wed, Apr 8, 2015 at 2:06 AM, Nathaniel Smith wrote: >> >> On Apr 5, 2015 7:04 AM, "Robert Kern" wrote: >> > >> > On Sat, Apr 4, 2015 at 10:38 PM, Nathaniel Smith wrote: >> > > >> > > On Apr 4, 2015 4:12 AM, "Todd" wrote: >> > > > >> > > > There was no break as large as this. In fact I would say this is >> > > > even a larger change than any individual change we saw in the python 2 >> > > > to 3 >> > > > switch. The basic mechanics of indexing are just too fundamental and >> > > > touch >> > > > on too many things to make this sort of change feasible. >> > > >> > > I'm afraid I'm not clever enough to know how large or feasible a >> > > change is without even seeing the proposed change. >> > >> > It doesn't take any cleverness. The change in question was to make the >> > default indexing semantics to orthogonal indexing. No matter the details of >> > the ultimate proposal to achieve that end, it has known minimum >> > consequences, at least in the broad outline. Current documentation and >> > books >> > become obsolete for a fundamental operation. Current code must be modified >> > by some step to continue working. These are consequences inherent in the >> > end, not just the means to the end; we don't need a concrete proposal in >> > front of us to know what they are. There are ways to mitigate these >> > consequences, but there are no silver bullets that eliminate them. And we >> > can compare those consequences to approaches like Jaime's that achieve a >> > majority of the benefits of such a change without any of the negative >> > consequences. That comparison does not bode well for any proposal. >> >> Ok, let me try to make my point another way. >> >> I don't actually care at this stage in the discussion whether the change >> is ultimately viable. And I don't think you should either. (For values of >> "you" that includes everyone in the discussion, not picking on Robert in >> particular :-).) >> >> My point is that rational, effective discussion requires giving ideas room >> to breath. Sometimes ideas turn out to be not as bad as they looked. >> Sometimes it turns out that they are, but there's some clever tweak that >> gives you 95% of the benefits for 5% of the cost. Sometimes you generate a >> better understanding of the tradeoffs that subsequently informs later design >> decisions. Sometimes working through the details makes both sides realize >> that there's a third option that solves both their problems. Sometimes you >> merely get a very specific understanding of why the whole approach is >> unreasonable that you can then, say, take to the pandas and netcdf >> developers as evidence of that you made a good faith effort and ask them to >> meet you half way. And all these things require understanding the specifics >> of what *exactly* works or doesn't work about about idea. IMHO, it's >> extremely misleading at this stage to make any assertion about whether >> Jaime's approach gives the "majority of benefits of such a change" is >> extremely misleading at this stage: not because it's wrong, but because it >> totally short-circuits the discussion about what benefits we care about. >> Jaime's patch certainly has merits, but does it help us make numpy and >> pandas/netcdf's more compatible? Does it make it easier for Eric to teach? >> Those are some specific merits that we might care about a lot, and for which >> Jaime's patch may or may not help much. But that kind of nuance gets lost >> when we jump straight to debating thumbs-up versus thumbs-down. > > And we can get all of that discussion from discussing Jaime's proposal. I > would argue that we will get better, more focused discussion from it since > it is actually a concrete proposal and not just a wish that numpy's indexing > semantics were something else. I think that a full airing and elaboration of > Jaime's proposal (as the final PR should look quite different than the > initial one to incorporate the what is found in the discussion) will give us > a satisficing solution. I certainly think that that is *more likely* to > arrive at a satisficing solution than an attempt to change the default > indexing semantics. I can name specific improvements that would specifically > address the concerns you named if you would like. Maybe it won't be *quite* > as good (for some parties) than if Numeric chose orthogonal indexing from > the get-go, but it will likely be much better for everyone than if numpy > broke backward compatibility on this feature now. > >> I cross-my-heart promise that under the current regime, no PR breaking >> fancy indexing would ever get anywhere *near* numpy master without >> *extensive* discussion and warnings on the list. The core devs just spent >> weeks quibbling about whether a PR that adds a new field to the end of the >> dtype struct would break ABI backcompat (we're now pretty sure it doesn't), >> and the current standard we enforce is that every PR that t
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: "fancy" vs. orthogonal)
That analogy fails because it suggests a private conversation. This list is extremely public. For example, I am just a user, and I am on it. I can tell you that as a long-time numpy user my reaction to the proposal to change indexing semantics was (i) OMG YMBFKM and then (ii) take a breath; this too will fade away. It is very reasonable to worry that some users will start at the same place but them move in a different direction, and that worry should affect how such proposals are floated and discussed. I am personally grateful that the idea's reception has been so chilly; it's very reassuring. fwiw, Alan On 4/7/2015 9:06 PM, Nathaniel Smith wrote: > If a grad student or junior colleague comes to you with an > idea where you see some potentially critical flaw, do you > yell THAT WILL NEVER WORK and kick them out of your > office? Or, do you maybe ask a few leading questions and > see where they go? ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Re: [Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: "fancy" vs. orthogonal)
On Wed, Apr 8, 2015 at 2:06 AM, Nathaniel Smith wrote: > > On Apr 5, 2015 7:04 AM, "Robert Kern" wrote: > > > > On Sat, Apr 4, 2015 at 10:38 PM, Nathaniel Smith wrote: > > > > > > On Apr 4, 2015 4:12 AM, "Todd" wrote: > > > > > > > > There was no break as large as this. In fact I would say this is even a larger change than any individual change we saw in the python 2 to 3 switch. The basic mechanics of indexing are just too fundamental and touch on too many things to make this sort of change feasible. > > > > > > I'm afraid I'm not clever enough to know how large or feasible a change is without even seeing the proposed change. > > > > It doesn't take any cleverness. The change in question was to make the default indexing semantics to orthogonal indexing. No matter the details of the ultimate proposal to achieve that end, it has known minimum consequences, at least in the broad outline. Current documentation and books become obsolete for a fundamental operation. Current code must be modified by some step to continue working. These are consequences inherent in the end, not just the means to the end; we don't need a concrete proposal in front of us to know what they are. There are ways to mitigate these consequences, but there are no silver bullets that eliminate them. And we can compare those consequences to approaches like Jaime's that achieve a majority of the benefits of such a change without any of the negative consequences. That comparison does not bode well for any proposal. > > Ok, let me try to make my point another way. > > I don't actually care at this stage in the discussion whether the change is ultimately viable. And I don't think you should either. (For values of "you" that includes everyone in the discussion, not picking on Robert in particular :-).) > > My point is that rational, effective discussion requires giving ideas room to breath. Sometimes ideas turn out to be not as bad as they looked. Sometimes it turns out that they are, but there's some clever tweak that gives you 95% of the benefits for 5% of the cost. Sometimes you generate a better understanding of the tradeoffs that subsequently informs later design decisions. Sometimes working through the details makes both sides realize that there's a third option that solves both their problems. Sometimes you merely get a very specific understanding of why the whole approach is unreasonable that you can then, say, take to the pandas and netcdf developers as evidence of that you made a good faith effort and ask them to meet you half way. And all these things require understanding the specifics of what *exactly* works or doesn't work about about idea. IMHO, it's extremely misleading at this stage to make any assertion about whether Jaime's approach gives the "majority of benefits of such a change" is extremely misleading at this stage: not because it's wrong, but because it totally short-circuits the discussion about what benefits we care about. Jaime's patch certainly has merits, but does it help us make numpy and pandas/netcdf's more compatible? Does it make it easier for Eric to teach? Those are some specific merits that we might care about a lot, and for which Jaime's patch may or may not help much. But that kind of nuance gets lost when we jump straight to debating thumbs-up versus thumbs-down. And we can get all of that discussion from discussing Jaime's proposal. I would argue that we will get better, more focused discussion from it since it is actually a concrete proposal and not just a wish that numpy's indexing semantics were something else. I think that a full airing and elaboration of Jaime's proposal (as the final PR should look quite different than the initial one to incorporate the what is found in the discussion) will give us a satisficing solution. I certainly think that that is *more likely* to arrive at a satisficing solution than an attempt to change the default indexing semantics. I can name specific improvements that would specifically address the concerns you named if you would like. Maybe it won't be *quite* as good (for some parties) than if Numeric chose orthogonal indexing from the get-go, but it will likely be much better for everyone than if numpy broke backward compatibility on this feature now. > I cross-my-heart promise that under the current regime, no PR breaking fancy indexing would ever get anywhere *near* numpy master without *extensive* discussion and warnings on the list. The core devs just spent weeks quibbling about whether a PR that adds a new field to the end of the dtype struct would break ABI backcompat (we're now pretty sure it doesn't), and the current standard we enforce is that every PR that touches public API needs a list discussion, even minor extensions with no compatibility issues at all. No one is going to sneak anything by anyone. That is not the issue. Ralf asked you not to invite such PRs in the first place. No one thinks that such a PR would get "snuck" in. That's not anyone's
[Numpy-discussion] On responding to dubious ideas (was: Re: Advanced indexing: "fancy" vs. orthogonal)
On Apr 5, 2015 7:04 AM, "Robert Kern" wrote: > > On Sat, Apr 4, 2015 at 10:38 PM, Nathaniel Smith wrote: > > > > On Apr 4, 2015 4:12 AM, "Todd" wrote: > > > > > > There was no break as large as this. In fact I would say this is even a larger change than any individual change we saw in the python 2 to 3 switch. The basic mechanics of indexing are just too fundamental and touch on too many things to make this sort of change feasible. > > > > I'm afraid I'm not clever enough to know how large or feasible a change is without even seeing the proposed change. > > It doesn't take any cleverness. The change in question was to make the default indexing semantics to orthogonal indexing. No matter the details of the ultimate proposal to achieve that end, it has known minimum consequences, at least in the broad outline. Current documentation and books become obsolete for a fundamental operation. Current code must be modified by some step to continue working. These are consequences inherent in the end, not just the means to the end; we don't need a concrete proposal in front of us to know what they are. There are ways to mitigate these consequences, but there are no silver bullets that eliminate them. And we can compare those consequences to approaches like Jaime's that achieve a majority of the benefits of such a change without any of the negative consequences. That comparison does not bode well for any proposal. Ok, let me try to make my point another way. I don't actually care at this stage in the discussion whether the change is ultimately viable. And I don't think you should either. (For values of "you" that includes everyone in the discussion, not picking on Robert in particular :-).) My point is that rational, effective discussion requires giving ideas room to breath. Sometimes ideas turn out to be not as bad as they looked. Sometimes it turns out that they are, but there's some clever tweak that gives you 95% of the benefits for 5% of the cost. Sometimes you generate a better understanding of the tradeoffs that subsequently informs later design decisions. Sometimes working through the details makes both sides realize that there's a third option that solves both their problems. Sometimes you merely get a very specific understanding of why the whole approach is unreasonable that you can then, say, take to the pandas and netcdf developers as evidence of that you made a good faith effort and ask them to meet you half way. And all these things require understanding the specifics of what *exactly* works or doesn't work about about idea. IMHO, it's extremely misleading at this stage to make any assertion about whether Jaime's approach gives the "majority of benefits of such a change" is extremely misleading at this stage: not because it's wrong, but because it totally short-circuits the discussion about what benefits we care about. Jaime's patch certainly has merits, but does it help us make numpy and pandas/netcdf's more compatible? Does it make it easier for Eric to teach? Those are some specific merits that we might care about a lot, and for which Jaime's patch may or may not help much. But that kind of nuance gets lost when we jump straight to debating thumbs-up versus thumbs-down. I cross-my-heart promise that under the current regime, no PR breaking fancy indexing would ever get anywhere *near* numpy master without *extensive* discussion and warnings on the list. The core devs just spent weeks quibbling about whether a PR that adds a new field to the end of the dtype struct would break ABI backcompat (we're now pretty sure it doesn't), and the current standard we enforce is that every PR that touches public API needs a list discussion, even minor extensions with no compatibility issues at all. No one is going to sneak anything by anyone. Plus, I dunno, our current approach to discussions just seems to make things hostile and shouty and unpleasant. If a grad student or junior colleague comes to you with an idea where you see some potentially critical flaw, do you yell THAT WILL NEVER WORK and kick them out of your office? Or, do you maybe ask a few leading questions and see where they go? I think things will work better if the next time something like this comes up, *one* person just says "hmm, interesting idea, but the backcompat issues seem pretty severe; do you have any ideas about how to mitigate that?", and then we let that point be taken as having been made and see where the discussion goes. Maybe we can all give it a try? -n ___ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
