On Tue, Dec 2, 2014 at 1:14 PM, Sean Busbey <bus...@cloudera.com> wrote:
> Responses below. I feel like most of these questions (with the noted > exception of details on my issue with one of the late compromise positions) > were previously answered, but I've tried to add additional detail below > where I may have been unclear in prior exchanges. > > While I agree that objections are the start of a conversation, votes are > meant to be provide closure so the community can move on to other work. I'd > ask that we try to bring this thread to a close. > > I understand the frustration with a long-running thread, but so long as your position makes no sense to me (and to the several others who have commented in this thread who you've also seemed to have failed to convince), I feel we must address it. > > On Mon, Dec 1, 2014 at 2:40 PM, Keith Turner <ke...@deenlo.com> wrote: > > > > > > > The question I was most curious about was why you are ok w/ making this > API > > change in 2.0 but not 1.7.0. I do not understand the reasoning behind > > this. > > > > As a matter of background, over time I have become more convinced of the > need for a stable, coherent API. Too many of the end users I have seen > endure substantial work because of our altering of the API. This applies > equally for forwards and backwards compatibility. When someone has a > requirement to use our software and they find that the particular version > they need to integrate with is different than they initially expected, I > would like them to not have to endure project delays merely for updating > use of our API. > > Your justification for this veto seems to have steadily shifted away from the reasoning you originally provided. You becoming more convinced of a need for a stable, coherent API is a *very* different reason than that which was initially expressed (that it didn't satisfy an underlying issue of property updates being consistent across a cluster) which was used to determine your vetos validity. I agree that API stability is a laudable goal, but blocking an API improvement on the basis of a forwards compatibility policy that *we haven't even discussed* or considered for adoption is ridiculous, in my view. If that is your goal, then we should revert all the ReplicationOperations API which was added to 1.7.0. Now, it might be reasonable to prefer forwards compatibility within a major release line, but between major releases? That is overkill, and effectively blocks improvements and new features. Now, granted, you might not find the particular feature very compelling of an addition, but if that were the burden for accepting contributions (that every developer should find new features compelling, otherwise veto), then there'd be a lot fewer features in Accumulo... including the recently added replication features to 1.7.0, which I personally don't find compelling within our API... but perhaps as an external function... but I'm not willing to block inclusion of a feature that others have valid use cases for, simply because I don't find it compelling enough to use it. I understand that we can have different metrics for our decisions about what to veto. Adding one new method to the API does *not* require *any* such endurance on behalf of users, especially when added to a "major version" 1.7.0. Major versions are where we can add breaking API changes, so why block a minor improvement which adds *no* such breakage? I could understand if this provided no value, and simply changed the look and feel of the API... if that were the case, I'd agree with you. It's not worth it. However, this actually provides functionality... important functionality... that satisfies the specified use cases, and you are preventing those use cases from being satisfied, and your reasoning for doing so makes no sense. Would you be okay with this change if it did not introduce the "NewTableConfiguration" object, and simply added an overloaded createTable method that took a Map of additional properties? Because that wouldn't change the look and feel of the API in any way? What I'm trying to understand by this question is whether you feel the "burden" of adding a new feature to the API is created by the change in the look-and-feel of the API, or by the new utility. If it's just the look-and-feel, then we can address that with a smaller change that adds an overloaded method to provide the same utility, and we've identified yet another opportunity for compromise. > For 2.0, we already have broad consensus around improving our API in > ACCUMULO-2589 despite the cost it puts on our user base; both because of a > better delineation of what is and is not expected to stay constant and > because we'll have a better formulation of a lifecycle for Accumulo related > resources. In that particular matter, it's the proper lifecycle that I > personally find compelling enough to broadly cause a burden. This is > probably colored by my having dealt with ACCUMULO-2128 and its ilk. > > Any conversation about 2.0 is hypothetical. There has been no formally submitted code for API improvements for 2.0, for consideration in a release. We cannot block improvements to 1.x on the basis of some forthcoming changes in the future. We're also talking about a feature that would be very useful for certain use cases long before 2.0 is ready for release. Again, what you find compelling should not be the metric for vetoing contributions. A more reasonable measure is whether it satisfies valid use cases without causing some demonstrable harm, and this change does satisfy valid use cases and can be done without affecting existing API (not even deprecating them, if we wish), and does so in a major version. > So, given that we're going to be asking our users to deal with a headache > come 2.0 (which will hopefully be in the next 6-9 months) I would prefer to > minimize asking them to take on dealing with changes in versions prior to > that. There may be some future issue that fixes something severe enough to > warrant changing my position on the matter. This issue did not. > > Again, I anticipate no such headache. You're speaking hypothetically about proposed changes for which no code has been submitted for consideration yet. This reasoning does not make sense. > I talked previously about my position on API changes in general in the > background info for Josh’s message: > > http://s.apache.org/HJg > > And in that message, you incorrectly suggested that the problem could be solved by addressing property updates... which is not true, because assignments and balancing will already be in play before that property update is issued. This is a completely separate issue being addressed, that would *still* be a problem, regardless of whether we address property updates. This issue is about *initial* table configuration, not table configuration *changes*. > and I had meant to cover the "we already agreed to break things in 2.0" > side in this admittedly short offer of compromise: > > http://s.apache.org/imf > > Again, hypothetical future code, that (with the exception of the removal of deprecated API), there is no anticipated breakage. I don't know what you're talking about. > > > On Mon, Dec 1, 2014 at 4:25 PM, Christopher <ctubb...@apache.org> wrote: > > > Explicit questions and outstanding requests for clarification since your > > last response (see previous emails for details and context): > > -> So, are you arguing for no more API additions until 2.0? > > > > My general preference, as mentioned above and previously in the background > info Josh asked for, is to avoid API changes prior to 2.0. I am not arguing > that this stance be taken on as a matter of course through this particular > vote. > > I think any change to our API, prior to 2.0 or after, should be carefully > considered. How we plan for users to interact with our software over time > matters. Through removals we will be directly impacting the ability of > operational users to adopt our newer versions. Through additions we will be > impacting how system integrators go about building applications above us. > It does neither us nor them any good for our API to change merely because > it improves our abstractions or condenses our support footprint. > > In the consideration of the above and the nice-to-have fixes this > particular patch brings, I think it can wait for the API overhaul in 2.0. > > The same two previous mailings I mentioned above to Keith are where I went > through this previously. > > > > > -> Please address the fact that there is no breakage here... > > -> Another reasonable compromise has also been proposed that seems to > > address all of your concerns. Please explain why it does not. > > > I think these are the same question, presuming your "other compromise" is > the one of adding the new API and leaving the extant create methods > undeprecated. If not, please let me know what other compromise I missed so > that I can respond accordingly. > > Yes, that is the other compromise I refer to. (compromise #2) > I covered the breakage part of this explicitly in my response to Keith's > question about which part of the API move I was concerned with: > > http://s.apache.org/nn5 > > Essentially, moving our table creation API to use a configuration object > instead of the myriad of different arguments is a shift in how we expect > users to interact with Accumulo. Even if the breakage doesn't happen right > now, this change is setting downstream users up for pain when the break > happens. To that same end, users attempting to proactively stay up to date > on our API will break if they have to move backwards. Yes, this a normal > part of API evolution. Yes, users will have to do this at some point. I'm > merely stating that we have already reached consensus on that point being > 2.0 and we should reserve using up the good will of our end users. > > Okay, so first, no, we have not reached consensus on this point being 2.0. We've only reached that in terms of removals... not additions. This entire paragraph is invalid if we accept compromise #2. Only "moving" would create that, yes, but "adding" a new method to do that does not. If we had agreed that no new additions would be added, then we should revert all of the replication API which was added. What if we were to add the method with a @Beta or @Experimental annotation in addition to compromise #2? (compromise #3). Also, what about an overloaded method which does not add the configuration object, but only an additional parameter (suggested above)? (compromise #4) > Similarly, simply expanding our API to have multiple long term ways of > doing table creation isn't tenable. For one, downstream users will ask > which method to use. Deprecation is how we normally offer them clear > guidance on where the API is going in the future. Without that, we'll just > be using some proxy for deprecation (either a javadoc or emails on user@). > Additionally, since the version that takes a single configuration object is > clearly the most sustainable approach, the other methods are likely to be > deprecated and then removed should we end up with additional major versions > prior to 2.0. > > I think we could probably reach consensus on the idea that we won't remove any deprecated APIs prior to 2.0, and we I bet could even reach consensus on the idea that no methods deprecated in 1.7.0 and later will be removed until 3.0, giving plenty of time for migration. > I covered some of this concern in my response to Brian in the first part of > my last response: > > http://s.apache.org/0um > > > > > -> Please explain how you see omitting this API addition is compatible > with > > [the goal of supporting non-major intermediate releases]. Please also > > explain why, if you consider 1.7 to be a major (expected) release, why > such > > an addition would not be appropriate, but would be appropriate for a > future > > major release (2.0). > > > I believe I covered this through a combination of the above explanations to > Keith and my response to Brian in the first part of my last response: > > http://s.apache.org/0um > > That we are having a 1.7 release at all is a matter of scheduling. There > are already too many things different in that development line for us to > release it as a follow on to 1.6 but not enough of our goals for 2.0 are > done to get that release out. That doesn't mean we should feel free to pack > as many breaking changes as we want into the release. > > > This is *NOT* a breaking change! You keep using that term, but I've repeatedly expressed why it is incorrect to do so. Please adjust your mode of thinking with regard to whether this is breaking or not, and select your terminology more carefully, to reflect the actual nature of the proposed patch (which does not break any API or necessarily influence any future breakage). > > > > > Brian also asked: > > -> I don’t see what breakage users would be required to deal with if the > > proposed changes were made. A new method would be added to the API and > some > > existing methods deprecated (presumably to be removed in 2.0). So how > would > > this hurt our users? > > > > > I think this is covered above. In summary, when we alter our API we need to > consider long term impact rather than just the immediate release. We are > already going to ask our users to handle major changes in a relatively > short time period. > > Okay, but we've already addressed a compromise that allows us to address this (don't deprecate, don't remove until at least 3.0). > > > > -> Given that you are ok with with the change in 2.0, it seems that your > > objection is not to the content of the change but rather the timing of > it. > > Given that users aren’t required to use the new API method added by the > > change, this objection and the veto seem invalid to me. Am I missing > > something? > > > > > I believe the rest of this message restates in detail my concerns about our > API evolution and specifically why I am on board with the planned breakage > in the 2.0 release. Let me know where there are particular gaps so I can > help clarify. > > Please everyone stop this divisive tactic of continuing to claim my veto is > invalid. The surface of our API and our impact on downstream users are > valid technical considerations. Our bylaws clearly state what is needed for > a veto to be sustained and I have already passed that bar. Let’s focus our > discussion on the underlying problem rather than technicalities of our > governance. > > It is not a "tactic", it is our *(at least, my) opinion that your veto is invalid (at least, the original veto justification). My request for Bill was to provide clarification to the reasoning, so we can understand precedence for future validity determinations... it was not a tactic to try to overturn the validity determination here. We can address that in a separate thread, if needed, but I thought the clarification might also add value here. To be clear, I think your original justification was invalid, but your concerns about API are valid (albeit insufficient, in my view, to warrant a veto, and very problematic and inconsistently applied to API additions, as Keith points out), and completely addressable by several of the proposed compromises that would still permit the functionality which is required for several valid use cases.
