Micah -- I would suggest that -- absent more opinions -- we vote about adopting the versioning scheme you described here (Format Version and Library Version)
On Wed, Jul 10, 2019 at 8:46 AM Wes McKinney <wesmck...@gmail.com> wrote: > > On Wed, Jul 10, 2019 at 12:43 AM Micah Kornfield <emkornfi...@gmail.com> > wrote: > > > > Hi Eric, > > Short answer: I think your understanding matches what I was proposing. > > Longer answer below. > > > >> So, for example, we release library v1.0.0 in a few months and then > >> library v2.0.0 a few months after that. In v2.0.0, C++, Python, and Java > >> didn't make any breaking API changes from 1.0.0. But C# made 3 API > >> breaking changes. This would be acceptable? > > > > Yes. I think all language bindings are going under rapid enough iteration > > that we are making at least a few small breaking API changes on each > > release even though we try to avoid it. I think it will be worth having > > further discussions on the release process once at least a few languages > > get to a more stable point. > > > > I agree with this. I think we are a pretty long ways away from making > API stability _guarantees_ in any of the implementations, though we > certainly should try to be courteous about the changes we do make, to > allow for graceful transitions over a period of 1-2 releases if > possible. > > > Thanks, > > Micah > > > > On Tue, Jul 9, 2019 at 2:26 PM Eric Erhardt <eric.erha...@microsoft.com> > > wrote: > >> > >> Just to be sure I fully understand the proposal: > >> > >> For the Library Version, we are going to increment the MAJOR version on > >> every normal release, and increment the MINOR version if we need to > >> release a patch/bug fix type of release. > >> > >> Since SemVer allows for API breaking changes on MAJOR versions, this > >> basically means, each library (C++, Python, C#, Java, etc) _can_ introduce > >> API breaking changes on every normal release (like we have been with the > >> 0.x.0 releases). > >> > >> So, for example, we release library v1.0.0 in a few months and then > >> library v2.0.0 a few months after that. In v2.0.0, C++, Python, and Java > >> didn't make any breaking API changes from 1.0.0. But C# made 3 API > >> breaking changes. This would be acceptable? > >> > >> If my understanding above is correct, then I think this is a good plan. > >> Initially I was concerned that the C# library wouldn't be free to make API > >> breaking changes with making the version `1.0.0`. The C# library is still > >> pretty inadequate, and I have a feeling there are a few things that will > >> need to change about it in the future. But with the above plan, this > >> concern won't be a problem. > >> > >> Eric > >> > >> -----Original Message----- > >> From: Micah Kornfield <emkornfi...@gmail.com> > >> Sent: Monday, July 1, 2019 10:02 PM > >> To: Wes McKinney <wesmck...@gmail.com> > >> Cc: dev@arrow.apache.org > >> Subject: Re: [Discuss] Compatibility Guarantees and Versioning Post "1.0.0" > >> > >> Hi Wes, > >> Thanks for your response. In regards to the protocol negotiation your > >> description of feature reporting (snipped below) is along the lines of > >> what I was thinking. It might not be necessary for 1.0.0, but at some > >> point might become useful. > >> > >> > >> > Note that we don't really have a mechanism for clients and servers to > >> > report to each other what features they support, so this could help > >> > with that when for applications where it might matter. > >> > >> > >> Thanks, > >> Micah > >> > >> > >> On Mon, Jul 1, 2019 at 12:54 PM Wes McKinney <wesmck...@gmail.com> wrote: > >> > >> > hi Micah, > >> > > >> > Sorry for the delay in feedback. I looked at the document and it seems > >> > like a reasonable perspective about forward- and > >> > backward-compatibility. > >> > > >> > It seems like the main thing you are proposing is to apply Semantic > >> > Versioning to Format and Library versions separately. That's an > >> > interesting idea, my thought had been to have a version number that is > >> > FORMAT_VERSION.LIBRARY_VERSION.PATCH_VERSION. But your proposal is > >> > more flexible in some ways, so let me clarify for others reading > >> > > >> > In what you are proposing, the next release would be: > >> > > >> > Format version: 1.0.0 > >> > Library version: 1.0.0 > >> > > >> > Suppose that 20 major versions down the road we stand at > >> > > >> > Format version: 1.5.0 > >> > Library version: 20.0.0 > >> > > >> > The minor version of the Format would indicate that there are > >> > additions, like new elements in the Type union, but otherwise backward > >> > and forward compatible. So the Minor version means "new things, but > >> > old clients will not be disrupted if those new things are not used". > >> > We've already been doing this since the V4 Format iteration but we > >> > have not had a way to signal that there may be new features. As a > >> > corollary to this, I wonder if we should create a dual version in the > >> > metadata > >> > > >> > PROTOCOL VERSION: (what is currently MetadataVersion, V2) FEATURE > >> > VERSION: not tracked at all > >> > > >> > So Minor version bumps in the format would trigger a bump in the > >> > FeatureVersion. Note that we don't really have a mechanism for clients > >> > and servers to report to each other what features they support, so > >> > this could help with that when for applications where it might matter. > >> > > >> > Should backward/forward compatibility be disrupted in the future, then > >> > a change to the major version would be required. So in year 2025, say, > >> > we might decide that we want to do: > >> > > >> > Format version: 2.0.0 > >> > Library version: 21.0.0 > >> > > >> > The Format version would live in the project's Documentation, so the > >> > Apache releases are only the library version. > >> > > >> > Regarding your open questions: > >> > > >> > 1. Should we clean up "warts" on the specification, like redundant > >> > information > >> > > >> > I don't think it's necessary. So if Metadata V5 is Format Version > >> > 1.0.0 (currently we are V4, but we're discussing some possible > >> > non-forward compatible changes...) I think that's OK. None of these > >> > things are "hurting" anything > >> > > >> > 2. Do we need additional mechanisms for marking some features as > >> > experimental? > >> > > >> > Not sure, but I think this can be mostly addressed through > >> > documentation. Flight will still be experimental in 1.0.0, for > >> > example. > >> > > >> > 3. Do we need protocol negotiation mechanisms in Flight > >> > > >> > Could you explain what you mean? Are you thinking if there is some > >> > major revamp of the protocol and you need to switch between a "V1 > >> > Flight Protocol" and a "V2 Flight Protocol"? > >> > > >> > - Wes > >> > > >> > On Thu, Jun 13, 2019 at 2:17 AM Micah Kornfield > >> > <emkornfi...@gmail.com> > >> > wrote: > >> > > > >> > > Hi Everyone, > >> > > I think there might be some ideas that we still need to reach > >> > > consensus > >> > on > >> > > for how the format and libraries evolve in a post-1.0.0 release world. > >> > > Specifically, I think we need to agree on definitions for > >> > > backwards/forwards compatibility and its implications for versioning > >> > > the format. > >> > > > >> > > To this end I put some thoughts down in a Google Doc [1] for the > >> > > purposes of discussion. Comments welcome. I will start threads for > >> > > any comments > >> > in > >> > > the document that seem to warrant further discussion, and once we > >> > > reach consensus I can create a patch to document what we decide on > >> > > as part of > >> > the > >> > > specification. > >> > > > >> > > Thanks, > >> > > Micah > >> > > > >> > > [1] > >> > > > >> > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdocs > >> > .google.com%2Fdocument%2Fd%2F1uBitWu57rDu85tNHn0NwstAbrlYqor9dPFg_7QaE-nc%2Fedit%23&data=02%7C01%7CEric.Erhardt%40microsoft.com%7C6fc59049ffb049c9ddb108d6fe99bebf%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636976334243577292&sdata=YNQ%2FgL5rvmvRqvvW%2Bxjmb%2F4KeEe2JHe1ruws2VP%2BvK4%3D&reserved=0 > >> >
