Re: [Discuss] Compatibility Guarantees and Versioning Post "1.0.0"

2019-07-13 Thread Micah Kornfield
SGTM could you or another PMC member start one?

Thanks,
Micah

On Saturday, July 13, 2019, Wes McKinney  wrote:

> Micah -- I would suggest that -- absent more opinions -- we vote about
> adopting the versioning scheme you described here (Format Version and
> Library Version)
>
> On Wed, Jul 10, 2019 at 8:46 AM Wes McKinney  wrote:
> >
> > On Wed, Jul 10, 2019 at 12:43 AM Micah Kornfield 
> wrote:
> > >
> > > Hi Eric,
> > > Short answer: I think your understanding matches what I was
> proposing.  Longer answer below.
> > >
> > >> So, for example, we release library v1.0.0 in a few months and then
> library v2.0.0 a few months after that.  In v2.0.0, C++, Python, and Java
> didn't make any breaking API changes from 1.0.0. But C# made 3 API breaking
> changes. This would be acceptable?
> > >
> > > Yes.  I think all language bindings are going under rapid enough
> iteration that we are making at least a few small breaking API changes on
> each release even though we try to avoid it.  I think it will be worth
> having further discussions on the release process once at least a few
> languages get to a more stable point.
> > >
> >
> > I agree with this. I think we are a pretty long ways away from making
> > API stability _guarantees_ in any of the implementations, though we
> > certainly should try to be courteous about the changes we do make, to
> > allow for graceful transitions over a period of 1-2 releases if
> > possible.
> >
> > > Thanks,
> > > Micah
> > >
> > > On Tue, Jul 9, 2019 at 2:26 PM Eric Erhardt <
> eric.erha...@microsoft.com> wrote:
> > >>
> > >> Just to be sure I fully understand the proposal:
> > >>
> > >> For the Library Version, we are going to increment the MAJOR version
> on every normal release, and increment the MINOR version if we need to
> release a patch/bug fix type of release.
> > >>
> > >> Since SemVer allows for API breaking changes on MAJOR versions, this
> basically means, each library (C++, Python, C#, Java, etc) _can_ introduce
> API breaking changes on every normal release (like we have been with the
> 0.x.0 releases).
> > >>
> > >> So, for example, we release library v1.0.0 in a few months and then
> library v2.0.0 a few months after that.  In v2.0.0, C++, Python, and Java
> didn't make any breaking API changes from 1.0.0. But C# made 3 API breaking
> changes. This would be acceptable?
> > >>
> > >> If my understanding above is correct, then I think this is a good
> plan. Initially I was concerned that the C# library wouldn't be free to
> make API breaking changes with making the version `1.0.0`. The C# library
> is still pretty inadequate, and I have a feeling there are a few things
> that will need to change about it in the future. But with the above plan,
> this concern won't be a problem.
> > >>
> > >> Eric
> > >>
> > >> -Original Message-
> > >> From: Micah Kornfield 
> > >> Sent: Monday, July 1, 2019 10:02 PM
> > >> To: Wes McKinney 
> > >> Cc: dev@arrow.apache.org
> > >> Subject: Re: [Discuss] Compatibility Guarantees and Versioning Post
> "1.0.0"
> > >>
> > >> Hi Wes,
> > >> Thanks for your response.  In regards to the protocol negotiation
> your description of feature reporting (snipped below) is along the lines of
> what I was thinking.  It might not be necessary for 1.0.0, but at some
> point might become useful.
> > >>
> > >>
> > >> >  Note that we don't really have a mechanism for clients and servers
> to
> > >> > report to each other what features they support, so this could help
> > >> > with that when for applications where it might matter.
> > >>
> > >>
> > >> Thanks,
> > >> Micah
> > >>
> > >>
> > >> On Mon, Jul 1, 2019 at 12:54 PM Wes McKinney 
> wrote:
> > >>
> > >> > hi Micah,
> > >> >
> > >> > Sorry for the delay in feedback. I looked at the document and it
> seems
> > >> > like a reasonable perspective about forward- and
> > >> > backward-compatibility.
> > >> >
> > >> > It seems like the main thing you are proposing is to apply Semantic
> > >> > Versioning to Format and Library versions separately. That's an
> > >> > interesting idea, my thought had been to have a version number that
> is
> >

Re: [Discuss] Compatibility Guarantees and Versioning Post "1.0.0"

2019-07-13 Thread Wes McKinney
Micah -- I would suggest that -- absent more opinions -- we vote about
adopting the versioning scheme you described here (Format Version and
Library Version)

On Wed, Jul 10, 2019 at 8:46 AM Wes McKinney  wrote:
>
> On Wed, Jul 10, 2019 at 12:43 AM Micah Kornfield  
> wrote:
> >
> > Hi Eric,
> > Short answer: I think your understanding matches what I was proposing.  
> > Longer answer below.
> >
> >> So, for example, we release library v1.0.0 in a few months and then 
> >> library v2.0.0 a few months after that.  In v2.0.0, C++, Python, and Java 
> >> didn't make any breaking API changes from 1.0.0. But C# made 3 API 
> >> breaking changes. This would be acceptable?
> >
> > Yes.  I think all language bindings are going under rapid enough iteration 
> > that we are making at least a few small breaking API changes on each 
> > release even though we try to avoid it.  I think it will be worth having 
> > further discussions on the release process once at least a few languages 
> > get to a more stable point.
> >
>
> I agree with this. I think we are a pretty long ways away from making
> API stability _guarantees_ in any of the implementations, though we
> certainly should try to be courteous about the changes we do make, to
> allow for graceful transitions over a period of 1-2 releases if
> possible.
>
> > Thanks,
> > Micah
> >
> > On Tue, Jul 9, 2019 at 2:26 PM Eric Erhardt  
> > wrote:
> >>
> >> Just to be sure I fully understand the proposal:
> >>
> >> For the Library Version, we are going to increment the MAJOR version on 
> >> every normal release, and increment the MINOR version if we need to 
> >> release a patch/bug fix type of release.
> >>
> >> Since SemVer allows for API breaking changes on MAJOR versions, this 
> >> basically means, each library (C++, Python, C#, Java, etc) _can_ introduce 
> >> API breaking changes on every normal release (like we have been with the 
> >> 0.x.0 releases).
> >>
> >> So, for example, we release library v1.0.0 in a few months and then 
> >> library v2.0.0 a few months after that.  In v2.0.0, C++, Python, and Java 
> >> didn't make any breaking API changes from 1.0.0. But C# made 3 API 
> >> breaking changes. This would be acceptable?
> >>
> >> If my understanding above is correct, then I think this is a good plan. 
> >> Initially I was concerned that the C# library wouldn't be free to make API 
> >> breaking changes with making the version `1.0.0`. The C# library is still 
> >> pretty inadequate, and I have a feeling there are a few things that will 
> >> need to change about it in the future. But with the above plan, this 
> >> concern won't be a problem.
> >>
> >> Eric
> >>
> >> -Original Message-
> >> From: Micah Kornfield 
> >> Sent: Monday, July 1, 2019 10:02 PM
> >> To: Wes McKinney 
> >> Cc: dev@arrow.apache.org
> >> Subject: Re: [Discuss] Compatibility Guarantees and Versioning Post "1.0.0"
> >>
> >> Hi Wes,
> >> Thanks for your response.  In regards to the protocol negotiation your 
> >> description of feature reporting (snipped below) is along the lines of 
> >> what I was thinking.  It might not be necessary for 1.0.0, but at some 
> >> point might become useful.
> >>
> >>
> >> >  Note that we don't really have a mechanism for clients and servers to
> >> > report to each other what features they support, so this could help
> >> > with that when for applications where it might matter.
> >>
> >>
> >> Thanks,
> >> Micah
> >>
> >>
> >> On Mon, Jul 1, 2019 at 12:54 PM Wes McKinney  wrote:
> >>
> >> > hi Micah,
> >> >
> >> > Sorry for the delay in feedback. I looked at the document and it seems
> >> > like a reasonable perspective about forward- and
> >> > backward-compatibility.
> >> >
> >> > It seems like the main thing you are proposing is to apply Semantic
> >> > Versioning to Format and Library versions separately. That's an
> >> > interesting idea, my thought had been to have a version number that is
> >> > FORMAT_VERSION.LIBRARY_VERSION.PATCH_VERSION. But your proposal is
> >> > more flexible in some ways, so let me clarify for others reading
> >> >
> >> > In what you are proposing, the next release would be:
> >> >
> >> > For

Re: [Discuss] Compatibility Guarantees and Versioning Post "1.0.0"

2019-07-10 Thread Wes McKinney
On Wed, Jul 10, 2019 at 12:43 AM Micah Kornfield  wrote:
>
> Hi Eric,
> Short answer: I think your understanding matches what I was proposing.  
> Longer answer below.
>
>> So, for example, we release library v1.0.0 in a few months and then library 
>> v2.0.0 a few months after that.  In v2.0.0, C++, Python, and Java didn't 
>> make any breaking API changes from 1.0.0. But C# made 3 API breaking 
>> changes. This would be acceptable?
>
> Yes.  I think all language bindings are going under rapid enough iteration 
> that we are making at least a few small breaking API changes on each release 
> even though we try to avoid it.  I think it will be worth having further 
> discussions on the release process once at least a few languages get to a 
> more stable point.
>

I agree with this. I think we are a pretty long ways away from making
API stability _guarantees_ in any of the implementations, though we
certainly should try to be courteous about the changes we do make, to
allow for graceful transitions over a period of 1-2 releases if
possible.

> Thanks,
> Micah
>
> On Tue, Jul 9, 2019 at 2:26 PM Eric Erhardt  
> wrote:
>>
>> Just to be sure I fully understand the proposal:
>>
>> For the Library Version, we are going to increment the MAJOR version on 
>> every normal release, and increment the MINOR version if we need to release 
>> a patch/bug fix type of release.
>>
>> Since SemVer allows for API breaking changes on MAJOR versions, this 
>> basically means, each library (C++, Python, C#, Java, etc) _can_ introduce 
>> API breaking changes on every normal release (like we have been with the 
>> 0.x.0 releases).
>>
>> So, for example, we release library v1.0.0 in a few months and then library 
>> v2.0.0 a few months after that.  In v2.0.0, C++, Python, and Java didn't 
>> make any breaking API changes from 1.0.0. But C# made 3 API breaking 
>> changes. This would be acceptable?
>>
>> If my understanding above is correct, then I think this is a good plan. 
>> Initially I was concerned that the C# library wouldn't be free to make API 
>> breaking changes with making the version `1.0.0`. The C# library is still 
>> pretty inadequate, and I have a feeling there are a few things that will 
>> need to change about it in the future. But with the above plan, this concern 
>> won't be a problem.
>>
>> Eric
>>
>> -Original Message-
>> From: Micah Kornfield 
>> Sent: Monday, July 1, 2019 10:02 PM
>> To: Wes McKinney 
>> Cc: dev@arrow.apache.org
>> Subject: Re: [Discuss] Compatibility Guarantees and Versioning Post "1.0.0"
>>
>> Hi Wes,
>> Thanks for your response.  In regards to the protocol negotiation your 
>> description of feature reporting (snipped below) is along the lines of what 
>> I was thinking.  It might not be necessary for 1.0.0, but at some point 
>> might become useful.
>>
>>
>> >  Note that we don't really have a mechanism for clients and servers to
>> > report to each other what features they support, so this could help
>> > with that when for applications where it might matter.
>>
>>
>> Thanks,
>> Micah
>>
>>
>> On Mon, Jul 1, 2019 at 12:54 PM Wes McKinney  wrote:
>>
>> > hi Micah,
>> >
>> > Sorry for the delay in feedback. I looked at the document and it seems
>> > like a reasonable perspective about forward- and
>> > backward-compatibility.
>> >
>> > It seems like the main thing you are proposing is to apply Semantic
>> > Versioning to Format and Library versions separately. That's an
>> > interesting idea, my thought had been to have a version number that is
>> > FORMAT_VERSION.LIBRARY_VERSION.PATCH_VERSION. But your proposal is
>> > more flexible in some ways, so let me clarify for others reading
>> >
>> > In what you are proposing, the next release would be:
>> >
>> > Format version: 1.0.0
>> > Library version: 1.0.0
>> >
>> > Suppose that 20 major versions down the road we stand at
>> >
>> > Format version: 1.5.0
>> > Library version: 20.0.0
>> >
>> > The minor version of the Format would indicate that there are
>> > additions, like new elements in the Type union, but otherwise backward
>> > and forward compatible. So the Minor version means "new things, but
>> > old clients will not be disrupted if those new things are not used".
>> > We've already been doing this since the V4 Format iteration but we
>> > have not had a way to 

Re: [Discuss] Compatibility Guarantees and Versioning Post "1.0.0"

2019-07-09 Thread Micah Kornfield
Hi Eric,
Short answer: I think your understanding matches what I was proposing.
Longer answer below.

So, for example, we release library v1.0.0 in a few months and then library
> v2.0.0 a few months after that.  In v2.0.0, C++, Python, and Java didn't
> make any breaking API changes from 1.0.0. But C# made 3 API breaking
> changes. This would be acceptable?

Yes.  I think all language bindings are going under rapid enough iteration
that we are making at least a few small breaking API changes on each
release even though we try to avoid it.  I think it will be worth having
further discussions on the release process once at least a few languages
get to a more stable point.

Thanks,
Micah

On Tue, Jul 9, 2019 at 2:26 PM Eric Erhardt 
wrote:

> Just to be sure I fully understand the proposal:
>
> For the Library Version, we are going to increment the MAJOR version on
> every normal release, and increment the MINOR version if we need to release
> a patch/bug fix type of release.
>
> Since SemVer allows for API breaking changes on MAJOR versions, this
> basically means, each library (C++, Python, C#, Java, etc) _can_ introduce
> API breaking changes on every normal release (like we have been with the
> 0.x.0 releases).
>
> So, for example, we release library v1.0.0 in a few months and then
> library v2.0.0 a few months after that.  In v2.0.0, C++, Python, and Java
> didn't make any breaking API changes from 1.0.0. But C# made 3 API breaking
> changes. This would be acceptable?
>
> If my understanding above is correct, then I think this is a good plan.
> Initially I was concerned that the C# library wouldn't be free to make API
> breaking changes with making the version `1.0.0`. The C# library is still
> pretty inadequate, and I have a feeling there are a few things that will
> need to change about it in the future. But with the above plan, this
> concern won't be a problem.
>
> Eric
>
> -Original Message-
> From: Micah Kornfield 
> Sent: Monday, July 1, 2019 10:02 PM
> To: Wes McKinney 
> Cc: dev@arrow.apache.org
> Subject: Re: [Discuss] Compatibility Guarantees and Versioning Post "1.0.0"
>
> Hi Wes,
> Thanks for your response.  In regards to the protocol negotiation your
> description of feature reporting (snipped below) is along the lines of what
> I was thinking.  It might not be necessary for 1.0.0, but at some point
> might become useful.
>
>
> >  Note that we don't really have a mechanism for clients and servers to
> > report to each other what features they support, so this could help
> > with that when for applications where it might matter.
>
>
> Thanks,
> Micah
>
>
> On Mon, Jul 1, 2019 at 12:54 PM Wes McKinney  wrote:
>
> > hi Micah,
> >
> > Sorry for the delay in feedback. I looked at the document and it seems
> > like a reasonable perspective about forward- and
> > backward-compatibility.
> >
> > It seems like the main thing you are proposing is to apply Semantic
> > Versioning to Format and Library versions separately. That's an
> > interesting idea, my thought had been to have a version number that is
> > FORMAT_VERSION.LIBRARY_VERSION.PATCH_VERSION. But your proposal is
> > more flexible in some ways, so let me clarify for others reading
> >
> > In what you are proposing, the next release would be:
> >
> > Format version: 1.0.0
> > Library version: 1.0.0
> >
> > Suppose that 20 major versions down the road we stand at
> >
> > Format version: 1.5.0
> > Library version: 20.0.0
> >
> > The minor version of the Format would indicate that there are
> > additions, like new elements in the Type union, but otherwise backward
> > and forward compatible. So the Minor version means "new things, but
> > old clients will not be disrupted if those new things are not used".
> > We've already been doing this since the V4 Format iteration but we
> > have not had a way to signal that there may be new features. As a
> > corollary to this, I wonder if we should create a dual version in the
> > metadata
> >
> > PROTOCOL VERSION: (what is currently MetadataVersion, V2) FEATURE
> > VERSION: not tracked at all
> >
> > So Minor version bumps in the format would trigger a bump in the
> > FeatureVersion. Note that we don't really have a mechanism for clients
> > and servers to report to each other what features they support, so
> > this could help with that when for applications where it might matter.
> >
> > Should backward/forward compatibility be disrupted in the future, then
> > a change to the major version would be required. So in year 2025, say,
> > we might decide

RE: [Discuss] Compatibility Guarantees and Versioning Post "1.0.0"

2019-07-09 Thread Eric Erhardt
Just to be sure I fully understand the proposal:

For the Library Version, we are going to increment the MAJOR version on every 
normal release, and increment the MINOR version if we need to release a 
patch/bug fix type of release.

Since SemVer allows for API breaking changes on MAJOR versions, this basically 
means, each library (C++, Python, C#, Java, etc) _can_ introduce API breaking 
changes on every normal release (like we have been with the 0.x.0 releases).

So, for example, we release library v1.0.0 in a few months and then library 
v2.0.0 a few months after that.  In v2.0.0, C++, Python, and Java didn't make 
any breaking API changes from 1.0.0. But C# made 3 API breaking changes. This 
would be acceptable?

If my understanding above is correct, then I think this is a good plan. 
Initially I was concerned that the C# library wouldn't be free to make API 
breaking changes with making the version `1.0.0`. The C# library is still 
pretty inadequate, and I have a feeling there are a few things that will need 
to change about it in the future. But with the above plan, this concern won't 
be a problem.

Eric

-Original Message-
From: Micah Kornfield  
Sent: Monday, July 1, 2019 10:02 PM
To: Wes McKinney 
Cc: dev@arrow.apache.org
Subject: Re: [Discuss] Compatibility Guarantees and Versioning Post "1.0.0"

Hi Wes,
Thanks for your response.  In regards to the protocol negotiation your 
description of feature reporting (snipped below) is along the lines of what I 
was thinking.  It might not be necessary for 1.0.0, but at some point might 
become useful.


>  Note that we don't really have a mechanism for clients and servers to 
> report to each other what features they support, so this could help 
> with that when for applications where it might matter.


Thanks,
Micah


On Mon, Jul 1, 2019 at 12:54 PM Wes McKinney  wrote:

> hi Micah,
>
> Sorry for the delay in feedback. I looked at the document and it seems 
> like a reasonable perspective about forward- and 
> backward-compatibility.
>
> It seems like the main thing you are proposing is to apply Semantic 
> Versioning to Format and Library versions separately. That's an 
> interesting idea, my thought had been to have a version number that is 
> FORMAT_VERSION.LIBRARY_VERSION.PATCH_VERSION. But your proposal is 
> more flexible in some ways, so let me clarify for others reading
>
> In what you are proposing, the next release would be:
>
> Format version: 1.0.0
> Library version: 1.0.0
>
> Suppose that 20 major versions down the road we stand at
>
> Format version: 1.5.0
> Library version: 20.0.0
>
> The minor version of the Format would indicate that there are 
> additions, like new elements in the Type union, but otherwise backward 
> and forward compatible. So the Minor version means "new things, but 
> old clients will not be disrupted if those new things are not used".
> We've already been doing this since the V4 Format iteration but we 
> have not had a way to signal that there may be new features. As a 
> corollary to this, I wonder if we should create a dual version in the 
> metadata
>
> PROTOCOL VERSION: (what is currently MetadataVersion, V2) FEATURE 
> VERSION: not tracked at all
>
> So Minor version bumps in the format would trigger a bump in the 
> FeatureVersion. Note that we don't really have a mechanism for clients 
> and servers to report to each other what features they support, so 
> this could help with that when for applications where it might matter.
>
> Should backward/forward compatibility be disrupted in the future, then 
> a change to the major version would be required. So in year 2025, say, 
> we might decide that we want to do:
>
> Format version: 2.0.0
> Library version: 21.0.0
>
> The Format version would live in the project's Documentation, so the 
> Apache releases are only the library version.
>
> Regarding your open questions:
>
> 1. Should we clean up "warts" on the specification, like redundant 
> information
>
> I don't think it's necessary. So if Metadata V5 is Format Version
> 1.0.0 (currently we are V4, but we're discussing some possible 
> non-forward compatible changes...) I think that's OK. None of these 
> things are "hurting" anything
>
> 2. Do we need additional mechanisms for marking some features as 
> experimental?
>
> Not sure, but I think this can be mostly addressed through 
> documentation. Flight will still be experimental in 1.0.0, for 
> example.
>
> 3. Do we need protocol negotiation mechanisms in Flight
>
> Could you explain what you mean? Are you thinking if there is some 
> major revamp of the protocol and you need to switch between a "V1 
> Flight Protocol" and a "V2 Flight Protocol"?
>
> - Wes
&g

Re: [Discuss] Compatibility Guarantees and Versioning Post "1.0.0"

2019-07-01 Thread Micah Kornfield
Hi Wes,
Thanks for your response.  In regards to the protocol negotiation your
description of feature reporting (snipped below) is along the lines of what
I was thinking.  It might not be necessary for 1.0.0, but at some point
might become useful.


>  Note that we don't really have a mechanism for clients
> and servers to report to each other what features they support, so
> this could help with that when for applications where it might matter.


Thanks,
Micah


On Mon, Jul 1, 2019 at 12:54 PM Wes McKinney  wrote:

> hi Micah,
>
> Sorry for the delay in feedback. I looked at the document and it seems
> like a reasonable perspective about forward- and
> backward-compatibility.
>
> It seems like the main thing you are proposing is to apply Semantic
> Versioning to Format and Library versions separately. That's an
> interesting idea, my thought had been to have a version number that is
> FORMAT_VERSION.LIBRARY_VERSION.PATCH_VERSION. But your proposal is
> more flexible in some ways, so let me clarify for others reading
>
> In what you are proposing, the next release would be:
>
> Format version: 1.0.0
> Library version: 1.0.0
>
> Suppose that 20 major versions down the road we stand at
>
> Format version: 1.5.0
> Library version: 20.0.0
>
> The minor version of the Format would indicate that there are
> additions, like new elements in the Type union, but otherwise backward
> and forward compatible. So the Minor version means "new things, but
> old clients will not be disrupted if those new things are not used".
> We've already been doing this since the V4 Format iteration but we
> have not had a way to signal that there may be new features. As a
> corollary to this, I wonder if we should create a dual version in the
> metadata
>
> PROTOCOL VERSION: (what is currently MetadataVersion, V2)
> FEATURE VERSION: not tracked at all
>
> So Minor version bumps in the format would trigger a bump in the
> FeatureVersion. Note that we don't really have a mechanism for clients
> and servers to report to each other what features they support, so
> this could help with that when for applications where it might matter.
>
> Should backward/forward compatibility be disrupted in the future, then
> a change to the major version would be required. So in year 2025, say,
> we might decide that we want to do:
>
> Format version: 2.0.0
> Library version: 21.0.0
>
> The Format version would live in the project's Documentation, so the
> Apache releases are only the library version.
>
> Regarding your open questions:
>
> 1. Should we clean up "warts" on the specification, like redundant
> information
>
> I don't think it's necessary. So if Metadata V5 is Format Version
> 1.0.0 (currently we are V4, but we're discussing some possible
> non-forward compatible changes...) I think that's OK. None of these
> things are "hurting" anything
>
> 2. Do we need additional mechanisms for marking some features as
> experimental?
>
> Not sure, but I think this can be mostly addressed through
> documentation. Flight will still be experimental in 1.0.0, for
> example.
>
> 3. Do we need protocol negotiation mechanisms in Flight
>
> Could you explain what you mean? Are you thinking if there is some
> major revamp of the protocol and you need to switch between a "V1
> Flight Protocol" and a "V2 Flight Protocol"?
>
> - Wes
>
> On Thu, Jun 13, 2019 at 2:17 AM Micah Kornfield 
> wrote:
> >
> > Hi Everyone,
> > I think there might be some ideas that we still need to reach consensus
> on
> > for how the format and libraries evolve in a post-1.0.0 release world.
> >  Specifically, I think we need to agree on definitions for
> > backwards/forwards compatibility and its implications for versioning the
> > format.
> >
> > To this end I put some thoughts down in a Google Doc [1] for the purposes
> > of discussion.  Comments welcome.  I will start threads for any comments
> in
> > the document that seem to warrant further discussion, and once we reach
> > consensus I can create a patch to document what we decide on as part of
> the
> > specification.
> >
> > Thanks,
> > Micah
> >
> > [1]
> >
> https://docs.google.com/document/d/1uBitWu57rDu85tNHn0NwstAbrlYqor9dPFg_7QaE-nc/edit#
>


Re: [Discuss] Compatibility Guarantees and Versioning Post "1.0.0"

2019-07-01 Thread Wes McKinney
hi Micah,

Sorry for the delay in feedback. I looked at the document and it seems
like a reasonable perspective about forward- and
backward-compatibility.

It seems like the main thing you are proposing is to apply Semantic
Versioning to Format and Library versions separately. That's an
interesting idea, my thought had been to have a version number that is
FORMAT_VERSION.LIBRARY_VERSION.PATCH_VERSION. But your proposal is
more flexible in some ways, so let me clarify for others reading

In what you are proposing, the next release would be:

Format version: 1.0.0
Library version: 1.0.0

Suppose that 20 major versions down the road we stand at

Format version: 1.5.0
Library version: 20.0.0

The minor version of the Format would indicate that there are
additions, like new elements in the Type union, but otherwise backward
and forward compatible. So the Minor version means "new things, but
old clients will not be disrupted if those new things are not used".
We've already been doing this since the V4 Format iteration but we
have not had a way to signal that there may be new features. As a
corollary to this, I wonder if we should create a dual version in the
metadata

PROTOCOL VERSION: (what is currently MetadataVersion, V2)
FEATURE VERSION: not tracked at all

So Minor version bumps in the format would trigger a bump in the
FeatureVersion. Note that we don't really have a mechanism for clients
and servers to report to each other what features they support, so
this could help with that when for applications where it might matter.

Should backward/forward compatibility be disrupted in the future, then
a change to the major version would be required. So in year 2025, say,
we might decide that we want to do:

Format version: 2.0.0
Library version: 21.0.0

The Format version would live in the project's Documentation, so the
Apache releases are only the library version.

Regarding your open questions:

1. Should we clean up "warts" on the specification, like redundant information

I don't think it's necessary. So if Metadata V5 is Format Version
1.0.0 (currently we are V4, but we're discussing some possible
non-forward compatible changes...) I think that's OK. None of these
things are "hurting" anything

2. Do we need additional mechanisms for marking some features as experimental?

Not sure, but I think this can be mostly addressed through
documentation. Flight will still be experimental in 1.0.0, for
example.

3. Do we need protocol negotiation mechanisms in Flight

Could you explain what you mean? Are you thinking if there is some
major revamp of the protocol and you need to switch between a "V1
Flight Protocol" and a "V2 Flight Protocol"?

- Wes

On Thu, Jun 13, 2019 at 2:17 AM Micah Kornfield  wrote:
>
> Hi Everyone,
> I think there might be some ideas that we still need to reach consensus on
> for how the format and libraries evolve in a post-1.0.0 release world.
>  Specifically, I think we need to agree on definitions for
> backwards/forwards compatibility and its implications for versioning the
> format.
>
> To this end I put some thoughts down in a Google Doc [1] for the purposes
> of discussion.  Comments welcome.  I will start threads for any comments in
> the document that seem to warrant further discussion, and once we reach
> consensus I can create a patch to document what we decide on as part of the
> specification.
>
> Thanks,
> Micah
>
> [1]
> https://docs.google.com/document/d/1uBitWu57rDu85tNHn0NwstAbrlYqor9dPFg_7QaE-nc/edit#


[Discuss] Compatibility Guarantees and Versioning Post "1.0.0"

2019-06-13 Thread Micah Kornfield
Hi Everyone,
I think there might be some ideas that we still need to reach consensus on
for how the format and libraries evolve in a post-1.0.0 release world.
 Specifically, I think we need to agree on definitions for
backwards/forwards compatibility and its implications for versioning the
format.

To this end I put some thoughts down in a Google Doc [1] for the purposes
of discussion.  Comments welcome.  I will start threads for any comments in
the document that seem to warrant further discussion, and once we reach
consensus I can create a patch to document what we decide on as part of the
specification.

Thanks,
Micah

[1]
https://docs.google.com/document/d/1uBitWu57rDu85tNHn0NwstAbrlYqor9dPFg_7QaE-nc/edit#