Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Nick Dimiduk]

Seems I misunderstood the conclusion of this discussion and pushed
prematurely HBASE-13339 (and perhaps HBASE-13149). On that ticket, Lars
states

 I don't recall that we agreed on this. Should be in trunk (2.0) no doubt.
There was a longer discussion about 1.1 and I seem to recall
 that we said we do not want to force a major Hadoop with a minor HBase
upgrade and that instead we'd document in the book how
 build with Hadoop 2.6.0.

My understanding of this discussion was that we have relaxed out statement
regarding adherence to semantic versioning guidelines. Thus a minor version
bump of the Hadoop dependency was acceptable, particularly considering the
bug found on 2.5. Furthermore, following Hadoop on it's Jackson version was
the prudent decision for allowing our tools to work using either common
invocation patterns.

I got this wrong? Can someone summarize the decision here so that I can
move forward with an RC?

Thanks,
Nick

On Sun, Apr 26, 2015 at 11:33 AM, Nick Dimiduk ndimi...@gmail.com wrote:

 I spoke with a couple of our HDFS folks on Friday, they sound confident
 that 2.7.1 will be considered production-stable. I think we can try bumping
 master to 2.7.1 when it's out.

 On Fri, Apr 24, 2015 at 10:03 AM, Sean Busbey bus...@cloudera.com wrote:

 On Fri, Apr 24, 2015 at 11:53 AM, Elliott Clark ecl...@apache.org
 wrote:

  Is 2.7.1 going to have that not for production tag or was that tag just
  because of the issues found late in the cycle that are now fixed?
 
 
 From reading from hadoop lists, I'm not sure. Including the non-production
 tag seemed to be a compromise position to get 2.7.0 out the door. The
 cycle
 covered so much ground that I'm not sure when it got decided. The
 discussion also got them onto figuring out what their general policy on
 production / non-production will be. When they come to a decision it'll be
 good to revisit and see what it means for us.

 At least some of the Hadoopers are in agreement that 2.7.1 should be
 labeled production ready. No one was opposed to the matter, but there were
 some noticeable gaps in assenting voices[1].

 It's supposed to be soon, so we'll see I guess?

 [1]:

 http://markmail.org/thread/zwzze6cqqgwq4rmw#query:+page:1+mid:u6pihxyy36pzz52e+state:results

 --
 Sean

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Jerry He]

A second thought on this. I wonder how it is going to work. The test suite
probably will run with old Jackson version with the compile.  We will ship
what we don't test with?
Whatever we do seem to imply similarly small risk ...
On Apr 25, 2015 8:02 PM, Nick Dimiduk ndimi...@gmail.com wrote:

 I'm game for giving it a shot if we can build some confidence it won't
 cause other issues.  However given we're moving to Hadoop 2.6 I think we've
 already relaxed the constraints and this won't buy us much. More trouble
 than it's worth?

 On Saturday, April 25, 2015, Jerry He jerry...@gmail.com wrote:

  This is an interesting idea from Sean.
 
   On the Jackson thing, are folks opposed to compiling with the current
  older
   version and then packaging the newer version? That would make sure we
  don't
   start using 1.9 features in a way that would prevent downstream users
  from
   downgrading to 1.8, which combined with a release note would address my
   concerns.
 
  I wonder if people have objections or concerns on this.  Maybe cause
  confusion down the road, or in the pom or assembly?
 
  Looks like we are moving in the direction of relaxing a little, and
 upgrade
  Jackson in HBase 1.1.
 
  Jerry
 

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Nick Dimiduk]

I spoke with a couple of our HDFS folks on Friday, they sound confident
that 2.7.1 will be considered production-stable. I think we can try bumping
master to 2.7.1 when it's out.

On Fri, Apr 24, 2015 at 10:03 AM, Sean Busbey bus...@cloudera.com wrote:

 On Fri, Apr 24, 2015 at 11:53 AM, Elliott Clark ecl...@apache.org wrote:

  Is 2.7.1 going to have that not for production tag or was that tag just
  because of the issues found late in the cycle that are now fixed?
 
 
 From reading from hadoop lists, I'm not sure. Including the non-production
 tag seemed to be a compromise position to get 2.7.0 out the door. The cycle
 covered so much ground that I'm not sure when it got decided. The
 discussion also got them onto figuring out what their general policy on
 production / non-production will be. When they come to a decision it'll be
 good to revisit and see what it means for us.

 At least some of the Hadoopers are in agreement that 2.7.1 should be
 labeled production ready. No one was opposed to the matter, but there were
 some noticeable gaps in assenting voices[1].

 It's supposed to be soon, so we'll see I guess?

 [1]:

 http://markmail.org/thread/zwzze6cqqgwq4rmw#query:+page:1+mid:u6pihxyy36pzz52e+state:results

 --
 Sean

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Nick Dimiduk]

I'm game for giving it a shot if we can build some confidence it won't
cause other issues.  However given we're moving to Hadoop 2.6 I think we've
already relaxed the constraints and this won't buy us much. More trouble
than it's worth?

On Saturday, April 25, 2015, Jerry He jerry...@gmail.com wrote:

 This is an interesting idea from Sean.

  On the Jackson thing, are folks opposed to compiling with the current
 older
  version and then packaging the newer version? That would make sure we
 don't
  start using 1.9 features in a way that would prevent downstream users
 from
  downgrading to 1.8, which combined with a release note would address my
  concerns.

 I wonder if people have objections or concerns on this.  Maybe cause
 confusion down the road, or in the pom or assembly?

 Looks like we are moving in the direction of relaxing a little, and upgrade
 Jackson in HBase 1.1.

 Jerry

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Jerry He]

This is an interesting idea from Sean.

 On the Jackson thing, are folks opposed to compiling with the current
older
 version and then packaging the newer version? That would make sure we
don't
 start using 1.9 features in a way that would prevent downstream users from
 downgrading to 1.8, which combined with a release note would address my
 concerns.

I wonder if people have objections or concerns on this.  Maybe cause
confusion down the road, or in the pom or assembly?

Looks like we are moving in the direction of relaxing a little, and upgrade
Jackson in HBase 1.1.

Jerry

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Nick Dimiduk]

On Thu, Apr 23, 2015 at 6:39 PM, Enis Söztutar enis@gmail.com wrote:

 I would be in favor of going 2.6.0 and jackson upgrade in 1.1.


I am also +1 bumping to Hadoop 2.6.x and Jackson 1.9.x in HBase 1.1.0. This
seems like the most practical solution to the problems outlined here. Let
me explore the API surface area differences between Hadoop 2.5, 2.6 and see
what might crop up for users.

-n

On Thu, Apr 23, 2015 at 5:34 PM, Nick Dimiduk ndimi...@gmail.com wrote:

  On Thu, Apr 23, 2015 at 4:13 PM, Stack st...@duboce.net wrote:
 
   Does this 'admission' help with the which-hadoop thread too?
  
 
  Right -- working toward semver.
 
  Are we now at liberty to bump to 2.6 or 2.7 even for branch-1.1? I still
  have Karthik's offer to roll a 2.5.3 with the HDFS issue resolved.
 
  What about the jackson issue with 2.5 YARN runtime?
 
  Thanks Sean and Josh for being our community conscience on these issues.
 
   Just want to make sure we're all on the same page (or find out if not).
   
On Thu, Apr 23, 2015 at 2:59 PM, Enis Söztutar e...@apache.org
  wrote:
   
 Then let's get Andrew's proposal committed:

 + APIs available in a patch version will be available in all later
 + patch versions. However, new APIs may be added which will not be
 + available in earlier patch versions.

 I propose the following change to the Client Binary compatibility
section
 of Section 11.1:

 - Old client code can run unchanged (no recompilation needed)
 against
   new
 jars.
 + Client code written to APIs available in a given patch release
 + can run unchanged (no recompilation needed) against the new
 + jars of later patch versions.

 We can even claim that if you are not using new APIs, we are binary
compat
 for downgrade. But you have to make sure that your code compiles
 with
that
 downgraded version.

 Enis



 On Thu, Apr 23, 2015 at 2:04 PM, Sean Busbey bus...@cloudera.com
wrote:

  I'm fine with us adding methods to public APIs in patch releases
 so
long
 as
  we stop claiming to follow semver. We can say we take the
  principles
   of
  semver as inspiration. This would reflect the current state of
 the
world
  WRT 1.0.1 and would still give us a reason keep the narrower
   definition
 of
  new features in minor versions.
 
 
  No longer claiming semver would also have the added benefit of
  making
it
  for me to easier to explain our compatibility promises to people.
   Right
 now
  I have to explain the difference between the things that get
 proper
 semver
  treatment (e.g. public api, wire compat) and which things are
downgraded
 to
  breaking-on-minor (e.g. LimitedPrivate, binary compatibility).
   Starting
  with the context of we like semver will be easier than we use
semver.
 
 
  Like Josh, my main concern is that we accurately advertise what
  we're
  doing. There are few things I've found more frustrating than
 being
  an
end
  user of a project that claims to follow semver without
  understanding
the
  burden that carries (and subsequently not meeting it).
 
  On Thu, Apr 23, 2015 at 3:48 PM, Stack st...@duboce.net wrote:
 
   I agree w/ the Enis characterization (so we need the callout on
semvar)
  but
   think we should practice what Seans says (patch is bug fixes
  only).
   St.Ack
  
   On Thu, Apr 23, 2015 at 1:31 PM, Sean Busbey 
  bus...@cloudera.com
  wrote:
  
Why don't we just focus development after a minor release on
  the
next
   minor
release instead of the next patch release?
   
We could limit backports to the patch releases to critical
  bugs,
 which
would cut down on how often someone has to deal with the pain
  of
 making
sure we don't add to public APIs. It also reduces the risk
   someone
  going
through an upgrade has, since there are fewer changes.
   
If someone fixes a bug and doesn't want to do the work of
  making
sure
  it
doesn't add methods in a patch release, they just don't
  backport
   to
  that
version and make a follow on e.g. backport to 1.0.z ticket.
   
   
   
On Thu, Apr 23, 2015 at 1:50 PM, Enis Söztutar 
   enis@gmail.com

   wrote:
   
 +1 to the proposal.

 The problem is that we have a very big API surface
 especially
with
  the
 coprocessors included in the report. Even simple bug fixes
  can
   introduce
 protected or public methods to base classes, which makes
  patch
  releases
 very hard to maintain. I would not want to spend the effort
  to
 spend
   tons
 of time trying to make a patch not introduce new methods in
   order
 to
 

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Stack]

On Fri, Apr 24, 2015 at 10:25 AM, Sean Busbey bus...@cloudera.com wrote:

 On Fri, Apr 24, 2015 at 12:21 PM, Stack st...@duboce.net wrote:

  On Fri, Apr 24, 2015 at 9:21 AM, Josh Elser josh.el...@gmail.com
 wrote:
 
   Cool, it seems that we have consensus. Let me try to give a patch since
   you all were nice enough to entertain me with a good discussion on the
   matter!
  
   https://issues.apache.org/jira/browse/HBASE-13554
  
  
  
  Thanks for filing the issue.
 
  Do we want to say SemVer in 2.0+?
 
  St.Ack
 
 
 Let's see how we do on establishing release cadence on 1+ and maintaining
 our existing compatibility promises before we set goals on how strict we'll
 be on versions in 2+. :)



Fair enough.
St.Ack



 --
 Sean

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Elliott Clark]

Is 2.7.1 going to have that not for production tag or was that tag just
because of the issues found late in the cycle that are now fixed?

On Fri, Apr 24, 2015 at 9:50 AM, Sean Busbey bus...@cloudera.com wrote:

 On Thu, Apr 23, 2015 at 7:34 PM, Nick Dimiduk ndimi...@gmail.com wrote:

  On Thu, Apr 23, 2015 at 4:13 PM, Stack st...@duboce.net wrote:
 
   Does this 'admission' help with the which-hadoop thread too?
  
 
  Right -- working toward semver.
 
  Are we now at liberty to bump to 2.6 or 2.7 even for branch-1.1? I still
  have Karthik's offer to roll a 2.5.3 with the HDFS issue resolved.
 
  What about the jackson issue with 2.5 YARN runtime?
 
  Thanks Sean and Josh for being our community conscience on these issues.
 
 
 I think it would still be a good idea to encourage Hadoop to put out 2.5.3
 and 2.6.1 with their respective HDFS issues resolved. It'll be good for our
 downstream users who may not be able to upgrade to a new major Hadoop
 version yet.

 I don't think we should bump to 2.7.0, given the not for production
 disclaimer in the release notes. To me, it would seem equivalent to if
 Phoenix had updated to use HBase 0.99.

 On the Jackson thing, are folks opposed to compiling with the current older
 version and then packaging the newer version? That would make sure we don't
 start using 1.9 features in a way that would prevent downstream users from
 downgrading to 1.8, which combined with a release note would address my
 concerns.


 --
 Sean

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Josh Elser]

Cool, it seems that we have consensus. Let me try to give a patch since 
you all were nice enough to entertain me with a good discussion on the 
matter!


https://issues.apache.org/jira/browse/HBASE-13554

Enis Söztutar wrote:

+1 to working toward semver. Is there any documentation that we would
like to clarify in the book given this enlightenment?

I would be in favor of going 2.6.0 and jackson upgrade in 1.1.

Enis

On Thu, Apr 23, 2015 at 5:34 PM, Nick Dimidukndimi...@gmail.com  wrote:


On Thu, Apr 23, 2015 at 4:13 PM, Stackst...@duboce.net  wrote:


Does this 'admission' help with the which-hadoop thread too?


Right -- working toward semver.

Are we now at liberty to bump to 2.6 or 2.7 even for branch-1.1? I still
have Karthik's offer to roll a 2.5.3 with the HDFS issue resolved.

What about the jackson issue with 2.5 YARN runtime?

Thanks Sean and Josh for being our community conscience on these issues.


Just want to make sure we're all on the same page (or find out if not).

On Thu, Apr 23, 2015 at 2:59 PM, Enis Söztutare...@apache.org

wrote:

Then let's get Andrew's proposal committed:

+ APIs available in a patch version will be available in all later
+ patch versions. However, new APIs may be added which will not be
+ available in earlier patch versions.

I propose the following change to the Client Binary compatibility

section

of Section 11.1:

- Old client code can run unchanged (no recompilation needed) against

new

jars.
+ Client code written to APIs available in a given patch release
+ can run unchanged (no recompilation needed) against the new
+ jars of later patch versions.

We can even claim that if you are not using new APIs, we are binary

compat

for downgrade. But you have to make sure that your code compiles with

that

downgraded version.

Enis



On Thu, Apr 23, 2015 at 2:04 PM, Sean Busbeybus...@cloudera.com

wrote:

I'm fine with us adding methods to public APIs in patch releases so

long

as

we stop claiming to follow semver. We can say we take the

principles

of

semver as inspiration. This would reflect the current state of the

world

WRT 1.0.1 and would still give us a reason keep the narrower

definition

of

new features in minor versions.


No longer claiming semver would also have the added benefit of

making

it

for me to easier to explain our compatibility promises to people.

Right

now

I have to explain the difference between the things that get proper

semver

treatment (e.g. public api, wire compat) and which things are

downgraded

to

breaking-on-minor (e.g. LimitedPrivate, binary compatibility).

Starting

with the context of we like semver will be easier than we use

semver.


Like Josh, my main concern is that we accurately advertise what

we're

doing. There are few things I've found more frustrating than being

an

end

user of a project that claims to follow semver without

understanding

the

burden that carries (and subsequently not meeting it).

On Thu, Apr 23, 2015 at 3:48 PM, Stackst...@duboce.net  wrote:


I agree w/ the Enis characterization (so we need the callout on

semvar)

but

think we should practice what Seans says (patch is bug fixes

only).

St.Ack

On Thu, Apr 23, 2015 at 1:31 PM, Sean Busbey

bus...@cloudera.com

wrote:

Why don't we just focus development after a minor release on

the

next

minor

release instead of the next patch release?

We could limit backports to the patch releases to critical

bugs,

which

would cut down on how often someone has to deal with the pain

of

making

sure we don't add to public APIs. It also reduces the risk

someone

going

through an upgrade has, since there are fewer changes.

If someone fixes a bug and doesn't want to do the work of

making

sure

it

doesn't add methods in a patch release, they just don't

backport

to

that

version and make a follow on e.g. backport to 1.0.z ticket.



On Thu, Apr 23, 2015 at 1:50 PM, Enis Söztutar

enis@gmail.com

wrote:

+1 to the proposal.

The problem is that we have a very big API surface especially

with

the

coprocessors included in the report. Even simple bug fixes

can

introduce

protected or public methods to base classes, which makes

patch

releases

very hard to maintain. I would not want to spend the effort

to

spend

tons

of time trying to make a patch not introduce new methods in

order

to

backport. That effort can be spent elsewhere IMO.

Looking at the report


https://people.apache.org/~enis/1.0.0_1.0.1RC2_compat_report.html,

nothing

strikes me as new functionality. Going from current 1.0.0

to

1.0.1RC2

should actually be as you would expect from upgrading a patch

release.

Yes, adding new API in patch releases will make downgrading

harder,

but I

think that is an acceptable tradeoff. We can document that if

your

application compiles (meaning that you are not using new API)

with

1.0.0,

then you can swap your jars in a binary compat manner.

Enis

On Thu, Apr 23, 2015 at 10:03 AM, Andrew Purtell

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Sean Busbey]

On Thu, Apr 23, 2015 at 7:34 PM, Nick Dimiduk ndimi...@gmail.com wrote:

 On Thu, Apr 23, 2015 at 4:13 PM, Stack st...@duboce.net wrote:

  Does this 'admission' help with the which-hadoop thread too?
 

 Right -- working toward semver.

 Are we now at liberty to bump to 2.6 or 2.7 even for branch-1.1? I still
 have Karthik's offer to roll a 2.5.3 with the HDFS issue resolved.

 What about the jackson issue with 2.5 YARN runtime?

 Thanks Sean and Josh for being our community conscience on these issues.


I think it would still be a good idea to encourage Hadoop to put out 2.5.3
and 2.6.1 with their respective HDFS issues resolved. It'll be good for our
downstream users who may not be able to upgrade to a new major Hadoop
version yet.

I don't think we should bump to 2.7.0, given the not for production
disclaimer in the release notes. To me, it would seem equivalent to if
Phoenix had updated to use HBase 0.99.

On the Jackson thing, are folks opposed to compiling with the current older
version and then packaging the newer version? That would make sure we don't
start using 1.9 features in a way that would prevent downstream users from
downgrading to 1.8, which combined with a release note would address my
concerns.


-- 
Sean

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Stack]

On Fri, Apr 24, 2015 at 9:21 AM, Josh Elser josh.el...@gmail.com wrote:

 Cool, it seems that we have consensus. Let me try to give a patch since
 you all were nice enough to entertain me with a good discussion on the
 matter!

 https://issues.apache.org/jira/browse/HBASE-13554



Thanks for filing the issue.

Do we want to say SemVer in 2.0+?

St.Ack



 Enis Söztutar wrote:

 +1 to working toward semver. Is there any documentation that we would
 like to clarify in the book given this enlightenment?

 I would be in favor of going 2.6.0 and jackson upgrade in 1.1.

 Enis

 On Thu, Apr 23, 2015 at 5:34 PM, Nick Dimidukndimi...@gmail.com  wrote:

  On Thu, Apr 23, 2015 at 4:13 PM, Stackst...@duboce.net  wrote:

  Does this 'admission' help with the which-hadoop thread too?

  Right -- working toward semver.

 Are we now at liberty to bump to 2.6 or 2.7 even for branch-1.1? I still
 have Karthik's offer to roll a 2.5.3 with the HDFS issue resolved.

 What about the jackson issue with 2.5 YARN runtime?

 Thanks Sean and Josh for being our community conscience on these issues.

  Just want to make sure we're all on the same page (or find out if not).

 On Thu, Apr 23, 2015 at 2:59 PM, Enis Söztutare...@apache.org

 wrote:

 Then let's get Andrew's proposal committed:

 + APIs available in a patch version will be available in all later
 + patch versions. However, new APIs may be added which will not be
 + available in earlier patch versions.

 I propose the following change to the Client Binary compatibility

 section

 of Section 11.1:

 - Old client code can run unchanged (no recompilation needed) against

 new

 jars.
 + Client code written to APIs available in a given patch release
 + can run unchanged (no recompilation needed) against the new
 + jars of later patch versions.

 We can even claim that if you are not using new APIs, we are binary

 compat

 for downgrade. But you have to make sure that your code compiles with

 that

 downgraded version.

 Enis



 On Thu, Apr 23, 2015 at 2:04 PM, Sean Busbeybus...@cloudera.com

 wrote:

 I'm fine with us adding methods to public APIs in patch releases so

 long

 as

 we stop claiming to follow semver. We can say we take the

 principles

 of

 semver as inspiration. This would reflect the current state of the

 world

 WRT 1.0.1 and would still give us a reason keep the narrower

 definition

 of

 new features in minor versions.


 No longer claiming semver would also have the added benefit of

 making

 it

 for me to easier to explain our compatibility promises to people.

 Right

 now

 I have to explain the difference between the things that get proper

 semver

 treatment (e.g. public api, wire compat) and which things are

 downgraded

 to

 breaking-on-minor (e.g. LimitedPrivate, binary compatibility).

 Starting

 with the context of we like semver will be easier than we use

 semver.


 Like Josh, my main concern is that we accurately advertise what

 we're

 doing. There are few things I've found more frustrating than being

 an

 end

 user of a project that claims to follow semver without

 understanding

 the

 burden that carries (and subsequently not meeting it).

 On Thu, Apr 23, 2015 at 3:48 PM, Stackst...@duboce.net  wrote:

  I agree w/ the Enis characterization (so we need the callout on

 semvar)

 but

 think we should practice what Seans says (patch is bug fixes

 only).

 St.Ack

 On Thu, Apr 23, 2015 at 1:31 PM, Sean Busbey

 bus...@cloudera.com

 wrote:

 Why don't we just focus development after a minor release on

 the

 next

 minor

 release instead of the next patch release?

 We could limit backports to the patch releases to critical

 bugs,

 which

 would cut down on how often someone has to deal with the pain

 of

 making

 sure we don't add to public APIs. It also reduces the risk

 someone

 going

 through an upgrade has, since there are fewer changes.

 If someone fixes a bug and doesn't want to do the work of

 making

 sure

 it

 doesn't add methods in a patch release, they just don't

 backport

 to

 that

 version and make a follow on e.g. backport to 1.0.z ticket.



 On Thu, Apr 23, 2015 at 1:50 PM, Enis Söztutar

 enis@gmail.com

 wrote:

 +1 to the proposal.

 The problem is that we have a very big API surface especially

 with

 the

 coprocessors included in the report. Even simple bug fixes

 can

 introduce

 protected or public methods to base classes, which makes

 patch

 releases

 very hard to maintain. I would not want to spend the effort

 to

 spend

 tons

 of time trying to make a patch not introduce new methods in

 order

 to

 backport. That effort can be spent elsewhere IMO.

 Looking at the report


 https://people.apache.org/~enis/1.0.0_1.0.1RC2_compat_report.html,

 nothing

 strikes me as new functionality. Going from current 1.0.0

 to

 1.0.1RC2

 should actually be as you would expect from upgrading a patch

 release.

 Yes, adding new API in patch releases will make downgrading

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Sean Busbey]

On Fri, Apr 24, 2015 at 11:53 AM, Elliott Clark ecl...@apache.org wrote:

 Is 2.7.1 going to have that not for production tag or was that tag just
 because of the issues found late in the cycle that are now fixed?


From reading from hadoop lists, I'm not sure. Including the non-production
tag seemed to be a compromise position to get 2.7.0 out the door. The cycle
covered so much ground that I'm not sure when it got decided. The
discussion also got them onto figuring out what their general policy on
production / non-production will be. When they come to a decision it'll be
good to revisit and see what it means for us.

At least some of the Hadoopers are in agreement that 2.7.1 should be
labeled production ready. No one was opposed to the matter, but there were
some noticeable gaps in assenting voices[1].

It's supposed to be soon, so we'll see I guess?

[1]:
http://markmail.org/thread/zwzze6cqqgwq4rmw#query:+page:1+mid:u6pihxyy36pzz52e+state:results

-- 
Sean

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Andrew Purtell]

Anyone disagree with the point of view put forward by Josh and Sean?


On Wed, Apr 22, 2015 at 10:35 AM, Josh Elser josh.el...@gmail.com wrote:

 Andy -- I understood your intent, but thanks for clarifying. (as well as
 taking the time to break this discussion out in the first place). I agree
 with your assessment.

 re: Sean's comments, if it wasn't clear by me asking in the first place, I
 also think sticking as close as possible to semver's rules is the best
 approach, although I'm getting the impression that there have been some
 previous reservations to doing so (especially by your comment about
 backporting features if there is demand is).

 I've found adhering to the bug-fix release restrictions can be a very
 painful and time-consuming task, so this is something to get a
 representative sampling of those who do the work to make sure everyone is
 on board.


 Sean Busbey wrote:

 I'd much rather we stick with the definitions used in Semantic Versioning.
 Our use is already confusing enough given our matrix of compatibilities
 that don't get major version for breaking protections.

 We've previously discussed how we'll do additional minor releases when
 there's sufficient interest in the new features present there. What's
 building that demand if any backwards compatible change can go back into a
 patch release?

 Would we have an easier time restraining ourselves if we had a regular
 schedule planned around new minor versions?


 On Wed, Apr 22, 2015 at 12:03 PM, Josh Elserjosh.el...@gmail.com
 wrote:

  While I can understand the desire to want to add things, I do think it
 makes things harder for users to reliably write code against versions of
 HBase which (by their view) should be completely compatible with one
 another.

 Take this extremely hypothetical situation: I'm new to HBase and start
 writing some code against HBase 1.0.1 which was just deployed at my
 $job. I
 don't _know_ what APIs are new, I just know what exists and treat that as
 acceptable for me to be using. Meanwhile in production, some other people
 find a bug with HBase 1.0.1 and roll back to 1.0.0 which they had been
 previously using. My reaction would be of course my code should work
 with
 HBase 1.0.0, I only used the public API when in fact this is not true.

 Personally, I think it's a little bold to say semver is even in use if
 this principal isn't being followed as it doesn't follow at all with my
 understanding on the guarantees defined by semver for bug-fix releases.

 That being said, if the intent *is* to allow ourselves to make these
 sorts
 of changes, I just think some sort of disclaimer should be present:

 - HBase uses Semantic Versioning for its release versioning
 + HBase uses Semantic Versioning for its release versioning with a caveat
 that methods and members might be added in newer bug-fix releases that
 were
 not present in the previous bug-fix release.


 Andrew Purtell wrote:

  [Subject changed]

 On Tue, Apr 21, 2015 at 8:47 PM, Josh Elserjosh.el...@gmail.com
  wrote:

   I was a little surprised when I noticed method additions to

 InterfaceAudience.Public annotated classes. This means that a user
 could
 write code against 1.0.1 that would not work against 1.0.0 which seems
 undesirable for a bugfix release. I read over the book section on
 compatibility and didn't see this addressed, so I thought I'd ask.


 Let's clarify this. It's not the first time this question has been
 asked.

 To get things moving:

 I propose the following addition to the Client API compatibility
 section
 of Section 11.1:

 + APIs available in a patch version will be available in all later
 + patch versions. However, new APIs may be added which will not be
 + available in earlier patch versions.

 I propose the following change to the Client Binary compatibility
 section
 of Section 11.1:

 - Old client code can run unchanged (no recompilation needed) against
 new
 jars.
 + Client code written to APIs available in a given patch release
 + can run unchanged (no recompilation needed) against the new
 + jars of later patch versions.


 What do you think?

 If these changes are (mostly) ok, then this clarifies in one direction.

 If these changes are not acceptable, I will propose edits that clarify
 toward the opposite meaning. ​








-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Enis Söztutar]

+1 to the proposal.

The problem is that we have a very big API surface especially with the
coprocessors included in the report. Even simple bug fixes can introduce
protected or public methods to base classes, which makes patch releases
very hard to maintain. I would not want to spend the effort to spend tons
of time trying to make a patch not introduce new methods in order to
backport. That effort can be spent elsewhere IMO.

Looking at the report
https://people.apache.org/~enis/1.0.0_1.0.1RC2_compat_report.html, nothing
strikes me as new functionality. Going from current 1.0.0 to 1.0.1RC2
should actually be as you would expect from upgrading a patch release.

Yes, adding new API in patch releases will make downgrading harder, but I
think that is an acceptable tradeoff. We can document that if your
application compiles (meaning that you are not using new API) with 1.0.0,
then you can swap your jars in a binary compat manner.

Enis

On Thu, Apr 23, 2015 at 10:03 AM, Andrew Purtell apurt...@apache.org
wrote:

 Anyone disagree with the point of view put forward by Josh and Sean?


 On Wed, Apr 22, 2015 at 10:35 AM, Josh Elser josh.el...@gmail.com wrote:

  Andy -- I understood your intent, but thanks for clarifying. (as well as
  taking the time to break this discussion out in the first place). I agree
  with your assessment.
 
  re: Sean's comments, if it wasn't clear by me asking in the first place,
 I
  also think sticking as close as possible to semver's rules is the best
  approach, although I'm getting the impression that there have been some
  previous reservations to doing so (especially by your comment about
  backporting features if there is demand is).
 
  I've found adhering to the bug-fix release restrictions can be a very
  painful and time-consuming task, so this is something to get a
  representative sampling of those who do the work to make sure everyone is
  on board.
 
 
  Sean Busbey wrote:
 
  I'd much rather we stick with the definitions used in Semantic
 Versioning.
  Our use is already confusing enough given our matrix of compatibilities
  that don't get major version for breaking protections.
 
  We've previously discussed how we'll do additional minor releases when
  there's sufficient interest in the new features present there. What's
  building that demand if any backwards compatible change can go back
 into a
  patch release?
 
  Would we have an easier time restraining ourselves if we had a regular
  schedule planned around new minor versions?
 
 
  On Wed, Apr 22, 2015 at 12:03 PM, Josh Elserjosh.el...@gmail.com
  wrote:
 
   While I can understand the desire to want to add things, I do think it
  makes things harder for users to reliably write code against versions
 of
  HBase which (by their view) should be completely compatible with one
  another.
 
  Take this extremely hypothetical situation: I'm new to HBase and start
  writing some code against HBase 1.0.1 which was just deployed at my
  $job. I
  don't _know_ what APIs are new, I just know what exists and treat that
 as
  acceptable for me to be using. Meanwhile in production, some other
 people
  find a bug with HBase 1.0.1 and roll back to 1.0.0 which they had been
  previously using. My reaction would be of course my code should work
  with
  HBase 1.0.0, I only used the public API when in fact this is not true.
 
  Personally, I think it's a little bold to say semver is even in use if
  this principal isn't being followed as it doesn't follow at all with my
  understanding on the guarantees defined by semver for bug-fix releases.
 
  That being said, if the intent *is* to allow ourselves to make these
  sorts
  of changes, I just think some sort of disclaimer should be present:
 
  - HBase uses Semantic Versioning for its release versioning
  + HBase uses Semantic Versioning for its release versioning with a
 caveat
  that methods and members might be added in newer bug-fix releases that
  were
  not present in the previous bug-fix release.
 
 
  Andrew Purtell wrote:
 
   [Subject changed]
 
  On Tue, Apr 21, 2015 at 8:47 PM, Josh Elserjosh.el...@gmail.com
   wrote:
 
I was a little surprised when I noticed method additions to
 
  InterfaceAudience.Public annotated classes. This means that a user
  could
  write code against 1.0.1 that would not work against 1.0.0 which
 seems
  undesirable for a bugfix release. I read over the book section on
  compatibility and didn't see this addressed, so I thought I'd ask.
 
 
  Let's clarify this. It's not the first time this question has been
  asked.
 
  To get things moving:
 
  I propose the following addition to the Client API compatibility
  section
  of Section 11.1:
 
  + APIs available in a patch version will be available in all later
  + patch versions. However, new APIs may be added which will not be
  + available in earlier patch versions.
 
  I propose the following change to the Client Binary compatibility
  section
  of Section 11.1:
 
  - Old client code 

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Sean Busbey]

Why don't we just focus development after a minor release on the next minor
release instead of the next patch release?

We could limit backports to the patch releases to critical bugs, which
would cut down on how often someone has to deal with the pain of making
sure we don't add to public APIs. It also reduces the risk someone going
through an upgrade has, since there are fewer changes.

If someone fixes a bug and doesn't want to do the work of making sure it
doesn't add methods in a patch release, they just don't backport to that
version and make a follow on e.g. backport to 1.0.z ticket.



On Thu, Apr 23, 2015 at 1:50 PM, Enis Söztutar enis@gmail.com wrote:

 +1 to the proposal.

 The problem is that we have a very big API surface especially with the
 coprocessors included in the report. Even simple bug fixes can introduce
 protected or public methods to base classes, which makes patch releases
 very hard to maintain. I would not want to spend the effort to spend tons
 of time trying to make a patch not introduce new methods in order to
 backport. That effort can be spent elsewhere IMO.

 Looking at the report
 https://people.apache.org/~enis/1.0.0_1.0.1RC2_compat_report.html, nothing
 strikes me as new functionality. Going from current 1.0.0 to 1.0.1RC2
 should actually be as you would expect from upgrading a patch release.

 Yes, adding new API in patch releases will make downgrading harder, but I
 think that is an acceptable tradeoff. We can document that if your
 application compiles (meaning that you are not using new API) with 1.0.0,
 then you can swap your jars in a binary compat manner.

 Enis

 On Thu, Apr 23, 2015 at 10:03 AM, Andrew Purtell apurt...@apache.org
 wrote:

  Anyone disagree with the point of view put forward by Josh and Sean?
 
 
  On Wed, Apr 22, 2015 at 10:35 AM, Josh Elser josh.el...@gmail.com
 wrote:
 
   Andy -- I understood your intent, but thanks for clarifying. (as well
 as
   taking the time to break this discussion out in the first place). I
 agree
   with your assessment.
  
   re: Sean's comments, if it wasn't clear by me asking in the first
 place,
  I
   also think sticking as close as possible to semver's rules is the best
   approach, although I'm getting the impression that there have been some
   previous reservations to doing so (especially by your comment about
   backporting features if there is demand is).
  
   I've found adhering to the bug-fix release restrictions can be a very
   painful and time-consuming task, so this is something to get a
   representative sampling of those who do the work to make sure everyone
 is
   on board.
  
  
   Sean Busbey wrote:
  
   I'd much rather we stick with the definitions used in Semantic
  Versioning.
   Our use is already confusing enough given our matrix of
 compatibilities
   that don't get major version for breaking protections.
  
   We've previously discussed how we'll do additional minor releases when
   there's sufficient interest in the new features present there. What's
   building that demand if any backwards compatible change can go back
  into a
   patch release?
  
   Would we have an easier time restraining ourselves if we had a regular
   schedule planned around new minor versions?
  
  
   On Wed, Apr 22, 2015 at 12:03 PM, Josh Elserjosh.el...@gmail.com
   wrote:
  
While I can understand the desire to want to add things, I do think
 it
   makes things harder for users to reliably write code against versions
  of
   HBase which (by their view) should be completely compatible with one
   another.
  
   Take this extremely hypothetical situation: I'm new to HBase and
 start
   writing some code against HBase 1.0.1 which was just deployed at my
   $job. I
   don't _know_ what APIs are new, I just know what exists and treat
 that
  as
   acceptable for me to be using. Meanwhile in production, some other
  people
   find a bug with HBase 1.0.1 and roll back to 1.0.0 which they had
 been
   previously using. My reaction would be of course my code should work
   with
   HBase 1.0.0, I only used the public API when in fact this is not
 true.
  
   Personally, I think it's a little bold to say semver is even in use
 if
   this principal isn't being followed as it doesn't follow at all with
 my
   understanding on the guarantees defined by semver for bug-fix
 releases.
  
   That being said, if the intent *is* to allow ourselves to make these
   sorts
   of changes, I just think some sort of disclaimer should be present:
  
   - HBase uses Semantic Versioning for its release versioning
   + HBase uses Semantic Versioning for its release versioning with a
  caveat
   that methods and members might be added in newer bug-fix releases
 that
   were
   not present in the previous bug-fix release.
  
  
   Andrew Purtell wrote:
  
[Subject changed]
  
   On Tue, Apr 21, 2015 at 8:47 PM, Josh Elserjosh.el...@gmail.com
wrote:
  
 I was a little surprised when I noticed method additions to
  
   

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Josh Elser]

Enis Söztutar wrote:

+1 to the proposal.

The problem is that we have a very big API surface especially with the
coprocessors included in the report. Even simple bug fixes can introduce
protected or public methods to base classes, which makes patch releases
very hard to maintain. I would not want to spend the effort to spend tons
of time trying to make a patch not introduce new methods in order to
backport. That effort can be spent elsewhere IMO.


Abso-friggin-lutely. Following bug-fix strictness is a _massive_ pain in 
the rear (been dealing with this over in Accumulo-landia and it sucks). 
IMO, the crux of what to consider here is finding the acceptable level 
to which HBase Devs want to hold themselves and how much knowledge is 
just expected of users.


Like you said, it is a lot of effort for what seems like ultimately very 
little effect on users. It will take constant effort by everyone landing 
patches in a bug-fix branch to not make the RM want to punch themself in 
the face.



Looking at the report
https://people.apache.org/~enis/1.0.0_1.0.1RC2_compat_report.html, nothing
strikes me as new functionality. Going from current 1.0.0 to 1.0.1RC2
should actually be as you would expect from upgrading a patch release.

Yes, adding new API in patch releases will make downgrading harder, but I
think that is an acceptable tradeoff. We can document that if your
application compiles (meaning that you are not using new API) with 1.0.0,
then you can swap your jars in a binary compat manner.


Like you pointed out in the other thread, though, choosing to follow 
semver is for the user. Your proposed solution adds in extra complexity 
on users that would otherwise be as simple as flipping the version on 
their Maven/Ivy configuration. Like you say, they should still be able 
to do things with the new jars against the old cluster, but it is a 
sharp-corner users would need to be aware of that isn't semver as 
advertised.


Really, I'm trying to play devil's advocate here. In the end, my only 
concern would be appropriate advertisement of what decision is made.



Enis

On Thu, Apr 23, 2015 at 10:03 AM, Andrew Purtellapurt...@apache.org
wrote:


Anyone disagree with the point of view put forward by Josh and Sean?

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Stack]

I agree w/ the Enis characterization (so we need the callout on semvar) but
think we should practice what Seans says (patch is bug fixes only).
St.Ack

On Thu, Apr 23, 2015 at 1:31 PM, Sean Busbey bus...@cloudera.com wrote:

 Why don't we just focus development after a minor release on the next minor
 release instead of the next patch release?

 We could limit backports to the patch releases to critical bugs, which
 would cut down on how often someone has to deal with the pain of making
 sure we don't add to public APIs. It also reduces the risk someone going
 through an upgrade has, since there are fewer changes.

 If someone fixes a bug and doesn't want to do the work of making sure it
 doesn't add methods in a patch release, they just don't backport to that
 version and make a follow on e.g. backport to 1.0.z ticket.



 On Thu, Apr 23, 2015 at 1:50 PM, Enis Söztutar enis@gmail.com wrote:

  +1 to the proposal.
 
  The problem is that we have a very big API surface especially with the
  coprocessors included in the report. Even simple bug fixes can introduce
  protected or public methods to base classes, which makes patch releases
  very hard to maintain. I would not want to spend the effort to spend tons
  of time trying to make a patch not introduce new methods in order to
  backport. That effort can be spent elsewhere IMO.
 
  Looking at the report
  https://people.apache.org/~enis/1.0.0_1.0.1RC2_compat_report.html,
 nothing
  strikes me as new functionality. Going from current 1.0.0 to 1.0.1RC2
  should actually be as you would expect from upgrading a patch release.
 
  Yes, adding new API in patch releases will make downgrading harder, but I
  think that is an acceptable tradeoff. We can document that if your
  application compiles (meaning that you are not using new API) with 1.0.0,
  then you can swap your jars in a binary compat manner.
 
  Enis
 
  On Thu, Apr 23, 2015 at 10:03 AM, Andrew Purtell apurt...@apache.org
  wrote:
 
   Anyone disagree with the point of view put forward by Josh and Sean?
  
  
   On Wed, Apr 22, 2015 at 10:35 AM, Josh Elser josh.el...@gmail.com
  wrote:
  
Andy -- I understood your intent, but thanks for clarifying. (as well
  as
taking the time to break this discussion out in the first place). I
  agree
with your assessment.
   
re: Sean's comments, if it wasn't clear by me asking in the first
  place,
   I
also think sticking as close as possible to semver's rules is the
 best
approach, although I'm getting the impression that there have been
 some
previous reservations to doing so (especially by your comment about
backporting features if there is demand is).
   
I've found adhering to the bug-fix release restrictions can be a very
painful and time-consuming task, so this is something to get a
representative sampling of those who do the work to make sure
 everyone
  is
on board.
   
   
Sean Busbey wrote:
   
I'd much rather we stick with the definitions used in Semantic
   Versioning.
Our use is already confusing enough given our matrix of
  compatibilities
that don't get major version for breaking protections.
   
We've previously discussed how we'll do additional minor releases
 when
there's sufficient interest in the new features present there.
 What's
building that demand if any backwards compatible change can go back
   into a
patch release?
   
Would we have an easier time restraining ourselves if we had a
 regular
schedule planned around new minor versions?
   
   
On Wed, Apr 22, 2015 at 12:03 PM, Josh Elserjosh.el...@gmail.com
wrote:
   
 While I can understand the desire to want to add things, I do think
  it
makes things harder for users to reliably write code against
 versions
   of
HBase which (by their view) should be completely compatible with
 one
another.
   
Take this extremely hypothetical situation: I'm new to HBase and
  start
writing some code against HBase 1.0.1 which was just deployed at my
$job. I
don't _know_ what APIs are new, I just know what exists and treat
  that
   as
acceptable for me to be using. Meanwhile in production, some other
   people
find a bug with HBase 1.0.1 and roll back to 1.0.0 which they had
  been
previously using. My reaction would be of course my code should
 work
with
HBase 1.0.0, I only used the public API when in fact this is not
  true.
   
Personally, I think it's a little bold to say semver is even in use
  if
this principal isn't being followed as it doesn't follow at all
 with
  my
understanding on the guarantees defined by semver for bug-fix
  releases.
   
That being said, if the intent *is* to allow ourselves to make
 these
sorts
of changes, I just think some sort of disclaimer should be present:
   
- HBase uses Semantic Versioning for its release versioning
+ HBase uses Semantic Versioning for its release versioning 

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Andrew Purtell]

So we like semver, not we use semver?

Just want to make sure we're all on the same page (or find out if not).

On Thu, Apr 23, 2015 at 2:59 PM, Enis Söztutar e...@apache.org wrote:

 Then let's get Andrew's proposal committed:

 + APIs available in a patch version will be available in all later
 + patch versions. However, new APIs may be added which will not be
 + available in earlier patch versions.

 I propose the following change to the Client Binary compatibility section
 of Section 11.1:

 - Old client code can run unchanged (no recompilation needed) against new
 jars.
 + Client code written to APIs available in a given patch release
 + can run unchanged (no recompilation needed) against the new
 + jars of later patch versions.

 We can even claim that if you are not using new APIs, we are binary compat
 for downgrade. But you have to make sure that your code compiles with that
 downgraded version.

 Enis



 On Thu, Apr 23, 2015 at 2:04 PM, Sean Busbey bus...@cloudera.com wrote:

  I'm fine with us adding methods to public APIs in patch releases so long
 as
  we stop claiming to follow semver. We can say we take the principles of
  semver as inspiration. This would reflect the current state of the world
  WRT 1.0.1 and would still give us a reason keep the narrower definition
 of
  new features in minor versions.
 
 
  No longer claiming semver would also have the added benefit of making it
  for me to easier to explain our compatibility promises to people. Right
 now
  I have to explain the difference between the things that get proper
 semver
  treatment (e.g. public api, wire compat) and which things are downgraded
 to
  breaking-on-minor (e.g. LimitedPrivate, binary compatibility). Starting
  with the context of we like semver will be easier than we use semver.
 
 
  Like Josh, my main concern is that we accurately advertise what we're
  doing. There are few things I've found more frustrating than being an end
  user of a project that claims to follow semver without understanding the
  burden that carries (and subsequently not meeting it).
 
  On Thu, Apr 23, 2015 at 3:48 PM, Stack st...@duboce.net wrote:
 
   I agree w/ the Enis characterization (so we need the callout on semvar)
  but
   think we should practice what Seans says (patch is bug fixes only).
   St.Ack
  
   On Thu, Apr 23, 2015 at 1:31 PM, Sean Busbey bus...@cloudera.com
  wrote:
  
Why don't we just focus development after a minor release on the next
   minor
release instead of the next patch release?
   
We could limit backports to the patch releases to critical bugs,
 which
would cut down on how often someone has to deal with the pain of
 making
sure we don't add to public APIs. It also reduces the risk someone
  going
through an upgrade has, since there are fewer changes.
   
If someone fixes a bug and doesn't want to do the work of making sure
  it
doesn't add methods in a patch release, they just don't backport to
  that
version and make a follow on e.g. backport to 1.0.z ticket.
   
   
   
On Thu, Apr 23, 2015 at 1:50 PM, Enis Söztutar enis@gmail.com
   wrote:
   
 +1 to the proposal.

 The problem is that we have a very big API surface especially with
  the
 coprocessors included in the report. Even simple bug fixes can
   introduce
 protected or public methods to base classes, which makes patch
  releases
 very hard to maintain. I would not want to spend the effort to
 spend
   tons
 of time trying to make a patch not introduce new methods in order
 to
 backport. That effort can be spent elsewhere IMO.

 Looking at the report
 https://people.apache.org/~enis/1.0.0_1.0.1RC2_compat_report.html,
nothing
 strikes me as new functionality. Going from current 1.0.0 to
  1.0.1RC2
 should actually be as you would expect from upgrading a patch
  release.

 Yes, adding new API in patch releases will make downgrading harder,
   but I
 think that is an acceptable tradeoff. We can document that if your
 application compiles (meaning that you are not using new API) with
   1.0.0,
 then you can swap your jars in a binary compat manner.

 Enis

 On Thu, Apr 23, 2015 at 10:03 AM, Andrew Purtell 
  apurt...@apache.org
 wrote:

  Anyone disagree with the point of view put forward by Josh and
  Sean?
 
 
  On Wed, Apr 22, 2015 at 10:35 AM, Josh Elser 
 josh.el...@gmail.com
  
 wrote:
 
   Andy -- I understood your intent, but thanks for clarifying.
 (as
   well
 as
   taking the time to break this discussion out in the first
  place). I
 agree
   with your assessment.
  
   re: Sean's comments, if it wasn't clear by me asking in the
 first
 place,
  I
   also think sticking as close as possible to semver's rules is
 the
best
   approach, although I'm getting the impression that there have
  been
some
   previous 

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Andrew Purtell]

 We also do it, so that users see less of This was in 0.98, but how come
it is not in 1.0.x. I think we will get substantially better over time.

It will. Eventually we will EOL 0.98

In the meantime we could clarify how 0.98 relates to later releases now
that releases 1.0.0 and up are being more strict about compatibility. There
will always be a 1.x that provides a straight upgrade path from 0.98, or if
not that is an error in 0.98 maintenance that would need to be caught and
corrected (hopefully before release). The part of it which may not be
obvious is which 1.x is the straight path upgrade from 0.98 could change
over time.

As 0.98 RM I'm also ok with being more strict about backporting features at
this point. I think many (most?) are moving on from 0.98 and we should be
encouraging new users to start with 1.0. However, let's use HBASE-13477 as
an example. Where we have people running 0.98 in production and cannot
upgrade right now beyond a later 0.98 point version, not having access to
highly useful metrics simply due to self-imposed policy isn't operator
friendly. Sure, we could say since you want it, upgrade :-) but that's
unsatisfying if upgrading just isn't possible. If we want to narrow down
when and why we'd put something in 0.98 that could only go into the latest
1.x, we can have that conversation.


On Thu, Apr 23, 2015 at 2:58 PM, Enis Söztutar enis@gmail.com wrote:

 On Thu, Apr 23, 2015 at 1:31 PM, Sean Busbey bus...@cloudera.com wrote:

  Why don't we just focus development after a minor release on the next
 minor
  release instead of the next patch release?
 
  We could limit backports to the patch releases to critical bugs, which
  would cut down on how often someone has to deal with the pain of making
  sure we don't add to public APIs. It also reduces the risk someone going
  through an upgrade has, since there are fewer changes.
 

 Big +1. I think we will get there, but devs/committers are used to
 backporting simple
 new features across all branches. Not to nit pick or anything, an example
 is
 HBASE-13168 which I failed to catch. Ideally it should not have been in
 1.0.1. We also
 do it, so that users see less of This was in 0.98, but how come it is not
 in 1.0.x. I think
 we will get substantially better over time.


 
  If someone fixes a bug and doesn't want to do the work of making sure it
  doesn't add methods in a patch release, they just don't backport to that
  version and make a follow on e.g. backport to 1.0.z ticket.
 
 
 
  On Thu, Apr 23, 2015 at 1:50 PM, Enis Söztutar enis@gmail.com
 wrote:
 
   +1 to the proposal.
  
   The problem is that we have a very big API surface especially with the
   coprocessors included in the report. Even simple bug fixes can
 introduce
   protected or public methods to base classes, which makes patch releases
   very hard to maintain. I would not want to spend the effort to spend
 tons
   of time trying to make a patch not introduce new methods in order to
   backport. That effort can be spent elsewhere IMO.
  
   Looking at the report
   https://people.apache.org/~enis/1.0.0_1.0.1RC2_compat_report.html,
  nothing
   strikes me as new functionality. Going from current 1.0.0 to 1.0.1RC2
   should actually be as you would expect from upgrading a patch release.
  
   Yes, adding new API in patch releases will make downgrading harder,
 but I
   think that is an acceptable tradeoff. We can document that if your
   application compiles (meaning that you are not using new API) with
 1.0.0,
   then you can swap your jars in a binary compat manner.
  
   Enis
  
   On Thu, Apr 23, 2015 at 10:03 AM, Andrew Purtell apurt...@apache.org
   wrote:
  
Anyone disagree with the point of view put forward by Josh and Sean?
   
   
On Wed, Apr 22, 2015 at 10:35 AM, Josh Elser josh.el...@gmail.com
   wrote:
   
 Andy -- I understood your intent, but thanks for clarifying. (as
 well
   as
 taking the time to break this discussion out in the first place). I
   agree
 with your assessment.

 re: Sean's comments, if it wasn't clear by me asking in the first
   place,
I
 also think sticking as close as possible to semver's rules is the
  best
 approach, although I'm getting the impression that there have been
  some
 previous reservations to doing so (especially by your comment about
 backporting features if there is demand is).

 I've found adhering to the bug-fix release restrictions can be a
 very
 painful and time-consuming task, so this is something to get a
 representative sampling of those who do the work to make sure
  everyone
   is
 on board.


 Sean Busbey wrote:

 I'd much rather we stick with the definitions used in Semantic
Versioning.
 Our use is already confusing enough given our matrix of
   compatibilities
 that don't get major version for breaking protections.

 We've previously discussed how we'll do additional minor releases
  

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Sean Busbey]

I'm fine with us adding methods to public APIs in patch releases so long as
we stop claiming to follow semver. We can say we take the principles of
semver as inspiration. This would reflect the current state of the world
WRT 1.0.1 and would still give us a reason keep the narrower definition of
new features in minor versions.


No longer claiming semver would also have the added benefit of making it
for me to easier to explain our compatibility promises to people. Right now
I have to explain the difference between the things that get proper semver
treatment (e.g. public api, wire compat) and which things are downgraded to
breaking-on-minor (e.g. LimitedPrivate, binary compatibility). Starting
with the context of we like semver will be easier than we use semver.


Like Josh, my main concern is that we accurately advertise what we're
doing. There are few things I've found more frustrating than being an end
user of a project that claims to follow semver without understanding the
burden that carries (and subsequently not meeting it).

On Thu, Apr 23, 2015 at 3:48 PM, Stack st...@duboce.net wrote:

 I agree w/ the Enis characterization (so we need the callout on semvar) but
 think we should practice what Seans says (patch is bug fixes only).
 St.Ack

 On Thu, Apr 23, 2015 at 1:31 PM, Sean Busbey bus...@cloudera.com wrote:

  Why don't we just focus development after a minor release on the next
 minor
  release instead of the next patch release?
 
  We could limit backports to the patch releases to critical bugs, which
  would cut down on how often someone has to deal with the pain of making
  sure we don't add to public APIs. It also reduces the risk someone going
  through an upgrade has, since there are fewer changes.
 
  If someone fixes a bug and doesn't want to do the work of making sure it
  doesn't add methods in a patch release, they just don't backport to that
  version and make a follow on e.g. backport to 1.0.z ticket.
 
 
 
  On Thu, Apr 23, 2015 at 1:50 PM, Enis Söztutar enis@gmail.com
 wrote:
 
   +1 to the proposal.
  
   The problem is that we have a very big API surface especially with the
   coprocessors included in the report. Even simple bug fixes can
 introduce
   protected or public methods to base classes, which makes patch releases
   very hard to maintain. I would not want to spend the effort to spend
 tons
   of time trying to make a patch not introduce new methods in order to
   backport. That effort can be spent elsewhere IMO.
  
   Looking at the report
   https://people.apache.org/~enis/1.0.0_1.0.1RC2_compat_report.html,
  nothing
   strikes me as new functionality. Going from current 1.0.0 to 1.0.1RC2
   should actually be as you would expect from upgrading a patch release.
  
   Yes, adding new API in patch releases will make downgrading harder,
 but I
   think that is an acceptable tradeoff. We can document that if your
   application compiles (meaning that you are not using new API) with
 1.0.0,
   then you can swap your jars in a binary compat manner.
  
   Enis
  
   On Thu, Apr 23, 2015 at 10:03 AM, Andrew Purtell apurt...@apache.org
   wrote:
  
Anyone disagree with the point of view put forward by Josh and Sean?
   
   
On Wed, Apr 22, 2015 at 10:35 AM, Josh Elser josh.el...@gmail.com
   wrote:
   
 Andy -- I understood your intent, but thanks for clarifying. (as
 well
   as
 taking the time to break this discussion out in the first place). I
   agree
 with your assessment.

 re: Sean's comments, if it wasn't clear by me asking in the first
   place,
I
 also think sticking as close as possible to semver's rules is the
  best
 approach, although I'm getting the impression that there have been
  some
 previous reservations to doing so (especially by your comment about
 backporting features if there is demand is).

 I've found adhering to the bug-fix release restrictions can be a
 very
 painful and time-consuming task, so this is something to get a
 representative sampling of those who do the work to make sure
  everyone
   is
 on board.


 Sean Busbey wrote:

 I'd much rather we stick with the definitions used in Semantic
Versioning.
 Our use is already confusing enough given our matrix of
   compatibilities
 that don't get major version for breaking protections.

 We've previously discussed how we'll do additional minor releases
  when
 there's sufficient interest in the new features present there.
  What's
 building that demand if any backwards compatible change can go
 back
into a
 patch release?

 Would we have an easier time restraining ourselves if we had a
  regular
 schedule planned around new minor versions?


 On Wed, Apr 22, 2015 at 12:03 PM, Josh Elserjosh.el...@gmail.com
 
 wrote:

  While I can understand the desire to want to add things, I do
 think
   it
 makes things harder for users to 

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Enis Söztutar]

On Thu, Apr 23, 2015 at 1:31 PM, Sean Busbey bus...@cloudera.com wrote:

 Why don't we just focus development after a minor release on the next minor
 release instead of the next patch release?

 We could limit backports to the patch releases to critical bugs, which
 would cut down on how often someone has to deal with the pain of making
 sure we don't add to public APIs. It also reduces the risk someone going
 through an upgrade has, since there are fewer changes.


Big +1. I think we will get there, but devs/committers are used to
backporting simple
new features across all branches. Not to nit pick or anything, an example
is
HBASE-13168 which I failed to catch. Ideally it should not have been in
1.0.1. We also
do it, so that users see less of This was in 0.98, but how come it is not
in 1.0.x. I think
we will get substantially better over time.



 If someone fixes a bug and doesn't want to do the work of making sure it
 doesn't add methods in a patch release, they just don't backport to that
 version and make a follow on e.g. backport to 1.0.z ticket.



 On Thu, Apr 23, 2015 at 1:50 PM, Enis Söztutar enis@gmail.com wrote:

  +1 to the proposal.
 
  The problem is that we have a very big API surface especially with the
  coprocessors included in the report. Even simple bug fixes can introduce
  protected or public methods to base classes, which makes patch releases
  very hard to maintain. I would not want to spend the effort to spend tons
  of time trying to make a patch not introduce new methods in order to
  backport. That effort can be spent elsewhere IMO.
 
  Looking at the report
  https://people.apache.org/~enis/1.0.0_1.0.1RC2_compat_report.html,
 nothing
  strikes me as new functionality. Going from current 1.0.0 to 1.0.1RC2
  should actually be as you would expect from upgrading a patch release.
 
  Yes, adding new API in patch releases will make downgrading harder, but I
  think that is an acceptable tradeoff. We can document that if your
  application compiles (meaning that you are not using new API) with 1.0.0,
  then you can swap your jars in a binary compat manner.
 
  Enis
 
  On Thu, Apr 23, 2015 at 10:03 AM, Andrew Purtell apurt...@apache.org
  wrote:
 
   Anyone disagree with the point of view put forward by Josh and Sean?
  
  
   On Wed, Apr 22, 2015 at 10:35 AM, Josh Elser josh.el...@gmail.com
  wrote:
  
Andy -- I understood your intent, but thanks for clarifying. (as well
  as
taking the time to break this discussion out in the first place). I
  agree
with your assessment.
   
re: Sean's comments, if it wasn't clear by me asking in the first
  place,
   I
also think sticking as close as possible to semver's rules is the
 best
approach, although I'm getting the impression that there have been
 some
previous reservations to doing so (especially by your comment about
backporting features if there is demand is).
   
I've found adhering to the bug-fix release restrictions can be a very
painful and time-consuming task, so this is something to get a
representative sampling of those who do the work to make sure
 everyone
  is
on board.
   
   
Sean Busbey wrote:
   
I'd much rather we stick with the definitions used in Semantic
   Versioning.
Our use is already confusing enough given our matrix of
  compatibilities
that don't get major version for breaking protections.
   
We've previously discussed how we'll do additional minor releases
 when
there's sufficient interest in the new features present there.
 What's
building that demand if any backwards compatible change can go back
   into a
patch release?
   
Would we have an easier time restraining ourselves if we had a
 regular
schedule planned around new minor versions?
   
   
On Wed, Apr 22, 2015 at 12:03 PM, Josh Elserjosh.el...@gmail.com
wrote:
   
 While I can understand the desire to want to add things, I do think
  it
makes things harder for users to reliably write code against
 versions
   of
HBase which (by their view) should be completely compatible with
 one
another.
   
Take this extremely hypothetical situation: I'm new to HBase and
  start
writing some code against HBase 1.0.1 which was just deployed at my
$job. I
don't _know_ what APIs are new, I just know what exists and treat
  that
   as
acceptable for me to be using. Meanwhile in production, some other
   people
find a bug with HBase 1.0.1 and roll back to 1.0.0 which they had
  been
previously using. My reaction would be of course my code should
 work
with
HBase 1.0.0, I only used the public API when in fact this is not
  true.
   
Personally, I think it's a little bold to say semver is even in use
  if
this principal isn't being followed as it doesn't follow at all
 with
  my
understanding on the guarantees defined by semver for bug-fix
  releases.
   
That being said, if the intent *is* to allow 

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Stack]

On Thu, Apr 23, 2015 at 3:08 PM, Andrew Purtell apurt...@apache.org wrote:

 So we like semver, not we use semver?


and Sean's

 No longer claiming semver would also have the added benefit of making it for
me to easier to explain our compatibility promises to people.

Yeah, 'we like semvar'/'we are almost semvar' + and are actively working
toward 'semvar' would give us leeway that we seem to need as we learn what
semvar implies.

Does this 'admission' help with the which-hadoop thread too?

St.Ack




 Just want to make sure we're all on the same page (or find out if not).

 On Thu, Apr 23, 2015 at 2:59 PM, Enis Söztutar e...@apache.org wrote:

  Then let's get Andrew's proposal committed:
 
  + APIs available in a patch version will be available in all later
  + patch versions. However, new APIs may be added which will not be
  + available in earlier patch versions.
 
  I propose the following change to the Client Binary compatibility
 section
  of Section 11.1:
 
  - Old client code can run unchanged (no recompilation needed) against new
  jars.
  + Client code written to APIs available in a given patch release
  + can run unchanged (no recompilation needed) against the new
  + jars of later patch versions.
 
  We can even claim that if you are not using new APIs, we are binary
 compat
  for downgrade. But you have to make sure that your code compiles with
 that
  downgraded version.
 
  Enis
 
 
 
  On Thu, Apr 23, 2015 at 2:04 PM, Sean Busbey bus...@cloudera.com
 wrote:
 
   I'm fine with us adding methods to public APIs in patch releases so
 long
  as
   we stop claiming to follow semver. We can say we take the principles of
   semver as inspiration. This would reflect the current state of the
 world
   WRT 1.0.1 and would still give us a reason keep the narrower definition
  of
   new features in minor versions.
  
  
   No longer claiming semver would also have the added benefit of making
 it
   for me to easier to explain our compatibility promises to people. Right
  now
   I have to explain the difference between the things that get proper
  semver
   treatment (e.g. public api, wire compat) and which things are
 downgraded
  to
   breaking-on-minor (e.g. LimitedPrivate, binary compatibility). Starting
   with the context of we like semver will be easier than we use
 semver.
  
  
   Like Josh, my main concern is that we accurately advertise what we're
   doing. There are few things I've found more frustrating than being an
 end
   user of a project that claims to follow semver without understanding
 the
   burden that carries (and subsequently not meeting it).
  
   On Thu, Apr 23, 2015 at 3:48 PM, Stack st...@duboce.net wrote:
  
I agree w/ the Enis characterization (so we need the callout on
 semvar)
   but
think we should practice what Seans says (patch is bug fixes only).
St.Ack
   
On Thu, Apr 23, 2015 at 1:31 PM, Sean Busbey bus...@cloudera.com
   wrote:
   
 Why don't we just focus development after a minor release on the
 next
minor
 release instead of the next patch release?

 We could limit backports to the patch releases to critical bugs,
  which
 would cut down on how often someone has to deal with the pain of
  making
 sure we don't add to public APIs. It also reduces the risk someone
   going
 through an upgrade has, since there are fewer changes.

 If someone fixes a bug and doesn't want to do the work of making
 sure
   it
 doesn't add methods in a patch release, they just don't backport to
   that
 version and make a follow on e.g. backport to 1.0.z ticket.



 On Thu, Apr 23, 2015 at 1:50 PM, Enis Söztutar enis@gmail.com
 
wrote:

  +1 to the proposal.
 
  The problem is that we have a very big API surface especially
 with
   the
  coprocessors included in the report. Even simple bug fixes can
introduce
  protected or public methods to base classes, which makes patch
   releases
  very hard to maintain. I would not want to spend the effort to
  spend
tons
  of time trying to make a patch not introduce new methods in order
  to
  backport. That effort can be spent elsewhere IMO.
 
  Looking at the report
 
 https://people.apache.org/~enis/1.0.0_1.0.1RC2_compat_report.html,
 nothing
  strikes me as new functionality. Going from current 1.0.0 to
   1.0.1RC2
  should actually be as you would expect from upgrading a patch
   release.
 
  Yes, adding new API in patch releases will make downgrading
 harder,
but I
  think that is an acceptable tradeoff. We can document that if
 your
  application compiles (meaning that you are not using new API)
 with
1.0.0,
  then you can swap your jars in a binary compat manner.
 
  Enis
 
  On Thu, Apr 23, 2015 at 10:03 AM, Andrew Purtell 
   apurt...@apache.org
  wrote:
 
   Anyone disagree with the point of view put forward by Josh and
   

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Josh Elser]

Stack wrote:

On Thu, Apr 23, 2015 at 3:08 PM, Andrew Purtellapurt...@apache.org  wrote:


  So we like semver, not we use semver?



and Sean's


  No longer claiming semver would also have the added benefit of making it for

me to easier to explain our compatibility promises to people.

Yeah, 'we like semvar'/'we are almost semvar' + and are actively working
toward 'semvar' would give us leeway that we seem to need as we learn what
semvar implies.

Does this 'admission' help with the which-hadoop thread too?

St.Ack



+1

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Ted Yu]

+1 as well. 



 On Apr 23, 2015, at 4:36 PM, Josh Elser josh.el...@gmail.com wrote:
 
 Stack wrote:
 On Thu, Apr 23, 2015 at 3:08 PM, Andrew Purtellapurt...@apache.org  wrote:
 
   So we like semver, not we use semver?
 
 
 and Sean's
 
   No longer claiming semver would also have the added benefit of making it 
  for
 me to easier to explain our compatibility promises to people.
 
 Yeah, 'we like semvar'/'we are almost semvar' + and are actively working
 toward 'semvar' would give us leeway that we seem to need as we learn what
 semvar implies.
 
 Does this 'admission' help with the which-hadoop thread too?
 
 St.Ack
 
 +1

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Nick Dimiduk]

On Thu, Apr 23, 2015 at 4:13 PM, Stack st...@duboce.net wrote:

 Does this 'admission' help with the which-hadoop thread too?


Right -- working toward semver.

Are we now at liberty to bump to 2.6 or 2.7 even for branch-1.1? I still
have Karthik's offer to roll a 2.5.3 with the HDFS issue resolved.

What about the jackson issue with 2.5 YARN runtime?

Thanks Sean and Josh for being our community conscience on these issues.

 Just want to make sure we're all on the same page (or find out if not).
 
  On Thu, Apr 23, 2015 at 2:59 PM, Enis Söztutar e...@apache.org wrote:
 
   Then let's get Andrew's proposal committed:
  
   + APIs available in a patch version will be available in all later
   + patch versions. However, new APIs may be added which will not be
   + available in earlier patch versions.
  
   I propose the following change to the Client Binary compatibility
  section
   of Section 11.1:
  
   - Old client code can run unchanged (no recompilation needed) against
 new
   jars.
   + Client code written to APIs available in a given patch release
   + can run unchanged (no recompilation needed) against the new
   + jars of later patch versions.
  
   We can even claim that if you are not using new APIs, we are binary
  compat
   for downgrade. But you have to make sure that your code compiles with
  that
   downgraded version.
  
   Enis
  
  
  
   On Thu, Apr 23, 2015 at 2:04 PM, Sean Busbey bus...@cloudera.com
  wrote:
  
I'm fine with us adding methods to public APIs in patch releases so
  long
   as
we stop claiming to follow semver. We can say we take the principles
 of
semver as inspiration. This would reflect the current state of the
  world
WRT 1.0.1 and would still give us a reason keep the narrower
 definition
   of
new features in minor versions.
   
   
No longer claiming semver would also have the added benefit of making
  it
for me to easier to explain our compatibility promises to people.
 Right
   now
I have to explain the difference between the things that get proper
   semver
treatment (e.g. public api, wire compat) and which things are
  downgraded
   to
breaking-on-minor (e.g. LimitedPrivate, binary compatibility).
 Starting
with the context of we like semver will be easier than we use
  semver.
   
   
Like Josh, my main concern is that we accurately advertise what we're
doing. There are few things I've found more frustrating than being an
  end
user of a project that claims to follow semver without understanding
  the
burden that carries (and subsequently not meeting it).
   
On Thu, Apr 23, 2015 at 3:48 PM, Stack st...@duboce.net wrote:
   
 I agree w/ the Enis characterization (so we need the callout on
  semvar)
but
 think we should practice what Seans says (patch is bug fixes only).
 St.Ack

 On Thu, Apr 23, 2015 at 1:31 PM, Sean Busbey bus...@cloudera.com
wrote:

  Why don't we just focus development after a minor release on the
  next
 minor
  release instead of the next patch release?
 
  We could limit backports to the patch releases to critical bugs,
   which
  would cut down on how often someone has to deal with the pain of
   making
  sure we don't add to public APIs. It also reduces the risk
 someone
going
  through an upgrade has, since there are fewer changes.
 
  If someone fixes a bug and doesn't want to do the work of making
  sure
it
  doesn't add methods in a patch release, they just don't backport
 to
that
  version and make a follow on e.g. backport to 1.0.z ticket.
 
 
 
  On Thu, Apr 23, 2015 at 1:50 PM, Enis Söztutar 
 enis@gmail.com
  
 wrote:
 
   +1 to the proposal.
  
   The problem is that we have a very big API surface especially
  with
the
   coprocessors included in the report. Even simple bug fixes can
 introduce
   protected or public methods to base classes, which makes patch
releases
   very hard to maintain. I would not want to spend the effort to
   spend
 tons
   of time trying to make a patch not introduce new methods in
 order
   to
   backport. That effort can be spent elsewhere IMO.
  
   Looking at the report
  
  https://people.apache.org/~enis/1.0.0_1.0.1RC2_compat_report.html,
  nothing
   strikes me as new functionality. Going from current 1.0.0 to
1.0.1RC2
   should actually be as you would expect from upgrading a patch
release.
  
   Yes, adding new API in patch releases will make downgrading
  harder,
 but I
   think that is an acceptable tradeoff. We can document that if
  your
   application compiles (meaning that you are not using new API)
  with
 1.0.0,
   then you can swap your jars in a binary compat manner.
  
   Enis
  
   On Thu, Apr 23, 2015 at 10:03 AM, Andrew Purtell 
apurt...@apache.org
   wrote:
  

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Sean Busbey]

On Thu, Apr 23, 2015 at 5:08 PM, Andrew Purtell apurt...@apache.org wrote:

 So we like semver, not we use semver?

 Just want to make sure we're all on the same page (or find out if not).


+1 from me.

I'd prefer we not make the statements about  binary compat for downgrading
because it's complicated and basically boils down to we don't make
promises about downgrading.

-- 
Sean

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Enis Söztutar]

+1 to working toward semver. Is there any documentation that we would
like to clarify in the book given this enlightenment?

I would be in favor of going 2.6.0 and jackson upgrade in 1.1.

Enis

On Thu, Apr 23, 2015 at 5:34 PM, Nick Dimiduk ndimi...@gmail.com wrote:

 On Thu, Apr 23, 2015 at 4:13 PM, Stack st...@duboce.net wrote:

  Does this 'admission' help with the which-hadoop thread too?
 

 Right -- working toward semver.

 Are we now at liberty to bump to 2.6 or 2.7 even for branch-1.1? I still
 have Karthik's offer to roll a 2.5.3 with the HDFS issue resolved.

 What about the jackson issue with 2.5 YARN runtime?

 Thanks Sean and Josh for being our community conscience on these issues.

  Just want to make sure we're all on the same page (or find out if not).
  
   On Thu, Apr 23, 2015 at 2:59 PM, Enis Söztutar e...@apache.org
 wrote:
  
Then let's get Andrew's proposal committed:
   
+ APIs available in a patch version will be available in all later
+ patch versions. However, new APIs may be added which will not be
+ available in earlier patch versions.
   
I propose the following change to the Client Binary compatibility
   section
of Section 11.1:
   
- Old client code can run unchanged (no recompilation needed) against
  new
jars.
+ Client code written to APIs available in a given patch release
+ can run unchanged (no recompilation needed) against the new
+ jars of later patch versions.
   
We can even claim that if you are not using new APIs, we are binary
   compat
for downgrade. But you have to make sure that your code compiles with
   that
downgraded version.
   
Enis
   
   
   
On Thu, Apr 23, 2015 at 2:04 PM, Sean Busbey bus...@cloudera.com
   wrote:
   
 I'm fine with us adding methods to public APIs in patch releases so
   long
as
 we stop claiming to follow semver. We can say we take the
 principles
  of
 semver as inspiration. This would reflect the current state of the
   world
 WRT 1.0.1 and would still give us a reason keep the narrower
  definition
of
 new features in minor versions.


 No longer claiming semver would also have the added benefit of
 making
   it
 for me to easier to explain our compatibility promises to people.
  Right
now
 I have to explain the difference between the things that get proper
semver
 treatment (e.g. public api, wire compat) and which things are
   downgraded
to
 breaking-on-minor (e.g. LimitedPrivate, binary compatibility).
  Starting
 with the context of we like semver will be easier than we use
   semver.


 Like Josh, my main concern is that we accurately advertise what
 we're
 doing. There are few things I've found more frustrating than being
 an
   end
 user of a project that claims to follow semver without
 understanding
   the
 burden that carries (and subsequently not meeting it).

 On Thu, Apr 23, 2015 at 3:48 PM, Stack st...@duboce.net wrote:

  I agree w/ the Enis characterization (so we need the callout on
   semvar)
 but
  think we should practice what Seans says (patch is bug fixes
 only).
  St.Ack
 
  On Thu, Apr 23, 2015 at 1:31 PM, Sean Busbey 
 bus...@cloudera.com
 wrote:
 
   Why don't we just focus development after a minor release on
 the
   next
  minor
   release instead of the next patch release?
  
   We could limit backports to the patch releases to critical
 bugs,
which
   would cut down on how often someone has to deal with the pain
 of
making
   sure we don't add to public APIs. It also reduces the risk
  someone
 going
   through an upgrade has, since there are fewer changes.
  
   If someone fixes a bug and doesn't want to do the work of
 making
   sure
 it
   doesn't add methods in a patch release, they just don't
 backport
  to
 that
   version and make a follow on e.g. backport to 1.0.z ticket.
  
  
  
   On Thu, Apr 23, 2015 at 1:50 PM, Enis Söztutar 
  enis@gmail.com
   
  wrote:
  
+1 to the proposal.
   
The problem is that we have a very big API surface especially
   with
 the
coprocessors included in the report. Even simple bug fixes
 can
  introduce
protected or public methods to base classes, which makes
 patch
 releases
very hard to maintain. I would not want to spend the effort
 to
spend
  tons
of time trying to make a patch not introduce new methods in
  order
to
backport. That effort can be spent elsewhere IMO.
   
Looking at the report
   
   https://people.apache.org/~enis/1.0.0_1.0.1RC2_compat_report.html,
   nothing
strikes me as new functionality. Going from current 1.0.0
 to
 1.0.1RC2
should actually be as you would expect from upgrading a patch
 release.
   

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Ted Yu]

I think these changes are Okay.

On Wed, Apr 22, 2015 at 9:29 AM, Andrew Purtell apurt...@apache.org wrote:

 [Subject changed]

 On Tue, Apr 21, 2015 at 8:47 PM, Josh Elser josh.el...@gmail.com wrote:

  I was a little surprised when I noticed method additions to
  InterfaceAudience.Public annotated classes. This means that a user could
  write code against 1.0.1 that would not work against 1.0.0 which seems
  undesirable for a bugfix release. I read over the book section on
  compatibility and didn't see this addressed, so I thought I'd ask.


 Let's clarify this. It's not the first time this question has been asked.

 To get things moving:

 I propose the following addition to the Client API compatibility section
 of Section 11.1:

 + APIs available in a patch version will be available in all later
 + patch versions. However, new APIs may be added which will not be
 + available in earlier patch versions.

 I propose the following change to the Client Binary compatibility section
 of Section 11.1:

 - Old client code can run unchanged (no recompilation needed) against new
 jars.
 + Client code written to APIs available in a given patch release
 + can run unchanged (no recompilation needed) against the new
 + jars of later patch versions.


 What do you think?

 If these changes are (mostly) ok, then this clarifies in one direction.

 If these changes are not acceptable, I will propose edits that clarify
 toward the opposite meaning. ​


 --
 Best regards,

- Andy

 Problems worthy of attack prove their worth by hitting back. - Piet Hein
 (via Tom White)

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Josh Elser]

While I can understand the desire to want to add things, I do think it 
makes things harder for users to reliably write code against versions of 
HBase which (by their view) should be completely compatible with one 
another.


Take this extremely hypothetical situation: I'm new to HBase and start 
writing some code against HBase 1.0.1 which was just deployed at my 
$job. I don't _know_ what APIs are new, I just know what exists and 
treat that as acceptable for me to be using. Meanwhile in production, 
some other people find a bug with HBase 1.0.1 and roll back to 1.0.0 
which they had been previously using. My reaction would be of course my 
code should work with HBase 1.0.0, I only used the public API when in 
fact this is not true.


Personally, I think it's a little bold to say semver is even in use if 
this principal isn't being followed as it doesn't follow at all with my 
understanding on the guarantees defined by semver for bug-fix releases.


That being said, if the intent *is* to allow ourselves to make these 
sorts of changes, I just think some sort of disclaimer should be present:


- HBase uses Semantic Versioning for its release versioning
+ HBase uses Semantic Versioning for its release versioning with a 
caveat that methods and members might be added in newer bug-fix releases 
that were not present in the previous bug-fix release.


Andrew Purtell wrote:

[Subject changed]

On Tue, Apr 21, 2015 at 8:47 PM, Josh Elserjosh.el...@gmail.com  wrote:


I was a little surprised when I noticed method additions to
InterfaceAudience.Public annotated classes. This means that a user could
write code against 1.0.1 that would not work against 1.0.0 which seems
undesirable for a bugfix release. I read over the book section on
compatibility and didn't see this addressed, so I thought I'd ask.



Let's clarify this. It's not the first time this question has been asked.

To get things moving:

I propose the following addition to the Client API compatibility section
of Section 11.1:

+ APIs available in a patch version will be available in all later
+ patch versions. However, new APIs may be added which will not be
+ available in earlier patch versions.

I propose the following change to the Client Binary compatibility section
of Section 11.1:

- Old client code can run unchanged (no recompilation needed) against new
jars.
+ Client code written to APIs available in a given patch release
+ can run unchanged (no recompilation needed) against the new
+ jars of later patch versions.


What do you think?

If these changes are (mostly) ok, then this clarifies in one direction.

If these changes are not acceptable, I will propose edits that clarify
toward the opposite meaning. ​

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Sean Busbey]

I'd much rather we stick with the definitions used in Semantic Versioning.
Our use is already confusing enough given our matrix of compatibilities
that don't get major version for breaking protections.

We've previously discussed how we'll do additional minor releases when
there's sufficient interest in the new features present there. What's
building that demand if any backwards compatible change can go back into a
patch release?

Would we have an easier time restraining ourselves if we had a regular
schedule planned around new minor versions?


On Wed, Apr 22, 2015 at 12:03 PM, Josh Elser josh.el...@gmail.com wrote:

 While I can understand the desire to want to add things, I do think it
 makes things harder for users to reliably write code against versions of
 HBase which (by their view) should be completely compatible with one
 another.

 Take this extremely hypothetical situation: I'm new to HBase and start
 writing some code against HBase 1.0.1 which was just deployed at my $job. I
 don't _know_ what APIs are new, I just know what exists and treat that as
 acceptable for me to be using. Meanwhile in production, some other people
 find a bug with HBase 1.0.1 and roll back to 1.0.0 which they had been
 previously using. My reaction would be of course my code should work with
 HBase 1.0.0, I only used the public API when in fact this is not true.

 Personally, I think it's a little bold to say semver is even in use if
 this principal isn't being followed as it doesn't follow at all with my
 understanding on the guarantees defined by semver for bug-fix releases.

 That being said, if the intent *is* to allow ourselves to make these sorts
 of changes, I just think some sort of disclaimer should be present:

 - HBase uses Semantic Versioning for its release versioning
 + HBase uses Semantic Versioning for its release versioning with a caveat
 that methods and members might be added in newer bug-fix releases that were
 not present in the previous bug-fix release.


 Andrew Purtell wrote:

 [Subject changed]

 On Tue, Apr 21, 2015 at 8:47 PM, Josh Elserjosh.el...@gmail.com  wrote:

  I was a little surprised when I noticed method additions to
 InterfaceAudience.Public annotated classes. This means that a user could
 write code against 1.0.1 that would not work against 1.0.0 which seems
 undesirable for a bugfix release. I read over the book section on
 compatibility and didn't see this addressed, so I thought I'd ask.



 Let's clarify this. It's not the first time this question has been asked.

 To get things moving:

 I propose the following addition to the Client API compatibility section
 of Section 11.1:

 + APIs available in a patch version will be available in all later
 + patch versions. However, new APIs may be added which will not be
 + available in earlier patch versions.

 I propose the following change to the Client Binary compatibility
 section
 of Section 11.1:

 - Old client code can run unchanged (no recompilation needed) against new
 jars.
 + Client code written to APIs available in a given patch release
 + can run unchanged (no recompilation needed) against the new
 + jars of later patch versions.


 What do you think?

 If these changes are (mostly) ok, then this clarifies in one direction.

 If these changes are not acceptable, I will propose edits that clarify
 toward the opposite meaning. ​





-- 
Sean

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Andrew Purtell]

Just to clarify something, I've proposed edits that clarify the de-facto
practice, since additional methods are turning up in patch releases, but am
not taking a position.

So far we don't have consensus.

 + HBase uses Semantic Versioning for its release versioning with a caveat
that methods and members might be added in newer bug-fix releases that were
not present in the previous bug-fix release.

This is a good point. We should make this change too, to be consistent, or
if we don't want to make this change, then we should:
- Reject my proposal (I will make a new one clarifying with the opposite
meaning)
- Sink the current 1.0.1 RC
- Remove any changes in branch-1.0 that added new methods to the API
surface before proposing a new 1.0.1 RC


On Wed, Apr 22, 2015 at 10:03 AM, Josh Elser josh.el...@gmail.com wrote:

 While I can understand the desire to want to add things, I do think it
 makes things harder for users to reliably write code against versions of
 HBase which (by their view) should be completely compatible with one
 another.

 Take this extremely hypothetical situation: I'm new to HBase and start
 writing some code against HBase 1.0.1 which was just deployed at my $job. I
 don't _know_ what APIs are new, I just know what exists and treat that as
 acceptable for me to be using. Meanwhile in production, some other people
 find a bug with HBase 1.0.1 and roll back to 1.0.0 which they had been
 previously using. My reaction would be of course my code should work with
 HBase 1.0.0, I only used the public API when in fact this is not true.

 Personally, I think it's a little bold to say semver is even in use if
 this principal isn't being followed as it doesn't follow at all with my
 understanding on the guarantees defined by semver for bug-fix releases.

 That being said, if the intent *is* to allow ourselves to make these sorts
 of changes, I just think some sort of disclaimer should be present:

 - HBase uses Semantic Versioning for its release versioning
 + HBase uses Semantic Versioning for its release versioning with a caveat
 that methods and members might be added in newer bug-fix releases that were
 not present in the previous bug-fix release.


 Andrew Purtell wrote:

 [Subject changed]

 On Tue, Apr 21, 2015 at 8:47 PM, Josh Elserjosh.el...@gmail.com  wrote:

  I was a little surprised when I noticed method additions to
 InterfaceAudience.Public annotated classes. This means that a user could
 write code against 1.0.1 that would not work against 1.0.0 which seems
 undesirable for a bugfix release. I read over the book section on
 compatibility and didn't see this addressed, so I thought I'd ask.



 Let's clarify this. It's not the first time this question has been asked.

 To get things moving:

 I propose the following addition to the Client API compatibility section
 of Section 11.1:

 + APIs available in a patch version will be available in all later
 + patch versions. However, new APIs may be added which will not be
 + available in earlier patch versions.

 I propose the following change to the Client Binary compatibility
 section
 of Section 11.1:

 - Old client code can run unchanged (no recompilation needed) against new
 jars.
 + Client code written to APIs available in a given patch release
 + can run unchanged (no recompilation needed) against the new
 + jars of later patch versions.


 What do you think?

 If these changes are (mostly) ok, then this clarifies in one direction.

 If these changes are not acceptable, I will propose edits that clarify
 toward the opposite meaning. ​





-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Re: Clarifying interface evolution freedom in patch releases (was: Re: [VOTE] Third release candidate for HBase 1.0.1 (RC2))

[Josh Elser]

Andy -- I understood your intent, but thanks for clarifying. (as well as 
taking the time to break this discussion out in the first place). I 
agree with your assessment.


re: Sean's comments, if it wasn't clear by me asking in the first place, 
I also think sticking as close as possible to semver's rules is the best 
approach, although I'm getting the impression that there have been some 
previous reservations to doing so (especially by your comment about 
backporting features if there is demand is).


I've found adhering to the bug-fix release restrictions can be a very 
painful and time-consuming task, so this is something to get a 
representative sampling of those who do the work to make sure everyone 
is on board.


Sean Busbey wrote:

I'd much rather we stick with the definitions used in Semantic Versioning.
Our use is already confusing enough given our matrix of compatibilities
that don't get major version for breaking protections.

We've previously discussed how we'll do additional minor releases when
there's sufficient interest in the new features present there. What's
building that demand if any backwards compatible change can go back into a
patch release?

Would we have an easier time restraining ourselves if we had a regular
schedule planned around new minor versions?


On Wed, Apr 22, 2015 at 12:03 PM, Josh Elserjosh.el...@gmail.com  wrote:


While I can understand the desire to want to add things, I do think it
makes things harder for users to reliably write code against versions of
HBase which (by their view) should be completely compatible with one
another.

Take this extremely hypothetical situation: I'm new to HBase and start
writing some code against HBase 1.0.1 which was just deployed at my $job. I
don't _know_ what APIs are new, I just know what exists and treat that as
acceptable for me to be using. Meanwhile in production, some other people
find a bug with HBase 1.0.1 and roll back to 1.0.0 which they had been
previously using. My reaction would be of course my code should work with
HBase 1.0.0, I only used the public API when in fact this is not true.

Personally, I think it's a little bold to say semver is even in use if
this principal isn't being followed as it doesn't follow at all with my
understanding on the guarantees defined by semver for bug-fix releases.

That being said, if the intent *is* to allow ourselves to make these sorts
of changes, I just think some sort of disclaimer should be present:

- HBase uses Semantic Versioning for its release versioning
+ HBase uses Semantic Versioning for its release versioning with a caveat
that methods and members might be added in newer bug-fix releases that were
not present in the previous bug-fix release.


Andrew Purtell wrote:


[Subject changed]

On Tue, Apr 21, 2015 at 8:47 PM, Josh Elserjosh.el...@gmail.com   wrote:

  I was a little surprised when I noticed method additions to

InterfaceAudience.Public annotated classes. This means that a user could
write code against 1.0.1 that would not work against 1.0.0 which seems
undesirable for a bugfix release. I read over the book section on
compatibility and didn't see this addressed, so I thought I'd ask.



Let's clarify this. It's not the first time this question has been asked.

To get things moving:

I propose the following addition to the Client API compatibility section
of Section 11.1:

+ APIs available in a patch version will be available in all later
+ patch versions. However, new APIs may be added which will not be
+ available in earlier patch versions.

I propose the following change to the Client Binary compatibility
section
of Section 11.1:

- Old client code can run unchanged (no recompilation needed) against new
jars.
+ Client code written to APIs available in a given patch release
+ can run unchanged (no recompilation needed) against the new
+ jars of later patch versions.


What do you think?

If these changes are (mostly) ok, then this clarifies in one direction.

If these changes are not acceptable, I will propose edits that clarify
toward the opposite meaning. ​