Re: [ceph-users] Ceph release cadence

[Joao Eduardo Luis]

I am happy with this branch of the thread!

I'm guessing this would start post-Mimic though, if no one objects and 
if we want to target a March release?


  -Joao

On 09/23/2017 02:58 AM, Sage Weil wrote:

On Fri, 22 Sep 2017, Gregory Farnum wrote:

On Fri, Sep 22, 2017 at 3:28 PM, Sage Weil  wrote:

Here is a concrete proposal for everyone to summarily shoot down (or
heartily endorse, depending on how your friday is going):

- 9 month cycle
- enforce a predictable release schedule with a freeze date and
   a release date.  (The actual .0 release of course depends on no blocker
   bugs being open; not sure how zealous 'train' style projects do
   this.)


Train projects basically commit to a feature freeze enough in advance
of the expected release date that it's feasible, and don't let people
fake it by rushing in stuff they "finished" the day before. I'm not
sure if every-9-month LTSes will be more conducive to that or not — if
we do scheduled releases, we still fundamentally need to be able to
say "nope, that feature we've been saying for 9 months we hope to have
out in this LTS won't make it until the next one". And we seem pretty
bad at that.


I'll be the first to say I'm no small part of the "we" there.  But I'm
also suggesting that's not a reason not to try to do better.  As I
said I think this will be easier than in the past because we don't
have as many headline features we're trying to wedge in.

In any case, is there an alternative way to get to the much-desired
regular cadence?


- no more even/odd pattern; all stable releases are created equal.
- support upgrades from up to 3 releases back.

This shortens the cycle a bit to relieve the "this feature must go in"
stress, without making it so short as to make the release pointless (e.g.,
infernalis, kraken).  (I also think that the feature pressure is much
lower now than it has been in the past.)

This creates more work for the developers because there are more upgrade
paths to consider: we no longer have strict "choke points" (like all
upgrades must go through luminous).  We could reserve the option to pick
specific choke point releases in the future, perhaps taking care to make
sure these are the releases that go into downstream distros.  We'll need
to be more systematic about the upgrade testing.


This sounds generally good to me — we did multiple-release upgrades
for a long time, and stuff is probably more complicated now but I
don't think it will actually be that big a deal.

3 releases back might be a bit much though — that's 27 months! (For
luminous, the beginning of 2015. Hammer.)


I'm *much* happier with 2 :) so no complaint from me.  I just heard a lot
of "2 years" and 2 releases (18 months) doesn't quite cover it.  Maybe
it's best to start with that, though?  It's still an improvement over the
current ~12 months.


Somewhat separately, several people expressed concern about having stable
releases to develop against.  This is somewhat orthogonal to what users
need.  To that end, we can do a dev checkpoint every 1 or 2 months
(preferences?), where we fork a 'next' branch and stabilize all of the
tests before moving on.  This is good practice anyway to avoid
accumulating low-frequency failures in the test suite that have to be
squashed at the end.


So this sounds like a fine idea to me, but how do we distinguish this
from the intermediate stable releases?

By which I mean, are we *really* going to do a stabilization branch
that will never get seen by users? What kind of testing and bug fixing
are we going to commit to doing against it, and how do we balance that
effort with feature work?

It seems like the same conflict we have now, only since the dev
checkpoints are less important they'll lose more often. Then we'll end
up having 9 months of scheduled work to debug for a user release
instead of 5 months that slipped to 7 or 8...


What if we frame this stabilization period in terms of stability of the
test suite.  That gives us something concrete to aim for, lets us move on
when we reach some threshold, and aligns perfectly with the thing that
makes it hard to safely land new code (noisy test results)...

sage



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release cadence

[Brady Deetz]

I'll be first to admit that most of my comments are anecdotal. But, I
suspect when it comes to storage many of us don't require a lot to get
scared back into our dark corners. In short it seems that the dev team
should get better at selecting features and delivering on the existing
scheduled cadence before shortening it. To me, the odd releases represent
feature previews for the next even release. If that's a fair way to look at
them, they could play a very important role in the stability of the even
release.

On Sep 22, 2017 8:59 PM, "Sage Weil"  wrote:

On Fri, 22 Sep 2017, Gregory Farnum wrote:
> On Fri, Sep 22, 2017 at 3:28 PM, Sage Weil  wrote:
> > Here is a concrete proposal for everyone to summarily shoot down (or
> > heartily endorse, depending on how your friday is going):
> >
> > - 9 month cycle
> > - enforce a predictable release schedule with a freeze date and
> >   a release date.  (The actual .0 release of course depends on no
blocker
> >   bugs being open; not sure how zealous 'train' style projects do
> >   this.)
>
> Train projects basically commit to a feature freeze enough in advance
> of the expected release date that it's feasible, and don't let people
> fake it by rushing in stuff they "finished" the day before. I'm not
> sure if every-9-month LTSes will be more conducive to that or not — if
> we do scheduled releases, we still fundamentally need to be able to
> say "nope, that feature we've been saying for 9 months we hope to have
> out in this LTS won't make it until the next one". And we seem pretty
> bad at that.

I'll be the first to say I'm no small part of the "we" there.  But I'm
also suggesting that's not a reason not to try to do better.  As I
said I think this will be easier than in the past because we don't
have as many headline features we're trying to wedge in.


That's excellent as long as it actually happens. Otherwise the collective
you may end up pushing worse code on a 9mo cycle than the current
theoretical 12mo cycle that is delayed when necessary. We all know that
software development never happens on time or on budget.


In any case, is there an alternative way to get to the much-desired
regular cadence?

> > - no more even/odd pattern; all stable releases are created equal.
> > - support upgrades from up to 3 releases back.
> >
> > This shortens the cycle a bit to relieve the "this feature must go in"
> > stress, without making it so short as to make the release pointless
(e.g.,
> > infernalis, kraken).  (I also think that the feature pressure is much
> > lower now than it has been in the past.)
> >
> > This creates more work for the developers because there are more upgrade
> > paths to consider: we no longer have strict "choke points" (like all
> > upgrades must go through luminous).  We could reserve the option to pick
> > specific choke point releases in the future, perhaps taking care to make
> > sure these are the releases that go into downstream distros.  We'll need
> > to be more systematic about the upgrade testing.
>
> This sounds generally good to me — we did multiple-release upgrades
> for a long time, and stuff is probably more complicated now but I
> don't think it will actually be that big a deal.
>
> 3 releases back might be a bit much though — that's 27 months! (For
> luminous, the beginning of 2015. Hammer.)

I'm *much* happier with 2 :) so no complaint from me.  I just heard a lot
of "2 years" and 2 releases (18 months) doesn't quite cover it.  Maybe
it's best to start with that, though?  It's still an improvement over the
current ~12 months.


A lot of vulnerabilities and bugs can come out in one year. As such, I
upgrade anything in my environment, at minimum, once a year. The "if it
ain't broke don't fix it" mentality is usually more dangerous than an
upgrade between minor releases. But... I will say that as my Ceph
environment grows, upgrades become increasingly difficult to manage and
anxiety increases with every node I add to my growing 2PB cluster.


> > Somewhat separately, several people expressed concern about having
stable
> > releases to develop against.  This is somewhat orthogonal to what users
> > need.  To that end, we can do a dev checkpoint every 1 or 2 months
> > (preferences?), where we fork a 'next' branch and stabilize all of the
> > tests before moving on.  This is good practice anyway to avoid
> > accumulating low-frequency failures in the test suite that have to be
> > squashed at the end.
>
> So this sounds like a fine idea to me, but how do we distinguish this
> from the intermediate stable releases?
>
> By which I mean, are we *really* going to do a stabilization branch
> that will never get seen by users? What kind of testing and bug fixing
> are we going to commit to doing against it, and how do we balance that
> effort with feature work?
>
> It seems like the same conflict we have now, only since the dev
> checkpoints are less important they'll lose more often. Then we'll end

Re: [ceph-users] Ceph release cadence

[Blair Bethwaite]

On 23 September 2017 at 11:58, Sage Weil  wrote:
> I'm *much* happier with 2 :) so no complaint from me.  I just heard a lot
> of "2 years" and 2 releases (18 months) doesn't quite cover it.  Maybe
> it's best to start with that, though?  It's still an improvement over the
> current ~12 months.

FWIW, I'd be happy with this if it means higher confidence in an
18-monthly skip-upgrade being a feasible operational approach.

-- 
Cheers,
~Blairo
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release cadence

[Sage Weil]

On Fri, 22 Sep 2017, Gregory Farnum wrote:
> On Fri, Sep 22, 2017 at 3:28 PM, Sage Weil  wrote:
> > Here is a concrete proposal for everyone to summarily shoot down (or
> > heartily endorse, depending on how your friday is going):
> >
> > - 9 month cycle
> > - enforce a predictable release schedule with a freeze date and
> >   a release date.  (The actual .0 release of course depends on no blocker
> >   bugs being open; not sure how zealous 'train' style projects do
> >   this.)
> 
> Train projects basically commit to a feature freeze enough in advance
> of the expected release date that it's feasible, and don't let people
> fake it by rushing in stuff they "finished" the day before. I'm not
> sure if every-9-month LTSes will be more conducive to that or not — if
> we do scheduled releases, we still fundamentally need to be able to
> say "nope, that feature we've been saying for 9 months we hope to have
> out in this LTS won't make it until the next one". And we seem pretty
> bad at that.

I'll be the first to say I'm no small part of the "we" there.  But I'm 
also suggesting that's not a reason not to try to do better.  As I 
said I think this will be easier than in the past because we don't 
have as many headline features we're trying to wedge in.

In any case, is there an alternative way to get to the much-desired 
regular cadence?

> > - no more even/odd pattern; all stable releases are created equal.
> > - support upgrades from up to 3 releases back.
> >
> > This shortens the cycle a bit to relieve the "this feature must go in"
> > stress, without making it so short as to make the release pointless (e.g.,
> > infernalis, kraken).  (I also think that the feature pressure is much
> > lower now than it has been in the past.)
> >
> > This creates more work for the developers because there are more upgrade
> > paths to consider: we no longer have strict "choke points" (like all
> > upgrades must go through luminous).  We could reserve the option to pick
> > specific choke point releases in the future, perhaps taking care to make
> > sure these are the releases that go into downstream distros.  We'll need
> > to be more systematic about the upgrade testing.
> 
> This sounds generally good to me — we did multiple-release upgrades
> for a long time, and stuff is probably more complicated now but I
> don't think it will actually be that big a deal.
> 
> 3 releases back might be a bit much though — that's 27 months! (For
> luminous, the beginning of 2015. Hammer.)

I'm *much* happier with 2 :) so no complaint from me.  I just heard a lot 
of "2 years" and 2 releases (18 months) doesn't quite cover it.  Maybe 
it's best to start with that, though?  It's still an improvement over the 
current ~12 months.

> > Somewhat separately, several people expressed concern about having stable
> > releases to develop against.  This is somewhat orthogonal to what users
> > need.  To that end, we can do a dev checkpoint every 1 or 2 months
> > (preferences?), where we fork a 'next' branch and stabilize all of the
> > tests before moving on.  This is good practice anyway to avoid
> > accumulating low-frequency failures in the test suite that have to be
> > squashed at the end.
> 
> So this sounds like a fine idea to me, but how do we distinguish this
> from the intermediate stable releases?
> 
> By which I mean, are we *really* going to do a stabilization branch
> that will never get seen by users? What kind of testing and bug fixing
> are we going to commit to doing against it, and how do we balance that
> effort with feature work?
> 
> It seems like the same conflict we have now, only since the dev
> checkpoints are less important they'll lose more often. Then we'll end
> up having 9 months of scheduled work to debug for a user release
> instead of 5 months that slipped to 7 or 8...

What if we frame this stabilization period in terms of stability of the 
test suite.  That gives us something concrete to aim for, lets us move on 
when we reach some threshold, and aligns perfectly with the thing that 
makes it hard to safely land new code (noisy test results)...

sage___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release cadence

[Gregory Farnum]

On Fri, Sep 22, 2017 at 3:28 PM, Sage Weil  wrote:
> Here is a concrete proposal for everyone to summarily shoot down (or
> heartily endorse, depending on how your friday is going):
>
> - 9 month cycle
> - enforce a predictable release schedule with a freeze date and
>   a release date.  (The actual .0 release of course depends on no blocker
>   bugs being open; not sure how zealous 'train' style projects do
>   this.)

Train projects basically commit to a feature freeze enough in advance
of the expected release date that it's feasible, and don't let people
fake it by rushing in stuff they "finished" the day before. I'm not
sure if every-9-month LTSes will be more conducive to that or not — if
we do scheduled releases, we still fundamentally need to be able to
say "nope, that feature we've been saying for 9 months we hope to have
out in this LTS won't make it until the next one". And we seem pretty
bad at that.

> - no more even/odd pattern; all stable releases are created equal.
> - support upgrades from up to 3 releases back.
>
> This shortens the cycle a bit to relieve the "this feature must go in"
> stress, without making it so short as to make the release pointless (e.g.,
> infernalis, kraken).  (I also think that the feature pressure is much
> lower now than it has been in the past.)
>
> This creates more work for the developers because there are more upgrade
> paths to consider: we no longer have strict "choke points" (like all
> upgrades must go through luminous).  We could reserve the option to pick
> specific choke point releases in the future, perhaps taking care to make
> sure these are the releases that go into downstream distros.  We'll need
> to be more systematic about the upgrade testing.

This sounds generally good to me — we did multiple-release upgrades
for a long time, and stuff is probably more complicated now but I
don't think it will actually be that big a deal.

3 releases back might be a bit much though — that's 27 months! (For
luminous, the beginning of 2015. Hammer.)

> Somewhat separately, several people expressed concern about having stable
> releases to develop against.  This is somewhat orthogonal to what users
> need.  To that end, we can do a dev checkpoint every 1 or 2 months
> (preferences?), where we fork a 'next' branch and stabilize all of the
> tests before moving on.  This is good practice anyway to avoid
> accumulating low-frequency failures in the test suite that have to be
> squashed at the end.

So this sounds like a fine idea to me, but how do we distinguish this
from the intermediate stable releases?

By which I mean, are we *really* going to do a stabilization branch
that will never get seen by users? What kind of testing and bug fixing
are we going to commit to doing against it, and how do we balance that
effort with feature work?

It seems like the same conflict we have now, only since the dev
checkpoints are less important they'll lose more often. Then we'll end
up having 9 months of scheduled work to debug for a user release
instead of 5 months that slipped to 7 or 8...

-Greg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release cadence

[Sage Weil]

Here is a concrete proposal for everyone to summarily shoot down (or 
heartily endorse, depending on how your friday is going):

- 9 month cycle
- enforce a predictable release schedule with a freeze date and 
  a release date.  (The actual .0 release of course depends on no blocker 
  bugs being open; not sure how zealous 'train' style projects do 
  this.)
- no more even/odd pattern; all stable releases are created equal.
- support upgrades from up to 3 releases back.

This shortens the cycle a bit to relieve the "this feature must go in" 
stress, without making it so short as to make the release pointless (e.g., 
infernalis, kraken).  (I also think that the feature pressure is much 
lower now than it has been in the past.)

This creates more work for the developers because there are more upgrade 
paths to consider: we no longer have strict "choke points" (like all 
upgrades must go through luminous).  We could reserve the option to pick 
specific choke point releases in the future, perhaps taking care to make 
sure these are the releases that go into downstream distros.  We'll need 
to be more systematic about the upgrade testing.

Somewhat separately, several people expressed concern about having stable 
releases to develop against.  This is somewhat orthogonal to what users 
need.  To that end, we can do a dev checkpoint every 1 or 2 months 
(preferences?), where we fork a 'next' branch and stabilize all of the 
tests before moving on.  This is good practice anyway to avoid 
accumulating low-frequency failures in the test suite that have to be 
squashed at the end.

The timing of the 9 mo cycle could be chosen to strategically avoid 
holiday periods.  E.g, Feb 1, May 1, Aug 1, Nov 1 (hard to miss both Dec 
and Aug, unfortunately).

Thoughts?  Counter-proposals?

sage
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release cadence

[Blair Bethwaite]

(Apologies if this is a double post - I think my phone turned it into
HTML and so bounced from ceph-devel)...

We currently use both upstream and distro (RHCS) versions on different
clusters. Downstream releases are still free to apply their own
models.

I like the idea of a predictable (and more regular) train model (and
I'd point out that trains can run a little late - there just needs to
be some process around that.

If you were to keep the even/odd release model then I'd suggest
considering a 4 monthly train cadence on these. If the even/odd model
is dropped (which would be my preference) then I like the sound of ~9
monthly releases splitting the difference between today's model.
OpenStack does 6 monthly cycles, and I think the data shows this is
too frequent for most operators (of course OpenStack is much more
complex and multifaceted as well) - many deployments are well behind
the stable deprecation schedule and it's the norm to hear people
talking about doing "skip upgrades".

On 7 September 2017 at 01:23, Sage Weil  wrote:
> Hi everyone,
>
> Traditionally, we have done a major named "stable" release twice a year,
> and every other such release has been an "LTS" release, with fixes
> backported for 1-2 years.
>
> With kraken and luminous we missed our schedule by a lot: instead of
> releasing in October and April we released in January and August.
>
> A few observations:
>
> - Not a lot of people seem to run the "odd" releases (e.g., infernalis,
> kraken).  This limits the value of actually making them.  It also means
> that those who *do* run them are running riskier code (fewer users -> more
> bugs).
>
> - The more recent requirement that upgrading clusters must make a stop at
> each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel
> -> lumninous) has been hugely helpful on the development side by reducing
> the amount of cross-version compatibility code to maintain and reducing
> the number of upgrade combinations to test.
>
> - When we try to do a time-based "train" release cadence, there always
> seems to be some "must-have" thing that delays the release a bit.  This
> doesn't happen as much with the odd releases, but it definitely happens
> with the LTS releases.  When the next LTS is a year away, it is hard to
> suck it up and wait that long.
>
> A couple of options:
>
> * Keep even/odd pattern, and continue being flexible with release dates
>
>   + flexible
>   - unpredictable
>   - odd releases of dubious value
>
> * Keep even/odd pattern, but force a 'train' model with a more regular
> cadence
>
>   + predictable schedule
>   - some features will miss the target and be delayed a year
>
> * Drop the odd releases but change nothing else (i.e., 12-month release
> cadence)
>
>   + eliminate the confusing odd releases with dubious value
>
> * Drop the odd releases, and aim for a ~9 month cadence. This splits the
> difference between the current even/odd pattern we've been doing.
>
>   + eliminate the confusing odd releases with dubious value
>   + waiting for the next release isn't quite as bad
>   - required upgrades every 9 months instead of ever 12 months
>
> * Drop the odd releases, but relax the "must upgrade through every LTS" to
> allow upgrades across 2 versions (e.g., luminous -> mimic or luminous ->
> nautilus).  Shorten release cycle (~6-9 months).
>
>   + more flexibility for users
>   + downstreams have greater choice in adopting an upstrema release
>   - more LTS branches to maintain
>   - more upgrade paths to consider
>
> Other options we should consider?  Other thoughts?
>
> Thanks!
> sage
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



-- 
Cheers,
~Blairo
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release cadence

[Blair Bethwaite]

On 7 September 2017 at 01:23, Sage Weil  wrote:
> * Drop the odd releases, and aim for a ~9 month cadence. This splits the
> difference between the current even/odd pattern we've been doing.
>
> + eliminate the confusing odd releases with dubious value
> + waiting for the next release isn't quite as bad
> - required upgrades every 9 months instead of ever 12 months
>
> * Drop the odd releases, but relax the "must upgrade through every LTS" to
> allow upgrades across 2 versions (e.g., luminous -> mimic or luminous ->
> nautilus). Shorten release cycle (~6-9 months).
>
> + more flexibility for users
> + downstreams have greater choice in adopting an upstrema release
> - more LTS branches to maintain
> - more upgrade paths to consider
>
> Other options we should consider? Other thoughts?

We currently use both upstream and distro (RHCS) versions on different
clusters. Downstream releases are still free to apply their own models.

I like the idea of a predictable (and more regular) train model (and I'd
point out that trains can run a little late - there just needs to be some
process around that.

If you were to keep the even/odd release model then I'd suggest considering
a 4 monthly train cadence on these. If the even/odd model is dropped (which
would be my preference) then I like the sound of ~9 monthly releases
splitting the difference between today's model. OpenStack does 6 monthly
cycles, and I think the data shows this is too frequent for most operators
(of course OpenStack is much more complex an multifaceted as well) - many
deployments are well behind the stable deprecation schedule and it's the
norm to hear people talking about doing "skip upgrades".

-- 
Cheers,
~Blairo
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release cadence

[Ben Hines]

We have generally been running the latest non LTS 'stable' release since my
cluster is slightly less mission critical than others, and there were
important features to us added in both Infernalis and Kraken. But i really
only care about RGW. If the rgw component could be split out of ceph into a
plugin and independently updated, it'd be awesome for us.

A minor bugfix to radosgw shouldn't be blocked by issues with RBD, for
example. I don't care at all.
Could have packages like:

ceph-core
ceph-radosgw
ceph-rbd ...
ceph-mgr..

Might increase the testing workload, but automation should take care of
that...

ceph-mgr is also similar. Minor (or even major) updates to the GUI
dashboard shouldn't be blocked rolling out to users because we're waiting
on a new RBD feature or critical RGW fix.

radosgw and mgr are really 'clients', after all.

-Ben

On Mon, Sep 11, 2017 at 3:30 PM, John Spray  wrote:

> On Wed, Sep 6, 2017 at 4:23 PM, Sage Weil  wrote:
> > Hi everyone,
> >
> > Traditionally, we have done a major named "stable" release twice a year,
> > and every other such release has been an "LTS" release, with fixes
> > backported for 1-2 years.
> >
> > With kraken and luminous we missed our schedule by a lot: instead of
> > releasing in October and April we released in January and August.
> >
> > A few observations:
> >
> > - Not a lot of people seem to run the "odd" releases (e.g., infernalis,
> > kraken).  This limits the value of actually making them.  It also means
> > that those who *do* run them are running riskier code (fewer users ->
> more
> > bugs).
> >
> > - The more recent requirement that upgrading clusters must make a stop at
> > each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel
> > -> lumninous) has been hugely helpful on the development side by reducing
> > the amount of cross-version compatibility code to maintain and reducing
> > the number of upgrade combinations to test.
> >
> > - When we try to do a time-based "train" release cadence, there always
> > seems to be some "must-have" thing that delays the release a bit.  This
> > doesn't happen as much with the odd releases, but it definitely happens
> > with the LTS releases.  When the next LTS is a year away, it is hard to
> > suck it up and wait that long.
> >
> > A couple of options:
> >
> > * Keep even/odd pattern, and continue being flexible with release dates
> >
> >   + flexible
> >   - unpredictable
> >   - odd releases of dubious value
> >
> > * Keep even/odd pattern, but force a 'train' model with a more regular
> > cadence
> >
> >   + predictable schedule
> >   - some features will miss the target and be delayed a year
> >
> > * Drop the odd releases but change nothing else (i.e., 12-month release
> > cadence)
> >
> >   + eliminate the confusing odd releases with dubious value
> >
> > * Drop the odd releases, and aim for a ~9 month cadence. This splits the
> > difference between the current even/odd pattern we've been doing.
> >
> >   + eliminate the confusing odd releases with dubious value
> >   + waiting for the next release isn't quite as bad
> >   - required upgrades every 9 months instead of ever 12 months
>
> This is my preferred option (second choice would be the next one up,
> i.e. same thing but annually).
>
> Our focus should be on delivering solid stuff, but not necessarily
> bending over backwards to enable people to run old stuff.  Our
> commitment to releases should be that there are either fixes for that
> release, or a newer (better) release to upgrade to.  Either way there
> is a solution on offer (and any user/vendor who wants to independently
> maintain other stable branches is free to do so).
>
> John
>
> > * Drop the odd releases, but relax the "must upgrade through every LTS"
> to
> > allow upgrades across 2 versions (e.g., luminous -> mimic or luminous ->
> > nautilus).  Shorten release cycle (~6-9 months).
> >
> >   + more flexibility for users
> >   + downstreams have greater choice in adopting an upstrema release
> >   - more LTS branches to maintain
> >   - more upgrade paths to consider
> >
> > Other options we should consider?  Other thoughts?
> >
> > Thanks!
> > sage
> > --
> > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> > the body of a message to majord...@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release cadence

[John Spray]

On Wed, Sep 6, 2017 at 4:23 PM, Sage Weil  wrote:
> Hi everyone,
>
> Traditionally, we have done a major named "stable" release twice a year,
> and every other such release has been an "LTS" release, with fixes
> backported for 1-2 years.
>
> With kraken and luminous we missed our schedule by a lot: instead of
> releasing in October and April we released in January and August.
>
> A few observations:
>
> - Not a lot of people seem to run the "odd" releases (e.g., infernalis,
> kraken).  This limits the value of actually making them.  It also means
> that those who *do* run them are running riskier code (fewer users -> more
> bugs).
>
> - The more recent requirement that upgrading clusters must make a stop at
> each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel
> -> lumninous) has been hugely helpful on the development side by reducing
> the amount of cross-version compatibility code to maintain and reducing
> the number of upgrade combinations to test.
>
> - When we try to do a time-based "train" release cadence, there always
> seems to be some "must-have" thing that delays the release a bit.  This
> doesn't happen as much with the odd releases, but it definitely happens
> with the LTS releases.  When the next LTS is a year away, it is hard to
> suck it up and wait that long.
>
> A couple of options:
>
> * Keep even/odd pattern, and continue being flexible with release dates
>
>   + flexible
>   - unpredictable
>   - odd releases of dubious value
>
> * Keep even/odd pattern, but force a 'train' model with a more regular
> cadence
>
>   + predictable schedule
>   - some features will miss the target and be delayed a year
>
> * Drop the odd releases but change nothing else (i.e., 12-month release
> cadence)
>
>   + eliminate the confusing odd releases with dubious value
>
> * Drop the odd releases, and aim for a ~9 month cadence. This splits the
> difference between the current even/odd pattern we've been doing.
>
>   + eliminate the confusing odd releases with dubious value
>   + waiting for the next release isn't quite as bad
>   - required upgrades every 9 months instead of ever 12 months

This is my preferred option (second choice would be the next one up,
i.e. same thing but annually).

Our focus should be on delivering solid stuff, but not necessarily
bending over backwards to enable people to run old stuff.  Our
commitment to releases should be that there are either fixes for that
release, or a newer (better) release to upgrade to.  Either way there
is a solution on offer (and any user/vendor who wants to independently
maintain other stable branches is free to do so).

John

> * Drop the odd releases, but relax the "must upgrade through every LTS" to
> allow upgrades across 2 versions (e.g., luminous -> mimic or luminous ->
> nautilus).  Shorten release cycle (~6-9 months).
>
>   + more flexibility for users
>   + downstreams have greater choice in adopting an upstrema release
>   - more LTS branches to maintain
>   - more upgrade paths to consider
>
> Other options we should consider?  Other thoughts?
>
> Thanks!
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release cadence

[Nathan Cutler]

From a backporter's perspective, the appealing options are the ones 
that reduce the number of stable releases in maintenance at any 
particular time.


In the current practice, there are always at least two LTS releases, and 
sometimes a non-LTS release as well, that are "live" and supposed to be 
getting backports. For example:


* when kraken was released, hammer and jewel were "live LTS" and kraken 
was "live non-LTS", for a total of three live releases.


* when luminous was released, hammer and kraken were declared EoL and 
there are now only two "live LTS" releases and no "live non-LTS".


During the period when there are three live releases, almost every 
bugfix seen as warranting a backport gets marked for backport to the two 
most recent stable releases. (For example, from January to August 2017 
with very few exceptions tracker issues got marked "Backport: jewel, 
kraken", not just "Backport: jewel".) This, of course, doubled the 
backporting workload, simply because if a bug is severe enough to 
backport to the most recent non-LTS release, it must be severe enough to 
be backported to the most recent LTS release as well. Unfortunately, 
there aren't enough developers working on backports to cover this double 
workload, so in practice the non-LTS release gets insufficient attention.


A "train" model could lower this backporting workload if it was 
accompanied by a declaration that the n-1 release gets backports for all 
important bugfixes and n-2 gets backports for critical bugfixes only 
(and n-3 gets EOLed).


Nathan
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release cadence

[Alexander Litvak]

As a user,  I woul like to add,  I would like to see a real 2 year support for 
LTS releases.  Hammer releases were sketchy at best in 2017.  When luminous was 
released The outstanding bugs were auto closed, good buy and good readance.


 Also the decision to drop certain OS support created a barrier to upgrade and 
looking at jewel and luminous upgrade path where you cannot easily go back 
after upgrade is completed doesn't add the confidence.


So making upgrades less radical may help production support to be more 
consistent and update process less dangerous.


I would say 9 month is a good reference point but for me  it is ready when it 
is really ready and tested.


Keeping development release may be better for devs and early adopters.  I don't 
believe production admins would go for intermediate one's as they being 
released now. 


This is only MHO and may be wrong.





⁣

On Sep 9, 2017, 15:32, at 15:32, Christian Theune  wrote:
>Hi,
>
>have been using Ceph for multiple years now. It’s unclear to me which
>of your options fits best, but here are my preferences:
>
>* Updates are risky in a way that we tend to rather not do them every
>year. Also, having seen jewel, we’ve been well off to avoid two
>major issues what would have bitten us and will upgrade from hammer in
>the next month or so.
>
>* Non-production releases are of not much value to me, as I have to
>keep our dev/staging/prod clusters in sync to work on our stuff.
>As you can never downgrade, there’s no value in it for me to try
>non-production releases (without frying dev for everyone).
>
>* I’d prefer stability over new features. *Specifically* that new
>features can be properly recombined with existing features (and each
>other) without leading to surprises. (E.g. cache tiering breaking with
>snapshots and then no way going back and a general notion of
>  “that combination wasn’t really well tested).
>
>* I’d prefer versions that I have to be maintained for
>production-critical issues maybe 2 years, so I can have some time after
>a new
>production release that overlaps with the new production release
>receiving important bug fixes until I switch.
>
>Maybe this is close to what your "Drop the odd releases, and aim for a
>~9 month cadence.” would say. Waiting for a feature for a year is a
>pain, but my personal goal for Ceph is that it first has to work
>properly, meaning: not loose your data, not "stopping the show”, and
>not drawing you into a corner you can’t get out.
>
>That’s my perspective as a user. As a fellow developer I feel your pain
>about wanting to release faster and reducing maintenance load, so
>thanks for asking!
>
>Hope this helps,
>Christian
>
>> On Sep 6, 2017, at 5:23 PM, Sage Weil  wrote:
>> 
>> Hi everyone,
>> 
>> Traditionally, we have done a major named "stable" release twice a
>year,
>> and every other such release has been an "LTS" release, with fixes
>> backported for 1-2 years.
>> 
>> With kraken and luminous we missed our schedule by a lot: instead of
>> releasing in October and April we released in January and August.
>> 
>> A few observations:
>> 
>> - Not a lot of people seem to run the "odd" releases (e.g.,
>infernalis,
>> kraken).  This limits the value of actually making them.  It also
>means
>> that those who *do* run them are running riskier code (fewer users ->
>more
>> bugs).
>> 
>> - The more recent requirement that upgrading clusters must make a
>stop at
>> each LTS (e.g., hammer -> luminous not supported, must go hammer ->
>jewel
>> -> lumninous) has been hugely helpful on the development side by
>reducing
>> the amount of cross-version compatibility code to maintain and
>reducing
>> the number of upgrade combinations to test.
>> 
>> - When we try to do a time-based "train" release cadence, there
>always
>> seems to be some "must-have" thing that delays the release a bit. 
>This
>> doesn't happen as much with the odd releases, but it definitely
>happens
>> with the LTS releases.  When the next LTS is a year away, it is hard
>to
>> suck it up and wait that long.
>> 
>> A couple of options:
>> 
>> * Keep even/odd pattern, and continue being flexible with release
>dates
>> 
>>  + flexible
>>  - unpredictable
>>  - odd releases of dubious value
>> 
>> * Keep even/odd pattern, but force a 'train' model with a more
>regular
>> cadence
>> 
>>  + predictable schedule
>>  - some features will miss the target and be delayed a year
>> 
>> * Drop the odd releases but change nothing else (i.e., 12-month
>release
>> cadence)
>> 
>>  + eliminate the confusing odd releases with dubious value
>> 
>> * Drop the odd releases, and aim for a ~9 month cadence. This splits
>the
>> difference between the current even/odd pattern we've been doing.
>> 
>>  + eliminate the confusing odd releases with dubious value
>>  + waiting for the next release isn't quite as bad
>>  - required upgrades every 9 months instead of ever 12 months
>> 
>> * Drop the odd releases, but relax the 

Re: [ceph-users] Ceph release cadence

[Sasha Litvak]

As a user,  I woul like to add,  I would like to see a real 2 year support
for LTS releases.  Hammer releases were sketchy at best in 2017.  When
luminous was released The outstanding bugs were auto closed, good buy and
good readance.

 Also the decision to drop certain OS support created a barrier to upgrade
and looking at jewel and luminous upgrade path where you cannot easily go
back after upgrade is completed doesn't add the confidence.

So making upgrades less radical may help production support to be more
consistent and update process less dangerous.

I would say 9 month is a good reference point but for me  it is ready when
it is really ready and tested.

Keeping development release may be better for devs and early adopters.  I
don't believe production admins would go for intermediate one's as they
being released now.

This is only MHO and may be wrong

On Saturday, September 9, 2017, Christian Theune  wrote:

> Hi,
>
> have been using Ceph for multiple years now. It’s unclear to me which of
> your options fits best, but here are my preferences:
>
> * Updates are risky in a way that we tend to rather not do them every
> year. Also, having seen jewel, we’ve been well off to avoid two
>   major issues what would have bitten us and will upgrade from hammer in
> the next month or so.
>
> * Non-production releases are of not much value to me, as I have to keep
> our dev/staging/prod clusters in sync to work on our stuff.
>   As you can never downgrade, there’s no value in it for me to try
> non-production releases (without frying dev for everyone).
>
> * I’d prefer stability over new features. *Specifically* that new features
> can be properly recombined with existing features (and each
>   other) without leading to surprises. (E.g. cache tiering breaking with
> snapshots and then no way going back and a general notion of
>   “that combination wasn’t really well tested).
>
> * I’d prefer versions that I have to be maintained for production-critical
> issues maybe 2 years, so I can have some time after a new
>   production release that overlaps with the new production release
> receiving important bug fixes until I switch.
>
> Maybe this is close to what your "Drop the odd releases, and aim for a ~9
> month cadence.” would say. Waiting for a feature for a year is a pain, but
> my personal goal for Ceph is that it first has to work properly, meaning:
> not loose your data, not "stopping the show”, and not drawing you into a
> corner you can’t get out.
>
> That’s my perspective as a user. As a fellow developer I feel your pain
> about wanting to release faster and reducing maintenance load, so thanks
> for asking!
>
> Hope this helps,
> Christian
>
> > On Sep 6, 2017, at 5:23 PM, Sage Weil >
> wrote:
> >
> > Hi everyone,
> >
> > Traditionally, we have done a major named "stable" release twice a year,
> > and every other such release has been an "LTS" release, with fixes
> > backported for 1-2 years.
> >
> > With kraken and luminous we missed our schedule by a lot: instead of
> > releasing in October and April we released in January and August.
> >
> > A few observations:
> >
> > - Not a lot of people seem to run the "odd" releases (e.g., infernalis,
> > kraken).  This limits the value of actually making them.  It also means
> > that those who *do* run them are running riskier code (fewer users ->
> more
> > bugs).
> >
> > - The more recent requirement that upgrading clusters must make a stop at
> > each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel
> > -> lumninous) has been hugely helpful on the development side by reducing
> > the amount of cross-version compatibility code to maintain and reducing
> > the number of upgrade combinations to test.
> >
> > - When we try to do a time-based "train" release cadence, there always
> > seems to be some "must-have" thing that delays the release a bit.  This
> > doesn't happen as much with the odd releases, but it definitely happens
> > with the LTS releases.  When the next LTS is a year away, it is hard to
> > suck it up and wait that long.
> >
> > A couple of options:
> >
> > * Keep even/odd pattern, and continue being flexible with release dates
> >
> >  + flexible
> >  - unpredictable
> >  - odd releases of dubious value
> >
> > * Keep even/odd pattern, but force a 'train' model with a more regular
> > cadence
> >
> >  + predictable schedule
> >  - some features will miss the target and be delayed a year
> >
> > * Drop the odd releases but change nothing else (i.e., 12-month release
> > cadence)
> >
> >  + eliminate the confusing odd releases with dubious value
> >
> > * Drop the odd releases, and aim for a ~9 month cadence. This splits the
> > difference between the current even/odd pattern we've been doing.
> >
> >  + eliminate the confusing odd releases with dubious value
> >  + waiting for the next release isn't quite as bad
> >  - required upgrades every 9 months instead of ever 12 

Re: [ceph-users] Ceph release cadence

[Sasha Litvak]

As a user,  I woul like to add,  I would like to see a real 2 year support
for LTS releases.  Hammer releases were sketchy at best in 2017.  When
luminous was released The outstanding bugs were auto closed, good buy and
good readance.

 Also the decision to drop certain OS support created a barrier to upgrade
and looking at jewel and luminous upgrade path where you cannot easily go
back after upgrade is completed doesn't add the confidence.

So making upgrades less radical may help production support to be more
consistent and update process less dangerous.

I would say 9 month is a good reference point but for me  it is ready when
it is really ready and tested.

Keeping development release may be better for devs and early adopters.  I
don't believe production admins would go for intermediate one's as they
being released now.

This is only MHO and may be wrong.

On Sep 9, 2017 15:32, "Christian Theune"  wrote:

> Hi,
>
> have been using Ceph for multiple years now. It’s unclear to me which of
> your options fits best, but here are my preferences:
>
> * Updates are risky in a way that we tend to rather not do them every
> year. Also, having seen jewel, we’ve been well off to avoid two
>   major issues what would have bitten us and will upgrade from hammer in
> the next month or so.
>
> * Non-production releases are of not much value to me, as I have to keep
> our dev/staging/prod clusters in sync to work on our stuff.
>   As you can never downgrade, there’s no value in it for me to try
> non-production releases (without frying dev for everyone).
>
> * I’d prefer stability over new features. *Specifically* that new features
> can be properly recombined with existing features (and each
>   other) without leading to surprises. (E.g. cache tiering breaking with
> snapshots and then no way going back and a general notion of
>   “that combination wasn’t really well tested).
>
> * I’d prefer versions that I have to be maintained for production-critical
> issues maybe 2 years, so I can have some time after a new
>   production release that overlaps with the new production release
> receiving important bug fixes until I switch.
>
> Maybe this is close to what your "Drop the odd releases, and aim for a ~9
> month cadence.” would say. Waiting for a feature for a year is a pain, but
> my personal goal for Ceph is that it first has to work properly, meaning:
> not loose your data, not "stopping the show”, and not drawing you into a
> corner you can’t get out.
>
> That’s my perspective as a user. As a fellow developer I feel your pain
> about wanting to release faster and reducing maintenance load, so thanks
> for asking!
>
> Hope this helps,
> Christian
>
> > On Sep 6, 2017, at 5:23 PM, Sage Weil  wrote:
> >
> > Hi everyone,
> >
> > Traditionally, we have done a major named "stable" release twice a year,
> > and every other such release has been an "LTS" release, with fixes
> > backported for 1-2 years.
> >
> > With kraken and luminous we missed our schedule by a lot: instead of
> > releasing in October and April we released in January and August.
> >
> > A few observations:
> >
> > - Not a lot of people seem to run the "odd" releases (e.g., infernalis,
> > kraken).  This limits the value of actually making them.  It also means
> > that those who *do* run them are running riskier code (fewer users ->
> more
> > bugs).
> >
> > - The more recent requirement that upgrading clusters must make a stop at
> > each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel
> > -> lumninous) has been hugely helpful on the development side by reducing
> > the amount of cross-version compatibility code to maintain and reducing
> > the number of upgrade combinations to test.
> >
> > - When we try to do a time-based "train" release cadence, there always
> > seems to be some "must-have" thing that delays the release a bit.  This
> > doesn't happen as much with the odd releases, but it definitely happens
> > with the LTS releases.  When the next LTS is a year away, it is hard to
> > suck it up and wait that long.
> >
> > A couple of options:
> >
> > * Keep even/odd pattern, and continue being flexible with release dates
> >
> >  + flexible
> >  - unpredictable
> >  - odd releases of dubious value
> >
> > * Keep even/odd pattern, but force a 'train' model with a more regular
> > cadence
> >
> >  + predictable schedule
> >  - some features will miss the target and be delayed a year
> >
> > * Drop the odd releases but change nothing else (i.e., 12-month release
> > cadence)
> >
> >  + eliminate the confusing odd releases with dubious value
> >
> > * Drop the odd releases, and aim for a ~9 month cadence. This splits the
> > difference between the current even/odd pattern we've been doing.
> >
> >  + eliminate the confusing odd releases with dubious value
> >  + waiting for the next release isn't quite as bad
> >  - required upgrades every 9 months instead of ever 12 months
> >
> > * Drop 

Re: [ceph-users] Ceph release cadence

[Christian Theune]

Hi,

have been using Ceph for multiple years now. It’s unclear to me which of your 
options fits best, but here are my preferences:

* Updates are risky in a way that we tend to rather not do them every year. 
Also, having seen jewel, we’ve been well off to avoid two
  major issues what would have bitten us and will upgrade from hammer in the 
next month or so.

* Non-production releases are of not much value to me, as I have to keep our 
dev/staging/prod clusters in sync to work on our stuff.
  As you can never downgrade, there’s no value in it for me to try 
non-production releases (without frying dev for everyone).

* I’d prefer stability over new features. *Specifically* that new features can 
be properly recombined with existing features (and each
  other) without leading to surprises. (E.g. cache tiering breaking with 
snapshots and then no way going back and a general notion of
  “that combination wasn’t really well tested).

* I’d prefer versions that I have to be maintained for production-critical 
issues maybe 2 years, so I can have some time after a new
  production release that overlaps with the new production release receiving 
important bug fixes until I switch.

Maybe this is close to what your "Drop the odd releases, and aim for a ~9 month 
cadence.” would say. Waiting for a feature for a year is a pain, but my 
personal goal for Ceph is that it first has to work properly, meaning: not 
loose your data, not "stopping the show”, and not drawing you into a corner you 
can’t get out.

That’s my perspective as a user. As a fellow developer I feel your pain about 
wanting to release faster and reducing maintenance load, so thanks for asking!

Hope this helps,
Christian

> On Sep 6, 2017, at 5:23 PM, Sage Weil  wrote:
> 
> Hi everyone,
> 
> Traditionally, we have done a major named "stable" release twice a year,
> and every other such release has been an "LTS" release, with fixes
> backported for 1-2 years.
> 
> With kraken and luminous we missed our schedule by a lot: instead of
> releasing in October and April we released in January and August.
> 
> A few observations:
> 
> - Not a lot of people seem to run the "odd" releases (e.g., infernalis,
> kraken).  This limits the value of actually making them.  It also means
> that those who *do* run them are running riskier code (fewer users -> more
> bugs).
> 
> - The more recent requirement that upgrading clusters must make a stop at
> each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel
> -> lumninous) has been hugely helpful on the development side by reducing
> the amount of cross-version compatibility code to maintain and reducing
> the number of upgrade combinations to test.
> 
> - When we try to do a time-based "train" release cadence, there always
> seems to be some "must-have" thing that delays the release a bit.  This
> doesn't happen as much with the odd releases, but it definitely happens
> with the LTS releases.  When the next LTS is a year away, it is hard to
> suck it up and wait that long.
> 
> A couple of options:
> 
> * Keep even/odd pattern, and continue being flexible with release dates
> 
>  + flexible
>  - unpredictable
>  - odd releases of dubious value
> 
> * Keep even/odd pattern, but force a 'train' model with a more regular
> cadence
> 
>  + predictable schedule
>  - some features will miss the target and be delayed a year
> 
> * Drop the odd releases but change nothing else (i.e., 12-month release
> cadence)
> 
>  + eliminate the confusing odd releases with dubious value
> 
> * Drop the odd releases, and aim for a ~9 month cadence. This splits the
> difference between the current even/odd pattern we've been doing.
> 
>  + eliminate the confusing odd releases with dubious value
>  + waiting for the next release isn't quite as bad
>  - required upgrades every 9 months instead of ever 12 months
> 
> * Drop the odd releases, but relax the "must upgrade through every LTS" to
> allow upgrades across 2 versions (e.g., luminous -> mimic or luminous ->
> nautilus).  Shorten release cycle (~6-9 months).
> 
>  + more flexibility for users
>  + downstreams have greater choice in adopting an upstrema release
>  - more LTS branches to maintain
>  - more upgrade paths to consider
> 
> Other options we should consider?  Other thoughts?
> 
> Thanks!
> sage
> --
> To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
> the body of a message to majord...@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Liebe Grüße,
Christian Theune

--
Christian Theune · c...@flyingcircus.io · +49 345 219401 0
Flying Circus Internet Operations GmbH · http://flyingcircus.io
Forsterstraße 29 · 06112 Halle (Saale) · Deutschland
HR Stendal HRB 21169 · Geschäftsführer: Christian Theune, Christian Zagrodnick



signature.asc
Description: Message signed with OpenPGP
___
ceph-users mailing list
ceph-users@lists.ceph.com

Re: [ceph-users] Ceph release cadence

[Scottix]

Personally I kind of like the current format and fundamentally we are
talking about Data storage which should be the most tested and scrutinized
piece of software on your computer. Having XYZ feature later than sooner
compared to oh I lost all my data. I am thinking of a recent FS that had a
feature they shouldn't have released. I appreciate the extra time it takes
to release to make it resilient.

Having a LTS version to rely on provides a good assurance the upgrade
process wil be thoroughly tested.
Having a version to do more experimental features keeps the new features at
bay, it follows the Ubuntu model basically.

I feel there were a lot of underpinning features in Luminous that
checkmarked a lot of checkboxes you have been wanting for a while. One
thing to consider possibly a lot of the core features become more
incremental.

I guess from my use case Ceph actually does everything I need it to do atm.
Yes new features and better processes make it better, but more or less I am
pretty content. Maybe I am a small minority in this logic.

On Fri, Sep 8, 2017 at 2:20 AM Matthew Vernon  wrote:

> Hi,
>
> On 06/09/17 16:23, Sage Weil wrote:
>
> > Traditionally, we have done a major named "stable" release twice a year,
> > and every other such release has been an "LTS" release, with fixes
> > backported for 1-2 years.
>
> We use the ceph version that comes with our distribution (Ubuntu LTS);
> those come out every 2 years (though we won't move to a brand-new
> distribution until we've done some testing!). So from my POV, LTS ceph
> releases that come out such that adjacent ceph LTSs fit neatly into
> adjacent Ubuntu LTSs is the ideal outcome. We're unlikely to ever try
> putting a non-LTS ceph version into production.
>
> I hope this isn't an unusual requirement :)
>
> Matthew
>
>
> --
>  The Wellcome Trust Sanger Institute is operated by Genome Research
>  Limited, a charity registered in England with number 1021457 and a
>  company registered in England with number 2742969, whose registered
>  office is 215 Euston Road, London, NW1 2BE.
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release cadence

[Matthew Vernon]

Hi,

On 06/09/17 16:23, Sage Weil wrote:

> Traditionally, we have done a major named "stable" release twice a year, 
> and every other such release has been an "LTS" release, with fixes 
> backported for 1-2 years.

We use the ceph version that comes with our distribution (Ubuntu LTS);
those come out every 2 years (though we won't move to a brand-new
distribution until we've done some testing!). So from my POV, LTS ceph
releases that come out such that adjacent ceph LTSs fit neatly into
adjacent Ubuntu LTSs is the ideal outcome. We're unlikely to ever try
putting a non-LTS ceph version into production.

I hope this isn't an unusual requirement :)

Matthew


-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE. 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release cadence

[Deepak Naidu]

>> Maybe I missed something, but I think Ceph does not support LTS releases for 
>> 3 years.
Yes, you are correct but it averages to 18mths sometime I see 20mths(Hammer). 
But anything with 1yr release cycle is not worth the time and having near 3yr 
support model is best for PROD.

http://docs.ceph.com/docs/master/releases/

--
Deepak

-Original Message-
From: Henrik Korkuc [mailto:li...@kirneh.eu] 
Sent: Wednesday, September 06, 2017 10:50 PM
To: Deepak Naidu; Sage Weil; ceph-de...@vger.kernel.org; 
ceph-maintain...@ceph.com; ceph-us...@ceph.com
Subject: Re: [ceph-users] Ceph release cadence

On 17-09-07 02:42, Deepak Naidu wrote:
> Hope collective feedback helps. So here's one.
>
>>> - Not a lot of people seem to run the "odd" releases (e.g., infernalis, 
>>> kraken).
> I think the more obvious reason companies/users wanting to use CEPH will 
> stick with LTS versions as it models the 3yr  support cycle.
Maybe I missed something, but I think Ceph does not support LTS releases for 3 
years.

>>> * Drop the odd releases, and aim for a ~9 month cadence. This splits the 
>>> difference between the current even/odd pattern we've been doing.
> Yes, provided an easy upgrade process.
>
>
> --
> Deepak
>
>
>
>
> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf 
> Of Sage Weil
> Sent: Wednesday, September 06, 2017 8:24 AM
> To: ceph-de...@vger.kernel.org; ceph-maintain...@ceph.com; 
> ceph-us...@ceph.com
> Subject: [ceph-users] Ceph release cadence
>
> Hi everyone,
>
> Traditionally, we have done a major named "stable" release twice a year, and 
> every other such release has been an "LTS" release, with fixes backported for 
> 1-2 years.
>
> With kraken and luminous we missed our schedule by a lot: instead of 
> releasing in October and April we released in January and August.
>
> A few observations:
>
> - Not a lot of people seem to run the "odd" releases (e.g., infernalis, 
> kraken).  This limits the value of actually making them.  It also means that 
> those who *do* run them are running riskier code (fewer users -> more bugs).
>
> - The more recent requirement that upgrading clusters must make a stop 
> at each LTS (e.g., hammer -> luminous not supported, must go hammer -> 
> jewel
> -> lumninous) has been hugely helpful on the development side by 
> -> reducing
> the amount of cross-version compatibility code to maintain and reducing the 
> number of upgrade combinations to test.
>
> - When we try to do a time-based "train" release cadence, there always seems 
> to be some "must-have" thing that delays the release a bit.  This doesn't 
> happen as much with the odd releases, but it definitely happens with the LTS 
> releases.  When the next LTS is a year away, it is hard to suck it up and 
> wait that long.
>
> A couple of options:
>
> * Keep even/odd pattern, and continue being flexible with release 
> dates
>
>+ flexible
>- unpredictable
>- odd releases of dubious value
>
> * Keep even/odd pattern, but force a 'train' model with a more regular 
> cadence
>
>+ predictable schedule
>- some features will miss the target and be delayed a year
>
> * Drop the odd releases but change nothing else (i.e., 12-month 
> release
> cadence)
>
>+ eliminate the confusing odd releases with dubious value
>   
> * Drop the odd releases, and aim for a ~9 month cadence. This splits the 
> difference between the current even/odd pattern we've been doing.
>
>+ eliminate the confusing odd releases with dubious value
>+ waiting for the next release isn't quite as bad
>- required upgrades every 9 months instead of ever 12 months
>
> * Drop the odd releases, but relax the "must upgrade through every LTS" to 
> allow upgrades across 2 versions (e.g., luminous -> mimic or luminous -> 
> nautilus).  Shorten release cycle (~6-9 months).
>
>+ more flexibility for users
>+ downstreams have greater choice in adopting an upstrema release
>- more LTS branches to maintain
>- more upgrade paths to consider
>
> Other options we should consider?  Other thoughts?
>
> Thanks!
> sage
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> --
> - This email message is for the sole use of the intended 
> recipient(s) and may contain confidential information.  Any 
> unauthorized review, use, disclosure or distribution is pro

Re: [ceph-users] Ceph release cadence

[Henrik Korkuc]

On 17-09-06 18:23, Sage Weil wrote:

Hi everyone,

Traditionally, we have done a major named "stable" release twice a year,
and every other such release has been an "LTS" release, with fixes
backported for 1-2 years.

With kraken and luminous we missed our schedule by a lot: instead of
releasing in October and April we released in January and August.

A few observations:

- Not a lot of people seem to run the "odd" releases (e.g., infernalis,
kraken).  This limits the value of actually making them.  It also means
that those who *do* run them are running riskier code (fewer users -> more
bugs).

- The more recent requirement that upgrading clusters must make a stop at
each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel
-> lumninous) has been hugely helpful on the development side by reducing
the amount of cross-version compatibility code to maintain and reducing
the number of upgrade combinations to test.

- When we try to do a time-based "train" release cadence, there always
seems to be some "must-have" thing that delays the release a bit.  This
doesn't happen as much with the odd releases, but it definitely happens
with the LTS releases.  When the next LTS is a year away, it is hard to
suck it up and wait that long.

A couple of options:

* Keep even/odd pattern, and continue being flexible with release dates

   + flexible
   - unpredictable
   - odd releases of dubious value

* Keep even/odd pattern, but force a 'train' model with a more regular
cadence

   + predictable schedule
   - some features will miss the target and be delayed a year

* Drop the odd releases but change nothing else (i.e., 12-month release
cadence)

   + eliminate the confusing odd releases with dubious value
  
* Drop the odd releases, and aim for a ~9 month cadence. This splits the

difference between the current even/odd pattern we've been doing.

   + eliminate the confusing odd releases with dubious value
   + waiting for the next release isn't quite as bad
   - required upgrades every 9 months instead of ever 12 months

* Drop the odd releases, but relax the "must upgrade through every LTS" to
allow upgrades across 2 versions (e.g., luminous -> mimic or luminous ->
nautilus).  Shorten release cycle (~6-9 months).

   + more flexibility for users
   + downstreams have greater choice in adopting an upstrema release
   - more LTS branches to maintain
   - more upgrade paths to consider

Other options we should consider?  Other thoughts?

What about this:
* drop odd releases
* have ~9 months release schedule
* no version jumping
* bugfix support for 2 release cycles
* list of major incoming features with their status, disabled by feature 
flag.
* have more QA passed dev releases so that people waiting for new 
features would be able to try them out


This way we trade shorter release cycle with longer bugfix support but 
no version jumping. This way stable folks could upgrade from 
"legacy-stable" to "old-stable" having already multiple minor fixes in 
both releases.


And bleading-edge people waiting for some features would know current 
status of new features (e.g. multi-active MDS going stable in L was a 
surprise for me). With dev releases to run in dev/staging env for testing.


Potentially shorter releases would have less features in, so it should 
be less risky to use them and of course shorter wait before new release.



Thanks!
sage
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release cadence

[Adrian Saul]

> * Drop the odd releases, and aim for a ~9 month cadence. This splits the
> difference between the current even/odd pattern we've been doing.
>
>   + eliminate the confusing odd releases with dubious value
>   + waiting for the next release isn't quite as bad
>   - required upgrades every 9 months instead of ever 12 months

As a user is probably closest to the ideal, although a production deployment 
might slip out of the LTS view in 18 months given once deployed they tend to 
stay static.

>From a testing perspective it would be good to know you could deploy the 
>"early access" version of a release and test with that rather than having to 
>switch release to productionise when that release is blessed.

Also, and this might be harder to achieve, but could krbd support for new 
releases be more aligned with kernel versions?  Or at the least a definitive 
map of what kernels and backports support which release.


Confidentiality: This email and any attachments are confidential and may be 
subject to copyright, legal or some other professional privilege. They are 
intended solely for the attention and use of the named addressee(s). They may 
only be copied, distributed or disclosed with the consent of the copyright 
owner. If you have received this email by mistake or by breach of the 
confidentiality clause, please notify the sender immediately by return email 
and delete or destroy all copies of the email. Any confidentiality, privilege 
or copyright is not waived or lost because this email has been sent to you by 
mistake.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release cadence

[Henrik Korkuc]

On 17-09-07 02:42, Deepak Naidu wrote:

Hope collective feedback helps. So here's one.


- Not a lot of people seem to run the "odd" releases (e.g., infernalis, kraken).

I think the more obvious reason companies/users wanting to use CEPH will stick 
with LTS versions as it models the 3yr  support cycle.
Maybe I missed something, but I think Ceph does not support LTS releases 
for 3 years.



* Drop the odd releases, and aim for a ~9 month cadence. This splits the 
difference between the current even/odd pattern we've been doing.

Yes, provided an easy upgrade process.


--
Deepak




-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Sage 
Weil
Sent: Wednesday, September 06, 2017 8:24 AM
To: ceph-de...@vger.kernel.org; ceph-maintain...@ceph.com; ceph-us...@ceph.com
Subject: [ceph-users] Ceph release cadence

Hi everyone,

Traditionally, we have done a major named "stable" release twice a year, and every other 
such release has been an "LTS" release, with fixes backported for 1-2 years.

With kraken and luminous we missed our schedule by a lot: instead of releasing 
in October and April we released in January and August.

A few observations:

- Not a lot of people seem to run the "odd" releases (e.g., infernalis, kraken).  
This limits the value of actually making them.  It also means that those who *do* run them 
are running riskier code (fewer users -> more bugs).

- The more recent requirement that upgrading clusters must make a stop at each LTS 
(e.g., hammer -> luminous not supported, must go hammer -> jewel
-> lumninous) has been hugely helpful on the development side by
-> reducing
the amount of cross-version compatibility code to maintain and reducing the 
number of upgrade combinations to test.

- When we try to do a time-based "train" release cadence, there always seems to be some 
"must-have" thing that delays the release a bit.  This doesn't happen as much with the 
odd releases, but it definitely happens with the LTS releases.  When the next LTS is a year away, 
it is hard to suck it up and wait that long.

A couple of options:

* Keep even/odd pattern, and continue being flexible with release dates

   + flexible
   - unpredictable
   - odd releases of dubious value

* Keep even/odd pattern, but force a 'train' model with a more regular cadence

   + predictable schedule
   - some features will miss the target and be delayed a year

* Drop the odd releases but change nothing else (i.e., 12-month release
cadence)

   + eliminate the confusing odd releases with dubious value
  
* Drop the odd releases, and aim for a ~9 month cadence. This splits the difference between the current even/odd pattern we've been doing.


   + eliminate the confusing odd releases with dubious value
   + waiting for the next release isn't quite as bad
   - required upgrades every 9 months instead of ever 12 months

* Drop the odd releases, but relax the "must upgrade through every LTS" to allow 
upgrades across 2 versions (e.g., luminous -> mimic or luminous -> nautilus).  Shorten 
release cycle (~6-9 months).

   + more flexibility for users
   + downstreams have greater choice in adopting an upstrema release
   - more LTS branches to maintain
   - more upgrade paths to consider

Other options we should consider?  Other thoughts?

Thanks!
sage
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
---
This email message is for the sole use of the intended recipient(s) and may 
contain
confidential information.  Any unauthorized review, use, disclosure or 
distribution
is prohibited.  If you are not the intended recipient, please contact the 
sender by
reply email and destroy all copies of the original message.
---
--
To unsubscribe from this list: send the line "unsubscribe ceph-devel" in
the body of a message to majord...@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release cadence

[Deepak Naidu]

Hope collective feedback helps. So here's one.

>>- Not a lot of people seem to run the "odd" releases (e.g., infernalis, 
>>kraken).  
I think the more obvious reason companies/users wanting to use CEPH will stick 
with LTS versions as it models the 3yr  support cycle.

>>* Drop the odd releases, and aim for a ~9 month cadence. This splits the 
>>difference between the current even/odd pattern we've been doing.
Yes, provided an easy upgrade process.


--
Deepak




-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Sage 
Weil
Sent: Wednesday, September 06, 2017 8:24 AM
To: ceph-de...@vger.kernel.org; ceph-maintain...@ceph.com; ceph-us...@ceph.com
Subject: [ceph-users] Ceph release cadence

Hi everyone,

Traditionally, we have done a major named "stable" release twice a year, and 
every other such release has been an "LTS" release, with fixes backported for 
1-2 years.

With kraken and luminous we missed our schedule by a lot: instead of releasing 
in October and April we released in January and August.

A few observations:

- Not a lot of people seem to run the "odd" releases (e.g., infernalis, 
kraken).  This limits the value of actually making them.  It also means that 
those who *do* run them are running riskier code (fewer users -> more bugs).

- The more recent requirement that upgrading clusters must make a stop at each 
LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel 
-> lumninous) has been hugely helpful on the development side by 
-> reducing
the amount of cross-version compatibility code to maintain and reducing the 
number of upgrade combinations to test.

- When we try to do a time-based "train" release cadence, there always seems to 
be some "must-have" thing that delays the release a bit.  This doesn't happen 
as much with the odd releases, but it definitely happens with the LTS releases. 
 When the next LTS is a year away, it is hard to suck it up and wait that long.

A couple of options:

* Keep even/odd pattern, and continue being flexible with release dates

  + flexible
  - unpredictable
  - odd releases of dubious value

* Keep even/odd pattern, but force a 'train' model with a more regular cadence

  + predictable schedule
  - some features will miss the target and be delayed a year

* Drop the odd releases but change nothing else (i.e., 12-month release
cadence)

  + eliminate the confusing odd releases with dubious value
 
* Drop the odd releases, and aim for a ~9 month cadence. This splits the 
difference between the current even/odd pattern we've been doing.

  + eliminate the confusing odd releases with dubious value
  + waiting for the next release isn't quite as bad
  - required upgrades every 9 months instead of ever 12 months

* Drop the odd releases, but relax the "must upgrade through every LTS" to 
allow upgrades across 2 versions (e.g., luminous -> mimic or luminous -> 
nautilus).  Shorten release cycle (~6-9 months).

  + more flexibility for users
  + downstreams have greater choice in adopting an upstrema release
  - more LTS branches to maintain
  - more upgrade paths to consider

Other options we should consider?  Other thoughts?

Thanks!
sage
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
---
This email message is for the sole use of the intended recipient(s) and may 
contain
confidential information.  Any unauthorized review, use, disclosure or 
distribution
is prohibited.  If you are not the intended recipient, please contact the 
sender by
reply email and destroy all copies of the original message.
---
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release cadence

[Eric Eastman]

I have been working with Ceph for the last several years and I help
support multiple Ceph clusters. I would like to have the team drop the
Even/Odd release schedule, and go to an all production release
schedule.  I would like releases on no more then a 9 month schedule,
with smaller incremental changes and predictable dates.  It would be
nice to be able to upgrade from at least the last 2 releases.

I would also like to see the RC schedule be more like the Linux kernel
or Samba releases, where we have an idea on how many RCs to expect and
how often they would come out, so we can schedule our testing, and
provide more helpful feedback during the RC period.

Eric

On Wed, Sep 6, 2017 at 9:23 AM, Sage Weil  wrote:
> Hi everyone,
>
> Traditionally, we have done a major named "stable" release twice a year,
> and every other such release has been an "LTS" release, with fixes
> backported for 1-2 years.
>
> With kraken and luminous we missed our schedule by a lot: instead of
> releasing in October and April we released in January and August.
>
> A few observations:
>
> - Not a lot of people seem to run the "odd" releases (e.g., infernalis,
> kraken).  This limits the value of actually making them.  It also means
> that those who *do* run them are running riskier code (fewer users -> more
> bugs).
>
> - The more recent requirement that upgrading clusters must make a stop at
> each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel
> -> lumninous) has been hugely helpful on the development side by reducing
> the amount of cross-version compatibility code to maintain and reducing
> the number of upgrade combinations to test.
>
> - When we try to do a time-based "train" release cadence, there always
> seems to be some "must-have" thing that delays the release a bit.  This
> doesn't happen as much with the odd releases, but it definitely happens
> with the LTS releases.  When the next LTS is a year away, it is hard to
> suck it up and wait that long.
>
> A couple of options:
>
> * Keep even/odd pattern, and continue being flexible with release dates
>
>   + flexible
>   - unpredictable
>   - odd releases of dubious value
>
> * Keep even/odd pattern, but force a 'train' model with a more regular
> cadence
>
>   + predictable schedule
>   - some features will miss the target and be delayed a year
>
> * Drop the odd releases but change nothing else (i.e., 12-month release
> cadence)
>
>   + eliminate the confusing odd releases with dubious value
>
> * Drop the odd releases, and aim for a ~9 month cadence. This splits the
> difference between the current even/odd pattern we've been doing.
>
>   + eliminate the confusing odd releases with dubious value
>   + waiting for the next release isn't quite as bad
>   - required upgrades every 9 months instead of ever 12 months
>
> * Drop the odd releases, but relax the "must upgrade through every LTS" to
> allow upgrades across 2 versions (e.g., luminous -> mimic or luminous ->
> nautilus).  Shorten release cycle (~6-9 months).
>
>   + more flexibility for users
>   + downstreams have greater choice in adopting an upstrema release
>   - more LTS branches to maintain
>   - more upgrade paths to consider
>
> Other options we should consider?  Other thoughts?
>
> Thanks!
> sage
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release cadence

[Joao Eduardo Luis]

On 09/06/2017 04:23 PM, Sage Weil wrote:

* Keep even/odd pattern, but force a 'train' model with a more regular
cadence

   + predictable schedule
   - some features will miss the target and be delayed a year


Personally, I think a predictable schedule is the way to go. Two major 
reasons come to mind:


1. Developers are actually aware of what the cut-off date is, and will 
plan accordingly; and,


2. Downstreams will have a better notion of when the next release is to 
be expected, and plan accordingly.


However, a one year wait for a release may be a can of worms waiting to 
be opened. Even though it would ideally provide us a lot more time to 
merge stuff and test it, there's also the downside that some stuff may 
be pushed further and further down the line, and eventually merged just 
before the window closes.


For that, I'd argue having an intermediate (staging?) release could be 
helpful, but I fear it would not be anything more than any other 
dev-release. Therefore, if we stick to a one-year cadence, let's have 
frequent dev-releases.


I would also like to argue for a hard cut-off date considerably before 
major holiday seasons. Because no one really wants to be dealing with 
bugs or releasing software while a considerable portion of developers 
are away.


Additionally,


* Drop the odd releases, but relax the "must upgrade through every LTS" to
allow upgrades across 2 versions (e.g., luminous -> mimic or luminous ->
nautilus).  Shorten release cycle (~6-9 months).


I can be on board with this too. As long as we have a very clear cut-off 
date regardless.


  -Joao
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release cadence

[Ken Dreyer]

On Wed, Sep 6, 2017 at 9:23 AM, Sage Weil  wrote:
> * Keep even/odd pattern, but force a 'train' model with a more regular
> cadence
>
>   + predictable schedule
>   - some features will miss the target and be delayed a year

This one (#2, regular release cadence) is the one I will value the most.

- Ken
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release cadence

[Jack]

Hi Sage,

The one option I do not want for Ceph is the last one: support upgrade
across multiple LTS versions

I'd rather wait 3 months for a better release (both in terms of
functions and quality) than seeing the Ceph team exhausted, having to
maintain for years a lot more releases and code


Others thoughts:
- I do not think time-based releases is a must-have. As a user, I
prefere quality over short-time releases, especially for a critical
software as Ceph (storage and stuff, I do not want creepy code here):
this eliminates #2
- for the same reason, I do not care if there is a release every 12
months, or every 9 months : a couple of months without the new release
is not business-critical, having a buggy software / not well tested
features is
- odd releases still allow bugs squashing, I guess it gives real-world
feedbacks too. Some people do have a "not so important" cluster, that
may use the odd releases

So:
#2 and especially #5: nope
#1, #3 or #4: I prefer #1, the others are fine too (if odd releases is
somehow a burden, for instance)

Regards,



So, to me, #1 is fine

On 06/09/2017 18:35, Alex Gorbachev wrote:
> On Wed, Sep 6, 2017 at 11:23 AM Sage Weil  wrote:
> 
>> Hi everyone,
>>
>> Traditionally, we have done a major named "stable" release twice a year,
>> and every other such release has been an "LTS" release, with fixes
>> backported for 1-2 years.
>>
>> With kraken and luminous we missed our schedule by a lot: instead of
>> releasing in October and April we released in January and August.
>>
>> A few observations:
>>
>> - Not a lot of people seem to run the "odd" releases (e.g., infernalis,
>> kraken).  This limits the value of actually making them.  It also means
>> that those who *do* run them are running riskier code (fewer users -> more
>> bugs).
>>
>> - The more recent requirement that upgrading clusters must make a stop at
>> each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel
>> -> lumninous) has been hugely helpful on the development side by reducing
>> the amount of cross-version compatibility code to maintain and reducing
>> the number of upgrade combinations to test.
>>
>> - When we try to do a time-based "train" release cadence, there always
>> seems to be some "must-have" thing that delays the release a bit.  This
>> doesn't happen as much with the odd releases, but it definitely happens
>> with the LTS releases.  When the next LTS is a year away, it is hard to
>> suck it up and wait that long.
>>
>> A couple of options:
>>
>> * Keep even/odd pattern, and continue being flexible with release dates
>>
>>   + flexible
>>   - unpredictable
>>   - odd releases of dubious value
>>
>> * Keep even/odd pattern, but force a 'train' model with a more regular
>> cadence
>>
>>   + predictable schedule
>>   - some features will miss the target and be delayed a year
>>
>> * Drop the odd releases but change nothing else (i.e., 12-month release
>> cadence)
>>
>>   + eliminate the confusing odd releases with dubious value
>>
>> * Drop the odd releases, and aim for a ~9 month cadence. This splits the
>> difference between the current even/odd pattern we've been doing.
>>
>>   + eliminate the confusing odd releases with dubious value
>>   + waiting for the next release isn't quite as bad
>>   - required upgrades every 9 months instead of ever 12 months
>>
>> * Drop the odd releases, but relax the "must upgrade through every LTS" to
>> allow upgrades across 2 versions (e.g., luminous -> mimic or luminous ->
>> nautilus).  Shorten release cycle (~6-9 months).
>>
>>   + more flexibility for users
>>   + downstreams have greater choice in adopting an upstrema release
>>   - more LTS branches to maintain
>>   - more upgrade paths to consider
>>
>> Other options we should consider?  Other thoughts?
> 
> 
> As a mission critical system user, I am in favor of dropping odd releases
> and going to a 9 month cycle.  We never run the odd releases as too risky.
> A good deal if functionality comes in updates, and usually the Ceph team
> brings them in gently, with the more experimental features off by default.
> 
> I suspect the 9 month even cycle will also make it easier to perform more
> incremental upgrades, i.e. small jumps rather than big leaps.
> 
> 
> 
>>
>> Thanks!
>> sage
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>>
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release cadence

[Alex Gorbachev]

On Wed, Sep 6, 2017 at 11:23 AM Sage Weil  wrote:

> Hi everyone,
>
> Traditionally, we have done a major named "stable" release twice a year,
> and every other such release has been an "LTS" release, with fixes
> backported for 1-2 years.
>
> With kraken and luminous we missed our schedule by a lot: instead of
> releasing in October and April we released in January and August.
>
> A few observations:
>
> - Not a lot of people seem to run the "odd" releases (e.g., infernalis,
> kraken).  This limits the value of actually making them.  It also means
> that those who *do* run them are running riskier code (fewer users -> more
> bugs).
>
> - The more recent requirement that upgrading clusters must make a stop at
> each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel
> -> lumninous) has been hugely helpful on the development side by reducing
> the amount of cross-version compatibility code to maintain and reducing
> the number of upgrade combinations to test.
>
> - When we try to do a time-based "train" release cadence, there always
> seems to be some "must-have" thing that delays the release a bit.  This
> doesn't happen as much with the odd releases, but it definitely happens
> with the LTS releases.  When the next LTS is a year away, it is hard to
> suck it up and wait that long.
>
> A couple of options:
>
> * Keep even/odd pattern, and continue being flexible with release dates
>
>   + flexible
>   - unpredictable
>   - odd releases of dubious value
>
> * Keep even/odd pattern, but force a 'train' model with a more regular
> cadence
>
>   + predictable schedule
>   - some features will miss the target and be delayed a year
>
> * Drop the odd releases but change nothing else (i.e., 12-month release
> cadence)
>
>   + eliminate the confusing odd releases with dubious value
>
> * Drop the odd releases, and aim for a ~9 month cadence. This splits the
> difference between the current even/odd pattern we've been doing.
>
>   + eliminate the confusing odd releases with dubious value
>   + waiting for the next release isn't quite as bad
>   - required upgrades every 9 months instead of ever 12 months
>
> * Drop the odd releases, but relax the "must upgrade through every LTS" to
> allow upgrades across 2 versions (e.g., luminous -> mimic or luminous ->
> nautilus).  Shorten release cycle (~6-9 months).
>
>   + more flexibility for users
>   + downstreams have greater choice in adopting an upstrema release
>   - more LTS branches to maintain
>   - more upgrade paths to consider
>
> Other options we should consider?  Other thoughts?


As a mission critical system user, I am in favor of dropping odd releases
and going to a 9 month cycle.  We never run the odd releases as too risky.
A good deal if functionality comes in updates, and usually the Ceph team
brings them in gently, with the more experimental features off by default.

I suspect the 9 month even cycle will also make it easier to perform more
incremental upgrades, i.e. small jumps rather than big leaps.



>
> Thanks!
> sage
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
-- 
--
Alex Gorbachev
Storcium
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release cadence

[Kingsley Tart]

On Wed, 2017-09-06 at 15:23 +, Sage Weil wrote:
> Hi everyone,
> 
> Traditionally, we have done a major named "stable" release twice a year, 
> and every other such release has been an "LTS" release, with fixes 
> backported for 1-2 years.
> 
> With kraken and luminous we missed our schedule by a lot: instead of 
> releasing in October and April we released in January and August.
> 
> A few observations:
[snip]

Firstly, I'd like to qualify my comments by saying that I haven't yet
tried Ceph[1], though I have been loosely following its progress. This
is partly because I've been busy doing other things.

[1] OK, this is a slight fib - I had a very brief play with it a few
years back but didn't really get anywhere with it and then got diverted
onto other things.

Unless I absolutely have to deploy now, I find myself doing this:

10 not long for new release, wait a bit
20 new release is here, but there's talk of a new one
30 goto 10

Having frequent minor updates and fixes is reassuring, but having
frequent major update changes with the "L" in "LTS" not being
particularly long tends to put me off a bit, largely because I find the
thought of upgrading something so mission critical quite daunting. I
can't speak from any Ceph experience on this one, obviously, but if
there's an easy rollback (even if it's never needed) without having to
rebuild the entire cluster then that would make me more willing to do
it.

-- 
Cheers,
Kingsley.

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Ceph release cadence

[Bryan Banister]

Very new to Ceph but long time but long time sys admin who is jaded/opinionated.

My 2 cents:
1) This sounds like a perfect thing to put in a poll and ask/beg people to 
vote.  Hopefully that will get you more of a response from a larger number of 
users.
2) Given that the value of the odd releases is "dubious", maybe those that are 
using these releases can give reason why they feel they need/want them?

I, personally, think having the even releases on a shorter cadence with the 
most users on each would be best, but I'm still new to this game,
-Bryan

-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Sage 
Weil
Sent: Wednesday, September 06, 2017 10:24 AM
To: ceph-de...@vger.kernel.org; ceph-maintain...@ceph.com; ceph-us...@ceph.com
Subject: [ceph-users] Ceph release cadence

Note: External Email
-

Hi everyone,

Traditionally, we have done a major named "stable" release twice a year,
and every other such release has been an "LTS" release, with fixes
backported for 1-2 years.

With kraken and luminous we missed our schedule by a lot: instead of
releasing in October and April we released in January and August.

A few observations:

- Not a lot of people seem to run the "odd" releases (e.g., infernalis,
kraken).  This limits the value of actually making them.  It also means
that those who *do* run them are running riskier code (fewer users -> more
bugs).

- The more recent requirement that upgrading clusters must make a stop at
each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel
-> lumninous) has been hugely helpful on the development side by reducing
the amount of cross-version compatibility code to maintain and reducing
the number of upgrade combinations to test.

- When we try to do a time-based "train" release cadence, there always
seems to be some "must-have" thing that delays the release a bit.  This
doesn't happen as much with the odd releases, but it definitely happens
with the LTS releases.  When the next LTS is a year away, it is hard to
suck it up and wait that long.

A couple of options:

* Keep even/odd pattern, and continue being flexible with release dates

  + flexible
  - unpredictable
  - odd releases of dubious value

* Keep even/odd pattern, but force a 'train' model with a more regular
cadence

  + predictable schedule
  - some features will miss the target and be delayed a year

* Drop the odd releases but change nothing else (i.e., 12-month release
cadence)

  + eliminate the confusing odd releases with dubious value

* Drop the odd releases, and aim for a ~9 month cadence. This splits the
difference between the current even/odd pattern we've been doing.

  + eliminate the confusing odd releases with dubious value
  + waiting for the next release isn't quite as bad
  - required upgrades every 9 months instead of ever 12 months

* Drop the odd releases, but relax the "must upgrade through every LTS" to
allow upgrades across 2 versions (e.g., luminous -> mimic or luminous ->
nautilus).  Shorten release cycle (~6-9 months).

  + more flexibility for users
  + downstreams have greater choice in adopting an upstrema release
  - more LTS branches to maintain
  - more upgrade paths to consider

Other options we should consider?  Other thoughts?

Thanks!
sage
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com




Note: This email is for the confidential use of the named addressee(s) only and 
may contain proprietary, confidential or privileged information. If you are not 
the intended recipient, you are hereby notified that any review, dissemination 
or copying of this email is strictly prohibited, and to please notify the 
sender immediately and destroy this email and any attachments. Email 
transmission cannot be guaranteed to be secure or error-free. The Company, 
therefore, does not make any guarantees as to the completeness or accuracy of 
this email or any attachments. This email is for informational purposes only 
and does not constitute a recommendation, offer, request or solicitation of any 
kind to buy, sell, subscribe, redeem or perform any type of transaction of a 
financial product.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Ceph release cadence

[Sage Weil]

Hi everyone,

Traditionally, we have done a major named "stable" release twice a year, 
and every other such release has been an "LTS" release, with fixes 
backported for 1-2 years.

With kraken and luminous we missed our schedule by a lot: instead of 
releasing in October and April we released in January and August.

A few observations:

- Not a lot of people seem to run the "odd" releases (e.g., infernalis, 
kraken).  This limits the value of actually making them.  It also means 
that those who *do* run them are running riskier code (fewer users -> more 
bugs).

- The more recent requirement that upgrading clusters must make a stop at 
each LTS (e.g., hammer -> luminous not supported, must go hammer -> jewel 
-> lumninous) has been hugely helpful on the development side by reducing 
the amount of cross-version compatibility code to maintain and reducing 
the number of upgrade combinations to test.

- When we try to do a time-based "train" release cadence, there always 
seems to be some "must-have" thing that delays the release a bit.  This 
doesn't happen as much with the odd releases, but it definitely happens 
with the LTS releases.  When the next LTS is a year away, it is hard to 
suck it up and wait that long.

A couple of options:

* Keep even/odd pattern, and continue being flexible with release dates

  + flexible
  - unpredictable
  - odd releases of dubious value

* Keep even/odd pattern, but force a 'train' model with a more regular 
cadence

  + predictable schedule
  - some features will miss the target and be delayed a year

* Drop the odd releases but change nothing else (i.e., 12-month release 
cadence)

  + eliminate the confusing odd releases with dubious value
 
* Drop the odd releases, and aim for a ~9 month cadence. This splits the 
difference between the current even/odd pattern we've been doing.

  + eliminate the confusing odd releases with dubious value
  + waiting for the next release isn't quite as bad
  - required upgrades every 9 months instead of ever 12 months

* Drop the odd releases, but relax the "must upgrade through every LTS" to 
allow upgrades across 2 versions (e.g., luminous -> mimic or luminous -> 
nautilus).  Shorten release cycle (~6-9 months).

  + more flexibility for users
  + downstreams have greater choice in adopting an upstrema release
  - more LTS branches to maintain
  - more upgrade paths to consider

Other options we should consider?  Other thoughts?

Thanks!
sage
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com