[openstack-dev] [all][qa] Migrating devstack jobs to Bionic (Ubuntu LTS 18.04)
Dear OpenStackers,
earlier this year Ubuntu released their current LTS version 18.04
codenamed "Bionic Beaver" and we are now facing the task to migrate
our devstack-based jobs to run on Bionic instead of the previous LTS
version 16.04 "Xenial Xerus".
The last time this has happened two years ago (migration from 14.04 to
16.04) and at that time it seems the migration was mostly driven by
the Infra team (see [1]), mostly because all of the job configuration
was still centrally hosted in a single repository
(openstack-infra/project-config). In the meantime, however, our CI
setup has been updated to use Zuul v3 and one of the new features that
come with this development is the introduction of per-project job
definitions.
So this new flexibility requires us to make a choice between the two
possible options we have for migrating jobs now:
1) Change the "devstack" base job to run on Bionic instances
instead of Xenial instances
2) Create new "devstack-bionic" and "tempest-full-bionic" base
jobs and migrate projects piecewise
Choosing option 1) would cause all projects that base their own jobs
on this job (possibly indirectly by e.g. being based on the
"tempest-full" job) switch automatically. So there would be the
possibility that some jobs would break and require to be fixed before
patches could be merged again in the affected project(s). To
accomodate those risks, QA team can give some time to projects to test
their jobs on Bionic with WIP patches (QA can provide Bionic base job
as WIP patch). This option does not require any pre/post migration
changes on project's jobs.
Choosing option 2) would avoid this by letting projects switch at
their own pace, but create the risk that some projects would never
migrate. It would also make further migrations, like the one expected
to happen when 20.04 is released, either having to follow the same
scheme or re-introduce the unversioned base job. Other point to note
down with this option is,
- project job definitions need to change their parent job from
"devstack" -> "devstack-bionic" or "tempest-full" ->
"tempest-full-bionic"
- QA needs to maintain existing jobs ("devstack", "tempest-full") and
bionic version jobs ("devstack-bionic", "tempest-full-bionic")
In order to prepare the decision, we have created a set of patches
that test the Bionic
jobs, you can find them under the common topic "devstack-bionic" [2].
There is also an
etherpad to give a summarized view of the results of these tests [3].
Please respond to this mail if you want to promote either of the above
options or
maybe want to propose an even better solution. You can also find us
for discussion
in the #openstack-qa IRC channel on freenode.
Infra team has tried both approaches during precise->trusty &
trusty->xenial migration[4].
Note that this mailing-list itself will soon be migrated, too, so if
you haven't subscribed
to the new list yet, this is a good time to act and avoid missing the
best parts [5].
Yours,
Jens (frickler@IRC)
[1] http://lists.openstack.org/pipermail/openstack-dev/2016-November/106906.html
[2] https://review.openstack.org/#/q/topic:devstack-bionic
[3] https://etherpad.openstack.org/p/devstack-bionic
[4]
http://eavesdrop.openstack.org/irclogs/%23openstack-qa/%23openstack-qa.2018-11-01.log.html#t2018-11-01T12:40:22
[5]
http://lists.openstack.org/pipermail/openstack-dev/2018-September/134911.html
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][qa] Migrating devstack jobs to Bionic (Ubuntu LTS 18.04)
Hi,
Thanks for bringing this up.
On Tue, Nov 06, 2018 at 01:12:55PM +0100, Dr. Jens Harbott (frickler) wrote:
> Dear OpenStackers,
>
> earlier this year Ubuntu released their current LTS version 18.04
> codenamed "Bionic Beaver" and we are now facing the task to migrate
> our devstack-based jobs to run on Bionic instead of the previous LTS
> version 16.04 "Xenial Xerus".
>
> The last time this has happened two years ago (migration from 14.04 to
> 16.04) and at that time it seems the migration was mostly driven by
> the Infra team (see [1]), mostly because all of the job configuration
> was still centrally hosted in a single repository
> (openstack-infra/project-config). In the meantime, however, our CI
> setup has been updated to use Zuul v3 and one of the new features that
> come with this development is the introduction of per-project job
> definitions.
>
> So this new flexibility requires us to make a choice between the two
> possible options we have for migrating jobs now:
>
> 1) Change the "devstack" base job to run on Bionic instances
> instead of Xenial instances
> 2) Create new "devstack-bionic" and "tempest-full-bionic" base
> jobs and migrate projects piecewise
>
> Choosing option 1) would cause all projects that base their own jobs
> on this job (possibly indirectly by e.g. being based on the
> "tempest-full" job) switch automatically. So there would be the
> possibility that some jobs would break and require to be fixed before
> patches could be merged again in the affected project(s). To
> accomodate those risks, QA team can give some time to projects to test
> their jobs on Bionic with WIP patches (QA can provide Bionic base job
> as WIP patch). This option does not require any pre/post migration
> changes on project's jobs.
>
> Choosing option 2) would avoid this by letting projects switch at
> their own pace, but create the risk that some projects would never
> migrate. It would also make further migrations, like the one expected
> to happen when 20.04 is released, either having to follow the same
> scheme or re-introduce the unversioned base job. Other point to note
> down with this option is,
>- project job definitions need to change their parent job from
> "devstack" -> "devstack-bionic" or "tempest-full" ->
> "tempest-full-bionic"
> - QA needs to maintain existing jobs ("devstack", "tempest-full") and
> bionic version jobs ("devstack-bionic", "tempest-full-bionic")
How about option 1) and also add jobs like "devstack-xenial" and
"tempest-full-xenial" which projects can use still for some time if their job on
Bionic would be broken now?
>
> In order to prepare the decision, we have created a set of patches
> that test the Bionic
> jobs, you can find them under the common topic "devstack-bionic" [2].
> There is also an
> etherpad to give a summarized view of the results of these tests [3].
>
> Please respond to this mail if you want to promote either of the above
> options or
> maybe want to propose an even better solution. You can also find us
> for discussion
> in the #openstack-qa IRC channel on freenode.
>
> Infra team has tried both approaches during precise->trusty &
> trusty->xenial migration[4].
>
> Note that this mailing-list itself will soon be migrated, too, so if
> you haven't subscribed
> to the new list yet, this is a good time to act and avoid missing the
> best parts [5].
>
> Yours,
> Jens (frickler@IRC)
>
>
> [1]
> http://lists.openstack.org/pipermail/openstack-dev/2016-November/106906.html
> [2] https://review.openstack.org/#/q/topic:devstack-bionic
> [3] https://etherpad.openstack.org/p/devstack-bionic
> [4]
> http://eavesdrop.openstack.org/irclogs/%23openstack-qa/%23openstack-qa.2018-11-01.log.html#t2018-11-01T12:40:22
> [5]
> http://lists.openstack.org/pipermail/openstack-dev/2018-September/134911.html
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
Slawek Kaplonski
Senior software engineer
Red Hat
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][qa] Migrating devstack jobs to Bionic (Ubuntu LTS 18.04)
On 2018-11-06 22:05:49 +0100 (+0100), Slawek Kaplonski wrote: [...] > also add jobs like "devstack-xenial" and "tempest-full-xenial" > which projects can use still for some time if their job on Bionic > would be broken now? [...] That opens the door to piecemeal migration, which (as we similarly saw during the Trusty to Xenial switch) will inevitably lead to projects who no longer gate on Xenial being unable to integration test against projects who don't yet support Bionic. At the same time, projects which have switched to Bionic will start merging changes which only work on Bionic without realizing it, so that projects which test on Xenial can't use them. In short, you'll be broken either way. On top of that, you can end up with projects that don't get around to switching completely before release comes, and then they're stuck having to manage a test platform transition on a stable branch. -- Jeremy Stanley signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][qa] Migrating devstack jobs to Bionic (Ubuntu LTS 18.04)
Hi, > Wiadomość napisana przez Jeremy Stanley w dniu > 06.11.2018, o godz. 22:25: > > On 2018-11-06 22:05:49 +0100 (+0100), Slawek Kaplonski wrote: > [...] >> also add jobs like "devstack-xenial" and "tempest-full-xenial" >> which projects can use still for some time if their job on Bionic >> would be broken now? > [...] > > That opens the door to piecemeal migration, which (as we similarly > saw during the Trusty to Xenial switch) will inevitably lead to > projects who no longer gate on Xenial being unable to integration > test against projects who don't yet support Bionic. At the same > time, projects which have switched to Bionic will start merging > changes which only work on Bionic without realizing it, so that > projects which test on Xenial can't use them. In short, you'll be > broken either way. On top of that, you can end up with projects that > don't get around to switching completely before release comes, and > then they're stuck having to manage a test platform transition on a > stable branch. I understand Your point here but will option 2) from first email lead to the same issues then? > -- > Jeremy Stanley > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev — Slawek Kaplonski Senior software engineer Red Hat __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][qa] Migrating devstack jobs to Bionic (Ubuntu LTS 18.04)
Thanks Jens.
As most of the base jobs are in QA repo, QA team will coordinate this migration
based on either of the approach mentioned below.
Another point to note - This migration will only target the zuulv3 jobs not the
legacy jobs. legacy jobs owner should migrate them to bionic when they will be
moved to zuulv3 native. Any risk of keeping the legacy on xenial till zullv3 ?
Tempest testing patch found that stable queens/pike jobs failing for bionic due
to not supported distro in devstack[1]. Fixing in
https://review.openstack.org/#/c/616017/ and will backport to pike too.
[1] https://review.openstack.org/#/c/611572/
http://logs.openstack.org/72/611572/1/check/tempest-full-queens/7cd3f21/job-output.txt.gz#_2018-11-01_09_57_07_551538
-gmann
On Tue, 06 Nov 2018 21:12:55 +0900 Dr. Jens Harbott (frickler)
wrote
> Dear OpenStackers,
>
> earlier this year Ubuntu released their current LTS version 18.04
> codenamed "Bionic Beaver" and we are now facing the task to migrate
> our devstack-based jobs to run on Bionic instead of the previous LTS
> version 16.04 "Xenial Xerus".
>
> The last time this has happened two years ago (migration from 14.04 to
> 16.04) and at that time it seems the migration was mostly driven by
> the Infra team (see [1]), mostly because all of the job configuration
> was still centrally hosted in a single repository
> (openstack-infra/project-config). In the meantime, however, our CI
> setup has been updated to use Zuul v3 and one of the new features that
> come with this development is the introduction of per-project job
> definitions.
>
> So this new flexibility requires us to make a choice between the two
> possible options we have for migrating jobs now:
>
> 1) Change the "devstack" base job to run on Bionic instances
> instead of Xenial instances
> 2) Create new "devstack-bionic" and "tempest-full-bionic" base
> jobs and migrate projects piecewise
>
> Choosing option 1) would cause all projects that base their own jobs
> on this job (possibly indirectly by e.g. being based on the
> "tempest-full" job) switch automatically. So there would be the
> possibility that some jobs would break and require to be fixed before
> patches could be merged again in the affected project(s). To
> accomodate those risks, QA team can give some time to projects to test
> their jobs on Bionic with WIP patches (QA can provide Bionic base job
> as WIP patch). This option does not require any pre/post migration
> changes on project's jobs.
>
> Choosing option 2) would avoid this by letting projects switch at
> their own pace, but create the risk that some projects would never
> migrate. It would also make further migrations, like the one expected
> to happen when 20.04 is released, either having to follow the same
> scheme or re-introduce the unversioned base job. Other point to note
> down with this option is,
>- project job definitions need to change their parent job from
> "devstack" -> "devstack-bionic" or "tempest-full" ->
> "tempest-full-bionic"
> - QA needs to maintain existing jobs ("devstack", "tempest-full") and
> bionic version jobs ("devstack-bionic", "tempest-full-bionic")
>
> In order to prepare the decision, we have created a set of patches
> that test the Bionic
> jobs, you can find them under the common topic "devstack-bionic" [2].
> There is also an
> etherpad to give a summarized view of the results of these tests [3].
>
> Please respond to this mail if you want to promote either of the above
> options or
> maybe want to propose an even better solution. You can also find us
> for discussion
> in the #openstack-qa IRC channel on freenode.
>
> Infra team has tried both approaches during precise->trusty &
> trusty->xenial migration[4].
>
> Note that this mailing-list itself will soon be migrated, too, so if
> you haven't subscribed
> to the new list yet, this is a good time to act and avoid missing the
> best parts [5].
>
> Yours,
> Jens (frickler@IRC)
>
>
> [1]
> http://lists.openstack.org/pipermail/openstack-dev/2016-November/106906.html
> [2] https://review.openstack.org/#/q/topic:devstack-bionic
> [3] https://etherpad.openstack.org/p/devstack-bionic
> [4]
> http://eavesdrop.openstack.org/irclogs/%23openstack-qa/%23openstack-qa.2018-11-01.log.html#t2018-11-01T12:40:22
> [5]
> http://lists.openstack.org/pipermail/openstack-dev/2018-September/134911.html
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubs
Re: [openstack-dev] [all][qa] Migrating devstack jobs to Bionic (Ubuntu LTS 18.04)
On Tue, Nov 6, 2018, at 2:02 PM, Ghanshyam Mann wrote: > Thanks Jens. > > As most of the base jobs are in QA repo, QA team will coordinate this > migration based on either of the approach mentioned below. > > Another point to note - This migration will only target the zuulv3 jobs > not the legacy jobs. legacy jobs owner should migrate them to bionic > when they will be moved to zuulv3 native. Any risk of keeping the legacy > on xenial till zullv3 ? > > Tempest testing patch found that stable queens/pike jobs failing for > bionic due to not supported distro in devstack[1]. Fixing in > https://review.openstack.org/#/c/616017/ and will backport to pike too. The existing stable branches should continue to test on xenial as that is what they were built on. We aren't asking that everything be ported forward to bionic. Instead the idea is that current development (aka master) switch to bionic and roll forward from that point. This applies to tempest jobs, functional jobs, and unittests, etc. Xenial isn't going away. It is there for the stable branches. > > [1] https://review.openstack.org/#/c/611572/ > > http://logs.openstack.org/72/611572/1/check/tempest-full-queens/7cd3f21/job-output.txt.gz#_2018-11-01_09_57_07_551538 > > > > -gmann __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][qa] Migrating devstack jobs to Bionic (Ubuntu LTS 18.04)
On Wed, 07 Nov 2018 06:51:32 +0900 Slawomir Kaplonski wrote > Hi, > > > Wiadomość napisana przez Jeremy Stanley w dniu > > 06.11.2018, o godz. 22:25: > > > > On 2018-11-06 22:05:49 +0100 (+0100), Slawek Kaplonski wrote: > > [...] > >> also add jobs like "devstack-xenial" and "tempest-full-xenial" > >> which projects can use still for some time if their job on Bionic > >> would be broken now? > > [...] > > > > That opens the door to piecemeal migration, which (as we similarly > > saw during the Trusty to Xenial switch) will inevitably lead to > > projects who no longer gate on Xenial being unable to integration > > test against projects who don't yet support Bionic. At the same > > time, projects which have switched to Bionic will start merging > > changes which only work on Bionic without realizing it, so that > > projects which test on Xenial can't use them. In short, you'll be > > broken either way. On top of that, you can end up with projects that > > don't get around to switching completely before release comes, and > > then they're stuck having to manage a test platform transition on a > > stable branch. > > I understand Your point here but will option 2) from first email lead to the > same issues then? seems so. approach 1 is less risky for such integrated testing issues and requires less work. In approach 1, we can coordinate the base job migration with project side testing with bionic. -gmann > > > -- > > Jeremy Stanley > > __ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > — > Slawek Kaplonski > Senior software engineer > Red Hat > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][qa] Migrating devstack jobs to Bionic (Ubuntu LTS 18.04)
On Wed, 07 Nov 2018 07:07:33 +0900 Clark Boylan wrote > On Tue, Nov 6, 2018, at 2:02 PM, Ghanshyam Mann wrote: > > Thanks Jens. > > > > As most of the base jobs are in QA repo, QA team will coordinate this > > migration based on either of the approach mentioned below. > > > > Another point to note - This migration will only target the zuulv3 jobs > > not the legacy jobs. legacy jobs owner should migrate them to bionic > > when they will be moved to zuulv3 native. Any risk of keeping the legacy > > on xenial till zullv3 ? > > > > Tempest testing patch found that stable queens/pike jobs failing for > > bionic due to not supported distro in devstack[1]. Fixing in > > https://review.openstack.org/#/c/616017/ and will backport to pike too. > > The existing stable branches should continue to test on xenial as that is > what they were built on. We aren't asking that everything be ported forward > to bionic. Instead the idea is that current development (aka master) switch > to bionic and roll forward from that point. Make sense. Thanks. We can keep stable branch jobs running on Tempest master on xenial only. -gmann > > This applies to tempest jobs, functional jobs, and unittests, etc. Xenial > isn't going away. It is there for the stable branches. > > > > > [1] https://review.openstack.org/#/c/611572/ > > > > http://logs.openstack.org/72/611572/1/check/tempest-full-queens/7cd3f21/job-output.txt.gz#_2018-11-01_09_57_07_551538 > > > > > > > > -gmann > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][qa] Migrating devstack jobs to Bionic (Ubuntu LTS 18.04)
Ghanshyam Mann writes: > On Wed, 07 Nov 2018 06:51:32 +0900 Slawomir Kaplonski > wrote > > Hi, > > > > > Wiadomość napisana przez Jeremy Stanley w dniu > 06.11.2018, o godz. 22:25: > > > > > > On 2018-11-06 22:05:49 +0100 (+0100), Slawek Kaplonski wrote: > > > [...] > > >> also add jobs like "devstack-xenial" and "tempest-full-xenial" > > >> which projects can use still for some time if their job on Bionic > > >> would be broken now? > > > [...] > > > > > > That opens the door to piecemeal migration, which (as we similarly > > > saw during the Trusty to Xenial switch) will inevitably lead to > > > projects who no longer gate on Xenial being unable to integration > > > test against projects who don't yet support Bionic. At the same > > > time, projects which have switched to Bionic will start merging > > > changes which only work on Bionic without realizing it, so that > > > projects which test on Xenial can't use them. In short, you'll be > > > broken either way. On top of that, you can end up with projects that > > > don't get around to switching completely before release comes, and > > > then they're stuck having to manage a test platform transition on a > > > stable branch. > > > > I understand Your point here but will option 2) from first email lead to > the same issues then? > > seems so. approach 1 is less risky for such integrated testing issues and > requires less work. In approach 1, we can coordinate the base job migration > with project side testing with bionic. > > -gmann I like the approach of updating the devstack jobs to move everything to Bionic at one time because it sounds like it presents less risk of us ending up with something that looks like it works together but doesn't actually because it's tested on a different platform, as well as being less likely to cause us to have to do major porting work in stable branches after the release. We'll need to take the same approach when updating the version of Python 3 used inside of devstack. Doug __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
