Re: [openstack-dev] [all][infra] Zuul v3 migration update
Excerpts from Clark Boylan's message of 2017-09-28 18:40:41 -0700: > On Wed, Sep 27, 2017, at 03:24 PM, Monty Taylor wrote: > > Hey everybody, > > > > We're there. It's ready. > > > > We've worked through all of the migration script issues and are happy > > with the results. The cutover trigger is primed and ready to go. > > > > But as it's 21:51 UTC / 16:52 US Central it's a short day to be > > available to respond to the questions folks may have... so we're going > > to postpone one more day. > > > > Since it's all ready to go we'll be looking at flipping the switch first > > thing in the morning. (basically as soon as the West Coast wakes up and > > is ready to go) > > > > The project-config repo should still be considered frozen except for > > migration-related changes. Hopefully we'll be able to flip the final > > switch early tomorrow. > > > > If you haven't yet, please see [1] for information about the transition. > > > > [1] https://docs.openstack.org/infra/manual/zuulv3.html > > > > Its done! Except for all the work to make jobs run properly. Early today Woo! > (PDT) we converted everything over to our auto generated Zuulv3 config. > Since then we've been working to address problems in job configs. > > These problems include: > Missing inclusion of the requirements repo for constraints in some > jobs > Configuration of python35 unittest jobs in some cases > Use of sudo checking not working properly > Multinode jobs not having multinode nodesets > > Known issues we will continue to work on: > Multinode devstack and grenade jobs are not working quite right > Releasenote jobs not working due to use of origin/ refs in git This problem should be fixed by the reno enhancement in https://review.openstack.org/#/c/508324/, which I will release when I'm given the all-clear for the release tagging jobs. > It looks like we may not have job branch exclusions in place for all > cases > The zuul-cloner shim may not work in all cases. We are tracking down > and fixing the broken corner cases. > > Keep in mind that with things in flux, there is a good chance that > changes enqueued to the gate will fail. It is a good idea to check > recent check queue results before approving changes. > > I don't think we've found any deal breaker problems at this point. I am > sure there are many more than I have listed above. Please feel free to > ask us about any errors. For the adventurous, fixing problems is likely > a great way to get familiar with the new system. You'll want to start by > fixing errors in openstack-infra/openstack-zuul-jobs/playbooks/legacy. > Once that stabilizes the next step is writing native job configs within > your project tree. Documentation can be found at > https://docs.openstack.org/infra/manual/zuulv3.html. I expect we'll > spend the next few days ironing out the transition. > > Thank you for your patience, > Clark > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][infra] Zuul v3 migration update
On Fri, Sep 29, 2017 at 2:40 AM, Clark Boylan wrote: > On Wed, Sep 27, 2017, at 03:24 PM, Monty Taylor wrote: > > Hey everybody, > > > > We're there. It's ready. > > > > We've worked through all of the migration script issues and are happy > > with the results. The cutover trigger is primed and ready to go. > > > > But as it's 21:51 UTC / 16:52 US Central it's a short day to be > > available to respond to the questions folks may have... so we're going > > to postpone one more day. > > > > Since it's all ready to go we'll be looking at flipping the switch first > > thing in the morning. (basically as soon as the West Coast wakes up and > > is ready to go) > > > > The project-config repo should still be considered frozen except for > > migration-related changes. Hopefully we'll be able to flip the final > > switch early tomorrow. > > > > If you haven't yet, please see [1] for information about the transition. > > > > [1] https://docs.openstack.org/infra/manual/zuulv3.html > > > > Its done! Except for all the work to make jobs run properly. Early today > (PDT) we converted everything over to our auto generated Zuulv3 config. > Since then we've been working to address problems in job configs. > > These problems include: > Missing inclusion of the requirements repo for constraints in some > jobs > Configuration of python35 unittest jobs in some cases > Use of sudo checking not working properly > Multinode jobs not having multinode nodesets > > Known issues we will continue to work on: > Multinode devstack and grenade jobs are not working quite right > Releasenote jobs not working due to use of origin/ refs in git > It looks like we may not have job branch exclusions in place for all > cases > The zuul-cloner shim may not work in all cases. We are tracking down > and fixing the broken corner cases. > > Keep in mind that with things in flux, there is a good chance that > changes enqueued to the gate will fail. It is a good idea to check > recent check queue results before approving changes. > > I don't think we've found any deal breaker problems at this point. I am > sure there are many more than I have listed above. Please feel free to > ask us about any errors. For the adventurous, fixing problems is likely > a great way to get familiar with the new system. You'll want to start by > fixing errors in openstack-infra/openstack-zuul-jobs/playbooks/legacy. > Once that stabilizes the next step is writing native job configs within > your project tree. Documentation can be found at > https://docs.openstack.org/infra/manual/zuulv3.html. I expect we'll > spend the next few days ironing out the transition. > > Thank you for your patience, > Clark > > Hi, We have a couple of failures in Bandit's gvate I wanted to flag up (fungi believes it might be from missing openstack/requirements as a required repo in the job definition): Glance Store: http://logs.openstack.org/44/504544/6/check/legacy-bandit-integration-glance_store/900f88a/job-output.txt.gz Magnum client: http://logs.openstack.org/44/504544/6/check/legacy-bandit-integration-python-magnumclient/ba50a54/job-output.txt.gz Thanks, and do let me know if I need to do anything on Bandits side. Luke __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][infra] Zuul v3 migration update
On Wed, Sep 27, 2017, at 03:24 PM, Monty Taylor wrote: > Hey everybody, > > We're there. It's ready. > > We've worked through all of the migration script issues and are happy > with the results. The cutover trigger is primed and ready to go. > > But as it's 21:51 UTC / 16:52 US Central it's a short day to be > available to respond to the questions folks may have... so we're going > to postpone one more day. > > Since it's all ready to go we'll be looking at flipping the switch first > thing in the morning. (basically as soon as the West Coast wakes up and > is ready to go) > > The project-config repo should still be considered frozen except for > migration-related changes. Hopefully we'll be able to flip the final > switch early tomorrow. > > If you haven't yet, please see [1] for information about the transition. > > [1] https://docs.openstack.org/infra/manual/zuulv3.html > Its done! Except for all the work to make jobs run properly. Early today (PDT) we converted everything over to our auto generated Zuulv3 config. Since then we've been working to address problems in job configs. These problems include: Missing inclusion of the requirements repo for constraints in some jobs Configuration of python35 unittest jobs in some cases Use of sudo checking not working properly Multinode jobs not having multinode nodesets Known issues we will continue to work on: Multinode devstack and grenade jobs are not working quite right Releasenote jobs not working due to use of origin/ refs in git It looks like we may not have job branch exclusions in place for all cases The zuul-cloner shim may not work in all cases. We are tracking down and fixing the broken corner cases. Keep in mind that with things in flux, there is a good chance that changes enqueued to the gate will fail. It is a good idea to check recent check queue results before approving changes. I don't think we've found any deal breaker problems at this point. I am sure there are many more than I have listed above. Please feel free to ask us about any errors. For the adventurous, fixing problems is likely a great way to get familiar with the new system. You'll want to start by fixing errors in openstack-infra/openstack-zuul-jobs/playbooks/legacy. Once that stabilizes the next step is writing native job configs within your project tree. Documentation can be found at https://docs.openstack.org/infra/manual/zuulv3.html. I expect we'll spend the next few days ironing out the transition. Thank you for your patience, Clark __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [all][infra] Zuul v3 migration update
Hey everybody, We're there. It's ready. We've worked through all of the migration script issues and are happy with the results. The cutover trigger is primed and ready to go. But as it's 21:51 UTC / 16:52 US Central it's a short day to be available to respond to the questions folks may have... so we're going to postpone one more day. Since it's all ready to go we'll be looking at flipping the switch first thing in the morning. (basically as soon as the West Coast wakes up and is ready to go) The project-config repo should still be considered frozen except for migration-related changes. Hopefully we'll be able to flip the final switch early tomorrow. If you haven't yet, please see [1] for information about the transition. [1] https://docs.openstack.org/infra/manual/zuulv3.html Thanks, Monty __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][infra] Zuul v3 migration update
On 09/27/2017 03:49 AM, Flavio Percoco wrote: Just wanted to say thanks to all of you for the hard work. I can only imagine how hard it must be to do this migration without causing downtimes. +1000 Thank you so much for the hard work the infra team has put into making this migration as painless for the community as possible. Your efforts have certainly not gone unnoticed. All the best, -jay __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [all][infra] Zuul v3 migration update
Just wanted to say thanks to all of you for the hard work. I can only imagine how hard it must be to do this migration without causing downtimes. Flavio On 26/09/17 18:04 -0500, Monty Taylor wrote: Hey everybody, We got significantly further along with our Zuul v3 rollout today. We uncovered some fun bugs in the migration but were able to fix most of them rather quickly. We've pretty much run out of daylight though for the majority of the team and there is a tricky zuul-cloner related issue to deal with, so we're not going to push things further tonight. We're leaving most of today's work in place, having gotten far enough that we feel comfortable not rolling back. The project-config repo should still be considered frozen except for migration-related changes. Hopefully we'll be able to flip the final switch early tomorrow. If you haven't yet, please see [1] for information about the transition. [1] https://docs.openstack.org/infra/manual/zuulv3.html Thanks, Monty __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- @flaper87 Flavio Percoco signature.asc Description: PGP signature __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [all][infra] Zuul v3 migration update
Hey everybody, We got significantly further along with our Zuul v3 rollout today. We uncovered some fun bugs in the migration but were able to fix most of them rather quickly. We've pretty much run out of daylight though for the majority of the team and there is a tricky zuul-cloner related issue to deal with, so we're not going to push things further tonight. We're leaving most of today's work in place, having gotten far enough that we feel comfortable not rolling back. The project-config repo should still be considered frozen except for migration-related changes. Hopefully we'll be able to flip the final switch early tomorrow. If you haven't yet, please see [1] for information about the transition. [1] https://docs.openstack.org/infra/manual/zuulv3.html Thanks, Monty __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [all][infra] Zuul v3 migration update
Hi, We started to migrate to Zuul v3 today, but ran into some problems which delayed us sufficiently that we're going to suspend activity until tomorrow. In the mean time, please consider the project-config repository frozen to all changes except those which are necessary for the migration. We should be able to lift this freeze as soon as we finish the migration. If you haven't yet, please see [1] for information about the transition. [1] https://docs.openstack.org/infra/manual/zuulv3.html Thanks, Jim __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev