Re: [OpenStack-Infra] Adding ARM64 cloud to infra
On 23/02/18 05:35, Ian Wienand wrote: > On 02/02/2018 05:15 PM, Ian Wienand wrote: >> - Once that is done, it should be straight forward to add a >>nodepool-builder in the cloud and have it build images, and zuul >>should be able to launch them just like any other node (famous last >>words). > > This roughly turned out to be correct :) > > In short, we now have ready xenial arm64 based nodes. If you request > an ubuntu-xenial-arm64 node it should "just work" > > There are some caveats: > > - I have manually installed a diskimage-builder with the changes from >[1] downwards onto nb03.openstack.org. These need to be finalised >and a release tagged before we can remove nb03 from the emergency >file (just means, don't run puppet on it). Reviews welcome! > > - I want to merge [2] and related changes to expose the image build >logs, and also the webapp end-points so we can monitor active >nodes, etc. It will take some baby-sitting so I plan on doing this >next week. > > - We have mirror.cn1.linaro.openstack.org, but it's not mirroring >anything that useful for arm64. We need to sort out mirroring of >ubuntu ports, maybe some wheel builds, etc. > > - There's currently capacity for 8 nodes. So please take that into >account when adding jobs. > > Everything seems in good shape at the moment. For posterity, here is > the first ever arm64 ready node: > > nodepool@nl03:/var/log/nodepool$ nodepool list | grep arm64 > | 0002683657 | linaro-cn1 | ubuntu-xenial-arm64 | > c7bb6da6-52e5-4aab-88f1-ec0f1b392a0c | 211.148.24.200 | > | ready| 00:00:03:43 | unlocked | > > :) Thank you, for the update! This is awesome news and great work \o/ Cheers, Gema > > -i > > [1] https://review.openstack.org/547161 > [2] https://review.openstack.org/543671 > -- Gema Gomez-Solano Tech Lead, SDI Linaro Ltd IRC: gema@#linaro on irc.freenode.net ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Adding ARM64 cloud to infra
On 15/01/18 22:51, Ian Wienand wrote: > On 01/16/2018 12:11 AM, Frank Jansen wrote: >> do you have any insight into the availability of a physical >> environment for the ARM64 cloud? > >> I’m curious, as there may be a need for downstream testing, which I >> would assume will want to make use of our existing OSP CI framework. > > Sorry, not 100% sure what you mean here? I think the theory is that > this would be an ARM64 based cloud attached to OpenStack infra and > thus run any jobs infra could ... +1, this is the idea indeed. Gema > > -i > > > ___ > OpenStack-Infra mailing list > OpenStack-Infra@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
Re: [OpenStack-Infra] Adding ARM64 cloud to infra
On 12/01/18 00:28, Clark Boylan wrote: > On Wed, Jan 10, 2018, at 1:41 AM, Gema Gomez wrote: >> Hi all, >> >> Linaro would like to add a new cloud to infra so that we can run tests >> on ARM64 going forward. This discussion has been ongoing for the good >> part of a year, apologies that it took us so long to get to a point >> where we feel comfortable going ahead in terms of stability of >> infrastructure and functionality. >> >> My team has been taking care of making OpenStack as multiarch as >> possible and making the experience of using an ARM64 cloud as close to >> using a traditional amd64 one as possible. We have the Linaro Developer >> Cloud program, which consists of a set of clouds that run on ARM64 >> hardware donated by the Linaro Enterprise Group members[1] and dedicated >> to enablement/testing of upstream projects. Until recently our clouds >> were run by an engineering team and were used to do enablement of >> different projects and bug fixes of OpenStack, now we have a dedicated >> admin and we are ready to take it a step forward. The clouds are >> currently running OpenStack Newton but we are going to be moving to >> Queens as soon as the release is out. Kolla has volunteered to be the >> first project for this experiment, they have been pushing us to start >> doing CI on our images so they also feel more comfortable accepting our >> changes. We will welcome any other project that wants to be part of this >> experiment, but we'll focus our engineering efforts initially on >> enabling Kolla CI. >> >> After some preliminary discussion with fungi and inc0, we are going to >> start small and grow from there. The initial plan is to add 2 projects: >> >> 1. Control-plane project that will host a nodepool builder with 8 vCPUs, >> 8 GB RAM, 1TB storage on a Cinder volume for the image building scratch >> space. A cache server with similar specs + 200GB on a cinder volume for >> AFS and Apache proxy caches. They will have a routable IP address. >> >> 2. Jobs project, we'll have capacity for 6 test instances initially and >> after initial assessment grow it as required (8 vCPUs/8 GB RAM, 80GB >> storage, 1 routable IP each). >> >> Is there anything else we are missing for the initial trial? Any >> questions/concerns before we start? I will try to have presence on the >> infra weekly calls/IRC channel or have someone from my team on them >> going forward. > > This plan looks good to me. The one question I had on IRC (and putting it > here for historical reasons) is whether or not Andrew FileSystem (AFS) will > build and run on arm64. OpenAFS is not in the linux kernel tree so this may > not work. The good news is mtreinish reports that after a quick test on some > of his hardware AFS was working. > >> >> In practical terms, once we've created the resources, is there a guide >> to getting the infra bits in place for it? Who to give the credentials >> to/etc? > > New clouds happen infrequently enough and require a reasonable amount of > communication to get going so I don't think we have written down a guide > beyond what we have on the donating resources page [2]. > > Typically what happens is we'll have an infra root act as the contact point > to set things up, you'll provide them with credentials via email (or whatever > communication system is most convenient) then they will immediately change > the password(s). It is also helpful if we can get a contact individual for > the cloud side and we'll record that in our passwords file so that any one of > our infra roots knows who to talk to should the need arise. > > Once the initial credential exchange happens the next step is for that infra > root to double check quotas and get the mirror host up and running as well as > image builder (and images) built. Once that is done you should be ready to > push changes to projects that add jobs using the new nodepool labels > (something like ubuntu-xenial-arm64). Sounds good. Quick update, we are working on applying a patch (https://review.openstack.org/#/c/489951/) to our Newton deployment so that uploaded images do not require any extra parameters. Once that is done we can give infra credentials. Who will be our infra counterpart for that? We are also happy to add engineers to work on any fixes required to make the infra tools work as seamlessly as possible with ARM64. Cheers! Gema >> >> Thanks! >> Gema > > Thank you! this is exciting. > >> >> [1] https://www.linaro.org/groups/leg/ > [2] https://docs.openstack.org/infra/system-config/contribute-cloud.html > >
Re: [OpenStack-Infra] Adding ARM64 cloud to infra
On 12/01/18 15:49, Paul Belanger wrote: > On Fri, Jan 12, 2018 at 11:17:33AM +0100, Marcin Juszkiewicz wrote: >> Wu dniu 12.01.2018 o 01:09, Ian Wienand pisze: >>> On 01/10/2018 08:41 PM, Gema Gomez wrote: >>>> 1. Control-plane project that will host a nodepool builder with 8 vCPUs, >>>> 8 GB RAM, 1TB storage on a Cinder volume for the image building scratch >>>> space. >>> Does this mean you're planning on using diskimage-builder to produce >>> the images to run tests on? I've seen occasional ARM things come by, >>> but of course diskimage-builder doesn't have CI for it (yet :) so it's >>> status is probably "unknown". >> >> I had a quick look at diskimage-builder tool. >> >> It looks to me that you always build MBR based image with one partition. >> This will have to be changed as AArch64 is UEFI based platform (both >> baremetal and VM) so disk needs to use GPT for partitioning and EFI >> System Partition needs to be present (with grub-efi binary on it). >> > This is often the case when bringing new images online, that some changes to > DIB > will be required to support them. I suspect somebody with access to AArch64 > hardware will first need to run build-image.sh[1] and paste the build.log. > That > will build an image locally for you using our DIB elements. > > [1] > http://git.openstack.org/cgit/openstack-infra/project-config/tree/tools/build-image.sh Yep, that won't be an issue. Will do that on Monday. >> I am aware that you like to build disk images on your own but have you >> considered using virt-install with generated preseed/kickstart files? It >> would move several arch related things (like bootloader) to be handled >> by distribution rules instead of handling them again in code. >> > I don't believe we want to look at using a new tool to build all our images, > switching to virt-install would be a large change. There are reasons why we > build images from scratch and don't believe switching to virt-install help > with that >> >> Sent a patch to make it choose proper grub package on aarch64. >> >> ___ >> OpenStack-Infra mailing list >> OpenStack-Infra@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra > > ___ > OpenStack-Infra mailing list > OpenStack-Infra@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra > -- Gema Gomez-Solano Tech Lead, SDI Linaro Ltd IRC: gema@#linaro on irc.freenode.net ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[OpenStack-Infra] Adding ARM64 cloud to infra
Hi all, Linaro would like to add a new cloud to infra so that we can run tests on ARM64 going forward. This discussion has been ongoing for the good part of a year, apologies that it took us so long to get to a point where we feel comfortable going ahead in terms of stability of infrastructure and functionality. My team has been taking care of making OpenStack as multiarch as possible and making the experience of using an ARM64 cloud as close to using a traditional amd64 one as possible. We have the Linaro Developer Cloud program, which consists of a set of clouds that run on ARM64 hardware donated by the Linaro Enterprise Group members[1] and dedicated to enablement/testing of upstream projects. Until recently our clouds were run by an engineering team and were used to do enablement of different projects and bug fixes of OpenStack, now we have a dedicated admin and we are ready to take it a step forward. The clouds are currently running OpenStack Newton but we are going to be moving to Queens as soon as the release is out. Kolla has volunteered to be the first project for this experiment, they have been pushing us to start doing CI on our images so they also feel more comfortable accepting our changes. We will welcome any other project that wants to be part of this experiment, but we'll focus our engineering efforts initially on enabling Kolla CI. After some preliminary discussion with fungi and inc0, we are going to start small and grow from there. The initial plan is to add 2 projects: 1. Control-plane project that will host a nodepool builder with 8 vCPUs, 8 GB RAM, 1TB storage on a Cinder volume for the image building scratch space. A cache server with similar specs + 200GB on a cinder volume for AFS and Apache proxy caches. They will have a routable IP address. 2. Jobs project, we'll have capacity for 6 test instances initially and after initial assessment grow it as required (8 vCPUs/8 GB RAM, 80GB storage, 1 routable IP each). Is there anything else we are missing for the initial trial? Any questions/concerns before we start? I will try to have presence on the infra weekly calls/IRC channel or have someone from my team on them going forward. In practical terms, once we've created the resources, is there a guide to getting the infra bits in place for it? Who to give the credentials to/etc? Thanks! Gema [1] https://www.linaro.org/groups/leg/ -- Gema Gomez-Solano Tech Lead, SDI Linaro Ltd IRC: gema@#linaro on irc.freenode.net ___ OpenStack-Infra mailing list OpenStack-Infra@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
