Re: [OpenStack-Infra] About aarch64 third party CI

2017-06-12 Thread Xinliang Liu 
On 13 June 2017 at 01:00, Ricardo Carrillo Cruz <
ricardo.carrillo.c...@gmail.com> wrote:

>
>
> 2017-06-09 22:18 GMT+02:00 Paul Belanger :
>
>> On Fri, Jun 09, 2017 at 07:58:44PM +, Jeremy Stanley wrote:
>> > On 2017-06-07 14:26:10 +0800 (+0800), Xinliang Liu wrote:
>> > [...]
>> > > we already have our own pre-built debian cloud image, could I just
>> > > use it and not use the one built by diskimage-builder?
>> > [...]
>> >
>> > The short answer is that nodepool doesn't currently have support for
>> > directly using an image provided independent of its own image build
>> > process. Clark was suggesting[*] in IRC today that it might be
>> > possible to inject records into Zookeeper (acting as a "fake"
>> > nodepool-builder daemon basically) to accomplish this, but nobody
>> > has yet implemented such a solution to our knowledge.
>> >
>> > Longer term, I think we do want a feature in nodepool to be able to
>> > specify the ID of a prebuilt image for a label/provider (at least we
>> > discussed that we wouldn't reject the idea if someone proposed a
>> > suitable implementation). Just be aware that nodepool's use of
>> > diskimage-builder to regularly rebuild images is intentional and
>> > useful since it ensures images are updated with the latest packages,
>> > kernels, warm caches and whatever else you specify in your elements
>> > so reducing job runtimes as they spend less effort updating these
>> > things on every run.
>> >
>> > [*] http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%
>> 23openstack-infra.2017-06-09.log.html#t2017-06-09T15:32:27-2 >
>> > --
>> > Jeremy Stanley
>>
>> Actually, I think 458073[1] aims to fix this use case.  I haven't tired it
>> myself but it adds support for using images which are not built and
>> managed by
>> nodepool.
>>
>> This is currently only on feature/zuulv3 branch.
>>
>> [1] https://review.openstack.org/#/c/458073/
>>
>>
> That's right, support for cloud-images on feature/zuulv3 is now merged and
> working.
> I just setup a Nodepool using this new feature over the weekend.
>
> This is a nodepool.yaml that can help you get going:
>
> http://paste.openstack.org/show/612191/
>

​Great! I will try and see what happen.

Thanks,
-xinliang
​

>
>
> HTH
>
>
>> > ___
>> > OpenStack-Infra mailing list
>> > OpenStack-Infra@lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>
>>
>> ___
>> OpenStack-Infra mailing list
>> OpenStack-Infra@lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
>
>
> ___
> OpenStack-Infra mailing list
> OpenStack-Infra@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] About aarch64 third party CI

2017-06-12 Thread Xinliang Liu 
Hi Jeremy,
Thanks for reply;-)

On 10 June 2017 at 03:58, Jeremy Stanley  wrote:

> On 2017-06-07 14:26:10 +0800 (+0800), Xinliang Liu wrote:
> [...]
> > we already have our own pre-built debian cloud image, could I just
> > use it and not use the one built by diskimage-builder?
> [...]
>
> The short answer is that nodepool doesn't currently have support for
> directly using an image provided independent of its own image build
> process. Clark was suggesting[*] in IRC today that it might be
> possible to inject records into Zookeeper (acting as a "fake"
> nodepool-builder daemon basically) to accomplish this, but nobody
> has yet implemented such a solution to our knowledge.
>

​Got it, thanks.


>
> Longer term, I think we do want a feature in nodepool to be able to
> specify the ID of a prebuilt image for a label/provider (at least we
> discussed that we wouldn't reject the idea if someone proposed a
> suitable implementation). Just be aware that nodepool's use of
> diskimage-builder to regularly rebuild images is intentional and
> useful since it ensures images are updated with the latest packages,
>

I think this is the point that make a really update gating image to run the
tests.
​

> kernels, warm caches and whatever else you specify in your elements
> so reducing job runtimes as they spend less effort updating these
> things on every run.
>

​How often nodepool will rebuild the images?
If it too frequency every day. ​Shall we just make a job to publish the
​pre-built gating images every day then other CI test just use them
(something like docker image, though it is container image)?
You know making a gating image need to include lots of elements, even
though with warm caches, when using diskimage-builder
we still need to rebuild step by step. What I mean is that building image
is taking a bit time.

Thanks,
-xinliang



>
> [*] http://eavesdrop.openstack.org/irclogs/%23openstack-
> infra/%23openstack-infra.2017-06-09.log.html#t2017-06-09T15:32:27-2 >
> --
> Jeremy Stanley
>
> ___
> OpenStack-Infra mailing list
> OpenStack-Infra@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] About aarch64 third party CI

2017-06-12 Thread James E. Blair 
Ricardo Carrillo Cruz  writes:

> This is a nodepool.yaml that can help you get going:
>
> http://paste.openstack.org/show/612191/

Glad it worked!

You can drop 'zmq-publishers' from the config entirely.

If 'images-dir' and 'diskimages' are required, then I would consider
that a bug; we should have default values for those so you don't need to
provide them in this case.

That config snippet also illustrates something I didn't quite realize at
the time I reviewed https://review.openstack.org/472959.  I don't think
we should be using UUIDs as keys in nodepool because they are hard for
humans to distinguish from each other.  It could make for somewhat
error-prone configuration.

So instead of:

cloud-images:
  - name: 9e884aab-a46e-46de-b57c-a044da0f45cd
pools:
  - name: main
labels:
  - name: xenial
cloud-image: 9e884aab-a46e-46de-b57c-a044da0f45cd

If someone wants to specify an image by id, we should have:

cloud-images:
  - name: mycloudimagename
id: 9e884aab-a46e-46de-b57c-a044da0f45cd
pools:
  - name: main
labels:
  - name: xenial
cloud-image: mycloudimagename

And then if you omit the 'id' field, we should just implicitly use
'name' as before.  This way it's easy to see which of several
cloud-images a label uses, and, when it's time to update the UUID for
that cloud image, that only needs to happen in one place.

-Jim

___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] About aarch64 third party CI

2017-06-12 Thread Ricardo Carrillo Cruz 
2017-06-09 22:18 GMT+02:00 Paul Belanger :

> On Fri, Jun 09, 2017 at 07:58:44PM +, Jeremy Stanley wrote:
> > On 2017-06-07 14:26:10 +0800 (+0800), Xinliang Liu wrote:
> > [...]
> > > we already have our own pre-built debian cloud image, could I just
> > > use it and not use the one built by diskimage-builder?
> > [...]
> >
> > The short answer is that nodepool doesn't currently have support for
> > directly using an image provided independent of its own image build
> > process. Clark was suggesting[*] in IRC today that it might be
> > possible to inject records into Zookeeper (acting as a "fake"
> > nodepool-builder daemon basically) to accomplish this, but nobody
> > has yet implemented such a solution to our knowledge.
> >
> > Longer term, I think we do want a feature in nodepool to be able to
> > specify the ID of a prebuilt image for a label/provider (at least we
> > discussed that we wouldn't reject the idea if someone proposed a
> > suitable implementation). Just be aware that nodepool's use of
> > diskimage-builder to regularly rebuild images is intentional and
> > useful since it ensures images are updated with the latest packages,
> > kernels, warm caches and whatever else you specify in your elements
> > so reducing job runtimes as they spend less effort updating these
> > things on every run.
> >
> > [*] http://eavesdrop.openstack.org/irclogs/%23openstack-
> infra/%23openstack-infra.2017-06-09.log.html#t2017-06-09T15:32:27-2 >
> > --
> > Jeremy Stanley
>
> Actually, I think 458073[1] aims to fix this use case.  I haven't tired it
> myself but it adds support for using images which are not built and
> managed by
> nodepool.
>
> This is currently only on feature/zuulv3 branch.
>
> [1] https://review.openstack.org/#/c/458073/
>
>
That's right, support for cloud-images on feature/zuulv3 is now merged and
working.
I just setup a Nodepool using this new feature over the weekend.

This is a nodepool.yaml that can help you get going:

http://paste.openstack.org/show/612191/

HTH


> > ___
> > OpenStack-Infra mailing list
> > OpenStack-Infra@lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
>
> ___
> OpenStack-Infra mailing list
> OpenStack-Infra@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

Re: [OpenStack-Infra] About aarch64 third party CI

2017-06-09 Thread Paul Belanger 
On Fri, Jun 09, 2017 at 07:58:44PM +, Jeremy Stanley wrote:
> On 2017-06-07 14:26:10 +0800 (+0800), Xinliang Liu wrote:
> [...]
> > we already have our own pre-built debian cloud image, could I just
> > use it and not use the one built by diskimage-builder?
> [...]
> 
> The short answer is that nodepool doesn't currently have support for
> directly using an image provided independent of its own image build
> process. Clark was suggesting[*] in IRC today that it might be
> possible to inject records into Zookeeper (acting as a "fake"
> nodepool-builder daemon basically) to accomplish this, but nobody
> has yet implemented such a solution to our knowledge.
> 
> Longer term, I think we do want a feature in nodepool to be able to
> specify the ID of a prebuilt image for a label/provider (at least we
> discussed that we wouldn't reject the idea if someone proposed a
> suitable implementation). Just be aware that nodepool's use of
> diskimage-builder to regularly rebuild images is intentional and
> useful since it ensures images are updated with the latest packages,
> kernels, warm caches and whatever else you specify in your elements
> so reducing job runtimes as they spend less effort updating these
> things on every run.
> 
> [*]  http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2017-06-09.log.html#t2017-06-09T15:32:27-2
>  >
> -- 
> Jeremy Stanley

Actually, I think 458073[1] aims to fix this use case.  I haven't tired it
myself but it adds support for using images which are not built and managed by
nodepool.

This is currently only on feature/zuulv3 branch.

[1] https://review.openstack.org/#/c/458073/

> ___
> OpenStack-Infra mailing list
> OpenStack-Infra@lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra


___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra

[OpenStack-Infra] About aarch64 third party CI

2017-06-07 Thread Xinliang Liu 
Hi,
This is xinliang from Linaro.

I am now setting up the aarch64 third party CI in an aarch64 cloud.
I create two ubunu16.04 vms, one for CI server and one for log server.
And  follow the steps from here[1]

I am now at the step: start nodepool
I realize that nodepool use diskimage-builder to build devstack gating
images.
I now manage to modify diskimage-builder to build a debian-minimal image
and trying to make a gating image this means add more elements into the
image.

The question is that we already have our own pre-built debian cloud image,
could I just use it and not use the one built by diskimage-builder?

[1] https://docs.openstack.org/infra/openstackci/third_party_ci.html.
[2] https://github.com/openstack/diskimage-builder

Best,
-xinliang
___
OpenStack-Infra mailing list
OpenStack-Infra@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra