Re: [openstack-dev] [ironic][nova][neutron][qa] Injecting code into grenade resource create phase

[Villalovos, John L]

> From: Jim Rollenhagen [mailto:j...@jimrollenhagen.com]
> On Thu, May 12, 2016 at 06:48:02AM -0400, Sean Dague wrote:
> > Let me make sure I understand the crux of the issue: services may need
> > to create additional custom networking in order for the rest of the
> > resource flow to work correctly. Ironic definitely needs this because
> > it's building up an interesting simulated network here for it's guests
> > to live on.
> >
> > That feels like we need a new pre_create (or early_create) phase that
> > ironic could hook into to do that same network create. I think the only
> > question in my mind is whether neutrons network create here should
> move
> > to early_create as well.
> 
> This would work well, if we moved neutron's net-create to here,
> because...
> 
> > Does ironic's ovs setup need to happen after the neutron create phase
> > here, or does it just need to happen before nova?
> 
> Yes, the network needs to already exist when we do this.
> 
> > Does ironic need to overwrite the $net_id that gets passed to Nova later?
> 
> Nope, it's the same ID. We just get the OVS tap device for that network
> and muck with it a bit. Then we set the shared property of the network:
> 
> neutron net-update $ironic_net_id --shared true
> 
> FWIW, to get the tap device, we do need to create a port on the network
> (which we later delete), but I don't think that has any effect on what
> you describe above.

As a note, a patch to Grenade has been posted to add an 'early_create' phase:
https://review.openstack.org/315501



__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][nova][neutron][qa] Injecting code into grenade resource create phase

[Jim Rollenhagen]

On Thu, May 12, 2016 at 10:51:57AM +0300, Vasyl Saienko wrote:
> Hello Jim,
> 
> Thanks for rising this question.
> 
> My personal feeling that we don't need to tune tests that won't pass due to
> design limitations.
> It is Ironic pre-requirement to have network access from control plane to
> server during provisioning.
> Until Ironic Neutron integration is completed we may skip this test.

Well, this isn't a tempest test or anything we can just skip. This is
part of grenade's workflow to do upgrades. It goes:

* run smoke tests on $old
* create some resources to persist across upgrades (this ensures we
  don't break existing resources when upgrading)
* upgrade to $new
* run more tempest tests

We can't just skip the resource create phase here.

// jim

> 
> On Thu, May 12, 2016 at 12:53 AM, Jim Rollenhagen 
> wrote:
> 
> > I've ran into a bit of a wedge working on ironic grenade tests.
> >
> > In a normal dsvm run, the ironic setup taps the control plane into the
> > tenant network (as that's how it's currently intended to be deployed).
> > That code is here[0].
> >
> > However, in a grenade run, during the resource create phase, a new
> > network is created. This happens in the neutron bits[1], and is used to
> > boot a server in the nova bits[2].
> >
> > Since the control plane can't communicate with the machine on that
> > network, our ramdisk doesn't reach back to ironic after booting up, and
> > provisioning fails[3][4].
> >
> > Curious if any grenade experts have thoughts on how we might be able to
> > set up that tap in between the neutron and nova resource creation.
> >
> > One alternative I've considered is a method to have nova resource
> > creation not boot an instance, and replicate that functionality in the
> > ironic plugin, after we tap into that network.
> >
> > I'm sure there's other alternatives here that I haven't thought of;
> > suggestions welcome. Thanks in advance. :)
> >
> > // jim
> >
> > [0]
> > https://github.com/openstack/ironic/blob/95ff5badbdea0898d7877e651893916008561760/devstack/lib/ironic#L653
> > [1]
> > https://github.com/openstack-dev/grenade/blob/fce63f40d21abea926d343e9cddd620e3f03684a/projects/50_neutron/resources.sh#L34
> > [2]
> > https://github.com/openstack-dev/grenade/blob/fce63f40d21abea926d343e9cddd620e3f03684a/projects/60_nova/resources.sh#L79
> > [3]
> > http://logs.openstack.org/65/311865/3/experimental/gate-grenade-dsvm-ironic/e635dec/logs/grenade.sh.txt.gz#_2016-05-10_18_10_03_648
> > [4]
> > http://logs.openstack.org/65/311865/3/experimental/gate-grenade-dsvm-ironic/e635dec/logs/old/screen-ir-cond.txt.gz?level=WARNING
> >
> > __
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >

> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][nova][neutron][qa] Injecting code into grenade resource create phase

[Jim Rollenhagen]

On Thu, May 12, 2016 at 06:48:02AM -0400, Sean Dague wrote:
> On 05/11/2016 05:53 PM, Jim Rollenhagen wrote:
> > I've ran into a bit of a wedge working on ironic grenade tests.
> > 
> > In a normal dsvm run, the ironic setup taps the control plane into the
> > tenant network (as that's how it's currently intended to be deployed).
> > That code is here[0].
> > 
> > However, in a grenade run, during the resource create phase, a new
> > network is created. This happens in the neutron bits[1], and is used to
> > boot a server in the nova bits[2].
> > 
> > Since the control plane can't communicate with the machine on that
> > network, our ramdisk doesn't reach back to ironic after booting up, and
> > provisioning fails[3][4].
> > 
> > Curious if any grenade experts have thoughts on how we might be able to
> > set up that tap in between the neutron and nova resource creation.
> > 
> > One alternative I've considered is a method to have nova resource
> > creation not boot an instance, and replicate that functionality in the
> > ironic plugin, after we tap into that network.
> > 
> > I'm sure there's other alternatives here that I haven't thought of;
> > suggestions welcome. Thanks in advance. :)
> > 
> > // jim
> > 
> > [0] 
> > https://github.com/openstack/ironic/blob/95ff5badbdea0898d7877e651893916008561760/devstack/lib/ironic#L653
> > [1] 
> > https://github.com/openstack-dev/grenade/blob/fce63f40d21abea926d343e9cddd620e3f03684a/projects/50_neutron/resources.sh#L34
> > [2] 
> > https://github.com/openstack-dev/grenade/blob/fce63f40d21abea926d343e9cddd620e3f03684a/projects/60_nova/resources.sh#L79
> > [3] 
> > http://logs.openstack.org/65/311865/3/experimental/gate-grenade-dsvm-ironic/e635dec/logs/grenade.sh.txt.gz#_2016-05-10_18_10_03_648
> > [4] 
> > http://logs.openstack.org/65/311865/3/experimental/gate-grenade-dsvm-ironic/e635dec/logs/old/screen-ir-cond.txt.gz?level=WARNING
> 
> Let me make sure I understand the crux of the issue: services may need
> to create additional custom networking in order for the rest of the
> resource flow to work correctly. Ironic definitely needs this because
> it's building up an interesting simulated network here for it's guests
> to live on.
> 
> That feels like we need a new pre_create (or early_create) phase that
> ironic could hook into to do that same network create. I think the only
> question in my mind is whether neutrons network create here should move
> to early_create as well.

This would work well, if we moved neutron's net-create to here,
because...

> Does ironic's ovs setup need to happen after the neutron create phase
> here, or does it just need to happen before nova?

Yes, the network needs to already exist when we do this.

> Does ironic need to overwrite the $net_id that gets passed to Nova later?

Nope, it's the same ID. We just get the OVS tap device for that network
and muck with it a bit. Then we set the shared property of the network:

neutron net-update $ironic_net_id --shared true

FWIW, to get the tap device, we do need to create a port on the network
(which we later delete), but I don't think that has any effect on what
you describe above.

// jim

> 
>   -Sean
> 
> -- 
> Sean Dague
> http://dague.net
> 
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][nova][neutron][qa] Injecting code into grenade resource create phase

[Sam Betts (sambetts)]

Its easy enough to setup devstack today to use a flat neutron network
instead of faking it using the tenant networks which we don¹t support, I
do it in my third party CI for real hardware, if we can make it work for
fake BM VMs too would it solve this issue, or would grenade override that?

For reference these are the extra settings I have to add to make it work
for real hardware with no changes to the Ironic devstack plugin:

PUBLIC_INTERFACE={phys_interface}

Q_L3_ENABLED=False
Q_USE_PROVIDER_NETWORKING=True
OVS_PHYSICAL_BRIDGE=br-{phys_interface}
PHYSICAL_NETWORK=private # Because the ironic devstack plugin looks for a
network called "private"
PROVIDER_NETWORK_TYPE=³flat²

ENABLE_ISOLATED_METADATA=True


Sam Betts

On 12/05/2016 11:48, "Sean Dague"  wrote:

>On 05/11/2016 05:53 PM, Jim Rollenhagen wrote:
>> I've ran into a bit of a wedge working on ironic grenade tests.
>> 
>> In a normal dsvm run, the ironic setup taps the control plane into the
>> tenant network (as that's how it's currently intended to be deployed).
>> That code is here[0].
>> 
>> However, in a grenade run, during the resource create phase, a new
>> network is created. This happens in the neutron bits[1], and is used to
>> boot a server in the nova bits[2].
>> 
>> Since the control plane can't communicate with the machine on that
>> network, our ramdisk doesn't reach back to ironic after booting up, and
>> provisioning fails[3][4].
>> 
>> Curious if any grenade experts have thoughts on how we might be able to
>> set up that tap in between the neutron and nova resource creation.
>> 
>> One alternative I've considered is a method to have nova resource
>> creation not boot an instance, and replicate that functionality in the
>> ironic plugin, after we tap into that network.
>> 
>> I'm sure there's other alternatives here that I haven't thought of;
>> suggestions welcome. Thanks in advance. :)
>> 
>> // jim
>> 
>> [0] 
>>https://github.com/openstack/ironic/blob/95ff5badbdea0898d7877e6518939160
>>08561760/devstack/lib/ironic#L653
>> [1] 
>>https://github.com/openstack-dev/grenade/blob/fce63f40d21abea926d343e9cdd
>>d620e3f03684a/projects/50_neutron/resources.sh#L34
>> [2] 
>>https://github.com/openstack-dev/grenade/blob/fce63f40d21abea926d343e9cdd
>>d620e3f03684a/projects/60_nova/resources.sh#L79
>> [3] 
>>http://logs.openstack.org/65/311865/3/experimental/gate-grenade-dsvm-iron
>>ic/e635dec/logs/grenade.sh.txt.gz#_2016-05-10_18_10_03_648
>> [4] 
>>http://logs.openstack.org/65/311865/3/experimental/gate-grenade-dsvm-iron
>>ic/e635dec/logs/old/screen-ir-cond.txt.gz?level=WARNING
>
>Let me make sure I understand the crux of the issue: services may need
>to create additional custom networking in order for the rest of the
>resource flow to work correctly. Ironic definitely needs this because
>it's building up an interesting simulated network here for it's guests
>to live on.
>
>That feels like we need a new pre_create (or early_create) phase that
>ironic could hook into to do that same network create. I think the only
>question in my mind is whether neutrons network create here should move
>to early_create as well.
>
>Does ironic's ovs setup need to happen after the neutron create phase
>here, or does it just need to happen before nova?
>
>Does ironic need to overwrite the $net_id that gets passed to Nova later?
>
>   -Sean
>
>-- 
>Sean Dague
>http://dague.net
>
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][nova][neutron][qa] Injecting code into grenade resource create phase

[Sean Dague]

On 05/11/2016 05:53 PM, Jim Rollenhagen wrote:
> I've ran into a bit of a wedge working on ironic grenade tests.
> 
> In a normal dsvm run, the ironic setup taps the control plane into the
> tenant network (as that's how it's currently intended to be deployed).
> That code is here[0].
> 
> However, in a grenade run, during the resource create phase, a new
> network is created. This happens in the neutron bits[1], and is used to
> boot a server in the nova bits[2].
> 
> Since the control plane can't communicate with the machine on that
> network, our ramdisk doesn't reach back to ironic after booting up, and
> provisioning fails[3][4].
> 
> Curious if any grenade experts have thoughts on how we might be able to
> set up that tap in between the neutron and nova resource creation.
> 
> One alternative I've considered is a method to have nova resource
> creation not boot an instance, and replicate that functionality in the
> ironic plugin, after we tap into that network.
> 
> I'm sure there's other alternatives here that I haven't thought of;
> suggestions welcome. Thanks in advance. :)
> 
> // jim
> 
> [0] 
> https://github.com/openstack/ironic/blob/95ff5badbdea0898d7877e651893916008561760/devstack/lib/ironic#L653
> [1] 
> https://github.com/openstack-dev/grenade/blob/fce63f40d21abea926d343e9cddd620e3f03684a/projects/50_neutron/resources.sh#L34
> [2] 
> https://github.com/openstack-dev/grenade/blob/fce63f40d21abea926d343e9cddd620e3f03684a/projects/60_nova/resources.sh#L79
> [3] 
> http://logs.openstack.org/65/311865/3/experimental/gate-grenade-dsvm-ironic/e635dec/logs/grenade.sh.txt.gz#_2016-05-10_18_10_03_648
> [4] 
> http://logs.openstack.org/65/311865/3/experimental/gate-grenade-dsvm-ironic/e635dec/logs/old/screen-ir-cond.txt.gz?level=WARNING

Let me make sure I understand the crux of the issue: services may need
to create additional custom networking in order for the rest of the
resource flow to work correctly. Ironic definitely needs this because
it's building up an interesting simulated network here for it's guests
to live on.

That feels like we need a new pre_create (or early_create) phase that
ironic could hook into to do that same network create. I think the only
question in my mind is whether neutrons network create here should move
to early_create as well.

Does ironic's ovs setup need to happen after the neutron create phase
here, or does it just need to happen before nova?

Does ironic need to overwrite the $net_id that gets passed to Nova later?

-Sean

-- 
Sean Dague
http://dague.net

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [ironic][nova][neutron][qa] Injecting code into grenade resource create phase

[Vasyl Saienko]

Hello Jim,

Thanks for rising this question.

My personal feeling that we don't need to tune tests that won't pass due to
design limitations.
It is Ironic pre-requirement to have network access from control plane to
server during provisioning.
Until Ironic Neutron integration is completed we may skip this test.

On Thu, May 12, 2016 at 12:53 AM, Jim Rollenhagen 
wrote:

> I've ran into a bit of a wedge working on ironic grenade tests.
>
> In a normal dsvm run, the ironic setup taps the control plane into the
> tenant network (as that's how it's currently intended to be deployed).
> That code is here[0].
>
> However, in a grenade run, during the resource create phase, a new
> network is created. This happens in the neutron bits[1], and is used to
> boot a server in the nova bits[2].
>
> Since the control plane can't communicate with the machine on that
> network, our ramdisk doesn't reach back to ironic after booting up, and
> provisioning fails[3][4].
>
> Curious if any grenade experts have thoughts on how we might be able to
> set up that tap in between the neutron and nova resource creation.
>
> One alternative I've considered is a method to have nova resource
> creation not boot an instance, and replicate that functionality in the
> ironic plugin, after we tap into that network.
>
> I'm sure there's other alternatives here that I haven't thought of;
> suggestions welcome. Thanks in advance. :)
>
> // jim
>
> [0]
> https://github.com/openstack/ironic/blob/95ff5badbdea0898d7877e651893916008561760/devstack/lib/ironic#L653
> [1]
> https://github.com/openstack-dev/grenade/blob/fce63f40d21abea926d343e9cddd620e3f03684a/projects/50_neutron/resources.sh#L34
> [2]
> https://github.com/openstack-dev/grenade/blob/fce63f40d21abea926d343e9cddd620e3f03684a/projects/60_nova/resources.sh#L79
> [3]
> http://logs.openstack.org/65/311865/3/experimental/gate-grenade-dsvm-ironic/e635dec/logs/grenade.sh.txt.gz#_2016-05-10_18_10_03_648
> [4]
> http://logs.openstack.org/65/311865/3/experimental/gate-grenade-dsvm-ironic/e635dec/logs/old/screen-ir-cond.txt.gz?level=WARNING
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [ironic][nova][neutron][qa] Injecting code into grenade resource create phase

[Jim Rollenhagen]

I've ran into a bit of a wedge working on ironic grenade tests.

In a normal dsvm run, the ironic setup taps the control plane into the
tenant network (as that's how it's currently intended to be deployed).
That code is here[0].

However, in a grenade run, during the resource create phase, a new
network is created. This happens in the neutron bits[1], and is used to
boot a server in the nova bits[2].

Since the control plane can't communicate with the machine on that
network, our ramdisk doesn't reach back to ironic after booting up, and
provisioning fails[3][4].

Curious if any grenade experts have thoughts on how we might be able to
set up that tap in between the neutron and nova resource creation.

One alternative I've considered is a method to have nova resource
creation not boot an instance, and replicate that functionality in the
ironic plugin, after we tap into that network.

I'm sure there's other alternatives here that I haven't thought of;
suggestions welcome. Thanks in advance. :)

// jim

[0] 
https://github.com/openstack/ironic/blob/95ff5badbdea0898d7877e651893916008561760/devstack/lib/ironic#L653
[1] 
https://github.com/openstack-dev/grenade/blob/fce63f40d21abea926d343e9cddd620e3f03684a/projects/50_neutron/resources.sh#L34
[2] 
https://github.com/openstack-dev/grenade/blob/fce63f40d21abea926d343e9cddd620e3f03684a/projects/60_nova/resources.sh#L79
[3] 
http://logs.openstack.org/65/311865/3/experimental/gate-grenade-dsvm-ironic/e635dec/logs/grenade.sh.txt.gz#_2016-05-10_18_10_03_648
[4] 
http://logs.openstack.org/65/311865/3/experimental/gate-grenade-dsvm-ironic/e635dec/logs/old/screen-ir-cond.txt.gz?level=WARNING

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev