On 8/1/2016 4:20 PM, Jim Rollenhagen wrote:
Yes, I know this is stupid late for these.
I'd like to request two exceptions to the non-priority feature freeze,
for a couple of features in the Ironic driver. These were not requested
at the normal time as I thought they were nowhere near ready.
Multitenant networking
======================
Ironic's top feature request for around 2 years now has been to make
networking safe for multitenant use, as opposed to a flat network
(including control plane access!) for all tenants. We've been working on
a solution for 3 cycles now, and finally have the Ironic pieces of it
done, after a heroic effort to finish things up this cycle.
There's just one patch left to make it work, in the virt driver in Nova.
That is here: https://review.openstack.org/#/c/297895/
It's important to note that this actually fixes some dead code we pushed
on before this feature was done, and is only ~50 lines, half of which
are comments/reno.
Reviewers on this unearthed a problem on the ironic side, which I expect
to be fixed in the next couple of days:
https://review.openstack.org/#/q/topic:bug/1608511
We also have CI for this feature in ironic, and I have a depends-on
testing all of this as a whole: https://review.openstack.org/#/c/347004/
Per Matt's request, I'm also adding that job to Nova's experimental
queue: https://review.openstack.org/#/c/349595/
A couple folks from the ironic team have also done some manual testing
of this feature, with the nova code in, using real switches.
Merging this patch would bring a *huge* win for deployers and operators,
and I don't think it's very risky. It'll be ready to go sometime this
week, once that ironic chain is merged.
I've reviewed this one and it looks good to me. It's dependent on
python-ironicclient>=1.5.0 which Jim has a g-r bump up as a dependency.
And the gate-tempest-dsvm-ironic-multitenant-network-nv job is testing
this and passing on the test patch in ironic (and that job is in the
nova experimental queue now).
The upgrade procedure had some people scratching their heads in IRC this
week so I've stated that we need clear documentation there, which will
probably live here:
http://docs.openstack.org/developer/ironic/deploy/upgrade-guide.html
Since Ironic isn't in here:
http://docs.openstack.org/ops-guide/ops_upgrades.html#update-services
But the docs in the Ironic repo say that Nova should be upgraded first
when going from Juno to Kilo so it's definitely important to get those
docs updated for upgrades from Mitaka to Newton, but Jim said he'd do
that this cycle.
Given how long people have been asking for this in Ironic and the Ironic
team has made it a priority to get it working on their side, and there
is CI already and a small change in Nova, I'm OK with giving a
non-priority FFE for this.
Multi-compute usage via a hash ring
===================================
One of the major problems with the ironic virt driver today is that we
don't support running multiple nova-compute daemons with the ironic driver
loaded, because each compute service manages all ironic nodes and stomps
on each other.
There's currently a hack in the ironic virt driver to
kind of make this work, but instance locking still isn't done:
https://github.com/openstack/ironic/blob/master/ironic/nova/compute/manager.py
That is also holding back removing the pluggable compute manager in nova:
https://github.com/openstack/nova/blob/master/nova/conf/service.py#L64-L69
And as someone that runs a deployment using this hack, I can tell you
first-hand that it doesn't work well.
We (the ironic and nova community) have been working on fixing this for
2-3 cycles now, trying to find a solution that isn't terrible and
doesn't break existing use cases. We've been conflating it with how we
schedule ironic instances and keep managing to find a big wedge with
each approach. The best approach we've found involves duplicating the
compute capabilities and affinity filters in ironic.
Some of us were talking at the nova midcycle and decided we should try
the hash ring approach (like ironic uses to shard nodes between
conductors) and see how it works out, even though people have said in
the past that wouldn't work. I did a proof of concept last week, and
started playing with five compute daemons in a devstack environment.
Two nerd-snipey days later and I had a fully working solution, with unit
tests, passing CI. That is here:
https://review.openstack.org/#/c/348443/
We'll need to work on CI for this with multiple compute services. That
shouldn't be crazy difficult, but I'm not sure we'll have it done this
cycle (and it might get interesting trying to test computes joining and
leaving the cluster).
It also needs some testing at scale, which is hard to do in the upstream
gate, but I'll be doing my best to ship this downstream as soon as I
can, and iterating on any problems we see there.
It's a huge win for operators, for only a few hundred lines (some of
which will be pulled out to oslo next cycle, as it's copied from
ironic). The single compute mode would still be recommended while we
iron out any issues here, and that mode is well-understood (as this will
behave the same in that case). We have a couple of nova cores on board
with helping get this through, and I think it's totally doable.
I'm much less familiar with this one and haven't reviewed the series
yet, but I know that Jim, Jay and Dan were talking about this at the
midcycle and it seems this is a breakthrough of sorts on a working
solution to this problem, so I'm open to an FFE on this iff Jay and Dan
are happy with it (as sponsors more or less).
I'd be happier if there was a multi-node CI job to verify this because
I've blocked lesser things on CI for FFEs this cycle.
Thanks for hearing me out,
// jim
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
Thanks,
Matt Riedemann
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev