On Thursday morning the Nova and Cinder teams got together for a
cross-project design summit session. The full etherpad is here [1].
This was all about volume multi-attach.
A subset of people from both teams were actually meeting weekly for four
weeks leading up to the summit to hash out some details which we hoped
to resolve at the summit. Unfortunately that didn't happen.
The first thing we wanted to settle was why do we actually want/need to
support volume multi-attach? Because "Cinder added it to their API in
Juno so Nova needs to support it" isn't a good reason. There are a few
drivers for this feature:
* The need for active/active and active/hot standby scenarios which
can't accept downtime due to attaching a new volume.
* Some database clusters, like Oracle RAC, require shared volumes. So
Trove is a stakeholder for this feature also.
* Other legacy application use cases were brought up that essentially
mean this is something they need to bridge a gap to adopting OpenStack.
So we agreed that while this is not really something we necessarily like
(because of the non-cloud legacy application nature of it), it is
necessary so we're going to continue trying to make it happen.
We then quickly went over what was completed in Mitaka and explained the
detach issue we ran into. The problem is when you have more than one
volume attached to the same instance on the same host, when you detach
one of them, both of the volume connections actually get terminated on
the host.
This problem is also complicated by the fact that some Cinder backends
will create one attachment per export/volume, whereas others will
multiplex all volumes onto one attachment.
Coming into the session we really had two competing solutions from the
Cinder team, one from Walter Boring and one from John Griffith. However,
during the session another idea was brought up from Dan Smith. The full
details are in the etherpad, but it's really an idea to abstract the
multiple volume attachments on the Cinder side that Nova only sees a
single volume, so Nova wouldn't have to change any of it's API handling
for Cinder volumes to be checking if they support multiattach or not,
and thus have to have conditional logic spread all over Nova (API,
compute, and virt drivers).
With Dan's idea we'd still have the disconnect/detach problem where Nova
would need to check if it can disconnect from the host if there is only
a single attachment left, but Nova has to do that regardless for all of
the proposed solutions.
It sounded like John Griffith had a similar idea before when looking at
this problem, and we spent a fair amount of time discussing it on both
sides during the session.
At the end of the session, we (Nova) came away with the following next
steps:
1. John Griffith (Cinder team) would work on a proof of concept for the
abstracted volume idea.
2. Cinder would work on adding a volume migration test to Tempest to be
run in the multi-node gate job (this needs to happen regardless). Scott
D'Angelo is going to work on this.
3. Ildikó Váncsa was going to work on the Nova volume disconnect ref
counting logic.
4. We'd meet on Friday during the Nova meetup session to discuss more
details.
--
So what happened then was we met on Friday and found out that we were
all speaking different languages on Thursday, because the Cinder team
didn't think that they were going to be going with this new abstracted
volume idea. After much wailing and gnashing of teeth we agreed to do a
hangout shortly after the summit to try and get back on the same page.
So we're doing that tomorrow (5/5) at 1600 UTC. This is going to be at
least myself, Ildikó, Scott and Walter on the call. Walter has been
creating diagrams of the flows through Nova and Cinder for various
interactions like attach/detach of volumes and volume-backed live
migration so that we can try to step back and see where the proposed
changes fit in.
It's sounding like regardless of solution there will be changes to the
Cinder API (at least for os-initialize_connection). There might be some
new APIs too, for example, an API to get connection_info during live
migration without Nova having to call os-initialize_connection to get it.
So we'll see what happens. We'll eventually figure out. After all, it's
only code, right? :)
[1] https://etherpad.openstack.org/p/newton-nova-cinder
--
Thanks,
Matt Riedemann
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
