A few of us have talked about writing a command to tell when you're
ready to upgrade (restart services with new code) for Ocata because we
have a few things landing in Ocata which are going from optional to
required, namely cells v2 and the placement API service.
We already have the 030 API DB migration for cells v2 which went into
the o-2 milestone today which breaks the API DB schema migrations if you
haven't run at least 'nova-manage cell_v2 simple_cell_setup'.
We have noted a few times in the placement/resource providers
discussions the need for something similar for the placement API, but
because that spans multiple databases (API and cell DBs), and because it
involves making sure the service is up and we can make REST API requests
to it, we can't do it in just a DB schema migration.
So today dansmith, jaypipes, sdague, edleafe and myself jumped on a call
to go over some notes / ideas being kicked around in an etherpad:
https://etherpad.openstack.org/p/nova-ocata-ready-for-upgrade
We agreed on writing a new CLI outside of nova-manage called nova-status
which can perform the upgrade readiness check for both cells v2 and the
placement API.
For cells v2 it's really going to check basically the same things as the
030 API DB schema migration.
For the placement API, it's going to do at least two things:
1. Try and make a request to / on the placement API endpoint from the
service catalog. This will at least check that (a) the placement
endpoint is in the service catalog, (b) nova.conf is configured with
credentials to make the request and (c) the service is running and
accepting requests.
2. Count the number of resource_providers in the API DB and compare that
to the number of compute_nodes in the cell DB and if there are fewer
resource providers than compute nodes, it's an issue which we'll flag in
the upgrade readiness CLI. This doesn't necessarily mean you can't
upgrade to Ocata, it just means there might be fewer computes available
for scheduling once you get to Ocata, so the chance of rebuilds and
NoValidHost increases until the computes are upgraded to Ocata and
configured to use the placement service to report
inventories/usage/allocations for RAM, CPU and disk.
That 2nd point is important because we also agreed to make the filter
scheduler NOT fallback to querying the compute_nodes table if there are
no resource providers available from the placement API. That means when
the scheduler gets a request to build or move a server, it's going to
query the placement API for possible resource providers to serve the
CPU/RAM/disk requirements for the build request spec and if nothing is
available it'll result in a NoValidHost failure. That is a change in
direction from a fallback plan we originally had in the spec here:
https://specs.openstack.org/openstack/nova-specs/specs/ocata/approved/resource-providers-scheduler-db-filters.html#other-deployer-impact
We're changing direction on that because we really want to make the
placement service required in Ocata and not delay it's usage for another
release, because as long as it's optional people are going to avoid
deploying and using it, which pushes us further out from forward
progress around the scheduler, placement service and resource tracker.
Regarding where this new CLI lived, and how it's deployed, and when it's
called, we had discussed a few options there, even talking about
splitting it out into it's own pip-installable package. We have a few
options but we aren't going to be totally clear on that until we get the
POC code written and then try to integrate it into grenade, so we're
basically deferring that discussion/decision for now. Wherever it is, we
know it needs to be run with the Ocata code (since that's where it's
going to be available), and after running the simple_cell_setup
command), and it needs to be run before restarting services with the
Ocata code. I'm not totally sure if it needs to be run after the DB
migrations or not, maybe Dan can clarify, but we'll sort it out for sure
when we integrate with grenade.
Anyway, the POC is started here:
https://review.openstack.org/#/c/411517/
I've got the basic framework in place and there is a patch on top that
does the cells v2 check. I plan on working on the placement API checks
tomorrow.
If you've read this far, congratulations. This email is really just
about communicating that things are happening because we have talked
about the need for this a few times, but hadn't hashed it out yet.
--
Thanks,
Matt Riedemann
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
