[Openstack-operators] 6 days left for the Forum Brainstorming Period...
Hello All! The Forum Brainstorming session ends September 11 and the topic submission phase begins September 12. Thank you to all of the projects that have created a wiki and begun the Brainstorming Phase. I'd like to encourage projects that have not yet created an etherpad to do so at https://wiki.openstack.org/wiki/Forum/Berlin2018 This is an opportunity to get feedback, vet ideas, and garner support from the community on your ideas. Don't rely only on a PTL to make the agenda... step on up and place the items you consider important front and center :) If you have questions or concerns about the process, please don't hesitate to reach out. Cheers, Jimmy ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [openstack-dev] [nova] [placement] extraction (technical) update
> I think there was a period in time where the nova_api database was created > where entires would try to get pulled out from the original nova database and > then checking nova_api if it doesn't exist afterwards (or vice versa). One > of the cases that this was done to deal with was for things like instance > types > or flavours. > > I don't know the exact details but I know that older instance types exist in > the nova db and the newer ones are sitting in nova_api. Something along > those lines? Yep, we've moved entire databases before in nova with minimal disruption to the users. Not just flavors, but several pieces of data came out of the "main" database and into the api database transparently. It's doable, but with placement being split to a separate project/repo/whatever, there's not really any option for being graceful about it in this case. > At this point, I'm thinking turn off placement, setup the new one, do > the migration > of the placement-specific tables (this can be a straightforward documented > task > OR it would be awesome if it was a placement command (something along > the lines of `placement-manage db import_from_nova`) which would import all > the right things > > The idea of having a command would be *extremely* useful for deployment tools > in automating the process and it also allows the placement team to selectively > decide what they want to onboard? Well, it's pretty cut-and-dried as all the tables in nova-api are either for nova or placement, so there's not much confusion about what belongs. I'm not sure that doing this import in python is really the most efficient way. I agree a placement-manage command would be ideal from an "easy button" point of view, but I think a couple lines of bash that call mysqldump are likely to vastly outperform us doing it natively in python. We could script exec()s of those commands from python, but.. I think I'd rather just see that as a shell script that people can easily alter/test on their own. Just curious, but in your case would the service catalog entry change at all? If you stand up the new placement in the exact same spot, it shouldn't, but I imagine some people will have the catalog entry change slightly (even if just because of a VIP or port change). Am I remembering correctly that the catalog can get cached in various places such that much of nova would need a restart to notice? --Dan ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [openstack-dev] [nova] [placement] extraction (technical) update
On 9/5/2018 10:03 AM, Mohammed Naser wrote: On Wed, Sep 5, 2018 at 10:57 AM Matt Riedemann wrote: On 9/5/2018 8:47 AM, Mohammed Naser wrote: Could placement not do what happened for a while when the nova_api database was created? Can you be more specific? I'm having a brain fart here and not remembering what you are referring to with respect to the nova_api DB. I think there was a period in time where the nova_api database was created where entires would try to get pulled out from the original nova database and then checking nova_api if it doesn't exist afterwards (or vice versa). One of the cases that this was done to deal with was for things like instance types or flavours. I don't know the exact details but I know that older instance types exist in the nova db and the newer ones are sitting in nova_api. Something along those lines? Yeah that more about supporting online data migrations *within* nova where new records were created in the API DB and old records would be looked up in both the API DB and then if not found there, in the cell (traditional nova DB). But you'd also be running the "nova-manage db online_data_migrations" CLI to force the migration of the records from the cell DB to the API DB. With Placement split out of nova, we can't really do that. You could point placement at the nova_api DB so it can pull existing records, but it would continue to create new records in the nova_api DB rather than the placement DB and at some point you have to make that data migration. Maybe you were thinking something like have temporary fallback code in placement such that if a record isn't found in the placement database, it queries a configured nova_api database? That'd be a ton of work at this point, and if it was something we were going to do, we should have agreed on that in YVR several months ago, definitely pre-extraction. I say this because I know that moving the database is a huge task for us, considering how big it can be in certain cases for us, and it means control plane outage too I'm pretty sure you were in the room in YVR when we talked about how operators were going to do the database migration and were mostly OK with what was discussed, which was a lot will just copy and take the downtime (I think CERN said around 10 minutes for them, but they aren't a public cloud either), but others might do something more sophisticated and nova shouldn't try to pick the best fit for all. If we're provided the list of tables used by placement, we could considerably make the downtime smaller because we don't have to pull in the other huge tables like instances/build requests/etc There are no instances records in the API DB, maybe you mean instance_mappings? But yes I get the point. What happens if things like server deletes happen while the placement service is down? The DELETE /allocations/{consumer_id} requests from nova to placement will fail with some keystoneauth1 exception, but because of our old friend @safe_connect we likely won't fail the server delete because we squash the exception from KSA: https://github.com/openstack/nova/blob/0f102089dd0b27c7d35f0cbba87332414032c0a4/nova/scheduler/client/report.py#L2069 However, you'd still have allocations in placement against resource providers (compute nodes) for instances that no longer exist, which means you're available capacity for scheduling new requests is diminished until those bogus allocations are purged from placement, which will take some scripting. In other words, not good things. I'm definitely interested in what you do plan to do for the database migration to minimize downtime. At this point, I'm thinking turn off placement, setup the new one, do the migration of the placement-specific tables (this can be a straightforward documented task OR it would be awesome if it was a placement command (something along the lines of `placement-manage db import_from_nova`) which would import all the right things You wouldn't also stop nova-api while doing this? Otherwise you're going to get into the data/resource tracking mess described above which will require some post-migration cleanup scripting. The idea of having a command would be*extremely* useful for deployment tools in automating the process and it also allows the placement team to selectively decide what they want to onboard? Just throwing ideas here. -- Thanks, Matt ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [openstack-dev] [nova] [placement] extraction (technical) update
On Wed, Sep 5, 2018 at 10:57 AM Matt Riedemann wrote: > > On 9/5/2018 8:47 AM, Mohammed Naser wrote: > > Could placement not do what happened for a while when the nova_api > > database was created? > > Can you be more specific? I'm having a brain fart here and not > remembering what you are referring to with respect to the nova_api DB. I think there was a period in time where the nova_api database was created where entires would try to get pulled out from the original nova database and then checking nova_api if it doesn't exist afterwards (or vice versa). One of the cases that this was done to deal with was for things like instance types or flavours. I don't know the exact details but I know that older instance types exist in the nova db and the newer ones are sitting in nova_api. Something along those lines? > > > > I say this because I know that moving the database is a huge task for > > us, considering how big it can be in certain cases for us, and it > > means control plane outage too > > I'm pretty sure you were in the room in YVR when we talked about how > operators were going to do the database migration and were mostly OK > with what was discussed, which was a lot will just copy and take the > downtime (I think CERN said around 10 minutes for them, but they aren't > a public cloud either), but others might do something more sophisticated > and nova shouldn't try to pick the best fit for all. If we're provided the list of tables used by placement, we could considerably make the downtime smaller because we don't have to pull in the other huge tables like instances/build requests/etc What happens if things like server deletes happen while the placement service is down? > I'm definitely interested in what you do plan to do for the database > migration to minimize downtime. At this point, I'm thinking turn off placement, setup the new one, do the migration of the placement-specific tables (this can be a straightforward documented task OR it would be awesome if it was a placement command (something along the lines of `placement-manage db import_from_nova`) which would import all the right things The idea of having a command would be *extremely* useful for deployment tools in automating the process and it also allows the placement team to selectively decide what they want to onboard? Just throwing ideas here. > +openstack-operators ML since this is an operators discussion now. > > -- > > Thanks, > > Matt -- Mohammed Naser — vexxhost - D. 514-316-8872 D. 800-910-1726 ext. 200 E. mna...@vexxhost.com W. http://vexxhost.com ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
Re: [Openstack-operators] [openstack-dev] [nova] [placement] extraction (technical) update
On 9/5/2018 8:47 AM, Mohammed Naser wrote: Could placement not do what happened for a while when the nova_api database was created? Can you be more specific? I'm having a brain fart here and not remembering what you are referring to with respect to the nova_api DB. I say this because I know that moving the database is a huge task for us, considering how big it can be in certain cases for us, and it means control plane outage too I'm pretty sure you were in the room in YVR when we talked about how operators were going to do the database migration and were mostly OK with what was discussed, which was a lot will just copy and take the downtime (I think CERN said around 10 minutes for them, but they aren't a public cloud either), but others might do something more sophisticated and nova shouldn't try to pick the best fit for all. I'm definitely interested in what you do plan to do for the database migration to minimize downtime. +openstack-operators ML since this is an operators discussion now. -- Thanks, Matt ___ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators