Re: [openstack-dev] [nova][scheduler] Updating Our Concept of Resources
Le 01/06/2015 14:40, Ed Leafe a écrit : Several of the discussions at Vancouver got me thinking about what seems to me to be a fundamental mis-match in Nova: the way we think about resources, and how we handle requesting/claiming them. I wrote down my initial thoughts (http://blog.leafe.com/rethinking-resources/), but as many of you took the week off after the summit, I waited until today to start a ML discussion. Nova started when "the cloud" was pretty much the VPS model of on-demand compute: you had big powerful servers that you wanted to virtualize into several smaller chunks, and you did that with hypervisors on those servers. Along with this came the concept of "flavors", which were the available portions of the overall server you could request. Over the years, though, that model has become just one of many (bare metal, NUMA, etc.), yet our internal model still tries to force everything into the flavor world. And with the advent of technologies such as LXD, it seems likely that the VPS model will begin to fade away and replaced by more efficient ways to share compute. We need to update our concept of a resource internally in Nova, both in the DB and in code, and stop thinking that every request should have a flavor. This is also the single biggest blocker in separating the Scheduler into a separate service that can be used to allocate resources in Cinder and Neutron, too, because we've tied it exclusively to the concept of a VPS resource. I'd like to start a discussion and get people's opinions on this. I have some ideas, of course, but I know that as these are not simple issues, getting input from many different perspectives is going to lead to a better approach. So: what do you think would be a better way to represent resources, as well as a better way to represent a request for some of those resources? -- Ed Leafe Long story short : https://blueprints.launchpad.net/nova/+spec/resource-objects Until we get this, it's difficult to discuss on different resources within Nova. Let's work by small steps and provide a versioned resource system before discussing on how other projects could add more resources to Nova. -Sylvain __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][scheduler] Updating Our Concept of Resources
On Jun 1, 2015, at 7:49 AM, Sylvain Bauza wrote: > Long story short : > https://blueprints.launchpad.net/nova/+spec/resource-objects > > Until we get this, it's difficult to discuss on different resources within > Nova. Let's work by small steps and provide a versioned resource system > before discussing on how other projects could add more resources to Nova. Yes, this is important in making it possible to transition to a different model in an organized way. I just wanted to start the discussion on what that new model might look like. -- Ed Leafe signature.asc Description: Message signed with OpenPGP using GPGMail __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][scheduler] Updating Our Concept of Resources
Ed Leafe said on Mon, Jun 01, 2015 at 07:40:17AM -0500: > We need to update our concept of a resource internally in Nova, both > in the DB and in code, and stop thinking that every request should > have a flavor. If you allocate all the memory of a box to high-mem instances, you may not be billing for all the CPU and disk which are now unusable. That's why flavors were introduced, afaik, and it's still a valid need. I totally agree the scheduler doesn't have to know anything about flavors though. We should push them out to request validation in the Nova API. This can be considered part of cleaning up the scheduler API. Alexis -- Nova Engineer, HP Cloud. AKA lealexis, lxsli. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][scheduler] Updating Our Concept of Resources
On Jun 2, 2015, at 5:58 AM, Alexis Lee wrote: > If you allocate all the memory of a box to high-mem instances, you may > not be billing for all the CPU and disk which are now unusable. That's > why flavors were introduced, afaik, and it's still a valid need. So we had a very good discussion at the weekly IRC meeting for the Scheduler, and we agreed to follow that up here on the ML. One thing that came up, noted in the quote above, is that I gave the impression in my first email that I thought flavors were useless. I think I did a better job in the original blog post of explaining that flavors are a great way to handle the sane division of a resource like a compute node. The issue I have with flavors is that we seem to be locked into the "everything that can be requested has to fit into the flavor", and that really doesn't make sense. Another concern was from the cloud provider's POV, which makes a flavor a convenient way of packaging cloud resources for sale. The customer can simply say "give me one of these" to specify a complex combination of virtualized resources. That's great, but it means that there has to be a flavor for every possible permutation of resources. If you restricted flavors to only represent the sane ways of dividing up compute nodes, any other features could be add-ons to the request. Something like ordering a pizza: offer the customer a fixed choice of sizes, but then let them specify any toppings in whatever combination they want. That's certainly more sane than presenting them with a menu with hundreds of pizza "flavors", each representing a different size/topping combination. > I totally agree the scheduler doesn't have to know anything about > flavors though. We should push them out to request validation in the > Nova API. This can be considered part of cleaning up the scheduler API. This idea was also discussed and seemed to get a lot of support. Basically, it means that by the time the request hits the scheduler, there is no "flavor" anymore; instead, the scheduler gets a request for so much RAM, so much disk, etc., and these amounts have already been validated at the API layer. So a customer requests a flavor just like they do now, and the API has the responsibility to verify that the flavor is valid, but then "unpacks" the flavor into its components and passes that on to compute. The end result is the same, but there would be no more need to store "flavors" anywhere but the front end. This has the added benefit of eliminating the problem with new flavors being propagated down to cells, since they would no longer need to have to translate what "flavor X" means. Don Dugger volunteered to write up a spec for removing flavors from the scheduler. So did I miss anything? :) -- Ed Leafe signature.asc Description: Message signed with OpenPGP using GPGMail __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][scheduler] Updating Our Concept of Resources
On 3 June 2015 at 13:53, Ed Leafe wrote: > On Jun 2, 2015, at 5:58 AM, Alexis Lee wrote: > >> If you allocate all the memory of a box to high-mem instances, you may >> not be billing for all the CPU and disk which are now unusable. That's >> why flavors were introduced, afaik, and it's still a valid need. > > So we had a very good discussion at the weekly IRC meeting for the Scheduler, > and we agreed to follow that up here on the ML. One thing that came up, noted > in the quote above, is that I gave the impression in my first email that I > thought flavors were useless. I think I did a better job in the original blog > post of explaining that flavors are a great way to handle the sane division > of a resource like a compute node. The issue I have with flavors is that we > seem to be locked into the "everything that can be requested has to fit into > the flavor", and that really doesn't make sense. > > Another concern was from the cloud provider's POV, which makes a flavor a > convenient way of packaging cloud resources for sale. The customer can simply > say "give me one of these" to specify a complex combination of virtualized > resources. That's great, but it means that there has to be a flavor for every > possible permutation of resources. If you restricted flavors to only > represent the sane ways of dividing up compute nodes, any other features > could be add-ons to the request. Something like ordering a pizza: offer the > customer a fixed choice of sizes, but then let them specify any toppings in > whatever combination they want. That's certainly more sane than presenting > them with a menu with hundreds of pizza "flavors", each representing a > different size/topping combination. I feel there is a lot to be said for treating "consumable" resources very separately to "free" options. For example grouping the vCPUs into sockets can be "free" in terms of capacity planning, so is a valid optional add on (assuming you are not doing some level of pinning to match that). For things where you are trying to find a specific compute node, that kind of attribute has clear capacity planning concerns, and is likely to have a specific "cost" associated with it. So we need to make sure its clear how that cost concept can be layered on top of the Nova API. For example "os_type" often changes the cost, and is implemented on top of flavors using a combination of protected image properties on glance and the way snapshots inherit image properties. >> I totally agree the scheduler doesn't have to know anything about >> flavors though. We should push them out to request validation in the >> Nova API. This can be considered part of cleaning up the scheduler API. > > This idea was also discussed and seemed to get a lot of support. Basically, > it means that by the time the request hits the scheduler, there is no > "flavor" anymore; instead, the scheduler gets a request for so much RAM, so > much disk, etc., and these amounts have already been validated at the API > layer. So a customer requests a flavor just like they do now, and the API has > the responsibility to verify that the flavor is valid, but then "unpacks" the > flavor into its components and passes that on to compute. The end result is > the same, but there would be no more need to store "flavors" anywhere but the > front end. This has the added benefit of eliminating the problem with new > flavors being propagated down to cells, since they would no longer need to > have to translate what "flavor X" means. Don Dugger volunteered to write up a > spec for removing flavors from the scheduler. > +1 for Nova translating the incoming request to a "resource request" the scheduler understands, given the resources it knows about. I would look at scoping that to "compute" resources, so its easier to add "volume" and "network" into that request at a later date. Thanks, John __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova][scheduler] Updating Our Concept of Resources
On 06/03/2015 02:13 PM, John Garbutt wrote: > On 3 June 2015 at 13:53, Ed Leafe wrote: >> On Jun 2, 2015, at 5:58 AM, Alexis Lee wrote: >> >>> If you allocate all the memory of a box to high-mem instances, you may >>> not be billing for all the CPU and disk which are now unusable. That's >>> why flavors were introduced, afaik, and it's still a valid need. >> >> So we had a very good discussion at the weekly IRC meeting for the >> Scheduler, and we agreed to follow that up here on the ML. One thing that >> came up, noted in the quote above, is that I gave the impression in my first >> email that I thought flavors were useless. I think I did a better job in the >> original blog post of explaining that flavors are a great way to handle the >> sane division of a resource like a compute node. The issue I have with >> flavors is that we seem to be locked into the "everything that can be >> requested has to fit into the flavor", and that really doesn't make sense. >> >> Another concern was from the cloud provider's POV, which makes a flavor a >> convenient way of packaging cloud resources for sale. The customer can >> simply say "give me one of these" to specify a complex combination of >> virtualized resources. That's great, but it means that there has to be a >> flavor for every possible permutation of resources. If you restricted >> flavors to only represent the sane ways of dividing up compute nodes, any >> other features could be add-ons to the request. Something like ordering a >> pizza: offer the customer a fixed choice of sizes, but then let them specify >> any toppings in whatever combination they want. That's certainly more sane >> than presenting them with a menu with hundreds of pizza "flavors", each >> representing a different size/topping combination. > > I feel there is a lot to be said for treating "consumable" resources > very separately to "free" options. > > For example grouping the vCPUs into sockets can be "free" in terms of > capacity planning, so is a valid optional add on (assuming you are not > doing some level of pinning to match that). > > For things where you are trying to find a specific compute node, that > kind of attribute has clear capacity planning concerns, and is likely > to have a specific "cost" associated with it. So we need to make sure > its clear how that cost concept can be layered on top of the Nova API. > For example "os_type" often changes the cost, and is implemented on > top of flavors using a combination of protected image properties on > glance and the way snapshots inherit image properties. > >>> I totally agree the scheduler doesn't have to know anything about >>> flavors though. We should push them out to request validation in the >>> Nova API. This can be considered part of cleaning up the scheduler API. >> >> This idea was also discussed and seemed to get a lot of support. Basically, >> it means that by the time the request hits the scheduler, there is no >> "flavor" anymore; instead, the scheduler gets a request for so much RAM, so >> much disk, etc., and these amounts have already been validated at the API >> layer. So a customer requests a flavor just like they do now, and the API >> has the responsibility to verify that the flavor is valid, but then >> "unpacks" the flavor into its components and passes that on to compute. The >> end result is the same, but there would be no more need to store "flavors" >> anywhere but the front end. This has the added benefit of eliminating the >> problem with new flavors being propagated down to cells, since they would no >> longer need to have to translate what "flavor X" means. Don Dugger >> volunteered to write up a spec for removing flavors from the scheduler. >> > > +1 for Nova translating the incoming request to a "resource request" > the scheduler understands, given the resources it knows about. > > I would look at scoping that to "compute" resources, so its easier to > add "volume" and "network" into that request at a later date. > I also agree with this pretty much completely. I feel that the single thing that made some of the scheduler discussions drag on for months is our lack of willingness to bite of the big chunk that is coming up with a solid API to the scheduler. Starting from nouns and verbs - it definitely seems like a good idea to pass in the _requested_ resources to a scheduler that knows about _avalible_ resources. [1] seems like an excellent start. I seem to remember Jay discussing at one point that not all of the things we want the scheduler to know about make sense to be modelled as resources (running instances for example) and it made a lot of sense to me, but it seems like it's the kind of thing that would be the easiest to figure out once you see the code (I also don't see it mentioned in [1] but I assume Jay dropped it to keep the scope of that BP manageable). N. [1] https://review.openstack.org/#/c/184534/1/specs/liberty/approved/resource-objects.rst
Re: [openstack-dev] [nova][scheduler] Updating Our Concept of Resources
Le 03/06/2015 16:02, Nikola Đipanov a écrit : On 06/03/2015 02:13 PM, John Garbutt wrote: On 3 June 2015 at 13:53, Ed Leafe wrote: On Jun 2, 2015, at 5:58 AM, Alexis Lee wrote: If you allocate all the memory of a box to high-mem instances, you may not be billing for all the CPU and disk which are now unusable. That's why flavors were introduced, afaik, and it's still a valid need. So we had a very good discussion at the weekly IRC meeting for the Scheduler, and we agreed to follow that up here on the ML. One thing that came up, noted in the quote above, is that I gave the impression in my first email that I thought flavors were useless. I think I did a better job in the original blog post of explaining that flavors are a great way to handle the sane division of a resource like a compute node. The issue I have with flavors is that we seem to be locked into the "everything that can be requested has to fit into the flavor", and that really doesn't make sense. Another concern was from the cloud provider's POV, which makes a flavor a convenient way of packaging cloud resources for sale. The customer can simply say "give me one of these" to specify a complex combination of virtualized resources. That's great, but it means that there has to be a flavor for every possible permutation of resources. If you restricted flavors to only represent the sane ways of dividing up compute nodes, any other features could be add-ons to the request. Something like ordering a pizza: offer the customer a fixed choice of sizes, but then let them specify any toppings in whatever combination they want. That's certainly more sane than presenting them with a menu with hundreds of pizza "flavors", each representing a different size/topping combination. I feel there is a lot to be said for treating "consumable" resources very separately to "free" options. For example grouping the vCPUs into sockets can be "free" in terms of capacity planning, so is a valid optional add on (assuming you are not doing some level of pinning to match that). For things where you are trying to find a specific compute node, that kind of attribute has clear capacity planning concerns, and is likely to have a specific "cost" associated with it. So we need to make sure its clear how that cost concept can be layered on top of the Nova API. For example "os_type" often changes the cost, and is implemented on top of flavors using a combination of protected image properties on glance and the way snapshots inherit image properties. I totally agree the scheduler doesn't have to know anything about flavors though. We should push them out to request validation in the Nova API. This can be considered part of cleaning up the scheduler API. This idea was also discussed and seemed to get a lot of support. Basically, it means that by the time the request hits the scheduler, there is no "flavor" anymore; instead, the scheduler gets a request for so much RAM, so much disk, etc., and these amounts have already been validated at the API layer. So a customer requests a flavor just like they do now, and the API has the responsibility to verify that the flavor is valid, but then "unpacks" the flavor into its components and passes that on to compute. The end result is the same, but there would be no more need to store "flavors" anywhere but the front end. This has the added benefit of eliminating the problem with new flavors being propagated down to cells, since they would no longer need to have to translate what "flavor X" means. Don Dugger volunteered to write up a spec for removing flavors from the scheduler. +1 for Nova translating the incoming request to a "resource request" the scheduler understands, given the resources it knows about. I would look at scoping that to "compute" resources, so its easier to add "volume" and "network" into that request at a later date. I also agree with this pretty much completely. I feel that the single thing that made some of the scheduler discussions drag on for months is our lack of willingness to bite of the big chunk that is coming up with a solid API to the scheduler. Starting from nouns and verbs - it definitely seems like a good idea to pass in the _requested_ resources to a scheduler that knows about _avalible_ resources. [1] seems like an excellent start. I seem to remember Jay discussing at one point that not all of the things we want the scheduler to know about make sense to be modelled as resources (running instances for example) and it made a lot of sense to me, but it seems like it's the kind of thing that would be the easiest to figure out once you see the code (I also don't see it mentioned in [1] but I assume Jay dropped it to keep the scope of that BP manageable). N. +1 to that. That's now 2 cycles that we're trying to make clean interfaces for the scheduler. We identified the relationship between the ResourceTracker and the Scheduler as one to be cleaned, and [1] is targeting th
Re: [openstack-dev] [nova][scheduler] Updating Our Concept of Resources
Good summary, Ed! Ed Leafe said on Wed, Jun 03, 2015 at 07:53:02AM -0500: > > I totally agree the scheduler doesn't have to know anything about > > flavors though. We should push them out to request validation in the > > Nova API. This can be considered part of cleaning up the scheduler API. > > This idea was also discussed and seemed to get a lot of support. ... > Don Dugger volunteered to write up a spec for removing flavors from > the scheduler. Yay <3 > So did I miss anything? :) One thing, we also discussed allowing a request to include multiple flavors. This could be used to mix-and-match pizza-base-flavors with pizza-topping-flavors. So to escape the pizza metaphor briefly, you could take XLarge VPS resources with a Medium SSD package and a Small network package. This approach prevents too much resource fragmentation and maintains a level of convenience for the user while allowing for an exponential number of flavor combinations. Alexis -- Nova Engineer, HP Cloud. AKA lealexis, lxsli. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
