[openstack-dev] [nova] Boston Forum session recap - claims in the scheduler (or conductor)
The etherpad for this session is here [1]. The goal for this session was to inform operators and get feedback on the plan for what we're doing with moving claims from the computes to the control layer (scheduler or conductor). We mostly talked about retries, which also came up in the cells v2 session that Dan Smith led [2] and will recap later. Without getting into too many details, in the cells v2 session we came to a compromise on build retries and said that we could pass hosts down to the cell so that the cell-level conductor could retry if needed (even though we expect doing claims at the top will fix the majority of reasons you'd have a reschedule in the first place). During the claims in the scheduler session, a new wrinkle came up which is the hosts that the scheduler returns to the top-level conductor may be in different cells. So if we have two cells, A and B, with hosts x and y in cell A and host z in cell B, we can't send z to A for retries, or x or y to B for retries. So we need some kind of post-filter/weigher filtering such that hosts are grouped by cell and then they can be sent to the cells for retries as necessary. There was also some side discussion asking if we somehow regressed pack-first strategies by using Placement in Ocata. John Garbutt and Dan Smith have the context on this (I think) so I'm hoping they can clarify if we really need to fix something in Ocata at this point, or is this more of a case of closing a loop-hole? We also spent a good chunk of the session talking about overhead calculations for memory_mb and disk_gb which happens in the compute and on a per-hypervisor basis. In the absence of automating ways to adjust for overhead, our solution for now is operators can adjust reserved host resource values (vcpus, memory, disk) via config options and be conservative or aggressive as they see fit. Chris Dent and I also noted that you can adjust those reserved values via the placement REST API but they will be overridden by the config in a periodic task - which may be a bug, if not at least a surprise to an operator. We didn't really get into this during the forum session, but there are different opinions within the nova dev team on how to do claims in the controller services (conductor vs scheduler). Sylvain Bauza has a series which uses the conductor service, and Ed Leafe has a series using the scheduler. More on that in the mailing list [3]. Next steps are going to be weighing both options between Sylvain and Ed, picking a path and moving forward, as we don't have a lot of time to sit on this fence if we're going to get it done in Pike. As a side request, it would be great if companies that have teams doing performance and scale testing could help out and compare before (Ocata) and after (Pike with claims in the controller) results, because we eventually want to deprecate the caching scheduler but that currently outperforms the filter scheduler at scale because of the retries involved when using the filter scheduler, and which we expect doing claims at the top will fix. [1] https://etherpad.openstack.org/p/BOS-forum-move-claims-from-compute-to-scheduler [2] https://etherpad.openstack.org/p/BOS-forum-cellsv2-developer-community-coordination [3] http://lists.openstack.org/pipermail/openstack-dev/2017-May/116949.html -- Thanks, Matt __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Boston Forum session recap - claims in the scheduler (or conductor)
Le 19/05/2017 02:55, Matt Riedemann a écrit : > The etherpad for this session is here [1]. The goal for this session was > to inform operators and get feedback on the plan for what we're doing > with moving claims from the computes to the control layer (scheduler or > conductor). > > We mostly talked about retries, which also came up in the cells v2 > session that Dan Smith led [2] and will recap later. > > Without getting into too many details, in the cells v2 session we came > to a compromise on build retries and said that we could pass hosts down > to the cell so that the cell-level conductor could retry if needed (even > though we expect doing claims at the top will fix the majority of > reasons you'd have a reschedule in the first place). > And during that session, we said that given cell-local conductors (when there is a reschedule) can't upcall the global (for all cells) schedulers, that's why we agreed to use the conductor to be calling Placement API for allocations. > During the claims in the scheduler session, a new wrinkle came up which > is the hosts that the scheduler returns to the top-level conductor may > be in different cells. So if we have two cells, A and B, with hosts x > and y in cell A and host z in cell B, we can't send z to A for retries, > or x or y to B for retries. So we need some kind of post-filter/weigher > filtering such that hosts are grouped by cell and then they can be sent > to the cells for retries as necessary. > That's already proposed for reviews in https://review.openstack.org/#/c/465175/ > There was also some side discussion asking if we somehow regressed > pack-first strategies by using Placement in Ocata. John Garbutt and Dan > Smith have the context on this (I think) so I'm hoping they can clarify > if we really need to fix something in Ocata at this point, or is this > more of a case of closing a loop-hole? > The problem is that the scheduler doesn't verify the cells when trying to find a destination for an instance, it's just using weights for packing. So, for example, say I have N hosts and 2 cells, the first weighting host could be in cell1 while the second could be in cell2. Then, even if the operator uses the weighers for packing, for example a RequestSpec with num_instances=2 could push one instance in cell1 and the other in cell2. From a scheduler point of view, I think we could possibly add a CellWeigher that would help to pack instances within the same cell. Anyway, that's not related to the claims series, so we could possibly backport it for Ocata hopefully. > We also spent a good chunk of the session talking about overhead > calculations for memory_mb and disk_gb which happens in the compute and > on a per-hypervisor basis. In the absence of automating ways to adjust > for overhead, our solution for now is operators can adjust reserved host > resource values (vcpus, memory, disk) via config options and be > conservative or aggressive as they see fit. Chris Dent and I also noted > that you can adjust those reserved values via the placement REST API but > they will be overridden by the config in a periodic task - which may be > a bug, if not at least a surprise to an operator. > > We didn't really get into this during the forum session, but there are > different opinions within the nova dev team on how to do claims in the > controller services (conductor vs scheduler). Sylvain Bauza has a series > which uses the conductor service, and Ed Leafe has a series using the > scheduler. More on that in the mailing list [3]. > Sorry, but I do remember we had a consensus on using conductor at least during the cells v2 session. What I'm a bit afraid is that we're duplicating efforts on a sole blueprint while we all agreed to go that way. > Next steps are going to be weighing both options between Sylvain and Ed, > picking a path and moving forward, as we don't have a lot of time to sit > on this fence if we're going to get it done in Pike. > There are multiple reasons why we chose to use conductor for that : - as I said earlier, conductors when rescheduling can't upcall a global scheduler, and we agreed to not have (for the moment) second-level schedulers for cellsv2 - eventually in 1 or 2 cycles, nova-scheduler will become a library that conductors can use for filtering/weighting reasons. The idea is to stop doing RPC calls to a separate service that requires its own HA (and we know we have problems with, given schedulers are stateful in memory). Instead, we should make the scheduler modules stateless so operators would only need to scale out conductors for performance. In that model, I think conductors should be the engines responsible for making allocations. - scheduler doesn't have any idea on whether the instance request is for a move operation or a boot, but conductors do know that logic. Instead of adding more specific conditionals into the scheduler so it would make it a library more nova-centric, it's far better to avoid of any
Re: [openstack-dev] [nova] Boston Forum session recap - claims in the scheduler (or conductor)
Le 19/05/2017 10:02, Sylvain Bauza a écrit : > > > Le 19/05/2017 02:55, Matt Riedemann a écrit : >> The etherpad for this session is here [1]. The goal for this session was >> to inform operators and get feedback on the plan for what we're doing >> with moving claims from the computes to the control layer (scheduler or >> conductor). >> >> We mostly talked about retries, which also came up in the cells v2 >> session that Dan Smith led [2] and will recap later. >> >> Without getting into too many details, in the cells v2 session we came >> to a compromise on build retries and said that we could pass hosts down >> to the cell so that the cell-level conductor could retry if needed (even >> though we expect doing claims at the top will fix the majority of >> reasons you'd have a reschedule in the first place). >> > > And during that session, we said that given cell-local conductors (when > there is a reschedule) can't upcall the global (for all cells) > schedulers, that's why we agreed to use the conductor to be calling > Placement API for allocations. > > >> During the claims in the scheduler session, a new wrinkle came up which >> is the hosts that the scheduler returns to the top-level conductor may >> be in different cells. So if we have two cells, A and B, with hosts x >> and y in cell A and host z in cell B, we can't send z to A for retries, >> or x or y to B for retries. So we need some kind of post-filter/weigher >> filtering such that hosts are grouped by cell and then they can be sent >> to the cells for retries as necessary. >> > > That's already proposed for reviews in > https://review.openstack.org/#/c/465175/ > > >> There was also some side discussion asking if we somehow regressed >> pack-first strategies by using Placement in Ocata. John Garbutt and Dan >> Smith have the context on this (I think) so I'm hoping they can clarify >> if we really need to fix something in Ocata at this point, or is this >> more of a case of closing a loop-hole? >> > > The problem is that the scheduler doesn't verify the cells when trying > to find a destination for an instance, it's just using weights for packing. > > So, for example, say I have N hosts and 2 cells, the first weighting > host could be in cell1 while the second could be in cell2. Then, even if > the operator uses the weighers for packing, for example a RequestSpec > with num_instances=2 could push one instance in cell1 and the other in > cell2. > > From a scheduler point of view, I think we could possibly add a > CellWeigher that would help to pack instances within the same cell. > Anyway, that's not related to the claims series, so we could possibly > backport it for Ocata hopefully. > Melanie actually made a good point about the current logic based on the `host_subset_size`config option. If you're leaving it defaulted to 1, in theory all instances coming along the scheduler would get a sorted list of hosts by weights and only pick the first one (ie. packing all the instances onto the same host) which is good for that (except of course some user request that fits all the space of the host and where a spread could be better by shuffling between multiple hosts). So, while I began deprecating that option because I thought the race condition would be fixed by conductor claims, I think we should keep it for the time being until we clearly identify whether it's still necessary. All what I said earlier above remains valid tho. In a world where 2 hosts are given as the less weighed ones, we could send instances from the same user request onto different cells, but that only ties the problem to a multi-instance boot problem, which is far less impactful. > >> We also spent a good chunk of the session talking about overhead >> calculations for memory_mb and disk_gb which happens in the compute and >> on a per-hypervisor basis. In the absence of automating ways to adjust >> for overhead, our solution for now is operators can adjust reserved host >> resource values (vcpus, memory, disk) via config options and be >> conservative or aggressive as they see fit. Chris Dent and I also noted >> that you can adjust those reserved values via the placement REST API but >> they will be overridden by the config in a periodic task - which may be >> a bug, if not at least a surprise to an operator. >> >> We didn't really get into this during the forum session, but there are >> different opinions within the nova dev team on how to do claims in the >> controller services (conductor vs scheduler). Sylvain Bauza has a series >> which uses the conductor service, and Ed Leafe has a series using the >> scheduler. More on that in the mailing list [3]. >> > > Sorry, but I do remember we had a consensus on using conductor at least > during the cells v2 session. > > What I'm a bit afraid is that we're duplicating efforts on a sole > blueprint while we all agreed to go that way. > >> Next steps are going to be weighing both options between Sylvain and Ed, >> picki
Re: [openstack-dev] [nova] Boston Forum session recap - claims in the scheduler (or conductor)
On 19 May 2017 at 10:03, Sylvain Bauza wrote: > > > Le 19/05/2017 10:02, Sylvain Bauza a écrit : >> >> >> Le 19/05/2017 02:55, Matt Riedemann a écrit : >>> The etherpad for this session is here [1]. The goal for this session was >>> to inform operators and get feedback on the plan for what we're doing >>> with moving claims from the computes to the control layer (scheduler or >>> conductor). >>> >>> We mostly talked about retries, which also came up in the cells v2 >>> session that Dan Smith led [2] and will recap later. >>> >>> Without getting into too many details, in the cells v2 session we came >>> to a compromise on build retries and said that we could pass hosts down >>> to the cell so that the cell-level conductor could retry if needed (even >>> though we expect doing claims at the top will fix the majority of >>> reasons you'd have a reschedule in the first place). >>> >> >> And during that session, we said that given cell-local conductors (when >> there is a reschedule) can't upcall the global (for all cells) >> schedulers, that's why we agreed to use the conductor to be calling >> Placement API for allocations. >> >> >>> During the claims in the scheduler session, a new wrinkle came up which >>> is the hosts that the scheduler returns to the top-level conductor may >>> be in different cells. So if we have two cells, A and B, with hosts x >>> and y in cell A and host z in cell B, we can't send z to A for retries, >>> or x or y to B for retries. So we need some kind of post-filter/weigher >>> filtering such that hosts are grouped by cell and then they can be sent >>> to the cells for retries as necessary. >>> >> >> That's already proposed for reviews in >> https://review.openstack.org/#/c/465175/ >> >> >>> There was also some side discussion asking if we somehow regressed >>> pack-first strategies by using Placement in Ocata. John Garbutt and Dan >>> Smith have the context on this (I think) so I'm hoping they can clarify >>> if we really need to fix something in Ocata at this point, or is this >>> more of a case of closing a loop-hole? >>> >> >> The problem is that the scheduler doesn't verify the cells when trying >> to find a destination for an instance, it's just using weights for packing. >> >> So, for example, say I have N hosts and 2 cells, the first weighting >> host could be in cell1 while the second could be in cell2. Then, even if >> the operator uses the weighers for packing, for example a RequestSpec >> with num_instances=2 could push one instance in cell1 and the other in >> cell2. >> >> From a scheduler point of view, I think we could possibly add a >> CellWeigher that would help to pack instances within the same cell. >> Anyway, that's not related to the claims series, so we could possibly >> backport it for Ocata hopefully. >> > > Melanie actually made a good point about the current logic based on the > `host_subset_size`config option. If you're leaving it defaulted to 1, in > theory all instances coming along the scheduler would get a sorted list > of hosts by weights and only pick the first one (ie. packing all the > instances onto the same host) which is good for that (except of course > some user request that fits all the space of the host and where a spread > could be better by shuffling between multiple hosts). > > So, while I began deprecating that option because I thought the race > condition would be fixed by conductor claims, I think we should keep it > for the time being until we clearly identify whether it's still necessary. > > All what I said earlier above remains valid tho. In a world where 2 > hosts are given as the less weighed ones, we could send instances from > the same user request onto different cells, but that only ties the > problem to a multi-instance boot problem, which is far less impactful. FWIW, I think we need to keep this. If you have *lots* of contention when picking your host, increasing host_subset_size should help reduce that contention (and maybe help increase the throughput). I haven't written a simulator to test it out, but it feels like we will still need to keep the fuzzy select. That might just be a different way to say the same thing mel was saying, not sure. Thanks, johnthetubaguy __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Boston Forum session recap - claims in the scheduler (or conductor)
Le 19/05/2017 12:19, John Garbutt a écrit : > On 19 May 2017 at 10:03, Sylvain Bauza wrote: >> >> >> Le 19/05/2017 10:02, Sylvain Bauza a écrit : >>> >>> >>> Le 19/05/2017 02:55, Matt Riedemann a écrit : The etherpad for this session is here [1]. The goal for this session was to inform operators and get feedback on the plan for what we're doing with moving claims from the computes to the control layer (scheduler or conductor). We mostly talked about retries, which also came up in the cells v2 session that Dan Smith led [2] and will recap later. Without getting into too many details, in the cells v2 session we came to a compromise on build retries and said that we could pass hosts down to the cell so that the cell-level conductor could retry if needed (even though we expect doing claims at the top will fix the majority of reasons you'd have a reschedule in the first place). >>> >>> And during that session, we said that given cell-local conductors (when >>> there is a reschedule) can't upcall the global (for all cells) >>> schedulers, that's why we agreed to use the conductor to be calling >>> Placement API for allocations. >>> >>> During the claims in the scheduler session, a new wrinkle came up which is the hosts that the scheduler returns to the top-level conductor may be in different cells. So if we have two cells, A and B, with hosts x and y in cell A and host z in cell B, we can't send z to A for retries, or x or y to B for retries. So we need some kind of post-filter/weigher filtering such that hosts are grouped by cell and then they can be sent to the cells for retries as necessary. >>> >>> That's already proposed for reviews in >>> https://review.openstack.org/#/c/465175/ >>> >>> There was also some side discussion asking if we somehow regressed pack-first strategies by using Placement in Ocata. John Garbutt and Dan Smith have the context on this (I think) so I'm hoping they can clarify if we really need to fix something in Ocata at this point, or is this more of a case of closing a loop-hole? >>> >>> The problem is that the scheduler doesn't verify the cells when trying >>> to find a destination for an instance, it's just using weights for packing. >>> >>> So, for example, say I have N hosts and 2 cells, the first weighting >>> host could be in cell1 while the second could be in cell2. Then, even if >>> the operator uses the weighers for packing, for example a RequestSpec >>> with num_instances=2 could push one instance in cell1 and the other in >>> cell2. >>> >>> From a scheduler point of view, I think we could possibly add a >>> CellWeigher that would help to pack instances within the same cell. >>> Anyway, that's not related to the claims series, so we could possibly >>> backport it for Ocata hopefully. >>> >> >> Melanie actually made a good point about the current logic based on the >> `host_subset_size`config option. If you're leaving it defaulted to 1, in >> theory all instances coming along the scheduler would get a sorted list >> of hosts by weights and only pick the first one (ie. packing all the >> instances onto the same host) which is good for that (except of course >> some user request that fits all the space of the host and where a spread >> could be better by shuffling between multiple hosts). >> >> So, while I began deprecating that option because I thought the race >> condition would be fixed by conductor claims, I think we should keep it >> for the time being until we clearly identify whether it's still necessary. >> >> All what I said earlier above remains valid tho. In a world where 2 >> hosts are given as the less weighed ones, we could send instances from >> the same user request onto different cells, but that only ties the >> problem to a multi-instance boot problem, which is far less impactful. > > FWIW, I think we need to keep this. > > If you have *lots* of contention when picking your host, increasing > host_subset_size should help reduce that contention (and maybe help > increase the throughput). I haven't written a simulator to test it > out, but it feels like we will still need to keep the fuzzy select. > That might just be a different way to say the same thing mel was > saying, not sure. > Yup, agreed, thanks to Mel, that's why I'm providing a new revision that is no longer removing this conf opt. Melanie, very good point! -Sylvain > Thanks, > johnthetubaguy > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.op
Re: [openstack-dev] [nova] Boston Forum session recap - claims in the scheduler (or conductor)
On Thu, 18 May 2017, Matt Riedemann wrote: We didn't really get into this during the forum session, but there are different opinions within the nova dev team on how to do claims in the controller services (conductor vs scheduler). Sylvain Bauza has a series which uses the conductor service, and Ed Leafe has a series using the scheduler. More on that in the mailing list [3]. Since we've got multiple threads going on this topic, I put some of my concerns in a comment on one of Ed's reviews: https://review.openstack.org/#/c/465171/3//COMMIT_MSG@30 It's a bit left fieldy but tries to ask about some of the long term concerns we may need to be thinking about here, with regard to other services using placement and maybe them needing a scheduler-like-thing too (because placement cannot do everything). -- Chris Dent ┬──┬◡ﾉ(° -°ﾉ) https://anticdent.org/ freenode: cdent tw: @anticdent__ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Boston Forum session recap - claims in the scheduler (or conductor)
On 5/19/2017 8:14 AM, Chris Dent wrote: On Thu, 18 May 2017, Matt Riedemann wrote: We didn't really get into this during the forum session, but there are different opinions within the nova dev team on how to do claims in the controller services (conductor vs scheduler). Sylvain Bauza has a series which uses the conductor service, and Ed Leafe has a series using the scheduler. More on that in the mailing list [3]. Since we've got multiple threads going on this topic, I put some of my concerns in a comment on one of Ed's reviews: https://review.openstack.org/#/c/465171/3//COMMIT_MSG@30 It's a bit left fieldy but tries to ask about some of the long term concerns we may need to be thinking about here, with regard to other services using placement and maybe them needing a scheduler-like-thing too (because placement cannot do everything). __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev Thanks for pointing this out Chris. I still need to read Ed's blog post (separate thread), but we've said for a few releases now that the long-term goal is to split placement out of nova, and leave nova-scheduler within nova, since the scheduler code is very nova-specific (think of all the affinity sync stuff). I appreciate thinking big and long-term, but I really don't want to spend a lot of time right now questioning every move we make and whether that might hurt the ability to split the scheduler code out in 5 years because some other project wants to use it - which is Gantt 2.0 and I haven't heard any other project express an interest in something like that. I don't know how other projects do claims today, and how different they are from how nova does them. So there would seem to be a lot of discovery involved before we could even go down that path, not to mention all of the nova-specific code that would need to come out of the scheduler, or be made generic/pluggable. Again, I think if we spend too much time thinking about that we're not going to get anything done in Pike. -- Thanks, Matt __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [nova] Boston Forum session recap - claims in the scheduler (or conductor)
Le 19/05/2017 15:14, Chris Dent a écrit : > On Thu, 18 May 2017, Matt Riedemann wrote: > >> We didn't really get into this during the forum session, but there are >> different opinions within the nova dev team on how to do claims in the >> controller services (conductor vs scheduler). Sylvain Bauza has a >> series which uses the conductor service, and Ed Leafe has a series >> using the scheduler. More on that in the mailing list [3]. > > Since we've got multiple threads going on this topic, I put some > of my concerns in a comment on one of Ed's reviews: > > https://review.openstack.org/#/c/465171/3//COMMIT_MSG@30 > > It's a bit left fieldy but tries to ask about some of the long term > concerns we may need to be thinking about here, with regard to other > services using placement and maybe them needing a > scheduler-like-thing too (because placement cannot do everything). > That's actually a good question that I would translate to : 'Are other projects having interest in scheduling other things than just instances ?'. To be honest, that's something I wondered since ages and I tried during the VM/BM Forum session [1] to ask operators/developers their usecases they'd like to see for placement given the priority they gave. If you look at the etherpad, you will see a couple of given usecases but none of them are related to a generic scheduler, rather a compute scheduler doing multi-projects affinity, which is already in our scope thanks to Placement. So, while I think it's a reasonable question to ask, it shouldn't divert our current priority effort as it can't be said a motivation. Also, I'm not particularly concerned by the interface between conductor and scheduler that we have, as that interface is flexible enough for not blocking us in the future, should we need to implement a generic scheduler. -Sylvain > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
