On Mon, Sep 4, 2017 at 9:11 PM, Jay Pipes <[email protected]> wrote:
On 09/01/2017 04:42 AM, Rua, Philippe (Nokia - FI/Espoo) wrote:
> Will it be possible to include network bandwidth as a resource in
Nova scheduling, for VM placement decision?
Yes.
See here for a related Neutron spec that mentions Placement:
https://review.openstack.org/#/c/396297/7/specs/pike/strict-minimum-bandwidth-support.rst
> Context: in telecommunication applications, the network traffic is
an important dimension of resource usage. For example, it is often
important to distribute "bandwidth-greedy" VMs to different compute
nodes. There were some earlier discussions on this topic, but I could
not find a concrete outcome. [1][2][3]
>
> After some reading, I wonder whether the Custom resource classes
can provide a generic mechanism? [4][5][6]
No :) Custom resource classes are antithetical to generic/standard
mechanisms.
We want to add two *standard* resource classes, one called
NET_INGRESS_BYTES_SEC and another called NET_EGRESS_BYTES_SEC which
would represent the total bandwidth in bytes per second the for
corresponding traffic directions.
While I agree that the end goal is to have standard resource classes
for bandwidth I think custom resource classes are generic enough to
model bandwidth resource. If you want to play with the bandwidth based
scheduling idea based on Pike then custom resource classes are
available as a tool for a proof of concept.
What would be the resource provider, though? There are at least two
potential answers here:
1) A network interface controller on the compute host
In this case, the NIC on the host would be a child provider of the
compute host resource provider. It would have an inventory record of
resource class NET_INGRESS_BYTES_SEC with a total value representing
the
entire bandwidth of the host NIC. Instances would consume some amount
of
NET_INGRESS_BYTES_SEC corresponding to *either* the Nova flavor (if
the
resources:NET_INGRESS_BYTES_SEC extra-spec is set) *or* to the sum of
consumed bandwidth amounts from the port profile of any ports
specified
when launching the instance (and thus would be part of the pci device
request collection attached to the build request).
2) A "network slice" of a network interface controller on the compute
host
In this case, assume that the NIC on the compute host has had its
total
bandwidth constrained via traffic control so that 50% of its available
ingress bandwidth is allocated to network A and 50% is allocated to
network B.
There would be multiple resources providers, each with an inventory
record of resource class NET_INGRESS_BYTES_SEC with a total value of
1/2
the total NIC bandwidth. Both of these resource providers would be
child
providers of the compute host resource provider. One of these child
resource providers will be decorated with the trait "CUSTOM_NETWORK_A"
and the other with trait "CUSTOM_NETWORK_B".
The scheduler would be able to determine which resource provider to
consume the NET_INGRESS_BYTES_SEC resources from by looking for a
resource provider that has both the required amount of
NET_INGRESS_BYTES_SEC as well as the trait required by the port
profile.
If, say, the port profile specifies that the port is to go on a NIC
with
access to network "A", then the build request would contain a request
to
the scheduler for CUSTOM_NETWORK_A trait...
The above setup can be simulated with custom resource classes and
individual resource providers per compute node connected to the given
compute node's resource provider via an aggregate. You most probably
need to simulate the above network traits with individual custom
resource classes in Pike.
I definitely don't think it is something I would do in production based
on Pike due to two reasons:
1) we have bugs in Pike GA that prevents nova to handle some edge cases
(especially in VM moving scenarios)
2) I agree with Jay that nested providers and neutron support will
allows us to do something much more cleaner in the future.
However I think Pike is a good base to build a PoC and gather feedback.
For example I already foresee a need to model OVS packet processing
limits and in the long run even include the capacity of the TOR
switches into the picture.
If you're coming to Denver, I encourage you to get with me, Sean
Mooney,
Moshe Levi and others who are interested in seeing this work move
forward.
@Jay: sign me up for this list.
Cheers,
gibi
Best,
-jay
> Here is what I have in mind:
> - The VM need is specified in the flavor extra-specs, e.g.
resources:CUSTOM_BANDWIDTH=123.
> - The compute node total capacity is specified in host aggregate
metadata, e.g. CUSTOM_BANDWIDTH=999.
> - Nova then takes care of the rest: scheduling where the free
capacity is sufficient, and performing simple resource usage
accounting (updating the compute node free network bandwidth capacity
as required).
>
> Is the outline above according to current plans?
> If not, what would be possible/needed in order to achieve the same
result, i.e. consider the VM network traffic need during VM placement?
>
> BR,
> Philippe
>
> [1]
https://blueprints.launchpad.net/nova/+spec/bandwidth-as-scheduler-metric
> [2] https://wiki.openstack.org/wiki/NetworkBandwidthEntitlement
> [3]
https://openstack.nimeyo.com/80515/openstack-scheduling-bandwidth-resources-nic_bw_kb-resource
> [4] https://docs.openstack.org/nova/latest/user/placement.html
> [5]
http://specs.openstack.org/openstack/nova-specs/priorities/pike-priorities.html#placement
> [6] https://review.openstack.org/#/c/473627/
>
>
__________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
[email protected]?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe:
[email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: [email protected]?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
