On Tue, Feb 02, 2016 at 04:33:27PM -0200, Eduardo Habkost wrote: > On Mon, Feb 01, 2016 at 04:35:17PM +1100, David Gibson wrote: > > Hi, > > > > It seems to me we're getting rather bogged down in how to proceed with > > an improved CPU hotplug (and hot unplug) interface, both generically > > and for ppc in particular. > > > > So here's a somewhat more concrete suggestion of a way forward, to see > > if we can get some consensus. > > > > The biggest difficulty I think we're grappling with is that device-add > > is actually *not* a great interface to cpu hotplug. Or rather, it's > > not great as the _only_ interface: in order to represent the many > > different constraints on how cpus can be plugged on various platforms, > > it's natural to use a heirarchy of cpu core / socket / package types > > specific to the specific platform or real-world cpu package being > > modeled. However, for the normal case of a regular homogenous (and at > > least slightly para-virtualized) server, that interface is nasty for > > management layers because they have to know the right type to > > instantiate. > > > > To address this, I'm proposing this two layer interface: > > > > Layer 1: Low-level, device-add based > > > > * a new, generic cpu-package QOM type represents a group of 1 or > > more cpu threads which can be hotplugged as a unit > > * cpu-package is abstract and can't be instantiated directly > > * archs and/or individual platforms have specific subtypes of > > cpu-package which can be instantiated > > * for platforms attempting to be faithful representations of real > > hardware these subtypes would match the specific characteristics > > of the real hardware devices. In addition to the cpu threads, > > they may have other on chip devices as sub-objects. > > * for platforms which are paravirtual - or which have existing > > firmware abstractions for cpu cores/sockets/packages/whatever - > > these could be more abstract, but would still be tied to that > > platform's constraints > > * Depending on the platform the cpu-package object could have > > further internal structure (e.g. a package object representing a > > socket contains package objects representing each core, which in > > turn contain cpu objects for each thread) > > * Some crazy platform that has multiple daughterboards each with > > several multi-chip-modules each with several chips, each > > with several cores each with several threads could represent > > that too. > > What exactly in this approach makes it depend on device-add? We > could have something very similar based on creation of QOM > objects, for example.
Uh.. I guess it doesn't. device_add just seemed the obvious thing to me. > > What would be common to all the cpu-package subtypes is: > > * A boolean "present" attribute ("realized" might already be > > suitable, but I'm not certain) > > "realized" might be suitable, but I am not even sure we want > cpu-package to be a TYPE_DEVICE subclass. It could be a simple > QOM class or even a QOM interface (machines could choose to > implement it as TYPE_DEVICE, or not). Yeah, I think doing it as a QOM interface makes sense. > > * A generic means of determining the number of cpu threads in the > > package, and enumerating those > > This could be based on QOM links. Yes, that makes sense. > > * A generic means of determining if the package is hotpluggable or > > not > > Isn't this a machine attribute, instead of a package attribute? Not necessarily. I was thinking of cases where for architectural reasons you can't hotplug chip/cpu/module 0 but can plug or unplug all the rest. > > * They'd get listed in a standard place in the QOM tree > > If we allow CPU thread enumeration and package enumeration be > based in QOM links, we can let machines implement those > interfaces without introducing QOM hierarchy requirements. > > We have one example where we would need to make this flexible > enough about QOM hierarchy, below (thread-based hotplug in x86). Makes sense. > > > > > This interface is suitable if you want complete control over > > constructing the system, including weird cases like heterogeneous > > machines (either totally different cpu types, or just different > > numbers of threads in different packages). > > > > The intention is that these objects would never look at the global cpu > > type or sockets/cores/threads numbers. The next level up would > > instead configure the packages to match those for the common case. > > > > Layer 2: Higher-level > > > > * not all machine types need support this model, but I'd expect > > all future versions of machine types designed for production use > > to do so > > * machine types don't construct cpu objects directly > > * instead they create enough cpu-package objects - of a subtype > > suitable for this machine - to provide maxcpus threads > > * the machine type would set the "present" bit on enough of the > > cpu packages to provide the base number of cpu threads > > Sounds interesting, and very simple for management code. What I > don't see is: what exactly makes it easier to implement just > Layer 1 and not Layer 2? > Implementing Layer 1 looks more difficult to me, because it > requires supporting creation of cpu-package objects on the fly, > using device_add (or whatever mechanism we choose for cpu-package > creation). Layer 2 lets the implementation choose freely when/how > exactly the other objects will be created and how exactly they > will appear in the device tree. They just need to do the right > thing when the "present" property is flipped. Hmm.. good point. Ok, how about this revised plan: 1. Implement the QOM backend structures for cpu packages, but don't allow them to be user instantiated 2. Implement Layer 2 in terms of (1) 3. When/if we need it, add the extra stuff necessary to allow direct instantiation of the cpu packages > > Management layers can then manage hotplug without knowing platform > > specifics by using qmp to toggle the "present" bit on packages. > > Platforms that allow thread-level pluggability can expose a package > > for every thread, those that allow core-level expose a package per > > core, those that have even less granularity expose a package at > > whatever grouping they can do hotplug on. > > > > Examples: > > > > For use with pc (or q35 or whatever) machine type, I'd expect a > > cpu-package subtype called, say "acpi-thread" which represents a > > single thread in the ACPI sense. Toggling those would trigger ACPI > > hotplug events as cpu_add does now. > > You have a good point here: I remember seeing suggestions of > making CPU hotplug tied to the socket/core/thread hierarchy > somehow. But this won't change the fact that x86 allows hotplug > of individual CPU threads. Right, this seems to be where we're bogged down - we seem to be going back and forth betweeh core level, socket level, thread level proposals without really looking at the big picture to come up with a scheme that works for all platforms. > In other words, if we make a /machine/socket/core/thread QOM > hierarchy, the cpu-packages for x86 won't necessarily represent > CPU sockets (but in other architectures, they might). The > interface needs to be generic enough to not assume anything about > the CPU topology level where CPU hotplug happens. Exactly. That's what I see as the key advantage of this proposal over earlier ones. > > For use with pseries, I'd expect a "papr-core" cpu-package subtype, > > which represents a single (paravirtual) core. Toggling present on > > this would trigger the PAPR hotplug events. A property would control > > the number of threads in the core (only settable before enabling > > present). > > > > For use with the powernv machine type (once ready for merge) I'd > > expect "POWER8-package" type which represents a POWER8 chip / module > > as close to the real hardware as we can get. It would have a fixed > > number of cores and threads within it as per the real hardware, and > > would also include xscoms and other per-module logic. > > > > From here to there: > > > > A suggested order of implementation to get there without too much risk > > of breaking things. > > > > 1. Fix bugs with creation / removal of CPU objects (Bharata's cpu > > hotplug series already has this) > > 2. Split creation and realization of CPU objects, so machine types > > must explicitly perform both steps (Bharata's series has this > > too) > > 3. Add the abstract cpu-package type, and define the generic > > interfaces it needs (Bharata's series has something that could be > > changed to this fairly easily) > > 4. For each machine type we care to convert: > > 4.1. Add platform suitable cpu-package subtypes > > 4.2. Convert the (latest version) machine type to instantiate > > packages instead of > > cpu threads directly > > Machines could even have the freedom to instantiate CPU threads > directly and then set up package objects for them. Reusing > generic code is useful, but it doesn't even need to be mandatory, > as long as the objects are available at the right place in the > QOM hierarchy. Ah, yes, I guess so. > > > 4.3. Add any necessary backwards compat goo > > 5. Teach libvirt how to toggle cpu-packages > > This is different from the very flexible QOM object > building/linking approach Andreas was talking about in last KVM > Forum. So, I know this stuff was discussed at KVM Forum, but unfortunately I never got a clear picture of what the outcome was. > But while I would love to have the ability to build > arbitrary QOM hierarchies with complex links between CPUs > sockets, cores, threads, etc, I believe we need an interface that > is: 1) generic enough for multiple architectures and machines to > implement them; 2) simple enough so that libvirt can use it > easily without requiring more arch-specific code. > > Also, your approach doesn't prevent the simple cpu-package > interface from having a complex QOM hierarchy hidden behind it. Exactly. -- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
signature.asc
Description: PGP signature