On Wed, Feb 03, 2016 at 12:50:02PM +1100, David Gibson wrote: > On Tue, Feb 02, 2016 at 04:33:27PM -0200, Eduardo Habkost wrote: > > On Mon, Feb 01, 2016 at 04:35:17PM +1100, David Gibson wrote: > > > Hi, > > > > > > It seems to me we're getting rather bogged down in how to proceed with > > > an improved CPU hotplug (and hot unplug) interface, both generically > > > and for ppc in particular. > > > > > > So here's a somewhat more concrete suggestion of a way forward, to see > > > if we can get some consensus. > > > > > > The biggest difficulty I think we're grappling with is that device-add > > > is actually *not* a great interface to cpu hotplug. Or rather, it's > > > not great as the _only_ interface: in order to represent the many > > > different constraints on how cpus can be plugged on various platforms, > > > it's natural to use a heirarchy of cpu core / socket / package types > > > specific to the specific platform or real-world cpu package being > > > modeled. However, for the normal case of a regular homogenous (and at > > > least slightly para-virtualized) server, that interface is nasty for > > > management layers because they have to know the right type to > > > instantiate. > > > > > > To address this, I'm proposing this two layer interface: > > > > > > Layer 1: Low-level, device-add based > > > > > > * a new, generic cpu-package QOM type represents a group of 1 or > > > more cpu threads which can be hotplugged as a unit > > > * cpu-package is abstract and can't be instantiated directly > > > * archs and/or individual platforms have specific subtypes of > > > cpu-package which can be instantiated > > > * for platforms attempting to be faithful representations of real > > > hardware these subtypes would match the specific characteristics > > > of the real hardware devices. In addition to the cpu threads, > > > they may have other on chip devices as sub-objects. > > > * for platforms which are paravirtual - or which have existing > > > firmware abstractions for cpu cores/sockets/packages/whatever - > > > these could be more abstract, but would still be tied to that > > > platform's constraints > > > * Depending on the platform the cpu-package object could have > > > further internal structure (e.g. a package object representing a > > > socket contains package objects representing each core, which in > > > turn contain cpu objects for each thread) > > > * Some crazy platform that has multiple daughterboards each with > > > several multi-chip-modules each with several chips, each > > > with several cores each with several threads could represent > > > that too. > > > > What exactly in this approach makes it depend on device-add? We > > could have something very similar based on creation of QOM > > objects, for example. > > Uh.. I guess it doesn't. device_add just seemed the obvious thing to > me. > > > > What would be common to all the cpu-package subtypes is: > > > * A boolean "present" attribute ("realized" might already be > > > suitable, but I'm not certain) > > > > "realized" might be suitable, but I am not even sure we want > > cpu-package to be a TYPE_DEVICE subclass. It could be a simple > > QOM class or even a QOM interface (machines could choose to > > implement it as TYPE_DEVICE, or not). > > Yeah, I think doing it as a QOM interface makes sense. > > > > * A generic means of determining the number of cpu threads in the > > > package, and enumerating those > > > > This could be based on QOM links. > > Yes, that makes sense. > > > > * A generic means of determining if the package is hotpluggable or > > > not > > > > Isn't this a machine attribute, instead of a package attribute? > > Not necessarily. I was thinking of cases where for architectural > reasons you can't hotplug chip/cpu/module 0 but can plug or unplug all > the rest.
Right, that makes sense too. But you can't query the cpu-package object unless it was already created. I guess that's another reason to start with the Layer 2 solution (where there's no need for on-the-fly creation of cpu-package objects). > > > > * They'd get listed in a standard place in the QOM tree > > > > If we allow CPU thread enumeration and package enumeration be > > based in QOM links, we can let machines implement those > > interfaces without introducing QOM hierarchy requirements. > > > > We have one example where we would need to make this flexible > > enough about QOM hierarchy, below (thread-based hotplug in x86). > > Makes sense. > > > > > > > > > This interface is suitable if you want complete control over > > > constructing the system, including weird cases like heterogeneous > > > machines (either totally different cpu types, or just different > > > numbers of threads in different packages). > > > > > > The intention is that these objects would never look at the global cpu > > > type or sockets/cores/threads numbers. The next level up would > > > instead configure the packages to match those for the common case. > > > > > > Layer 2: Higher-level > > > > > > * not all machine types need support this model, but I'd expect > > > all future versions of machine types designed for production use > > > to do so > > > * machine types don't construct cpu objects directly > > > * instead they create enough cpu-package objects - of a subtype > > > suitable for this machine - to provide maxcpus threads > > > * the machine type would set the "present" bit on enough of the > > > cpu packages to provide the base number of cpu threads > > > > Sounds interesting, and very simple for management code. What I > > don't see is: what exactly makes it easier to implement just > > Layer 1 and not Layer 2? > > > > Implementing Layer 1 looks more difficult to me, because it > > requires supporting creation of cpu-package objects on the fly, > > using device_add (or whatever mechanism we choose for cpu-package > > creation). Layer 2 lets the implementation choose freely when/how > > exactly the other objects will be created and how exactly they > > will appear in the device tree. They just need to do the right > > thing when the "present" property is flipped. > > Hmm.. good point. Ok, how about this revised plan: > > 1. Implement the QOM backend structures for cpu packages, but don't > allow them to be user instantiated > 2. Implement Layer 2 in terms of (1) > 3. When/if we need it, add the extra stuff necessary to allow direct > instantiation of the cpu packages Makes sense to me. With generic mechanisms to let management enumerate the socket/core/thread IDs contained/available in each cpu-package slot, this should be good enough for all architectures and still give management enough information to decide how to plug/unplug specific CPUs. I have another suggestion: if we are going to expose these objects where new CPUs could be plugged by just flipping a property, what about naming them "cpu-slot"? Because they wouldn't represent an actual CPU package, but just a slot where a CPU package could be plugged in. The initial (simpler) solution could involve just flipping a property in the cpu-slot. Later, we may allow more complex scenarios where complex QOM objects are created before linking them to the cpu-slot. > > > > Management layers can then manage hotplug without knowing platform > > > specifics by using qmp to toggle the "present" bit on packages. > > > Platforms that allow thread-level pluggability can expose a package > > > for every thread, those that allow core-level expose a package per > > > core, those that have even less granularity expose a package at > > > whatever grouping they can do hotplug on. > > > > > > Examples: > > > > > > For use with pc (or q35 or whatever) machine type, I'd expect a > > > cpu-package subtype called, say "acpi-thread" which represents a > > > single thread in the ACPI sense. Toggling those would trigger ACPI > > > hotplug events as cpu_add does now. > > > > You have a good point here: I remember seeing suggestions of > > making CPU hotplug tied to the socket/core/thread hierarchy > > somehow. But this won't change the fact that x86 allows hotplug > > of individual CPU threads. > > Right, this seems to be where we're bogged down - we seem to be going > back and forth betweeh core level, socket level, thread level > proposals without really looking at the big picture to come up with a > scheme that works for all platforms. > > > In other words, if we make a /machine/socket/core/thread QOM > > hierarchy, the cpu-packages for x86 won't necessarily represent > > CPU sockets (but in other architectures, they might). The > > interface needs to be generic enough to not assume anything about > > the CPU topology level where CPU hotplug happens. > > Exactly. That's what I see as the key advantage of this proposal over > earlier ones. Right. > > > > For use with pseries, I'd expect a "papr-core" cpu-package subtype, > > > which represents a single (paravirtual) core. Toggling present on > > > this would trigger the PAPR hotplug events. A property would control > > > the number of threads in the core (only settable before enabling > > > present). > > > > > > For use with the powernv machine type (once ready for merge) I'd > > > expect "POWER8-package" type which represents a POWER8 chip / module > > > as close to the real hardware as we can get. It would have a fixed > > > number of cores and threads within it as per the real hardware, and > > > would also include xscoms and other per-module logic. > > > > > > From here to there: > > > > > > A suggested order of implementation to get there without too much risk > > > of breaking things. > > > > > > 1. Fix bugs with creation / removal of CPU objects (Bharata's cpu > > > hotplug series already has this) > > > 2. Split creation and realization of CPU objects, so machine types > > > must explicitly perform both steps (Bharata's series has this > > > too) > > > 3. Add the abstract cpu-package type, and define the generic > > > interfaces it needs (Bharata's series has something that could be > > > changed to this fairly easily) > > > 4. For each machine type we care to convert: > > > 4.1. Add platform suitable cpu-package subtypes > > > 4.2. Convert the (latest version) machine type to instantiate > > > packages instead of > > > cpu threads directly > > > > Machines could even have the freedom to instantiate CPU threads > > directly and then set up package objects for them. Reusing > > generic code is useful, but it doesn't even need to be mandatory, > > as long as the objects are available at the right place in the > > QOM hierarchy. > > Ah, yes, I guess so. > > > > > > 4.3. Add any necessary backwards compat goo > > > 5. Teach libvirt how to toggle cpu-packages > > > > This is different from the very flexible QOM object > > building/linking approach Andreas was talking about in last KVM > > Forum. > > So, I know this stuff was discussed at KVM Forum, but unfortunately I > never got a clear picture of what the outcome was. I didn't see a clear conclusion after the KVM Forum discussions, except for some ideas to use QOM object creation/linking to implement something flexible enough for all architectures. But I don't think we were moving towards something generic enough that would allow management to implement CPU hotplug without arch-specific code to setup CPU sockets/packages/cores/threads/etc. > > But while I would love to have the ability to build > > arbitrary QOM hierarchies with complex links between CPUs > > sockets, cores, threads, etc, I believe we need an interface that > > is: 1) generic enough for multiple architectures and machines to > > implement them; 2) simple enough so that libvirt can use it > > easily without requiring more arch-specific code. > > > > Also, your approach doesn't prevent the simple cpu-package > > interface from having a complex QOM hierarchy hidden behind it. > > Exactly. -- Eduardo