Hi Daniel and folks,
Let me pick up this thread again to discuss some updates from my work
on QOM CPU topology. I would like to know if I'm on the right track. :-)
On Mon, Feb 13, 2023 at 01:38:28PM +0000, Daniel P. Berrangé wrote:
>
[snip]
>
> IIUC the functionality offered by -hybrid should be a superset
> of the -smp functionality. IOW, -smp ought to be possible to
> re-implement -smp as an alias for -hybrid, such that internally
> code only ever has to deal with the modern approach. Having to
> keep support for both -smp and -hybrid throughout the code is
> undesirable IMHO. Keeping the compat at the CLI parsing level
> limits the burden.
>
>
> As a more general thought, rather than introducing a new top level
> command line argument -hybrid, I'm thinking we should possibly just
> define this all using QOM and thus re-use the existing -object
> argument.
>
> I'm also finding the above example command lines quite difficult
> to understand, as there is alot of implicit linkage and expansion
> between the different levels. With devices we're much more
> explicitly with the parent/child relationships, and have to
> express everything with no automatic expansion, linking it all
> together via the id=/bus= properties. This is quite a bit more
> verbose, but it is also very effective at letting us express
> arbitrarily complex relationships.
>
> I think it would be worth exploring that approach for the CPU
> topology expression too.
>
> If we followed the more explicit device approach to modelling
> then instead of:
>
> -cpu core,...
> -cpu atom,...
> -hybrid socket,sockets=1
> -hybrid die,dies=1
> -hybrid cluster,clusters=4
> -hybrid core,cores=1,coretype="core",threads=2,clusterid=0-2
> -hybrid core,cores=4,coretype="atom",threads=1
>
> we would end up with something like
>
> -object cpu-socket,id=sock0
> -object cpu-die,id=die0,parent=sock0
> -object cpu-cluster,id=cluster0,parent=die0
> -object cpu-cluster,id=cluster1,parent=die0
> -object cpu-cluster,id=cluster2,parent=die0
> -object cpu-cluster,id=cluster3,parent=die0
> -object x86-cpu-model-atom,id=cpu0,parent=cluster0
> -object x86-cpu-model-atom,id=cpu1,parent=cluster0
> -object x86-cpu-model-atom,id=cpu2,parent=cluster0
> -object x86-cpu-model-atom,id=cpu3,parent=cluster0
> -object x86-cpu-model-core,id=cpu4,parent=cluster0,threads=2
> -object x86-cpu-model-atom,id=cpu5,parent=cluster1
> -object x86-cpu-model-atom,id=cpu6,parent=cluster1
> -object x86-cpu-model-atom,id=cpu7,parent=cluster1
> -object x86-cpu-model-atom,id=cpu8,parent=cluster1
> -object x86-cpu-model-core,id=cpu9,parent=cluster1,threads=2
> -object x86-cpu-model-atom,id=cpu10,parent=cluster2
> -object x86-cpu-model-atom,id=cpu11,parent=cluster2
> -object x86-cpu-model-atom,id=cpu12,parent=cluster2
> -object x86-cpu-model-atom,id=cpu13,parent=cluster2
> -object x86-cpu-model-core,id=cpu14,parent=cluster2,threads=2
> -object x86-cpu-model-atom,id=cpu15,parent=cluster3
> -object x86-cpu-model-atom,id=cpu16,parent=cluster3
> -object x86-cpu-model-atom,id=cpu17,parent=cluster3
> -object x86-cpu-model-atom,id=cpu18,parent=cluster3
> -object x86-cpu-model-core,id=cpu19,parent=cluster3,threads=2
>
> The really obvious downside is that it is much more verbose.
I find the "core" and "cluster" already have the abstraction as the
device, and Andreas also tried to abstract "socket" as the device.
So I find in this QOM way, all CPU topology levels should be
abstracted to devices and created via "-device". And I also need to
extend "cluster" device to support VM case as a general CPU topology
abstraction, not only just used in TCG emulation.
I've done a preliminary POC so far, where I can create the CPU topology
via "-device", and also added the ability to create the "child"
relationship in "-device" (since neither the "link" relationship nor
the bus looks like a good representation of the CPU hierarchy).
There's an example:
-device cpu-socket,id=sock0 \
-device cpu-die,id=die0,parent=sock0 \
-device cpu-die,id=die1,parent=sock0 \
-device cpu-cluster,id=cluster0,parent=die0 \
-device cpu-cluster,id=cluster1,parent=die0 \
-device cpu-cluster,id=cluster2,parent=die1 \
-device x86-intel-core,id=core0,parent=cluster0,threads=3 \
-device x86-intel-atom,id=core1,parent=cluster0,threads=2 \
-device x86-core,id=core2,parent=cluster1,threads=1 \
-device x86-intel-core,id=core3,parent=cluster2,threads=5 \
with the above format, I could build the the more accurate canonical
path for CPUs like this:
(QEMU) query-hotpluggable-cpus
{
"arguments": {},
"execute": "query-hotpluggable-cpus"
}
{
"return": [
{
"props": {
"cluster-id": 0,
"core-id": 0,
"die-id": 1,
"socket-id": 0,
"thread-id": 4
},
"qom-path":
"/machine/peripheral/cpu-slot/sock0/die1/cluster2/core3/host-x86_64-cpu[10]",
"type": "host-x86_64-cpu",
"vcpus-count": 1
},
{
"props": {
"cluster-id": 0,
"core-id": 0,
"die-id": 1,
"socket-id": 0,
"thread-id": 3
},
"qom-path":
"/machine/peripheral/cpu-slot/sock0/die1/cluster2/core3/host-x86_64-cpu[9]",
"type": "host-x86_64-cpu",
"vcpus-count": 1
},
{
"props": {
"cluster-id": 0,
"core-id": 0,
"die-id": 1,
"socket-id": 0,
"thread-id": 2
},
"qom-path":
"/machine/peripheral/cpu-slot/sock0/die1/cluster2/core3/host-x86_64-cpu[8]",
"type": "host-x86_64-cpu",
"vcpus-count": 1
},
{
"props": {
"cluster-id": 0,
"core-id": 0,
"die-id": 1,
"socket-id": 0,
"thread-id": 1
},
"qom-path":
"/machine/peripheral/cpu-slot/sock0/die1/cluster2/core3/host-x86_64-cpu[7]",
"type": "host-x86_64-cpu",
"vcpus-count": 1
},
{
"props": {
"cluster-id": 0,
"core-id": 0,
"die-id": 1,
"socket-id": 0,
"thread-id": 0
},
"qom-path":
"/machine/peripheral/cpu-slot/sock0/die1/cluster2/core3/host-x86_64-cpu[6]",
"type": "host-x86_64-cpu",
"vcpus-count": 1
},
{
"props": {
"cluster-id": 1,
"core-id": 0,
"die-id": 0,
"socket-id": 0,
"thread-id": 0
},
"qom-path":
"/machine/peripheral/cpu-slot/sock0/die0/cluster1/core2/host-x86_64-cpu[5]",
"type": "host-x86_64-cpu",
"vcpus-count": 1
},
{
"props": {
"cluster-id": 0,
"core-id": 1,
"die-id": 0,
"socket-id": 0,
"thread-id": 1
},
"qom-path":
"/machine/peripheral/cpu-slot/sock0/die0/cluster0/core1/host-x86_64-cpu[4]",
"type": "host-x86_64-cpu",
"vcpus-count": 1
},
{
"props": {
"cluster-id": 0,
"core-id": 1,
"die-id": 0,
"socket-id": 0,
"thread-id": 0
},
"qom-path":
"/machine/peripheral/cpu-slot/sock0/die0/cluster0/core1/host-x86_64-cpu[3]",
"type": "host-x86_64-cpu",
"vcpus-count": 1
},
{
"props": {
"cluster-id": 0,
"core-id": 0,
"die-id": 0,
"socket-id": 0,
"thread-id": 2
},
"qom-path":
"/machine/peripheral/cpu-slot/sock0/die0/cluster0/core0/host-x86_64-cpu[2]",
"type": "host-x86_64-cpu",
"vcpus-count": 1
},
{
"props": {
"cluster-id": 0,
"core-id": 0,
"die-id": 0,
"socket-id": 0,
"thread-id": 1
},
"qom-path":
"/machine/peripheral/cpu-slot/sock0/die0/cluster0/core0/host-x86_64-cpu[1]",
"type": "host-x86_64-cpu",
"vcpus-count": 1
},
{
"props": {
"cluster-id": 0,
"core-id": 0,
"die-id": 0,
"socket-id": 0,
"thread-id": 0
},
"qom-path":
"/machine/peripheral/cpu-slot/sock0/die0/cluster0/core0/host-x86_64-cpu[0]",
"type": "host-x86_64-cpu",
"vcpus-count": 1
}
]
}
Of course, I'm still a bit far from the full QOM topology support, but
would like to hear comments at the POC stage as well! :-)
Thanks and BR,
Zhao
>
> This example only has 20 CPUs. For a VM with say 1000 CPUs
> this will be very big, but that doesn't neccesarily make it
> wrong.
>
> On the flipside
>
> * It is really clear exactly how many CPUs I've added
>
> * The relationship between the topology levels is clear
>
> * Every CPU has a unique ID given that can be used in
> later QMP commands
>
> * Whether or not 'threads' are permitted is now a property
> of the specific CPU model implementation, not the global
> config. IOW we can express that some CPU models allowing
> for threads, and some don't.
>
> * The -cpu arg is also obsoleted, replaced by the
> -object x86-cpu-model-core. This might facilitate the
> modelling of machines with CPUs from different architectures.
>
>
> We could potentially compress the leaf node level by expressing
> how many instances of an object we want. it we want. ie, define
> a more convenient shorthand syntax to creating many instances of
> an object. so eg
>
> -object-set $TYPE,$PROPS,idbase=foo,count=4
>
> would be functionally identical to
>
> -object $TYPE,$PROPS,id=foo.0
> -object $TYPE,$PROPS,id=foo.1
> -object $TYPE,$PROPS,id=foo.2
> -object $TYPE,$PROPS,id=foo.3
>
> QEMU just expands it and creates all the objects internally.
>
> So the huge example I have above for 20 cpus would become much
> shorter: e.g.
>
> -object cpu-socket,id=sock0
> -object cpu-die,id=die0,parent=sock0
> -object cpu-cluster,id=cluster0,parent=die0
> -object cpu-cluster,id=cluster1,parent=die0
> -object cpu-cluster,id=cluster2,parent=die0
> -object cpu-cluster,id=cluster3,parent=die0
> -object-set x86-cpu-core-atom,idbase=cpu0,parent=cluster0,count=4
> -object-set x86-cpu-core-core,id=cpu1,parent=cluster0,threads=2,count=1
> -object-set x86-cpu-core-atom,idbase=cpu2,parent=cluster1,count=4
> -object-set x86-cpu-core-core,id=cpu3,parent=cluster1,threads=2,count=1
> -object-set x86-cpu-core-atom,idbase=cpu4,parent=cluster2,count=4
> -object-set x86-cpu-core-core,id=cpu5,parent=cluster2,threads=2,count=1
> -object-set x86-cpu-core-atom,idbase=cpu6,parent=cluster3,count=4
> -object-set x86-cpu-core-core,id=cpu7,parent=cluster3,threads=2,count=1
>
> IOW, the size of the CLI config only depends on the number of elements
> in the hierarchy, and is independant of the number of leaf CPU cores.
>
> Obviously in describing all of the above, I've ignored any complexity
> of dealing with our existing code implementation and pain of getting
> it converted to the new model.
>
> With regards,
> Daniel
> --
> |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org -o- https://fstop138.berrange.com :|
> |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
>
