On 4/3/23 23:38, Bui Quang Minh wrote:
On 4/3/23 17:27, David Woodhouse wrote:
On Wed, 2023-03-29 at 22:30 +0700, Bui Quang Minh wrote:
I do some more testing on my hardware, your point is correct when dest
== 0xffffffff, the interrupt is delivered to all APICs regardless of
their mode.
To be precisely, it only broadcasts to CPUs in xAPIC mode if the IPI
destination mode is physical. In case the destination mode is logical,
flat model/cluster model rule applies to determine if the xAPIC CPUs
accept the IPI. Wow, this is so complicated :)
So even if you send to *all* of the first 8 CPUs in a cluster (e.g.
cluster 0x0001 giving a destination 0x000100FF, a CPU in xAPIC mode
doesn't see that as a broadcast because it's logical mode?
I mean if the destination is 0xffffffff, the xAPIC CPU will see
destination as 0xff. 0xff is broadcast in physical destination mode
only, in logical destination, it may not be a broadcast. It may depend
on the whether it is flat model or cluster model in logical destination
mode.
In flat model, 8 bits are used as mask, so in theory, this model can
only support 8 CPUs, each CPU reserves its own bit by setting the upper
8 bits of APIC LDR register. In Intel SDM, it is said that 0xff can be
interpreted as a broadcast, this is true in normal case, but I think if
the CPU its APIC LDR to 0, the IPI should not deliver to this CPU. This
also matches with the current flat model of logical destination mode
implementation of userspace APIC in Qemu before my series. However, I
see this seems like a corner case, I didn't test it on real hardware.
With cluster model, when writing the above paragraphs, I think that 0xff
will be delivered to all APICs (mask = 0xf) of cluster 15 (0xf).
However, reading the SDM more carefully, I see that the there are only
15 clusters with address from 0 to 14 so there is no cluster with
address 15. 0xff is interpreted as broadcast to all APICs in all
clusters too.
In conclusion, IPI with destination 0xffffffff can be a broadcast to all
xAPIC CPUs too if we just ignore the corner case in flat model of
logical destination mode (we may need to test more)
I do some tests on my machine with custom Linux kernel module (I can't
use kvm-unit-tests because the display on my laptop is on embedded
display port not serial port so I don't know how to get the output). I
find out that
- In flat model, if the CPU intentionally set its 8 bit address to 0 in
APIC_LDR, the 0xff logical IPI does not deliver to this CPU.
- In cluster model, the 4 higher bits used as cluster address, if these
4 bits is equal to 0xf, the IPI is broadcasted to all cluster. The 4
lower bits are used to address APICs in the cluster. If the CPU
intentionally set these 4 lower bits to 0 in APIC_LDR, same as the flat
model, this CPU does not receive the logical IPI. For example, the CPUs
set its address to 0x20 (cluster 2, address 0 in cluster), the logical
IPI with destination == 0xff does deliver to cluster 2 but does not
deliver to this CPU.
So I choose to stick with the current implementation, 0xffffffff in IPI
is seen as 0xff IPI in xAPIC CPUs. This IPI if in physical mode is a
broadcast but not in logical mode.