2016-08-09 16:19+0800, Chao Gao: > On Tue, Aug 09, 2016 at 02:18:15PM +0800, Peter Xu wrote: >>On Tue, Aug 09, 2016 at 12:33:17PM +0800, Chao Gao wrote: >>> On Mon, Aug 08, 2016 at 04:57:14PM +0800, Peter Xu wrote: >>> >On Mon, Aug 08, 2016 at 03:41:23PM +0800, Chao Gao wrote: >>> >> HI, everyone. >>> >> >>> >> We have done some tests after merging this patch set into the lastest >>> >> qemu >>> >> master. In kvm aspect, we use the lastest kvm linux-next branch. Here are >>> >> some problems we have met. >>> >> >>> >> 1. We can't boot up a 288 vcpus linux guest with CLI: >>> >> qemu-system-x86_64 -boot c -m 4096 -sdl -monitor pty --enable-kvm \ >>> >> -M kernel-irqchip=split -serial stdio -bios bios.bin -smp cpus=288 \ >>> >> -hda vdisk.img -device intel-iommu,intremap=on -machine q35. >>> >> The problem exists, even after we only assign 32 vcpus to the linux >>> >> guest. >>> >> Maybe the output "do_IRQ: 146.113 No irq handler for vector (irq -1)" is >>> >> a clue. >>> >> The output of qemu and kernel is in attachments. Do you have any idea >>> >> about the problem and how to solve it? >>> > >>> >IIUC, we need to wait for Radim's QEMU patches to finally enable 288 >>> >vcpus? >>> > >>> >Btw, could you please try adding this to the QEMU cmdline when testing >>> >with 32 vcpus: >>> > >>> > -global ioapic.version=0x20 >>> > >>> >I see that you were running RHEL 7.2 guest with a default e1000. In >>> >that case, we may need to boost ioapic version to 0x20. >>> >>> It doesn't work. My host machine has 16 cpus. When I assign 4 or 8 vcpus to >>> the guest >>> or 255 vcpus but set "kernel-irqchip=off", the guest work well. Maybe when >>> irqchip >>> is in kernel, intremap can only handle situations that vcpus number is less >>> than >>> physical cpus'. Do you think it's right? >> >>I don't think so. Vcpu number should not be related to host cpu >>numbers. >> >>I think the problem is with x2apic. Currently, x2apic is enabled in >>vIOMMU when kernel irqchip is used. This is problematic, since >>actually we are throughing away dest_id[31:8] without Radim's patches, >>meanwhile I see that by default x2apic is using cluster mode. >> >>In cluster mode, 8 bits will possibly not suffice (I think the reason >>why >17 vcpus will bring trouble is that each cluster has 16 vcpus, >>we'll have trouble if we have more than one cluster). >> >>To temporarily solve your issue, you should not only need "-global >>ioapic.version=0x20" in QEMU command line, but also add "x2apic_phys" >>to you guest kernel boot parameter, to force guest boot with x2apic >>physical mode (not cluster mode). Though this can only work for <255 >>vcpus. IMHO we may still need to wait for Radim's patches to test >255 >>case. > > ok, after adding "x2apic_phys" to guest kernel boot parameter, I > boot up a 288(yes, 288) vcpus guest successfully with command > qemu-system-x86_64 -boot c -m 4096 -sdl --enable-kvm \ > -M kernel-irqchip=split -bios bios.bin -smp cpus=288 -hda vdisk.img \ > -device intel-iommu,intremap=on -machine q35 > Also, I can see interrupts on those cpu with inital apicid>255 from > /proc/cpuinfo and /proc/interrupts.
Great, thanks for testing! Only IPIs will be correctly delivered to apic_id > 255 without few more patches on the QEMU side, though.