Hi Jean and Ard,
On 6/1/24 01:09, Jean-Philippe Brucker wrote:
On Fri, May 31, 2024 at 04:23:13PM +1000, Gavin Shan wrote:
I got a chance to try CCA software components, suggested by [1]. However, the
edk2
is stuck somewhere. I didn't reach to stage of loading guest kernel yet. I'm
replying
to see if anyone has a idea.
...
INFO: BL31: Preparing for EL3 exit to normal world
INFO: Entry point address = 0x60000000
INFO: SPSR = 0x3c9
UEFI firmware (version built at 01:31:23 on May 31 2024)
The boot is stuck and no more output after that. I tried adding more verbose
output
from edk2 and found it's stuck at the following point.
ArmVirtPkg/PrePi/PrePi.c::PrePiMain
rmVirtPkg/Library/PlatformPeiLib/PlatformPeiLib.c::PlatformPeim
#ifdef MDE_CPU_AARCH64
//
// Set the SMCCC conduit to SMC if executing at EL2, which is typically the
// exception level that services HVCs rather than the one that invokes them.
//
if (ArmReadCurrentEL () == AARCH64_EL2) {
Status = PcdSetBoolS (PcdMonitorConduitHvc, FALSE); // The function
is never returned in my case
ASSERT_EFI_ERROR (Status);
}
#endif
I'm able to reproduce this even without RME. This code was introduced
recently by c98f7f755089 ("ArmVirtPkg: Use dynamic PCD to set the SMCCC
conduit"). Maybe Ard (Cc'd) knows what could be going wrong here.
A slightly reduced reproducer:
$ cd edk2/
$ build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemuKernel.dsc
$ cd ..
$ git clone https://github.com/ARM-software/arm-trusted-firmware.git tf-a
$ cd tf-a/
$ make -j CROSS_COMPILE=aarch64-linux-gnu- PLAT=qemu DEBUG=1 LOG_LEVEL=40
QEMU_USE_GIC_DRIVER=QEMU_GICV3
BL33=../edk2/Build/ArmVirtQemuKernel-AARCH64/DEBUG_GCC5/FV/QEMU_EFI.fd all fip
&& \
dd if=build/qemu/debug/bl1.bin of=flash.bin && \
dd if=build/qemu/debug/fip.bin of=flash.bin seek=64 bs=4096
$ qemu-system-aarch64 -M virt,virtualization=on,secure=on,gic-version=3 -cpu
max -m 2G -smp 8 -monitor none -serial mon:stdio -nographic -bios flash.bin
Thanks for the hints. Eventually, I'm able to start the host with
'edk2-stable202402'.
Note that 'edk2-stable202405' doesn't work. However, I failed to build the edk2
for
guest and unable to start the guest successfully, more information is provided
below.
--> host's edk2
# git clone g...@github.com:gwshan/edk2.git edk2
# cd edk2; git checkout edk2-stable202402 -b stable202402
# git submodule update --init --recursive; \
source edksetup.sh; make -j -C BaseTools; \
export GCC5_AARCH64_PREFIX=; \
build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemuKernel.dsc
---> tf-a: rebuild using commands as you suggested.
---> Boot host
/home/gavin/sandbox/qemu.main/build/qemu-system-aarch64
\
-M virt,virtualization=on,secure=on,gic-version=3,acpi=off
\
-cpu max,x-rme=on -m 8G -smp 8
\
-nodefaults -monitor none -serial mon:stdio -nographic
\
-bios /home/gavin/sandbox/CCA/tf-a/flash.bin
\
-kernel /home/gavin/sandbox/CCA/linux/arch/arm64/boot/Image
\
-drive
format=raw,if=none,file=/home/gavin/sandbox/CCA/buildroot/output/images/rootfs.ext4,id=hd0
\
-device virtio-blk-pci,drive=hd0
\
-netdev
tap,id=tap0,vhost=false,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
\
-device virtio-net-pci,netdev=tap0,mac=52:54:00:f1:26:b0
\
-append root=/dev/vda console=ttyAMA0
\
-device virtio-9p-device,fsdev=shr0,mount_tag=shr0
\
-fsdev local,security_model=none,path=/home/gavin/sandbox/CCA,id=shr0
:
NOTICE: Booting Trusted Firmware
NOTICE: BL1: v2.10.0(debug):99e0b97aa-dirty
NOTICE: BL1: Built : 00:31:35, May 31 2024
INFO: BL1: RAM 0xe0ee000 - 0xe0f7000
INFO: BL1: Loading BL2
:
Booting RMM v.0.4.0(debug) 17924bc Built with GCC 11.4.1
RMM-EL3 Interface v.0.2
Boot Manifest Interface v.0.3
RMI/RSI ABI v.1.0/1.0 built: May 31 2024 00:21:59
INFO: RMM init end.
:
UEFI firmware (version built at 04:07:42 on Jun 1 2024)
PlatformPeim: PL011 UART (console) @ 0x9000000
PlatformPeim: PL011 UART (debug) @ 0x9000000
:
EFI stub: Booting Linux Kernel...
EFI stub: EFI_RNG_PROTOCOL unavailable
:
Welcome to Buildroot
buildroot login:
# ifconfig eth0 | grep 'inet addr'
inet addr:10.26.1.212 Bcast:10.26.1.255 Mask:255.255.255.0
---> guest edk2
# git clone https://git.codelinaro.org/linaro/dcap/edk2.git edk2-guest
# cd edk2-guest; git checkout origin/cca/v2 -b cca/v2
# git submodule update --init --recursive; \
source edksetup.sh; make -j -C BaseTools; \
export GCC5_AARCH64_PREFIX=; \
build -b DEBUG -a AARCH64 -t GCC5 -p ArmVirtPkg/ArmVirtQemu.dsc
:
WriteSections64():
/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore/DEBUG/ArmPlatformPrePeiCore.dll
AARCH64 small code model requires identical ELF and PE/COFF section offsets
modulo 4 KB.
cp -p -f
/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/OvmfPkg/VirtioFsDxe/VirtioFsDxe/DEBUG/VirtioFsDxe.dll
/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/OvmfPkg/VirtioFsDxe/VirtioFsDxe/DEBUG/VirtioFsDxe.debug
cp -p -f
/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Disk/PartitionDxe/PartitionDxe/DEBUG/PartitionDxe.debug
/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/PartitionDxe.debug
"gcc" -MMD -MF
/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/CryptoPkg/Library/OpensslLib/OpensslLibCrypto/OUTPUT/openssl/crypto/asn1/x_sig.obj.deps
@/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/CryptoPkg/Library/OpensslLib/OpensslLibCrypto/OUTPUT/cc_resp.txt
-c -o
/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/CryptoPkg/Library/OpensslLib/OpensslLibCrypto/OUTPUT/openssl/crypto/asn1/x_sig.obj
/home/gavin/sandbox/CCA/edk2-guest/CryptoPkg/Library/OpensslLib/openssl/crypto/asn1/x_sig.c
"GenFw" -e DXE_CORE -o
/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/OUTPUT/DxeCore.efi
/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
GenSec -s EFI_SECTION_USER_INTERFACE -n ArmCpuDxe -o
/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/FV/Ffs/B8D9777E-D72A-451F-9BDB-BAFB52A68415ArmCpuDxe/B8D9777E-D72A-451F-9BDB-BAFB52A68415SEC3.ui
cp -p -f
/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/DisplayEngineDxe/DisplayEngineDxe/DEBUG/*.map
/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/DisplayEngineDxe/DisplayEngineDxe/OUTPUT
cp -p -f
/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Disk/UdfDxe/UdfDxe/OUTPUT/UdfDxe.efi
/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Universal/Disk/UdfDxe/UdfDxe/DEBUG
GenFw: ERROR 3000: Invalid
:
build.py...
: error 7000: Failed to execute command
make tbuild
[/home/gavin/sandbox/CCA/edk2-guest/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore]
build.py...
: error F002: Failed to build module
/home/gavin/sandbox/CCA/edk2-guest/ArmPlatformPkg/PrePeiCore/PrePeiCoreUniCore.inf
[AARCH64, GCC5, DEBUG]
- Failed -
Build end time: 05:42:19, Jun.01 2024
Build total time: 00:00:31
---> Use the edk2 image from the latest QEMU source
# cd /home/gavin/sandbox/CCA
# cp /home/gavin/sandbox/qemu.main/build/pc-bios/edk2-aarch64-code.fd ./
---> Start the guest and no output from the console
host# mount | grep 9p
shr0 on /mnt/shr0 type 9p (rw,relatime,access=client,trans=virtio)
host# export SHR_DIR="/mnt/shr0"
host# qemu-system-aarch64 -accel kvm \
-machine virt,gic-version=3,confidential-guest-support=rme0 \
-cpu host -smp 2 -m 512M \
-object 'rme-guest,id=rme0,measurement-algo=sha512' \
-monitor none -serial mon:stdio -nographic \
-bios /mnt/edk2-aarch64-code.fd \
-kernel ${SHR_DIR}/linux/arch/arm64/boot/Image \
-initrd ${SHR_DIR}/buildroot/output/images/rootfs.cpio \
-append 'console=ttyAMA0'
:
<no output from the console>
:
(QEMU) q
There are some messages from host's console indicating RMI/RMM servicing
states when the guest is running at background. After the guest is terminated,
the host crashes.
SMC_RMM_RTT_CREATE 102dff000 122c2e000 1e00000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE 102dff000 1234a7000 2000000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE 102dff000 1235bd000 2200000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE 102dff000 12387c000 2400000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE 102dff000 123a5a000 2600000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE 102dff000 12407d000 2800000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE 102dff000 124109000 2a00000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE 102dff000 123e49000 2c00000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE 102dff000 124275000 2e00000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE 102dff000 123138000 3000000 3 > RMI_SUCCESS
SMC_RMM_RTT_CREATE 102dff000 124d07000 3200000 3 > RMI_SUCCESS
:
:
[22768.994481] rcu: INFO: rcu_preempt self-detected stall on CPU
[22769.006861] rcu: 3-....: (2751 ticks this GP)
idle=93ec/1/0x4000000000000000 softirq=114451/115721 fqs=1160
[22769.020475] rcu: (t=5257 jiffies g=531913 q=7 ncpus=8)
[22769.030547] CPU: 3 PID: 198 Comm: qemu-system-aar Not tainted
6.9.0-rc1-gavin-gfcfc92d6ff07 #1
[22769.041847] Hardware name: QEMU QEMU Virtual Machine, BIOS unknown 2/2/2022
[22769.050548] pstate: 60402009 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[22769.059382] pc : kvm_realm_unmap_range+0x308/0x32c
[22769.070275] lr : kvm_realm_unmap_range+0x304/0x32c
[22769.075893] sp : ffff800080a3b930
[22769.079929] x29: ffff800080a3b930 x28: 00000000003d7000 x27: 00000000003d6000
[22769.092990] x26: 00000000c4000152 x25: ffffffffffffffff x24: 0000000000000000
[22769.101150] x23: 0000010000000000 x22: 00000000c4000155 x21: 0000000102dff000
[22769.109056] x20: ffff8000801a5e00 x19: 0000000000000000 x18: 0000000000000001
[22769.117042] x17: 0000000000000000 x16: 000000000000000e x15: 0000000000000000
[22769.124991] x14: 0000ffff7fa14000 x13: 0000000000000002 x12: 000000000010d594
[22769.134213] x11: 0000000000000002 x10: 00000000ffffffff x9 : ffffffffffffffff
[22769.142951] x8 : 0000000000000000 x7 : 0000000000000000 x6 : 00000000c2dff000
[22769.151413] x5 : 0000000102f56000 x4 : 0000000000000015 x3 : 0000000000000000
[22769.159932] x2 : 0000000000000000 x1 : 0000000000000000 x0 : 0000000000000000
[22769.169674] Call trace:
[22769.174026] kvm_realm_unmap_range+0x308/0x32c
[22769.181046] __unmap_stage2_range+0x60/0x7c
[22769.186396] kvm_free_stage2_pgd+0xa0/0xd4
[22769.191766] kvm_arch_flush_shadow_all+0x1c/0x34
[22769.197879] kvm_mmu_notifier_release+0x30/0x84
[22769.203304] __mmu_notifier_release+0x7c/0x1f8
[22769.209340] exit_mmap+0x264/0x274
[22769.213992] __mmput+0x40/0x150
[22769.218635] mmput+0x50/0x5c
[22769.222606] do_exit+0x288/0x92c
[22769.226935] do_group_exit+0x34/0x90
[22769.231359] get_signal+0x814/0x820
[22769.236537] do_signal+0x90/0x1320
[22769.241145] do_notify_resume+0xc8/0x140
[22769.246458] el0_svc+0xc8/0xdc
[22769.250913] el0t_64_sync_handler+0x13c/0x158
[22769.256045] el0t_64_sync+0x190/0x194
Thanks,
Gavin
