Hi Wei, Michal, > On Dec 8, 2023, at 19:13, Wei Chen <wei.c...@arm.com> wrote: > > Hi Henry and Michal, > >>>>>>> On 08/12/2023 06:46, Henry Wang wrote: >>>>>>>> diff --git a/automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp >>>>>>>> b/automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp >>>>>>>> new file mode 100755 >>>>>>>> index 0000000000..25d9a5f81c >>>>>>>> --- /dev/null >>>>>>>> +++ b/automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp >>>>>>>> @@ -0,0 +1,73 @@ >>>>>>>> +#!/usr/bin/expect >>>>>>>> + >>>>>>>> +set timeout 2000 >>>>>>> Do we really need such a big timeout (~30 min)? >>>>>>> Looking at your test job, it took 16 mins (quite a lot but I know FVP >>>>>>> is slow >>>>>>> + send_slow slows things down) >>>>>> >>>>>> This is a really good question. I did have the same question while >>>>>> working on >>>>>> the negative test today. The timeout 2000 indeed will fail the job at >>>>>> about 30min, >>>>>> and waiting for it is indeed not really pleasant. >>>>>> >>>>>> But my second thought would be - from my observation, the overall time >>>>>> now >>>>>> would vary between 15min ~ 20min, and having a 10min margin is not that >>>>>> crazy >>>>>> given that we probably will do more testing from the job in the future, >>>>>> and if the >>>>>> GitLab Arm worker is high loaded, FVP will probably become slower. And >>>>>> normally >>>>>> we don’t even trigger the timeout as the job will normally pass. So I >>>>>> decided >>>>>> to keep this. >>>>>> >>>>>> Mind sharing your thoughts about the better value of the timeout? >>>>>> Probably 25min? >>>>> From what you said that the average is 15-20, I think we can leave it set >>>>> to 30. >>>>> But I wonder if we can do something to decrease the average time. ~20 min >>>>> is a lot >>>>> even for FVP :) Have you tried setting send_slow to something lower than >>>>> 100ms? >>>>> That said, we don't send too many chars to FVP, so I doubt it would play >>>>> a major role >>>>> in the overall time. >>>> >>>> I agree with the send_slow part. Actually I do have the same concern, here >>>> are my current >>>> understanding and I think you will definitely help with your knowledge: >>>> If you check the full log of Dom0 booting, for example [1], you will find >>>> that we wasted so >>>> much time in starting the services of the OS (modloop, udev-settle, etc). >>>> All of these services >>>> are retried many times but in the end they are still not up, and from my >>>> understanding they >>>> won’t affect the actual test(?) If we can somehow get rid of these >>>> services from rootfs, I think >>>> we can save a lot of time. >>>> >>>> And honestly, I noticed that qemu-alpine-arm64-gcc suffers from the same >>>> problem and it also >>>> takes around 15min to finish. So if we managed to tailor the services from >>>> the filesystem, we >>>> can save a lot of time. >>> That is not true. Qemu runs the tests relatively fast within few minutes. >>> The reason you see e.g. 12 mins >>> for some Qemu jobs comes from the timeout we set in Qemu scripts. We don't >>> have yet the solution (we could >>> do the same as Qubes script) to detect the test success early and exit >>> before timeout. That is why currently >>> the only way for Qemu tests to finish is by reaching the timeout. >>> >>> So the problem is not with the rootfs and services (the improvement would >>> not be significant) but with >>> the simulation being slow. That said, this is something we all know and I >>> expect FVP to only be used in scenarios >>> which cannot be tested using Qemu or real HW. >> Ok, you made a point. Let me do some experiments to see if I can improve. >> Otherwise maybe >> we can live with this until a better solution. >> Kind regards, >> Henry > > QEMU works like FVP enabled use_real_time flag. How about enable > use_real_time flag in CI for most test cases, but disable it for > some time sensitive test cases? Normally, enable use_real_time > will give several times improvement of FVP performance.
I am seeing below from the FVP parameter lists of the one we are currently using. The "Deprecated" word worries me a bit (the old version FVP does not have the “Deprecated" though). ``` bp.refcounter.use_real_time=0 # (bool , init-time) default = '0' : **Deprecated, this parameter will be removed in future versions** Update the Generic Timer counter at a real-time base frequency instead of simulator time ``` Also, from my testing in the GitLab pipeline, I was not able to see significant time improvement. So I guess instead I will try what Julien suggests to see if things can be better. Kind regards, Henry > > Cheers, > Wei Chen > >>> >>> ~Michal
