date:20231207

[RFC PATCH v1 0/1] ARCH_FIXED_CONFIG introduction for randconfig

2023-12-07 Thread Oleksii Kurochko

Brief Overview:
In the earlier patch series [1], it was introduced a comprehensive set of 
changes
enabling a full Xen build for RISC-V.
This early support primarily provides the minimum stubs required for the RISC-V
Xen build. At this stage of development, many configs are deemed unnecessary and
are expected to be disabled.

Without ARCH_FIXED_CONFIG (or an alternative mechanism), the alternative is
updating CI's build.yaml in multiple instances with the same configs,
mirroring the approach taken in [1] to prevent the inadvertent activation of
unsupported configs.

For example, in scenarios like dom0less, we can exclude grant tables by setting
"CONFIG_GRANT_TABLE=n" in ARCH_FIXED_CONFIG. This eliminates the need for 
intricate
modifications to Kconfig configurations with conditions like "depends on X86 || 
ARM"
or the introduction of HAS_* conditions followed by Kconfig updates with
"depends on HAS_*," as illustrated in examples [2] or [3].

It might be useful for other architectures as well, especially for PPC, which is
currently under development.

There are several open questions:
- Does introduction of ARCH_FIXED_CONFIG make sense?
- Should ARCH_FIXED_CONFIG be re-used for *defconfig?

[1] 
https://lore.kernel.org/xen-devel/b4e85f8f58787b4d179022973ce25673d6b56e36.1700761381.git.oleksii.kuroc...@gmail.com/
[2] 
https://lore.kernel.org/xen-devel/cdc20255540a66ba0b6946ac6d48c11029cd3385.1701453087.git.oleksii.kuroc...@gmail.com/
[3] 
https://lore.kernel.org/xen-devel/d42a34866edc70a12736b5c6976aa1b44b4ebd8a.1701453087.git.oleksii.kuroc...@gmail.com/

Oleksii Kurochko (1):
  xen/Makefile: introduce ARCH_FIXED_CONFIG for randconfig

 xen/Makefile | 5 -
 1 file changed, 4 insertions(+), 1 deletion(-)

-- 
2.43.0

Re: [XEN PATCH] automation/eclair_analysis: file exclusion automation

2023-12-07 Thread Julien Grall





On 07/12/2023 11:39, Nicola Vetrini wrote:

+-doc_begin="libfdt is out of scope."
+-file_tag+={out_of_scope,"^xen/include/xen/libfdt/.*$"}


AFAICT, before this was marked as "adopted". But this is now moved to 
"out_of_scope". Can you explain why?


It also feels somewhat unrelated to the rest of the patch.

Cheers,

--
Julien Grall

xen | Failed pipeline for staging | 25147005

2023-12-07 Thread GitLab



Pipeline #1098955890 has failed!

Project: xen ( https://gitlab.com/xen-project/xen )
Branch: staging ( https://gitlab.com/xen-project/xen/-/commits/staging )

Commit: 25147005 ( 
https://gitlab.com/xen-project/xen/-/commit/25147005daf5a4e121b96496d6d208fac05fca35
 )
Commit Message: xen/sched: do some minor cleanup of sched_move_...
Commit Author: Juergen Gross ( https://gitlab.com/jgross1 )
Committed by: George Dunlap


Pipeline #1098955890 ( 
https://gitlab.com/xen-project/xen/-/pipelines/1098955890 ) triggered by Ganis 
( https://gitlab.com/ganis )
had 12 failed jobs.

Job #5703366092 ( https://gitlab.com/xen-project/xen/-/jobs/5703366092/raw )

Stage: build
Name: archlinux-gcc
Job #5703366143 ( https://gitlab.com/xen-project/xen/-/jobs/5703366143/raw )

Stage: build
Name: debian-bookworm-32-gcc-debug
Job #5703366193 ( https://gitlab.com/xen-project/xen/-/jobs/5703366193/raw )

Stage: build
Name: opensuse-tumbleweed-gcc
Job #5703366199 ( https://gitlab.com/xen-project/xen/-/jobs/5703366199/raw )

Stage: test
Name: build-each-commit-gcc
Job #5703366195 ( https://gitlab.com/xen-project/xen/-/jobs/5703366195/raw )

Stage: build
Name: opensuse-tumbleweed-gcc-debug
Job #5703366130 ( https://gitlab.com/xen-project/xen/-/jobs/5703366130/raw )

Stage: build
Name: debian-bookworm-gcc
Job #5703366094 ( https://gitlab.com/xen-project/xen/-/jobs/5703366094/raw )

Stage: build
Name: archlinux-gcc-debug
Job #5703366149 ( https://gitlab.com/xen-project/xen/-/jobs/5703366149/raw )

Stage: build
Name: ubuntu-trusty-gcc
Job #5703365823 ( https://gitlab.com/xen-project/xen/-/jobs/5703365823/raw )

Stage: build
Name: alpine-3.18-gcc
Job #5703365831 ( https://gitlab.com/xen-project/xen/-/jobs/5703365831/raw )

Stage: build
Name: alpine-3.18-gcc-debug
Job #5703366151 ( https://gitlab.com/xen-project/xen/-/jobs/5703366151/raw )

Stage: build
Name: ubuntu-trusty-gcc-debug
Job #5703366134 ( https://gitlab.com/xen-project/xen/-/jobs/5703366134/raw )

Stage: build
Name: debian-bookworm-gcc-debug

-- 
You're receiving this email because of your account on gitlab.com.

Re: [XEN PATCH] automation/eclair_analysis: file exclusion automation

2023-12-07 Thread Nicola Vetrini


On 2023-12-07 18:08, Julien Grall wrote:

On 07/12/2023 11:39, Nicola Vetrini wrote:

+-doc_begin="libfdt is out of scope."
+-file_tag+={out_of_scope,"^xen/include/xen/libfdt/.*$"}


AFAICT, before this was marked as "adopted". But this is now moved to 
"out_of_scope". Can you explain why?


It also feels somewhat unrelated to the rest of the patch.

Cheers,


I mistakenly changed the tag. It is not unrelated, as it't not part of 
exclude-list.json (perhaps unintentionally). The manual exclusions that 
remain in out_of_scope.ecl are there for this reason, since I wanted to 
keep the set of excluded files as it was before.


--
Nicola Vetrini, BSc
Software Engineer, BUGSENG srl (https://bugseng.com)

Re: [XEN PATCH] automation/eclair_analysis: file exclusion automation

2023-12-07 Thread Julien Grall


Hi Nicola,

On 07/12/2023 17:53, Nicola Vetrini wrote:

On 2023-12-07 18:08, Julien Grall wrote:

On 07/12/2023 11:39, Nicola Vetrini wrote:

+-doc_begin="libfdt is out of scope."
+-file_tag+={out_of_scope,"^xen/include/xen/libfdt/.*$"}


AFAICT, before this was marked as "adopted". But this is now moved to 
"out_of_scope". Can you explain why?


It also feels somewhat unrelated to the rest of the patch.

Cheers,


I mistakenly changed the tag. It is not unrelated, as it't not part of 
exclude-list.json (perhaps unintentionally). The manual exclusions that 
remain in out_of_scope.ecl are there for this reason, since I wanted to 
keep the set of excluded files as it was before.


Given that common/libfdt/* is part of the exclude-list.json, I can't see 
why include/xen/libfdt/* are not. So can you add it?


Cheers,

--
Julien Grall

Re: [PATCH v2 07/39] xen/riscv: introduce arch-riscv/hvm/save.h

2023-12-07 Thread Shawn Anastasio

On 12/5/23 9:59 AM, Jan Beulich wrote:
> On 24.11.2023 11:30, Oleksii Kurochko wrote:
>> --- a/xen/include/public/hvm/save.h
>> +++ b/xen/include/public/hvm/save.h
>> @@ -91,6 +91,8 @@ DECLARE_HVM_SAVE_TYPE(END, 0, struct hvm_save_end);
>>  #include "../arch-arm/hvm/save.h"
>>  #elif defined(__powerpc64__)
>>  #include "../arch-ppc.h"
>> +#elif defined(__riscv)
>> +#include "../arch-riscv/hvm/save.h"
>>  #else
>>  #error "unsupported architecture"
>>  #endif
> 
> The PPC part here looks bogus altogether. Shawn?
>

I think my original intention here was to avoid creating yet another
empty header while still having a place to put PPC-specific definitions
that might be required.

See as how the ARM file is entirely empty though, I doubt we'll be any
different, so this could definitely be dropped.

> Jan

Thanks,
Shawn

[xen-unstable-smoke test] 184026: tolerable all pass - PUSHED

2023-12-07 Thread osstest service owner

flight 184026 xen-unstable-smoke real [real]
http://logs.test-lab.xenproject.org/osstest/logs/184026/

Failures :-/ but no regressions.

Tests which did not succeed, but are not blocking:
 test-amd64-amd64-libvirt 15 migrate-support-checkfail   never pass
 test-arm64-arm64-xl-xsm  15 migrate-support-checkfail   never pass
 test-arm64-arm64-xl-xsm  16 saverestore-support-checkfail   never pass
 test-armhf-armhf-xl  15 migrate-support-checkfail   never pass
 test-armhf-armhf-xl  16 saverestore-support-checkfail   never pass

version targeted for testing:
 xen  d2b7c442b4a066bb670ee83e24800cabc415241d
baseline version:
 xen  02d0a615b32d03702f79807fa5e88f0cf78dde84

Last test of basis   184025  2023-12-07 13:02:05 Z0 days
Testing same since   184026  2023-12-07 16:02:06 Z0 days1 attempts


People who touched revisions under test:
  George Dunlap 
  Juergen Gross 
  Julien Grall 

jobs:
 build-arm64-xsm  pass
 build-amd64  pass
 build-armhf  pass
 build-amd64-libvirt  pass
 test-armhf-armhf-xl  pass
 test-arm64-arm64-xl-xsm  pass
 test-amd64-amd64-xl-qemuu-debianhvm-amd64pass
 test-amd64-amd64-libvirt pass



sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images

Logs, config files, etc. are available at
http://logs.test-lab.xenproject.org/osstest/logs

Explanation of these reports, and of osstest in general, is at
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master

Test harness code can be found at
http://xenbits.xen.org/gitweb?p=osstest.git;a=summary


Pushing revision :

To xenbits.xen.org:/home/xen/git/xen.git
   02d0a615b3..d2b7c442b4  d2b7c442b4a066bb670ee83e24800cabc415241d -> smoke

Re: [PATCH] MAINTAINERS: Hand over the release manager role to Oleksii Kurochko

2023-12-07 Thread Julien Grall


Hi,

On 07/12/2023 16:20, Henry Wang wrote:

I've finished the opportunity to do two releases (4.17 and 4.18)
and Oleksii Kurochko has volunteered to be the next release manager.


Henry, thanks for your time as release manager.
Oleksii, thanks for stepping up and good luck for the role!


Hand over the role to him by changing the maintainership of the
CHANGELOG.md.

Signed-off-by: Henry Wang 


Acked-by: Julien Grall 

I didn't hear any objection during the community call. But I will give 
until Tuesday morning (UK time) just in case. If there are none, then I 
will commit it.


Cheers,

--
Julien Grall

Re: [PATCH] Mini-OS: don't use objcopy --dump-section

2023-12-07 Thread Andrew Cooper

On 06/12/2023 4:17 pm, Juergen Gross wrote:
> The objcopy option "--dump-section" isn't supported in binutils before
> version 2.25, which are still supported by the Xen build system.
>
> Avoid that by using the "-O binary" format together with
> "--only-section". This requires to set the "alloc" section flag.
>
> Fixes: 33411a11f848 ("Mini-OS: hide all symbols not exported via 
> EXPORT_SYMBOLS()")
> Signed-off-by: Juergen Gross 

Reviewed-by: Andrew Cooper 

And FAOD, I'm taking this, plus a bump to the minios rev right now to
unbreak Gitlab CI, as it's been broken for 2 now.

~Andrew

xen | Failed pipeline for staging | d2b7c442

2023-12-07 Thread GitLab



Pipeline #1099081266 has failed!

Project: xen ( https://gitlab.com/xen-project/xen )
Branch: staging ( https://gitlab.com/xen-project/xen/-/commits/staging )

Commit: d2b7c442 ( 
https://gitlab.com/xen-project/xen/-/commit/d2b7c442b4a066bb670ee83e24800cabc415241d
 )
Commit Message: CODING_STYLE: Add a section of the naming conve...
Commit Author: Julien Grall


Pipeline #1099081266 ( 
https://gitlab.com/xen-project/xen/-/pipelines/1099081266 ) triggered by Ganis 
( https://gitlab.com/ganis )
had 12 failed jobs.

Job #5704236009 ( https://gitlab.com/xen-project/xen/-/jobs/5704236009/raw )

Stage: build
Name: alpine-3.18-gcc-debug
Job #5704236006 ( https://gitlab.com/xen-project/xen/-/jobs/5704236006/raw )

Stage: build
Name: alpine-3.18-gcc
Job #5704236300 ( https://gitlab.com/xen-project/xen/-/jobs/5704236300/raw )

Stage: build
Name: archlinux-gcc
Job #5704236325 ( https://gitlab.com/xen-project/xen/-/jobs/5704236325/raw )

Stage: build
Name: debian-bookworm-gcc
Job #5704236327 ( https://gitlab.com/xen-project/xen/-/jobs/5704236327/raw )

Stage: build
Name: debian-bookworm-gcc-debug
Job #5704236339 ( https://gitlab.com/xen-project/xen/-/jobs/5704236339/raw )

Stage: build
Name: ubuntu-trusty-gcc
Job #5704236388 ( https://gitlab.com/xen-project/xen/-/jobs/5704236388/raw )

Stage: build
Name: opensuse-tumbleweed-gcc-debug
Job #5704236386 ( https://gitlab.com/xen-project/xen/-/jobs/5704236386/raw )

Stage: build
Name: opensuse-tumbleweed-gcc
Job #5704236391 ( https://gitlab.com/xen-project/xen/-/jobs/5704236391/raw )

Stage: test
Name: build-each-commit-gcc
Job #5704236303 ( https://gitlab.com/xen-project/xen/-/jobs/5704236303/raw )

Stage: build
Name: archlinux-gcc-debug
Job #5704236333 ( https://gitlab.com/xen-project/xen/-/jobs/5704236333/raw )

Stage: build
Name: debian-bookworm-32-gcc-debug
Job #5704236342 ( https://gitlab.com/xen-project/xen/-/jobs/5704236342/raw )

Stage: build
Name: ubuntu-trusty-gcc-debug

-- 
You're receiving this email because of your account on gitlab.com.

[ovmf test] 184027: all pass - PUSHED

2023-12-07 Thread osstest service owner

flight 184027 ovmf real [real]
http://logs.test-lab.xenproject.org/osstest/logs/184027/

Perfect :-)
All tests in this flight passed as required
version targeted for testing:
 ovmf 238690a30d02d3f95f0355c88c35dc0e4232342a
baseline version:
 ovmf 553dfb0f57ae8a666938873cf836a33549568c87

Last test of basis   184023  2023-12-07 10:14:25 Z0 days
Testing same since   184027  2023-12-07 17:11:00 Z0 days1 attempts


People who touched revisions under test:
  Corvin Köhne 

jobs:
 build-amd64-xsm  pass
 build-i386-xsm   pass
 build-amd64  pass
 build-i386   pass
 build-amd64-libvirt  pass
 build-i386-libvirt   pass
 build-amd64-pvopspass
 build-i386-pvops pass
 test-amd64-amd64-xl-qemuu-ovmf-amd64 pass
 test-amd64-i386-xl-qemuu-ovmf-amd64  pass



sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images

Logs, config files, etc. are available at
http://logs.test-lab.xenproject.org/osstest/logs

Explanation of these reports, and of osstest in general, is at
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master

Test harness code can be found at
http://xenbits.xen.org/gitweb?p=osstest.git;a=summary


Pushing revision :

To xenbits.xen.org:/home/xen/git/osstest/ovmf.git
   553dfb0f57..238690a30d  238690a30d02d3f95f0355c88c35dc0e4232342a -> 
xen-tested-master

Re: [RFC PATCH v1 1/1] xen/Makefile: introduce ARCH_FIXED_CONFIG for randconfig

2023-12-07 Thread Andrew Cooper

On 07/12/2023 5:03 pm, Oleksii Kurochko wrote:
> ARCH_FIXED_CONFIG is required in the case of randconfig
> and CI for configs that aren't ready or are not
> supposed to be implemented for specific architecture.
> These configs should always be disabled to prevent randconfig
> related tests from failing.
>
> Signed-off-by: Oleksii Kurochko 
> ---
>  xen/Makefile | 5 -
>  1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/xen/Makefile b/xen/Makefile
> index ca571103c8..8ae8fe1480 100644
> --- a/xen/Makefile
> +++ b/xen/Makefile
> @@ -336,11 +336,14 @@ ifeq ($(config-build),y)
>  # *config targets only - make sure prerequisites are updated, and descend
>  # in tools/kconfig to make the *config target
>  
> +ARCH_FORCED_CONFIG := $(srctree)/arch/$(SRCARCH)/configs/randomforced.config
> +
>  # Create a file for KCONFIG_ALLCONFIG which depends on the environment.
>  # This will be use by kconfig targets 
> allyesconfig/allmodconfig/allnoconfig/randconfig
>  filechk_kconfig_allconfig = \
>  $(if $(findstring n,$(XEN_HAS_CHECKPOLICY)), echo 
> 'CONFIG_XSM_FLASK_POLICY=n';) \
> -$(if $(KCONFIG_ALLCONFIG), cat $(KCONFIG_ALLCONFIG);) \
> +$(if $(KCONFIG_ALLCONFIG), cat $(KCONFIG_ALLCONFIG); \
> +$(if $(wildcard $(ARCH_FORCED_CONFIG)), cat $(ARCH_FORCED_CONFIG);) ) \
>  :
>  
>  .allconfig.tmp: FORCE

We already have infrastructure for this.  What's wrong with
EXTRA_FIXED_RANDCONFIG?

---8<---

CI: Revert "automation: Drop ppc64le-*randconfig jobs", fix Randconfig
with existing infrastructure
    
This reverts commit cbb71b95dd708b1e26899bbe1e7bf9a85081fd60.

Signed-off-by: Andrew Cooper 

diff --git a/automation/gitlab-ci/build.yaml
b/automation/gitlab-ci/build.yaml
index 32af30ccedc9..346d0400ed09 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -538,6 +538,7 @@ archlinux-current-gcc-riscv64-randconfig:
 RANDCONFIG: y
 EXTRA_FIXED_RANDCONFIG:
   CONFIG_COVERAGE=n
+  CONFIG_GRANT_TABLE=n
 
 archlinux-current-gcc-riscv64-debug-randconfig:
   extends: .gcc-riscv64-cross-build-debug
@@ -547,6 +548,7 @@ archlinux-current-gcc-riscv64-debug-randconfig:
 RANDCONFIG: y
 EXTRA_FIXED_RANDCONFIG:
   CONFIG_COVERAGE=n
+  CONFIG_GRANT_TABLE=n
 
 # Power cross-build
 debian-bullseye-gcc-ppc64le:
@@ -563,6 +565,26 @@ debian-bullseye-gcc-ppc64le-debug:
 KBUILD_DEFCONFIG: ppc64_defconfig
 HYPERVISOR_ONLY: y
 
+debian-bullseye-gcc-ppc64le-randconfig:
+  extends: .gcc-ppc64le-cross-build
+  variables:
+    CONTAINER: debian:bullseye-ppc64le
+    KBUILD_DEFCONFIG: ppc64_defconfig
+    RANDCONFIG: y
+    EXTRA_FIXED_RANDCONFIG:
+  CONFIG_COVERAGE=n
+  CONFIG_GRANT_TABLE=n
+
+debian-bullseye-gcc-ppc64le-debug-randconfig:
+  extends: .gcc-ppc64le-cross-build-debug
+  variables:
+    CONTAINER: debian:bullseye-ppc64le
+    KBUILD_DEFCONFIG: ppc64_defconfig
+    RANDCONFIG: y
+    EXTRA_FIXED_RANDCONFIG:
+  CONFIG_COVERAGE=n
+  CONFIG_GRANT_TABLE=n
+
 # Yocto test jobs
 yocto-qemuarm64:
   extends: .yocto-test-arm64

Re: preparations for 4.17.3

2023-12-07 Thread Andrew Cooper

On 07/12/2023 7:06 am, Jan Beulich wrote:
> All,
>
> the release is about due. Please point out backports you find missing
> from the respective staging branch, but which you consider relevant.

e3c409d59ac8 "x86/x2apic: introduce a mixed physical/cluster mode"

There's now clear evidence on the "[PATCH] x86/x2apic: introduce a mixed
physical/cluster mode" that it fixes something on real systems.  At a
guess, a platform erratum concerning the use of external cluster mode
interrupts, but nevertheless its a genuine improvement too.

I've got backports to 4.17 and 4.13 easily to hand if you want.

~Andrew

Re: [RFC PATCH v1 1/1] xen/Makefile: introduce ARCH_FIXED_CONFIG for randconfig

2023-12-07 Thread Oleksii

On Thu, 2023-12-07 at 20:17 +, Andrew Cooper wrote:
> On 07/12/2023 5:03 pm, Oleksii Kurochko wrote:
> > ARCH_FIXED_CONFIG is required in the case of randconfig
> > and CI for configs that aren't ready or are not
> > supposed to be implemented for specific architecture.
> > These configs should always be disabled to prevent randconfig
> > related tests from failing.
> > 
> > Signed-off-by: Oleksii Kurochko 
> > ---
> >  xen/Makefile | 5 -
> >  1 file changed, 4 insertions(+), 1 deletion(-)
> > 
> > diff --git a/xen/Makefile b/xen/Makefile
> > index ca571103c8..8ae8fe1480 100644
> > --- a/xen/Makefile
> > +++ b/xen/Makefile
> > @@ -336,11 +336,14 @@ ifeq ($(config-build),y)
> >  # *config targets only - make sure prerequisites are updated, and
> > descend
> >  # in tools/kconfig to make the *config target
> >  
> > +ARCH_FORCED_CONFIG :=
> > $(srctree)/arch/$(SRCARCH)/configs/randomforced.config
> > +
> >  # Create a file for KCONFIG_ALLCONFIG which depends on the
> > environment.
> >  # This will be use by kconfig targets
> > allyesconfig/allmodconfig/allnoconfig/randconfig
> >  filechk_kconfig_allconfig = \
> >  $(if $(findstring n,$(XEN_HAS_CHECKPOLICY)), echo
> > 'CONFIG_XSM_FLASK_POLICY=n';) \
> > -    $(if $(KCONFIG_ALLCONFIG), cat $(KCONFIG_ALLCONFIG);) \
> > +    $(if $(KCONFIG_ALLCONFIG), cat $(KCONFIG_ALLCONFIG); \
> > +    $(if $(wildcard $(ARCH_FORCED_CONFIG)), cat
> > $(ARCH_FORCED_CONFIG);) ) \
> >  :
> >  
> >  .allconfig.tmp: FORCE
> 
> We already have infrastructure for this.  What's wrong with
> EXTRA_FIXED_RANDCONFIG?
Everything is fine; I don't know why there was only an issue with
CONFIG_GRANT_TABLE on PPC. On the RISC-V side, there were more configs
issues, prompting me to include all the configurations not implemented
for RISC-V in EXTRA_FIXED_RANDCONFIG. You can find the added
configurations in this commit:
https://lore.kernel.org/xen-devel/b4e85f8f58787b4d179022973ce25673d6b56e36.1700761381.git.oleksii.kuroc...@gmail.com/#Z31automation:gitlab-ci:build.yaml

One challenge is that the same configurations need to be added multiple
times for each build test using randconfig.

Another reason for this approach is a suggestion from Jan (probably I
misunderstood it), who proposed using a template to instruct randconfig
not to modify currently unnecessary configurations. You can find the
suggestion and discussion here:
https://lore.kernel.org/xen-devel/008d0c66-6816-4d12-9e1f-1878e982f...@suse.com/

Perhaps we could enhance the build script to fetch "fixed" configs from
the architecture-specific fixed-defconfig instead of modifying the
Makefile directly.

> 
> ---8<---
> 
> CI: Revert "automation: Drop ppc64le-*randconfig jobs", fix
> Randconfig
> with existing infrastructure
>     
> This reverts commit cbb71b95dd708b1e26899bbe1e7bf9a85081fd60.
> 
> Signed-off-by: Andrew Cooper 
> 
> diff --git a/automation/gitlab-ci/build.yaml
> b/automation/gitlab-ci/build.yaml
> index 32af30ccedc9..346d0400ed09 100644
> --- a/automation/gitlab-ci/build.yaml
> +++ b/automation/gitlab-ci/build.yaml
> @@ -538,6 +538,7 @@ archlinux-current-gcc-riscv64-randconfig:
>  RANDCONFIG: y
>  EXTRA_FIXED_RANDCONFIG:
>    CONFIG_COVERAGE=n
> +  CONFIG_GRANT_TABLE=n
>  
>  archlinux-current-gcc-riscv64-debug-randconfig:
>    extends: .gcc-riscv64-cross-build-debug
> @@ -547,6 +548,7 @@ archlinux-current-gcc-riscv64-debug-randconfig:
>  RANDCONFIG: y
>  EXTRA_FIXED_RANDCONFIG:
>    CONFIG_COVERAGE=n
> +  CONFIG_GRANT_TABLE=n
>  
>  # Power cross-build
>  debian-bullseye-gcc-ppc64le:
> @@ -563,6 +565,26 @@ debian-bullseye-gcc-ppc64le-debug:
>  KBUILD_DEFCONFIG: ppc64_defconfig
>  HYPERVISOR_ONLY: y
>  
> +debian-bullseye-gcc-ppc64le-randconfig:
> +  extends: .gcc-ppc64le-cross-build
> +  variables:
> +    CONTAINER: debian:bullseye-ppc64le
> +    KBUILD_DEFCONFIG: ppc64_defconfig
> +    RANDCONFIG: y
> +    EXTRA_FIXED_RANDCONFIG:
> +  CONFIG_COVERAGE=n
> +  CONFIG_GRANT_TABLE=n
> +
> +debian-bullseye-gcc-ppc64le-debug-randconfig:
> +  extends: .gcc-ppc64le-cross-build-debug
> +  variables:
> +    CONTAINER: debian:bullseye-ppc64le
> +    KBUILD_DEFCONFIG: ppc64_defconfig
> +    RANDCONFIG: y
> +    EXTRA_FIXED_RANDCONFIG:
> +  CONFIG_COVERAGE=n
> +  CONFIG_GRANT_TABLE=n
> +
>  # Yocto test jobs
>  yocto-qemuarm64:
>    extends: .yocto-test-arm64
> 

~ Oleksii

Re: [PATCH] MAINTAINERS: Hand over the release manager role to Oleksii Kurochko

2023-12-07 Thread Oleksii

Hi Julien and Henry,

On Thu, 2023-12-07 at 18:46 +, Julien Grall wrote:
> Hi,
> 
> On 07/12/2023 16:20, Henry Wang wrote:
> > I've finished the opportunity to do two releases (4.17 and 4.18)
> > and Oleksii Kurochko has volunteered to be the next release
> > manager.
> 
> Henry, thanks for your time as release manager.
> Oleksii, thanks for stepping up and good luck for the role!
Thank you very much.

Just one question: Is it necessary to provide my ACK?

> 
> > Hand over the role to him by changing the maintainership of the
> > CHANGELOG.md.
> > 
> > Signed-off-by: Henry Wang 
> 
> Acked-by: Julien Grall 
> 
> I didn't hear any objection during the community call. But I will
> give 
> until Tuesday morning (UK time) just in case. If there are none, then
> I 
> will commit it.
> 
> Cheers,
> 

~ Oleksii

[xen-unstable test] 184020: tolerable FAIL - PUSHED

2023-12-07 Thread osstest service owner

flight 184020 xen-unstable real [real]
flight 184029 xen-unstable real-retest [real]
http://logs.test-lab.xenproject.org/osstest/logs/184020/
http://logs.test-lab.xenproject.org/osstest/logs/184029/

Failures :-/ but no regressions.

Tests which are failing intermittently (not blocking):
 test-armhf-armhf-xl-credit1   8 xen-bootfail pass in 184029-retest

Tests which did not succeed, but are not blocking:
 test-armhf-armhf-xl-credit1 15 migrate-support-check fail in 184029 never pass
 test-armhf-armhf-xl-credit1 16 saverestore-support-check fail in 184029 never 
pass
 test-amd64-amd64-xl-qemut-win7-amd64 19 guest-stopfail like 184005
 test-amd64-i386-xl-qemuu-win7-amd64 19 guest-stop fail like 184005
 test-amd64-amd64-xl-qemuu-ws16-amd64 19 guest-stopfail like 184005
 test-armhf-armhf-libvirt-qcow2 15 saverestore-support-check   fail like 184005
 test-amd64-i386-xl-qemut-ws16-amd64 19 guest-stop fail like 184005
 test-armhf-armhf-libvirt-raw 15 saverestore-support-checkfail  like 184005
 test-amd64-i386-xl-qemut-win7-amd64 19 guest-stop fail like 184005
 test-armhf-armhf-libvirt 16 saverestore-support-checkfail  like 184005
 test-amd64-amd64-xl-qemut-ws16-amd64 19 guest-stopfail like 184005
 test-amd64-amd64-qemuu-nested-amd 20 debian-hvm-install/l1/l2 fail like 184005
 test-amd64-i386-xl-qemuu-ws16-amd64 19 guest-stop fail like 184005
 test-amd64-amd64-xl-qemuu-win7-amd64 19 guest-stopfail like 184005
 test-amd64-i386-libvirt  15 migrate-support-checkfail   never pass
 test-amd64-i386-libvirt-xsm  15 migrate-support-checkfail   never pass
 test-amd64-i386-xl-pvshim14 guest-start  fail   never pass
 test-amd64-amd64-libvirt 15 migrate-support-checkfail   never pass
 test-amd64-amd64-libvirt-xsm 15 migrate-support-checkfail   never pass
 test-arm64-arm64-xl  15 migrate-support-checkfail   never pass
 test-arm64-arm64-xl  16 saverestore-support-checkfail   never pass
 test-arm64-arm64-xl-credit1  15 migrate-support-checkfail   never pass
 test-arm64-arm64-xl-credit1  16 saverestore-support-checkfail   never pass
 test-arm64-arm64-xl-xsm  15 migrate-support-checkfail   never pass
 test-arm64-arm64-xl-xsm  16 saverestore-support-checkfail   never pass
 test-arm64-arm64-libvirt-xsm 15 migrate-support-checkfail   never pass
 test-arm64-arm64-libvirt-xsm 16 saverestore-support-checkfail   never pass
 test-arm64-arm64-xl-credit2  15 migrate-support-checkfail   never pass
 test-arm64-arm64-xl-credit2  16 saverestore-support-checkfail   never pass
 test-arm64-arm64-xl-thunderx 15 migrate-support-checkfail   never pass
 test-arm64-arm64-xl-thunderx 16 saverestore-support-checkfail   never pass
 test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm 13 migrate-support-check 
fail never pass
 test-armhf-armhf-xl-multivcpu 15 migrate-support-checkfail  never pass
 test-armhf-armhf-xl-multivcpu 16 saverestore-support-checkfail  never pass
 test-amd64-i386-libvirt-qemuu-debianhvm-amd64-xsm 13 migrate-support-check 
fail never pass
 test-armhf-armhf-xl-arndale  15 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-arndale  16 saverestore-support-checkfail   never pass
 test-armhf-armhf-xl-rtds 15 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-rtds 16 saverestore-support-checkfail   never pass
 test-arm64-arm64-libvirt-raw 14 migrate-support-checkfail   never pass
 test-arm64-arm64-libvirt-raw 15 saverestore-support-checkfail   never pass
 test-amd64-amd64-libvirt-vhd 14 migrate-support-checkfail   never pass
 test-arm64-arm64-xl-vhd  14 migrate-support-checkfail   never pass
 test-arm64-arm64-xl-vhd  15 saverestore-support-checkfail   never pass
 test-armhf-armhf-libvirt-qcow2 14 migrate-support-checkfail never pass
 test-armhf-armhf-libvirt-raw 14 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-vhd  14 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-vhd  15 saverestore-support-checkfail   never pass
 test-amd64-i386-libvirt-raw  14 migrate-support-checkfail   never pass
 test-armhf-armhf-xl  15 migrate-support-checkfail   never pass
 test-armhf-armhf-xl  16 saverestore-support-checkfail   never pass
 test-armhf-armhf-xl-credit2  15 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-credit2  16 saverestore-support-checkfail   never pass
 test-armhf-armhf-libvirt 15 migrate-support-checkfail   never pass

version targeted for testing:
 xen  d4bfd3899886d0fbe259c20660dadb1e00170f2d
baseline version:
 xen  01da0aeecd41435cea8bd2fe0f547e0a474f6e45

Last test of basis   184005  2023-12-06 06:14:42 Z

Re: [RFC PATCH] xen/arm: Add emulation of Debug Data Transfer Registers

2023-12-07 Thread Stefano Stabellini

On Thu, 7 Dec 2023, Julien Grall wrote:
> Hi Stefano,
> 
> On 05/12/2023 23:21, Stefano Stabellini wrote:
> > On Tue, 5 Dec 2023, Julien Grall wrote:
> > > I agree that crashing a guest is bad, but is lying to the domain really
> > > better? The consequence here is not that bad and hopefully it would be
> > > fairly
> > > easy to find. But this is not always the case. So I definitely would place
> > > a
> > > half-backed emulation more severe than a guest crash.
> > 
> > 
> > I see where Julien is coming from, but I would go with option two:
> > "emulate DCC the same way as KVM". That's because I don't think we can
> > get away with crashing the guest in all cases. Although the issue came
> > up with a Linux guest, it could have been triggered by a proprietary
> > operating system that we cannot change, and I think Xen should support
> > running unmodified operating systems.
> > 
> > If we go with a "half-backed emulation" solution, as Julien wrote, then
> > it is better to be more similar to other hypervisors, that's why I chose
> > option two instead of option three.
> > 
> > But at the same time I recognize the validity of Julien's words and it
> > makes me wonder if we should have a KCONFIG option or command line
> > option to switch the Xen behavior. We could use it to gate all the
> > "half-backed emulation" we do for compatibility.  Something like:
> > 
> > config PARTIAL_EMULATION
> >  bool "Partial Emulation"
> >  ---help---
> >Enables partial, not spec compliant, emulation of certain
> > register
> >  interfaces (e.g DCC UART) for guest compatibility. If you disable
> >  this option, Xen will crash the guest if the guest tries to access
> >  interfaces not fully emulated or virtualized.
> > 
> >  If you enable this option, the guest might misbehave due to non-spec
> >  compliant emulation done by Xen.
> 
> As I wrote to Ayan on Matrix today, I am not in favor of the emulation. Yet, I
> am not going to oppose (as in Nack it) if the other maintainers agree with it.

Thanks for being flexible


> The KConfig would be nice, the question is whether we want to (security)
> support such configuration? E.g. could this potentially introduce a security
> issue in the guest?

The important question is whether it could introduce a security issue in
Xen. If we think it wouldn't increase the attack surface significantly
then I would security support it otherwise not.


> Regarding the  emulation itself, I actually prefer 3 because at least the
> Linux drivers will be able to bail out rather than trying to use them.

I don't have a strong opinion between 2 and 3

[ovmf test] 184028: all pass - PUSHED

2023-12-07 Thread osstest service owner

flight 184028 ovmf real [real]
http://logs.test-lab.xenproject.org/osstest/logs/184028/

Perfect :-)
All tests in this flight passed as required
version targeted for testing:
 ovmf eccdab611c01aa40b6cefcfbcb4d23e54b4c0ec6
baseline version:
 ovmf 238690a30d02d3f95f0355c88c35dc0e4232342a

Last test of basis   184027  2023-12-07 17:11:00 Z0 days
Testing same since   184028  2023-12-07 20:11:01 Z0 days1 attempts


People who touched revisions under test:
  Corvin Köhne 
  Gerd Hoffmann 
  Laszlo Ersek 

jobs:
 build-amd64-xsm  pass
 build-i386-xsm   pass
 build-amd64  pass
 build-i386   pass
 build-amd64-libvirt  pass
 build-i386-libvirt   pass
 build-amd64-pvopspass
 build-i386-pvops pass
 test-amd64-amd64-xl-qemuu-ovmf-amd64 pass
 test-amd64-i386-xl-qemuu-ovmf-amd64  pass



sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images

Logs, config files, etc. are available at
http://logs.test-lab.xenproject.org/osstest/logs

Explanation of these reports, and of osstest in general, is at
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master

Test harness code can be found at
http://xenbits.xen.org/gitweb?p=osstest.git;a=summary


Pushing revision :

To xenbits.xen.org:/home/xen/git/osstest/ovmf.git
   238690a30d..eccdab611c  eccdab611c01aa40b6cefcfbcb4d23e54b4c0ec6 -> 
xen-tested-master

[xen-unstable-smoke test] 184030: tolerable all pass - PUSHED

2023-12-07 Thread osstest service owner

flight 184030 xen-unstable-smoke real [real]
http://logs.test-lab.xenproject.org/osstest/logs/184030/

Failures :-/ but no regressions.

Tests which did not succeed, but are not blocking:
 test-amd64-amd64-libvirt 15 migrate-support-checkfail   never pass
 test-arm64-arm64-xl-xsm  15 migrate-support-checkfail   never pass
 test-arm64-arm64-xl-xsm  16 saverestore-support-checkfail   never pass
 test-armhf-armhf-xl  15 migrate-support-checkfail   never pass
 test-armhf-armhf-xl  16 saverestore-support-checkfail   never pass

version targeted for testing:
 xen  bc4fe94a69d4dab103c37045d97e589ef75f8647
baseline version:
 xen  d2b7c442b4a066bb670ee83e24800cabc415241d

Last test of basis   184026  2023-12-07 16:02:06 Z0 days
Testing same since   184030  2023-12-07 21:02:00 Z0 days1 attempts


People who touched revisions under test:
  Andrew Cooper 
  Juergen Gross 

jobs:
 build-arm64-xsm  pass
 build-amd64  pass
 build-armhf  pass
 build-amd64-libvirt  pass
 test-armhf-armhf-xl  pass
 test-arm64-arm64-xl-xsm  pass
 test-amd64-amd64-xl-qemuu-debianhvm-amd64pass
 test-amd64-amd64-libvirt pass



sg-report-flight on osstest.test-lab.xenproject.org
logs: /home/logs/logs
images: /home/logs/images

Logs, config files, etc. are available at
http://logs.test-lab.xenproject.org/osstest/logs

Explanation of these reports, and of osstest in general, is at
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README.email;hb=master
http://xenbits.xen.org/gitweb/?p=osstest.git;a=blob;f=README;hb=master

Test harness code can be found at
http://xenbits.xen.org/gitweb?p=osstest.git;a=summary


Pushing revision :

To xenbits.xen.org:/home/xen/git/xen.git
   d2b7c442b4..bc4fe94a69  bc4fe94a69d4dab103c37045d97e589ef75f8647 -> smoke

xen | Failed pipeline for staging | bc4fe94a

2023-12-07 Thread GitLab



Pipeline #1099444749 has failed!

Project: xen ( https://gitlab.com/xen-project/xen )
Branch: staging ( https://gitlab.com/xen-project/xen/-/commits/staging )

Commit: bc4fe94a ( 
https://gitlab.com/xen-project/xen/-/commit/bc4fe94a69d4dab103c37045d97e589ef75f8647
 )
Commit Message: tools/libs/evtchn: replace assert()s in stubdom...
Commit Author: Juergen Gross ( https://gitlab.com/jgross1 )
Committed by: Andrew Cooper ( https://gitlab.com/andyhhp )


Pipeline #1099444749 ( 
https://gitlab.com/xen-project/xen/-/pipelines/1099444749 ) triggered by Ganis 
( https://gitlab.com/ganis )
had 1 failed job.

Job #5706590895 ( https://gitlab.com/xen-project/xen/-/jobs/5706590895/raw )

Stage: test
Name: build-each-commit-gcc

-- 
You're receiving this email because of your account on gitlab.com.

Re: [PATCH] MAINTAINERS: Hand over the release manager role to Oleksii Kurochko

2023-12-07 Thread Henry Wang

Hi Julien, Oleksii,

> On Dec 8, 2023, at 05:09, Oleksii  wrote:
> 
> Hi Julien and Henry,
> 
> On Thu, 2023-12-07 at 18:46 +, Julien Grall wrote:
>> Hi,
>> 
>> On 07/12/2023 16:20, Henry Wang wrote:
>>> I've finished the opportunity to do two releases (4.17 and 4.18)
>>> and Oleksii Kurochko has volunteered to be the next release
>>> manager.
>> 
>> Henry, thanks for your time as release manager.

Thank you Julien for your kind support as the release technician!

>> Oleksii, thanks for stepping up and good luck for the role!
> Thank you very much.
> 
> Just one question: Is it necessary to provide my ACK?

Oleksii: Yes please, your ack means you are happy with taking the role.

Also, any thing that you need, please don’t hesitate to reach out. I will stick
to the community (more as the code contributor though) so I will be more
than happy to help.

>>> Hand over the role to him by changing the maintainership of the
>>> CHANGELOG.md.
>>> 
>>> Signed-off-by: Henry Wang 
>> 
>> Acked-by: Julien Grall 
>> 
>> I didn't hear any objection during the community call. But I will
>> give 
>> until Tuesday morning (UK time) just in case. If there are none, then
>> I 
>> will commit it.

Thanks Julien.

Kind regards,
Henry

>> 
>> Cheers,
>> 
> 
> ~ Oleksii

Re: [PATCH] docs/misra/rules.rst: add more rules

2023-12-07 Thread Stefano Stabellini

On Thu, 7 Dec 2023, Jan Beulich wrote:
> On 07.12.2023 03:42, Stefano Stabellini wrote:
> > On Wed, 6 Dec 2023, Jan Beulich wrote:
> >> On 06.12.2023 04:02, Stefano Stabellini wrote:
> >>> --- a/docs/misra/rules.rst
> >>> +++ b/docs/misra/rules.rst
> >>> @@ -462,11 +462,23 @@ maintainers if you want to suggest a change.
> >>>  
> >>> while(0) and while(1) and alike are allowed.
> >>>  
> >>> +   * - `Rule 16.3 
> >>> `_
> >>> + - Required
> >>> + - An unconditional break statement shall terminate every
> >>> +   switch-clause
> >>> + - In addition to break, also other flow control statements such as
> >>> +   continue, return, goto are allowed.
> >>> +
> >>> * - `Rule 16.7 
> >>> `_
> >>>   - Required
> >>>   - A switch-expression shall not have essentially Boolean type
> >>>   -
> >>>  
> >>> +   * - `Rule 17.1 
> >>> `_
> >>> + - Required
> >>> + - The features of  shall not be used
> >>> + -
> >>
> >> Did we really accept this without any constraint (warranting mentioning
> >> here)?
> > 
> > We agreed that in certain situations stdarg.h is OK to use and in those
> > cases we would add a deviation. Would you like me to add something to
> > that effect here? I could do that but it would sound a bit vague.  Also
> > if we want to specify a project-wide deviation it would be better
> > documented in docs/misra/deviations.rst. I would leave Rule 17.1 without
> > a note.
> 
> I can see your point, and I don't have a good suggestion on possible text.
> Still I wouldn't feel well ack-ing this in its present shape.

What about:

 - It is understood that in some limited circumstances  is
   appropriate to use, such as the implementation of printk. Those
   cases will be dealt with using deviations as usual, see
   docs/misra/deviations.rst and
   docs/misra/documenting-violations.rst.



> >>> @@ -478,12 +490,24 @@ maintainers if you want to suggest a change.
> >>> have an explicit return statement with an expression
> >>>   -
> >>>  
> >>> +   * - `Rule 17.5 
> >>> `_
> >>> + - Advisory
> >>> + - The function argument corresponding to a parameter declared to
> >>> +   have an array type shall have an appropriate number of elements
> >>> + -
> >>> +
> >>> * - `Rule 17.6 
> >>> `_
> >>>   - Mandatory
> >>>   - The declaration of an array parameter shall not contain the
> >>> static keyword between the [ ]
> >>>   -
> >>>  
> >>> +   * - `Rule 17.7 
> >>> `_
> >>> + - Required
> >>> + - The value returned by a function having non-void return type
> >>> +   shall be used
> >>> + -
> >>
> >> Same question here.
> > 
> > Here I was also thinking it might be good to add a comment. Maybe we could
> > add:
> > 
> >  - Please beware that this rule has many violations in the Xen
> >codebase today, and its adoption is aspirational. However, when
> >submitting new patches please try to decrease the number of
> >violations when possible.
> 
> Yea, I think this would be good to add.

I sent out a patch with this addition, and removing Rule 17.1 as that
one is a bit more complicated.

[PATCH v2] docs/misra/rules.rst: add more rules

2023-12-07 Thread Stefano Stabellini

Add the rules accepted in the last three MISRA C working group meetings.

Signed-off-by: Stefano Stabellini 
---
Changes in v2:
- remove 17.1 for now, to be a separate patch
- add a clarification comment for 17.7
---
 docs/misra/rules.rst | 34 ++
 1 file changed, 34 insertions(+)

diff --git a/docs/misra/rules.rst b/docs/misra/rules.rst
index 75921b9a34..2b570af0e0 100644
--- a/docs/misra/rules.rst
+++ b/docs/misra/rules.rst
@@ -462,6 +462,13 @@ maintainers if you want to suggest a change.
 
while(0) and while(1) and alike are allowed.
 
+   * - `Rule 16.3 
`_
+ - Required
+ - An unconditional break statement shall terminate every
+   switch-clause
+ - In addition to break, also other flow control statements such as
+   continue, return, goto are allowed.
+
* - `Rule 16.7 
`_
  - Required
  - A switch-expression shall not have essentially Boolean type
@@ -478,12 +485,27 @@ maintainers if you want to suggest a change.
have an explicit return statement with an expression
  -
 
+   * - `Rule 17.5 
`_
+ - Advisory
+ - The function argument corresponding to a parameter declared to
+   have an array type shall have an appropriate number of elements
+ -
+
* - `Rule 17.6 
`_
  - Mandatory
  - The declaration of an array parameter shall not contain the
static keyword between the [ ]
  -
 
+   * - `Rule 17.7 
`_
+ - Required
+ - The value returned by a function having non-void return type
+   shall be used
+ - Please beware that this rule has many violations in the Xen
+   codebase today, and its adoption is aspirational. However, when
+   submitting new patches please try to decrease the number of
+   violations when possible.
+
* - `Rule 18.3 
`_
  - Required
  - The relational operators > >= < and <= shall not be applied to objects 
of pointer type except where they point into the same object
@@ -498,6 +520,11 @@ maintainers if you want to suggest a change.
instances where Eclair is unable to verify that the code is valid
in regard to Rule 19.1. Caution reports are not violations.
 
+   * - `Rule 20.4 
`_
+ - Required
+ - A macro shall not be defined with the same name as a keyword
+ -
+
* - `Rule 20.7 
`_
  - Required
  - Expressions resulting from the expansion of macro parameters
@@ -506,6 +533,13 @@ maintainers if you want to suggest a change.
as function arguments, as macro arguments, array indices, lhs in
assignments
 
+   * - `Rule 20.9 
`_
+ - Required
+ - All identifiers used in the controlling expression of #if or
+   #elif preprocessing directives shall be #define'd before
+   evaluation
+ -
+
* - `Rule 20.13 
`_
  - Required
  - A line whose first token is # shall be a valid preprocessing
-- 
2.25.1

[PATCH] docs/misra/rules.rst: add Rule 16.2

2023-12-07 Thread Stefano Stabellini



Signed-off-by: Stefano Stabellini 

diff --git a/docs/misra/rules.rst b/docs/misra/rules.rst
index 2b570af0e0..7cb9544a96 100644
--- a/docs/misra/rules.rst
+++ b/docs/misra/rules.rst
@@ -469,6 +469,15 @@ maintainers if you want to suggest a change.
  - In addition to break, also other flow control statements such as
continue, return, goto are allowed.
 
+   * - `Rule 16.2 
`_
+ - Required
+ - A switch label shall only be used when the most closely-enclosing
+   compound statement is the body of a switch statement
+ - The x86 emulator (xen/arch/x86/x86_emulate*) is exempt from
+   compliance with this rule. Efforts to make the x86 emulator
+   adhere to Rule 16.2 would result in increased complexity and
+   maintenance difficulty, and could potentially introduce bugs. 
+
* - `Rule 16.7 
`_
  - Required
  - A switch-expression shall not have essentially Boolean type

Re: [XEN PATCH] automation/eclair: add deviations for MISRA C:2012 Rule 16.3

2023-12-07 Thread Stefano Stabellini

On Thu, 7 Dec 2023, Julien Grall wrote:
> Hi Federico,
> 
> On 07/12/2023 09:08, Federico Serafini wrote:
> > MISRA C:2012 Rule 16.3 states that an unconditional break statement
> > shall terminate every switch-clause.
> > 
> > Update ECLAIR configuration to take into account:
> > - continue, goto, return statements;
> > - functions and macros that do not give the control back;
> > - fallthrough comments and pseudo-keywords.
> > 
> > Update docs/misra/deviations.rst accordingly.
> > 
> > Signed-off-by: Federico Serafini 
> > ---
> >   .../eclair_analysis/ECLAIR/deviations.ecl | 18 ++
> >   docs/misra/deviations.rst | 24 +++
> >   2 files changed, 42 insertions(+)
> 
> It would be good that this is depending on to be accepted:
> 
> https://lore.kernel.org/alpine.DEB.2.22.394.2312051859440.110490@ubuntu-linux-20-04-desktop.
> 
> > 
> > diff --git a/automation/eclair_analysis/ECLAIR/deviations.ecl
> > b/automation/eclair_analysis/ECLAIR/deviations.ecl
> > index b0c79741b5..df0b58a010 100644
> > --- a/automation/eclair_analysis/ECLAIR/deviations.ecl
> > +++ b/automation/eclair_analysis/ECLAIR/deviations.ecl
> > @@ -321,6 +321,24 @@ statements are deliberate"
> >   -config=MC3R1.R14.3,statements={deliberate ,
> > "wrapped(any(),node(if_stmt))" }
> >   -doc_end
> >   +#
> > +# Series 16.
> > +#
> > +
> > +-doc_begin="Switch clauses ending with continue, goto, return statements
> > are safe."
> > +-config=MC3R1.R16.3,terminals+={safe,
> > "node(continue_stmt||goto_stmt||return_stmt)"}
> > +-doc_end
> > +
> > +-doc_begin="Switch clauses not ending with the break statement are safe if
> > a function/macro that does not give the control back is present."
> > +-config=MC3R1.R16.3,terminals+={safe,
> > "call(decl(name(__builtin_unreachable||do_unexpected_trap||fatal_trap||machine_halt||machine_restart||maybe_reboot||panic)))"}
> > +-config=MC3R1.R16.3,terminals+={safe,"macro(name(BUG||BUG_ON))"}
> > +-doc_end
> > +
> > +-doc_begin="Switch clauses not ending with the break statement are safe if
> > an explicit comment or pseudo-keyword indicating the fallthrough intention
> > is present."
> > +-config=MC3R1.R16.3,reports+={safe,
> > "any_area(any_loc(any_exp(text(^(?s).*([fF]all[- ]?[tT]hrough|FALL[-
> > ]?THROUGH).*$,0..1"}
> > +-config=MC3R1.R16.3,reports+={safe, "any_area(text(^(?s).*([fF]all[-
> > ]?[tT]hrough|FALL[- ]?THROUGH).*$,0..1))"}
> 
> This is not trivial to read. Can you document the exhaustive list of keywords
> you are actually trying to deviate on? Also, did you consider to harmonize to
> only a few?
> 
> > +-doc_end
> > +
> >   #
> >   # Series 20.
> >   #
> > diff --git a/docs/misra/deviations.rst b/docs/misra/deviations.rst
> > index 6e7c4f25b8..fecd2bae8e 100644
> > --- a/docs/misra/deviations.rst
> > +++ b/docs/misra/deviations.rst
> > @@ -270,6 +270,30 @@ Deviations related to MISRA C:2012 Rules:
> >  statements are deliberate.
> >- Project-wide deviation; tagged as `disapplied` for ECLAIR.
> >   +   * - R16.3
> > + - Switch clauses ending with continue, goto, return statements are
> > safe.
> > + - Tagged as `safe` for ECLAIR.
> > +
> > +   * - R16.3
> > + - Switch clauses not ending with the break statement are safe if a
> > +   function/macro that does not give the control back is present.
> > + - Tagged as `safe` for ECLAIR, such functions/macros are:
> > +- __builtin_unreachable
> > +- do_unexpected_trap
> > +- fatal_trap
> > +- machine_halt
> > +- machine_restart
> > +- maybe_reboot
> > +- panic
> > +- BUG
> 
> To me, it seems to be odd to deviate R16.3 by function. Some of those have an
> attribute noreboot. Can this be used?

Just to clarify, I think Julien meant "noreturn" which is defined as
__attribute__((__noreturn__))

I think we need to deviate by function, rather than using SAF, because
we would have to use SAF in every switch rather than at the declaration
of panic and friends. But if we could use noreturn, that would be
awesome.


> > +- BUG_ON
> 
> BUG_ON() can return if the condition is false. So it doesn't look correct to
> deviate with the argument that the function doesn't give the control back...

+1


> > +
> > +   * - R16.3
> > + - Switch clauses not ending with the break statement are safe if an
> > +   explicit comment or pseudo-keyword indicating the fallthrough
> > intention
> > +   is present.
> > + - Tagged as `safe` for ECLAIR.
> > +
> >  * - R20.7
> >- Code violating Rule 20.7 is safe when macro parameters are used:
> >  (1) as function arguments;

Re: [XEN PATCH v2 3/5] x86/mm: remove compat_subarch_memory_op()

2023-12-07 Thread Stefano Stabellini

On Thu, 7 Dec 2023, Federico Serafini wrote:
> Remove remove compat_subarch_memory_op() declaration: there is no
> definition and there are no calls to such function in the XEN project.
> 
> Signed-off-by: Federico Serafini 

Reviewed-by: Stefano Stabellini

Re: [XEN PATCH v2 5/5] AMD/IOMMU: address violations of MISRA C:2012 Rule 8.2

2023-12-07 Thread Stefano Stabellini

On Thu, 7 Dec 2023, Federico Serafini wrote:
> Add missing parameter names to address violations of MISRA C:2012
> Rule 8.2. Remove trailing spaces and use C standard types to comply
> with XEN coding style. No functional change.
> 
> Signed-off-by: Federico Serafini 

Reviewed-by: Stefano Stabellini

Re: [XEN PATCH v2 4/5] x86/mm: address violations of MISRA C:2012 Rule 8.2

2023-12-07 Thread Stefano Stabellini

On Thu, 7 Dec 2023, Federico Serafini wrote:
> Add missing parameter names. No functional change.
> 
> Signed-off-by: Federico Serafini 

Reviewed-by: Stefano Stabellini

Re: [XEN PATCH v2 2/5] xen/acpi: address violations of MISRA C:2012 Rule 8.2

2023-12-07 Thread Stefano Stabellini

On Thu, 7 Dec 2023, Federico Serafini wrote:
> Add missing parameter names. No functional change.
> 
> Signed-off-by: Federico Serafini 

Reviewed-by: Stefano Stabellini

Re: [PATCH 1/3] AMD/IOMMU: address violations of MISRA C:2012 Rule 14.4

2023-12-07 Thread Stefano Stabellini

On Thu, 7 Dec 2023, Simone Ballarin wrote:
> From: Maria Celeste Cesario 
> 
> The xen sources contain violations of MISRA C:2012 Rule 14.4 whose
> headline states:
> "The controlling expression of an if statement and the controlling
> expression of an iteration-statement shall have essentially Boolean type".
> 
> Add comparisons to avoid using enum constants as controlling expressions
> to comply with Rule 14.4.
> No functional change.
> 
> Signed-off-by: Maria Celeste Cesario  
> Signed-off-by: Simone Ballarin  

Reviewed-by: Stefano Stabellini

Re: [PATCH 2/3] xen/x86: address violations of MISRA C:2012 Rule 14.4

2023-12-07 Thread Stefano Stabellini

On Thu, 7 Dec 2023, Simone Ballarin wrote:
> From: Maria Celeste Cesario 
> 
> The xen sources contain violations of MISRA C:2012 Rule 14.4 whose
> headline states:
> "The controlling expression of an if statement and the controlling
> expression of an iteration-statement shall have essentially Boolean type".
> 
> Add comparisons to avoid using enum constants as controlling expressions
> to comply with Rule 14.4.
> No functional change.
> 
> Signed-off-by: Maria Celeste Cesario  
> Signed-off-by: Simone Ballarin  

I think it would have been better to put all the iommu_intremap changed
in the same patch (patch #1). But anyway:

Reviewed-by: Stefano Stabellini

Re: [PATCH 3/3] xen: address violations of MISRA C:2012 Rule 14.4

2023-12-07 Thread Stefano Stabellini

On Thu, 7 Dec 2023, Simone Ballarin wrote:
> From: Maria Celeste Cesario 
> 
> The xen sources contain violations of MISRA C:2012 Rule 14.4 whose
> headline states:
> "The controlling expression of an if statement and the controlling
> expression of an iteration-statement shall have essentially Boolean type".
> 
> Struct domain member is_dying is an anonymous enum designed to act as boolean.
> Add deviation to mark its uses in controlling expressions as deliberate.
> 
> Signed-off-by: Maria Celeste Cesario 
> Signed-off-by: Simone Ballarin 

Reviewed-by: Stefano Stabellini

Re: [RFC PATCH v1 1/1] xen/Makefile: introduce ARCH_FIXED_CONFIG for randconfig

2023-12-07 Thread Stefano Stabellini

On Thu, 7 Dec 2023, Oleksii wrote:
> On Thu, 2023-12-07 at 20:17 +, Andrew Cooper wrote:
> > On 07/12/2023 5:03 pm, Oleksii Kurochko wrote:
> > > ARCH_FIXED_CONFIG is required in the case of randconfig
> > > and CI for configs that aren't ready or are not
> > > supposed to be implemented for specific architecture.
> > > These configs should always be disabled to prevent randconfig
> > > related tests from failing.
> > > 
> > > Signed-off-by: Oleksii Kurochko 
> > > ---
> > >  xen/Makefile | 5 -
> > >  1 file changed, 4 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/xen/Makefile b/xen/Makefile
> > > index ca571103c8..8ae8fe1480 100644
> > > --- a/xen/Makefile
> > > +++ b/xen/Makefile
> > > @@ -336,11 +336,14 @@ ifeq ($(config-build),y)
> > >  # *config targets only - make sure prerequisites are updated, and
> > > descend
> > >  # in tools/kconfig to make the *config target
> > >  
> > > +ARCH_FORCED_CONFIG :=
> > > $(srctree)/arch/$(SRCARCH)/configs/randomforced.config
> > > +
> > >  # Create a file for KCONFIG_ALLCONFIG which depends on the
> > > environment.
> > >  # This will be use by kconfig targets
> > > allyesconfig/allmodconfig/allnoconfig/randconfig
> > >  filechk_kconfig_allconfig = \
> > >  $(if $(findstring n,$(XEN_HAS_CHECKPOLICY)), echo
> > > 'CONFIG_XSM_FLASK_POLICY=n';) \
> > > -    $(if $(KCONFIG_ALLCONFIG), cat $(KCONFIG_ALLCONFIG);) \
> > > +    $(if $(KCONFIG_ALLCONFIG), cat $(KCONFIG_ALLCONFIG); \
> > > +    $(if $(wildcard $(ARCH_FORCED_CONFIG)), cat
> > > $(ARCH_FORCED_CONFIG);) ) \
> > >  :
> > >  
> > >  .allconfig.tmp: FORCE
> > 
> > We already have infrastructure for this.  What's wrong with
> > EXTRA_FIXED_RANDCONFIG?
> Everything is fine; I don't know why there was only an issue with
> CONFIG_GRANT_TABLE on PPC. On the RISC-V side, there were more configs
> issues, prompting me to include all the configurations not implemented
> for RISC-V in EXTRA_FIXED_RANDCONFIG. You can find the added
> configurations in this commit:
> https://lore.kernel.org/xen-devel/b4e85f8f58787b4d179022973ce25673d6b56e36.1700761381.git.oleksii.kuroc...@gmail.com/#Z31automation:gitlab-ci:build.yaml
> 
> One challenge is that the same configurations need to be added multiple
> times for each build test using randconfig.

That's a lot of extra configs to add. Could you use a yaml anchor or a
.something to include? So that you define the full list only once at the
top of the file and then reuse it everywhere as needed.


> Another reason for this approach is a suggestion from Jan (probably I
> misunderstood it), who proposed using a template to instruct randconfig
> not to modify currently unnecessary configurations. You can find the
> suggestion and discussion here:
> https://lore.kernel.org/xen-devel/008d0c66-6816-4d12-9e1f-1878e982f...@suse.com/
> 
> Perhaps we could enhance the build script to fetch "fixed" configs from
> the architecture-specific fixed-defconfig instead of modifying the
> Makefile directly.

Sorry I missed the original thread somehow. Please use "automation" as
subject line tag for automation patches.

[linux-linus test] 184021: regressions - FAIL

2023-12-07 Thread osstest service owner

flight 184021 linux-linus real [real]
http://logs.test-lab.xenproject.org/osstest/logs/184021/

Regressions :-(

Tests which did not succeed and are blocking,
including tests which could not be run:
 build-i386-pvops  6 kernel-build fail REGR. vs. 183973

Tests which did not succeed, but are not blocking:
 test-armhf-armhf-libvirt 16 saverestore-support-checkfail  like 183973
 test-amd64-amd64-xl-qemut-win7-amd64 19 guest-stopfail like 183973
 test-amd64-amd64-xl-qemuu-win7-amd64 19 guest-stopfail like 183973
 test-amd64-amd64-xl-qemuu-ws16-amd64 19 guest-stopfail like 183973
 test-armhf-armhf-libvirt-raw 15 saverestore-support-checkfail  like 183973
 test-amd64-amd64-xl-qemut-ws16-amd64 19 guest-stopfail like 183973
 test-armhf-armhf-libvirt-qcow2 15 saverestore-support-check   fail like 183973
 test-amd64-amd64-qemuu-nested-amd 20 debian-hvm-install/l1/l2 fail like 183973
 test-amd64-amd64-libvirt-xsm 15 migrate-support-checkfail   never pass
 test-amd64-amd64-libvirt 15 migrate-support-checkfail   never pass
 test-arm64-arm64-xl-thunderx 15 migrate-support-checkfail   never pass
 test-arm64-arm64-xl  15 migrate-support-checkfail   never pass
 test-arm64-arm64-xl-thunderx 16 saverestore-support-checkfail   never pass
 test-arm64-arm64-xl  16 saverestore-support-checkfail   never pass
 test-arm64-arm64-libvirt-xsm 15 migrate-support-checkfail   never pass
 test-arm64-arm64-libvirt-xsm 16 saverestore-support-checkfail   never pass
 test-arm64-arm64-xl-credit2  15 migrate-support-checkfail   never pass
 test-arm64-arm64-xl-credit2  16 saverestore-support-checkfail   never pass
 test-arm64-arm64-xl-credit1  15 migrate-support-checkfail   never pass
 test-arm64-arm64-xl-credit1  16 saverestore-support-checkfail   never pass
 test-amd64-amd64-libvirt-qemuu-debianhvm-amd64-xsm 13 migrate-support-check 
fail never pass
 test-armhf-armhf-xl-multivcpu 15 migrate-support-checkfail  never pass
 test-armhf-armhf-xl-multivcpu 16 saverestore-support-checkfail  never pass
 test-armhf-armhf-libvirt 15 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-arndale  15 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-arndale  16 saverestore-support-checkfail   never pass
 test-arm64-arm64-xl-xsm  15 migrate-support-checkfail   never pass
 test-arm64-arm64-xl-xsm  16 saverestore-support-checkfail   never pass
 test-armhf-armhf-xl-rtds 15 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-rtds 16 saverestore-support-checkfail   never pass
 test-amd64-amd64-libvirt-raw 14 migrate-support-checkfail   never pass
 test-armhf-armhf-xl  15 migrate-support-checkfail   never pass
 test-armhf-armhf-xl  16 saverestore-support-checkfail   never pass
 test-arm64-arm64-libvirt-raw 14 migrate-support-checkfail   never pass
 test-arm64-arm64-libvirt-raw 15 saverestore-support-checkfail   never pass
 test-arm64-arm64-xl-vhd  14 migrate-support-checkfail   never pass
 test-arm64-arm64-xl-vhd  15 saverestore-support-checkfail   never pass
 test-armhf-armhf-libvirt-raw 14 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-vhd  14 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-vhd  15 saverestore-support-checkfail   never pass
 test-amd64-amd64-libvirt-qcow2 14 migrate-support-checkfail never pass
 test-armhf-armhf-xl-credit2  15 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-credit2  16 saverestore-support-checkfail   never pass
 test-armhf-armhf-xl-credit1  15 migrate-support-checkfail   never pass
 test-armhf-armhf-xl-credit1  16 saverestore-support-checkfail   never pass
 test-armhf-armhf-libvirt-qcow2 14 migrate-support-checkfail never pass

version targeted for testing:
 linuxbee0e7762ad2c6025b9f5245c040fcc36ef2bde8
baseline version:
 linux815fb87b753055df2d9e50f6cd80eb10235fe3e9

Last test of basis   183973  2023-12-02 07:48:26 Z5 days
Failing since183977  2023-12-03 00:12:06 Z5 days   10 attempts
Testing same since   183993  2023-12-05 09:00:54 Z2 days4 attempts


People who touched revisions under test:
  Alex Williamson 
  Brett Creeley 
  Dan Carpenter 
  David Howells 
  Dmitry Antipov 
  Eugenio Pérez 
  Jason Gunthorpe 
  Jason Wang 
  Jens Axboe 
  Joao Martins 
  JP Kobryn 
  Juergen Gross 
  Linus Torvalds 
  Masami Hiramatsu (Google) 
  Michael Ellerman 
  Michael S. Tsirkin 
  Namjae Jeon 
  Nicholas Piggin 
  Paulo Alcantara (SUSE) 
  Paulo Alcantara 
  Robin Murphy 
  Sean Christopherson 
  Shannon Nelson 
  Steve French 
  Steve Sistare 
  Takashi Sakamoto 
  Timoth

Re: [PATCH 1/5] automation: Add a Dockerfile for running FVP_Base jobs

2023-12-07 Thread Stefano Stabellini

On Thu, 7 Dec 2023, Henry Wang wrote:
> Fixed Virtual Platforms (FVPs) are complete simulations of an Arm
> system, including processor, memory and peripherals. These are set
> out in a "programmer's view", which gives programmers a comprehensive
> model on which to build and test software. FVP can be configured to
> different setups by its cmdline parameters, and hence having the FVP
> in CI will provide us with the flexibility to test Arm features and
> setups that we find difficult to use real hardware or emulators.
> 
> This commit adds a Dockerfile for the new arm64v8 container with
> FVP installed, based on the debian bookworm-arm64v8 image. This
> container will be used to run the FVP test jobs. Compared to the
> debian bookworm-arm64v8 image, the packages in the newly added FVP
> container does not contain the `u-boot-qemu`, and adds the `expect`
> to run expect scripts introduced by following commits, `telnet` to
> connect to FVP, and `tftpd-hpa` to provide the TFTP service for
> the FVP.
> 
> Signed-off-by: Henry Wang 
> ---
>  .../debian/bookworm-arm64v8-fvp.dockerfile| 64 +++
>  1 file changed, 64 insertions(+)
>  create mode 100644 automation/build/debian/bookworm-arm64v8-fvp.dockerfile
> 
> diff --git a/automation/build/debian/bookworm-arm64v8-fvp.dockerfile 
> b/automation/build/debian/bookworm-arm64v8-fvp.dockerfile
> new file mode 100644
> index 00..3b87dc5a5b
> --- /dev/null
> +++ b/automation/build/debian/bookworm-arm64v8-fvp.dockerfile

Please move this container under automation/tests-artifacts/ because the
container is only meant to be used for tests as opposed as to build Xen.
I know that in other cases we have reused the build container but that
just because it was already there an working for the purpose.

Also please name it based on the fvp rather than debian:

automation/tests-artifacts/armfvp/11.23_9-arm64v8.dockerfile

Everything else looks fine.



> @@ -0,0 +1,64 @@
> +FROM --platform=linux/arm64/v8 debian:bookworm
> +LABEL maintainer.name="The Xen Project" \
> +  maintainer.email="xen-devel@lists.xenproject.org"
> +
> +ARG FVP_BASE_VERSION="11.23_9_Linux64_armv8l"
> +
> +ENV DEBIAN_FRONTEND=noninteractive
> +ENV USER root
> +
> +RUN mkdir /build
> +WORKDIR /build
> +
> +# build depends
> +RUN apt-get update && \
> +apt-get --quiet --yes install \
> +build-essential \
> +zlib1g-dev \
> +libncurses5-dev \
> +libssl-dev \
> +python3-dev \
> +python3-setuptools \
> +xorg-dev \
> +uuid-dev \
> +libyajl-dev \
> +libaio-dev \
> +libglib2.0-dev \
> +clang \
> +libpixman-1-dev \
> +pkg-config \
> +flex \
> +bison \
> +acpica-tools \
> +libfdt-dev \
> +bin86 \
> +bcc \
> +liblzma-dev \
> +libnl-3-dev \
> +ocaml-nox \
> +libfindlib-ocaml-dev \
> +markdown \
> +transfig \
> +pandoc \
> +checkpolicy \
> +wget \
> +git \
> +nasm \
> +# for test phase, fvp-smoke-* jobs
> +u-boot-tools \
> +expect \
> +device-tree-compiler \
> +curl \
> +cpio \
> +busybox-static \
> +telnet \
> +tftpd-hpa \
> +&& \
> +apt-get autoremove -y && \
> +apt-get clean && \
> +rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
> +
> +RUN wget 
> https://developer.arm.com/-/media/Files/downloads/ecosystem-models/FVP_Base_RevC-2xAEMvA_${FVP_BASE_VERSION}.tgz
>  && \
> +mkdir -p /FVP/FVP_Base_RevC-2xAEMvA && \
> +tar -xvzf FVP_Base_RevC-2xAEMvA_${FVP_BASE_VERSION}.tgz -C 
> /FVP/FVP_Base_RevC-2xAEMvA && \
> +rm FVP_Base_RevC-2xAEMvA_${FVP_BASE_VERSION}.tgz
> -- 
> 2.25.1
>

Re: [PATCH 2/5] automation: Add the Dockerfile to build TF-A and U-Boot for FVP

2023-12-07 Thread Stefano Stabellini

On Thu, 7 Dec 2023, Henry Wang wrote:
> Unlike the emulators that currently being used in the CI pipelines,
> the FVP must start at EL3. Therefore we need the firmware, i.e. the
> TrustedFirmware-A (TF-A), for corresponding functionality.
> 
> There is a dedicated board (vexpress_fvp) in U-Boot (serve as the
> BL33 of the TF-A) for the FVP platform, so the U-Boot should also be
> compiled for the FVP platform instead of reusing the U-Boot for the
> existing emulators used in the CI pipelines.
> 
> To avoid compiling TF-A and U-Boot everytime in the job, adding a
> Dockerfile to the test artifacts to build TF-A v2.9.0 and U-Boot
> v2023.10 for FVP. The binaries for the TF-A and U-Boot, as well as
> the device tree for the FVP platform, will be saved (and exported by
> the CI job introduced by following commits). Note that, a patch for
> the TF-A will be applied before building to enable the virtio-net
> and the virtio-rng device on the FVP. The virtio-net device will
> provide the networking service for FVP, and the virtio-rng device
> will improve the speed of the FVP.
> 
> Signed-off-by: Henry Wang 

Reviewed-by: Stefano Stabellini 


> ---
>  .../2023.10-2.9.0-arm64v8.dockerfile  | 48 +++
>  1 file changed, 48 insertions(+)
>  create mode 100644 
> automation/tests-artifacts/armfvp-uboot-tfa/2023.10-2.9.0-arm64v8.dockerfile
> 
> diff --git 
> a/automation/tests-artifacts/armfvp-uboot-tfa/2023.10-2.9.0-arm64v8.dockerfile
>  
> b/automation/tests-artifacts/armfvp-uboot-tfa/2023.10-2.9.0-arm64v8.dockerfile
> new file mode 100644
> index 00..6566b60545
> --- /dev/null
> +++ 
> b/automation/tests-artifacts/armfvp-uboot-tfa/2023.10-2.9.0-arm64v8.dockerfile
> @@ -0,0 +1,48 @@
> +FROM --platform=linux/arm64/v8 debian:bookworm
> +LABEL maintainer.name="The Xen Project" \
> +  maintainer.email="xen-devel@lists.xenproject.org"
> +
> +ENV DEBIAN_FRONTEND=noninteractive
> +ENV UBOOT_VERSION="2023.10"
> +ENV TFA_VERSION="v2.9.0"
> +ENV USER root
> +
> +RUN mkdir /build
> +WORKDIR /build
> +
> +# build depends
> +RUN apt-get update && \
> +apt-get --quiet --yes install \
> +build-essential \
> +libssl-dev \
> +bc \
> +curl \
> +flex \
> +bison \
> +git \
> +device-tree-compiler && \
> +apt-get autoremove -y && \
> +apt-get clean && \
> +rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
> +
> +# Build U-Boot and TF-A
> +RUN curl -fsSLO 
> https://ftp.denx.de/pub/u-boot/u-boot-"$UBOOT_VERSION".tar.bz2 && \
> +tar xvf u-boot-"$UBOOT_VERSION".tar.bz2 && \
> +cd u-boot-"$UBOOT_VERSION" && \
> +make -j$(nproc) V=1 vexpress_fvp_defconfig && \
> +make -j$(nproc) V=1 all && \
> +cd .. && \
> +git clone --branch "$TFA_VERSION" --depth 1 
> https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git && \
> +cd trusted-firmware-a && \
> +curl -fsSLO 
> https://git.yoctoproject.org/meta-arm/plain/meta-arm-bsp/recipes-bsp/trusted-firmware-a/files/fvp-base/0001-fdts-fvp-base-Add-stdout-path-and-virtio-net-and-rng.patch
>  \
> + --output 
> 0001-fdts-fvp-base-Add-stdout-path-and-virtio-net-and-rng.patch && \
> +git config --global user.email "y...@example.com" && \
> +git config --global user.name "Your Name" && \
> +git am 0001-fdts-fvp-base-Add-stdout-path-and-virtio-net-and-rng.patch 
> && \
> +make -j$(nproc) DEBUG=1 PLAT=fvp ARCH=aarch64 
> FVP_DT_PREFIX=fvp-base-gicv3-psci-1t all && \
> +make -j$(nproc) DEBUG=1 PLAT=fvp ARCH=aarch64 
> FVP_DT_PREFIX=fvp-base-gicv3-psci-1t fip 
> BL33=../u-boot-"$UBOOT_VERSION"/u-boot.bin && \
> +cp build/fvp/debug/bl1.bin / && \
> +cp build/fvp/debug/fip.bin / && \
> +cp build/fvp/debug/fdts/fvp-base-gicv3-psci-1t.dtb / && \
> +cd /build && \
> +rm -rf u-boot-"$UBOOT_VERSION" trusted-firmware-a
> -- 
> 2.25.1
>

Re: [PATCH 3/5] automation: Add the expect script with test case for FVP

2023-12-07 Thread Stefano Stabellini

On Thu, 7 Dec 2023, Henry Wang wrote:
> To interact with the FVP (for example entering the U-Boot shell
> and transferring the files by TFTP), we need to connect the
> corresponding port by the telnet first. Use an expect script to
> simplify the automation of the whole "interacting with FVP" stuff.
> 
> The expect script will firstly detect the IP of the host, then
> connect to the telnet port of the FVP, set the `serverip` and `ipaddr`
> for the TFTP service in the U-Boot shell, and finally boot Xen from
> U-Boot and wait for the expected results by Xen, Dom0 and DomU.

I am not an expert in "expect" but this script looks great


> Signed-off-by: Henry Wang 
> ---
>  .../expect/fvp-base-smoke-dom0-arm64.exp  | 73 +++
>  1 file changed, 73 insertions(+)
>  create mode 100755 automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp
> 
> diff --git a/automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp 
> b/automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp
> new file mode 100755
> index 00..25d9a5f81c
> --- /dev/null
> +++ b/automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp
> @@ -0,0 +1,73 @@
> +#!/usr/bin/expect
> +
> +set timeout 2000
> +
> +# Command to use to run must be given as first argument
> +# if options are required, quotes must be used:
> +# xxx.exp "cmd opt1 opt2"
> +set runcmd [lindex $argv 0]
> +
> +# Maximum number of line to be printed, this can be used to prevent runs to
> +# go forever on errors when Xen is rebooting
> +set maxline 1000
> +
> +# Configure slow parameters used with send -s
> +# This allows us to slow down console writes to prevent issues with slow
> +# emulators or targets.
> +# Format here is: {NUM TIME} which reads as wait TIME seconds each NUM of
> +# characters, here we send 1 char each 100ms
> +set send_slow {1 .1}
> +
> +proc test_boot {{maxline} {host_ip}} {
> +expect_after {
> +-re "(.*)\r" {
> +if {$maxline != 0} {
> +set maxline [expr {$maxline - 1}]
> +if {$maxline <= 0} {
> +send_user "ERROR-Toomuch!\n"
> +exit 2
> +}
> +}
> +exp_continue
> +}
> +timeout {send_user "ERROR-Timeout!\n"; exit 3}
> +eof {send_user "ERROR-EOF!\n"; exit 4}
> +}

Why do we need this "expect_after" ?


> +# Extract the telnet port numbers from FVP output, because the telnet 
> ports
> +# are not guaranteed to be fixed numbers.
> +expect -re {terminal_0: Listening for serial connection on port [0-9]+}
> +set terminal_0 $expect_out(0,string)
> +if {[regexp {port (\d+)} $terminal_0 match port_0]} {
> +puts "terminal_0 port is $port_0"
> +} else {
> +puts "terminal_0 port not found"
> +exit 5
> +}
> +
> +spawn bash -c "telnet localhost $port_0"
> +expect -re "Hit any key to stop autoboot.*"
> +send -s "  \r"
> +send -s "setenv serverip $host_ip; setenv ipaddr $host_ip; tftpb 
> 0x8020 boot.scr; source 0x8020\r"
> +
> +# Initial Xen boot
> +expect -re "\(XEN\).*Freed .* init memory."
> +
> +# Dom0 and DomU
> +expect -re "Domain-0.*"
> +expect -re "BusyBox.*"
> +expect -re "/ #.*"

This is clear, excellent


> +}
> +
> +# Get host IP
> +spawn bash -c "hostname -I | awk '{print \$1}'"
> +expect -re {(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})}

Why d{1,3}?


> +set host_ip $expect_out(0,string)
> +
> +# Start the FVP and run the test
> +spawn bash -c "$runcmd"
> +
> +test_boot 2000 "$host_ip"
> +
> +send_user "\nExecution with SUCCESS\n"

Won't this always return SUCCESS even in case of failure?


> +exit 0
> -- 
> 2.25.1
>

Re: [PATCH 4/5] automation: Add the script for the FVP smoke test

2023-12-07 Thread Stefano Stabellini

On Thu, 7 Dec 2023, Henry Wang wrote:
> This commit adds the shell script for the FVP smoke test. Similarly
> as the QEMU jobs, the shell script will firstly prepare the DomU
> BusyBox image, use the ImageBuilder to arrange the binaries in memory
> and generate the U-Boot script, then start the test. To provide the
> TFTP service for the FVP, the shell script will also start the TFTP
> service, and copy the binaries needed by test to the TFTP directory
> used by the TFTP server.
> 
> Signed-off-by: Henry Wang 
> ---
>  .../scripts/fvp-base-smoke-dom0-arm64.sh  | 117 ++
>  1 file changed, 117 insertions(+)
>  create mode 100755 automation/scripts/fvp-base-smoke-dom0-arm64.sh
> 
> diff --git a/automation/scripts/fvp-base-smoke-dom0-arm64.sh 
> b/automation/scripts/fvp-base-smoke-dom0-arm64.sh
> new file mode 100755
> index 00..716a63b0a8
> --- /dev/null
> +++ b/automation/scripts/fvp-base-smoke-dom0-arm64.sh
> @@ -0,0 +1,117 @@
> +#!/bin/bash
> +
> +set -ex
> +
> +# DomU Busybox
> +cd binaries
> +mkdir -p initrd
> +mkdir -p initrd/bin
> +mkdir -p initrd/sbin
> +mkdir -p initrd/etc
> +mkdir -p initrd/dev
> +mkdir -p initrd/proc
> +mkdir -p initrd/sys
> +mkdir -p initrd/lib
> +mkdir -p initrd/var
> +mkdir -p initrd/mnt
> +cp /bin/busybox initrd/bin/busybox
> +initrd/bin/busybox --install initrd/bin
> +echo "#!/bin/sh
> +
> +mount -t proc proc /proc
> +mount -t sysfs sysfs /sys
> +mount -t devtmpfs devtmpfs /dev
> +/bin/sh" > initrd/init
> +chmod +x initrd/init
> +cd initrd
> +find . | cpio --create --format='newc' | gzip > ../initrd.cpio.gz
> +cd ..
> +
> +mkdir -p rootfs
> +cd rootfs
> +tar xvzf ../initrd.tar.gz
> +mkdir proc
> +mkdir run
> +mkdir srv
> +mkdir sys
> +rm var/run
> +cp -ar ../dist/install/* .
> +mv ../initrd.cpio.gz ./root
> +cp ../Image ./root
> +echo "name=\"test\"
> +memory=512
> +vcpus=1
> +kernel=\"/root/Image\"
> +ramdisk=\"/root/initrd.cpio.gz\"
> +extra=\"console=hvc0 root=/dev/ram0 rdinit=/bin/sh\"
> +" > root/test.cfg
> +echo "#!/bin/bash
> +
> +export LD_LIBRARY_PATH=/usr/local/lib
> +bash /etc/init.d/xencommons start
> +
> +xl list
> +
> +xl create -c /root/test.cfg
> +
> +" > etc/local.d/xen.start
> +chmod +x etc/local.d/xen.start
> +echo "rc_verbose=yes" >> etc/rc.conf
> +find . |cpio -H newc -o|gzip > ../xen-rootfs.cpio.gz
> +cd ../..
> +
> +# Start a TFTP server to provide TFTP service to FVP
> +service tftpd-hpa start
> +
> +# ImageBuilder
> +echo 'MEMORY_START="0x8000"
> +MEMORY_END="0xFF00"
> +
> +DEVICE_TREE="fvp-base-gicv3-psci-1t.dtb"
> +XEN="xen"
> +DOM0_KERNEL="Image"
> +DOM0_RAMDISK="xen-rootfs.cpio.gz"
> +XEN_CMD="console=dtuart dom0_mem=1024M console_timestamps=boot"
> +
> +NUM_DOMUS=0
> +
> +LOAD_CMD="tftpb"
> +UBOOT_SOURCE="boot.source"
> +UBOOT_SCRIPT="boot.scr"' > binaries/config
> +rm -rf imagebuilder
> +git clone https://gitlab.com/ViryaOS/imagebuilder
> +bash imagebuilder/scripts/uboot-script-gen -t tftp -d binaries/ -c 
> binaries/config
> +
> +# Copy files to the TFTP directory to use
> +cp ./binaries/boot.scr /srv/tftp/
> +cp ./binaries/Image /srv/tftp/
> +cp ./binaries/xen-rootfs.cpio.gz /srv/tftp/
> +cp ./binaries/xen /srv/tftp/
> +cp ./binaries/fvp-base-gicv3-psci-1t.dtb /srv/tftp/
> +
> +# Start FVP
> +TERM0_CFG="-C bp.terminal_0.mode=telnet -C bp.terminal_0.start_telnet=0"
> +TERM1_CFG="-C bp.terminal_1.mode=telnet -C bp.terminal_1.start_telnet=0"
> +TERM2_CFG="-C bp.terminal_2.mode=telnet -C bp.terminal_2.start_telnet=0"
> +TERM3_CFG="-C bp.terminal_3.mode=telnet -C bp.terminal_3.start_telnet=0"
> +
> +VIRTIO_USER_NETWORK_CFG="-C bp.virtio_net.enabled=1 \
> +-C bp.virtio_net.hostbridge.userNetworking=1 \
> +-C bp.virtio_net.hostbridge.userNetPorts=8022=22 \
> +-C bp.virtio_net.transport=legacy"
> +
> +./automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp \
> +
> "/FVP/FVP_Base_RevC-2xAEMvA/Base_RevC_AEMvA_pkg/models/Linux64_armv8l_GCC-9.3/FVP_Base_RevC-2xAEMvA
>  \
> +-C bp.vis.disable_visualisation=1 \
> +-C bp.ve_sysregs.exit_on_shutdown=1 \
> +-C bp.secure_memory=0 \
> +-C cache_state_modelled=0 \
> +-C cluster0.has_arm_v8-4=1 \
> +-C cluster1.has_arm_v8-4=1 \
> +${TERM0_CFG} ${TERM1_CFG} ${TERM2_CFG} ${TERM3_CFG} \
> +${VIRTIO_USER_NETWORK_CFG} \
> +-C bp.secureflashloader.fname=$(pwd)/binaries/bl1.bin \
> +-C bp.flashloader0.fname=$(pwd)/binaries/fip.bin" |& \
> +tee smoke.serial

This script is clear and it is fine:

Reviewed-by: Stefano Stabellini 

My only concern is that the expect checks on what booted (Xen, Dom0,
DomU) are inside the script fvp-base-smoke-dom0-arm64.exp rather than in
this script. So if in the future we are going to have multiple tests
with different configurations (for instance see
qemu-smoke-dom0less-arm64.sh) we'll have to find a way to reuse
fvp-base-smoke-dom0-arm64.exp somehow.

Re: [PATCH 5/5] automation: Add the arm64 FVP build and Dom0 smoke test jobs

2023-12-07 Thread Stefano Stabellini

On Thu, 7 Dec 2023, Henry Wang wrote:
> Add a job in the build stage to export the TF-A, U-Boot and the
> device tree for the FVP platform from the test artifact container.
> 
> Add a FVP smoke test job in the test stage to do the same test as
> the `qemu-smoke-dom0-arm64-gcc` job.
> 
> Signed-off-by: Henry Wang 

Reviewed-by: Stefano Stabellini 


> ---
> Although it does not affect the functionality, I am still quite
> confused about why the logs displayed by GitLab UI, for
> example [1], is much less than the actual output (saved in the
> artifacts, see [2]). Had a discussion with Michal on Matrix
> and we agree that the log in gitlab UI is usually capped.
> 
> [1] https://gitlab.com/xen-project/people/henryw/xen/-/jobs/5700569676
> [2] 
> https://gitlab.com/xen-project/people/henryw/xen/-/jobs/5700569676/artifacts/file/smoke.serial
> ---
>  automation/gitlab-ci/build.yaml | 17 +
>  automation/gitlab-ci/test.yaml  | 22 ++
>  2 files changed, 39 insertions(+)
> 
> diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
> index 32af30cced..89d2f01302 100644
> --- a/automation/gitlab-ci/build.yaml
> +++ b/automation/gitlab-ci/build.yaml
> @@ -344,6 +344,23 @@ kernel-6.1.19-export:
>tags:
>  - x86_64
>  
> +armfvp-uboot-tfa-2023.10-2.9.0-arm64-export:
> +  extends: .test-jobs-artifact-common
> +  image: 
> registry.gitlab.com/xen-project/xen/tests-artifacts/armfvp-uboot-tfa:2023.10-2.9.0-arm64v8
> +  script:
> +- |
> +   mkdir binaries && \
> +   cp /bl1.bin binaries/bl1.bin && \
> +   cp /fip.bin binaries/fip.bin && \
> +   cp /fvp-base-gicv3-psci-1t.dtb binaries/fvp-base-gicv3-psci-1t.dtb
> +  artifacts:
> +paths:
> +  - binaries/bl1.bin
> +  - binaries/fip.bin
> +  - binaries/fvp-base-gicv3-psci-1t.dtb
> +  tags:
> +- arm64
> +
>  # Jobs below this line
>  
>  # Build jobs needed for tests
> diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
> index 6aabdb9d15..47e00d0a0b 100644
> --- a/automation/gitlab-ci/test.yaml
> +++ b/automation/gitlab-ci/test.yaml
> @@ -96,6 +96,19 @@
>tags:
>  - xilinx
>  
> +.fvp-arm64:
> +  extends: .test-jobs-common
> +  variables:
> +CONTAINER: debian:bookworm-arm64v8-fvp
> +LOGFILE: fvp-smoke-arm64.log
> +  artifacts:
> +paths:
> +  - smoke.serial
> +  - '*.log'
> +when: always
> +  tags:
> +- arm64
> +
>  .adl-x86-64:
>extends: .test-jobs-common
>variables:
> @@ -459,3 +472,12 @@ qemu-smoke-ppc64le-powernv9-gcc:
>needs:
>  - qemu-system-ppc64-8.1.0-ppc64-export
>  - debian-bullseye-gcc-ppc64le-debug
> +
> +fvp-smoke-dom0-arm64-gcc-debug:
> +  extends: .fvp-arm64
> +  script:
> +- ./automation/scripts/fvp-base-smoke-dom0-arm64.sh 2>&1 | tee ${LOGFILE}
> +  needs:
> +- *arm64-test-needs
> +- armfvp-uboot-tfa-2023.10-2.9.0-arm64-export
> +- alpine-3.18-gcc-debug-arm64
> -- 
> 2.25.1
>

Re: [PATCH 1/5] automation: Add a Dockerfile for running FVP_Base jobs

2023-12-07 Thread Stefano Stabellini

On Thu, 7 Dec 2023, Stefano Stabellini wrote:
> On Thu, 7 Dec 2023, Henry Wang wrote:
> > Fixed Virtual Platforms (FVPs) are complete simulations of an Arm
> > system, including processor, memory and peripherals. These are set
> > out in a "programmer's view", which gives programmers a comprehensive
> > model on which to build and test software. FVP can be configured to
> > different setups by its cmdline parameters, and hence having the FVP
> > in CI will provide us with the flexibility to test Arm features and
> > setups that we find difficult to use real hardware or emulators.
> > 
> > This commit adds a Dockerfile for the new arm64v8 container with
> > FVP installed, based on the debian bookworm-arm64v8 image. This
> > container will be used to run the FVP test jobs. Compared to the
> > debian bookworm-arm64v8 image, the packages in the newly added FVP
> > container does not contain the `u-boot-qemu`, and adds the `expect`
> > to run expect scripts introduced by following commits, `telnet` to
> > connect to FVP, and `tftpd-hpa` to provide the TFTP service for
> > the FVP.
> > 
> > Signed-off-by: Henry Wang 
> > ---
> >  .../debian/bookworm-arm64v8-fvp.dockerfile| 64 +++
> >  1 file changed, 64 insertions(+)
> >  create mode 100644 automation/build/debian/bookworm-arm64v8-fvp.dockerfile
> > 
> > diff --git a/automation/build/debian/bookworm-arm64v8-fvp.dockerfile 
> > b/automation/build/debian/bookworm-arm64v8-fvp.dockerfile
> > new file mode 100644
> > index 00..3b87dc5a5b
> > --- /dev/null
> > +++ b/automation/build/debian/bookworm-arm64v8-fvp.dockerfile
> 
> Please move this container under automation/tests-artifacts/ because the
> container is only meant to be used for tests as opposed as to build Xen.
> I know that in other cases we have reused the build container but that
> just because it was already there an working for the purpose.
> 
> Also please name it based on the fvp rather than debian:
> 
> automation/tests-artifacts/armfvp/11.23_9-arm64v8.dockerfile
> 
> Everything else looks fine.

I take it back. We even have
automation/build/ubuntu/xenial-xilinx.dockerfile and
automation/build/debian/bookworm-cppcheck.dockerfile

At one point I think we should separate the build containers from the
ones we use for testing but I won't ask it here.

Reviewed-by: Stefano Stabellini

Re: [PATCH 4/5] automation: Add the script for the FVP smoke test

2023-12-07 Thread Henry Wang

Hi Stefano,

> On Dec 8, 2023, at 09:41, Stefano Stabellini  wrote:
> 
> On Thu, 7 Dec 2023, Henry Wang wrote:
>> This commit adds the shell script for the FVP smoke test. Similarly
>> as the QEMU jobs, the shell script will firstly prepare the DomU
>> BusyBox image, use the ImageBuilder to arrange the binaries in memory
>> and generate the U-Boot script, then start the test. To provide the
>> TFTP service for the FVP, the shell script will also start the TFTP
>> service, and copy the binaries needed by test to the TFTP directory
>> used by the TFTP server.
>> 
>> Signed-off-by: Henry Wang 
>> ---
>> .../scripts/fvp-base-smoke-dom0-arm64.sh  | 117 ++
>> 1 file changed, 117 insertions(+)
>> create mode 100755 automation/scripts/fvp-base-smoke-dom0-arm64.sh
> 
> This script is clear and it is fine:
> 
> Reviewed-by: Stefano Stabellini 

Thanks!

> 
> My only concern is that the expect checks on what booted (Xen, Dom0,
> DomU) are inside the script fvp-base-smoke-dom0-arm64.exp rather than in
> this script. So if in the future we are going to have multiple tests
> with different configurations (for instance see
> qemu-smoke-dom0less-arm64.sh) we'll have to find a way to reuse
> fvp-base-smoke-dom0-arm64.exp somehow.

We do have ways to reuse expect script, for example, we can extract the common
code (variables and test logic) to the common.tcl file and source the 
common.tcl file
in the expect script to reuse the code. The reason why I didn’t do that in this 
series
is that currently there is only one script so I feel that there is not much 
benefit to do
this instantly :) Let’s wait to see if there is more comments from others, and 
I am
definitely open to refactor if there is the need to extract the common logic 
(for example
when we add dom0less tests in the future).

Kind regards,
Henry

Re: [PATCH 3/5] automation: Add the expect script with test case for FVP

2023-12-07 Thread Henry Wang

Hi Stefano,

> On Dec 8, 2023, at 09:38, Stefano Stabellini  wrote:
> 
> On Thu, 7 Dec 2023, Henry Wang wrote:
>> To interact with the FVP (for example entering the U-Boot shell
>> and transferring the files by TFTP), we need to connect the
>> corresponding port by the telnet first. Use an expect script to
>> simplify the automation of the whole "interacting with FVP" stuff.
>> 
>> The expect script will firstly detect the IP of the host, then
>> connect to the telnet port of the FVP, set the `serverip` and `ipaddr`
>> for the TFTP service in the U-Boot shell, and finally boot Xen from
>> U-Boot and wait for the expected results by Xen, Dom0 and DomU.
> 
> I am not an expert in "expect" but this script looks great

:)

> 
> 
>> Signed-off-by: Henry Wang 
>> ---
>> .../expect/fvp-base-smoke-dom0-arm64.exp  | 73 +++
>> 1 file changed, 73 insertions(+)
>> create mode 100755 automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp
>> 
>> +
>> +proc test_boot {{maxline} {host_ip}} {
>> +expect_after {
>> +-re "(.*)\r" {
>> +if {$maxline != 0} {
>> +set maxline [expr {$maxline - 1}]
>> +if {$maxline <= 0} {
>> +send_user "ERROR-Toomuch!\n"
>> +exit 2
>> +}
>> +}
>> +exp_continue
>> +}
>> +timeout {send_user "ERROR-Timeout!\n"; exit 3}
>> +eof {send_user "ERROR-EOF!\n"; exit 4}
>> +}
> 
> Why do we need this "expect_after" ?

It is basically for the error handling. Either there is too many characters 
triggered
by some misconfiguration of Xen/Kernel leading to Xen/Kernel constantly reboots,
or the code stuck somehow, we have a way to stop the script instead of waiting 
in
the infinity loop.

>> +# Extract the telnet port numbers from FVP output, because the telnet 
>> ports
>> +# are not guaranteed to be fixed numbers.
>> +expect -re {terminal_0: Listening for serial connection on port [0-9]+}
>> +set terminal_0 $expect_out(0,string)
>> +if {[regexp {port (\d+)} $terminal_0 match port_0]} {
>> +puts "terminal_0 port is $port_0"
>> +} else {
>> +puts "terminal_0 port not found"
>> +exit 5
>> +}
>> +
>> +spawn bash -c "telnet localhost $port_0"
>> +expect -re "Hit any key to stop autoboot.*"
>> +send -s "  \r"
>> +send -s "setenv serverip $host_ip; setenv ipaddr $host_ip; tftpb 
>> 0x8020 boot.scr; source 0x8020\r"
>> +
>> +# Initial Xen boot
>> +expect -re "\(XEN\).*Freed .* init memory."
>> +
>> +# Dom0 and DomU
>> +expect -re "Domain-0.*"
>> +expect -re "BusyBox.*"
>> +expect -re "/ #.*"
> 
> This is clear, excellent
> 
>> +}
>> +
>> +# Get host IP
>> +spawn bash -c "hostname -I | awk '{print \$1}'"
>> +expect -re {(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})}
> 
> Why d{1,3}?

I think that is because the IP pattern, we at least have one digit and at most 
3.

>> +set host_ip $expect_out(0,string)
>> +
>> +# Start the FVP and run the test
>> +spawn bash -c "$runcmd"
>> +
>> +test_boot 2000 "$host_ip"
>> +
>> +send_user "\nExecution with SUCCESS\n"
> 
> Won't this always return SUCCESS even in case of failure?

IMHO, if things fails, we have various exit code (1-5) for each failure case. 
For example,
if the FVP port somehow cannot be found, we exit with error code 5. This will 
basically lead
us to the only case where the failure is caused by the script fails to wait for 
the expected
string/regexp, and this case we have the timeout failure triggered by my 
above-mentioned
expect_after block.

Kind regards,
Henry

>> +exit 0
>> -- 
>> 2.25.1
>>

Re: [PATCH 3/5] automation: Add the expect script with test case for FVP

2023-12-07 Thread Henry Wang

Hi Stefano,

> On Dec 8, 2023, at 10:03, Henry Wang  wrote:
> 
> Hi Stefano,
> 
>> On Dec 8, 2023, at 09:38, Stefano Stabellini  wrote:
>>> +set host_ip $expect_out(0,string)
>>> +
>>> +# Start the FVP and run the test
>>> +spawn bash -c "$runcmd"
>>> +
>>> +test_boot 2000 "$host_ip"
>>> +
>>> +send_user "\nExecution with SUCCESS\n"
>> 
>> Won't this always return SUCCESS even in case of failure?
> 
> IMHO, if things fails, we have various exit code (1-5) for each failure case. 
> For example,
> if the FVP port somehow cannot be found, we exit with error code 5. This will 
> basically lead
> us to the only case where the failure is caused by the script fails to wait 
> for the expected
> string/regexp, and this case we have the timeout failure triggered by my 
> above-mentioned
> expect_after block.

I did a test to see if I break the expect script by adding below hunk:
```
--- a/automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp
+++ b/automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp
@@ -51,6 +51,7 @@ proc test_boot {{maxline} {host_ip}} {
 send -s "setenv serverip $host_ip; setenv ipaddr $host_ip; tftpb 
0x8020 boot.scr; source 0x8020\r"

 # Initial Xen boot
+expect -re "this is a hack to break the build"
 expect -re "\(XEN\).*Freed .* init memory."

 # Dom0 and DomU
```
The timeout did happen in the expect script after the set timeout, see [1]

However the job still passes, and I believe this is caused by the shell script:
```
./automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp \

"/FVP/FVP_Base_RevC-2xAEMvA/Base_RevC_AEMvA_pkg/models/Linux64_armv8l_GCC-9.3/FVP_Base_RevC-2xAEMvA
 \
-C bp.vis.disable_visualisation=1 \
-C bp.ve_sysregs.exit_on_shutdown=1 \
-C bp.secure_memory=0 \
-C cache_state_modelled=0 \
-C cluster0.has_arm_v8-4=1 \
-C cluster1.has_arm_v8-4=1 \
${TERM0_CFG} ${TERM1_CFG} ${TERM2_CFG} ${TERM3_CFG} \
${VIRTIO_USER_NETWORK_CFG} \
-C bp.secureflashloader.fname=$(pwd)/binaries/bl1.bin \
-C bp.flashloader0.fname=$(pwd)/binaries/fip.bin" |& \
tee smoke.serial

exit 0
```

The “|& tee smoke.serial” hides the error propagated by the expect script. I 
will send a v2 to fix it.

[1] 
https://gitlab.com/xen-project/people/henryw/xen/-/jobs/5708263782/artifacts/file/smoke.serial

Kind regards,
Henry

> 
> Kind regards,
> Henry
> 
>>> +exit 0
>>> -- 
>>> 2.25.1
>>> 
>

[PATCH v2 0/5] automation: Support running FVP Dom0 smoke test for Arm

2023-12-07 Thread Henry Wang

This series adds the support for running FVP Dom0 smoke test for Arm on
the Arm64 GitLab CI runner. Detailed changes please refer to the commit
message of each commit.

An example test pipeline with these patches applied (with the docker
registry changed to my own registry and unrelated job removed) can be
found at:
https://gitlab.com/xen-project/people/henryw/xen/-/pipelines/1099757245

The second example of a negative test with breaking the expect script
by a "never met" condition can be found at:
https://gitlab.com/xen-project/people/henryw/xen/-/pipelines/1099757601
The job will fail as expected after the timeout set by the expect
script.

Henry Wang (5):
  automation: Add a Dockerfile for running FVP_Base jobs
  automation: Add the Dockerfile to build TF-A and U-Boot for FVP
  automation: Add the expect script with test case for FVP
  automation: Add the script for the FVP smoke test
  automation: Add the arm64 FVP build and Dom0 smoke test jobs

 .../debian/bookworm-arm64v8-fvp.dockerfile|  64 ++
 automation/gitlab-ci/build.yaml   |  17 +++
 automation/gitlab-ci/test.yaml|  22 
 .../expect/fvp-base-smoke-dom0-arm64.exp  |  73 +++
 .../scripts/fvp-base-smoke-dom0-arm64.sh  | 120 ++
 .../2023.10-2.9.0-arm64v8.dockerfile  |  48 +++
 6 files changed, 344 insertions(+)
 create mode 100644 automation/build/debian/bookworm-arm64v8-fvp.dockerfile
 create mode 100755 automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp
 create mode 100755 automation/scripts/fvp-base-smoke-dom0-arm64.sh
 create mode 100644 
automation/tests-artifacts/armfvp-uboot-tfa/2023.10-2.9.0-arm64v8.dockerfile

-- 
2.25.1

[PATCH v2 2/5] automation: Add the Dockerfile to build TF-A and U-Boot for FVP

2023-12-07 Thread Henry Wang

Unlike the emulators that currently being used in the CI pipelines,
the FVP must start at EL3. Therefore we need the firmware, i.e. the
TrustedFirmware-A (TF-A), for corresponding functionality.

There is a dedicated board (vexpress_fvp) in U-Boot (serve as the
BL33 of the TF-A) for the FVP platform, so the U-Boot should also be
compiled for the FVP platform instead of reusing the U-Boot for the
existing emulators used in the CI pipelines.

To avoid compiling TF-A and U-Boot everytime in the job, adding a
Dockerfile to the test artifacts to build TF-A v2.9.0 and U-Boot
v2023.10 for FVP. The binaries for the TF-A and U-Boot, as well as
the device tree for the FVP platform, will be saved (and exported by
the CI job introduced by following commits). Note that, a patch for
the TF-A will be applied before building to enable the virtio-net
and the virtio-rng device on the FVP. The virtio-net device will
provide the networking service for FVP, and the virtio-rng device
will improve the speed of the FVP.

Signed-off-by: Henry Wang 
Reviewed-by: Stefano Stabellini 
---
v2:
- Add Stefano's Reviewed-by tag.
---
 .../2023.10-2.9.0-arm64v8.dockerfile  | 48 +++
 1 file changed, 48 insertions(+)
 create mode 100644 
automation/tests-artifacts/armfvp-uboot-tfa/2023.10-2.9.0-arm64v8.dockerfile

diff --git 
a/automation/tests-artifacts/armfvp-uboot-tfa/2023.10-2.9.0-arm64v8.dockerfile 
b/automation/tests-artifacts/armfvp-uboot-tfa/2023.10-2.9.0-arm64v8.dockerfile
new file mode 100644
index 00..6566b60545
--- /dev/null
+++ 
b/automation/tests-artifacts/armfvp-uboot-tfa/2023.10-2.9.0-arm64v8.dockerfile
@@ -0,0 +1,48 @@
+FROM --platform=linux/arm64/v8 debian:bookworm
+LABEL maintainer.name="The Xen Project" \
+  maintainer.email="xen-devel@lists.xenproject.org"
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV UBOOT_VERSION="2023.10"
+ENV TFA_VERSION="v2.9.0"
+ENV USER root
+
+RUN mkdir /build
+WORKDIR /build
+
+# build depends
+RUN apt-get update && \
+apt-get --quiet --yes install \
+build-essential \
+libssl-dev \
+bc \
+curl \
+flex \
+bison \
+git \
+device-tree-compiler && \
+apt-get autoremove -y && \
+apt-get clean && \
+rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
+
+# Build U-Boot and TF-A
+RUN curl -fsSLO https://ftp.denx.de/pub/u-boot/u-boot-"$UBOOT_VERSION".tar.bz2 
&& \
+tar xvf u-boot-"$UBOOT_VERSION".tar.bz2 && \
+cd u-boot-"$UBOOT_VERSION" && \
+make -j$(nproc) V=1 vexpress_fvp_defconfig && \
+make -j$(nproc) V=1 all && \
+cd .. && \
+git clone --branch "$TFA_VERSION" --depth 1 
https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git && \
+cd trusted-firmware-a && \
+curl -fsSLO 
https://git.yoctoproject.org/meta-arm/plain/meta-arm-bsp/recipes-bsp/trusted-firmware-a/files/fvp-base/0001-fdts-fvp-base-Add-stdout-path-and-virtio-net-and-rng.patch
 \
+ --output 
0001-fdts-fvp-base-Add-stdout-path-and-virtio-net-and-rng.patch && \
+git config --global user.email "y...@example.com" && \
+git config --global user.name "Your Name" && \
+git am 0001-fdts-fvp-base-Add-stdout-path-and-virtio-net-and-rng.patch && \
+make -j$(nproc) DEBUG=1 PLAT=fvp ARCH=aarch64 
FVP_DT_PREFIX=fvp-base-gicv3-psci-1t all && \
+make -j$(nproc) DEBUG=1 PLAT=fvp ARCH=aarch64 
FVP_DT_PREFIX=fvp-base-gicv3-psci-1t fip 
BL33=../u-boot-"$UBOOT_VERSION"/u-boot.bin && \
+cp build/fvp/debug/bl1.bin / && \
+cp build/fvp/debug/fip.bin / && \
+cp build/fvp/debug/fdts/fvp-base-gicv3-psci-1t.dtb / && \
+cd /build && \
+rm -rf u-boot-"$UBOOT_VERSION" trusted-firmware-a
-- 
2.25.1

[PATCH v2 4/5] automation: Add the script for the FVP smoke test

2023-12-07 Thread Henry Wang

This commit adds the shell script for the FVP smoke test. Similarly
as the QEMU jobs, the shell script will firstly prepare the DomU
BusyBox image, use the ImageBuilder to arrange the binaries in memory
and generate the U-Boot script, then start the test. To provide the
TFTP service for the FVP, the shell script will also start the TFTP
service, and copy the binaries needed by test to the TFTP directory
used by the TFTP server.

Signed-off-by: Henry Wang 
---
v2:
- Set pipefail before running the expect script, so that the error
  won't be hid by pipe and the tee to the logfile.
---
 .../scripts/fvp-base-smoke-dom0-arm64.sh  | 120 ++
 1 file changed, 120 insertions(+)
 create mode 100755 automation/scripts/fvp-base-smoke-dom0-arm64.sh

diff --git a/automation/scripts/fvp-base-smoke-dom0-arm64.sh 
b/automation/scripts/fvp-base-smoke-dom0-arm64.sh
new file mode 100755
index 00..99097dad51
--- /dev/null
+++ b/automation/scripts/fvp-base-smoke-dom0-arm64.sh
@@ -0,0 +1,120 @@
+#!/bin/bash
+
+set -ex
+
+# DomU Busybox
+cd binaries
+mkdir -p initrd
+mkdir -p initrd/bin
+mkdir -p initrd/sbin
+mkdir -p initrd/etc
+mkdir -p initrd/dev
+mkdir -p initrd/proc
+mkdir -p initrd/sys
+mkdir -p initrd/lib
+mkdir -p initrd/var
+mkdir -p initrd/mnt
+cp /bin/busybox initrd/bin/busybox
+initrd/bin/busybox --install initrd/bin
+echo "#!/bin/sh
+
+mount -t proc proc /proc
+mount -t sysfs sysfs /sys
+mount -t devtmpfs devtmpfs /dev
+/bin/sh" > initrd/init
+chmod +x initrd/init
+cd initrd
+find . | cpio --create --format='newc' | gzip > ../initrd.cpio.gz
+cd ..
+
+mkdir -p rootfs
+cd rootfs
+tar xvzf ../initrd.tar.gz
+mkdir proc
+mkdir run
+mkdir srv
+mkdir sys
+rm var/run
+cp -ar ../dist/install/* .
+mv ../initrd.cpio.gz ./root
+cp ../Image ./root
+echo "name=\"test\"
+memory=512
+vcpus=1
+kernel=\"/root/Image\"
+ramdisk=\"/root/initrd.cpio.gz\"
+extra=\"console=hvc0 root=/dev/ram0 rdinit=/bin/sh\"
+" > root/test.cfg
+echo "#!/bin/bash
+
+export LD_LIBRARY_PATH=/usr/local/lib
+bash /etc/init.d/xencommons start
+
+xl list
+
+xl create -c /root/test.cfg
+
+" > etc/local.d/xen.start
+chmod +x etc/local.d/xen.start
+echo "rc_verbose=yes" >> etc/rc.conf
+find . |cpio -H newc -o|gzip > ../xen-rootfs.cpio.gz
+cd ../..
+
+# Start a TFTP server to provide TFTP service to FVP
+service tftpd-hpa start
+
+# ImageBuilder
+echo 'MEMORY_START="0x8000"
+MEMORY_END="0xFF00"
+
+DEVICE_TREE="fvp-base-gicv3-psci-1t.dtb"
+XEN="xen"
+DOM0_KERNEL="Image"
+DOM0_RAMDISK="xen-rootfs.cpio.gz"
+XEN_CMD="console=dtuart dom0_mem=1024M console_timestamps=boot"
+
+NUM_DOMUS=0
+
+LOAD_CMD="tftpb"
+UBOOT_SOURCE="boot.source"
+UBOOT_SCRIPT="boot.scr"' > binaries/config
+rm -rf imagebuilder
+git clone https://gitlab.com/ViryaOS/imagebuilder
+bash imagebuilder/scripts/uboot-script-gen -t tftp -d binaries/ -c 
binaries/config
+
+# Copy files to the TFTP directory to use
+cp ./binaries/boot.scr /srv/tftp/
+cp ./binaries/Image /srv/tftp/
+cp ./binaries/xen-rootfs.cpio.gz /srv/tftp/
+cp ./binaries/xen /srv/tftp/
+cp ./binaries/fvp-base-gicv3-psci-1t.dtb /srv/tftp/
+
+# Start FVP
+TERM0_CFG="-C bp.terminal_0.mode=telnet -C bp.terminal_0.start_telnet=0"
+TERM1_CFG="-C bp.terminal_1.mode=telnet -C bp.terminal_1.start_telnet=0"
+TERM2_CFG="-C bp.terminal_2.mode=telnet -C bp.terminal_2.start_telnet=0"
+TERM3_CFG="-C bp.terminal_3.mode=telnet -C bp.terminal_3.start_telnet=0"
+
+VIRTIO_USER_NETWORK_CFG="-C bp.virtio_net.enabled=1 \
+-C bp.virtio_net.hostbridge.userNetworking=1 \
+-C bp.virtio_net.hostbridge.userNetPorts=8022=22 \
+-C bp.virtio_net.transport=legacy"
+
+# Set the pipefail so that the error code from the expect script won't
+# be hid by pipe and the tee command.
+set -o pipefail
+./automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp \
+
"/FVP/FVP_Base_RevC-2xAEMvA/Base_RevC_AEMvA_pkg/models/Linux64_armv8l_GCC-9.3/FVP_Base_RevC-2xAEMvA
 \
+-C bp.vis.disable_visualisation=1 \
+-C bp.ve_sysregs.exit_on_shutdown=1 \
+-C bp.secure_memory=0 \
+-C cache_state_modelled=0 \
+-C cluster0.has_arm_v8-4=1 \
+-C cluster1.has_arm_v8-4=1 \
+${TERM0_CFG} ${TERM1_CFG} ${TERM2_CFG} ${TERM3_CFG} \
+${VIRTIO_USER_NETWORK_CFG} \
+-C bp.secureflashloader.fname=$(pwd)/binaries/bl1.bin \
+-C bp.flashloader0.fname=$(pwd)/binaries/fip.bin" |& \
+tee smoke.serial
+
+exit 0
-- 
2.25.1

[PATCH v2 3/5] automation: Add the expect script with test case for FVP

2023-12-07 Thread Henry Wang

To interact with the FVP (for example entering the U-Boot shell
and transferring the files by TFTP), we need to connect the
corresponding port by the telnet first. Use an expect script to
simplify the automation of the whole "interacting with FVP" stuff.

The expect script will firstly detect the IP of the host, then
connect to the telnet port of the FVP, set the `serverip` and `ipaddr`
for the TFTP service in the U-Boot shell, and finally boot Xen from
U-Boot and wait for the expected results by Xen, Dom0 and DomU.

Signed-off-by: Henry Wang 
---
v2:
- No change.
---
 .../expect/fvp-base-smoke-dom0-arm64.exp  | 73 +++
 1 file changed, 73 insertions(+)
 create mode 100755 automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp

diff --git a/automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp 
b/automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp
new file mode 100755
index 00..25d9a5f81c
--- /dev/null
+++ b/automation/scripts/expect/fvp-base-smoke-dom0-arm64.exp
@@ -0,0 +1,73 @@
+#!/usr/bin/expect
+
+set timeout 2000
+
+# Command to use to run must be given as first argument
+# if options are required, quotes must be used:
+# xxx.exp "cmd opt1 opt2"
+set runcmd [lindex $argv 0]
+
+# Maximum number of line to be printed, this can be used to prevent runs to
+# go forever on errors when Xen is rebooting
+set maxline 1000
+
+# Configure slow parameters used with send -s
+# This allows us to slow down console writes to prevent issues with slow
+# emulators or targets.
+# Format here is: {NUM TIME} which reads as wait TIME seconds each NUM of
+# characters, here we send 1 char each 100ms
+set send_slow {1 .1}
+
+proc test_boot {{maxline} {host_ip}} {
+expect_after {
+-re "(.*)\r" {
+if {$maxline != 0} {
+set maxline [expr {$maxline - 1}]
+if {$maxline <= 0} {
+send_user "ERROR-Toomuch!\n"
+exit 2
+}
+}
+exp_continue
+}
+timeout {send_user "ERROR-Timeout!\n"; exit 3}
+eof {send_user "ERROR-EOF!\n"; exit 4}
+}
+
+# Extract the telnet port numbers from FVP output, because the telnet ports
+# are not guaranteed to be fixed numbers.
+expect -re {terminal_0: Listening for serial connection on port [0-9]+}
+set terminal_0 $expect_out(0,string)
+if {[regexp {port (\d+)} $terminal_0 match port_0]} {
+puts "terminal_0 port is $port_0"
+} else {
+puts "terminal_0 port not found"
+exit 5
+}
+
+spawn bash -c "telnet localhost $port_0"
+expect -re "Hit any key to stop autoboot.*"
+send -s "  \r"
+send -s "setenv serverip $host_ip; setenv ipaddr $host_ip; tftpb 
0x8020 boot.scr; source 0x8020\r"
+
+# Initial Xen boot
+expect -re "\(XEN\).*Freed .* init memory."
+
+# Dom0 and DomU
+expect -re "Domain-0.*"
+expect -re "BusyBox.*"
+expect -re "/ #.*"
+}
+
+# Get host IP
+spawn bash -c "hostname -I | awk '{print \$1}'"
+expect -re {(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})}
+set host_ip $expect_out(0,string)
+
+# Start the FVP and run the test
+spawn bash -c "$runcmd"
+
+test_boot 2000 "$host_ip"
+
+send_user "\nExecution with SUCCESS\n"
+exit 0
-- 
2.25.1

[PATCH v2 1/5] automation: Add a Dockerfile for running FVP_Base jobs

2023-12-07 Thread Henry Wang

Fixed Virtual Platforms (FVPs) are complete simulations of an Arm
system, including processor, memory and peripherals. These are set
out in a "programmer's view", which gives programmers a comprehensive
model on which to build and test software. FVP can be configured to
different setups by its cmdline parameters, and hence having the FVP
in CI will provide us with the flexibility to test Arm features and
setups that we find difficult to use real hardware or emulators.

This commit adds a Dockerfile for the new arm64v8 container with
FVP installed, based on the debian bookworm-arm64v8 image. This
container will be used to run the FVP test jobs. Compared to the
debian bookworm-arm64v8 image, the packages in the newly added FVP
container does not contain the `u-boot-qemu`, and adds the `expect`
to run expect scripts introduced by following commits, `telnet` to
connect to FVP, and `tftpd-hpa` to provide the TFTP service for
the FVP.

Signed-off-by: Henry Wang 
Reviewed-by: Stefano Stabellini 
---
v2:
- Add Stefano's Reviewed-by tag.
---
 .../debian/bookworm-arm64v8-fvp.dockerfile| 64 +++
 1 file changed, 64 insertions(+)
 create mode 100644 automation/build/debian/bookworm-arm64v8-fvp.dockerfile

diff --git a/automation/build/debian/bookworm-arm64v8-fvp.dockerfile 
b/automation/build/debian/bookworm-arm64v8-fvp.dockerfile
new file mode 100644
index 00..3b87dc5a5b
--- /dev/null
+++ b/automation/build/debian/bookworm-arm64v8-fvp.dockerfile
@@ -0,0 +1,64 @@
+FROM --platform=linux/arm64/v8 debian:bookworm
+LABEL maintainer.name="The Xen Project" \
+  maintainer.email="xen-devel@lists.xenproject.org"
+
+ARG FVP_BASE_VERSION="11.23_9_Linux64_armv8l"
+
+ENV DEBIAN_FRONTEND=noninteractive
+ENV USER root
+
+RUN mkdir /build
+WORKDIR /build
+
+# build depends
+RUN apt-get update && \
+apt-get --quiet --yes install \
+build-essential \
+zlib1g-dev \
+libncurses5-dev \
+libssl-dev \
+python3-dev \
+python3-setuptools \
+xorg-dev \
+uuid-dev \
+libyajl-dev \
+libaio-dev \
+libglib2.0-dev \
+clang \
+libpixman-1-dev \
+pkg-config \
+flex \
+bison \
+acpica-tools \
+libfdt-dev \
+bin86 \
+bcc \
+liblzma-dev \
+libnl-3-dev \
+ocaml-nox \
+libfindlib-ocaml-dev \
+markdown \
+transfig \
+pandoc \
+checkpolicy \
+wget \
+git \
+nasm \
+# for test phase, fvp-smoke-* jobs
+u-boot-tools \
+expect \
+device-tree-compiler \
+curl \
+cpio \
+busybox-static \
+telnet \
+tftpd-hpa \
+&& \
+apt-get autoremove -y && \
+apt-get clean && \
+rm -rf /var/lib/apt/lists* /tmp/* /var/tmp/*
+
+RUN wget 
https://developer.arm.com/-/media/Files/downloads/ecosystem-models/FVP_Base_RevC-2xAEMvA_${FVP_BASE_VERSION}.tgz
 && \
+mkdir -p /FVP/FVP_Base_RevC-2xAEMvA && \
+tar -xvzf FVP_Base_RevC-2xAEMvA_${FVP_BASE_VERSION}.tgz -C 
/FVP/FVP_Base_RevC-2xAEMvA && \
+rm FVP_Base_RevC-2xAEMvA_${FVP_BASE_VERSION}.tgz
-- 
2.25.1

[PATCH v2 5/5] automation: Add the arm64 FVP build and Dom0 smoke test jobs

2023-12-07 Thread Henry Wang

Add a job in the build stage to export the TF-A, U-Boot and the
device tree for the FVP platform from the test artifact container.

Add a FVP smoke test job in the test stage to do the same test as
the `qemu-smoke-dom0-arm64-gcc` job.

Signed-off-by: Henry Wang 
Reviewed-by: Stefano Stabellini 
---
v2:
- Add Stefano's Reviewed-by tag.

Although it does not affect the functionality, I am still quite
confused about why the logs displayed by GitLab UI, for
example [1], is much less than the actual output (saved in the
artifacts, see [2]). Had a discussion with Michal on Matrix
and we agree that the log in gitlab UI is usually capped.

[1] https://gitlab.com/xen-project/people/henryw/xen/-/jobs/5690876361
[2] 
https://gitlab.com/xen-project/people/henryw/xen/-/jobs/5690876361/artifacts/file/smoke.serial
---
 automation/gitlab-ci/build.yaml | 17 +
 automation/gitlab-ci/test.yaml  | 22 ++
 2 files changed, 39 insertions(+)

diff --git a/automation/gitlab-ci/build.yaml b/automation/gitlab-ci/build.yaml
index 32af30cced..89d2f01302 100644
--- a/automation/gitlab-ci/build.yaml
+++ b/automation/gitlab-ci/build.yaml
@@ -344,6 +344,23 @@ kernel-6.1.19-export:
   tags:
 - x86_64
 
+armfvp-uboot-tfa-2023.10-2.9.0-arm64-export:
+  extends: .test-jobs-artifact-common
+  image: 
registry.gitlab.com/xen-project/xen/tests-artifacts/armfvp-uboot-tfa:2023.10-2.9.0-arm64v8
+  script:
+- |
+   mkdir binaries && \
+   cp /bl1.bin binaries/bl1.bin && \
+   cp /fip.bin binaries/fip.bin && \
+   cp /fvp-base-gicv3-psci-1t.dtb binaries/fvp-base-gicv3-psci-1t.dtb
+  artifacts:
+paths:
+  - binaries/bl1.bin
+  - binaries/fip.bin
+  - binaries/fvp-base-gicv3-psci-1t.dtb
+  tags:
+- arm64
+
 # Jobs below this line
 
 # Build jobs needed for tests
diff --git a/automation/gitlab-ci/test.yaml b/automation/gitlab-ci/test.yaml
index 6aabdb9d15..47e00d0a0b 100644
--- a/automation/gitlab-ci/test.yaml
+++ b/automation/gitlab-ci/test.yaml
@@ -96,6 +96,19 @@
   tags:
 - xilinx
 
+.fvp-arm64:
+  extends: .test-jobs-common
+  variables:
+CONTAINER: debian:bookworm-arm64v8-fvp
+LOGFILE: fvp-smoke-arm64.log
+  artifacts:
+paths:
+  - smoke.serial
+  - '*.log'
+when: always
+  tags:
+- arm64
+
 .adl-x86-64:
   extends: .test-jobs-common
   variables:
@@ -459,3 +472,12 @@ qemu-smoke-ppc64le-powernv9-gcc:
   needs:
 - qemu-system-ppc64-8.1.0-ppc64-export
 - debian-bullseye-gcc-ppc64le-debug
+
+fvp-smoke-dom0-arm64-gcc-debug:
+  extends: .fvp-arm64
+  script:
+- ./automation/scripts/fvp-base-smoke-dom0-arm64.sh 2>&1 | tee ${LOGFILE}
+  needs:
+- *arm64-test-needs
+- armfvp-uboot-tfa-2023.10-2.9.0-arm64-export
+- alpine-3.18-gcc-debug-arm64
-- 
2.25.1

Re: [RFC KERNEL PATCH v2 2/3] xen/pvh: Unmask irq for passthrough device in PVH dom0

2023-12-07 Thread Chen, Jiqian

Thank Stefano and Juergen, I will use this approach in next version.

On 2023/12/7 14:43, Juergen Gross wrote:
> On 07.12.23 03:18, Stefano Stabellini wrote:
>> On Tue, 5 Dec 2023, Chen, Jiqian wrote:
>>> When PVH dom0 enable a device, it will get trigger and polarity from ACPI 
>>> (see acpi_pci_irq_enable)
>>> I have a version of patch which tried that way, see below:
>>
>> This approach looks much better. I think this patch is OKish. Juergen,
>> what do you think?
> 
> The approach seems to be fine.
> 
> 
> Juergen
> 
>>
>>
>>> diff --git a/arch/x86/xen/enlighten_pvh.c b/arch/x86/xen/enlighten_pvh.c
>>> index ada3868c02c2..43e1bda9f946 100644
>>> --- a/arch/x86/xen/enlighten_pvh.c
>>> +++ b/arch/x86/xen/enlighten_pvh.c
>>> @@ -1,6 +1,7 @@
>>>   // SPDX-License-Identifier: GPL-2.0
>>>   #include 
>>>   #include 
>>> +#include 
>>>
>>>   #include 
>>>
>>> @@ -25,6 +26,127 @@
>>>   bool __ro_after_init xen_pvh;
>>>   EXPORT_SYMBOL_GPL(xen_pvh);
>>>
>>> +typedef struct gsi_info {
>>> +   int gsi;
>>> +   int trigger;
>>> +   int polarity;
>>> +   int pirq;
>>> +} gsi_info_t;
>>> +
>>> +struct acpi_prt_entry {
>>> +   struct acpi_pci_id  id;
>>> +   u8  pin;
>>> +   acpi_handle link;
>>> +   u32 index;  /* GSI, or link _CRS index 
>>> */
>>> +};
>>> +
>>> +static int xen_pvh_get_gsi_info(struct pci_dev *dev,
>>> +   gsi_info_t 
>>> *gsi_info)
>>> +{
>>> +   int gsi;
>>> +   u8 pin = 0;
>>> +   struct acpi_prt_entry *entry;
>>> +   int trigger = ACPI_LEVEL_SENSITIVE;
>>> +   int polarity = acpi_irq_model == ACPI_IRQ_MODEL_GIC ?
>>> + ACPI_ACTIVE_HIGH : ACPI_ACTIVE_LOW;
>>> +
>>> +   if (dev)
>>> +   pin = dev->pin;
>>> +   if (!pin) {
>>> +   xen_raw_printk("No interrupt pin configured\n");
>>> +   return -EINVAL;
>>> +   }
>>> +
>>> +   entry = acpi_pci_irq_lookup(dev, pin);
>>> +   if (entry) {
>>> +   if (entry->link)
>>> +   gsi = acpi_pci_link_allocate_irq(entry->link,
>>> +    entry->index,
>>> +    &trigger, 
>>> &polarity,
>>> +    NULL);
>>> +   else
>>> +   gsi = entry->index;
>>> +   } else
>>> +   return -EINVAL;
>>> +
>>> +   gsi_info->gsi = gsi;
>>> +   gsi_info->trigger = trigger;
>>> +   gsi_info->polarity = polarity;
>>> +
>>> +   return 0;
>>> +}
>>> +
>>> +static int xen_pvh_map_pirq(gsi_info_t *gsi_info)
>>> +{
>>> +   struct physdev_map_pirq map_irq;
>>> +   int ret;
>>> +
>>> +   map_irq.domid = DOMID_SELF;
>>> +   map_irq.type = MAP_PIRQ_TYPE_GSI;
>>> +   map_irq.index = gsi_info->gsi;
>>> +   map_irq.pirq = gsi_info->gsi;
>>> +
>>> +   ret = HYPERVISOR_physdev_op(PHYSDEVOP_map_pirq, &map_irq);
>>> +   gsi_info->pirq = map_irq.pirq;
>>> +
>>> +   return ret;
>>> +}
>>> +
>>> +static int xen_pvh_unmap_pirq(gsi_info_t *gsi_info)
>>> +{
>>> +   struct physdev_unmap_pirq unmap_irq;
>>> +
>>> +   unmap_irq.domid = DOMID_SELF;
>>> +   unmap_irq.pirq = gsi_info->pirq;
>>> +
>>> +   return HYPERVISOR_physdev_op(PHYSDEVOP_unmap_pirq, &unmap_irq);
>>> +}
>>> +
>>> +static int xen_pvh_setup_gsi(gsi_info_t *gsi_info)
>>> +{
>>> +   struct physdev_setup_gsi setup_gsi;
>>> +
>>> +   setup_gsi.gsi = gsi_info->gsi;
>>> +   setup_gsi.triggering = (gsi_info->trigger == ACPI_EDGE_SENSITIVE ? 
>>> 0 : 1);
>>> +   setup_gsi.polarity = (gsi_info->polarity == ACPI_ACTIVE_HIGH ? 0 : 
>>> 1);
>>> +
>>> +   return HYPERVISOR_physdev_op(PHYSDEVOP_setup_gsi, &setup_gsi);
>>> +}
>>> +
>>> +int xen_pvh_passthrough_gsi(struct pci_dev *dev)
>>> +{
>>> +   int ret;
>>> +   gsi_info_t gsi_info;
>>> +
>>> +   if (!dev) {
>>> +   return -EINVAL;
>>> +   }
>>> +
>>> +   ret = xen_pvh_get_gsi_info(dev, &gsi_info);
>>> +   if (ret) {
>>> +   xen_raw_printk("Fail to get gsi info!\n");
>>> +   return ret;
>>> +   }
>>> +
>>> +   ret = xen_pvh_map_pirq(&gsi_info);
>>> +   if (ret) {
>>> +   xen_raw_printk("Fail to map pirq for gsi (%d)!\n", 
>>> gsi_info.gsi);
>>> +   return ret;
>>> +   }
>>> +
>>> +   ret = xen_pvh_setup_gsi(&gsi_info);
>>> +   if (ret == -EEXIST) {
>>> +   ret = 0;
>>> +   xen_raw_printk("Already setup the GSI :%u\n", gsi_info.gsi);
>>> +   } else if (ret) {
>>> +   xen_raw_printk("Fail to setup gsi (%d)!\n", gsi_info.gsi);
>>> +   xen_pvh_unmap_pirq(&gsi_info);
>>> +   }
>>> +
>>> +   return ret;
>>> +}
>>> +EXPORT_SYMBOL_GPL(xen_pvh_passthrough_gsi);
>>> +
>>>

Re: [PATCH] docs/misra/rules.rst: add more rules

2023-12-07 Thread Jan Beulich

On 08.12.2023 01:08, Stefano Stabellini wrote:
> On Thu, 7 Dec 2023, Jan Beulich wrote:
>> On 07.12.2023 03:42, Stefano Stabellini wrote:
>>> On Wed, 6 Dec 2023, Jan Beulich wrote:
 On 06.12.2023 04:02, Stefano Stabellini wrote:
> --- a/docs/misra/rules.rst
> +++ b/docs/misra/rules.rst
> @@ -462,11 +462,23 @@ maintainers if you want to suggest a change.
>  
> while(0) and while(1) and alike are allowed.
>  
> +   * - `Rule 16.3 
> `_
> + - Required
> + - An unconditional break statement shall terminate every
> +   switch-clause
> + - In addition to break, also other flow control statements such as
> +   continue, return, goto are allowed.
> +
> * - `Rule 16.7 
> `_
>   - Required
>   - A switch-expression shall not have essentially Boolean type
>   -
>  
> +   * - `Rule 17.1 
> `_
> + - Required
> + - The features of  shall not be used
> + -

 Did we really accept this without any constraint (warranting mentioning
 here)?
>>>
>>> We agreed that in certain situations stdarg.h is OK to use and in those
>>> cases we would add a deviation. Would you like me to add something to
>>> that effect here? I could do that but it would sound a bit vague.  Also
>>> if we want to specify a project-wide deviation it would be better
>>> documented in docs/misra/deviations.rst. I would leave Rule 17.1 without
>>> a note.
>>
>> I can see your point, and I don't have a good suggestion on possible text.
>> Still I wouldn't feel well ack-ing this in its present shape.
> 
> What about:
> 
>  - It is understood that in some limited circumstances  is
>appropriate to use, such as the implementation of printk. Those
>cases will be dealt with using deviations as usual, see
>docs/misra/deviations.rst and
>docs/misra/documenting-violations.rst.

Looks okay. Would also look okay if it was just the first sentence.

Jan

Re: [PATCH v2] docs/misra/rules.rst: add more rules

2023-12-07 Thread Jan Beulich

On 08.12.2023 01:09, Stefano Stabellini wrote:
> Add the rules accepted in the last three MISRA C working group meetings.
> 
> Signed-off-by: Stefano Stabellini 

Acked-by: Jan Beulich 

> --- a/docs/misra/rules.rst
> +++ b/docs/misra/rules.rst
> @@ -462,6 +462,13 @@ maintainers if you want to suggest a change.
>  
> while(0) and while(1) and alike are allowed.
>  
> +   * - `Rule 16.3 
> `_
> + - Required
> + - An unconditional break statement shall terminate every
> +   switch-clause
> + - In addition to break, also other flow control statements such as
> +   continue, return, goto are allowed.

To eliminate any room for doubt, maybe add "unconditional" also again here?

Jan

Re: [PATCH v2] tools/libs/evtchn: replace assert()s in stubdom with proper locking

2023-12-07 Thread Jan Beulich

On 07.12.2023 07:25, Juergen Gross wrote:
> In tools/libs/evtchn/minios.c there are assert()s for the current
> thread being the main thread when binding an event channel.
> 
> As Mini-OS is supporting multiple threads, there is no real reason
> why the binding shouldn't be allowed to happen in any other thread.
> 
> Drop the assert()s and replace them with proper locking of the
> port_list.
> 
> Signed-off-by: Juergen Gross 

Is this a change I should pick up for backport?

Jan

Re: [PATCH v2] tools/libs/evtchn: replace assert()s in stubdom with proper locking

2023-12-07 Thread Juergen Gross


On 08.12.23 08:34, Jan Beulich wrote:

On 07.12.2023 07:25, Juergen Gross wrote:

In tools/libs/evtchn/minios.c there are assert()s for the current
thread being the main thread when binding an event channel.

As Mini-OS is supporting multiple threads, there is no real reason
why the binding shouldn't be allowed to happen in any other thread.

Drop the assert()s and replace them with proper locking of the
port_list.

Signed-off-by: Juergen Gross 


Is this a change I should pick up for backport?


This patch isn't really fixing a bug, but more enhancing functionality.

I don't think a backport is needed.


Juergen


OpenPGP_0xB0DE9DD628BF132F.asc
Description: OpenPGP public key


OpenPGP_signature.asc
Description: OpenPGP digital signature

< 1 2

101 - 154 of 154 matches

Mail list logo