-----Original Message-----
From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Laszlo Ersek
Sent: Wednesday, August 19, 2020 5:27 AM
To: edk2-devel-groups-io <devel@edk2.groups.io>; Sean Brogan
<sean.bro...@microsoft.com>; Bret Barkelew
<bret.barke...@microsoft.com>
Cc: Vladimir Olovyannikov <vladimir.olovyanni...@broadcom.com>; Rebecca Cran
<rebe...@bsdio.com>; Tom Lendacky
<thomas.lenda...@amd.com>
Subject: [edk2-devel] running CI locally
Hi!
I'd like to test CI locally. I'm going through ".pytool/Readme.md" with
the tree checked out at 7e6f150b6902 (= current HEAD). I'm doing this in
a RHEL8 VM, with a python3 virtual environment set up / entered.
* My first note is that the command
pip install --upgrade pip-requirements.txt
under "Prerequisets", has a small typo; it should be
pip install --upgrade -r pip-requirements.txt
(the "-r" option is missing).
(
After adding "-r", the following components are now installed in my
virtual env:
- edk2-pytool-library: 0.10.12
- edk2-pytool-extensions: 0.13.9
- antlr4-python3-runtime: 4.7.1
- pyyaml: 5.3.1
Stating this because it might matter for the rest of my email.
)
* Second, when I run the following command:
stuart_update -c .pytool/CISettings.py TOOL_CHAIN_TAG=GCC5
I get the following warnings:
WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm@2.14.02: [Nuget] We
failed to install this version 2.14.02 of mu_nasm
WARNING - [SDE] Failed to fetch NugetDependecy: iasl@20190215.0.0: [Nuget] We
failed to install this version 20190215.0.0 of iasl
(repeated one more time:)
WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm@2.14.02: [Nuget] We
failed to install this version 2.14.02 of mu_nasm
WARNING - [SDE] Failed to fetch NugetDependecy: iasl@20190215.0.0: [Nuget] We
failed to install this version 20190215.0.0 of iasl
and then finally:
ERROR - We were unable to successfully update 2 dependencies in environment
ERROR - Error
The virtual machine has NASM installed (2.13.03-2.el8) and IASL too
(acpica-tools-20180629-3.el8).
Where do the NASM and IASL version requirements (2.14.02 and
20190215.0.0, respectively) come from?
Hm... After a git-grep for those version numbers, I find:
- BaseTools/Bin/nasm_ext_dep.yaml
- BaseTools/Bin/iasl_ext_dep.yaml
I was about to say that these version requirements are too strict: for
example, "BaseTools/Conf/tools_def.template" requires "NASM 2.10 or
later for use with the GCC toolchain family". What I have installed
satisfies that, and so CI shouldn't require anything more recent.
*However*, both of the above YAML files have very helpful comments, so I
understand these high versions are downloaded afresh, and only for the
CI run.
And so my question becomes: why do the "nuget" downloads fail for me
(because, presumably, they work fine in the central CI env on github /
Azure); and how can I fix the issue if it pops up again?
I've checked "nuget.org" in my browser, and it has:
- https://www.nuget.org/packages/mu_nasm/ --> 2.14.2
- https://www.nuget.org/packages/iasl/ --> 20190215.0.0
... On a hunch, I've attempted adding the "--verbose" option to the
"stuart_update" command; this is the output (excerpt):
DEBUG - Verify 'mu_nasm' returning 'False'.
DEBUG - Verify 'iasl' returning 'False'.
DEBUG - Creating 4 threads for the SDE update
UpdatingDEBUG - Verify 'mu_nasm' returning 'False'.
DEBUG - Verify 'gcc_aarch64_linux' returning 'True'.
DEBUG - Cleaning dependency directory for 'mu_nasm'...
DEBUG - Verify 'gcc_arm_linux' returning 'True'.
INFO - Cmd to run is: mono
/root/py3venv/lib/python3.6/site-packages/edk2toolext/bin/NuGet.exe locals
global-packages -list
DEBUG - Verify 'iasl' returning 'False'.
INFO - ------------------------------------------------
DEBUG - Cleaning dependency directory for 'iasl'...
DEBUG - Verify 'gcc_riscv64_unknown' returning 'True'.
INFO - --------------Cmd Output Starting---------------
INFO - Cmd to run is: mono
/root/py3venv/lib/python3.6/site-packages/edk2toolext/bin/NuGet.exe locals
global-packages -list
INFO - ------------------------------------------------
INFO - ------------------------------------------------
INFO - --------------Cmd Output Starting---------------
INFO - ------------------------------------------------
INFO - /bin/sh: mono: command not found
INFO - /bin/sh: mono: command not found
INFO - ------------------------------------------------
INFO - --------------Cmd Output Finished---------------
INFO - --------- Running Time (mm:ss): 00:00 ----------
INFO - ----------- Return Code: 0x0000007f ------------
INFO - ------------------------------------------------
INFO - Nuget was unable to provide global packages cache location.
INFO - Cmd to run is: mono
/root/py3venv/lib/python3.6/site-packages/edk2toolext/bin/NuGet.exe install
mu_nasm -Source
https://api.nuget.org/v3/index.json -ExcludeVersion -NonInteractive -Version
2.14.02 -Verbosity detailed -OutputDirectory
"/root/src/rhel8/edk2/BaseTools/Bin/mu_nasm_extdep_temp"
INFO - ------------------------------------------------
INFO - --------------Cmd Output Starting---------------
INFO - ------------------------------------------------
INFO - ------------------------------------------------
INFO - --------------Cmd Output Finished---------------
INFO - --------- Running Time (mm:ss): 00:00 ----------
INFO - ----------- Return Code: 0x0000007f ------------
INFO - ------------------------------------------------
INFO - Nuget was unable to provide global packages cache location.
INFO - Cmd to run is: mono
/root/py3venv/lib/python3.6/site-packages/edk2toolext/bin/NuGet.exe install
iasl -Source
https://api.nuget.org/v3/index.json -ExcludeVersion -NonInteractive -Version
20190215.0.0 -Verbosity detailed -OutputDirectory
"/root/src/rhel8/edk2/BaseTools/Bin/iasl_extdep_temp"
INFO - ------------------------------------------------
INFO - --------------Cmd Output Starting---------------
INFO - /bin/sh: mono: command not found
INFO - ------------------------------------------------
INFO - ------------------------------------------------
INFO - --------------Cmd Output Finished---------------
INFO - --------- Running Time (mm:ss): 00:00 ----------
INFO - ----------- Return Code: 0x0000007f ------------
INFO - ------------------------------------------------
WARNING - [SDE] Failed to fetch NugetDependecy: mu_nasm@2.14.02: [Nuget] We
failed to install this version 2.14.02 of mu_nasm
Aha! So "mono" is another requirement.
After installing the "mono-complete" package from EPEL8 (together with
its dependencies), the "stuart_update" command completes fine for me.
Can we spell out the "mono" dependency in ".pytool/Readme.md" somewhere,
or can we perhaps check for mono automatically?
* Third, attempting
stuart_ci_build -c .pytool/CISettings.py -p OvmfPkg -a IA32,X64 -t DEBUG
TOOL_CHAIN_TAG=GCC5
doesn't seem to do anything:
SECTION - Init SDE
SECTION - Loading Plugins
SECTION - Start Invocable Tool
SECTION - Getting Environment
SECTION - Loading plugins
SECTION - Building OvmfPkg Package
PROGRESS - --Running OvmfPkg: Compiler Plugin DEBUG --
WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
PROGRESS - Overall Build Status: Success
SECTION - Summary
PROGRESS - Success
If I remove the "-t DEBUG" option, a bit more happens (but it also
fails):
SECTION - Init SDE
SECTION - Loading Plugins
SECTION - Start Invocable Tool
SECTION - Getting Environment
SECTION - Loading plugins
SECTION - Building OvmfPkg Package
PROGRESS - --Running OvmfPkg: Char Encoding Check Test NO-TARGET --
PROGRESS - --->Test Success: Char Encoding Check Test NO-TARGET
PROGRESS - --Running OvmfPkg: Compiler Plugin DEBUG --
WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
PROGRESS - --Running OvmfPkg: Compiler Plugin RELEASE --
WARNING - --->Test Skipped: in plugin! Compiler Plugin RELEASE
PROGRESS - --Running OvmfPkg: Dependency Check Test NO-TARGET --
PROGRESS - --->Test Success: Dependency Check Test NO-TARGET
PROGRESS - --Running OvmfPkg: Dsc Complete Check Test NO-TARGET --
PROGRESS - --->Test Success: Dsc Complete Check Test NO-TARGET
PROGRESS - --Running OvmfPkg: Guid Check Test NO-TARGET --
PROGRESS - --->Test Success: Guid Check Test NO-TARGET
PROGRESS - --Running OvmfPkg: Library Class Check Test NO-TARGET --
PROGRESS - --->Test Success: Library Class Check Test NO-TARGET
PROGRESS - --Running OvmfPkg: Spell Check Test NO-TARGET --
WARNING - NodeJs not installed. Test can't run
WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
PROGRESS - --Running OvmfPkg: EccCheck Test NO-TARGET --
PROGRESS - --->Test Success: EccCheck Test NO-TARGET
PROGRESS - --Running OvmfPkg: License Check Test NO-TARGET --
PROGRESS - --->Test Success: License Check Test NO-TARGET
PROGRESS - --Running OvmfPkg: Host Unit Test Compiler Plugin NOOPT --
WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin NOOPT
PROGRESS - --Running OvmfPkg: Host Unit Test Dsc Complete Check Test NO-TARGET
--
PROGRESS - --->Test Success: Host Unit Test Dsc Complete Check Test NO-TARGET
PROGRESS - Overall Build Status: Success
SECTION - Summary
PROGRESS - Success
I'm not sure about the "NodeJs not installed. Test can't run" message --
I guess it only relates to the "Spell Check Test".
But what's missing for the actual build? I've noticed the following
warning:
Test Skipped: in plugin! Compiler Plugin DEBUG
But even if I add "--verbose", the only lines around that warning are:
INFO - Running on Package: OvmfPkg
PROGRESS - --Running OvmfPkg: Compiler Plugin DEBUG --
WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
PROGRESS - Overall Build Status: Success
Why is the build skipped?
--*--
Summary:
- the typo in "pip install --upgrade -r pip-requirements.txt" should be fixed
- "mono" should be documented, or auto-installed, as a dependency
- I don't know why "stuart_ci_build" skips the actual build.
Thanks!
Laszlo
