Laszlo,
Idea being if the package supports host based unit tests it will scan
package to make sure all host based unit tests and libraries are listed
in the DSC.
As to why you see a different set of packages...that is interesting. If
you can include your full build logs i can look a little closer. I know
on the Azure CI servers we don't run the
On 8/21/2020 12:23 AM, Laszlo Ersek wrote:
Hi Mike, Sean;
On 08/19/20 19:59, Laszlo Ersek wrote:
I'll report back with more results.
I installed a Fedora 32 Server VM.
Installed the "mono-complete" package (+its deps).
Installed the "nodejs" package.
Installed "cspell" (+deps) with "npm" (with the latter coming from the
nodejs package).
Edk2 checked out at current HEAD
(5a6d764e1d073d28e8f398289ccb5592bf9a72ba).
* Platform CI results:
- <https://github.com/tianocore/edk2/tree/master/OvmfPkg/PlatformCI>:
Built successfully for IA32, X64, and IA32X64 (toolchain: GCC5).
- <https://github.com/tianocore/edk2/tree/master/ArmVirtPkg/PlatformCI>:
Built successfully for ARM and AARCH64 (toolchain: GCC5).
* Core CI results:
- <https://github.com/tianocore/edk2/blob/master/.pytool/Readme.md>
Complete run successful (toolchain: GCC5); no package / arch / target
/ test restrictions.
I got the following WARNINGs:
- ArmVirtPkg, EmulatorPkg, and OvmfPkg are covered by Platform CI, not
Core CI -- these are expected:
PROGRESS - --Running ArmVirtPkg: Compiler Plugin DEBUG --
WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
PROGRESS - --Running ArmVirtPkg: Compiler Plugin RELEASE --
WARNING - --->Test Skipped: in plugin! Compiler Plugin RELEASE
PROGRESS - --Running EmulatorPkg: Compiler Plugin DEBUG --
WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
PROGRESS - --Running EmulatorPkg: Compiler Plugin RELEASE --
WARNING - --->Test Skipped: in plugin! Compiler Plugin RELEASE
PROGRESS - --Running OvmfPkg: Compiler Plugin DEBUG --
WARNING - --->Test Skipped: in plugin! Compiler Plugin DEBUG
PROGRESS - --Running OvmfPkg: Compiler Plugin RELEASE --
WARNING - --->Test Skipped: in plugin! Compiler Plugin RELEASE
- ArmVirtPkg, DynamicTablesPkg, CryptoPkg, EmulatorPkg, FatPkg,
NetworkPkg, OvmfPkg, PcAtChipsetPkg, SecurityPkg and ShellPkg are
not covered by host-based unit tests -- I think also expected:
PROGRESS - --Running ArmVirtPkg: Host Unit Test Compiler Plugin NOOPT --
WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin
NOOPT
PROGRESS - --Running DynamicTablesPkg: Host Unit Test Compiler Plugin
NOOPT --
WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin
NOOPT
PROGRESS - --Running CryptoPkg: Host Unit Test Compiler Plugin NOOPT --
WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin
NOOPT
PROGRESS - --Running EmulatorPkg: Host Unit Test Compiler Plugin
NOOPT --
WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin
NOOPT
PROGRESS - --Running FatPkg: Host Unit Test Compiler Plugin NOOPT --
WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin
NOOPT
PROGRESS - --Running NetworkPkg: Host Unit Test Compiler Plugin NOOPT --
WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin
NOOPT
PROGRESS - --Running OvmfPkg: Host Unit Test Compiler Plugin NOOPT --
WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin
NOOPT
PROGRESS - --Running PcAtChipsetPkg: Host Unit Test Compiler Plugin
NOOPT --
WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin
NOOPT
PROGRESS - --Running SecurityPkg: Host Unit Test Compiler Plugin
NOOPT --
WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin
NOOPT
PROGRESS - --Running ShellPkg: Host Unit Test Compiler Plugin NOOPT --
WARNING - --->Test Skipped: in plugin! Host Unit Test Compiler Plugin
NOOPT
- I'm not really sure about the following warnings. They were emitted
for a subset of the above packages. I read the docs at
<https://github.com/tianocore/edk2/blob/master/.pytool/Readme.md#host-module-inclusion-test---hostunittestdsccompletecheck>
but I still don't understand :)
Each plugin/test has a readme as well. Not sure if this helps explain
the HostUnitTestDscCompleteCheck more.
https://github.com/tianocore/edk2/tree/master/.pytool/Plugin/HostUnitTestDscCompleteCheck
As to why you don't see it skipped for ArmVirtPkg or a few others: I
think this is a bug.
In Azure CI i see it as test passed.
PROGRESS - --Running ArmVirtPkg: Host Unit Test Dsc Complete Check Test
NO-TARGET --
PROGRESS - --->Test Success: Host Unit Test Dsc Complete Check Test
NO-TARGET
When i look at the code
https://github.com/tianocore/edk2/blob/master/.pytool/Plugin/HostUnitTestDscCompleteCheck/HostUnitTestDscCompleteCheck.py#L59
I see that how those packages *.ci.yaml have it configured is different
than those that show skipped below. It avoids the skip conditions but
since those packages don't have any host unit test libraries or modules
there is no failure.
This should probably be a bugzilla on the
HostUnitTestDscCompleteCheck.py as the reporting is not correct. Test
should show up as skipped.
You can see them here showing up as success
https://dev.azure.com/tianocore/edk2-ci/_build/results?buildId=11309&view=logs&j=780eaa17-fc2d-5131-b9d2-21b19bc9bec5&t=0791c90f-e7d8-5372-c9c5-edbe30bf6aeb&l=32
PROGRESS - --Running CryptoPkg: Host Unit Test Dsc Complete Check
Test NO-TARGET --
WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete
Check Test NO-TARGET
PROGRESS - --Running FatPkg: Host Unit Test Dsc Complete Check Test
NO-TARGET --
WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete
Check Test NO-TARGET
PROGRESS - --Running NetworkPkg: Host Unit Test Dsc Complete Check
Test NO-TARGET --
WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete
Check Test NO-TARGET
PROGRESS - --Running PcAtChipsetPkg: Host Unit Test Dsc Complete
Check Test NO-TARGET --
WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete
Check Test NO-TARGET
PROGRESS - --Running SecurityPkg: Host Unit Test Dsc Complete Check
Test NO-TARGET --
WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete
Check Test NO-TARGET
PROGRESS - --Running ShellPkg: Host Unit Test Dsc Complete Check Test
NO-TARGET --
WARNING - --->Test Skipped: in plugin! Host Unit Test Dsc Complete
Check Test NO-TARGET
- Still related to host-based unit tests, I believe the following
warnings, for FmpDevicePkg, MdeModulePkg, MdePkg, UefiCpuPkg, and
UnitTestFrameworkPkg, report that running the tests on the build
host (understandably) prevents the tests from covering all edk2
architectures:
Yes this message is because we override the arch for NOOPT to match that
of the build system since we are going to run these executables in the
host environment.
PROGRESS - --Running FmpDevicePkg: Host Unit Test Compiler Plugin
NOOPT --
WARNING - Allowing Override for key TARGET_ARCH
PROGRESS - --Running MdeModulePkg: Host Unit Test Compiler Plugin
NOOPT --
WARNING - Allowing Override for key TARGET_ARCH
PROGRESS - --Running MdePkg: Host Unit Test Compiler Plugin NOOPT --
WARNING - Allowing Override for key TARGET_ARCH
PROGRESS - --Running UefiCpuPkg: Host Unit Test Compiler Plugin NOOPT --
WARNING - Allowing Override for key TARGET_ARCH
PROGRESS - --Running UnitTestFrameworkPkg: Host Unit Test Compiler
Plugin NOOPT --
WARNING - Allowing Override for key TARGET_ARCH
- The below seems to be related to the "spell checking in audit
mode" known issues at
<https://github.com/tianocore/edk2/blob/master/.pytool/Readme.md>.
What does "audit mode" mean?
see readme here.
https://github.com/tianocore/edk2/tree/master/.pytool/Plugin/SpellCheck#configuration
Basically runs the tests and reports all the errors as warnings and then
reports the test as skipped.
It was hopefully to encourage package maintainers to fix their spelling
issues but not breaking the build on day 1.
FWIW, the packages listed below (EmulatorPkg, MdeModulePkg,
MdePkg, NetworkPkg, OvmfPkg, ShellPkg, UefiCpuPkg) is a proper
subset of the packges noted in ".pytool/Readme.md" -- the latter
mentions CryptoPkg and SecurityPkg in addition.
PROGRESS - --Running EmulatorPkg: Spell Check Test NO-TARGET --
WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
PROGRESS - --Running MdeModulePkg: Spell Check Test NO-TARGET --
WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
PROGRESS - --Running MdePkg: Spell Check Test NO-TARGET --
WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
PROGRESS - --Running NetworkPkg: Spell Check Test NO-TARGET --
WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
PROGRESS - --Running OvmfPkg: Spell Check Test NO-TARGET --
WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
PROGRESS - --Running ShellPkg: Spell Check Test NO-TARGET --
WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
PROGRESS - --Running UefiCpuPkg: Spell Check Test NO-TARGET --
WARNING - --->Test Skipped: in plugin! Spell Check Test NO-TARGET
- Not sure about the last one:
PROGRESS - --Running FatPkg: Library Class Check Test NO-TARGET --
WARNING - --->Test Skipped: in plugin! Library Class Check Test
NO-TARGET
* One suggestion (in addition to the others made in this thread):
- Please add "BaseToolsBuild" to "BaseTools/.gitignore".
There is a bugzilla for this already.
https://bugzilla.tianocore.org/show_bug.cgi?id=2581
* A question:
- On github, there is some logic that restricts the full Core CI build
to packages that may be affected by a patch series. Is this available
locally?
According to the Core CI documentation, the "stuart_ci_build" command
can be limited with "-p" / "-a" / "-t" flags; given a commit range,
how do we calculate the "tightest" flag values?
A couple things here.
You can always look at the azurepipelines files to get a better
understanding of the CI process. Hopefully there is no magic (besides
being able to read yaml). You can see here:
https://github.com/tianocore/edk2/blob/master/.azurepipelines/templates/pr-gate-steps.yml#L35
It is actually another stuart tool.
stuart_pr_eval.
We have bug to document it.
https://github.com/tianocore/edk2-pytool-extensions/issues/89
But the python file is pretty well documented. There are 4 policies that
can trigger the need for a change to build a package.
See here
https://github.com/tianocore/edk2-pytool-extensions/blob/master/edk2toolext/invocables/edk2_pr_eval.py#L117
Thank you very much!
Laszlo
