Re: [EXTERNAL] Re: [edk2-devel] generating the test matrix for local CI runs

2021-03-04 Thread Laszlo Ersek
On 03/04/21 19:32, Bret Barkelew wrote:

> Can you share the specific PR youre failing so that we can talk in
> concrete terms?

Sure.

In January, I posted the following patch series:

  [edk2-devel] [PATCH 0/8]
  ShellPkg, ArmVirtPkg, OvmfPkg: shell usability improvements

  https://edk2.groups.io/g/devel/message/69590
  https://www.redhat.com/archives/edk2-devel-archive/2021-January/msg00070.html
  Message-Id: 20210104154235.31785-1-ler...@redhat.com

This series cleared my local CI just fine (both Windows and Linux VM),
before I posted it.

When review was done, I attempted to merge the series:

  https://github.com/tianocore/edk2/pull/1336

Although the PR does not show it now (or at least I cannot make it show
it), the github CI uncovered a real problem in my patch set, one that my
local run missed.

Namely, I had added an OrderedCollectionLib class dependency to the
(singleton) UefiShellCommandLib instance, but failed to resolve the
OrderedCollectionLib class in the EmulatorPkg and UefiPayloadPkg DSC
files. Consequently, those DSC files would no longer build. My local CI
runs didn't extend to these packages, but -- thankfully -- the
github.com-based CI did.

In v2, I added one extra patch for each of EmulatorPkg and
UefiPayloadPkg DSC:

  [edk2-devel] [PATCH v2 00/10]
  multiple packages: shell usability improvements

  https://edk2.groups.io/g/devel/message/70216
  https://www.redhat.com/archives/edk2-devel-archive/2021-January/msg00700.html
  Message-Id: 20210113085453.10168-1-ler...@redhat.com

(What's more, I tracked down the DSC files in edk2-platforms that used
to build the shell, and I resolved the OrderedCollectionLib class in
them as well -- see the links inside the above-referenced blurbs.)

The v2 CI succeded on github (personal, then merge):

  https://github.com/tianocore/edk2/pull/1337
  https://github.com/tianocore/edk2/pull/1370

In summary, the bug (in my local CI usage) was that I didn't cover
EmulatorPkg and UefiPayloadPkg.

So, I'd like if the local test builds determined the affected set of
packages as well -- I'd like to be *unable* to miss EmulatorPkg and
UefiPayloadPkg in the above type of situation.

Thanks!
Laszlo



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#72450): https://edk2.groups.io/g/devel/message/72450
Mute This Topic: https://groups.io/mt/81083112/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-




Re: [EXTERNAL] Re: [edk2-devel] generating the test matrix for local CI runs

2021-03-04 Thread Laszlo Ersek
On 03/04/21 19:20, Bret Barkelew wrote:
> Thinking further, it will skip some tests if you haven’t installed the 
> prerequisites (e.g. spell check). We could potentially add a single script to 
> install the prerequisites, but then there are the prereqs for the prereqs 
> (e.g. Node).
> 
> To more clearly address your question, the “tests” that a GitHub CI is broken 
> up into are arbitrary subdivisions of the tests that run locally (when you 
> run “stuart_ci_build” without parameters) so that the workload can be divided 
> amongst the available worker agents. If you were to compare the individual 
> test logs against your local, monolithic test log, you would find that all 
> cases were covered in both (assuming the aforementioned prereqs are 
> installed).

Thank you -- I hope to test this later.

(1) A question about deps: if a dep is missing, can I force an error
(rather than a silent skip)?

Again, I distinctly remember doing what I then considered "due
diligence" locally, only to see github reject the PR later. If that was
because some deps were missing locally, I'd really like to turn that
scenario into a local error.

A script for installing all dependencies would help of course, but not
as a *replacement* for this kind of deps enforcement -- instead, only
for solving the missing dependency problems. I'd still want the local CI
to break hard if it couldn't offer the exact coverage as github.com.

Another question below:

> 
> - Bret
> 
> From: Bret Barkelew via 
> groups.io<mailto:bret.barkelew=microsoft@groups.io>
> Sent: Thursday, March 4, 2021 9:57 AM
> To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; 
> rebe...@nuviainc.com<mailto:rebe...@nuviainc.com>; 
> ler...@redhat.com<mailto:ler...@redhat.com>
> Cc: Sean Brogan<mailto:sean.bro...@microsoft.com>; Kinney, Michael 
> D<mailto:michael.d.kin...@intel.com>
> Subject: Re: [EXTERNAL] Re: [edk2-devel] generating the test matrix for local 
> CI runs
> 
> *An* answer is:
> - There’s a DevOps virtual environment that you can run that will pretend to 
> be a build agent. I played with it years ago, but that was before the CI 
> tooling (and it was years ago) so I don’t know how the capabilities compare.
> 
> My real answer is:
> - If you run the three commands -- setup, update, and build -- without any 
> parameters other than your TOOL_CHAIN_TAG, it should run all tests on all 
> packages for that particular OS/toolchain.

(2) You mention "for that particular OS/toolchain".

How can I make sure that I'm using the exact same toolchain as github would?

I'd like to tell the tooling only the OS (Windows vs. Linux), and then
the toolchain should be auto-selected, same way as it is auto-selected
on github. For example, if I'm in my Windows 10 VM, then I should be
*unable* to request anything else than the latest VS toolchain;
similarly, in the Linux VM, I should be *unable* to ask for (e.g.) the
GCC49 toolchain; only the GCC50 toolchain should be usable.

When I submit a github PR, I'm not asked to pick toolchains for CI; can
we automate the toolchain selection locally too?

(Sorry if I misunderstood your comment above -- if you are saying that
it's permitted, but *not required*, to specify TOOL_CHAIN_TAG, then my
question (2) falls away.)

Thanks!
Laszlo
> 
> - Bret
> 
> From: Rebecca Cran via groups.io<mailto:rebecca=nuviainc@groups.io>
> Sent: Thursday, March 4, 2021 9:45 AM
> To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; 
> ler...@redhat.com<mailto:ler...@redhat.com>
> Cc: Sean Brogan<mailto:sean.bro...@microsoft.com>; Bret 
> Barkelew<mailto:bret.barke...@microsoft.com>; Kinney, Michael 
> D<mailto:michael.d.kin...@intel.com>
> Subject: [EXTERNAL] Re: [edk2-devel] generating the test matrix for local CI 
> runs
> 
> On 3/4/21 10:04 AM, Laszlo Ersek wrote:
>> (Sigh, my address book pulled one over me -- resending to the correct
>> list address now. Please ignore the previous posting that you may have
>> gotten "in private".)
>>
>> Hi All,
>>
>> while I can run specific CI tests locally, using the commands listed at:
>>
>> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2F.pytool&data=04%7C01%7Cbret.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608d8df354b05%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504767278995847%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=l8WblhD%2B7dpkFaCZ8roSfNUmj1imytHi5eUT%2F%2BKrLCo%3D&reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2F.pytool&

Re: [EXTERNAL] Re: [edk2-devel] generating the test matrix for local CI runs

2021-03-04 Thread Bret Barkelew via groups.io
Digging in even further, it looks like you’re asking about tests across test 
pipelines, including some of the platform pipelines. I don’t believe there’s a 
cohesive way to do that, as some of them require platform-specific parameters 
(such as the --FlashRom parameter for OvmfPkg).

Can you share the specific PR you’re failing so that we can talk in concrete 
terms?

My guess (from experience failing this myself) is that you are failing the 
spell check remotely and not running it locally (because Node, npm, or cspell 
are not installed). It’s possible that we should investigate documenting 
cross-pipeline and cross-package CI dependencies in a central place. Right now 
it’s distributed (e.g. did you follow the steps here: 
https://github.com/tianocore/edk2/tree/master/.pytool#spell-checking---cspell).

- Bret

From: Bret Barkelew via groups.io<mailto:bret.barkelew=microsoft@groups.io>
Sent: Thursday, March 4, 2021 10:22 AM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; 
rebe...@nuviainc.com<mailto:rebe...@nuviainc.com>; 
ler...@redhat.com<mailto:ler...@redhat.com>
Cc: Sean Brogan<mailto:sean.bro...@microsoft.com>; Kinney, Michael 
D<mailto:michael.d.kin...@intel.com>
Subject: Re: [EXTERNAL] Re: [edk2-devel] generating the test matrix for local 
CI runs

Thinking further, it will skip some tests if you haven’t installed the 
prerequisites (e.g. spell check). We could potentially add a single script to 
install the prerequisites, but then there are the prereqs for the prereqs (e.g. 
Node).

To more clearly address your question, the “tests” that a GitHub CI is broken 
up into are arbitrary subdivisions of the tests that run locally (when you run 
“stuart_ci_build” without parameters) so that the workload can be divided 
amongst the available worker agents. If you were to compare the individual test 
logs against your local, monolithic test log, you would find that all cases 
were covered in both (assuming the aforementioned prereqs are installed).

- Bret

From: Bret Barkelew via groups.io<mailto:bret.barkelew=microsoft@groups.io>
Sent: Thursday, March 4, 2021 9:57 AM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; 
rebe...@nuviainc.com<mailto:rebe...@nuviainc.com>; 
ler...@redhat.com<mailto:ler...@redhat.com>
Cc: Sean Brogan<mailto:sean.bro...@microsoft.com>; Kinney, Michael 
D<mailto:michael.d.kin...@intel.com>
Subject: Re: [EXTERNAL] Re: [edk2-devel] generating the test matrix for local 
CI runs

*An* answer is:
- There’s a DevOps virtual environment that you can run that will pretend to be 
a build agent. I played with it years ago, but that was before the CI tooling 
(and it was years ago) so I don’t know how the capabilities compare.

My real answer is:
- If you run the three commands -- setup, update, and build -- without any 
parameters other than your TOOL_CHAIN_TAG, it should run all tests on all 
packages for that particular OS/toolchain.

- Bret

From: Rebecca Cran via groups.io<mailto:rebecca=nuviainc@groups.io>
Sent: Thursday, March 4, 2021 9:45 AM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; 
ler...@redhat.com<mailto:ler...@redhat.com>
Cc: Sean Brogan<mailto:sean.bro...@microsoft.com>; Bret 
Barkelew<mailto:bret.barke...@microsoft.com>; Kinney, Michael 
D<mailto:michael.d.kin...@intel.com>
Subject: [EXTERNAL] Re: [edk2-devel] generating the test matrix for local CI 
runs

On 3/4/21 10:04 AM, Laszlo Ersek wrote:
> (Sigh, my address book pulled one over me -- resending to the correct
> list address now. Please ignore the previous posting that you may have
> gotten "in private".)
>
> Hi All,
>
> while I can run specific CI tests locally, using the commands listed at:
>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2F.pytool&data=04%7C01%7Cbret.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608d8df354b05%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504767278995847%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=l8WblhD%2B7dpkFaCZ8roSfNUmj1imytHi5eUT%2F%2BKrLCo%3D&reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2F.pytool&data=04%7C01%7CBret.Barkelew%40microsoft.com%7C17527a8762f346729ca508d8df3a3118%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504789795390940%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=nKugKAbKDAjSZASwvAAnvDTvaev1IezlePOxvURGrF8%3D&reserved=0>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2FArmVirtPkg%2FPlatformCI&data=04%7C01%7Cbret.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608d8df354b05%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C

Re: [EXTERNAL] Re: [edk2-devel] generating the test matrix for local CI runs

2021-03-04 Thread Bret Barkelew via groups.io
Thinking further, it will skip some tests if you haven’t installed the 
prerequisites (e.g. spell check). We could potentially add a single script to 
install the prerequisites, but then there are the prereqs for the prereqs (e.g. 
Node).

To more clearly address your question, the “tests” that a GitHub CI is broken 
up into are arbitrary subdivisions of the tests that run locally (when you run 
“stuart_ci_build” without parameters) so that the workload can be divided 
amongst the available worker agents. If you were to compare the individual test 
logs against your local, monolithic test log, you would find that all cases 
were covered in both (assuming the aforementioned prereqs are installed).

- Bret

From: Bret Barkelew via groups.io<mailto:bret.barkelew=microsoft@groups.io>
Sent: Thursday, March 4, 2021 9:57 AM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; 
rebe...@nuviainc.com<mailto:rebe...@nuviainc.com>; 
ler...@redhat.com<mailto:ler...@redhat.com>
Cc: Sean Brogan<mailto:sean.bro...@microsoft.com>; Kinney, Michael 
D<mailto:michael.d.kin...@intel.com>
Subject: Re: [EXTERNAL] Re: [edk2-devel] generating the test matrix for local 
CI runs

*An* answer is:
- There’s a DevOps virtual environment that you can run that will pretend to be 
a build agent. I played with it years ago, but that was before the CI tooling 
(and it was years ago) so I don’t know how the capabilities compare.

My real answer is:
- If you run the three commands -- setup, update, and build -- without any 
parameters other than your TOOL_CHAIN_TAG, it should run all tests on all 
packages for that particular OS/toolchain.

- Bret

From: Rebecca Cran via groups.io<mailto:rebecca=nuviainc@groups.io>
Sent: Thursday, March 4, 2021 9:45 AM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; 
ler...@redhat.com<mailto:ler...@redhat.com>
Cc: Sean Brogan<mailto:sean.bro...@microsoft.com>; Bret 
Barkelew<mailto:bret.barke...@microsoft.com>; Kinney, Michael 
D<mailto:michael.d.kin...@intel.com>
Subject: [EXTERNAL] Re: [edk2-devel] generating the test matrix for local CI 
runs

On 3/4/21 10:04 AM, Laszlo Ersek wrote:
> (Sigh, my address book pulled one over me -- resending to the correct
> list address now. Please ignore the previous posting that you may have
> gotten "in private".)
>
> Hi All,
>
> while I can run specific CI tests locally, using the commands listed at:
>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2F.pytool&data=04%7C01%7Cbret.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608d8df354b05%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504767278995847%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=l8WblhD%2B7dpkFaCZ8roSfNUmj1imytHi5eUT%2F%2BKrLCo%3D&reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2F.pytool&data=04%7C01%7CBret.Barkelew%40microsoft.com%7C385eef1d988f4a56155a08d8df36ecce%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504774296518259%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=UPVktVmDdChYhoc6zErhuz57Dllw21gqcwi9crhUdp8%3D&reserved=0>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2FArmVirtPkg%2FPlatformCI&data=04%7C01%7Cbret.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608d8df354b05%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504767278995847%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=8yG7Mro9Li4XeCvfmPhfReBbipz47nyAjUmMKMGfh%2Bo%3D&reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2FArmVirtPkg%2FPlatformCI&data=04%7C01%7CBret.Barkelew%40microsoft.com%7C385eef1d988f4a56155a08d8df36ecce%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504774296518259%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=mcoSK9x7FSSJzlI86mdvs%2Btp7zdeF42IJn62mJxxozo%3D&reserved=0>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2FOvmfPkg%2FPlatformCI&data=04%7C01%7Cbret.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608d8df354b05%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504767278995847%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=NUhAJ8zkITwPkvCrUqb7ZogSQTJuBXv0eL6Au0sYC2M%3D&reserved=0<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2FOvmfPkg%2FPlatformCI&data=04%7C01%7CBret.Barkelew%40microsoft.com%7C385eef

Re: [EXTERNAL] Re: [edk2-devel] generating the test matrix for local CI runs

2021-03-04 Thread Bret Barkelew via groups.io
*An* answer is:
- There’s a DevOps virtual environment that you can run that will pretend to be 
a build agent. I played with it years ago, but that was before the CI tooling 
(and it was years ago) so I don’t know how the capabilities compare.

My real answer is:
- If you run the three commands -- setup, update, and build -- without any 
parameters other than your TOOL_CHAIN_TAG, it should run all tests on all 
packages for that particular OS/toolchain.

- Bret

From: Rebecca Cran via groups.io<mailto:rebecca=nuviainc@groups.io>
Sent: Thursday, March 4, 2021 9:45 AM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; 
ler...@redhat.com<mailto:ler...@redhat.com>
Cc: Sean Brogan<mailto:sean.bro...@microsoft.com>; Bret 
Barkelew<mailto:bret.barke...@microsoft.com>; Kinney, Michael 
D<mailto:michael.d.kin...@intel.com>
Subject: [EXTERNAL] Re: [edk2-devel] generating the test matrix for local CI 
runs

On 3/4/21 10:04 AM, Laszlo Ersek wrote:
> (Sigh, my address book pulled one over me -- resending to the correct
> list address now. Please ignore the previous posting that you may have
> gotten "in private".)
>
> Hi All,
>
> while I can run specific CI tests locally, using the commands listed at:
>
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2F.pytool&data=04%7C01%7Cbret.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608d8df354b05%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504767278995847%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=l8WblhD%2B7dpkFaCZ8roSfNUmj1imytHi5eUT%2F%2BKrLCo%3D&reserved=0
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2FArmVirtPkg%2FPlatformCI&data=04%7C01%7Cbret.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608d8df354b05%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504767278995847%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=8yG7Mro9Li4XeCvfmPhfReBbipz47nyAjUmMKMGfh%2Bo%3D&reserved=0
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Ftianocore%2Fedk2%2Ftree%2Fmaster%2FOvmfPkg%2FPlatformCI&data=04%7C01%7Cbret.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608d8df354b05%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504767278995847%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=NUhAJ8zkITwPkvCrUqb7ZogSQTJuBXv0eL6Au0sYC2M%3D&reserved=0
>
> I don't know how I can determine and drive the whole "test plan" locally.
>
> When I submit a PR to github, "something" generates (say) 83 test cases.
> How can I run that "something" locally?
>
> Assume I have the following:
> - a Linux builder VM, with up-to-date tools,
> - a Windows builder VM, with up-to-date tools,
> - a topic branch, on top of "master".
>
> How do I generate and run those same 83 tests, locally?
>
> It has happened to me that all my local CI steps succeeded (meaning both
> the Windows and the Linux builder), but github still rejected the PR --
> the reason was that I failed to think of a particular test that github
> would run. Composing 83 "stuart" command lines isn't something I'd like
> to do manually.

FWIW it looks like people have been looking for similar functionality of
testing a pipeline locally, and not found it.

e.g.:
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fazure-pipelines-agent%2Fissues%2F1764&data=04%7C01%7Cbret.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608d8df354b05%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504767278995847%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=XMxZesyKxVN9Wg3SyRpUbnnmKzojevHCqQhzQ1Pc4%2Bg%3D&reserved=0
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdevelopercommunity.visualstudio.com%2Ft%2Fyaml-how-to-test-yaml-locally-before-commit%2F1302759&data=04%7C01%7Cbret.barkelew%40microsoft.com%7Ce4e5493578cf45b804c608d8df354b05%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637504767278995847%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=gwGZAQJAIXIS0LODFpglrCf5Ua2JeMnNm3A3auCrhx8%3D&reserved=0

--
Rebecca Cran







-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#72438): https://edk2.groups.io/g/devel/message/72438
Mute This Topic: https://groups.io/mt/81083112/21656
Group Owner: devel+ow...@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-