From: Pierre Gondois <pierre.gond...@arm.com> To enable CI support of the tianocore/edk2-platforms repository, add YML configuration files used to run Continuous Integration (CI) checks on Azure Pipelines agents.
These files are copies of the files from the tianocore/edk2 repository. Any modification to the tianocore/edk2 files must be reflected on the tianocore/edk2-platforms copies. The following files have been modified: - .azurepipelines/templates/platform-build-run-steps.yml - .azurepipelines/templates/pr-gate-build-job.yml - .azurepipelines/templates/pr-gate-steps.yml The sections modified are marked with the following comments: -EDK2_PLATFORMS_MODIF_START -EDK2_PLATFORMS_MODIF_END Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3509 Change-Id: I79cc9f5358fe57ad36dedf23054c7bb0bc0246a6 Cc: Ard Biesheuvel <ardb+tianoc...@kernel.org> Cc: Leif Lindholm <l...@nuviainc.com> Cc: Sean Brogan <sean.bro...@microsoft.com> Cc: Bret Barkelew <bret.barke...@microsoft.com> Cc: Michael D Kinney <michael.d.kin...@intel.com> Cc: Liming Gao <gaolim...@byosoft.com.cn> Cc: Sami Mujawar <sami.muja...@arm.com> Signed-off-by: Sami Mujawar <sami.muja...@arm.com> Signed-off-by: Pierre Gondois <pierre.gond...@arm.com> --- Notes: v3: - Align with the latest version in the edk2 repository. [Sean] - Replace Readme(s)'s content with a reference to the edk2 repository. [Sean] - Add step to checkout edk2's latest master in the CI. [Pierre] v4: - Checkout edk2 repository in Ubuntu-PatchCheck.yml to get the PatchCheck.py script. [Pierre] .azurepipelines/ReadMe.md | 5 + .azurepipelines/Ubuntu-GCC5.yml | 21 +++ .azurepipelines/Ubuntu-PatchCheck.yml | 45 ++++++ .azurepipelines/Windows-VS2019.yml | 20 +++ .azurepipelines/templates/ReadMe.md | 5 + .../templates/basetools-build-steps.yml | 37 +++++ .../templates/platform-build-run-steps.yml | 151 ++++++++++++++++++ .../templates/pr-gate-build-job.yml | 43 +++++ .azurepipelines/templates/pr-gate-steps.yml | 149 +++++++++++++++++ .../templates/spell-check-prereq-steps.yml | 22 +++ .pytool/Readme.md | 9 ++ 11 files changed, 507 insertions(+) create mode 100644 .azurepipelines/ReadMe.md create mode 100644 .azurepipelines/Ubuntu-GCC5.yml create mode 100644 .azurepipelines/Ubuntu-PatchCheck.yml create mode 100644 .azurepipelines/Windows-VS2019.yml create mode 100644 .azurepipelines/templates/ReadMe.md create mode 100644 .azurepipelines/templates/basetools-build-steps.yml create mode 100644 .azurepipelines/templates/platform-build-run-steps.yml create mode 100644 .azurepipelines/templates/pr-gate-build-job.yml create mode 100644 .azurepipelines/templates/pr-gate-steps.yml create mode 100644 .azurepipelines/templates/spell-check-prereq-steps.yml diff --git a/.azurepipelines/ReadMe.md b/.azurepipelines/ReadMe.md new file mode 100644 index 000000000000..de69a4ca220d --- /dev/null +++ b/.azurepipelines/ReadMe.md @@ -0,0 +1,5 @@ +EDK2_PLATFORMS_MODIF_START +As the content of this folder has been imported from the tianocore repository at: +https://github.com/tianocore/edk2 +Please use the Readme.md that can be found there. +EDK2_PLATFORMS_MODIF_END diff --git a/.azurepipelines/Ubuntu-GCC5.yml b/.azurepipelines/Ubuntu-GCC5.yml new file mode 100644 index 000000000000..3760c6efe105 --- /dev/null +++ b/.azurepipelines/Ubuntu-GCC5.yml @@ -0,0 +1,21 @@ +## @file +# Azure Pipeline build file for a build using ubuntu and GCC5 +# +# Copyright (c) Microsoft Corporation. +# Copyright (c) 2020, Hewlett Packard Enterprise Development LP. All rights reserved.<BR> +# SPDX-License-Identifier: BSD-2-Clause-Patent +## +trigger: +- master +- stable/* +pr: +- master +- stable/* + +jobs: +- template: templates/pr-gate-build-job.yml + parameters: + tool_chain_tag: 'GCC5' + vm_image: 'ubuntu-latest' + arch_list: "IA32,X64,ARM,AARCH64,RISCV64" + diff --git a/.azurepipelines/Ubuntu-PatchCheck.yml b/.azurepipelines/Ubuntu-PatchCheck.yml new file mode 100644 index 000000000000..7b565a25559c --- /dev/null +++ b/.azurepipelines/Ubuntu-PatchCheck.yml @@ -0,0 +1,45 @@ +## @file +# Azure Pipielines YML file that evalues the patch series in a PR using the +# python script BaseTools/Scripts/PatchCheck.py. +# +# NOTE: This example monitors pull requests against the edk2-ci branch. Most +# environments would replace 'edk2-ci' with 'master'. +# +# Copyright (c) 2021, Arm Limited. All rights reserved.<BR> +# Copyright (c) 2019 - 2020, Intel Corporation. All rights reserved.<BR> +# SPDX-License-Identifier: BSD-2-Clause-Patent +# +# https://github.com/tianocore +# +## + +trigger: none + +pr: +- master +- stable/* + +pool: + vmImage: 'ubuntu-latest' + +steps: +- checkout: self + clean: true + +# EDK2_PLATFORMS_MODIF_START: + # As PatchCheck.py is in the edk2 repository, checkout the repo. +- script: git submodule update --init --remote --checkout edk2 + displayName: Checkout edk2's latest master +# EDK2_PLATFORMS_MODIF_END + +- task: UsePythonVersion@0 + inputs: + versionSpec: '3.7.x' + architecture: 'x64' + +# EDK2_PLATFORMS_MODIF_START: +- script: | + git fetch origin $(System.PullRequest.TargetBranch):$(System.PullRequest.TargetBranch) + python edk2/BaseTools/Scripts/PatchCheck.py $(System.PullRequest.TargetBranch)..$(System.PullRequest.SourceCommitId) + displayName: 'Use PatchCheck.py to verify patch series in pull request' +# EDK2_PLATFORMS_MODIF_END diff --git a/.azurepipelines/Windows-VS2019.yml b/.azurepipelines/Windows-VS2019.yml new file mode 100644 index 000000000000..22f2d88c2c6a --- /dev/null +++ b/.azurepipelines/Windows-VS2019.yml @@ -0,0 +1,20 @@ +## @file +# Azure Pipeline build file for a build using Windows and VS2019 +# +# Copyright (c) Microsoft Corporation. +# SPDX-License-Identifier: BSD-2-Clause-Patent +## +trigger: +- master +- stable/* + +pr: +- master +- stable/* + +jobs: +- template: templates/pr-gate-build-job.yml + parameters: + tool_chain_tag: 'VS2019' + vm_image: 'windows-latest' + arch_list: "IA32,X64" diff --git a/.azurepipelines/templates/ReadMe.md b/.azurepipelines/templates/ReadMe.md new file mode 100644 index 000000000000..de69a4ca220d --- /dev/null +++ b/.azurepipelines/templates/ReadMe.md @@ -0,0 +1,5 @@ +EDK2_PLATFORMS_MODIF_START +As the content of this folder has been imported from the tianocore repository at: +https://github.com/tianocore/edk2 +Please use the Readme.md that can be found there. +EDK2_PLATFORMS_MODIF_END diff --git a/.azurepipelines/templates/basetools-build-steps.yml b/.azurepipelines/templates/basetools-build-steps.yml new file mode 100644 index 000000000000..d8c108c6e212 --- /dev/null +++ b/.azurepipelines/templates/basetools-build-steps.yml @@ -0,0 +1,37 @@ +## @file +# File templates/basetools-build-job.yml +# +# template file to build basetools +# +# Copyright (c) Microsoft Corporation. +# SPDX-License-Identifier: BSD-2-Clause-Patent +## +parameters: + tool_chain_tag: '' + +steps: +- ${{ if contains(parameters.tool_chain_tag, 'GCC') }}: + - bash: sudo apt-get update + displayName: Update apt + condition: and(gt(variables.pkg_count, 0), succeeded()) + + - bash: sudo apt-get install gcc g++ make uuid-dev + displayName: Install required tools + condition: and(gt(variables.pkg_count, 0), succeeded()) + +- task: CmdLine@1 + displayName: Build Base Tools from source + inputs: + filename: python + arguments: BaseTools/Edk2ToolsBuild.py -t ${{ parameters.tool_chain_tag }} + condition: and(gt(variables.pkg_count, 0), succeeded()) + +- task: CopyFiles@2 + displayName: "Copy base tools build log" + inputs: + targetFolder: '$(Build.ArtifactStagingDirectory)' + SourceFolder: 'BaseTools/BaseToolsBuild' + contents: | + BASETOOLS_BUILD*.* + flattenFolders: true + condition: and(gt(variables.pkg_count, 0), succeededOrFailed()) diff --git a/.azurepipelines/templates/platform-build-run-steps.yml b/.azurepipelines/templates/platform-build-run-steps.yml new file mode 100644 index 000000000000..960a11ced5ee --- /dev/null +++ b/.azurepipelines/templates/platform-build-run-steps.yml @@ -0,0 +1,151 @@ + +## @file +# File steps.yml +# +# template file containing the steps to build +# +# Copyright (c) 2021, Arm Limited. All rights reserved.<BR> +# Copyright (c) Microsoft Corporation. +# SPDX-License-Identifier: BSD-2-Clause-Patent +## + +parameters: +- name: tool_chain_tag + type: string + default: '' +- name: build_pkg + type: string + default: '' +- name: build_target + type: string + default: '' +- name: build_arch + type: string + default: '' +- name: build_file + type: string + default: '' +- name: build_flags + type: string + default: '' +- name: run_flags + type: string + default: '' + +- name: extra_install_step + type: stepList + default: [] + +steps: +- checkout: self + clean: true + fetchDepth: 1 + +- task: UsePythonVersion@0 + inputs: + versionSpec: "3.8.x" + architecture: "x64" + +- script: pip install -r pip-requirements.txt --upgrade + displayName: 'Install/Upgrade pip modules' + +# Set default +- bash: echo "##vso[task.setvariable variable=pkg_count]${{ 1 }}" + +# Fetch the target branch so that pr_eval can diff them. +# Seems like azure pipelines/github changed checkout process in nov 2020. +- script: git fetch origin $(System.PullRequest.targetBranch) + displayName: fetch target branch + condition: eq(variables['Build.Reason'], 'PullRequest') + +# trim the package list if this is a PR +- task: CmdLine@1 + displayName: Check if ${{ parameters.build_pkg }} need testing + inputs: + filename: stuart_pr_eval + arguments: -c ${{ parameters.build_file }} -t ${{ parameters.build_target}} -a ${{ parameters.build_arch}} --pr-target origin/$(System.PullRequest.targetBranch) --output-count-format-string "##vso[task.setvariable variable=pkg_count;isOutpout=true]{pkgcount}" + condition: eq(variables['Build.Reason'], 'PullRequest') + + # Setup repo +- task: CmdLine@1 + displayName: Setup + inputs: + filename: stuart_setup + arguments: -c ${{ parameters.build_file }} TOOL_CHAIN_TAG=${{ parameters.tool_chain_tag}} -t ${{ parameters.build_target}} -a ${{ parameters.build_arch}} ${{ parameters.build_flags}} + condition: and(gt(variables.pkg_count, 0), succeeded()) + +# EDK2_PLATFORMS_MODIF_START: + # As edk2-platforms may rely on new edk2 modifications, checkout edk2's latest master +- script: git submodule update --remote --checkout edk2 + displayName: Checkout edk2's latest master + condition: and(gt(variables.pkg_count, 0), succeeded()) +# EDK2_PLATFORMS_MODIF_END + +# Stuart Update +- task: CmdLine@1 + displayName: Update + inputs: + filename: stuart_update + arguments: -c ${{ parameters.build_file }} TOOL_CHAIN_TAG=${{ parameters.tool_chain_tag}} -t ${{ parameters.build_target}} -a ${{ parameters.build_arch}} ${{ parameters.build_flags}} + condition: and(gt(variables.pkg_count, 0), succeeded()) + +# EDK2_PLATFORMS_MODIF_START: +# The base tools are imported in .pytool/CISettings.py via the 'edk2basetools' python module. +# # build basetools +# # do this after setup and update so that code base dependencies +# # are all resolved. +# - template: basetools-build-steps.yml +# parameters: +# tool_chain_tag: ${{ parameters.tool_chain_tag }} +# EDK2_PLATFORMS_MODIF_END + +# Potential Extra steps +- ${{ parameters.extra_install_step }} + +# Build +- task: CmdLine@1 + displayName: Build + inputs: + filename: stuart_build + arguments: -c ${{ parameters.build_file }} TOOL_CHAIN_TAG=${{ parameters.tool_chain_tag}} TARGET=${{ parameters.build_target}} -a ${{ parameters.build_arch}} ${{ parameters.build_flags}} + condition: and(gt(variables.pkg_count, 0), succeeded()) + +# Run +- task: CmdLine@1 + displayName: Run to shell + inputs: + filename: stuart_build + arguments: -c ${{ parameters.build_file }} TOOL_CHAIN_TAG=${{ parameters.tool_chain_tag}} TARGET=${{ parameters.build_target}} -a ${{ parameters.build_arch}} ${{ parameters.build_flags}} ${{ parameters.run_flags }} --FlashOnly + condition: and(and(gt(variables.pkg_count, 0), succeeded()), eq(variables['Run'], true)) + timeoutInMinutes: 1 + +# Copy the build logs to the artifact staging directory +- task: CopyFiles@2 + displayName: "Copy build logs" + inputs: + targetFolder: "$(Build.ArtifactStagingDirectory)" + SourceFolder: "Build" + contents: | + BUILDLOG_*.txt + BUILDLOG_*.md + CI_*.txt + CI_*.md + CISETUP.txt + SETUPLOG.txt + UPDATE_LOG.txt + PREVALLOG.txt + TestSuites.xml + **/BUILD_TOOLS_REPORT.html + **/OVERRIDELOG.TXT + BASETOOLS_BUILD*.* + flattenFolders: true + condition: succeededOrFailed() + +# Publish build artifacts to Azure Artifacts/TFS or a file share +- task: PublishBuildArtifacts@1 + continueOnError: true + displayName: "Publish build logs" + inputs: + pathtoPublish: "$(Build.ArtifactStagingDirectory)" + artifactName: "Build Logs $(System.JobName)" + condition: succeededOrFailed() diff --git a/.azurepipelines/templates/pr-gate-build-job.yml b/.azurepipelines/templates/pr-gate-build-job.yml new file mode 100644 index 000000000000..9bb8e2819793 --- /dev/null +++ b/.azurepipelines/templates/pr-gate-build-job.yml @@ -0,0 +1,43 @@ +## @file +# File templates/pr-gate-build-job.yml +# +# template file used to build supported packages. +# +# Copyright (c) Microsoft Corporation. +# Copyright (c) 2020 - 2021, ARM Limited. All rights reserved.<BR> +# SPDX-License-Identifier: BSD-2-Clause-Patent +## + +parameters: + tool_chain_tag: '' + vm_image: '' + arch_list: '' + +# Build step +jobs: + +- job: Build_${{ parameters.tool_chain_tag }} + +# EDK2_PLATFORMS_MODIF_START: +# Build edk2-platforms packages + #Use matrix to speed up the build process + strategy: + matrix: + TARGET_ARM_PLATFORMS: + Build.Pkgs: 'JunoPkg' + Build.Targets: 'DEBUG,RELEASE,NO-TARGET,NOOPT' +# EDK2_PLATFORMS_MODIF_END + + workspace: + clean: all + + pool: + vmImage: ${{ parameters.vm_image }} + + steps: + - template: pr-gate-steps.yml + parameters: + tool_chain_tag: ${{ parameters.tool_chain_tag }} + build_pkgs: $(Build.Pkgs) + build_targets: $(Build.Targets) + build_archs: ${{ parameters.arch_list }} diff --git a/.azurepipelines/templates/pr-gate-steps.yml b/.azurepipelines/templates/pr-gate-steps.yml new file mode 100644 index 000000000000..1936d5a10780 --- /dev/null +++ b/.azurepipelines/templates/pr-gate-steps.yml @@ -0,0 +1,149 @@ +## @file +# File templates/pr-gate-steps.yml +# +# template file containing the steps to build +# +# Copyright (c) 2021, Arm Limited. All rights reserved.<BR> +# Copyright (c) Microsoft Corporation. +# SPDX-License-Identifier: BSD-2-Clause-Patent +## + +parameters: + tool_chain_tag: '' + build_pkgs: '' + build_targets: '' + build_archs: '' + +steps: +- checkout: self + clean: true + fetchDepth: 1 + +- task: UsePythonVersion@0 + inputs: + versionSpec: '3.8.x' + architecture: 'x64' + +- script: pip install -r pip-requirements.txt --upgrade + displayName: 'Install/Upgrade pip modules' + +# Set default +- bash: | + echo "##vso[task.setvariable variable=pkgs_to_build]${{ parameters.build_pkgs }}" + echo "##vso[task.setvariable variable=pkg_count]${{ 1 }}" + +# Fetch the target branch so that pr_eval can diff them. +# Seems like azure pipelines/github changed checkout process in nov 2020. +- script: git fetch origin $(System.PullRequest.targetBranch) + displayName: fetch target branch + condition: eq(variables['Build.Reason'], 'PullRequest') + +# trim the package list if this is a PR +- task: CmdLine@1 + displayName: Check if ${{ parameters.build_pkgs }} need testing + inputs: + filename: stuart_pr_eval + arguments: -c .pytool/CISettings.py -p ${{ parameters.build_pkgs }} --pr-target origin/$(System.PullRequest.targetBranch) --output-csv-format-string "##vso[task.setvariable variable=pkgs_to_build;isOutpout=true]{pkgcsv}" --output-count-format-string "##vso[task.setvariable variable=pkg_count;isOutpout=true]{pkgcount}" + condition: eq(variables['Build.Reason'], 'PullRequest') + +# install spell check prereqs +- template: spell-check-prereq-steps.yml + +# Build repo +- task: CmdLine@1 + displayName: Setup ${{ parameters.build_pkgs }} ${{ parameters.build_archs}} + inputs: + filename: stuart_setup + arguments: -c .pytool/CISettings.py -p $(pkgs_to_build) -t ${{ parameters.build_targets}} -a ${{ parameters.build_archs}} TOOL_CHAIN_TAG=${{ parameters.tool_chain_tag}} + condition: and(gt(variables.pkg_count, 0), succeeded()) + +# EDK2_PLATFORMS_MODIF_START: + # As edk2-platforms may rely on new edk2 modifications, checkout edk2's latest master +- script: git submodule update --remote --checkout edk2 + displayName: Checkout edk2's latest master + condition: and(gt(variables.pkg_count, 0), succeeded()) +# EDK2_PLATFORMS_MODIF_END + +- task: CmdLine@1 + displayName: Update ${{ parameters.build_pkgs }} ${{ parameters.build_archs}} + inputs: + filename: stuart_update + arguments: -c .pytool/CISettings.py -p $(pkgs_to_build) -t ${{ parameters.build_targets}} -a ${{ parameters.build_archs}} TOOL_CHAIN_TAG=${{ parameters.tool_chain_tag}} + condition: and(gt(variables.pkg_count, 0), succeeded()) + +# build basetools +# do this after setup and update so that code base dependencies +# are all resolved. +# EDK2_PLATFORMS_MODIF_START: +# The base tools are imported in .pytool/CISettings.py via the 'edk2basetools' python module. +# - template: basetools-build-steps.yml +# parameters: +# tool_chain_tag: ${{ parameters.tool_chain_tag }} +# EDK2_PLATFORMS_MODIF_END + +- task: CmdLine@1 + displayName: Build and Test ${{ parameters.build_pkgs }} ${{ parameters.build_archs}} + inputs: + filename: stuart_ci_build + arguments: -c .pytool/CISettings.py -p $(pkgs_to_build) -t ${{ parameters.build_targets}} -a ${{ parameters.build_archs}} TOOL_CHAIN_TAG=${{ parameters.tool_chain_tag}} + condition: and(gt(variables.pkg_count, 0), succeeded()) + +# Publish Test Results to Azure Pipelines/TFS +- task: PublishTestResults@2 + displayName: 'Publish junit test results' + continueOnError: true + condition: and( succeededOrFailed(),gt(variables.pkg_count, 0)) + inputs: + testResultsFormat: 'JUnit' # Options: JUnit, NUnit, VSTest, xUnit + testResultsFiles: 'Build/TestSuites.xml' + #searchFolder: '$(System.DefaultWorkingDirectory)' # Optional + mergeTestResults: true # Optional + testRunTitle: $(System.JobName) # Optional + #buildPlatform: # Optional + #buildConfiguration: # Optional + publishRunAttachments: true # Optional + +# Publish Test Results to Azure Pipelines/TFS +- task: PublishTestResults@2 + displayName: 'Publish host based test results for $(System.JobName)' + continueOnError: true + condition: and( succeededOrFailed(), gt(variables.pkg_count, 0)) + inputs: + testResultsFormat: 'JUnit' # Options: JUnit, NUnit, VSTest, xUnit + testResultsFiles: 'Build/**/*.result.xml' + #searchFolder: '$(System.DefaultWorkingDirectory)' # Optional + mergeTestResults: false # Optional + testRunTitle: ${{ parameters.build_pkgs }} # Optional + #buildPlatform: # Optional + #buildConfiguration: # Optional + publishRunAttachments: true # Optional + +# Copy the build logs to the artifact staging directory +- task: CopyFiles@2 + displayName: "Copy build logs" + inputs: + targetFolder: '$(Build.ArtifactStagingDirectory)' + SourceFolder: 'Build' + contents: | + BUILDLOG_*.txt + BUILDLOG_*.md + CI_*.txt + CI_*.md + CISETUP.txt + SETUPLOG.txt + UPDATE_LOG.txt + PREVALLOG.txt + TestSuites.xml + **/BUILD_TOOLS_REPORT.html + **/OVERRIDELOG.TXT + flattenFolders: true + condition: succeededOrFailed() + +# Publish build artifacts to Azure Artifacts/TFS or a file share +- task: PublishBuildArtifacts@1 + continueOnError: true + displayName: "Publish build logs" + inputs: + pathtoPublish: '$(Build.ArtifactStagingDirectory)' + artifactName: 'Build Logs $(System.JobName)' + condition: succeededOrFailed() diff --git a/.azurepipelines/templates/spell-check-prereq-steps.yml b/.azurepipelines/templates/spell-check-prereq-steps.yml new file mode 100644 index 000000000000..98ee3cfa6bc6 --- /dev/null +++ b/.azurepipelines/templates/spell-check-prereq-steps.yml @@ -0,0 +1,22 @@ +## @file +# File templates/spell-check-prereq-steps.yml +# +# template file used to install spell checking prerequisits +# +# Copyright (c) Microsoft Corporation. +# SPDX-License-Identifier: BSD-2-Clause-Patent +## + +parameters: + none: '' + +steps: +- task: NodeTool@0 + inputs: + versionSpec: '14.x' + #checkLatest: false # Optional + condition: and(gt(variables.pkg_count, 0), succeeded()) + +- script: npm install -g cspell + displayName: 'Install cspell npm' + condition: and(gt(variables.pkg_count, 0), succeeded()) diff --git a/.pytool/Readme.md b/.pytool/Readme.md index 9e59b30043fc..0d740caa40a8 100644 --- a/.pytool/Readme.md +++ b/.pytool/Readme.md @@ -9,6 +9,15 @@ For more detailed status look at the test results of the latest CI run on the repo readme. +## edk2 submodule + +It is possible that the edk2-platforms repository relies on new modifications +in the edk2 repository. The edk2-platforms CI uses the edk2 submodule. Thus, +the edk2 submodule might need to be updated to run the CI properly. + +To rebase the edk2 submodule on the latest master, run: +* `git submodule update --remote --rebase edk2` + ## Readme As the content of the .pytool folder has been imported from the tianocore repository at: -- 2.17.1 -=-=-=-=-=-=-=-=-=-=-=- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#82705): https://edk2.groups.io/g/devel/message/82705 Mute This Topic: https://groups.io/mt/86606570/21656 Group Owner: devel+ow...@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [arch...@mail-archive.com] -=-=-=-=-=-=-=-=-=-=-=-
