This is an automated email from the ASF dual-hosted git repository. ruifengz pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new ad7e1bff534 [SPARK-46150][INFRA] Combine python codegen check and protobuf braking change ad7e1bff534 is described below commit ad7e1bff534a1a61aebe587f8cd850d34f4fa190 Author: Ruifeng Zheng <ruife...@apache.org> AuthorDate: Wed Nov 29 14:43:52 2023 +0800 [SPARK-46150][INFRA] Combine python codegen check and protobuf braking change ### What changes were proposed in this pull request? Combine python codegen check and protobuf braking change ### Why are the changes needed? 1, `uses: bufbuild/buf-setup-actionv1` for breaking change detection always install the latest version of `buf`, then we no longer need to manually upgrade the `buf` installation for python code gen check; ``` Run bufbuild/buf-setup-actionv1 with: github_token: *** version: 1.[2](https://github.com/zhengruifeng/spark/actions/runs/7027951456/job/19123179757#step:4:2)8.1 buf_domain: buf.build Setting up buf version "1.28.1" Resolving the download URL for the current platform... Downloading buf version "1.28.1" from https://github.com/bufbuild/buf/releases/download/v1.28.1/buf-Linux-x86_64.tar.gz Successfully downloaded buf version "1.28.1" from https://github.com/bufbuild/buf/releases/download/v1.28.1/buf-Linux-x86_64.tar.gz Extracting buf... /usr/bin/tar xz --warning=no-unknown-keyword --overwrite -C /home/runner/work/_temp/94eec429-14[3](https://github.com/zhengruifeng/spark/actions/runs/7027951456/job/19123179757#step:4:3)2-[4](https://github.com/zhengruifeng/spark/actions/runs/7027951456/job/19123179757#step:4:4)0[5](https://github.com/zhengruifeng/spark/actions/runs/7027951456/job/19123179757#step:4:5)9-8c21-8e9a[6](https://github.com/zhengruifeng/spark/actions/runs/7027951456/job/19123179757#step:4:7)140902f -f /home [...] Successfully extracted buf to /home/runner/work/_temp/[9](https://github.com/zhengruifeng/spark/actions/runs/7027951456/job/19123179757#step:4:10)4eec429-1432-4059-8c21-8e9a6140902f Adding buf to the cache... Successfully cached buf to /opt/hostedtoolcache/buf/1.28.1/x64 Adding buf binary to PATH Successfully setup buf version 1.28.1 1.28.1 ``` 2, to simplify the job `lint` ### Does this PR introduce _any_ user-facing change? no, infra-only ### How was this patch tested? ci ### Was this patch authored or co-authored using generative AI tooling? no Closes #44067 from zhengruifeng/infra_move_py_codegen_check. Authored-by: Ruifeng Zheng <ruife...@apache.org> Signed-off-by: Ruifeng Zheng <ruife...@apache.org> --- .github/workflows/build_and_test.yml | 33 ++++++++++++++++++++++----------- 1 file changed, 22 insertions(+), 11 deletions(-) diff --git a/.github/workflows/build_and_test.yml b/.github/workflows/build_and_test.yml index 14018fee473..4612b504ccd 100644 --- a/.github/workflows/build_and_test.yml +++ b/.github/workflows/build_and_test.yml @@ -98,7 +98,7 @@ jobs: \"java-other-versions\": \"true\", \"lint\" : \"true\", \"k8s-integration-tests\" : \"true\", - \"breaking-changes-buf\" : \"true\", + \"buf\" : \"true\", \"ui\" : \"true\", }" echo $precondition # For debugging @@ -557,11 +557,10 @@ jobs: name: test-results-sparkr--${{ inputs.java }}-${{ inputs.hadoop }}-hive2.3 path: "**/target/test-reports/*.xml" - breaking-changes-buf: + buf: needs: [precondition] - if: (!cancelled()) && fromJson(needs.precondition.outputs.required).breaking-changes-buf == 'true' - # Change 'branch-3.5' to 'branch-4.0' in master branch after cutting branch-4.0 branch. - name: Breaking change detection with Buf (branch-3.5) + if: (!cancelled()) && fromJson(needs.precondition.outputs.required).buf == 'true' + name: Protobuf breaking change detection and Python CodeGen check runs-on: ubuntu-22.04 steps: - name: Checkout Spark repository @@ -580,12 +579,22 @@ jobs: uses: bufbuild/buf-setup-action@v1 with: github_token: ${{ secrets.GITHUB_TOKEN }} - - name: Detect breaking changes + # Change 'branch-3.5' to 'branch-4.0' in master branch after cutting branch-4.0 branch. + - name: Breaking change detection against branch-3.5 uses: bufbuild/buf-breaking-action@v1 with: input: connector/connect/common/src/main against: 'https://github.com/apache/spark.git#branch=branch-3.5,subdir=connector/connect/common/src/main' - + - name: Install Python 3.9 + uses: actions/setup-python@v4 + with: + python-version: '3.9' + - name: Install dependencies for Python CodeGen check + run: | + python3.9 -m pip install 'black==23.9.1' 'protobuf==4.25.1' 'mypy==0.982' 'mypy-protobuf==3.3.0' + python3.9 -m pip list + - name: Python CodeGen check + run: ./dev/connect-check-protos.py # Static analysis, and documentation build lint: @@ -697,8 +706,9 @@ jobs: python3.9 -m pip install 'pandas-stubs==1.2.0.53' ipython 'grpcio==1.59.3' 'grpc-stubs==1.24.11' 'googleapis-common-protos-stubs==2.2.0' - name: Python linter run: PYTHON_EXECUTABLE=python3.9 ./dev/lint-python - - name: Install dependencies for Python code generation check - if: inputs.branch != 'branch-3.3' && inputs.branch != 'branch-3.4' + # Should delete this section after SPARK 3.5 EOL. + - name: Install dependencies for Python code generation check for branch-3.5 + if: inputs.branch == 'branch-3.5' run: | # See more in "Installation" https://docs.buf.build/installation#tarball curl -LO https://github.com/bufbuild/buf/releases/download/v1.28.1/buf-Linux-x86_64.tar.gz @@ -706,8 +716,9 @@ jobs: tar -xvzf buf-Linux-x86_64.tar.gz -C $HOME/buf --strip-components 1 rm buf-Linux-x86_64.tar.gz python3.9 -m pip install 'protobuf==4.25.1' 'mypy-protobuf==3.3.0' - - name: Python code generation check - if: inputs.branch != 'branch-3.3' && inputs.branch != 'branch-3.4' + # Should delete this section after SPARK 3.5 EOL. + - name: Python code generation check for branch-3.5 + if: inputs.branch == 'branch-3.5' run: if test -f ./dev/connect-check-protos.py; then PATH=$PATH:$HOME/buf/bin PYTHON_EXECUTABLE=python3.9 ./dev/connect-check-protos.py; fi # Should delete this section after SPARK 3.5 EOL. - name: Install JavaScript linter dependencies for branch-3.3, branch-3.4, branch-3.5 --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org