This is an automated email from the ASF dual-hosted git repository.
ruifengz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new ad7e1bff534 [SPARK-46150][INFRA] Combine python codegen check and
protobuf braking change
ad7e1bff534 is described below
commit ad7e1bff534a1a61aebe587f8cd850d34f4fa190
Author: Ruifeng Zheng <ruife...@apache.org>
AuthorDate: Wed Nov 29 14:43:52 2023 +0800
[SPARK-46150][INFRA] Combine python codegen check and protobuf braking
change
### What changes were proposed in this pull request?
Combine python codegen check and protobuf braking change
### Why are the changes needed?
1, `uses: bufbuild/buf-setup-actionv1` for breaking change detection always
install the latest version of `buf`, then we no longer need to manually upgrade
the `buf` installation for python code gen check;
```
Run bufbuild/buf-setup-actionv1
with:
github_token: ***
version:
1.[2](https://github.com/zhengruifeng/spark/actions/runs/7027951456/job/19123179757#step:4:2)8.1
buf_domain: buf.build
Setting up buf version "1.28.1"
Resolving the download URL for the current platform...
Downloading buf version "1.28.1" from
https://github.com/bufbuild/buf/releases/download/v1.28.1/buf-Linux-x86_64.tar.gz
Successfully downloaded buf version "1.28.1" from
https://github.com/bufbuild/buf/releases/download/v1.28.1/buf-Linux-x86_64.tar.gz
Extracting buf...
/usr/bin/tar xz --warning=no-unknown-keyword --overwrite -C
/home/runner/work/_temp/94eec429-14[3](https://github.com/zhengruifeng/spark/actions/runs/7027951456/job/19123179757#step:4:3)2-[4](https://github.com/zhengruifeng/spark/actions/runs/7027951456/job/19123179757#step:4:4)0[5](https://github.com/zhengruifeng/spark/actions/runs/7027951456/job/19123179757#step:4:5)9-8c21-8e9a[6](https://github.com/zhengruifeng/spark/actions/runs/7027951456/job/19123179757#step:4:7)140902f
-f /home [...]
Successfully extracted buf to
/home/runner/work/_temp/[9](https://github.com/zhengruifeng/spark/actions/runs/7027951456/job/19123179757#step:4:10)4eec429-1432-4059-8c21-8e9a6140902f
Adding buf to the cache...
Successfully cached buf to /opt/hostedtoolcache/buf/1.28.1/x64
Adding buf binary to PATH
Successfully setup buf version 1.28.1
1.28.1
```
2, to simplify the job `lint`
### Does this PR introduce _any_ user-facing change?
no, infra-only
### How was this patch tested?
ci
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #44067 from zhengruifeng/infra_move_py_codegen_check.
Authored-by: Ruifeng Zheng <ruife...@apache.org>
Signed-off-by: Ruifeng Zheng <ruife...@apache.org>
---
.github/workflows/build_and_test.yml | 33 ++++++++++++++++++++++-----------
1 file changed, 22 insertions(+), 11 deletions(-)
diff --git a/.github/workflows/build_and_test.yml
b/.github/workflows/build_and_test.yml
index 14018fee473..4612b504ccd 100644
--- a/.github/workflows/build_and_test.yml
+++ b/.github/workflows/build_and_test.yml
@@ -98,7 +98,7 @@ jobs:
\"java-other-versions\": \"true\",
\"lint\" : \"true\",
\"k8s-integration-tests\" : \"true\",
- \"breaking-changes-buf\" : \"true\",
+ \"buf\" : \"true\",
\"ui\" : \"true\",
}"
echo $precondition # For debugging
@@ -557,11 +557,10 @@ jobs:
name: test-results-sparkr--${{ inputs.java }}-${{ inputs.hadoop
}}-hive2.3
path: "**/target/test-reports/*.xml"
- breaking-changes-buf:
+ buf:
needs: [precondition]
- if: (!cancelled()) &&
fromJson(needs.precondition.outputs.required).breaking-changes-buf == 'true'
- # Change 'branch-3.5' to 'branch-4.0' in master branch after cutting
branch-4.0 branch.
- name: Breaking change detection with Buf (branch-3.5)
+ if: (!cancelled()) && fromJson(needs.precondition.outputs.required).buf ==
'true'
+ name: Protobuf breaking change detection and Python CodeGen check
runs-on: ubuntu-22.04
steps:
- name: Checkout Spark repository
@@ -580,12 +579,22 @@ jobs:
uses: bufbuild/buf-setup-action@v1
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
- - name: Detect breaking changes
+ # Change 'branch-3.5' to 'branch-4.0' in master branch after cutting
branch-4.0 branch.
+ - name: Breaking change detection against branch-3.5
uses: bufbuild/buf-breaking-action@v1
with:
input: connector/connect/common/src/main
against:
'https://github.com/apache/spark.git#branch=branch-3.5,subdir=connector/connect/common/src/main'
-
+ - name: Install Python 3.9
+ uses: actions/setup-python@v4
+ with:
+ python-version: '3.9'
+ - name: Install dependencies for Python CodeGen check
+ run: |
+ python3.9 -m pip install 'black==23.9.1' 'protobuf==4.25.1'
'mypy==0.982' 'mypy-protobuf==3.3.0'
+ python3.9 -m pip list
+ - name: Python CodeGen check
+ run: ./dev/connect-check-protos.py
# Static analysis, and documentation build
lint:
@@ -697,8 +706,9 @@ jobs:
python3.9 -m pip install 'pandas-stubs==1.2.0.53' ipython
'grpcio==1.59.3' 'grpc-stubs==1.24.11' 'googleapis-common-protos-stubs==2.2.0'
- name: Python linter
run: PYTHON_EXECUTABLE=python3.9 ./dev/lint-python
- - name: Install dependencies for Python code generation check
- if: inputs.branch != 'branch-3.3' && inputs.branch != 'branch-3.4'
+ # Should delete this section after SPARK 3.5 EOL.
+ - name: Install dependencies for Python code generation check for
branch-3.5
+ if: inputs.branch == 'branch-3.5'
run: |
# See more in "Installation"
https://docs.buf.build/installation#tarball
curl -LO
https://github.com/bufbuild/buf/releases/download/v1.28.1/buf-Linux-x86_64.tar.gz
@@ -706,8 +716,9 @@ jobs:
tar -xvzf buf-Linux-x86_64.tar.gz -C $HOME/buf --strip-components 1
rm buf-Linux-x86_64.tar.gz
python3.9 -m pip install 'protobuf==4.25.1' 'mypy-protobuf==3.3.0'
- - name: Python code generation check
- if: inputs.branch != 'branch-3.3' && inputs.branch != 'branch-3.4'
+ # Should delete this section after SPARK 3.5 EOL.
+ - name: Python code generation check for branch-3.5
+ if: inputs.branch == 'branch-3.5'
run: if test -f ./dev/connect-check-protos.py; then
PATH=$PATH:$HOME/buf/bin PYTHON_EXECUTABLE=python3.9
./dev/connect-check-protos.py; fi
# Should delete this section after SPARK 3.5 EOL.
- name: Install JavaScript linter dependencies for branch-3.3, branch-3.4,
branch-3.5
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
