Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on PR #24426: URL: https://github.com/apache/flink/pull/24426#issuecomment-2205212666 Great, thanks @XComp, @HuangXingBo for the help -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on PR #24426: URL: https://github.com/apache/flink/pull/24426#issuecomment-2176882275 Hey @HuangXingBo, I have run two builds, one is with default using `auditwheel` (together with patchelf) and another without `auditwheel`. Both on the `manylinux2014` on Linux. - With auditwheel: https://github.com/morazow/flink/actions/runs/9566314731 - Without auditwheel: https://github.com/morazow/flink/actions/runs/9566526379 I also updated the build workflow to make explicit some of the steps, previously it was running auditwheel by default on linux. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on PR #24426: URL: https://github.com/apache/flink/pull/24426#issuecomment-2175766542 Hey @HuangXingBo, I think one of the runs above was with `manylinux2014` and without auditwheel step. But I will do another workflow run to create the wheels. I'll let you know -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
HuangXingBo commented on PR #24426: URL: https://github.com/apache/flink/pull/24426#issuecomment-2175616974 @morazow I am not sure whether our wheel package can be manylinux2014 without auditwheel. We can try to remove auditwheel to see the effect -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on PR #24426: URL: https://github.com/apache/flink/pull/24426#issuecomment-2175396160 Hello @HuangXingBo, thanks for the feedback! Is the current state fine? It uses the `manylinux2014` for Linux: ``` [tool.cibuildwheel.linux] archs = ["x86_64"] manylinux-x86_64-image = "manylinux2014" ``` Should I also keep the `auditwheel` steps? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
HuangXingBo commented on PR #24426: URL: https://github.com/apache/flink/pull/24426#issuecomment-2171014919 @morazow Since a lot of our dependencies are manyLinux2014, I think we can indeed build manyLinux2014. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on PR #24426: URL: https://github.com/apache/flink/pull/24426#issuecomment-2167770793 Thanks @HuangXingBo for the feedback! Should we still support `manylinux1`? It seems already end of life for some years -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1639647084
##
.github/workflows/nightly.yml:
##
@@ -28,69 +28,131 @@ jobs:
name: "Pre-compile Checks"
uses: ./.github/workflows/template.pre-compile-checks.yml
- java8:
-name: "Java 8"
-uses: ./.github/workflows/template.flink-ci.yml
-with:
- workflow-caller-id: java8
- environment: 'PROFILE="-Dinclude_hadoop_aws"'
- jdk_version: 8
-secrets:
- s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
- s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
- s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
- java11:
-name: "Java 11"
-uses: ./.github/workflows/template.flink-ci.yml
-with:
- workflow-caller-id: java11
- environment: 'PROFILE="-Dinclude_hadoop_aws -Djdk11 -Pjava11-target"'
- jdk_version: 11
-secrets:
- s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
- s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
- s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
- java17:
-name: "Java 17"
-uses: ./.github/workflows/template.flink-ci.yml
-with:
- workflow-caller-id: java17
- environment: 'PROFILE="-Dinclude_hadoop_aws -Djdk11 -Djdk17
-Pjava17-target"'
- jdk_version: 17
-secrets:
- s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
- s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
- s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
- java21:
-name: "Java 21"
-uses: ./.github/workflows/template.flink-ci.yml
-with:
- workflow-caller-id: java21
- environment: 'PROFILE="-Dinclude_hadoop_aws -Djdk11 -Djdk17 -Djdk21
-Pjava21-target"'
- jdk_version: 21
-secrets:
- s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
- s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
- s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
- hadoop313:
-name: "Hadoop 3.1.3"
-uses: ./.github/workflows/template.flink-ci.yml
-with:
- workflow-caller-id: hadoop313
- environment: 'PROFILE="-Dflink.hadoop.version=3.2.3
-Phadoop3-tests,hive3"'
- jdk_version: 8
-secrets:
- s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
- s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
- s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
- adaptive-scheduler:
-name: "AdaptiveScheduler"
-uses: ./.github/workflows/template.flink-ci.yml
-with:
- workflow-caller-id: adaptive-scheduler
- environment: 'PROFILE="-Penable-adaptive-scheduler"'
- jdk_version: 8
-secrets:
- s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
- s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
- s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+# java8:
Review Comment:
Yes, I will put it back once we finalize the PR, commented just to run to
create the wheels.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
XComp commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1639305664
##
.github/workflows/nightly.yml:
##
@@ -28,69 +28,131 @@ jobs:
name: "Pre-compile Checks"
uses: ./.github/workflows/template.pre-compile-checks.yml
- java8:
-name: "Java 8"
-uses: ./.github/workflows/template.flink-ci.yml
-with:
- workflow-caller-id: java8
- environment: 'PROFILE="-Dinclude_hadoop_aws"'
- jdk_version: 8
-secrets:
- s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
- s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
- s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
- java11:
-name: "Java 11"
-uses: ./.github/workflows/template.flink-ci.yml
-with:
- workflow-caller-id: java11
- environment: 'PROFILE="-Dinclude_hadoop_aws -Djdk11 -Pjava11-target"'
- jdk_version: 11
-secrets:
- s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
- s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
- s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
- java17:
-name: "Java 17"
-uses: ./.github/workflows/template.flink-ci.yml
-with:
- workflow-caller-id: java17
- environment: 'PROFILE="-Dinclude_hadoop_aws -Djdk11 -Djdk17
-Pjava17-target"'
- jdk_version: 17
-secrets:
- s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
- s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
- s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
- java21:
-name: "Java 21"
-uses: ./.github/workflows/template.flink-ci.yml
-with:
- workflow-caller-id: java21
- environment: 'PROFILE="-Dinclude_hadoop_aws -Djdk11 -Djdk17 -Djdk21
-Pjava21-target"'
- jdk_version: 21
-secrets:
- s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
- s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
- s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
- hadoop313:
-name: "Hadoop 3.1.3"
-uses: ./.github/workflows/template.flink-ci.yml
-with:
- workflow-caller-id: hadoop313
- environment: 'PROFILE="-Dflink.hadoop.version=3.2.3
-Phadoop3-tests,hive3"'
- jdk_version: 8
-secrets:
- s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
- s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
- s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
- adaptive-scheduler:
-name: "AdaptiveScheduler"
-uses: ./.github/workflows/template.flink-ci.yml
-with:
- workflow-caller-id: adaptive-scheduler
- environment: 'PROFILE="-Penable-adaptive-scheduler"'
- jdk_version: 8
-secrets:
- s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
- s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
- s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+# java8:
Review Comment:
That's just disabled code to fasten the test run. It will be reverted before
merging the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
HuangXingBo commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1639115041
##
.github/workflows/nightly.yml:
##
@@ -28,69 +28,131 @@ jobs:
name: "Pre-compile Checks"
uses: ./.github/workflows/template.pre-compile-checks.yml
- java8:
-name: "Java 8"
-uses: ./.github/workflows/template.flink-ci.yml
-with:
- workflow-caller-id: java8
- environment: 'PROFILE="-Dinclude_hadoop_aws"'
- jdk_version: 8
-secrets:
- s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
- s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
- s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
- java11:
-name: "Java 11"
-uses: ./.github/workflows/template.flink-ci.yml
-with:
- workflow-caller-id: java11
- environment: 'PROFILE="-Dinclude_hadoop_aws -Djdk11 -Pjava11-target"'
- jdk_version: 11
-secrets:
- s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
- s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
- s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
- java17:
-name: "Java 17"
-uses: ./.github/workflows/template.flink-ci.yml
-with:
- workflow-caller-id: java17
- environment: 'PROFILE="-Dinclude_hadoop_aws -Djdk11 -Djdk17
-Pjava17-target"'
- jdk_version: 17
-secrets:
- s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
- s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
- s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
- java21:
-name: "Java 21"
-uses: ./.github/workflows/template.flink-ci.yml
-with:
- workflow-caller-id: java21
- environment: 'PROFILE="-Dinclude_hadoop_aws -Djdk11 -Djdk17 -Djdk21
-Pjava21-target"'
- jdk_version: 21
-secrets:
- s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
- s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
- s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
- hadoop313:
-name: "Hadoop 3.1.3"
-uses: ./.github/workflows/template.flink-ci.yml
-with:
- workflow-caller-id: hadoop313
- environment: 'PROFILE="-Dflink.hadoop.version=3.2.3
-Phadoop3-tests,hive3"'
- jdk_version: 8
-secrets:
- s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
- s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
- s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
- adaptive-scheduler:
-name: "AdaptiveScheduler"
-uses: ./.github/workflows/template.flink-ci.yml
-with:
- workflow-caller-id: adaptive-scheduler
- environment: 'PROFILE="-Penable-adaptive-scheduler"'
- jdk_version: 8
-secrets:
- s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
- s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
- s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+# java8:
Review Comment:
Is this code no longer needed? Why not just delete it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
HuangXingBo commented on PR #24426: URL: https://github.com/apache/flink/pull/24426#issuecomment-2167064115 Thanks @morazow for the update. I downloaded the wheel package and found that the wheel package after repair is manylinux_2_5. It may be related to not using `dev/glibc_version_fix.sh`. If the version of glibc is too new, it is impossible to generate manylinux1 package. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on PR #24426: URL: https://github.com/apache/flink/pull/24426#issuecomment-2160556166 Thanks @HuangXingBo for the update! I have changed the GitHub actions to run the `auditwheel` repair after installing `patchelf` in Linux. Could you please have another look? You can find the build wheel artifacts for this change here: https://github.com/morazow/flink/actions/runs/9464510932 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
HuangXingBo commented on PR #24426: URL: https://github.com/apache/flink/pull/24426#issuecomment-2154360942 I downloaded the wheel package from https://github.com/morazow/flink/actions/runs/9224143298 and found that the wheel package for Linux is manylinux2014, not manylinux1, which may cause it to fail to install on some old machines. I used `dev/glibc_version_fix.sh` and `patchelf` in build-wheels.sh to solve this problem previously. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
XComp commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1626117295
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,46 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+
+ build_python_wheels:
+name: "Build Python Wheels on ${{ matrix.os_name }}"
+runs-on: ${{ matrix.os }}
+strategy:
+ fail-fast: false
+ matrix:
+include:
+ - os: ubuntu-latest
+os_name: linux
+ - os: macos-latest
+os_name: macos
+steps:
+ - name: "Checkout the repository"
+uses: actions/checkout@v4
+with:
+ fetch-depth: 0
+ persist-credentials: false
+ - name: "Stringify workflow name"
+uses: "./.github/actions/stringify"
+id: stringify_workflow
+with:
+ value: ${{ github.workflow }}
+ - name: "Build python wheels for ${{ matrix.os_name }}"
+uses: pypa/cibuildwheel@v2.16.5
Review Comment:
Looks fine from my side. But I am not familiar with the whole buildwheel
logic. @HuangXingBo can you do another pass over it and approve the changes
once more?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1617850492
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,46 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+
+ build_python_wheels:
+name: "Build Python Wheels on ${{ matrix.os_name }}"
+runs-on: ${{ matrix.os }}
+strategy:
+ fail-fast: false
+ matrix:
+include:
+ - os: ubuntu-latest
+os_name: linux
+ - os: macos-latest
+os_name: macos
+steps:
+ - name: "Checkout the repository"
+uses: actions/checkout@v4
+with:
+ fetch-depth: 0
+ persist-credentials: false
+ - name: "Stringify workflow name"
+uses: "./.github/actions/stringify"
+id: stringify_workflow
+with:
+ value: ${{ github.workflow }}
+ - name: "Build python wheels for ${{ matrix.os_name }}"
+uses: pypa/cibuildwheel@v2.16.5
Review Comment:
Great, updated ✔️
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
XComp commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1616244045
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,46 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+
+ build_python_wheels:
+name: "Build Python Wheels on ${{ matrix.os_name }}"
+runs-on: ${{ matrix.os }}
+strategy:
+ fail-fast: false
+ matrix:
+include:
+ - os: ubuntu-latest
+os_name: linux
+ - os: macos-latest
+os_name: macos
+steps:
+ - name: "Checkout the repository"
+uses: actions/checkout@v4
+with:
+ fetch-depth: 0
+ persist-credentials: false
+ - name: "Stringify workflow name"
+uses: "./.github/actions/stringify"
+id: stringify_workflow
+with:
+ value: ${{ github.workflow }}
+ - name: "Build python wheels for ${{ matrix.os_name }}"
+uses: pypa/cibuildwheel@v2.16.5
Review Comment:
That should work
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1616229933
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,46 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+
+ build_python_wheels:
+name: "Build Python Wheels on ${{ matrix.os_name }}"
+runs-on: ${{ matrix.os }}
+strategy:
+ fail-fast: false
+ matrix:
+include:
+ - os: ubuntu-latest
+os_name: linux
+ - os: macos-latest
+os_name: macos
+steps:
+ - name: "Checkout the repository"
+uses: actions/checkout@v4
+with:
+ fetch-depth: 0
+ persist-credentials: false
+ - name: "Stringify workflow name"
+uses: "./.github/actions/stringify"
+id: stringify_workflow
+with:
+ value: ${{ github.workflow }}
+ - name: "Build python wheels for ${{ matrix.os_name }}"
+uses: pypa/cibuildwheel@v2.16.5
Review Comment:
Ahh I see. I would be fine to use generic install aproach, right?
```
# Used to host cibuildwheel
- uses: actions/setup-python@v5
- name: Install cibuildwheel
run: python -m pip install cibuildwheel==2.18.1
- name: Build wheels
run: python -m cibuildwheel --output-dir wheelhouse
```
cibuildwheel docs also suggest this generic way for GitHub actions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
XComp commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1616177599
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,46 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+
+ build_python_wheels:
+name: "Build Python Wheels on ${{ matrix.os_name }}"
+runs-on: ${{ matrix.os }}
+strategy:
+ fail-fast: false
+ matrix:
+include:
+ - os: ubuntu-latest
+os_name: linux
+ - os: macos-latest
+os_name: macos
+steps:
+ - name: "Checkout the repository"
+uses: actions/checkout@v4
+with:
+ fetch-depth: 0
+ persist-credentials: false
+ - name: "Stringify workflow name"
+uses: "./.github/actions/stringify"
+id: stringify_workflow
+with:
+ value: ${{ github.workflow }}
+ - name: "Build python wheels for ${{ matrix.os_name }}"
+uses: pypa/cibuildwheel@v2.16.5
Review Comment:
Interesting approach. It's just that Apache doesn't allow actions other than
the ones from the `apache`, `github` and `actions`
([source](https://infra.apache.org/github-actions-policy.html)). We could use a
fixed version for the custom action and review that specific version. But
`pypa/cibuildwheel@v2.16.5` seems to be a bigger project. I'm wondering whether
it's worth it or just use the previous approach (even though I liked your
intention ). WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on PR #24426: URL: https://github.com/apache/flink/pull/24426#issuecomment-2133390885 Hey @XComp, could you please have look to the PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on PR #24426: URL: https://github.com/apache/flink/pull/24426#issuecomment-2129494739 Test run built wheel artifacts: https://github.com/morazow/flink/actions/runs/9224143298 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on PR #24426: URL: https://github.com/apache/flink/pull/24426#issuecomment-2129446733 Hey @XComp, @HuangXingBo , What do you think of the latest commit changes for migration? This way both Linux & MacOS platform use similar build system, we also don't depend on bash script on Linux. All the wheel build requirements are defined in the `pyproject.toml` (python versions, architecture, etc) that are used for both platforms. For Linux I had to add `manylinux2014` version since `manylinux1` does not support python3.10+ versions. Please let me know what you think -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1613261308
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,51 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+
+ build_python_wheels:
+name: "Build Python Wheels on ${{ matrix.os }}"
+runs-on: ${{ matrix.os }}
+strategy:
+ fail-fast: false
+ matrix:
+include:
+ - os: ubuntu-latest
+os_name: linux
+python-version: 3.9
+ - os: macos-latest
+os_name: macos
+python-version: 3.9
Review Comment:
Yes on macos the only pyproject.toml is used by cibuildwheel. Tools seems to
be well documented, maybe we could also use it for linux builds.
It uses the `cp38-*`, etc for Python versions,
https://cibuildwheel.pypa.io/en/stable/options/#build-skip. For Linux we are
running bash scripts for building wheel, but we could let the cibuildwheel
action do for both platforms.
We can ask @HuangXingBo for final review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1613227700
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,51 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+
+ build_python_wheels:
+name: "Build Python Wheels on ${{ matrix.os }}"
Review Comment:
Additionally, I checked the wheel artifacts from above latest Azure
pipelines run, they are more or less same size as GitHub actions run
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1613226372
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,51 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+
+ build_python_wheels:
+name: "Build Python Wheels on ${{ matrix.os }}"
Review Comment:
Hey @XComp,
I think this is fine. The Azure Pipelines
(https://github.com/apache/flink/blob/master/tools/azure-pipelines/build-python-wheels.yml)
also run like that, builds each wheel in a separate job.
From [GitHub migration
doc](https://docs.github.com/en/actions/migrating-to-github-actions/manually-migrating-to-github-actions/migrating-from-azure-pipelines-to-github-actions#migrating-jobs-and-steps):
```
- Jobs contain a series of steps that run sequentially.
- Jobs run on separate virtual machines or in separate containers.
- Jobs run in parallel by default, but can be configured to run sequentially.
```
So should be fine to build each wheel separately without the Flink build
similar to the Azure Pipelines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
XComp commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1612942822
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,51 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+
+ build_python_wheels:
+name: "Build Python Wheels on ${{ matrix.os }}"
Review Comment:
I'm not talking about the subfolder here. I mean the actual build: the
wheels build doesn't depend on the compile workflow right now, i.e. it doesn't
wait for Flink to be build. Therefore, the Flink build (or rather the relevant
Flink dependency) isn't properly packaged with the wheels. ...as far as I
understand.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
XComp commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1612936573
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,51 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+
+ build_python_wheels:
+name: "Build Python Wheels on ${{ matrix.os }}"
+runs-on: ${{ matrix.os }}
+strategy:
+ fail-fast: false
+ matrix:
+include:
+ - os: ubuntu-latest
+os_name: linux
+python-version: 3.9
+ - os: macos-latest
+os_name: macos
+python-version: 3.9
Review Comment:
> You're checking [Xingbo
Huang](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=hxb)'s fork
where the master branch doesn't seem to be up-to-date.
[apache/flink:flink-python/dev/build-wheels.sh](https://github.com/apache/flink/blob/master/flink-python/dev/build-wheels.sh#L19-L26)
does, indeed, have 3.11 added to the python version list.
Ok, I [responded in the Jira
issue](https://issues.apache.org/jira/browse/FLINK-34582?focusedCommentId=17849185=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17849185).
Looks like the Python version doesn't make a diffrence here for Linux. We
might want to add this as a comment and refer to the build script stating that
the version is specified there.
In the mean time: Can you find out where the Python versions are configured
for the MacOS builds? It appears that we also have the very same list of Python
versions used for building macos wheels (see [this Azure pipeline
run](https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59794=logs=f73b5736-8355-5390-ec71-4dfdec0ce6c5=90f7230e-bf5a-531b-8566-ad48d3e03bbb=49)
as an example).
I only found
[apache/flink:flink-python/pyproject.toml:30](https://github.com/apache/flink/blob/942d23636b13df7a118274d307943f5524cd/flink-python/pyproject.toml#L30).
That might be the right configuration. But I don't understand why the Python
version needed to be updated in the workflow yaml to 3.11 in FLINK-34582 in
that case. We might need a comment there to clarify things for macos as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
XComp commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1612936573
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,51 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+
+ build_python_wheels:
+name: "Build Python Wheels on ${{ matrix.os }}"
+runs-on: ${{ matrix.os }}
+strategy:
+ fail-fast: false
+ matrix:
+include:
+ - os: ubuntu-latest
+os_name: linux
+python-version: 3.9
+ - os: macos-latest
+os_name: macos
+python-version: 3.9
Review Comment:
> You're checking [Xingbo
Huang](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=hxb)'s fork
where the master branch doesn't seem to be up-to-date.
[apache/flink:flink-python/dev/build-wheels.sh](https://github.com/apache/flink/blob/master/flink-python/dev/build-wheels.sh#L19-L26)
does, indeed, have 3.11 added to the python version list.
Ok, I [responded in the Jira
issue](https://issues.apache.org/jira/browse/FLINK-34582?focusedCommentId=17849185=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17849185).
Looks like the Python version doesn't make a diffrence here for Linux. We
might want to add this as a comment and refer to the build script stating that
the version is specified there.
In the mean time: Can you find out where the Python versions are configured
for the MacOS builds? It appears that we also have the very same list of Python
versions used for building macos wheels (see [this Azure pipeline
run](https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59794=logs=f73b5736-8355-5390-ec71-4dfdec0ce6c5=90f7230e-bf5a-531b-8566-ad48d3e03bbb=49)
as an example). I only found
[apache/flink:flink-python/pyproject.toml:30](https://github.com/apache/flink/blob/942d23636b13df7a118274d307943f5524cd/flink-python/pyproject.toml#L30).
That might be the right configuration. But I don't understand why the Python
version needed to be updated in the workflow yaml to 3.11 in FLINK-34582 in
that case. We might need a comment there to clarify things for macos as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
XComp commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1612936573
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,51 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+
+ build_python_wheels:
+name: "Build Python Wheels on ${{ matrix.os }}"
+runs-on: ${{ matrix.os }}
+strategy:
+ fail-fast: false
+ matrix:
+include:
+ - os: ubuntu-latest
+os_name: linux
+python-version: 3.9
+ - os: macos-latest
+os_name: macos
+python-version: 3.9
Review Comment:
> You're checking [Xingbo
Huang](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=hxb)'s fork
where the master branch doesn't seem to be up-to-date.
[apache/flink:flink-python/dev/build-wheels.sh](https://github.com/apache/flink/blob/master/flink-python/dev/build-wheels.sh#L19-L26)
does, indeed, have 3.11 added to the python version list.
Ok, I [responded in the Jira
issue](https://issues.apache.org/jira/browse/FLINK-34582?focusedCommentId=17849185=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17849185).
Looks like the Python version doesn't make a diffrence here for Linux. We
might want to add this as a comment and refer to the build script stating that
the version is specified there.
In the mean time: Can you find out where the Python versions are configured
for the MacOS builds? It appears that we also have the very same list of Python
versions used for building macos wheels (see [this Azure pipeline
run](https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59794=logs=f73b5736-8355-5390-ec71-4dfdec0ce6c5=90f7230e-bf5a-531b-8566-ad48d3e03bbb=49]
as an example). I only found
[apache/flink:flink-python/pyproject.toml:30](https://github.com/apache/flink/blob/942d23636b13df7a118274d307943f5524cd/flink-python/pyproject.toml#L30).
That might be the right configuration. But I don't understand why the Python
version needed to be updated in the workflow yaml to 3.11 in FLINK-34582 in
that case. We might need a comment there to clarify things for macos as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
XComp commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1612936573
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,51 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+
+ build_python_wheels:
+name: "Build Python Wheels on ${{ matrix.os }}"
+runs-on: ${{ matrix.os }}
+strategy:
+ fail-fast: false
+ matrix:
+include:
+ - os: ubuntu-latest
+os_name: linux
+python-version: 3.9
+ - os: macos-latest
+os_name: macos
+python-version: 3.9
Review Comment:
> You're checking [Xingbo
Huang](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=hxb)'s fork
where the master branch doesn't seem to be up-to-date.
[apache/flink:flink-python/dev/build-wheels.sh](https://github.com/apache/flink/blob/master/flink-python/dev/build-wheels.sh#L19-L26)
does, indeed, have 3.11 added to the python version list.
Ok, I [responded in the Jira
issue](https://issues.apache.org/jira/browse/FLINK-34582?focusedCommentId=17849185=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17849185).
Looks like the Python version doesn't make a diffrence here for Linux. We
might want to add this as a comment and refer to the build script stating that
the version is specified there.
In the mean time: Can you find out where the Python versions are configured
for the MacOS builds? It appears that we also have the very same list of Python
versions used for building macos wheels (see [this Azure pipeline
run|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59794=logs=f73b5736-8355-5390-ec71-4dfdec0ce6c5=90f7230e-bf5a-531b-8566-ad48d3e03bbb=49]
as an example). I only found
[apache/flink:flink-python/pyproject.toml:30](https://github.com/apache/flink/blob/942d23636b13df7a118274d307943f5524cd/flink-python/pyproject.toml#L30).
That might be the right configuration. But I don't understand why the Python
version needed to be updated in the workflow yaml to 3.11 in FLINK-34582 in
that case. We might need a comment there to clarify things for macos as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
XComp commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1612936573
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,51 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+
+ build_python_wheels:
+name: "Build Python Wheels on ${{ matrix.os }}"
+runs-on: ${{ matrix.os }}
+strategy:
+ fail-fast: false
+ matrix:
+include:
+ - os: ubuntu-latest
+os_name: linux
+python-version: 3.9
+ - os: macos-latest
+os_name: macos
+python-version: 3.9
Review Comment:
> You're checking [Xingbo
Huang](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=hxb)'s fork
where the master branch doesn't seem to be up-to-date.
[apache/flink:flink-python/dev/build-wheels.sh](https://github.com/apache/flink/blob/master/flink-python/dev/build-wheels.sh#L19-L26)
does, indeed, have 3.11 added to the python version list.
Ok, I [responded in the Jira
issue](https://issues.apache.org/jira/browse/FLINK-34582?focusedCommentId=17849185=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17849185).
Looks like the Python version doesn't make a diffrence here for Linux. We
might want to add this as a comment and refer to the build script stating that
the version is specified there.
In the mean time: Can you find out where the Python versions are configured
for the MacOS builds? It appears that we also have the very same list of Python
versions used for building macos wheels (see [this Azure pipeline
run|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59794=logs=f73b5736-8355-5390-ec71-4dfdec0ce6c5=90f7230e-bf5a-531b-8566-ad48d3e03bbb=49]
as an example). I only found
[apache/flink:flink-python/pyproject.toml:30|https://github.com/apache/flink/blob/942d23636b13df7a118274d307943f5524cd/flink-python/pyproject.toml#L30).
That might be the right configuration. But I don't understand why the Python
version needed to be updated in the workflow yaml to 3.11 in FLINK-34582 in
that case. We might need a comment there to clarify things for macos as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
XComp commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1612936573
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,51 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+
+ build_python_wheels:
+name: "Build Python Wheels on ${{ matrix.os }}"
+runs-on: ${{ matrix.os }}
+strategy:
+ fail-fast: false
+ matrix:
+include:
+ - os: ubuntu-latest
+os_name: linux
+python-version: 3.9
+ - os: macos-latest
+os_name: macos
+python-version: 3.9
Review Comment:
> You're checking [Xingbo
Huang](https://issues.apache.org/jira/secure/ViewProfile.jspa?name=hxb)'s fork
where the master branch doesn't seem to be up-to-date.
[apache/flink:flink-python/dev/build-wheels.sh](https://github.com/apache/flink/blob/master/flink-python/dev/build-wheels.sh#L19-L26)
does, indeed, have 3.11 added to the python version list.
Ok, I [responded in the Jira
issue](https://issues.apache.org/jira/browse/FLINK-34582?focusedCommentId=17849185=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17849185).
Looks like the Python version doesn't make a diffrence here for Linux. We
might want to add this as a comment and refer to the build script stating that
the version is specified there.
In the mean time: Can you find out where the Python versions are configured
for the MacOS builds? It appears that we also have the very same list of Python
versions used for building macos wheels (see [this Azure pipeline
run|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=59794=logs=f73b5736-8355-5390-ec71-4dfdec0ce6c5=90f7230e-bf5a-531b-8566-ad48d3e03bbb=49]
as an example). I only found
[apache/flink:flink-python/pyproject.toml:30|https://github.com/apache/flink/blob/942d23636b13df7a118274d307943f5524cd/flink-python/pyproject.toml#L30].
That might be the right configuration. But I don't understand why the Python
version needed to be updated in the workflow yaml to 3.11 in FLINK-34582 in
that case. We might need a comment there to clarify things for macos as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on PR #24426: URL: https://github.com/apache/flink/pull/24426#issuecomment-2127412535 Artifacts from new nightly run:  -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1611819234
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,51 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+
+ build_python_wheels:
+name: "Build Python Wheels on ${{ matrix.os }}"
Review Comment:
Ahh yeah, from the ticket there is no folders so we may need to change tar
to strip parent folder with `-C` option.
From test runs:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1611807738
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,51 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+
+ build_python_wheels:
+name: "Build Python Wheels on ${{ matrix.os }}"
+runs-on: ${{ matrix.os }}
+strategy:
+ fail-fast: false
+ matrix:
+include:
+ - os: ubuntu-latest
+os_name: linux
+python-version: 3.9
+ - os: macos-latest
+os_name: macos
+python-version: 3.9
Review Comment:
Yes, indeed, the linux version of build scripts uses versions until 3.10 but
not 3.11. I have asked on the ticket.
https://github.com/HuangXingBo/flink/blob/master/flink-python/dev/build-wheels.sh#L19-L26
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1611806322
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,51 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+
+ build_python_wheels:
+name: "Build Python Wheels on ${{ matrix.os }}"
Review Comment:
We do that in the `tar` command for `flink-python/dist`, right?
Both os version build and create this folder that then we archive and upload
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on code in PR #24426: URL: https://github.com/apache/flink/pull/24426#discussion_r1611695577 ## .github/workflows/wheels.yml: ## @@ -0,0 +1,74 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# This workflow is meant as an extended CI run that includes certain features that shall be tested +# and JDK versions that are supported but not considered default. + +name: "Wheels Test" Review Comment: Ahh got it, on my fork the nightly build was all grayed out. I thought this was due to some secret requirement from original repo. But it is a dispatch workflow, I will try with what suggested after addressing other suggestions -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
XComp commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1611659779
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,51 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+
+ build_python_wheels:
+name: "Build Python Wheels on ${{ matrix.os }}"
+runs-on: ${{ matrix.os }}
+strategy:
+ fail-fast: false
+ matrix:
+include:
+ - os: ubuntu-latest
+os_name: linux
+python-version: 3.9
+ - os: macos-latest
+os_name: macos
+python-version: 3.9
+steps:
+ - name: "Checkout the repository"
+uses: actions/checkout@v4
+with:
+ fetch-depth: 0
+ persist-credentials: false
+ - name: "Install python"
+uses: actions/setup-python@v5
+with:
+ python-version: ${{ matrix.python-version }}
+ - name: "Stringify workflow name"
+uses: "./.github/actions/stringify"
+id: stringify_workflow
+with:
+ value: ${{ github.workflow }}
+ - name: "Build python wheels for ${{ matrix.os }}"
+if: matrix.os_name == 'linux'
+run: |
+ cd flink-python
+ bash dev/build-wheels.sh
+ - name: "Build python wheels for ${{ matrix.os }}"
+if: matrix.os_name == 'macos'
+run: |
+ cd flink-python
+ python -m pip install --upgrade pip
+ pip install cibuildwheel==2.8.0
Review Comment:
```suggestion
pip install cibuildwheel==2.16.5
```
Looks like there was an update on Azure in
[FLINK-34582](https://issues.apache.org/jira/browse/FLINK-34582) which we
should include. Maybe, put this in a separate commit and use
[FLINK-34582](https://issues.apache.org/jira/browse/FLINK-34582) as the issue
ID in the commit message. We should document this in
[FLINK-34582](https://issues.apache.org/jira/browse/FLINK-34582) after this PR
is merged.
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,51 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+
+ build_python_wheels:
+name: "Build Python Wheels on ${{ matrix.os }}"
+runs-on: ${{ matrix.os }}
+strategy:
+ fail-fast: false
+ matrix:
+include:
+ - os: ubuntu-latest
+os_name: linux
+python-version: 3.9
+ - os: macos-latest
+os_name: macos
+python-version: 3.9
+steps:
+ - name: "Checkout the repository"
+uses: actions/checkout@v4
+with:
+ fetch-depth: 0
+ persist-credentials: false
+ - name: "Install python"
+uses: actions/setup-python@v5
+with:
+ python-version: ${{ matrix.python-version }}
+ - name: "Stringify workflow name"
+uses: "./.github/actions/stringify"
+id: stringify_workflow
+with:
+ value: ${{ github.workflow }}
+ - name: "Build python wheels for ${{ matrix.os }}"
Review Comment:
```suggestion
- name: "Build python wheels for ${{ matrix.os_name }}"
```
I feel like using the `os_name` is less tied to GHA here. WDYT?
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,51 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+
+ build_python_wheels:
+name: "Build Python Wheels on ${{ matrix.os }}"
+runs-on: ${{ matrix.os }}
+strategy:
+ fail-fast: false
+ matrix:
+include:
+ - os: ubuntu-latest
+os_name: linux
+python-version: 3.9
+ - os: macos-latest
+os_name: macos
+python-version: 3.9
Review Comment:
Looks like the python version also got changed in
[FLINK-34582](https://issues.apache.org/jira/browse/FLINK-34582) 樂 We should
check why we don't upgrade the Python version for Linux.
##
.github/workflows/wheels.yml:
##
@@ -0,0 +1,74 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND,
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1611583785
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,51 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+
+ build_python_wheels:
+name: "Build Python Wheels on ${{ matrix.os }}"
+runs-on: ${{ matrix.os }}
+strategy:
+ fail-fast: false
+ matrix:
+os: [ ubuntu-latest, macos-latest ]
Review Comment:
Test run with include matrix setting:
https://github.com/morazow/flink/actions/runs/9207684232
Let's keep it with only include, latest commit, it's more understandable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on PR #24426: URL: https://github.com/apache/flink/pull/24426#issuecomment-2126952694 Test run with include matrix setting: https://github.com/morazow/flink/actions/runs/9207684232 Let's keep it like this it more understandable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1611578612
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,51 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+
+ build_python_wheels:
+name: "Build Python Wheels on ${{ matrix.os }}"
+runs-on: ${{ matrix.os }}
+strategy:
+ fail-fast: false
+ matrix:
+os: [ ubuntu-latest, macos-latest ]
Review Comment:
From doc it extends if there is already matrix definition:
https://docs.github.com/en/actions/using-jobs/using-a-matrix-for-your-jobs#expanding-or-adding-matrix-configurations
But I think I got your suggestion, to make it clear let me define similar
include based matrix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on PR #24426: URL: https://github.com/apache/flink/pull/24426#issuecomment-2126937800 Test run of Python wheel build: https://github.com/morazow/flink/actions/runs/9207285570 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
XComp commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1611564251
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,51 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+
+ build_python_wheels:
+name: "Build Python Wheels on ${{ matrix.os }}"
+runs-on: ${{ matrix.os }}
+strategy:
+ fail-fast: false
+ matrix:
+os: [ ubuntu-latest, macos-latest ]
Review Comment:
How come it works without it in the matrix definition of
`template.flink-ci.yml` that only uses `include` in the config? What's the
difference there? 樂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1611521749
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,51 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+
+ build_python_wheels:
+name: "Build Python Wheels on ${{ matrix.os }}"
+runs-on: ${{ matrix.os }}
+strategy:
+ fail-fast: false
+ matrix:
+os: [ ubuntu-latest, macos-latest ]
Review Comment:
I think we still need the os parameter, apparently include is only for
extending the existing matrix setting
Without os param run:
https://github.com/morazow/flink/actions/runs/9206750374
With both os and include:
https://github.com/morazow/flink/actions/runs/9207285570
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1611469958
##
.github/workflows/template.flink-ci.yml:
##
@@ -80,7 +80,7 @@ jobs:
# timeout in minutes - this environment variable is required by
uploading_watchdog.sh
GHA_JOB_TIMEOUT: 240
outputs:
- stringified-workflow-name: ${{
steps.workflow-prep-step.outputs.stringified-workflow-name }}
+ stringified-workflow-name: ${{
steps.workflow-prep-step.outputs.stringified_value }}
steps:
Review Comment:
@XComp, I kept this `outputs.stringified-workflow-name` as is, it is used in
the later jobs.
Initially I thought this could be confusing, but maybe all right to keep it
as is, the `stringified_value` will be used only within the compile steps
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on code in PR #24426: URL: https://github.com/apache/flink/pull/24426#discussion_r1611448720 ## .github/actions/stringify_workflow/action.yml: ## @@ -0,0 +1,47 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +--- +name: "Stringifies Workflow" Review Comment: Yes agreed, I will update it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1611447937
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,51 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+
+ build_python_wheels:
+name: "Build Python Wheels on ${{ matrix.os }}"
+runs-on: ${{ matrix.os }}
+strategy:
+ fail-fast: false
+ matrix:
+os: [ ubuntu-latest, macos-latest ]
Review Comment:
Ahh yes correct, removing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
XComp commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1611407508
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,51 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+
+ build_python_wheels:
+name: "Build Python Wheels on ${{ matrix.os }}"
+runs-on: ${{ matrix.os }}
+strategy:
+ fail-fast: false
+ matrix:
Review Comment:
Good idea to use matrix here
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,51 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+
+ build_python_wheels:
+name: "Build Python Wheels on ${{ matrix.os }}"
+runs-on: ${{ matrix.os }}
+strategy:
+ fail-fast: false
+ matrix:
+os: [ ubuntu-latest, macos-latest ]
Review Comment:
That line is obsolete since you have this parameter already defined in the
`.include` map below
##
.github/actions/stringify_workflow/action.yml:
##
@@ -0,0 +1,47 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+---
+name: "Stringifies Workflow"
Review Comment:
I feel we can make this more generic and call it just "Stringify". There
should be only one input variable `value` and one output `stringified_value`.
That makes the action open for future use-cases other than workflow names. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on PR #24426: URL: https://github.com/apache/flink/pull/24426#issuecomment-2124739983 Thanks @XComp, I have addressed your findings, please have a look. I used Python version `3.9` since it was in the Azure pipelines, let me know if we need to change it. Separated also the workflow stringify to another action, please have a look -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1609840918
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,65 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ prepare:
+name: Prepare
+runs-on: ubuntu-latest
+outputs:
+ stringified-workflow-name: ${{
steps.workflow-prep-step.outputs.stringified-workflow-name }}
+steps:
+ - name: "Stringify workflow name"
+id: workflow-prep-step
+run: |
+ # Similar to the template.flink-ci.yml compile job
+ stringified_workflow_name=$(echo "${{ github.workflow }}" | tr -C
'[:alnum:]._' '-' | tr '[:upper:]' '[:lower:]' | sed -e 's/--*/-/g' -e
's/^-*//g' -e 's/-*$//g')
+ echo "stringified-workflow-name=${stringified_workflow_name}" >>
$GITHUB_OUTPUT
+ linux:
+needs: prepare
+runs-on: ubuntu-latest
+steps:
+ - name: Checkout the repository
+uses: actions/checkout@v4
+with:
+ fetch-depth: 0
+ persist-credentials: false
+ - name: Install python
+uses: actions/setup-python@v5
+with:
+ python-version: ${{ env.PYTHON_VERSION }}
+ - name: Build wheels
+run: |
+ cd flink-python
+ bash dev/build-wheels.sh
+ - name: Tar artifact
+run: tar -czvf python-wheel.tar.gz flink-python/dist
+ - name: Upload artifact
+uses: actions/upload-artifact@v4
+with:
+ name: wheel_${{ needs.prepare.outputs.stringified-workflow-name
}}-${{ github.run_number }}
Review Comment:
Yes correct, in Azure pipeline, it is ${Agent.OS}, here I will use `linux`
and `macos` to distinguish the wheels
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1609830007
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,65 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ prepare:
+name: Prepare
+runs-on: ubuntu-latest
+outputs:
+ stringified-workflow-name: ${{
steps.workflow-prep-step.outputs.stringified-workflow-name }}
+steps:
+ - name: "Stringify workflow name"
+id: workflow-prep-step
+run: |
+ # Similar to the template.flink-ci.yml compile job
+ stringified_workflow_name=$(echo "${{ github.workflow }}" | tr -C
'[:alnum:]._' '-' | tr '[:upper:]' '[:lower:]' | sed -e 's/--*/-/g' -e
's/^-*//g' -e 's/-*$//g')
+ echo "stringified-workflow-name=${stringified_workflow_name}" >>
$GITHUB_OUTPUT
+ linux:
+needs: prepare
+runs-on: ubuntu-latest
+steps:
+ - name: Checkout the repository
+uses: actions/checkout@v4
+with:
+ fetch-depth: 0
+ persist-credentials: false
+ - name: Install python
+uses: actions/setup-python@v5
+with:
+ python-version: ${{ env.PYTHON_VERSION }}
Review Comment:
Good catch, it is missing from env, added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on PR #24426: URL: https://github.com/apache/flink/pull/24426#issuecomment-2124583186 Thanks @XComp, going through your suggestions, I will address them, thanks No indeed, I am going to squash them, it should be single commit -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
XComp commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1609654509
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,65 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ prepare:
+name: Prepare
+runs-on: ubuntu-latest
+outputs:
+ stringified-workflow-name: ${{
steps.workflow-prep-step.outputs.stringified-workflow-name }}
+steps:
+ - name: "Stringify workflow name"
+id: workflow-prep-step
+run: |
+ # Similar to the template.flink-ci.yml compile job
Review Comment:
Can move this logic into a custom action? 樂
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,65 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ prepare:
+name: Prepare
+runs-on: ubuntu-latest
+outputs:
+ stringified-workflow-name: ${{
steps.workflow-prep-step.outputs.stringified-workflow-name }}
+steps:
+ - name: "Stringify workflow name"
+id: workflow-prep-step
+run: |
+ # Similar to the template.flink-ci.yml compile job
+ stringified_workflow_name=$(echo "${{ github.workflow }}" | tr -C
'[:alnum:]._' '-' | tr '[:upper:]' '[:lower:]' | sed -e 's/--*/-/g' -e
's/^-*//g' -e 's/-*$//g')
+ echo "stringified-workflow-name=${stringified_workflow_name}" >>
$GITHUB_OUTPUT
+ linux:
+needs: prepare
+runs-on: ubuntu-latest
+steps:
+ - name: Checkout the repository
+uses: actions/checkout@v4
+with:
+ fetch-depth: 0
+ persist-credentials: false
+ - name: Install python
+uses: actions/setup-python@v5
+with:
+ python-version: ${{ env.PYTHON_VERSION }}
Review Comment:
Where is `PYTHON_VERSION` defined? 樂
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,65 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ prepare:
+name: Prepare
Review Comment:
Prepare seems to be too generic in this context. What about
`prepare-python-wheel-upload`? But we could get rid of this one job if we move
the stringification into a custom action anyway, I guess. 樂
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,65 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ prepare:
+name: Prepare
+runs-on: ubuntu-latest
+outputs:
+ stringified-workflow-name: ${{
steps.workflow-prep-step.outputs.stringified-workflow-name }}
+steps:
+ - name: "Stringify workflow name"
+id: workflow-prep-step
+run: |
+ # Similar to the template.flink-ci.yml compile job
+ stringified_workflow_name=$(echo "${{ github.workflow }}" | tr -C
'[:alnum:]._' '-' | tr '[:upper:]' '[:lower:]' | sed -e 's/--*/-/g' -e
's/^-*//g' -e 's/-*$//g')
+ echo "stringified-workflow-name=${stringified_workflow_name}" >>
$GITHUB_OUTPUT
+ linux:
+needs: prepare
+runs-on: ubuntu-latest
+steps:
+ - name: Checkout the repository
+uses: actions/checkout@v4
+with:
+ fetch-depth: 0
+ persist-credentials: false
+ - name: Install python
+uses: actions/setup-python@v5
+with:
+ python-version: ${{ env.PYTHON_VERSION }}
+ - name: Build wheels
+run: |
+ cd flink-python
+ bash dev/build-wheels.sh
+ - name: Tar artifact
+run: tar -czvf python-wheel.tar.gz flink-python/dist
Review Comment:
```suggestion
run: tar -czvf python-wheel-linux.tar.gz flink-python/dist
```
We should try to avoid having ambigious names. Here it might be not really
necessary because we're updating the name in the `upload-artifact` step anyway.
But it might help certain readers with understanding the code.
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,65 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ prepare:
+name: Prepare
+runs-on: ubuntu-latest
+outputs:
+ stringified-workflow-name: ${{
steps.workflow-prep-step.outputs.stringified-workflow-name }}
+steps:
+ - name: "Stringify workflow name"
+id: workflow-prep-step
+run: |
+ # Similar to the template.flink-ci.yml compile job
+ stringified_workflow_name=$(echo "${{ github.workflow }}" | tr -C
'[:alnum:]._' '-' | tr '[:upper:]' '[:lower:]'
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
XComp commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1540948523
##
.github/workflows/template.python-wheels-ci.yml:
##
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Python Wheels CI
+
+on:
+ workflow_call:
+
+permissions: read-all
+
+jobs:
+ linux:
+runs-on: ubuntu-latest
+steps:
+ - name: Checkout the repository
+uses: actions/checkout@v4
+with:
+ fetch-depth: 0
+ persist-credentials: false
+ - name: Build wheels
+run: |
+ cd flink-python
+ bash dev/build-wheels.sh
+ - name: Tar artifact
+run: tar -cvf python-wheel.tar flink-python/dist
+ - name: Upload artifact
+uses: actions/upload-artifact@v4
+with:
+ name: wheel_${{ runner.os }}_${{ github.sha }}_${{ github.run_number
}}
Review Comment:
What are you referring to with "them"? The `name`, I guess?
I'm wondering whether we should rely on what's provided in
[apache/flink:.github/workflows/template.flink-ci.yml:106](https://github.com/apache/flink/blob/master/.github/workflows/template.flink-ci.yml#L106)
with the stringified workflow name. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on PR #24426: URL: https://github.com/apache/flink/pull/24426#issuecomment-2118852480 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on PR #24426: URL: https://github.com/apache/flink/pull/24426#issuecomment-2041795538 Thanks @XComp, I am going to address your suggestions. Please have a look once you are back -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1555171619
##
.github/workflows/template.python-wheels-ci.yml:
##
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Python Wheels CI
+
+on:
+ workflow_call:
+
+permissions: read-all
+
+jobs:
+ linux:
+runs-on: ubuntu-latest
+steps:
+ - name: Checkout the repository
+uses: actions/checkout@v4
+with:
+ fetch-depth: 0
+ persist-credentials: false
+ - name: Build wheels
+run: |
+ cd flink-python
+ bash dev/build-wheels.sh
+ - name: Tar artifact
+run: tar -cvf python-wheel.tar flink-python/dist
Review Comment:
Yes, good idea!
##
.github/workflows/template.python-wheels-ci.yml:
##
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Python Wheels CI
+
+on:
+ workflow_call:
+
+permissions: read-all
+
+jobs:
+ linux:
+runs-on: ubuntu-latest
+steps:
+ - name: Checkout the repository
+uses: actions/checkout@v4
+with:
+ fetch-depth: 0
+ persist-credentials: false
+ - name: Build wheels
+run: |
+ cd flink-python
+ bash dev/build-wheels.sh
+ - name: Tar artifact
+run: tar -cvf python-wheel.tar flink-python/dist
+ - name: Upload artifact
+uses: actions/upload-artifact@v4
+with:
+ name: wheel_${{ runner.os }}_${{ github.sha }}_${{ github.run_number
}}
+ path: python-wheel.tar
+ macos:
+runs-on: macos-latest
+steps:
+ - name: Checkout the repository
+uses: actions/checkout@v4
+with:
+ fetch-depth: 0
+ persist-credentials: false
+ - name: Install python
Review Comment:
Yes good idea, I took the same steps from Azure pipelines. But I agree
having same Python version on both Linux/MacOs jobs is good idea. I will change
workflow.
I didn't check which version is installed in the Linux VM
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,6 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ python-wheels:
Review Comment:
Yes, that should be fine, I would also prefer the inlined version.
I wanted to keep the structure similar to other jobs, but python wheels are
not used more than once. I will change it to the inlined version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1555166750
##
.github/workflows/template.python-wheels-ci.yml:
##
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Python Wheels CI
+
+on:
+ workflow_call:
+
+permissions: read-all
+
+jobs:
+ linux:
+runs-on: ubuntu-latest
+steps:
+ - name: Checkout the repository
+uses: actions/checkout@v4
+with:
+ fetch-depth: 0
+ persist-credentials: false
+ - name: Build wheels
+run: |
+ cd flink-python
+ bash dev/build-wheels.sh
+ - name: Tar artifact
+run: tar -cvf python-wheel.tar flink-python/dist
+ - name: Upload artifact
+uses: actions/upload-artifact@v4
+with:
+ name: wheel_${{ runner.os }}_${{ github.sha }}_${{ github.run_number
}}
Review Comment:
Hey @XComp,
I meant to make the python-wheels-ci GitHub workflow and Azure pipelines.
Here the main concern is to keep the Python Wheels artifact names similar.
Since the Azure pipelines used `Agent.JobName` it would be fine to use the
stringrified workflow name, with `wheel` prefixed.
Would that be good option?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
XComp commented on PR #24426: URL: https://github.com/apache/flink/pull/24426#issuecomment-2033791965 fyi: I won't be responsive till the end of April at least -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
XComp commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1540950575
##
.github/workflows/template.python-wheels-ci.yml:
##
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Python Wheels CI
+
+on:
+ workflow_call:
+
+permissions: read-all
+
+jobs:
+ linux:
+runs-on: ubuntu-latest
+steps:
+ - name: Checkout the repository
+uses: actions/checkout@v4
+with:
+ fetch-depth: 0
+ persist-credentials: false
+ - name: Build wheels
+run: |
+ cd flink-python
+ bash dev/build-wheels.sh
+ - name: Tar artifact
+run: tar -cvf python-wheel.tar flink-python/dist
Review Comment:
Would it make sense to also compress it here? Or is there a reason why
compression is not enabled :thinking:
##
.github/workflows/template.python-wheels-ci.yml:
##
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Python Wheels CI
+
+on:
+ workflow_call:
+
+permissions: read-all
+
+jobs:
+ linux:
+runs-on: ubuntu-latest
+steps:
+ - name: Checkout the repository
+uses: actions/checkout@v4
+with:
+ fetch-depth: 0
+ persist-credentials: false
+ - name: Build wheels
+run: |
+ cd flink-python
+ bash dev/build-wheels.sh
+ - name: Tar artifact
+run: tar -cvf python-wheel.tar flink-python/dist
+ - name: Upload artifact
+uses: actions/upload-artifact@v4
+with:
+ name: wheel_${{ runner.os }}_${{ github.sha }}_${{ github.run_number
}}
Review Comment:
What are you referring to with "them"? The `name`, I guess?
I'm wondering whether we should rely on what's provided in
[apache/flink:.github/workflows/template.flink-ci.yml:106}(https://github.com/apache/flink/blob/master/.github/workflows/template.flink-ci.yml#L106)
with the stringified workflow name. WDYT?
##
.github/workflows/nightly.yml:
##
@@ -94,3 +94,6 @@ jobs:
s3_bucket: ${{ secrets.IT_CASE_S3_BUCKET }}
s3_access_key: ${{ secrets.IT_CASE_S3_ACCESS_KEY }}
s3_secret_key: ${{ secrets.IT_CASE_S3_SECRET_KEY }}
+ python-wheels:
Review Comment:
I'm wondering whether we should just inline the two jobs. Adding them as a
template makes the workflow appear on the left side of the workflow list in
GitHub Actions (analogously to [the entry for the
template.flink-ci.yml](https://github.com/apache/flink/actions/workflows/template.flink-ci.yml)
without any additional utilization.
One argument to keep the template workflow could be that we would be able to
enable `workflow_dispatch` for the wheel generation. But I'm not aware of such
a use-case. :thinking:
WDYT? :thinking:
##
.github/workflows/template.python-wheels-ci.yml:
##
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow commented on code in PR #24426:
URL: https://github.com/apache/flink/pull/24426#discussion_r1508821620
##
.github/workflows/template.python-wheels-ci.yml:
##
@@ -0,0 +1,67 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Python Wheels CI
+
+on:
+ workflow_call:
+
+permissions: read-all
+
+jobs:
+ linux:
+runs-on: ubuntu-latest
+steps:
+ - name: Checkout the repository
+uses: actions/checkout@v4
+with:
+ fetch-depth: 0
+ persist-credentials: false
+ - name: Build wheels
+run: |
+ cd flink-python
+ bash dev/build-wheels.sh
+ - name: Tar artifact
+run: tar -cvf python-wheel.tar flink-python/dist
+ - name: Upload artifact
+uses: actions/upload-artifact@v4
+with:
+ name: wheel_${{ runner.os }}_${{ github.sha }}_${{ github.run_number
}}
Review Comment:
This is slightly different than the Azure pipelines build. There
`Agent.JobName` is used.
```
- task: PublishPipelineArtifact@0
inputs:
artifactName: 'wheel_$(Agent.OS)_$(Agent.JobName)'
targetPath: 'flink-python/dist'
```
Could you please suggest how could we make them similar?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
flinkbot commented on PR #24426: URL: https://github.com/apache/flink/pull/24426#issuecomment-1972963471 ## CI report: * e1c8bf3cfbaf7f74feff2aa4b32cf41230d099c9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [FLINK-34487][ci] Adds Python Wheels nightly GHA workflow [flink]
morazow opened a new pull request, #24426: URL: https://github.com/apache/flink/pull/24426 ## What is the purpose of the change Integrates Python Wheels build into the GitHub Actions (GHA) nightly workflow. ## Brief change log - Adds Python Wheels GHA workflow, `template.python-wheels-ci.yml` - Integrates the Python Wheels workflow to the nightly GHA workflow ## Verifying this change This changes does not change code. But the successful nightly GHA run should verify the change. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
