TobKed commented on a change in pull request #11877:
URL: https://github.com/apache/beam/pull/11877#discussion_r436067638
##########
File path: .github/workflows/build_wheels.yml
##########
@@ -0,0 +1,141 @@
+name: Build python wheels
+
+on:
+ push:
+ branches:
+ - master
+ - release-*
+ tags:
+ - v*
+
+jobs:
+
+ build_source:
+ runs-on: ubuntu-18.04
+ steps:
+ - name: Checkout code
+ uses: actions/checkout@v2
+ - name: Install python
+ uses: actions/setup-python@v2
+ with:
+ python-version: 3.7
+ - name: Get build dependencies
+ working-directory: ./sdks/python
+ run: python3 -m pip install cython && python3 -m pip install -r
build-requirements.txt
+ - name: Install wheels
+ run: python3 -m pip install wheel
+ - name: Buld source
+ working-directory: ./sdks/python
+ run: python3 setup.py sdist --formats=gztar,zip
+ - name: Unzip source
+ working-directory: ./sdks/python
+ run: unzip dist/$(ls dist | grep .zip | head -n 1)
+ - name: Rename source directory
+ working-directory: ./sdks/python
+ run: mv $(ls | grep apache-beam) apache-beam-source
+ - name: Upload source
+ uses: actions/upload-artifact@v2
+ with:
+ name: source
+ path: sdks/python/apache-beam-source
+ - name: Upload compressed sources
+ uses: actions/upload-artifact@v2
+ with:
+ name: source_gztar_zip
Review comment:
1. Currently two steps: `List sources on GCS bucket` and `Copy wheels to
GCS bucket` are listing files of specific types. Instead of this two separate
steps I could create job which will list all files in specific gcs folder. I
think it would be much cleaner and explicit. Did I understand correctly your
idea?
About cleaning up these GCS locations I consider two options:
- setting lifecycle management on the bucket which will delete files older
than some arbitrary age, e.g. 365 days. I think advantage of this is that will
be maintenance free.
- creating another scheduled workflow on github actions which will delete
gcs folders if corresponding branch does not exist anymore. Could be scheduled
to run e.g. once pre week.
Which option has more sense for you?
2. "Upload" steps perform file upload as artifacts so they could be passed
between jobs and being available for download for 90 days (if not deleted
earlier). These artifacts are picked up later by "Upload to GCS" jobs. What do
you think about renaming these steps e.g.: "Upload wheels" -> "Upload wheels as
artifacts" ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
