Andrushika commented on code in PR #73527:
URL: https://github.com/apache/airflow/pull/73527#discussion_r4070215065
##########
.github/actions/install-prek/action.yml:
##########
@@ -143,16 +149,28 @@ runs:
shell: bash
run: cat ~/.cache/prek/prek.log || true
if: always()
+ # A pull request that restored the cache has nothing new to save: the
stash it got (usually the
+ # one from the base branch) is what it would upload again, and
re-compressing and uploading
+ # ~7 GB costs about five minutes per run. Save on a miss, and on non-PR
runs (the canary on
+ # the default branch) so the base-branch stash stays fresh within its
retention window.
- name: "Prepare .tar file from prek cache"
run: |
- tar -C ~ -czf /tmp/cache-prek.tar.gz .cache/prek
+ if command -v pigz > /dev/null 2>&1; then
+ tar -C ~ -cf - .cache/prek | pigz > /tmp/cache-prek.tar.gz
+ else
+ tar -C ~ -czf /tmp/cache-prek.tar.gz .cache/prek
+ fi
shell: bash
- if: inputs.save-cache == 'true'
+ if: >
+ inputs.save-cache == 'true'
+ && (steps.restore-prek-tar.outputs.tar-restored != 'true' ||
github.event_name != 'pull_request')
- name: "Save prek cache"
uses:
apache/infrastructure-actions/stash/save@61dcea11f19e2bbe1263f14d72235e8da17d3ad0
# save/v1.0.0
with:
key: ${{ steps.cache-key.outputs.key }}
path: /tmp/cache-prek.tar.gz
if-no-files-found: 'error'
retention-days: '2'
Review Comment:
The tarball here is already gzip, but stash/save passes compression-level
straight to upload-artifact, which compresses again by default.
We could set compression-level: '0' on this same step and skip that second
pass.
```suggestion
retention-days: '2'
compression-level: '0'
```
##########
.github/actions/install-prek/action.yml:
##########
@@ -143,16 +149,28 @@ runs:
shell: bash
run: cat ~/.cache/prek/prek.log || true
if: always()
+ # A pull request that restored the cache has nothing new to save: the
stash it got (usually the
+ # one from the base branch) is what it would upload again, and
re-compressing and uploading
+ # ~7 GB costs about five minutes per run. Save on a miss, and on non-PR
runs (the canary on
+ # the default branch) so the base-branch stash stays fresh within its
retention window.
- name: "Prepare .tar file from prek cache"
run: |
- tar -C ~ -czf /tmp/cache-prek.tar.gz .cache/prek
+ if command -v pigz > /dev/null 2>&1; then
+ tar -C ~ -cf - .cache/prek | pigz > /tmp/cache-prek.tar.gz
+ else
+ tar -C ~ -czf /tmp/cache-prek.tar.gz .cache/prek
+ fi
shell: bash
- if: inputs.save-cache == 'true'
+ if: >
+ inputs.save-cache == 'true'
Review Comment:
On a cache hit this skips the save, but a restored tarball can still hold a
partial environment. prek install-hooks repairs it, and with this condition the
repaired copy is never uploaded.
For example, a PR that misses once saves its own stash. If that stash is
partial, every later push on that PR restores it, repairs it, and drops the
repair until the stash expires two days later.
I think two days is an acceptable bound here. Just noting it so the
trade-off is visible next to the condition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
