aoelvp94 opened a new pull request, #75566: URL: https://github.com/apache/airflow/pull/75566
related: #73015 `GCSDagBundle` stages the bundle by downloading the prefix one object at a time (`GCSHook.sync_to_local_dir`), so a bundle of N small files costs one request per object on every component that stages it. With ephemeral workers (KubernetesExecutor) that cost is on the critical path of every task start, and a refresh still pays a metadata GET per object when nothing changed. This is the GCS counterpart of #73015. It adds an opt-in `archive_object` kwarg to `GCSDagBundle`: when set, the bundle is staged from a single `.tar.gz` object instead — one metadata read (staging is skipped when the ETag matches the last staged archive), one download, a safe unpack (`tarfile` with `filter="data"`), and a rename swap that keeps the previous tree inside the staging directory so a failed swap restores it. `archive_object` and `prefix` are mutually exclusive — passing both raises `ValueError` at construction. When `archive_object` is set the archive is the only source of the Dags: there is no second copy to keep in sync and no fallback to a per-object sync. A missing archive raises `AirflowNotFoundException` during initialization. Existing bucket, prefix, and version errors are unchanged. Publishing the archive is the deployer's job (documented: `tar -C ./dags -czf dags.tar.gz .` and `gcloud storage cp`). Measured with a fully local reproduction (Airflow 3.3.1, fake-gcs, identical 400-file / ~3 MiB set — https://github.com/aoelvp94/airflow-gcs-dag-bundle-staging-demo): | staging strategy | wall-clock (median) | requests served | |---|---:|---:| | per-object sync | 1.10 s | 403 | | single archive | 0.20 s | 4 | The clock times are a loopback floor (about 3 ms per call). The request count is the part that tracks the S3 result: 400 downloads plus 2 lists and 1 bucket GET, versus 1 download, 2 metadata reads, and 1 bucket GET. Nothing changes when `archive_object` is omitted. Covered by 8 new unit tests (mutual exclusion, unpack, ETag skip, ETag change, missing archive, corrupt archive, failed swap, console link) plus the existing `GCSDagBundle` tests. Docs updated in `providers/google/docs/bundles/index.rst`. `prek` static checks and `pytest providers/google/tests/unit/google/cloud/bundles/test_gcs.py` pass locally (17 passed). --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: Grok in Cursor following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. Made with [Cursor](https://cursor.com) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
