aoelvp94 opened a new pull request, #75657: URL: https://github.com/apache/airflow/pull/75657
related: #73015 related: #75566 `WasbDagBundle` stages the bundle by downloading the prefix one blob at a time (`WasbHook.sync_to_local_dir`), so a bundle of N small files costs one request per blob on every component that stages it. With ephemeral workers (KubernetesExecutor) that cost is on the critical path of every task start. This is the Azure counterpart of #73015 and #75566. It adds an opt-in `archive_blob` argument on the existing `WasbDagBundle`, not a separate class. With `archive_blob` unset, behavior is unchanged. When it is set, the bundle is staged from a single `.tar.gz` blob: one properties read (staging is skipped when the ETag matches), one download pinned with `If-Match` on that ETag, a safe unpack (`tarfile` with `filter="data"`), and a rename swap that keeps the previous tree inside the staging directory so a failed swap restores it. Between the two renames the directory is briefly missing, because readers do not take the bundle lock. `archive_blob` and `prefix` are mutually exclusive. Passing both raises `ValueError` when the bundle is constructed. A missing archive raises `ValueError`, same as a missing container or prefix. There is no fallback to a per-blob sync. Publishing the archive is the deployer's job (documented: `tar -C ./dags -czf dags.tar.gz .` and `az storage blob upload`). Measured with a fully local reproduction (Airflow 3.3, Azurite 3.35.0, identical 400-file / ~3 MiB set — https://github.com/aoelvp94/airflow-azure-dag-bundle-staging-demo): | staging strategy | wall-clock (median) | requests served | |---|---:|---:| | per-blob sync | 1.08 s | 403 | | single archive | 0.12 s | 4 | The clock times are a loopback floor. The request count is 1 container GET, 2 lists, and 400 downloads, versus 1 container GET, 2 metadata reads, and 1 download. The released provider 15.2.1 does not ship `WasbDagBundle` yet (it is on main via #67016), so the demo vendors that class. This PR changes the class on main. Covered by unit tests for mutual exclusion, unpack, an unchanged ETag, an empty ETag, an ETag change, a missing archive, an `If-Match` failure, a corrupt archive, a failed swap, and the blob URL. Existing `WasbDagBundle` tests still pass. `pytest providers/microsoft/azure/tests/unit/microsoft/azure/bundles/test_wasb.py` passed locally (19 tests) and `prek` passed on the changed files. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: Grok in Cursor following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. Made with [Cursor](https://cursor.com) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
