shahar1 opened a new pull request, #77152: URL: https://github.com/apache/airflow/pull/77152
Proof of concept for [AIP-129: Backfill for Asset Consumers with Dated Partitions](https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/462817309/AIP-129+Backfill+for+Asset+Consumers+with+Dated+Partitions). This draft makes the implementation available for the AIP discussion. It is not ready to merge and does not imply approval of the proposal. Allow a supported `PartitionedAssetTimetable` consumer to backfill partitions from existing asset events without rerunning producers. Support calendar rollups and identity-mapped consumers, select the newest event for each required input, and retain existing partitioned-backfill reprocessing behavior. The branch includes internal Python support for explicit partition keys and detailed planning results. The REST routes support date-range creation and the existing dry run. ### Validation Rebased onto `upstream/main` at `8a016b710ff`. Using Breeze with Python 3.11, ran: ```bash breeze run --backend sqlite pytest airflow-core/tests/unit/models/test_backfill_consumer.py airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_backfills.py::TestConsumerBackfill airflow-core/tests/unit/models/test_backfill.py -q breeze run --backend postgres pytest airflow-core/tests/unit/models/test_backfill_consumer.py airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_backfills.py::TestConsumerBackfill airflow-core/tests/unit/models/test_backfill.py -q breeze run --backend mysql pytest airflow-core/tests/unit/models/test_backfill_consumer.py airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_backfills.py::TestConsumerBackfill airflow-core/tests/unit/models/test_backfill.py -q ``` - SQLite: 163 passed, 2 skipped. - PostgreSQL 14: 165 passed. - MySQL 8.0: 165 passed. - Fast pre-commit hooks and manual-stage hooks passed, excluding the repository-prescribed interactive/dev-server hooks. These are targeted results, not a claim that the full CI matrix or documentation builds passed. ### Remaining work and known limits - Add explicit-key request fields and CLI support. - Expose detailed per-key results through the REST API and UI. - Filter detailed preview asset information according to the caller's permissions before exposing it. - Add configuration metadata for the candidate-key limit. - Complete the supported-schedule restrictions and design decisions documented in the AIP. - Live scheduling remains independent and can create another run for the same key. - The existing backfill lookup can prefer an older finished run over a queued run with no `start_date`; this draft does not fix that pre-existing behavior. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — GitHub Copilot (Copilot SDK in VS Code; review, validation and PR preparation) Generated-by: GitHub Copilot (Copilot SDK in VS Code) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- Drafted-by: GitHub Copilot (Copilot SDK in VS Code) (no human review before posting) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
