1fanwang opened a new pull request, #66453: URL: https://github.com/apache/airflow/pull/66453
## What this is A discussion-only demo PR stacked on top of the AIP-96 v2 PR set (#66402, #66410, #66445). **Not for merge.** The intent is to give reviewers something concrete to react to alongside the [AIP-96 v2 cwiki page](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-96+Resumable+Operators). ## What's added A self-contained `ResumableExampleOperator` in `airflow-core/src/airflow/example_dags/plugins/` plus a small example DAG that uses it. The 'external job' is simulated in-process (a sleep loop with a SIGTERM handler) so the demo runs without external infrastructure. The operator demonstrates the v1 integration pattern from AIP-96 v2: ```python def execute(self, context): external_job_id = context['task_state'].get(RESUME_KEY) if external_job_id is None: external_job_id = _submit_external_job(...) context['task_state'].set(RESUME_KEY, external_job_id) # ... poll, with SIGTERM handler that raises AirflowTaskCheckpointed ``` A real operator (Databricks, EMR, Spark-on-K8s) follows the same shape but calls actual APIs in submit / poll. ## Reviewer questions - Is this the right shape for the v1 integration pattern, or should AIP-96 ship a helper base class (`ResumableExternalJobOperator` with a declared `resume_attribute`) as part of v1 instead? - Should the SIGTERM handler be installed by the framework rather than by the operator? Tradeoff: framework convenience vs. operator-author control. - Where should resumable operators live — in providers per-platform, or a generic helper in `airflow-core`? ## To exercise the resume flow end-to-end (manual) 1. Spin up airflow standalone with this branch. 2. Trigger `example_resumable_operator` DAG. 3. While the task is RUNNING, send SIGTERM to the worker process. 4. Observe task transitions to CHECKPOINTED. 5. Scheduler dispatches next attempt at next scheduling tick. 6. Operator reads prior `external_job_id` from task_state and resumes polling without resubmitting. A formal e2e test for this flow is planned but not in this PR. ## Stacking This PR depends on: - #66402 (CHECKPOINTED state + AirflowTaskCheckpointed exception) - #66410 (on_task_instance_checkpointed listener hook) - #66445 (supervisor → API server → DB wiring) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
