Hi all,
Event-driven shouldn't mean polling faster. AIP-125: Webhooks [1] lets any
system push an event into Airflow, and Airflow acts on it the moment it
arrives: start a Dag, wake a waiting task, or cancel a run.
This follows the "Support Inbound webhooks for Airflow" thread [2], and I'd
like to start the discussion on the AIP here.
Thanks Kaxil, Jarek and Constance for the feedback there. I reshaped the
proposal: it's no longer only about turning webhooks into asset events.
The idea
Airflow gets its own webhook endpoint, POST /webhooks/<name>. Each sender
gets a name, like github, and an Airflow connection that holds its shared
secret. Airflow checks the sender's signature, records the delivery and
acts on it. Support for each service lives in its provider: the GitHub
provider knows GitHub's webhooks, and the standard provider has generic
sources for any service that signs its requests or sends a secret header.
The endpoint can run as its own process, airflow api-server --apps
webhooks, so you can put it on the internet without the UI or the REST API.
Every delivery shows up on a new Webhook Deliveries page, along with what
Airflow did with it.
It comes in three phases:
1. Dags run when an event arrives. A Dag schedules on a webhook topic:
preview_requested = WebhookTopic(
webhook="github",
path="dheerajturaga/inbound-webhooks-demo",
event="pull_request",
action="labeled",
detail="deploy-preview",
)
@dag(schedule=preview_requested)
def preview_environment(): ...
2. Tasks wait for an external event. WaitForWebhookOperator,
@task.webhook_wait, or wait_for_webhook() from any operator. While it
waits, the task holds no worker slot and no trigger, and the matching
webhook wakes it right away.
3. Dags and tasks cancel when an event says so. A Dag or a task lists the
events that cancel it with cancel_on, for example "cancel the preview when
its pull request is closed".
The earlier feedback
- More than assets: Phase 2 resumes a waiting task and Phase 3 cancels a
run or a task. Phase 1 still starts Dags through asset events, so the
scheduler creates those runs the way it does today.
- Pointing at one run: a wait matches on fields of the event, such as the
pull request number. A cancel links the event to the run by what it's
about, so closing pull request 12 only cancels the run for pull request 12.
A manual run can join in by passing the key in its conf.
- Batching: as Constance said, from 3.4 each event gets its own run by
default, and cancel_on relies on that.
- Per-task callback tokens and answering HITL tasks from Slack are not in
this AIP. Both can sit on the same endpoint, and I'd like to propose them
separately once this lands.
Try it
- The provider this started from runs Phase 1 in production at my company:
https://pypi.org/project/airflow-provider-inbound-webhooks/ (its README has
a quick start that needs no third-party account).
- All three phases are implemented in core as a stack of draft PRs:
https://github.com/apache/airflow/stack/75579
All feedback welcome.
Thanks,
Dheeraj
[1]
https://cwiki.apache.org/confluence/spaces/AIRFLOW/pages/451980090/AIP-125+Webhooks
[2] https://lists.apache.org/thread/p3dpnor4zwko8tldpncj6ftj1bpp18q7
