GitHub user nathadfield edited a discussion: Event-driven scheduling: duplicate DAG runs when an `AssetWatcher` runs on multiple triggerers
## The problem When you run more than one triggerer (the default for HA), an `AssetWatcher` that polls some external state seems to fire once *per triggerer* for the same change, so the consumer DAG runs twice (N times for N triggerers). We hit this on 3.2.1 with a watcher that polls a Git repo's `main` HEAD and triggers a re-index when it advances. With two triggerers, every merge kicks off two identical runs of a fairly expensive pipeline. Cron and manual runs are fine; only the asset-triggered ones double up, and the count tracks the number of triggerers. The watcher *definition* is deduped (one trigger backs the asset), but the *events* aren't: each triggerer runs its own copy of the watcher, independently notices the same change, emits its own event, and each event creates a run. So it's effectively at-least-once delivery with no built-in way to get "one run per change" for a plain poller. ## What's already there (and why it doesn't seem to cover this) A couple of adjacent things exist: - **Shared-stream triggers** (3.3, #66584 / #67523) let sibling watchers share one poll loop and add a producer-side ack channel for message brokers. But the docs are explicit that the saving is "at the poll-loop and upstream-I/O layer, not at the persistence or scheduling layer", so it reduces polling, not the number of events/runs. For a plain (non-broker) poller across triggerers, you'd still get one event per triggerer. - The docs **do** say subscribers must be idempotent, but in the context of the ack channel and triggerer restart (broker redelivery). The plain multi-triggerer poll-watcher duplication doesn't seem to be called out. So this may already be considered expected behaviour, and part of what I'm asking is whether it is. ## How this differs from #54491 / #63507 Those are scheduler-side: multiple *schedulers* turning a **single** event into two runs (fixed with a row-level lock, #60773). Here there are genuinely **two events** from two *triggerers*, so that fix doesn't apply. ## What we do about it today The usual "make the consumer idempotent" approach: the DAG's first step skips if it's already processed this particular change. It works, but every author of an event-driven DAG has to reinvent it, and the duplicate run still gets scheduled before it's skipped. ## Possible directions - A **dedupe key on `AssetWatcher`** (derived from the event) so identical events collapse into one run. Feels like the most natural fit, and complements shared-stream, which deliberately stops at the poll layer. - **Coalescing identical events** on the scheduler side within a short window. - An opt-in **single-runner watcher** (one triggerer owns it, with failover) for cases where you'd rather have exactly-once than zero-gap coverage. More architectural, hence a discussion. - Failing any of that, **document** the plain multi-triggerer poll case explicitly, alongside the existing idempotency notes. ## Questions - Is triggerer-side duplication for a plain poller a known/accepted property, or a gap worth closing at the scheduling layer (given shared-stream intentionally stops short of it)? - If it's worth closing, is a dedupe key on `AssetWatcher` the right first step, or does the single-runner idea need an AIP? - Is there a way to make a polling watcher exactly-once across triggerers today that I've missed? Related: #54491, #63507, #60773, #66584, #67523. GitHub link: https://github.com/apache/airflow/discussions/69319 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
