Leondon9 commented on PR #70225: URL: https://github.com/apache/airflow/pull/70225#issuecomment-5181591290
I opened the original discussion behind this (#67375), so let me add the concrete case, since the "why task-level?" question is the crux. The workload is a **time-scheduled** batch consumer (`@daily`, etc.) whose SQL is written against its own `data_interval_start` / `_end`. Mid-run it needs to wait for an upstream asset partition (e.g. `orders / dt=<interval>`) produced by a *different* Dag on its own cadence. Why the existing mechanisms don't quite cover it: - **Asset-schedule the consumer / `AssetOrTimeSchedule`:** asset-triggered runs get a *degenerate* data interval — `DataInterval.exact(run_after)`, pinned to the trigger instant — not the window the consumer's SQL is built on. Moving the wait onto the schedule changes the run's contract, even though the consumer still wants its own time interval. - **`RollupMapper` + `WaitPolicy`:** scheduler-side — it gates whether a *run is created* from partition-aligned events. It can't gate *between tasks* in an already-running, time-scheduled Dag, which is where the wait belongs here. On the visibility concern (the strongest objection, I think): I don't think satisfying it requires scheduling on the asset. Airflow already separates lineage from scheduling — inlets/outlets draw the dependency graph independently of what triggers a run. A task-level sensor can declare the awaited asset as an inlet, so the dependency stays visible to the platform (asset graph, lineage, OpenLineage) without coupling the run to asset-triggering. That keeps both: the consumer keeps its time schedule and data interval, and the data dependency stays first-class. So I'd argue this is a genuine gap for time-partitioned batch consumers — the alternatives either change the run's contract or act at the wrong layer — and a task-level sensor is complementary to asset scheduling, not a replacement for it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
