ashb commented on pull request #17576: URL: https://github.com/apache/airflow/pull/17576#issuecomment-916413420
@malthe Valid point -- I hadn't considered something like "give my stream processing up to X minutes to process the data for this interval and make it available". But I think that should be _explicit_ in the dag. If this task should wait until 30mins after the interval is over, then to make it easy to reason about that should be a (Async)TimeDeltaSensor. Explicit is better than magic, and using pre_execute to run custom code to skip feels like magic to me. This whole feature just adds a lot of scope for a lot of "accidental complexity" in both Airflow itself, and in the DAGs people write. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
