Hi all, Thanks a lot for the feedback and inputs. Greatly greatly appreciated!
Let me try to answer the questions & share my findings one by one: To Daniel’s question: the two examples I shared earlier may not be super solid. I actually “made up” them, and the use case we are having in mind may be more complex, like dynamically deciding execution order among TIs. The whole idea is about to allow a customized/flexible scheduling when needed. A common concern I see from folks are potential impact on performance. I totally agree with Ash to limit it only to power users who truly know what they are doing, and Jarek was making a great point that clear guidance/examples should be provided if we add this as a feature. I also tried to follow the approach Ash shared (set `deps` in operator class from `task_policy`), and then I realized a few points worth sharing here: - Adding a custom `ti_dep` into `deps` in operator class from `task_policy` itself would fail. It actually fails the DAG parsing, and it requires register the custom `ti_dep` in the Plugins. That’s when I realize actually we already support custom `ti_dep` in Airflow (https://github.com/apache/airflow/pull/22698) since 2.3.0. Without adding your custom `ti_dep` via Plugin properly, you will not be able to set that in the operator class’s `deps` from `task_policy`. - After adding my own dummy custom ti_dep via Airflow Plugin, it’s actually failing the whole scheduling, and may result in “Task deadlock” for TI execution. The questions I have for the current implementation are: - There is zero documentation about the “add custom ti_deps in Airflow Plugin” (https://github.com/apache/airflow/pull/22698). Was that a miss or was it intended? - Setting `deps` in operator class from `task_policy` may not be the solution that I have been looking for: it means the custom ti_deps will be evaluated almost everywhere, in eluding when we serialize operator info (like `_serialize_node`), while we may only want to evaluate it at the TI scheduling stage. I would love to hear more thoughts from you. Many thanks! XD
