Hi everyone, Sending a new message to everyone to gather feedback on the AIP-31 about Airflow functional DAG API. This was initially discussed and proposed in [DISCUSS] Airflow functional DAGs. After leaving open a small doc to iterate on the proposal for a couple weeks, I decided to move forward to formalize it as an AIP document in confluence. I will still be tracking comments on the doc but would rather move the conversation here.
AIP - 31 focuses on solving the issue about implicit message passing in Airflow by extending the DAG/Operator API to include a way to set up a message passing dependency on the same DAG file. This is complimentary to task dependency declaration and is intended to be used as another option to declare dependencies by declaring message dependencies. In addition, AIP-31 proposes a way to declare PythonOperators from a function using decorators. This should help embed custom behavior into DAGs without needing to create custom operators for everything. Changes proposed: • Add __call__ function in BaseOperator: Add a functional interface to replace class attributes on execution time. • XComArg class: This object is a reference to an XCom value that has not been created and will need to be resolved in the future. • PythonFunctionalOperator and Python Function Operator: Extend PythonOperator to map op_args and op_kwargs from a decorated interface for easier set up and add a decorator to create PythonOperators from a function in an easier way. See AIP document for a DAG example. Any help on how to proceed with this will be appreciated (also joined Slack and posted in. #airflow-creative). My guess is once we agree on what the API should look like, next step is to do a vote and if successful to create JIRA issues and GitHub PRs with the mentioned changes. Please let me know if there’s any aspect that people feel strongly opinionated against or aspects that are not clear and I should work on expanding further. Best, Gerard Casas Saez Twitter | Cortex | @casassaez
