I have another discussion to start. We've recently talked to a number of customers who are extending airflow. It's often the case that people are modifying airflow's source code and later have a hard time with updating it when newer versions of Airflow are released.
This is a common trait - we've heard the same story from at least three of our customers and it was also mentioned today at Slack by one of the users: https://apache-airflow.slack.com/archives/CSS36QQS1/p1582014236100800 and I heard some anecdotal evidence of people doing it at many events I spoke. We have a plugin mechanism that allows for some extensibility but I wonder if we should do something more complete. Maybe we could gather from people a list of ways people are extending Airflow currently by modifying it's code and maybe we can come up with some "extension points" that we might introduce to Airflow to let them add custom functionality they want rather than modifying Airflow's code. I think many of the extensions do not need to modify Airflow's source code, and there won't be many extensions (maybe even we already have all that we need but people do not know that they can extend airflow without modifying the code. I think it would require a bit more description (and maybe some verification) of what internal API of Airflow is for those extensions so that we can keep backwards compatibility Let me know what you think? Is it worth it? Do you see any problems with trying to manage that? Maybe that's something we could introduce in 2.0 (at least by better documenting what is "an extension" and providing some examples on how Airflow can be extended. J. -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
