I think the kinds of plugins you are talking make sense in some contexts (e.g. custom views when clicking on a task in the UI, e.g. ability to visualize the data an ETL job provides) but we should be careful allowing extensions to more core parts, it will become very hard to change/maintain the project. From what I've seen the biggest gaps in customization/lowest hanging fruit is extending the REST API and making it more robust instead of experimental.
On Tue, Feb 18, 2020 at 7:33 AM Jarek Potiuk <jarek.pot...@polidea.com> wrote: > I have another discussion to start. We've recently talked to a number of > customers who are extending airflow. It's often the case that people are > modifying airflow's source code and later have a hard time with updating it > when newer versions of Airflow are released. > > This is a common trait - we've heard the same story from at least three of > our customers and it was also mentioned today at Slack by one of the users: > https://apache-airflow.slack.com/archives/CSS36QQS1/p1582014236100800 and > I heard some anecdotal evidence of people doing it at many events I spoke. > > We have a plugin mechanism that allows for some extensibility but I wonder > if we should do something more complete. Maybe we could gather from people > a list of ways people are extending Airflow currently by modifying it's > code and maybe we can come up with some "extension points" that we might > introduce to Airflow to let them add custom functionality they want rather > than modifying Airflow's code. > > I think many of the extensions do not need to modify Airflow's source code, > and there won't be many extensions (maybe even we already have all that we > need but people do not know that they can extend airflow without modifying > the code. I think it would require a bit more description (and maybe some > verification) of what internal API of Airflow is for those extensions so > that we can keep backwards compatibility > > Let me know what you think? Is it worth it? Do you see any problems with > trying to manage that? Maybe that's something we could introduce in 2.0 (at > least by better documenting what is "an extension" and providing some > examples on how Airflow can be extended. > > J. > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> >
