Hello All, Since we are voting now on AIP-10, Multi-layered/Multi-staging docker image. I would like to start discussion on the follow-up AIP-7 Simplified Development Workflow.
Here is a short 10 min screencast/video showing Airflow Breeze in action: https://youtu.be/ffKFHV6f3PQ . It shows how you can start running Airflow Integration Tests (same that are run in CI) in less than 10 minutes from the scratch. For now - I reused our "*Airflow Breeze*" catchy name, logo and tagline ("It's a Breeze to develop Airflow") - which we will gladly donate to the community. I am not sure if community will like the name and we can change it of course, but I personally think it's funny and catchy and to-the point ;). It looks like it might be a good timing now to introduce it - for those who watched Ash's "Learn how to hack on Airflow" - i believe "Simplified Development Workflow" automates a lot of the manual tweaking that Ash shown. I am already quite advanced with the implementation as I ported most of it from an earlier environment that we developed in our company Polidea and I use it all the time in testing AIP-10. Several people were already involved in testing it and I saw some good comments. I encourage you to take a look and comment (in the AIP or in Github directly). Especially I would like some feedback from Linux users as I am developing mostly on Mac. I tested it on Linux and it should work but Docker on Mac works a bit differently than on Linux and I expect some small problems to be revealed when someone start using it more. Here are al the resources: - AIP-7: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-7+Simplified+development+workflow - The PR I am working on https://github.com/apache/airflow/pull/4932 - The current Breeze documentation in my branch: https://github.com/PolideaInternal/airflow/blob/simplified-development-workflow/BREEZE.rst - And once again the video: https://youtu.be/ffKFHV6f3PQ The documentation is pretty comprehensive (also mostly ported from our development environment). It describes various ways you can run tests with Airflow - including debugging using IDE integration (IntelliJ), debugging with ipdb in console etc. J. -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> E: [email protected]
