Hello everyone, On top of moving out from Travis (I will resume working on it next week) I thought about some ways to improve the feedback cycle time we have with CI (super long now).
Maybe we should consider being smarter with running tests only when really needed. After introducing pre-commit framework, I thought that it is rather smart in selecting which tests to run locally based on what files changed. It's not perfect of course and there are edge cases where we change .xml files and python tests stop running (for example), but maybe we can introduce some smartness in our test execution scripts based on similar principles. First of all we should always run all tests after master is merged (catches edge cases missed by partial running in PR). Then in PRs we could run partial tests: 1) Do not run Python tests if none of .py files change 2) Do not run Doc generation if neither .py nor .rst files change 3) [This might be controversial / not catching a lot of problems] - only run related tests - for corresponding packages - when .py files change. 4) Only run "real unit tests" if none of the operators/hooks/sensors change (we would have to introduce a way to distinguish unit tests - requiring only basic Airflow + database - from integration tests where hooks/sensors/operators change). This could be done using the slimmer/smaller/future production CI image - without the overhead of starting up all the services. 5) Run only static checks + real/related tests in PR for "drafts/in-progress PRs" but only trigger full tests when someone marks it as "Ready to merge" (via label or comment in PR). Of course if we can make it faster with no-Travis setup running full tests in all PRs makes more sense, but maybe some of those options above are also acceptable. Comments? J. -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
