Maybe your 4th point covers this but there are frequent Doc only changes. In this case we should not run "real-test" but only static checks - mypy, pylint. flake8, doc generation.
On Fri, Aug 23, 2019 at 2:39 PM Jarek Potiuk <jarek.pot...@polidea.com> wrote: > Hello everyone, > > On top of moving out from Travis (I will resume working on it next week) I > thought about some ways to improve the feedback cycle time we have with CI > (super long now). > > Maybe we should consider being smarter with running tests only when really > needed. > > After introducing pre-commit framework, I thought that it is rather smart > in selecting which tests to run locally based on what files changed. It's > not perfect of course and there are edge cases where we change .xml files > and python tests stop running (for example), but maybe we can introduce > some smartness in our test execution scripts based on similar principles. > > First of all we should always run all tests after master is merged (catches > edge cases missed by partial running in PR). Then in PRs we could run > partial tests: > > 1) Do not run Python tests if none of .py files change > > 2) Do not run Doc generation if neither .py nor .rst files change > > 3) [This might be controversial / not catching a lot of problems] - only > run related tests - for corresponding packages - when .py files change. > > 4) Only run "real unit tests" if none of the operators/hooks/sensors change > (we would have to introduce a way to distinguish unit tests - requiring > only basic Airflow + database - from integration tests where > hooks/sensors/operators change). This could be done using the > slimmer/smaller/future production CI image - without the overhead of > starting up all the services. > > 5) Run only static checks + real/related tests in PR for > "drafts/in-progress PRs" but only trigger full tests when someone marks it > as "Ready to merge" (via label or comment in PR). > > Of course if we can make it faster with no-Travis setup running full tests > in all PRs makes more sense, but maybe some of those options above are also > acceptable. > > Comments? > > J. > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> >
