Awesome Jarek, that was my fear as well. Good to hear that we still run the full test suite.\
Cheers, Fokko Op vr 18 okt. 2019 om 15:47 schreef Daniel Imberman < daniel.imber...@gmail.com>: > Oh then that sounds fine in that case :) > > via Newton Mail [ > https://cloudmagic.com/k/d/mailapp?ct=dx&cv=10.0.32&pv=10.14.5&source=email_footer_2 > ] > On Fri, Oct 18, 2019 at 10:46 AM, Jarek Potiuk <jarek.pot...@polidea.com> > wrote: > It's even more than daily. In the latest version every time after we merge > to master full suite of tests will be run - which means that we can > immediately see which commit caused the problem. It's only at the PR level > the tests will be "smart". > > On Fri, Oct 18, 2019 at 3:43 PM Daniel Imberman <daniel.imber...@gmail.com > > > wrote: > > > I’m still somewhat hesitant on this as it could allow regressions to peak > > through, though as long as we’re still doing the daily build, and with > our > > soon-to-be-created prerelease load testing I think we should be okay. > > > > On Fri, Oct 18, 2019 at 10:30 AM, Jarek Potiuk <jarek.pot...@polidea.com > > > > wrote: > > Seems that our tests got way smarter now :). > > > > I just implemented the "smartness" it with > > https://github.com/apache/airflow/pull/6321 and trying to workaround > > Kubernetes problem we have :). > > The doc changes are short enough that there is no need to optimise those. > > Doc-test only should execute much much faster now :). > > For details see https://issues.apache.org/jira/browse/AIRFLOW-5649 > > > > J. > > > > On Fri, Aug 23, 2019 at 4:09 PM James Meickle > > <jmeic...@quantopian.com.invalid> wrote: > > > > > GitHub recently introduced the idea of "Draft" PRs: > > > https://github.blog/2019-02-14-introducing-draft-pull-requests/ > > > > > > Could we do something similar either with that system or something > else? > > > Run a minimal set until it's marked as "ready for testing", and then > run > > a > > > larger suite. > > > > > > On Fri, Aug 23, 2019 at 10:01 AM Kaxil Naik <kaxiln...@gmail.com> > wrote: > > > > > > > Maybe your 4th point covers this but there are frequent Doc only > > changes. > > > > In this case we should not run "real-test" but only static checks - > > mypy, > > > > pylint. flake8, doc generation. > > > > > > > > On Fri, Aug 23, 2019 at 2:39 PM Jarek Potiuk < > jarek.pot...@polidea.com > > > > > > > wrote: > > > > > > > > > Hello everyone, > > > > > > > > > > On top of moving out from Travis (I will resume working on it next > > > week) > > > > I > > > > > thought about some ways to improve the feedback cycle time we have > > with > > > > CI > > > > > (super long now). > > > > > > > > > > Maybe we should consider being smarter with running tests only when > > > > really > > > > > needed. > > > > > > > > > > After introducing pre-commit framework, I thought that it is rather > > > smart > > > > > in selecting which tests to run locally based on what files > changed. > > > It's > > > > > not perfect of course and there are edge cases where we change .xml > > > files > > > > > and python tests stop running (for example), but maybe we can > > introduce > > > > > some smartness in our test execution scripts based on similar > > > principles. > > > > > > > > > > First of all we should always run all tests after master is merged > > > > (catches > > > > > edge cases missed by partial running in PR). Then in PRs we could > run > > > > > partial tests: > > > > > > > > > > 1) Do not run Python tests if none of .py files change > > > > > > > > > > 2) Do not run Doc generation if neither .py nor .rst files change > > > > > > > > > > 3) [This might be controversial / not catching a lot of problems] - > > > only > > > > > run related tests - for corresponding packages - when .py files > > change. > > > > > > > > > > 4) Only run "real unit tests" if none of the > operators/hooks/sensors > > > > change > > > > > (we would have to introduce a way to distinguish unit tests - > > requiring > > > > > only basic Airflow + database - from integration tests where > > > > > hooks/sensors/operators change). This could be done using the > > > > > slimmer/smaller/future production CI image - without the overhead > of > > > > > starting up all the services. > > > > > > > > > > 5) Run only static checks + real/related tests in PR for > > > > > "drafts/in-progress PRs" but only trigger full tests when someone > > marks > > > > it > > > > > as "Ready to merge" (via label or comment in PR). > > > > > > > > > > Of course if we can make it faster with no-Travis setup running > full > > > > tests > > > > > in all PRs makes more sense, but maybe some of those options above > > are > > > > also > > > > > acceptable. > > > > > > > > > > Comments? > > > > > > > > > > J. > > > > > > > > > > -- > > > > > > > > > > Jarek Potiuk > > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > > > > > -- > > > > Jarek Potiuk > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] <https://www.polidea.com/> > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/>
