I concur. There are subtle differences in SQLAlchemy behavior depending on the backend... In that context testing against multiple DBMS makes sense and adds value.
@Conrad: I do see your point though, testing libraries should not be Airflow's responsibility but from a pragmatic point of view we should continue to do so because it makes the development process more robust. On Fri, Jun 28, 2019 at 8:21 AM Driesprong, Fokko <fo...@driesprong.frl> wrote: > Yes Conrad, this is actually true. > > However I think it is important to test different backends, as an example, > we had a lot of issues with MySQL's timezone behavior, besides that, it > would be possible to test specific database evolution steps for certain > backends. > > Cheers, Fokko > > Op vr 28 jun. 2019 om 14:11 schreef Conrad Dean <conrad.p.d...@gmail.com>: > > > Regarding the matrix of backends to test -- is this testing specific > > platform integrations, or if everything is routed through SQLAlchemy's > ORM > > does this test suite just end up testing SQLAlchemy? > > > > It makes sense to have the ability to swap backends out for the sake of > > reproducing issues when something comes up, but exhaustively providing > > coverage for a library that Airflow uses feels a bit out-of-place. This > > would only make sense if the project has its own extensions or plugins on > > top of SQLAlchemy, or uses its own forked version of it. > > > > If the test suite was limited to one backend for CI it would reduce the > > build times by quite a bit. > > > > On Thu, Jun 27, 2019 at 9:28 AM Jarek Potiuk <jarek.pot...@polidea.com> > > wrote: > > > > > Yeah. I also have a working version of Cloud build configuration and we > > can > > > run the tests on cloud build if we can get some credits from Google. > And > > > the changes from the upcoming CI image will make it much easier to run > > > tests on any CI provider. Except Kubernetes tests they are pretty much > > > CI-agnostic. Kubernetes tests will likely be also fixed soon. > > > > > > Another idea: I thought that in the future we can also run only subset > of > > > postgres/mysql/sqlite tests on all combinations. I think there are just > > > handful of tests that are specific for backend (and we already know > which > > > ones they are - they are skipped-if). > > > > > > J. > > > > > > Principal Software Engineer > > > Phone: +48660796129 > > > > > > czw., 27 cze 2019, 15:12 użytkownik Philippe Gagnon < > > philgagn...@gmail.com > > > > > > > napisał: > > > > > > > I think the combinations that you are proposing are sensible for > > > pre-merge > > > > checks. > > > > > > > > I am working on a proposal to offload extra combinations to another > CI > > > > provider (Azure DevOps specifically seems like a good candidate), > > either > > > > pre or post merge. Ideally I'd like to run more combinations > pre-merge > > > but > > > > there is a trade-off to be conscious of here between development > > velocity > > > > and quality assurance, which I think this issue highlights quite > well. > > > > > > > > Please let me know your thoughts > > > > > > > > Philippe > > > > > > > > On Thu, Jun 27, 2019 at 9:05 AM Jarek Potiuk < > jarek.pot...@polidea.com > > > > > > > wrote: > > > > > > > > > Agree that we should be thoughtful about others as well: In the > > latest > > > > push > > > > > (few minutes ago) of the upcoming official CI image i implemented > the > > > > > change we discussed in the Github where we limit the number of > > > > combinations > > > > > we test: > > > > > > > > > > You can see it yourself: > > > > > https://travis-ci.org/apache/airflow/builds/551305240 > > > > > > > > > > Those are the combinations I propose: > > > > > > > > > > Python: 3.6 > > > > > BACKEND=mysql ENV=docker > > > > > > > > > > Python: 3.6 > > > > > BACKEND=postgres ENV=docker > > > > > > > > > > Python: 3.5 > > > > > BACKEND=sqlite ENV=docker > > > > > > > > > > Python: 3.6 > > > > > BACKEND=postgres ENV=kubernetes KUBERNETES_VERSION=v1.13.0 > > > > > > > > > > J, > > > > > > > > > > > > > > > On Thu, Jun 27, 2019 at 11:00 AM Driesprong, Fokko > > > <fo...@driesprong.frl > > > > > > > > > > wrote: > > > > > > > > > > > We got this message last year: > > > > > > > > > > > > > Hello, Airflow PPMC. > > > > > > > While going through the usage statistics for our Travis CI > > > service, I > > > > > > > have noticed that the Airflow project is using an abnormally > > large > > > > > > > amount of resources, 2600 hours per month or the equivalent of > > > having > > > > > > > almost 4 machines building airflow non-stop 24/7. As this is > not > > > > free, > > > > > > > but rather costing us money, I'm contacting you with the > > intention > > > of > > > > > > > figuring out ways to reduce the use of Travis for the project. > > > > > > > > > > > > > We would greatly prefer that the project itself comes up with a > > > > > solution > > > > > > > to lower the usage of Travis, as we'd hate to simply turn it > off > > > for > > > > > > > you, but the usage is at a rather severe level, totaling more > > than > > > > 21% > > > > > > > of the total build time of all projects using Travis, so > > something > > > > > > > actionable should be decided upon and (preferably) completed by > > the > > > > end > > > > > > > of May that will reduce the consumption of Travis resources. > > > > > > > > > > > > > Alternately, if you are unable to lower the pressure on Travis, > > the > > > > > > > podling and/or IPMC may ask the board of directors for a > separate > > > > > budget > > > > > > > for additional build nodes to cope with the added load - I'll > > leave > > > > > this > > > > > > > for the podling and IPMC to decide on. > > > > > > > > > > > > > Please let us know when you have decided on a plan to remedy > this > > > > > > situation. > > > > > > > > > > > > > With regards, > > > > > > > Daniel on behalf of ASF Infrastructure. > > > > > > > > > > > > I think more and more projects are still migrating to the ASF > > Travis, > > > > so > > > > > I > > > > > > think natural that there is more load. However, this still leaves > > the > > > > > > question if we have to run the full matrix. > > > > > > > > > > > > Cheers, Fokko > > > > > > > > > > > > > > > > > > > > > > > > Op do 27 jun. 2019 om 10:56 schreef Jarek Potiuk < > > > > > jarek.pot...@polidea.com > > > > > > >: > > > > > > > > > > > > > I think we should really involve infra to increase the slot > > number > > > or > > > > > > maybe > > > > > > > even somehow allocate slots per project. > > > > > > > The problem is that we cannot control what other apache > projects > > > are > > > > > > doing, > > > > > > > so even if we decrease our runtime, it's the other projects > that > > > > might > > > > > > hold > > > > > > > us in the queue :( > > > > > > > > > > > > > > J. > > > > > > > > > > > > > > On Thu, Jun 27, 2019 at 10:19 AM Driesprong, Fokko > > > > > <fo...@driesprong.frl > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > I've noticed this at other Apache projects as well, sometimes > > it > > > > > takes > > > > > > up > > > > > > > > to 7-8 hours. The only thing we can do, is reduce the runtime > > of > > > > the > > > > > > jobs > > > > > > > > so we take less slots :-) > > > > > > > > > > > > > > > > Cheers, Fokko > > > > > > > > > > > > > > > > Op wo 26 jun. 2019 om 21:59 schreef Jarek Potiuk < > > > > > > > jarek.pot...@polidea.com > > > > > > > > >: > > > > > > > > > > > > > > > > > Yep. That's what I suggested as the reason in the ticket - > I > > > > guess > > > > > > > INFRA > > > > > > > > > are the only people who can do anything about it (increase > > > > > > concurrency > > > > > > > ? > > > > > > > > > pay more for Travis :)? ). > > > > > > > > > > > > > > > > > > On Wed, Jun 26, 2019 at 9:51 PM Ash Berlin-Taylor < > > > > a...@apache.org> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > I asked Travis on twitter and they said it was due to the > > > > Apache > > > > > > > other > > > > > > > > > > projects build queues > > > > > > > > > > > > > > > > > > > > https://twitter.com/travisci/status/1143893051460526080 > > > > > > > > > > > > > > > > > > > > -ash > > > > > > > > > > > > > > > > > > > > On 26 June 2019 20:48:33 BST, Jarek Potiuk < > > > > > > jarek.pot...@polidea.com > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > >> > > > > > > > > > >> Hello everyone, > > > > > > > > > >> > > > > > > > > > >> For the last few days the Travis builds for > apache/airflow > > > > > project > > > > > > > are > > > > > > > > > >> waiting in a queue for hours. This is not a normal > > > situation. > > > > > I've > > > > > > > > > opened > > > > > > > > > >> INFRA ticket for that: > > > > > > > > > https://issues.apache.org/jira/browse/INFRA-18657 > > > > > > > > > >> > > > > > > > > > >> J. > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > > > Jarek Potiuk > > > > > > > > > Polidea <https://www.polidea.com/> | Principal Software > > > Engineer > > > > > > > > > > > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > > > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > Jarek Potiuk > > > > > > > Polidea <https://www.polidea.com/> | Principal Software > Engineer > > > > > > > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Jarek Potiuk > > > > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > > > > > > > M: +48 660 796 129 <+48660796129> > > > > > [image: Polidea] <https://www.polidea.com/> > > > > > > > > > > > > > > >
