Seems tests are stable - but the kerberos problem is happening often enough to take a look. I will see what I can do to make it stable. seems that might be a race between kerberos initialising and tests starting to run.
On Mon, Jan 13, 2020 at 8:58 PM Jarek Potiuk <[email protected]> wrote: > Just merged the change with integration separation/slimming down the tests > on CI. https://github.com/apache/airflow/pull/7091 > > It looks like it is far more stable, I just had one failure with kerberos > not starting (which also happened sometimes with old tests). We will look > in the future at some of the "xfailed/xpassed" tests - those that we know > are problematic. We have 8 of them now. > > Also Breeze is now much more enjoyable to use. Pls. take a look at the > docs. > > J. > > On Wed, Jan 8, 2020 at 2:23 PM Jarek Potiuk <[email protected]> > wrote: > >> I like what you've done with the separate integrations, and that coupled >>> with pytest markers and better "import error" handling in the tests would >>> make it easier to run a sub-set of the tests without having to install >>> everything (for instance not having to install mysql client libs. >> >> >> Cool. That's exactly what I am working on in >> https://github.com/apache/airflow/pull/7091 -> I want to get all the >> tests run in integration-less CI, select all those that failed and treat >> them appropriately. >> >> >>> Admittedly less of a worry with breeze/docker, but still would be nice >>> to skip/deselct tests when deps aren't there) >>> >> >> Yeah. For me it's the same. I think we had recently a few discussions >> with first time users that they have difficulty contributing because they >> do not know how to reproduce failing CI reliably locally. I think the >> resource of Breeze environment for simple tests was a big >> blocker/difficulty for some users so slimming it down and making it >> integration-less by default will be really helpful. I will also make it the >> "default" way of reproducing tests - i will remove the separate bash >> scripts which were an intermediate step. This is the same work especially >> that I use the same mechanism and ... well - it will be far easier for me >> to have integration - specific cases working in CI if i also have Breeze >> to support it (eating my own dog food). >> >> >>> Most of these PRs are merged now, I've glanced over #7091 and like the >>> look of it, good work! You'll let us know when we should take a deeper look? >>> >> >> Yep I will. I hope today/tomorrow - most of it is ready. I also managed >> to VASTLY simplified running kubernetes kind (One less docker image, >> everything runs in the same docker engine as the airflow-testing itself) in >> https://github.com/apache/airflow/pull/6516 which is prerequisite for >> #7091 - so both will need to be reviewed. I marke >> >> >>> For cassandra tests specifically I'm not sure there is a huge amount of >>> value in actually running the tests against cassandra -- we are using the >>> official python module for it, and the test is basically running these >>> queries - DROP TABLE IF EXISTS, CREATE TABLE, INSERT INTO TABLE, and then >>> running hook.record_exists -- that seems like it's testing cassandra >>> itself, when I think all we should do is test that hook.record_exists calls >>> the execute method on the connection with the right string. I'll knock up a >>> PR for this. >>> Do we think it's worth keeping the non-mocked/integration tests too? >>> >> >> I would not remove them just yet. Let's see how it works when I separate >> it out. I have a feeling that we have very little number of those >> integration tests overall so maybe it will be stable and fast enough when >> we only run those in a separate job. I think it's good to have different >> levels of tests (unit/integration/system) as they find different types of >> problems. As long as we can have integration/system tests clearly >> separated, stable and easy to disable/enable - I am all for having >> different types of tests. There is this old and well established concept of >> Test Pyramid https://martinfowler.com/bliki/TestPyramid.html which >> applies very accurately to our case. By adding markers/categorising the >> tests and seeing how many of those tests we have, how stable they are, how >> long they are and (eventtually) how much it costs us - we can make better >> decisions. >> >> J. >> >> > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> > > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
