I would expect acceptable test run time to be somewhat bimodal: maximum
around 100ms for unit tests (which should be the majority of tests) and
minutes for functional and integration tests. It would be good for Travis
to run all these tests on every PR, but it would be nice if I could limit
my local test runs to only the unit tests.
My biggest pain point when working with calcite is the size of the core
project. Individual tests are mostly fast, but just building core takes 100
seconds. It would also be easier to run just the relevant tests if there
were several smaller maven projects.
Andrew
On Mon, Sep 10, 2018 at 4:53 AM Vladimir Sitnikov <
sitnikov.vladi...@gmail.com> wrote:
> Michael>just because a test is slow of course doesn't mean it's not useful
>
> Does "slow test" impact your development experience in a bad way?
> Does "slow test" impact contributor's experience in a bad way?
> Do things like "MaterializationTest 160sec" and "LatticeTest 212sec" make
> materialization/lattice much stronger?
>
> Spoiler: yes, yes, no. In the latter case, most of the time is spent not in
> materialization/lattice engine, but most of the time is spent in data
> loading.
> That is the tests cover very little cases, and they spend too much time in
> testing HSQLDB (or whatever DB is used to load the data).
> In other words, current MaterializationTest/LatticeTest are much less
> related to Calcite than they relate to H*DB.
>
> Michael>I'm not sure it makes sense to have one single duration that's
> acceptable
> for all tests
>
> Of course there are always exceptions. Like the one I have listed with
> "Cassandra startup".
>
> Let me rephrase: "what test duration would make your eyebrow to raise?"
> What test duration would raise both of your eyebrows?
>
> Michael>We could also put some slower tests behind a define since
> just because a test is slow of course doesn't mean it's not useful.
>
> Calcite is an intermediate framework. It is not a database. It is not a
> training framework for machine learning.
> Thus Calcite should not take much time to perform its duties.
>
> For instance, if simple query fetches 100 rows by primary key literals
> takes 80 seconds to "prepare" in Calcite, I would definitely call it a bug.
> It does not mean that that case is "useful to run in a day-to-day"
> codebase.
> We can have it with @Ignore("the test is too slow, see CALCITE-XXXX"),
> however it makes absolutely no sense in running the test on each test
> execution.
>
> Michael>If we are going to pick a single duration,
>
> Note: I'm not suggesting to veto tests based on its duration.
>
> Suppose I'm reviewing a PR, and I see that the added test there takes 95
> seconds on my machine.
> Is it reasonable to ask the author to reduce test duration?
>
> Michael>then I think it should be much higher than 5 seconds
>
> What's your idea on the reasonable duration for a single test?
>
> Just in case. https://issues.apache.org/jira/browse/CALCITE-2509 shows
> that
> a simple
> coalesce(vInt(0), vInt(1), vInt(2), vInt(3), vInt(4), vInt(5), vInt(6),
> vInt(7), vInt(8), vInt(9), vIntNotNull(0))
> takes 30+seconds in RexSimplifyTest.
>
> Would you mind if I add that kind of enabled-by-default test?
>
> Vladimir
>
