I fully understand you. Although I have that luxury to use more containers, I simply feel that rerunning the same code with different configurations which do not impact that code is just a waste of resources and money.
- - -- --- ----- -------- ------------- Jacek Lewandowski czw., 15 lut 2024 o 08:41 Štefan Miklošovič <stefan.mikloso...@gmail.com> napisał(a): > By the way, I am not sure if it is all completely transparent and > understood by everybody but let me guide you through a typical patch which > is meant to be applied from 4.0 to trunk (4 branches) to see how it looks > like. > > I do not have the luxury of running CircleCI on 100 containers, I have > just 25. So what takes around 2.5h for 100 containers takes around 6-7 for > 25. That is a typical java11_pre-commit_tests for trunk. Then I have to > provide builds for java17_pre-commit_tests too, that takes around 3-4 hours > because it just tests less, let's round it up to 10 hours for trunk. > > Then I need to do this for 5.0 as well, basically double the time because > as I am writing this the difference is not too big between these two > branches. So 20 hours. > > Then I need to build 4.1 and 4.0 too, 4.0 is very similar to 4.1 when it > comes to the number of tests, nevertheless, there are workflows for Java 8 > and Java 11 for each so lets say this takes 10 hours again. So together I'm > 35. > > To schedule all the builds, trigger them, monitor their progress etc is > work in itself. I am scripting this like crazy to not touch the UI in > Circle at all and I made my custom scripts which call Circle API and it > triggers the builds from the console to speed this up because as soon as a > developer is meant to be clicking around all day, needing to tracking the > progress, it gets old pretty quickly. > > Thank god this is just a patch from 4.0, when it comes to 3.0 and 3.11 > just add more hours to that. > > So all in all, a typical 4.0 - trunk patch is tested for two days at > least, that's when all is nice and I do not need to rework it and rurun it > again ... Does this all sound flexible and speedy enough for people? > > If we dropped the formal necessity to build various jvms it would > significantly speed up the development. > > > On Thu, Feb 15, 2024 at 8:10 AM Jacek Lewandowski < > lewandowski.ja...@gmail.com> wrote: > >> Excellent point, I was saying for some time that IMHO we can reduce >>> to running in CI at least pre-commit: >>> 1) Build J11 2) build J17 >>> 3) run tests with build 11 + runtime 11 >>> 4) run tests with build 11 and runtime 17. >> >> >> Ekaterina, I was thinking more about: >> 1) build J11 >> 2) build J17 >> 3) run tests with build J11 + runtime J11 >> 4) run smoke tests with build J17 and runtime J17 >> >> Again, I don't see value in running build J11 and J17 runtime >> additionally to J11 runtime - just pick one unless we change something >> specific to JVM >> >> If we need to decide whether to test the latest or default, I think we >> should pick the latest because this is actually Cassandra 5.0 defined as a >> set of new features that will shine on the website. >> >> Also - we have configurations which test some features but they more like >> dimensions: >> - commit log compression >> - sstable compression >> - CDC >> - Trie memtables >> - Trie SSTable format >> - Extended deletion time >> ... >> >> Currently, with what we call the default configuration is tested with: >> - no compression, no CDC, no extended deletion time >> - *commit log compression + sstable compression*, no cdc, no extended >> deletion time >> - no compression, *CDC enabled*, no extended deletion time >> - no compression, no CDC, *enabled extended deletion time* >> >> This applies only to unit tests of course >> >> Then, are we going to test all of those scenarios with the "latest" >> configuration? I'm asking because the latest configuration is mostly about >> tries and UCS and has nothing to do with compression or CDC. Then why the >> default configuration should be tested more thoroughly than latest which >> enables essential Cassandra 5.0 features? >> >> I propose to significantly reduce that stuff. Let's distinguish the >> packages of tests that need to be run with CDC enabled / disabled, with >> commitlog compression enabled / disabled, tests that verify sstable formats >> (mostly io and index I guess), and leave other parameters set as with the >> latest configuration - this is the easiest way I think. >> >> For dtests we have vnodes/no-vnodes, offheap/onheap, and nothing about >> other stuff. To me running no-vnodes makes no sense because no-vnodes is >> just a special case of vnodes=1. On the other hand offheap/onheap buffers >> could be tested in unit tests. In short, I'd run dtests only with the >> default and latest configuration. >> >> Sorry for being too wordy, >> >> >> czw., 15 lut 2024 o 07:39 Štefan Miklošovič <stefan.mikloso...@gmail.com> >> napisał(a): >> >>> Something along what Paulo is proposing makes sense to me. To sum it up, >>> knowing what workflows we have now: >>> >>> java17_pre-commit_tests >>> java11_pre-commit_tests >>> java17_separate_tests >>> java11_separate_tests >>> >>> We would have couple more, together like: >>> >>> java17_pre-commit_tests >>> java17_pre-commit_tests-latest-yaml >>> java11_pre-commit_tests >>> java11_pre-commit_tests-latest-yaml >>> java17_separate_tests >>> java17_separate_tests-default-yaml >>> java11_separate_tests >>> java11_separate_tests-latest-yaml >>> >>> To go over Paulo's plan, his steps 1-3 for 5.0 would result in requiring >>> just one workflow >>> >>> java11_pre-commit_tests >>> >>> when no configuration is touched and two workflows >>> >>> java11_pre-commit_tests >>> java11_pre-commit_tests-latest-yaml >>> >>> when there is some configuration change. >>> >>> Now the term "some configuration change" is quite tricky and it is not >>> always easy to evaluate if both default and latest yaml workflows need to >>> be executed. It might happen that a change is of such a nature that it does >>> not change the configuration but it is necessary to verify that it still >>> works with both scenarios. -latest.yaml config might be such that a change >>> would make sense to do in isolation for default config only but it would >>> not work with -latest.yaml too. I don't know if this is just a theoretical >>> problem or not but my gut feeling is that we would be safer if we just >>> required both default and latest yaml workflows together. >>> >>> Even if we do, we basically replace "two jvms" builds for "two yamls" >>> builds but I consider "two yamls" builds to be more valuable in general >>> than "two jvms" builds. It would take basically the same amount of time, we >>> would just reoriented our building matrix from different jvms to different >>> yamls. >>> >>> For releases we would for sure need to just run it across jvms too. >>> >>> On Thu, Feb 15, 2024 at 7:05 AM Paulo Motta <pa...@apache.org> wrote: >>> >>>> > Perhaps it is also a good opportunity to distinguish subsets of >>>> tests which make sense to run with a configuration matrix. >>>> >>>> Agree. I think we should define a “standard/golden” configuration for >>>> each branch and minimally require precommit tests for that configuration. >>>> Assignees and reviewers can determine if additional test variants are >>>> required based on the patch scope. >>>> >>>> Nightly and prerelease tests can be run to catch any issues outside the >>>> standard configuration based on the supported configuration matrix. >>>> >>>> On Wed, 14 Feb 2024 at 15:32 Jacek Lewandowski < >>>> lewandowski.ja...@gmail.com> wrote: >>>> >>>>> śr., 14 lut 2024 o 17:30 Josh McKenzie <jmcken...@apache.org> >>>>> napisał(a): >>>>> >>>>>> When we have failing tests people do not spend the time to figure out >>>>>> if their logic caused a regression and merge, making things more >>>>>> unstable… >>>>>> so when we merge failing tests that leads to people merging even more >>>>>> failing tests... >>>>>> >>>>>> What's the counter position to this Jacek / Berenguer? >>>>>> >>>>> >>>>> For how long are we going to deceive ourselves? Are we shipping those >>>>> features or not? Perhaps it is also a good opportunity to distinguish >>>>> subsets of tests which make sense to run with a configuration matrix. >>>>> >>>>> If we don't add those tests to the pre-commit pipeline, "people do >>>>> not spend the time to figure out if their logic caused a regression and >>>>> merge, making things more unstable…" >>>>> I think it is much more valuable to test those various configurations >>>>> rather than test against j11 and j17 separately. I can see a really little >>>>> value in doing that. >>>>> >>>>> >>>>>
