First I want to take a moment to say thanks to Andres and Josh for this new improvement. I truly believe it will save us a lot of time and efforts in the flaky tests fight
Pre-CASSANDRA-15712 there was default config for free tier and HIGHRES option(bumping all resources to max). We added MIDRES as a balanced version which gives you the chance to be able to run all tests but with less resources. What do I mean? It is still to be used by people with paid accounts but who want to consider the cost/resources spent. It was based on experimental runs which proved that with half the resources(compared to max out) for some suites we get similar and reasonable time to run for those test suites. One confusion I want to clear here - MIDRES doesn’t mean we will use medium containers but mid/less than the HIGHRES resources. Some Python tests cannot be run with medium containers, for example. Now whether some people still use HIGHRES or not I can neither confirm nor deny. I guess we can make a survey on that. I think improvement in choosing parallelism dynamically will be probably a great improvement too. We need to test. But if I recall correctly the moment we bump container size is when we bump to use way more credits. To be double-checked though, it’s been a while. Hope this info helps. On the topic of docs - I have one draft version for CircleCI doc/instructions but it has to be updated with all latest developments. I created it last year but never published as there was always something about to happen and for me to have to wait and update it accordingly. I can share with anyone willing to finish it so they don’t have to start from scratch. I don’t think I will have the time to do it in the next few weeks at least. On Tue, 18 Oct 2022 at 13:30, Andrés de la Peña <adelap...@apache.org> wrote: > The -h profile works but it spends a lot of resources for slightly faster > results. The -m profile is better value in terms of speed per resources. I > guess -h can be used if one wants to get results as soon as possible, no > matter the cost. Ekaterina might be better informed than me, given her work > on CASSANDRA-15712. > > In any case, the new multiplexer doesn't change the resources > configuration at all. We might want to reevaluate that config in the > future, and probably follow David's suggestion of deciding parallelism as a > function of the number of tests to be run. > > On Tue, 18 Oct 2022 at 18:13, Josh McKenzie <jmcken...@apache.org> wrote: > >> * Running the .circleci/generate.sh script with -l/-m/-h flags will use >> git diff to automatically detect the new or modified tests and will add >> them to the lists of tests to be repeated. The pre-commit workflow will >> automatically start repeated runs for these tests. The only exception to >> this are Python dtests, that should be specified manually. >> >> Of note: the -h profile should not be used (correct me if I'm wrong here >> Andres). Use -l for the free tier on circle or -m for paid. >> >> Will have some follow up tickets regarding job naming, default config >> type, and updating documentation shortly. >> >> On Tue, Oct 18, 2022, at 12:33 PM, Andrés de la Peña wrote: >> >> Just to let you know that CASSANDRA-17939 has just been committed. >> >> It changes the way the CircleCI multiplexer works, in line with the >> recent changes in our release criteria: >> >> * The default number of repeated tests iterations is 500, except for long >> and upgrade tests. >> * It is possible to specify multiple test classes and methods to be >> repeated into the same config push. So patches altering dozens of tests >> won't require dozens of config pushes anymore. >> * Running the .circleci/generate.sh script with -l/-m/-h flags will use >> git diff to automatically detect the new or modified tests and will add >> them to the lists of tests to be repeated. The pre-commit workflow will >> automatically start repeated runs for these tests. The only exception to >> this are Python dtests, that should be specified manually. >> * The CircleCI jobs are rearranged so for every regular job there is a >> companion job to run the repeated tests associated to that job. Those >> companion jobs will only be visible if there are repeated tests to run. >> Here >> <https://app.circleci.com/pipelines/github/adelapena/cassandra/2278/workflows/e339f5d4-0e16-4d4a-bde8-f0d9b9f3912d> >> is an example run with repeated tests for all the test suites, and here >> <https://app.circleci.com/pipelines/github/adelapena/cassandra/2269/workflows/d8907cbc-dbca-4d21-bdb9-1a4c58e1a412> >> is the same workflow without any repeated tests. >> >> Some documentation on how to use it can be found here: >> https://github.com/apache/cassandra/blob/trunk/.circleci/readme.md#running-tests-in-a-loop >> >> >>
