[
https://issues.apache.org/jira/browse/CASSANDRA-15686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17077751#comment-17077751
]
David Capwell commented on CASSANDRA-15686:
-------------------------------------------
[~spod] can you also take a look at this JIRA?
bq. Confirm the build is running as normal by using the config-2_1.yml directly
renamed as config.yml, see build #42 here:
https://app.circleci.com/pipelines/github/newkek/cassandra?branch=chg
cool. That basically means we don't really need to generate LOWER as we could
just use that. We may still want the script to do that just to validate the
YAML is "correct", but wouldn't need to commit that.
bq. the dtests don't build successfully either on medium
I see this as a bug honestly. There are some tests which expect high resources
but we don't run those in Circle CI (see
[here|https://github.com/apache/cassandra/blob/trunk/.circleci/config-2_1.yml#L418],
we explicitly disable the high resource tests with
--skip-resource-intensive-tests). I personally feel that these should be fixed
as part of the flaky test cleanup work many people are working on; everyone
should be able to run the regression suite, if that isn't true it should be see
as a bug IMO.
bq. it would allow people with large instances access to build without having
to switch their config. Does that make sense?
IMO higher should only be faster, not the only correct config, so I personally
feel that the default should reflect what new members are able to do and people
who have special accounts have an extra step of switching to higher.
bq. Hm afaict running with multiple runners is fine for the unit tests
I would need to look closer. We spin up storage at least which I thought was
shared (directory layout), so the system tables could conflict and cause race
condition problems (even if every create table has no conflict).
If I am wrong (happens a lot), then faster builds are awesome! =)
> Improvements in circle CI default config
> ----------------------------------------
>
> Key: CASSANDRA-15686
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15686
> Project: Cassandra
> Issue Type: Bug
> Components: Build
> Reporter: Kevin Gallardo
> Priority: Normal
>
> I have been looking at and played around with the [default CircleCI
> config|https://github.com/apache/cassandra/blob/trunk/.circleci/config.yml],
> a few comments/questions regarding the following topics:
> * Python dtests do not run successfully (200-300 failures) on {{medium}}
> instances, they seem to only run with small flaky failures on {{large}}
> instances or higher
> * Python Upgrade tests:
> ** Do not seem to run without many failures on any instance types / any
> parallelism setting
> ** Do not seem to parallelize well, it seems each container is going to
> download multiple C* versions
> ** Additionally it seems the configuration is not up to date, as currently
> we get errors because {{JAVA8_HOME}} is not set
> * Unit tests do not seem to parallelize optimally, number of test runners do
> not reflect the available CPUs on the container. Ideally if # of runners == #
> of CPUs, build time is improved, on any type of instances.
> ** For instance when using the current configuration, running on medium
> instances, build will use 1 junit test runner, but 2 CPUs are available. If
> using 2 runners, the build time is reduced from 19min (at the current main
> config of parallelism=4) to 12min.
> * There are some typos in the file, some dtests say "Run Unit Tests" but
> they are JVM dtests (see
> [here|https://github.com/apache/cassandra/blob/trunk/.circleci/config.yml#L1077],
>
> [here|https://github.com/apache/cassandra/blob/trunk/.circleci/config.yml#L1386])
> So some ways to process these would be:
> * Do the Python dtests run successfully for anyone on {{medium}} instances?
> If not, would it make sense to bump them to {{large}} so that they can be run
> successfully?
> * Does anybody ever run the python upgrade tests on CircleCI and what is the
> configuration that makes it work?
> * Would it make sense to either hardcode the number of test runners in the
> unit tests with `-Dtest.runners` in the config file to reflect the number of
> CPUs on the instances, or change the build so that it is able to detect the
> appropriate number of core available automatically?
> Additionally, it seems this default config file (config.yml) is not as well
> maintained as the
> [{{config-2_1.yml}}|https://github.com/apache/cassandra/blob/trunk/.circleci/config-2_1.yml]
> (+its lowres/highres) version in the same folder (from CASSANDRA-14806).
> What is the reasoning for maintaining these 2 versions of the build? Could
> the better maintained version be used as the default? We could generate a
> lowres version of the new config-2_1.yml, and rename it {{config.yml}} so
> that it gets picked up by CircleCI automatically instead of the current
> default.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
