[
https://issues.apache.org/jira/browse/CASSANDRA-16006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Semb Wever updated CASSANDRA-16006:
---
Description:
Currently dtests in Jenkins take ~10 hours.
Using the Jenkins Matrix plugin these jobs can be split into smaller lists of
dtests and run in parallel. This is the approach CircleCI takes.
This approach was [trialed|https://github.com/apache/cassandra-builds/pull/29]
with the dtest-upgrade jobs (which are not yet part of the branch pipelines,
and haven't previously worked at all due to their duration).
In addition to the Matrix plugin, the Priority-Sorter and Matrix Reloaded
plugins also needed to be added.
The splits will occupy all executors, and multiple builds will lead to a long
build queue. More important builds (artifacts and unit tests) need a way to be
prioritised in such saturated situations.
Splits can fail for silly reasons (false-positive), like
[full|https://issues.apache.org/jira/browse/INFRA-19793] /tmp disks, or
[connectivity issues|https://issues.apache.org/jira/browse/INFRA-20107] between
the donated agent servers. The Matrix Reloaded plugin makes it easy to rebuilt
just those failed splits.
was:
Currently dtests in Jenkins take ~10 hours.
Using the Jenkins Matrix plugin these jobs can be split into smaller lists of
dtests and run in parallel. This is the approach CircleCI takes.
This approach was [trialed|https://github.com/apache/cassandra-builds/pull/29]
with the dtest-upgrade jobs (which are not yet part of the branch pipelines,
and haven't previously worked at all due to their duration).
In addition to the Matrix plugin, the Priority-Sorter and Matrix Reloaded
plugins also needed to be added.
The splits will occupy all executors, and multiple builds will lead to a long
build queue. More important builds (artifacts and unit tests) need a way to be
prioritised in such saturated situations.
Splits can fail for silly reasons (false-positive), like full /tmp disks, or
connectivity issues between the donated agent servers. The Matrix Reloaded
plugin makes it easy to rebuilt just those failed splits.
> Parallelise Jenkins dtests
> --
>
> Key: CASSANDRA-16006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16006
> Project: Cassandra
> Issue Type: Improvement
> Components: CI
>Reporter: Michael Semb Wever
>Assignee: Michael Semb Wever
>Priority: Normal
> Fix For: 4.0-rc
>
>
> Currently dtests in Jenkins take ~10 hours.
> Using the Jenkins Matrix plugin these jobs can be split into smaller lists of
> dtests and run in parallel. This is the approach CircleCI takes.
> This approach was
> [trialed|https://github.com/apache/cassandra-builds/pull/29] with the
> dtest-upgrade jobs (which are not yet part of the branch pipelines, and
> haven't previously worked at all due to their duration).
> In addition to the Matrix plugin, the Priority-Sorter and Matrix Reloaded
> plugins also needed to be added.
> The splits will occupy all executors, and multiple builds will lead to a long
> build queue. More important builds (artifacts and unit tests) need a way to
> be prioritised in such saturated situations.
> Splits can fail for silly reasons (false-positive), like
> [full|https://issues.apache.org/jira/browse/INFRA-19793] /tmp disks, or
> [connectivity issues|https://issues.apache.org/jira/browse/INFRA-20107]
> between the donated agent servers. The Matrix Reloaded plugin makes it easy
> to rebuilt just those failed splits.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
