[jira] [Updated] (CASSANDRA-16006) Parallelise Jenkins dtests
[ https://issues.apache.org/jira/browse/CASSANDRA-16006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-16006: --- Fix Version/s: (was: 4.0-rc) 4.0-beta2 Source Control Link: https://github.com/apache/cassandra-builds/commit/6fc260ae265e7c8f8c29445264ae143e1693cf14 Resolution: Fixed Status: Resolved (was: Ready to Commit) Committed as [6fc260ae265e7c8f8c29445264ae143e1693cf14 |https://github.com/apache/cassandra-builds/commit/6fc260ae265e7c8f8c29445264ae143e1693cf14]. > Parallelise Jenkins dtests > -- > > Key: CASSANDRA-16006 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16006 > Project: Cassandra > Issue Type: Improvement > Components: CI >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > Fix For: 4.0-beta2 > > > Currently dtests in Jenkins take ~10 hours. > Using the Jenkins Matrix plugin these jobs can be split into smaller lists of > dtests and run in parallel. This is the approach CircleCI takes. > This approach was > [trialed|https://github.com/apache/cassandra-builds/pull/29] with the > dtest-upgrade jobs (which are not yet part of the branch pipelines, and > haven't previously worked at all due to their duration). > In addition to the Matrix plugin, the Priority-Sorter and Matrix Reloaded > plugins also needed to be added. > The splits will occupy all executors, and multiple builds will lead to a long > build queue. More important builds (artifacts and unit tests) need a way to > be prioritised in such saturated situations. > Splits can fail for silly reasons (false-positive), like > [full|https://issues.apache.org/jira/browse/INFRA-19793] /tmp disks, or > [connectivity issues|https://issues.apache.org/jira/browse/INFRA-20107] > between the donated agent servers. The Matrix Reloaded plugin makes it easy > to rebuilt just those failed splits. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16006) Parallelise Jenkins dtests
[ https://issues.apache.org/jira/browse/CASSANDRA-16006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eduard Tudenhoefner updated CASSANDRA-16006: Status: Ready to Commit (was: Review In Progress) > Parallelise Jenkins dtests > -- > > Key: CASSANDRA-16006 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16006 > Project: Cassandra > Issue Type: Improvement > Components: CI >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > Fix For: 4.0-rc > > > Currently dtests in Jenkins take ~10 hours. > Using the Jenkins Matrix plugin these jobs can be split into smaller lists of > dtests and run in parallel. This is the approach CircleCI takes. > This approach was > [trialed|https://github.com/apache/cassandra-builds/pull/29] with the > dtest-upgrade jobs (which are not yet part of the branch pipelines, and > haven't previously worked at all due to their duration). > In addition to the Matrix plugin, the Priority-Sorter and Matrix Reloaded > plugins also needed to be added. > The splits will occupy all executors, and multiple builds will lead to a long > build queue. More important builds (artifacts and unit tests) need a way to > be prioritised in such saturated situations. > Splits can fail for silly reasons (false-positive), like > [full|https://issues.apache.org/jira/browse/INFRA-19793] /tmp disks, or > [connectivity issues|https://issues.apache.org/jira/browse/INFRA-20107] > between the donated agent servers. The Matrix Reloaded plugin makes it easy > to rebuilt just those failed splits. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16006) Parallelise Jenkins dtests
[ https://issues.apache.org/jira/browse/CASSANDRA-16006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eduard Tudenhoefner updated CASSANDRA-16006: Reviewers: Eduard Tudenhoefner, Eduard Tudenhoefner (was: Eduard Tudenhoefner) Eduard Tudenhoefner, Eduard Tudenhoefner Status: Review In Progress (was: Patch Available) changes LGTM > Parallelise Jenkins dtests > -- > > Key: CASSANDRA-16006 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16006 > Project: Cassandra > Issue Type: Improvement > Components: CI >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > Fix For: 4.0-rc > > > Currently dtests in Jenkins take ~10 hours. > Using the Jenkins Matrix plugin these jobs can be split into smaller lists of > dtests and run in parallel. This is the approach CircleCI takes. > This approach was > [trialed|https://github.com/apache/cassandra-builds/pull/29] with the > dtest-upgrade jobs (which are not yet part of the branch pipelines, and > haven't previously worked at all due to their duration). > In addition to the Matrix plugin, the Priority-Sorter and Matrix Reloaded > plugins also needed to be added. > The splits will occupy all executors, and multiple builds will lead to a long > build queue. More important builds (artifacts and unit tests) need a way to > be prioritised in such saturated situations. > Splits can fail for silly reasons (false-positive), like > [full|https://issues.apache.org/jira/browse/INFRA-19793] /tmp disks, or > [connectivity issues|https://issues.apache.org/jira/browse/INFRA-20107] > between the donated agent servers. The Matrix Reloaded plugin makes it easy > to rebuilt just those failed splits. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16006) Parallelise Jenkins dtests
[ https://issues.apache.org/jira/browse/CASSANDRA-16006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-16006: --- Test and Documentation Plan: Tested on local jenkins. Tested on ci-cassandra.a.o with dtest-upgrade jobs. (was: Tested on local jenkins. Test on ci-cassandra.a.o with dtest-upgrade jobs.) > Parallelise Jenkins dtests > -- > > Key: CASSANDRA-16006 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16006 > Project: Cassandra > Issue Type: Improvement > Components: CI >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > Fix For: 4.0-rc > > > Currently dtests in Jenkins take ~10 hours. > Using the Jenkins Matrix plugin these jobs can be split into smaller lists of > dtests and run in parallel. This is the approach CircleCI takes. > This approach was > [trialed|https://github.com/apache/cassandra-builds/pull/29] with the > dtest-upgrade jobs (which are not yet part of the branch pipelines, and > haven't previously worked at all due to their duration). > In addition to the Matrix plugin, the Priority-Sorter and Matrix Reloaded > plugins also needed to be added. > The splits will occupy all executors, and multiple builds will lead to a long > build queue. More important builds (artifacts and unit tests) need a way to > be prioritised in such saturated situations. > Splits can fail for silly reasons (false-positive), like > [full|https://issues.apache.org/jira/browse/INFRA-19793] /tmp disks, or > [connectivity issues|https://issues.apache.org/jira/browse/INFRA-20107] > between the donated agent servers. The Matrix Reloaded plugin makes it easy > to rebuilt just those failed splits. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16006) Parallelise Jenkins dtests
[ https://issues.apache.org/jira/browse/CASSANDRA-16006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-16006: --- Test and Documentation Plan: Tested on local jenkins. Test on ci-cassandra.a.o with dtest-upgrade jobs. Status: Patch Available (was: In Progress) > Parallelise Jenkins dtests > -- > > Key: CASSANDRA-16006 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16006 > Project: Cassandra > Issue Type: Improvement > Components: CI >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > Fix For: 4.0-rc > > > Currently dtests in Jenkins take ~10 hours. > Using the Jenkins Matrix plugin these jobs can be split into smaller lists of > dtests and run in parallel. This is the approach CircleCI takes. > This approach was > [trialed|https://github.com/apache/cassandra-builds/pull/29] with the > dtest-upgrade jobs (which are not yet part of the branch pipelines, and > haven't previously worked at all due to their duration). > In addition to the Matrix plugin, the Priority-Sorter and Matrix Reloaded > plugins also needed to be added. > The splits will occupy all executors, and multiple builds will lead to a long > build queue. More important builds (artifacts and unit tests) need a way to > be prioritised in such saturated situations. > Splits can fail for silly reasons (false-positive), like > [full|https://issues.apache.org/jira/browse/INFRA-19793] /tmp disks, or > [connectivity issues|https://issues.apache.org/jira/browse/INFRA-20107] > between the donated agent servers. The Matrix Reloaded plugin makes it easy > to rebuilt just those failed splits. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16006) Parallelise Jenkins dtests
[ https://issues.apache.org/jira/browse/CASSANDRA-16006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-16006: --- Description: Currently dtests in Jenkins take ~10 hours. Using the Jenkins Matrix plugin these jobs can be split into smaller lists of dtests and run in parallel. This is the approach CircleCI takes. This approach was [trialed|https://github.com/apache/cassandra-builds/pull/29] with the dtest-upgrade jobs (which are not yet part of the branch pipelines, and haven't previously worked at all due to their duration). In addition to the Matrix plugin, the Priority-Sorter and Matrix Reloaded plugins also needed to be added. The splits will occupy all executors, and multiple builds will lead to a long build queue. More important builds (artifacts and unit tests) need a way to be prioritised in such saturated situations. Splits can fail for silly reasons (false-positive), like [full|https://issues.apache.org/jira/browse/INFRA-19793] /tmp disks, or [connectivity issues|https://issues.apache.org/jira/browse/INFRA-20107] between the donated agent servers. The Matrix Reloaded plugin makes it easy to rebuilt just those failed splits. was: Currently dtests in Jenkins take ~10 hours. Using the Jenkins Matrix plugin these jobs can be split into smaller lists of dtests and run in parallel. This is the approach CircleCI takes. This approach was [trialed|https://github.com/apache/cassandra-builds/pull/29] with the dtest-upgrade jobs (which are not yet part of the branch pipelines, and haven't previously worked at all due to their duration). In addition to the Matrix plugin, the Priority-Sorter and Matrix Reloaded plugins also needed to be added. The splits will occupy all executors, and multiple builds will lead to a long build queue. More important builds (artifacts and unit tests) need a way to be prioritised in such saturated situations. Splits can fail for silly reasons (false-positive), like full /tmp disks, or connectivity issues between the donated agent servers. The Matrix Reloaded plugin makes it easy to rebuilt just those failed splits. > Parallelise Jenkins dtests > -- > > Key: CASSANDRA-16006 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16006 > Project: Cassandra > Issue Type: Improvement > Components: CI >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > Fix For: 4.0-rc > > > Currently dtests in Jenkins take ~10 hours. > Using the Jenkins Matrix plugin these jobs can be split into smaller lists of > dtests and run in parallel. This is the approach CircleCI takes. > This approach was > [trialed|https://github.com/apache/cassandra-builds/pull/29] with the > dtest-upgrade jobs (which are not yet part of the branch pipelines, and > haven't previously worked at all due to their duration). > In addition to the Matrix plugin, the Priority-Sorter and Matrix Reloaded > plugins also needed to be added. > The splits will occupy all executors, and multiple builds will lead to a long > build queue. More important builds (artifacts and unit tests) need a way to > be prioritised in such saturated situations. > Splits can fail for silly reasons (false-positive), like > [full|https://issues.apache.org/jira/browse/INFRA-19793] /tmp disks, or > [connectivity issues|https://issues.apache.org/jira/browse/INFRA-20107] > between the donated agent servers. The Matrix Reloaded plugin makes it easy > to rebuilt just those failed splits. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16006) Parallelise Jenkins dtests
[ https://issues.apache.org/jira/browse/CASSANDRA-16006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Semb Wever updated CASSANDRA-16006: --- Change Category: Performance Complexity: Normal Fix Version/s: 4.0-rc Assignee: Michael Semb Wever Status: Open (was: Triage Needed) WIP [here|https://github.com/apache/cassandra-builds/compare/master...thelastpickle:mck/jenkins-dtest-parallel] > Parallelise Jenkins dtests > -- > > Key: CASSANDRA-16006 > URL: https://issues.apache.org/jira/browse/CASSANDRA-16006 > Project: Cassandra > Issue Type: Improvement > Components: CI >Reporter: Michael Semb Wever >Assignee: Michael Semb Wever >Priority: Normal > Fix For: 4.0-rc > > > Currently dtests in Jenkins take ~10 hours. > Using the Jenkins Matrix plugin these jobs can be split into smaller lists of > dtests and run in parallel. This is the approach CircleCI takes. > This approach was > [trialed|https://github.com/apache/cassandra-builds/pull/29] with the > dtest-upgrade jobs (which are not yet part of the branch pipelines, and > haven't previously worked at all due to their duration). > In addition to the Matrix plugin, the Priority-Sorter and Matrix Reloaded > plugins also needed to be added. > The splits will occupy all executors, and multiple builds will lead to a long > build queue. More important builds (artifacts and unit tests) need a way to > be prioritised in such saturated situations. > Splits can fail for silly reasons (false-positive), like full /tmp disks, or > connectivity issues between the donated agent servers. The Matrix Reloaded > plugin makes it easy to rebuilt just those failed splits. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
