[
https://issues.apache.org/jira/browse/HDDS-16397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ryan Blough reassigned HDDS-16397:
----------------------------------
Assignee: Ryan Blough
> Investigate wall-clock improvement for Ozone Github CI workflows
> ----------------------------------------------------------------
>
> Key: HDDS-16397
> URL: https://issues.apache.org/jira/browse/HDDS-16397
> Project: Apache Ozone
> Issue Type: Improvement
> Components: CI
> Affects Versions: 2.2.0, 2.3.0
> Reporter: Ryan Blough
> Assignee: Ryan Blough
> Priority: Major
>
> I have been doing some experimenting to try and speed up the ci-cd pipeline.
> The core motivation is that we do not seem to use the resources of the
> default github runners very heavily; in my checks it looks like ~6GB out of
> 15.6GB peak memory consumption, and only during short bursts does CPU
> utilization get to the high 90th-percentile.
> The ci pipeline appears to have several options for greater or lesser
> caching, parallelism, and work scoping, but they are not currently being
> exploited. The plan is to experiment with these options and see if we can get
> lower wall-clock time for some of these workflows. The target is stuff like:
> - Implement mvn -T 1C for reactor parallelism. Chief target is the build job.
> - Activate the parallel-tests profile and try up to 4 test threads.
> - Further partition the om, ozone, hdds integration profiles.
> - check.yml accepts 'split' that does not seem to go anywhere - follow
> through to a work division property
> - populate_cache.yml has a 'workflow_call' that no workflow invokes
> - JUnit has parallel execution options, but we don't have thread-safety
> verified for many tests; undertake thread safety verification to allow
> parallel execution from JUnit
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
