[
https://issues.apache.org/jira/browse/CAMEL-27163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18148165#comment-18148165
]
Aurélien Pupier commented on CAMEL-27163:
-----------------------------------------
h3. CAMEL-27163 status: what was done, tried, discarded, and what remains
_AI-generated by IBM Bob 2.2.1 on behalf of Aurélien Pupier (@apupier)._
h4. Baseline (ubuntu-avx / jdk21)
* Build without coverage: 36:06. Build with -Pcoverage: 51:23.
* Sonar analysis: 53:37 (54:26 for the Jenkins step).
* Sum of the three stages: about 1h46.
h4. Done and merged, measured on CI (build 2495)
# PR #30341, instrument only org/apache/camel/** in the coverage profile.
** Coverage build 51:23 -> 43:41.
** Median per-module ratio vs. the build without coverage: 1.65 -> 1.41.
** Outliers: camel-reactor 10:12 -> 0:59, camel-pqc 6:43 -> 1:51.
** Locally the old configuration ran out of memory on a full build.
# PR #30387, one JaCoCo report per module instead of one 110 MB aggregate
parsed by each of the 729 modules.
** JaCoCo XML Report Importer: 1,219 s -> 4.5 s.
** Sonar analysis total: 53:37 -> 39:36.
** New steps before the analysis: merge 7 s + per-module reports 1:14.
** The "coverage" module was removed; sonar-build.yml was adapted (disabled
workflow, not tested).
** Net on the three stages: about 1h46 -> about 1h25 (about 21 of the 28
minutes targeted).
** Caveat: the other sensors were 15-35% slower in the 2495 run than in 2483
(JavaSensor 1,077 s -> 1,291 s, SCM 327 s -> 388 s, javabugs 161 s -> 186 s,
security 81 s -> 112 s), so the real saving of the importer fix is larger than
the 14 minutes seen between the two analyses.
** Side effect to check on SonarCloud: the old aggregate covered only the 502
modules listed in the coverage pom. Locally on the same build, the aggregate
gave 56.6% and per-module reports 65.5% (69.9% on a later clean full rebuild).
The reported coverage of main will likely change. Issue counts (bugs, smells,
vulnerabilities) were identical in all local runs.
h4. Tried and discarded (with the reason)
* sonar.java.experimental.batchModeSizeInKB (local, JavaSensor time, 3
repeats): default 413 KB 191 s; 2048 KB mean 190 s; 8192 KB mean 179 s; 32768
KB 173 s. The differences are within the run-to-run spread, so the current 2048
stays. sonar.java.fileByFile=true is 3x slower (577 s).
* Garbage collection of the scanner engine (G1, GC log): 616 pauses, 2.5 s
total over a 310 s run (0.8%), no full GC, heap after GC under 1 GB. No -Xmx is
passed to the engine (JVM default). Not a bottleneck locally.
* -Dsonar.verbose=true has no effect with the Maven plugin (no DEBUG output);
mvn -X works but gave no extra timing information.
* Analysis on a separate agent: it still has to wait for the coverage build, so
the wall time does not change.
* Ruled out by decision: sonar.java.skipUnchanged (cannot be used on main),
disabling the SCM blame or the Sonar SCM algorithm option (already the fast
algorithm), disabling javabugs or the Java security analysis, excluding
src/generated from the analysis.
* Coverage overhead, GC or heap as the cause, on camel-jms (local): module 5:13
-> 5:31 (+6%); GC pauses 2.9 s -> 3.9 s over 72 test forks; max heap after GC
about 460 MB in both; no full GC. So changing -Xmx would not help and I did not
run the Xmx variations.
* Coverage overhead, local reproduction of the CI slowdown: camel-jms alone
+7%; 135 container-free modules in parallel with -T1C (load average 74-94) +19%
on JMS and a median ratio of 1.20, against +52% and 1.41 on CI. Load amplifies
the overhead but does not explain the CI figure.
h4. What is still slow on CI (build 2495)
* Sonar JavaSensor: 1,291 s (about 21 min). Local profile
(sonar.java.performance.measure): rules about 45% with no dominant rule, ECJ
parsing about 40%, conversion about 12%. On CI it costs about 0.76 s per module
plus about 21 ms per file, against about 0.12 s and 3.4 ms locally, and
1,077-1,291 s against about 190 s overall, so I have no explanation for the
5-6x gap.
* SCM blame 388 s, javabugs 186 s, security 112 s (kept by decision).
* Coverage build: still 41% above the build without coverage. Largest
per-module excess (coverage minus no coverage, from build 2470 vs 2495): JMS
+318 s, MongoDB +229 s, Kubernetes +192 s, Azure ServiceBus +164 s, Netty +161
s, Cassandra CQL +152 s, Netty HTTP +143 s. The top 12 modules account for
about 32 minutes of the 159 minutes of summed per-module excess.
h4. Remaining ideas (none tested yet)
# One-off CI diagnostic on the heaviest modules (JMS, MongoDB, Kubernetes,
Netty, Cassandra): per-fork duration, -Xlog:gc and jacoco.exec write time,
because the CI overhead could not be reproduced locally.
# JaCoCo agent options for the test forks (for example exclclassloader),
probably a small effect.
# One-off CI run with -Dsonar.scanner.javaOpts="-Xlog:gc*:file=gc.log" to check
GC and heap on the CI agent for the Sonar analysis.
# Find out why JavaSensor is 5-6x slower on CI than locally: compare the CPU
and memory of the agent, check whether the Sonar step runs while other jobs use
the node, and check the per-module fixed cost of 0.76 s.
# Smaller items: Java architecture sensor (24 s), text and secrets sensor (12
s), CPD indexer (12 s), the 1:14 report step.
# Remaining gap: about 5 minutes above the 1h20 of the ppc64le job, estimated
from the stage sums, not measured on one pipeline run.
> Improve time for test coverage and sonar analysis
> -------------------------------------------------
>
> Key: CAMEL-27163
> URL: https://issues.apache.org/jira/browse/CAMEL-27163
> Project: Camel
> Issue Type: Task
> Components: build system
> Affects Versions: 4.24.0
> Reporter: Aurélien Pupier
> Assignee: Aurélien Pupier
> Priority: Major
> Attachments: main_build-2483_filter-239.txt
>
>
> the current main bottleneck for build on Jenkins is related to Sonar and test
> coverage.
> The test coverage reports is moving build and test time from 34' to 55'
> Then the sonar report (reusing the test coverage reports and also doing issue
> analysis) is taking 54'.
> So it is adding 1h15 to the build on Ubuntu, jumping to a 1h48'.
> note that it will be very interesting to gain 28' as the next most timely job
> is taking 1h20 on ppc64le
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)