[
https://issues.apache.org/jira/browse/HADOOP-15190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18097131#comment-18097131
]
Ferenc Erdelyi commented on HADOOP-15190:
-----------------------------------------
h2. Update — address review feedback (Clover sweep, module naming, module-list
guard)
Thanks [~K0K0V0K] for the review! This update addresses all three points, plus a
groupId defect and a plugin-management cleanup surfaced while verifying.
h3. 1. Remaining Clover references
A full repo sweep shows *no live Clover references remain* — none in any
{{pom.xml}},
{{dev-support/}}, {{*.sh}}, or {{*.properties}}. The {{BUILDING.txt}} line the
review
flagged is already replaced in the latest push:
{noformat}
- * Run clover : mvn test -Pclover
+ * Run JaCoCo coverage : mvn verify -Dhadoop.skip-jacoco=false
-Dmaven.test.failure.ignore=true
{noformat}
The only other {{clover}} hits are in
{{src/site/markdown/release/*/CHANGELOG.*.md}} and
{{RELEASENOTES.*.md}} — immutable generated release history, intentionally left
untouched.
h3. 2. Module naming
Dropped the tool name from the module so it survives a future coverage-tool
swap without a
rename. artifactId stays the generic {{hadoop-coverage}}:
{noformat}
- <description>Hadoop JaCoCo Aggregate Coverage</description>
- <name>Hadoop JaCoCo Aggregate Coverage</name>
+ <description>Hadoop Aggregate Code Coverage</description>
+ <name>Hadoop Aggregate Code Coverage</name>
{noformat}
h3. 3. Guard against a forgotten module
Rather than remove the explicit dependency list (JaCoCo's {{report-aggregate}}
requires
each aggregated module to be a declared reactor dependency — there is no "all
modules"
mode), the list is now *guarded* so it cannot silently drift:
* New script {{dev-support/bin/check-coverage-modules.sh}} scans the source
tree for
_test-bearing_ modules (jar packaging + a {{src/test/java}} directory), and
fails the
build if any such module is neither listed in {{hadoop-coverage/pom.xml}} nor
in an
allowlist.
* New file {{dev-support/coverage-modules-allowlist.txt}} documents the
intentional
exclusions ({{hadoop-mapreduce-examples}} — already excluded via the
{{**/examples/**}}
class pattern; {{hadoop-client-integration-tests}} — a test-only module).
* Wired via {{exec-maven-plugin}} at the {{validate}} phase, *ungated by
hadoop.skip-jacoco*
(list correctness is independent of whether coverage is collected), so it
runs on every
build that includes the module — including the full-reactor {{mvn install}}
in precommit.
* The script fails loudly (exit 2) if the scan finds zero modules (e.g. a wrong
repo root),
rather than passing vacuously, and documents its intentional scope in the
header: modules
are matched by artifactId only (coordinate resolution is Maven's job), and
"covered"
artifactIds are read from the coverage pom's single top-level
{{<dependencies>}} block.
Audit of the current list: 71 declared vs 71 test-bearing modules, 69
intersecting; the
two differences each way are intentional/harmless, so *no coverage is lost
today* — the
guard is purely preventive.
h3. Additional fixes found during verification
*mawo-core groupId.* The {{hadoop-yarn-applications-mawo-core}} dependency
declared the
wrong groupId ({{org.apache.hadoop}}); the module's real groupId is
{{org.apache.hadoop.applications.mawo}}. In a clean reactor this would not
match the
reactor module and would pull a stale artifact or fail to resolve. Corrected:
{noformat}
- <groupId>org.apache.hadoop</groupId>
+ <groupId>org.apache.hadoop.applications.mawo</groupId>
<artifactId>hadoop-yarn-applications-mawo-core</artifactId>
{noformat}
*exec-maven-plugin version management.* Since {{hadoop-coverage}}'s parent is
{{hadoop-main}} (which did not manage {{exec-maven-plugin}}), the plugin
version is now
declared once in the root {{pom.xml}} {{pluginManagement}} (alongside jacoco)
and the
duplicate declaration was removed from {{hadoop-project/pom.xml}} — single
source of truth.
Verified that {{hadoop-common}}'s effective POM still resolves
{{exec-maven-plugin}} to
{{1.3.1}} (no regression), and {{hadoop-coverage}} inherits it version-free.
h3. Verification
h4. Positive test — clean tree passes
{noformat}
$ bash dev-support/bin/check-coverage-modules.sh "$PWD"
"$PWD/hadoop-coverage/pom.xml"
check-coverage-modules: OK (71 test-bearing modules accounted for)
$ echo $?
0
{noformat}
h4. Negative test — a missing module fails the build
Running against a copy of the pom with {{hadoop-distcp}} removed:
{noformat}
check-coverage-modules: FAILED
The following test-bearing modules are missing from the coverage aggregate:
- hadoop-distcp
Add each as a <dependency> in hadoop-coverage/pom.xml so its coverage
is aggregated, or, if it is intentionally excluded, add it to
dev-support/coverage-modules-allowlist.txt (with a reason).
$ echo $?
1
{noformat}
h4. Maven wiring — guard runs at the validate phase (version inherited from
root)
{noformat}
$ mvn validate -pl hadoop-coverage
[INFO] Building Hadoop Aggregate Code Coverage 3.6.0-SNAPSHOT
[INFO] --- exec:1.3.1:exec (check-coverage-modules) @ hadoop-coverage ---
check-coverage-modules: OK (71 test-bearing modules accounted for)
[INFO] BUILD SUCCESS
{noformat}
h4. Aggregate report still builds end-to-end
Scoped run (nfs + coverage) confirms guard -> tests -> report-aggregate all
succeed:
{noformat}
$ mvn verify -pl hadoop-common-project/hadoop-nfs,hadoop-coverage \
-Dhadoop.skip-jacoco=false -Dmaven.test.failure.ignore=true
[INFO] Tests run: 21, Failures: 0, Errors: 0, Skipped: 0
check-coverage-modules: OK (71 test-bearing modules accounted for)
[INFO] --- jacoco:0.8.15:report-aggregate (report-aggregate) @ hadoop-coverage
---
[INFO] Analyzed bundle 'hadoop-nfs' with 77 classes
[INFO] BUILD SUCCESS
{noformat}
Report produced at {{hadoop-coverage/target/site/jacoco-aggregate/index.html}}.
h3. Files changed
|| File || Change ||
| {{pom.xml}} | centralize exec-maven-plugin version in pluginManagement
(alongside jacoco) |
| {{hadoop-project/pom.xml}} | remove duplicate exec-maven-plugin
version/management (promoted to root) |
| {{hadoop-coverage/pom.xml}} | generic name/description; mawo-core groupId
fix; exec-maven-plugin guard wiring (inherits version) |
| {{dev-support/bin/check-coverage-modules.sh}} | new — module-list guard |
| {{dev-support/coverage-modules-allowlist.txt}} | new — documented intentional
exclusions |
{{shellcheck}} passes in precommit (yetus); the script follows Hadoop's shell
conventions.
> Use Jacoco to generate Unit Test coverage reports
> -------------------------------------------------
>
> Key: HADOOP-15190
> URL: https://issues.apache.org/jira/browse/HADOOP-15190
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Reporter: Duo Xu
> Assignee: Duo Xu
> Priority: Minor
> Labels: pull-request-available
> Attachments: HADOOP-15190-design_2026_July_16.txt,
> HADOOP-15190.01.patch, aggregate_coverage_report_demo_2026_july_17.png,
> hadoop_nfs_jacoco_report_2026_july_17.png, jacoco_report_2018_01_25.JPG
>
>
> Currently Hadoop is using maven-clover2-plugin for code coverage, which is
> outdated. And Atlassian open-sourced clover last year so license cannot be
> purchased although we can switch to use the license-free version called
> "openclover".
> This Jira is to replace clover with Jacoco, which is actively maintained by
> the community.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
