yangdong2398 commented on PR #8347: URL: https://github.com/apache/hadoop/pull/8347#issuecomment-4133580524
> @yangdong2398 I have conducted some analysis on the dependencies of this module with cc and summarized them as follows: > > ## Compile-Scope Dependency Tree > ``` > org.apache.hadoop:hadoop-bos:3.6.0-SNAPSHOT > └── com.baidubce:bce-java-sdk:0.10.330 > ├── org.apache.commons:commons-lang3:3.18.0 > ├── org.slf4j:slf4j-api:1.7.36 > ├── commons-codec:commons-codec:1.15 > ├── commons-io:commons-io:2.16.1 > ├── org.apache.httpcomponents:httpasyncclient:4.0.2 > │ ├── org.apache.httpcomponents:httpcore:4.4.13 > │ ├── org.apache.httpcomponents:httpcore-nio:4.3.2 > │ ├── org.apache.httpcomponents:httpclient:4.5.13 > │ └── commons-logging:commons-logging:1.3.0 > ├── joda-time:joda-time:2.3 > ├── org.eclipse.paho:org.eclipse.paho.client.mqttv3:1.2.1 > ├── javax.validation:validation-api:1.1.0.Final > └── org.bouncycastle:bcprov-jdk15on:1.59 > ``` > > ## License Summary > Artifact Version License ASF Category APL 2.0? > com.baidubce:bce-java-sdk 0.10.330 Apache License 2.0 A Yes > org.apache.commons:commons-lang3 3.18.0 Apache License 2.0 A Yes > org.slf4j:slf4j-api 1.7.36 MIT License A Compatible > commons-codec:commons-codec 1.15 Apache License 2.0 A Yes > commons-io:commons-io 2.16.1 Apache License 2.0 A Yes > org.apache.httpcomponents:httpasyncclient 4.0.2 Apache License 2.0 A Yes > org.apache.httpcomponents:httpcore 4.4.13 Apache License 2.0 A Yes > org.apache.httpcomponents:httpcore-nio 4.3.2 Apache License 2.0 A Yes > org.apache.httpcomponents:httpclient 4.5.13 Apache License 2.0 A Yes > commons-logging:commons-logging 1.3.0 Apache License 2.0 A Yes > joda-time:joda-time 2.3 Apache License 2.0 A Yes > **org.eclipse.paho:org.eclipse.paho.client.mqttv3** **1.2.1** **EPL 1.0 / EDL 1.0** **B** **No** > javax.validation:validation-api 1.1.0.Final Apache License 2.0 A Yes > org.bouncycastle:bcprov-jdk15on 1.59 MIT License (Bouncy Castle Licence) A Compatible > **Result:** 12 of 14 dependencies are Apache License 2.0. Two (slf4j-api, bcprov-jdk15on) are MIT — ASF Category A, fully compatible. **One dependency (`org.eclipse.paho:org.eclipse.paho.client.mqttv3`) is EPL 1.0 — ASF Category B, requiring special handling.** > > ## Open-Source License Risks > ### Risk 1: `org.eclipse.paho:org.eclipse.paho.client.mqttv3:1.2.1` — EPL 1.0 (ASF Category B) > **JAR:** `org.eclipse.paho.client.mqttv3-1.2.1.jar` > > **Risk:** This library is dual-licensed under **EPL 1.0** (Eclipse Public License) and **EDL 1.0** (Eclipse Distribution License / BSD-3-Clause). EPL 1.0 is classified as **ASF Category B** — weak copyleft. Per [ASF third-party licensing policy](https://www.apache.org/legal/resolved.html): > > * Category B dependencies **must not** appear in source releases > * Category B dependencies **may** appear in binary distributions, but **must** be prominently labeled in `LICENSE-binary` and/or `NOTICE-binary` > * The PR description claims _"No changes required to LICENSE, LICENSE-binary, or NOTICE-binary files as dependencies are shaded and not exposed in distribution"_ — **this is incorrect**. Shading does not eliminate licensing obligations; the EPL-licensed bytecode is still present in the shaded JAR > > **How to address (Option A — Preferred):** Exclude the dependency in `pom.xml`. This is an MQTT client library with no obvious relevance to a Baidu Object Storage connector — it is almost certainly an unused transitive dependency of `bce-java-sdk`: > > ``` > <dependency> > <groupId>com.baidubce</groupId> > <artifactId>bce-java-sdk</artifactId> > <version>${bce-java-sdk.version}</version> > <exclusions> > <!-- existing exclusions... --> > <exclusion> > <groupId>org.eclipse.paho</groupId> > <artifactId>org.eclipse.paho.client.mqttv3</artifactId> > </exclusion> > </exclusions> > </dependency> > ``` > > **How to address (Option B — If exclusion is not possible):** If the BOS SDK demonstrably requires this library at runtime: > > 1. Add an EPL 1.0 entry to `LICENSE-binary` documenting the inclusion > 2. Add an entry to `NOTICE-binary` if the library carries a NOTICE file > 3. Ensure the JAR is only included in binary distributions, never in source releases > 4. The shading relocation must also be documented (the EPL license text applies to the relocated classes under `bfs.org.eclipse.paho.*`) > > ### Risk 2: `org.bouncycastle:bcprov-jdk15on:1.59` — Known CVEs > **JAR:** `bcprov-jdk15on-1.59.jar` > > **Risk:** While the Bouncy Castle Licence (MIT-style) is ASF Category A and license-compatible, version **1.59** (released 2018) has multiple known security vulnerabilities: > > * **[CVE-2018-1000613](https://github.com/advisories/GHSA-4446-656p-f54g)** — Unsafe class construction via `ObjectInputStream` > * **[CVE-2020-15522](https://github.com/advisories/GHSA-6xx3-rg99-gc3p)** — Timing side-channel in EC key generation > * **[CVE-2020-26939](https://github.com/advisories/GHSA-72m5-fvvv-55m6)** — Observable differences in scalar multiplication for `P-521` > * **[CVE-2023-33201](https://github.com/advisories/GHSA-hr8g-6v94-x4m9)** — LDAP injection via `X.500` name processing > > This is not a license risk per se, but a **security and compliance risk** — including a dependency with known, unpatched CVEs may fail OWASP dependency-check and could be flagged in ASF release audits. > > **How to address (Option A — Preferred):** Exclude the dependency if it is not actually used by the hadoop-bos code path: > > ``` > <exclusion> > <groupId>org.bouncycastle</groupId> > <artifactId>bcprov-jdk15on</artifactId> > </exclusion> > ``` > > **How to address (Option B):** If the BOS SDK requires it at runtime, upgrade to a non-vulnerable version. Note that `bcprov-jdk15on` is EOL; the replacement artifact is `bcprov-jdk18on` (currently at 1.78+). This would need to be coordinated with the BOS SDK's compatibility. > > ### Risk 3: Shading does not neutralize licensing obligations — PR description is misleading > **Affected JARs:** All shaded transitive dependencies > > **Risk:** The PR description states: _"No changes required to LICENSE, LICENSE-binary, or NOTICE-binary files as dependencies are shaded and not exposed in distribution."_ This reflects a fundamental misunderstanding of ASF licensing policy. Shading (bytecode relocation) changes the package namespace but does **not** change the license that applies to the code. Specifically: > > * The EPL 1.0 code from Eclipse Paho remains EPL 1.0 after relocation to `bfs.org.eclipse.paho.*` > * The MIT code from Bouncy Castle and SLF4J remains MIT-licensed after relocation > * Any NOTICE files from transitive dependencies must still be aggregated > > **How to address:** > > 1. If Risk 1 is resolved by excluding Eclipse Paho, the remaining dependencies are all Category A (APL 2.0 or MIT), which do not require `LICENSE-binary` entries but still require `NOTICE-binary` aggregation if they carry NOTICE files > 2. Audit each shaded dependency for NOTICE files and aggregate them > 3. Correct the PR description to accurately reflect licensing obligations @LuciferYang Thank you for the thorough dependency analysis! Regarding the risks you identified: Risk 1 (org.eclipse.paho) & Risk 2 (bcprov-jdk15on): Both dependencies have already been excluded in the pom.xml (lines 95-102). They are transitive dependencies of bce-java-sdk not needed by the BOS storage connector. You can verify this in the [pom.xml exclusions section](https://github.com/apache/hadoop/pull/8347/files#diff-hadoop-cloud-storage-project/hadoop-bos/pom.xml). Risk 3 (Shading and LICENSE obligations): I agree the PR description was misleading. The shaded dependencies are all Apache License 2.0 or compatible: bce-java-sdk — Apache 2.0 httpcomponents (httpclient, httpcore, httpcore-nio, httpasyncclient) — Apache 2.0 commons-codec — Apache 2.0 joda-time — Apache 2.0 guava + failureaccess — Apache 2.0 org.json:json — JSON License (ASF Category B) I will review whether org.json:json needs special handling or can be excluded/replaced per ASF policy. I will also update the PR description to accurately reflect the licensing situation. @roryqi Regarding JAR conflicts: all potentially conflicting transitive dependencies (jackson, guava, httpcomponents, commons-codec, etc.) are shaded under org.apache.hadoop.fs.bos.shaded.* prefix to avoid runtime classpath conflicts with Hadoop's own dependencies. The shading includes are explicitly scoped — only the listed artifacts are bundled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
