kevinjqliu opened a new issue, #3690: URL: https://github.com/apache/parquet-java/issues/3690
## Background We need to identify which published JARs bundle third-party code. ASF release policy requires JARs to place [LICENSE and NOTICE in `META-INF`](https://www.apache.org/legal/release-policy.html#what-are-the-requirements-to-distribute-other-artifacts-in-addition-to-the-source-package) and requires the [LICENSE file to cover every bundled non-Apache component](https://www.apache.org/legal/release-policy.html#the-license-file). Any [required third-party notices](https://www.apache.org/legal/resolved.html#what-are-required-third-party-notices) also need to be included. This issue documents the findings and provides a script to verify them. ## Findings Cross-checking the final JAR contents against the Maven Shade configuration found: - 7 binary JARs with bundled third-party classes - 13 Javadoc JARs with JDK-generated assets - 39 JARs with no bundled third-party code | Artifact | Maven configuration | Final JAR contents | |---|---|---| | `parquet-avro-*.jar` | Explicitly shades FastUtil | FastUtil | | `parquet-column-*.jar` | Inherits FastUtil/OpenHFT Shade rules | FastUtil, OpenHFT | | `parquet-format-structures-*.jar` | Explicitly includes `libthrift` | Apache Thrift | | `parquet-hadoop-*.jar` | Shade processes Jackson and FastUtil; filters/minimization remove Jackson | FastUtil only | | `parquet-jackson-*.jar` | Explicitly includes Jackson artifacts | Jackson | | `parquet-hadoop-bundle-*.jar` | Includes other `org.apache.parquet:parquet-*` JARs | Carries their already-shaded third-party classes | | `parquet-cli-*-runtime.jar` | `shadedArtifactAttached=true`, classifier `runtime`, includes all dependencies | Full runtime dependency set | A few edge cases: - `parquet-variant`, `parquet-protobuf`, and `parquet-thrift` run Shade, but filtering and minimization leave no third-party classes in the final JARs. - `parquet-benchmarks.jar` bundles the full dependency tree, but it is a local-only uber-JAR. The published versioned `parquet-benchmarks-*.jar` contains only project classes. - The 15 source and 15 test JARs contain project output rather than dependency code. - The 13 Javadoc JARs contain JDK-generated JavaScript and CSS. [ASF policy explicitly permits these assets in Maven Javadoc JARs](https://www.apache.org/legal/resolved.html#from-java-9-onwards-javadoc-can-include-search-functionality-that-includes-javascript-under-other-open-source-licenses-can-apache-projects-include-this-javadoc). <details> <summary>Setup and verification commands</summary> ### Setup Run the setup commands from the repository root with Java 17. Maven output is left visible so build progress and failures are easy to follow. ```bash java -version ./mvnw clean package --batch-mode -DskipTests ./mvnw --batch-mode process-resources source:jar-no-fork javadoc:jar ``` ### Verification After setup completes, run the verification separately: ```bash bash <<'EOF' set -euo pipefail export LC_ALL=C tmp_dir=$(mktemp -d) trap 'rm -rf "$tmp_dir"' EXIT candidate_jars="$tmp_dir/candidates" third_party_jars="$tmp_dir/third-party" javadoc_jars="$tmp_dir/javadocs" clean_jars="$tmp_dir/clean" : >"$third_party_jars" : >"$javadoc_jars" : >"$clean_jars" find . -mindepth 3 -maxdepth 3 -type f \ -path './*/target/*.jar' \ ! -name 'original-*.jar' \ ! -name 'parquet-benchmarks.jar' \ -print0 >"$candidate_jars" count_non_parquet_classes() { jar tf "$1" | awk ' /\.class$/ { path=$0 sub(/^META-INF\/versions\/[0-9]+\//, "", path) if (path !~ /^org\/apache\/parquet\//) count++ } END { print count+0 } ' } count_external_sources() { jar tf "$1" | awk ' /\.java$/ && $0 !~ /^org\/apache\/parquet\// { count++ } END { print count+0 } ' } count_extra_test_classes() { local jar_file=$1 local module_dir=$2 local jar_classes="$tmp_dir/jar-classes" local project_classes="$tmp_dir/project-classes" jar tf "$jar_file" | awk '/\.class$/' | sort -u >"$jar_classes" : >"$project_classes" if [[ -d "$module_dir/target/test-classes" ]]; then ( cd "$module_dir/target/test-classes" find . -type f -name '*.class' -print | sed 's#^\./##' | sort -u ) >"$project_classes" fi comm -13 "$project_classes" "$jar_classes" | awk 'END { print NR+0 }' } printf 'Inspecting existing JARs...\n' jar_count=0 while IFS= read -r -d '' jar_file; do jar_count=$((jar_count + 1)) jar_name=$(basename "$jar_file") module_dir=${jar_file%/target/*} case "$jar_name" in *-javadoc.jar) unzip -p "$jar_file" script.js >"$tmp_dir/script.js" if grep -Fq 'Oracle and/or its affiliates' "$tmp_dir/script.js"; then printf '%s\tJDK-generated JavaScript/CSS\n' "$jar_file" >>"$javadoc_jars" else printf '%s\n' "$jar_file" >>"$clean_jars" fi continue ;; *-sources.jar) bundled_count=$(count_external_sources "$jar_file") ;; *-tests.jar) bundled_count=$(count_extra_test_classes "$jar_file" "$module_dir") ;; *) bundled_count=$(count_non_parquet_classes "$jar_file") ;; esac if [[ $bundled_count -gt 0 ]]; then printf '%s\t%d bundled classes or sources\n' \ "$jar_file" "$bundled_count" >>"$third_party_jars" else printf '%s\n' "$jar_file" >>"$clean_jars" fi done <"$candidate_jars" if [[ $jar_count -eq 0 ]]; then printf 'No JARs found. Run the setup commands first.\n' >&2 exit 1 fi printf '\n=== JARs containing bundled third-party code ===\n\n' sort "$third_party_jars" printf '\n=== Javadoc JARs containing JDK-provided assets ===\n\n' sort "$javadoc_jars" printf '\n=== JARs with no bundled third-party code found ===\n\n' sort "$clean_jars" printf '\nInspected %d published JARs.\n' "$jar_count" EOF ``` </details> The Java 17 build completed successfully, and the legal-file check verified all 59 JARs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
