[
https://issues.apache.org/jira/browse/HDDS-17775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai updated HDDS-17775:
------------------------------------
Fix Version/s: 2.3.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Remove commons-io, kerby-util and unrelocated macOS natives from the Ozone
> filesystem jar
> -----------------------------------------------------------------------------------------
>
> Key: HDDS-17775
> URL: https://issues.apache.org/jira/browse/HDDS-17775
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Client
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.3.0
>
>
> h3. Problem
> Two libraries entered the client dependencies after 2.2.0, each for a single
> call. Both are shaded into \{{ozone-filesystem-hadoop3}}, which the 2.2.0 jar
> did not include:
> ||library||added by||client use||compressed in the jar||
> |\{{commons-io:commons-io}}|\{{hadoop-hdds/client/pom.xml}}, in the merge of
> the HDDS-10685 short-circuit read branch (879707874b)|one
> \{{IOUtils.closeQuietly(sock)}} in \{{DomainSocketFactory}}|512 KB, 428
> classes|
> |\{{org.apache.kerby:kerby-util}}|\{{hadoop-ozone/common/pom.xml}}, by
> HDDS-15140, which moved \{{AWSV4AuthValidator}} into \{{ozone-common}}|two
> \{{Hex.encode(byte[])}} calls in \{{AWSV4AuthValidator}}|31 KB, 22 classes|
> Nothing else in the jar references either library. I scanned the constant
> pools of all classes in the shaded jar: each library has exactly one
> referencing class.
> In addition, two macOS aarch64 netty native libraries ship under their Ratis
> names, not relocated:
> {noformat}
> META-INF/native/liborg_apache_ratis_thirdparty_netty_transport_native_kqueue_aarch_64.jnilib
> (82 KB)
> META-INF/native/liborg_apache_ratis_thirdparty_netty_resolver_dns_native_macos_aarch_64.jnilib
> (59 KB)
> {noformat}
> {\{ozonefs-shaded/pom.xml}} unpacks the netty natives, renames each to the
> Ozone-relocated name with
> {\{copy-rename-maven-plugin}}, and excludes the original name in the shade
> filter. Both lists have only the x86_64 kqueue
> and resolver-dns natives. ratis-thirdparty 1.1.1 added the aarch_64 builds of
> both, so they are neither renamed nor
> excluded. The relocated classes look for
> \{{liborg_apache_ozone_shaded_org_apache_ratis_thirdparty_netty_...}}, so on
> Apple-silicon macOS the kqueue transport and the native DNS resolver can't
> load. Netty falls back to NIO and the JDK
> resolver, which is harmless, but the two files are dead weight.
> h3. Proposed fix
> * \{{DomainSocketFactory}}: use Ozone's
> \{{org.apache.hadoop.hdds.utils.IOUtils.closeQuietly}} (hdds-common), and
> remove \{{commons-io}} from \{{hadoop-hdds/client/pom.xml}}. \{{hdds-client}}
> has no test uses of commons-io.
> * \{{AWSV4AuthValidator}}: use Hadoop's
> \{{org.apache.hadoop.util.StringUtils.byteToHexString(byte[])}}. The class
> already imports Hadoop \{{StringUtils}} for \{{split}}, so the change needs
> no new import or dependency. Remove
> {\{kerby-util}} from \{{hadoop-ozone/common/pom.xml}}.
> ** The output is identical. A check over all 256 byte values and the empty
> array gave the same string from kerby
> {\{Hex.encode}} and \{{byteToHexString}}, so S3 signature comparison doesn't
> change.
> ** Don't use Ozone's \{{org.apache.hadoop.hdds.StringUtils.bytes2Hex}}. It is
> a debug format, uppercase with spaces
> (\{{"00 01 0A"}}), and would break every signature check.
> ** \{{java.util.HexFormat}} is not an option, because \{{ozone-common}}
> targets Java 8.
> ** \{{s3gateway}} keeps its own \{{kerby-util}} dependency
> (\{{AWSSignatureProcessor}}, \{{StringToSignProducer}}). It is
> server-side and out of scope.
> * \{{hadoop-ozone/ozonefs-shaded/pom.xml}}: for
> \{{netty_transport_native_kqueue_aarch_64}} and
> {\{netty_resolver_dns_native_macos_aarch_64}}, add a
> \{{copy-rename-maven-plugin}} \{{fileSet}} to the relocated name and
> an \{{<exclude>}} for the Ratis name, following the existing x86_64 entries.
> Alternatively, exclude them only, if
> Apple-silicon clients don't need the native transport. NIO is the fallback
> either way.
> * Optional: extend the \{{ban-rocksdb}} enforcer rule from HDDS-17772 (or add
> a sibling rule) to ban \{{commons-io}} and
> {\{kerby-util}} in the client modules too, so they can't come back unnoticed.
> h3. Impact
> * \{{ozone-filesystem-hadoop3}} shrinks by about 0.55 MB compressed and 450
> entries, plus 36 KB if the natives are only
> excluded.
> * Two fewer third-party libraries in client classpaths, shaded jars and their
> LICENSE/NOTICE.
> * commons-io and kerby-util: no behavior change. Natives: with renaming,
> Apple-silicon macOS clients can use netty's
> kqueue transport and native DNS resolver; with exclusion only, nothing
> changes, because they can't load today.
> h3. How to check
> {noformat}
> mvn dependency:tree -pl :ozone-filesystem-shaded
> -Dincludes=commons-io,org.apache.kerby # empty after the fix
> unzip -l hadoop-ozone/ozonefs-hadoop3/target/ozone-filesystem-hadoop3-*.jar |
> grep -cE 'shaded/org/apache/(commons/io|kerby)/' # 0
> unzip -l hadoop-ozone/ozonefs-hadoop3/target/ozone-filesystem-hadoop3-*.jar |
> grep -c 'native/liborg_apache_ratis_thirdparty' # 0
> {noformat}
> Tests: \{{TestAWSV4AuthValidator}}, \{{TestS3SecurityUtil}} and
> \{{TestS3DerivedKey}} (\{{ozone-manager}}) for the
> signature path, and \{{TestXceiverClientManagerSC}} (needs libhadoop) for the
> short-circuit path.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]