Andrey Yarovoy created HDDS-17775:
-------------------------------------

             Summary: Remove commons-io, kerby-util and unrelocated macOS 
natives from the Ozone filesystem jar
                 Key: HDDS-17775
                 URL: https://issues.apache.org/jira/browse/HDDS-17775
             Project: Apache Ozone
          Issue Type: Bug
          Components: Ozone Client
            Reporter: Andrey Yarovoy


h3. Problem
Two libraries entered the client dependencies after 2.2.0, each for a single 
call. Both are shaded into \{{ozone-filesystem-hadoop3}}, which the 2.2.0 jar 
did not include:

||library||added by||client use||compressed in the jar||
|\{{commons-io:commons-io}}|\{{hadoop-hdds/client/pom.xml}}, in the merge of 
the HDDS-10685 short-circuit read branch (879707874b)|one 
\{{IOUtils.closeQuietly(sock)}} in \{{DomainSocketFactory}}|512 KB, 428 classes|
|\{{org.apache.kerby:kerby-util}}|\{{hadoop-ozone/common/pom.xml}}, by 
HDDS-15140, which moved \{{AWSV4AuthValidator}} into \{{ozone-common}}|two 
\{{Hex.encode(byte[])}} calls in \{{AWSV4AuthValidator}}|31 KB, 22 classes|

Nothing else in the jar references either library. I scanned the constant pools 
of all classes in the shaded jar: each library has exactly one referencing 
class.

In addition, two macOS aarch64 netty native libraries ship under their Ratis 
names, not relocated:
{noformat}
META-INF/native/liborg_apache_ratis_thirdparty_netty_transport_native_kqueue_aarch_64.jnilib
 (82 KB)
META-INF/native/liborg_apache_ratis_thirdparty_netty_resolver_dns_native_macos_aarch_64.jnilib
 (59 KB)
{noformat}
{\{ozonefs-shaded/pom.xml}} unpacks the netty natives, renames each to the 
Ozone-relocated name with
{\{copy-rename-maven-plugin}}, and excludes the original name in the shade 
filter. Both lists have only the x86_64 kqueue
and resolver-dns natives. ratis-thirdparty 1.1.1 added the aarch_64 builds of 
both, so they are neither renamed nor
excluded. The relocated classes look for 
\{{liborg_apache_ozone_shaded_org_apache_ratis_thirdparty_netty_...}}, so on
Apple-silicon macOS the kqueue transport and the native DNS resolver can't 
load. Netty falls back to NIO and the JDK
resolver, which is harmless, but the two files are dead weight.

h3. Proposed fix
* \{{DomainSocketFactory}}: use Ozone's 
\{{org.apache.hadoop.hdds.utils.IOUtils.closeQuietly}} (hdds-common), and
remove \{{commons-io}} from \{{hadoop-hdds/client/pom.xml}}. \{{hdds-client}} 
has no test uses of commons-io.
* \{{AWSV4AuthValidator}}: use Hadoop's 
\{{org.apache.hadoop.util.StringUtils.byteToHexString(byte[])}}. The class
already imports Hadoop \{{StringUtils}} for \{{split}}, so the change needs no 
new import or dependency. Remove
{\{kerby-util}} from \{{hadoop-ozone/common/pom.xml}}.
** The output is identical. A check over all 256 byte values and the empty 
array gave the same string from kerby
{\{Hex.encode}} and \{{byteToHexString}}, so S3 signature comparison doesn't 
change.
** Don't use Ozone's \{{org.apache.hadoop.hdds.StringUtils.bytes2Hex}}. It is a 
debug format, uppercase with spaces
(\{{"00 01 0A"}}), and would break every signature check.
** \{{java.util.HexFormat}} is not an option, because \{{ozone-common}} targets 
Java 8.
** \{{s3gateway}} keeps its own \{{kerby-util}} dependency 
(\{{AWSSignatureProcessor}}, \{{StringToSignProducer}}). It is
server-side and out of scope.
* \{{hadoop-ozone/ozonefs-shaded/pom.xml}}: for 
\{{netty_transport_native_kqueue_aarch_64}} and
{\{netty_resolver_dns_native_macos_aarch_64}}, add a 
\{{copy-rename-maven-plugin}} \{{fileSet}} to the relocated name and
an \{{<exclude>}} for the Ratis name, following the existing x86_64 entries. 
Alternatively, exclude them only, if
Apple-silicon clients don't need the native transport. NIO is the fallback 
either way.
* Optional: extend the \{{ban-rocksdb}} enforcer rule from HDDS-17772 (or add a 
sibling rule) to ban \{{commons-io}} and
{\{kerby-util}} in the client modules too, so they can't come back unnoticed.

h3. Impact
* \{{ozone-filesystem-hadoop3}} shrinks by about 0.55 MB compressed and 450 
entries, plus 36 KB if the natives are only
excluded.
* Two fewer third-party libraries in client classpaths, shaded jars and their 
LICENSE/NOTICE.
* commons-io and kerby-util: no behavior change. Natives: with renaming, 
Apple-silicon macOS clients can use netty's
kqueue transport and native DNS resolver; with exclusion only, nothing changes, 
because they can't load today.

h3. How to check
{noformat}
mvn dependency:tree -pl :ozone-filesystem-shaded 
-Dincludes=commons-io,org.apache.kerby # empty after the fix
unzip -l hadoop-ozone/ozonefs-hadoop3/target/ozone-filesystem-hadoop3-*.jar | 
grep -cE 'shaded/org/apache/(commons/io|kerby)/' # 0
unzip -l hadoop-ozone/ozonefs-hadoop3/target/ozone-filesystem-hadoop3-*.jar | 
grep -c 'native/liborg_apache_ratis_thirdparty' # 0
{noformat}
Tests: \{{TestAWSV4AuthValidator}}, \{{TestS3SecurityUtil}} and 
\{{TestS3DerivedKey}} (\{{ozone-manager}}) for the
signature path, and \{{TestXceiverClientManagerSC}} (needs libhadoop) for the 
short-circuit path.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to