[
https://issues.apache.org/jira/browse/HDDS-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy updated HDDS-17778:
----------------------------------
Status: Patch Available (was: Open)
> Replace URIBuilder and http ParseException with JDK classes to drop
> httpclient and httpcore from Ozone client jars
> ------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17778
> URL: https://issues.apache.org/jira/browse/HDDS-17778
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Client
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
> Labels: pull-request-available
>
> h3. Problem
> {{ozone-common}} and {{ozone-filesystem-common}} declare {{httpclient}}
> 4.5.14 and {{httpcore}} 4.4.16 as compile dependencies. The client uses them
> for only two classes, in four places:
> ||class||used in||for||
> |{{org.apache.http.client.utils.URIBuilder}}|{{{}BasicOzoneFileSystem#initialize{}}},
> {{BasicRootedOzoneFileSystem#initialize}}|building {{o3fs://<authority>}} /
> {{ofs://<authority>}} from scheme and authority|
> |{{org.apache.http.client.utils.URIBuilder}}|{{OMNodeDetails#getOMDBCheckpointEndpointUrl}}|building
> the OM DB checkpoint URL with two constant boolean parameters|
> |{{org.apache.http.ParseException}}|{{{}OFSPath#initOFSPath{}}}, two
> {{throw}} statements|an unchecked exception for a non-ofs scheme, or when the
> temp bucket name can't be resolved|
>
>
>
> The shaded filesystem jar excludes {{{}hadoop-common{}}}, so these
> declarations are the only reason httpclient and httpcore are in it. Their
> relocated classes take *0.89 MB* (compressed) in
> {{{}ozone-filesystem-hadoop3{}}}:
> {noformat}
> ozone-filesystem-shaded
> \- ozone-filesystem-common:compile
> \- httpclient:4.5.14:compile
> \- httpcore:4.4.16:compile
> {noformat}
> That's small, but it's free to remove. The jar is shipped with every job that
> uses {{-libjars}} or {{{}spark.jars{}}}. It is 155 MB on master, 71 MB with
> HDDS-17772.
> h3. Proposed change
> # *Filesystem URIs:* replace {{new
> URIBuilder().setScheme(s).setHost(authority).build()}} with {{{}new URI(s,
> authority, null, null, null){}}}. Keep {{{}URIBuilder{}}}'s one extra
> behaviour: it brackets an unbracketed IPv6 literal (for {{{}ofs://::1/{}}},
> {{URI#getAuthority()}} returns {{{}::1{}}}). Bracket the authority when
> {{InetAddresses.isInetAddress(authority)}} and it contains {{{}:{}}}. A small
> helper in {{OzoneFSUtils}} serves both file systems.
> # *OM checkpoint URL:* replace it with {{{}new URI(scheme, address, path,
> query, null).toURL(){}}}. The query is
> {{{}includeSnapshotData=true&flushBeforeCheckpoint=<flag>{}}}. Both values
> are constants, so nothing needs encoding.
> # *{{{}OFSPath{}}}:* throw {{IllegalArgumentException}} with the same
> messages. It is unchecked, like {{{}org.apache.http.ParseException{}}}, and
> nothing in the code base catches the http {{{}ParseException{}}}. The three
> {{catch (ParseException ...)}} in {{{}TrashPolicyOzone{}}},
> {{S3ConditionalRequest}} and {{ReconUtils}} are
> {{{}java.text.ParseException{}}}.
> # *Build:*
> ** remove {{httpclient}} and {{httpcore}} from
> {{hadoop-ozone/common/pom.xml}} and
> {{{}hadoop-ozone/ozonefs-common/pom.xml{}}};
> ** add {{org.apache.httpcomponents}} to the banned dependencies of the
> client modules, next to the RocksDB rule from HDDS-17772.
> {{commons-codec}} also arrives through httpclient, but {{hdds-common}}
> declares and uses it, so it stays.
> h3. Behaviour
> Checked {{URIBuilder}} against {{java.net.URI}} side by side (the bracketing
> helper of step 1 is not built yet):
>
> ||authority||URIBuilder||java.net.URI 5-arg||
> |\{{bucket.vol}}, \{{bucket.vol.om1}}, \{{omservice}} |same|same|
> |\{{om1:9862}}, \{{bucket.vol.om1:9862}}, \{{10.0.0.1:9874}} |same|same|
> |\{{[::1]:9862}} |\{{ofs://[::1]:9862}} |same|
> |\{{::1}} (unbracketed)|\{{ofs://[::1]}} |\{{ofs://::1}}, but unreachable:
> both file systems reject it first (2.3.0-SNAPSHOT, all four forms
> \{{ofs://::1/}}, \{{ofs://::1:9862/}}, \{{o3fs://bucket.vol.::1/}},
> \{{o3fs://bucket.vol.::1:9862/}})|
> |empty|\{{URISyntaxException}} |\{{URISyntaxException}} |
> |\{{https}}, \{{om1:9874}} / \{{[::1]:9875}}, path, two parameters|same|same|
> {{o3fs}} / {{ofs}} {{getUri()}} and the OM checkpoint URL stay byte-for-byte
> the same.
> {{OFSPath}} throws {{IllegalArgumentException}} instead of
> {{{}org.apache.http.ParseException{}}}, with the same message.
> h3. Compatibility
> * No wire, proto, config or on-disk changes.
> * Applications that depend on the unshaded {{ozone-common}} or
> {{ozone-filesystem-common}} and use httpclient without declaring it lose it
> as a transitive dependency. Inside Ozone, {{{}s3gateway{}}},
> {{{}cli-shell{}}}, {{vapor}} and the integration test modules all declare
> httpclient themselves.
> h3. Testing
> * *Unit:*
> ** {{getUri()}} of both file systems for host, host:port, service id,
> bracketed IPv6 with port, and an unbracketed IPv6 literal;
> ** {{getOMDBCheckpointEndpointUrl}} for http and https, with {{useV2}} and
> {{flush}} both ways, against the URL that {{URIBuilder}} builds today;
> ** {{OFSPath}} with a non-ofs scheme throws {{IllegalArgumentException}}
> with the old message.
> * *Existing:* {{{}TestOzoneFileSystem{}}},
> {{{}TestRootedOzoneFileSystem{}}}, {{{}TestOFSPath{}}}, and the OM bootstrap
> and checkpoint tests.
> * *Jar contents:* {{ozone-filesystem-hadoop3}} has no
> {{org/apache/ozone/shaded/org/apache/http}} and no
> {{{}META-INF/maven/org.apache.httpcomponents{}}}. Record its size before and
> after.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]