[ 
https://issues.apache.org/jira/browse/HDDS-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrey Yarovoy updated HDDS-17778:
----------------------------------
    Description: 
h3. Problem

{{ozone-common}} and {{ozone-filesystem-common}} declare {{httpclient}} 4.5.14 
and {{httpcore}} 4.4.16 as compile dependencies. The client uses them for only 
two classes, in four places:
 
||class||used in||for||
|\{{org.apache.http.client.utils.URIBuilder}}|\{{BasicOzoneFileSystem#initialize}},
 \{{BasicRootedOzoneFileSystem#initialize}}|building \{{o3fs://<authority>}} / 
\{{ofs://<authority>}} from scheme and authority|
|\{{org.apache.http.client.utils.URIBuilder}}|\{{OMNodeDetails#getOMDBCheckpointEndpointUrl}}|building
 the OM DB checkpoint URL with two constant boolean parameters|
|\{{org.apache.http.ParseException}}|\{{OFSPath#initOFSPath}}, two \{{throw}} 
statements|an unchecked exception for a non-ofs scheme, or when the temp bucket 
name can't be resolved|

The shaded filesystem jar excludes {{{}hadoop-common{}}}, so these declarations 
are the only reason httpclient and httpcore are in it. Their relocated classes 
take *0.89 MB* (compressed) in {{{}ozone-filesystem-hadoop3{}}}:
{noformat}
ozone-filesystem-shaded
\- ozone-filesystem-common:compile
\- httpclient:4.5.14:compile
\- httpcore:4.4.16:compile
{noformat}
That's small, but it's free to remove. The jar is shipped with every job that 
uses {{-libjars}} or {{{}spark.jars{}}}. It is 155 MB on master, 71 MB with 
HDDS-17772.
h3. Proposed change
 # *Filesystem URIs:* replace {{new 
URIBuilder().setScheme(s).setHost(authority).build()}} with {{{}new URI(s, 
authority, null, null, null){}}}. Keep {{{}URIBuilder{}}}'s one extra 
behaviour: it brackets an unbracketed IPv6 literal (for {{{}ofs://::1/{}}}, 
{{URI#getAuthority()}} returns {{{}::1{}}}). Bracket the authority when 
{{InetAddresses.isInetAddress(authority)}} and it contains {{{}:{}}}. A small 
helper in {{OzoneFSUtils}} serves both file systems.
 # *OM checkpoint URL:* replace it with {{{}new URI(scheme, address, path, 
query, null).toURL(){}}}. The query is 
{{{}includeSnapshotData=true&flushBeforeCheckpoint=<flag>{}}}. Both values are 
constants, so nothing needs encoding.
 # *{{{}OFSPath{}}}:* throw {{IllegalArgumentException}} with the same 
messages. It is unchecked, like {{{}org.apache.http.ParseException{}}}, and 
nothing in the code base catches the http {{{}ParseException{}}}. The three 
{{catch (ParseException ...)}} in {{{}TrashPolicyOzone{}}}, 
{{S3ConditionalRequest}} and {{ReconUtils}} are 
{{{}java.text.ParseException{}}}.
 # *Build:*
 ** remove {{httpclient}} and {{httpcore}} from {{hadoop-ozone/common/pom.xml}} 
and {{{}hadoop-ozone/ozonefs-common/pom.xml{}}};
 ** add {{org.apache.httpcomponents}} to the banned dependencies of the client 
modules, next to the RocksDB rule from HDDS-17772.

{{commons-codec}} also arrives through httpclient, but {{hdds-common}} declares 
and uses it, so it stays.
h3. Behaviour

Checked {{URIBuilder}} against {{java.net.URI}} side by side (the bracketing 
helper of step 1 is not built yet):


||authority||URIBuilder||java.net.URI 5-arg||
|\{{bucket.vol}}, \{{bucket.vol.om1}}, \{{omservice}}|same|same|
|\{{om1:9862}}, \{{bucket.vol.om1:9862}}, \{{10.0.0.1:9874}}|same|same|
|\{{[::1]:9862}}|\{{ofs://[::1]:9862}}|same|
|\{{::1}} (unbracketed)|\{{ofs://[::1]}}|\{{ofs://::1}}, *so bracket it (step 
1)*|
|empty|\{{URISyntaxException}}|\{{URISyntaxException}}|
|\{{https}}, \{{om1:9874}} / \{{[::1]:9875}}, path, two parameters|same|same|
  * {{o3fs}} / {{ofs}} {{getUri()}} and the OM checkpoint URL stay 
byte-for-byte the same.
 * {{OFSPath}} throws {{IllegalArgumentException}} instead of 
{{{}org.apache.http.ParseException{}}}, with the same message.

h3. Compatibility
 * No wire, proto, config or on-disk changes.
 * Applications that depend on the unshaded {{ozone-common}} or 
{{ozone-filesystem-common}} and use httpclient without declaring it lose it as 
a transitive dependency. Inside Ozone, {{{}s3gateway{}}}, {{{}cli-shell{}}}, 
{{vapor}} and the integration test modules all declare httpclient themselves.

h3. Testing
 * *Unit:*
 ** {{getUri()}} of both file systems for host, host:port, service id, 
bracketed IPv6 with port, and an unbracketed IPv6 literal;
 ** {{getOMDBCheckpointEndpointUrl}} for http and https, with {{useV2}} and 
{{flush}} both ways, against the URL that {{URIBuilder}} builds today;
 ** {{OFSPath}} with a non-ofs scheme throws {{IllegalArgumentException}} with 
the old message.
 * *Existing:* {{{}TestOzoneFileSystem{}}}, {{{}TestRootedOzoneFileSystem{}}}, 
{{{}TestOFSPath{}}}, and the OM bootstrap and checkpoint tests.
 * *Jar contents:* {{ozone-filesystem-hadoop3}} has no 
{{org/apache/ozone/shaded/org/apache/http}} and no 
{{{}META-INF/maven/org.apache.httpcomponents{}}}. Record its size before and 
after.

  was:
h3. Problem

{{ozone-common}} and {{ozone-filesystem-common}} declare {{httpclient}} 4.5.14 
and {{httpcore}} 4.4.16 as compile dependencies. The client uses them for only 
two classes, in four places:
||class||used in||for||
|\{{org.apache.http.client.utils.URIBuilder}}|\{{BasicOzoneFileSystem#initialize}},
 \{{BasicRootedOzoneFileSystem#initialize}}|building \{{o3fs://<authority>}} / 
\{{ofs://<authority>}} from scheme and authority|
|\{{org.apache.http.client.utils.URIBuilder}}|\{{OMNodeDetails#getOMDBCheckpointEndpointUrl}}|building
 the OM DB checkpoint URL with two constant boolean parameters|
|\{{org.apache.http.ParseException}}|\{{OFSPath#initOFSPath}}, two 
\{{throw}}s|an unchecked exception for a non-ofs scheme, or when the temp 
bucket name can't be resolved|
 
The shaded filesystem jar excludes {{{}hadoop-common{}}}, so these declarations 
are the only reason httpclient and httpcore are in it. Their relocated classes 
take *0.89 MB* (compressed) in {{{}ozone-filesystem-hadoop3{}}}:
{noformat}
ozone-filesystem-shaded
\- ozone-filesystem-common:compile
\- httpclient:4.5.14:compile
\- httpcore:4.4.16:compile
{noformat}
That's small, but it's free to remove. The jar is shipped with every job that 
uses {{-libjars}} or {{{}spark.jars{}}}. It is 155 MB on master, 71 MB with 
HDDS-17772.
h3. Proposed change
 # *Filesystem URIs:* replace {{new 
URIBuilder().setScheme(s).setHost(authority).build()}} with {{{}new URI(s, 
authority, null, null, null){}}}. Keep {{{}URIBuilder{}}}'s one extra 
behaviour: it brackets an unbracketed IPv6 literal (for {{{}ofs://::1/{}}}, 
{{URI#getAuthority()}} returns {{{}::1{}}}). Bracket the authority when 
{{InetAddresses.isInetAddress(authority)}} and it contains {{{}:{}}}. A small 
helper in {{OzoneFSUtils}} serves both file systems.
 # *OM checkpoint URL:* replace it with {{{}new URI(scheme, address, path, 
query, null).toURL(){}}}. The query is 
{{{}includeSnapshotData=true&flushBeforeCheckpoint=<flag>{}}}. Both values are 
constants, so nothing needs encoding.
 # *{{{}OFSPath{}}}:* throw {{IllegalArgumentException}} with the same 
messages. It is unchecked, like {{{}org.apache.http.ParseException{}}}, and 
nothing in the code base catches the http {{{}ParseException{}}}. The three 
{{catch (ParseException ...)}} in {{{}TrashPolicyOzone{}}}, 
{{S3ConditionalRequest}} and {{ReconUtils}} are 
{{{}java.text.ParseException{}}}.
 # *Build:*
 ** remove {{httpclient}} and {{httpcore}} from {{hadoop-ozone/common/pom.xml}} 
and {{{}hadoop-ozone/ozonefs-common/pom.xml{}}};
 ** add {{org.apache.httpcomponents}} to the banned dependencies of the client 
modules, next to the RocksDB rule from HDDS-17772.

{{commons-codec}} also arrives through httpclient, but {{hdds-common}} declares 
and uses it, so it stays.
h3. Behaviour

Checked {{URIBuilder}} against {{java.net.URI}} side by side (the bracketing 
helper of step 1 is not built yet):
||authority||URIBuilder||java.net.URI 5-arg||
|{{{}bucket.vol{}}}, {{{}bucket.vol.om1{}}}, \{ Unknown macro: \{omservice}
}|same|same|
|{{{}om1:9862{}}}, {{{}bucket.vol.om1:9862{}}}, \{ Unknown macro: \{10.0.0.1}
}|same|same|
|{Unknown macro: \{[::1]}}|{Unknown macro: \\{ofs}}|same|
|{{::1}} (unbracketed)|{Unknown macro: \\{ofs}}|{{{}ofs://::1{}}}, *so bracket 
it (step 1)*|
|empty|{Unknown macro: \\{URISyntaxException}}|{Unknown macro: 
\\{URISyntaxException}}|
|{{{}https{}}}, {{om1:9874}} / {{{}[::1]:9875{}}}, path, two 
parameters|same|same|
 * {{o3fs}} / {{ofs}} {{getUri()}} and the OM checkpoint URL stay byte-for-byte 
the same.
 * {{OFSPath}} throws {{IllegalArgumentException}} instead of 
{{{}org.apache.http.ParseException{}}}, with the same message.

h3. Compatibility
 * No wire, proto, config or on-disk changes.
 * Applications that depend on the unshaded {{ozone-common}} or 
{{ozone-filesystem-common}} and use httpclient without declaring it lose it as 
a transitive dependency. Inside Ozone, {{{}s3gateway{}}}, {{{}cli-shell{}}}, 
{{vapor}} and the integration test modules all declare httpclient themselves.

h3. Testing
 * *Unit:*
 ** {{getUri()}} of both file systems for host, host:port, service id, 
bracketed IPv6 with port, and an unbracketed IPv6 literal;
 ** {{getOMDBCheckpointEndpointUrl}} for http and https, with {{useV2}} and 
{{flush}} both ways, against the URL that {{URIBuilder}} builds today;
 ** {{OFSPath}} with a non-ofs scheme throws {{IllegalArgumentException}} with 
the old message.
 * *Existing:* {{{}TestOzoneFileSystem{}}}, {{{}TestRootedOzoneFileSystem{}}}, 
{{{}TestOFSPath{}}}, and the OM bootstrap and checkpoint tests.
 * *Jar contents:* {{ozone-filesystem-hadoop3}} has no 
{{org/apache/ozone/shaded/org/apache/http}} and no 
{{{}META-INF/maven/org.apache.httpcomponents{}}}. Record its size before and 
after.


> Replace URIBuilder and http ParseException with JDK classes to drop 
> httpclient and httpcore from Ozone client jars
> ------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDDS-17778
>                 URL: https://issues.apache.org/jira/browse/HDDS-17778
>             Project: Apache Ozone
>          Issue Type: Improvement
>          Components: Ozone Client
>            Reporter: Andrey Yarovoy
>            Assignee: Andrey Yarovoy
>            Priority: Major
>
> h3. Problem
> {{ozone-common}} and {{ozone-filesystem-common}} declare {{httpclient}} 
> 4.5.14 and {{httpcore}} 4.4.16 as compile dependencies. The client uses them 
> for only two classes, in four places:
>  
> ||class||used in||for||
> |\{{org.apache.http.client.utils.URIBuilder}}|\{{BasicOzoneFileSystem#initialize}},
>  \{{BasicRootedOzoneFileSystem#initialize}}|building \{{o3fs://<authority>}} 
> / \{{ofs://<authority>}} from scheme and authority|
> |\{{org.apache.http.client.utils.URIBuilder}}|\{{OMNodeDetails#getOMDBCheckpointEndpointUrl}}|building
>  the OM DB checkpoint URL with two constant boolean parameters|
> |\{{org.apache.http.ParseException}}|\{{OFSPath#initOFSPath}}, two \{{throw}} 
> statements|an unchecked exception for a non-ofs scheme, or when the temp 
> bucket name can't be resolved|
> The shaded filesystem jar excludes {{{}hadoop-common{}}}, so these 
> declarations are the only reason httpclient and httpcore are in it. Their 
> relocated classes take *0.89 MB* (compressed) in 
> {{{}ozone-filesystem-hadoop3{}}}:
> {noformat}
> ozone-filesystem-shaded
> \- ozone-filesystem-common:compile
> \- httpclient:4.5.14:compile
> \- httpcore:4.4.16:compile
> {noformat}
> That's small, but it's free to remove. The jar is shipped with every job that 
> uses {{-libjars}} or {{{}spark.jars{}}}. It is 155 MB on master, 71 MB with 
> HDDS-17772.
> h3. Proposed change
>  # *Filesystem URIs:* replace {{new 
> URIBuilder().setScheme(s).setHost(authority).build()}} with {{{}new URI(s, 
> authority, null, null, null){}}}. Keep {{{}URIBuilder{}}}'s one extra 
> behaviour: it brackets an unbracketed IPv6 literal (for {{{}ofs://::1/{}}}, 
> {{URI#getAuthority()}} returns {{{}::1{}}}). Bracket the authority when 
> {{InetAddresses.isInetAddress(authority)}} and it contains {{{}:{}}}. A small 
> helper in {{OzoneFSUtils}} serves both file systems.
>  # *OM checkpoint URL:* replace it with {{{}new URI(scheme, address, path, 
> query, null).toURL(){}}}. The query is 
> {{{}includeSnapshotData=true&flushBeforeCheckpoint=<flag>{}}}. Both values 
> are constants, so nothing needs encoding.
>  # *{{{}OFSPath{}}}:* throw {{IllegalArgumentException}} with the same 
> messages. It is unchecked, like {{{}org.apache.http.ParseException{}}}, and 
> nothing in the code base catches the http {{{}ParseException{}}}. The three 
> {{catch (ParseException ...)}} in {{{}TrashPolicyOzone{}}}, 
> {{S3ConditionalRequest}} and {{ReconUtils}} are 
> {{{}java.text.ParseException{}}}.
>  # *Build:*
>  ** remove {{httpclient}} and {{httpcore}} from 
> {{hadoop-ozone/common/pom.xml}} and 
> {{{}hadoop-ozone/ozonefs-common/pom.xml{}}};
>  ** add {{org.apache.httpcomponents}} to the banned dependencies of the 
> client modules, next to the RocksDB rule from HDDS-17772.
> {{commons-codec}} also arrives through httpclient, but {{hdds-common}} 
> declares and uses it, so it stays.
> h3. Behaviour
> Checked {{URIBuilder}} against {{java.net.URI}} side by side (the bracketing 
> helper of step 1 is not built yet):
> ||authority||URIBuilder||java.net.URI 5-arg||
> |\{{bucket.vol}}, \{{bucket.vol.om1}}, \{{omservice}}|same|same|
> |\{{om1:9862}}, \{{bucket.vol.om1:9862}}, \{{10.0.0.1:9874}}|same|same|
> |\{{[::1]:9862}}|\{{ofs://[::1]:9862}}|same|
> |\{{::1}} (unbracketed)|\{{ofs://[::1]}}|\{{ofs://::1}}, *so bracket it (step 
> 1)*|
> |empty|\{{URISyntaxException}}|\{{URISyntaxException}}|
> |\{{https}}, \{{om1:9874}} / \{{[::1]:9875}}, path, two parameters|same|same|
>   * {{o3fs}} / {{ofs}} {{getUri()}} and the OM checkpoint URL stay 
> byte-for-byte the same.
>  * {{OFSPath}} throws {{IllegalArgumentException}} instead of 
> {{{}org.apache.http.ParseException{}}}, with the same message.
> h3. Compatibility
>  * No wire, proto, config or on-disk changes.
>  * Applications that depend on the unshaded {{ozone-common}} or 
> {{ozone-filesystem-common}} and use httpclient without declaring it lose it 
> as a transitive dependency. Inside Ozone, {{{}s3gateway{}}}, 
> {{{}cli-shell{}}}, {{vapor}} and the integration test modules all declare 
> httpclient themselves.
> h3. Testing
>  * *Unit:*
>  ** {{getUri()}} of both file systems for host, host:port, service id, 
> bracketed IPv6 with port, and an unbracketed IPv6 literal;
>  ** {{getOMDBCheckpointEndpointUrl}} for http and https, with {{useV2}} and 
> {{flush}} both ways, against the URL that {{URIBuilder}} builds today;
>  ** {{OFSPath}} with a non-ofs scheme throws {{IllegalArgumentException}} 
> with the old message.
>  * *Existing:* {{{}TestOzoneFileSystem{}}}, 
> {{{}TestRootedOzoneFileSystem{}}}, {{{}TestOFSPath{}}}, and the OM bootstrap 
> and checkpoint tests.
>  * *Jar contents:* {{ozone-filesystem-hadoop3}} has no 
> {{org/apache/ozone/shaded/org/apache/http}} and no 
> {{{}META-INF/maven/org.apache.httpcomponents{}}}. Record its size before and 
> after.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to