[
https://issues.apache.org/jira/browse/HDDS-17778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Yarovoy updated HDDS-17778:
----------------------------------
Description:
h3. Problem
{{ozone-common}} and {{ozone-filesystem-common}} declare {{httpclient}} 4.5.14
and {{httpcore}} 4.4.16 as compile dependencies. The client uses them for only
two classes, in four places:
||class||used in||for||
|{Unknown macro: \{org.apache.http.client.utils.URIBuilder}
}|{{{}BasicOzoneFileSystem#initialize{}}}, {
Unknown macro: \{BasicRootedOzoneFileSystem#initialize}
}|building {{o3fs://<authority>}} / {{ofs://<authority>}} from scheme and
authority|
|{Unknown macro: \{org.apache.http.client.utils.URIBuilder}
}|{Unknown macro: \{OMNodeDetails#getOMDBCheckpointEndpointUrl}
}|building the OM DB checkpoint URL with two constant boolean parameters|
|{Unknown macro: \{org.apache.http.ParseException}
}|{{{}OFSPath#initOFSPath{}}}, two \{{throw}}s|an unchecked exception for a
non-ofs scheme, or when the temp bucket name can't be resolved|
The shaded filesystem jar excludes {{{}hadoop-common{}}}, so these declarations
are the only reason httpclient and httpcore are in it. Their relocated classes
take *0.89 MB* (compressed) in {{{}ozone-filesystem-hadoop3{}}}:
{noformat}
ozone-filesystem-shaded
\- ozone-filesystem-common:compile
\- httpclient:4.5.14:compile
\- httpcore:4.4.16:compile
{noformat}
That's small, but it's free to remove. The jar is shipped with every job that
uses {{-libjars}} or {{{}spark.jars{}}}. It is 155 MB on master, 71 MB with
HDDS-17772.
h3. Proposed change
# *Filesystem URIs:* replace {{new
URIBuilder().setScheme(s).setHost(authority).build()}} with {{{}new URI(s,
authority, null, null, null){}}}. Keep {{{}URIBuilder{}}}'s one extra
behaviour: it brackets an unbracketed IPv6 literal (for {{{}ofs://::1/{}}},
{{URI#getAuthority()}} returns {{{}::1{}}}). Bracket the authority when
{{InetAddresses.isInetAddress(authority)}} and it contains {{{}:{}}}. A small
helper in {{OzoneFSUtils}} serves both file systems.
# *OM checkpoint URL:* replace it with {{{}new URI(scheme, address, path,
query, null).toURL(){}}}. The query is
{{{}includeSnapshotData=true&flushBeforeCheckpoint=<flag>{}}}. Both values are
constants, so nothing needs encoding.
# *{{{}OFSPath{}}}:* throw {{IllegalArgumentException}} with the same
messages. It is unchecked, like {{{}org.apache.http.ParseException{}}}, and
nothing in the code base catches the http {{{}ParseException{}}}. The three
{{catch (ParseException ...)}} in {{{}TrashPolicyOzone{}}},
{{S3ConditionalRequest}} and {{ReconUtils}} are
{{{}java.text.ParseException{}}}.
# *Build:*
** remove {{httpclient}} and {{httpcore}} from {{hadoop-ozone/common/pom.xml}}
and {{{}hadoop-ozone/ozonefs-common/pom.xml{}}};
** add {{org.apache.httpcomponents}} to the banned dependencies of the client
modules, next to the RocksDB rule from HDDS-17772.
{{commons-codec}} also arrives through httpclient, but {{hdds-common}} declares
and uses it, so it stays.
h3. Behaviour
Checked {{URIBuilder}} against {{java.net.URI}} side by side (the bracketing
helper of step 1 is not built yet):
||authority||URIBuilder||java.net.URI 5-arg||
|{{{}bucket.vol{}}}, {{{}bucket.vol.om1{}}}, {
Unknown macro: \{omservice}
}|same|same|
|{{{}om1:9862{}}}, {{{}bucket.vol.om1:9862{}}}, {
Unknown macro: \{10.0.0.1}
}|same|same|
|{Unknown macro: {[::1]}
}|{Unknown macro: \{ofs}
}|same|
|{{::1}} (unbracketed)|{Unknown macro: \{ofs}
}|{{{}ofs://::1{}}}, *so bracket it (step 1)*|
|empty|{Unknown macro: \{URISyntaxException}
}|{Unknown macro: \{URISyntaxException}
}|
|{{{}https{}}}, {{om1:9874}} / {{{}[::1]:9875{}}}, path, two
parameters|same|same|
* {{o3fs}} / {{ofs}} {{getUri()}} and the OM checkpoint URL stay byte-for-byte
the same.
* {{OFSPath}} throws {{IllegalArgumentException}} instead of
{{{}org.apache.http.ParseException{}}}, with the same message.
h3. Compatibility
* No wire, proto, config or on-disk changes.
* Applications that depend on the unshaded {{ozone-common}} or
{{ozone-filesystem-common}} and use httpclient without declaring it lose it as
a transitive dependency. Inside Ozone, {{{}s3gateway{}}}, {{{}cli-shell{}}},
{{vapor}} and the integration test modules all declare httpclient themselves.
h3. Testing
* *Unit:*
** {{getUri()}} of both file systems for host, host:port, service id,
bracketed IPv6 with port, and an unbracketed IPv6 literal;
** {{getOMDBCheckpointEndpointUrl}} for http and https, with {{useV2}} and
{{flush}} both ways, against the URL that {{URIBuilder}} builds today;
** {{OFSPath}} with a non-ofs scheme throws {{IllegalArgumentException}} with
the old message.
* *Existing:* {{{}TestOzoneFileSystem{}}}, {{{}TestRootedOzoneFileSystem{}}},
{{{}TestOFSPath{}}}, and the OM bootstrap and checkpoint tests.
* *Jar contents:* {{ozone-filesystem-hadoop3}} has no
{{org/apache/ozone/shaded/org/apache/http}} and no
{{{}META-INF/maven/org.apache.httpcomponents{}}}. Record its size before and
after.
was:
h3. Problem
{\{ozone-common}} and \{{ozone-filesystem-common}} declare \{{httpclient}}
4.5.14 and \{{httpcore}} 4.4.16 as compile dependencies. The client uses them
for only two classes, in four places:
||class||used in||for||
|\{{org.apache.http.client.utils.URIBuilder}}|\{{BasicOzoneFileSystem#initialize}},
\{{BasicRootedOzoneFileSystem#initialize}}|building \{{o3fs://<authority>}} /
\{{ofs://<authority>}} from scheme and authority|
|\{{org.apache.http.client.utils.URIBuilder}}|\{{OMNodeDetails#getOMDBCheckpointEndpointUrl}}|building
the OM DB checkpoint URL with two constant boolean parameters|
|\{{org.apache.http.ParseException}}|\{{OFSPath#initOFSPath}}, two
\{{throw}}s|an unchecked exception for a non-ofs scheme, or when the temp
bucket name can't be resolved|
The shaded filesystem jar excludes \{{hadoop-common}}, so these declarations
are the only reason httpclient and httpcore are in it. Their relocated classes
take *0.89 MB* (compressed) in \{{ozone-filesystem-hadoop3}}:
{noformat}
ozone-filesystem-shaded
\- ozone-filesystem-common:compile
\- httpclient:4.5.14:compile
\- httpcore:4.4.16:compile
{noformat}
That's small, but it's free to remove. The jar is shipped with every job that
uses \{{-libjars}} or \{{spark.jars}}. It is 155 MB on master, 71 MB with
HDDS-17772.
h3. Proposed change
# *Filesystem URIs:* replace \{{new
URIBuilder().setScheme(s).setHost(authority).build()}} with \{{new URI(s,
authority, null, null, null)}}. Keep \{{URIBuilder}}'s one extra behaviour: it
brackets an unbracketed IPv6 literal (for \{{ofs://::1/}},
\{{URI#getAuthority()}} returns \{{::1}}). Bracket the authority when
\{{InetAddresses.isInetAddress(authority)}} and it contains \{{:}}. A small
helper in \{{OzoneFSUtils}} serves both file systems.
# *OM checkpoint URL:* replace it with \{{new URI(scheme, address, path, query,
null).toURL()}}. The query is
\{{includeSnapshotData=true&flushBeforeCheckpoint=<flag>}}. Both values are
constants, so nothing needs encoding.
# *\{{OFSPath}}:* throw \{{IllegalArgumentException}} with the same messages.
It is unchecked, like \{{org.apache.http.ParseException}}, and nothing in the
code base catches the http \{{ParseException}}. The three \{{catch
(ParseException ...)}} in \{{TrashPolicyOzone}}, \{{S3ConditionalRequest}} and
\{{ReconUtils}} are \{{java.text.ParseException}}.
# *Build:*
#* remove \{{httpclient}} and \{{httpcore}} from
\{{hadoop-ozone/common/pom.xml}} and \{{hadoop-ozone/ozonefs-common/pom.xml}};
#* add \{{org.apache.httpcomponents}} to the banned dependencies of the client
modules, next to the RocksDB rule from HDDS-17772.
{\{commons-codec}} also arrives through httpclient, but \{{hdds-common}}
declares and uses it, so it stays.
h3. Behaviour
Checked \{{URIBuilder}} against \{{java.net.URI}} side by side (the bracketing
helper of step 1 is not built yet):
||authority||URIBuilder||java.net.URI 5-arg||
|\{{bucket.vol}}, \{{bucket.vol.om1}}, \{{omservice}}|same|same|
|\{{om1:9862}}, \{{bucket.vol.om1:9862}}, \{{10.0.0.1:9874}}|same|same|
|\{{[::1]:9862}}|\{{ofs://[::1]:9862}}|same|
|\{{::1}} (unbracketed)|\{{ofs://[::1]}}|\{{ofs://::1}}, *so bracket it (step
1)*|
|empty|\{{URISyntaxException}}|\{{URISyntaxException}}|
|\{{https}}, \{{om1:9874}} / \{{[::1]:9875}}, path, two parameters|same|same|
* \{{o3fs}} / \{{ofs}} \{{getUri()}} and the OM checkpoint URL stay
byte-for-byte the same.
* \{{OFSPath}} throws \{{IllegalArgumentException}} instead of
\{{org.apache.http.ParseException}}, with the same message.
h3. Compatibility
* No wire, proto, config or on-disk changes.
* Applications that depend on the unshaded \{{ozone-common}} or
\{{ozone-filesystem-common}} and use httpclient without declaring it lose it as
a transitive dependency. Inside Ozone, \{{s3gateway}}, \{{cli-shell}},
\{{vapor}} and the integration test modules all declare httpclient themselves.
h3. Testing
* *Unit:*
** \{{getUri()}} of both file systems for host, host:port, service id,
bracketed IPv6 with port, and an unbracketed IPv6 literal;
** \{{getOMDBCheckpointEndpointUrl}} for http and https, with \{{useV2}} and
\{{flush}} both ways, against the URL that \{{URIBuilder}} builds today;
** \{{OFSPath}} with a non-ofs scheme throws \{{IllegalArgumentException}} with
the old message.
* *Existing:* \{{TestOzoneFileSystem}}, \{{TestRootedOzoneFileSystem}},
\{{TestOFSPath}}, and the OM bootstrap and checkpoint tests.
* *Jar contents:* \{{ozone-filesystem-hadoop3}} has no
\{{org/apache/ozone/shaded/org/apache/http}} and no
\{{META-INF/maven/org.apache.httpcomponents}}. Record its size before and after.
> Replace URIBuilder and http ParseException with JDK classes to drop
> httpclient and httpcore from Ozone client jars
> ------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17778
> URL: https://issues.apache.org/jira/browse/HDDS-17778
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Client
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
>
> h3. Problem
> {{ozone-common}} and {{ozone-filesystem-common}} declare {{httpclient}}
> 4.5.14 and {{httpcore}} 4.4.16 as compile dependencies. The client uses them
> for only two classes, in four places:
> ||class||used in||for||
> |{Unknown macro: \{org.apache.http.client.utils.URIBuilder}
> }|{{{}BasicOzoneFileSystem#initialize{}}}, {
> Unknown macro: \{BasicRootedOzoneFileSystem#initialize}
> }|building {{o3fs://<authority>}} / {{ofs://<authority>}} from scheme and
> authority|
> |{Unknown macro: \{org.apache.http.client.utils.URIBuilder}
> }|{Unknown macro: \{OMNodeDetails#getOMDBCheckpointEndpointUrl}
> }|building the OM DB checkpoint URL with two constant boolean parameters|
> |{Unknown macro: \{org.apache.http.ParseException}
> }|{{{}OFSPath#initOFSPath{}}}, two \{{throw}}s|an unchecked exception for a
> non-ofs scheme, or when the temp bucket name can't be resolved|
> The shaded filesystem jar excludes {{{}hadoop-common{}}}, so these
> declarations are the only reason httpclient and httpcore are in it. Their
> relocated classes take *0.89 MB* (compressed) in
> {{{}ozone-filesystem-hadoop3{}}}:
> {noformat}
> ozone-filesystem-shaded
> \- ozone-filesystem-common:compile
> \- httpclient:4.5.14:compile
> \- httpcore:4.4.16:compile
> {noformat}
> That's small, but it's free to remove. The jar is shipped with every job that
> uses {{-libjars}} or {{{}spark.jars{}}}. It is 155 MB on master, 71 MB with
> HDDS-17772.
> h3. Proposed change
> # *Filesystem URIs:* replace {{new
> URIBuilder().setScheme(s).setHost(authority).build()}} with {{{}new URI(s,
> authority, null, null, null){}}}. Keep {{{}URIBuilder{}}}'s one extra
> behaviour: it brackets an unbracketed IPv6 literal (for {{{}ofs://::1/{}}},
> {{URI#getAuthority()}} returns {{{}::1{}}}). Bracket the authority when
> {{InetAddresses.isInetAddress(authority)}} and it contains {{{}:{}}}. A small
> helper in {{OzoneFSUtils}} serves both file systems.
> # *OM checkpoint URL:* replace it with {{{}new URI(scheme, address, path,
> query, null).toURL(){}}}. The query is
> {{{}includeSnapshotData=true&flushBeforeCheckpoint=<flag>{}}}. Both values
> are constants, so nothing needs encoding.
> # *{{{}OFSPath{}}}:* throw {{IllegalArgumentException}} with the same
> messages. It is unchecked, like {{{}org.apache.http.ParseException{}}}, and
> nothing in the code base catches the http {{{}ParseException{}}}. The three
> {{catch (ParseException ...)}} in {{{}TrashPolicyOzone{}}},
> {{S3ConditionalRequest}} and {{ReconUtils}} are
> {{{}java.text.ParseException{}}}.
> # *Build:*
> ** remove {{httpclient}} and {{httpcore}} from
> {{hadoop-ozone/common/pom.xml}} and
> {{{}hadoop-ozone/ozonefs-common/pom.xml{}}};
> ** add {{org.apache.httpcomponents}} to the banned dependencies of the
> client modules, next to the RocksDB rule from HDDS-17772.
> {{commons-codec}} also arrives through httpclient, but {{hdds-common}}
> declares and uses it, so it stays.
> h3. Behaviour
> Checked {{URIBuilder}} against {{java.net.URI}} side by side (the bracketing
> helper of step 1 is not built yet):
> ||authority||URIBuilder||java.net.URI 5-arg||
> |{{{}bucket.vol{}}}, {{{}bucket.vol.om1{}}}, {
> Unknown macro: \{omservice}
> }|same|same|
> |{{{}om1:9862{}}}, {{{}bucket.vol.om1:9862{}}}, {
> Unknown macro: \{10.0.0.1}
> }|same|same|
> |{Unknown macro: {[::1]}
> }|{Unknown macro: \{ofs}
> }|same|
> |{{::1}} (unbracketed)|{Unknown macro: \{ofs}
> }|{{{}ofs://::1{}}}, *so bracket it (step 1)*|
> |empty|{Unknown macro: \{URISyntaxException}
> }|{Unknown macro: \{URISyntaxException}
> }|
> |{{{}https{}}}, {{om1:9874}} / {{{}[::1]:9875{}}}, path, two
> parameters|same|same|
> * {{o3fs}} / {{ofs}} {{getUri()}} and the OM checkpoint URL stay
> byte-for-byte the same.
> * {{OFSPath}} throws {{IllegalArgumentException}} instead of
> {{{}org.apache.http.ParseException{}}}, with the same message.
> h3. Compatibility
> * No wire, proto, config or on-disk changes.
> * Applications that depend on the unshaded {{ozone-common}} or
> {{ozone-filesystem-common}} and use httpclient without declaring it lose it
> as a transitive dependency. Inside Ozone, {{{}s3gateway{}}},
> {{{}cli-shell{}}}, {{vapor}} and the integration test modules all declare
> httpclient themselves.
> h3. Testing
> * *Unit:*
> ** {{getUri()}} of both file systems for host, host:port, service id,
> bracketed IPv6 with port, and an unbracketed IPv6 literal;
> ** {{getOMDBCheckpointEndpointUrl}} for http and https, with {{useV2}} and
> {{flush}} both ways, against the URL that {{URIBuilder}} builds today;
> ** {{OFSPath}} with a non-ofs scheme throws {{IllegalArgumentException}}
> with the old message.
> * *Existing:* {{{}TestOzoneFileSystem{}}},
> {{{}TestRootedOzoneFileSystem{}}}, {{{}TestOFSPath{}}}, and the OM bootstrap
> and checkpoint tests.
> * *Jar contents:* {{ozone-filesystem-hadoop3}} has no
> {{org/apache/ozone/shaded/org/apache/http}} and no
> {{{}META-INF/maven/org.apache.httpcomponents{}}}. Record its size before and
> after.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]