yandrey321 opened a new pull request, #12012:
URL: https://github.com/apache/ozone/pull/12012
## What changes were proposed in this pull request?
**Problem.** `ozone-common` and `ozone-filesystem-common` declare
`httpclient` and `httpcore` only for two classes, used in four places:
- `URIBuilder` builds the `o3fs://` / `ofs://` URI in `BasicOzoneFileSystem`
and `BasicRootedOzoneFileSystem`, and
the OM DB checkpoint URL in `OMNodeDetails`;
- `org.apache.http.ParseException` is thrown twice in `OFSPath`.
Those declarations are why the shaded filesystem jar bundles both libraries:
773 entries, about 0.97 MB compressed. The
jar is shipped with every job that uses `-libjars` or `spark.jars`.
**Implementation idea.** Use JDK classes instead, then remove the two
dependency declarations:
- **Filesystem URIs:** `new URI(scheme + "://" + authority)`. That is the
string `URIBuilder` builds and parses, so
the result is the same, including the `URISyntaxException` for an illegal
authority.
- **Checkpoint URL:** the 5-argument `URI` constructor, with the query built
from two constant keys and booleans.
- **`OFSPath`:** throws `IllegalArgumentException` with the same messages.
- **Guard:** a `ban-httpcomponents` enforcer rule in
`ozone-filesystem-shaded`.
**Key decisions.**
- **No IPv6 bracketing for the filesystem URIs.** `URIBuilder` brackets an
unbracketed IPv6 literal, and the JDK
parse doesn't. Both file systems reject unbracketed IPv6 authorities
earlier in `initialize`, so the input never
gets that far.
- **String parse rather than the 5-argument constructor for the filesystem
URIs.** `getAuthority()` returns the
decoded authority. The multi-argument constructors would quote it, so
`ofs://om%20x/` would be accepted instead of
failing with "Invalid Ozone endpoint", and `ofs://om%2Fx/` would come out
differently.
- **The ban is enabled only in `ozone-filesystem-shaded`.** The unshaded
client modules still get `httpclient` from
`hadoop-common` / `hadoop-hdfs-client`, so a ban there would fail the
build. The shaded module catches both ways
back: a re-added declaration, and code that uses httpclient, because
dependency analysis would then require a
declaration.
**Side effects.**
- `OFSPath` throws `IllegalArgumentException` instead of
`org.apache.http.ParseException`. Both are unchecked, and
nothing catches the http one. The `catch (ParseException ...)` blocks in
`TrashPolicyOzone`, `S3ConditionalRequest`
and `ReconUtils` are `java.text.ParseException`.
- Applications that depend on the unshaded `ozone-common` /
`ozone-filesystem-common` artifacts no longer get httpclient
from them; they still get it from Hadoop. All Ozone modules that use
httpclient (`s3gateway`, `cli-shell`, `vapor`,
integration tests) declare it themselves.
- No wire, proto, config or on-disk changes.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-17778
## How was this patch tested?
Unit + integration tests.
**Jar contents:** `ozone-filesystem-hadoop3` no longer has
`org/apache/ozone/shaded/org/apache/http`,
`mozilla/public-suffix-list.txt` or
`META-INF/maven/org.apache.httpcomponents`. Nothing else changed apart from
that. It is now 69.2 MB on disk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]