mattcasters opened a new pull request, #8287: URL: https://github.com/apache/hop/pull/8287
Adds a marketplace HDFS VFS plugin so Hop pipelines (including Parquet File Output) can write to HDFS without shipping Hadoop client JARs. Parquet already writes through `HopVfs.getOutputStream()`. Matching Cloudera parcel libraries in Hop would fight the Hadoop 3.4.2 bits already on the classpath and would break on a 7.1.8 (Hadoop 3.1.1) → 7.3.2 (Hadoop 3.4.2) upgrade. This plugin talks the stable WebHDFS REST API instead. ## What changed - **`hop-tech-hadoop` marketplace plugin.** Named VFS connection (scheme is the connection name, e.g. `cdp:///warehouse/...`). Does **not** register a global `hdfs://` provider — Native Spark already uses that as a Hadoop FileSystem URI. - **Transports:** HttpFS (one gateway, recommended from OpenShift), Knox, WebHDFS (NameNode HTTP plus DataNode redirects). - **Kerberos:** JAAS keytab login, SPNEGO via the JDK GSS API, process-wide re-login at ~80% of TGT lifetime. No `kinit` binary, no `UserGroupInformation`. - **Streaming writes.** Piped HTTP PUT so Parquet is not buffered in heap. - **Docs** for the connection, VFS, and writing Parquet from OpenShift to CDP. ## Tests - 20 unit tests against an in-process WebHDFS stub (mkdir, list, HttpFS create, WebHDFS noredirect create, streaming close, path encoding, named-connection scheme). - Docker IT project `integration-tests/hadoop` (unsecured WebHDFS + text/Parquet). Not run against a live cluster in this change. ## OpenShift / CDP note From a remote container, use **HttpFS or Knox**. Raw WebHDFS and native HDFS RPC both need DataNode ports from the pod. Fixes #8285 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
