mattcasters opened a new pull request, #8487: URL: https://github.com/apache/hop/pull/8487
Fixes #8440 ## Root cause Writing several Parquet files in parallel over the Hadoop-free HDFS VFS driver (WebHDFS, HTTPS, Kerberos, TLS) stalled after a burst of rows. Stopping the pipeline left the Parquet File Output copies in **Halting** for minutes. This was a lock, not a slow DataNode. 1. `HdfsWebHdfsClient` wrapped every `httpClient.execute(...)` — including the streaming CREATE PUT body — in `HdfsKerberosSession.doAs()`. 2. `doAs()` held the process-wide `JVM_KERBEROS` lock for the **entire action**. One copy could keep that lock for the life of a Parquet file. 3. SPNEGO is not required on DataNode Location URLs, but the PUT still ran inside `doAs()`. 4. The other copies blocked in `HopVfs.getOutputStream()` → `exists()` / CREATE (`GETFILESTATUS`) waiting for the same lock. 5. Default rowset size is 10 000. The previous transform filled the blocked copies’ rowsets and could no longer feed the copy that still held the lock. That copy’s upload thread sat in `PipedInputStream.read()` **still holding `JVM_KERBEROS`**. Deadlock. 6. Stop → `HdfsStreamingOutputStream.close()` → unbounded `upload.get()` with no HTTP abort, which is the Halting status. There is no socket timeout while the client is still waiting for more body bytes. A 1 MiB pipe made the stall appear as soon as the first copy entered its PUT. Contributing risks on the same path: - HttpClient 5 pool defaults (5 per route, 25 total). HttpFS/Knox put every PUT and `GETFILESTATUS` on one host. Four long PUTs leave one slot for metadata; `connectionRequestTimeout` default is 3 minutes. - Expect-continue was not explicitly disabled. - Close never cancelled the in-flight `HttpPut`. ## Fixes - **Kerberos:** `doAs()` snapshots the Subject under `JVM_KERBEROS` and releases the lock before the action. New `gss()` holds the lock only for SPNEGO `initSecContext` / re-login. HTTP execute (including the file body) is no longer inside `doAs()`. - **Pool:** always install a pooling manager (HTTP and HTTPS): 64 per route, 128 total, 30s connection-request timeout, 30s connect, 300s socket/response timeout, expect-continue off, idle eviction, validate-after-inactivity. - **Streaming close:** 8 MiB pipe; `close()` waits up to 120s then cancels the `HttpPut` and the upload future. ## Tests `./mvnw -pl plugins/tech/hadoop test` - Four overlapping CREATEs against the in-process WebHDFS stub, including with a globally locked `doAs` (fails if execute is wrapped again). - `doAs` does not hold `JVM_KERBEROS` while the action runs. - Close times out and unblocks when the uploader never reads. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
