[
https://issues.apache.org/jira/browse/HUDI-5238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Kudinkin updated HUDI-5238:
----------------------------------
Description:
Originally reported at [https://github.com/apache/hudi/issues/7234]
-------
Root-cause:
Basically, the reason it’s failing is following: # GCS uses
PipeInputStream/PipeOutputStream comprising reading/writing ends of the “pipe”
it’s using for unidirectional comm b/w Threads
# PipeInputStream (for whatever reason) remembers the thread that actually
wrote into the pipe
# In BoundedInMemoryQueue we’re bootstrapping new executors (read, threads)
for reading and _writing_ (it’s only used in HoodieMergeHandle, and in
bulk-insert)
# When we’re done writing in HoodieMergeHelper, we’re shutting down *first*
BIMQ, then the HoodieMergeHandle, and that’s exactly the reason why it’s failing
Issue has been introduced at [https://github.com/apache/hudi/pull/4264/files]
> Hudi throwing "PipeBroken" exception during Merging on GCS
> ----------------------------------------------------------
>
> Key: HUDI-5238
> URL: https://issues.apache.org/jira/browse/HUDI-5238
> Project: Apache Hudi
> Issue Type: Bug
> Affects Versions: 0.12.1
> Reporter: Alexey Kudinkin
> Priority: Blocker
> Fix For: 0.13.0
>
>
> Originally reported at [https://github.com/apache/hudi/issues/7234]
> -------
>
> Root-cause:
> Basically, the reason it’s failing is following: # GCS uses
> PipeInputStream/PipeOutputStream comprising reading/writing ends of the
> “pipe” it’s using for unidirectional comm b/w Threads
> # PipeInputStream (for whatever reason) remembers the thread that actually
> wrote into the pipe
> # In BoundedInMemoryQueue we’re bootstrapping new executors (read, threads)
> for reading and _writing_ (it’s only used in HoodieMergeHandle, and in
> bulk-insert)
> # When we’re done writing in HoodieMergeHelper, we’re shutting down *first*
> BIMQ, then the HoodieMergeHandle, and that’s exactly the reason why it’s
> failing
>
> Issue has been introduced at [https://github.com/apache/hudi/pull/4264/files]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
