[ https://issues.apache.org/jira/browse/HUDI-5238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexey Kudinkin reassigned HUDI-5238: ------------------------------------- Assignee: Alexey Kudinkin > Hudi throwing "PipeBroken" exception during Merging on GCS > ---------------------------------------------------------- > > Key: HUDI-5238 > URL: https://issues.apache.org/jira/browse/HUDI-5238 > Project: Apache Hudi > Issue Type: Bug > Affects Versions: 0.12.1 > Reporter: Alexey Kudinkin > Assignee: Alexey Kudinkin > Priority: Blocker > Fix For: 0.13.0 > > > Originally reported at [https://github.com/apache/hudi/issues/7234] > ------- > > Root-cause: > Basically, the reason it’s failing is following: # GCS uses > PipeInputStream/PipeOutputStream comprising reading/writing ends of the > “pipe” it’s using for unidirectional comm b/w Threads > # PipeInputStream (for whatever reason) remembers the thread that actually > wrote into the pipe > # In BoundedInMemoryQueue we’re bootstrapping new executors (read, threads) > for reading and _writing_ (it’s only used in HoodieMergeHandle, and in > bulk-insert) > # When we’re done writing in HoodieMergeHelper, we’re shutting down *first* > BIMQ, then the HoodieMergeHandle, and that’s exactly the reason why it’s > failing > > Issue has been introduced at [https://github.com/apache/hudi/pull/4264/files] -- This message was sent by Atlassian Jira (v8.20.10#820010)