[ https://issues.apache.org/jira/browse/HUDI-5238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raymond Xu updated HUDI-5238: ----------------------------- Sprint: 2022/11/15, 2022/11/29, 2022/12/12, 0.13.0 Final Sprint, 0.13.0 Final Sprint 2, 0.13.0 Final Sprint 3, Sprint 2023-01-31, Sprint 2023-02-14 (was: 2022/11/15, 2022/11/29, 2022/12/12, 0.13.0 Final Sprint, 0.13.0 Final Sprint 2, 0.13.0 Final Sprint 3, Sprint 2023-01-31) > Hudi throwing "PipeBroken" exception during Merging on GCS > ---------------------------------------------------------- > > Key: HUDI-5238 > URL: https://issues.apache.org/jira/browse/HUDI-5238 > Project: Apache Hudi > Issue Type: Bug > Affects Versions: 0.12.1 > Reporter: Alexey Kudinkin > Assignee: Alexey Kudinkin > Priority: Critical > Labels: pull-request-available > Fix For: 0.14.0 > > > Originally reported at [https://github.com/apache/hudi/issues/7234] > ------- > > Root-cause: > Basically, the reason it’s failing is following: # GCS uses > PipeInputStream/PipeOutputStream comprising reading/writing ends of the > “pipe” it’s using for unidirectional comm b/w Threads > # PipeInputStream (for whatever reason) remembers the thread that actually > wrote into the pipe > # In BoundedInMemoryQueue we’re bootstrapping new executors (read, threads) > for reading and _writing_ (it’s only used in HoodieMergeHandle, and in > bulk-insert) > # When we’re done writing in HoodieMergeHelper, we’re shutting down *first* > BIMQ, then the HoodieMergeHandle, and that’s exactly the reason why it’s > failing > > Issue has been introduced at [https://github.com/apache/hudi/pull/4264/files] -- This message was sent by Atlassian Jira (v8.20.10#820010)