Yue Zhang created HUDI-2435: ------------------------------- Summary: Tuning clustering job handle errors Key: HUDI-2435 URL: https://issues.apache.org/jira/browse/HUDI-2435 Project: Apache Hudi Issue Type: Task Reporter: Yue Zhang
Before clustering job/ async clustering finished, hudi will perform errors check using JavaRDD<WriteStatus> writeResponse It is a collect spark action, when executor is crashed by accident and the cache of JavaRDD<WriteStatus> is lost, so that this collect action will trigger a complete compute and create unexpected marker files or data files. We should use Option<HoodieCommitMetadata> commitMetadata to do handle errors action instead. -- This message was sent by Atlassian Jira (v8.3.4#803005)