yihua opened a new pull request #3921: URL: https://github.com/apache/hudi/pull/3921
Reverts apache/hudi#3769 The original PR apache/hudi#3769 causes the `java.io.FileNotFoundException` for the partition path upon the first commit to the MOR table with Hudi kafka-connect sink using Java client. ``` [2021-11-03 11:27:48,702] ERROR [hudi-sink|task-0] WorkerSinkTask{id=hudi-sink-0} RetriableException from SinkTask: (org.apache.kafka.connect.runtime.WorkerSinkTask:600) org.apache.kafka.connect.errors.RetriableException: Intermittent write errors for Hudi for the topic/partition: hudi-test-topic:0 , ensuring kafka connect will retry at org.apache.hudi.connect.HoodieSinkTask.put(HoodieSinkTask.java:117) at org.apache.kafka.connect.runtime.WorkerSinkTask.deliverMessages(WorkerSinkTask.java:581) at org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:329) at org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:232) at org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:201) at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:186) at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:241) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.hudi.exception.HoodieIOException: Error writing records and ending commit 20211103112647 for partition 0 at org.apache.hudi.connect.transaction.ConnectTransactionParticipant.handleEndCommit(ConnectTransactionParticipant.java:198) at org.apache.hudi.connect.transaction.ConnectTransactionParticipant.processRecords(ConnectTransactionParticipant.java:124) at org.apache.hudi.connect.HoodieSinkTask.put(HoodieSinkTask.java:114) ... 11 more Caused by: java.io.IOException: org.apache.hudi.exception.HoodieIOException: Write records failed ... 14 more Caused by: org.apache.hudi.exception.HoodieIOException: Write records failed at org.apache.hudi.connect.writers.BufferedConnectWriter.flushRecords(BufferedConnectWriter.java:125) at org.apache.hudi.connect.writers.AbstractConnectWriter.close(AbstractConnectWriter.java:95) at org.apache.hudi.connect.transaction.ConnectTransactionParticipant.handleEndCommit(ConnectTransactionParticipant.java:177) ... 13 more Caused by: java.io.IOException: org.apache.hudi.exception.HoodieUpsertException: Failed to upsert for commit time 20211103112647 ... 16 more Caused by: org.apache.hudi.exception.HoodieUpsertException: Failed to upsert for commit time 20211103112647 at org.apache.hudi.table.action.deltacommit.JavaUpsertPreppedDeltaCommitActionExecutor.execute(JavaUpsertPreppedDeltaCommitActionExecutor.java:96) at org.apache.hudi.table.HoodieJavaMergeOnReadTable.upsertPrepped(HoodieJavaMergeOnReadTable.java:50) at org.apache.hudi.table.HoodieJavaMergeOnReadTable.upsertPrepped(HoodieJavaMergeOnReadTable.java:40) at org.apache.hudi.client.HoodieJavaWriteClient.upsertPreppedRecords(HoodieJavaWriteClient.java:120) at org.apache.hudi.connect.writers.BufferedConnectWriter.flushRecords(BufferedConnectWriter.java:112) ... 15 more Caused by: org.apache.hudi.exception.HoodieIOException: Failed to list base files in partition partition-0 at org.apache.hudi.common.table.view.AbstractTableFileSystemView.lambda$ensurePartitionLoadedCorrectly$9(AbstractTableFileSystemView.java:300) at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660) at org.apache.hudi.common.table.view.AbstractTableFileSystemView.ensurePartitionLoadedCorrectly(AbstractTableFileSystemView.java:281) at org.apache.hudi.common.table.view.AbstractTableFileSystemView.getLatestFileSlice(AbstractTableFileSystemView.java:577) at org.apache.hudi.common.table.view.PriorityBasedFileSystemView.execute(PriorityBasedFileSystemView.java:101) at org.apache.hudi.common.table.view.PriorityBasedFileSystemView.getLatestFileSlice(PriorityBasedFileSystemView.java:252) at org.apache.hudi.io.HoodieAppendHandle.init(HoodieAppendHandle.java:135) at org.apache.hudi.io.HoodieAppendHandle.doAppend(HoodieAppendHandle.java:349) at org.apache.hudi.table.action.deltacommit.JavaUpsertPreppedDeltaCommitActionExecutor.lambda$execute$0(JavaUpsertPreppedDeltaCommitActionExecutor.java:83) at java.util.HashMap.forEach(HashMap.java:1289) at org.apache.hudi.table.action.deltacommit.JavaUpsertPreppedDeltaCommitActionExecutor.execute(JavaUpsertPreppedDeltaCommitActionExecutor.java:80) ... 19 more Caused by: java.io.FileNotFoundException: File file:/tmp/hoodie/hudi-test-topic/partition-0 does not exist at org.apache.hadoop.fs.RawLocalFileSystem.listStatus(RawLocalFileSystem.java:431) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1517) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1557) at org.apache.hadoop.fs.ChecksumFileSystem.listStatus(ChecksumFileSystem.java:674) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1517) at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1557) at org.apache.hudi.common.fs.FSUtils.getAllDataFilesInPartition(FSUtils.java:453) at org.apache.hudi.metadata.FileSystemBackedTableMetadata.getAllFilesInPartition(FileSystemBackedTableMetadata.java:62) at org.apache.hudi.metadata.BaseTableMetadata.getAllFilesInPartition(BaseTableMetadata.java:129) at org.apache.hudi.metadata.HoodieMetadataFileSystemView.listPartition(HoodieMetadataFileSystemView.java:65) at org.apache.hudi.common.table.view.AbstractTableFileSystemView.lambda$ensurePartitionLoadedCorrectly$9(AbstractTableFileSystemView.java:290) ... 29 more ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org