[ https://issues.apache.org/jira/browse/HUDI-2648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raymond Xu updated HUDI-2648: ----------------------------- Component/s: writer-core > Retry FileSystem action when caught runtime exception > ------------------------------------------------------ > > Key: HUDI-2648 > URL: https://issues.apache.org/jira/browse/HUDI-2648 > Project: Apache Hudi > Issue Type: Improvement > Components: writer-core > Reporter: Yue Zhang > Assignee: Yue Zhang > Priority: Major > Labels: pull-request-available > Fix For: 0.11.0 > > > Hoodie will do lots of list/get/put/delete etc actions on filesystem. > Sometimes will meet the fileSystem performance issue or short service > suspension, for example > Using hoodie on S3, and will throw lots of exception like thisĀ > {code:java} > com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 503, AWS > Service: Amazon S3, AWS Request ID: J58R9W4E324DWJ1Y, AWS Error Code: > SlowDown, AWS Error Message: Please reduce your request rate., S3 Extended > Request ID: > Pkd6VCJKTuq3NJ6vUQJB1TUznKGvAfjIZkmJOxt+j6oAD5roz+ojKf9rtcBOgUBw4sWeDwxdTv4=com.amazonaws.services.s3.model.AmazonS3Exception: > Status Code: 503, AWS Service: Amazon S3, AWS Request ID: J58R9W4E324DWJ1Y, > AWS Error Code: SlowDown, AWS Error Message: Please reduce your request > rate., S3 Extended Request ID: > Pkd6VCJKTuq3NJ6vUQJB1TUznKGvAfjIZkmJOxt+j6oAD5roz+ojKf9rtcBOgUBw4sWeDwxdTv4= > at > com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:798) > at > com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:421) > at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232) at > com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3528) at > com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1393) > at > org.apache.hadoop.fs.s3a.S3AFileSystem.createEmptyObject(S3AFileSystem.java:1191) > at > org.apache.hadoop.fs.s3a.S3AFileSystem.createFakeDirectory(S3AFileSystem.java:1168) > at org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:871) at > org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1881) at > org.apache.hudi.common.fs.HoodieWrapperFileSystem.lambda$mkdirs$25(HoodieWrapperFileSystem.java:632) > at > org.apache.hudi.common.fs.HoodieWrapperFileSystem.executeFuncWithTimeMetrics(HoodieWrapperFileSystem.java:100) > at > org.apache.hudi.common.fs.HoodieWrapperFileSystem.mkdirs(HoodieWrapperFileSystem.java:631) > at org.apache.hudi.table.MarkerFiles.create(MarkerFiles.java:195) at > org.apache.hudi.io.HoodieWriteHandle.createMarkerFile(HoodieWriteHandle.java:129) > at > org.apache.hudi.io.HoodieCreateHandle.lambda$new$1(HoodieCreateHandle.java:97) > at org.apache.hudi.common.util.RetryHelper.start(RetryHelper.java:61) at > org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:100) at > org.apache.hudi.io.HoodieCreateHandle.<init>(HoodieCreateHandle.java:67) at > org.apache.hudi.io.CreateHandleFactory.create(CreateHandleFactory.java:34) at > org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:83) > at > org.apache.hudi.execution.CopyOnWriteInsertHandler.consumeOneRecord(CopyOnWriteInsertHandler.java:40) > at > org.apache.hudi.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:37) > at > org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:121) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > {code} > It's because of meet the throttle of S3 list/get/put/delete. > This exception will cause app crashed, which can be solved by retry wrapper. -- This message was sent by Atlassian Jira (v8.20.1#820001)