Re: [I] [SUPPORT]FileID of partition path xxx=xx does not exist. [hudi]
CaesarWangX commented on issue #11202: URL: https://github.com/apache/hudi/issues/11202#issuecomment-2121555896 Hi @ad1happy2go, Is there any progress？ Thank you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [SUPPORT]FileID of partition path xxx=xx does not exist. [hudi]
CaesarWangX commented on issue #11202: URL: https://github.com/apache/hudi/issues/11202#issuecomment-2113723533 Hi @ad1happy2go, yes, because we set "hoodie.cleaner.policy.failed.writes" = "NEVER", so it's not running rollback. And the reason we manually deleted the delta commit file is to reproduce the issue of field not found issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [SUPPORT]FileID of partition path xxx=xx does not exist. [hudi]
ad1happy2go commented on issue #11202: URL: https://github.com/apache/hudi/issues/11202#issuecomment-2112722424 @CaesarWangX Is this the last delta commit in your timeline? Why we are deleting the delta commit file manually? You are saying if you do that, it's not running rollback? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [SUPPORT]FileID of partition path xxx=xx does not exist. [hudi]
CaesarWangX commented on issue #11202: URL: https://github.com/apache/hudi/issues/11202#issuecomment-2109495556 @ad1happy2go you can follow the following step to reproduce it Steps to reproduce the behavior: Filegroup has only one data file. Delete the deltacommit corresponding to this filegroup (only delete deltacommit, keep request and inflight). New data will continue to be written to this filegroup (determined by getSmallFiles). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [SUPPORT]FileID of partition path xxx=xx does not exist. [hudi]
CaesarWangX commented on issue #11202: URL: https://github.com/apache/hudi/issues/11202#issuecomment-2109494369 @ad1happy2go Sure, here it is. "hoodie.datasource.write.table.type" = "MERGE_ON_READ" "hoodie.table.name" = "smart_event" "hoodie.datasource.write.recordkey.field" = "rowkey" "hoodie.datasource.write.operation" = "insert" "hoodie.datasource.write.hive_style_partitioning" = "true" "hoodie.datasource.hive_sync.partition_fields" = "dt" "hoodie.datasource.hive_sync.partition_extractor_class" = "org.apache.hudi.hive.MultiPartKeysValueExtractor" "hoodie.datasource.write.precombine.field" = "log_time" "hoodie.upsert.shuffle.parallelism" = "200" "hoodie.insert.shuffle.parallelism" = "200" "hoodie.parquet.small.file.limit" = "441600" "hoodie.parquet.max.file.size" = "829120" "hoodie.index.type" = "BLOOM" "hoodie.bloom.index.prune.by.ranges" = "true" "hoodie.bloom.index.use.caching" = "true" "hoodie.bloom.index.use.treebased.filter" = "true" "hoodie.bloom.index.bucketized.checking" = "true" "hoodie.bloom.index.filter.type" = "DYNAMIC_V0" "hoodie.cleaner.policy" = "KEEP_LATEST_COMMITS" "hoodie.clean.automatic" = "true" "hoodie.clean.async" = "true" "hoodie.keep.min.commits" = "5" "hoodie.keep.max.commits" = "7" "hoodie.cleaner.parallelism" = "200" "hoodie.embed.timeline.server" = "false" "hoodie.datasource.compaction.async.enable" = "true" "hoodie.cleaner.commits.retained" = "4" "hoodie.copyonwrite.insert.auto.split" = "true" "hoodie.merge.allow.duplicate.on.inserts" = "true" "hoodie.metadata.enable" = "false" "hoodie.cleaner.policy.failed.writes" = "NEVER" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [SUPPORT]FileID of partition path xxx=xx does not exist. [hudi]
ad1happy2go commented on issue #11202: URL: https://github.com/apache/hudi/issues/11202#issuecomment-2109478367 @CaesarWangX Can you also post writer configuration please -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [SUPPORT]FileID of partition path xxx=xx does not exist. [hudi]
CaesarWangX commented on issue #11202: URL: https://github.com/apache/hudi/issues/11202#issuecomment-2109455182 @ad1happy2go @danny0405 I'm sorry, I have re uploaded a new trace log -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [SUPPORT]FileID of partition path xxx=xx does not exist. [hudi]
ad1happy2go commented on issue #11202: URL: https://github.com/apache/hudi/issues/11202#issuecomment-2109436536 @CaesarWangX That's interesting. All confusion is as in your stack trace it is pointing to "FlinkMergeHandle" code which ideally spark streaming should not use. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [SUPPORT]FileID of partition path xxx=xx does not exist. [hudi]
danny0405 commented on issue #11202: URL: https://github.com/apache/hudi/issues/11202#issuecomment-2109224081 ```java Caused by: java.util.NoSuchElementException: FileID x of partition path dt=2019-02-20 does not exist. at org.apache.hudi.io.HoodieMergeHandle.getLatestBaseFile(HoodieMergeHandle.java:159) at org.apache.hudi.io.HoodieMergeHandle.(HoodieMergeHandle.java:121) at org.apache.hudi.io.FlinkMergeHandle.(FlinkMergeHandle.java:70) at org.apache.hudi.io.FlinkConcatHandle.(FlinkConcatHandle.java:53) at org.apache.hudi.client.HoodieFlinkWriteClient.getOrCreateWriteHandle(HoodieFlinkWriteClient.java:557) at org.apache.hudi.client.HoodieFlinkWriteClient.insert(HoodieFlinkWriteClient.java:175) at org.apache.hudi.sink.StreamWriteFunction.lambda$initWriteFunction$0(StreamWriteFunction.java:181) at org.apache.hudi.sink.StreamWriteFunction.lambda$flushRemaining$7(StreamWriteFunction.java:461) ``` The error msg indicates that you enabled the inline clustering for Flink, can you disable that and try again by using the async clustering instead. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [SUPPORT]FileID of partition path xxx=xx does not exist. [hudi]
CaesarWangX commented on issue #11202: URL: https://github.com/apache/hudi/issues/11202#issuecomment-2109198672 The reason we do not use metadata table is that in spark structured streaming, enabling the metadata table will affect the efficiency of micro batch, as there will be additional list operations. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [SUPPORT]FileID of partition path xxx=xx does not exist. [hudi]
CaesarWangX commented on issue #11202: URL: https://github.com/apache/hudi/issues/11202#issuecomment-2109155774 Hi @danny0405 @xushiyan , We are using spark3.4.1 and hudi0.14.0. Updated the context and please help look into this. Thank you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [SUPPORT]FileID of partition path xxx=xx does not exist. [hudi]
CaesarWangX commented on issue #11202: URL: https://github.com/apache/hudi/issues/11202#issuecomment-2109118458 Hi @danny0405 , we don't need the metadata table, so as i mentioned, we set metadata.enable=false. We are using hudi in AWS EMR, so we don't have chance to use hudi0.14.1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [I] [SUPPORT]FileID of partition path xxx=xx does not exist. [hudi]
danny0405 commented on issue #11202: URL: https://github.com/apache/hudi/issues/11202#issuecomment-2109071887 Did you use Hudi 0.14.0 release? Did you enable the metadata table? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[I] [SUPPORT]FileID of partition path xxx=xx does not exist. [hudi]
CaesarWangX opened a new issue, #11202: URL: https://github.com/apache/hudi/issues/11202 **_Tips before filing an issue_** - Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)? - Join the mailing list to engage in conversations and get faster support at dev-subscr...@hudi.apache.org. - If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly. **Describe the problem you faced** After upgrading from Hudi 0.11 to Hudi 0.14, we occasionally encounter this error. Once it occurs, the task continues to fail even after restarting, and the error message remains the same with the FileID. I have reviewed the following two issues, but unfortunately, there is no effective solution available. https://github.com/apache/hudi/issues/5886 https://github.com/apache/hudi/issues/8890 Fortunately, by examining the source code, I have successfully replicated this issue and attempted to modify the source code to prevent task failures. Any information or suggestions are helpful, thank you. **To Reproduce** Steps to reproduce the behavior: 1. Filegroup has only one data file. 2. Delete the deltacommit corresponding to this filegroup (only delete deltacommit, keep request and inflight). 3. New data will continue to be written to this filegroup (determined by getSmallFiles). BWT, Under normal circumstances, the task fails due to this reason when these three conditions are coincidentally met at the same time. How challenging it is indeed! Therefore, reproducing this error is quite difficult. Essentially, it occurs when the job fails after completing data writing to the filegroup but before committing the deltacommit. **Expected behavior** A clear and concise description of what you expected to happen. **Environment Description** * Hudi version : 0.14.0, MOR, INSERT * Spark version : 3.4.1 * Storage (HDFS/S3/GCS..) : * Running on Docker? (yes/no) : **Additional context** **Not finished yet, will provide more information later.** After examining the code, the key difference between Hudi 0.11 and 0.14 is as follows: **AbstractTableFileSystemView** Hudi0.11: https://github.com/apache/hudi/assets/12985552/78f519f6-b77f-4798-ac03-69647a374a7b;> https://github.com/apache/hudi/assets/12985552/adf624b0-1901-4cba-b61d-baed3725f590;> Hudi0.14: **Stacktrace** Caused by: java.util.NoSuchElementException: FileID x of partition path dt=2019-02-20 does not exist. at org.apache.hudi.io.HoodieMergeHandle.getLatestBaseFile(HoodieMergeHandle.java:159) at org.apache.hudi.io.HoodieMergeHandle.(HoodieMergeHandle.java:121) at org.apache.hudi.io.FlinkMergeHandle.(FlinkMergeHandle.java:70) at org.apache.hudi.io.FlinkConcatHandle.(FlinkConcatHandle.java:53) at org.apache.hudi.client.HoodieFlinkWriteClient.getOrCreateWriteHandle(HoodieFlinkWriteClient.java:557) at org.apache.hudi.client.HoodieFlinkWriteClient.insert(HoodieFlinkWriteClient.java:175) at org.apache.hudi.sink.StreamWriteFunction.lambda$initWriteFunction$0(StreamWriteFunction.java:181) at org.apache.hudi.sink.StreamWriteFunction.lambda$flushRemaining$7(StreamWriteFunction.java:461) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
