[
https://issues.apache.org/jira/browse/HUDI-6802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Y Ethan Guo closed HUDI-6802.
-----------------------------
Resolution: Fixed
> Use completion time in Spark FileIndex for listing
> --------------------------------------------------
>
> Key: HUDI-6802
> URL: https://issues.apache.org/jira/browse/HUDI-6802
> Project: Apache Hudi
> Issue Type: New Feature
> Reporter: Ethan Guo (this is the old account; please use "yihua")
> Assignee: Jonathan Vexler
> Priority: Blocker
> Fix For: 1.0.0
>
> Original Estimate: 4h
> Remaining Estimate: 4h
>
> We need to make sure that the file index (BaseHoodieTableFileIndex,
> SparkHoodieTableFileIndex, HoodieFileIndex, subclasses, etc.) should
> incorporate completion time when listing the files for different query types
> (snapshot, read-optimized, incremental, CDC) in Spark. This should already
> be supported as the completion-time-based listing is done in the file system
> view. Still, we should revisit the code to make sure there is no gap on the
> Spark side.
> The log file ordering will be tackled separately. This ticket makes sure
> that we return the accurate collection of log files based on the completion
> time, especially when NBCC is enabled and there are concurrent writers
> updating the same file group.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
