[
https://issues.apache.org/jira/browse/HUDI-3232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17480722#comment-17480722
]
sivabalan narayanan commented on HUDI-3232:
-------------------------------------------
this definitely looks beneficial.
may I know how you plan to get this done.
Do you plan to add some filtering in below method in HoodieTableMetaClient.
{code:java}
public List<HoodieInstant> scanHoodieInstantsFromFileSystem(Path timelinePath,
Set<String> includedExtensions,
boolean applyLayoutVersionFilters) throws IOException { {code}
something like,
HTMC (HoodieTableMetaClient) will maintain last loaded instant or mod time of
last loaded instant file. So, every time when reload is invoked, we just filter
for files modified after the last known mod time.
Or you had something else in mind ?
> support reload timeline Incrementally
> -------------------------------------
>
> Key: HUDI-3232
> URL: https://issues.apache.org/jira/browse/HUDI-3232
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Common Core, incremental-query, writer-core
> Reporter: Yann Byron
> Priority: Major
>
> Recently, call `HoodieTableMetaClient.reloadActiveTimeline` many times in one
> operation, and this will reload the timeline fully.
> Perhaps, to support to reload in Incremental mode will increase the
> performance.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
