[
https://issues.apache.org/jira/browse/HUDI-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen updated HUDI-1623:
-----------------------------
Description:
We suggest a new file format naming for the *completed* metadata file:
${start_time}.${action}.${completion_time}
We also need a global *Time Generator* that can ensure the monotonical
increasing generation of the timestamp, for example, maybe hold a mutex lock
with the last generated timestamp backing up there. Say it may holds a lock
{*}L1{*}. For each instant time generation, it needs guard from the lock.
Before creating the completed file, we also need a lock guard from L1.
Something to note:
1. we only add completion timestamp to the completed metadata file;
2. we only add lock guard to the completed metadata file creation, not the
whole commiting procedure;
3. for normal instant time generation, we also need a lock.
was:
We suggest a new file format naming for the *completed* metadata file:
${start_time}.${action}.${completion_time}
We also need a global *Time Generator* that can ensure the monotonical **
generation of the timestamp, for example, maybe hold a mutex lock with the last
generated timestamp backing up. Say it may holds a lock {*}L1{*}. For each
instant time generation, it needs guard from the lock.
Before creating the completed file, we also need a lock guard from L1.
Something to note:
1. we only add completion timestamp to the completed metadata file;
2. we only add lock guard to the completed metadata file creation, not the
whole commiting procedure;
3. for normal instant time generation, we also need a lock.
> Support start_commit_time & end_commit_times for serializable incremental pull
> ------------------------------------------------------------------------------
>
> Key: HUDI-1623
> URL: https://issues.apache.org/jira/browse/HUDI-1623
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Common Core
> Reporter: Nishith Agarwal
> Assignee: Danny Chen
> Priority: Critical
> Fix For: 1.0.0
>
>
> We suggest a new file format naming for the *completed* metadata file:
> ${start_time}.${action}.${completion_time}
>
> We also need a global *Time Generator* that can ensure the monotonical
> increasing generation of the timestamp, for example, maybe hold a mutex lock
> with the last generated timestamp backing up there. Say it may holds a lock
> {*}L1{*}. For each instant time generation, it needs guard from the lock.
>
> Before creating the completed file, we also need a lock guard from L1.
>
> Something to note:
> 1. we only add completion timestamp to the completed metadata file;
> 2. we only add lock guard to the completed metadata file creation, not the
> whole commiting procedure;
> 3. for normal instant time generation, we also need a lock.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
