[jira] [Updated] (HUDI-2892) Pending Clustering may stain the ActiveTimeLine and lead to incomplete query results
[ https://issues.apache.org/jira/browse/HUDI-2892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2892: -- Fix Version/s: 0.10.1 > Pending Clustering may stain the ActiveTimeLine and lead to incomplete query > results > > > Key: HUDI-2892 > URL: https://issues.apache.org/jira/browse/HUDI-2892 > Project: Apache Hudi > Issue Type: Bug >Reporter: Yue Zhang >Assignee: Yue Zhang >Priority: Major > Labels: pull-request-available > Fix For: 0.11.0, 0.10.1 > > > details could find in https://github.com/apache/hudi/issues/4163 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-2892) Pending Clustering may stain the ActiveTimeLine and lead to incomplete query results
[ https://issues.apache.org/jira/browse/HUDI-2892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-2892: -- Fix Version/s: 0.11.0 > Pending Clustering may stain the ActiveTimeLine and lead to incomplete query > results > > > Key: HUDI-2892 > URL: https://issues.apache.org/jira/browse/HUDI-2892 > Project: Apache Hudi > Issue Type: Bug >Reporter: Yue Zhang >Priority: Major > Labels: pull-request-available > Fix For: 0.11.0 > > > details could find in https://github.com/apache/hudi/issues/4163 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-2892) Pending Clustering may stain the ActiveTimeLine and lead to incomplete query results
[ https://issues.apache.org/jira/browse/HUDI-2892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-2892: - Labels: pull-request-available (was: ) > Pending Clustering may stain the ActiveTimeLine and lead to incomplete query > results > > > Key: HUDI-2892 > URL: https://issues.apache.org/jira/browse/HUDI-2892 > Project: Apache Hudi > Issue Type: Bug >Reporter: Yue Zhang >Priority: Major > Labels: pull-request-available > > details could find in https://github.com/apache/hudi/issues/4163 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-2892) Pending Clustering may stain the ActiveTimeLine and lead to incomplete query results
[ https://issues.apache.org/jira/browse/HUDI-2892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yue Zhang updated HUDI-2892: Description: details could find in https://github.com/apache/hudi/issues/4163 (was: Step 1 Do a normal hudi insert drwxr-xr-x 3 yuezhang FREEWHEELMEDIA\Domain Users 96 11 30 11:39 .aux/ drwxr-xr-x 2 yuezhang FREEWHEELMEDIA\Domain Users 64 11 30 11:39 .temp/ -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 5485 11 30 11:39 20211130113918979.commit -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:39 20211130113918979.commit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:39 20211130113918979.inflight drwxr-xr-x 2 yuezhang FREEWHEELMEDIA\Domain Users 64 11 30 11:39 archived/ -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 553 11 30 11:39 hoodie.properties Step 2 Build a clustering plan but don't execute this plan 20211130114103632.replacecommit.requested will cluster data files from 20211130113918979.commit drwxr-xr-x 3 yuezhang FREEWHEELMEDIA\Domain Users 96 11 30 11:39 .aux/ drwxr-xr-x 2 yuezhang FREEWHEELMEDIA\Domain Users 64 11 30 11:39 .temp/ -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 5485 11 30 11:39 20211130113918979.commit -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:39 20211130113918979.commit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:39 20211130113918979.inflight -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 2976 11 30 11:41 20211130114103632.replacecommit.requested drwxr-xr-x 2 yuezhang FREEWHEELMEDIA\Domain Users 64 11 30 11:39 archived/ -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 553 11 30 11:39 hoodie.properties Step 3 Do a few times hudi insert and trigger several archivals drwxr-xr-x 3 yuezhang FREEWHEELMEDIA\Domain Users 96 11 30 11:39 .aux/ drwxr-xr-x 2 yuezhang FREEWHEELMEDIA\Domain Users 64 11 30 11:44 .temp/ -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 5485 11 30 11:39 20211130113918979.commit -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:39 20211130113918979.commit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:39 20211130113918979.inflight -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 2976 11 30 11:41 20211130114103632.replacecommit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 5485 11 30 11:41 20211130114122881.commit -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:41 20211130114122881.commit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:41 20211130114122881.inflight -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 5485 11 30 11:42 20211130114207164.commit -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:42 20211130114207164.commit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:42 20211130114207164.inflight -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 5485 11 30 11:44 20211130114351703.commit -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:43 20211130114351703.commit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:43 20211130114351703.inflight drwxr-xr-x 2 yuezhang FREEWHEELMEDIA\Domain Users 64 11 30 11:39 archived/ -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 553 11 30 11:39 hoodie.properties drwxr-xr-x 3 yuezhang FREEWHEELMEDIA\Domain Users 96 11 30 13:17 .aux/ drwxr-xr-x 2 yuezhang FREEWHEELMEDIA\Domain Users 64 11 30 13:23 .temp/ -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 2976 11 30 13:17 20211130114103632.replacecommit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 5485 11 30 13:18 20211130131825336.commit -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 13:18 20211130131825336.commit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 13:18 20211130131825336.inflight -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 5485 11 30 13:23 20211130132256488.commit -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 13:22 20211130132256488.commit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 13:22 20211130132256488.inflight -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 5485 11 30 13:23 20211130132327154.commit -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 13:23 20211130132327154.commit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 13:23 20211130132327154.inflight drwxr-xr-x 6 yuezhang FREEWHEELMEDIA\Domain Users 192 11 30 13:23 archived/ -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 553 11 30 13:17 hoodie.properties 20211130114122881.commit 20211130114207164.commit and 20211130114351703.commit
[jira] [Updated] (HUDI-2892) Pending Clustering may stain the ActiveTimeLine and lead to incomplete query results
[ https://issues.apache.org/jira/browse/HUDI-2892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yue Zhang updated HUDI-2892: Description: Step 1 Do a normal hudi insert drwxr-xr-x 3 yuezhang FREEWHEELMEDIA\Domain Users 96 11 30 11:39 .aux/ drwxr-xr-x 2 yuezhang FREEWHEELMEDIA\Domain Users 64 11 30 11:39 .temp/ -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 5485 11 30 11:39 20211130113918979.commit -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:39 20211130113918979.commit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:39 20211130113918979.inflight drwxr-xr-x 2 yuezhang FREEWHEELMEDIA\Domain Users 64 11 30 11:39 archived/ -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 553 11 30 11:39 hoodie.properties Step 2 Build a clustering plan but don't execute this plan 20211130114103632.replacecommit.requested will cluster data files from 20211130113918979.commit drwxr-xr-x 3 yuezhang FREEWHEELMEDIA\Domain Users 96 11 30 11:39 .aux/ drwxr-xr-x 2 yuezhang FREEWHEELMEDIA\Domain Users 64 11 30 11:39 .temp/ -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 5485 11 30 11:39 20211130113918979.commit -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:39 20211130113918979.commit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:39 20211130113918979.inflight -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 2976 11 30 11:41 20211130114103632.replacecommit.requested drwxr-xr-x 2 yuezhang FREEWHEELMEDIA\Domain Users 64 11 30 11:39 archived/ -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 553 11 30 11:39 hoodie.properties Step 3 Do a few times hudi insert and trigger several archivals drwxr-xr-x 3 yuezhang FREEWHEELMEDIA\Domain Users 96 11 30 11:39 .aux/ drwxr-xr-x 2 yuezhang FREEWHEELMEDIA\Domain Users 64 11 30 11:44 .temp/ -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 5485 11 30 11:39 20211130113918979.commit -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:39 20211130113918979.commit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:39 20211130113918979.inflight -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 2976 11 30 11:41 20211130114103632.replacecommit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 5485 11 30 11:41 20211130114122881.commit -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:41 20211130114122881.commit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:41 20211130114122881.inflight -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 5485 11 30 11:42 20211130114207164.commit -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:42 20211130114207164.commit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:42 20211130114207164.inflight -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 5485 11 30 11:44 20211130114351703.commit -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:43 20211130114351703.commit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:43 20211130114351703.inflight drwxr-xr-x 2 yuezhang FREEWHEELMEDIA\Domain Users 64 11 30 11:39 archived/ -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 553 11 30 11:39 hoodie.properties drwxr-xr-x 3 yuezhang FREEWHEELMEDIA\Domain Users 96 11 30 13:17 .aux/ drwxr-xr-x 2 yuezhang FREEWHEELMEDIA\Domain Users 64 11 30 13:23 .temp/ -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 2976 11 30 13:17 20211130114103632.replacecommit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 5485 11 30 13:18 20211130131825336.commit -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 13:18 20211130131825336.commit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 13:18 20211130131825336.inflight -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 5485 11 30 13:23 20211130132256488.commit -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 13:22 20211130132256488.commit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 13:22 20211130132256488.inflight -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 5485 11 30 13:23 20211130132327154.commit -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 13:23 20211130132327154.commit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 13:23 20211130132327154.inflight drwxr-xr-x 6 yuezhang FREEWHEELMEDIA\Domain Users 192 11 30 13:23 archived/ -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 553 11 30 13:17 hoodie.properties 20211130114122881.commit 20211130114207164.commit and 20211130114351703.commit were archived. Step 4 Do query to check record numbers and based hudi
[jira] [Updated] (HUDI-2892) Pending Clustering may stain the ActiveTimeLine and lead to incomplete query results
[ https://issues.apache.org/jira/browse/HUDI-2892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yue Zhang updated HUDI-2892: Description: **Describe the problem you faced** If there's a pending clustering instant still existed in active timeline after several archival actions. Next time we finish this pending clustering instant, this clustering instant may stain the ActiveTimeLine and lead to incomplete query results **To Reproduce** **Step 1** Do a normal hudi insert ``` drwxr-xr-x 3 yuezhang FREEWHEELMEDIA\Domain Users 96 11 30 11:39 .aux/ drwxr-xr-x 2 yuezhang FREEWHEELMEDIA\Domain Users 64 11 30 11:39 .temp/ -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 5485 11 30 11:39 20211130113918979.commit -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:39 20211130113918979.commit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:39 20211130113918979.inflight drwxr-xr-x 2 yuezhang FREEWHEELMEDIA\Domain Users 64 11 30 11:39 archived/ -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 553 11 30 11:39 hoodie.properties ``` **Step 2** Build a clustering plan but don't execute this plan 20211130114103632.replacecommit.requested will cluster data files from 20211130113918979.commit ``` drwxr-xr-x 3 yuezhang FREEWHEELMEDIA\Domain Users 96 11 30 11:39 .aux/ drwxr-xr-x 2 yuezhang FREEWHEELMEDIA\Domain Users 64 11 30 11:39 .temp/ -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 5485 11 30 11:39 20211130113918979.commit -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:39 20211130113918979.commit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:39 20211130113918979.inflight -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 2976 11 30 11:41 20211130114103632.replacecommit.requested drwxr-xr-x 2 yuezhang FREEWHEELMEDIA\Domain Users 64 11 30 11:39 archived/ -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 553 11 30 11:39 hoodie.properties ``` **Step 3** Do a few times hudi insert and trigger several archivals ``` drwxr-xr-x 3 yuezhang FREEWHEELMEDIA\Domain Users 96 11 30 11:39 .aux/ drwxr-xr-x 2 yuezhang FREEWHEELMEDIA\Domain Users 64 11 30 11:44 .temp/ -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 5485 11 30 11:39 20211130113918979.commit -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:39 20211130113918979.commit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:39 20211130113918979.inflight -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 2976 11 30 11:41 20211130114103632.replacecommit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 5485 11 30 11:41 20211130114122881.commit -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:41 20211130114122881.commit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:41 20211130114122881.inflight -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 5485 11 30 11:42 20211130114207164.commit -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:42 20211130114207164.commit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:42 20211130114207164.inflight -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 5485 11 30 11:44 20211130114351703.commit -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:43 20211130114351703.commit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 11:43 20211130114351703.inflight drwxr-xr-x 2 yuezhang FREEWHEELMEDIA\Domain Users 64 11 30 11:39 archived/ -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 553 11 30 11:39 hoodie.properties ``` ``` drwxr-xr-x 3 yuezhang FREEWHEELMEDIA\Domain Users 96 11 30 13:17 .aux/ drwxr-xr-x 2 yuezhang FREEWHEELMEDIA\Domain Users 64 11 30 13:23 .temp/ -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 2976 11 30 13:17 20211130114103632.replacecommit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 5485 11 30 13:18 20211130131825336.commit -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 13:18 20211130131825336.commit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 13:18 20211130131825336.inflight -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 5485 11 30 13:23 20211130132256488.commit -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 13:22 20211130132256488.commit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 13:22 20211130132256488.inflight -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 5485 11 30 13:23 20211130132327154.commit -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 13:23 20211130132327154.commit.requested -rw-r--r-- 1 yuezhang FREEWHEELMEDIA\Domain Users 0 11 30 13:23
