[jira] [Updated] (HUDI-3026) HoodieAppendhandle may result in duplicate key for hbase index
[ https://issues.apache.org/jira/browse/HUDI-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-3026: Component/s: compaction table-service > HoodieAppendhandle may result in duplicate key for hbase index > -- > > Key: HUDI-3026 > URL: https://issues.apache.org/jira/browse/HUDI-3026 > Project: Apache Hudi > Issue Type: Bug > Components: compaction, table-service >Reporter: ZiyueGuan >Assignee: ZiyueGuan >Priority: Critical > Labels: pull-request-available > Original Estimate: 1h > Remaining Estimate: 1h > > Problem: a same key may occur in two file group when Hbase index is used. > These two file group will have same FileID prefix. As Hbase index is global, > this is unexpected > How to repro: > We should have a table w/o record sorted in spark. Let's say we have five > records with key 1,2,3,4,5 to write. They may be iterated in different order. > In the first attempt 1, we write three records 5,4,3 to > fileID_1_log.1_attempt1. But this attempt failed. Spark will have a try in > the second task attempt (attempt 2), we write four records 1,2,3,4 to > fileID_1_log.1_attempt2. And then, we find this filegroup is large enough by > call canWrite. So hudi write record 5 to fileID_2_log.1_attempt2 and finish > this commit. > When we do compaction, fileID_1_log.1_attempt1 and fileID_1_log.1_attempt2 > will be compacted. And we finally got 543 + 1234 = 12345 in fileID_1 while we > also got 5 in fileID_2. Record 5 will appear in two fileGroup. > Reason: Markerfile doesn't reconcile log file as code show in > [https://github.com/apache/hudi/blob/9a2030ab3190acf600ce4820be9a08929595763e/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java#L553.] > And log file is actually not fail-safe. > I'm not sure if [~danny0405] have found this problem too as I find > FlinkAppendHandle had been made to always return true. But it was just > changed back recently. > Solution: > We may have a quick fix by making canWrite in HoodieAppendHandle always > return true. However, I think there may be a more elegant solution that we > use append result to generate compaction plan rather than list log file, in > which we will have a more granular control on log block instead of log file. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-3026) HoodieAppendhandle may result in duplicate key for hbase index
[ https://issues.apache.org/jira/browse/HUDI-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prashant Wason updated HUDI-3026: - Fix Version/s: 0.14.1 (was: 0.14.0) > HoodieAppendhandle may result in duplicate key for hbase index > -- > > Key: HUDI-3026 > URL: https://issues.apache.org/jira/browse/HUDI-3026 > Project: Apache Hudi > Issue Type: Bug >Reporter: ZiyueGuan >Assignee: ZiyueGuan >Priority: Critical > Labels: pull-request-available > Fix For: 0.14.1 > > Original Estimate: 1h > Remaining Estimate: 1h > > Problem: a same key may occur in two file group when Hbase index is used. > These two file group will have same FileID prefix. As Hbase index is global, > this is unexpected > How to repro: > We should have a table w/o record sorted in spark. Let's say we have five > records with key 1,2,3,4,5 to write. They may be iterated in different order. > In the first attempt 1, we write three records 5,4,3 to > fileID_1_log.1_attempt1. But this attempt failed. Spark will have a try in > the second task attempt (attempt 2), we write four records 1,2,3,4 to > fileID_1_log.1_attempt2. And then, we find this filegroup is large enough by > call canWrite. So hudi write record 5 to fileID_2_log.1_attempt2 and finish > this commit. > When we do compaction, fileID_1_log.1_attempt1 and fileID_1_log.1_attempt2 > will be compacted. And we finally got 543 + 1234 = 12345 in fileID_1 while we > also got 5 in fileID_2. Record 5 will appear in two fileGroup. > Reason: Markerfile doesn't reconcile log file as code show in > [https://github.com/apache/hudi/blob/9a2030ab3190acf600ce4820be9a08929595763e/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java#L553.] > And log file is actually not fail-safe. > I'm not sure if [~danny0405] have found this problem too as I find > FlinkAppendHandle had been made to always return true. But it was just > changed back recently. > Solution: > We may have a quick fix by making canWrite in HoodieAppendHandle always > return true. However, I think there may be a more elegant solution that we > use append result to generate compaction plan rather than list log file, in > which we will have a more granular control on log block instead of log file. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-3026) HoodieAppendhandle may result in duplicate key for hbase index
[ https://issues.apache.org/jira/browse/HUDI-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ethan Guo updated HUDI-3026: Fix Version/s: 0.14.0 (was: 0.13.0) > HoodieAppendhandle may result in duplicate key for hbase index > -- > > Key: HUDI-3026 > URL: https://issues.apache.org/jira/browse/HUDI-3026 > Project: Apache Hudi > Issue Type: Bug >Reporter: ZiyueGuan >Assignee: ZiyueGuan >Priority: Critical > Labels: pull-request-available > Fix For: 0.14.0 > > Original Estimate: 1h > Remaining Estimate: 1h > > Problem: a same key may occur in two file group when Hbase index is used. > These two file group will have same FileID prefix. As Hbase index is global, > this is unexpected > How to repro: > We should have a table w/o record sorted in spark. Let's say we have five > records with key 1,2,3,4,5 to write. They may be iterated in different order. > In the first attempt 1, we write three records 5,4,3 to > fileID_1_log.1_attempt1. But this attempt failed. Spark will have a try in > the second task attempt (attempt 2), we write four records 1,2,3,4 to > fileID_1_log.1_attempt2. And then, we find this filegroup is large enough by > call canWrite. So hudi write record 5 to fileID_2_log.1_attempt2 and finish > this commit. > When we do compaction, fileID_1_log.1_attempt1 and fileID_1_log.1_attempt2 > will be compacted. And we finally got 543 + 1234 = 12345 in fileID_1 while we > also got 5 in fileID_2. Record 5 will appear in two fileGroup. > Reason: Markerfile doesn't reconcile log file as code show in > [https://github.com/apache/hudi/blob/9a2030ab3190acf600ce4820be9a08929595763e/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java#L553.] > And log file is actually not fail-safe. > I'm not sure if [~danny0405] have found this problem too as I find > FlinkAppendHandle had been made to always return true. But it was just > changed back recently. > Solution: > We may have a quick fix by making canWrite in HoodieAppendHandle always > return true. However, I think there may be a more elegant solution that we > use append result to generate compaction plan rather than list log file, in > which we will have a more granular control on log block instead of log file. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-3026) HoodieAppendhandle may result in duplicate key for hbase index
[ https://issues.apache.org/jira/browse/HUDI-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sagar Sumit updated HUDI-3026: -- Fix Version/s: 0.13.0 (was: 0.12.0) > HoodieAppendhandle may result in duplicate key for hbase index > -- > > Key: HUDI-3026 > URL: https://issues.apache.org/jira/browse/HUDI-3026 > Project: Apache Hudi > Issue Type: Bug >Reporter: ZiyueGuan >Assignee: ZiyueGuan >Priority: Critical > Labels: pull-request-available > Fix For: 0.13.0 > > Original Estimate: 1h > Remaining Estimate: 1h > > Problem: a same key may occur in two file group when Hbase index is used. > These two file group will have same FileID prefix. As Hbase index is global, > this is unexpected > How to repro: > We should have a table w/o record sorted in spark. Let's say we have five > records with key 1,2,3,4,5 to write. They may be iterated in different order. > In the first attempt 1, we write three records 5,4,3 to > fileID_1_log.1_attempt1. But this attempt failed. Spark will have a try in > the second task attempt (attempt 2), we write four records 1,2,3,4 to > fileID_1_log.1_attempt2. And then, we find this filegroup is large enough by > call canWrite. So hudi write record 5 to fileID_2_log.1_attempt2 and finish > this commit. > When we do compaction, fileID_1_log.1_attempt1 and fileID_1_log.1_attempt2 > will be compacted. And we finally got 543 + 1234 = 12345 in fileID_1 while we > also got 5 in fileID_2. Record 5 will appear in two fileGroup. > Reason: Markerfile doesn't reconcile log file as code show in > [https://github.com/apache/hudi/blob/9a2030ab3190acf600ce4820be9a08929595763e/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java#L553.] > And log file is actually not fail-safe. > I'm not sure if [~danny0405] have found this problem too as I find > FlinkAppendHandle had been made to always return true. But it was just > changed back recently. > Solution: > We may have a quick fix by making canWrite in HoodieAppendHandle always > return true. However, I think there may be a more elegant solution that we > use append result to generate compaction plan rather than list log file, in > which we will have a more granular control on log block instead of log file. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HUDI-3026) HoodieAppendhandle may result in duplicate key for hbase index
[ https://issues.apache.org/jira/browse/HUDI-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3026: - Fix Version/s: 0.12.0 (was: 0.11.0) > HoodieAppendhandle may result in duplicate key for hbase index > -- > > Key: HUDI-3026 > URL: https://issues.apache.org/jira/browse/HUDI-3026 > Project: Apache Hudi > Issue Type: Bug >Reporter: ZiyueGuan >Assignee: ZiyueGuan >Priority: Critical > Labels: pull-request-available > Fix For: 0.12.0 > > Original Estimate: 1h > Remaining Estimate: 1h > > Problem: a same key may occur in two file group when Hbase index is used. > These two file group will have same FileID prefix. As Hbase index is global, > this is unexpected > How to repro: > We should have a table w/o record sorted in spark. Let's say we have five > records with key 1,2,3,4,5 to write. They may be iterated in different order. > In the first attempt 1, we write three records 5,4,3 to > fileID_1_log.1_attempt1. But this attempt failed. Spark will have a try in > the second task attempt (attempt 2), we write four records 1,2,3,4 to > fileID_1_log.1_attempt2. And then, we find this filegroup is large enough by > call canWrite. So hudi write record 5 to fileID_2_log.1_attempt2 and finish > this commit. > When we do compaction, fileID_1_log.1_attempt1 and fileID_1_log.1_attempt2 > will be compacted. And we finally got 543 + 1234 = 12345 in fileID_1 while we > also got 5 in fileID_2. Record 5 will appear in two fileGroup. > Reason: Markerfile doesn't reconcile log file as code show in > [https://github.com/apache/hudi/blob/9a2030ab3190acf600ce4820be9a08929595763e/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java#L553.] > And log file is actually not fail-safe. > I'm not sure if [~danny0405] have found this problem too as I find > FlinkAppendHandle had been made to always return true. But it was just > changed back recently. > Solution: > We may have a quick fix by making canWrite in HoodieAppendHandle always > return true. However, I think there may be a more elegant solution that we > use append result to generate compaction plan rather than list log file, in > which we will have a more granular control on log block instead of log file. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-3026) HoodieAppendhandle may result in duplicate key for hbase index
[ https://issues.apache.org/jira/browse/HUDI-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3026: - Sprint: Cont' improve - 2022/02/07, Cont' improve - 2022/02/14, Cont' improve - 2022/03/01, Cont' improve - 2022/03/7 (was: Cont' improve - 2022/02/07, Cont' improve - 2022/02/14, Cont' improve - 2022/03/01) > HoodieAppendhandle may result in duplicate key for hbase index > -- > > Key: HUDI-3026 > URL: https://issues.apache.org/jira/browse/HUDI-3026 > Project: Apache Hudi > Issue Type: Bug >Reporter: ZiyueGuan >Assignee: ZiyueGuan >Priority: Critical > Labels: pull-request-available > Fix For: 0.11.0 > > Original Estimate: 1h > Remaining Estimate: 1h > > Problem: a same key may occur in two file group when Hbase index is used. > These two file group will have same FileID prefix. As Hbase index is global, > this is unexpected > How to repro: > We should have a table w/o record sorted in spark. Let's say we have five > records with key 1,2,3,4,5 to write. They may be iterated in different order. > In the first attempt 1, we write three records 5,4,3 to > fileID_1_log.1_attempt1. But this attempt failed. Spark will have a try in > the second task attempt (attempt 2), we write four records 1,2,3,4 to > fileID_1_log.1_attempt2. And then, we find this filegroup is large enough by > call canWrite. So hudi write record 5 to fileID_2_log.1_attempt2 and finish > this commit. > When we do compaction, fileID_1_log.1_attempt1 and fileID_1_log.1_attempt2 > will be compacted. And we finally got 543 + 1234 = 12345 in fileID_1 while we > also got 5 in fileID_2. Record 5 will appear in two fileGroup. > Reason: Markerfile doesn't reconcile log file as code show in > [https://github.com/apache/hudi/blob/9a2030ab3190acf600ce4820be9a08929595763e/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java#L553.] > And log file is actually not fail-safe. > I'm not sure if [~danny0405] have found this problem too as I find > FlinkAppendHandle had been made to always return true. But it was just > changed back recently. > Solution: > We may have a quick fix by making canWrite in HoodieAppendHandle always > return true. However, I think there may be a more elegant solution that we > use append result to generate compaction plan rather than list log file, in > which we will have a more granular control on log block instead of log file. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-3026) HoodieAppendhandle may result in duplicate key for hbase index
[ https://issues.apache.org/jira/browse/HUDI-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3026: - Sprint: Cont' improve - 2022/02/07, Cont' improve - 2022/02/14, Cont' improve - 2022/03/01 (was: Cont' improve - 2022/02/07, Cont' improve - 2022/02/14) > HoodieAppendhandle may result in duplicate key for hbase index > -- > > Key: HUDI-3026 > URL: https://issues.apache.org/jira/browse/HUDI-3026 > Project: Apache Hudi > Issue Type: Bug >Reporter: ZiyueGuan >Assignee: ZiyueGuan >Priority: Critical > Labels: pull-request-available > Fix For: 0.11.0 > > Original Estimate: 1h > Remaining Estimate: 1h > > Problem: a same key may occur in two file group when Hbase index is used. > These two file group will have same FileID prefix. As Hbase index is global, > this is unexpected > How to repro: > We should have a table w/o record sorted in spark. Let's say we have five > records with key 1,2,3,4,5 to write. They may be iterated in different order. > In the first attempt 1, we write three records 5,4,3 to > fileID_1_log.1_attempt1. But this attempt failed. Spark will have a try in > the second task attempt (attempt 2), we write four records 1,2,3,4 to > fileID_1_log.1_attempt2. And then, we find this filegroup is large enough by > call canWrite. So hudi write record 5 to fileID_2_log.1_attempt2 and finish > this commit. > When we do compaction, fileID_1_log.1_attempt1 and fileID_1_log.1_attempt2 > will be compacted. And we finally got 543 + 1234 = 12345 in fileID_1 while we > also got 5 in fileID_2. Record 5 will appear in two fileGroup. > Reason: Markerfile doesn't reconcile log file as code show in > [https://github.com/apache/hudi/blob/9a2030ab3190acf600ce4820be9a08929595763e/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java#L553.] > And log file is actually not fail-safe. > I'm not sure if [~danny0405] have found this problem too as I find > FlinkAppendHandle had been made to always return true. But it was just > changed back recently. > Solution: > We may have a quick fix by making canWrite in HoodieAppendHandle always > return true. However, I think there may be a more elegant solution that we > use append result to generate compaction plan rather than list log file, in > which we will have a more granular control on log block instead of log file. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-3026) HoodieAppendhandle may result in duplicate key for hbase index
[ https://issues.apache.org/jira/browse/HUDI-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3026: -- Sprint: Cont' improve - 2022/02/07, Cont' improve - 2022/02/14 (was: Cont' improve - 2022/02/07, Cont' improve - 2022/02/14, Cont' improve - 2022/02/21) > HoodieAppendhandle may result in duplicate key for hbase index > -- > > Key: HUDI-3026 > URL: https://issues.apache.org/jira/browse/HUDI-3026 > Project: Apache Hudi > Issue Type: Bug >Reporter: ZiyueGuan >Assignee: ZiyueGuan >Priority: Critical > Labels: pull-request-available > Fix For: 0.11.0 > > Original Estimate: 1h > Remaining Estimate: 1h > > Problem: a same key may occur in two file group when Hbase index is used. > These two file group will have same FileID prefix. As Hbase index is global, > this is unexpected > How to repro: > We should have a table w/o record sorted in spark. Let's say we have five > records with key 1,2,3,4,5 to write. They may be iterated in different order. > In the first attempt 1, we write three records 5,4,3 to > fileID_1_log.1_attempt1. But this attempt failed. Spark will have a try in > the second task attempt (attempt 2), we write four records 1,2,3,4 to > fileID_1_log.1_attempt2. And then, we find this filegroup is large enough by > call canWrite. So hudi write record 5 to fileID_2_log.1_attempt2 and finish > this commit. > When we do compaction, fileID_1_log.1_attempt1 and fileID_1_log.1_attempt2 > will be compacted. And we finally got 543 + 1234 = 12345 in fileID_1 while we > also got 5 in fileID_2. Record 5 will appear in two fileGroup. > Reason: Markerfile doesn't reconcile log file as code show in > [https://github.com/apache/hudi/blob/9a2030ab3190acf600ce4820be9a08929595763e/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java#L553.] > And log file is actually not fail-safe. > I'm not sure if [~danny0405] have found this problem too as I find > FlinkAppendHandle had been made to always return true. But it was just > changed back recently. > Solution: > We may have a quick fix by making canWrite in HoodieAppendHandle always > return true. However, I think there may be a more elegant solution that we > use append result to generate compaction plan rather than list log file, in > which we will have a more granular control on log block instead of log file. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-3026) HoodieAppendhandle may result in duplicate key for hbase index
[ https://issues.apache.org/jira/browse/HUDI-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3026: -- Sprint: Cont' improve - 2022/02/07, Cont' improve - 2022/02/14, Cont' improve - 2022/02/21 (was: Cont' improve - 2022/02/07, Cont' improve - 2022/02/14) > HoodieAppendhandle may result in duplicate key for hbase index > -- > > Key: HUDI-3026 > URL: https://issues.apache.org/jira/browse/HUDI-3026 > Project: Apache Hudi > Issue Type: Bug >Reporter: ZiyueGuan >Assignee: ZiyueGuan >Priority: Critical > Labels: pull-request-available > Fix For: 0.11.0 > > Original Estimate: 1h > Remaining Estimate: 1h > > Problem: a same key may occur in two file group when Hbase index is used. > These two file group will have same FileID prefix. As Hbase index is global, > this is unexpected > How to repro: > We should have a table w/o record sorted in spark. Let's say we have five > records with key 1,2,3,4,5 to write. They may be iterated in different order. > In the first attempt 1, we write three records 5,4,3 to > fileID_1_log.1_attempt1. But this attempt failed. Spark will have a try in > the second task attempt (attempt 2), we write four records 1,2,3,4 to > fileID_1_log.1_attempt2. And then, we find this filegroup is large enough by > call canWrite. So hudi write record 5 to fileID_2_log.1_attempt2 and finish > this commit. > When we do compaction, fileID_1_log.1_attempt1 and fileID_1_log.1_attempt2 > will be compacted. And we finally got 543 + 1234 = 12345 in fileID_1 while we > also got 5 in fileID_2. Record 5 will appear in two fileGroup. > Reason: Markerfile doesn't reconcile log file as code show in > [https://github.com/apache/hudi/blob/9a2030ab3190acf600ce4820be9a08929595763e/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java#L553.] > And log file is actually not fail-safe. > I'm not sure if [~danny0405] have found this problem too as I find > FlinkAppendHandle had been made to always return true. But it was just > changed back recently. > Solution: > We may have a quick fix by making canWrite in HoodieAppendHandle always > return true. However, I think there may be a more elegant solution that we > use append result to generate compaction plan rather than list log file, in > which we will have a more granular control on log block instead of log file. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-3026) HoodieAppendhandle may result in duplicate key for hbase index
[ https://issues.apache.org/jira/browse/HUDI-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3026: -- Sprint: Cont' improve - 2022/02/07, Cont' improve - 2022/02/14 (was: Cont' improve - 2022/02/07) > HoodieAppendhandle may result in duplicate key for hbase index > -- > > Key: HUDI-3026 > URL: https://issues.apache.org/jira/browse/HUDI-3026 > Project: Apache Hudi > Issue Type: Bug >Reporter: ZiyueGuan >Assignee: ZiyueGuan >Priority: Critical > Labels: pull-request-available > Fix For: 0.11.0 > > Original Estimate: 1h > Remaining Estimate: 1h > > Problem: a same key may occur in two file group when Hbase index is used. > These two file group will have same FileID prefix. As Hbase index is global, > this is unexpected > How to repro: > We should have a table w/o record sorted in spark. Let's say we have five > records with key 1,2,3,4,5 to write. They may be iterated in different order. > In the first attempt 1, we write three records 5,4,3 to > fileID_1_log.1_attempt1. But this attempt failed. Spark will have a try in > the second task attempt (attempt 2), we write four records 1,2,3,4 to > fileID_1_log.1_attempt2. And then, we find this filegroup is large enough by > call canWrite. So hudi write record 5 to fileID_2_log.1_attempt2 and finish > this commit. > When we do compaction, fileID_1_log.1_attempt1 and fileID_1_log.1_attempt2 > will be compacted. And we finally got 543 + 1234 = 12345 in fileID_1 while we > also got 5 in fileID_2. Record 5 will appear in two fileGroup. > Reason: Markerfile doesn't reconcile log file as code show in > [https://github.com/apache/hudi/blob/9a2030ab3190acf600ce4820be9a08929595763e/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java#L553.] > And log file is actually not fail-safe. > I'm not sure if [~danny0405] have found this problem too as I find > FlinkAppendHandle had been made to always return true. But it was just > changed back recently. > Solution: > We may have a quick fix by making canWrite in HoodieAppendHandle always > return true. However, I think there may be a more elegant solution that we > use append result to generate compaction plan rather than list log file, in > which we will have a more granular control on log block instead of log file. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-3026) HoodieAppendhandle may result in duplicate key for hbase index
[ https://issues.apache.org/jira/browse/HUDI-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3026: -- Status: In Progress (was: Open) > HoodieAppendhandle may result in duplicate key for hbase index > -- > > Key: HUDI-3026 > URL: https://issues.apache.org/jira/browse/HUDI-3026 > Project: Apache Hudi > Issue Type: Bug >Reporter: ZiyueGuan >Assignee: ZiyueGuan >Priority: Critical > Labels: pull-request-available > Fix For: 0.11.0 > > Original Estimate: 1h > Remaining Estimate: 1h > > Problem: a same key may occur in two file group when Hbase index is used. > These two file group will have same FileID prefix. As Hbase index is global, > this is unexpected > How to repro: > We should have a table w/o record sorted in spark. Let's say we have five > records with key 1,2,3,4,5 to write. They may be iterated in different order. > In the first attempt 1, we write three records 5,4,3 to > fileID_1_log.1_attempt1. But this attempt failed. Spark will have a try in > the second task attempt (attempt 2), we write four records 1,2,3,4 to > fileID_1_log.1_attempt2. And then, we find this filegroup is large enough by > call canWrite. So hudi write record 5 to fileID_2_log.1_attempt2 and finish > this commit. > When we do compaction, fileID_1_log.1_attempt1 and fileID_1_log.1_attempt2 > will be compacted. And we finally got 543 + 1234 = 12345 in fileID_1 while we > also got 5 in fileID_2. Record 5 will appear in two fileGroup. > Reason: Markerfile doesn't reconcile log file as code show in > [https://github.com/apache/hudi/blob/9a2030ab3190acf600ce4820be9a08929595763e/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java#L553.] > And log file is actually not fail-safe. > I'm not sure if [~danny0405] have found this problem too as I find > FlinkAppendHandle had been made to always return true. But it was just > changed back recently. > Solution: > We may have a quick fix by making canWrite in HoodieAppendHandle always > return true. However, I think there may be a more elegant solution that we > use append result to generate compaction plan rather than list log file, in > which we will have a more granular control on log block instead of log file. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-3026) HoodieAppendhandle may result in duplicate key for hbase index
[ https://issues.apache.org/jira/browse/HUDI-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3026: -- Priority: Critical (was: Major) > HoodieAppendhandle may result in duplicate key for hbase index > -- > > Key: HUDI-3026 > URL: https://issues.apache.org/jira/browse/HUDI-3026 > Project: Apache Hudi > Issue Type: Bug >Reporter: ZiyueGuan >Assignee: ZiyueGuan >Priority: Critical > Labels: pull-request-available > Fix For: 0.11.0 > > Original Estimate: 1h > Remaining Estimate: 1h > > Problem: a same key may occur in two file group when Hbase index is used. > These two file group will have same FileID prefix. As Hbase index is global, > this is unexpected > How to repro: > We should have a table w/o record sorted in spark. Let's say we have five > records with key 1,2,3,4,5 to write. They may be iterated in different order. > In the first attempt 1, we write three records 5,4,3 to > fileID_1_log.1_attempt1. But this attempt failed. Spark will have a try in > the second task attempt (attempt 2), we write four records 1,2,3,4 to > fileID_1_log.1_attempt2. And then, we find this filegroup is large enough by > call canWrite. So hudi write record 5 to fileID_2_log.1_attempt2 and finish > this commit. > When we do compaction, fileID_1_log.1_attempt1 and fileID_1_log.1_attempt2 > will be compacted. And we finally got 543 + 1234 = 12345 in fileID_1 while we > also got 5 in fileID_2. Record 5 will appear in two fileGroup. > Reason: Markerfile doesn't reconcile log file as code show in > [https://github.com/apache/hudi/blob/9a2030ab3190acf600ce4820be9a08929595763e/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java#L553.] > And log file is actually not fail-safe. > I'm not sure if [~danny0405] have found this problem too as I find > FlinkAppendHandle had been made to always return true. But it was just > changed back recently. > Solution: > We may have a quick fix by making canWrite in HoodieAppendHandle always > return true. However, I think there may be a more elegant solution that we > use append result to generate compaction plan rather than list log file, in > which we will have a more granular control on log block instead of log file. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-3026) HoodieAppendhandle may result in duplicate key for hbase index
[ https://issues.apache.org/jira/browse/HUDI-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3026: -- Remaining Estimate: 1h Original Estimate: 1h > HoodieAppendhandle may result in duplicate key for hbase index > -- > > Key: HUDI-3026 > URL: https://issues.apache.org/jira/browse/HUDI-3026 > Project: Apache Hudi > Issue Type: Bug >Reporter: ZiyueGuan >Assignee: ZiyueGuan >Priority: Major > Labels: pull-request-available > Fix For: 0.11.0 > > Original Estimate: 1h > Remaining Estimate: 1h > > Problem: a same key may occur in two file group when Hbase index is used. > These two file group will have same FileID prefix. As Hbase index is global, > this is unexpected > How to repro: > We should have a table w/o record sorted in spark. Let's say we have five > records with key 1,2,3,4,5 to write. They may be iterated in different order. > In the first attempt 1, we write three records 5,4,3 to > fileID_1_log.1_attempt1. But this attempt failed. Spark will have a try in > the second task attempt (attempt 2), we write four records 1,2,3,4 to > fileID_1_log.1_attempt2. And then, we find this filegroup is large enough by > call canWrite. So hudi write record 5 to fileID_2_log.1_attempt2 and finish > this commit. > When we do compaction, fileID_1_log.1_attempt1 and fileID_1_log.1_attempt2 > will be compacted. And we finally got 543 + 1234 = 12345 in fileID_1 while we > also got 5 in fileID_2. Record 5 will appear in two fileGroup. > Reason: Markerfile doesn't reconcile log file as code show in > [https://github.com/apache/hudi/blob/9a2030ab3190acf600ce4820be9a08929595763e/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java#L553.] > And log file is actually not fail-safe. > I'm not sure if [~danny0405] have found this problem too as I find > FlinkAppendHandle had been made to always return true. But it was just > changed back recently. > Solution: > We may have a quick fix by making canWrite in HoodieAppendHandle always > return true. However, I think there may be a more elegant solution that we > use append result to generate compaction plan rather than list log file, in > which we will have a more granular control on log block instead of log file. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-3026) HoodieAppendhandle may result in duplicate key for hbase index
[ https://issues.apache.org/jira/browse/HUDI-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3026: -- Fix Version/s: 0.11.0 > HoodieAppendhandle may result in duplicate key for hbase index > -- > > Key: HUDI-3026 > URL: https://issues.apache.org/jira/browse/HUDI-3026 > Project: Apache Hudi > Issue Type: Bug >Reporter: ZiyueGuan >Assignee: ZiyueGuan >Priority: Major > Labels: pull-request-available > Fix For: 0.11.0 > > > Problem: a same key may occur in two file group when Hbase index is used. > These two file group will have same FileID prefix. As Hbase index is global, > this is unexpected > How to repro: > We should have a table w/o record sorted in spark. Let's say we have five > records with key 1,2,3,4,5 to write. They may be iterated in different order. > In the first attempt 1, we write three records 5,4,3 to > fileID_1_log.1_attempt1. But this attempt failed. Spark will have a try in > the second task attempt (attempt 2), we write four records 1,2,3,4 to > fileID_1_log.1_attempt2. And then, we find this filegroup is large enough by > call canWrite. So hudi write record 5 to fileID_2_log.1_attempt2 and finish > this commit. > When we do compaction, fileID_1_log.1_attempt1 and fileID_1_log.1_attempt2 > will be compacted. And we finally got 543 + 1234 = 12345 in fileID_1 while we > also got 5 in fileID_2. Record 5 will appear in two fileGroup. > Reason: Markerfile doesn't reconcile log file as code show in > [https://github.com/apache/hudi/blob/9a2030ab3190acf600ce4820be9a08929595763e/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java#L553.] > And log file is actually not fail-safe. > I'm not sure if [~danny0405] have found this problem too as I find > FlinkAppendHandle had been made to always return true. But it was just > changed back recently. > Solution: > We may have a quick fix by making canWrite in HoodieAppendHandle always > return true. However, I think there may be a more elegant solution that we > use append result to generate compaction plan rather than list log file, in > which we will have a more granular control on log block instead of log file. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-3026) HoodieAppendhandle may result in duplicate key for hbase index
[ https://issues.apache.org/jira/browse/HUDI-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sivabalan narayanan updated HUDI-3026: -- Reviewers: sivabalan narayanan > HoodieAppendhandle may result in duplicate key for hbase index > -- > > Key: HUDI-3026 > URL: https://issues.apache.org/jira/browse/HUDI-3026 > Project: Apache Hudi > Issue Type: Bug >Reporter: ZiyueGuan >Assignee: ZiyueGuan >Priority: Major > Labels: pull-request-available > Fix For: 0.11.0 > > > Problem: a same key may occur in two file group when Hbase index is used. > These two file group will have same FileID prefix. As Hbase index is global, > this is unexpected > How to repro: > We should have a table w/o record sorted in spark. Let's say we have five > records with key 1,2,3,4,5 to write. They may be iterated in different order. > In the first attempt 1, we write three records 5,4,3 to > fileID_1_log.1_attempt1. But this attempt failed. Spark will have a try in > the second task attempt (attempt 2), we write four records 1,2,3,4 to > fileID_1_log.1_attempt2. And then, we find this filegroup is large enough by > call canWrite. So hudi write record 5 to fileID_2_log.1_attempt2 and finish > this commit. > When we do compaction, fileID_1_log.1_attempt1 and fileID_1_log.1_attempt2 > will be compacted. And we finally got 543 + 1234 = 12345 in fileID_1 while we > also got 5 in fileID_2. Record 5 will appear in two fileGroup. > Reason: Markerfile doesn't reconcile log file as code show in > [https://github.com/apache/hudi/blob/9a2030ab3190acf600ce4820be9a08929595763e/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java#L553.] > And log file is actually not fail-safe. > I'm not sure if [~danny0405] have found this problem too as I find > FlinkAppendHandle had been made to always return true. But it was just > changed back recently. > Solution: > We may have a quick fix by making canWrite in HoodieAppendHandle always > return true. However, I think there may be a more elegant solution that we > use append result to generate compaction plan rather than list log file, in > which we will have a more granular control on log block instead of log file. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-3026) HoodieAppendhandle may result in duplicate key for hbase index
[ https://issues.apache.org/jira/browse/HUDI-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raymond Xu updated HUDI-3026: - Sprint: Cont' improve - 2022/02/07 > HoodieAppendhandle may result in duplicate key for hbase index > -- > > Key: HUDI-3026 > URL: https://issues.apache.org/jira/browse/HUDI-3026 > Project: Apache Hudi > Issue Type: Bug >Reporter: ZiyueGuan >Assignee: ZiyueGuan >Priority: Major > Labels: pull-request-available > > Problem: a same key may occur in two file group when Hbase index is used. > These two file group will have same FileID prefix. As Hbase index is global, > this is unexpected > How to repro: > We should have a table w/o record sorted in spark. Let's say we have five > records with key 1,2,3,4,5 to write. They may be iterated in different order. > In the first attempt 1, we write three records 5,4,3 to > fileID_1_log.1_attempt1. But this attempt failed. Spark will have a try in > the second task attempt (attempt 2), we write four records 1,2,3,4 to > fileID_1_log.1_attempt2. And then, we find this filegroup is large enough by > call canWrite. So hudi write record 5 to fileID_2_log.1_attempt2 and finish > this commit. > When we do compaction, fileID_1_log.1_attempt1 and fileID_1_log.1_attempt2 > will be compacted. And we finally got 543 + 1234 = 12345 in fileID_1 while we > also got 5 in fileID_2. Record 5 will appear in two fileGroup. > Reason: Markerfile doesn't reconcile log file as code show in > [https://github.com/apache/hudi/blob/9a2030ab3190acf600ce4820be9a08929595763e/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java#L553.] > And log file is actually not fail-safe. > I'm not sure if [~danny0405] have found this problem too as I find > FlinkAppendHandle had been made to always return true. But it was just > changed back recently. > Solution: > We may have a quick fix by making canWrite in HoodieAppendHandle always > return true. However, I think there may be a more elegant solution that we > use append result to generate compaction plan rather than list log file, in > which we will have a more granular control on log block instead of log file. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HUDI-3026) HoodieAppendhandle may result in duplicate key for hbase index
[ https://issues.apache.org/jira/browse/HUDI-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZiyueGuan updated HUDI-3026: Description: Problem: a same key may occur in two file group when Hbase index is used. These two file group will have same FileID prefix. As Hbase index is global, this is unexpected How to repro: We should have a table w/o record sorted in spark. Let's say we have five records with key 1,2,3,4,5 to write. They may be iterated in different order. In the first attempt 1, we write three records 5,4,3 to fileID_1_log.1_attempt1. But this attempt failed. Spark will have a try in the second task attempt (attempt 2), we write four records 1,2,3,4 to fileID_1_log.1_attempt2. And then, we find this filegroup is large enough by call canWrite. So hudi write record 5 to fileID_2_log.1_attempt2 and finish this commit. When we do compaction, fileID_1_log.1_attempt1 and fileID_1_log.1_attempt2 will be compacted. And we finally got 543 + 1234 = 12345 in fileID_1 while we also got 5 in fileID_2. Record 5 will appear in two fileGroup. Reason: Markerfile doesn't reconcile log file as code show in [https://github.com/apache/hudi/blob/9a2030ab3190acf600ce4820be9a08929595763e/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java#L553.] And log file is actually not fail-safe. I'm not sure if [~danny0405] have found this problem too as I find FlinkAppendHandle had been made to always return true. But it was just changed back recently. Solution: We may have a quick fix by making canWrite in HoodieAppendHandle always return true. However, I think there may be a more elegant solution that we use append result to generate compaction plan rather than list log file, in which we will have a more granular control on log block instead of log file. was: Problem: a same key may occur in two file group when Hbase index is used. These two file group will have same FileID prefix. As Hbase index is global, this is unexpected How to repro: We should have a table w/o record sorted in spark. Let's say we have 1,2,3,4,5 records to write. They may be iterated in different order. In the first attempt 1, we write 543 to fileID_1_log.1_attempt1. But this attempt failed. Spark will have a try in the second task attempt (attempt 2), we write 1234 to fileID_1_log.1_attempt2. And then, we find this filegroup is large enough by call canWrite. So hudi write record 5 to fileID_2_log.1_attempt2 and finish this commit. When we do compaction, fileID_1_log.1_attempt1 and fileID_1_log.1_attempt2 will be compacted. And we finally got 543 + 1234 = 12345 in fileID_1 while we also got 5 in fileID_2. Record 5 will appear in two fileGroup. Reason: Markerfile doesn't reconcile log file as code show in [https://github.com/apache/hudi/blob/9a2030ab3190acf600ce4820be9a08929595763e/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java#L553.] And log file is actually not fail-safe. I'm not sure if [~danny0405] have found this problem too as I find FlinkAppendHandle had been made to always return true. But it was just changed back recently. Solution: We may have a quick fix by making canWrite in HoodieAppendHandle always return true. However, I think there may be a more elegant solution that we use append result to generate compaction plan rather than list log file, in which we will have a more granular control on log block instead of log file. > HoodieAppendhandle may result in duplicate key for hbase index > -- > > Key: HUDI-3026 > URL: https://issues.apache.org/jira/browse/HUDI-3026 > Project: Apache Hudi > Issue Type: Bug >Reporter: ZiyueGuan >Assignee: ZiyueGuan >Priority: Major > Labels: pull-request-available > > Problem: a same key may occur in two file group when Hbase index is used. > These two file group will have same FileID prefix. As Hbase index is global, > this is unexpected > How to repro: > We should have a table w/o record sorted in spark. Let's say we have five > records with key 1,2,3,4,5 to write. They may be iterated in different order. > In the first attempt 1, we write three records 5,4,3 to > fileID_1_log.1_attempt1. But this attempt failed. Spark will have a try in > the second task attempt (attempt 2), we write four records 1,2,3,4 to > fileID_1_log.1_attempt2. And then, we find this filegroup is large enough by > call canWrite. So hudi write record 5 to fileID_2_log.1_attempt2 and finish > this commit. > When we do compaction, fileID_1_log.1_attempt1 and fileID_1_log.1_attempt2 > will be compacted. And we finally got 543 + 1234 = 12345 in fileID_1 while we > also got 5 in fileID_2. Record 5 will appear in two fileGroup. > Reason: Markerfile doesn't reconcile log file
[jira] [Updated] (HUDI-3026) HoodieAppendhandle may result in duplicate key for hbase index
[ https://issues.apache.org/jira/browse/HUDI-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ZiyueGuan updated HUDI-3026: Description: Problem: a same key may occur in two file group when Hbase index is used. These two file group will have same FileID prefix. As Hbase index is global, this is unexpected How to repro: We should have a table w/o record sorted in spark. Let's say we have 1,2,3,4,5 records to write. They may be iterated in different order. In the first attempt 1, we write 543 to fileID_1_log.1_attempt1. But this attempt failed. Spark will have a try in the second task attempt (attempt 2), we write 1234 to fileID_1_log.1_attempt2. And then, we find this filegroup is large enough by call canWrite. So hudi write record 5 to fileID_2_log.1_attempt2 and finish this commit. When we do compaction, fileID_1_log.1_attempt1 and fileID_1_log.1_attempt2 will be compacted. And we finally got 543 + 1234 = 12345 in fileID_1 while we also got 5 in fileID_2. Record 5 will appear in two fileGroup. Reason: Markerfile doesn't reconcile log file as code show in [https://github.com/apache/hudi/blob/9a2030ab3190acf600ce4820be9a08929595763e/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java#L553.] And log file is actually not fail-safe. I'm not sure if [~danny0405] have found this problem too as I find FlinkAppendHandle had been made to always return true. But it was just changed back recently. Solution: We may have a quick fix by making canWrite in HoodieAppendHandle always return true. However, I think there may be a more elegant solution that we use append result to generate compaction plan rather than list log file, in which we will have a more granular control on log block instead of log file. was: Problem: a same key may occur in two file group. These two file group will have same FileID prefix. How to repro: We should have a table w/o record sorted in spark. Let's say we have 1,2,3,4,5 records to write. They may be iterated in different order. In the first attempt 1, we write 543 to fileID_1_log.1_attempt1. But this attempt failed. Spark will have a try in the second task attempt (attempt 2), we write 1234 to fileID_1_log.1_attempt2. And then, we find this filegroup is large enough by call canWrite. So hudi write record 5 to fileID_2_log.1_attempt2 and finish this commit. When we do compaction, fileID_1_log.1_attempt1 and fileID_1_log.1_attempt2 will be compacted. And we finally got 543 + 1234 = 12345 in fileID_1 while we also got 5 in fileID_2. Record 5 will appear in two fileGroup. Reason: Markerfile doesn't reconcile log file as code show in [https://github.com/apache/hudi/blob/9a2030ab3190acf600ce4820be9a08929595763e/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java#L553.] And log file is actually not fail-safe. I'm not sure if [~danny0405] have found this problem too as I find FlinkAppendHandle had been made to always return true. But it was just changed back recently. Solution: We may have a quick fix by making canWrite in HoodieAppendHandle always return true. However, I think there may be a more elegant solution that we use append result to generate compaction plan rather than list log file, in which we will have a more granular control on log block instead of log file. > HoodieAppendhandle may result in duplicate key for hbase index > -- > > Key: HUDI-3026 > URL: https://issues.apache.org/jira/browse/HUDI-3026 > Project: Apache Hudi > Issue Type: Bug >Reporter: ZiyueGuan >Assignee: ZiyueGuan >Priority: Major > Labels: pull-request-available > > Problem: a same key may occur in two file group when Hbase index is used. > These two file group will have same FileID prefix. As Hbase index is global, > this is unexpected > How to repro: > We should have a table w/o record sorted in spark. Let's say we have > 1,2,3,4,5 records to write. They may be iterated in different order. > In the first attempt 1, we write 543 to fileID_1_log.1_attempt1. But this > attempt failed. Spark will have a try in the second task attempt (attempt 2), > we write 1234 to fileID_1_log.1_attempt2. And then, we find this filegroup > is large enough by call canWrite. So hudi write record 5 to > fileID_2_log.1_attempt2 and finish this commit. > When we do compaction, fileID_1_log.1_attempt1 and fileID_1_log.1_attempt2 > will be compacted. And we finally got 543 + 1234 = 12345 in fileID_1 while we > also got 5 in fileID_2. Record 5 will appear in two fileGroup. > Reason: Markerfile doesn't reconcile log file as code show in > [https://github.com/apache/hudi/blob/9a2030ab3190acf600ce4820be9a08929595763e/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/Ho
[jira] [Updated] (HUDI-3026) HoodieAppendhandle may result in duplicate key for hbase index
[ https://issues.apache.org/jira/browse/HUDI-3026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HUDI-3026: - Labels: pull-request-available (was: ) > HoodieAppendhandle may result in duplicate key for hbase index > -- > > Key: HUDI-3026 > URL: https://issues.apache.org/jira/browse/HUDI-3026 > Project: Apache Hudi > Issue Type: Bug >Reporter: ZiyueGuan >Assignee: ZiyueGuan >Priority: Major > Labels: pull-request-available > > Problem: a same key may occur in two file group. These two file group will > have same FileID prefix. > How to repro: > We should have a table w/o record sorted in spark. Let's say we have > 1,2,3,4,5 records to write. They may be iterated in different order. > In the first attempt 1, we write 543 to fileID_1_log.1_attempt1. But this > attempt failed. Spark will have a try in the second task attempt (attempt 2), > we write 1234 to fileID_1_log.1_attempt2. And then, we find this filegroup > is large enough by call canWrite. So hudi write record 5 to > fileID_2_log.1_attempt2 and finish this commit. > When we do compaction, fileID_1_log.1_attempt1 and fileID_1_log.1_attempt2 > will be compacted. And we finally got 543 + 1234 = 12345 in fileID_1 while we > also got 5 in fileID_2. Record 5 will appear in two fileGroup. > Reason: Markerfile doesn't reconcile log file as code show in > [https://github.com/apache/hudi/blob/9a2030ab3190acf600ce4820be9a08929595763e/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/HoodieTable.java#L553.] > And log file is actually not fail-safe. > I'm not sure if [~danny0405] have found this problem too as I find > FlinkAppendHandle had been made to always return true. But it was just > changed back recently. > Solution: > We may have a quick fix by making canWrite in HoodieAppendHandle always > return true. However, I think there may be a more elegant solution that we > use append result to generate compaction plan rather than list log file, in > which we will have a more granular control on log block instead of log file. -- This message was sent by Atlassian Jira (v8.20.1#820001)
