Re: [PR] Support changelog scan for table with delete files [iceberg]

aokolnychyi commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2407845436 Will start looking into this PR today and should be able to finish over the weekend. -- This is an automated message from the Apache Git Service. To respond to the message, please

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2374326949 @flyrain you worked on this area and implemented some of the changelog support; can you please review? -- This is an automated message from the Apache Git Service. To respond to the me

Re: [PR] Support changelog scan for table with delete files [iceberg]

pvary commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2373258387 @RussellSpitzer, @aokolnychyi: would it be possible to take a look at this PR? I did my best to review it, but I'm not an expert in this part of the code. Thanks, Peter -- This is a

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2372741865 @pvary can you please help move this forward then? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above

Re: [PR] Support changelog scan for table with delete files [iceberg]

pvary commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2367649336 This looks good to me. @dramaticlly: Any more comments, before we try to involve the guys who are more experienced with the core/spark parts? -- This is an automated message from the

Re: [PR] Support changelog scan for table with delete files [iceberg]

pvary commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1771032932 ## core/src/test/java/org/apache/iceberg/TestBaseIncrementalChangelogScan.java: ## @@ -132,6 +131,175 @@ public void testFileDeletes() { assertThat(t1.existingDele

Re: [PR] Support changelog scan for table with delete files [iceberg]

pvary commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1771031788 ## core/src/test/java/org/apache/iceberg/TestBaseIncrementalChangelogScan.java: ## @@ -132,6 +131,139 @@ public void testFileDeletes() { assertThat(t1.existingDele

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1767624705 ## core/src/test/java/org/apache/iceberg/TestBaseIncrementalChangelogScan.java: ## @@ -132,6 +131,175 @@ public void testFileDeletes() { assertThat(t1.existingDel

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1767624705 ## core/src/test/java/org/apache/iceberg/TestBaseIncrementalChangelogScan.java: ## @@ -132,6 +131,175 @@ public void testFileDeletes() { assertThat(t1.existingDel

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1767624705 ## core/src/test/java/org/apache/iceberg/TestBaseIncrementalChangelogScan.java: ## @@ -132,6 +131,175 @@ public void testFileDeletes() { assertThat(t1.existingDel

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1767624705 ## core/src/test/java/org/apache/iceberg/TestBaseIncrementalChangelogScan.java: ## @@ -132,6 +131,175 @@ public void testFileDeletes() { assertThat(t1.existingDel

Re: [PR] Support changelog scan for table with delete files [iceberg]

pvary commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1767544018 ## core/src/test/java/org/apache/iceberg/TestBaseIncrementalChangelogScan.java: ## @@ -132,6 +131,175 @@ public void testFileDeletes() { assertThat(t1.existingDele

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1767321018 ## core/src/test/java/org/apache/iceberg/TestBaseIncrementalChangelogScan.java: ## @@ -132,6 +131,175 @@ public void testFileDeletes() { assertThat(t1.existingDel

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1767319143 ## core/src/test/java/org/apache/iceberg/TestBaseIncrementalChangelogScan.java: ## @@ -132,6 +131,175 @@ public void testFileDeletes() { assertThat(t1.existingDel

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1767295969 ## core/src/test/java/org/apache/iceberg/TestBaseIncrementalChangelogScan.java: ## @@ -132,6 +131,175 @@ public void testFileDeletes() { assertThat(t1.existingDel

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1767285190 ## core/src/test/java/org/apache/iceberg/TestBaseIncrementalChangelogScan.java: ## @@ -132,6 +131,175 @@ public void testFileDeletes() { assertThat(t1.existingDel

Re: [PR] Support changelog scan for table with delete files [iceberg]

dramaticlly commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1767264496 ## core/src/test/java/org/apache/iceberg/TestBaseIncrementalChangelogScan.java: ## @@ -132,6 +131,175 @@ public void testFileDeletes() { assertThat(t1.existi

Re: [PR] Support changelog scan for table with delete files [iceberg]

dramaticlly commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1767226037 ## core/src/test/java/org/apache/iceberg/TestBaseIncrementalChangelogScan.java: ## @@ -132,6 +131,175 @@ public void testFileDeletes() { assertThat(t1.existi

Re: [PR] Support changelog scan for table with delete files [iceberg]

pvary commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1766615044 ## core/src/test/java/org/apache/iceberg/TestBaseIncrementalChangelogScan.java: ## @@ -132,6 +131,175 @@ public void testFileDeletes() { assertThat(t1.existingDele

Re: [PR] Support changelog scan for table with delete files [iceberg]

pvary commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1766608610 ## core/src/test/java/org/apache/iceberg/TestBaseIncrementalChangelogScan.java: ## @@ -132,6 +131,175 @@ public void testFileDeletes() { assertThat(t1.existingDele

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1765955792 ## core/src/test/java/org/apache/iceberg/TestBaseIncrementalChangelogScan.java: ## @@ -132,6 +131,175 @@ public void testFileDeletes() { assertThat(t1.existingDel

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1765955792 ## core/src/test/java/org/apache/iceberg/TestBaseIncrementalChangelogScan.java: ## @@ -132,6 +131,175 @@ public void testFileDeletes() { assertThat(t1.existingDel

Re: [PR] Support changelog scan for table with delete files [iceberg]

dramaticlly commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1765491165 ## core/src/test/java/org/apache/iceberg/TestBaseIncrementalChangelogScan.java: ## @@ -132,6 +131,175 @@ public void testFileDeletes() { assertThat(t1.existi

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1764249470 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -133,51 +128,124 @@ private static Map computeSnapshotOrdinals(Deque snapsh ret

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1763998376 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -133,51 +128,124 @@ private static Map computeSnapshotOrdinals(Deque snapsh ret

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1763979054 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -133,51 +131,149 @@ private static Map computeSnapshotOrdinals(Deque snapsh ret

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1763976533 ## core/src/test/java/org/apache/iceberg/TestBaseIncrementalChangelogScan.java: ## @@ -132,6 +131,139 @@ public void testFileDeletes() { assertThat(t1.existingDel

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1763976533 ## core/src/test/java/org/apache/iceberg/TestBaseIncrementalChangelogScan.java: ## @@ -132,6 +131,139 @@ public void testFileDeletes() { assertThat(t1.existingDel

Re: [PR] Support changelog scan for table with delete files [iceberg]

pvary commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1762795906 ## core/src/test/java/org/apache/iceberg/TestBaseIncrementalChangelogScan.java: ## @@ -132,6 +131,139 @@ public void testFileDeletes() { assertThat(t1.existingDele

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1762263665 ## core/src/test/java/org/apache/iceberg/TestBaseIncrementalChangelogScan.java: ## @@ -132,6 +131,139 @@ public void testFileDeletes() { assertThat(t1.existingDel

Re: [PR] Support changelog scan for table with delete files [iceberg]

pvary commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1759686766 ## core/src/test/java/org/apache/iceberg/TestBaseIncrementalChangelogScan.java: ## @@ -132,6 +131,139 @@ public void testFileDeletes() { assertThat(t1.existingDele

Re: [PR] Support changelog scan for table with delete files [iceberg]

pvary commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1759685604 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -63,33 +60,43 @@ protected CloseableIterable doPlanFiles( return CloseableItera

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1759573665 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -63,33 +60,43 @@ protected CloseableIterable doPlanFiles( return CloseableIter

Re: [PR] Support changelog scan for table with delete files [iceberg]

pvary commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2348057437 > @pvary I added some tests in `TestBaseIncrementalChangelogScan`. However, at that level, we can only check what scan tasks (`AddedRowsScanTask`, `DeletedRowsScanTask`, `DeletedDataFileS

Re: [PR] Support changelog scan for table with delete files [iceberg]

pvary commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1758196928 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -63,33 +60,43 @@ protected CloseableIterable doPlanFiles( return CloseableItera

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2348055779 @pvary I added some tests in `TestBaseIncrementalChangelogScan`. However, at that level, we can only check what scan tasks (`AddedRowsScanTask`, `DeletedRowsScanTask`, `DeletedDataFileSc

Re: [PR] Support changelog scan for table with delete files [iceberg]

pvary commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1758196928 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -63,33 +60,43 @@ protected CloseableIterable doPlanFiles( return CloseableItera

Re: [PR] Support changelog scan for table with delete files [iceberg]

pvary commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1758193054 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -133,51 +131,149 @@ private static Map computeSnapshotOrdinals(Deque snapsh retu

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1758191734 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -63,33 +60,43 @@ protected CloseableIterable doPlanFiles( return CloseableIter

Re: [PR] Support changelog scan for table with delete files [iceberg]

pvary commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1758186873 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -133,51 +131,149 @@ private static Map computeSnapshotOrdinals(Deque snapsh retu

Re: [PR] Support changelog scan for table with delete files [iceberg]

pvary commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1758181104 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -63,33 +60,43 @@ protected CloseableIterable doPlanFiles( return CloseableItera

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1757711124 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -133,51 +131,149 @@ private static Map computeSnapshotOrdinals(Deque snapsh ret

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1757711124 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -133,51 +131,149 @@ private static Map computeSnapshotOrdinals(Deque snapsh ret

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1757707990 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -133,51 +131,149 @@ private static Map computeSnapshotOrdinals(Deque snapsh ret

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1757706938 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -133,51 +131,149 @@ private static Map computeSnapshotOrdinals(Deque snapsh ret

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1757701740 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/ChangelogRowReader.java: ## @@ -112,13 +149,62 @@ private CloseableIterable openChangelogScanTask(Ch

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1757688389 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/ChangelogRowReader.java: ## @@ -112,13 +149,62 @@ private CloseableIterable openChangelogScanTask(Ch

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1757688389 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/ChangelogRowReader.java: ## @@ -112,13 +149,62 @@ private CloseableIterable openChangelogScanTask(Ch

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1757688389 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/ChangelogRowReader.java: ## @@ -112,13 +149,62 @@ private CloseableIterable openChangelogScanTask(Ch

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1757679042 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestChangelogReader.java: ## @@ -191,39 +214,359 @@ public void testMixDeleteAndInsert() throws IOExc

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1757391061 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -63,33 +60,43 @@ protected CloseableIterable doPlanFiles( return CloseableIter

Re: [PR] Support changelog scan for table with delete files [iceberg]

dramaticlly commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1755259576 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -63,33 +60,43 @@ protected CloseableIterable doPlanFiles( return Closeabl

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1753070517 ## gradle.properties: ## @@ -32,4 +32,4 @@ org.gradle.parallel=true org.gradle.configureondemand=true # explicitly disable the configuration cache org.gradle.config

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2335008435 I did not have time to come back to this work this week as I hoped, but I think I should have some time next week. -- This is an automated message from the Apache Git Service. To respo

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1747872923 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -63,33 +60,43 @@ protected CloseableIterable doPlanFiles( return CloseableIter

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1747869556 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/ChangelogRowReader.java: ## @@ -112,13 +149,62 @@ private CloseableIterable openChangelogScanTask(Ch

Re: [PR] Support changelog scan for table with delete files [iceberg]

manuzhang commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1740294635 ## gradle.properties: ## @@ -32,4 +32,4 @@ org.gradle.parallel=true org.gradle.configureondemand=true # explicitly disable the configuration cache org.gradle.con

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2313986585 @pvary and @dramaticlly, thank you both for reviewing. I am busy with some other work at the moment, but I'll return to this by next week. -- This is an automated message from the Apac

Re: [PR] Support changelog scan for table with delete files [iceberg]

dramaticlly commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1733659899 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/ChangelogRowReader.java: ## @@ -112,13 +149,62 @@ private CloseableIterable openChangelogScanTa

Re: [PR] Support changelog scan for table with delete files [iceberg]

dramaticlly commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1733661588 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -133,51 +131,149 @@ private static Map computeSnapshotOrdinals(Deque snapsh

Re: [PR] Support changelog scan for table with delete files [iceberg]

dramaticlly commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1733677814 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -63,33 +60,43 @@ protected CloseableIterable doPlanFiles( return Closeabl

Re: [PR] Support changelog scan for table with delete files [iceberg]

dramaticlly commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1733661588 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -133,51 +131,149 @@ private static Map computeSnapshotOrdinals(Deque snapsh

Re: [PR] Support changelog scan for table with delete files [iceberg]

dramaticlly commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1733661339 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -133,51 +131,149 @@ private static Map computeSnapshotOrdinals(Deque snapsh

Re: [PR] Support changelog scan for table with delete files [iceberg]

dramaticlly commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1733659899 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/ChangelogRowReader.java: ## @@ -112,13 +149,62 @@ private CloseableIterable openChangelogScanTa

Re: [PR] Support changelog scan for table with delete files [iceberg]

dramaticlly commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1733658370 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/ChangelogRowReader.java: ## @@ -112,13 +149,62 @@ private CloseableIterable openChangelogScanTa

Re: [PR] Support changelog scan for table with delete files [iceberg]

dramaticlly commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1733657338 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/ChangelogRowReader.java: ## @@ -112,13 +149,62 @@ private CloseableIterable openChangelogScanTa

Re: [PR] Support changelog scan for table with delete files [iceberg]

pvary commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2305694139 Could we add some tests for the `BaseIncrementalChangelogScan` directly? It would be good if we don't depend on Spark to test core functionality. Thanks, Peter -- This is an autom

Re: [PR] Support changelog scan for table with delete files [iceberg]

pvary commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1727847483 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -133,51 +131,149 @@ private static Map computeSnapshotOrdinals(Deque snapsh retu

Re: [PR] Support changelog scan for table with delete files [iceberg]

pvary commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1727828125 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -133,51 +131,149 @@ private static Map computeSnapshotOrdinals(Deque snapsh retu

Re: [PR] Support changelog scan for table with delete files [iceberg]

pvary commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1727796525 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -63,33 +60,43 @@ protected CloseableIterable doPlanFiles( return CloseableItera

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1727409290 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -63,33 +60,43 @@ protected CloseableIterable doPlanFiles( return CloseableIter

Re: [PR] Support changelog scan for table with delete files [iceberg]

pvary commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1726833603 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -63,33 +60,43 @@ protected CloseableIterable doPlanFiles( return CloseableItera

Re: [PR] Support changelog scan for table with delete files [iceberg]

pvary commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1726782294 ## core/src/main/java/org/apache/iceberg/BaseIncrementalChangelogScan.java: ## @@ -63,33 +60,43 @@ protected CloseableIterable doPlanFiles( return CloseableItera

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1726394464 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestChangelogReader.java: ## @@ -191,39 +214,361 @@ public void testMixDeleteAndInsert() throws IOExc

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2303850176 @pvary I have fixed the implementation so that existing deletes are applied before new deletes are emitted. I have fixed the test case accordingly. (I also renamed `testFlinkScenario1` a

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1726387609 ## spark/v3.5/spark/src/main/java/org/apache/iceberg/spark/source/ChangelogRowReader.java: ## @@ -112,13 +149,62 @@ private CloseableIterable openChangelogScanTask(Ch

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1726379602 ## data/src/main/java/org/apache/iceberg/data/DeleteFilter.java: ## @@ -197,31 +197,31 @@ record -> deleteSet.contains(projectRow.wrap(asStructLike(record))); }

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1726371909 ## gradle.properties: ## @@ -32,4 +32,4 @@ org.gradle.parallel=true org.gradle.configureondemand=true # explicitly disable the configuration cache org.gradle.config

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2302563661 @pvary thanks for posting on the dev list thread! In your description of scenario 1, you wrote: > > When the changelog scan reads the 3rd snapshot, it should consider:

Re: [PR] Support changelog scan for table with delete files [iceberg]

pvary commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2302465047 > @pvary thanks for reviewing the tests! Just a quick response for now. For scenario 1, the behavior is as I thought you described, yet in your comment on the test code, you indicated tha

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2302439337 @pvary thanks for reviewing the tests! Just a quick response for now. For scenario 1, the behavior is as I thought you described, yet in your comment on the test code, you indicated th

Re: [PR] Support changelog scan for table with delete files [iceberg]

flyrain commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2302430012 cc @dramaticlly -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. T

Re: [PR] Support changelog scan for table with delete files [iceberg]

pvary commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2302344676 > @pvary I have added `testFlinkScenario1` and `testFlinkScenario2` to `TestChangelogReader` based on your two scenarios. Please check the expected results. (I will rename the tests later

Re: [PR] Support changelog scan for table with delete files [iceberg]

pvary commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1725269441 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestChangelogReader.java: ## @@ -191,39 +214,361 @@ public void testMixDeleteAndInsert() throws IOExce

Re: [PR] Support changelog scan for table with delete files [iceberg]

pvary commented on code in PR #10935: URL: https://github.com/apache/iceberg/pull/10935#discussion_r1725175321 ## spark/v3.5/spark/src/test/java/org/apache/iceberg/spark/source/TestChangelogReader.java: ## @@ -191,39 +214,361 @@ public void testMixDeleteAndInsert() throws IOExce

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2297933397 @pvary I have added `testFlinkScenario1` and `testFlinkScenario2` to `TestChangelogReader`. Please check the expected results. (I will rename the tests later with more descriptive names.

Re: [PR] Support changelog scan for table with delete files [iceberg]

pvary commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2295126370 > For the positional delete case, I do not believe deleting the same position in the same data file again can happen I agree, that this should be the case. My only concern is, that

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2294958003 @pvary thank you for your interest and for the Flink scenarios, which is very helpful as I am unfamiliar with Flink. Regarding https://github.com/apache/iceberg/pull/9888, please r

Re: [PR] Support changelog scan for table with delete files [iceberg]

pvary commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2294754336 @wypoon, @manuzhang: I'm interested in providing Flink streaming CDC reads. That would require a working changelog scan planning in Iceberg core. So I would be happy to help with my revie

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2294451831 @aokolnychyi @flyrain @stevenzwu @szehon-ho can you please review this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitH

Re: [PR] Support changelog scan for table with delete files [iceberg]

wypoon commented on PR #10935: URL: https://github.com/apache/iceberg/pull/10935#issuecomment-2294429604 @manuzhang thank you for pointing me to your PR, https://github.com/apache/iceberg/pull/9888. I have tried testing it and left comments on your PR. -- This is an automated message fro