mxm opened a new pull request, #15996: URL: https://github.com/apache/iceberg/pull/15996
In order to get rid of equality deletes, we first need to be able to resolving existing equality delete files into positional deletes. For this PR, we opted to use deletion vectors over regular delete files due to their efficiency in terms of space and lookup. The idea is to allow writers to write equality deletes like they normally due (e.g. via `IcebergSink`), but they commit to a staging branch first. The maintenance task monitors the staging branch, builds an index from the equality fields values of the data files, then converts the equality delete files into DVs, and commits everything back to the main branch, where it can be read efficiently using only positional deletes. This workflow will require additional changes to `IcebergSink`, to write data to the staging branch, instead of the main branch and run the here-added `ConvertEqualityDeletes` maintenance task in-line, via its post commit topology. I was tempted to add this right-away, but decided to keep the scope limited to the maintenance task only. The maintenance task topology looks like this: ``` Planner (p=1) -> Reader (p=N) -> Worker (p=N, keyed) -> DVResolver (p=1) -> DVMerger (p=N) -> Committer (p=1) ``` ## Limitations For keeping the scope somewhat limited, I skipped these: - Row lineage is not preserved (can be handled in a follow-up) - No staging branch lifecycle management, e.g. create / clean up branch (will be added with the integration into IcebergSink) - DVResolver does a full manifest scan (no partition pruning) ## Follow-ups Those, I'm planning to work on next: - Integrate ConvertEqualityDeletes maintenance task as in-line compaction in IcebergSink - Clean up processed snapshots on the staging branch - Add documentation ## Commits The breakdown is as follows: 1. 842751804648ee384f7089bc8d2fa34f9a0ea9bd Data model and key serialization - data exchange types and key serialization 2. 5928dc24f34c185f8efad8f4c28363fe26cde96e Operators - file reader and keyed PK index worker with buffering 3. e5285ad03785dda0339cc0534be6ef433ead9b54 DV resolution and writing - group positions by data file, merge with existing DVs 4. 691e20979ae3d73f88f09d3dde502ea9c884ffb4 Planner - staging branch scanning, orchestration, reindex detection 5. 4fb06084e9387638f2dbd137242ec3ee96e4d061 Committer - RowDelta assembly, idempotency via snapshot properties, V3 enforcement 6. 76ab1d996fc02bacb4d102568252bf6d82c89286 API and integration tests - ConvertEqualityDeletes builder, end-to-end tests These could be put into separate PRs, if needed. ## Metrics Under the maintenance metric group: - Planner: processedEqDeleteFileNum, processedStagingSnapshotNum, skippedNoOpCycles, reindexCount - Worker: indexedKeyNum, resolvedDeleteNum - Committer: addedDvNum, commitDurationMs, addedDataFileNum, addedDataFileSize, error ## References 1. Design document: https://docs.google.com/document/d/1Jz4Fjt-6jRmwqbgHX_u0ohuyTB9ytDzfslS7lYraIjk/edit?tab=t.5qlfbj9bk8i3 (This PR implements the "use lock to avoid conflicts" solution) 2. Mailing list discussion: https://lists.apache.org/thread/gldqrycmo24r0vo77wzcj9s9b04rtvy7 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
