deniskuzZ opened a new pull request, #6924: URL: https://github.com/apache/hive/pull/6924
### What changes were proposed in this pull request? - HiveStorageHandler#acidDistributeColumns: how the deleted records of the rewritten query are distributed - V3: DISTRIBUTE BY FILE__PATH, no SORT BY, so all deletes of a data file reach one DV writer - MERGE with UPDATE and DELETE clauses on V3: one delete branch; the update's insert branch is not counted, rows affected unchanged - HiveStorageHandler#configureJobConf(FileSinkDesc, JobConf): called for every file sink while the job is built - Iceberg plans the previous deletes of a V3 merge-on-read delete sink there, once per job, into a gzipped file, instead of a full-table planFiles per task - The file path is unique per job build and recorded in `iceberg.mr.rewritable.deletes.<table>`; writers load it once per daemon through the query ObjectCache, and a DV writer without it fails - Commit and abort delete the recorded file through the table's FileIO - Stacked on HIVE-30111 ### Why are the changes needed? Bug-Fix: 1TRC V3 DML wrote 2,299 duplicate DVs and ran HS2 out of memory at commit (heap 0.37 -> 11.7 GB in 40 s). After: 17,016 DVs for 17,016 files, HS2 peak 5.15 GB ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? - TestHiveIcebergDeletionVectors, TestHiveIcebergRewritableDeletes - iceberg_v3_dml_distribution.q (plans of V3 DELETE, UPDATE, MERGE), iceberg_mv_rebuild_dv_per_file.q -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
