prathik-onehouse opened a new pull request, #20572:
URL: https://github.com/apache/hudi/pull/20572

   ### Describe the issue this Pull Request addresses
   
   closes #19770
   
   A row-writer bulk insert `INSERT OVERWRITE` in dynamic overwrite mode on a 
table with `hoodie.populate.meta.fields=false` skips the pending clustering 
guard. `prepareForBulkInsert` stubs the meta columns as null in that case, so 
`DatasetBulkInsertOverwriteCommitActionExecutor#resolveTargetPartitions` reads 
no real partitions from `_hoodie_partition_path`, finds no file groups to 
replace, and the overwrite replaces file groups that a pending clustering plan 
owns. Rows can be lost once that plan commits.
   
   ### Summary and Changelog
   
   When meta fields are not populated, the dynamic arm of 
`resolveTargetPartitions` now derives the partitions with the key generator the 
row writer uses to route each row (new 
`HoodieDatasetBulkInsertHelper#getDistinctPartitionPaths`), so the guard 
rejects an overwrite that overlaps pending clustering. 
`BaseDatasetBulkInsertCommitActionExecutor#execute` reads 
`writeConfig.populateMetaFields()` instead of the raw config key, matching 
`prepareForBulkInsert` and the writer. The pending clustering SQL test in 
`TestInsertTable2` now runs both overwrite modes with and without meta fields; 
without the fix, the dynamic case with meta fields off lets the overlapping 
overwrite through.
   
   ### Impact
   
   Overwrites that overlap pending clustering on tables without meta fields now 
fail with the same "Not allowed to update the clustering file group" error as 
tables with meta fields. When clustering is pending, the guard runs one extra 
Spark job over the prepared rows to collect their partitions.
   
   ### Risk Level
   
   low. The new path only runs for dynamic overwrite with meta fields off while 
clustering is pending, and it uses the same key generator the writer already 
uses.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] fix(spark): reject bulk ins... prathik-onehouse (via GitHub) via commits

Reply via email to