prathik-onehouse opened a new pull request, #20573:
URL: https://github.com/apache/hudi/pull/20573

   ### Describe the issue this Pull Request addresses
   
   closes #19635
   
   `sync_validate` fails or reports a misleading error in three cases. The 
catch-up count rebuilt every catch-up instant as a commit from its timestamp, 
so a MOR target (deltacommits) or a replacecommit (insert overwrite, 
clustering) in the catch-up range failed with "Cannot find the instant". In 
mode `complete`, a failed Hive JDBC connection was replaced by a 
`NullPointerException` from closing the null connection in the `finally` block, 
hiding the real `SQLException`. A `sourceLatestCommit != null` check could 
never be false.
   
   ### Summary and Changelog
   
   The catch-up count now reads each catch-up instant as it is on the timeline, 
with its own action, through `TimelineUtils.getCommitMetadata`, and only counts 
completed instants, so a write still in flight is not read as if it had 
committed. `countRecords` closes the result set and the connection only when 
they were opened, so the `SQLException` reaches the user. The dead null check 
is removed. `TestValidateHoodieSyncProcedure` covers a MOR target, a 
replacecommit in the catch-up range, a pending instant in the catch-up range, 
and both record-count modes on an unusable HiveServer2 URL. Without the fix 
those four tests fail and the other four pass.
   
   ### Impact
   
   `sync_validate` now returns a catch-up count for MOR targets and for 
catch-up ranges with replacecommits instead of failing, and a Hive connection 
failure in mode `complete` is reported as the `SQLException`. No config or API 
change.
   
   ### Risk Level
   
   low. The change is limited to the `sync_validate` procedure; plain commit 
catch-up behaves as before.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] fix(spark): count sync_vali... prathik-onehouse (via GitHub) via commits

Reply via email to