hudi-bot opened a new issue, #15115:
URL: https://github.com/apache/hudi/issues/15115
Currently, block-merging is configurable to be either lazy or non-lazy.
However non-lazy sequence is incorrect – it will be merging blocks before
actually rolling back reverted ones. To make sure users do not accidentally hit
this issue, we need to revisit MOR block merging sequence and make sure that
following invariants are upheld
# Blocks are merged in 2 passes:
## First we load all blocks, while dropping rolled back ones, then
## We merge them in another forward-pass
# We should try to avoid having 2 merging sequences and instead consolidate
on just one: right now we have "block + block", and "base + block", but we
should be able to just get away with just the latter (this will simplify
merging sequence quite substantially, for ex in respect to handling of
deletions)
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-3828
- Type: Bug
- Fix version(s):
- 1.1.0
---
## Comments
18/Jan/23 20:33;alexey.kudinkin;This is addressed in the new scanV2
implementation;;;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
