Thanks Vinoth for proposing a clean and extendable design. The overall design looks great. Another rollout option is to only use consolidated log index for index lookup if latest "valid" log block has been written in new format. If that is not the case, we can revert to scanning previous log blocks for index lookup. Balaji.V On Tuesday, October 29, 2019, 07:52:00 PM PDT, Bhavani Sudha <bhavanisud...@gmail.com> wrote: I vote for the second option. Also it can give time to analyze on how to deal with backwards compatibility. I ll take a look at the RFC later tonight and get back.
On Sun, Oct 27, 2019 at 10:24 AM Vinoth Chandar <vin...@apache.org> wrote: > One issue I have some open questions myself > > Is it ok to assume log will have old data block versions, followed by new > data block versions. For e.g, if rollout new code, then revert back then > there could be an arbitrary mix of new and old data blocks. Handling this > might make design/code fairly complex. Alternatively we can keep it simple > for now, disable by default and only advise to enable for new tables or > when hudi version is stable > > > On Sun, Oct 27, 2019 at 12:13 AM Vinoth Chandar <vin...@apache.org> wrote: > > > > > > https://cwiki.apache.org/confluence/display/HUDI/RFC-6+Add+indexing+support+to+the+log+file > > > > > > Feedback welcome, on this RFC tackling HUDI-86 > > >
