great proposal, +1 as well! Best, Gary
On Tue, Mar 22, 2022 at 4:20 AM Alexey Kudinkin <[email protected]> wrote: > Hello, everyone! > > @Surya, first of all, wanted to say that i think this is a great proposal! > > > A new compaction strategy can be added, but we thought it might > > complicate the existing logic and need to rely on some hacks, especially > > since Compaction action writes to a base file and places a .commit file > > upon completion. Whereas, in our use case we are not concerned with the > > base file at all, instead we are merging blocks and writing back to the > log > > file. So, we thought it is better to use a new action(called > > LogCompaction), which works at a log file level and writes back to the > log > > file. Since log files are in general added by deltacommit, upon > completion > > LogCompaction can place a .deltacommit. > > > Did I understand your proposal correctly, that you're suggesting that the > Minor compaction, unlike Major one, be performed as part of the Delta > Commit? > > I think Sagar's question is very valid from the perspective of configuring > and balancing major/minor compactions. Nevertheless, I think we can cover > and discuss it as part of the RFC process. > > > LogCompaction is not a replacement for regular compaction. LogCompaction > > is performed as a minor compaction so as to reduce the no. of log blocks > to > > consider. It does not consider base files while merging the log blocks. > To > > merge log files with base file Compaction action is still needed. By > using > > LogCompaction action frequently, the frequency with which we do full > scale > > compaction is reduced. > > Consider a scenario in which, after 'X' no. of LogCompaction actions, for > > some file groups the log files size becomes comparable to that of base > file > > size, in this scenario LogCompaction action is going to take close to the > > same amount of time as compaction action. So, now instead of > LogCompaction, > > full scale Compaction needs to be performed on those file groups. In > future > > we can also introduce logic to determine what is the right > > action(Compaction or LogCompaction) to be performed depending on the > state > > of the file group. > > > To be able to reason about strategy for both major/minor compactions we > need to clearly define the criteria we're optimizing on (is it # of RPC to > HDFS/Object Storage, I/O, etc). That would also help us to measure > objectively the performance improvements from the introduction of minor > compaction. > > On Mon, Mar 21, 2022 at 8:26 AM Vinoth Chandar <[email protected]> wrote: > > > +1 overall > > > > On Sat, Mar 19, 2022 at 5:02 PM Surya Prasanna < > [email protected] > > > > > wrote: > > > > > Hi Sagar, > > > Sorry for the delay in response. Thanks for the questions. > > > > > > 1. Trying to understand the main goal. Is it to balance the tradeoff > > > between read and write amplification for metadata table? Or is it > purely > > to > > > optimize for reads? > > > > On large tables, write amplification is a side effect of frequent > > > compactions. So, instead of increasing the frequency of full > compaction, > > we > > > are proposing minor compaction(LogCompaction) to be done frequently to > > > merge only the log blocks and write a new log block. By merging the > > blocks, > > > there are less no. of blocks to deal with during read, that way we are > > > optimizing for read performance and potentially avoiding the write > > > amplification problem. > > > > > > 2. Why do we need a separate action? Why can't any of the existing > > > compaction strategies (or a new one if needed) help to achieve this? > > > > A new compaction strategy can be added, but we thought it might > > > complicate the existing logic and need to rely on some hacks, > especially > > > since Compaction action writes to a base file and places a .commit file > > > upon completion. Whereas, in our use case we are not concerned with the > > > base file at all, instead we are merging blocks and writing back to the > > log > > > file. So, we thought it is better to use a new action(called > > > LogCompaction), which works at a log file level and writes back to the > > log > > > file. Since log files are in general added by deltacommit, upon > > completion > > > LogCompaction can place a .deltacommit. > > > > > > 3. Is the proposed LogCompaction a replacement for regular compaction > for > > > metadata table i.e. if LogCompaction is enabled then compaction cannot > be > > > done? > > > > LogCompaction is not a replacement for regular compaction. > > LogCompaction > > > is performed as a minor compaction so as to reduce the no. of log > blocks > > to > > > consider. It does not consider base files while merging the log blocks. > > To > > > merge log files with base file Compaction action is still needed. By > > using > > > LogCompaction action frequently, the frequency with which we do full > > scale > > > compaction is reduced. > > > Consider a scenario in which, after 'X' no. of LogCompaction actions, > for > > > some file groups the log files size becomes comparable to that of base > > file > > > size, in this scenario LogCompaction action is going to take close to > the > > > same amount of time as compaction action. So, now instead of > > LogCompaction, > > > full scale Compaction needs to be performed on those file groups. In > > future > > > we can also introduce logic to determine what is the right > > > action(Compaction or LogCompaction) to be performed depending on the > > state > > > of the file group. > > > > > > Thanks, > > > Surya > > > > > > > > > On Fri, Mar 18, 2022 at 11:22 PM Surya Prasanna Yalla <[email protected] > > > > > wrote: > > > > > > > > > > > > > > > ---------- Forwarded message --------- > > > > From: sagar sumit <[email protected]> > > > > Date: Wed, Mar 16, 2022 at 11:17 PM > > > > Subject: Re: [DISCUSS] New RFC to create LogCompaction action for MOR > > > > tables? > > > > To: <[email protected]> > > > > > > > > > > > > Hi Surya, > > > > > > > > This is a very interesting idea! I'll be looking forward to RFC. > > > > > > > > I have a few high-level questions: > > > > > > > > 1. Trying to understand the main goal. Is it to balance the tradeoff > > > > between read and write amplification for metadata table? Or is it > > purely > > > to > > > > optimize for reads? > > > > 2. Why do we need a separate action? Why can't any of the existing > > > > compaction strategies (or a new one if needed) help to achieve this? > > > > 3. Is the proposed LogCompaction a replacement for regular compaction > > for > > > > metadata table i.e. if LogCompaction is enabled then compaction > cannot > > be > > > > done? > > > > > > > > Regards, > > > > Sagar > > > > > > > > On Thu, Mar 17, 2022 at 12:51 AM Surya Prasanna < > > > > [email protected]> > > > > wrote: > > > > > > > > > Hi Team, > > > > > > > > > > > > > > > Record level index uses a metadata table which is a MOR table type. > > > > > > > > > > Each delta commit in the metadata table creates multiple hfile log > > > blocks > > > > > and so to read them multiple file handles have to be opened which > > might > > > > > cause issues in read performance. To reduce the read performance, > > > > > compaction can be run frequently which basically merges all the log > > > > blocks > > > > > to base file and creates another version of base file. If this is > > done > > > > > frequently, it would cause write amplification. > > > > > > > > > > Instead of merging all the log blocks to base file and doing a full > > > > > compaction, minor compaction can be done which basically merges log > > > > blocks > > > > > and creates one new log block. > > > > > > > > > > This can be achieved by adding a new action to Hudi called > > > LogCompaction > > > > > and requires a RFC. Please let me know what you think. > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Surya > > > > > > > > > > > > > > >
