[ https://issues.apache.org/jira/browse/FLINK-27557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17539352#comment-17539352 ]
Jane Chan edited comment on FLINK-27557 at 5/20/22 2:59 AM: ------------------------------------------------------------ {quote}Can we just use overwrite? * overwrite specific manifest entries. * overwrite whole table or some whole partition. (eg: when rescale in compaction){quote} Overwrite means the writer cannot accept the specified manifest entries as restored files, so how to pass them to FileStoreCommit as compact before(mark as delete) is a new question. Beyond that, overwrite corresponds to the commit kind "OVERWRITE", but it should be more suitable to use "COMPACT" in this situation. I rethink it and maybe we don't need to generate the new files for the normal compaction. * The simplest way is to build the levels from the restored files, and don't sink record to MemTable, and submit compaction when precommit is invoked. * In the future, maybe we can introduce a new dedicated compaction source to restore LSM, perform compaction and commit, which avoids the shuffle cost. was (Author: qingyue): {quote}Can we just use overwrite? * overwrite specific manifest entries. * overwrite whole table or some whole partition. (eg: when rescale in compaction){quote} Overwrite means the writer cannot accept the specified manifest entries as restored files, so how to pass them to FileStoreCommit as compact before(mark as delete) is a new question. Beyond that, overwrite corresponds to the commit kind "OVERWRITE", but it should be more suitable to use "COMPACT" in this situation. I rethink it and maybe we don't need to generate the new files for the normal compaction. * The simplest way is to build the levels from the restored files, and don't sink record to MemTable, and submit compaction when precommit is invoked. * In the future, maybe we can introduce a new dedicated compaction source to restore LSM, perform compaction and commit, which avoids the shuffle cost. !normal-compaction.png! !rescale-bucket-compaction.png! !optimized-normal-compaction.png! > Create the empty writer for 'ALTER TABLE ... COMPACT' > ----------------------------------------------------- > > Key: FLINK-27557 > URL: https://issues.apache.org/jira/browse/FLINK-27557 > Project: Flink > Issue Type: Sub-task > Components: Table Store > Affects Versions: table-store-0.2.0 > Reporter: Jane Chan > Priority: Major > Fix For: table-store-0.2.0 > > Attachments: normal-compaction.png, optimized-normal-compaction.png, > rescale-bucket-compaction.png > > > Currently, FileStoreWrite only creates an empty writer for the \{{INSERT > OVERWRITE}} clause. We should also create the empty writer for manual > compaction. -- This message was sent by Atlassian Jira (v8.20.7#820007)