[
https://issues.apache.org/jira/browse/FLINK-27557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jane Chan updated FLINK-27557:
------------------------------
Summary: Trigger compaction when prepareCommit is triggered (was: Let
FileStoreWrite accept pre-planned manifest entries and let MergeTreeWriter
trigger compaction when commit)
> Trigger compaction when prepareCommit is triggered
> --------------------------------------------------
>
> Key: FLINK-27557
> URL: https://issues.apache.org/jira/browse/FLINK-27557
> Project: Flink
> Issue Type: Sub-task
> Components: Table Store
> Affects Versions: table-store-0.2.0
> Reporter: Jane Chan
> Priority: Major
> Fix For: table-store-0.2.0
>
> Attachments: normal-compaction.png, optimized-normal-compaction.png,
> rescale-bucket-compaction.png
>
>
> Currently, FileStoreWrite will scan and plan files when creating a non-empty
> writer. We should also create a non-empty writer for non-rescale compaction
> cases but instead use the pre-planned manifest entries.
> The sink records should be dumped, and when prepareCommit is invoked, it
> should trigger submitCompaction. This will generate two snapshots, the first
> commit kind is "ADD" with empty new files, and the second commit kind is
> "COMPACT" with specified manifest entries marked as deleted and compacted
> files marked as added.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
