[
https://issues.apache.org/jira/browse/FLINK-27557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jane Chan updated FLINK-27557:
------------------------------
Description:
Currently, FileStoreWrite will scan and plan files when creating a non-empty
writer. We should also create a non-empty writer for non-rescale compaction
cases but instead use the pre-planned manifest entries.
The sink records should be dumped, and when prepareCommit is invoked, it should
trigger submitCompaction. This will generate two snapshots, the first commit
kind is "ADD" with empty new files, and the second commit kind is "COMPACT"
with specified manifest entries marked as deleted and compacted files marked as
added.
was:Currently, FileStoreWrite will scan and plan files when creating a
non-empty writer. We should also create a non-empty writer for non-rescale
compaction cases but use the pre-planned manifest entries instead.
> Let FileStoreWrite accept pre-planned manifest entries and let
> MergeTreeWriter trigger compaction when commit
> -------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-27557
> URL: https://issues.apache.org/jira/browse/FLINK-27557
> Project: Flink
> Issue Type: Sub-task
> Components: Table Store
> Affects Versions: table-store-0.2.0
> Reporter: Jane Chan
> Priority: Major
> Fix For: table-store-0.2.0
>
> Attachments: normal-compaction.png, optimized-normal-compaction.png,
> rescale-bucket-compaction.png
>
>
> Currently, FileStoreWrite will scan and plan files when creating a non-empty
> writer. We should also create a non-empty writer for non-rescale compaction
> cases but instead use the pre-planned manifest entries.
> The sink records should be dumped, and when prepareCommit is invoked, it
> should trigger submitCompaction. This will generate two snapshots, the first
> commit kind is "ADD" with empty new files, and the second commit kind is
> "COMPACT" with specified manifest entries marked as deleted and compacted
> files marked as added.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
