[ https://issues.apache.org/jira/browse/FLINK-27526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jane Chan updated FLINK-27526: ------------------------------ Description: Currently, TableStore does not support changing the number of the bucket (denoted by config option {{{}table-storage.bucket{}}}) once the managed table is created. The reason is that the LSM tree is built under bucket level, and thus writing with different bucket numbers will cause the same record to be hashed to another bucket and leads to data corruption. In the release-0.1, TableStore will detect this change and throw an exception when scanning. See FLINK-27316. However, this is not flexible and user-friendly. To be more specific, if the bucket number remains unchanged, the number of files under each bucket will grow fast as time passes, slowing down the scan speed to restore the LSM tree and finally influencing read and write latency. In this ticket, we aim to support changing the bucket number via {{ALTER TABLE SET (...)}} and utilize the compaction mechanism to provide a way for users to reorganize the existing data layout. {code:sql} -- alter catalog metadata ALTER TABLE managed_table SET('bucket' = 'bucket-num'); -- alter configuration under session level, the successive compact operations will compact data by the new bucket number SET 'table-store.compact-rescale' = 'true'; ALTER TABLE managed_table PARTITION partition_spec[, PARTITION partition_spec, ...] COMPACT; -- per command setting ALTER TABLE managed_table /*+OPTIONS('compact-rescale' = 'true')*/ PARTITION partition_spec[, PARTITION partition_spec, ...] COMPACT; {code} was: Currently, TableStore does not support changing the number of the bucket (denoted by config option {{{}table-storage.bucket{}}}) once the managed table is created. The reason is that the LSM tree is built under bucket level, and thus writing with different bucket numbers will cause the same record to be hashed to another bucket and leads to data corruption. In the release-0.1, TableStore will detect this change and throw an exception when scanning. See FLINK-27316. However, this is not flexible and user-friendly. To be more specific, if the bucket number remains unchanged, the number of files under each bucket will grow fast as time passes, slowing down the scan speed to restore the LSM tree and finally influencing read and write latency. In this ticket, we aim to support changing the bucket number via {{ALTER TABLE SET (...)}} and provide a way for users to reorganize the existing data layout. {code:sql} -- alter catalog metadata ALTER TABLE managed_table SET('bucket' = 'bucket-num'); -- alter configuration under session level, the successive compact operations will compact data by the new bucket number SET 'table-store.compact-rescale' = 'true'; ALTER TABLE managed_table PARTITION partition_spec[, PARTITION partition_spec, ...] COMPACT; -- per command setting ALTER TABLE managed_table /*+OPTIONS('compact-rescale' = 'true')*/ PARTITION partition_spec[, PARTITION partition_spec, ...] COMPACT; {code} > Support scaling bucket number for FileStore > ------------------------------------------- > > Key: FLINK-27526 > URL: https://issues.apache.org/jira/browse/FLINK-27526 > Project: Flink > Issue Type: New Feature > Components: Table Store > Affects Versions: table-store-0.2.0 > Reporter: Jane Chan > Assignee: Jane Chan > Priority: Major > Fix For: table-store-0.2.0 > > > Currently, TableStore does not support changing the number of the bucket > (denoted by config option {{{}table-storage.bucket{}}}) once the managed > table is created. The reason is that the LSM tree is built under bucket > level, and thus writing with different bucket numbers will cause the same > record to be hashed to another bucket and leads to data corruption. In the > release-0.1, TableStore will detect this change and throw an exception when > scanning. See FLINK-27316. > However, this is not flexible and user-friendly. To be more specific, if the > bucket number remains unchanged, the number of files under each bucket will > grow fast as time passes, slowing down the scan speed to restore the LSM tree > and finally influencing read and write latency. > In this ticket, we aim to support changing the bucket number via {{ALTER > TABLE SET (...)}} and utilize the compaction mechanism to provide a way for > users to reorganize the existing data layout. > > {code:sql} > -- alter catalog metadata > ALTER TABLE managed_table SET('bucket' = 'bucket-num'); > -- alter configuration under session level, the successive compact operations > will compact data by the new bucket number > SET 'table-store.compact-rescale' = 'true'; > ALTER TABLE managed_table PARTITION partition_spec[, PARTITION > partition_spec, ...] COMPACT; > -- per command setting > ALTER TABLE managed_table /*+OPTIONS('compact-rescale' = 'true')*/ PARTITION > partition_spec[, PARTITION partition_spec, ...] COMPACT; > {code} -- This message was sent by Atlassian Jira (v8.20.7#820007)