[
https://issues.apache.org/jira/browse/FLINK-35152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
tumengyao updated FLINK-35152:
------------------------------
Summary: Flink CDC Doris Sink Auto create table event should support
setting auto partition fields for each table (was: Flink CDC Doris/Starrocks
Sink Auto create table event should support setting auto partition fields for
each table)
> Flink CDC Doris Sink Auto create table event should support setting auto
> partition fields for each table
> ---------------------------------------------------------------------------------------------------------
>
> Key: FLINK-35152
> URL: https://issues.apache.org/jira/browse/FLINK-35152
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
> Affects Versions: 3.1.0
> Reporter: tumengyao
> Priority: Minor
> Labels: Doris
>
> In some scenarios, when creating a physical table in Doris, appropriate
> partition fields need to be selected to speed up the efficiency of data query
> and calculation. In addition, partition tables support more applications,
> such as hot and cold data layering and so on.
> The current Flink CDC Doris Sink's create table event creates a table with no
> partitions set.
> The Auto Partition function supported by doris 2.1.x simplifies the creation
> and management of partitions. We just need to add some configuration items to
> the Flink CDC job. To tell Flink CDC which fields Doris Sink will use in the
> create table event to create partitions, you can get a partition table in
> Doris.
> Here's an example:
> source: Mysql
> source_table:
> CREATE TABLE table1 (
> col1 INT AUTO_INCREMENT PRIMARY KEY,
> col2 DECIMAL(18, 2),
> col3 VARCHAR(500),
> col4 TEXT,
> col5 DATETIME DEFAULT CURRENT_TIMESTAMP
> );
> If you want to specify the partition of table test.table1, you need to add
> sink-table-partition-keys , sink-table-partition-type information ,...., to
> mysql_to_doris.yaml
> route:
> source-table: test.table1
> sink-table:ods.ods_table1
> sink-table-partition-key:col5
> sink-table-partition-func-call-expr:date_trunc(`col5`, 'month')
> sink-table-partition-type:auto range
> The auto range partition in Doris 2.1.x does not support null partitions. So
> you need to set test.table1.col5 == null then '1990-01-01 00:00:00' else
> test.table1.col5 end
> Now after submitting the mysql_to_doris.ymal Flink CDC job, an ods.ods_table1
> data table should appear in the Doris database
> The data table DDL is as follows:
> CREATE TABLE table1 (
> col1 INT ,
> col5 DATETIME not null,
> col2 DECIMAL(18, 2),
> col3 VARCHAR(500),
> col4 STRING
> ) unique KEY(`col1`,`col5`)
> AUTO PARTITION BY RANGE date_trunc(`col5`, 'month')()
> DISTRIBUTED BY HASH (`id`) BUCKETS AUTO
> PROPERTIES (
> ...
> );
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
