[ https://issues.apache.org/jira/browse/FLINK-16818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068413#comment-17068413 ]
Jingsong Lee commented on FLINK-16818: -------------------------------------- Hi [~zhangjun], actually Spark not do shuffle before sink, but Flink add a shuffle before sink, so there is only a single task to write file in Flink. After FLIP-115, will add a config option to control this, and we will think about "default not shuffle before sink". Another thing is 10 G file, there is no rolling policy for Flink batch sink, IMO, it doesn't so matter, but we will add rolling policy in FLIP-115 too. > Optimize data skew when flink write data to hive dynamic partition table > ------------------------------------------------------------------------ > > Key: FLINK-16818 > URL: https://issues.apache.org/jira/browse/FLINK-16818 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive > Affects Versions: 1.10.0 > Environment: {code:java} > {code} > Reporter: Jun Zhang > Priority: Major > Fix For: 1.11.0 > > > I read the source table data of hive through flink sql, and then write the > target table of hive. The target table is a partitioned table. When the data > of a partition is particularly large, data skew occurs, resulting in a > particularly long execution time. > By default Configuration, the same sql, hive on spark takes five minutes, and > flink takes about 40 minutes. > example: > > {code:java} > // the schema of myparttable > name string, > age int, > PARTITIONED BY ( > type string, > day string > ) > INSERT OVERWRITE myparttable SELECT name, age, type,day from sourcetable; > {code} > -- This message was sent by Atlassian Jira (v8.3.4#803005)