现象: CREATE TABLE test.xxx_5min ( ......
) PARTITIONED BY (dt string , hm string) stored as orc TBLPROPERTIES( 'sink.partition-commit.trigger'='process-time', 'sink.partition-commit.delay'='5 min', 'sink.partition-commit.policy.kind'='metastore,success-file', 'sink.rolling-policy.file-size'='128MB', 'sink.rolling-policy.check-interval' ='30s', 'sink.rolling-policy.rollover-interval'='5min' ); dt = FROM_UNIXTIME((UNIX_TIMESTAMP()/300 * 300) ,'yyyyMMdd') hm = FROM_UNIXTIME((UNIX_TIMESTAMP()/300 * 300) ,'HHmm') 5min产生一个分区, ,checkpoint频率:30s 问题: 1.flink 1.11 steaming写入为什么是1min产生一个文件,而且文件大小没有到128M,如果参数sink.rolling-policy.rollover-interval'='5min 文件滚动时间 5min 滚动大小128M生效的话,就不应该产生这么小的问题,文件大小没有按照预期控制,为啥? 2.小文件问题该如何解决?有什么好的思路 3. 标记文件_Success文件为啥上报延迟? 如果是 12:30的分区,5min的分区,理论上应该12:35左右的时候就应该提交partition?