Hi Chen, Table Filesystem/Hive sink file compaction has been merged into master, detail in [1]. It is included in Flink 1.12.
Hope you can have a try and test. [1]https://issues.apache.org/jira/browse/FLINK-19345 Best, Jingsong On Thu, Nov 19, 2020 at 2:31 PM Chen Qin <qinnc...@gmail.com> wrote: > Hi there, > > We are testing out writing Kafka to hive table as parquet format. > Currently, we have seen user has to choose to create lots of small files in > min level folder to gain latency benefits. I recall FF2020 Global folks > mentioned implement compaction logic during the checkpointing time. Wonder > how that goes? Love collaborate on this topic. > > Chen > Pinterest > -- Best, Jingsong Lee
