We just added this feature to 1.12 [1][2], it would be great that you can download the 1.12 RC to test it out, and give us some feedback.
In case you will wonder why I linked 2 jiras, it's because both FileSystem & Hive connector share the same option options and also the implementations. [1] https://issues.apache.org/jira/browse/FLINK-19875 [2] https://issues.apache.org/jira/browse/FLINK-19886 Best, Kurt On Thu, Nov 19, 2020 at 2:31 PM Chen Qin <qinnc...@gmail.com> wrote: > Hi there, > > We are testing out writing Kafka to hive table as parquet format. > Currently, we have seen user has to choose to create lots of small files in > min level folder to gain latency benefits. I recall FF2020 Global folks > mentioned implement compaction logic during the checkpointing time. Wonder > how that goes? Love collaborate on this topic. > > Chen > Pinterest >
