Hello,

I am using the StreamingFileSink BulkFormat in my Flink stream processing
job to output parquet files to S3. Now the
StreamingFileSink.BulkFormatBuilder
<https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/streaming/api/functions/sink/filesystem/StreamingFileSink.BulkFormatBuilder.html>,
does not have an option to set a custom rolling policy. It will roll the
files whenever the checkpoint triggers
<https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/streaming/api/functions/sink/filesystem/rollingpolicies/OnCheckpointRollingPolicy.html>.
It would be better to have a rolling policy based on both *size* and *time*.
One option is to write our own StreamingFileSink, which does accept a
custom rolling policy, but I suspect there might be some reason for this
behaviour.
I would like to get the opinion of Flink experts on this. And if there are
any potential workarounds to get the desired behaviour.

Kind regards
Ayush Verma

Reply via email to