[ https://issues.apache.org/jira/browse/FLINK-18303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17136241#comment-17136241 ]
Jingsong Lee commented on FLINK-18303: -------------------------------------- First, I don't think this is a technical bug, but it can be a user oriented bug. The options control the "see the part file" for csv,json are: # rolling-policy.time-interval # bucket check interval # checkpoint interval As long as one of the three options is not set small, it will not achieve the desired effect. > Filesystem connector doesn't flush part files after rolling interval > -------------------------------------------------------------------- > > Key: FLINK-18303 > URL: https://issues.apache.org/jira/browse/FLINK-18303 > Project: Flink > Issue Type: Bug > Components: Connectors / FileSystem, Table SQL / Ecosystem > Reporter: Jark Wu > Assignee: Jark Wu > Priority: Major > Fix For: 1.11.0 > > > I have set "execution.checkpointing.interval" to "5s" and > "sink.rolling-policy.time-interval" to "2s". However, it still take 60 > seconds to see the first part file. > This can be reproduced by the following code in SQL CLI: > {code:sql} > CREATE TABLE CsvTable ( > event_timestamp STRING, > `user` STRING, > message STRING, > duplicate_count BIGINT, > constant STRING > ) WITH ( > 'connector' = 'filesystem', > 'path' = '$RESULT', > 'format' = 'csv', > 'sink.rolling-policy.time-interval' = '2s' > ); > INSERT INTO CsvTable -- read from Kafka Avro, and write into Filesystem Csv > SELECT AvroTable.*, RegReplace('Test constant folding.', 'Test', 'Success') > AS constant > FROM AvroTable; > {code} > This is found when I migrate SQLClientKafkaITCase to use DDL (FLINK-18086). -- This message was sent by Atlassian Jira (v8.3.4#803005)