Hi all, One thing to note is that, the CSVBulkReader does not support the splittable property. Previously with TextInputFormat we were able to use the block size to split them, but in Streaming world this is not there.
Regards Ram On Wed, Mar 8, 2023 at 7:22 AM yuxia <luoyu...@alumni.sjtu.edu.cn> wrote: > Hi, as the doc said: > 'The BulkFormat reads and decodes batches of records at a time.' So, the > bulk is not binded to column format, the bulk writer for csv is indeed > implemented in the Flink code. Actaully, you can use either Row or Bulk > depending on what style you would like to write data. > > As for the doc missing for CSV BulkFormat and not public in flink-csv, I > really don't know why. I guess the reason maybe Flink won't expose it the > datastream api, but only expose to table api. > > Best regards, > Yuxia > > ------------------------------ > *发件人: *"User" <user@flink.apache.org> > *收件人: *"User" <user@flink.apache.org> > *发送时间: *星期二, 2023年 3 月 07日 下午 7:35:47 > *主题: *CSV File Sink in Streaming Use Case > > Hi, > > I am working on a Java DataStream application and need to implement a File > sink with CSV format. > > I see that I have two options here - Row and Bulk ( > https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/datastream/filesystem/#format-types-1 > ) > > So for CSV file distribution which one should I use? Row or Bulk? > > I think the documentation is confusing for File connectors. Because I can > see an example for PyFlink which uses a BulkWriter for CSV. But the same > class is not public in flink-csv. So does Flink not support CSVBulkWriter > for Java? > > And for Table API File sink explicitly supports CSV for Row format. But > fails to mention anything about CSV in DataStream File sink. > > This all is just really confusing. Any leads on this are much appreciated. > > Thanks > >
