[
https://issues.apache.org/jira/browse/FLINK-7274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16102902#comment-16102902
]
Fabian Hueske commented on FLINK-7274:
--------------------------------------
[RFC 4180|https://tools.ietf.org/html/rfc4180] says
> Each field may or may not be enclosed in double quotes...
So, yes according to this, quotes should be OK. On the other hand, CSV is a
pretty liberal format and everybody treats it a bit differently.
The CsvInputFormat was never designed to adhere to the RFC specification. The
focus was rather on efficiency (via tight integration with Flink's interfaces)
than feature completeness. Therefore, the parsing logic is a custom
implementation and does not rely on a CSV library.
However, there is some recent work on an RFC compliant CsvTableSource that
should become available soon (see FLINK-7050)
> ParserError NUMERIC_VALUE_FORMAT_ERROR
> --------------------------------------
>
> Key: FLINK-7274
> URL: https://issues.apache.org/jira/browse/FLINK-7274
> Project: Flink
> Issue Type: Bug
> Affects Versions: 1.3.1
> Reporter: Egor Litvinenko
> Labels: csvparser
> Attachments: Untitled 1.csv
>
>
> {code:java}
> DataSet<Row> dataSet = env
> .readCsvFile("/file/test-data.csv")
> .fieldDelimiter(",")
> .parseQuotedStrings('"')
> .ignoreFirstLine()
> .types(String.class, Double.class, Double.class,
> Double.class, Double.class)
> {code}
> {code:log}
> Caused by: org.apache.flink.api.common.io.ParseException: Line could not be
> parsed:
> '"1950-01-01","73.20101635771319","87.25023810870184","36.0149972876981","46.43200584961114"'
> ParserError NUMERIC_VALUE_FORMAT_ERROR
> Expect field types: class java.lang.String, class java.lang.Double, class
> java.lang.Double, class java.lang.Double, class java.lang.Double
> {code}
> Test data example:
> "ID","F1","F2","F3","F4"
> "1950-01-01","73.20101635771319","87.25023810870184","36.0149972876981","46.43200584961114"
> "1950-01-02","22.265361054145394","57.02164143464855","67.24219049572051","43.058275223048035"
> "1950-01-03","45.674551461704915","86.35170144091485","16.18842554618568","6.748071385147735"
> "1950-01-04","8.890850738221644","20.490727535158946","58.32831367590852","17.916755029167952"
> "1950-01-05","38.07336923931018","27.223155544419697","92.67895969507504","60.027033750000335"
> If generate this data without quote char, it will be fine.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
