[
https://issues.apache.org/jira/browse/FLINK-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15477387#comment-15477387
]
ASF GitHub Bot commented on FLINK-3921:
---------------------------------------
Github user greghogan commented on a diff in the pull request:
https://github.com/apache/flink/pull/2060#discussion_r78203358
--- Diff:
flink-core/src/main/java/org/apache/flink/types/parser/FieldParser.java ---
@@ -75,8 +76,30 @@
/** Invalid Boolean value **/
BOOLEAN_INVALID
}
-
+
+ private Charset charset = Charset.forName("US-ASCII");
+
private ParseErrorState errorState = ParseErrorState.NONE;
+
+ /**
+ * Parses the value of a field from the byte array.
+ * The start position within the byte array and the array's valid
length is given.
+ * The content of the value is delimited by a field delimiter.
+ *
+ * @param bytes The byte array that holds the value.
+ * @param startPos The index where the field starts
+ * @param limit The limit unto which the byte contents is valid for the
parser. The limit is the
+ * position one after the last valid byte.
+ * @param delim The field delimiter character
+ * @param reuse An optional reusable field to hold the value
+ * @param charset The charset to parse with
+ *
+ * @return The index of the next delimiter, if the field was parsed
correctly. A value less than 0 otherwise.
+ */
+ public int parseField(byte[] bytes, int startPos, int limit, byte[]
delim, T reuse, Charset charset){
+ this.charset = charset;
--- End diff --
Is this method needed? `GenericCsvInputFormat.open` can `setCharset` on
each newly instantiated `FieldParser`, and in the case where a user decided to
change charset on an open file `GenericCsvInputFormat.setCharset` could go
through and `setCharset` on the list of `FieldParser`s.
> StringParser not specifying encoding to use
> -------------------------------------------
>
> Key: FLINK-3921
> URL: https://issues.apache.org/jira/browse/FLINK-3921
> Project: Flink
> Issue Type: Improvement
> Components: Core
> Affects Versions: 1.0.3
> Reporter: Tatu Saloranta
> Assignee: Rekha Joshi
> Priority: Trivial
>
> Class `flink.types.parser.StringParser` has javadocs indicating that contents
> are expected to be Ascii, similar to `StringValueParser`. That makes sense,
> but when constructing actual instance, no encoding is specified; on line 66
> f.ex:
> this.result = new String(bytes, startPos+1, i - startPos - 2);
> which leads to using whatever default platform encoding is. If contents
> really are always Ascii (would not count on that as parser is used from CSV
> reader), not a big deal, but it can lead to the usual Latin-1-VS-UTF-8 issues.
> So I think that encoding should be explicitly specified, whatever is to be
> used: javadocs claim ascii, so could be "us-ascii", but could well be UTF-8
> or even ISO-8859-1.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
