fhueske commented on a change in pull request #6823: [FLINK-10134] UTF-16 support for TextInputFormat bug refixed URL: https://github.com/apache/flink/pull/6823#discussion_r225553146
########## File path: flink-core/src/main/java/org/apache/flink/api/common/io/DelimitedInputFormat.java ########## @@ -62,26 +64,41 @@ // Charset is not serializable private transient Charset charset; + /** + * The charset of bom in the file to process. + */ + private transient Charset bomCharset; + + /** + * The Map to record the BOM encoding of all files. + */ + private transient final Map<String, Charset> fileBomCharsetMap; + + /** + * The stepSize to record different encoding formats. + */ + protected transient int charsetStepSize = 1; Review comment: UTF-16 is variable length encoded. Hence, the number of bytes of a character depend on the character not only on the charset. I'd remove this variable and move it to `TextInputFormat` which uses it in a specific way. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services