[
https://issues.apache.org/jira/browse/SPARK-26016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16782542#comment-16782542
]
Maxim Gekk commented on SPARK-26016:
------------------------------------
[~srowen] JSON datasource uses Text datasource in schema inferring to split
text input by lines. For that, line separator should be converted to a correct
sequence of bytes and passed to Hadoop's Line Reader. Here we need correct
encoding in Text datasource. Text datasource itself does copy bytes from Hadoop
Line Reader to InternalRow:
[https://github.com/apache/spark/blob/36a2e6371b4d173c3e03cc0d869c39335a0d7682/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/text/TextFileFormat.scala#L135]
. After that, the bytes are read by Jackson parser from streams decoded
according to encoding:
[https://github.com/apache/spark/blob/5264164a67df498b73facae207eda12ee133be7d/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/CreateJacksonParser.scala#L88-L93]
The PR you pointed out just makes necessary changes in Text datasource to
support encodings in JSON datasource.
> Encoding not working when using a map / mapPartitions call
> ----------------------------------------------------------
>
> Key: SPARK-26016
> URL: https://issues.apache.org/jira/browse/SPARK-26016
> Project: Spark
> Issue Type: Bug
> Components: Java API
> Affects Versions: 2.4.0
> Reporter: Chris Caspanello
> Priority: Major
> Attachments: spark-sandbox.zip
>
>
> Attached you will find a project with unit tests showing the issue at hand.
> If I read in a ISO-8859-1 encoded file and simply write out what was read;
> the contents in the part file matches what was read. Which is great.
> However, the second I use a map / mapPartitions function it looks like the
> encoding is not correct. In addition a simple collectAsList and writing that
> list of strings to a file does not work either. I don't think I'm doing
> anything wrong. Can someone please investigate? I think this is a bug.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
