[
https://issues.apache.org/jira/browse/SPARK-26016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16782520#comment-16782520
]
Sean Owen commented on SPARK-26016:
-----------------------------------
Yeah, the TL;DR from reading the code is that it writes UTF-8, right?
df.write.text always writes UTF-8 strings; I think that's the most immediate
explanation for all of this. df.read.text also only reads UTF-8, as it pushes
down to the Hadoop input formats and Text works in terms of UTF-8. I think it's
working insofar as ISO-8859-1's character mapping is a subset of UTF-8's. I'm
not as sure why the write path doesn't accidentally work; BOMs? I think this is
80% right at least as an explanation.
The problem is that this isn't documented anywhere I can find. I think people
are just accustomed to always working in UTF-8 and the fact that you can read
some common encodings as UTF-8 anyway. That much can easily be documented.
It's exacerbated by the fact that there's an 'encoding' option on
DataFrameReader/Writer, which does nothing but affect how the line separator is
treated. (I don't quite get why you could use a non-UTF-8 encoding of your line
separator if everything else assumes UTF-8!)
[~maxgekk] it looks like this option was added in
https://github.com/apache/spark/pull/20937/files#diff-cbd2ad0c69c1517fbbc42b587fd37e25R44
; was it necessary for the text data source? for the reason above I'm not sure
about how it works. I see it's also used for JSON, separately. Is it meant to
be 'hidden'?
> Encoding not working when using a map / mapPartitions call
> ----------------------------------------------------------
>
> Key: SPARK-26016
> URL: https://issues.apache.org/jira/browse/SPARK-26016
> Project: Spark
> Issue Type: Bug
> Components: Java API
> Affects Versions: 2.4.0
> Reporter: Chris Caspanello
> Priority: Major
> Attachments: spark-sandbox.zip
>
>
> Attached you will find a project with unit tests showing the issue at hand.
> If I read in a ISO-8859-1 encoded file and simply write out what was read;
> the contents in the part file matches what was read. Which is great.
> However, the second I use a map / mapPartitions function it looks like the
> encoding is not correct. In addition a simple collectAsList and writing that
> list of strings to a file does not work either. I don't think I'm doing
> anything wrong. Can someone please investigate? I think this is a bug.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
