[ https://issues.apache.org/jira/browse/SPARK-1849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000460#comment-14000460 ]
Harry Brundage commented on SPARK-1849: --------------------------------------- I disagree - the data isn't badly encoded, just not UTF-8 encoded, which when we're talking about data from the internet really isn't all that uncommon. You could extend my specific problem of some lines in the source file being a different encoding to a file entirely encoded in iso-8859-1, which is likely something Spark should deal with considering all the effort put into supporting Windows. I don't think asking users to drop down to writing custom {{InputFormat}}s to deal with the realities of large data is a good move if Spark wants to become the fast and general data processing engine for large scale data. I could certainly use {{sc.hadoopFile}} to load in my data and work with the {{org.apache.hadoop.io.Text}} objects myself, but A) why force everyone dealing with this issue to go through the pain of figuring that out, and B) I'm in PySpark where I can't actually do that without fancy Py4J trickery. I think encoding issues should be in your face. > Broken UTF-8 encoded data gets character replacements and thus can't be > "fixed" > ------------------------------------------------------------------------------- > > Key: SPARK-1849 > URL: https://issues.apache.org/jira/browse/SPARK-1849 > Project: Spark > Issue Type: Bug > Reporter: Harry Brundage > Fix For: 1.0.0, 0.9.1 > > Attachments: encoding_test > > > I'm trying to process a file which isn't valid UTF-8 data inside hadoop using > Spark via {{sc.textFile()}}. Is this possible, and if not, is this a bug that > we should fix? It looks like {{HadoopRDD}} uses > {{org.apache.hadoop.io.Text.toString}} on all the data it ever reads, which I > believe replaces invalid UTF-8 byte sequences with the UTF-8 replacement > character, \uFFFD. Some example code mimicking what {{sc.textFile}} does > underneath: > {code} > scala> sc.textFile(path).collect()(0) > res8: String = ?pple > scala> sc.hadoopFile(path, classOf[TextInputFormat], classOf[LongWritable], > classOf[Text]).map(pair => pair._2.toString).collect()(0).getBytes() > res9: Array[Byte] = Array(-17, -65, -67, 112, 112, 108, 101) > scala> sc.hadoopFile(path, classOf[TextInputFormat], classOf[LongWritable], > classOf[Text]).map(pair => pair._2.getBytes).collect()(0) > res10: Array[Byte] = Array(-60, 112, 112, 108, 101) > {code} > In the above example, the first two snippets show the string representation > and byte representation of the example line of text. The third snippet shows > what happens if you call {{getBytes}} on the {{Text}} object which comes back > from hadoop land: we get the real bytes in the file out. > Now, I think this is a bug, though you may disagree. The text inside my file > is perfectly valid iso-8859-1 encoded bytes, which I would like to be able to > rescue and re-encode into UTF-8, because I want my application to be smart > like that. I think Spark should give me the raw broken string so I can > re-encode, but I can't get at the original bytes in order to guess at what > the source encoding might be, as they have already been replaced. I'm dealing > with data from some CDN access logs which are to put it nicely diversely > encoded, but I think a use case Spark should fully support. So, my suggested > fix, which I'd like some guidance, is to change {{textFile}} to spit out > broken strings by not using {{Text}}'s UTF-8 encoding. > Further compounding this issue is that my application is actually in PySpark, > but we can talk about how bytes fly through to Scala land after this if we > agree that this is an issue at all. -- This message was sent by Atlassian JIRA (v6.2#6252)