Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/20849 When I was trying to remove the flexible format for lineSep (recordDelimiter), I faced to a problem. I cannot fix the test: https://github.com/MaxGekk/spark-1/blob/54fd42b64e0715540010c4d59b8b4f7a4a1b0876/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala#L2071-L2081 There are no any combination of charset and lineSep that allow me to read the file. Here is the structure of the file: ``` BOM json_record1 delimiter json_record2 delimiter ``` The delimiter in hex: **x0d 00 0a 00** . Basically it is `\r\n` in UTF-16LE. If I set: ``` .option("charset", "UTF-16LE").option("lineSep", "\r\n") ``` The first record is ignored because it contains BOM which `UTF-16LE` must not contain. As the result I am getting only the the second record. If I set `UTF-16`, I am getting the first record because it contains BOM (as UTF-16 string must contain) but the second is rejected. How does it work in the case if `.option("recordDelimiter", "x0d 00 0a 00")` and charset is not specified. The answer is charset auto-detection of jackson-json. Hadoop's LineRecord Reader just splits the json by the delimiter and we have: ``` Seq("BOM json_record1", "json_record2") ``` The first string is detected according to BOM. And BOM is removed from the result by jackson. The second string is detected according to its chars as `UTF-16LE`. And we are getting correct result. So, if we don't support lineSep format in which sequence of bytes are expressed explicitly, we cannot read unicode jsons with BOM in per-line mode.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org