Github user MaxGekk commented on a diff in the pull request: https://github.com/apache/spark/pull/21909#discussion_r210704902 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala --- @@ -2223,21 +2223,31 @@ class JsonSuite extends QueryTest with SharedSQLContext with TestJsonData { checkAnswer(jsonDF, Seq(Row("Chris", "Baird"))) } - test("SPARK-23723: specified encoding is not matched to actual encoding") { - val fileName = "test-data/utf16LE.json" - val schema = new StructType().add("firstName", StringType).add("lastName", StringType) - val exception = intercept[SparkException] { - spark.read.schema(schema) - .option("mode", "FAILFAST") - .option("multiline", "true") - .options(Map("encoding" -> "UTF-16BE")) - .json(testFile(fileName)) - .count() + def doCount(bypassParser: Boolean, multiLine: Boolean): Long = { + var result: Long = -1 + withSQLConf(SQLConf.BYPASS_PARSER_FOR_EMPTY_SCHEMA.key -> bypassParser.toString) { + val fileName = "test-data/utf16LE.json" + val schema = new StructType().add("firstName", StringType).add("lastName", StringType) + result = spark.read.schema(schema) + .option("mode", "FAILFAST") --- End diff -- > Does the mode matter? I just want to have an explicit error in the test instead of `0` for `count()` ( `DROPMALFORMED`), or full table of nulls or an exception (`PERMISSIVE`) since an exception is expected result. > What happened if users use DROPMALFORMED before this PR? It depends on `multiLine`. If it is `true`, behaviour before and after PR is the same since the optimization doesn't impact on the `multiLine` mode. For `multiLine` equals to `false`, after the PR the result is `5` (total number of lines), before the PR - `0` in the `DROPMALFORMED` mode. We can enable this optimization for the `PERMISSIVE` mode only to exclude any deviation in outputs.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org