This is an automated email from the ASF dual-hosted git repository. maxgekk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new cc1674d66ef [SPARK-41948][SQL] Fix NPE for error classes: CANNOT_PARSE_JSON_FIELD cc1674d66ef is described below commit cc1674d66ef34f540aa7bd5c7e465605e264e040 Author: panbingkun <pbk1...@gmail.com> AuthorDate: Mon Jan 23 15:15:59 2023 +0300 [SPARK-41948][SQL] Fix NPE for error classes: CANNOT_PARSE_JSON_FIELD ### What changes were proposed in this pull request? The pr aims to fix NPE for error classes: CANNOT_PARSE_JSON_FIELD. ### Why are the changes needed? 1. When I want to delete redundant 'toString()' in code block as follow <img width="722" alt="image" src="https://user-images.githubusercontent.com/15246973/211269145-0f087bb1-dc93-480c-9f9d-afde5ac1c8de.png"> I found the UT("select from_json('[1, \"2\", 3]', 'array<int>')") failed. #### Why can it succeed before deletion? `parse.getCurrentName.toString()` => null.toString() => throw NPE, but follow logical can cover it, https://github.com/apache/spark/blob/15a0f55246bee7b043bd6081f53744fbf74403eb/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonParser.scala#L569-L573 But obviously this is not our original intention. #### After deletion, The IllegalArgumentException will be thrown. `parse.getCurrentName` => throw java.lang.IllegalArgumentException as follow: `Caused by: java.lang.IllegalArgumentException: Cannot resolve variable 'fieldName' (enableSubstitutionInVariables=false). at org.apache.commons.text.StringSubstitutor.substitute(StringSubstitutor.java:1532) at org.apache.commons.text.StringSubstitutor.substitute(StringSubstitutor.java:1389) at org.apache.commons.text.StringSubstitutor.replace(StringSubstitutor.java:893) at org.apache.spark.ErrorClassesJsonReader.getErrorMessage(ErrorClassesJSONReader.scala:51) ... 140 more ` Above code can't handle IllegalArgumentException, so the UT failed. So, we should consider the case where `parse.getCurrentName` is null. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. Existed UT. Closes #39466 from panbingkun/SPARK-41948. Authored-by: panbingkun <pbk1...@gmail.com> Signed-off-by: Max Gekk <max.g...@gmail.com> --- .../main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala | 4 ++-- sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala | 3 ++- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala index 8128c460602..9c8c764cf92 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala @@ -1443,8 +1443,8 @@ private[sql] object QueryExecutionErrors extends QueryErrorsBase { new SparkRuntimeException( errorClass = "CANNOT_PARSE_JSON_FIELD", messageParameters = Map( - "fieldName" -> parser.getCurrentName.toString(), - "fieldValue" -> parser.getText.toString(), + "fieldName" -> toSQLValue(parser.getCurrentName, StringType), + "fieldValue" -> parser.getText, "jsonType" -> jsonType.toString(), "dataType" -> toSQLType(dataType))) } diff --git a/sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala index 6e16533eb30..57c54e88229 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/JsonFunctionsSuite.scala @@ -27,6 +27,7 @@ import org.apache.commons.lang3.exception.ExceptionUtils import org.apache.spark.{SparkException, SparkRuntimeException} import org.apache.spark.sql.catalyst.InternalRow import org.apache.spark.sql.catalyst.expressions.{Literal, StructsToJson} +import org.apache.spark.sql.catalyst.expressions.Cast._ import org.apache.spark.sql.functions._ import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.test.SharedSparkSession @@ -785,7 +786,7 @@ class JsonFunctionsSuite extends QueryTest with SharedSparkSession { exception = ExceptionUtils.getRootCause(exception).asInstanceOf[SparkRuntimeException], errorClass = "CANNOT_PARSE_JSON_FIELD", parameters = Map( - "fieldName" -> "a", + "fieldName" -> toSQLValue("a", StringType), "fieldValue" -> "1", "jsonType" -> "VALUE_STRING", "dataType" -> "\"INT\"") --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org