This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push: new 39e9b55 [SPARK-31171][SQL][FOLLOWUP] update document 39e9b55 is described below commit 39e9b554ea171e71ea152c1d3a59f72e8918dfd2 Author: Wenchen Fan <wenc...@databricks.com> AuthorDate: Thu Mar 19 07:29:31 2020 +0900 [SPARK-31171][SQL][FOLLOWUP] update document ### What changes were proposed in this pull request? A followup of https://github.com/apache/spark/pull/27936 to update document. ### Why are the changes needed? correct document ### Does this PR introduce any user-facing change? no ### How was this patch tested? N/A Closes #27950 from cloud-fan/null. Authored-by: Wenchen Fan <wenc...@databricks.com> Signed-off-by: Takeshi Yamamuro <yamam...@apache.org> (cherry picked from commit 8643e5d9c50294f59b01988d99d447a38776178e) Signed-off-by: Takeshi Yamamuro <yamam...@apache.org> --- docs/sql-ref-ansi-compliance.md | 7 ++++++- .../spark/sql/catalyst/expressions/collectionOperations.scala | 6 +++--- sql/core/src/main/scala/org/apache/spark/sql/functions.scala | 4 ++++ 3 files changed, 13 insertions(+), 4 deletions(-) diff --git a/docs/sql-ref-ansi-compliance.md b/docs/sql-ref-ansi-compliance.md index 27e60b4..bc5bde6 100644 --- a/docs/sql-ref-ansi-compliance.md +++ b/docs/sql-ref-ansi-compliance.md @@ -21,7 +21,7 @@ license: | Since Spark 3.0, Spark SQL introduces two experimental options to comply with the SQL standard: `spark.sql.ansi.enabled` and `spark.sql.storeAssignmentPolicy` (See a table below for details). -When `spark.sql.ansi.enabled` is set to `true`, Spark SQL follows the standard in basic behaviours (e.g., arithmetic operations, type conversion, and SQL parsing). +When `spark.sql.ansi.enabled` is set to `true`, Spark SQL follows the standard in basic behaviours (e.g., arithmetic operations, type conversion, SQL functions and SQL parsing). Moreover, Spark SQL has an independent option to control implicit casting behaviours when inserting rows in a table. The casting behaviours are defined as store assignment rules in the standard. @@ -140,6 +140,11 @@ SELECT * FROM t; {% endhighlight %} +### SQL Functions + +The behavior of some SQL functions can be different under ANSI mode (`spark.sql.ansi.enabled=true`). + - `size`: This function returns null for null input under ANSI mode. + ### SQL Keywords When `spark.sql.ansi.enabled` is true, Spark SQL will use the ANSI mode parser. diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala index 6d95909..8b61bc4 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala @@ -77,9 +77,9 @@ trait BinaryArrayExpressionWithImplicitCast extends BinaryExpression @ExpressionDescription( usage = """ _FUNC_(expr) - Returns the size of an array or a map. - The function returns -1 if its input is null and spark.sql.legacy.sizeOfNull is set to true. - If spark.sql.legacy.sizeOfNull is set to false, the function returns null for null input. - By default, the spark.sql.legacy.sizeOfNull parameter is set to true. + The function returns null for null input if spark.sql.legacy.sizeOfNull is set to false or + spark.sql.ansi.enabled is set to true. Otherwise, the function returns -1 for null input. + With the default settings, the function returns -1 for null input. """, examples = """ Examples: diff --git a/sql/core/src/main/scala/org/apache/spark/sql/functions.scala b/sql/core/src/main/scala/org/apache/spark/sql/functions.scala index e2d3d55..69383d4 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/functions.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/functions.scala @@ -3957,6 +3957,10 @@ object functions { /** * Returns length of array or map. * + * The function returns null for null input if spark.sql.legacy.sizeOfNull is set to false or + * spark.sql.ansi.enabled is set to true. Otherwise, the function returns -1 for null input. + * With the default settings, the function returns -1 for null input. + * * @group collection_funcs * @since 1.5.0 */ --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org