[spark] branch master updated: [SPARK-41780][SQL] Should throw INVALID_PARAMETER_VALUE.PATTERN when the parameters `regexp` is invalid
This is an automated email from the ASF dual-hosted git repository. maxgekk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 15a0f55246b [SPARK-41780][SQL] Should throw INVALID_PARAMETER_VALUE.PATTERN when the parameters `regexp` is invalid 15a0f55246b is described below commit 15a0f55246bee7b043bd6081f53744fbf74403eb Author: panbingkun AuthorDate: Mon Jan 9 11:37:54 2023 +0300 [SPARK-41780][SQL] Should throw INVALID_PARAMETER_VALUE.PATTERN when the parameters `regexp` is invalid ### What changes were proposed in this pull request? In the PR, I propose to throw error classes - `INVALID_PARAMETER_VALUE.PATTERN` when the parameters `regexp` in regexp_replace & regexp_extract & rlike is invalid. ### Why are the changes needed? Clear error prompt should improve user experience with Spark SQL. The original error prompt is: https://user-images.githubusercontent.com/15246973/210493673-c1de9927-9a18-4f9d-a94c-48735b6c5e5a.png";> Valid: [a\\d]{0,2} Invalid: [a\\d]{0, 2} ![image](https://user-images.githubusercontent.com/15246973/210494925-cb6c8043-de02-4c8e-9b40-225350422340.png) ### Does this PR introduce _any_ user-facing change? Yes. ### How was this patch tested? Add new UT. Pass GA. Closes #39383 from panbingkun/SPARK-41780. Authored-by: panbingkun Signed-off-by: Max Gekk --- .../catalyst/expressions/regexpExpressions.scala | 20 ++-- .../expressions/RegexpExpressionsSuite.scala | 29 - .../apache/spark/sql/StringFunctionsSuite.scala| 38 ++ 3 files changed, 76 insertions(+), 11 deletions(-) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala index c86dcfb3b96..29510bc3852 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala @@ -57,7 +57,12 @@ abstract class StringRegexExpression extends BinaryExpression null } else { // Let it raise exception if couldn't compile the regex string -Pattern.compile(escape(str)) +try { + Pattern.compile(escape(str)) +} catch { + case e: PatternSyntaxException => +throw QueryExecutionErrors.invalidPatternError(prettyName, e.getPattern, e) +} } protected def pattern(str: String) = if (cache == null) compile(str) else cache @@ -634,7 +639,12 @@ case class RegExpReplace(subject: Expression, regexp: Expression, rep: Expressio if (!p.equals(lastRegex)) { // regex value changed lastRegex = p.asInstanceOf[UTF8String].clone() - pattern = Pattern.compile(lastRegex.toString) + try { +pattern = Pattern.compile(lastRegex.toString) + } catch { +case e: PatternSyntaxException => + throw QueryExecutionErrors.invalidPatternError(prettyName, e.getPattern, e) + } } if (!r.equals(lastReplacementInUTF8)) { // replacement string changed @@ -688,7 +698,11 @@ case class RegExpReplace(subject: Expression, regexp: Expression, rep: Expressio if (!$regexp.equals($termLastRegex)) { // regex value changed $termLastRegex = $regexp.clone(); -$termPattern = $classNamePattern.compile($termLastRegex.toString()); +try { + $termPattern = $classNamePattern.compile($termLastRegex.toString()); +} catch (java.util.regex.PatternSyntaxException e) { + throw QueryExecutionErrors.invalidPatternError("$prettyName", e.getPattern(), e); +} } if (!$rep.equals($termLastReplacementInUTF8)) { // replacement string changed diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/RegexpExpressionsSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/RegexpExpressionsSuite.scala index 8b5e303849c..af051a1a9bc 100644 --- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/RegexpExpressionsSuite.scala +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/RegexpExpressionsSuite.scala @@ -279,14 +279,27 @@ class RegexpExpressionsSuite extends SparkFunSuite with ExpressionEvalHelper { checkLiteralRow("abc" rlike _, "^bc", false) checkLiteralRow("abc" rlike _, "^ab", true) checkLiteralRow("abc" rlike _, "^bc", false) - -intercept[java.util.regex.PatternSyntaxException] { - evaluateWithoutCodegen("ac" rlike "**") -} -intercept[java.util.regex.PatternSyntaxException] { - val regex = $"a".string.at(0) - evaluateWithoutCodegen("ac" rlike regex, create_row("**"
[spark] branch master updated: [MINOR] Fix wrong file name
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new c2a5cbf2088 [MINOR] Fix wrong file name c2a5cbf2088 is described below commit c2a5cbf2088b33d9a518ffe1409db05d373a3b25 Author: dengziming AuthorDate: Mon Jan 9 19:17:09 2023 +0900 [MINOR] Fix wrong file name ### What changes were proposed in this pull request? Fix the filename in doc ### Why are the changes needed? improve docs and output tips. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Not needed. Closes #39465 from dengziming/filename-typo. Authored-by: dengziming Signed-off-by: Hyukjin Kwon --- dev/connect-check-protos.py | 2 +- dev/connect-gen-protos.sh | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/dev/connect-check-protos.py b/dev/connect-check-protos.py index b902274b1f4..a38a399d877 100755 --- a/dev/connect-check-protos.py +++ b/dev/connect-check-protos.py @@ -18,7 +18,7 @@ # # Utility for checking whether generated codes in PySpark are out of sync. -# usage: ./dev/check-codegen-python.py +# usage: ./dev/connect-check-protos.py import os import sys diff --git a/dev/connect-gen-protos.sh b/dev/connect-gen-protos.sh index cb5b66379b2..c4fc8d385f2 100755 --- a/dev/connect-gen-protos.sh +++ b/dev/connect-gen-protos.sh @@ -20,7 +20,7 @@ set -ex if [[ $# -gt 1 ]]; then echo "Illegal number of parameters." - echo "Usage: ./dev/generate_protos.sh [path]" + echo "Usage: ./dev/connect-gen-protos.sh [path]" exit -1 fi - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (c2a5cbf2088 -> e08af6d753e)
This is an automated email from the ASF dual-hosted git repository. ruifengz pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from c2a5cbf2088 [MINOR] Fix wrong file name add e08af6d753e [SPARK-41945][CONNECT][PYTHON] Python: connect client lost column data with pyarrow.Table.to_pylist No new revisions were added by this update. Summary of changes: python/pyspark/sql/connect/dataframe.py| 24 +++--- .../sql/tests/connect/test_connect_basic.py| 18 ++-- .../sql/tests/connect/test_parity_functions.py | 5 - 3 files changed, 33 insertions(+), 14 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-41944][CONNECT] Pass configurations when local remote mode is on
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 55fe522bbb4 [SPARK-41944][CONNECT] Pass configurations when local remote mode is on 55fe522bbb4 is described below commit 55fe522bbb45ffb4ca59ec664ec0b59d41fa35da Author: Hyukjin Kwon AuthorDate: Mon Jan 9 21:42:32 2023 +0900 [SPARK-41944][CONNECT] Pass configurations when local remote mode is on ### What changes were proposed in this pull request? This PR mainly proposes to pass the user-specified configurations to local remote mode. Previously, all user-specific configurations were ignored in case of PySpark shell such as`./bin/pyspark` or plain Python interpreter - PySpark application submission case was fine. Now, configurations are properly passed to the server side, e.g., `./bin/pyspark --remote local --conf aaa=bbb` and `aaa=bbb` is properly passed to the server side. For `spark.master` and `spark.plugins`, user-specific configurations are respected. If they are unset, they are automatically set, e.g., `org.apache.spark.sql.connect.SparkConnectPlugin`. If they are set, users have to provide the proper values to overwrite them, meaning that either: ```bash ./bin/pyspark --remote local --conf spark.plugins="other.Plugin,org.apache.spark.sql.connect.SparkConnectPlugin" ``` or ```bash ./bin/pyspark --remote local ``` In addition, this PR fixes the related code as below: - Adds `spark.local.connect` internal configuration to be used in Spark Submit (so we don't have to parse and manipulate user specified arguments in Python in order to remove `--remote` or `spark.remote` configuration). - Adds some more validation on arguments in `SparkSubmitCommandBuilder` so invalid combination can fail fast (e.g., setting both remote and master like `--master ...` and `--conf spark.remote=...`) - In dev mode, do not set `spark.jars` anymore since it adds the jars into the class path of the JVM through `addJarToCurrentClassLoader`. ### Why are the changes needed? To correctly pass the configurations specified from users. ### Does this PR introduce _any_ user-facing change? No, Spark Connect has not been released yet. This is kind of a followup of https://github.com/apache/spark/pull/39441 to complete its support. ### How was this patch tested? Manually tested all combinations such as: ```bash ./bin/pyspark --conf spark.remote=local ./bin/pyspark --conf spark.remote=local --conf spark.jars=a ./bin/pyspark --conf spark.remote=local --jars /.../spark/connector/connect/server/target/scala-2.12/spark-connect-assembly-3.4.0-SNAPSHOT.jar ./bin/spark-submit --conf spark.remote=local --jars /.../spark/connector/connect/server/target/scala-2.12/spark-connect-assembly-3.4.0-SNAPSHOT.jar app.py ./bin/pyspark --conf spark.remote=local --conf spark.jars=/.../spark/connector/connect/server/target/scala-2.12/spark-connect-assembly-3.4.0-SNAPSHOT.jar ./bin/pyspark --master "local[*]" --remote "local" ./bin/spark-submit --conf spark.remote=local app.py ./bin/spark-submit --master="local[*]" --conf spark.remote=local app.py ./bin/spark-submit --master="local[*]" --remote=local app.py ./bin/pyspark --master "local[*]" --remote "local" ./bin/pyspark --master "local[*]" --remote "local" ./bin/pyspark --master "local[*]" --conf spark.remote="local" ./bin/spark-submit --master="local[*]" --remote=local app.py ./bin/pyspark --remote local ``` Closes #39463 from HyukjinKwon/SPARK-41933-conf. Authored-by: Hyukjin Kwon Signed-off-by: Hyukjin Kwon --- .../org/apache/spark/deploy/SparkSubmit.scala | 12 ++- .../apache/spark/deploy/SparkSubmitArguments.scala | 3 +- .../org/apache/spark/launcher/SparkLauncher.java | 4 + .../spark/launcher/SparkSubmitCommandBuilder.java | 20 ++-- python/pyspark/sql/connect/session.py | 118 ++--- 5 files changed, 82 insertions(+), 75 deletions(-) diff --git a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala index 745836dfbef..65e4367b33a 100644 --- a/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala +++ b/core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala @@ -226,12 +226,13 @@ private[spark] class SparkSubmit extends Logging { val childArgs = new ArrayBuffer[String]() val childClasspath = new ArrayBuffer[String]() val sparkConf = args.toSparkConf() +if (sparkConf.contains("spark.local.connect")) sparkConf.remove("spark.remote") var childMainClass = "" // Set the cluster manager val clusterManager: Int = ar
[spark] branch master updated: [SPARK-41860][SQL] Make AvroScanBuilder and JsonScanBuilder case classes
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 1f1c0016fad [SPARK-41860][SQL] Make AvroScanBuilder and JsonScanBuilder case classes 1f1c0016fad is described below commit 1f1c0016fad234ba9d939a8269adc3059842687a Author: Lorenzo Martini AuthorDate: Mon Jan 9 09:53:22 2023 -0800 [SPARK-41860][SQL] Make AvroScanBuilder and JsonScanBuilder case classes ### What changes were proposed in this pull request? Make `AvroScanBuilder` and `JsonScanBuilder` case classes from normal classes. All the other existing `ScanBuilder`s inheriting from `FileScanBuilder` are case classes which is very nice and useful. However the Json and the Avro ones aren't, which makes it quite inconvenient when working with those. I don't see any particular reason why these two would not be case classes as well given they are very similar to the other ones and very simple and stateless. Especially when `AvroScan` and `JsonScan` are also already case classes like the other scans. ### Why are the changes needed? When working on using different possible `FileScanBuilder` is really problematic that these 2 ScanBuilders are not case classes, especially given all the others are. This is a very simple change that helps development greatly and allows for more flexibility / feature deelopment when working with ScanBuilders. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? No need for testing as it's a simple change from `class` to `case class` Closes #39366 from LorenzoMartini/patch-1. Authored-by: Lorenzo Martini Signed-off-by: Dongjoon Hyun --- .../src/main/scala/org/apache/spark/sql/v2/avro/AvroScanBuilder.scala | 2 +- .../spark/sql/execution/datasources/v2/json/JsonScanBuilder.scala | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/connector/avro/src/main/scala/org/apache/spark/sql/v2/avro/AvroScanBuilder.scala b/connector/avro/src/main/scala/org/apache/spark/sql/v2/avro/AvroScanBuilder.scala index 08dcaa7d087..754c58e65b0 100644 --- a/connector/avro/src/main/scala/org/apache/spark/sql/v2/avro/AvroScanBuilder.scala +++ b/connector/avro/src/main/scala/org/apache/spark/sql/v2/avro/AvroScanBuilder.scala @@ -24,7 +24,7 @@ import org.apache.spark.sql.sources.Filter import org.apache.spark.sql.types.StructType import org.apache.spark.sql.util.CaseInsensitiveStringMap -class AvroScanBuilder ( +case class AvroScanBuilder ( sparkSession: SparkSession, fileIndex: PartitioningAwareFileIndex, schema: StructType, diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/json/JsonScanBuilder.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/json/JsonScanBuilder.scala index 788ccfac6e7..dcae6bd3fd0 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/json/JsonScanBuilder.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/json/JsonScanBuilder.scala @@ -24,7 +24,7 @@ import org.apache.spark.sql.sources.Filter import org.apache.spark.sql.types.StructType import org.apache.spark.sql.util.CaseInsensitiveStringMap -class JsonScanBuilder ( +case class JsonScanBuilder ( sparkSession: SparkSession, fileIndex: PartitioningAwareFileIndex, schema: StructType, - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (1f1c0016fad -> c292bd7a345)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 1f1c0016fad [SPARK-41860][SQL] Make AvroScanBuilder and JsonScanBuilder case classes add c292bd7a345 [SPARK-41848][CORE] Fixing task over-scheduled with TaskResourceProfile No new revisions were added by this update. Summary of changes: .../executor/CoarseGrainedExecutorBackend.scala| 3 +- .../scala/org/apache/spark/executor/Executor.scala | 2 +- .../cluster/CoarseGrainedClusterMessage.scala | 10 ++- .../cluster/CoarseGrainedSchedulerBackend.scala| 16 ++-- .../CoarseGrainedExecutorBackendSuite.scala| 25 +- .../CoarseGrainedSchedulerBackendSuite.scala | 96 +- 6 files changed, 135 insertions(+), 17 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-41947][CORE][DOCS] Update the contents of error class guidelines
This is an automated email from the ASF dual-hosted git repository. maxgekk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 786594734bd [SPARK-41947][CORE][DOCS] Update the contents of error class guidelines 786594734bd is described below commit 786594734bd79017ebd42eb117b62958afad07bb Author: itholic AuthorDate: Mon Jan 9 23:24:09 2023 +0300 [SPARK-41947][CORE][DOCS] Update the contents of error class guidelines ### What changes were proposed in this pull request? This PR proposes to update error class guidelines for `core/src/main/resources/error/README.md`. ### Why are the changes needed? Because some of contents are out of date, and no longer valid for current behavior. ### Does this PR introduce _any_ user-facing change? No. It fixed the developer guidelines for error class. ### How was this patch tested? The existing CI should pass. Closes #39464 from itholic/SPARK-41947. Authored-by: itholic Signed-off-by: Max Gekk --- core/src/main/resources/error/README.md | 16 +--- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/core/src/main/resources/error/README.md b/core/src/main/resources/error/README.md index 23e62cd25fb..8ea9e37c27f 100644 --- a/core/src/main/resources/error/README.md +++ b/core/src/main/resources/error/README.md @@ -8,9 +8,9 @@ and message parameters rather than an arbitrary error message. 1. Check if the error is an internal error. Internal errors are bugs in the code that we do not expect users to encounter; this does not include unsupported operations. If true, use the error class `INTERNAL_ERROR` and skip to step 4. -2. Check if an appropriate error class already exists in `error-class.json`. +2. Check if an appropriate error class already exists in `error-classes.json`. If true, use the error class and skip to step 4. -3. Add a new class to `error-class.json`; keep in mind the invariants below. +3. Add a new class to `error-classes.json`; keep in mind the invariants below. 4. Check if the exception type already extends `SparkThrowable`. If true, skip to step 6. 5. Mix `SparkThrowable` into the exception. @@ -24,10 +24,10 @@ Throw with arbitrary error message: ### After -`error-class.json` +`error-classes.json` "PROBLEM_BECAUSE": { - "message": ["Problem %s because %s"], + "message": ["Problem because "], "sqlState": "X" } @@ -35,16 +35,18 @@ Throw with arbitrary error message: class SparkTestException( errorClass: String, -messageParameters: Seq[String]) +messageParameters: Map[String, String]) extends TestException(SparkThrowableHelper.getMessage(errorClass, messageParameters)) with SparkThrowable { - def getErrorClass: String = errorClass + override def getMessageParameters: java.util.Map[String, String] = messageParameters.asJava + + override def getErrorClass: String = errorClass } Throw with error class and message parameters: -throw new SparkTestException("PROBLEM_BECAUSE", Seq("A", "B")) +throw new SparkTestException("PROBLEM_BECAUSE", Map("problem" -> "A", "cause" -> "B")) ## Access fields - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-41951][DOCS] Update SQL migration guide and documentations
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new c32ba2cb992 [SPARK-41951][DOCS] Update SQL migration guide and documentations c32ba2cb992 is described below commit c32ba2cb9925386498afb63cf42542e283391ace Author: Dongjoon Hyun AuthorDate: Mon Jan 9 15:16:41 2023 -0800 [SPARK-41951][DOCS] Update SQL migration guide and documentations ### What changes were proposed in this pull request? This PR aims to make `SQL migration guide` and documentations up-to-date for Apache Spark 3.4.0. ### Why are the changes needed? - We changed the default values of the configurations at Apache Spark 3.4.0. - In addition, I added the missed non-internal configurations because they are already exposed to the users in the previous Spark releases. ### Does this PR introduce _any_ user-facing change? No, this is a documentation-only change. ### How was this patch tested? Manual review. Closes #39470 from dongjoon-hyun/SPARK-41951. Authored-by: Dongjoon Hyun Signed-off-by: Dongjoon Hyun --- docs/sql-data-sources-orc.md | 3 +-- docs/sql-migration-guide.md| 1 + docs/sql-performance-tuning.md | 58 ++ 3 files changed, 60 insertions(+), 2 deletions(-) diff --git a/docs/sql-data-sources-orc.md b/docs/sql-data-sources-orc.md index d4a50c9fab9..2b00b771b09 100644 --- a/docs/sql-data-sources-orc.md +++ b/docs/sql-data-sources-orc.md @@ -37,7 +37,6 @@ For example, historically, `native` implementation handles `CHAR/VARCHAR` with S `native` implementation supports a vectorized ORC reader and has been the default ORC implementation since Spark 2.3. The vectorized reader is used for the native ORC tables (e.g., the ones created using the clause `USING ORC`) when `spark.sql.orc.impl` is set to `native` and `spark.sql.orc.enableVectorizedReader` is set to `true`. -For nested data types (array, map and struct), vectorized reader is disabled by default. Set `spark.sql.orc.enableNestedColumnVectorizedReader` to `true` to enable vectorized reader for these types. For the Hive ORC serde tables (e.g., the ones created using the clause `USING HIVE OPTIONS (fileFormat 'ORC')`), the vectorized reader is used when `spark.sql.hive.convertMetastoreOrc` is also set to `true`, and is turned on by default. @@ -173,7 +172,7 @@ When reading from Hive metastore ORC tables and inserting to Hive metastore ORC spark.sql.orc.enableNestedColumnVectorizedReader -false +true Enables vectorized orc decoding in native implementation for nested data types (array, map and struct). If spark.sql.orc.enableVectorizedReader is set to diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md index df48bf45b7d..0f80bb176f5 100644 --- a/docs/sql-migration-guide.md +++ b/docs/sql-migration-guide.md @@ -35,6 +35,7 @@ license: | - Valid values for `fmt` are case-insensitive `hex`, `base64`, `utf-8`, `utf8`. - Since Spark 3.4, Spark throws only `PartitionsAlreadyExistException` when it creates partitions but some of them exist already. In Spark 3.3 or earlier, Spark can throw either `PartitionsAlreadyExistException` or `PartitionAlreadyExistsException`. - Since Spark 3.4, Spark will do validation for partition spec in ALTER PARTITION to follow the behavior of `spark.sql.storeAssignmentPolicy` which may cause an exception if type conversion fails, e.g. `ALTER TABLE .. ADD PARTITION(p='a')` if column `p` is int type. To restore the legacy behavior, set `spark.sql.legacy.skipTypeValidationOnAlterPartition` to `true`. + - Since Spark 3.4, vectorized readers are enabled by default for the nested data types (array, map and struct). To restore the legacy behavior, set `spark.sql.orc.enableNestedColumnVectorizedReader` and `spark.sql.parquet.enableNestedColumnVectorizedReader` to `false`. ## Upgrading from Spark SQL 3.2 to 3.3 diff --git a/docs/sql-performance-tuning.md b/docs/sql-performance-tuning.md index 6ac39d90527..d5503a2e62d 100644 --- a/docs/sql-performance-tuning.md +++ b/docs/sql-performance-tuning.md @@ -286,6 +286,27 @@ This feature coalesces the post shuffle partitions based on the map output stati +### Spliting skewed shuffle partitions + + Property NameDefaultMeaningSince Version + + spark.sql.adaptive.optimizeSkewsInRebalancePartitions.enabled + true + + When true and spark.sql.adaptive.enabled is true, Spark will optimize the skewed shuffle partitions in RebalancePartitions and split them to smaller ones according to the target size (specified by spark.sql.adaptive.advisoryPartitionSizeInBytes), to avoid data skew. + + 3.2.0 + + + spark.sql.adaptive.rebal
[spark] branch master updated (c32ba2cb992 -> 9974b956ac0)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from c32ba2cb992 [SPARK-41951][DOCS] Update SQL migration guide and documentations add 9974b956ac0 [SPARK-41589][PYTHON][ML] PyTorch Distributor Baseline API Changes No new revisions were added by this update. Summary of changes: dev/sparktestsupport/modules.py| 1 + python/docs/source/reference/pyspark.ml.rst| 13 + python/pyspark/ml/__init__.py | 2 + .../python => python/pyspark/ml/torch}/__init__.py | 0 python/pyspark/ml/torch/distributor.py | 281 + .../pyspark/ml/torch/tests}/__init__.py| 0 python/pyspark/ml/torch/tests/test_distributor.py | 190 ++ 7 files changed, 487 insertions(+) copy {examples/src/main/python => python/pyspark/ml/torch}/__init__.py (100%) create mode 100644 python/pyspark/ml/torch/distributor.py copy {examples/src/main/python => python/pyspark/ml/torch/tests}/__init__.py (100%) create mode 100644 python/pyspark/ml/torch/tests/test_distributor.py - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-41232][SQL][PYTHON] Adding array_append function
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new c13fea90595 [SPARK-41232][SQL][PYTHON] Adding array_append function c13fea90595 is described below commit c13fea90595cb1489046135c45c92d3bacb85818 Author: Ankit Prakash Gupta AuthorDate: Tue Jan 10 10:31:57 2023 +0900 [SPARK-41232][SQL][PYTHON] Adding array_append function ### What changes were proposed in this pull request? [SPARK-41232] Adding array_append function in spark sql, Pyspark **Syntax**: array_append(arr, element) **Arguments:** arr: Array of anytype of elements in which the element has to be appended. element: Separate element type which has to be appended in the arr array. The type of element has to match with the type of elements array is holding. `select array_append(array(1, 2, 3), 4);` | array_append(array(1, 2, 3), 4) | | --| | [1, 2, 3, 4] | The mainstream database supports array_append show below: **Snowflake** https://docs.snowflake.com/en/developer-guide/snowpark/reference/python/api/snowflake.snowpark.functions.array_append.html **PostgreSQL** https://www.postgresql.org/docs/9.1/functions-array.html **MySQL** https://dev.mysql.com/doc/refman/5.7/en/json-modification-functions.html#function_json-array-append ### Why are the changes needed? New API - array_append -> To append element in a column or a value to another array column at the end ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Unit-tests have been added Closes #38865 from infoankitp/SPARK-41232. Authored-by: Ankit Prakash Gupta Signed-off-by: Hyukjin Kwon --- .../source/reference/pyspark.sql/functions.rst | 1 + python/pyspark/sql/functions.py| 30 + .../sql/catalyst/analysis/FunctionRegistry.scala | 1 + .../expressions/collectionOperations.scala | 147 + .../expressions/CollectionExpressionsSuite.scala | 75 +++ .../scala/org/apache/spark/sql/functions.scala | 12 ++ .../sql-functions/sql-expression-schema.md | 1 + .../src/test/resources/sql-tests/inputs/array.sql | 11 ++ .../resources/sql-tests/results/ansi/array.sql.out | 72 ++ .../test/resources/sql-tests/results/array.sql.out | 72 ++ .../apache/spark/sql/DataFrameFunctionsSuite.scala | 101 ++ 11 files changed, 523 insertions(+) diff --git a/python/docs/source/reference/pyspark.sql/functions.rst b/python/docs/source/reference/pyspark.sql/functions.rst index 2dc09cf2762..ddc8eab90f7 100644 --- a/python/docs/source/reference/pyspark.sql/functions.rst +++ b/python/docs/source/reference/pyspark.sql/functions.rst @@ -155,6 +155,7 @@ Collection Functions concat array_position element_at +array_append array_sort array_remove array_distinct diff --git a/python/pyspark/sql/functions.py b/python/pyspark/sql/functions.py index c533ca7be6e..60276b2a0b1 100644 --- a/python/pyspark/sql/functions.py +++ b/python/pyspark/sql/functions.py @@ -7775,6 +7775,36 @@ def array_compact(col: "ColumnOrName") -> Column: return _invoke_function_over_columns("array_compact", col) +@try_remote_functions +def array_append(col1: "ColumnOrName", col2: "ColumnOrName") -> Column: +""" +Collection function: returns an array of the elements in col1 along +with the added element in col2 at the last of the array. + +.. versionadded:: 3.4.0 + +Parameters +-- +col1 : :class:`~pyspark.sql.Column` or str +name of column containing array +col2 : :class:`~pyspark.sql.Column` or str +name of column containing element + +Returns +--- +:class:`~pyspark.sql.Column` +an array of values from first array along with the element. + +Examples + +>>> from pyspark.sql import Row +>>> df = spark.createDataFrame([Row(c1=["b", "a", "c"], c2="c")]) +>>> df.select(array_append(df.c1, df.c2)).collect() +[Row(array_append(c1, c2)=['b', 'a', 'c', 'c'])] +""" +return _invoke_function_over_columns("array_append", col1, col2) + + @try_remote_functions def explode(col: "ColumnOrName") -> Column: """ diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala index e28c77a10a4..01d672e20c6 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala @@ -
[spark] branch master updated: [SPARK-41943][CORE] Use java api to create files and grant permissions
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new b22946ed8b5 [SPARK-41943][CORE] Use java api to create files and grant permissions b22946ed8b5 is described below commit b22946ed8b5f41648a32a4e0c4c40226141a06a0 Author: smallzhongfeng AuthorDate: Tue Jan 10 10:40:44 2023 +0800 [SPARK-41943][CORE] Use java api to create files and grant permissions ### What changes were proposed in this pull request? For method `createDirWithPermission770`, using java api to create files and grant permissions instead of calling shell commands. ### Why are the changes needed? Safer and more efficient. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Origin uts. Closes #39448 from smallzhongfeng/java-api-mkdir. Authored-by: smallzhongfeng Signed-off-by: Wenchen Fan --- .../org/apache/spark/storage/DiskBlockManager.scala | 16 +--- 1 file changed, 5 insertions(+), 11 deletions(-) diff --git a/core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala b/core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala index e29f3fc1b80..a7ed9226c57 100644 --- a/core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala +++ b/core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala @@ -19,7 +19,7 @@ package org.apache.spark.storage import java.io.{File, IOException} import java.nio.file.Files -import java.nio.file.attribute.PosixFilePermission +import java.nio.file.attribute.{PosixFilePermission, PosixFilePermissions} import java.util.UUID import scala.collection.mutable.HashMap @@ -301,9 +301,6 @@ private[spark] class DiskBlockManager( * Create a directory that is writable by the group. * Grant the permission 770 "rwxrwx---" to the directory so the shuffle server can * create subdirs/files within the merge folder. - * TODO: Find out why can't we create a dir using java api with permission 770 - * Files.createDirectories(mergeDir.toPath, PosixFilePermissions.asFileAttribute( - * PosixFilePermissions.fromString("rwxrwx---"))) */ def createDirWithPermission770(dirToCreate: File): Unit = { var attempts = 0 @@ -315,16 +312,13 @@ private[spark] class DiskBlockManager( throw SparkCoreErrors.failToCreateDirectoryError(dirToCreate.getAbsolutePath, maxAttempts) } try { -val builder = new ProcessBuilder().command( - "mkdir", "-p", "-m770", dirToCreate.getAbsolutePath) -val proc = builder.start() -val exitCode = proc.waitFor() +dirToCreate.mkdirs() +Files.setPosixFilePermissions( + dirToCreate.toPath, PosixFilePermissions.fromString("rwxrwx---")) if (dirToCreate.exists()) { created = dirToCreate } -logDebug( - s"Created directory at ${dirToCreate.getAbsolutePath} with permission " + -s"770 and exitCode $exitCode") +logDebug(s"Created directory at ${dirToCreate.getAbsolutePath} with permission 770") } catch { case e: SecurityException => logWarning(s"Failed to create directory ${dirToCreate.getAbsolutePath} " + - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-41950][PS][BUILD] Pin scikit-learn to 1.1.X
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new c4a33ecb161 [SPARK-41950][PS][BUILD] Pin scikit-learn to 1.1.X c4a33ecb161 is described below commit c4a33ecb1617bf7abda462662134f147104cbd5a Author: Hyukjin Kwon AuthorDate: Tue Jan 10 12:16:52 2023 +0900 [SPARK-41950][PS][BUILD] Pin scikit-learn to 1.1.X ### What changes were proposed in this pull request? After scikit-learn 1.2.0 release (https://github.com/scikit-learn/scikit-learn/releases), the unittest in pandas API on Spark is broken. ```python from mlflow.tracking import MlflowClient, set_tracking_uri from sklearn.linear_model import LinearRegression import mlflow.sklearn from tempfile import mkdtemp d = mkdtemp("pandas_on_spark_mlflow") set_tracking_uri("file:%s"%d) client = MlflowClient() exp_id = mlflow.create_experiment("my_experiment") exp = mlflow.set_experiment("my_experiment") train = pd.DataFrame({"x1": np.arange(8), "x2": np.arange(8)**2, "y": np.log(2 + np.arange(8))}) train_x = train[["x1", "x2"]] train_y = train[["y"]] with mlflow.start_run(): lr = LinearRegression() lr.fit(train_x, train_y) mlflow.sklearn.log_model(lr, "model") from pyspark.pandas.mlflow import load_model run_info = client.search_runs(exp_id)[-1].info model = load_model("runs:/{run_id}/model".format(run_id=run_info.run_id)) prediction_df = ps.DataFrame({"x1": [2.0], "x2": [4.0]}) prediction_df["prediction"] = model.predict(prediction_df) print(prediction_df) ``` https://github.com/apache/spark/blob/06ec98b0d6a51e0c3ffec70e78d86d577b0e7a72/python/pyspark/pandas/mlflow.py#L134-L202 ``` File "/__w/spark/spark/python/pyspark/pandas/mlflow.py", line 172, in pyspark.pandas.mlflow.load_model Failed example: prediction_df Exception raised: Traceback (most recent call last): File "/usr/lib/python3.9/doctest.py", line 1336, in __run exec(compile(example.source, filename, "single", File "", line 1, in prediction_df File "/__w/spark/spark/python/pyspark/pandas/frame.py", line 13322, in __repr__ pdf = cast("DataFrame", self._get_or_create_repr_pandas_cache(max_display_count)) File "/__w/spark/spark/python/pyspark/pandas/frame.py", line 13313, in _get_or_create_repr_pandas_cache self, "_repr_pandas_cache", {n: self.head(n + 1)._to_internal_pandas()} File "/__w/spark/spark/python/pyspark/pandas/frame.py", line 13308, in _to_internal_pandas return self._internal.to_pandas_frame File "/__w/spark/spark/python/pyspark/pandas/utils.py", line 588, in wrapped_lazy_property setattr(self, attr_name, fn(self)) File "/__w/spark/spark/python/pyspark/pandas/internal.py", line 1056, in to_pandas_frame pdf = sdf.toPandas() File "/__w/spark/spark/python/pyspark/sql/pandas/conversion.py", line 208, in toPandas pdf = pd.DataFrame.from_records(self.collect(), columns=self.columns) File "/__w/spark/spark/python/pyspark/sql/dataframe.py", line 1197, in collect sock_info = self._jdf.collectToPython() File "/__w/spark/spark/python/lib/py4j-0.10.9.7-src.zip/py4j/java_gateway.py", line 1322, in __call__ return_value = get_return_value( File "/__w/spark/spark/python/pyspark/sql/utils.py", line 209, in deco raise converted from None pyspark.sql.utils.PythonException: An exception was thrown from the Python worker. Please see the stack trace below. Traceback (most recent call last): File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/worker.py", line 829, in main process() File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/worker.py", line 821, in process serializer.dump_stream(out_iter, outfile) File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", line 345, in dump_stream return ArrowStreamSerializer.dump_stream(self, init_stream_yield_batches(), stream) File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", line 86, in dump_stream for batch in iterator: File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/sql/pandas/serializers.py", line 338, in init_stream_yield_batches for series in iterator: File "/__w/spark/spark/python/lib/pyspark.zip/pyspark/worker.py", line 519, in func for result_batch, result_type in result_iter: File "/usr/local/lib/python3.9/dist-packages/mlflow/pyfunc/__init__.py", line 1253, in ud
[GitHub] [spark-website] drgnchan opened a new pull request, #431: Add language tab for word count code example
drgnchan opened a new pull request, #431: URL: https://github.com/apache/spark-website/pull/431 Before https://user-images.githubusercontent.com/40224023/211459050-6c824af6-4ed2-46ab-8f94-25e5fd02d979.png";> After https://user-images.githubusercontent.com/40224023/211459062-76816b76-7a31-4211-a024-8aecd3f81fa8.png";> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-41914][SQL] FileFormatWriter materializes AQE plan before accessing outputOrdering
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new c0911981887 [SPARK-41914][SQL] FileFormatWriter materializes AQE plan before accessing outputOrdering c0911981887 is described below commit c091198188789afb1282bc76419cf6e1397b0bc9 Author: Enrico Minack AuthorDate: Tue Jan 10 13:10:07 2023 +0800 [SPARK-41914][SQL] FileFormatWriter materializes AQE plan before accessing outputOrdering ### What changes were proposed in this pull request? The `FileFormatWriter` materializes an `AdaptiveQueryPlan` before accessing the plan's `outputOrdering`. This is required when planned writing is disabled (`spark.sql.optimizer.plannedWrite.enabled` is `true` by default). With planned writing enabled `FileFormatWriter` gets the final plan already. ### Why are the changes needed? `FileFormatWriter` enforces an ordering if the written plan does not provide that ordering. An `AdaptiveQueryPlan` does not know its final ordering, in which case `FileFormatWriter` enforces the ordering (e.g. by column `"a"`) even if the plan provides a compatible ordering (e.g. by columns `"a", "b"`). In case of spilling, that order (e.g. by columns `"a", "b"`) gets broken (see SPARK-40588). ### Does this PR introduce _any_ user-facing change? This fixes SPARK-40588 for 3.4, which was introduced in 3.0. This restores behaviour from Spark 2.4. ### How was this patch tested? The final plan that is written to files is now stored in `FileFormatWriter.executedPlan` (similar to existing `FileFormatWriter.outputOrderingMatched`). Unit tests assert the outermost sort order written to files. The actual plan written into the files changed from (taken from `"SPARK-41914: v1 write with AQE and in-partition sorted - non-string partition column"`): ``` Sort [input[2, int, false] ASC NULLS FIRST], false, 0 +- *(3) Sort [key#13 ASC NULLS FIRST, value#14 ASC NULLS FIRST], false, 0 +- *(3) Project [b#24, value#14, key#13] +- *(3) BroadcastHashJoin [key#13], [a#23], Inner, BuildLeft, false :- BroadcastQueryStage 2 : +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)),false), [plan_id=376] : +- AQEShuffleRead local :+- ShuffleQueryStage 0 : +- Exchange hashpartitioning(key#13, 5), ENSURE_REQUIREMENTS, [plan_id=328] : +- *(1) SerializeFromObject [knownnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$TestData, true])).key AS key#13, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, knownnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$TestData, true])).value, true, false, true) AS value#14] : +- Scan[obj#12] +- AQEShuffleRead local +- ShuffleQueryStage 1 +- Exchange hashpartitioning(a#23, 5), ENSURE_REQUIREMENTS, [plan_id=345] +- *(2) SerializeFromObject [knownnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$TestData2, true])).a AS a#23, knownnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$TestData2, true])).b AS b#24] +- Scan[obj#22] ``` where `FileFormatWriter` enforces order with `Sort [input[2, int, false] ASC NULLS FIRST], false, 0`, to ``` *(3) Sort [key#13 ASC NULLS FIRST, value#14 ASC NULLS FIRST], false, 0 +- *(3) Project [b#24, value#14, key#13] +- *(3) BroadcastHashJoin [key#13], [a#23], Inner, BuildLeft, false :- BroadcastQueryStage 2 : +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)),false), [plan_id=375] : +- AQEShuffleRead local :+- ShuffleQueryStage 0 : +- Exchange hashpartitioning(key#13, 5), ENSURE_REQUIREMENTS, [plan_id=327] : +- *(1) SerializeFromObject [knownnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$TestData, true])).key AS key#13, staticinvoke(class org.apache.spark.unsafe.types.UTF8String, StringType, fromString, knownnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$TestData, true])).value, true, false, true) AS value#14] : +- Scan[obj#12] +- AQEShuffleRead local +- ShuffleQueryStage 1 +- Exchange hashpartitioning(a#23, 5), ENSURE_REQUIREMENTS, [plan_id=344] +- *(2) SerializeFromObject [knownnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$TestData2, true])).a AS a#23, knownnotnull(assertnotnull(input
[spark] branch master updated: [SPARK-41957][CONNECT][PYTHON] Enable the doctest for `DataFrame.hint`
This is an automated email from the ASF dual-hosted git repository. ruifengz pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 7c1de057dd5 [SPARK-41957][CONNECT][PYTHON] Enable the doctest for `DataFrame.hint` 7c1de057dd5 is described below commit 7c1de057dd59dbef62f153c75005bb67ea809ac3 Author: Ruifeng Zheng AuthorDate: Tue Jan 10 13:57:50 2023 +0800 [SPARK-41957][CONNECT][PYTHON] Enable the doctest for `DataFrame.hint` ### What changes were proposed in this pull request? 1, Enable the doctest for `DataFrame.hint` 2, add additional test for JOIN hint ### Why are the changes needed? for test coverage ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? updated test and added UT Closes #39472 from zhengruifeng/connect_fix_hint. Authored-by: Ruifeng Zheng Signed-off-by: Ruifeng Zheng --- python/pyspark/sql/connect/dataframe.py| 2 -- python/pyspark/sql/dataframe.py| 2 +- python/pyspark/sql/tests/connect/test_connect_basic.py | 15 +++ 3 files changed, 16 insertions(+), 3 deletions(-) diff --git a/python/pyspark/sql/connect/dataframe.py b/python/pyspark/sql/connect/dataframe.py index 7f36272f538..5ff3d59ddd6 100644 --- a/python/pyspark/sql/connect/dataframe.py +++ b/python/pyspark/sql/connect/dataframe.py @@ -1627,8 +1627,6 @@ def _test() -> None: del pyspark.sql.connect.dataframe.DataFrame.drop.__doc__ del pyspark.sql.connect.dataframe.DataFrame.join.__doc__ -del pyspark.sql.connect.dataframe.DataFrame.hint.__doc__ - # TODO(SPARK-41886): The doctest output has different order del pyspark.sql.connect.dataframe.DataFrame.intersect.__doc__ diff --git a/python/pyspark/sql/dataframe.py b/python/pyspark/sql/dataframe.py index 79809cbea53..a437400fb90 100644 --- a/python/pyspark/sql/dataframe.py +++ b/python/pyspark/sql/dataframe.py @@ -1116,7 +1116,7 @@ class DataFrame(PandasMapOpsMixin, PandasConversionMixin): >>> df = spark.createDataFrame([(2, "Alice"), (5, "Bob")], schema=["age", "name"]) >>> df2 = spark.createDataFrame([Row(height=80, name="Tom"), Row(height=85, name="Bob")]) ->>> df.join(df2, "name").explain() +>>> df.join(df2, "name").explain() # doctest: +SKIP == Physical Plan == ... ... +- SortMergeJoin ... diff --git a/python/pyspark/sql/tests/connect/test_connect_basic.py b/python/pyspark/sql/tests/connect/test_connect_basic.py index 322594127e7..c946c735a1d 100644 --- a/python/pyspark/sql/tests/connect/test_connect_basic.py +++ b/python/pyspark/sql/tests/connect/test_connect_basic.py @@ -1302,6 +1302,21 @@ class SparkConnectBasicTests(SparkConnectSQLTestCase): with self.assertRaises(SparkConnectException): self.connect.read.table(self.tbl_name).hint("REPARTITION", "id", 3).toPandas() +def test_join_hint(self): + +cdf1 = self.connect.createDataFrame([(2, "Alice"), (5, "Bob")], schema=["age", "name"]) +cdf2 = self.connect.createDataFrame( +[Row(height=80, name="Tom"), Row(height=85, name="Bob")] +) + +self.assertTrue( +"BroadcastHashJoin" in cdf1.join(cdf2.hint("BROADCAST"), "name")._explain_string() +) +self.assertTrue("SortMergeJoin" in cdf1.join(cdf2.hint("MERGE"), "name")._explain_string()) +self.assertTrue( +"ShuffledHashJoin" in cdf1.join(cdf2.hint("SHUFFLE_HASH"), "name")._explain_string() +) + def test_empty_dataset(self): # SPARK-41005: Test arrow based collection with empty dataset. self.assertTrue( - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-41595][SQL] Support generator function explode/explode_outer in the FROM clause
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new eca21c7ad58 [SPARK-41595][SQL] Support generator function explode/explode_outer in the FROM clause eca21c7ad58 is described below commit eca21c7ad582c9c374108518842ad816843e1224 Author: allisonwang-db AuthorDate: Tue Jan 10 14:09:06 2023 +0800 [SPARK-41595][SQL] Support generator function explode/explode_outer in the FROM clause ### What changes were proposed in this pull request? This PR supports using table-valued generator functions in the FROM clause of a query. A generator function can be registered in the table function registry and resolved as a table function during analysis. Note this PR only adds support for two built-in generator functions: `explode` and `explode_outer` with literal input values. We will support more generator functions and LATERAL references in separate PRs. ### Why are the changes needed? To make table-valued generator functions more user-friendly and consistent with Spark's built-in table function Range. ### Does this PR introduce _any_ user-facing change? Yes. Before this PR, the built-in generator function explode/explode_outer cannot be used in the FROM clause: ``` select * from explode(array(1, 2)) AnalysisException: could not resolve `explode` to a table-valued function; ``` After this PR, we can support this usage: ``` select * from explode(array(1, 2)) +---+ |col| +---+ | 1| | 2| +---+ ``` ### How was this patch tested? New SQL query tests. Closes #39133 from allisonwang-db/spark-41595-explode-in-from. Authored-by: allisonwang-db Signed-off-by: Wenchen Fan --- .../spark/sql/catalyst/analysis/Analyzer.scala | 52 +++-- .../sql/catalyst/analysis/FunctionRegistry.scala | 23 +- .../spark/sql/catalyst/analysis/unresolved.scala | 65 -- .../sql/catalyst/catalog/SessionCatalog.scala | 4 +- .../spark/sql/catalyst/parser/AstBuilder.scala | 8 +- .../spark/sql/catalyst/trees/TreePatterns.scala| 1 + .../sql/catalyst/analysis/AnalysisSuite.scala | 6 +- .../sql/catalyst/parser/PlanParserSuite.scala | 12 +- .../sql-tests/inputs/table-valued-functions.sql| 33 +++ .../results/table-valued-functions.sql.out | 257 + 10 files changed, 410 insertions(+), 51 deletions(-) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala index 00a24357226..3b3a011db97 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala @@ -2431,7 +2431,7 @@ class Analyzer(override val catalogManager: CatalogManager) def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsUpWithPruning( _.containsAnyPattern(UNRESOLVED_FUNC, UNRESOLVED_FUNCTION, GENERATOR, -UNRESOLVED_TABLE_VALUED_FUNCTION), ruleId) { +UNRESOLVED_TABLE_VALUED_FUNCTION, UNRESOLVED_TVF_ALIASES), ruleId) { // Resolve functions with concrete relations from v2 catalog. case u @ UnresolvedFunctionName(nameParts, cmd, requirePersistentFunc, mismatchHint, _) => lookupBuiltinOrTempFunction(nameParts) @@ -2453,7 +2453,7 @@ class Analyzer(override val catalogManager: CatalogManager) // Resolve table-valued function references. case u: UnresolvedTableValuedFunction if u.functionArgs.forall(_.resolved) => withPosition(u) { - val resolvedFunc = try { + try { resolveBuiltinOrTempTableFunction(u.name, u.functionArgs).getOrElse { val CatalogAndIdentifier(catalog, ident) = expandIdentifier(u.name) if (CatalogV2Util.isSessionCatalog(catalog)) { @@ -2470,27 +2470,26 @@ class Analyzer(override val catalogManager: CatalogManager) errorClass = "_LEGACY_ERROR_TEMP_2308", messageParameters = Map("name" -> u.name.quoted)) } - // If alias names assigned, add `Project` with the aliases - if (u.outputNames.nonEmpty) { -val outputAttrs = resolvedFunc.output -// Checks if the number of the aliases is equal to expected one -if (u.outputNames.size != outputAttrs.size) { - u.failAnalysis( -errorClass = "_LEGACY_ERROR_TEMP_2307", -messageParameters = Map( - "funcName" -> u.name.quoted, - "aliasesNum" -> u.outputNames.size.toString, - "outColsNum" -> outputAttrs.size.toString)) -
[GitHub] [spark-website] dongjoon-hyun closed pull request #431: Add language tab for word count code example
dongjoon-hyun closed pull request #431: Add language tab for word count code example URL: https://github.com/apache/spark-website/pull/431 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark-website] branch asf-site updated: Add language tab for word count code example
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new c94c6ff3d Add language tab for word count code example c94c6ff3d is described below commit c94c6ff3dba01748958aafff39e408c8c001972f Author: drgnchan <40224023+drgnc...@users.noreply.github.com> AuthorDate: Mon Jan 9 22:45:03 2023 -0800 Add language tab for word count code example Before https://user-images.githubusercontent.com/40224023/211459050-6c824af6-4ed2-46ab-8f94-25e5fd02d979.png";> After https://user-images.githubusercontent.com/40224023/211459062-76816b76-7a31-4211-a024-8aecd3f81fa8.png";> Author: drgnchan <40224023+drgnc...@users.noreply.github.com> Closes #431 from drgnchan/asf-site. --- examples.md| 5 + site/examples.html | 6 ++ 2 files changed, 11 insertions(+) diff --git a/examples.md b/examples.md index 8061ac430..d9362784d 100644 --- a/examples.md +++ b/examples.md @@ -26,6 +26,11 @@ In this page, we will show examples using RDD API as well as examples using high Word count In this example, we use a few transformations to build a dataset of (String, Int) pairs called counts and then save it to a file. + + Python + Scala + Java + diff --git a/site/examples.html b/site/examples.html index b41f9a095..71cf0c60a 100644 --- a/site/examples.html +++ b/site/examples.html @@ -142,6 +142,12 @@ In this page, we will show examples using RDD API as well as examples using high Word count In this example, we use a few transformations to build a dataset of (String, Int) pairs called counts and then save it to a file. + + Python + Scala + Java + + - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org