[spark] branch master updated (10d7acc81c5 -> d5f758ea359)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 10d7acc81c5 [SPARK-43889][PYTHON] add check for column name for `__dir__()` to filter out illegal column name add d5f758ea359 [SPARK-43792][SQL][PYTHON][CONNECT] Add optional pattern for Catalog.listCatalogs No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/catalog/Catalog.scala | 8 .../org/apache/spark/sql/internal/CatalogImpl.scala | 12 .../scala/org/apache/spark/sql/CatalogSuite.scala | 5 + .../src/main/protobuf/spark/connect/catalog.proto | 5 - .../sql/connect/planner/SparkConnectPlanner.scala | 6 +- project/MimaExcludes.scala | 4 +++- python/pyspark/sql/catalog.py | 21 +++-- python/pyspark/sql/connect/catalog.py | 4 ++-- python/pyspark/sql/connect/plan.py | 8 ++-- python/pyspark/sql/connect/proto/catalog_pb2.py | 4 ++-- python/pyspark/sql/connect/proto/catalog_pb2.pyi| 14 ++ .../org/apache/spark/sql/catalog/Catalog.scala | 7 +++ .../org/apache/spark/sql/catalog/interface.scala| 7 ++- .../org/apache/spark/sql/internal/CatalogImpl.scala | 10 ++ .../apache/spark/sql/internal/CatalogSuite.scala| 4 15 files changed, 103 insertions(+), 16 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-43882][SQL] Assign name to _LEGACY_ERROR_TEMP_2122
This is an automated email from the ASF dual-hosted git repository. maxgekk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 2687d784fe4 [SPARK-43882][SQL] Assign name to _LEGACY_ERROR_TEMP_2122 2687d784fe4 is described below commit 2687d784fe4d20af321f11074139c0ce382bbaef Author: Jia Fan AuthorDate: Wed May 31 10:26:15 2023 +0300 [SPARK-43882][SQL] Assign name to _LEGACY_ERROR_TEMP_2122 ### What changes were proposed in this pull request? This PR proposes to assign name to _LEGACY_ERROR_TEMP_2122, "FAILED_PARSE_STRUCT_TYPE". ### Why are the changes needed? Assign proper name to _LEGACY_ERROR_TEMP_2122 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Add new test Closes #41381 from Hisoka-X/SPARK-43882_LEGACY_ERROR_TEMP_2122. Authored-by: Jia Fan Signed-off-by: Max Gekk --- core/src/main/resources/error/error-classes.json | 11 ++- .../org/apache/spark/sql/errors/QueryExecutionErrors.scala| 4 ++-- .../apache/spark/sql/errors/QueryExecutionErrorsSuite.scala | 10 ++ 3 files changed, 18 insertions(+), 7 deletions(-) diff --git a/core/src/main/resources/error/error-classes.json b/core/src/main/resources/error/error-classes.json index 8c3ba1e190d..7f2b1975855 100644 --- a/core/src/main/resources/error/error-classes.json +++ b/core/src/main/resources/error/error-classes.json @@ -634,6 +634,12 @@ ], "sqlState" : "38000" }, + "FAILED_PARSE_STRUCT_TYPE" : { +"message" : [ + "Failed parsing struct: ." +], +"sqlState" : "22018" + }, "FAILED_RENAME_PATH" : { "message" : [ "Failed to rename to as destination already exists." @@ -4563,11 +4569,6 @@ "Do not support type ." ] }, - "_LEGACY_ERROR_TEMP_2122" : { -"message" : [ - "Failed parsing : ." -] - }, "_LEGACY_ERROR_TEMP_2124" : { "message" : [ "Failed to merge decimal types with incompatible scale and ." diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala index 5daa8ed3b7f..7ce3e7a9e7e 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/errors/QueryExecutionErrors.scala @@ -1305,8 +1305,8 @@ private[sql] object QueryExecutionErrors extends QueryErrorsBase { def failedParsingStructTypeError(raw: String): SparkRuntimeException = { new SparkRuntimeException( - errorClass = "_LEGACY_ERROR_TEMP_2122", - messageParameters = Map("simpleString" -> StructType.simpleString, "raw" -> raw)) + errorClass = "FAILED_PARSE_STRUCT_TYPE", + messageParameters = Map("raw" -> toSQLValue(raw, StringType))) } def cannotMergeDecimalTypesWithIncompatibleScaleError( diff --git a/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala index 4bcb1d115b7..6d2c2600cbb 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/errors/QueryExecutionErrorsSuite.scala @@ -633,6 +633,16 @@ class QueryExecutionErrorsSuite "config" -> s${SQLConf.ANSI_ENABLED.key})) } + test("FAILED_PARSE_STRUCT_TYPE: parsing invalid struct type") { +val raw = """{"type":"array","elementType":"integer","containsNull":false}""" +checkError( + exception = intercept[SparkRuntimeException] { +StructType.fromString(raw) + }, + errorClass = "FAILED_PARSE_STRUCT_TYPE", + parameters = Map("raw" -> s"'$raw'")) + } + test("CAST_OVERFLOW: from long to ANSI intervals") { Seq( LongType -> "9223372036854775807L", - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-43081][ML][FOLLOW-UP] Improve torch distributor data loader code
This is an automated email from the ASF dual-hosted git repository. weichenxu123 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new c2060e7c0a3 [SPARK-43081][ML][FOLLOW-UP] Improve torch distributor data loader code c2060e7c0a3 is described below commit c2060e7c0a332c20f527adeb34a52042237430e4 Author: Weichen Xu AuthorDate: Wed May 31 16:34:19 2023 +0800 [SPARK-43081][ML][FOLLOW-UP] Improve torch distributor data loader code ### What changes were proposed in this pull request? ### Why are the changes needed? Improve torch distributor data loader code: * Add a verification that num_processes must match input spark dataframe partitions. This makes user debug easier when they set mismatched input dataframe, otherwise torch package will raise intricate error information. * Improve column value conversion in torch dataloader. Avoid comparing type operation for every column values. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? UT. Closes #41382 from WeichenXu123/improve-torch-dataloader. Authored-by: Weichen Xu Signed-off-by: Weichen Xu --- python/pyspark/ml/torch/data.py| 50 -- python/pyspark/ml/torch/distributor.py | 6 2 files changed, 36 insertions(+), 20 deletions(-) diff --git a/python/pyspark/ml/torch/data.py b/python/pyspark/ml/torch/data.py index a52b96a9392..d421683c16d 100644 --- a/python/pyspark/ml/torch/data.py +++ b/python/pyspark/ml/torch/data.py @@ -17,7 +17,7 @@ import torch import numpy as np -from typing import Any, Iterator +from typing import Any, Callable, Iterator from pyspark.sql.types import StructType @@ -26,27 +26,37 @@ class _SparkPartitionTorchDataset(torch.utils.data.IterableDataset): self.arrow_file_path = arrow_file_path self.num_samples = num_samples self.field_types = [field.dataType.simpleString() for field in schema] +self.field_converters = [ +_SparkPartitionTorchDataset._get_field_converter(field_type) +for field_type in self.field_types +] @staticmethod -def _extract_field_value(value: Any, field_type: str) -> Any: -# TODO: avoid checking field type for every row. +def _get_field_converter(field_type: str) -> Callable[[Any], Any]: if field_type == "vector": -if value["type"] == 1: -# dense vector -return value["values"] -if value["type"] == 0: -# sparse vector -size = int(value["size"]) -sparse_array = np.zeros(size, dtype=np.float64) -sparse_array[value["indices"]] = value["values"] -return sparse_array -if field_type in ["float", "double", "int", "bigint", "smallint"]: -return value -raise ValueError( -"SparkPartitionTorchDataset does not support loading data from field of " -f"type {field_type}." -) +def converter(value: Any) -> Any: +if value["type"] == 1: +# dense vector +return value["values"] +if value["type"] == 0: +# sparse vector +size = int(value["size"]) +sparse_array = np.zeros(size, dtype=np.float64) +sparse_array[value["indices"]] = value["values"] +return sparse_array + +elif field_type in ["float", "double", "int", "bigint", "smallint"]: + +def converter(value: Any) -> Any: +return value + +else: +raise ValueError( +"SparkPartitionTorchDataset does not support loading data from field of " +f"type {field_type}." +) +return converter def __iter__(self) -> Iterator[Any]: from pyspark.sql.pandas.serializers import ArrowStreamSerializer @@ -71,8 +81,8 @@ class _SparkPartitionTorchDataset(torch.utils.data.IterableDataset): batch_pdf = batch.to_pandas() for row in batch_pdf.itertuples(index=False): yield [ - _SparkPartitionTorchDataset._extract_field_value(value, field_type) -for value, field_type in zip(row, self.field_types) +field_converter(value) +for value, field_converter in zip(row, self.field_converters) ] count += 1 if count == self.num_samples: diff --git a/python/pyspark/ml/torch/distributor.py b/python/pyspark/ml/torch/distributor.py index 0249e6b4b2c..711f76db09b 100644 ---
[spark] branch master updated (c2060e7c0a3 -> 3457b4be356)
This is an automated email from the ASF dual-hosted git repository. maxgekk pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from c2060e7c0a3 [SPARK-43081][ML][FOLLOW-UP] Improve torch distributor data loader code add 3457b4be356 [SPARK-43852][SPARK-43853][SPARK-43854][SPARK-43855][SPARK-43856] Assign names to the error class _LEGACY_ERROR_TEMP_2418-2425 No new revisions were added by this update. Summary of changes: core/src/main/resources/error/error-classes.json | 57 -- .../sql/tests/pandas/test_pandas_udf_scalar.py | 4 +- python/pyspark/sql/tests/test_udf.py | 4 +- .../sql/catalyst/analysis/CheckAnalysis.scala | 18 +++ .../sql/catalyst/analysis/AnalysisErrorSuite.scala | 28 --- .../apache/spark/sql/DataFrameAsOfJoinSuite.scala | 29 ++- .../apache/spark/sql/LateralColumnAliasSuite.scala | 32 .../sql/hive/execution/AggregationQuerySuite.scala | 25 ++ .../spark/sql/hive/execution/HiveUDAFSuite.scala | 15 -- .../spark/sql/hive/execution/UDAQuerySuite.scala | 25 ++ 10 files changed, 145 insertions(+), 92 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (3457b4be356 -> fead25ac4d6)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 3457b4be356 [SPARK-43852][SPARK-43853][SPARK-43854][SPARK-43855][SPARK-43856] Assign names to the error class _LEGACY_ERROR_TEMP_2418-2425 add fead25ac4d6 [SPARK-43775][SQL] DataSource V2: Allow representing updates as deletes and inserts No new revisions were added by this update. Summary of changes: .../spark/sql/connector/write/SupportsDelta.java | 10 .../sql/catalyst/analysis/CheckAnalysis.scala | 1 + .../catalyst/analysis/RewriteRowLevelCommand.scala | 31 - .../sql/catalyst/analysis/RewriteUpdateTable.scala | 54 ++ .../catalog/InMemoryRowLevelOperationTable.scala | 5 ++ ...taBasedUpdateAsDeleteAndInsertTableSuite.scala} | 3 +- .../spark/sql/connector/UpdateTableSuiteBase.scala | 15 ++ 7 files changed, 108 insertions(+), 11 deletions(-) copy sql/core/src/test/scala/org/apache/spark/sql/connector/{DeltaBasedUpdateTableSuite.scala => DeltaBasedUpdateAsDeleteAndInsertTableSuite.scala} (94%) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (fead25ac4d6 -> d3f76c6ca07)
This is an automated email from the ASF dual-hosted git repository. hvanhovell pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from fead25ac4d6 [SPARK-43775][SQL] DataSource V2: Allow representing updates as deletes and inserts add d3f76c6ca07 [SPARK-43894][PYTHON] Fix bug in df.cache() No new revisions were added by this update. Summary of changes: python/pyspark/sql/connect/dataframe.py| 4 +--- python/pyspark/sql/tests/connect/test_connect_basic.py | 6 ++ 2 files changed, 7 insertions(+), 3 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.4 updated: [SPARK-43894][PYTHON] Fix bug in df.cache()
This is an automated email from the ASF dual-hosted git repository. hvanhovell pushed a commit to branch branch-3.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.4 by this push: new 0e1401dc71b [SPARK-43894][PYTHON] Fix bug in df.cache() 0e1401dc71b is described below commit 0e1401dc71b5aee540a54fc6a36a1857b13390b4 Author: Martin Grund AuthorDate: Wed May 31 11:55:19 2023 -0400 [SPARK-43894][PYTHON] Fix bug in df.cache() ### What changes were proposed in this pull request? Previously calling `df.cache()` would result in an invalid plan input exception because we did not invoke `persist()` with the right arguments. This patch simplifies the logic and makes it compatible to the behavior in Spark itself. ### Why are the changes needed? Bug ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Added UT Closes #41399 from grundprinzip/df_cache. Authored-by: Martin Grund Signed-off-by: Herman van Hovell (cherry picked from commit d3f76c6ca07a7a11fd228dde770186c0fbc3f03f) Signed-off-by: Herman van Hovell --- python/pyspark/sql/connect/dataframe.py| 4 +--- python/pyspark/sql/tests/connect/test_connect_basic.py | 6 ++ 2 files changed, 7 insertions(+), 3 deletions(-) diff --git a/python/pyspark/sql/connect/dataframe.py b/python/pyspark/sql/connect/dataframe.py index ca2e1b7a0dc..03049109061 100644 --- a/python/pyspark/sql/connect/dataframe.py +++ b/python/pyspark/sql/connect/dataframe.py @@ -1544,9 +1544,7 @@ class DataFrame: def cache(self) -> "DataFrame": if self._plan is None: raise Exception("Cannot cache on empty plan.") -relation = self._plan.plan(self._session.client) -self._session.client._analyze(method="persist", relation=relation) -return self +return self.persist() cache.__doc__ = PySparkDataFrame.cache.__doc__ diff --git a/python/pyspark/sql/tests/connect/test_connect_basic.py b/python/pyspark/sql/tests/connect/test_connect_basic.py index 008b95d6f14..b051b9233c8 100644 --- a/python/pyspark/sql/tests/connect/test_connect_basic.py +++ b/python/pyspark/sql/tests/connect/test_connect_basic.py @@ -3032,6 +3032,12 @@ class SparkConnectBasicTests(SparkConnectSQLTestCase): message_parameters={"attr_name": "_jreader"}, ) +def test_df_caache(self): +df = self.connect.range(10) +df.cache() +self.assert_eq(10, df.count()) +self.assertTrue(df.is_cached) + class SparkConnectSessionTests(SparkConnectSQLTestCase): def _check_no_active_session_error(self, e: PySparkException): - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-43888][CORE] Relocate Logging to common/utils
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 51359b3 [SPARK-43888][CORE] Relocate Logging to common/utils 51359b3 is described below commit 51359b3e57de8421b1929045de9ff48890b8 Author: Rui Wang AuthorDate: Wed May 31 10:02:14 2023 -0700 [SPARK-43888][CORE] Relocate Logging to common/utils ### What changes were proposed in this pull request? We can relocate logging to common/utils so share it among Spark Connect client and Spark core. ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? Towards the goal that Spark Connect client do not need to depend on Spark core. ### How was this patch tested? Existing UT Closes #41391 from amaliujia/refactor_logging. Authored-by: Rui Wang Signed-off-by: Dongjoon Hyun --- common/utils/pom.xml | 29 ++ .../scala/org/apache/spark/internal/Logging.scala | 6 ++--- core/pom.xml | 29 -- 3 files changed, 32 insertions(+), 32 deletions(-) diff --git a/common/utils/pom.xml b/common/utils/pom.xml index 53df20b646d..ee10a606182 100644 --- a/common/utils/pom.xml +++ b/common/utils/pom.xml @@ -51,6 +51,35 @@ org.apache.commons commons-text + + org.slf4j + slf4j-api + + + + org.slf4j + jul-to-slf4j + + + org.slf4j + jcl-over-slf4j + + + org.apache.logging.log4j + log4j-slf4j2-impl + + + org.apache.logging.log4j + log4j-api + + + org.apache.logging.log4j + log4j-core + + + org.apache.logging.log4j + log4j-1.2-api + target/scala-${scala.binary.version}/classes diff --git a/core/src/main/scala/org/apache/spark/internal/Logging.scala b/common/utils/src/main/scala/org/apache/spark/internal/Logging.scala similarity index 98% rename from core/src/main/scala/org/apache/spark/internal/Logging.scala rename to common/utils/src/main/scala/org/apache/spark/internal/Logging.scala index b6e3914622a..83e01330ce3 100644 --- a/core/src/main/scala/org/apache/spark/internal/Logging.scala +++ b/common/utils/src/main/scala/org/apache/spark/internal/Logging.scala @@ -27,7 +27,7 @@ import org.apache.logging.log4j.core.filter.AbstractFilter import org.slf4j.{Logger, LoggerFactory} import org.apache.spark.internal.Logging.SparkShellLoggingFilter -import org.apache.spark.util.Utils +import org.apache.spark.util.SparkClassUtils /** * Utility trait for classes that want to log data. Creates a SLF4J logger for the class and allows @@ -133,7 +133,7 @@ trait Logging { if (Logging.islog4j2DefaultConfigured()) { Logging.defaultSparkLog4jConfig = true val defaultLogProps = "org/apache/spark/log4j2-defaults.properties" -Option(Utils.getSparkClassLoader.getResource(defaultLogProps)) match { + Option(SparkClassUtils.getSparkClassLoader.getResource(defaultLogProps)) match { case Some(url) => val context = LogManager.getContext(false).asInstanceOf[LoggerContext] context.setConfigLocation(url.toURI) @@ -197,7 +197,7 @@ private[spark] object Logging { try { // We use reflection here to handle the case where users remove the // slf4j-to-jul bridge order to route their logs to JUL. -val bridgeClass = Utils.classForName("org.slf4j.bridge.SLF4JBridgeHandler") +val bridgeClass = SparkClassUtils.classForName("org.slf4j.bridge.SLF4JBridgeHandler") bridgeClass.getMethod("removeHandlersForRootLogger").invoke(null) val installed = bridgeClass.getMethod("isInstalled").invoke(null).asInstanceOf[Boolean] if (!installed) { diff --git a/core/pom.xml b/core/pom.xml index 4413ba6b634..79bf8a21635 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -218,35 +218,6 @@ com.google.code.findbugs jsr305 - - org.slf4j - slf4j-api - - - - org.slf4j - jul-to-slf4j - - - org.slf4j - jcl-over-slf4j - - - org.apache.logging.log4j - log4j-slf4j2-impl - - - org.apache.logging.log4j - log4j-api - - - org.apache.logging.log4j - log4j-core - - - org.apache.logging.log4j - log4j-1.2-api - com.ning - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-43867][SQL] Improve suggested candidates for unresolved attribute
This is an automated email from the ASF dual-hosted git repository. maxgekk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new a8893422752 [SPARK-43867][SQL] Improve suggested candidates for unresolved attribute a8893422752 is described below commit a88934227523334550e451e437ce013772001079 Author: Max Gekk AuthorDate: Wed May 31 21:04:44 2023 +0300 [SPARK-43867][SQL] Improve suggested candidates for unresolved attribute ### What changes were proposed in this pull request? In the PR, I propose to change the approach of stripping the common part of candidate qualifiers in `StringUtils.orderSuggestedIdentifiersBySimilarity`: 1. If all candidates have the same qualifier including namespace and table name, drop it. It should be dropped if the base string (unresolved attribute) doesn't include a namespace and table name. For example: - `[ns1.table1.col1, ns1.table1.col2] -> [col1, col2]` for unresolved attribute `col0` - `[ns1.table1.col1, ns1.table1.col2] -> [table1.col1, table1.col2]` for unresolved attribute `table1.col0` 2. If all candidates belong to the same namespace, just drop it. It should be dropped for any non-fully qualified unresolved attribute. For example: - `[ns1.table1.col1, ns1.table2.col2] -> [table1.col1, table2.col2]` for unresolved attribute `col0` or `table0.col0` - `[ns1.table1.col1, ns1.table1.col2] -> [ns1.table1.col1, ns1.table1.col2]` for unresolved attribute `ns0.table0.col0` 4. Otherwise take the suggested candidates AS IS. 5. Sort the candidate list using the levenshtein distance. ### Why are the changes needed? This should improve user experience with Spark SQL by simplifying the error message about an unresolved attribute. ### Does this PR introduce _any_ user-facing change? Yes, it changes the error message. ### How was this patch tested? By running the existing test suites: ``` $ build/sbt "test:testOnly *AnalysisErrorSuite" $ build/sbt "test:testOnly *QueryCompilationErrorsSuite" $ PYSPARK_PYTHON=python3 build/sbt "sql/testOnly org.apache.spark.sql.SQLQueryTestSuite" $ build/sbt "test:testOnly *DatasetUnpivotSuite" $ build/sbt "test:testOnly *DatasetSuite" ``` Closes #41368 from MaxGekk/fix-suggested-column-list. Authored-by: Max Gekk Signed-off-by: Max Gekk --- .../sql/catalyst/analysis/CheckAnalysis.scala | 3 +- .../plans/logical/basicLogicalOperators.scala | 2 +- .../spark/sql/catalyst/util/StringUtils.scala | 46 +- .../sql/catalyst/analysis/AnalysisErrorSuite.scala | 4 +- .../spark/sql/catalyst/util/StringUtilsSuite.scala | 5 ++- .../columnresolution-negative.sql.out | 2 +- .../analyzer-results/group-by-all.sql.out | 2 +- .../analyzer-results/join-lateral.sql.out | 2 +- .../postgreSQL/aggregates_part1.sql.out| 2 +- .../analyzer-results/postgreSQL/join.sql.out | 6 +-- .../udf/postgreSQL/udf-aggregates_part1.sql.out| 2 +- .../udf/postgreSQL/udf-join.sql.out| 6 +-- .../results/columnresolution-negative.sql.out | 2 +- .../sql-tests/results/group-by-all.sql.out | 2 +- .../sql-tests/results/join-lateral.sql.out | 2 +- .../results/postgreSQL/aggregates_part1.sql.out| 2 +- .../sql-tests/results/postgreSQL/join.sql.out | 6 +-- .../udf/postgreSQL/udf-aggregates_part1.sql.out| 2 +- .../results/udf/postgreSQL/udf-join.sql.out| 6 +-- .../org/apache/spark/sql/DatasetUnpivotSuite.scala | 2 +- .../spark/sql/connector/DataSourceV2SQLSuite.scala | 4 +- .../sql/errors/QueryCompilationErrorsSuite.scala | 3 +- 22 files changed, 53 insertions(+), 60 deletions(-) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala index c46dff1c4bf..594c0b666e8 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala @@ -139,7 +139,8 @@ trait CheckAnalysis extends PredicateHelper with LookupCatalog with QueryErrorsB a: Attribute, errorClass: String): Nothing = { val missingCol = a.sql -val candidates = operator.inputSet.toSeq.map(_.qualifiedName) +val candidates = operator.inputSet.toSeq + .map(attr => attr.qualifier :+ attr.name) val orderedCandidates = StringUtils.orderSuggestedIdentifiersBySimilarity(missingCol, candidates) throw QueryCompilationErrors.unresolvedAttributeError( diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/basicLogicalOperators.scala b/sql/catalys
[spark] branch master updated (a8893422752 -> 47ddac13144)
This is an automated email from the ASF dual-hosted git repository. ueshin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from a8893422752 [SPARK-43867][SQL] Improve suggested candidates for unresolved attribute add 47ddac13144 [SPARK-43804][PYTHON][TESTS] Test on nested structs support in Pandas UDF No new revisions were added by this update. Summary of changes: .../sql/tests/connect/test_parity_pandas_udf.py| 3 - .../sql/tests/pandas/test_pandas_udf_scalar.py | 107 + 2 files changed, 89 insertions(+), 21 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-43895][CONNECT][GO] Prepare the go package path
This is an automated email from the ASF dual-hosted git repository. gengliang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 419f46ab621 [SPARK-43895][CONNECT][GO] Prepare the go package path 419f46ab621 is described below commit 419f46ab621e72e64dc9b897e416e56ea348cf1e Author: Martin Grund AuthorDate: Wed May 31 13:17:09 2023 -0700 [SPARK-43895][CONNECT][GO] Prepare the go package path ### What changes were proposed in this pull request? This patch adds the golang package path to the proto files to be consumed from the golang repo. ### Why are the changes needed? Preparation for the go client. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? No functional change. Closes #41403 from grundprinzip/SPARK-43895. Authored-by: Martin Grund Signed-off-by: Gengliang Wang --- connector/connect/common/src/main/protobuf/buf.yaml | 2 ++ connector/connect/common/src/main/protobuf/spark/connect/base.proto | 1 + .../connect/common/src/main/protobuf/spark/connect/catalog.proto| 1 + .../connect/common/src/main/protobuf/spark/connect/commands.proto | 1 + .../connect/common/src/main/protobuf/spark/connect/common.proto | 1 + .../common/src/main/protobuf/spark/connect/example_plugins.proto| 1 + .../common/src/main/protobuf/spark/connect/expressions.proto| 1 + .../connect/common/src/main/protobuf/spark/connect/relations.proto | 1 + .../connect/common/src/main/protobuf/spark/connect/types.proto | 1 + python/pyspark/sql/connect/proto/base_pb2.py| 6 -- python/pyspark/sql/connect/proto/catalog_pb2.py | 6 -- python/pyspark/sql/connect/proto/commands_pb2.py| 6 -- python/pyspark/sql/connect/proto/common_pb2.py | 6 -- python/pyspark/sql/connect/proto/example_plugins_pb2.py | 6 -- python/pyspark/sql/connect/proto/expressions_pb2.py | 6 -- python/pyspark/sql/connect/proto/relations_pb2.py | 6 -- python/pyspark/sql/connect/proto/types_pb2.py | 6 -- 17 files changed, 42 insertions(+), 16 deletions(-) diff --git a/connector/connect/common/src/main/protobuf/buf.yaml b/connector/connect/common/src/main/protobuf/buf.yaml index 496e97af3fa..f17614a8dc4 100644 --- a/connector/connect/common/src/main/protobuf/buf.yaml +++ b/connector/connect/common/src/main/protobuf/buf.yaml @@ -18,6 +18,8 @@ version: v1 breaking: use: - FILE + except: +- FILE_SAME_GO_PACKAGE lint: use: - DEFAULT diff --git a/connector/connect/common/src/main/protobuf/spark/connect/base.proto b/connector/connect/common/src/main/protobuf/spark/connect/base.proto index f54e28e3b61..e869712858a 100644 --- a/connector/connect/common/src/main/protobuf/spark/connect/base.proto +++ b/connector/connect/common/src/main/protobuf/spark/connect/base.proto @@ -28,6 +28,7 @@ import "spark/connect/types.proto"; option java_multiple_files = true; option java_package = "org.apache.spark.connect.proto"; +option go_package = "internal/generated"; // A [[Plan]] is the structure that carries the runtime information for the execution from the // client to the server. A [[Plan]] can either be of the type [[Relation]] which is a reference diff --git a/connector/connect/common/src/main/protobuf/spark/connect/catalog.proto b/connector/connect/common/src/main/protobuf/spark/connect/catalog.proto index 9729c102269..f048dbc7f25 100644 --- a/connector/connect/common/src/main/protobuf/spark/connect/catalog.proto +++ b/connector/connect/common/src/main/protobuf/spark/connect/catalog.proto @@ -24,6 +24,7 @@ import "spark/connect/types.proto"; option java_multiple_files = true; option java_package = "org.apache.spark.connect.proto"; +option go_package = "internal/generated"; // Catalog messages are marked as unstable. message Catalog { diff --git a/connector/connect/common/src/main/protobuf/spark/connect/commands.proto b/connector/connect/common/src/main/protobuf/spark/connect/commands.proto index 87d76c5d63f..e716364f69b 100644 --- a/connector/connect/common/src/main/protobuf/spark/connect/commands.proto +++ b/connector/connect/common/src/main/protobuf/spark/connect/commands.proto @@ -26,6 +26,7 @@ package spark.connect; option java_multiple_files = true; option java_package = "org.apache.spark.connect.proto"; +option go_package = "internal/generated"; // A [[Command]] is an operation that is executed by the server that does not directly consume or // produce a relational result. diff --git a/connector/connect/common/src/main/protobuf/spark/connect/common.proto b/connector/connect/common/src/main/protobuf/spark/connect/common.proto index 42cac88ea3f..5c538cf108
[spark] branch master updated: [SPARK-43333][SQL] Allow Avro to convert union type to SQL with field name stable with type
This is an automated email from the ASF dual-hosted git repository. gengliang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 8f2afb88d42 [SPARK-4][SQL] Allow Avro to convert union type to SQL with field name stable with type 8f2afb88d42 is described below commit 8f2afb88d42af04f36c84972d9ebcb5dabc91260 Author: Siying Dong AuthorDate: Wed May 31 13:32:02 2023 -0700 [SPARK-4][SQL] Allow Avro to convert union type to SQL with field name stable with type ### What changes were proposed in this pull request? Introduce AvroOption "enableStableIdentifiersForUnionType". If it is set to true (default remains to be false), Avro's union is converted to SQL schema by naming field name "member_" + type name. This is to try to keep field name stable with type name. ### Why are the changes needed? The purpose of this is twofold: To allow adding or removing types to the union without affecting the record names of other member types. If the new or removed type is not ordered last, then existing queries referencing "member2" may need to be rewritten to reference "member1" or "member3". Referencing the type name in the query is more readable than referencing "member0". For example, our system produces an avro schema from a Java type structure where subtyping maps to union types whose members are ordered lexicographically. Adding a subtype can therefore easily result in all references to "member2" needing to be updated to "member3". ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Add a unit test that covers all types supported in union, as well as some potential name conflict cases. Closes #41263 from siying/avro_stable_union. Authored-by: Siying Dong Signed-off-by: Gengliang Wang --- .../apache/spark/sql/avro/AvroDataToCatalyst.scala | 2 +- .../org/apache/spark/sql/avro/AvroOptions.scala| 10 + .../org/apache/spark/sql/avro/AvroUtils.scala | 2 +- .../apache/spark/sql/avro/SchemaConverters.scala | 62 -- .../apache/spark/sql/avro/AvroFunctionsSuite.scala | 9 +- .../org/apache/spark/sql/avro/AvroSuite.scala | 222 ++--- docs/sql-data-sources-avro.md | 8 +- 7 files changed, 274 insertions(+), 41 deletions(-) diff --git a/connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDataToCatalyst.scala b/connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDataToCatalyst.scala index f8718edd97f..59f2999bdd3 100644 --- a/connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDataToCatalyst.scala +++ b/connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroDataToCatalyst.scala @@ -39,7 +39,7 @@ private[sql] case class AvroDataToCatalyst( override def inputTypes: Seq[AbstractDataType] = Seq(BinaryType) override lazy val dataType: DataType = { -val dt = SchemaConverters.toSqlType(expectedSchema).dataType +val dt = SchemaConverters.toSqlType(expectedSchema, options).dataType parseMode match { // With PermissiveMode, the output Catalyst row might contain columns of null values for // corrupt records, even if some of the columns are not nullable in the user-provided schema. diff --git a/connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroOptions.scala b/connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroOptions.scala index 95001bb8150..c8057ca5879 100644 --- a/connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroOptions.scala +++ b/connector/avro/src/main/scala/org/apache/spark/sql/avro/AvroOptions.scala @@ -130,6 +130,9 @@ private[sql] class AvroOptions( val datetimeRebaseModeInRead: String = parameters .get(DATETIME_REBASE_MODE) .getOrElse(SQLConf.get.getConf(SQLConf.AVRO_REBASE_MODE_IN_READ)) + + val useStableIdForUnionType: Boolean = +parameters.get(STABLE_ID_FOR_UNION_TYPE).map(_.toBoolean).getOrElse(false) } private[sql] object AvroOptions extends DataSourceOptions { @@ -154,4 +157,11 @@ private[sql] object AvroOptions extends DataSourceOptions { // datasource similarly to the SQL config `spark.sql.avro.datetimeRebaseModeInRead`, // and can be set to the same values: `EXCEPTION`, `LEGACY` or `CORRECTED`. val DATETIME_REBASE_MODE = newOption("datetimeRebaseMode") + // If it is set to true, Avro schema is deserialized into Spark SQL schema, and the Avro Union + // type is transformed into a structure where the field names remain consistent with their + // respective types. The resulting field names are converted to lowercase, e.g. member_int or + // member_string. If two user-defined type names or a user-defined type name and a built-in + // type name are identical regardless of case, an exception will be raised. However, in other + // cases, the field
[spark-connect-go] branch master updated: [SPARK-43895] Basic Repository Layout
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark-connect-go.git The following commit(s) were added to refs/heads/master by this push: new aa4cef7 [SPARK-43895] Basic Repository Layout aa4cef7 is described below commit aa4cef71a43192b70120ce075a8e2e9076207f09 Author: Martin Grund AuthorDate: Thu Jun 1 09:10:21 2023 +0900 [SPARK-43895] Basic Repository Layout ## Proposed Changes In order to prepare the implementation of the Go client. This patch prepares the repository layout. * Adds the git submodule for the Spark repository, currently pointing at `master` for the reference to the proto files. * Adds the Makefile and the necessary outline for the build system * Adds the `buf` generation logic for the protos. ## Testing `make gen && make check && make test` Closes #3 from grundprinzip/SPARK-43895. Lead-authored-by: Martin Grund Co-authored-by: Hyukjin Kwon Signed-off-by: Hyukjin Kwon --- .gitignore| 23 + .gitmodules | 3 ++ Makefile | 105 ++ README.md | 9 + buf.gen.yaml | 23 + buf.work.yaml | 19 +++ go.mod| 53 + go.sum| 61 ++ spark | 1 + 9 files changed, 297 insertions(+) diff --git a/.gitignore b/.gitignore new file mode 100644 index 000..0bca2cf --- /dev/null +++ b/.gitignore @@ -0,0 +1,23 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +# All generated files +internal/generated +internal/generated.out + +# Ignore Coverage Files +coverage* diff --git a/.gitmodules b/.gitmodules new file mode 100644 index 000..59db423 --- /dev/null +++ b/.gitmodules @@ -0,0 +1,3 @@ +[submodule "spark"] + path = spark + url = https://github.com/apache/spark.git diff --git a/Makefile b/Makefile new file mode 100644 index 000..a9829da --- /dev/null +++ b/Makefile @@ -0,0 +1,105 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +FIRST_GOPATH := $(firstword $(subst :, ,$(GOPATH))) +PKGS := $(shell go list ./... | grep -v /tests | grep -v /xcpb | grep -v /gpb) +GOFILES_NOVENDOR := $(shell find . -name vendor -prune -o -type f -name '*.go' -not -name '*.pb.go' -print) +GOFILES_BUILD := $(shell find . -type f -name '*.go' -not -name '*_test.go') +PROTOFILES:= $(shell find . -name vendor -prune -o -type f -name '*.proto' -print) + +ALLGOFILES := $(shell find . -type f -name '*.go') +DATE := $(shell date -u -d "@$(SOURCE_DATE_EPOCH)" '+%FT%T%z' 2>/dev/null || date -u '+%FT%T%z') + +BUILDFLAGS_NOPIE := +#BUILDFLAGS_NOPIE := -trimpath -ldflags="-s -w -X main.version=$(GOPASS_VERSION) -X main.commit=$(GOPASS_REVISION) -X main.date=$(DATE)" -gcflags="-trimpath=$(GOPATH)" -asmflags="-trimpath=$(GOPATH)" +BUILDFLAGS?= $(BUILDFLAGS_NOPIE) -buildmode=pie +TESTFLAGS ?= +PWD := $(shell pwd) +PREFIX?= $(GOPATH) +BINDIR?= $(PREFIX)/bin +GO:= GO111MODULE=on go +GOOS ?= $(shell go version | cut -d' ' -f4 | c
[spark-connect-go] branch master updated: [MINOR] Fix the merge script to work
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark-connect-go.git The following commit(s) were added to refs/heads/master by this push: new ce56131 [MINOR] Fix the merge script to work ce56131 is described below commit ce56131597dfd555aa24573ca122887757d50a98 Author: Hyukjin Kwon AuthorDate: Thu Jun 1 09:12:07 2023 +0900 [MINOR] Fix the merge script to work --- merge_connect_go_pr.py | 6 +- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/merge_connect_go_pr.py b/merge_connect_go_pr.py index 417c0ff..bb514a8 100755 --- a/merge_connect_go_pr.py +++ b/merge_connect_go_pr.py @@ -478,7 +478,11 @@ def main(): branches = get_json("%s/branches" % GITHUB_API_BASE) branch_names = list(filter(lambda x: x.startswith("branch-"), [x["name"] for x in branches])) # Assumes branch names can be sorted lexicographically -latest_branch = sorted(branch_names, reverse=True)[0] +if len(branch_names) == 0: +# Remove this when we have a branch. It fails now because we don't have branch-*. +latest_branch = "master" +else: +latest_branch = sorted(branch_names, reverse=True)[0] pr_num = input("Which pull request would you like to merge? (e.g. 34): ") pr = get_json("%s/pulls/%s" % (GITHUB_API_BASE, pr_num)) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (8f2afb88d42 -> e3eba0b8877)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from 8f2afb88d42 [SPARK-4][SQL] Allow Avro to convert union type to SQL with field name stable with type add e3eba0b8877 [SPARK-43896][TESTS][PS][CONNECT] Enable `test_iterrows` and `test_itertuples` on Connect No new revisions were added by this update. Summary of changes: .../pyspark/pandas/tests/connect/indexes/test_parity_indexing.py | 8 1 file changed, 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-43516][ML][FOLLOW-UP] Make `pyspark.mlv2` module supports python < 3.9
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 33a76811b23 [SPARK-43516][ML][FOLLOW-UP] Make `pyspark.mlv2` module supports python < 3.9 33a76811b23 is described below commit 33a76811b23f2249cf9343fdc4ef654d12bd23b5 Author: Weichen Xu AuthorDate: Thu Jun 1 10:24:39 2023 +0900 [SPARK-43516][ML][FOLLOW-UP] Make `pyspark.mlv2` module supports python < 3.9 ### What changes were proposed in this pull request? Make `pyspark.mlv2` module supports python < 3.9 We need to change some type hints definition to make them compatible with python < 3.9 ### Why are the changes needed? pyspark master still need to support python 3.7 and python 3.8 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Manually run `pyspark.mlv2` tests against python 3.7 or python 3.8 Closes #41405 from WeichenXu123/fix-tpye-hints. Authored-by: Weichen Xu Signed-off-by: Hyukjin Kwon --- python/pyspark/mlv2/base.py | 5 +++-- python/pyspark/mlv2/feature.py| 7 +++ python/pyspark/mlv2/summarizer.py | 8 python/pyspark/mlv2/util.py | 7 +++ 4 files changed, 13 insertions(+), 14 deletions(-) diff --git a/python/pyspark/mlv2/base.py b/python/pyspark/mlv2/base.py index 4c0d4652928..dc503db71c0 100644 --- a/python/pyspark/mlv2/base.py +++ b/python/pyspark/mlv2/base.py @@ -16,7 +16,6 @@ # from abc import ABCMeta, abstractmethod -from collections.abc import Callable import pandas as pd @@ -28,6 +27,8 @@ from typing import ( TypeVar, Union, TYPE_CHECKING, +Tuple, +Callable, ) from pyspark import since @@ -123,7 +124,7 @@ class Transformer(Params, metaclass=ABCMeta): """ raise NotImplementedError() -def _output_columns(self) -> list[tuple[str, str]]: +def _output_columns(self) -> List[Tuple[str, str]]: """ Return a list of output transformed columns, each elements in the list is a tuple of (column_name, column_spark_type) diff --git a/python/pyspark/mlv2/feature.py b/python/pyspark/mlv2/feature.py index 6bbcdf7eaac..cecff362823 100644 --- a/python/pyspark/mlv2/feature.py +++ b/python/pyspark/mlv2/feature.py @@ -15,10 +15,9 @@ # limitations under the License. # -from collections.abc import Callable import numpy as np import pandas as pd -from typing import Any, Union +from typing import Any, Union, List, Tuple, Callable from pyspark.sql import DataFrame from pyspark.mlv2.base import Estimator, Model @@ -60,7 +59,7 @@ class MaxAbsScalerModel(Model, HasInputCol, HasOutputCol): def _input_column_name(self) -> str: return self.getInputCol() -def _output_columns(self) -> list[tuple[str, str]]: +def _output_columns(self) -> List[Tuple[str, str]]: return [(self.getOutputCol(), "array")] def _get_transform_fn(self) -> Callable[["pd.Series"], Any]: @@ -108,7 +107,7 @@ class StandardScalerModel(Model, HasInputCol, HasOutputCol): def _input_column_name(self) -> str: return self.getInputCol() -def _output_columns(self) -> list[tuple[str, str]]: +def _output_columns(self) -> List[Tuple[str, str]]: return [(self.getOutputCol(), "array")] def _get_transform_fn(self) -> Callable[["pd.Series"], Any]: diff --git a/python/pyspark/mlv2/summarizer.py b/python/pyspark/mlv2/summarizer.py index 6bf03c26e19..18776b71eb2 100644 --- a/python/pyspark/mlv2/summarizer.py +++ b/python/pyspark/mlv2/summarizer.py @@ -17,7 +17,7 @@ import numpy as np import pandas as pd -from typing import Any, Union +from typing import Any, Union, List, Dict from pyspark.sql import DataFrame from pyspark.mlv2.util import aggregate_dataframe @@ -46,7 +46,7 @@ class SummarizerAggState: self.max_values = np.maximum(self.max_values, state.max_values) return self -def to_result(self, metrics: list[str]) -> dict[str, Any]: +def to_result(self, metrics: List[str]) -> Dict[str, Any]: result = {} for metric in metrics: @@ -75,8 +75,8 @@ class SummarizerAggState: def summarize_dataframe( -dataframe: Union["DataFrame", "pd.DataFrame"], column: str, metrics: list[str] -) -> dict[str, Any]: +dataframe: Union["DataFrame", "pd.DataFrame"], column: str, metrics: List[str] +) -> Dict[str, Any]: """ Summarize an array type column over a spark dataframe or a pandas dataframe diff --git a/python/pyspark/mlv2/util.py b/python/pyspark/mlv2/util.py index de2ffb3d7c1..9aebb3fa9a3 100644 --- a/python/pyspark/mlv2/util.py +++ b/python/pyspark/mlv2/util.py @@ -16,8 +16,7 @@ # import pandas as pd -from collections.abc import Callable, Iterable -from typing import Any, Union +from typing im
[spark] branch master updated: [SPARK-43898][CORE] Automatically register `immutable.ArraySeq$ofRef` to `KryoSerializer` for Scala 2.13
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 398435fcd6e [SPARK-43898][CORE] Automatically register `immutable.ArraySeq$ofRef` to `KryoSerializer` for Scala 2.13 398435fcd6e is described below commit 398435fcd6eb8623219c8d6c3e5966463ebe4429 Author: yangjie01 AuthorDate: Wed May 31 19:23:30 2023 -0700 [SPARK-43898][CORE] Automatically register `immutable.ArraySeq$ofRef` to `KryoSerializer` for Scala 2.13 SPARK_GENERATE_BENCHMARK_FILES=1 build/sbt clean "sql/Test/runMain org.apache.spark.sql.execution.benchmark.TPCDSQueryBenchmark --data-location /Users/yangjie01/Tools/tpcds-sf-1 --query-filter q17" -Pscala-2.13 ### What changes were proposed in this pull request? This pr aims to automatically register `immutable.ArraySeq$ofRef` to `KryoSerializer` for Scala 2.13 to make `TPCDSQueryBenchmark` can run successfully using Scala 2.13. ### Why are the changes needed? Scala 2.13 introduced `scala.collection.immutable.ArraySeq$ofRef`, but it has not been registered with `KryoSerializer`, so when run `TPCDSQueryBenchmark` using `KryoSerializer` and enabled `spark.kryo.registrationRequired`(This is the default behavior after SPARK-42074), there will be the following error: ``` Error: Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Failed to serialize task 741, not attempting to retry it. Exception during serialization: java.io.IOException: java.lang.IllegalArgumentException: Class is not registered: scala.collection.immutable.ArraySeq$ofRef Note: To register this class use: kryo.register(scala.collection.immutable.ArraySeq$ofRef.class); at org.apache.spark.scheduler.DAGScheduler.failJobAndIndependentStages(DAGScheduler.scala:2815) at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2(DAGScheduler.scala:2751) at org.apache.spark.scheduler.DAGScheduler.$anonfun$abortStage$2$adapted(DAGScheduler.scala:2750) at scala.collection.immutable.List.foreach(List.scala:333) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:2750) at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1(DAGScheduler.scala:1218) at org.apache.spark.scheduler.DAGScheduler.$anonfun$handleTaskSetFailed$1$adapted(DAGScheduler.scala:1218) at scala.Option.foreach(Option.scala:437) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:1218) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:3014) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2953) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2942) at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:983) at org.apache.spark.SparkContext.runJob(SparkContext.scala:2285) at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.writeWithV2(WriteToDataSourceV2Exec.scala:385) at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.writeWithV2$(WriteToDataSourceV2Exec.scala:359) at org.apache.spark.sql.execution.datasources.v2.OverwriteByExpressionExec.writeWithV2(WriteToDataSourceV2Exec.scala:243) at org.apache.spark.sql.execution.datasources.v2.V2ExistingTableWriteExec.run(WriteToDataSourceV2Exec.scala:337) at org.apache.spark.sql.execution.datasources.v2.V2ExistingTableWriteExec.run$(WriteToDataSourceV2Exec.scala:336) at org.apache.spark.sql.execution.datasources.v2.OverwriteByExpressionExec.run(WriteToDataSourceV2Exec.scala:243) at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:43) at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result(V2CommandExec.scala:43) at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.executeCollect(V2CommandExec.scala:49) at org.apache.spark.sql.execution.QueryExecution$$anonfun$eagerlyExecuteCommands$1.$anonfun$applyOrElse$1(QueryExecution.scala:98) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$6(SQLExecution.scala:118) at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:195) at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:103) at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:825)
[spark] branch master updated: [SPARK-43504][K8S] Mounts the hadoop config map on the executor pod
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 10ee643c674 [SPARK-43504][K8S] Mounts the hadoop config map on the executor pod 10ee643c674 is described below commit 10ee643c6746bbf110be6d680fe1c3559e522720 Author: fwang12 AuthorDate: Wed May 31 21:30:05 2023 -0700 [SPARK-43504][K8S] Mounts the hadoop config map on the executor pod ### What changes were proposed in this pull request? In this pr, for spark on k8s, the hadoop config map will be mounted in executor side as well. Before, the hadoop config map is only mounted in driver side. ### Why are the changes needed? Since [SPARK-25815](https://issues.apache.org/jira/browse/SPARK-25815) [,](https://github.com/apache/spark/pull/22911,) the hadoop config map will not be mounted in executor side. Per the https://github.com/apache/spark/pull/22911 description: > The main two things that don't need to happen in executors anymore are: > 1. adding the Hadoop config to the executor pods: this is not needed > since the Spark driver will serialize the Hadoop config and send > it to executors when running tasks. But in fact, the executor still need the hadoop configuration. ![image](https://github.com/apache/spark/assets/6757692/ff6374c9-7ebd-4472-a85c-99c75a737e2a) As shown in above picture, the driver can resolve `hdfs://zeus`, but the executor can not. so we still need to mount the hadoop config map in executor side. ### Does this PR introduce _any_ user-facing change? Yes, users do not need to take workarounds to make executors load the hadoop configuration. Such as: - including hadoop conf in executor image - placing hadoop conf files under `SPARK_CONF_DIR`. ### How was this patch tested? UT. Closes #41181 from turboFei/exec_hadoop_conf. Authored-by: fwang12 Signed-off-by: Dongjoon Hyun --- .../k8s/features/HadoopConfDriverFeatureStep.scala | 8 +++ .../features/HadoopConfExecutorFeatureStep.scala | 63 + .../cluster/k8s/KubernetesExecutorBuilder.scala| 1 + .../HadoopConfExecutorFeatureStepSuite.scala | 82 ++ 4 files changed, 154 insertions(+) diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/HadoopConfDriverFeatureStep.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/HadoopConfDriverFeatureStep.scala index 069a57d3dc4..45a5b8d7dae 100644 --- a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/HadoopConfDriverFeatureStep.scala +++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/HadoopConfDriverFeatureStep.scala @@ -104,6 +104,14 @@ private[spark] class HadoopConfDriverFeatureStep(conf: KubernetesConf) } } + override def getAdditionalPodSystemProperties(): Map[String, String] = { +if (hasHadoopConf) { + Map(HADOOP_CONFIG_MAP_NAME -> existingConfMap.getOrElse(newConfigMapName)) +} else { + Map.empty +} + } + override def getAdditionalKubernetesResources(): Seq[HasMetadata] = { if (confDir.isDefined) { val fileMap = confFiles.map { file => diff --git a/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/HadoopConfExecutorFeatureStep.scala b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/HadoopConfExecutorFeatureStep.scala new file mode 100644 index 000..8a2773c1ac3 --- /dev/null +++ b/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/HadoopConfExecutorFeatureStep.scala @@ -0,0 +1,63 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.deploy.k8s.features + +import io.fabric8.kubernetes.api.model.{ContainerBuilder, PodBuilder, VolumeBuilder} + +import org.apache.spark.deploy.k8s.{KubernetesConf, SparkPod} +impo
[spark] branch master updated: [SPARK-43421][SS] Implement Changelog based Checkpointing for RocksDB State Store Provider
This is an automated email from the ASF dual-hosted git repository. kabhwan pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new c02c8be43c6 [SPARK-43421][SS] Implement Changelog based Checkpointing for RocksDB State Store Provider c02c8be43c6 is described below commit c02c8be43c64dbd6bffceb53f8b18cd19a0d2f2e Author: Chaoqin Li AuthorDate: Thu Jun 1 14:34:24 2023 +0900 [SPARK-43421][SS] Implement Changelog based Checkpointing for RocksDB State Store Provider ### What changes were proposed in this pull request? In order to reduce the checkpoint duration and end to end latency, we propose Changelog Based Checkpointing for RocksDB State Store Provider. Below is the mechanism. 1. Changelog checkpoint: Upon each put() delete() call to local rocksdb instance, log the operation to a changelog file. During the state change commit, sync the compressed change log of the current batch to DFS as checkpointDir/{version}.delta. 2. Version reconstruction: For version j, find latest snapshot i.zip such that i <= j, load snapshot i, and replay i+1.delta ~ j.delta. This is used in loading the initial state as well as creating the latest version snapshot. Note: If a query is shutdown without exception, there won’t be changelog replay during query restart because a maintenance task is executed before the state store instance is unloaded. 3. Background snapshot: A maintenance thread in executors will launch maintenance tasks periodically. Inside the maintenance task, sync the latest RocksDB local snapshot to DFS as checkpointDir/{version}.zip. Snapshot enables faster failure recovery and allows old versions to be purged. 4. Garbage collection: Inside the maintenance task, delete snapshot and delta files from DFS for versions that is out of retention range(default retained version number is 100) ### Why are the changes needed? We have identified state checkpointing latency as one of the major performance bottlenecks for stateful streaming queries. Currently, RocksDB state store pauses the RocksDB instances to upload a snapshot to the cloud when committing a batch, which is heavy weight and has unpredictable performance. With changelog based checkpointing, we allow the RocksDB instance to run uninterruptibly, which improves RocksDB operation performance. This also dramatically reduces the commit time and batch duration because we are uploading a smaller amount of data during state commit. With this change, stateful query with RocksDB state store will have lower and more predictable latency. ### How was this patch tested? Add unit test for changelog checkpointing utility. Add unit test and integration test that check backward compatibility with existing checkpoint. Enable RocksDB state store unit test and stateful streaming query integration test to run with changelog checkpointing enabled. Closes #41099 from chaoqin-li1123/changelog. Authored-by: Chaoqin Li Signed-off-by: Jungtaek Lim --- docs/structured-streaming-programming-guide.md | 18 + .../sql/execution/streaming/state/RocksDB.scala| 187 -- .../streaming/state/RocksDBFileManager.scala | 119 -- .../state/RocksDBStateStoreProvider.scala | 8 +- .../streaming/state/StateStoreChangelog.scala | 167 + .../state/RocksDBStateStoreIntegrationSuite.scala | 60 ++- .../streaming/state/RocksDBStateStoreSuite.scala | 80 +++- .../execution/streaming/state/RocksDBSuite.scala | 402 + .../state/StateStoreCompatibilitySuite.scala | 2 +- .../streaming/state/StateStoreSuite.scala | 109 +++--- .../streaming/FlatMapGroupsWithStateSuite.scala| 3 + .../sql/streaming/RocksDBStateStoreTest.scala | 52 +++ .../sql/streaming/StreamingAggregationSuite.scala | 3 + .../streaming/StreamingDeduplicationSuite.scala| 3 + 14 files changed, 1012 insertions(+), 201 deletions(-) diff --git a/docs/structured-streaming-programming-guide.md b/docs/structured-streaming-programming-guide.md index 267596a3899..53d5919d4dc 100644 --- a/docs/structured-streaming-programming-guide.md +++ b/docs/structured-streaming-programming-guide.md @@ -2320,6 +2320,11 @@ Here are the configs regarding to RocksDB instance of the state store provider: Whether we perform a range compaction of RocksDB instance for commit operation False + + spark.sql.streaming.stateStore.rocksdb.changelogCheckpointing.enabled +Whether to upload changelog instead of snapshot during RocksDB StateStore commit +False + spark.sql.streaming.stateStore.rocksdb.blockSizeKB Approximate size in KB of user data packed per block for a RocksDB BlockBasedTable, which is a RocksDB's default SST file format. @@ -2389,6 +2394,19 @@ If you want to cap R
[spark] branch master updated (c02c8be43c6 -> c8c6b0c3308)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git from c02c8be43c6 [SPARK-43421][SS] Implement Changelog based Checkpointing for RocksDB State Store Provider add c8c6b0c3308 Revert [SPARK-43836][SPARK-43845][SPARK-43858][SPARK-43858] to keep Scala 2.12 for Spark 3.x No new revisions were added by this update. Summary of changes: .github/workflows/benchmark.yml| 4 +- .../{build_scala212.yml => build_scala213.yml} | 4 +- assembly/pom.xml | 4 +- common/kvstore/pom.xml | 4 +- common/network-common/pom.xml | 4 +- common/network-shuffle/pom.xml | 4 +- common/network-yarn/pom.xml| 4 +- common/sketch/pom.xml | 4 +- common/tags/pom.xml| 4 +- common/unsafe/pom.xml | 4 +- common/utils/pom.xml | 4 +- connector/avro/pom.xml | 8 +- connector/connect/client/jvm/pom.xml | 4 +- connector/connect/common/pom.xml | 4 +- connector/connect/server/pom.xml | 8 +- connector/docker-integration-tests/pom.xml | 4 +- connector/kafka-0-10-assembly/pom.xml | 4 +- connector/kafka-0-10-sql/pom.xml | 8 +- connector/kafka-0-10-token-provider/pom.xml| 4 +- connector/kafka-0-10/pom.xml | 8 +- connector/kinesis-asl-assembly/pom.xml | 4 +- connector/kinesis-asl/pom.xml | 4 +- connector/protobuf/pom.xml | 8 +- connector/spark-ganglia-lgpl/pom.xml | 4 +- core/pom.xml | 8 +- dev/deps/spark-deps-hadoop-3-hive-2.3 | 43 +++ dev/mima | 8 +- dev/test-dependencies.sh | 2 +- docs/_plugins/copy_api_dirs.rb | 14 +-- examples/pom.xml | 4 +- graphx/pom.xml | 4 +- hadoop-cloud/pom.xml | 4 +- launcher/pom.xml | 4 +- mllib-local/pom.xml| 4 +- mllib/pom.xml | 8 +- pom.xml| 137 +++-- repl/pom.xml | 4 +- resource-managers/kubernetes/core/pom.xml | 4 +- .../kubernetes/integration-tests/pom.xml | 4 +- resource-managers/mesos/pom.xml| 4 +- resource-managers/yarn/pom.xml | 4 +- sql/catalyst/pom.xml | 8 +- sql/core/pom.xml | 8 +- sql/hive-thriftserver/pom.xml | 8 +- sql/hive/pom.xml | 8 +- streaming/pom.xml | 8 +- tools/pom.xml | 4 +- 47 files changed, 213 insertions(+), 207 deletions(-) rename .github/workflows/{build_scala212.yml => build_scala213.yml} (93%) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark-docker] branch master updated: [SPARK-43370] Switch spark user only when run driver and executor
This is an automated email from the ASF dual-hosted git repository. yikun pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark-docker.git The following commit(s) were added to refs/heads/master by this push: new 2dc12d9 [SPARK-43370] Switch spark user only when run driver and executor 2dc12d9 is described below commit 2dc12d96910710aa6ee2d717c4c723ddd75127a1 Author: Yikun Jiang AuthorDate: Thu Jun 1 14:36:17 2023 +0800 [SPARK-43370] Switch spark user only when run driver and executor ### What changes were proposed in this pull request? Switch spark user only when run driver and executor ### Why are the changes needed? Address doi comments: question 7 [1] [1] https://github.com/docker-library/official-images/pull/13089#issuecomment-1533540388 [2] https://github.com/docker-library/official-images/pull/13089#issuecomment-1561793792 ### Does this PR introduce _any_ user-facing change? Yes ### How was this patch tested? 1. test mannuly ``` cd ~/spark-docker/3.4.0/scala2.12-java11-ubuntu $ docker build . -t spark-test $ docker run -ti spark-test bash sparkafa78af05cf8:/opt/spark/work-dir$ $ docker run --user root -ti spark-test bash root095e0d7651fd:/opt/spark/work-dir# ``` 2. ci passed Closes: https://github.com/apache/spark-docker/pull/44 Closes #43 from Yikun/SPARK-43370. Authored-by: Yikun Jiang Signed-off-by: Yikun Jiang --- 3.4.0/scala2.12-java11-python3-r-ubuntu/Dockerfile | 4 3.4.0/scala2.12-java11-python3-ubuntu/Dockerfile | 4 3.4.0/scala2.12-java11-r-ubuntu/Dockerfile | 4 3.4.0/scala2.12-java11-ubuntu/Dockerfile | 2 ++ 3.4.0/scala2.12-java11-ubuntu/entrypoint.sh| 23 +++--- Dockerfile.template| 2 ++ entrypoint.sh.template | 23 +++--- r-python.template | 4 8 files changed, 44 insertions(+), 22 deletions(-) diff --git a/3.4.0/scala2.12-java11-python3-r-ubuntu/Dockerfile b/3.4.0/scala2.12-java11-python3-r-ubuntu/Dockerfile index 7734100..0f1962f 100644 --- a/3.4.0/scala2.12-java11-python3-r-ubuntu/Dockerfile +++ b/3.4.0/scala2.12-java11-python3-r-ubuntu/Dockerfile @@ -16,6 +16,8 @@ # FROM spark:3.4.0-scala2.12-java11-ubuntu +USER root + RUN set -ex; \ apt-get update; \ apt install -y python3 python3-pip; \ @@ -24,3 +26,5 @@ RUN set -ex; \ rm -rf /var/lib/apt/lists/* ENV R_HOME /usr/lib/R + +USER spark diff --git a/3.4.0/scala2.12-java11-python3-ubuntu/Dockerfile b/3.4.0/scala2.12-java11-python3-ubuntu/Dockerfile index 6c12c30..258d806 100644 --- a/3.4.0/scala2.12-java11-python3-ubuntu/Dockerfile +++ b/3.4.0/scala2.12-java11-python3-ubuntu/Dockerfile @@ -16,8 +16,12 @@ # FROM spark:3.4.0-scala2.12-java11-ubuntu +USER root + RUN set -ex; \ apt-get update; \ apt install -y python3 python3-pip; \ rm -rf /var/cache/apt/*; \ rm -rf /var/lib/apt/lists/* + +USER spark diff --git a/3.4.0/scala2.12-java11-r-ubuntu/Dockerfile b/3.4.0/scala2.12-java11-r-ubuntu/Dockerfile index 24cd41a..4c928c6 100644 --- a/3.4.0/scala2.12-java11-r-ubuntu/Dockerfile +++ b/3.4.0/scala2.12-java11-r-ubuntu/Dockerfile @@ -16,6 +16,8 @@ # FROM spark:3.4.0-scala2.12-java11-ubuntu +USER root + RUN set -ex; \ apt-get update; \ apt install -y r-base r-base-dev; \ @@ -23,3 +25,5 @@ RUN set -ex; \ rm -rf /var/lib/apt/lists/* ENV R_HOME /usr/lib/R + +USER spark diff --git a/3.4.0/scala2.12-java11-ubuntu/Dockerfile b/3.4.0/scala2.12-java11-ubuntu/Dockerfile index 205b399..a680106 100644 --- a/3.4.0/scala2.12-java11-ubuntu/Dockerfile +++ b/3.4.0/scala2.12-java11-ubuntu/Dockerfile @@ -77,4 +77,6 @@ ENV SPARK_HOME /opt/spark WORKDIR /opt/spark/work-dir +USER spark + ENTRYPOINT [ "/opt/entrypoint.sh" ] diff --git a/3.4.0/scala2.12-java11-ubuntu/entrypoint.sh b/3.4.0/scala2.12-java11-ubuntu/entrypoint.sh index 716f1af..6def3f9 100755 --- a/3.4.0/scala2.12-java11-ubuntu/entrypoint.sh +++ b/3.4.0/scala2.12-java11-ubuntu/entrypoint.sh @@ -69,6 +69,13 @@ elif ! [ -z ${SPARK_HOME+x} ]; then SPARK_CLASSPATH="$SPARK_HOME/conf:$SPARK_CLASSPATH"; fi +# Switch to spark if no USER specified (root by default) otherwise use USER directly +switch_spark_if_root() { + if [ $(id -u) -eq 0 ]; then +echo gosu spark + fi +} + case "$1" in driver) shift 1 @@ -78,6 +85,8 @@ case "$1" in --deploy-mode client "$@" ) +# Execute the container CMD under tini for better hygiene +exec $(switch_spark_if_root) /usr/bin/tini -s -- "${CMD[@]}" ;; executor) shift 1 @@ -96,20 +105,12 @@ case "$1" in --resourceProfileId $SPARK_RESOURCE_PROFILE_ID --podName $SPARK_EXECUTOR_POD_NAME ) +# Execute the container CMD under tini for better hygiene +
[spark-connect-go] branch master updated: [SPARK-43909] Adding PR Template
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark-connect-go.git The following commit(s) were added to refs/heads/master by this push: new ec893d2 [SPARK-43909] Adding PR Template ec893d2 is described below commit ec893d29544a38da43298f72bc66a06e054bb147 Author: Martin Grund AuthorDate: Thu Jun 1 15:37:34 2023 +0900 [SPARK-43909] Adding PR Template This patch adds the PR template to the repository. Closes #4 from grundprinzip/SPARK-43909-p1. Authored-by: Martin Grund Signed-off-by: Hyukjin Kwon --- .github/PULL_REQUEST_TEMPLATE | 49 +++ 1 file changed, 49 insertions(+) diff --git a/.github/PULL_REQUEST_TEMPLATE b/.github/PULL_REQUEST_TEMPLATE new file mode 100644 index 000..241e3c1 --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE @@ -0,0 +1,49 @@ + + +### What changes were proposed in this pull request? + + + +### Why are the changes needed? + + + +### Does this PR introduce _any_ user-facing change? + + + +### How was this patch tested? + - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org