[spark] branch master updated: [SPARK-24774][SQL][FOLLOWUP] Remove unused code in SchemaConverters.scala
This is an automated email from the ASF dual-hosted git repository. maxgekk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 59c55dd [SPARK-24774][SQL][FOLLOWUP] Remove unused code in SchemaConverters.scala 59c55dd is described below commit 59c55dd4c6f7772ef7949653679a2b76211788e8 Author: Gengliang Wang AuthorDate: Wed Nov 3 08:43:25 2021 +0300 [SPARK-24774][SQL][FOLLOWUP] Remove unused code in SchemaConverters.scala ### What changes were proposed in this pull request? As MaxGekk pointed out in https://github.com/apache/spark/pull/22037/files#r741373793, there is some unused code in SchemaConverters.scala. The UUID generator was for generating `fix` avro field names but we figure out a better solution during PR review. This PR is to remove the dead code. ### Why are the changes needed? Code clean up ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existing UT. Closes #34472 from gengliangwang/SPARK-24774-followup. Authored-by: Gengliang Wang Signed-off-by: Max Gekk --- .../src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala | 4 1 file changed, 4 deletions(-) diff --git a/external/avro/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala b/external/avro/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala index 1c9b06b..347364c 100644 --- a/external/avro/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala +++ b/external/avro/src/main/scala/org/apache/spark/sql/avro/SchemaConverters.scala @@ -18,14 +18,12 @@ package org.apache.spark.sql.avro import scala.collection.JavaConverters._ -import scala.util.Random import org.apache.avro.{LogicalTypes, Schema, SchemaBuilder} import org.apache.avro.LogicalTypes.{Date, Decimal, LocalTimestampMicros, LocalTimestampMillis, TimestampMicros, TimestampMillis} import org.apache.avro.Schema.Type._ import org.apache.spark.annotation.DeveloperApi -import org.apache.spark.sql.catalyst.util.RandomUUIDGenerator import org.apache.spark.sql.types._ import org.apache.spark.sql.types.Decimal.minBytesForPrecision @@ -35,8 +33,6 @@ import org.apache.spark.sql.types.Decimal.minBytesForPrecision */ @DeveloperApi object SchemaConverters { - private lazy val uuidGenerator = RandomUUIDGenerator(new Random().nextLong()) - private lazy val nullSchema = Schema.create(Schema.Type.NULL) /** - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (78cc91c -> 90c23eb)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 78cc91c [SPARK-32567][SQL] Add code-gen for full outer shuffled hash join add 90c23eb [SPARK-37191][SQL] Allow merging DecimalTypes with different precision values No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/types/StructType.scala| 10 ++-- .../apache/spark/sql/types/StructTypeSuite.scala | 28 +- .../datasources/parquet/ParquetQuerySuite.scala| 22 + 3 files changed, 51 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-32567][SQL] Add code-gen for full outer shuffled hash join
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 78cc91c [SPARK-32567][SQL] Add code-gen for full outer shuffled hash join 78cc91c is described below commit 78cc91c962abd48d7ec2e9721d1e1429f802dced Author: Cheng Su AuthorDate: Wed Nov 3 11:18:12 2021 +0800 [SPARK-32567][SQL] Add code-gen for full outer shuffled hash join ### What changes were proposed in this pull request? As title. This PR is to add code-gen support for FULL OUTER shuffled hash join. The main change is in `ShuffledHashJoinExec.scala:doProduce()` to generate code for FULL OUTER join. * `ShuffledHashJoinExec.scala:codegenFullOuterJoinWithUniqueKey()` is the code for join with unique join key from build side. * `ShuffledHashJoinExec.scala:codegenFullOuterJoinWithNonUniqueKey()` is the code for join with non-unique key. Example query: ``` val df1 = spark.range(5).select($"id".as("k1")) val df2 = spark.range(10).select($"id".as("k2")) df1.join(df2.hint("SHUFFLE_HASH"), $"k1" === $"k2" % 3 && $"k1" + 3 =!= $"k2", "full_outer") ``` Generated code for example query: https://gist.github.com/c21/828b782ee81827f4148939cb50314a7b ### Why are the changes needed? Improve query performance for FULL OUTER shuffled hash join. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? * Added unit test in `WholeStageCodegenSuite`. * Existing unit test in `OuterJoinSuite`. Closes #3 from c21/shj-codegen. Authored-by: Cheng Su Signed-off-by: Wenchen Fan --- .../joins/BroadcastNestedLoopJoinExec.scala| 2 +- .../spark/sql/execution/joins/HashJoin.scala | 4 +- .../sql/execution/joins/JoinCodegenSupport.scala | 50 ++-- .../sql/execution/joins/ShuffledHashJoinExec.scala | 260 - .../sql/execution/joins/SortMergeJoinExec.scala| 3 +- .../sql/execution/WholeStageCodegenSuite.scala | 45 +++- .../spark/sql/execution/joins/OuterJoinSuite.scala | 21 ++ 7 files changed, 349 insertions(+), 36 deletions(-) diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastNestedLoopJoinExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastNestedLoopJoinExec.scala index 77a30b7..0677211 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastNestedLoopJoinExec.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/BroadcastNestedLoopJoinExec.scala @@ -463,7 +463,7 @@ case class BroadcastNestedLoopJoinExec( private def codegenOuter(ctx: CodegenContext, input: Seq[ExprCode]): String = { val (buildRowArray, buildRowArrayTerm) = prepareBroadcast(ctx) val (buildRow, checkCondition, _) = getJoinCondition(ctx, input, streamed, broadcast) -val buildVars = genBuildSideVars(ctx, buildRow, broadcast) +val buildVars = genOneSideJoinVars(ctx, buildRow, broadcast, setDefaultValue = true) val resultVars = buildSide match { case BuildLeft => buildVars ++ input diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala index f87acb8..0e8bb84 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/HashJoin.scala @@ -444,7 +444,7 @@ trait HashJoin extends JoinCodegenSupport { val HashedRelationInfo(relationTerm, keyIsUnique, _) = prepareRelation(ctx) val (keyEv, anyNull) = genStreamSideJoinKey(ctx, input) val matched = ctx.freshName("matched") -val buildVars = genBuildSideVars(ctx, matched, buildPlan) +val buildVars = genOneSideJoinVars(ctx, matched, buildPlan, setDefaultValue = true) val numOutput = metricTerm(ctx, "numOutputRows") // filter the output via condition @@ -646,7 +646,7 @@ trait HashJoin extends JoinCodegenSupport { val existsVar = ctx.freshName("exists") val matched = ctx.freshName("matched") -val buildVars = genBuildSideVars(ctx, matched, buildPlan) +val buildVars = genOneSideJoinVars(ctx, matched, buildPlan, setDefaultValue = false) val checkCondition = if (condition.isDefined) { val expr = condition.get // evaluate the variables from build side that used by condition diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/JoinCodegenSupport.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/JoinCodegenSupport.scala index 96aa0be..75f0a35 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/joins/JoinCodegenSupport.scala +++
[spark] branch master updated (293c085 -> d246010)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 293c085 [SPARK-36895][SQL][FOLLOWUP] CREATE INDEX command should rely on the analyzer framework to resolve columns add d246010 [SPARK-36935][SQL] Extend ParquetSchemaConverter to compute Parquet repetition & definition level No new revisions were added by this update. Summary of changes: .../java/org/apache/parquet/io/ColumnIOUtil.java} | 26 +- .../parquet/SpecificParquetRecordReaderBase.java |2 + .../datasources/parquet/ParquetColumn.scala| 55 + .../datasources/parquet/ParquetRowConverter.scala | 11 +- .../parquet/ParquetSchemaConverter.scala | 198 +++- .../datasources/parquet/ParquetSchemaSuite.scala | 1241 +++- 6 files changed, 1433 insertions(+), 100 deletions(-) copy sql/{hive-thriftserver/src/main/java/org/apache/hive/service/cli/OperationStatus.java => core/src/main/java/org/apache/parquet/io/ColumnIOUtil.java} (59%) create mode 100644 sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetColumn.scala - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-36895][SQL][FOLLOWUP] CREATE INDEX command should rely on the analyzer framework to resolve columns
This is an automated email from the ASF dual-hosted git repository. huaxingao pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 293c085 [SPARK-36895][SQL][FOLLOWUP] CREATE INDEX command should rely on the analyzer framework to resolve columns 293c085 is described below commit 293c085d677220e71966d98c25cde8a06ae78468 Author: Wenchen Fan AuthorDate: Tue Nov 2 14:39:42 2021 -0700 [SPARK-36895][SQL][FOLLOWUP] CREATE INDEX command should rely on the analyzer framework to resolve columns ### What changes were proposed in this pull request? This PR leverages the existing framework to resolve columns in the CREATE INDEX command. ### Why are the changes needed? To fail earlier instead of passing invalid column names to v2 sources. ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? new test Closes #34467 from cloud-fan/col. Lead-authored-by: Wenchen Fan Co-authored-by: Wenchen Fan Signed-off-by: Huaxin Gao --- .../org/apache/spark/sql/catalyst/analysis/Analyzer.scala | 15 +++ .../org/apache/spark/sql/catalyst/parser/AstBuilder.scala | 4 ++-- .../spark/sql/catalyst/plans/logical/v2Commands.scala | 12 +++- .../apache/spark/sql/catalyst/parser/DDLParserSuite.scala | 6 +++--- .../execution/datasources/v2/DataSourceV2Strategy.scala | 5 - .../apache/spark/sql/connector/DataSourceV2SQLSuite.scala | 11 --- 6 files changed, 35 insertions(+), 18 deletions(-) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala index f0a1c8c..068886e 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala @@ -272,7 +272,7 @@ class Analyzer(override val catalogManager: CatalogManager) ResolveInsertInto :: ResolveRelations :: ResolvePartitionSpec :: - ResolveAlterTableCommands :: + ResolveFieldNameAndPosition :: AddMetadataColumns :: DeduplicateRelations :: ResolveReferences :: @@ -3529,11 +3529,18 @@ class Analyzer(override val catalogManager: CatalogManager) } /** - * Rule to mostly resolve, normalize and rewrite column names based on case sensitivity - * for alter table column commands. + * Rule to resolve, normalize and rewrite field names based on case sensitivity for commands. */ - object ResolveAlterTableCommands extends Rule[LogicalPlan] { + object ResolveFieldNameAndPosition extends Rule[LogicalPlan] { def apply(plan: LogicalPlan): LogicalPlan = plan.resolveOperatorsUp { + case cmd: CreateIndex if cmd.table.resolved && + cmd.columns.exists(_._1.isInstanceOf[UnresolvedFieldName]) => +val table = cmd.table.asInstanceOf[ResolvedTable] +cmd.copy(columns = cmd.columns.map { + case (u: UnresolvedFieldName, prop) => resolveFieldNames(table, u.name, u) -> prop + case other => other +}) + case a: AlterTableCommand if a.table.resolved && hasUnresolvedFieldName(a) => val table = a.table.asInstanceOf[ResolvedTable] a.transformExpressions { diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala index 722a055..a16674f 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/parser/AstBuilder.scala @@ -4429,7 +4429,7 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg } val columns = ctx.columns.multipartIdentifierProperty.asScala - .map(_.multipartIdentifier.getText).toSeq + .map(_.multipartIdentifier).map(typedVisit[Seq[String]]).toSeq val columnsProperties = ctx.columns.multipartIdentifierProperty.asScala .map(x => (Option(x.options).map(visitPropertyKeyValues).getOrElse(Map.empty))).toSeq val options = Option(ctx.options).map(visitPropertyKeyValues).getOrElse(Map.empty) @@ -4439,7 +4439,7 @@ class AstBuilder extends SqlBaseBaseVisitor[AnyRef] with SQLConfHelper with Logg indexName, indexType, ctx.EXISTS != null, - columns.map(FieldReference(_).asInstanceOf[FieldReference]).zip(columnsProperties), + columns.map(UnresolvedFieldName(_)).zip(columnsProperties), options) } diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/v2Commands.scala index
[spark] branch master updated (ec6a3ae -> b78167a)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ec6a3ae [SPARK-37176][SQL] Sync JsonInferSchema#infer method's exception handle logic with JacksonParser#parse method add b78167a [SPARK-37066][SQL] Improve error message to show file path when failed to read next file No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/avro/AvroLogicalTypeSuite.scala | 2 +- .../apache/spark/sql/errors/QueryExecutionErrors.scala| 8 .../spark/sql/execution/datasources/FileScanRDD.scala | 13 ++--- .../execution/datasources/v2/FilePartitionReader.scala| 15 +++ .../execution/datasources/FileSourceStrategySuite.scala | 4 ++-- .../spark/sql/execution/datasources/csv/CSVSuite.scala| 4 ++-- .../datasources/parquet/ParquetSchemaSuite.scala | 6 ++ 7 files changed, 24 insertions(+), 28 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-37176][SQL] Sync JsonInferSchema#infer method's exception handle logic with JacksonParser#parse method
This is an automated email from the ASF dual-hosted git repository. maxgekk pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new ec6a3ae [SPARK-37176][SQL] Sync JsonInferSchema#infer method's exception handle logic with JacksonParser#parse method ec6a3ae is described below commit ec6a3ae6dff1dc9c63978ae14a1793ccd771 Author: Xianjin YE AuthorDate: Tue Nov 2 12:40:09 2021 +0300 [SPARK-37176][SQL] Sync JsonInferSchema#infer method's exception handle logic with JacksonParser#parse method ### What changes were proposed in this pull request? Change `JsonInferSchema#infer`'s exception handle logic to be aligned with `JacksonParser#parse` ### Why are the changes needed? To reduce behavior inconsistency, users can have the same expectation for schema infer and json parse when dealing with some malformed input. ### Does this PR introduce _any_ user-facing change? Yes. Before this patch, json's inferring schema could be failed for some malformed input but succeeded when parsing. After this patch, they have the same exception handle logic. ### How was this patch tested? Added one new test and modify one exist test to cover the new case. Closes #34455 from advancedxy/SPARK-37176. Authored-by: Xianjin YE Signed-off-by: Max Gekk --- .../spark/sql/catalyst/json/JsonInferSchema.scala | 33 +++- .../test/resources/test-data/malformed_utf8.json | 3 ++ .../sql/execution/datasources/json/JsonSuite.scala | 35 ++ 3 files changed, 63 insertions(+), 8 deletions(-) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JsonInferSchema.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JsonInferSchema.scala index 3b17cde..3b62b16 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JsonInferSchema.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JsonInferSchema.scala @@ -17,6 +17,8 @@ package org.apache.spark.sql.catalyst.json +import java.io.CharConversionException +import java.nio.charset.MalformedInputException import java.util.Comparator import scala.util.control.Exception.allCatch @@ -45,6 +47,18 @@ private[sql] class JsonInferSchema(options: JSONOptions) extends Serializable { legacyFormat = FAST_DATE_FORMAT, isParsing = true) + private def handleJsonErrorsByParseMode(parseMode: ParseMode, + columnNameOfCorruptRecord: String, e: Throwable): Option[StructType] = { +parseMode match { + case PermissiveMode => +Some(StructType(Seq(StructField(columnNameOfCorruptRecord, StringType + case DropMalformedMode => +None + case FailFastMode => +throw QueryExecutionErrors.malformedRecordsDetectedInSchemaInferenceError(e) +} + } + /** * Infer the type of a collection of json records in three stages: * 1. Infer the type of each record @@ -68,14 +82,17 @@ private[sql] class JsonInferSchema(options: JSONOptions) extends Serializable { Some(inferField(parser)) } } catch { - case e @ (_: RuntimeException | _: JsonProcessingException) => parseMode match { -case PermissiveMode => - Some(StructType(Seq(StructField(columnNameOfCorruptRecord, StringType -case DropMalformedMode => - None -case FailFastMode => - throw QueryExecutionErrors.malformedRecordsDetectedInSchemaInferenceError(e) - } + case e @ (_: RuntimeException | _: JsonProcessingException | +_: MalformedInputException) => +handleJsonErrorsByParseMode(parseMode, columnNameOfCorruptRecord, e) + case e: CharConversionException if options.encoding.isEmpty => +val msg = + """JSON parser cannot handle a character in its input. +|Specifying encoding as an input option explicitly might help to resolve the issue. +|""".stripMargin + e.getMessage +val wrappedCharException = new CharConversionException(msg) +wrappedCharException.initCause(e) +handleJsonErrorsByParseMode(parseMode, columnNameOfCorruptRecord, wrappedCharException) } }.reduceOption(typeMerger).toIterator } diff --git a/sql/core/src/test/resources/test-data/malformed_utf8.json b/sql/core/src/test/resources/test-data/malformed_utf8.json new file mode 100644 index 000..c57eb43 --- /dev/null +++ b/sql/core/src/test/resources/test-data/malformed_utf8.json @@ -0,0 +1,3 @@ +{"a": 1} +{"a": 1} +� \ No newline at end of file diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala
Subscribe
- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-37156][PYTHON] Inline type hints for python/pyspark/storagelevel.py
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 73e8628 [SPARK-37156][PYTHON] Inline type hints for python/pyspark/storagelevel.py 73e8628 is described below commit 73e8628f48db3f17a39f2154b54cbdea3d31e92c Author: dchvn AuthorDate: Tue Nov 2 16:43:17 2021 +0900 [SPARK-37156][PYTHON] Inline type hints for python/pyspark/storagelevel.py ### What changes were proposed in this pull request? Inline type hints for python/pyspark/storagelevel.py ### Why are the changes needed? We can take advantage of static type checking within the functions by inlining the type hints. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Existing tests Closes #34437 from dchvn/SPARK-37156. Authored-by: dchvn Signed-off-by: Hyukjin Kwon --- python/pyspark/storagelevel.py | 25 +--- python/pyspark/storagelevel.pyi | 43 - 2 files changed, 22 insertions(+), 46 deletions(-) diff --git a/python/pyspark/storagelevel.py b/python/pyspark/storagelevel.py index ecf8e5c..51fdebd 100644 --- a/python/pyspark/storagelevel.py +++ b/python/pyspark/storagelevel.py @@ -17,6 +17,8 @@ __all__ = ["StorageLevel"] +from typing import ClassVar + class StorageLevel(object): @@ -29,18 +31,35 @@ class StorageLevel(object): formats. """ -def __init__(self, useDisk, useMemory, useOffHeap, deserialized, replication=1): +DISK_ONLY: ClassVar["StorageLevel"] +DISK_ONLY_2: ClassVar["StorageLevel"] +DISK_ONLY_3: ClassVar["StorageLevel"] +MEMORY_ONLY: ClassVar["StorageLevel"] +MEMORY_ONLY_2: ClassVar["StorageLevel"] +MEMORY_AND_DISK: ClassVar["StorageLevel"] +MEMORY_AND_DISK_2: ClassVar["StorageLevel"] +OFF_HEAP: ClassVar["StorageLevel"] +MEMORY_AND_DISK_DESER: ClassVar["StorageLevel"] + +def __init__( +self, +useDisk: bool, +useMemory: bool, +useOffHeap: bool, +deserialized: bool, +replication: int = 1, +): self.useDisk = useDisk self.useMemory = useMemory self.useOffHeap = useOffHeap self.deserialized = deserialized self.replication = replication -def __repr__(self): +def __repr__(self) -> str: return "StorageLevel(%s, %s, %s, %s, %s)" % ( self.useDisk, self.useMemory, self.useOffHeap, self.deserialized, self.replication) -def __str__(self): +def __str__(self) -> str: result = "" result += "Disk " if self.useDisk else "" result += "Memory " if self.useMemory else "" diff --git a/python/pyspark/storagelevel.pyi b/python/pyspark/storagelevel.pyi deleted file mode 100644 index 2eb0585..000 --- a/python/pyspark/storagelevel.pyi +++ /dev/null @@ -1,43 +0,0 @@ -# -# Licensed to the Apache Software Foundation (ASF) under one -# or more contributor license agreements. See the NOTICE file -# distributed with this work for additional information -# regarding copyright ownership. The ASF licenses this file -# to you under the Apache License, Version 2.0 (the -# "License"); you may not use this file except in compliance -# with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, -# software distributed under the License is distributed on an -# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -# KIND, either express or implied. See the License for the -# specific language governing permissions and limitations -# under the License. - -from typing import ClassVar - -class StorageLevel: -DISK_ONLY: ClassVar[StorageLevel] -DISK_ONLY_2: ClassVar[StorageLevel] -MEMORY_ONLY: ClassVar[StorageLevel] -MEMORY_ONLY_2: ClassVar[StorageLevel] -DISK_ONLY_3: ClassVar[StorageLevel] -MEMORY_AND_DISK: ClassVar[StorageLevel] -MEMORY_AND_DISK_2: ClassVar[StorageLevel] -OFF_HEAP: ClassVar[StorageLevel] - -useDisk: bool -useMemory: bool -useOffHeap: bool -deserialized: bool -replication: int -def __init__( -self, -useDisk: bool, -useMemory: bool, -useOffHeap: bool, -deserialized: bool, -replication: int = ..., -) -> None: ... - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (b4a6eb6 -> 6d42230)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from b4a6eb6 [SPARK-37164][SQL] Add ExpressionBuilder for functions with complex overloads add 6d42230 [SPARK-37168][SQL] Improve error messages for SQL functions and operators under ANSI mode No new revisions were added by this update. Summary of changes: core/src/main/resources/error/error-classes.json | 6 +- .../catalyst/expressions/datetimeExpressions.scala | 90 ++ .../catalyst/expressions/intervalExpressions.scala | 13 +++- .../spark/sql/catalyst/util/DateTimeUtils.scala| 6 +- .../spark/sql/errors/QueryExecutionErrors.scala| 48 ++-- .../expressions/StringExpressionsSuite.scala | 2 +- .../resources/sql-tests/results/ansi/array.sql.out | 14 ++-- .../resources/sql-tests/results/ansi/date.sql.out | 8 +- .../resources/sql-tests/results/ansi/map.sql.out | 4 +- .../sql-tests/results/ansi/timestamp.sql.out | 20 ++--- .../sql-tests/results/postgreSQL/date.sql.out | 6 +- .../results/timestampNTZ/timestamp-ansi.sql.out| 20 ++--- 12 files changed, 155 insertions(+), 82 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (320fa07 -> b4a6eb6)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 320fa07 [SPARK-37159][SQL][TESTS] Change HiveExternalCatalogVersionsSuite to be able to test with Java 17 add b4a6eb6 [SPARK-37164][SQL] Add ExpressionBuilder for functions with complex overloads No new revisions were added by this update. Summary of changes: .../sql/catalyst/analysis/FunctionRegistry.scala | 20 ++- .../catalyst/expressions/stringExpressions.scala | 163 +++-- .../spark/sql/errors/QueryCompilationErrors.scala | 4 +- .../expressions/StringExpressionsSuite.scala | 4 +- .../scala/org/apache/spark/sql/functions.scala | 4 +- .../sql-functions/sql-expression-schema.md | 4 +- .../sql-tests/inputs/string-functions.sql | 8 +- .../results/ansi/string-functions.sql.out | 16 +- .../sql-tests/results/string-functions.sql.out | 28 ++-- 9 files changed, 137 insertions(+), 114 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org