[spark] branch master updated (f47e0f8 -> 323a6e8)
This is an automated email from the ASF dual-hosted git repository. viirya pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f47e0f8 [SPARK-35261][SQL] Support static magic method for stateless Java ScalarFunction add 323a6e8 [SPARK-35232][SQL] Nested column pruning should retain column metadata No new revisions were added by this update. Summary of changes: .../sql/catalyst/expressions/SchemaPruning.scala | 8 ++- .../catalyst/expressions/SchemaPruningSuite.scala | 14 + .../sql/connector/catalog/InMemoryTable.scala | 68 -- 3 files changed, 70 insertions(+), 20 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (b4ec9e2 -> f47e0f8)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from b4ec9e2 [SPARK-35321][SQL] Don't register Hive permanent functions when creating Hive client add f47e0f8 [SPARK-35261][SQL] Support static magic method for stateless Java ScalarFunction No new revisions were added by this update. Summary of changes: .../catalog/functions/ScalarFunction.java | 60 ++--- .../spark/sql/catalyst/analysis/Analyzer.scala | 5 +- .../V2FunctionBenchmark-jdk11-results.txt | 44 ++ .../benchmarks/V2FunctionBenchmark-results.txt | 44 ++ .../connector/catalog/functions/JavaLongAdd.java | 130 ++ .../connector/catalog/functions/JavaStrLen.java| 48 +++ .../sql/connector/DataSourceV2FunctionSuite.scala | 22 +++ .../connector/functions/V2FunctionBenchmark.scala | 147 + 8 files changed, 461 insertions(+), 39 deletions(-) create mode 100644 sql/core/benchmarks/V2FunctionBenchmark-jdk11-results.txt create mode 100644 sql/core/benchmarks/V2FunctionBenchmark-results.txt create mode 100644 sql/core/src/test/java/test/org/apache/spark/sql/connector/catalog/functions/JavaLongAdd.java create mode 100644 sql/core/src/test/scala/org/apache/spark/sql/connector/functions/V2FunctionBenchmark.scala - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated (b190e2b -> 5268a38)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git. from b190e2b [SPARK-35288][SQL] StaticInvoke should find the method without exact argument classes match add 5268a38 [SPARK-35321][SQL][3.0] Don't register Hive permanent functions when creating Hive client No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/hive/client/HiveClientImpl.scala | 4 ++-- .../scala/org/apache/spark/sql/hive/client/HiveShim.scala | 11 +++ 2 files changed, 13 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.1 updated (a4c34ba -> 6fbea6a)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-3.1 in repository https://gitbox.apache.org/repos/asf/spark.git. from a4c34ba [SPARK-35288][SQL] StaticInvoke should find the method without exact argument classes match add 6fbea6a [SPARK-35321][SQL][3.1] Don't register Hive permanent functions when creating Hive client No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/hive/client/HiveClientImpl.scala | 4 ++-- .../scala/org/apache/spark/sql/hive/client/HiveShim.scala | 11 +++ 2 files changed, 13 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (33fbf56 -> b4ec9e2)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 33fbf56 [SPARK-35288][SQL] StaticInvoke should find the method without exact argument classes match add b4ec9e2 [SPARK-35321][SQL] Don't register Hive permanent functions when creating Hive client No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/hive/client/HiveClientImpl.scala | 4 ++-- .../scala/org/apache/spark/sql/hive/client/HiveShim.scala | 11 +++ 2 files changed, 13 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [SPARK-35288][SQL] StaticInvoke should find the method without exact argument classes match
This is an automated email from the ASF dual-hosted git repository. viirya pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new 7733510 [SPARK-35288][SQL] StaticInvoke should find the method without exact argument classes match 7733510 is described below commit 7733510d0403625c41710d7e79f810117aac2ced Author: Liang-Chi Hsieh AuthorDate: Fri May 7 09:07:57 2021 -0700 [SPARK-35288][SQL] StaticInvoke should find the method without exact argument classes match ### What changes were proposed in this pull request? This patch proposes to make StaticInvoke able to find method with given method name even the parameter types do not exactly match to argument classes. ### Why are the changes needed? Unlike `Invoke`, `StaticInvoke` only tries to get the method with exact argument classes. If the calling method's parameter types are not exactly matched with the argument classes, `StaticInvoke` cannot find the method. `StaticInvoke` should be able to find the method under the cases too. ### Does this PR introduce _any_ user-facing change? Yes. `StaticInvoke` can find a method even the argument classes are not exactly matched. ### How was this patch tested? Unit test. Closes #32413 from viirya/static-invoke. Authored-by: Liang-Chi Hsieh Signed-off-by: Liang-Chi Hsieh (cherry picked from commit 33fbf5647b4a5587c78ac51339c0cbc9d70547a4) Signed-off-by: Liang-Chi Hsieh --- .../sql/catalyst/expressions/objects/objects.scala | 56 -- .../expressions/ObjectExpressionsSuite.scala | 34 +++-- 2 files changed, 60 insertions(+), 30 deletions(-) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala index 43e8105..fb4132a 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala @@ -139,6 +139,34 @@ trait InvokeLike extends Expression with NonSQLExpression { } } } + + final def findMethod(cls: Class[_], functionName: String, argClasses: Seq[Class[_]]): Method = { +// Looking with function name + argument classes first. +try { + cls.getMethod(functionName, argClasses: _*) +} catch { + case _: NoSuchMethodException => +// For some cases, e.g. arg class is Object, `getMethod` cannot find the method. +// We look at function name + argument length +val m = cls.getMethods.filter { m => + m.getName == functionName && m.getParameterCount == arguments.length +} +if (m.isEmpty) { + sys.error(s"Couldn't find $functionName on $cls") +} else if (m.length > 1) { + // More than one matched method signature. Exclude synthetic one, e.g. generic one. + val realMethods = m.filter(!_.isSynthetic) + if (realMethods.length > 1) { +// Ambiguous case, we don't know which method to choose, just fail it. +sys.error(s"Found ${realMethods.length} $functionName on $cls") + } else { +realMethods.head + } +} else { + m.head +} +} + } } /** @@ -230,7 +258,7 @@ case class StaticInvoke( override def children: Seq[Expression] = arguments lazy val argClasses = ScalaReflection.expressionJavaClasses(arguments) - @transient lazy val method = cls.getDeclaredMethod(functionName, argClasses : _*) + @transient lazy val method = findMethod(cls, functionName, argClasses) override def eval(input: InternalRow): Any = { invoke(null, method, arguments, input, dataType) @@ -317,31 +345,7 @@ case class Invoke( @transient lazy val method = targetObject.dataType match { case ObjectType(cls) => - // Looking with function name + argument classes first. - try { -Some(cls.getMethod(encodedFunctionName, argClasses: _*)) - } catch { -case _: NoSuchMethodException => - // For some cases, e.g. arg class is Object, `getMethod` cannot find the method. - // We look at function name + argument length - val m = cls.getMethods.filter { m => -m.getName == encodedFunctionName && m.getParameterCount == arguments.length - } - if (m.isEmpty) { -sys.error(s"Couldn't find $encodedFunctionName on $cls") - } else if (m.length > 1) { -// More than one matched method signature. Exclude synthetic one, e.g. generic one. -val realMethods = m.filter(!_.isSynthetic) -if (realMethods.length > 1) { - // Ambiguous case, we don't know
[spark] branch branch-3.0 updated: [SPARK-35288][SQL] StaticInvoke should find the method without exact argument classes match
This is an automated email from the ASF dual-hosted git repository. viirya pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new b190e2b [SPARK-35288][SQL] StaticInvoke should find the method without exact argument classes match b190e2b is described below commit b190e2b36dba9ad85c2b30a2b693aa9defd6de02 Author: Liang-Chi Hsieh AuthorDate: Fri May 7 09:07:57 2021 -0700 [SPARK-35288][SQL] StaticInvoke should find the method without exact argument classes match ### What changes were proposed in this pull request? This patch proposes to make StaticInvoke able to find method with given method name even the parameter types do not exactly match to argument classes. ### Why are the changes needed? Unlike `Invoke`, `StaticInvoke` only tries to get the method with exact argument classes. If the calling method's parameter types are not exactly matched with the argument classes, `StaticInvoke` cannot find the method. `StaticInvoke` should be able to find the method under the cases too. ### Does this PR introduce _any_ user-facing change? Yes. `StaticInvoke` can find a method even the argument classes are not exactly matched. ### How was this patch tested? Unit test. Closes #32413 from viirya/static-invoke. Authored-by: Liang-Chi Hsieh Signed-off-by: Liang-Chi Hsieh (cherry picked from commit 33fbf5647b4a5587c78ac51339c0cbc9d70547a4) Signed-off-by: Liang-Chi Hsieh --- .../sql/catalyst/expressions/objects/objects.scala | 56 -- .../expressions/ObjectExpressionsSuite.scala | 34 +++-- 2 files changed, 60 insertions(+), 30 deletions(-) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala index 066188a..e5e 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala @@ -141,6 +141,34 @@ trait InvokeLike extends Expression with NonSQLExpression { } } } + + final def findMethod(cls: Class[_], functionName: String, argClasses: Seq[Class[_]]): Method = { +// Looking with function name + argument classes first. +try { + cls.getMethod(functionName, argClasses: _*) +} catch { + case _: NoSuchMethodException => +// For some cases, e.g. arg class is Object, `getMethod` cannot find the method. +// We look at function name + argument length +val m = cls.getMethods.filter { m => + m.getName == functionName && m.getParameterCount == arguments.length +} +if (m.isEmpty) { + sys.error(s"Couldn't find $functionName on $cls") +} else if (m.length > 1) { + // More than one matched method signature. Exclude synthetic one, e.g. generic one. + val realMethods = m.filter(!_.isSynthetic) + if (realMethods.length > 1) { +// Ambiguous case, we don't know which method to choose, just fail it. +sys.error(s"Found ${realMethods.length} $functionName on $cls") + } else { +realMethods.head + } +} else { + m.head +} +} + } } /** @@ -232,7 +260,7 @@ case class StaticInvoke( override def children: Seq[Expression] = arguments lazy val argClasses = ScalaReflection.expressionJavaClasses(arguments) - @transient lazy val method = cls.getDeclaredMethod(functionName, argClasses : _*) + @transient lazy val method = findMethod(cls, functionName, argClasses) override def eval(input: InternalRow): Any = { invoke(null, method, arguments, input, dataType) @@ -319,31 +347,7 @@ case class Invoke( @transient lazy val method = targetObject.dataType match { case ObjectType(cls) => - // Looking with function name + argument classes first. - try { -Some(cls.getMethod(encodedFunctionName, argClasses: _*)) - } catch { -case _: NoSuchMethodException => - // For some cases, e.g. arg class is Object, `getMethod` cannot find the method. - // We look at function name + argument length - val m = cls.getMethods.filter { m => -m.getName == encodedFunctionName && m.getParameterCount == arguments.length - } - if (m.isEmpty) { -sys.error(s"Couldn't find $encodedFunctionName on $cls") - } else if (m.length > 1) { -// More than one matched method signature. Exclude synthetic one, e.g. generic one. -val realMethods = m.filter(!_.isSynthetic) -if (realMethods.length > 1) { - // Ambiguous case, we don't know
[spark] branch branch-3.1 updated: [SPARK-35288][SQL] StaticInvoke should find the method without exact argument classes match
This is an automated email from the ASF dual-hosted git repository. viirya pushed a commit to branch branch-3.1 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.1 by this push: new a4c34ba [SPARK-35288][SQL] StaticInvoke should find the method without exact argument classes match a4c34ba is described below commit a4c34ba161557c8857a1b639441fe79912021b23 Author: Liang-Chi Hsieh AuthorDate: Fri May 7 09:07:57 2021 -0700 [SPARK-35288][SQL] StaticInvoke should find the method without exact argument classes match ### What changes were proposed in this pull request? This patch proposes to make StaticInvoke able to find method with given method name even the parameter types do not exactly match to argument classes. ### Why are the changes needed? Unlike `Invoke`, `StaticInvoke` only tries to get the method with exact argument classes. If the calling method's parameter types are not exactly matched with the argument classes, `StaticInvoke` cannot find the method. `StaticInvoke` should be able to find the method under the cases too. ### Does this PR introduce _any_ user-facing change? Yes. `StaticInvoke` can find a method even the argument classes are not exactly matched. ### How was this patch tested? Unit test. Closes #32413 from viirya/static-invoke. Authored-by: Liang-Chi Hsieh Signed-off-by: Liang-Chi Hsieh (cherry picked from commit 33fbf5647b4a5587c78ac51339c0cbc9d70547a4) Signed-off-by: Liang-Chi Hsieh --- .../sql/catalyst/expressions/objects/objects.scala | 56 -- .../expressions/ObjectExpressionsSuite.scala | 34 +++-- 2 files changed, 60 insertions(+), 30 deletions(-) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala index 53a2bfd..f78e3f5 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala @@ -141,6 +141,34 @@ trait InvokeLike extends Expression with NonSQLExpression { } } } + + final def findMethod(cls: Class[_], functionName: String, argClasses: Seq[Class[_]]): Method = { +// Looking with function name + argument classes first. +try { + cls.getMethod(functionName, argClasses: _*) +} catch { + case _: NoSuchMethodException => +// For some cases, e.g. arg class is Object, `getMethod` cannot find the method. +// We look at function name + argument length +val m = cls.getMethods.filter { m => + m.getName == functionName && m.getParameterCount == arguments.length +} +if (m.isEmpty) { + sys.error(s"Couldn't find $functionName on $cls") +} else if (m.length > 1) { + // More than one matched method signature. Exclude synthetic one, e.g. generic one. + val realMethods = m.filter(!_.isSynthetic) + if (realMethods.length > 1) { +// Ambiguous case, we don't know which method to choose, just fail it. +sys.error(s"Found ${realMethods.length} $functionName on $cls") + } else { +realMethods.head + } +} else { + m.head +} +} + } } /** @@ -232,7 +260,7 @@ case class StaticInvoke( override def children: Seq[Expression] = arguments lazy val argClasses = ScalaReflection.expressionJavaClasses(arguments) - @transient lazy val method = cls.getDeclaredMethod(functionName, argClasses : _*) + @transient lazy val method = findMethod(cls, functionName, argClasses) override def eval(input: InternalRow): Any = { invoke(null, method, arguments, input, dataType) @@ -319,31 +347,7 @@ case class Invoke( @transient lazy val method = targetObject.dataType match { case ObjectType(cls) => - // Looking with function name + argument classes first. - try { -Some(cls.getMethod(encodedFunctionName, argClasses: _*)) - } catch { -case _: NoSuchMethodException => - // For some cases, e.g. arg class is Object, `getMethod` cannot find the method. - // We look at function name + argument length - val m = cls.getMethods.filter { m => -m.getName == encodedFunctionName && m.getParameterCount == arguments.length - } - if (m.isEmpty) { -sys.error(s"Couldn't find $encodedFunctionName on $cls") - } else if (m.length > 1) { -// More than one matched method signature. Exclude synthetic one, e.g. generic one. -val realMethods = m.filter(!_.isSynthetic) -if (realMethods.length > 1) { - // Ambiguous case, we don't know
[spark] branch master updated (6f0ef93 -> 33fbf56)
This is an automated email from the ASF dual-hosted git repository. viirya pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 6f0ef93 [SPARK-35297][CORE][DOC][MINOR] Modify the comment about the executor add 33fbf56 [SPARK-35288][SQL] StaticInvoke should find the method without exact argument classes match No new revisions were added by this update. Summary of changes: .../sql/catalyst/expressions/objects/objects.scala | 56 -- .../expressions/ObjectExpressionsSuite.scala | 34 +++-- 2 files changed, 60 insertions(+), 30 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (2634dba -> 6f0ef93)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 2634dba [SPARK-35175][BUILD] Add linter for JavaScript source files add 6f0ef93 [SPARK-35297][CORE][DOC][MINOR] Modify the comment about the executor No new revisions were added by this update. Summary of changes: core/src/main/scala/org/apache/spark/executor/Executor.scala | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (72d3266 -> d3b92ee)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 72d3266 [SPARK-35144][SQL] Migrate to transformWithPruning for object rules add d3b92ee [SPARK-35021][SQL] Group exception messages in connector/catalog No new revisions were added by this update. Summary of changes: .../sql/connector/catalog/CatalogV2Implicits.scala | 36 +-- .../sql/connector/catalog/CatalogV2Util.scala | 5 ++- .../sql/connector/catalog/LookupCatalog.scala | 6 ++-- .../spark/sql/errors/QueryCompilationErrors.scala | 42 -- 4 files changed, 60 insertions(+), 29 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-35144][SQL] Migrate to transformWithPruning for object rules
This is an automated email from the ASF dual-hosted git repository. gengliang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 72d3266 [SPARK-35144][SQL] Migrate to transformWithPruning for object rules 72d3266 is described below commit 72d32662d470e286a639783fed8dcf6c3948 Author: Yingyi Bu AuthorDate: Fri May 7 18:36:28 2021 +0800 [SPARK-35144][SQL] Migrate to transformWithPruning for object rules ### What changes were proposed in this pull request? Added the following TreePattern enums: - APPEND_COLUMNS - DESERIALIZE_TO_OBJECT - LAMBDA_VARIABLE - MAP_OBJECTS - SERIALIZE_FROM_OBJECT - PROJECT - TYPED_FILTER Added tree traversal pruning to the following rules dealing with objects: - EliminateSerialization - CombineTypedFilters - EliminateMapObjects - ObjectSerializerPruning ### Why are the changes needed? Reduce the number of tree traversals and hence improve the query compilation latency. ### How was this patch tested? Existing tests. Closes #32451 from sigmod/object. Authored-by: Yingyi Bu Signed-off-by: Gengliang Wang --- .../spark/sql/catalyst/expressions/objects/objects.scala | 6 +- .../org/apache/spark/sql/catalyst/optimizer/objects.scala | 15 ++- .../catalyst/plans/logical/basicLogicalOperators.scala| 2 ++ .../apache/spark/sql/catalyst/plans/logical/object.scala | 8 .../spark/sql/catalyst/rules/RuleIdCollection.scala | 5 + .../apache/spark/sql/catalyst/trees/TreePatterns.scala| 7 +++ 6 files changed, 37 insertions(+), 6 deletions(-) diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala index 469c895..40378a3 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala @@ -33,7 +33,7 @@ import org.apache.spark.sql.catalyst.expressions._ import org.apache.spark.sql.catalyst.expressions.codegen._ import org.apache.spark.sql.catalyst.expressions.codegen.Block._ import org.apache.spark.sql.catalyst.trees.TernaryLike -import org.apache.spark.sql.catalyst.trees.TreePattern.{NULL_CHECK, TreePattern} +import org.apache.spark.sql.catalyst.trees.TreePattern._ import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, ArrayData, GenericArrayData, MapData} import org.apache.spark.sql.errors.QueryExecutionErrors import org.apache.spark.sql.types._ @@ -669,6 +669,8 @@ case class LambdaVariable( private val accessor: (InternalRow, Int) => Any = InternalRow.getAccessor(dataType, nullable) + final override val nodePatterns: Seq[TreePattern] = Seq(LAMBDA_VARIABLE) + // Interpreted execution of `LambdaVariable` always get the 0-index element from input row. override def eval(input: InternalRow): Any = { assert(input.numFields == 1, @@ -781,6 +783,8 @@ case class MapObjects private( override def second: Expression = lambdaFunction override def third: Expression = inputData + final override val nodePatterns: Seq[TreePattern] = Seq(MAP_OBJECTS) + // The data with UserDefinedType are actually stored with the data type of its sqlType. // When we want to apply MapObjects on it, we have to use it. lazy private val inputDataType = inputData.dataType match { diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/objects.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/objects.scala index 97712a0..52544ff 100644 --- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/objects.scala +++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/objects.scala @@ -24,6 +24,7 @@ import org.apache.spark.sql.catalyst.expressions._ import org.apache.spark.sql.catalyst.expressions.objects._ import org.apache.spark.sql.catalyst.plans.logical._ import org.apache.spark.sql.catalyst.rules._ +import org.apache.spark.sql.catalyst.trees.TreePattern._ import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType, UserDefinedType} /* @@ -35,7 +36,8 @@ import org.apache.spark.sql.types.{ArrayType, DataType, MapType, StructType, Use * representation of data item. For example back to back map operations. */ object EliminateSerialization extends Rule[LogicalPlan] { - def apply(plan: LogicalPlan): LogicalPlan = plan transform { + def apply(plan: LogicalPlan): LogicalPlan = plan.transformWithPruning( +_.containsAnyPattern(DESERIALIZE_TO_OBJECT, APPEND_COLUMNS, TYPED_FILTER), ruleId) { case d @ DeserializeToObject(_, _, s: SerializeFromObject) if d.outputObj
[spark] branch master updated (cf2c4ba -> 9aa18df)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from cf2c4ba [SPARK-35020][SQL] Group exception messages in catalyst/util add 9aa18df [SPARK-35333][SQL] Skip object null check in Invoke if possible No new revisions were added by this update. Summary of changes: .../sql/catalyst/expressions/objects/objects.scala | 27 -- 1 file changed, 20 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (e83910f -> cf2c4ba)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from e83910f [SPARK-26164][SQL][FOLLOWUP] WriteTaskStatsTracker should know which file the row is written to add cf2c4ba [SPARK-35020][SQL] Group exception messages in catalyst/util No new revisions were added by this update. Summary of changes: .../sql/catalyst/util/ArrayBasedMapBuilder.scala | 16 +++ .../catalyst/util/DateTimeFormatterHelper.scala| 21 +++-- .../spark/sql/catalyst/util/DateTimeUtils.scala| 7 +-- .../spark/sql/catalyst/util/IntervalUtils.scala| 3 +- .../apache/spark/sql/catalyst/util/TypeUtils.scala | 4 +- .../spark/sql/errors/QueryCompilationErrors.scala | 4 ++ .../spark/sql/errors/QueryExecutionErrors.scala| 54 +- 7 files changed, 77 insertions(+), 32 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (33c1034 -> e83910f)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 33c1034 [SPARK-34701][SQL][FOLLOW-UP] Children/innerChildren should be mutually exclusive for AnalysisOnlyCommand add e83910f [SPARK-26164][SQL][FOLLOWUP] WriteTaskStatsTracker should know which file the row is written to No new revisions were added by this update. Summary of changes: .../datasources/BasicWriteStatsTracker.scala | 2 +- .../datasources/FileFormatDataWriter.scala | 4 +- .../execution/datasources/WriteStatsTracker.scala | 3 +- .../CustomWriteTaskStatsTrackerSuite.scala | 72 ++ 4 files changed, 77 insertions(+), 4 deletions(-) create mode 100644 sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/CustomWriteTaskStatsTrackerSuite.scala - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org