[spark] branch branch-3.0 updated: [SPARK-30964][CORE][WEBUI] Accelerate InMemoryStore with a new index
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 3e26876 [SPARK-30964][CORE][WEBUI] Accelerate InMemoryStore with a new index 3e26876 is described below commit 3e268763c345076801f1ff6f75bab67ecab87e8f Author: Gengliang Wang AuthorDate: Mon Mar 2 15:48:48 2020 +0800 [SPARK-30964][CORE][WEBUI] Accelerate InMemoryStore with a new index ### What changes were proposed in this pull request? Spark uses the class `InMemoryStore` as the KV storage for live UI and history server(by default if no LevelDB file path is provided). In `InMemoryStore`, all the task data in one application is stored in a hashmap, which key is the task ID and the value is the task data. This fine for getting or deleting with a provided task ID. However, Spark stage UI always shows all the task data in one stage and the current implementation is to look up all the values in the hashmap. The time complexity is O(numOfTasks). Also, when there are too many stages (>spark.ui.retainedStages), Spark will linearly try to look up all the task data of the stages to be deleted as well. This can be very bad for a large application with many stages and tasks. We can improve it by allowing the natural key of an entity to have a real parent index. So that on each lookup with parent node provided, Spark can look up all the natural keys(in our case, the task IDs) first, and then find the data with the natural keys in the hashmap. ### Why are the changes needed? The in-memory KV store becomes really slow for large applications. We can improve it with a new index. The performance can be 10 times, 100 times, even 1000 times faster. This is also possible to make the Spark driver more stable for large applications. ### Does this PR introduce any user-facing change? No ### How was this patch tested? Existing unit tests. Also, I run a benchmark with the following code ``` val store = new InMemoryStore() val numberOfTasksPerStage = 1 (0 until 1000).map { sId => (0 until numberOfTasksPerStage).map { taskId => val task = newTaskData(sId * numberOfTasksPerStage + taskId, "SUCCESS", sId) store.write(task) } } val appStatusStore = new AppStatusStore(store) var start = System.nanoTime() appStatusStore.taskSummary(2, attemptId, Array(0, 0.25, 0.5, 0.75, 1)) println("task summary run time: " + ((System.nanoTime() - start) / 100)) val stageIds = Seq(1, 11, 66, 88) val stageKeys = stageIds.map(Array(_, attemptId)) start = System.nanoTime() store.removeAllByIndexValues(classOf[TaskDataWrapper], TaskIndexNames.STAGE, stageKeys.asJavaCollection) println("clean up tasks run time: " + ((System.nanoTime() - start) / 100)) ``` Task summary before the changes: 98642ms Task summary after the changes: 120ms Task clean up before the changes: 4900ms Task clean up before the changes: 4ms It's 800x faster after the changes in the micro-benchmark. Closes #27716 from gengliangwang/liveUIStore. Authored-by: Gengliang Wang Signed-off-by: Wenchen Fan (cherry picked from commit 6b641430c37e0115bee781fed7360187b988313d) Signed-off-by: Wenchen Fan --- .../apache/spark/util/kvstore/InMemoryStore.java | 116 + .../org/apache/spark/util/kvstore/KVTypeInfo.java | 7 +- .../apache/spark/util/kvstore/LevelDBTypeInfo.java | 5 +- .../scala/org/apache/spark/status/storeTypes.scala | 2 +- 4 files changed, 105 insertions(+), 25 deletions(-) diff --git a/common/kvstore/src/main/java/org/apache/spark/util/kvstore/InMemoryStore.java b/common/kvstore/src/main/java/org/apache/spark/util/kvstore/InMemoryStore.java index b33c538..db08740 100644 --- a/common/kvstore/src/main/java/org/apache/spark/util/kvstore/InMemoryStore.java +++ b/common/kvstore/src/main/java/org/apache/spark/util/kvstore/InMemoryStore.java @@ -163,6 +163,12 @@ public class InMemoryStore implements KVStore { } } + /** + * An alias class for the type "ConcurrentHashMap, Boolean>", which is used + * as a concurrent hashset for storing natural keys and the boolean value doesn't matter. + */ + private static class NaturalKeys extends ConcurrentHashMap, Boolean> {} + private static class InstanceList { /** @@ -205,11 +211,19 @@ public class InMemoryStore implements KVStore { private final KVTypeInfo ti; private final KVTypeInfo.Accessor naturalKey; private final ConcurrentMap, T> data; +private final String naturalParentIndexName; +private final Boolean hasNaturalParentIndex; +// A mapping from parent to the natu
[spark] branch master updated (6b64143 -> f24a460)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 6b64143 [SPARK-30964][CORE][WEBUI] Accelerate InMemoryStore with a new index add f24a460 [SPARK-30993][SQL] Use its sql type for UDT when checking the type of length (fixed/var) or mutable No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/expressions/UnsafeRow.java | 8 + .../codegen/GenerateUnsafeRowJoinerSuite.scala | 41 +- .../apache/spark/sql/UserDefinedTypeSuite.scala| 37 +++ 3 files changed, 85 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (6b64143 -> f24a460)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 6b64143 [SPARK-30964][CORE][WEBUI] Accelerate InMemoryStore with a new index add f24a460 [SPARK-30993][SQL] Use its sql type for UDT when checking the type of length (fixed/var) or mutable No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/expressions/UnsafeRow.java | 8 + .../codegen/GenerateUnsafeRowJoinerSuite.scala | 41 +- .../apache/spark/sql/UserDefinedTypeSuite.scala| 37 +++ 3 files changed, 85 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-30993][SQL] Use its sql type for UDT when checking the type of length (fixed/var) or mutable
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 6c4977d [SPARK-30993][SQL] Use its sql type for UDT when checking the type of length (fixed/var) or mutable 6c4977d is described below commit 6c4977d38f13628abfa24129ae6844146672d96d Author: Jungtaek Lim (HeartSaVioR) AuthorDate: Mon Mar 2 22:33:11 2020 +0800 [SPARK-30993][SQL] Use its sql type for UDT when checking the type of length (fixed/var) or mutable ### What changes were proposed in this pull request? This patch fixes the bug of UnsafeRow which misses to handle the UDT specifically, in `isFixedLength` and `isMutable`. These methods don't check its SQL type for UDT, always treating UDT as variable-length, and non-mutable. It doesn't bring any issue if UDT is used to represent complicated type, but when UDT is used to represent some type which is matched with fixed length of SQL type, it exposes the chance of correctness issues, as these informations sometimes decide how the value should be handled. We got report from user mailing list which suspected as mapGroupsWithState looks like handling UDT incorrectly, but after some investigation it was from GenerateUnsafeRowJoiner in shuffle phase. https://github.com/apache/spark/blob/0e2ca11d80c3921387d7b077cb64c3a0c06b08d7/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeRowJoiner.scala#L32-L43 Here updating position should not happen on fixed-length column, but due to this bug, the value of UDT having fixed-length as sql type would be modified, which actually corrupts the value. ### Why are the changes needed? Misclassifying of the type of length for UDT can corrupt the value when the row is presented to the input of GenerateUnsafeRowJoiner, which brings correctness issue. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? New UT added. Closes #27747 from HeartSaVioR/SPARK-30993. Authored-by: Jungtaek Lim (HeartSaVioR) Signed-off-by: Wenchen Fan (cherry picked from commit f24a46011c8cba086193f697d653b6eccd029e8f) Signed-off-by: Wenchen Fan --- .../spark/sql/catalyst/expressions/UnsafeRow.java | 8 + .../codegen/GenerateUnsafeRowJoinerSuite.scala | 41 +- .../apache/spark/sql/UserDefinedTypeSuite.scala| 37 +++ 3 files changed, 85 insertions(+), 1 deletion(-) diff --git a/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java b/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java index 23e7d1f..034894b 100644 --- a/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java +++ b/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeRow.java @@ -95,6 +95,10 @@ public final class UnsafeRow extends InternalRow implements Externalizable, Kryo } public static boolean isFixedLength(DataType dt) { +if (dt instanceof UserDefinedType) { + return isFixedLength(((UserDefinedType) dt).sqlType()); +} + if (dt instanceof DecimalType) { return ((DecimalType) dt).precision() <= Decimal.MAX_LONG_DIGITS(); } else { @@ -103,6 +107,10 @@ public final class UnsafeRow extends InternalRow implements Externalizable, Kryo } public static boolean isMutable(DataType dt) { +if (dt instanceof UserDefinedType) { + return isMutable(((UserDefinedType) dt).sqlType()); +} + return mutableFieldTypes.contains(dt) || dt instanceof DecimalType || dt instanceof CalendarIntervalType; } diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeRowJoinerSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeRowJoinerSuite.scala index 81e2993..fb1ea7b 100644 --- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeRowJoinerSuite.scala +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateUnsafeRowJoinerSuite.scala @@ -17,13 +17,15 @@ package org.apache.spark.sql.catalyst.expressions.codegen +import java.time.{LocalDateTime, ZoneOffset} + import scala.util.Random import org.apache.spark.SparkFunSuite import org.apache.spark.sql.RandomDataGenerator import org.apache.spark.sql.catalyst.{CatalystTypeConverters, InternalRow} import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder -import org.apache.spark.sql.catalyst.expressions.{JoinedRow, UnsafeProjection, UnsafeRow} +import org.apache.spark.sql.catalyst.expressions.{GenericInternalRow, JoinedRow, UnsafeProjection, UnsafeRow}
[spark] branch master updated (f24a460 -> ac12276)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f24a460 [SPARK-30993][SQL] Use its sql type for UDT when checking the type of length (fixed/var) or mutable add ac12276 [SPARK-30813][ML] Fix Matrices.sprand comments No new revisions were added by this update. Summary of changes: mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala | 2 +- mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala| 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f24a460 -> ac12276)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f24a460 [SPARK-30993][SQL] Use its sql type for UDT when checking the type of length (fixed/var) or mutable add ac12276 [SPARK-30813][ML] Fix Matrices.sprand comments No new revisions were added by this update. Summary of changes: mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala | 2 +- mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala| 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-30813][ML] Fix Matrices.sprand comments
This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new d64d6ee [SPARK-30813][ML] Fix Matrices.sprand comments d64d6ee is described below commit d64d6ee8ac2e9a105da63cd5e310f486de42b76c Author: Wu, Xiaochang AuthorDate: Mon Mar 2 08:56:17 2020 -0600 [SPARK-30813][ML] Fix Matrices.sprand comments ### What changes were proposed in this pull request? Fix mistakes in comments ### Why are the changes needed? There are mistakes in comments ### Does this PR introduce any user-facing change? No ### How was this patch tested? N/A Closes #27564 from xwu99/fix-mllib-sprand-comment. Authored-by: Wu, Xiaochang Signed-off-by: Sean Owen (cherry picked from commit ac122762f5091a7abf62d17595e0f5a99374ac5c) Signed-off-by: Sean Owen --- mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala | 2 +- mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala| 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala b/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala index 61d35c8..34e4366 100644 --- a/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala +++ b/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala @@ -1103,7 +1103,7 @@ object Matrices { DenseMatrix.rand(numRows, numCols, rng) /** - * Generate a `SparseMatrix` consisting of `i.i.d.` gaussian random numbers. + * Generate a `SparseMatrix` consisting of `i.i.d.` uniform random numbers. * @param numRows number of rows of the matrix * @param numCols number of columns of the matrix * @param density the desired density for the matrix diff --git a/mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala b/mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala index 83187d6..57edc96 100644 --- a/mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala +++ b/mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala @@ -1055,7 +1055,7 @@ object Matrices { DenseMatrix.rand(numRows, numCols, rng) /** - * Generate a `SparseMatrix` consisting of `i.i.d.` gaussian random numbers. + * Generate a `SparseMatrix` consisting of `i.i.d.` uniform random numbers. * @param numRows number of rows of the matrix * @param numCols number of columns of the matrix * @param density the desired density for the matrix - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [SPARK-30813][ML] Fix Matrices.sprand comments
This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new cd8f86a [SPARK-30813][ML] Fix Matrices.sprand comments cd8f86a is described below commit cd8f86aafeb2ee5796476bd2cc14e71851611062 Author: Wu, Xiaochang AuthorDate: Mon Mar 2 08:56:17 2020 -0600 [SPARK-30813][ML] Fix Matrices.sprand comments ### What changes were proposed in this pull request? Fix mistakes in comments ### Why are the changes needed? There are mistakes in comments ### Does this PR introduce any user-facing change? No ### How was this patch tested? N/A Closes #27564 from xwu99/fix-mllib-sprand-comment. Authored-by: Wu, Xiaochang Signed-off-by: Sean Owen (cherry picked from commit ac122762f5091a7abf62d17595e0f5a99374ac5c) Signed-off-by: Sean Owen --- mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala | 2 +- mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala| 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala b/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala index 14428c6..5e3c37a 100644 --- a/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala +++ b/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala @@ -1103,7 +1103,7 @@ object Matrices { DenseMatrix.rand(numRows, numCols, rng) /** - * Generate a `SparseMatrix` consisting of `i.i.d.` gaussian random numbers. + * Generate a `SparseMatrix` consisting of `i.i.d.` uniform random numbers. * @param numRows number of rows of the matrix * @param numCols number of columns of the matrix * @param density the desired density for the matrix diff --git a/mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala b/mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala index e474cfa..0c29198 100644 --- a/mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala +++ b/mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala @@ -1055,7 +1055,7 @@ object Matrices { DenseMatrix.rand(numRows, numCols, rng) /** - * Generate a `SparseMatrix` consisting of `i.i.d.` gaussian random numbers. + * Generate a `SparseMatrix` consisting of `i.i.d.` uniform random numbers. * @param numRows number of rows of the matrix * @param numCols number of columns of the matrix * @param density the desired density for the matrix - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-30813][ML] Fix Matrices.sprand comments
This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new d64d6ee [SPARK-30813][ML] Fix Matrices.sprand comments d64d6ee is described below commit d64d6ee8ac2e9a105da63cd5e310f486de42b76c Author: Wu, Xiaochang AuthorDate: Mon Mar 2 08:56:17 2020 -0600 [SPARK-30813][ML] Fix Matrices.sprand comments ### What changes were proposed in this pull request? Fix mistakes in comments ### Why are the changes needed? There are mistakes in comments ### Does this PR introduce any user-facing change? No ### How was this patch tested? N/A Closes #27564 from xwu99/fix-mllib-sprand-comment. Authored-by: Wu, Xiaochang Signed-off-by: Sean Owen (cherry picked from commit ac122762f5091a7abf62d17595e0f5a99374ac5c) Signed-off-by: Sean Owen --- mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala | 2 +- mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala| 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala b/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala index 61d35c8..34e4366 100644 --- a/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala +++ b/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala @@ -1103,7 +1103,7 @@ object Matrices { DenseMatrix.rand(numRows, numCols, rng) /** - * Generate a `SparseMatrix` consisting of `i.i.d.` gaussian random numbers. + * Generate a `SparseMatrix` consisting of `i.i.d.` uniform random numbers. * @param numRows number of rows of the matrix * @param numCols number of columns of the matrix * @param density the desired density for the matrix diff --git a/mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala b/mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala index 83187d6..57edc96 100644 --- a/mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala +++ b/mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala @@ -1055,7 +1055,7 @@ object Matrices { DenseMatrix.rand(numRows, numCols, rng) /** - * Generate a `SparseMatrix` consisting of `i.i.d.` gaussian random numbers. + * Generate a `SparseMatrix` consisting of `i.i.d.` uniform random numbers. * @param numRows number of rows of the matrix * @param numCols number of columns of the matrix * @param density the desired density for the matrix - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [SPARK-30813][ML] Fix Matrices.sprand comments
This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new cd8f86a [SPARK-30813][ML] Fix Matrices.sprand comments cd8f86a is described below commit cd8f86aafeb2ee5796476bd2cc14e71851611062 Author: Wu, Xiaochang AuthorDate: Mon Mar 2 08:56:17 2020 -0600 [SPARK-30813][ML] Fix Matrices.sprand comments ### What changes were proposed in this pull request? Fix mistakes in comments ### Why are the changes needed? There are mistakes in comments ### Does this PR introduce any user-facing change? No ### How was this patch tested? N/A Closes #27564 from xwu99/fix-mllib-sprand-comment. Authored-by: Wu, Xiaochang Signed-off-by: Sean Owen (cherry picked from commit ac122762f5091a7abf62d17595e0f5a99374ac5c) Signed-off-by: Sean Owen --- mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala | 2 +- mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala| 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala b/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala index 14428c6..5e3c37a 100644 --- a/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala +++ b/mllib-local/src/main/scala/org/apache/spark/ml/linalg/Matrices.scala @@ -1103,7 +1103,7 @@ object Matrices { DenseMatrix.rand(numRows, numCols, rng) /** - * Generate a `SparseMatrix` consisting of `i.i.d.` gaussian random numbers. + * Generate a `SparseMatrix` consisting of `i.i.d.` uniform random numbers. * @param numRows number of rows of the matrix * @param numCols number of columns of the matrix * @param density the desired density for the matrix diff --git a/mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala b/mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala index e474cfa..0c29198 100644 --- a/mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala +++ b/mllib/src/main/scala/org/apache/spark/mllib/linalg/Matrices.scala @@ -1055,7 +1055,7 @@ object Matrices { DenseMatrix.rand(numRows, numCols, rng) /** - * Generate a `SparseMatrix` consisting of `i.i.d.` gaussian random numbers. + * Generate a `SparseMatrix` consisting of `i.i.d.` uniform random numbers. * @param numRows number of rows of the matrix * @param numCols number of columns of the matrix * @param density the desired density for the matrix - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ac12276 -> b517f99)
This is an automated email from the ASF dual-hosted git repository. jiangxb1987 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ac12276 [SPARK-30813][ML] Fix Matrices.sprand comments add b517f99 [SPARK-30969][CORE] Remove resource coordination support from Standalone No new revisions were added by this update. Summary of changes: .gitignore | 1 - .../main/scala/org/apache/spark/SparkContext.scala | 25 +- .../spark/deploy/StandaloneResourceUtils.scala | 263 + .../org/apache/spark/deploy/worker/Worker.scala| 27 +-- .../org/apache/spark/internal/config/package.scala | 17 -- .../main/scala/org/apache/spark/util/Utils.scala | 22 -- .../scala/org/apache/spark/SparkContextSuite.scala | 6 +- .../apache/spark/deploy/worker/WorkerSuite.scala | 74 +- .../resource/ResourceDiscoveryPluginSuite.scala| 4 - docs/configuration.md | 27 +-- docs/spark-standalone.md | 10 +- python/pyspark/tests/test_context.py | 2 - python/pyspark/tests/test_taskcontext.py | 2 - 13 files changed, 17 insertions(+), 463 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (ac12276 -> b517f99)
This is an automated email from the ASF dual-hosted git repository. jiangxb1987 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from ac12276 [SPARK-30813][ML] Fix Matrices.sprand comments add b517f99 [SPARK-30969][CORE] Remove resource coordination support from Standalone No new revisions were added by this update. Summary of changes: .gitignore | 1 - .../main/scala/org/apache/spark/SparkContext.scala | 25 +- .../spark/deploy/StandaloneResourceUtils.scala | 263 + .../org/apache/spark/deploy/worker/Worker.scala| 27 +-- .../org/apache/spark/internal/config/package.scala | 17 -- .../main/scala/org/apache/spark/util/Utils.scala | 22 -- .../scala/org/apache/spark/SparkContextSuite.scala | 6 +- .../apache/spark/deploy/worker/WorkerSuite.scala | 74 +- .../resource/ResourceDiscoveryPluginSuite.scala| 4 - docs/configuration.md | 27 +-- docs/spark-standalone.md | 10 +- python/pyspark/tests/test_context.py | 2 - python/pyspark/tests/test_taskcontext.py | 2 - 13 files changed, 17 insertions(+), 463 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-30969][CORE] Remove resource coordination support from Standalone
This is an automated email from the ASF dual-hosted git repository. jiangxb1987 pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 148262f [SPARK-30969][CORE] Remove resource coordination support from Standalone 148262f is described below commit 148262f3fbdf7c5da7cd147cf43bf5ebab5f5244 Author: yi.wu AuthorDate: Mon Mar 2 11:23:07 2020 -0800 [SPARK-30969][CORE] Remove resource coordination support from Standalone ### What changes were proposed in this pull request? Remove automatically resource coordination support from Standalone. ### Why are the changes needed? Resource coordination is mainly designed for the scenario where multiple workers launched on the same host. However, it's, actually, a non-existed scenario for today's Spark. Because, Spark now can start multiple executors in a single Worker, while it only allow one executor per Worker at very beginning. So, now, it really help nothing for user to launch multiple workers on the same host. Thus, it's not worth for us to bring over complicated implementation and potential high maintain [...] ### Does this PR introduce any user-facing change? No, it's Spark 3.0 feature. ### How was this patch tested? Pass Jenkins. Closes #27722 from Ngone51/abandon_coordination. Authored-by: yi.wu Signed-off-by: Xingbo Jiang (cherry picked from commit b517f991fe0c95a186872d38be6a2091d9326195) Signed-off-by: Xingbo Jiang --- .gitignore | 1 - .../main/scala/org/apache/spark/SparkContext.scala | 25 +- .../spark/deploy/StandaloneResourceUtils.scala | 263 + .../org/apache/spark/deploy/worker/Worker.scala| 27 +-- .../org/apache/spark/internal/config/package.scala | 17 -- .../main/scala/org/apache/spark/util/Utils.scala | 22 -- .../scala/org/apache/spark/SparkContextSuite.scala | 6 +- .../apache/spark/deploy/worker/WorkerSuite.scala | 74 +- .../resource/ResourceDiscoveryPluginSuite.scala| 4 - docs/configuration.md | 27 +-- docs/spark-standalone.md | 10 +- 11 files changed, 17 insertions(+), 459 deletions(-) diff --git a/.gitignore b/.gitignore index 798e8ac..198fdee 100644 --- a/.gitignore +++ b/.gitignore @@ -72,7 +72,6 @@ scalastyle-on-compile.generated.xml scalastyle-output.xml scalastyle.txt spark-*-bin-*.tgz -spark-resources/ spark-tests.log src_managed/ streaming-tests.log diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala b/core/src/main/scala/org/apache/spark/SparkContext.scala index 91188d5..bcbb7e4 100644 --- a/core/src/main/scala/org/apache/spark/SparkContext.scala +++ b/core/src/main/scala/org/apache/spark/SparkContext.scala @@ -41,7 +41,6 @@ import org.apache.hadoop.mapreduce.lib.input.{FileInputFormat => NewFileInputFor import org.apache.spark.annotation.DeveloperApi import org.apache.spark.broadcast.Broadcast import org.apache.spark.deploy.{LocalSparkCluster, SparkHadoopUtil} -import org.apache.spark.deploy.StandaloneResourceUtils._ import org.apache.spark.executor.{ExecutorMetrics, ExecutorMetricsSource} import org.apache.spark.input.{FixedLengthBinaryInputFormat, PortableDataStream, StreamInputFormat, WholeTextFileInputFormat} import org.apache.spark.internal.Logging @@ -250,15 +249,6 @@ class SparkContext(config: SparkConf) extends Logging { def isLocal: Boolean = Utils.isLocalMaster(_conf) - private def isClientStandalone: Boolean = { -val isSparkCluster = master match { - case SparkMasterRegex.SPARK_REGEX(_) => true - case SparkMasterRegex.LOCAL_CLUSTER_REGEX(_, _, _) => true - case _ => false -} -deployMode == "client" && isSparkCluster - } - /** * @return true if context is stopped or in the midst of stopping. */ @@ -396,17 +386,7 @@ class SparkContext(config: SparkConf) extends Logging { _driverLogger = DriverLogger(_conf) val resourcesFileOpt = conf.get(DRIVER_RESOURCES_FILE) -val allResources = getOrDiscoverAllResources(_conf, SPARK_DRIVER_PREFIX, resourcesFileOpt) -_resources = { - // driver submitted in client mode under Standalone may have conflicting resources with - // other drivers/workers on this host. We should sync driver's resources info into - // SPARK_RESOURCES/SPARK_RESOURCES_COORDINATE_DIR/ to avoid collision. - if (isClientStandalone) { -acquireResources(_conf, SPARK_DRIVER_PREFIX, allResources, Utils.getProcessId) - } else { -allResources - } -} +_resources = getOrDiscoverAllResources(_conf, SPARK_DRIVER_PREFIX, resourcesFileOpt) logResourceInfo(SPARK_DRIVER_PREFIX, _resources) // log out spark.app.name in the Spark driver logs @@ -2019,9 +1999,6 @@ class Spark
[spark] branch branch-3.0 updated: [SPARK-30969][CORE] Remove resource coordination support from Standalone
This is an automated email from the ASF dual-hosted git repository. jiangxb1987 pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 148262f [SPARK-30969][CORE] Remove resource coordination support from Standalone 148262f is described below commit 148262f3fbdf7c5da7cd147cf43bf5ebab5f5244 Author: yi.wu AuthorDate: Mon Mar 2 11:23:07 2020 -0800 [SPARK-30969][CORE] Remove resource coordination support from Standalone ### What changes were proposed in this pull request? Remove automatically resource coordination support from Standalone. ### Why are the changes needed? Resource coordination is mainly designed for the scenario where multiple workers launched on the same host. However, it's, actually, a non-existed scenario for today's Spark. Because, Spark now can start multiple executors in a single Worker, while it only allow one executor per Worker at very beginning. So, now, it really help nothing for user to launch multiple workers on the same host. Thus, it's not worth for us to bring over complicated implementation and potential high maintain [...] ### Does this PR introduce any user-facing change? No, it's Spark 3.0 feature. ### How was this patch tested? Pass Jenkins. Closes #27722 from Ngone51/abandon_coordination. Authored-by: yi.wu Signed-off-by: Xingbo Jiang (cherry picked from commit b517f991fe0c95a186872d38be6a2091d9326195) Signed-off-by: Xingbo Jiang --- .gitignore | 1 - .../main/scala/org/apache/spark/SparkContext.scala | 25 +- .../spark/deploy/StandaloneResourceUtils.scala | 263 + .../org/apache/spark/deploy/worker/Worker.scala| 27 +-- .../org/apache/spark/internal/config/package.scala | 17 -- .../main/scala/org/apache/spark/util/Utils.scala | 22 -- .../scala/org/apache/spark/SparkContextSuite.scala | 6 +- .../apache/spark/deploy/worker/WorkerSuite.scala | 74 +- .../resource/ResourceDiscoveryPluginSuite.scala| 4 - docs/configuration.md | 27 +-- docs/spark-standalone.md | 10 +- 11 files changed, 17 insertions(+), 459 deletions(-) diff --git a/.gitignore b/.gitignore index 798e8ac..198fdee 100644 --- a/.gitignore +++ b/.gitignore @@ -72,7 +72,6 @@ scalastyle-on-compile.generated.xml scalastyle-output.xml scalastyle.txt spark-*-bin-*.tgz -spark-resources/ spark-tests.log src_managed/ streaming-tests.log diff --git a/core/src/main/scala/org/apache/spark/SparkContext.scala b/core/src/main/scala/org/apache/spark/SparkContext.scala index 91188d5..bcbb7e4 100644 --- a/core/src/main/scala/org/apache/spark/SparkContext.scala +++ b/core/src/main/scala/org/apache/spark/SparkContext.scala @@ -41,7 +41,6 @@ import org.apache.hadoop.mapreduce.lib.input.{FileInputFormat => NewFileInputFor import org.apache.spark.annotation.DeveloperApi import org.apache.spark.broadcast.Broadcast import org.apache.spark.deploy.{LocalSparkCluster, SparkHadoopUtil} -import org.apache.spark.deploy.StandaloneResourceUtils._ import org.apache.spark.executor.{ExecutorMetrics, ExecutorMetricsSource} import org.apache.spark.input.{FixedLengthBinaryInputFormat, PortableDataStream, StreamInputFormat, WholeTextFileInputFormat} import org.apache.spark.internal.Logging @@ -250,15 +249,6 @@ class SparkContext(config: SparkConf) extends Logging { def isLocal: Boolean = Utils.isLocalMaster(_conf) - private def isClientStandalone: Boolean = { -val isSparkCluster = master match { - case SparkMasterRegex.SPARK_REGEX(_) => true - case SparkMasterRegex.LOCAL_CLUSTER_REGEX(_, _, _) => true - case _ => false -} -deployMode == "client" && isSparkCluster - } - /** * @return true if context is stopped or in the midst of stopping. */ @@ -396,17 +386,7 @@ class SparkContext(config: SparkConf) extends Logging { _driverLogger = DriverLogger(_conf) val resourcesFileOpt = conf.get(DRIVER_RESOURCES_FILE) -val allResources = getOrDiscoverAllResources(_conf, SPARK_DRIVER_PREFIX, resourcesFileOpt) -_resources = { - // driver submitted in client mode under Standalone may have conflicting resources with - // other drivers/workers on this host. We should sync driver's resources info into - // SPARK_RESOURCES/SPARK_RESOURCES_COORDINATE_DIR/ to avoid collision. - if (isClientStandalone) { -acquireResources(_conf, SPARK_DRIVER_PREFIX, allResources, Utils.getProcessId) - } else { -allResources - } -} +_resources = getOrDiscoverAllResources(_conf, SPARK_DRIVER_PREFIX, resourcesFileOpt) logResourceInfo(SPARK_DRIVER_PREFIX, _resources) // log out spark.app.name in the Spark driver logs @@ -2019,9 +1999,6 @@ class Spark
[spark] branch master updated (b517f99 -> f0010c8)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from b517f99 [SPARK-30969][CORE] Remove resource coordination support from Standalone add f0010c8 [SPARK-31003][TESTS] Fix incorrect uses of assume() in tests No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/catalyst/expressions/OrderingSuite.scala | 2 +- sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala | 2 +- .../scala/org/apache/spark/sql/execution/command/DDLSuite.scala | 4 ++-- .../src/test/scala/org/apache/spark/sql/internal/CatalogSuite.scala | 6 +++--- .../test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala | 2 +- .../scala/org/apache/spark/sql/sources/BucketedWriteSuite.scala | 2 +- .../scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala| 6 +++--- .../src/test/scala/org/apache/spark/sql/hive/test/TestHive.scala| 2 +- .../apache/spark/sql/sources/BucketedReadWithHiveSupportSuite.scala | 2 +- .../spark/sql/sources/BucketedWriteWithHiveSupportSuite.scala | 2 +- 10 files changed, 15 insertions(+), 15 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (b517f99 -> f0010c8)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from b517f99 [SPARK-30969][CORE] Remove resource coordination support from Standalone add f0010c8 [SPARK-31003][TESTS] Fix incorrect uses of assume() in tests No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/catalyst/expressions/OrderingSuite.scala | 2 +- sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala | 2 +- .../scala/org/apache/spark/sql/execution/command/DDLSuite.scala | 4 ++-- .../src/test/scala/org/apache/spark/sql/internal/CatalogSuite.scala | 6 +++--- .../test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala | 2 +- .../scala/org/apache/spark/sql/sources/BucketedWriteSuite.scala | 2 +- .../scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala| 6 +++--- .../src/test/scala/org/apache/spark/sql/hive/test/TestHive.scala| 2 +- .../apache/spark/sql/sources/BucketedReadWithHiveSupportSuite.scala | 2 +- .../spark/sql/sources/BucketedWriteWithHiveSupportSuite.scala | 2 +- 10 files changed, 15 insertions(+), 15 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-31003][TESTS] Fix incorrect uses of assume() in tests
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 8cb23f0 [SPARK-31003][TESTS] Fix incorrect uses of assume() in tests 8cb23f0 is described below commit 8cb23f0cb5b20b7e49fdd16c52d6451e901d9a7a Author: Josh Rosen AuthorDate: Mon Mar 2 15:20:45 2020 -0800 [SPARK-31003][TESTS] Fix incorrect uses of assume() in tests ### What changes were proposed in this pull request? This patch fixes several incorrect uses of `assume()` in our tests. If a call to `assume(condition)` fails then it will cause the test to be marked as skipped instead of failed: this feature allows test cases to be skipped if certain prerequisites are missing. For example, we use this to skip certain tests when running on Windows (or when Python dependencies are unavailable). In contrast, `assert(condition)` will fail the test if the condition doesn't hold. If `assume()` is accidentally substituted for `assert()`then the resulting test will be marked as skipped in cases where it should have failed, undermining the purpose of the test. This patch fixes several such cases, replacing certain `assume()` calls with `assert()`. Credit to ahirreddy for spotting this problem. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Existing tests. Closes #27754 from JoshRosen/fix-assume-vs-assert. Lead-authored-by: Josh Rosen Co-authored-by: Josh Rosen Signed-off-by: Dongjoon Hyun (cherry picked from commit f0010c81e2ef9b8859b39917bb62b48d739a4a22) Signed-off-by: Dongjoon Hyun --- .../org/apache/spark/sql/catalyst/expressions/OrderingSuite.scala | 2 +- sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala | 2 +- .../scala/org/apache/spark/sql/execution/command/DDLSuite.scala | 4 ++-- .../src/test/scala/org/apache/spark/sql/internal/CatalogSuite.scala | 6 +++--- .../test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala | 2 +- .../scala/org/apache/spark/sql/sources/BucketedWriteSuite.scala | 2 +- .../scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala| 6 +++--- .../src/test/scala/org/apache/spark/sql/hive/test/TestHive.scala| 2 +- .../apache/spark/sql/sources/BucketedReadWithHiveSupportSuite.scala | 2 +- .../spark/sql/sources/BucketedWriteWithHiveSupportSuite.scala | 2 +- 10 files changed, 15 insertions(+), 15 deletions(-) diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/OrderingSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/OrderingSuite.scala index 94e251d..4488902 100644 --- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/OrderingSuite.scala +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/OrderingSuite.scala @@ -106,7 +106,7 @@ class OrderingSuite extends SparkFunSuite with ExpressionEvalHelper { StructField("a", dataType, nullable = true) :: StructField("b", dataType, nullable = true) :: Nil) val maybeDataGenerator = RandomDataGenerator.forType(rowType, nullable = false) -assume(maybeDataGenerator.isDefined) +assert(maybeDataGenerator.isDefined) val randGenerator = maybeDataGenerator.get val toCatalyst = CatalystTypeConverters.createToCatalystConverter(rowType) for (_ <- 1 to 50) { diff --git a/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala index cd2c681..8189353 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala @@ -195,7 +195,7 @@ class CachedTableSuite extends QueryTest with SQLTestUtils } test("SPARK-1669: cacheTable should be idempotent") { -assume(!spark.table("testData").logicalPlan.isInstanceOf[InMemoryRelation]) +assert(!spark.table("testData").logicalPlan.isInstanceOf[InMemoryRelation]) spark.catalog.cacheTable("testData") assertCached(spark.table("testData")) diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala index 6c824c2..5a67dce 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala @@ -1033,7 +1033,7 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils { df.write.insertInto("students") spark.catalog.cacheTable("students") checkAnswer(spark.table("students"), df) -assume(spark.catalog.isCache
[spark] branch branch-3.0 updated: [SPARK-31003][TESTS] Fix incorrect uses of assume() in tests
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 8cb23f0 [SPARK-31003][TESTS] Fix incorrect uses of assume() in tests 8cb23f0 is described below commit 8cb23f0cb5b20b7e49fdd16c52d6451e901d9a7a Author: Josh Rosen AuthorDate: Mon Mar 2 15:20:45 2020 -0800 [SPARK-31003][TESTS] Fix incorrect uses of assume() in tests ### What changes were proposed in this pull request? This patch fixes several incorrect uses of `assume()` in our tests. If a call to `assume(condition)` fails then it will cause the test to be marked as skipped instead of failed: this feature allows test cases to be skipped if certain prerequisites are missing. For example, we use this to skip certain tests when running on Windows (or when Python dependencies are unavailable). In contrast, `assert(condition)` will fail the test if the condition doesn't hold. If `assume()` is accidentally substituted for `assert()`then the resulting test will be marked as skipped in cases where it should have failed, undermining the purpose of the test. This patch fixes several such cases, replacing certain `assume()` calls with `assert()`. Credit to ahirreddy for spotting this problem. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Existing tests. Closes #27754 from JoshRosen/fix-assume-vs-assert. Lead-authored-by: Josh Rosen Co-authored-by: Josh Rosen Signed-off-by: Dongjoon Hyun (cherry picked from commit f0010c81e2ef9b8859b39917bb62b48d739a4a22) Signed-off-by: Dongjoon Hyun --- .../org/apache/spark/sql/catalyst/expressions/OrderingSuite.scala | 2 +- sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala | 2 +- .../scala/org/apache/spark/sql/execution/command/DDLSuite.scala | 4 ++-- .../src/test/scala/org/apache/spark/sql/internal/CatalogSuite.scala | 6 +++--- .../test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala | 2 +- .../scala/org/apache/spark/sql/sources/BucketedWriteSuite.scala | 2 +- .../scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala| 6 +++--- .../src/test/scala/org/apache/spark/sql/hive/test/TestHive.scala| 2 +- .../apache/spark/sql/sources/BucketedReadWithHiveSupportSuite.scala | 2 +- .../spark/sql/sources/BucketedWriteWithHiveSupportSuite.scala | 2 +- 10 files changed, 15 insertions(+), 15 deletions(-) diff --git a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/OrderingSuite.scala b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/OrderingSuite.scala index 94e251d..4488902 100644 --- a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/OrderingSuite.scala +++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/OrderingSuite.scala @@ -106,7 +106,7 @@ class OrderingSuite extends SparkFunSuite with ExpressionEvalHelper { StructField("a", dataType, nullable = true) :: StructField("b", dataType, nullable = true) :: Nil) val maybeDataGenerator = RandomDataGenerator.forType(rowType, nullable = false) -assume(maybeDataGenerator.isDefined) +assert(maybeDataGenerator.isDefined) val randGenerator = maybeDataGenerator.get val toCatalyst = CatalystTypeConverters.createToCatalystConverter(rowType) for (_ <- 1 to 50) { diff --git a/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala index cd2c681..8189353 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala @@ -195,7 +195,7 @@ class CachedTableSuite extends QueryTest with SQLTestUtils } test("SPARK-1669: cacheTable should be idempotent") { -assume(!spark.table("testData").logicalPlan.isInstanceOf[InMemoryRelation]) +assert(!spark.table("testData").logicalPlan.isInstanceOf[InMemoryRelation]) spark.catalog.cacheTable("testData") assertCached(spark.table("testData")) diff --git a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala index 6c824c2..5a67dce 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/execution/command/DDLSuite.scala @@ -1033,7 +1033,7 @@ abstract class DDLSuite extends QueryTest with SQLTestUtils { df.write.insertInto("students") spark.catalog.cacheTable("students") checkAnswer(spark.table("students"), df) -assume(spark.catalog.isCache
[spark] branch branch-2.4 updated (cd8f86a -> 0b71b4d)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git. from cd8f86a [SPARK-30813][ML] Fix Matrices.sprand comments add 0b71b4d [SPARK-31003][TESTS] Fix incorrect uses of assume() in tests No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/catalyst/expressions/OrderingSuite.scala | 2 +- sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala | 2 +- .../scala/org/apache/spark/sql/execution/command/DDLSuite.scala | 4 ++-- .../src/test/scala/org/apache/spark/sql/internal/CatalogSuite.scala | 6 +++--- .../test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala | 2 +- .../scala/org/apache/spark/sql/sources/BucketedWriteSuite.scala | 2 +- .../src/main/scala/org/apache/spark/sql/hive/test/TestHive.scala| 2 +- .../scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala| 6 +++--- .../apache/spark/sql/sources/BucketedReadWithHiveSupportSuite.scala | 2 +- .../spark/sql/sources/BucketedWriteWithHiveSupportSuite.scala | 2 +- 10 files changed, 15 insertions(+), 15 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated (cd8f86a -> 0b71b4d)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git. from cd8f86a [SPARK-30813][ML] Fix Matrices.sprand comments add 0b71b4d [SPARK-31003][TESTS] Fix incorrect uses of assume() in tests No new revisions were added by this update. Summary of changes: .../org/apache/spark/sql/catalyst/expressions/OrderingSuite.scala | 2 +- sql/core/src/test/scala/org/apache/spark/sql/CachedTableSuite.scala | 2 +- .../scala/org/apache/spark/sql/execution/command/DDLSuite.scala | 4 ++-- .../src/test/scala/org/apache/spark/sql/internal/CatalogSuite.scala | 6 +++--- .../test/scala/org/apache/spark/sql/sources/BucketedReadSuite.scala | 2 +- .../scala/org/apache/spark/sql/sources/BucketedWriteSuite.scala | 2 +- .../src/main/scala/org/apache/spark/sql/hive/test/TestHive.scala| 2 +- .../scala/org/apache/spark/sql/hive/execution/HiveDDLSuite.scala| 6 +++--- .../apache/spark/sql/sources/BucketedReadWithHiveSupportSuite.scala | 2 +- .../spark/sql/sources/BucketedWriteWithHiveSupportSuite.scala | 2 +- 10 files changed, 15 insertions(+), 15 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f0010c8 -> 473a28c)
This is an automated email from the ASF dual-hosted git repository. lixiao pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f0010c8 [SPARK-31003][TESTS] Fix incorrect uses of assume() in tests add 473a28c [SPARK-30991] Refactor AQE readers and RDDs No new revisions were added by this update. Summary of changes: .../spark/sql/execution/ShuffledRowRDD.scala | 142 - .../apache/spark/sql/execution/SparkPlanInfo.scala | 2 +- .../adaptive/CustomShuffleReaderExec.scala | 81 .../execution/adaptive/CustomShuffledRowRDD.scala | 113 .../execution/adaptive/LocalShuffledRowRDD.scala | 112 .../adaptive/OptimizeLocalShuffleReader.scala | 88 +++-- .../execution/adaptive/OptimizeSkewedJoin.scala| 72 ++- .../adaptive/ReduceNumShufflePartitions.scala | 49 ++- .../adaptive/ShufflePartitionsCoalescer.scala | 23 ++-- .../execution/exchange/ShuffleExchangeExec.scala | 12 +- .../ReduceNumShufflePartitionsSuite.scala | 28 ++-- .../ShufflePartitionsCoalescerSuite.scala | 101 ++- .../adaptive/AdaptiveQueryExecSuite.scala | 23 ++-- 13 files changed, 317 insertions(+), 529 deletions(-) create mode 100644 sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/CustomShuffleReaderExec.scala delete mode 100644 sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/CustomShuffledRowRDD.scala delete mode 100644 sql/core/src/main/scala/org/apache/spark/sql/execution/adaptive/LocalShuffledRowRDD.scala - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-30991] Refactor AQE readers and RDDs
This is an automated email from the ASF dual-hosted git repository. lixiao pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 597 [SPARK-30991] Refactor AQE readers and RDDs 597 is described below commit 597b5507448980e4fadbad85ffb104808081 Author: maryannxue AuthorDate: Mon Mar 2 16:04:00 2020 -0800 [SPARK-30991] Refactor AQE readers and RDDs ### What changes were proposed in this pull request? This PR combines `CustomShuffledRowRDD` and `LocalShuffledRowRDD` into `ShuffledRowRDD`, and creates `CustomShuffleReaderExec` to unify and replace all existing AQE readers: `CoalescedShuffleReaderExec`, `LocalShuffleReaderExec` and `SkewJoinShuffleReaderExec`. ### Why are the changes needed? To reduce code redundancy. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Passed existing UTs. Closes #27742 from maryannxue/aqe-readers. Authored-by: maryannxue Signed-off-by: gatorsmile (cherry picked from commit 473a28c1d032993c7fa515b39f2cb1e3105d65d3) Signed-off-by: gatorsmile --- .../spark/sql/execution/ShuffledRowRDD.scala | 142 - .../apache/spark/sql/execution/SparkPlanInfo.scala | 2 +- .../adaptive/CustomShuffleReaderExec.scala | 81 .../execution/adaptive/CustomShuffledRowRDD.scala | 113 .../execution/adaptive/LocalShuffledRowRDD.scala | 112 .../adaptive/OptimizeLocalShuffleReader.scala | 88 +++-- .../execution/adaptive/OptimizeSkewedJoin.scala| 72 ++- .../adaptive/ReduceNumShufflePartitions.scala | 49 ++- .../adaptive/ShufflePartitionsCoalescer.scala | 23 ++-- .../execution/exchange/ShuffleExchangeExec.scala | 12 +- .../ReduceNumShufflePartitionsSuite.scala | 28 ++-- .../ShufflePartitionsCoalescerSuite.scala | 101 ++- .../adaptive/AdaptiveQueryExecSuite.scala | 23 ++-- 13 files changed, 317 insertions(+), 529 deletions(-) diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/ShuffledRowRDD.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/ShuffledRowRDD.scala index 4c19f95..eb02259 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/ShuffledRowRDD.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/ShuffledRowRDD.scala @@ -26,17 +26,28 @@ import org.apache.spark.sql.catalyst.InternalRow import org.apache.spark.sql.execution.metric.{SQLMetric, SQLShuffleReadMetricsReporter} import org.apache.spark.sql.internal.SQLConf +sealed trait ShufflePartitionSpec + +// A partition that reads data of one or more reducers, from `startReducerIndex` (inclusive) to +// `endReducerIndex` (exclusive). +case class CoalescedPartitionSpec( + startReducerIndex: Int, endReducerIndex: Int) extends ShufflePartitionSpec + +// A partition that reads partial data of one reducer, from `startMapIndex` (inclusive) to +// `endMapIndex` (exclusive). +case class PartialReducerPartitionSpec( + reducerIndex: Int, startMapIndex: Int, endMapIndex: Int) extends ShufflePartitionSpec + +// A partition that reads partial data of one mapper, from `startReducerIndex` (inclusive) to +// `endReducerIndex` (exclusive). +case class PartialMapperPartitionSpec( + mapIndex: Int, startReducerIndex: Int, endReducerIndex: Int) extends ShufflePartitionSpec + /** - * The [[Partition]] used by [[ShuffledRowRDD]]. A post-shuffle partition - * (identified by `postShufflePartitionIndex`) contains a range of pre-shuffle partitions - * (`startPreShufflePartitionIndex` to `endPreShufflePartitionIndex - 1`, inclusive). + * The [[Partition]] used by [[ShuffledRowRDD]]. */ -private final class ShuffledRowRDDPartition( -val postShufflePartitionIndex: Int, -val startPreShufflePartitionIndex: Int, -val endPreShufflePartitionIndex: Int) extends Partition { - override val index: Int = postShufflePartitionIndex -} +private final case class ShuffledRowRDDPartition( + index: Int, spec: ShufflePartitionSpec) extends Partition /** * A dummy partitioner for use with records whose partition ids have been pre-computed (i.e. for @@ -94,8 +105,7 @@ class CoalescedPartitioner(val parent: Partitioner, val partitionStartIndices: A * interfaces / internals. * * This RDD takes a [[ShuffleDependency]] (`dependency`), - * and an optional array of partition start indices as input arguments - * (`specifiedPartitionStartIndices`). + * and an array of [[ShufflePartitionSpec]] as input arguments. * * The `dependency` has the parent RDD of this RDD, which represents the dataset before shuffle * (i.e. map output). Elements of this RDD are (partitionId, Row) pairs. @@ -103,79 +113,97 @@ class CoalescedPartitioner(val parent: Partitioner, val
[spark] branch master updated (473a28c -> 3956e95)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 473a28c [SPARK-30991] Refactor AQE readers and RDDs add 3956e95 [SPARK-25202][SQL][FOLLOW-UP] Keep the old parameter name 'pattern' at split in Scala API No new revisions were added by this update. Summary of changes: .../main/scala/org/apache/spark/sql/functions.scala | 20 ++-- .../scala/org/apache/spark/sql/DataFrameSuite.scala | 2 +- 2 files changed, 11 insertions(+), 11 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-25202][SQL][FOLLOW-UP] Keep the old parameter name 'pattern' at split in Scala API
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new fafa8d8 [SPARK-25202][SQL][FOLLOW-UP] Keep the old parameter name 'pattern' at split in Scala API fafa8d8 is described below commit fafa8d887ded4ca779bcd9f1cef88cc40aded0ae Author: HyukjinKwon AuthorDate: Tue Mar 3 10:24:50 2020 +0900 [SPARK-25202][SQL][FOLLOW-UP] Keep the old parameter name 'pattern' at split in Scala API ### What changes were proposed in this pull request? To address the concern pointed out in https://github.com/apache/spark/pull/7. This will make `split` source-compatible by removing minimal cosmetic changes. ### Why are the changes needed? For source compatibility. ### Does this PR introduce any user-facing change? No (it will prevent potential user-facing change from the original PR) ### How was this patch tested? Unittest was changed (in order for us to detect that source compatibility easily). Closes #27756 from HyukjinKwon/SPARK-25202. Authored-by: HyukjinKwon Signed-off-by: HyukjinKwon (cherry picked from commit 3956e95f059ba9599c3cfde29225177d29b2494a) Signed-off-by: HyukjinKwon --- .../main/scala/org/apache/spark/sql/functions.scala | 20 ++-- .../scala/org/apache/spark/sql/DataFrameSuite.scala | 2 +- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/sql/core/src/main/scala/org/apache/spark/sql/functions.scala b/sql/core/src/main/scala/org/apache/spark/sql/functions.scala index c60df14..c6e8cf7 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/functions.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/functions.scala @@ -2460,25 +2460,25 @@ object functions { def soundex(e: Column): Column = withExpr { SoundEx(e.expr) } /** - * Splits str around matches of the given regex. + * Splits str around matches of the given pattern. * * @param str a string expression to split - * @param regex a string representing a regular expression. The regex string should be - * a Java regular expression. + * @param pattern a string representing a regular expression. The regex string should be + *a Java regular expression. * * @group string_funcs * @since 1.5.0 */ - def split(str: Column, regex: String): Column = withExpr { -StringSplit(str.expr, Literal(regex), Literal(-1)) + def split(str: Column, pattern: String): Column = withExpr { +StringSplit(str.expr, Literal(pattern), Literal(-1)) } /** - * Splits str around matches of the given regex. + * Splits str around matches of the given pattern. * * @param str a string expression to split - * @param regex a string representing a regular expression. The regex string should be - * a Java regular expression. + * @param pattern a string representing a regular expression. The regex string should be + *a Java regular expression. * @param limit an integer expression which controls the number of times the regex is applied. * * limit greater than 0: The resulting array's length will not be more than limit, @@ -2491,8 +2491,8 @@ object functions { * @group string_funcs * @since 3.0.0 */ - def split(str: Column, regex: String, limit: Int): Column = withExpr { -StringSplit(str.expr, Literal(regex), Literal(limit)) + def split(str: Column, pattern: String, limit: Int): Column = withExpr { +StringSplit(str.expr, Literal(pattern), Literal(limit)) } /** diff --git a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala index 42a9073..e74d553 100644 --- a/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala +++ b/sql/core/src/test/scala/org/apache/spark/sql/DataFrameSuite.scala @@ -196,7 +196,7 @@ class DataFrameSuite extends QueryTest test("explode on output of array-valued function") { val df = Seq(("1,2"), ("4"), ("7,8,9")).toDF("csv") checkAnswer( - df.select(explode(split($"csv", ","))), + df.select(explode(split($"csv", pattern = ","))), Row("1") :: Row("2") :: Row("4") :: Row("7") :: Row("8") :: Row("9") :: Nil) } - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (3956e95 -> 1fac06c)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 3956e95 [SPARK-25202][SQL][FOLLOW-UP] Keep the old parameter name 'pattern' at split in Scala API add 1fac06c Revert "[SPARK-30808][SQL] Enable Java 8 time API in Thrift server" No new revisions were added by this update. Summary of changes: .../apache/spark/sql/execution/HiveResult.scala| 61 -- .../scala/org/apache/spark/sql/SQLQuerySuite.scala | 3 +- .../org/apache/spark/sql/SQLQueryTestSuite.scala | 2 +- .../spark/sql/execution/HiveResultSuite.scala | 25 + .../SparkExecuteStatementOperation.scala | 10 +--- .../sql/hive/thriftserver/SparkSQLDriver.scala | 5 +- .../sql/hive/execution/HiveComparisonTest.scala| 4 +- 7 files changed, 43 insertions(+), 67 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (3956e95 -> 1fac06c)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 3956e95 [SPARK-25202][SQL][FOLLOW-UP] Keep the old parameter name 'pattern' at split in Scala API add 1fac06c Revert "[SPARK-30808][SQL] Enable Java 8 time API in Thrift server" No new revisions were added by this update. Summary of changes: .../apache/spark/sql/execution/HiveResult.scala| 61 -- .../scala/org/apache/spark/sql/SQLQuerySuite.scala | 3 +- .../org/apache/spark/sql/SQLQueryTestSuite.scala | 2 +- .../spark/sql/execution/HiveResultSuite.scala | 25 + .../SparkExecuteStatementOperation.scala | 10 +--- .../sql/hive/thriftserver/SparkSQLDriver.scala | 5 +- .../sql/hive/execution/HiveComparisonTest.scala| 4 +- 7 files changed, 43 insertions(+), 67 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: Revert "[SPARK-30808][SQL] Enable Java 8 time API in Thrift server"
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new 4fa447c Revert "[SPARK-30808][SQL] Enable Java 8 time API in Thrift server" 4fa447c is described below commit 4fa447cabc1c09170edd9da0e586660a7ae0db74 Author: Kent Yao AuthorDate: Tue Mar 3 14:21:20 2020 +0800 Revert "[SPARK-30808][SQL] Enable Java 8 time API in Thrift server" This reverts commit afaeb29599593f021c9ea47e52f8c70013a4afef. ### What changes were proposed in this pull request? Based on the result and comment from https://github.com/apache/spark/pull/27552#discussion_r385531744 In the hive module, server-side provides datetime values simply use `value.toSting`, and the client-side regenerates the results back in `HiveBaseResultSet` with `java.sql.Date(Timestamp).valueOf`. there will be inconsistency between client and server if we use java8 APIs ### Why are the changes needed? the change is still unclear enough ### Does this PR introduce any user-facing change? no ### How was this patch tested? Nah Closes #27733 from yaooqinn/SPARK-30808. Authored-by: Kent Yao Signed-off-by: Wenchen Fan (cherry picked from commit 1fac06c4307c8c7a5a48a50952d48ee5b9ebccb2) Signed-off-by: Wenchen Fan --- .../apache/spark/sql/execution/HiveResult.scala| 61 -- .../scala/org/apache/spark/sql/SQLQuerySuite.scala | 3 +- .../org/apache/spark/sql/SQLQueryTestSuite.scala | 2 +- .../spark/sql/execution/HiveResultSuite.scala | 25 + .../SparkExecuteStatementOperation.scala | 10 +--- .../sql/hive/thriftserver/SparkSQLDriver.scala | 5 +- .../sql/hive/execution/HiveComparisonTest.scala| 4 +- 7 files changed, 43 insertions(+), 67 deletions(-) diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala index b191840..5a2f16d 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala @@ -21,10 +21,9 @@ import java.nio.charset.StandardCharsets import java.sql.{Date, Timestamp} import java.time.{Instant, LocalDate} -import org.apache.spark.sql.{Dataset, Row} +import org.apache.spark.sql.Row import org.apache.spark.sql.catalyst.util.{DateFormatter, DateTimeUtils, TimestampFormatter} import org.apache.spark.sql.execution.command.{DescribeCommandBase, ExecutedCommandExec, ShowTablesCommand} -import org.apache.spark.sql.functions._ import org.apache.spark.sql.internal.SQLConf import org.apache.spark.sql.types._ import org.apache.spark.unsafe.types.CalendarInterval @@ -37,43 +36,27 @@ object HiveResult { * Returns the result as a hive compatible sequence of strings. This is used in tests and * `SparkSQLDriver` for CLI applications. */ - def hiveResultString(ds: Dataset[_]): Seq[String] = { -val executedPlan = ds.queryExecution.executedPlan -executedPlan match { - case ExecutedCommandExec(_: DescribeCommandBase) => -// If it is a describe command for a Hive table, we want to have the output format -// be similar with Hive. -executedPlan.executeCollectPublic().map { - case Row(name: String, dataType: String, comment) => -Seq(name, dataType, - Option(comment.asInstanceOf[String]).getOrElse("")) - .map(s => String.format(s"%-20s", s)) - .mkString("\t") -} - // SHOW TABLES in Hive only output table names, - // while ours output database, table name, isTemp. - case command @ ExecutedCommandExec(s: ShowTablesCommand) if !s.isExtended => -command.executeCollect().map(_.getString(1)) - case _ => -val sessionWithJava8DatetimeEnabled = { - val cloned = ds.sparkSession.cloneSession() - cloned.conf.set(SQLConf.DATETIME_JAVA8API_ENABLED.key, true) - cloned -} -sessionWithJava8DatetimeEnabled.withActive { - // We cannot collect the original dataset because its encoders could be created - // with disabled Java 8 date-time API. - val result: Seq[Seq[Any]] = Dataset.ofRows(ds.sparkSession, ds.logicalPlan) -.queryExecution -.executedPlan -.executeCollectPublic().map(_.toSeq).toSeq - // We need the types so we can output struct field names - val types = executedPlan.output.map(_.dataType) - // Reformat to match hive tab delimited output. - result.map(_.zip(types).map(e => toHiveString(e))) -.map(_.mkString("\t")) -} -} + def hiveResultString(executedPlan: SparkPlan): Seq[