[spark] branch master updated (37c806a -> db420f7)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 37c806a [SPARK-32958][SQL] Prune unnecessary columns from JsonToStructs add db420f7 [SPARK-33049][CORE] Decommission shuffle block test is flaky No new revisions were added by this update. Summary of changes: .../spark/storage/BlockManagerDecommissionIntegrationSuite.scala | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (37c806a -> db420f7)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 37c806a [SPARK-32958][SQL] Prune unnecessary columns from JsonToStructs add db420f7 [SPARK-33049][CORE] Decommission shuffle block test is flaky No new revisions were added by this update. Summary of changes: .../spark/storage/BlockManagerDecommissionIntegrationSuite.scala | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (37c806a -> db420f7)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 37c806a [SPARK-32958][SQL] Prune unnecessary columns from JsonToStructs add db420f7 [SPARK-33049][CORE] Decommission shuffle block test is flaky No new revisions were added by this update. Summary of changes: .../spark/storage/BlockManagerDecommissionIntegrationSuite.scala | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (37c806a -> db420f7)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 37c806a [SPARK-32958][SQL] Prune unnecessary columns from JsonToStructs add db420f7 [SPARK-33049][CORE] Decommission shuffle block test is flaky No new revisions were added by this update. Summary of changes: .../spark/storage/BlockManagerDecommissionIntegrationSuite.scala | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (37c806a -> db420f7)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 37c806a [SPARK-32958][SQL] Prune unnecessary columns from JsonToStructs add db420f7 [SPARK-33049][CORE] Decommission shuffle block test is flaky No new revisions were added by this update. Summary of changes: .../spark/storage/BlockManagerDecommissionIntegrationSuite.scala | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (9b21fdd -> 37c806a)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 9b21fdd [SPARK-32949][FOLLOW-UP][R][SQL] Reindent lines in SparkR timestamp_seconds add 37c806a [SPARK-32958][SQL] Prune unnecessary columns from JsonToStructs No new revisions were added by this update. Summary of changes: .../sql/catalyst/optimizer/OptimizeJsonExprs.scala | 16 ++ .../optimizer/OptimizeJsonExprsSuite.scala | 58 ++ 2 files changed, 74 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (9b21fdd -> 37c806a)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 9b21fdd [SPARK-32949][FOLLOW-UP][R][SQL] Reindent lines in SparkR timestamp_seconds add 37c806a [SPARK-32958][SQL] Prune unnecessary columns from JsonToStructs No new revisions were added by this update. Summary of changes: .../sql/catalyst/optimizer/OptimizeJsonExprs.scala | 16 ++ .../optimizer/OptimizeJsonExprsSuite.scala | 58 ++ 2 files changed, 74 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (9b21fdd -> 37c806a)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 9b21fdd [SPARK-32949][FOLLOW-UP][R][SQL] Reindent lines in SparkR timestamp_seconds add 37c806a [SPARK-32958][SQL] Prune unnecessary columns from JsonToStructs No new revisions were added by this update. Summary of changes: .../sql/catalyst/optimizer/OptimizeJsonExprs.scala | 16 ++ .../optimizer/OptimizeJsonExprsSuite.scala | 58 ++ 2 files changed, 74 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (9b21fdd -> 37c806a)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 9b21fdd [SPARK-32949][FOLLOW-UP][R][SQL] Reindent lines in SparkR timestamp_seconds add 37c806a [SPARK-32958][SQL] Prune unnecessary columns from JsonToStructs No new revisions were added by this update. Summary of changes: .../sql/catalyst/optimizer/OptimizeJsonExprs.scala | 16 ++ .../optimizer/OptimizeJsonExprsSuite.scala | 58 ++ 2 files changed, 74 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (9b21fdd -> 37c806a)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 9b21fdd [SPARK-32949][FOLLOW-UP][R][SQL] Reindent lines in SparkR timestamp_seconds add 37c806a [SPARK-32958][SQL] Prune unnecessary columns from JsonToStructs No new revisions were added by this update. Summary of changes: .../sql/catalyst/optimizer/OptimizeJsonExprs.scala | 16 ++ .../optimizer/OptimizeJsonExprsSuite.scala | 58 ++ 2 files changed, 74 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f86171a -> 9b21fdd)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f86171a [SPARK-33043][ML] Handle spark.driver.maxResultSize=0 in RowMatrix heuristic computation add 9b21fdd [SPARK-32949][FOLLOW-UP][R][SQL] Reindent lines in SparkR timestamp_seconds No new revisions were added by this update. Summary of changes: R/pkg/R/functions.R | 14 +++--- 1 file changed, 7 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f86171a -> 9b21fdd)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f86171a [SPARK-33043][ML] Handle spark.driver.maxResultSize=0 in RowMatrix heuristic computation add 9b21fdd [SPARK-32949][FOLLOW-UP][R][SQL] Reindent lines in SparkR timestamp_seconds No new revisions were added by this update. Summary of changes: R/pkg/R/functions.R | 14 +++--- 1 file changed, 7 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f86171a -> 9b21fdd)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f86171a [SPARK-33043][ML] Handle spark.driver.maxResultSize=0 in RowMatrix heuristic computation add 9b21fdd [SPARK-32949][FOLLOW-UP][R][SQL] Reindent lines in SparkR timestamp_seconds No new revisions were added by this update. Summary of changes: R/pkg/R/functions.R | 14 +++--- 1 file changed, 7 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f86171a -> 9b21fdd)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f86171a [SPARK-33043][ML] Handle spark.driver.maxResultSize=0 in RowMatrix heuristic computation add 9b21fdd [SPARK-32949][FOLLOW-UP][R][SQL] Reindent lines in SparkR timestamp_seconds No new revisions were added by this update. Summary of changes: R/pkg/R/functions.R | 14 +++--- 1 file changed, 7 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f86171a -> 9b21fdd)
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f86171a [SPARK-33043][ML] Handle spark.driver.maxResultSize=0 in RowMatrix heuristic computation add 9b21fdd [SPARK-32949][FOLLOW-UP][R][SQL] Reindent lines in SparkR timestamp_seconds No new revisions were added by this update. Summary of changes: R/pkg/R/functions.R | 14 +++--- 1 file changed, 7 insertions(+), 7 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] megelon commented on pull request #292: :rocket: Including ApacheSparkBogotá Meetup on community page :rocket:
megelon commented on pull request #292: URL: https://github.com/apache/spark-website/pull/292#issuecomment-703147391 Thank you for your help!!! This means a lot for me and for the Bogotá community! :rocket: Your work rock's (seriously)!!! :guitar: This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33043][ML] Handle spark.driver.maxResultSize=0 in RowMatrix heuristic computation
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new c9b6271 [SPARK-33043][ML] Handle spark.driver.maxResultSize=0 in
RowMatrix heuristic computation
c9b6271 is described below
commit c9b62711fdec24160c4bdeff8fc09eedb0b75ee0
Author: Sean Owen
AuthorDate: Sat Oct 3 13:12:55 2020 -0500
[SPARK-33043][ML] Handle spark.driver.maxResultSize=0 in RowMatrix
heuristic computation
### What changes were proposed in this pull request?
RowMatrix contains a computation based on spark.driver.maxResultSize.
However, when this value is set to 0, the computation fails (log of 0). The fix
is simply to correctly handle this setting, which means unlimited result size,
by using a tree depth of 1 in the RowMatrix method.
### Why are the changes needed?
Simple bug fix to make several Spark ML functions which use RowMatrix run
correctly in this case.
### Does this PR introduce _any_ user-facing change?
Not other than the bug fix of course.
### How was this patch tested?
Existing RowMatrix tests plus a new test.
Closes #29925 from srowen/SPARK-33043.
Authored-by: Sean Owen
Signed-off-by: Sean Owen
(cherry picked from commit f86171aea43479f54ac2bbbca8f128baa3fc4a8c)
Signed-off-by: Sean Owen
---
.../apache/spark/mllib/linalg/distributed/RowMatrix.scala | 6 +-
.../spark/mllib/linalg/distributed/RowMatrixSuite.scala | 15 +++
2 files changed, 20 insertions(+), 1 deletion(-)
diff --git
a/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
b/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
index 20e26ce..07b9d91 100644
---
a/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
+++
b/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
@@ -786,11 +786,15 @@ class RowMatrix @Since("1.0.0") (
* Based on the formulae: (numPartitions)^(1/depth) * objectSize <=
DriverMaxResultSize
* @param aggregatedObjectSizeInBytes the size, in megabytes, of the object
being tree aggregated
*/
- private[spark] def getTreeAggregateIdealDepth(aggregatedObjectSizeInBytes:
Long) = {
+ private[spark] def getTreeAggregateIdealDepth(aggregatedObjectSizeInBytes:
Long): Int = {
require(aggregatedObjectSizeInBytes > 0,
"Cannot compute aggregate depth heuristic based on a zero-size object to
aggregate")
val maxDriverResultSizeInBytes = rows.conf.get[Long](MAX_RESULT_SIZE)
+if (maxDriverResultSizeInBytes <= 0) {
+ // Unlimited result size, so 1 is OK
+ return 1
+}
require(maxDriverResultSizeInBytes > aggregatedObjectSizeInBytes,
s"Cannot aggregate object of size $aggregatedObjectSizeInBytes Bytes, "
diff --git
a/mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/RowMatrixSuite.scala
b/mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/RowMatrixSuite.scala
index 0a4b119..adc4eee 100644
---
a/mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/RowMatrixSuite.scala
+++
b/mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/RowMatrixSuite.scala
@@ -25,6 +25,7 @@ import breeze.linalg.{norm => brzNorm, svd => brzSvd,
DenseMatrix => BDM, DenseV
import breeze.numerics.abs
import org.apache.spark.SparkFunSuite
+import org.apache.spark.internal.config.MAX_RESULT_SIZE
import org.apache.spark.mllib.linalg.{Matrices, Vector, Vectors}
import org.apache.spark.mllib.random.RandomRDDs
import org.apache.spark.mllib.util.{LocalClusterSparkContext,
MLlibTestSparkContext}
@@ -121,6 +122,20 @@ class RowMatrixSuite extends SparkFunSuite with
MLlibTestSparkContext {
assert(objectBiggerThanResultSize.getMessage.contains("it's bigger than
maxResultSize"))
}
+ test("SPARK-33043: getTreeAggregateIdealDepth with unlimited driver size") {
+val originalMaxResultSize = sc.conf.get[Long](MAX_RESULT_SIZE)
+sc.conf.set(MAX_RESULT_SIZE, 0L)
+try {
+ val nbPartitions = 100
+ val vectors = sc.emptyRDD[Vector]
+.repartition(nbPartitions)
+ val rowMat = new RowMatrix(vectors)
+ assert(rowMat.getTreeAggregateIdealDepth(700 * 1024 * 1024) === 1)
+} finally {
+ sc.conf.set(MAX_RESULT_SIZE, originalMaxResultSize)
+}
+ }
+
test("similar columns") {
val colMags = Vectors.dense(math.sqrt(126), math.sqrt(66), math.sqrt(94))
val expected = BDM(
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33043][ML] Handle spark.driver.maxResultSize=0 in RowMatrix heuristic computation
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new c9b6271 [SPARK-33043][ML] Handle spark.driver.maxResultSize=0 in
RowMatrix heuristic computation
c9b6271 is described below
commit c9b62711fdec24160c4bdeff8fc09eedb0b75ee0
Author: Sean Owen
AuthorDate: Sat Oct 3 13:12:55 2020 -0500
[SPARK-33043][ML] Handle spark.driver.maxResultSize=0 in RowMatrix
heuristic computation
### What changes were proposed in this pull request?
RowMatrix contains a computation based on spark.driver.maxResultSize.
However, when this value is set to 0, the computation fails (log of 0). The fix
is simply to correctly handle this setting, which means unlimited result size,
by using a tree depth of 1 in the RowMatrix method.
### Why are the changes needed?
Simple bug fix to make several Spark ML functions which use RowMatrix run
correctly in this case.
### Does this PR introduce _any_ user-facing change?
Not other than the bug fix of course.
### How was this patch tested?
Existing RowMatrix tests plus a new test.
Closes #29925 from srowen/SPARK-33043.
Authored-by: Sean Owen
Signed-off-by: Sean Owen
(cherry picked from commit f86171aea43479f54ac2bbbca8f128baa3fc4a8c)
Signed-off-by: Sean Owen
---
.../apache/spark/mllib/linalg/distributed/RowMatrix.scala | 6 +-
.../spark/mllib/linalg/distributed/RowMatrixSuite.scala | 15 +++
2 files changed, 20 insertions(+), 1 deletion(-)
diff --git
a/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
b/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
index 20e26ce..07b9d91 100644
---
a/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
+++
b/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
@@ -786,11 +786,15 @@ class RowMatrix @Since("1.0.0") (
* Based on the formulae: (numPartitions)^(1/depth) * objectSize <=
DriverMaxResultSize
* @param aggregatedObjectSizeInBytes the size, in megabytes, of the object
being tree aggregated
*/
- private[spark] def getTreeAggregateIdealDepth(aggregatedObjectSizeInBytes:
Long) = {
+ private[spark] def getTreeAggregateIdealDepth(aggregatedObjectSizeInBytes:
Long): Int = {
require(aggregatedObjectSizeInBytes > 0,
"Cannot compute aggregate depth heuristic based on a zero-size object to
aggregate")
val maxDriverResultSizeInBytes = rows.conf.get[Long](MAX_RESULT_SIZE)
+if (maxDriverResultSizeInBytes <= 0) {
+ // Unlimited result size, so 1 is OK
+ return 1
+}
require(maxDriverResultSizeInBytes > aggregatedObjectSizeInBytes,
s"Cannot aggregate object of size $aggregatedObjectSizeInBytes Bytes, "
diff --git
a/mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/RowMatrixSuite.scala
b/mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/RowMatrixSuite.scala
index 0a4b119..adc4eee 100644
---
a/mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/RowMatrixSuite.scala
+++
b/mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/RowMatrixSuite.scala
@@ -25,6 +25,7 @@ import breeze.linalg.{norm => brzNorm, svd => brzSvd,
DenseMatrix => BDM, DenseV
import breeze.numerics.abs
import org.apache.spark.SparkFunSuite
+import org.apache.spark.internal.config.MAX_RESULT_SIZE
import org.apache.spark.mllib.linalg.{Matrices, Vector, Vectors}
import org.apache.spark.mllib.random.RandomRDDs
import org.apache.spark.mllib.util.{LocalClusterSparkContext,
MLlibTestSparkContext}
@@ -121,6 +122,20 @@ class RowMatrixSuite extends SparkFunSuite with
MLlibTestSparkContext {
assert(objectBiggerThanResultSize.getMessage.contains("it's bigger than
maxResultSize"))
}
+ test("SPARK-33043: getTreeAggregateIdealDepth with unlimited driver size") {
+val originalMaxResultSize = sc.conf.get[Long](MAX_RESULT_SIZE)
+sc.conf.set(MAX_RESULT_SIZE, 0L)
+try {
+ val nbPartitions = 100
+ val vectors = sc.emptyRDD[Vector]
+.repartition(nbPartitions)
+ val rowMat = new RowMatrix(vectors)
+ assert(rowMat.getTreeAggregateIdealDepth(700 * 1024 * 1024) === 1)
+} finally {
+ sc.conf.set(MAX_RESULT_SIZE, originalMaxResultSize)
+}
+ }
+
test("similar columns") {
val colMags = Vectors.dense(math.sqrt(126), math.sqrt(66), math.sqrt(94))
val expected = BDM(
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (5af62a2 -> f86171a)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 5af62a2 [SPARK-33052][SQL][TEST] Make all the database versions up-to-date for integration tests add f86171a [SPARK-33043][ML] Handle spark.driver.maxResultSize=0 in RowMatrix heuristic computation No new revisions were added by this update. Summary of changes: .../apache/spark/mllib/linalg/distributed/RowMatrix.scala | 6 +- .../spark/mllib/linalg/distributed/RowMatrixSuite.scala | 15 +++ 2 files changed, 20 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33043][ML] Handle spark.driver.maxResultSize=0 in RowMatrix heuristic computation
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new c9b6271 [SPARK-33043][ML] Handle spark.driver.maxResultSize=0 in
RowMatrix heuristic computation
c9b6271 is described below
commit c9b62711fdec24160c4bdeff8fc09eedb0b75ee0
Author: Sean Owen
AuthorDate: Sat Oct 3 13:12:55 2020 -0500
[SPARK-33043][ML] Handle spark.driver.maxResultSize=0 in RowMatrix
heuristic computation
### What changes were proposed in this pull request?
RowMatrix contains a computation based on spark.driver.maxResultSize.
However, when this value is set to 0, the computation fails (log of 0). The fix
is simply to correctly handle this setting, which means unlimited result size,
by using a tree depth of 1 in the RowMatrix method.
### Why are the changes needed?
Simple bug fix to make several Spark ML functions which use RowMatrix run
correctly in this case.
### Does this PR introduce _any_ user-facing change?
Not other than the bug fix of course.
### How was this patch tested?
Existing RowMatrix tests plus a new test.
Closes #29925 from srowen/SPARK-33043.
Authored-by: Sean Owen
Signed-off-by: Sean Owen
(cherry picked from commit f86171aea43479f54ac2bbbca8f128baa3fc4a8c)
Signed-off-by: Sean Owen
---
.../apache/spark/mllib/linalg/distributed/RowMatrix.scala | 6 +-
.../spark/mllib/linalg/distributed/RowMatrixSuite.scala | 15 +++
2 files changed, 20 insertions(+), 1 deletion(-)
diff --git
a/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
b/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
index 20e26ce..07b9d91 100644
---
a/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
+++
b/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
@@ -786,11 +786,15 @@ class RowMatrix @Since("1.0.0") (
* Based on the formulae: (numPartitions)^(1/depth) * objectSize <=
DriverMaxResultSize
* @param aggregatedObjectSizeInBytes the size, in megabytes, of the object
being tree aggregated
*/
- private[spark] def getTreeAggregateIdealDepth(aggregatedObjectSizeInBytes:
Long) = {
+ private[spark] def getTreeAggregateIdealDepth(aggregatedObjectSizeInBytes:
Long): Int = {
require(aggregatedObjectSizeInBytes > 0,
"Cannot compute aggregate depth heuristic based on a zero-size object to
aggregate")
val maxDriverResultSizeInBytes = rows.conf.get[Long](MAX_RESULT_SIZE)
+if (maxDriverResultSizeInBytes <= 0) {
+ // Unlimited result size, so 1 is OK
+ return 1
+}
require(maxDriverResultSizeInBytes > aggregatedObjectSizeInBytes,
s"Cannot aggregate object of size $aggregatedObjectSizeInBytes Bytes, "
diff --git
a/mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/RowMatrixSuite.scala
b/mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/RowMatrixSuite.scala
index 0a4b119..adc4eee 100644
---
a/mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/RowMatrixSuite.scala
+++
b/mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/RowMatrixSuite.scala
@@ -25,6 +25,7 @@ import breeze.linalg.{norm => brzNorm, svd => brzSvd,
DenseMatrix => BDM, DenseV
import breeze.numerics.abs
import org.apache.spark.SparkFunSuite
+import org.apache.spark.internal.config.MAX_RESULT_SIZE
import org.apache.spark.mllib.linalg.{Matrices, Vector, Vectors}
import org.apache.spark.mllib.random.RandomRDDs
import org.apache.spark.mllib.util.{LocalClusterSparkContext,
MLlibTestSparkContext}
@@ -121,6 +122,20 @@ class RowMatrixSuite extends SparkFunSuite with
MLlibTestSparkContext {
assert(objectBiggerThanResultSize.getMessage.contains("it's bigger than
maxResultSize"))
}
+ test("SPARK-33043: getTreeAggregateIdealDepth with unlimited driver size") {
+val originalMaxResultSize = sc.conf.get[Long](MAX_RESULT_SIZE)
+sc.conf.set(MAX_RESULT_SIZE, 0L)
+try {
+ val nbPartitions = 100
+ val vectors = sc.emptyRDD[Vector]
+.repartition(nbPartitions)
+ val rowMat = new RowMatrix(vectors)
+ assert(rowMat.getTreeAggregateIdealDepth(700 * 1024 * 1024) === 1)
+} finally {
+ sc.conf.set(MAX_RESULT_SIZE, originalMaxResultSize)
+}
+ }
+
test("similar columns") {
val colMags = Vectors.dense(math.sqrt(126), math.sqrt(66), math.sqrt(94))
val expected = BDM(
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (5af62a2 -> f86171a)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 5af62a2 [SPARK-33052][SQL][TEST] Make all the database versions up-to-date for integration tests add f86171a [SPARK-33043][ML] Handle spark.driver.maxResultSize=0 in RowMatrix heuristic computation No new revisions were added by this update. Summary of changes: .../apache/spark/mllib/linalg/distributed/RowMatrix.scala | 6 +- .../spark/mllib/linalg/distributed/RowMatrixSuite.scala | 15 +++ 2 files changed, 20 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33043][ML] Handle spark.driver.maxResultSize=0 in RowMatrix heuristic computation
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new c9b6271 [SPARK-33043][ML] Handle spark.driver.maxResultSize=0 in
RowMatrix heuristic computation
c9b6271 is described below
commit c9b62711fdec24160c4bdeff8fc09eedb0b75ee0
Author: Sean Owen
AuthorDate: Sat Oct 3 13:12:55 2020 -0500
[SPARK-33043][ML] Handle spark.driver.maxResultSize=0 in RowMatrix
heuristic computation
### What changes were proposed in this pull request?
RowMatrix contains a computation based on spark.driver.maxResultSize.
However, when this value is set to 0, the computation fails (log of 0). The fix
is simply to correctly handle this setting, which means unlimited result size,
by using a tree depth of 1 in the RowMatrix method.
### Why are the changes needed?
Simple bug fix to make several Spark ML functions which use RowMatrix run
correctly in this case.
### Does this PR introduce _any_ user-facing change?
Not other than the bug fix of course.
### How was this patch tested?
Existing RowMatrix tests plus a new test.
Closes #29925 from srowen/SPARK-33043.
Authored-by: Sean Owen
Signed-off-by: Sean Owen
(cherry picked from commit f86171aea43479f54ac2bbbca8f128baa3fc4a8c)
Signed-off-by: Sean Owen
---
.../apache/spark/mllib/linalg/distributed/RowMatrix.scala | 6 +-
.../spark/mllib/linalg/distributed/RowMatrixSuite.scala | 15 +++
2 files changed, 20 insertions(+), 1 deletion(-)
diff --git
a/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
b/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
index 20e26ce..07b9d91 100644
---
a/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
+++
b/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
@@ -786,11 +786,15 @@ class RowMatrix @Since("1.0.0") (
* Based on the formulae: (numPartitions)^(1/depth) * objectSize <=
DriverMaxResultSize
* @param aggregatedObjectSizeInBytes the size, in megabytes, of the object
being tree aggregated
*/
- private[spark] def getTreeAggregateIdealDepth(aggregatedObjectSizeInBytes:
Long) = {
+ private[spark] def getTreeAggregateIdealDepth(aggregatedObjectSizeInBytes:
Long): Int = {
require(aggregatedObjectSizeInBytes > 0,
"Cannot compute aggregate depth heuristic based on a zero-size object to
aggregate")
val maxDriverResultSizeInBytes = rows.conf.get[Long](MAX_RESULT_SIZE)
+if (maxDriverResultSizeInBytes <= 0) {
+ // Unlimited result size, so 1 is OK
+ return 1
+}
require(maxDriverResultSizeInBytes > aggregatedObjectSizeInBytes,
s"Cannot aggregate object of size $aggregatedObjectSizeInBytes Bytes, "
diff --git
a/mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/RowMatrixSuite.scala
b/mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/RowMatrixSuite.scala
index 0a4b119..adc4eee 100644
---
a/mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/RowMatrixSuite.scala
+++
b/mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/RowMatrixSuite.scala
@@ -25,6 +25,7 @@ import breeze.linalg.{norm => brzNorm, svd => brzSvd,
DenseMatrix => BDM, DenseV
import breeze.numerics.abs
import org.apache.spark.SparkFunSuite
+import org.apache.spark.internal.config.MAX_RESULT_SIZE
import org.apache.spark.mllib.linalg.{Matrices, Vector, Vectors}
import org.apache.spark.mllib.random.RandomRDDs
import org.apache.spark.mllib.util.{LocalClusterSparkContext,
MLlibTestSparkContext}
@@ -121,6 +122,20 @@ class RowMatrixSuite extends SparkFunSuite with
MLlibTestSparkContext {
assert(objectBiggerThanResultSize.getMessage.contains("it's bigger than
maxResultSize"))
}
+ test("SPARK-33043: getTreeAggregateIdealDepth with unlimited driver size") {
+val originalMaxResultSize = sc.conf.get[Long](MAX_RESULT_SIZE)
+sc.conf.set(MAX_RESULT_SIZE, 0L)
+try {
+ val nbPartitions = 100
+ val vectors = sc.emptyRDD[Vector]
+.repartition(nbPartitions)
+ val rowMat = new RowMatrix(vectors)
+ assert(rowMat.getTreeAggregateIdealDepth(700 * 1024 * 1024) === 1)
+} finally {
+ sc.conf.set(MAX_RESULT_SIZE, originalMaxResultSize)
+}
+ }
+
test("similar columns") {
val colMags = Vectors.dense(math.sqrt(126), math.sqrt(66), math.sqrt(94))
val expected = BDM(
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (5af62a2 -> f86171a)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 5af62a2 [SPARK-33052][SQL][TEST] Make all the database versions up-to-date for integration tests add f86171a [SPARK-33043][ML] Handle spark.driver.maxResultSize=0 in RowMatrix heuristic computation No new revisions were added by this update. Summary of changes: .../apache/spark/mllib/linalg/distributed/RowMatrix.scala | 6 +- .../spark/mllib/linalg/distributed/RowMatrixSuite.scala | 15 +++ 2 files changed, 20 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-3.0 updated: [SPARK-33043][ML] Handle spark.driver.maxResultSize=0 in RowMatrix heuristic computation
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-3.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.0 by this push:
new c9b6271 [SPARK-33043][ML] Handle spark.driver.maxResultSize=0 in
RowMatrix heuristic computation
c9b6271 is described below
commit c9b62711fdec24160c4bdeff8fc09eedb0b75ee0
Author: Sean Owen
AuthorDate: Sat Oct 3 13:12:55 2020 -0500
[SPARK-33043][ML] Handle spark.driver.maxResultSize=0 in RowMatrix
heuristic computation
### What changes were proposed in this pull request?
RowMatrix contains a computation based on spark.driver.maxResultSize.
However, when this value is set to 0, the computation fails (log of 0). The fix
is simply to correctly handle this setting, which means unlimited result size,
by using a tree depth of 1 in the RowMatrix method.
### Why are the changes needed?
Simple bug fix to make several Spark ML functions which use RowMatrix run
correctly in this case.
### Does this PR introduce _any_ user-facing change?
Not other than the bug fix of course.
### How was this patch tested?
Existing RowMatrix tests plus a new test.
Closes #29925 from srowen/SPARK-33043.
Authored-by: Sean Owen
Signed-off-by: Sean Owen
(cherry picked from commit f86171aea43479f54ac2bbbca8f128baa3fc4a8c)
Signed-off-by: Sean Owen
---
.../apache/spark/mllib/linalg/distributed/RowMatrix.scala | 6 +-
.../spark/mllib/linalg/distributed/RowMatrixSuite.scala | 15 +++
2 files changed, 20 insertions(+), 1 deletion(-)
diff --git
a/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
b/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
index 20e26ce..07b9d91 100644
---
a/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
+++
b/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala
@@ -786,11 +786,15 @@ class RowMatrix @Since("1.0.0") (
* Based on the formulae: (numPartitions)^(1/depth) * objectSize <=
DriverMaxResultSize
* @param aggregatedObjectSizeInBytes the size, in megabytes, of the object
being tree aggregated
*/
- private[spark] def getTreeAggregateIdealDepth(aggregatedObjectSizeInBytes:
Long) = {
+ private[spark] def getTreeAggregateIdealDepth(aggregatedObjectSizeInBytes:
Long): Int = {
require(aggregatedObjectSizeInBytes > 0,
"Cannot compute aggregate depth heuristic based on a zero-size object to
aggregate")
val maxDriverResultSizeInBytes = rows.conf.get[Long](MAX_RESULT_SIZE)
+if (maxDriverResultSizeInBytes <= 0) {
+ // Unlimited result size, so 1 is OK
+ return 1
+}
require(maxDriverResultSizeInBytes > aggregatedObjectSizeInBytes,
s"Cannot aggregate object of size $aggregatedObjectSizeInBytes Bytes, "
diff --git
a/mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/RowMatrixSuite.scala
b/mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/RowMatrixSuite.scala
index 0a4b119..adc4eee 100644
---
a/mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/RowMatrixSuite.scala
+++
b/mllib/src/test/scala/org/apache/spark/mllib/linalg/distributed/RowMatrixSuite.scala
@@ -25,6 +25,7 @@ import breeze.linalg.{norm => brzNorm, svd => brzSvd,
DenseMatrix => BDM, DenseV
import breeze.numerics.abs
import org.apache.spark.SparkFunSuite
+import org.apache.spark.internal.config.MAX_RESULT_SIZE
import org.apache.spark.mllib.linalg.{Matrices, Vector, Vectors}
import org.apache.spark.mllib.random.RandomRDDs
import org.apache.spark.mllib.util.{LocalClusterSparkContext,
MLlibTestSparkContext}
@@ -121,6 +122,20 @@ class RowMatrixSuite extends SparkFunSuite with
MLlibTestSparkContext {
assert(objectBiggerThanResultSize.getMessage.contains("it's bigger than
maxResultSize"))
}
+ test("SPARK-33043: getTreeAggregateIdealDepth with unlimited driver size") {
+val originalMaxResultSize = sc.conf.get[Long](MAX_RESULT_SIZE)
+sc.conf.set(MAX_RESULT_SIZE, 0L)
+try {
+ val nbPartitions = 100
+ val vectors = sc.emptyRDD[Vector]
+.repartition(nbPartitions)
+ val rowMat = new RowMatrix(vectors)
+ assert(rowMat.getTreeAggregateIdealDepth(700 * 1024 * 1024) === 1)
+} finally {
+ sc.conf.set(MAX_RESULT_SIZE, originalMaxResultSize)
+}
+ }
+
test("similar columns") {
val colMags = Vectors.dense(math.sqrt(126), math.sqrt(66), math.sqrt(94))
val expected = BDM(
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (5af62a2 -> f86171a)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 5af62a2 [SPARK-33052][SQL][TEST] Make all the database versions up-to-date for integration tests add f86171a [SPARK-33043][ML] Handle spark.driver.maxResultSize=0 in RowMatrix heuristic computation No new revisions were added by this update. Summary of changes: .../apache/spark/mllib/linalg/distributed/RowMatrix.scala | 6 +- .../spark/mllib/linalg/distributed/RowMatrixSuite.scala | 15 +++ 2 files changed, 20 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (5af62a2 -> f86171a)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 5af62a2 [SPARK-33052][SQL][TEST] Make all the database versions up-to-date for integration tests add f86171a [SPARK-33043][ML] Handle spark.driver.maxResultSize=0 in RowMatrix heuristic computation No new revisions were added by this update. Summary of changes: .../apache/spark/mllib/linalg/distributed/RowMatrix.scala | 6 +- .../spark/mllib/linalg/distributed/RowMatrixSuite.scala | 15 +++ 2 files changed, 20 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark-website] branch asf-site updated: :rocket: Including ApacheSparkBogotá Meetup on community page :rocket:
This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/spark-website.git The following commit(s) were added to refs/heads/asf-site by this push: new 757cc46 :rocket: Including ApacheSparkBogotá Meetup on community page :rocket: 757cc46 is described below commit 757cc46c85d5b4ad072fe25c32c3dbadc300e3da Author: miguel diaz AuthorDate: Sat Oct 3 10:16:01 2020 -0500 :rocket: Including ApacheSparkBogotá Meetup on community page :rocket: Hello, I am trying again. :sweat_smile: I am Co-organizer of Apache Spark Bogotá Meetup from Colombia https://www.meetup.com/es/Apache-Spark-Bogota/ And would like to include the community on the following web page https://spark.apache.org/community.html This time I didn't use jekill because as you see new version update a lot of things, please let me know if now it is good to go. I change the .md and the .html community files. Author: miguel diaz Closes #292 from megelon/asbog. --- community.md| 5 - site/community.html | 3 +++ 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/community.md b/community.md index dca08c0..e8f2cf7 100644 --- a/community.md +++ b/community.md @@ -139,9 +139,12 @@ Spark Meetups are grass-roots events organized and hosted by individuals in the https://www.meetup.com/SanKir-Big-Data-Group/;>Bangalore Spark Meetup -https://www.meetup.com/Boston-Apache-Spark-User-Group/;>Boston Spark Meetup +https://www.meetup.com/es/Apache-Spark-Bogota/;>Bogotá Spark Meetup +https://www.meetup.com/Boston-Apache-Spark-User-Group/;>Boston Spark Meetup + + https://www.meetup.com/Boulder-Denver-Spark-Meetup/;>Boulder/Denver Spark Meetup diff --git a/site/community.html b/site/community.html index 337dc8a..f129ac2 100644 --- a/site/community.html +++ b/site/community.html @@ -345,6 +345,9 @@ vulnerabilities, and for information on known security issues. https://www.meetup.com/SanKir-Big-Data-Group/;>Bangalore Spark Meetup +https://www.meetup.com/es/Apache-Spark-Bogota/;>Bogotá Spark Meetup + + https://www.meetup.com/Boston-Apache-Spark-User-Group/;>Boston Spark Meetup - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] srowen closed pull request #292: :rocket: Including ApacheSparkBogotá Meetup on community page :rocket:
srowen closed pull request #292: URL: https://github.com/apache/spark-website/pull/292 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] srowen commented on pull request #292: :rocket: Including ApacheSparkBogotá Meetup on community page :rocket:
srowen commented on pull request #292: URL: https://github.com/apache/spark-website/pull/292#issuecomment-703118961 Merged to asf-site This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] megelon opened a new pull request #292: :rocket: Including ApacheSparkBogotá Meetup on community page :rocket:
megelon opened a new pull request #292: URL: https://github.com/apache/spark-website/pull/292 Hello, I am trying again. :sweat_smile: I am Co-organizer of Apache Spark Bogotá Meetup from Colombia https://www.meetup.com/es/Apache-Spark-Bogota/ And would like to include the community on the following web page https://spark.apache.org/community.html This time I didn't use jekill because as you see new version update a lot of things, please let me know if now it is good to go. I change the .md and the .html community files. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (1299c8a -> 5af62a2)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1299c8a [SPARK-33037][SHUFFLE] Remove knownManagers to support user's custom shuffle manager plugin add 5af62a2 [SPARK-33052][SQL][TEST] Make all the database versions up-to-date for integration tests No new revisions were added by this update. Summary of changes: .../src/test/resources/mariadb_docker_entrypoint.sh | 2 +- .../scala/org/apache/spark/sql/jdbc/DB2IntegrationSuite.scala | 9 - .../org/apache/spark/sql/jdbc/DB2KrbIntegrationSuite.scala| 9 - .../apache/spark/sql/jdbc/MariaDBKrbIntegrationSuite.scala| 4 +++- .../apache/spark/sql/jdbc/MsSqlServerIntegrationSuite.scala | 10 +- .../org/apache/spark/sql/jdbc/MySQLIntegrationSuite.scala | 11 +-- .../org/apache/spark/sql/jdbc/PostgresIntegrationSuite.scala | 9 - .../apache/spark/sql/jdbc/PostgresKrbIntegrationSuite.scala | 9 - 8 files changed, 54 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (1299c8a -> 5af62a2)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1299c8a [SPARK-33037][SHUFFLE] Remove knownManagers to support user's custom shuffle manager plugin add 5af62a2 [SPARK-33052][SQL][TEST] Make all the database versions up-to-date for integration tests No new revisions were added by this update. Summary of changes: .../src/test/resources/mariadb_docker_entrypoint.sh | 2 +- .../scala/org/apache/spark/sql/jdbc/DB2IntegrationSuite.scala | 9 - .../org/apache/spark/sql/jdbc/DB2KrbIntegrationSuite.scala| 9 - .../apache/spark/sql/jdbc/MariaDBKrbIntegrationSuite.scala| 4 +++- .../apache/spark/sql/jdbc/MsSqlServerIntegrationSuite.scala | 10 +- .../org/apache/spark/sql/jdbc/MySQLIntegrationSuite.scala | 11 +-- .../org/apache/spark/sql/jdbc/PostgresIntegrationSuite.scala | 9 - .../apache/spark/sql/jdbc/PostgresKrbIntegrationSuite.scala | 9 - 8 files changed, 54 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (1299c8a -> 5af62a2)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1299c8a [SPARK-33037][SHUFFLE] Remove knownManagers to support user's custom shuffle manager plugin add 5af62a2 [SPARK-33052][SQL][TEST] Make all the database versions up-to-date for integration tests No new revisions were added by this update. Summary of changes: .../src/test/resources/mariadb_docker_entrypoint.sh | 2 +- .../scala/org/apache/spark/sql/jdbc/DB2IntegrationSuite.scala | 9 - .../org/apache/spark/sql/jdbc/DB2KrbIntegrationSuite.scala| 9 - .../apache/spark/sql/jdbc/MariaDBKrbIntegrationSuite.scala| 4 +++- .../apache/spark/sql/jdbc/MsSqlServerIntegrationSuite.scala | 10 +- .../org/apache/spark/sql/jdbc/MySQLIntegrationSuite.scala | 11 +-- .../org/apache/spark/sql/jdbc/PostgresIntegrationSuite.scala | 9 - .../apache/spark/sql/jdbc/PostgresKrbIntegrationSuite.scala | 9 - 8 files changed, 54 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (1299c8a -> 5af62a2)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1299c8a [SPARK-33037][SHUFFLE] Remove knownManagers to support user's custom shuffle manager plugin add 5af62a2 [SPARK-33052][SQL][TEST] Make all the database versions up-to-date for integration tests No new revisions were added by this update. Summary of changes: .../src/test/resources/mariadb_docker_entrypoint.sh | 2 +- .../scala/org/apache/spark/sql/jdbc/DB2IntegrationSuite.scala | 9 - .../org/apache/spark/sql/jdbc/DB2KrbIntegrationSuite.scala| 9 - .../apache/spark/sql/jdbc/MariaDBKrbIntegrationSuite.scala| 4 +++- .../apache/spark/sql/jdbc/MsSqlServerIntegrationSuite.scala | 10 +- .../org/apache/spark/sql/jdbc/MySQLIntegrationSuite.scala | 11 +-- .../org/apache/spark/sql/jdbc/PostgresIntegrationSuite.scala | 9 - .../apache/spark/sql/jdbc/PostgresKrbIntegrationSuite.scala | 9 - 8 files changed, 54 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (1299c8a -> 5af62a2)
This is an automated email from the ASF dual-hosted git repository. yamamuro pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 1299c8a [SPARK-33037][SHUFFLE] Remove knownManagers to support user's custom shuffle manager plugin add 5af62a2 [SPARK-33052][SQL][TEST] Make all the database versions up-to-date for integration tests No new revisions were added by this update. Summary of changes: .../src/test/resources/mariadb_docker_entrypoint.sh | 2 +- .../scala/org/apache/spark/sql/jdbc/DB2IntegrationSuite.scala | 9 - .../org/apache/spark/sql/jdbc/DB2KrbIntegrationSuite.scala| 9 - .../apache/spark/sql/jdbc/MariaDBKrbIntegrationSuite.scala| 4 +++- .../apache/spark/sql/jdbc/MsSqlServerIntegrationSuite.scala | 10 +- .../org/apache/spark/sql/jdbc/MySQLIntegrationSuite.scala | 11 +-- .../org/apache/spark/sql/jdbc/PostgresIntegrationSuite.scala | 9 - .../apache/spark/sql/jdbc/PostgresKrbIntegrationSuite.scala | 9 - 8 files changed, 54 insertions(+), 9 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] srowen commented on pull request #291: :) bogotá-meetup
srowen commented on pull request #291: URL: https://github.com/apache/spark-website/pull/291#issuecomment-703106778 You would have to push to your branch in your fork, not the main one. Make sure you made a branch against your fork? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] megelon commented on pull request #291: :) bogotá-meetup
megelon commented on pull request #291: URL: https://github.com/apache/spark-website/pull/291#issuecomment-703089369 :hand: I am pushing the new branch but recieved this message :sweat_smile: ERROR: Permission to apache/spark-website.git denied to megelon. Can you help me? Please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] megelon closed pull request #291: :) bogotá-meetup
megelon closed pull request #291: URL: https://github.com/apache/spark-website/pull/291 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[GitHub] [spark-website] megelon commented on pull request #291: :) bogotá-meetup
megelon commented on pull request #291: URL: https://github.com/apache/spark-website/pull/291#issuecomment-703088849 No problem. I am closing this PR and starting a cleaner one. :grimacing: This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
