svn commit: r27962 - in /dev/spark/2.4.0-SNAPSHOT-2018_07_07_00_01-044b33b-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s
Author: pwendell Date: Sat Jul 7 07:16:10 2018 New Revision: 27962 Log: Apache Spark 2.4.0-SNAPSHOT-2018_07_07_00_01-044b33b docs [This commit notification would consist of 1467 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-24757][SQL] Improving the error message for broadcast timeouts
Repository: spark Updated Branches: refs/heads/master 044b33b2e -> 79c668942 [SPARK-24757][SQL] Improving the error message for broadcast timeouts ## What changes were proposed in this pull request? In the PR, I propose to provide a tip to user how to resolve the issue of timeout expiration for broadcast joins. In particular, they can increase the timeout via **spark.sql.broadcastTimeout** or disable the broadcast at all by setting **spark.sql.autoBroadcastJoinThreshold** to `-1`. ## How was this patch tested? It tested manually from `spark-shell`: ``` scala> spark.conf.set("spark.sql.broadcastTimeout", 1) scala> val df = spark.range(100).join(spark.range(15).as[Long].map { x => Thread.sleep(5000) x }).where("id = value") scala> df.count() ``` ``` org.apache.spark.SparkException: Could not execute broadcast in 1 secs. You can increase the timeout for broadcasts via spark.sql.broadcastTimeout or disable broadcast join by setting spark.sql.autoBroadcastJoinThreshold to -1 at org.apache.spark.sql.execution.exchange.BroadcastExchangeExec.doExecuteBroadcast(BroadcastExchangeExec.scala:150) ``` Author: Maxim Gekk Closes #21727 from MaxGekk/broadcast-timeout-error. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/79c66894 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/79c66894 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/79c66894 Branch: refs/heads/master Commit: 79c66894296840cc4a5bf6c8718ecfd2b08bcca8 Parents: 044b33b Author: Maxim Gekk Authored: Sat Jul 7 22:16:48 2018 +0200 Committer: Herman van Hovell Committed: Sat Jul 7 22:16:48 2018 +0200 -- .../sql/execution/exchange/BroadcastExchangeExec.scala | 13 - 1 file changed, 12 insertions(+), 1 deletion(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/79c66894/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala -- diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala index c55f9b8..a80673c 100644 --- a/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala +++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala @@ -17,6 +17,8 @@ package org.apache.spark.sql.execution.exchange +import java.util.concurrent.TimeoutException + import scala.concurrent.{ExecutionContext, Future} import scala.concurrent.duration._ import scala.util.control.NonFatal @@ -140,7 +142,16 @@ case class BroadcastExchangeExec( } override protected[sql] def doExecuteBroadcast[T](): broadcast.Broadcast[T] = { -ThreadUtils.awaitResult(relationFuture, timeout).asInstanceOf[broadcast.Broadcast[T]] +try { + ThreadUtils.awaitResult(relationFuture, timeout).asInstanceOf[broadcast.Broadcast[T]] +} catch { + case ex: TimeoutException => +logError(s"Could not execute broadcast in ${timeout.toSeconds} secs.", ex) +throw new SparkException(s"Could not execute broadcast in ${timeout.toSeconds} secs. " + + s"You can increase the timeout for broadcasts via ${SQLConf.BROADCAST_TIMEOUT.key} or " + + s"disable broadcast join by setting ${SQLConf.AUTO_BROADCASTJOIN_THRESHOLD.key} to -1", + ex) +} } } - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
svn commit: r27979 - in /dev/spark/2.4.0-SNAPSHOT-2018_07_07_16_01-79c6689-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s
Author: pwendell Date: Sat Jul 7 23:15:55 2018 New Revision: 27979 Log: Apache Spark 2.4.0-SNAPSHOT-2018_07_07_16_01-79c6689 docs [This commit notification would consist of 1467 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] Git Push Summary
Repository: spark Updated Tags: refs/tags/v2.3.2-rc1 [created] 4df06b451 - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[2/2] spark git commit: Preparing development version 2.3.3-SNAPSHOT
Preparing development version 2.3.3-SNAPSHOT Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/72eb97ce Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/72eb97ce Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/72eb97ce Branch: refs/heads/branch-2.3 Commit: 72eb97ce96c27e296aa6ed8198246000a78fb10b Parents: 4df06b4 Author: Saisai Shao Authored: Sun Jul 8 01:24:55 2018 + Committer: Saisai Shao Committed: Sun Jul 8 01:24:55 2018 + -- R/pkg/DESCRIPTION | 2 +- assembly/pom.xml | 2 +- common/kvstore/pom.xml| 2 +- common/network-common/pom.xml | 2 +- common/network-shuffle/pom.xml| 2 +- common/network-yarn/pom.xml | 2 +- common/sketch/pom.xml | 2 +- common/tags/pom.xml | 2 +- common/unsafe/pom.xml | 2 +- core/pom.xml | 2 +- docs/_config.yml | 4 ++-- examples/pom.xml | 2 +- external/docker-integration-tests/pom.xml | 2 +- external/flume-assembly/pom.xml | 2 +- external/flume-sink/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka-0-10-assembly/pom.xml | 2 +- external/kafka-0-10-sql/pom.xml | 2 +- external/kafka-0-10/pom.xml | 2 +- external/kafka-0-8-assembly/pom.xml | 2 +- external/kafka-0-8/pom.xml| 2 +- external/kinesis-asl-assembly/pom.xml | 2 +- external/kinesis-asl/pom.xml | 2 +- external/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- hadoop-cloud/pom.xml | 2 +- launcher/pom.xml | 2 +- mllib-local/pom.xml | 2 +- mllib/pom.xml | 2 +- pom.xml | 2 +- python/pyspark/version.py | 2 +- repl/pom.xml | 2 +- resource-managers/kubernetes/core/pom.xml | 2 +- resource-managers/mesos/pom.xml | 2 +- resource-managers/yarn/pom.xml| 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive-thriftserver/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- 41 files changed, 42 insertions(+), 42 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/72eb97ce/R/pkg/DESCRIPTION -- diff --git a/R/pkg/DESCRIPTION b/R/pkg/DESCRIPTION index 8df2635..6ec4966 100644 --- a/R/pkg/DESCRIPTION +++ b/R/pkg/DESCRIPTION @@ -1,6 +1,6 @@ Package: SparkR Type: Package -Version: 2.3.2 +Version: 2.3.3 Title: R Frontend for Apache Spark Description: Provides an R Frontend for Apache Spark. Authors@R: c(person("Shivaram", "Venkataraman", role = c("aut", "cre"), http://git-wip-us.apache.org/repos/asf/spark/blob/72eb97ce/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index 57485fc..f8b15cc 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ org.apache.spark spark-parent_2.11 -2.3.2 +2.3.3-SNAPSHOT ../pom.xml http://git-wip-us.apache.org/repos/asf/spark/blob/72eb97ce/common/kvstore/pom.xml -- diff --git a/common/kvstore/pom.xml b/common/kvstore/pom.xml index 53e58c2..e412a47 100644 --- a/common/kvstore/pom.xml +++ b/common/kvstore/pom.xml @@ -22,7 +22,7 @@ org.apache.spark spark-parent_2.11 -2.3.2 +2.3.3-SNAPSHOT ../../pom.xml http://git-wip-us.apache.org/repos/asf/spark/blob/72eb97ce/common/network-common/pom.xml -- diff --git a/common/network-common/pom.xml b/common/network-common/pom.xml index d05647c..d8f9a3d 100644 --- a/common/network-common/pom.xml +++ b/common/network-common/pom.xml @@ -22,7 +22,7 @@ org.apache.spark spark-parent_2.11 -2.3.2 +2.3.3-SNAPSHOT ../../pom.xml http://git-wip-us.apache.org/repos/asf/spark/blob/72eb97ce/common/network-shuffle/pom.xml -- diff --git a/common/network-shuffle/pom.xml b/common/network-shuffle/pom.xml index 8d46761..a1a4f87 100644 --- a/common/network-shuffle/pom.xml +++ b/common/network-shuffle/pom.xml @@ -22,7 +22,7 @@ org.apache.spark spark-parent_2.11 -2.3.2
[1/2] spark git commit: Preparing Spark release v2.3.2-rc1
Repository: spark Updated Branches: refs/heads/branch-2.3 64c72b4de -> 72eb97ce9 Preparing Spark release v2.3.2-rc1 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/4df06b45 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/4df06b45 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/4df06b45 Branch: refs/heads/branch-2.3 Commit: 4df06b45160241dbb331153efbb25703f913c192 Parents: 64c72b4 Author: Saisai Shao Authored: Sun Jul 8 01:24:42 2018 + Committer: Saisai Shao Committed: Sun Jul 8 01:24:42 2018 + -- assembly/pom.xml | 2 +- common/kvstore/pom.xml| 2 +- common/network-common/pom.xml | 2 +- common/network-shuffle/pom.xml| 2 +- common/network-yarn/pom.xml | 2 +- common/sketch/pom.xml | 2 +- common/tags/pom.xml | 2 +- common/unsafe/pom.xml | 2 +- core/pom.xml | 2 +- docs/_config.yml | 2 +- examples/pom.xml | 2 +- external/docker-integration-tests/pom.xml | 2 +- external/flume-assembly/pom.xml | 2 +- external/flume-sink/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka-0-10-assembly/pom.xml | 2 +- external/kafka-0-10-sql/pom.xml | 2 +- external/kafka-0-10/pom.xml | 2 +- external/kafka-0-8-assembly/pom.xml | 2 +- external/kafka-0-8/pom.xml| 2 +- external/kinesis-asl-assembly/pom.xml | 2 +- external/kinesis-asl/pom.xml | 2 +- external/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- hadoop-cloud/pom.xml | 2 +- launcher/pom.xml | 2 +- mllib-local/pom.xml | 2 +- mllib/pom.xml | 2 +- pom.xml | 2 +- python/pyspark/version.py | 2 +- repl/pom.xml | 2 +- resource-managers/kubernetes/core/pom.xml | 2 +- resource-managers/mesos/pom.xml | 2 +- resource-managers/yarn/pom.xml| 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive-thriftserver/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- 40 files changed, 40 insertions(+), 40 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/4df06b45/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index 02bf39b..57485fc 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ org.apache.spark spark-parent_2.11 -2.3.2-SNAPSHOT +2.3.2 ../pom.xml http://git-wip-us.apache.org/repos/asf/spark/blob/4df06b45/common/kvstore/pom.xml -- diff --git a/common/kvstore/pom.xml b/common/kvstore/pom.xml index 646fdfb..53e58c2 100644 --- a/common/kvstore/pom.xml +++ b/common/kvstore/pom.xml @@ -22,7 +22,7 @@ org.apache.spark spark-parent_2.11 -2.3.2-SNAPSHOT +2.3.2 ../../pom.xml http://git-wip-us.apache.org/repos/asf/spark/blob/4df06b45/common/network-common/pom.xml -- diff --git a/common/network-common/pom.xml b/common/network-common/pom.xml index 76c7dcf..d05647c 100644 --- a/common/network-common/pom.xml +++ b/common/network-common/pom.xml @@ -22,7 +22,7 @@ org.apache.spark spark-parent_2.11 -2.3.2-SNAPSHOT +2.3.2 ../../pom.xml http://git-wip-us.apache.org/repos/asf/spark/blob/4df06b45/common/network-shuffle/pom.xml -- diff --git a/common/network-shuffle/pom.xml b/common/network-shuffle/pom.xml index f2661fe..8d46761 100644 --- a/common/network-shuffle/pom.xml +++ b/common/network-shuffle/pom.xml @@ -22,7 +22,7 @@ org.apache.spark spark-parent_2.11 -2.3.2-SNAPSHOT +2.3.2 ../../pom.xml http://git-wip-us.apache.org/repos/asf/spark/blob/4df06b45/common/network-yarn/pom.xml -- diff --git a/common/network-yarn/pom.xml b/common/network-yarn/pom.xml index 229d466..11c290d 100644 --- a/common/network-yarn/pom.xml +++ b/common/network-yarn/pom.xml @@ -22,7 +22,7 @@ org.apache.spark spark-parent_2.11 -2.3.2-SNAPSHOT +2.3.2 ../../pom.xml h
svn commit: r27981 - /dev/spark/v2.3.2-rc1-bin/
Author: jshao Date: Sun Jul 8 04:41:12 2018 New Revision: 27981 Log: Apache Spark 2.3.2 Added: dev/spark/v2.3.2-rc1-bin/ dev/spark/v2.3.2-rc1-bin/SparkR_2.3.2.tar.gz (with props) dev/spark/v2.3.2-rc1-bin/SparkR_2.3.2.tar.gz.asc dev/spark/v2.3.2-rc1-bin/SparkR_2.3.2.tar.gz.sha512 dev/spark/v2.3.2-rc1-bin/pyspark-2.3.2.tar.gz (with props) dev/spark/v2.3.2-rc1-bin/pyspark-2.3.2.tar.gz.asc dev/spark/v2.3.2-rc1-bin/pyspark-2.3.2.tar.gz.sha512 dev/spark/v2.3.2-rc1-bin/spark-2.3.2-bin-hadoop2.6.tgz (with props) dev/spark/v2.3.2-rc1-bin/spark-2.3.2-bin-hadoop2.6.tgz.asc dev/spark/v2.3.2-rc1-bin/spark-2.3.2-bin-hadoop2.6.tgz.sha512 dev/spark/v2.3.2-rc1-bin/spark-2.3.2-bin-hadoop2.7.tgz (with props) dev/spark/v2.3.2-rc1-bin/spark-2.3.2-bin-hadoop2.7.tgz.asc dev/spark/v2.3.2-rc1-bin/spark-2.3.2-bin-hadoop2.7.tgz.sha512 dev/spark/v2.3.2-rc1-bin/spark-2.3.2-bin-without-hadoop.tgz (with props) dev/spark/v2.3.2-rc1-bin/spark-2.3.2-bin-without-hadoop.tgz.asc dev/spark/v2.3.2-rc1-bin/spark-2.3.2-bin-without-hadoop.tgz.sha512 dev/spark/v2.3.2-rc1-bin/spark-2.3.2.tgz (with props) dev/spark/v2.3.2-rc1-bin/spark-2.3.2.tgz.asc dev/spark/v2.3.2-rc1-bin/spark-2.3.2.tgz.sha512 Added: dev/spark/v2.3.2-rc1-bin/SparkR_2.3.2.tar.gz == Binary file - no diff available. Propchange: dev/spark/v2.3.2-rc1-bin/SparkR_2.3.2.tar.gz -- svn:mime-type = application/octet-stream Added: dev/spark/v2.3.2-rc1-bin/SparkR_2.3.2.tar.gz.asc == --- dev/spark/v2.3.2-rc1-bin/SparkR_2.3.2.tar.gz.asc (added) +++ dev/spark/v2.3.2-rc1-bin/SparkR_2.3.2.tar.gz.asc Sun Jul 8 04:41:12 2018 @@ -0,0 +1,16 @@ +-BEGIN PGP SIGNATURE- + +iQIcBAABCgAGBQJbQYuSAAoJENsLIaASlz/QmO4QALqSjYMcRMPssb5VNctoIxS6 +E57s9FRJjOHmxAzTp/5re44crpeWiZVNtMesT/BCg5clS53R/YAjAJOxivx8sV6s +l2cwIt/xZKsQF3PYQnv1WVaGN9So5dlSYZGPyAm9bOp1es5MEAvAn7FBms2ttGsv +M6KYiWPF2DLzo/OVGb7YMH6e9TkjUt/aFcE5ll3UQ4vRNcvxOEO/zpc1FbOEh3sF +cLNRCHV3LWILSatpHznsEg3IkfE35rj09qYt7Ia2ECpCvzK6vbJMRy27duz64yUI +wGD1Yt6CBl1F0BkV/Gh0Vm0x1OnXLNSJzuNh9P3xwD7Dmi251lBasKQr8bNb3Mqs +jsFVGnn8QEKegDkHdfuMwZBaZCeCqA/QhxGSCp20r8p7JwecCFJJEhakR3kgoQ9T +HZ+vIyQpziW78WabHpwEhGGaygWENGhI3TVQpXTGm0JOiJv9RAmvHMZ7tn7yyXI9 +CnFR73xrk4OGjODSdVc+4nS+lB088+j0An1V0c6/3GP/D80MQzj7TSMADYlq/PwO +9mJw0fllpJPWErlDey9WSYhyaLqo3YzrtoB7r+C9ENm6on70muJ4TBMF7Vz77BXi +4IKsB8RE4TFETYhzPW+gkwKu0skM2QfWq4dpLmTBKGefpnEW0hZ93/GzTzeaZ+N0 +ifR8izt4HDcMRD8YSG6t +=iiTK +-END PGP SIGNATURE- Added: dev/spark/v2.3.2-rc1-bin/SparkR_2.3.2.tar.gz.sha512 == --- dev/spark/v2.3.2-rc1-bin/SparkR_2.3.2.tar.gz.sha512 (added) +++ dev/spark/v2.3.2-rc1-bin/SparkR_2.3.2.tar.gz.sha512 Sun Jul 8 04:41:12 2018 @@ -0,0 +1,3 @@ +SparkR_2.3.2.tar.gz: 87320EB2 61E1637A 4099ABE8 FAB19E69 729F9D84 4696838C + 6F3CFB5B 75607A71 3E45080E 2ABCBFF7 5552D248 0D5078B2 + 366E2F44 B291B98C 26820CD2 E94E0D3B Added: dev/spark/v2.3.2-rc1-bin/pyspark-2.3.2.tar.gz == Binary file - no diff available. Propchange: dev/spark/v2.3.2-rc1-bin/pyspark-2.3.2.tar.gz -- svn:mime-type = application/octet-stream Added: dev/spark/v2.3.2-rc1-bin/pyspark-2.3.2.tar.gz.asc == --- dev/spark/v2.3.2-rc1-bin/pyspark-2.3.2.tar.gz.asc (added) +++ dev/spark/v2.3.2-rc1-bin/pyspark-2.3.2.tar.gz.asc Sun Jul 8 04:41:12 2018 @@ -0,0 +1,16 @@ +-BEGIN PGP SIGNATURE- + +iQIcBAABCgAGBQJbQY82AAoJENsLIaASlz/Qj0YQAMOpAIzO/TkDEguG1sPW9aRs +XzdjA0jx/FWYRUFuMwpmGd+AXusw0MScc1tBDOesgGjzgMr20O7a5O+8DEXDTjrr +qgwBY+jn3l/nQRM3UxoMKWLYgrjTCIeXQwER4hKspiElx3BIRs0NdBmUtafauc9l +E0I0Ysy5s5Easpi+vCqT7iEbG+1uHaQVtvJAArbJLv2CgCfRyLw+WjV01tmeG9+h +0Wh/im+nKQphqXd3yhIXNNq43wTFQpVrALcO5F5BhAmXoVr8S+fYbtk09NKiLubZ +/0lMJmN/HIbABxcGWpmqzWmmgI5vr8NxKq1sZ8MyJ1WPUjtJ2VQEYY68pY03sBFi +BxwFr09oboyRIezHQAcc9ptWvupbxcmZWtRUaBVriGoacFavGbpHCKXAlVsPyl0a +ikUGijvaWWPDu8R29HYyqFpQxguSJWD2kdjVzEyqke/m/5X45lmdwnu9wHj9NUa6 +jHDiAi+D3FptBW8SMVAfeqit423Vv1JDOzI+6IrWEtJIYFvYGn+lliRtzaG0GWUJ +IgCK08huzG22pWVvx0m/2NcHMiddPbbNpx/nQ5KvKflVsNyScQD1yggEcW9pFZqC +J0gg5eGidTpb98DSuvaIK+Ucv6fMQF+3CbS6YVgOh6wrB2FXSXlO5ZOeS1WoHvF4 +5Anwy5d6AD+WuWOG4XEG +=Ez4U +-END PGP SIGNATURE- Added: dev/spark/v2.3.2-rc1-bin/pyspark-2.3.2.tar.gz.sha512 == --- dev/spark/v2.3.2-rc1-bin/pyspark-2.3.2.tar.gz.sha512 (added) +++ dev/spark/v2.3.2-rc1-bin/pyspark-2.3.2.tar.gz.sha512 Sun Jul 8 04:41:12 2018 @@ -0,0 +1,3 @@ +pyspa
svn commit: r27982 - in /dev/spark/2.3.3-SNAPSHOT-2018_07_07_22_01-72eb97c-docs: ./ _site/ _site/api/ _site/api/R/ _site/api/java/ _site/api/java/lib/ _site/api/java/org/ _site/api/java/org/apache/ _s
Author: pwendell Date: Sun Jul 8 05:15:27 2018 New Revision: 27982 Log: Apache Spark 2.3.3-SNAPSHOT-2018_07_07_22_01-72eb97c docs [This commit notification would consist of 1443 parts, which exceeds the limit of 50 ones, so it was shortened to the summary.] - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org