[spark] branch master updated (7f36cd2 -> 8d1b5ba)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 7f36cd2 [SPARK-28570][CORE][SHUFFLE] Make UnsafeShuffleWriter use the new API add 8d1b5ba [SPARK-28906][BUILD] Fix incorrect information in bin/spark-submit --version No new revisions were added by this update. Summary of changes: dev/create-release/do-release-docker.sh | 1 + dev/create-release/release-build.sh | 3 +-- 2 files changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [SPARK-28906][BUILD] Fix incorrect information in bin/spark-submit --version
This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new 9e6a1b8 [SPARK-28906][BUILD] Fix incorrect information in bin/spark-submit --version 9e6a1b8 is described below commit 9e6a1b85c3443894f857b9127b7c4511a2d45c79 Author: Kazuaki Ishizaki AuthorDate: Wed Sep 11 08:12:44 2019 -0500 [SPARK-28906][BUILD] Fix incorrect information in bin/spark-submit --version ### What changes were proposed in this pull request? This PR allows `bin/spark-submit --version` to show the correct information while the previous versions, which were created by `dev/create-release/do-release-docker.sh`, show incorrect information. There are two root causes to show incorrect information: 1. Did not pass `USER` environment variable to the docker container 1. Did not keep `.git` directory in the work directory ### Why are the changes needed? The information is missing while the previous versions show the correct information. ### Does this PR introduce any user-facing change? Yes, the following is the console output in branch-2.3 ``` $ bin/spark-submit --version Welcome to __ / __/__ ___ _/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.3.4 /_/ Using Scala version 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_212 Branch HEAD Compiled by user ishizaki on 2019-09-02T02:18:10Z Revision 8c6f8150f3c6298ff4e1c7e06028f12d7eaf0210 Url https://gitbox.apache.org/repos/asf/spark.git Type --help for more information. ``` Without this PR, the console output is as follows ``` $ spark-submit --version Welcome to __ / __/__ ___ _/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.3.4 /_/ Using Scala version 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_212 Branch Compiled by user on 2019-08-26T08:29:39Z Revision Url Type --help for more information. ``` ### How was this patch tested? After building the package, I manually executed `bin/spark-submit --version` Closes #25655 from kiszk/SPARK-28906. Authored-by: Kazuaki Ishizaki Signed-off-by: Sean Owen (cherry picked from commit 8d1b5ba76651c049c957ad48ae60f2c8656dff49) Signed-off-by: Sean Owen --- dev/create-release/do-release-docker.sh | 1 + dev/create-release/release-build.sh | 3 +-- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/dev/create-release/do-release-docker.sh b/dev/create-release/do-release-docker.sh index c1a122e..f643c06 100755 --- a/dev/create-release/do-release-docker.sh +++ b/dev/create-release/do-release-docker.sh @@ -127,6 +127,7 @@ GPG_KEY=$GPG_KEY ASF_PASSWORD=$ASF_PASSWORD GPG_PASSPHRASE=$GPG_PASSPHRASE RELEASE_STEP=$RELEASE_STEP +USER=$USER EOF JAVA_VOL= diff --git a/dev/create-release/release-build.sh b/dev/create-release/release-build.sh index affb4dc..8bcf8da 100755 --- a/dev/create-release/release-build.sh +++ b/dev/create-release/release-build.sh @@ -170,7 +170,6 @@ DEST_DIR_NAME="$SPARK_PACKAGE_VERSION" git clean -d -f -x rm .gitignore -rm -rf .git cd .. if [[ "$1" == "package" ]]; then @@ -185,7 +184,7 @@ if [[ "$1" == "package" ]]; then rm -r spark-$SPARK_VERSION/licenses-binary fi - tar cvzf spark-$SPARK_VERSION.tgz spark-$SPARK_VERSION + tar cvzf spark-$SPARK_VERSION.tgz --exclude spark-$SPARK_VERSION/.git spark-$SPARK_VERSION echo $GPG_PASSPHRASE | $GPG --passphrase-fd 0 --armour --output spark-$SPARK_VERSION.tgz.asc \ --detach-sig spark-$SPARK_VERSION.tgz echo $GPG_PASSPHRASE | $GPG --passphrase-fd 0 --print-md \ - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-27492][DOC][YARN][K8S][CORE] Resource scheduling high level user docs
This is an automated email from the ASF dual-hosted git repository. tgraves pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new b425f8e [SPARK-27492][DOC][YARN][K8S][CORE] Resource scheduling high level user docs b425f8e is described below commit b425f8ee6599f53f47d7d4a8f0c27f2ba7d2eab9 Author: Thomas Graves AuthorDate: Wed Sep 11 08:22:36 2019 -0500 [SPARK-27492][DOC][YARN][K8S][CORE] Resource scheduling high level user docs ### What changes were proposed in this pull request? Document the resource scheduling feature - https://issues.apache.org/jira/browse/SPARK-24615 Add general docs, yarn, kubernetes, and standalone cluster specific ones. ### Why are the changes needed? Help users understand the feature ### Does this PR introduce any user-facing change? docs ### How was this patch tested? N/A Closes #25698 from tgravescs/SPARK-27492-gpu-sched-docs. Authored-by: Thomas Graves Signed-off-by: Thomas Graves --- docs/configuration.md | 14 +- docs/running-on-kubernetes.md | 11 +++ docs/running-on-yarn.md | 14 ++ docs/spark-standalone.md | 12 4 files changed, 50 insertions(+), 1 deletion(-) diff --git a/docs/configuration.md b/docs/configuration.md index 9933283..5cf42d5 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -230,7 +230,7 @@ of the most common options to set are: write to STDOUT a JSON string in the format of the ResourceInformation class. This has a name and an array of addresses. For a client-submitted driver in Standalone, discovery script must assign different resource addresses to this driver comparing to workers' and -other dirvers' when spark.resources.coordinate.enable is off. +other drivers' when spark.resources.coordinate.enable is off. @@ -2617,3 +2617,15 @@ Also, you can modify or add configurations at runtime: --conf spark.hadoop.abc.def=xyz \ myApp.jar {% endhighlight %} + +# Custom Resource Scheduling and Configuration Overview + +GPUs and other accelerators have been widely used for accelerating special workloads, e.g., +deep learning and signal processing. Spark now supports requesting and scheduling generic resources, such as GPUs, with a few caveats. The current implementation requires that the resource have addresses that can be allocated by the scheduler. It requires your cluster manager to support and be properly configured with the resources. + +There are configurations available to request resources for the driver: spark.driver.resource.{resourceName}.amount, request resources for the executor(s): spark.executor.resource.{resourceName}.amount and specify the requirements for each task: spark.task.resource.{resourceName}.amount. The spark.driver.resource.{resourceName}.discoveryScript config is required on YARN, Kubernetes and a client side Driver on Spark Standalone. spa [...] + +Spark will use the configurations specified to first request containers with the corresponding resources from the cluster manager. Once it gets the container, Spark launches an Executor in that container which will discover what resources the container has and the addresses associated with each resource. The Executor will register with the Driver and report back the resources available to that Executor. The Spark scheduler can then schedule tasks to each Executor and assign specific reso [...] + +See your cluster manager specific page for requirements and details on each of - [YARN](running-on-yarn.html#resource-allocation-and-configuration-overview), [Kubernetes](running-on-kubernetes.html#resource-allocation-and-configuration-overview) and [Standalone Mode](spark-standalone.html#resource-allocation-and-configuration-overview). It is currently not available with Mesos or local mode. If using local-cluster mode see the Spark Standalone documentation but be aware only a single wor [...] + diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md index 2d4e5cd..4ef738e 100644 --- a/docs/running-on-kubernetes.md +++ b/docs/running-on-kubernetes.md @@ -1266,3 +1266,14 @@ The following affect the driver and executor containers. All other containers in + +### Resource Allocation and Configuration Overview + +Please make sure to have read the Custom Resource Scheduling and Configuration Overview section on the [configuration page](configuration.html). This section only talks about the Kubernetes specific aspects of resource scheduling. + +The user is responsible to properly configuring the Kubernetes cluster to have the resources available and ideally isolate each resource per container so that a resource is not shared between multiple containers. If the resource is not isolated the user is responsible for writin
[spark] branch master updated (b425f8e -> f263909)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from b425f8e [SPARK-27492][DOC][YARN][K8S][CORE] Resource scheduling high level user docs add f263909 [SPARK-23243][CORE][FOLLOWUP] Remove todo added by SPARK-23207 No new revisions were added by this update. Summary of changes: core/src/main/scala/org/apache/spark/rdd/RDD.scala | 2 -- 1 file changed, 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f263909 -> 2736efa)
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f263909 [SPARK-23243][CORE][FOLLOWUP] Remove todo added by SPARK-23207 add 2736efa [SPARK-26989][CORE][TEST] DAGSchedulerSuite: ensure listeners are fully processed before checking recorded values No new revisions were added by this update. Summary of changes: .../apache/spark/scheduler/DAGSchedulerSuite.scala | 138 +++-- 1 file changed, 71 insertions(+), 67 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-29007][STREAMING][MLLIB][TESTS] Enforce not leaking SparkContext in tests which creates new StreamingContext with new SparkContext
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new b62ef8f [SPARK-29007][STREAMING][MLLIB][TESTS] Enforce not leaking SparkContext in tests which creates new StreamingContext with new SparkContext b62ef8f is described below commit b62ef8f7935ae5c9a4a5e7e8a17aa5d7375c85b1 Author: Jungtaek Lim (HeartSaVioR) AuthorDate: Wed Sep 11 10:29:13 2019 -0700 [SPARK-29007][STREAMING][MLLIB][TESTS] Enforce not leaking SparkContext in tests which creates new StreamingContext with new SparkContext ### What changes were proposed in this pull request? This patch enforces tests to prevent leaking newly created SparkContext while is created via initializing StreamingContext. Leaking SparkContext in test would make most of following tests being failed as well, so this patch applies defensive programming, trying its best to ensure SparkContext is cleaned up. ### Why are the changes needed? We got some case in CI build where SparkContext is being leaked and other tests are affected by leaked SparkContext. Ideally we should isolate the environment among tests if possible. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Modified UTs. Closes #25709 from HeartSaVioR/SPARK-29007. Authored-by: Jungtaek Lim (HeartSaVioR) Signed-off-by: Marcelo Vanzin --- external/kafka-0-10/pom.xml| 7 ++ .../kafka010/DirectKafkaStreamSuite.scala | 24 +++ .../streaming/kinesis/KinesisStreamSuite.scala | 37 -- mllib/pom.xml | 7 ++ .../StreamingLogisticRegressionSuite.scala | 16 ++--- .../mllib/clustering/StreamingKMeansSuite.scala| 13 +--- .../StreamingLinearRegressionSuite.scala | 16 ++--- .../apache/spark/streaming/CheckpointSuite.scala | 18 ++--- .../spark/streaming/DStreamClosureSuite.scala | 16 + .../apache/spark/streaming/DStreamScopeSuite.scala | 23 +++--- .../spark/streaming/LocalStreamingContext.scala| 83 ++ .../apache/spark/streaming/MapWithStateSuite.scala | 30 +++- .../streaming/ReceiverInputDStreamSuite.scala | 16 ++--- .../spark/streaming/StreamingContextSuite.scala| 52 ++ .../spark/streaming/StreamingListenerSuite.scala | 11 +-- .../org/apache/spark/streaming/TestSuiteBase.scala | 30 .../spark/streaming/WindowOperationsSuite.scala| 19 ++--- .../scheduler/ExecutorAllocationManagerSuite.scala | 19 ++--- .../scheduler/InputInfoTrackerSuite.scala | 22 ++ .../streaming/scheduler/RateControllerSuite.scala | 6 +- .../ui/StreamingJobProgressListenerSuite.scala | 16 ++--- 21 files changed, 240 insertions(+), 241 deletions(-) diff --git a/external/kafka-0-10/pom.xml b/external/kafka-0-10/pom.xml index 397de87..d11569d 100644 --- a/external/kafka-0-10/pom.xml +++ b/external/kafka-0-10/pom.xml @@ -47,6 +47,13 @@ org.apache.spark + spark-streaming_${scala.binary.version} + ${project.version} + test-jar + test + + + org.apache.spark spark-core_${scala.binary.version} ${project.version} test-jar diff --git a/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala b/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala index 4d3e476..26b41e6 100644 --- a/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala +++ b/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala @@ -18,8 +18,8 @@ package org.apache.spark.streaming.kafka010 import java.io.File -import java.lang.{ Long => JLong } -import java.util.{ Arrays, HashMap => JHashMap, Map => JMap, UUID } +import java.lang.{Long => JLong} +import java.util.{Arrays, HashMap => JHashMap, Map => JMap, UUID} import java.util.concurrent.ConcurrentHashMap import java.util.concurrent.ConcurrentLinkedQueue import java.util.concurrent.atomic.AtomicLong @@ -31,13 +31,12 @@ import scala.util.Random import org.apache.kafka.clients.consumer._ import org.apache.kafka.common.TopicPartition import org.apache.kafka.common.serialization.StringDeserializer -import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll} import org.scalatest.concurrent.Eventually import org.apache.spark.{SparkConf, SparkFunSuite} import org.apache.spark.internal.Logging import org.apache.spark.rdd.RDD -import org.apache.spark.streaming.{Milliseconds, StreamingContext, Time} +import org.apache.spark.streaming.{LocalStreamingContext, Milliseconds, StreamingContext, Time} import org.apache.spark.streaming.dstream.DStre
[spark] branch master updated (b62ef8f -> c18f849)
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from b62ef8f [SPARK-29007][STREAMING][MLLIB][TESTS] Enforce not leaking SparkContext in tests which creates new StreamingContext with new SparkContext add c18f849 [SPARK-24663][STREAMING][TESTS] StreamingContextSuite: Wait until slow receiver has been initialized, but with hard timeout No new revisions were added by this update. Summary of changes: .../org/apache/spark/streaming/StreamingContextSuite.scala | 12 1 file changed, 8 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (c18f849 -> 7ce0f2b)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from c18f849 [SPARK-24663][STREAMING][TESTS] StreamingContextSuite: Wait until slow receiver has been initialized, but with hard timeout add 7ce0f2b [SPARK-29041][PYTHON] Allows createDataFrame to accept bytes as binary type No new revisions were added by this update. Summary of changes: python/pyspark/sql/tests/test_serde.py | 4 python/pyspark/sql/types.py| 2 +- 2 files changed, 5 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (7ce0f2b -> eec728a)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 7ce0f2b [SPARK-29041][PYTHON] Allows createDataFrame to accept bytes as binary type add eec728a [SPARK-29057][SQL] remove InsertIntoTable No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/analysis/Analyzer.scala | 4 +-- .../sql/catalyst/analysis/CheckAnalysis.scala | 9 +++--- .../plans/logical/basicLogicalOperators.scala | 36 -- .../org/apache/spark/sql/DataFrameWriter.scala | 9 +++--- .../execution/datasources/DataSourceStrategy.scala | 11 --- .../datasources/FallBackFileSourceV2.scala | 7 +++-- .../spark/sql/execution/datasources/rules.scala| 25 +++ .../spark/sql/util/DataFrameCallbackSuite.scala| 7 +++-- .../org/apache/spark/sql/hive/HiveStrategies.scala | 15 - .../org/apache/spark/sql/hive/InsertSuite.scala| 1 - 10 files changed, 46 insertions(+), 78 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [MINOR][DOCS] Fix few typos in the java docs
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 8f632d7 [MINOR][DOCS] Fix few typos in the java docs 8f632d7 is described below commit 8f632d70455156010f0e87288541304ad2164a52 Author: dengziming AuthorDate: Thu Sep 12 09:30:03 2019 +0900 [MINOR][DOCS] Fix few typos in the java docs JIRA :https://issues.apache.org/jira/browse/SPARK-29050 'a hdfs' change into 'an hdfs' 'an unique' change into 'a unique' 'an url' change into 'a url' 'a error' change into 'an error' Closes #25756 from dengziming/feature_fix_typos. Authored-by: dengziming Signed-off-by: HyukjinKwon --- R/pkg/R/context.R | 4 ++-- core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala | 2 +- core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala | 2 +- core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala | 2 +- core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala| 2 +- docs/spark-standalone.md | 2 +- .../org/apache/spark/streaming/kinesis/KinesisCheckpointer.scala | 2 +- python/pyspark/context.py | 2 +- .../sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala | 4 ++-- .../scala/org/apache/spark/sql/streaming/StreamingQueryListener.scala | 4 ++-- sql/core/src/main/scala/org/apache/spark/sql/streaming/progress.scala | 2 +- .../src/test/resources/ql/src/test/queries/clientpositive/load_fs2.q | 2 +- .../main/scala/org/apache/spark/streaming/dstream/InputDStream.scala | 4 ++-- 13 files changed, 17 insertions(+), 17 deletions(-) diff --git a/R/pkg/R/context.R b/R/pkg/R/context.R index 51ae2d2..93ba130 100644 --- a/R/pkg/R/context.R +++ b/R/pkg/R/context.R @@ -301,7 +301,7 @@ broadcastRDD <- function(sc, object) { #' Set the checkpoint directory #' #' Set the directory under which RDDs are going to be checkpointed. The -#' directory must be a HDFS path if running on a cluster. +#' directory must be an HDFS path if running on a cluster. #' #' @param sc Spark Context to use #' @param dirName Directory path @@ -446,7 +446,7 @@ setLogLevel <- function(level) { #' Set checkpoint directory #' #' Set the directory under which SparkDataFrame are going to be checkpointed. The directory must be -#' a HDFS path if running on a cluster. +#' an HDFS path if running on a cluster. #' #' @rdname setCheckpointDir #' @param directory Directory path to checkpoint to diff --git a/core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala b/core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala index 330c2f6..3485128 100644 --- a/core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala +++ b/core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala @@ -609,7 +609,7 @@ class JavaSparkContext(val sc: SparkContext) extends Closeable { /** * Set the directory under which RDDs are going to be checkpointed. The directory must - * be a HDFS path if running on a cluster. + * be an HDFS path if running on a cluster. */ def setCheckpointDir(dir: String) { sc.setCheckpointDir(dir) diff --git a/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala b/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala index c96640a..b552444 100644 --- a/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala +++ b/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala @@ -124,7 +124,7 @@ private[spark] class MetricsSystem private ( * If either ID is not available, this defaults to just using . * * @param source Metric source to be named by this method. - * @return An unique metric name for each combination of + * @return A unique metric name for each combination of * application, executor/driver and metric source. */ private[spark] def buildRegistryName(source: Source): String = { diff --git a/core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala b/core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala index d188bdd..49e32d0 100644 --- a/core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala +++ b/core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala @@ -27,7 +27,7 @@ import org.apache.spark.util.Utils /** * :: DeveloperApi :: - * This class represent an unique identifier for a BlockManager. + * This class represent a unique identifier for a BlockManager. * * The first 2 constructors of this class are made private to ensure that BlockManagerId objects * can be created only using the apply method in the companion object. This allows de-duplication diff --git a/core/src/test/scala/org/ap
[spark] branch branch-2.4 updated: [MINOR][DOCS] Fix few typos in the java docs
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new ecb2052 [MINOR][DOCS] Fix few typos in the java docs ecb2052 is described below commit ecb2052bf0cf7dea749cca10d864f7383eeb1224 Author: dengziming AuthorDate: Thu Sep 12 09:30:03 2019 +0900 [MINOR][DOCS] Fix few typos in the java docs JIRA :https://issues.apache.org/jira/browse/SPARK-29050 'a hdfs' change into 'an hdfs' 'an unique' change into 'a unique' 'an url' change into 'a url' 'a error' change into 'an error' Closes #25756 from dengziming/feature_fix_typos. Authored-by: dengziming Signed-off-by: HyukjinKwon (cherry picked from commit 8f632d70455156010f0e87288541304ad2164a52) Signed-off-by: HyukjinKwon --- R/pkg/R/context.R | 4 ++-- core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala | 2 +- core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala | 2 +- core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala | 2 +- core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala| 2 +- docs/spark-standalone.md | 2 +- .../org/apache/spark/streaming/kinesis/KinesisCheckpointer.scala | 2 +- python/pyspark/context.py | 2 +- .../sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala | 4 ++-- .../scala/org/apache/spark/sql/streaming/StreamingQueryListener.scala | 4 ++-- sql/core/src/main/scala/org/apache/spark/sql/streaming/progress.scala | 2 +- .../src/test/resources/ql/src/test/queries/clientpositive/load_fs2.q | 2 +- .../main/scala/org/apache/spark/streaming/dstream/InputDStream.scala | 4 ++-- 13 files changed, 17 insertions(+), 17 deletions(-) diff --git a/R/pkg/R/context.R b/R/pkg/R/context.R index b49f7c3..f1a6b84 100644 --- a/R/pkg/R/context.R +++ b/R/pkg/R/context.R @@ -297,7 +297,7 @@ broadcastRDD <- function(sc, object) { #' Set the checkpoint directory #' #' Set the directory under which RDDs are going to be checkpointed. The -#' directory must be a HDFS path if running on a cluster. +#' directory must be an HDFS path if running on a cluster. #' #' @param sc Spark Context to use #' @param dirName Directory path @@ -442,7 +442,7 @@ setLogLevel <- function(level) { #' Set checkpoint directory #' #' Set the directory under which SparkDataFrame are going to be checkpointed. The directory must be -#' a HDFS path if running on a cluster. +#' an HDFS path if running on a cluster. #' #' @rdname setCheckpointDir #' @param directory Directory path to checkpoint to diff --git a/core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala b/core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala index 09c8384..09e9910 100644 --- a/core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala +++ b/core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala @@ -713,7 +713,7 @@ class JavaSparkContext(val sc: SparkContext) /** * Set the directory under which RDDs are going to be checkpointed. The directory must - * be a HDFS path if running on a cluster. + * be an HDFS path if running on a cluster. */ def setCheckpointDir(dir: String) { sc.setCheckpointDir(dir) diff --git a/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala b/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala index 3457a26..657d75c 100644 --- a/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala +++ b/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala @@ -122,7 +122,7 @@ private[spark] class MetricsSystem private ( * If either ID is not available, this defaults to just using . * * @param source Metric source to be named by this method. - * @return An unique metric name for each combination of + * @return A unique metric name for each combination of * application, executor/driver and metric source. */ private[spark] def buildRegistryName(source: Source): String = { diff --git a/core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala b/core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala index d4a59c3..83cd4f0 100644 --- a/core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala +++ b/core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala @@ -27,7 +27,7 @@ import org.apache.spark.util.Utils /** * :: DeveloperApi :: - * This class represent an unique identifier for a BlockManager. + * This class represent a unique identifier for a BlockManager. * * The first 2 constructors of this class are made private to ensure that BlockManagerId objects * can be created only using the apply method
[spark] branch master updated (8f632d7 -> 850833f)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8f632d7 [MINOR][DOCS] Fix few typos in the java docs add 850833f [SPARK-29046][SQL] Fix NPE in SQLConf.get when active SparkContext is stopping No new revisions were added by this update. Summary of changes: .../scala/org/apache/spark/sql/internal/SQLConf.scala | 3 ++- .../org/apache/spark/sql/internal/SQLConfSuite.scala | 19 +++ 2 files changed, 21 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (850833f -> 6768431)
This is an automated email from the ASF dual-hosted git repository. yumwang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 850833f [SPARK-29046][SQL] Fix NPE in SQLConf.get when active SparkContext is stopping add 6768431 [SPARK-29045][SQL][TESTS] Drop table to avoid test failure in SQLMetricsSuite No new revisions were added by this update. Summary of changes: .../sql/execution/metric/SQLMetricsTestUtils.scala | 46 +++--- 1 file changed, 24 insertions(+), 22 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org