[spark] branch master updated (7f36cd2 -> 8d1b5ba)
This is an automated email from the ASF dual-hosted git repository. srowen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 7f36cd2 [SPARK-28570][CORE][SHUFFLE] Make UnsafeShuffleWriter use the new API add 8d1b5ba [SPARK-28906][BUILD] Fix incorrect information in bin/spark-submit --version No new revisions were added by this update. Summary of changes: dev/create-release/do-release-docker.sh | 1 + dev/create-release/release-build.sh | 3 +-- 2 files changed, 2 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [SPARK-28906][BUILD] Fix incorrect information in bin/spark-submit --version
This is an automated email from the ASF dual-hosted git repository.
srowen pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new 9e6a1b8 [SPARK-28906][BUILD] Fix incorrect information in
bin/spark-submit --version
9e6a1b8 is described below
commit 9e6a1b85c3443894f857b9127b7c4511a2d45c79
Author: Kazuaki Ishizaki
AuthorDate: Wed Sep 11 08:12:44 2019 -0500
[SPARK-28906][BUILD] Fix incorrect information in bin/spark-submit --version
### What changes were proposed in this pull request?
This PR allows `bin/spark-submit --version` to show the correct information
while the previous versions, which were created by
`dev/create-release/do-release-docker.sh`, show incorrect information.
There are two root causes to show incorrect information:
1. Did not pass `USER` environment variable to the docker container
1. Did not keep `.git` directory in the work directory
### Why are the changes needed?
The information is missing while the previous versions show the correct
information.
### Does this PR introduce any user-facing change?
Yes, the following is the console output in branch-2.3
```
$ bin/spark-submit --version
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.3.4
/_/
Using Scala version 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_212
Branch HEAD
Compiled by user ishizaki on 2019-09-02T02:18:10Z
Revision 8c6f8150f3c6298ff4e1c7e06028f12d7eaf0210
Url https://gitbox.apache.org/repos/asf/spark.git
Type --help for more information.
```
Without this PR, the console output is as follows
```
$ spark-submit --version
Welcome to
__
/ __/__ ___ _/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.3.4
/_/
Using Scala version 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_212
Branch
Compiled by user on 2019-08-26T08:29:39Z
Revision
Url
Type --help for more information.
```
### How was this patch tested?
After building the package, I manually executed `bin/spark-submit --version`
Closes #25655 from kiszk/SPARK-28906.
Authored-by: Kazuaki Ishizaki
Signed-off-by: Sean Owen
(cherry picked from commit 8d1b5ba76651c049c957ad48ae60f2c8656dff49)
Signed-off-by: Sean Owen
---
dev/create-release/do-release-docker.sh | 1 +
dev/create-release/release-build.sh | 3 +--
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/dev/create-release/do-release-docker.sh
b/dev/create-release/do-release-docker.sh
index c1a122e..f643c06 100755
--- a/dev/create-release/do-release-docker.sh
+++ b/dev/create-release/do-release-docker.sh
@@ -127,6 +127,7 @@ GPG_KEY=$GPG_KEY
ASF_PASSWORD=$ASF_PASSWORD
GPG_PASSPHRASE=$GPG_PASSPHRASE
RELEASE_STEP=$RELEASE_STEP
+USER=$USER
EOF
JAVA_VOL=
diff --git a/dev/create-release/release-build.sh
b/dev/create-release/release-build.sh
index affb4dc..8bcf8da 100755
--- a/dev/create-release/release-build.sh
+++ b/dev/create-release/release-build.sh
@@ -170,7 +170,6 @@ DEST_DIR_NAME="$SPARK_PACKAGE_VERSION"
git clean -d -f -x
rm .gitignore
-rm -rf .git
cd ..
if [[ "$1" == "package" ]]; then
@@ -185,7 +184,7 @@ if [[ "$1" == "package" ]]; then
rm -r spark-$SPARK_VERSION/licenses-binary
fi
- tar cvzf spark-$SPARK_VERSION.tgz spark-$SPARK_VERSION
+ tar cvzf spark-$SPARK_VERSION.tgz --exclude spark-$SPARK_VERSION/.git
spark-$SPARK_VERSION
echo $GPG_PASSPHRASE | $GPG --passphrase-fd 0 --armour --output
spark-$SPARK_VERSION.tgz.asc \
--detach-sig spark-$SPARK_VERSION.tgz
echo $GPG_PASSPHRASE | $GPG --passphrase-fd 0 --print-md \
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-27492][DOC][YARN][K8S][CORE] Resource scheduling high level user docs
This is an automated email from the ASF dual-hosted git repository.
tgraves pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new b425f8e [SPARK-27492][DOC][YARN][K8S][CORE] Resource scheduling high
level user docs
b425f8e is described below
commit b425f8ee6599f53f47d7d4a8f0c27f2ba7d2eab9
Author: Thomas Graves
AuthorDate: Wed Sep 11 08:22:36 2019 -0500
[SPARK-27492][DOC][YARN][K8S][CORE] Resource scheduling high level user docs
### What changes were proposed in this pull request?
Document the resource scheduling feature -
https://issues.apache.org/jira/browse/SPARK-24615
Add general docs, yarn, kubernetes, and standalone cluster specific ones.
### Why are the changes needed?
Help users understand the feature
### Does this PR introduce any user-facing change?
docs
### How was this patch tested?
N/A
Closes #25698 from tgravescs/SPARK-27492-gpu-sched-docs.
Authored-by: Thomas Graves
Signed-off-by: Thomas Graves
---
docs/configuration.md | 14 +-
docs/running-on-kubernetes.md | 11 +++
docs/running-on-yarn.md | 14 ++
docs/spark-standalone.md | 12
4 files changed, 50 insertions(+), 1 deletion(-)
diff --git a/docs/configuration.md b/docs/configuration.md
index 9933283..5cf42d5 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -230,7 +230,7 @@ of the most common options to set are:
write to STDOUT a JSON string in the format of the ResourceInformation
class. This has a
name and an array of addresses. For a client-submitted driver in
Standalone, discovery
script must assign different resource addresses to this driver comparing
to workers' and
-other dirvers' when spark.resources.coordinate.enable is off.
+other drivers' when spark.resources.coordinate.enable is off.
@@ -2617,3 +2617,15 @@ Also, you can modify or add configurations at runtime:
--conf spark.hadoop.abc.def=xyz \
myApp.jar
{% endhighlight %}
+
+# Custom Resource Scheduling and Configuration Overview
+
+GPUs and other accelerators have been widely used for accelerating special
workloads, e.g.,
+deep learning and signal processing. Spark now supports requesting and
scheduling generic resources, such as GPUs, with a few caveats. The current
implementation requires that the resource have addresses that can be allocated
by the scheduler. It requires your cluster manager to support and be properly
configured with the resources.
+
+There are configurations available to request resources for the driver:
spark.driver.resource.{resourceName}.amount, request resources for
the executor(s): spark.executor.resource.{resourceName}.amount and
specify the requirements for each task:
spark.task.resource.{resourceName}.amount. The
spark.driver.resource.{resourceName}.discoveryScript config is
required on YARN, Kubernetes and a client side Driver on Spark Standalone.
spa [...]
+
+Spark will use the configurations specified to first request containers with
the corresponding resources from the cluster manager. Once it gets the
container, Spark launches an Executor in that container which will discover
what resources the container has and the addresses associated with each
resource. The Executor will register with the Driver and report back the
resources available to that Executor. The Spark scheduler can then schedule
tasks to each Executor and assign specific reso [...]
+
+See your cluster manager specific page for requirements and details on each of
- [YARN](running-on-yarn.html#resource-allocation-and-configuration-overview),
[Kubernetes](running-on-kubernetes.html#resource-allocation-and-configuration-overview)
and [Standalone
Mode](spark-standalone.html#resource-allocation-and-configuration-overview). It
is currently not available with Mesos or local mode. If using local-cluster
mode see the Spark Standalone documentation but be aware only a single wor [...]
+
diff --git a/docs/running-on-kubernetes.md b/docs/running-on-kubernetes.md
index 2d4e5cd..4ef738e 100644
--- a/docs/running-on-kubernetes.md
+++ b/docs/running-on-kubernetes.md
@@ -1266,3 +1266,14 @@ The following affect the driver and executor containers.
All other containers in
+
+### Resource Allocation and Configuration Overview
+
+Please make sure to have read the Custom Resource Scheduling and Configuration
Overview section on the [configuration page](configuration.html). This section
only talks about the Kubernetes specific aspects of resource scheduling.
+
+The user is responsible to properly configuring the Kubernetes cluster to have
the resources available and ideally isolate each resource per container so that
a resource is not shared between multiple containers. If the resource is not
isolated the user is responsible for writin
[spark] branch master updated (b425f8e -> f263909)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from b425f8e [SPARK-27492][DOC][YARN][K8S][CORE] Resource scheduling high level user docs add f263909 [SPARK-23243][CORE][FOLLOWUP] Remove todo added by SPARK-23207 No new revisions were added by this update. Summary of changes: core/src/main/scala/org/apache/spark/rdd/RDD.scala | 2 -- 1 file changed, 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (f263909 -> 2736efa)
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from f263909 [SPARK-23243][CORE][FOLLOWUP] Remove todo added by SPARK-23207 add 2736efa [SPARK-26989][CORE][TEST] DAGSchedulerSuite: ensure listeners are fully processed before checking recorded values No new revisions were added by this update. Summary of changes: .../apache/spark/scheduler/DAGSchedulerSuite.scala | 138 +++-- 1 file changed, 71 insertions(+), 67 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-29007][STREAMING][MLLIB][TESTS] Enforce not leaking SparkContext in tests which creates new StreamingContext with new SparkContext
This is an automated email from the ASF dual-hosted git repository.
vanzin pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new b62ef8f [SPARK-29007][STREAMING][MLLIB][TESTS] Enforce not leaking
SparkContext in tests which creates new StreamingContext with new SparkContext
b62ef8f is described below
commit b62ef8f7935ae5c9a4a5e7e8a17aa5d7375c85b1
Author: Jungtaek Lim (HeartSaVioR)
AuthorDate: Wed Sep 11 10:29:13 2019 -0700
[SPARK-29007][STREAMING][MLLIB][TESTS] Enforce not leaking SparkContext in
tests which creates new StreamingContext with new SparkContext
### What changes were proposed in this pull request?
This patch enforces tests to prevent leaking newly created SparkContext
while is created via initializing StreamingContext. Leaking SparkContext in
test would make most of following tests being failed as well, so this patch
applies defensive programming, trying its best to ensure SparkContext is
cleaned up.
### Why are the changes needed?
We got some case in CI build where SparkContext is being leaked and other
tests are affected by leaked SparkContext. Ideally we should isolate the
environment among tests if possible.
### Does this PR introduce any user-facing change?
No.
### How was this patch tested?
Modified UTs.
Closes #25709 from HeartSaVioR/SPARK-29007.
Authored-by: Jungtaek Lim (HeartSaVioR)
Signed-off-by: Marcelo Vanzin
---
external/kafka-0-10/pom.xml| 7 ++
.../kafka010/DirectKafkaStreamSuite.scala | 24 +++
.../streaming/kinesis/KinesisStreamSuite.scala | 37 --
mllib/pom.xml | 7 ++
.../StreamingLogisticRegressionSuite.scala | 16 ++---
.../mllib/clustering/StreamingKMeansSuite.scala| 13 +---
.../StreamingLinearRegressionSuite.scala | 16 ++---
.../apache/spark/streaming/CheckpointSuite.scala | 18 ++---
.../spark/streaming/DStreamClosureSuite.scala | 16 +
.../apache/spark/streaming/DStreamScopeSuite.scala | 23 +++---
.../spark/streaming/LocalStreamingContext.scala| 83 ++
.../apache/spark/streaming/MapWithStateSuite.scala | 30 +++-
.../streaming/ReceiverInputDStreamSuite.scala | 16 ++---
.../spark/streaming/StreamingContextSuite.scala| 52 ++
.../spark/streaming/StreamingListenerSuite.scala | 11 +--
.../org/apache/spark/streaming/TestSuiteBase.scala | 30
.../spark/streaming/WindowOperationsSuite.scala| 19 ++---
.../scheduler/ExecutorAllocationManagerSuite.scala | 19 ++---
.../scheduler/InputInfoTrackerSuite.scala | 22 ++
.../streaming/scheduler/RateControllerSuite.scala | 6 +-
.../ui/StreamingJobProgressListenerSuite.scala | 16 ++---
21 files changed, 240 insertions(+), 241 deletions(-)
diff --git a/external/kafka-0-10/pom.xml b/external/kafka-0-10/pom.xml
index 397de87..d11569d 100644
--- a/external/kafka-0-10/pom.xml
+++ b/external/kafka-0-10/pom.xml
@@ -47,6 +47,13 @@
org.apache.spark
+ spark-streaming_${scala.binary.version}
+ ${project.version}
+ test-jar
+ test
+
+
+ org.apache.spark
spark-core_${scala.binary.version}
${project.version}
test-jar
diff --git
a/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala
b/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala
index 4d3e476..26b41e6 100644
---
a/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala
+++
b/external/kafka-0-10/src/test/scala/org/apache/spark/streaming/kafka010/DirectKafkaStreamSuite.scala
@@ -18,8 +18,8 @@
package org.apache.spark.streaming.kafka010
import java.io.File
-import java.lang.{ Long => JLong }
-import java.util.{ Arrays, HashMap => JHashMap, Map => JMap, UUID }
+import java.lang.{Long => JLong}
+import java.util.{Arrays, HashMap => JHashMap, Map => JMap, UUID}
import java.util.concurrent.ConcurrentHashMap
import java.util.concurrent.ConcurrentLinkedQueue
import java.util.concurrent.atomic.AtomicLong
@@ -31,13 +31,12 @@ import scala.util.Random
import org.apache.kafka.clients.consumer._
import org.apache.kafka.common.TopicPartition
import org.apache.kafka.common.serialization.StringDeserializer
-import org.scalatest.{BeforeAndAfter, BeforeAndAfterAll}
import org.scalatest.concurrent.Eventually
import org.apache.spark.{SparkConf, SparkFunSuite}
import org.apache.spark.internal.Logging
import org.apache.spark.rdd.RDD
-import org.apache.spark.streaming.{Milliseconds, StreamingContext, Time}
+import org.apache.spark.streaming.{LocalStreamingContext, Milliseconds,
StreamingContext, Time}
import org.apache.spark.streaming.dstream.DStre
[spark] branch master updated (b62ef8f -> c18f849)
This is an automated email from the ASF dual-hosted git repository. vanzin pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from b62ef8f [SPARK-29007][STREAMING][MLLIB][TESTS] Enforce not leaking SparkContext in tests which creates new StreamingContext with new SparkContext add c18f849 [SPARK-24663][STREAMING][TESTS] StreamingContextSuite: Wait until slow receiver has been initialized, but with hard timeout No new revisions were added by this update. Summary of changes: .../org/apache/spark/streaming/StreamingContextSuite.scala | 12 1 file changed, 8 insertions(+), 4 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (c18f849 -> 7ce0f2b)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from c18f849 [SPARK-24663][STREAMING][TESTS] StreamingContextSuite: Wait until slow receiver has been initialized, but with hard timeout add 7ce0f2b [SPARK-29041][PYTHON] Allows createDataFrame to accept bytes as binary type No new revisions were added by this update. Summary of changes: python/pyspark/sql/tests/test_serde.py | 4 python/pyspark/sql/types.py| 2 +- 2 files changed, 5 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (7ce0f2b -> eec728a)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 7ce0f2b [SPARK-29041][PYTHON] Allows createDataFrame to accept bytes as binary type add eec728a [SPARK-29057][SQL] remove InsertIntoTable No new revisions were added by this update. Summary of changes: .../spark/sql/catalyst/analysis/Analyzer.scala | 4 +-- .../sql/catalyst/analysis/CheckAnalysis.scala | 9 +++--- .../plans/logical/basicLogicalOperators.scala | 36 -- .../org/apache/spark/sql/DataFrameWriter.scala | 9 +++--- .../execution/datasources/DataSourceStrategy.scala | 11 --- .../datasources/FallBackFileSourceV2.scala | 7 +++-- .../spark/sql/execution/datasources/rules.scala| 25 +++ .../spark/sql/util/DataFrameCallbackSuite.scala| 7 +++-- .../org/apache/spark/sql/hive/HiveStrategies.scala | 15 - .../org/apache/spark/sql/hive/InsertSuite.scala| 1 - 10 files changed, 46 insertions(+), 78 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [MINOR][DOCS] Fix few typos in the java docs
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 8f632d7 [MINOR][DOCS] Fix few typos in the java docs
8f632d7 is described below
commit 8f632d70455156010f0e87288541304ad2164a52
Author: dengziming
AuthorDate: Thu Sep 12 09:30:03 2019 +0900
[MINOR][DOCS] Fix few typos in the java docs
JIRA :https://issues.apache.org/jira/browse/SPARK-29050
'a hdfs' change into 'an hdfs'
'an unique' change into 'a unique'
'an url' change into 'a url'
'a error' change into 'an error'
Closes #25756 from dengziming/feature_fix_typos.
Authored-by: dengziming
Signed-off-by: HyukjinKwon
---
R/pkg/R/context.R | 4 ++--
core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala | 2 +-
core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala | 2 +-
core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala | 2 +-
core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala| 2 +-
docs/spark-standalone.md | 2 +-
.../org/apache/spark/streaming/kinesis/KinesisCheckpointer.scala | 2 +-
python/pyspark/context.py | 2 +-
.../sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala | 4 ++--
.../scala/org/apache/spark/sql/streaming/StreamingQueryListener.scala | 4 ++--
sql/core/src/main/scala/org/apache/spark/sql/streaming/progress.scala | 2 +-
.../src/test/resources/ql/src/test/queries/clientpositive/load_fs2.q | 2 +-
.../main/scala/org/apache/spark/streaming/dstream/InputDStream.scala | 4 ++--
13 files changed, 17 insertions(+), 17 deletions(-)
diff --git a/R/pkg/R/context.R b/R/pkg/R/context.R
index 51ae2d2..93ba130 100644
--- a/R/pkg/R/context.R
+++ b/R/pkg/R/context.R
@@ -301,7 +301,7 @@ broadcastRDD <- function(sc, object) {
#' Set the checkpoint directory
#'
#' Set the directory under which RDDs are going to be checkpointed. The
-#' directory must be a HDFS path if running on a cluster.
+#' directory must be an HDFS path if running on a cluster.
#'
#' @param sc Spark Context to use
#' @param dirName Directory path
@@ -446,7 +446,7 @@ setLogLevel <- function(level) {
#' Set checkpoint directory
#'
#' Set the directory under which SparkDataFrame are going to be checkpointed.
The directory must be
-#' a HDFS path if running on a cluster.
+#' an HDFS path if running on a cluster.
#'
#' @rdname setCheckpointDir
#' @param directory Directory path to checkpoint to
diff --git
a/core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala
b/core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala
index 330c2f6..3485128 100644
--- a/core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala
+++ b/core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala
@@ -609,7 +609,7 @@ class JavaSparkContext(val sc: SparkContext) extends
Closeable {
/**
* Set the directory under which RDDs are going to be checkpointed. The
directory must
- * be a HDFS path if running on a cluster.
+ * be an HDFS path if running on a cluster.
*/
def setCheckpointDir(dir: String) {
sc.setCheckpointDir(dir)
diff --git a/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala
b/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala
index c96640a..b552444 100644
--- a/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala
+++ b/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala
@@ -124,7 +124,7 @@ private[spark] class MetricsSystem private (
* If either ID is not available, this defaults to just using .
*
* @param source Metric source to be named by this method.
- * @return An unique metric name for each combination of
+ * @return A unique metric name for each combination of
* application, executor/driver and metric source.
*/
private[spark] def buildRegistryName(source: Source): String = {
diff --git a/core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala
b/core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala
index d188bdd..49e32d0 100644
--- a/core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala
+++ b/core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala
@@ -27,7 +27,7 @@ import org.apache.spark.util.Utils
/**
* :: DeveloperApi ::
- * This class represent an unique identifier for a BlockManager.
+ * This class represent a unique identifier for a BlockManager.
*
* The first 2 constructors of this class are made private to ensure that
BlockManagerId objects
* can be created only using the apply method in the companion object. This
allows de-duplication
diff --git a/core/src/test/scala/org/ap
[spark] branch branch-2.4 updated: [MINOR][DOCS] Fix few typos in the java docs
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-2.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-2.4 by this push:
new ecb2052 [MINOR][DOCS] Fix few typos in the java docs
ecb2052 is described below
commit ecb2052bf0cf7dea749cca10d864f7383eeb1224
Author: dengziming
AuthorDate: Thu Sep 12 09:30:03 2019 +0900
[MINOR][DOCS] Fix few typos in the java docs
JIRA :https://issues.apache.org/jira/browse/SPARK-29050
'a hdfs' change into 'an hdfs'
'an unique' change into 'a unique'
'an url' change into 'a url'
'a error' change into 'an error'
Closes #25756 from dengziming/feature_fix_typos.
Authored-by: dengziming
Signed-off-by: HyukjinKwon
(cherry picked from commit 8f632d70455156010f0e87288541304ad2164a52)
Signed-off-by: HyukjinKwon
---
R/pkg/R/context.R | 4 ++--
core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala | 2 +-
core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala | 2 +-
core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala | 2 +-
core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala| 2 +-
docs/spark-standalone.md | 2 +-
.../org/apache/spark/streaming/kinesis/KinesisCheckpointer.scala | 2 +-
python/pyspark/context.py | 2 +-
.../sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala | 4 ++--
.../scala/org/apache/spark/sql/streaming/StreamingQueryListener.scala | 4 ++--
sql/core/src/main/scala/org/apache/spark/sql/streaming/progress.scala | 2 +-
.../src/test/resources/ql/src/test/queries/clientpositive/load_fs2.q | 2 +-
.../main/scala/org/apache/spark/streaming/dstream/InputDStream.scala | 4 ++--
13 files changed, 17 insertions(+), 17 deletions(-)
diff --git a/R/pkg/R/context.R b/R/pkg/R/context.R
index b49f7c3..f1a6b84 100644
--- a/R/pkg/R/context.R
+++ b/R/pkg/R/context.R
@@ -297,7 +297,7 @@ broadcastRDD <- function(sc, object) {
#' Set the checkpoint directory
#'
#' Set the directory under which RDDs are going to be checkpointed. The
-#' directory must be a HDFS path if running on a cluster.
+#' directory must be an HDFS path if running on a cluster.
#'
#' @param sc Spark Context to use
#' @param dirName Directory path
@@ -442,7 +442,7 @@ setLogLevel <- function(level) {
#' Set checkpoint directory
#'
#' Set the directory under which SparkDataFrame are going to be checkpointed.
The directory must be
-#' a HDFS path if running on a cluster.
+#' an HDFS path if running on a cluster.
#'
#' @rdname setCheckpointDir
#' @param directory Directory path to checkpoint to
diff --git
a/core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala
b/core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala
index 09c8384..09e9910 100644
--- a/core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala
+++ b/core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala
@@ -713,7 +713,7 @@ class JavaSparkContext(val sc: SparkContext)
/**
* Set the directory under which RDDs are going to be checkpointed. The
directory must
- * be a HDFS path if running on a cluster.
+ * be an HDFS path if running on a cluster.
*/
def setCheckpointDir(dir: String) {
sc.setCheckpointDir(dir)
diff --git a/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala
b/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala
index 3457a26..657d75c 100644
--- a/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala
+++ b/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala
@@ -122,7 +122,7 @@ private[spark] class MetricsSystem private (
* If either ID is not available, this defaults to just using .
*
* @param source Metric source to be named by this method.
- * @return An unique metric name for each combination of
+ * @return A unique metric name for each combination of
* application, executor/driver and metric source.
*/
private[spark] def buildRegistryName(source: Source): String = {
diff --git a/core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala
b/core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala
index d4a59c3..83cd4f0 100644
--- a/core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala
+++ b/core/src/main/scala/org/apache/spark/storage/BlockManagerId.scala
@@ -27,7 +27,7 @@ import org.apache.spark.util.Utils
/**
* :: DeveloperApi ::
- * This class represent an unique identifier for a BlockManager.
+ * This class represent a unique identifier for a BlockManager.
*
* The first 2 constructors of this class are made private to ensure that
BlockManagerId objects
* can be created only using the apply method
[spark] branch master updated (8f632d7 -> 850833f)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 8f632d7 [MINOR][DOCS] Fix few typos in the java docs add 850833f [SPARK-29046][SQL] Fix NPE in SQLConf.get when active SparkContext is stopping No new revisions were added by this update. Summary of changes: .../scala/org/apache/spark/sql/internal/SQLConf.scala | 3 ++- .../org/apache/spark/sql/internal/SQLConfSuite.scala | 19 +++ 2 files changed, 21 insertions(+), 1 deletion(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (850833f -> 6768431)
This is an automated email from the ASF dual-hosted git repository. yumwang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 850833f [SPARK-29046][SQL] Fix NPE in SQLConf.get when active SparkContext is stopping add 6768431 [SPARK-29045][SQL][TESTS] Drop table to avoid test failure in SQLMetricsSuite No new revisions were added by this update. Summary of changes: .../sql/execution/metric/SQLMetricsTestUtils.scala | 46 +++--- 1 file changed, 24 insertions(+), 22 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
