spark git commit: [SPARK-6175] Fix standalone executor log links when ephemeral ports or SPARK_PUBLIC_DNS are used
Repository: spark
Updated Branches:
refs/heads/master 0bfacd5c5 - 424a86a1e
[SPARK-6175] Fix standalone executor log links when ephemeral ports or
SPARK_PUBLIC_DNS are used
This patch fixes two issues with the executor log viewing links added in Spark
1.3. In standalone mode, the log URLs might include a port value of 0 rather
than the actual bound port of the UI, which broke the ability to view logs from
workers whose web UIs had been configured to bind to ephemeral ports. In
addition, the URLs used workers' local hostnames instead of respecting
SPARK_PUBLIC_DNS, which prevented this feature from working properly on Spark
EC2 clusters because the links would point to internal DNS names instead of
external ones.
I included tests for both of these bugs:
- We now browse to the URLs and verify that they point to the expected pages.
- To test SPARK_PUBLIC_DNS, I changed the code that reads the environment
variable to do so via `SparkConf.getenv`, then used a custom SparkConf subclass
to mock the environment variable (this pattern is used elsewhere in Spark's
tests).
Author: Josh Rosen joshro...@databricks.com
Closes #4903 from JoshRosen/SPARK-6175 and squashes the following commits:
5577f41 [Josh Rosen] Remove println
cfec135 [Josh Rosen] Use webUi.boundPort and publicAddress in log links
27918c7 [Josh Rosen] Add failing unit tests for standalone log URL viewing
c250fbe [Josh Rosen] Respect SparkConf in local-cluster Workers.
422a2ef [Josh Rosen] Use conf.getenv to read SPARK_PUBLIC_DNS
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/424a86a1
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/424a86a1
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/424a86a1
Branch: refs/heads/master
Commit: 424a86a1ed2a3e6dd54cf8b09fe2f13a1311b7e6
Parents: 0bfacd5
Author: Josh Rosen joshro...@databricks.com
Authored: Thu Mar 5 12:04:00 2015 -0800
Committer: Josh Rosen joshro...@databricks.com
Committed: Thu Mar 5 12:04:00 2015 -0800
--
.../apache/spark/deploy/LocalSparkCluster.scala | 2 +-
.../org/apache/spark/deploy/master/Master.scala | 2 +-
.../spark/deploy/worker/ExecutorRunner.scala| 4 +-
.../org/apache/spark/deploy/worker/Worker.scala | 9 ++--
.../main/scala/org/apache/spark/ui/WebUI.scala | 2 +-
.../apache/spark/deploy/JsonProtocolSuite.scala | 2 +-
.../spark/deploy/LogUrlsStandaloneSuite.scala | 54
.../deploy/worker/ExecutorRunnerTest.scala | 2 +-
8 files changed, 57 insertions(+), 20 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/424a86a1/core/src/main/scala/org/apache/spark/deploy/LocalSparkCluster.scala
--
diff --git
a/core/src/main/scala/org/apache/spark/deploy/LocalSparkCluster.scala
b/core/src/main/scala/org/apache/spark/deploy/LocalSparkCluster.scala
index 0401b15..3ab425a 100644
--- a/core/src/main/scala/org/apache/spark/deploy/LocalSparkCluster.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/LocalSparkCluster.scala
@@ -59,7 +59,7 @@ class LocalSparkCluster(
/* Start the Workers */
for (workerNum - 1 to numWorkers) {
val (workerSystem, _) = Worker.startSystemAndActor(localHostname, 0, 0,
coresPerWorker,
-memoryPerWorker, masters, null, Some(workerNum))
+memoryPerWorker, masters, null, Some(workerNum), _conf)
workerActorSystems += workerSystem
}
http://git-wip-us.apache.org/repos/asf/spark/blob/424a86a1/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
--
diff --git a/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
b/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
index 4584b73..1581429 100644
--- a/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
@@ -96,7 +96,7 @@ private[spark] class Master(
val webUi = new MasterWebUI(this, webUiPort)
val masterPublicAddress = {
-val envVar = System.getenv(SPARK_PUBLIC_DNS)
+val envVar = conf.getenv(SPARK_PUBLIC_DNS)
if (envVar != null) envVar else host
}
http://git-wip-us.apache.org/repos/asf/spark/blob/424a86a1/core/src/main/scala/org/apache/spark/deploy/worker/ExecutorRunner.scala
--
diff --git
a/core/src/main/scala/org/apache/spark/deploy/worker/ExecutorRunner.scala
b/core/src/main/scala/org/apache/spark/deploy/worker/ExecutorRunner.scala
index 066d46c..023f3c6 100644
--- a/core/src/main/scala/org/apache/spark/deploy/worker/ExecutorRunner.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/worker/ExecutorRunner.scala
@@ -44,6 +44,7 @@
spark git commit: SPARK-6182 [BUILD] spark-parent pom needs to be published for both 2.10 and 2.11
Repository: spark Updated Branches: refs/heads/master e06c7dfbc - c9cfba0ce SPARK-6182 [BUILD] spark-parent pom needs to be published for both 2.10 and 2.11 Option 1 of 2: Convert spark-parent module name to spark-parent_2.10 / spark-parent_2.11 Author: Sean Owen so...@cloudera.com Closes #4912 from srowen/SPARK-6182.1 and squashes the following commits: eff60de [Sean Owen] Convert spark-parent module name to spark-parent_2.10 / spark-parent_2.11 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c9cfba0c Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c9cfba0c Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c9cfba0c Branch: refs/heads/master Commit: c9cfba0cebe3eb546e3e96f3e5b9b89a74c5b7de Parents: e06c7df Author: Sean Owen so...@cloudera.com Authored: Thu Mar 5 11:31:48 2015 -0800 Committer: Patrick Wendell patr...@databricks.com Committed: Thu Mar 5 11:31:48 2015 -0800 -- assembly/pom.xml | 2 +- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume-sink/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka-assembly/pom.xml | 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/java8-tests/pom.xml| 2 +- extras/kinesis-asl/pom.xml| 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- network/common/pom.xml| 2 +- network/shuffle/pom.xml | 2 +- network/yarn/pom.xml | 2 +- pom.xml | 2 +- repl/pom.xml | 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive-thriftserver/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- 28 files changed, 28 insertions(+), 28 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/c9cfba0c/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index 3d1ed0d..cbf5b6c 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -20,7 +20,7 @@ modelVersion4.0.0/modelVersion parent groupIdorg.apache.spark/groupId -artifactIdspark-parent/artifactId +artifactIdspark-parent_2.10/artifactId version1.3.0-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/c9cfba0c/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index 510e926..1fe6106 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -20,7 +20,7 @@ modelVersion4.0.0/modelVersion parent groupIdorg.apache.spark/groupId -artifactIdspark-parent/artifactId +artifactIdspark-parent_2.10/artifactId version1.3.0-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/c9cfba0c/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index c993781..fab776d 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -20,7 +20,7 @@ modelVersion4.0.0/modelVersion parent groupIdorg.apache.spark/groupId -artifactIdspark-parent/artifactId +artifactIdspark-parent_2.10/artifactId version1.3.0-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/c9cfba0c/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index 8caad2b..994071d 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -20,7 +20,7 @@ modelVersion4.0.0/modelVersion parent groupIdorg.apache.spark/groupId -artifactIdspark-parent/artifactId +artifactIdspark-parent_2.10/artifactId version1.3.0-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/c9cfba0c/external/flume-sink/pom.xml -- diff --git a/external/flume-sink/pom.xml b/external/flume-sink/pom.xml index 0706f1e..96c2787 100644 --- a/external/flume-sink/pom.xml +++ b/external/flume-sink/pom.xml @@ -20,7 +20,7 @@ modelVersion4.0.0/modelVersion parent groupIdorg.apache.spark/groupId -artifactIdspark-parent/artifactId +artifactIdspark-parent_2.10/artifactId
spark git commit: [SPARK-6175] Fix standalone executor log links when ephemeral ports or SPARK_PUBLIC_DNS are used
Repository: spark
Updated Branches:
refs/heads/branch-1.3 ae315d2b8 - 988b49821
[SPARK-6175] Fix standalone executor log links when ephemeral ports or
SPARK_PUBLIC_DNS are used
This patch fixes two issues with the executor log viewing links added in Spark
1.3. In standalone mode, the log URLs might include a port value of 0 rather
than the actual bound port of the UI, which broke the ability to view logs from
workers whose web UIs had been configured to bind to ephemeral ports. In
addition, the URLs used workers' local hostnames instead of respecting
SPARK_PUBLIC_DNS, which prevented this feature from working properly on Spark
EC2 clusters because the links would point to internal DNS names instead of
external ones.
I included tests for both of these bugs:
- We now browse to the URLs and verify that they point to the expected pages.
- To test SPARK_PUBLIC_DNS, I changed the code that reads the environment
variable to do so via `SparkConf.getenv`, then used a custom SparkConf subclass
to mock the environment variable (this pattern is used elsewhere in Spark's
tests).
Author: Josh Rosen joshro...@databricks.com
Closes #4903 from JoshRosen/SPARK-6175 and squashes the following commits:
5577f41 [Josh Rosen] Remove println
cfec135 [Josh Rosen] Use webUi.boundPort and publicAddress in log links
27918c7 [Josh Rosen] Add failing unit tests for standalone log URL viewing
c250fbe [Josh Rosen] Respect SparkConf in local-cluster Workers.
422a2ef [Josh Rosen] Use conf.getenv to read SPARK_PUBLIC_DNS
(cherry picked from commit 424a86a1ed2a3e6dd54cf8b09fe2f13a1311b7e6)
Signed-off-by: Josh Rosen joshro...@databricks.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/988b4982
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/988b4982
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/988b4982
Branch: refs/heads/branch-1.3
Commit: 988b498218c963d44790ac74e643ce76d02fce0f
Parents: ae315d2
Author: Josh Rosen joshro...@databricks.com
Authored: Thu Mar 5 12:04:00 2015 -0800
Committer: Josh Rosen joshro...@databricks.com
Committed: Thu Mar 5 12:04:14 2015 -0800
--
.../apache/spark/deploy/LocalSparkCluster.scala | 2 +-
.../org/apache/spark/deploy/master/Master.scala | 2 +-
.../spark/deploy/worker/ExecutorRunner.scala| 4 +-
.../org/apache/spark/deploy/worker/Worker.scala | 9 ++--
.../main/scala/org/apache/spark/ui/WebUI.scala | 2 +-
.../apache/spark/deploy/JsonProtocolSuite.scala | 2 +-
.../spark/deploy/LogUrlsStandaloneSuite.scala | 54
.../deploy/worker/ExecutorRunnerTest.scala | 2 +-
8 files changed, 57 insertions(+), 20 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/988b4982/core/src/main/scala/org/apache/spark/deploy/LocalSparkCluster.scala
--
diff --git
a/core/src/main/scala/org/apache/spark/deploy/LocalSparkCluster.scala
b/core/src/main/scala/org/apache/spark/deploy/LocalSparkCluster.scala
index 0401b15..3ab425a 100644
--- a/core/src/main/scala/org/apache/spark/deploy/LocalSparkCluster.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/LocalSparkCluster.scala
@@ -59,7 +59,7 @@ class LocalSparkCluster(
/* Start the Workers */
for (workerNum - 1 to numWorkers) {
val (workerSystem, _) = Worker.startSystemAndActor(localHostname, 0, 0,
coresPerWorker,
-memoryPerWorker, masters, null, Some(workerNum))
+memoryPerWorker, masters, null, Some(workerNum), _conf)
workerActorSystems += workerSystem
}
http://git-wip-us.apache.org/repos/asf/spark/blob/988b4982/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
--
diff --git a/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
b/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
index 148485c..259888f 100644
--- a/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
+++ b/core/src/main/scala/org/apache/spark/deploy/master/Master.scala
@@ -96,7 +96,7 @@ private[spark] class Master(
val webUi = new MasterWebUI(this, webUiPort)
val masterPublicAddress = {
-val envVar = System.getenv(SPARK_PUBLIC_DNS)
+val envVar = conf.getenv(SPARK_PUBLIC_DNS)
if (envVar != null) envVar else host
}
http://git-wip-us.apache.org/repos/asf/spark/blob/988b4982/core/src/main/scala/org/apache/spark/deploy/worker/ExecutorRunner.scala
--
diff --git
a/core/src/main/scala/org/apache/spark/deploy/worker/ExecutorRunner.scala
b/core/src/main/scala/org/apache/spark/deploy/worker/ExecutorRunner.scala
index bea04cd..2ec10f8 100644
---
spark git commit: SPARK-6182 [BUILD] spark-parent pom needs to be published for both 2.10 and 2.11
Repository: spark Updated Branches: refs/heads/branch-1.3 f8205d3be - ae315d2b8 SPARK-6182 [BUILD] spark-parent pom needs to be published for both 2.10 and 2.11 Option 1 of 2: Convert spark-parent module name to spark-parent_2.10 / spark-parent_2.11 Author: Sean Owen so...@cloudera.com Closes #4912 from srowen/SPARK-6182.1 and squashes the following commits: eff60de [Sean Owen] Convert spark-parent module name to spark-parent_2.10 / spark-parent_2.11 (cherry picked from commit c9cfba0cebe3eb546e3e96f3e5b9b89a74c5b7de) Signed-off-by: Patrick Wendell patr...@databricks.com Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/ae315d2b Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/ae315d2b Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/ae315d2b Branch: refs/heads/branch-1.3 Commit: ae315d2b86e217fddcc18c0b43e3028fe590bbbe Parents: f8205d3 Author: Sean Owen so...@cloudera.com Authored: Thu Mar 5 11:31:48 2015 -0800 Committer: Patrick Wendell patr...@databricks.com Committed: Thu Mar 5 11:32:06 2015 -0800 -- assembly/pom.xml | 2 +- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume-sink/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka-assembly/pom.xml | 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/java8-tests/pom.xml| 2 +- extras/kinesis-asl/pom.xml| 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- network/common/pom.xml| 2 +- network/shuffle/pom.xml | 2 +- network/yarn/pom.xml | 2 +- pom.xml | 2 +- repl/pom.xml | 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive-thriftserver/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- 28 files changed, 28 insertions(+), 28 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/ae315d2b/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index c1bcdbb..1215fba 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -20,7 +20,7 @@ modelVersion4.0.0/modelVersion parent groupIdorg.apache.spark/groupId -artifactIdspark-parent/artifactId +artifactIdspark-parent_2.10/artifactId version1.3.0-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/ae315d2b/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index 510e926..1fe6106 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -20,7 +20,7 @@ modelVersion4.0.0/modelVersion parent groupIdorg.apache.spark/groupId -artifactIdspark-parent/artifactId +artifactIdspark-parent_2.10/artifactId version1.3.0-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/ae315d2b/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index c993781..fab776d 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -20,7 +20,7 @@ modelVersion4.0.0/modelVersion parent groupIdorg.apache.spark/groupId -artifactIdspark-parent/artifactId +artifactIdspark-parent_2.10/artifactId version1.3.0-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/ae315d2b/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index 8caad2b..994071d 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -20,7 +20,7 @@ modelVersion4.0.0/modelVersion parent groupIdorg.apache.spark/groupId -artifactIdspark-parent/artifactId +artifactIdspark-parent_2.10/artifactId version1.3.0-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/ae315d2b/external/flume-sink/pom.xml -- diff --git a/external/flume-sink/pom.xml b/external/flume-sink/pom.xml index 0706f1e..96c2787 100644 --- a/external/flume-sink/pom.xml +++ b/external/flume-sink/pom.xml @@ -20,7 +20,7 @@
spark git commit: [SPARK-6090][MLLIB] add a basic BinaryClassificationMetrics to PySpark/MLlib
Repository: spark
Updated Branches:
refs/heads/master c9cfba0ce - 0bfacd5c5
[SPARK-6090][MLLIB] add a basic BinaryClassificationMetrics to PySpark/MLlib
A simple wrapper around the Scala implementation. `DataFrame` is used for
serialization/deserialization. Methods that return `RDD`s are not supported in
this PR.
davies If we recognize Scala's `Product`s in Py4J, we can easily add wrappers
for Scala methods that returns `RDD[(Double, Double)]`. Is it easy to register
serializer for `Product` in PySpark?
Author: Xiangrui Meng m...@databricks.com
Closes #4863 from mengxr/SPARK-6090 and squashes the following commits:
009a3a3 [Xiangrui Meng] provide schema
dcddab5 [Xiangrui Meng] add a basic BinaryClassificationMetrics to PySpark/MLlib
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/0bfacd5c
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/0bfacd5c
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/0bfacd5c
Branch: refs/heads/master
Commit: 0bfacd5c5dd7d10a69bcbcbda630f0843d1cf285
Parents: c9cfba0
Author: Xiangrui Meng m...@databricks.com
Authored: Thu Mar 5 11:50:09 2015 -0800
Committer: Xiangrui Meng m...@databricks.com
Committed: Thu Mar 5 11:50:09 2015 -0800
--
.../BinaryClassificationMetrics.scala | 8 ++
python/docs/pyspark.mllib.rst | 7 ++
python/pyspark/mllib/evaluation.py | 83
python/run-tests| 1 +
4 files changed, 99 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/0bfacd5c/mllib/src/main/scala/org/apache/spark/mllib/evaluation/BinaryClassificationMetrics.scala
--
diff --git
a/mllib/src/main/scala/org/apache/spark/mllib/evaluation/BinaryClassificationMetrics.scala
b/mllib/src/main/scala/org/apache/spark/mllib/evaluation/BinaryClassificationMetrics.scala
index ced042e..c1d1a22 100644
---
a/mllib/src/main/scala/org/apache/spark/mllib/evaluation/BinaryClassificationMetrics.scala
+++
b/mllib/src/main/scala/org/apache/spark/mllib/evaluation/BinaryClassificationMetrics.scala
@@ -22,6 +22,7 @@ import org.apache.spark.Logging
import org.apache.spark.SparkContext._
import org.apache.spark.mllib.evaluation.binary._
import org.apache.spark.rdd.{RDD, UnionRDD}
+import org.apache.spark.sql.DataFrame
/**
* :: Experimental ::
@@ -53,6 +54,13 @@ class BinaryClassificationMetrics(
*/
def this(scoreAndLabels: RDD[(Double, Double)]) = this(scoreAndLabels, 0)
+ /**
+ * An auxiliary constructor taking a DataFrame.
+ * @param scoreAndLabels a DataFrame with two double columns: score and label
+ */
+ private[mllib] def this(scoreAndLabels: DataFrame) =
+this(scoreAndLabels.map(r = (r.getDouble(0), r.getDouble(1
+
/** Unpersist intermediate RDDs used in the computation. */
def unpersist() {
cumulativeCounts.unpersist()
http://git-wip-us.apache.org/repos/asf/spark/blob/0bfacd5c/python/docs/pyspark.mllib.rst
--
diff --git a/python/docs/pyspark.mllib.rst b/python/docs/pyspark.mllib.rst
index b706c5e..1510147 100644
--- a/python/docs/pyspark.mllib.rst
+++ b/python/docs/pyspark.mllib.rst
@@ -16,6 +16,13 @@ pyspark.mllib.clustering module
:members:
:undoc-members:
+pyspark.mllib.evaluation module
+---
+
+.. automodule:: pyspark.mllib.evaluation
+ :members:
+ :undoc-members:
+
pyspark.mllib.feature module
---
http://git-wip-us.apache.org/repos/asf/spark/blob/0bfacd5c/python/pyspark/mllib/evaluation.py
--
diff --git a/python/pyspark/mllib/evaluation.py
b/python/pyspark/mllib/evaluation.py
new file mode 100644
index 000..16cb49c
--- /dev/null
+++ b/python/pyspark/mllib/evaluation.py
@@ -0,0 +1,83 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the License); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an AS IS BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+from pyspark.mllib.common import JavaModelWrapper
+from
Git Push Summary
Repository: spark Updated Tags: refs/tags/v1.3.0-rc3 [deleted] 6fb4af2fb - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: Updating CHANGES.txt for Spark 1.3
Repository: spark Updated Branches: refs/heads/branch-1.3 a0cbfe4da - 4ab990c2f Updating CHANGES.txt for Spark 1.3 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/4ab990c2 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/4ab990c2 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/4ab990c2 Branch: refs/heads/branch-1.3 Commit: 4ab990c2f1098720526010b45a86c1b00852e869 Parents: a0cbfe4 Author: Patrick Wendell patr...@databricks.com Authored: Thu Mar 5 15:00:48 2015 -0800 Committer: Patrick Wendell patr...@databricks.com Committed: Thu Mar 5 15:00:48 2015 -0800 -- CHANGES.txt | 40 1 file changed, 40 insertions(+) -- http://git-wip-us.apache.org/repos/asf/spark/blob/4ab990c2/CHANGES.txt -- diff --git a/CHANGES.txt b/CHANGES.txt index 81e8431..d3713de 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -3,6 +3,46 @@ Spark Change Log Release 1.3.0 + [SQL] Make Strategies a public developer API + Michael Armbrust mich...@databricks.com + 2015-03-05 14:50:25 -0800 + Commit: 556e0de, github.com/apache/spark/pull/4920 + + [SPARK-6163][SQL] jsonFile should be backed by the data source API + Yin Huai yh...@databricks.com + 2015-03-05 14:49:44 -0800 + Commit: 083fed5, github.com/apache/spark/pull/4896 + + [SPARK-6145][SQL] fix ORDER BY on nested fields + Wenchen Fan cloud0...@outlook.com, Michael Armbrust mich...@databricks.com + 2015-03-05 14:49:01 -0800 + Commit: e358f55, github.com/apache/spark/pull/4918 + + [SPARK-6175] Fix standalone executor log links when ephemeral ports or SPARK_PUBLIC_DNS are used + Josh Rosen joshro...@databricks.com + 2015-03-05 12:04:00 -0800 + Commit: 988b498, github.com/apache/spark/pull/4903 + + SPARK-6182 [BUILD] spark-parent pom needs to be published for both 2.10 and 2.11 + Sean Owen so...@cloudera.com + 2015-03-05 11:31:48 -0800 + Commit: ae315d2, github.com/apache/spark/pull/4912 + + Revert [SPARK-6153] [SQL] promote guava dep for hive-thriftserver + Cheng Lian l...@databricks.com + 2015-03-05 17:58:18 +0800 + Commit: f8205d3 + + [SPARK-6153] [SQL] promote guava dep for hive-thriftserver + Daoyuan Wang daoyuan.w...@intel.com + 2015-03-05 16:35:17 +0800 + Commit: b92d925, github.com/apache/spark/pull/4884 + + Updating CHANGES file + Patrick Wendell patr...@databricks.com + 2015-03-04 21:19:49 -0800 + Commit: 87eac3c + SPARK-5143 [BUILD] [WIP] spark-network-yarn 2.11 depends on spark-network-shuffle 2.10 Sean Owen so...@cloudera.com 2015-03-04 21:00:51 -0800 - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SQL] Make Strategies a public developer API
Repository: spark
Updated Branches:
refs/heads/branch-1.3 083fed555 - 556e0de1a
[SQL] Make Strategies a public developer API
Author: Michael Armbrust mich...@databricks.com
Closes #4920 from marmbrus/openStrategies and squashes the following commits:
cbc35c0 [Michael Armbrust] [SQL] Make Strategies a public developer API
(cherry picked from commit eb48fd6e9d55fb034c00e61374bb9c2a86a82fb8)
Signed-off-by: Michael Armbrust mich...@databricks.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/556e0de1
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/556e0de1
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/556e0de1
Branch: refs/heads/branch-1.3
Commit: 556e0de1aded8d0b0606063e4e66ee144bcdb5bc
Parents: 083fed5
Author: Michael Armbrust mich...@databricks.com
Authored: Thu Mar 5 14:50:25 2015 -0800
Committer: Michael Armbrust mich...@databricks.com
Committed: Thu Mar 5 14:50:58 2015 -0800
--
sql/core/src/main/scala/org/apache/spark/sql/package.scala | 7 +--
1 file changed, 5 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/556e0de1/sql/core/src/main/scala/org/apache/spark/sql/package.scala
--
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/package.scala
b/sql/core/src/main/scala/org/apache/spark/sql/package.scala
index 02e5b01..3f97a11 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/package.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/package.scala
@@ -34,10 +34,13 @@ import org.apache.spark.sql.execution.SparkPlan
package object sql {
/**
- * Converts a logical plan into zero or more SparkPlans.
+ * Converts a logical plan into zero or more SparkPlans. This API is
exposed for experimenting
+ * with the query planner and is not designed to be stable across spark
releases. Developers
+ * writing libraries should instead consider using the stable APIs provided
in
+ * [[org.apache.spark.sql.sources]]
*/
@DeveloperApi
- protected[sql] type Strategy =
org.apache.spark.sql.catalyst.planning.GenericStrategy[SparkPlan]
+ type Strategy =
org.apache.spark.sql.catalyst.planning.GenericStrategy[SparkPlan]
/**
* Type alias for [[DataFrame]]. Kept here for backward source compatibility
for Scala.
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[2/2] spark git commit: Preparing development version 1.3.1-SNAPSHOT
Preparing development version 1.3.1-SNAPSHOT Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c152f9a7 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c152f9a7 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c152f9a7 Branch: refs/heads/branch-1.3 Commit: c152f9a7ed82328747672c2c7017353da32630c7 Parents: 4aaf48d Author: Patrick Wendell patr...@databricks.com Authored: Thu Mar 5 23:02:08 2015 + Committer: Patrick Wendell patr...@databricks.com Committed: Thu Mar 5 23:02:08 2015 + -- assembly/pom.xml | 2 +- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume-sink/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka-assembly/pom.xml | 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/java8-tests/pom.xml| 2 +- extras/kinesis-asl/pom.xml| 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- network/common/pom.xml| 2 +- network/shuffle/pom.xml | 2 +- network/yarn/pom.xml | 2 +- pom.xml | 2 +- repl/pom.xml | 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive-thriftserver/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- 28 files changed, 28 insertions(+), 28 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/c152f9a7/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index 71d80fb..114dde7 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0/version +version1.3.1-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/c152f9a7/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index 538788a..dea41f8 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0/version +version1.3.1-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/c152f9a7/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index a239e00..ad8a582 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0/version +version1.3.1-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/c152f9a7/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index 97356c0..73ab234 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0/version +version1.3.1-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/c152f9a7/external/flume-sink/pom.xml -- diff --git a/external/flume-sink/pom.xml b/external/flume-sink/pom.xml index b7bcaf3..1a5aaf5 100644 --- a/external/flume-sink/pom.xml +++ b/external/flume-sink/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0/version +version1.3.1-SNAPSHOT/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/c152f9a7/external/flume/pom.xml -- diff --git a/external/flume/pom.xml b/external/flume/pom.xml index 7b65ea3..d5539d9 100644 --- a/external/flume/pom.xml +++ b/external/flume/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0/version +version1.3.1-SNAPSHOT/version relativePath../../pom.xml/relativePath /parent
[2/2] spark git commit: Revert Preparing Spark release v1.3.0-rc3
Revert Preparing Spark release v1.3.0-rc3 This reverts commit 6fb4af2fbeb3d1b888191a2fa1042c80e3ef2d60. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/a0cbfe4d Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/a0cbfe4d Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/a0cbfe4d Branch: refs/heads/branch-1.3 Commit: a0cbfe4da1cff5cea990a917a21e5eaaa2c725aa Parents: d6b9dce Author: Patrick Wendell patr...@databricks.com Authored: Thu Mar 5 14:56:25 2015 -0800 Committer: Patrick Wendell patr...@databricks.com Committed: Thu Mar 5 14:56:25 2015 -0800 -- assembly/pom.xml | 2 +- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume-sink/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka-assembly/pom.xml | 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/java8-tests/pom.xml| 2 +- extras/kinesis-asl/pom.xml| 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- network/common/pom.xml| 2 +- network/shuffle/pom.xml | 2 +- network/yarn/pom.xml | 2 +- pom.xml | 2 +- repl/pom.xml | 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive-thriftserver/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- 28 files changed, 28 insertions(+), 28 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/a0cbfe4d/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index 71d80fb..1215fba 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0/version +version1.3.0-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/a0cbfe4d/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index 538788a..1fe6106 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0/version +version1.3.0-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/a0cbfe4d/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index a239e00..fab776d 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0/version +version1.3.0-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/a0cbfe4d/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index 97356c0..994071d 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0/version +version1.3.0-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/a0cbfe4d/external/flume-sink/pom.xml -- diff --git a/external/flume-sink/pom.xml b/external/flume-sink/pom.xml index b7bcaf3..96c2787 100644 --- a/external/flume-sink/pom.xml +++ b/external/flume-sink/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0/version +version1.3.0-SNAPSHOT/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/a0cbfe4d/external/flume/pom.xml -- diff --git a/external/flume/pom.xml b/external/flume/pom.xml index 7b65ea3..172d447 100644 --- a/external/flume/pom.xml +++ b/external/flume/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0/version +version1.3.0-SNAPSHOT/version
Git Push Summary
Repository: spark Updated Tags: refs/tags/v1.3.0-rc3 [created] 6fb4af2fb - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[2/2] spark git commit: Preparing development version 1.3.1-SNAPSHOT
Preparing development version 1.3.1-SNAPSHOT Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5097f869 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/5097f869 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/5097f869 Branch: refs/heads/branch-1.3 Commit: 5097f869efbdb75d3b87bcbd8e621e7c12356942 Parents: 6fb4af2 Author: Patrick Wendell patr...@databricks.com Authored: Thu Mar 5 22:40:05 2015 + Committer: Patrick Wendell patr...@databricks.com Committed: Thu Mar 5 22:40:05 2015 + -- assembly/pom.xml | 2 +- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume-sink/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka-assembly/pom.xml | 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/java8-tests/pom.xml| 2 +- extras/kinesis-asl/pom.xml| 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- network/common/pom.xml| 2 +- network/shuffle/pom.xml | 2 +- network/yarn/pom.xml | 2 +- pom.xml | 2 +- repl/pom.xml | 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive-thriftserver/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- 28 files changed, 28 insertions(+), 28 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/5097f869/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index 71d80fb..114dde7 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0/version +version1.3.1-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/5097f869/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index 538788a..dea41f8 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0/version +version1.3.1-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/5097f869/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index a239e00..ad8a582 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0/version +version1.3.1-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/5097f869/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index 97356c0..73ab234 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0/version +version1.3.1-SNAPSHOT/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/5097f869/external/flume-sink/pom.xml -- diff --git a/external/flume-sink/pom.xml b/external/flume-sink/pom.xml index b7bcaf3..1a5aaf5 100644 --- a/external/flume-sink/pom.xml +++ b/external/flume-sink/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0/version +version1.3.1-SNAPSHOT/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/5097f869/external/flume/pom.xml -- diff --git a/external/flume/pom.xml b/external/flume/pom.xml index 7b65ea3..d5539d9 100644 --- a/external/flume/pom.xml +++ b/external/flume/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0/version +version1.3.1-SNAPSHOT/version relativePath../../pom.xml/relativePath /parent
[1/2] spark git commit: Preparing Spark release v1.3.0-rc3
Repository: spark Updated Branches: refs/heads/branch-1.3 988b49821 - 5097f869e Preparing Spark release v1.3.0-rc3 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6fb4af2f Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6fb4af2f Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/6fb4af2f Branch: refs/heads/branch-1.3 Commit: 6fb4af2fbeb3d1b888191a2fa1042c80e3ef2d60 Parents: 988b498 Author: Patrick Wendell patr...@databricks.com Authored: Thu Mar 5 22:40:04 2015 + Committer: Patrick Wendell patr...@databricks.com Committed: Thu Mar 5 22:40:04 2015 + -- assembly/pom.xml | 2 +- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume-sink/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka-assembly/pom.xml | 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/java8-tests/pom.xml| 2 +- extras/kinesis-asl/pom.xml| 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- network/common/pom.xml| 2 +- network/shuffle/pom.xml | 2 +- network/yarn/pom.xml | 2 +- pom.xml | 2 +- repl/pom.xml | 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive-thriftserver/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- 28 files changed, 28 insertions(+), 28 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/6fb4af2f/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index 1215fba..71d80fb 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0-SNAPSHOT/version +version1.3.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/6fb4af2f/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index 1fe6106..538788a 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0-SNAPSHOT/version +version1.3.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/6fb4af2f/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index fab776d..a239e00 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0-SNAPSHOT/version +version1.3.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/6fb4af2f/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index 994071d..97356c0 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0-SNAPSHOT/version +version1.3.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/6fb4af2f/external/flume-sink/pom.xml -- diff --git a/external/flume-sink/pom.xml b/external/flume-sink/pom.xml index 96c2787..b7bcaf3 100644 --- a/external/flume-sink/pom.xml +++ b/external/flume-sink/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0-SNAPSHOT/version +version1.3.0/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/6fb4af2f/external/flume/pom.xml -- diff --git a/external/flume/pom.xml b/external/flume/pom.xml index 172d447..7b65ea3 100644 --- a/external/flume/pom.xml +++ b/external/flume/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0-SNAPSHOT/version +
spark git commit: [SPARK-6145][SQL] fix ORDER BY on nested fields
Repository: spark
Updated Branches:
refs/heads/branch-1.3 5097f869e - e358f
[SPARK-6145][SQL] fix ORDER BY on nested fields
Based on #4904 with style errors fixed.
`LogicalPlan#resolve` will not only produce `Attribute`, but also `GetField`
chain.
So in `ResolveSortReferences`, after resolve the ordering expressions, we
should not just collect the `Attribute` results, but also `Attribute` at the
bottom of `GetField` chain.
Author: Wenchen Fan cloud0...@outlook.com
Author: Michael Armbrust mich...@databricks.com
Closes #4918 from marmbrus/pr/4904 and squashes the following commits:
997f84e [Michael Armbrust] fix style
3eedbfc [Wenchen Fan] fix 6145
(cherry picked from commit 5873c713cc47af311f517ea33a6110993a410377)
Signed-off-by: Michael Armbrust mich...@databricks.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/e358f555
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/e358f555
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/e358f555
Branch: refs/heads/branch-1.3
Commit: e358f1745f89ac0b247388f125a8f4e164ec
Parents: 5097f86
Author: Wenchen Fan cloud0...@outlook.com
Authored: Thu Mar 5 14:49:01 2015 -0800
Committer: Michael Armbrust mich...@databricks.com
Committed: Thu Mar 5 14:49:13 2015 -0800
--
.../scala/org/apache/spark/sql/catalyst/SqlParser.scala | 2 +-
.../org/apache/spark/sql/catalyst/analysis/Analyzer.scala | 5 +++--
.../test/scala/org/apache/spark/sql/SQLQuerySuite.scala | 10 ++
3 files changed, 14 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/e358f555/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
--
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
index c363a5e..54ab13c 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
@@ -385,7 +385,7 @@ class SqlParser extends AbstractSparkSQLParser {
protected lazy val dotExpressionHeader: Parser[Expression] =
(ident ~ .) ~ ident ~ rep(. ~ ident) ^^ {
- case i1 ~ i2 ~ rest = UnresolvedAttribute(i1 + . + i2 +
rest.mkString(., ., ))
+ case i1 ~ i2 ~ rest = UnresolvedAttribute((Seq(i1, i2) ++
rest).mkString(.))
}
protected lazy val dataType: Parser[DataType] =
http://git-wip-us.apache.org/repos/asf/spark/blob/e358f555/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
--
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
index e4e5425..7753331 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
@@ -310,7 +310,7 @@ class Analyzer(catalog: Catalog,
}
/**
- * In many dialects of SQL is it valid to sort by attributes that are not
present in the SELECT
+ * In many dialects of SQL it is valid to sort by attributes that are not
present in the SELECT
* clause. This rule detects such queries and adds the required attributes
to the original
* projection, so that they will be available during sorting. Another
projection is added to
* remove these attributes after sorting.
@@ -321,7 +321,8 @@ class Analyzer(catalog: Catalog,
if !s.resolved p.resolved =
val unresolved = ordering.flatMap(_.collect { case
UnresolvedAttribute(name) = name })
val resolved = unresolved.flatMap(child.resolve(_, resolver))
-val requiredAttributes = AttributeSet(resolved.collect { case a:
Attribute = a })
+val requiredAttributes =
+ AttributeSet(resolved.flatMap(_.collect { case a: Attribute = a }))
val missingInProject = requiredAttributes -- p.output
if (missingInProject.nonEmpty) {
http://git-wip-us.apache.org/repos/asf/spark/blob/e358f555/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
--
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
index 097bf0d..4dedcd3 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
@@ -1049,4 +1049,14 @@ class SQLQuerySuite extends QueryTest with
BeforeAndAfterAll {
spark git commit: [SPARK-6145][SQL] fix ORDER BY on nested fields
Repository: spark
Updated Branches:
refs/heads/master 424a86a1e - 5873c713c
[SPARK-6145][SQL] fix ORDER BY on nested fields
Based on #4904 with style errors fixed.
`LogicalPlan#resolve` will not only produce `Attribute`, but also `GetField`
chain.
So in `ResolveSortReferences`, after resolve the ordering expressions, we
should not just collect the `Attribute` results, but also `Attribute` at the
bottom of `GetField` chain.
Author: Wenchen Fan cloud0...@outlook.com
Author: Michael Armbrust mich...@databricks.com
Closes #4918 from marmbrus/pr/4904 and squashes the following commits:
997f84e [Michael Armbrust] fix style
3eedbfc [Wenchen Fan] fix 6145
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5873c713
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/5873c713
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/5873c713
Branch: refs/heads/master
Commit: 5873c713cc47af311f517ea33a6110993a410377
Parents: 424a86a
Author: Wenchen Fan cloud0...@outlook.com
Authored: Thu Mar 5 14:49:01 2015 -0800
Committer: Michael Armbrust mich...@databricks.com
Committed: Thu Mar 5 14:49:01 2015 -0800
--
.../scala/org/apache/spark/sql/catalyst/SqlParser.scala | 2 +-
.../org/apache/spark/sql/catalyst/analysis/Analyzer.scala | 5 +++--
.../test/scala/org/apache/spark/sql/SQLQuerySuite.scala | 10 ++
3 files changed, 14 insertions(+), 3 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/5873c713/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
--
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
index c363a5e..54ab13c 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala
@@ -385,7 +385,7 @@ class SqlParser extends AbstractSparkSQLParser {
protected lazy val dotExpressionHeader: Parser[Expression] =
(ident ~ .) ~ ident ~ rep(. ~ ident) ^^ {
- case i1 ~ i2 ~ rest = UnresolvedAttribute(i1 + . + i2 +
rest.mkString(., ., ))
+ case i1 ~ i2 ~ rest = UnresolvedAttribute((Seq(i1, i2) ++
rest).mkString(.))
}
protected lazy val dataType: Parser[DataType] =
http://git-wip-us.apache.org/repos/asf/spark/blob/5873c713/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
--
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
index e4e5425..7753331 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
@@ -310,7 +310,7 @@ class Analyzer(catalog: Catalog,
}
/**
- * In many dialects of SQL is it valid to sort by attributes that are not
present in the SELECT
+ * In many dialects of SQL it is valid to sort by attributes that are not
present in the SELECT
* clause. This rule detects such queries and adds the required attributes
to the original
* projection, so that they will be available during sorting. Another
projection is added to
* remove these attributes after sorting.
@@ -321,7 +321,8 @@ class Analyzer(catalog: Catalog,
if !s.resolved p.resolved =
val unresolved = ordering.flatMap(_.collect { case
UnresolvedAttribute(name) = name })
val resolved = unresolved.flatMap(child.resolve(_, resolver))
-val requiredAttributes = AttributeSet(resolved.collect { case a:
Attribute = a })
+val requiredAttributes =
+ AttributeSet(resolved.flatMap(_.collect { case a: Attribute = a }))
val missingInProject = requiredAttributes -- p.output
if (missingInProject.nonEmpty) {
http://git-wip-us.apache.org/repos/asf/spark/blob/5873c713/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
--
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
index 097bf0d..4dedcd3 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
@@ -1049,4 +1049,14 @@ class SQLQuerySuite extends QueryTest with
BeforeAndAfterAll {
rdd.toDF().registerTempTable(distinctData)
checkAnswer(sql(SELECT COUNT(DISTINCT key,value) FROM distinctData),
Row(2))
}
+
+
spark git commit: [SPARK-6163][SQL] jsonFile should be backed by the data source API
Repository: spark
Updated Branches:
refs/heads/master 5873c713c - 1b4bb25c1
[SPARK-6163][SQL] jsonFile should be backed by the data source API
jira: https://issues.apache.org/jira/browse/SPARK-6163
Author: Yin Huai yh...@databricks.com
Closes #4896 from yhuai/SPARK-6163 and squashes the following commits:
45e023e [Yin Huai] Address @chenghao-intel's comment.
2e8734e [Yin Huai] Use JSON data source for jsonFile.
92a4a33 [Yin Huai] Test.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/1b4bb25c
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/1b4bb25c
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/1b4bb25c
Branch: refs/heads/master
Commit: 1b4bb25c10d72132d7f4f3835ef9a3b94b2349e0
Parents: 5873c71
Author: Yin Huai yh...@databricks.com
Authored: Thu Mar 5 14:49:44 2015 -0800
Committer: Michael Armbrust mich...@databricks.com
Committed: Thu Mar 5 14:49:44 2015 -0800
--
.../scala/org/apache/spark/sql/SQLContext.scala | 12 +++--
.../org/apache/spark/sql/json/JsonSuite.scala | 28
2 files changed, 32 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/1b4bb25c/sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala
--
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala
b/sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala
index ce800e0..9c49e84 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala
@@ -542,20 +542,16 @@ class SQLContext(@transient val sparkContext:
SparkContext)
* @group specificdata
*/
@Experimental
- def jsonFile(path: String, schema: StructType): DataFrame = {
-val json = sparkContext.textFile(path)
-jsonRDD(json, schema)
- }
+ def jsonFile(path: String, schema: StructType): DataFrame =
+load(json, schema, Map(path - path))
/**
* :: Experimental ::
* @group specificdata
*/
@Experimental
- def jsonFile(path: String, samplingRatio: Double): DataFrame = {
-val json = sparkContext.textFile(path)
-jsonRDD(json, samplingRatio)
- }
+ def jsonFile(path: String, samplingRatio: Double): DataFrame =
+load(json, Map(path - path, samplingRatio -
samplingRatio.toString))
/**
* Loads an RDD[String] storing JSON objects (one object per record),
returning the result as a
http://git-wip-us.apache.org/repos/asf/spark/blob/1b4bb25c/sql/core/src/test/scala/org/apache/spark/sql/json/JsonSuite.scala
--
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/json/JsonSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/json/JsonSuite.scala
index 9d94d34..0c21f72 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/json/JsonSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/json/JsonSuite.scala
@@ -19,6 +19,8 @@ package org.apache.spark.sql.json
import java.sql.{Date, Timestamp}
+import org.scalactic.Tolerance._
+
import org.apache.spark.sql.TestData._
import org.apache.spark.sql.catalyst.util._
import org.apache.spark.sql.functions._
@@ -551,6 +553,32 @@ class JsonSuite extends QueryTest {
jsonDF.registerTempTable(jsonTable)
}
+ test(jsonFile should be based on JSONRelation) {
+val file = getTempFilePath(json)
+val path = file.toString
+sparkContext.parallelize(1 to 100).map(i = s{a: 1, b:
str$i}).saveAsTextFile(path)
+val jsonDF = jsonFile(path, 0.49)
+
+val analyzed = jsonDF.queryExecution.analyzed
+assert(
+ analyzed.isInstanceOf[LogicalRelation],
+ The DataFrame returned by jsonFile should be based on JSONRelation.)
+val relation = analyzed.asInstanceOf[LogicalRelation].relation
+assert(
+ relation.isInstanceOf[JSONRelation],
+ The DataFrame returned by jsonFile should be based on JSONRelation.)
+assert(relation.asInstanceOf[JSONRelation].path === path)
+assert(relation.asInstanceOf[JSONRelation].samplingRatio === (0.49 +-
0.001))
+
+val schema = StructType(StructField(a, LongType, true) :: Nil)
+val logicalRelation =
+ jsonFile(path,
schema).queryExecution.analyzed.asInstanceOf[LogicalRelation]
+val relationWithSchema =
logicalRelation.relation.asInstanceOf[JSONRelation]
+assert(relationWithSchema.path === path)
+assert(relationWithSchema.schema === schema)
+assert(relationWithSchema.samplingRatio 0.99)
+ }
+
test(Loading a JSON dataset from a text file) {
val file = getTempFilePath(json)
val path = file.toString
-
To unsubscribe, e-mail:
[1/2] spark git commit: Revert Preparing development version 1.3.1-SNAPSHOT
Repository: spark Updated Branches: refs/heads/branch-1.3 556e0de1a - a0cbfe4da Revert Preparing development version 1.3.1-SNAPSHOT This reverts commit 5097f869efbdb75d3b87bcbd8e621e7c12356942. Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d6b9dce1 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d6b9dce1 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d6b9dce1 Branch: refs/heads/branch-1.3 Commit: d6b9dce18f45b5144db2b59656d64db1af25c11b Parents: 556e0de Author: Patrick Wendell patr...@databricks.com Authored: Thu Mar 5 14:56:23 2015 -0800 Committer: Patrick Wendell patr...@databricks.com Committed: Thu Mar 5 14:56:23 2015 -0800 -- assembly/pom.xml | 2 +- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume-sink/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka-assembly/pom.xml | 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/java8-tests/pom.xml| 2 +- extras/kinesis-asl/pom.xml| 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- network/common/pom.xml| 2 +- network/shuffle/pom.xml | 2 +- network/yarn/pom.xml | 2 +- pom.xml | 2 +- repl/pom.xml | 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive-thriftserver/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- 28 files changed, 28 insertions(+), 28 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/d6b9dce1/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index 114dde7..71d80fb 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.1-SNAPSHOT/version +version1.3.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/d6b9dce1/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index dea41f8..538788a 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.1-SNAPSHOT/version +version1.3.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/d6b9dce1/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index ad8a582..a239e00 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.1-SNAPSHOT/version +version1.3.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/d6b9dce1/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index 73ab234..97356c0 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.1-SNAPSHOT/version +version1.3.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/d6b9dce1/external/flume-sink/pom.xml -- diff --git a/external/flume-sink/pom.xml b/external/flume-sink/pom.xml index 1a5aaf5..b7bcaf3 100644 --- a/external/flume-sink/pom.xml +++ b/external/flume-sink/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.1-SNAPSHOT/version +version1.3.0/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/d6b9dce1/external/flume/pom.xml -- diff --git a/external/flume/pom.xml b/external/flume/pom.xml index d5539d9..7b65ea3 100644 --- a/external/flume/pom.xml +++ b/external/flume/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId
spark git commit: [SQL] Make Strategies a public developer API
Repository: spark
Updated Branches:
refs/heads/master 1b4bb25c1 - eb48fd6e9
[SQL] Make Strategies a public developer API
Author: Michael Armbrust mich...@databricks.com
Closes #4920 from marmbrus/openStrategies and squashes the following commits:
cbc35c0 [Michael Armbrust] [SQL] Make Strategies a public developer API
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/eb48fd6e
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/eb48fd6e
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/eb48fd6e
Branch: refs/heads/master
Commit: eb48fd6e9d55fb034c00e61374bb9c2a86a82fb8
Parents: 1b4bb25
Author: Michael Armbrust mich...@databricks.com
Authored: Thu Mar 5 14:50:25 2015 -0800
Committer: Michael Armbrust mich...@databricks.com
Committed: Thu Mar 5 14:50:25 2015 -0800
--
sql/core/src/main/scala/org/apache/spark/sql/package.scala | 7 +--
1 file changed, 5 insertions(+), 2 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/eb48fd6e/sql/core/src/main/scala/org/apache/spark/sql/package.scala
--
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/package.scala
b/sql/core/src/main/scala/org/apache/spark/sql/package.scala
index 02e5b01..3f97a11 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/package.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/package.scala
@@ -34,10 +34,13 @@ import org.apache.spark.sql.execution.SparkPlan
package object sql {
/**
- * Converts a logical plan into zero or more SparkPlans.
+ * Converts a logical plan into zero or more SparkPlans. This API is
exposed for experimenting
+ * with the query planner and is not designed to be stable across spark
releases. Developers
+ * writing libraries should instead consider using the stable APIs provided
in
+ * [[org.apache.spark.sql.sources]]
*/
@DeveloperApi
- protected[sql] type Strategy =
org.apache.spark.sql.catalyst.planning.GenericStrategy[SparkPlan]
+ type Strategy =
org.apache.spark.sql.catalyst.planning.GenericStrategy[SparkPlan]
/**
* Type alias for [[DataFrame]]. Kept here for backward source compatibility
for Scala.
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-6163][SQL] jsonFile should be backed by the data source API
Repository: spark
Updated Branches:
refs/heads/branch-1.3 e358f - 083fed555
[SPARK-6163][SQL] jsonFile should be backed by the data source API
jira: https://issues.apache.org/jira/browse/SPARK-6163
Author: Yin Huai yh...@databricks.com
Closes #4896 from yhuai/SPARK-6163 and squashes the following commits:
45e023e [Yin Huai] Address @chenghao-intel's comment.
2e8734e [Yin Huai] Use JSON data source for jsonFile.
92a4a33 [Yin Huai] Test.
(cherry picked from commit 1b4bb25c10d72132d7f4f3835ef9a3b94b2349e0)
Signed-off-by: Michael Armbrust mich...@databricks.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/083fed55
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/083fed55
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/083fed55
Branch: refs/heads/branch-1.3
Commit: 083fed55567e6fc8dfc97b7b28dfa69873d80dfd
Parents: e358f55
Author: Yin Huai yh...@databricks.com
Authored: Thu Mar 5 14:49:44 2015 -0800
Committer: Michael Armbrust mich...@databricks.com
Committed: Thu Mar 5 14:49:56 2015 -0800
--
.../scala/org/apache/spark/sql/SQLContext.scala | 12 +++--
.../org/apache/spark/sql/json/JsonSuite.scala | 28
2 files changed, 32 insertions(+), 8 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/083fed55/sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala
--
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala
b/sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala
index ce800e0..9c49e84 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/SQLContext.scala
@@ -542,20 +542,16 @@ class SQLContext(@transient val sparkContext:
SparkContext)
* @group specificdata
*/
@Experimental
- def jsonFile(path: String, schema: StructType): DataFrame = {
-val json = sparkContext.textFile(path)
-jsonRDD(json, schema)
- }
+ def jsonFile(path: String, schema: StructType): DataFrame =
+load(json, schema, Map(path - path))
/**
* :: Experimental ::
* @group specificdata
*/
@Experimental
- def jsonFile(path: String, samplingRatio: Double): DataFrame = {
-val json = sparkContext.textFile(path)
-jsonRDD(json, samplingRatio)
- }
+ def jsonFile(path: String, samplingRatio: Double): DataFrame =
+load(json, Map(path - path, samplingRatio -
samplingRatio.toString))
/**
* Loads an RDD[String] storing JSON objects (one object per record),
returning the result as a
http://git-wip-us.apache.org/repos/asf/spark/blob/083fed55/sql/core/src/test/scala/org/apache/spark/sql/json/JsonSuite.scala
--
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/json/JsonSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/json/JsonSuite.scala
index 9d94d34..0c21f72 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/json/JsonSuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/json/JsonSuite.scala
@@ -19,6 +19,8 @@ package org.apache.spark.sql.json
import java.sql.{Date, Timestamp}
+import org.scalactic.Tolerance._
+
import org.apache.spark.sql.TestData._
import org.apache.spark.sql.catalyst.util._
import org.apache.spark.sql.functions._
@@ -551,6 +553,32 @@ class JsonSuite extends QueryTest {
jsonDF.registerTempTable(jsonTable)
}
+ test(jsonFile should be based on JSONRelation) {
+val file = getTempFilePath(json)
+val path = file.toString
+sparkContext.parallelize(1 to 100).map(i = s{a: 1, b:
str$i}).saveAsTextFile(path)
+val jsonDF = jsonFile(path, 0.49)
+
+val analyzed = jsonDF.queryExecution.analyzed
+assert(
+ analyzed.isInstanceOf[LogicalRelation],
+ The DataFrame returned by jsonFile should be based on JSONRelation.)
+val relation = analyzed.asInstanceOf[LogicalRelation].relation
+assert(
+ relation.isInstanceOf[JSONRelation],
+ The DataFrame returned by jsonFile should be based on JSONRelation.)
+assert(relation.asInstanceOf[JSONRelation].path === path)
+assert(relation.asInstanceOf[JSONRelation].samplingRatio === (0.49 +-
0.001))
+
+val schema = StructType(StructField(a, LongType, true) :: Nil)
+val logicalRelation =
+ jsonFile(path,
schema).queryExecution.analyzed.asInstanceOf[LogicalRelation]
+val relationWithSchema =
logicalRelation.relation.asInstanceOf[JSONRelation]
+assert(relationWithSchema.path === path)
+assert(relationWithSchema.schema === schema)
+assert(relationWithSchema.samplingRatio 0.99)
+ }
+
test(Loading a JSON dataset from a text file) {
val file = getTempFilePath(json)
val path =
[1/2] spark git commit: Preparing Spark release v1.3.0-rc3
Repository: spark Updated Branches: refs/heads/branch-1.3 4ab990c2f - c152f9a7e Preparing Spark release v1.3.0-rc3 Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/4aaf48d4 Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/4aaf48d4 Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/4aaf48d4 Branch: refs/heads/branch-1.3 Commit: 4aaf48d46d13129f0f9bdafd771dd80fe568a7dc Parents: 4ab990c Author: Patrick Wendell patr...@databricks.com Authored: Thu Mar 5 23:02:07 2015 + Committer: Patrick Wendell patr...@databricks.com Committed: Thu Mar 5 23:02:07 2015 + -- assembly/pom.xml | 2 +- bagel/pom.xml | 2 +- core/pom.xml | 2 +- examples/pom.xml | 2 +- external/flume-sink/pom.xml | 2 +- external/flume/pom.xml| 2 +- external/kafka-assembly/pom.xml | 2 +- external/kafka/pom.xml| 2 +- external/mqtt/pom.xml | 2 +- external/twitter/pom.xml | 2 +- external/zeromq/pom.xml | 2 +- extras/java8-tests/pom.xml| 2 +- extras/kinesis-asl/pom.xml| 2 +- extras/spark-ganglia-lgpl/pom.xml | 2 +- graphx/pom.xml| 2 +- mllib/pom.xml | 2 +- network/common/pom.xml| 2 +- network/shuffle/pom.xml | 2 +- network/yarn/pom.xml | 2 +- pom.xml | 2 +- repl/pom.xml | 2 +- sql/catalyst/pom.xml | 2 +- sql/core/pom.xml | 2 +- sql/hive-thriftserver/pom.xml | 2 +- sql/hive/pom.xml | 2 +- streaming/pom.xml | 2 +- tools/pom.xml | 2 +- yarn/pom.xml | 2 +- 28 files changed, 28 insertions(+), 28 deletions(-) -- http://git-wip-us.apache.org/repos/asf/spark/blob/4aaf48d4/assembly/pom.xml -- diff --git a/assembly/pom.xml b/assembly/pom.xml index 1215fba..71d80fb 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0-SNAPSHOT/version +version1.3.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/4aaf48d4/bagel/pom.xml -- diff --git a/bagel/pom.xml b/bagel/pom.xml index 1fe6106..538788a 100644 --- a/bagel/pom.xml +++ b/bagel/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0-SNAPSHOT/version +version1.3.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/4aaf48d4/core/pom.xml -- diff --git a/core/pom.xml b/core/pom.xml index fab776d..a239e00 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0-SNAPSHOT/version +version1.3.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/4aaf48d4/examples/pom.xml -- diff --git a/examples/pom.xml b/examples/pom.xml index 994071d..97356c0 100644 --- a/examples/pom.xml +++ b/examples/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0-SNAPSHOT/version +version1.3.0/version relativePath../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/4aaf48d4/external/flume-sink/pom.xml -- diff --git a/external/flume-sink/pom.xml b/external/flume-sink/pom.xml index 96c2787..b7bcaf3 100644 --- a/external/flume-sink/pom.xml +++ b/external/flume-sink/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0-SNAPSHOT/version +version1.3.0/version relativePath../../pom.xml/relativePath /parent http://git-wip-us.apache.org/repos/asf/spark/blob/4aaf48d4/external/flume/pom.xml -- diff --git a/external/flume/pom.xml b/external/flume/pom.xml index 172d447..7b65ea3 100644 --- a/external/flume/pom.xml +++ b/external/flume/pom.xml @@ -21,7 +21,7 @@ parent groupIdorg.apache.spark/groupId artifactIdspark-parent_2.10/artifactId -version1.3.0-SNAPSHOT/version +
Git Push Summary
Repository: spark Updated Tags: refs/tags/v1.3.0-rc3 [created] 4aaf48d46 - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: [SPARK-6153] [SQL] promote guava dep for hive-thriftserver
Repository: spark
Updated Branches:
refs/heads/branch-1.3 b1e916240 - b92d925d9
[SPARK-6153] [SQL] promote guava dep for hive-thriftserver
For package thriftserver, guava is used at runtime.
/cc pwendell
Author: Daoyuan Wang daoyuan.w...@intel.com
Closes #4884 from adrian-wang/test and squashes the following commits:
4600ae7 [Daoyuan Wang] only promote for thriftserver
44dda18 [Daoyuan Wang] promote guava dep for hive
(cherry picked from commit e06c7dfbc2331db2d1c365959c12aaac640a610a)
Signed-off-by: Cheng Lian l...@databricks.com
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b92d925d
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b92d925d
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b92d925d
Branch: refs/heads/branch-1.3
Commit: b92d925d9dcfa601d65d4e6590f62f2aaf59dad2
Parents: b1e9162
Author: Daoyuan Wang daoyuan.w...@intel.com
Authored: Thu Mar 5 16:35:17 2015 +0800
Committer: Cheng Lian l...@databricks.com
Committed: Thu Mar 5 16:35:54 2015 +0800
--
sql/hive-thriftserver/pom.xml | 5 +
1 file changed, 5 insertions(+)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/b92d925d/sql/hive-thriftserver/pom.xml
--
diff --git a/sql/hive-thriftserver/pom.xml b/sql/hive-thriftserver/pom.xml
index 123a1f6..279987f 100644
--- a/sql/hive-thriftserver/pom.xml
+++ b/sql/hive-thriftserver/pom.xml
@@ -42,6 +42,11 @@
version${project.version}/version
/dependency
dependency
+ groupIdcom.google.guava/groupId
+ artifactIdguava/artifactId
+ scoperuntime/scope
+/dependency
+dependency
groupId${hive.group}/groupId
artifactIdhive-cli/artifactId
/dependency
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
spark git commit: Revert [SPARK-6153] [SQL] promote guava dep for hive-thriftserver
Repository: spark
Updated Branches:
refs/heads/branch-1.3 b92d925d9 - f8205d3be
Revert [SPARK-6153] [SQL] promote guava dep for hive-thriftserver
This reverts commit b92d925d9dcfa601d65d4e6590f62f2aaf59dad2.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f8205d3b
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/f8205d3b
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/f8205d3b
Branch: refs/heads/branch-1.3
Commit: f8205d3be30c76c60e361b3426fc82e0da425580
Parents: b92d925
Author: Cheng Lian l...@databricks.com
Authored: Thu Mar 5 17:58:18 2015 +0800
Committer: Cheng Lian l...@databricks.com
Committed: Thu Mar 5 17:58:18 2015 +0800
--
sql/hive-thriftserver/pom.xml | 5 -
1 file changed, 5 deletions(-)
--
http://git-wip-us.apache.org/repos/asf/spark/blob/f8205d3b/sql/hive-thriftserver/pom.xml
--
diff --git a/sql/hive-thriftserver/pom.xml b/sql/hive-thriftserver/pom.xml
index 279987f..123a1f6 100644
--- a/sql/hive-thriftserver/pom.xml
+++ b/sql/hive-thriftserver/pom.xml
@@ -42,11 +42,6 @@
version${project.version}/version
/dependency
dependency
- groupIdcom.google.guava/groupId
- artifactIdguava/artifactId
- scoperuntime/scope
-/dependency
-dependency
groupId${hive.group}/groupId
artifactIdhive-cli/artifactId
/dependency
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
