spark git commit: [SPARK-19732][SQL][PYSPARK] Add fill functions for nulls in bool fields of datasets

2017-06-02 Thread ueshin 
Repository: spark
Updated Branches:
  refs/heads/master 864d94fe8 -> 6cbc61d10


[SPARK-19732][SQL][PYSPARK] Add fill functions for nulls in bool fields of 
datasets

## What changes were proposed in this pull request?

Allow fill/replace of NAs with booleans, both in Python and Scala

## How was this patch tested?

Unit tests, doctests

This PR is original work from me and I license this work to the Spark project

Author: Ruben Berenguel Montoro 
Author: Ruben Berenguel 

Closes #18164 from rberenguel/SPARK-19732-fillna-bools.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6cbc61d1
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6cbc61d1
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/6cbc61d1

Branch: refs/heads/master
Commit: 6cbc61d1070584ffbc34b1f53df352c9162f414a
Parents: 864d94f
Author: Ruben Berenguel Montoro 
Authored: Sat Jun 3 14:56:42 2017 +0900
Committer: Takuya UESHIN 
Committed: Sat Jun 3 14:56:42 2017 +0900

--
 python/pyspark/sql/dataframe.py | 23 ++---
 python/pyspark/sql/tests.py | 34 +++-
 .../apache/spark/sql/DataFrameNaFunctions.scala | 30 +++--
 .../spark/sql/DataFrameNaFunctionsSuite.scala   | 21 
 4 files changed, 94 insertions(+), 14 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/6cbc61d1/python/pyspark/sql/dataframe.py
--
diff --git a/python/pyspark/sql/dataframe.py b/python/pyspark/sql/dataframe.py
index 8d8b938..99abfcc 100644
--- a/python/pyspark/sql/dataframe.py
+++ b/python/pyspark/sql/dataframe.py
@@ -1289,7 +1289,7 @@ class DataFrame(object):
 """Replace null values, alias for ``na.fill()``.
 :func:`DataFrame.fillna` and :func:`DataFrameNaFunctions.fill` are 
aliases of each other.
 
-:param value: int, long, float, string, or dict.
+:param value: int, long, float, string, bool or dict.
 Value to replace null values with.
 If the value is a dict, then `subset` is ignored and `value` must 
be a mapping
 from column name (string) to replacement value. The replacement 
value must be
@@ -1309,6 +1309,15 @@ class DataFrame(object):
 | 50|50| null|
 +---+--+-+
 
+>>> df5.na.fill(False).show()
+++---+-+
+| age|   name|  spy|
+++---+-+
+|  10|  Alice|false|
+|   5|Bob|false|
+|null|Mallory| true|
+++---+-+
+
 >>> df4.na.fill({'age': 50, 'name': 'unknown'}).show()
 +---+--+---+
 |age|height|   name|
@@ -1319,10 +1328,13 @@ class DataFrame(object):
 | 50|  null|unknown|
 +---+--+---+
 """
-if not isinstance(value, (float, int, long, basestring, dict)):
-raise ValueError("value should be a float, int, long, string, or 
dict")
+if not isinstance(value, (float, int, long, basestring, bool, dict)):
+raise ValueError("value should be a float, int, long, string, bool 
or dict")
+
+# Note that bool validates isinstance(int), but we don't want to
+# convert bools to floats
 
-if isinstance(value, (int, long)):
+if not isinstance(value, bool) and isinstance(value, (int, long)):
 value = float(value)
 
 if isinstance(value, dict):
@@ -1819,6 +1831,9 @@ def _test():
Row(name='Bob', age=5, height=None),
Row(name='Tom', age=None, height=None),
Row(name=None, age=None, 
height=None)]).toDF()
+globs['df5'] = sc.parallelize([Row(name='Alice', spy=False, age=10),
+   Row(name='Bob', spy=None, age=5),
+   Row(name='Mallory', spy=True, 
age=None)]).toDF()
 globs['sdf'] = sc.parallelize([Row(name='Tom', time=1479441846),
Row(name='Bob', time=1479442946)]).toDF()
 

http://git-wip-us.apache.org/repos/asf/spark/blob/6cbc61d1/python/pyspark/sql/tests.py
--
diff --git a/python/pyspark/sql/tests.py b/python/pyspark/sql/tests.py
index acea911..845e1c7 100644
--- a/python/pyspark/sql/tests.py
+++ b/python/pyspark/sql/tests.py
@@ -1697,40 +1697,58 @@ class SQLTests(ReusedPySparkTestCase):
 schema = StructType([
 StructField("name", StringType(), True),
 StructField("age", IntegerType(), True),
-StructField("height", DoubleType(), True)])
+

spark git commit: [SPARK-20974][BUILD] we should run REPL tests if SQL module has code changes

2017-06-02 Thread wenchen 
Repository: spark
Updated Branches:
  refs/heads/branch-2.0 9952b53b5 -> 0f3598820


[SPARK-20974][BUILD] we should run REPL tests if SQL module has code changes

## What changes were proposed in this pull request?

REPL module depends on SQL module, so we should run REPL tests if SQL module 
has code changes.

## How was this patch tested?

N/A

Author: Wenchen Fan 

Closes #18191 from cloud-fan/test.

(cherry picked from commit 864d94fe879a32de324da65a844e62a0260b222d)
Signed-off-by: Wenchen Fan 


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/0f359882
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/0f359882
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/0f359882

Branch: refs/heads/branch-2.0
Commit: 0f35988200f862939d9edb1a9bffeaf315e645ba
Parents: 9952b53
Author: Wenchen Fan 
Authored: Fri Jun 2 21:59:52 2017 -0700
Committer: Wenchen Fan 
Committed: Fri Jun 2 22:00:32 2017 -0700

--
 dev/run-tests.py|  2 +-
 dev/sparktestsupport/modules.py | 13 +
 2 files changed, 14 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/0f359882/dev/run-tests.py
--
diff --git a/dev/run-tests.py b/dev/run-tests.py
index ad9d4ac..43e3bf6 100755
--- a/dev/run-tests.py
+++ b/dev/run-tests.py
@@ -111,7 +111,7 @@ def determine_modules_to_test(changed_modules):
 >>> x = [x.name for x in determine_modules_to_test([modules.sql])]
 >>> x # doctest: +NORMALIZE_WHITESPACE
 ['sql', 'hive', 'mllib', 'sql-kafka-0-10', 'examples', 'hive-thriftserver',
- 'pyspark-sql', 'sparkr', 'pyspark-mllib', 'pyspark-ml']
+ 'pyspark-sql', 'repl', 'sparkr', 'pyspark-mllib', 'pyspark-ml']
 """
 modules_to_test = set()
 for module in changed_modules:

http://git-wip-us.apache.org/repos/asf/spark/blob/0f359882/dev/sparktestsupport/modules.py
--
diff --git a/dev/sparktestsupport/modules.py b/dev/sparktestsupport/modules.py
index c5d1a07..8f3e13c 100644
--- a/dev/sparktestsupport/modules.py
+++ b/dev/sparktestsupport/modules.py
@@ -123,6 +123,7 @@ sql = Module(
 ],
 )
 
+
 hive = Module(
 name="hive",
 dependencies=[sql],
@@ -142,6 +143,18 @@ hive = Module(
 )
 
 
+repl = Module(
+name="repl",
+dependencies=[hive],
+source_file_regexes=[
+"repl/",
+],
+sbt_test_goals=[
+"repl/test",
+],
+)
+
+
 hive_thriftserver = Module(
 name="hive-thriftserver",
 dependencies=[hive],


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

spark git commit: [SPARK-20974][BUILD] we should run REPL tests if SQL module has code changes

2017-06-02 Thread wenchen 
Repository: spark
Updated Branches:
  refs/heads/branch-2.1 0b25a7d93 -> afab8557b


[SPARK-20974][BUILD] we should run REPL tests if SQL module has code changes

## What changes were proposed in this pull request?

REPL module depends on SQL module, so we should run REPL tests if SQL module 
has code changes.

## How was this patch tested?

N/A

Author: Wenchen Fan 

Closes #18191 from cloud-fan/test.

(cherry picked from commit 864d94fe879a32de324da65a844e62a0260b222d)
Signed-off-by: Wenchen Fan 


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/afab8557
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/afab8557
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/afab8557

Branch: refs/heads/branch-2.1
Commit: afab8557b069dff233bc187ddad46d071eeb6137
Parents: 0b25a7d
Author: Wenchen Fan 
Authored: Fri Jun 2 21:59:52 2017 -0700
Committer: Wenchen Fan 
Committed: Fri Jun 2 22:00:19 2017 -0700

--
 dev/run-tests.py|  2 +-
 dev/sparktestsupport/modules.py | 13 +
 2 files changed, 14 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/afab8557/dev/run-tests.py
--
diff --git a/dev/run-tests.py b/dev/run-tests.py
index ab285ac..f24aac9 100755
--- a/dev/run-tests.py
+++ b/dev/run-tests.py
@@ -111,7 +111,7 @@ def determine_modules_to_test(changed_modules):
 >>> x = [x.name for x in determine_modules_to_test([modules.sql])]
 >>> x # doctest: +NORMALIZE_WHITESPACE
 ['sql', 'hive', 'mllib', 'sql-kafka-0-10', 'examples', 'hive-thriftserver',
- 'pyspark-sql', 'sparkr', 'pyspark-mllib', 'pyspark-ml']
+ 'pyspark-sql', 'repl', 'sparkr', 'pyspark-mllib', 'pyspark-ml']
 """
 modules_to_test = set()
 for module in changed_modules:

http://git-wip-us.apache.org/repos/asf/spark/blob/afab8557/dev/sparktestsupport/modules.py
--
diff --git a/dev/sparktestsupport/modules.py b/dev/sparktestsupport/modules.py
index 0cf078c..9e293e7 100644
--- a/dev/sparktestsupport/modules.py
+++ b/dev/sparktestsupport/modules.py
@@ -123,6 +123,7 @@ sql = Module(
 ],
 )
 
+
 hive = Module(
 name="hive",
 dependencies=[sql],
@@ -142,6 +143,18 @@ hive = Module(
 )
 
 
+repl = Module(
+name="repl",
+dependencies=[hive],
+source_file_regexes=[
+"repl/",
+],
+sbt_test_goals=[
+"repl/test",
+],
+)
+
+
 hive_thriftserver = Module(
 name="hive-thriftserver",
 dependencies=[hive],


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

spark git commit: [SPARK-20974][BUILD] we should run REPL tests if SQL module has code changes

2017-06-02 Thread wenchen 
Repository: spark
Updated Branches:
  refs/heads/branch-2.2 478874eff -> c8bbab664


[SPARK-20974][BUILD] we should run REPL tests if SQL module has code changes

## What changes were proposed in this pull request?

REPL module depends on SQL module, so we should run REPL tests if SQL module 
has code changes.

## How was this patch tested?

N/A

Author: Wenchen Fan 

Closes #18191 from cloud-fan/test.

(cherry picked from commit 864d94fe879a32de324da65a844e62a0260b222d)
Signed-off-by: Wenchen Fan 


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/c8bbab66
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/c8bbab66
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/c8bbab66

Branch: refs/heads/branch-2.2
Commit: c8bbab6643533681d1f422cd5c9615638f6a9282
Parents: 478874e
Author: Wenchen Fan 
Authored: Fri Jun 2 21:59:52 2017 -0700
Committer: Wenchen Fan 
Committed: Fri Jun 2 22:00:04 2017 -0700

--
 dev/run-tests.py|  2 +-
 dev/sparktestsupport/modules.py | 13 +
 2 files changed, 14 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/c8bbab66/dev/run-tests.py
--
diff --git a/dev/run-tests.py b/dev/run-tests.py
index 818a0c9..72d148d 100755
--- a/dev/run-tests.py
+++ b/dev/run-tests.py
@@ -111,7 +111,7 @@ def determine_modules_to_test(changed_modules):
 >>> x = [x.name for x in determine_modules_to_test([modules.sql])]
 >>> x # doctest: +NORMALIZE_WHITESPACE
 ['sql', 'hive', 'mllib', 'sql-kafka-0-10', 'examples', 'hive-thriftserver',
- 'pyspark-sql', 'sparkr', 'pyspark-mllib', 'pyspark-ml']
+ 'pyspark-sql', 'repl', 'sparkr', 'pyspark-mllib', 'pyspark-ml']
 """
 modules_to_test = set()
 for module in changed_modules:

http://git-wip-us.apache.org/repos/asf/spark/blob/c8bbab66/dev/sparktestsupport/modules.py
--
diff --git a/dev/sparktestsupport/modules.py b/dev/sparktestsupport/modules.py
index 78b5b8b..2971e0d 100644
--- a/dev/sparktestsupport/modules.py
+++ b/dev/sparktestsupport/modules.py
@@ -123,6 +123,7 @@ sql = Module(
 ],
 )
 
+
 hive = Module(
 name="hive",
 dependencies=[sql],
@@ -142,6 +143,18 @@ hive = Module(
 )
 
 
+repl = Module(
+name="repl",
+dependencies=[hive],
+source_file_regexes=[
+"repl/",
+],
+sbt_test_goals=[
+"repl/test",
+],
+)
+
+
 hive_thriftserver = Module(
 name="hive-thriftserver",
 dependencies=[hive],


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

spark git commit: [SPARK-20974][BUILD] we should run REPL tests if SQL module has code changes

2017-06-02 Thread wenchen 
Repository: spark
Updated Branches:
  refs/heads/master 6de41e951 -> 864d94fe8


[SPARK-20974][BUILD] we should run REPL tests if SQL module has code changes

## What changes were proposed in this pull request?

REPL module depends on SQL module, so we should run REPL tests if SQL module 
has code changes.

## How was this patch tested?

N/A

Author: Wenchen Fan 

Closes #18191 from cloud-fan/test.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/864d94fe
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/864d94fe
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/864d94fe

Branch: refs/heads/master
Commit: 864d94fe879a32de324da65a844e62a0260b222d
Parents: 6de41e9
Author: Wenchen Fan 
Authored: Fri Jun 2 21:59:52 2017 -0700
Committer: Wenchen Fan 
Committed: Fri Jun 2 21:59:52 2017 -0700

--
 dev/run-tests.py|  2 +-
 dev/sparktestsupport/modules.py | 13 +
 2 files changed, 14 insertions(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/864d94fe/dev/run-tests.py
--
diff --git a/dev/run-tests.py b/dev/run-tests.py
index 818a0c9..72d148d 100755
--- a/dev/run-tests.py
+++ b/dev/run-tests.py
@@ -111,7 +111,7 @@ def determine_modules_to_test(changed_modules):
 >>> x = [x.name for x in determine_modules_to_test([modules.sql])]
 >>> x # doctest: +NORMALIZE_WHITESPACE
 ['sql', 'hive', 'mllib', 'sql-kafka-0-10', 'examples', 'hive-thriftserver',
- 'pyspark-sql', 'sparkr', 'pyspark-mllib', 'pyspark-ml']
+ 'pyspark-sql', 'repl', 'sparkr', 'pyspark-mllib', 'pyspark-ml']
 """
 modules_to_test = set()
 for module in changed_modules:

http://git-wip-us.apache.org/repos/asf/spark/blob/864d94fe/dev/sparktestsupport/modules.py
--
diff --git a/dev/sparktestsupport/modules.py b/dev/sparktestsupport/modules.py
index 78b5b8b..2971e0d 100644
--- a/dev/sparktestsupport/modules.py
+++ b/dev/sparktestsupport/modules.py
@@ -123,6 +123,7 @@ sql = Module(
 ],
 )
 
+
 hive = Module(
 name="hive",
 dependencies=[sql],
@@ -142,6 +143,18 @@ hive = Module(
 )
 
 
+repl = Module(
+name="repl",
+dependencies=[hive],
+source_file_regexes=[
+"repl/",
+],
+sbt_test_goals=[
+"repl/test",
+],
+)
+
+
 hive_thriftserver = Module(
 name="hive-thriftserver",
 dependencies=[hive],


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

spark git commit: [SPARK-17078][SQL][FOLLOWUP] Simplify explain cost command

2017-06-02 Thread wenchen 
Repository: spark
Updated Branches:
  refs/heads/master 0eb1fc6cd -> 6de41e951


[SPARK-17078][SQL][FOLLOWUP] Simplify explain cost command

## What changes were proposed in this pull request?

Usually when using explain cost command, users want to see the stats of plan. 
Since stats is only showed in optimized plan, it is more direct and convenient 
to include only optimized plan and physical plan in the output.

## How was this patch tested?

Enhanced existing test.

Author: Zhenhua Wang 

Closes #18190 from wzhfy/simplifyExplainCost.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6de41e95
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6de41e95
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/6de41e95

Branch: refs/heads/master
Commit: 6de41e951fd6172ab7d603474abded0ee7417cde
Parents: 0eb1fc6
Author: Zhenhua Wang 
Authored: Fri Jun 2 17:36:00 2017 -0700
Committer: Wenchen Fan 
Committed: Fri Jun 2 17:36:00 2017 -0700

--
 .../spark/sql/execution/QueryExecution.scala| 28 ++--
 .../spark/sql/execution/command/commands.scala  |  2 +-
 .../sql/hive/execution/HiveExplainSuite.scala   |  6 +
 3 files changed, 21 insertions(+), 15 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/6de41e95/sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala
--
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala
index 1ba9a79..34998cb 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala
@@ -200,11 +200,7 @@ class QueryExecution(val sparkSession: SparkSession, val 
logical: LogicalPlan) {
   """.stripMargin.trim
   }
 
-  override def toString: String = completeString(appendStats = false)
-
-  def toStringWithStats: String = completeString(appendStats = true)
-
-  private def completeString(appendStats: Boolean): String = {
+  override def toString: String = {
 def output = Utils.truncatedString(
   analyzed.output.map(o => s"${o.name}: ${o.dataType.simpleString}"), ", ")
 val analyzedPlan = Seq(
@@ -212,25 +208,29 @@ class QueryExecution(val sparkSession: SparkSession, val 
logical: LogicalPlan) {
   stringOrError(analyzed.treeString(verbose = true))
 ).filter(_.nonEmpty).mkString("\n")
 
-val optimizedPlanString = if (appendStats) {
-  // trigger to compute stats for logical plans
-  optimizedPlan.stats(sparkSession.sessionState.conf)
-  optimizedPlan.treeString(verbose = true, addSuffix = true)
-} else {
-  optimizedPlan.treeString(verbose = true)
-}
-
 s"""== Parsed Logical Plan ==
|${stringOrError(logical.treeString(verbose = true))}
|== Analyzed Logical Plan ==
|$analyzedPlan
|== Optimized Logical Plan ==
-   |${stringOrError(optimizedPlanString)}
+   |${stringOrError(optimizedPlan.treeString(verbose = true))}
|== Physical Plan ==
|${stringOrError(executedPlan.treeString(verbose = true))}
 """.stripMargin.trim
   }
 
+  def stringWithStats: String = {
+// trigger to compute stats for logical plans
+optimizedPlan.stats(sparkSession.sessionState.conf)
+
+// only show optimized logical plan and physical plan
+s"""== Optimized Logical Plan ==
+|${stringOrError(optimizedPlan.treeString(verbose = true, addSuffix = 
true))}
+|== Physical Plan ==
+|${stringOrError(executedPlan.treeString(verbose = true))}
+""".stripMargin.trim
+  }
+
   /** A special namespace for commands that can be used to debug query 
execution. */
   // scalastyle:off
   object debug {

http://git-wip-us.apache.org/repos/asf/spark/blob/6de41e95/sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala
--
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala
index 99d81c4..2d82fcf 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/command/commands.scala
@@ -127,7 +127,7 @@ case class ExplainCommand(
   } else if (extended) {
 queryExecution.toString
   } else if (cost) {
-queryExecution.toStringWithStats
+queryExecution.stringWithStats
   } else {
 queryExecution.simpleString
   }

[1/2] spark git commit: Preparing Spark release v2.2.0-rc4

2017-06-02 Thread pwendell 
Repository: spark
Updated Branches:
  refs/heads/branch-2.2 b560c975b -> 478874eff


Preparing Spark release v2.2.0-rc4


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/377cfa8a
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/377cfa8a
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/377cfa8a

Branch: refs/heads/branch-2.2
Commit: 377cfa8ac7ff7a8a6a6d273182e18ea7dc25ce7e
Parents: b560c97
Author: Patrick Wendell 
Authored: Fri Jun 2 17:20:38 2017 -0700
Committer: Patrick Wendell 
Committed: Fri Jun 2 17:20:38 2017 -0700

--
 assembly/pom.xml  | 2 +-
 common/network-common/pom.xml | 2 +-
 common/network-shuffle/pom.xml| 2 +-
 common/network-yarn/pom.xml   | 2 +-
 common/sketch/pom.xml | 2 +-
 common/tags/pom.xml   | 2 +-
 common/unsafe/pom.xml | 2 +-
 core/pom.xml  | 2 +-
 docs/_config.yml  | 2 +-
 examples/pom.xml  | 2 +-
 external/docker-integration-tests/pom.xml | 2 +-
 external/flume-assembly/pom.xml   | 2 +-
 external/flume-sink/pom.xml   | 2 +-
 external/flume/pom.xml| 2 +-
 external/kafka-0-10-assembly/pom.xml  | 2 +-
 external/kafka-0-10-sql/pom.xml   | 2 +-
 external/kafka-0-10/pom.xml   | 2 +-
 external/kafka-0-8-assembly/pom.xml   | 2 +-
 external/kafka-0-8/pom.xml| 2 +-
 external/kinesis-asl-assembly/pom.xml | 2 +-
 external/kinesis-asl/pom.xml  | 2 +-
 external/spark-ganglia-lgpl/pom.xml   | 2 +-
 graphx/pom.xml| 2 +-
 launcher/pom.xml  | 2 +-
 mllib-local/pom.xml   | 2 +-
 mllib/pom.xml | 2 +-
 pom.xml   | 2 +-
 python/pyspark/version.py | 2 +-
 repl/pom.xml  | 2 +-
 resource-managers/mesos/pom.xml   | 2 +-
 resource-managers/yarn/pom.xml| 2 +-
 sql/catalyst/pom.xml  | 2 +-
 sql/core/pom.xml  | 2 +-
 sql/hive-thriftserver/pom.xml | 2 +-
 sql/hive/pom.xml  | 2 +-
 streaming/pom.xml | 2 +-
 tools/pom.xml | 2 +-
 37 files changed, 37 insertions(+), 37 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/377cfa8a/assembly/pom.xml
--
diff --git a/assembly/pom.xml b/assembly/pom.xml
index 9d8607d..3a7003f 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -21,7 +21,7 @@
   
 org.apache.spark
 spark-parent_2.11
-2.2.0-SNAPSHOT
+2.2.0
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/spark/blob/377cfa8a/common/network-common/pom.xml
--
diff --git a/common/network-common/pom.xml b/common/network-common/pom.xml
index 8657af7..5e9ffd1 100644
--- a/common/network-common/pom.xml
+++ b/common/network-common/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.11
-2.2.0-SNAPSHOT
+2.2.0
 ../../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/spark/blob/377cfa8a/common/network-shuffle/pom.xml
--
diff --git a/common/network-shuffle/pom.xml b/common/network-shuffle/pom.xml
index 24c10fb..c3e10d1 100644
--- a/common/network-shuffle/pom.xml
+++ b/common/network-shuffle/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.11
-2.2.0-SNAPSHOT
+2.2.0
 ../../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/spark/blob/377cfa8a/common/network-yarn/pom.xml
--
diff --git a/common/network-yarn/pom.xml b/common/network-yarn/pom.xml
index 5d4bde9..e66a8b4 100644
--- a/common/network-yarn/pom.xml
+++ b/common/network-yarn/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.11
-2.2.0-SNAPSHOT
+2.2.0
 ../../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/spark/blob/377cfa8a/common/sketch/pom.xml
--
diff --git a/common/sketch/pom.xml b/common/sketch/pom.xml
index 1356c47..1a1f652 100644
--- a/common/sketch/pom.xml
+++ b/common/sketch/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.11
-2.2.0-SNAPSHOT
+2.2.0
 ../../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/spark/blob/377cfa8a/common/tags/pom.xml

[2/2] spark git commit: Preparing development version 2.2.1-SNAPSHOT

2017-06-02 Thread pwendell 
Preparing development version 2.2.1-SNAPSHOT


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/478874ef
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/478874ef
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/478874ef

Branch: refs/heads/branch-2.2
Commit: 478874eff53e6615c98eb2de5be5bbacd036ecce
Parents: 377cfa8
Author: Patrick Wendell 
Authored: Fri Jun 2 17:20:42 2017 -0700
Committer: Patrick Wendell 
Committed: Fri Jun 2 17:20:42 2017 -0700

--
 R/pkg/DESCRIPTION | 2 +-
 assembly/pom.xml  | 2 +-
 common/network-common/pom.xml | 2 +-
 common/network-shuffle/pom.xml| 2 +-
 common/network-yarn/pom.xml   | 2 +-
 common/sketch/pom.xml | 2 +-
 common/tags/pom.xml   | 2 +-
 common/unsafe/pom.xml | 2 +-
 core/pom.xml  | 2 +-
 docs/_config.yml  | 4 ++--
 examples/pom.xml  | 2 +-
 external/docker-integration-tests/pom.xml | 2 +-
 external/flume-assembly/pom.xml   | 2 +-
 external/flume-sink/pom.xml   | 2 +-
 external/flume/pom.xml| 2 +-
 external/kafka-0-10-assembly/pom.xml  | 2 +-
 external/kafka-0-10-sql/pom.xml   | 2 +-
 external/kafka-0-10/pom.xml   | 2 +-
 external/kafka-0-8-assembly/pom.xml   | 2 +-
 external/kafka-0-8/pom.xml| 2 +-
 external/kinesis-asl-assembly/pom.xml | 2 +-
 external/kinesis-asl/pom.xml  | 2 +-
 external/spark-ganglia-lgpl/pom.xml   | 2 +-
 graphx/pom.xml| 2 +-
 launcher/pom.xml  | 2 +-
 mllib-local/pom.xml   | 2 +-
 mllib/pom.xml | 2 +-
 pom.xml   | 2 +-
 python/pyspark/version.py | 2 +-
 repl/pom.xml  | 2 +-
 resource-managers/mesos/pom.xml   | 2 +-
 resource-managers/yarn/pom.xml| 2 +-
 sql/catalyst/pom.xml  | 2 +-
 sql/core/pom.xml  | 2 +-
 sql/hive-thriftserver/pom.xml | 2 +-
 sql/hive/pom.xml  | 2 +-
 streaming/pom.xml | 2 +-
 tools/pom.xml | 2 +-
 38 files changed, 39 insertions(+), 39 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/478874ef/R/pkg/DESCRIPTION
--
diff --git a/R/pkg/DESCRIPTION b/R/pkg/DESCRIPTION
index 879c1f8..cfa49b9 100644
--- a/R/pkg/DESCRIPTION
+++ b/R/pkg/DESCRIPTION
@@ -1,6 +1,6 @@
 Package: SparkR
 Type: Package
-Version: 2.2.0
+Version: 2.2.1
 Title: R Frontend for Apache Spark
 Description: The SparkR package provides an R Frontend for Apache Spark.
 Authors@R: c(person("Shivaram", "Venkataraman", role = c("aut", "cre"),

http://git-wip-us.apache.org/repos/asf/spark/blob/478874ef/assembly/pom.xml
--
diff --git a/assembly/pom.xml b/assembly/pom.xml
index 3a7003f..da7b0c9 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -21,7 +21,7 @@
   
 org.apache.spark
 spark-parent_2.11
-2.2.0
+2.2.1-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/spark/blob/478874ef/common/network-common/pom.xml
--
diff --git a/common/network-common/pom.xml b/common/network-common/pom.xml
index 5e9ffd1..7577253 100644
--- a/common/network-common/pom.xml
+++ b/common/network-common/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.11
-2.2.0
+2.2.1-SNAPSHOT
 ../../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/spark/blob/478874ef/common/network-shuffle/pom.xml
--
diff --git a/common/network-shuffle/pom.xml b/common/network-shuffle/pom.xml
index c3e10d1..558864a 100644
--- a/common/network-shuffle/pom.xml
+++ b/common/network-shuffle/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.11
-2.2.0
+2.2.1-SNAPSHOT
 ../../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/spark/blob/478874ef/common/network-yarn/pom.xml
--
diff --git a/common/network-yarn/pom.xml b/common/network-yarn/pom.xml
index e66a8b4..de66617 100644
--- a/common/network-yarn/pom.xml
+++ b/common/network-yarn/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.11
-2.2.0
+2.2.1-SNAPSHOT
 ../../pom.xml

[spark] Git Push Summary

2017-06-02 Thread pwendell 
Repository: spark
Updated Tags:  refs/tags/v2.2.0-rc4 [created] 377cfa8ac

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

spark git commit: Revert "[SPARK-20946][SQL] simplify the config setting logic in SparkSession.getOrCreate"

2017-06-02 Thread yhuai 
Repository: spark
Updated Branches:
  refs/heads/branch-2.2 6c628e75e -> b560c975b


Revert "[SPARK-20946][SQL] simplify the config setting logic in 
SparkSession.getOrCreate"

This reverts commit e11d90bf8deb553fd41b8837e3856c11486c2503.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b560c975
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b560c975
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b560c975

Branch: refs/heads/branch-2.2
Commit: b560c975b7cdc8828fc9e27cbca740c5e550b9cd
Parents: 6c628e7
Author: Yin Huai 
Authored: Fri Jun 2 15:36:21 2017 -0700
Committer: Yin Huai 
Committed: Fri Jun 2 15:37:38 2017 -0700

--
 .../spark/ml/recommendation/ALSSuite.scala  |  4 +++-
 .../apache/spark/ml/tree/impl/TreeTests.scala   |  2 ++
 .../org/apache/spark/sql/SparkSession.scala | 25 +---
 3 files changed, 21 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/b560c975/mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala
--
diff --git 
a/mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala 
b/mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala
index 23f2256..701040f 100644
--- a/mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala
+++ b/mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala
@@ -820,13 +820,15 @@ class ALSCleanerSuite extends SparkFunSuite {
   FileUtils.listFiles(localDir, TrueFileFilter.INSTANCE, 
TrueFileFilter.INSTANCE).asScala.toSet
 try {
   conf.set("spark.local.dir", localDir.getAbsolutePath)
-  val sc = new SparkContext("local[2]", "ALSCleanerSuite", conf)
+  val sc = new SparkContext("local[2]", "test", conf)
   try {
 sc.setCheckpointDir(checkpointDir.getAbsolutePath)
 // Generate test data
 val (training, _) = ALSSuite.genImplicitTestData(sc, 20, 5, 1, 0.2, 0)
 // Implicitly test the cleaning of parents during ALS training
 val spark = SparkSession.builder
+  .master("local[2]")
+  .appName("ALSCleanerSuite")
   .sparkContext(sc)
   .getOrCreate()
 import spark.implicits._

http://git-wip-us.apache.org/repos/asf/spark/blob/b560c975/mllib/src/test/scala/org/apache/spark/ml/tree/impl/TreeTests.scala
--
diff --git a/mllib/src/test/scala/org/apache/spark/ml/tree/impl/TreeTests.scala 
b/mllib/src/test/scala/org/apache/spark/ml/tree/impl/TreeTests.scala
index b6894b3..92a2369 100644
--- a/mllib/src/test/scala/org/apache/spark/ml/tree/impl/TreeTests.scala
+++ b/mllib/src/test/scala/org/apache/spark/ml/tree/impl/TreeTests.scala
@@ -43,6 +43,8 @@ private[ml] object TreeTests extends SparkFunSuite {
   categoricalFeatures: Map[Int, Int],
   numClasses: Int): DataFrame = {
 val spark = SparkSession.builder()
+  .master("local[2]")
+  .appName("TreeTests")
   .sparkContext(data.sparkContext)
   .getOrCreate()
 import spark.implicits._

http://git-wip-us.apache.org/repos/asf/spark/blob/b560c975/sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
--
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala 
b/sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
index bf37b76..d2bf350 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
@@ -757,8 +757,6 @@ object SparkSession {
 
 private[this] var userSuppliedContext: Option[SparkContext] = None
 
-// The `SparkConf` inside the given `SparkContext` may get changed if you 
specify some options
-// for this builder.
 private[spark] def sparkContext(sparkContext: SparkContext): Builder = 
synchronized {
   userSuppliedContext = Option(sparkContext)
   this
@@ -856,7 +854,7 @@ object SparkSession {
  *
  * @since 2.2.0
  */
-def withExtensions(f: SparkSessionExtensions => Unit): Builder = 
synchronized {
+def withExtensions(f: SparkSessionExtensions => Unit): Builder = {
   f(extensions)
   this
 }
@@ -901,14 +899,22 @@ object SparkSession {
 
 // No active nor global default session. Create a new one.
 val sparkContext = userSuppliedContext.getOrElse {
+  // set app name if not given
+  val randomAppName = java.util.UUID.randomUUID().toString
   val sparkConf = new SparkConf()
-  options.get("spark.master").foreach(sparkConf.setMaster)
-  // set a random app

spark git commit: [MINOR][SQL] Update the description of spark.sql.files.ignoreCorruptFiles and spark.sql.columnNameOfCorruptRecord

2017-06-02 Thread wenchen 
Repository: spark
Updated Branches:
  refs/heads/branch-2.2 0c4227940 -> 6c628e75e


[MINOR][SQL] Update the description of spark.sql.files.ignoreCorruptFiles and 
spark.sql.columnNameOfCorruptRecord

### What changes were proposed in this pull request?
1. The description of `spark.sql.files.ignoreCorruptFiles` is not accurate. 
When the file does not exist, we will issue the error message.
```
org.apache.spark.sql.AnalysisException: Path does not exist: 
file:/nonexist/path;
```

2. `spark.sql.columnNameOfCorruptRecord` also affects the CSV format. The 
current description only mentions JSON format.

### How was this patch tested?
N/A

Author: Xiao Li 

Closes #18184 from gatorsmile/updateMessage.

(cherry picked from commit 2a780ac7fe21df7c336885f8e814c1b866e04285)
Signed-off-by: Wenchen Fan 


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/6c628e75
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/6c628e75
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/6c628e75

Branch: refs/heads/branch-2.2
Commit: 6c628e75e84e99acb365216cbaf413296ee9138e
Parents: 0c42279
Author: Xiao Li 
Authored: Fri Jun 2 12:58:29 2017 -0700
Committer: Wenchen Fan 
Committed: Fri Jun 2 12:58:37 2017 -0700

--
 .../src/main/scala/org/apache/spark/sql/internal/SQLConf.scala | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/6c628e75/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
--
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index e741b4e..1ea9eb5 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -345,7 +345,8 @@ object SQLConf {
 .createWithDefault(true)
 
   val COLUMN_NAME_OF_CORRUPT_RECORD = 
buildConf("spark.sql.columnNameOfCorruptRecord")
-.doc("The name of internal column for storing raw/un-parsed JSON records 
that fail to parse.")
+.doc("The name of internal column for storing raw/un-parsed JSON and CSV 
records that fail " +
+  "to parse.")
 .stringConf
 .createWithDefault("_corrupt_record")
 
@@ -535,8 +536,7 @@ object SQLConf {
 
   val IGNORE_CORRUPT_FILES = buildConf("spark.sql.files.ignoreCorruptFiles")
 .doc("Whether to ignore corrupt files. If true, the Spark jobs will 
continue to run when " +
-  "encountering corrupted or non-existing and contents that have been read 
will still be " +
-  "returned.")
+  "encountering corrupted files and the contents that have been read will 
still be returned.")
 .booleanConf
 .createWithDefault(false)
 


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

spark git commit: [MINOR][SQL] Update the description of spark.sql.files.ignoreCorruptFiles and spark.sql.columnNameOfCorruptRecord

2017-06-02 Thread wenchen 
Repository: spark
Updated Branches:
  refs/heads/master 16186cdcb -> 2a780ac7f


[MINOR][SQL] Update the description of spark.sql.files.ignoreCorruptFiles and 
spark.sql.columnNameOfCorruptRecord

### What changes were proposed in this pull request?
1. The description of `spark.sql.files.ignoreCorruptFiles` is not accurate. 
When the file does not exist, we will issue the error message.
```
org.apache.spark.sql.AnalysisException: Path does not exist: 
file:/nonexist/path;
```

2. `spark.sql.columnNameOfCorruptRecord` also affects the CSV format. The 
current description only mentions JSON format.

### How was this patch tested?
N/A

Author: Xiao Li 

Closes #18184 from gatorsmile/updateMessage.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/2a780ac7
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/2a780ac7
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/2a780ac7

Branch: refs/heads/master
Commit: 2a780ac7fe21df7c336885f8e814c1b866e04285
Parents: 16186cd
Author: Xiao Li 
Authored: Fri Jun 2 12:58:29 2017 -0700
Committer: Wenchen Fan 
Committed: Fri Jun 2 12:58:29 2017 -0700

--
 .../src/main/scala/org/apache/spark/sql/internal/SQLConf.scala | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/2a780ac7/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
--
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 1739b0c..54bee02 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -345,7 +345,8 @@ object SQLConf {
 .createWithDefault(true)
 
   val COLUMN_NAME_OF_CORRUPT_RECORD = 
buildConf("spark.sql.columnNameOfCorruptRecord")
-.doc("The name of internal column for storing raw/un-parsed JSON records 
that fail to parse.")
+.doc("The name of internal column for storing raw/un-parsed JSON and CSV 
records that fail " +
+  "to parse.")
 .stringConf
 .createWithDefault("_corrupt_record")
 
@@ -535,8 +536,7 @@ object SQLConf {
 
   val IGNORE_CORRUPT_FILES = buildConf("spark.sql.files.ignoreCorruptFiles")
 .doc("Whether to ignore corrupt files. If true, the Spark jobs will 
continue to run when " +
-  "encountering corrupted or non-existing and contents that have been read 
will still be " +
-  "returned.")
+  "encountering corrupted files and the contents that have been read will 
still be returned.")
 .booleanConf
 .createWithDefault(false)
 


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[spark] Git Push Summary

2017-06-02 Thread pwendell 
Repository: spark
Updated Tags:  refs/tags/v2.2.0-rc3 [created] cc5dbd55b

-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

[1/2] spark git commit: Preparing Spark release v2.2.0-rc3

2017-06-02 Thread pwendell 
Repository: spark
Updated Branches:
  refs/heads/branch-2.2 9a4a8e1b0 -> 0c4227940


Preparing Spark release v2.2.0-rc3


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/cc5dbd55
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/cc5dbd55
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/cc5dbd55

Branch: refs/heads/branch-2.2
Commit: cc5dbd55b0b312a661d21a4b605ce5ead2ba5218
Parents: 9a4a8e1
Author: Patrick Wendell 
Authored: Fri Jun 2 12:07:53 2017 -0700
Committer: Patrick Wendell 
Committed: Fri Jun 2 12:07:53 2017 -0700

--
 R/pkg/DESCRIPTION | 2 +-
 assembly/pom.xml  | 2 +-
 common/network-common/pom.xml | 2 +-
 common/network-shuffle/pom.xml| 2 +-
 common/network-yarn/pom.xml   | 2 +-
 common/sketch/pom.xml | 2 +-
 common/tags/pom.xml   | 2 +-
 common/unsafe/pom.xml | 2 +-
 core/pom.xml  | 2 +-
 docs/_config.yml  | 4 ++--
 examples/pom.xml  | 2 +-
 external/docker-integration-tests/pom.xml | 2 +-
 external/flume-assembly/pom.xml   | 2 +-
 external/flume-sink/pom.xml   | 2 +-
 external/flume/pom.xml| 2 +-
 external/kafka-0-10-assembly/pom.xml  | 2 +-
 external/kafka-0-10-sql/pom.xml   | 2 +-
 external/kafka-0-10/pom.xml   | 2 +-
 external/kafka-0-8-assembly/pom.xml   | 2 +-
 external/kafka-0-8/pom.xml| 2 +-
 external/kinesis-asl-assembly/pom.xml | 2 +-
 external/kinesis-asl/pom.xml  | 2 +-
 external/spark-ganglia-lgpl/pom.xml   | 2 +-
 graphx/pom.xml| 2 +-
 launcher/pom.xml  | 2 +-
 mllib-local/pom.xml   | 2 +-
 mllib/pom.xml | 2 +-
 pom.xml   | 2 +-
 python/pyspark/version.py | 2 +-
 repl/pom.xml  | 2 +-
 resource-managers/mesos/pom.xml   | 2 +-
 resource-managers/yarn/pom.xml| 2 +-
 sql/catalyst/pom.xml  | 2 +-
 sql/core/pom.xml  | 2 +-
 sql/hive-thriftserver/pom.xml | 2 +-
 sql/hive/pom.xml  | 2 +-
 streaming/pom.xml | 2 +-
 tools/pom.xml | 2 +-
 38 files changed, 39 insertions(+), 39 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/cc5dbd55/R/pkg/DESCRIPTION
--
diff --git a/R/pkg/DESCRIPTION b/R/pkg/DESCRIPTION
index cfa49b9..879c1f8 100644
--- a/R/pkg/DESCRIPTION
+++ b/R/pkg/DESCRIPTION
@@ -1,6 +1,6 @@
 Package: SparkR
 Type: Package
-Version: 2.2.1
+Version: 2.2.0
 Title: R Frontend for Apache Spark
 Description: The SparkR package provides an R Frontend for Apache Spark.
 Authors@R: c(person("Shivaram", "Venkataraman", role = c("aut", "cre"),

http://git-wip-us.apache.org/repos/asf/spark/blob/cc5dbd55/assembly/pom.xml
--
diff --git a/assembly/pom.xml b/assembly/pom.xml
index da7b0c9..3a7003f 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -21,7 +21,7 @@
   
 org.apache.spark
 spark-parent_2.11
-2.2.1-SNAPSHOT
+2.2.0
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/spark/blob/cc5dbd55/common/network-common/pom.xml
--
diff --git a/common/network-common/pom.xml b/common/network-common/pom.xml
index 7577253..5e9ffd1 100644
--- a/common/network-common/pom.xml
+++ b/common/network-common/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.11
-2.2.1-SNAPSHOT
+2.2.0
 ../../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/spark/blob/cc5dbd55/common/network-shuffle/pom.xml
--
diff --git a/common/network-shuffle/pom.xml b/common/network-shuffle/pom.xml
index 558864a..c3e10d1 100644
--- a/common/network-shuffle/pom.xml
+++ b/common/network-shuffle/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.11
-2.2.1-SNAPSHOT
+2.2.0
 ../../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/spark/blob/cc5dbd55/common/network-yarn/pom.xml
--
diff --git a/common/network-yarn/pom.xml b/common/network-yarn/pom.xml
index de66617..e66a8b4 100644
--- a/common/network-yarn/pom.xml
+++ b/common/network-yarn/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark

[2/2] spark git commit: Preparing development version 2.2.0-SNAPSHOT

2017-06-02 Thread pwendell 
Preparing development version 2.2.0-SNAPSHOT


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/0c422794
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/0c422794
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/0c422794

Branch: refs/heads/branch-2.2
Commit: 0c422794041e5ede35faeedcad2fb0112e7420b2
Parents: cc5dbd5
Author: Patrick Wendell 
Authored: Fri Jun 2 12:07:58 2017 -0700
Committer: Patrick Wendell 
Committed: Fri Jun 2 12:07:58 2017 -0700

--
 assembly/pom.xml  | 2 +-
 common/network-common/pom.xml | 2 +-
 common/network-shuffle/pom.xml| 2 +-
 common/network-yarn/pom.xml   | 2 +-
 common/sketch/pom.xml | 2 +-
 common/tags/pom.xml   | 2 +-
 common/unsafe/pom.xml | 2 +-
 core/pom.xml  | 2 +-
 docs/_config.yml  | 2 +-
 examples/pom.xml  | 2 +-
 external/docker-integration-tests/pom.xml | 2 +-
 external/flume-assembly/pom.xml   | 2 +-
 external/flume-sink/pom.xml   | 2 +-
 external/flume/pom.xml| 2 +-
 external/kafka-0-10-assembly/pom.xml  | 2 +-
 external/kafka-0-10-sql/pom.xml   | 2 +-
 external/kafka-0-10/pom.xml   | 2 +-
 external/kafka-0-8-assembly/pom.xml   | 2 +-
 external/kafka-0-8/pom.xml| 2 +-
 external/kinesis-asl-assembly/pom.xml | 2 +-
 external/kinesis-asl/pom.xml  | 2 +-
 external/spark-ganglia-lgpl/pom.xml   | 2 +-
 graphx/pom.xml| 2 +-
 launcher/pom.xml  | 2 +-
 mllib-local/pom.xml   | 2 +-
 mllib/pom.xml | 2 +-
 pom.xml   | 2 +-
 python/pyspark/version.py | 2 +-
 repl/pom.xml  | 2 +-
 resource-managers/mesos/pom.xml   | 2 +-
 resource-managers/yarn/pom.xml| 2 +-
 sql/catalyst/pom.xml  | 2 +-
 sql/core/pom.xml  | 2 +-
 sql/hive-thriftserver/pom.xml | 2 +-
 sql/hive/pom.xml  | 2 +-
 streaming/pom.xml | 2 +-
 tools/pom.xml | 2 +-
 37 files changed, 37 insertions(+), 37 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/0c422794/assembly/pom.xml
--
diff --git a/assembly/pom.xml b/assembly/pom.xml
index 3a7003f..9d8607d 100644
--- a/assembly/pom.xml
+++ b/assembly/pom.xml
@@ -21,7 +21,7 @@
   
 org.apache.spark
 spark-parent_2.11
-2.2.0
+2.2.0-SNAPSHOT
 ../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/spark/blob/0c422794/common/network-common/pom.xml
--
diff --git a/common/network-common/pom.xml b/common/network-common/pom.xml
index 5e9ffd1..8657af7 100644
--- a/common/network-common/pom.xml
+++ b/common/network-common/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.11
-2.2.0
+2.2.0-SNAPSHOT
 ../../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/spark/blob/0c422794/common/network-shuffle/pom.xml
--
diff --git a/common/network-shuffle/pom.xml b/common/network-shuffle/pom.xml
index c3e10d1..24c10fb 100644
--- a/common/network-shuffle/pom.xml
+++ b/common/network-shuffle/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.11
-2.2.0
+2.2.0-SNAPSHOT
 ../../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/spark/blob/0c422794/common/network-yarn/pom.xml
--
diff --git a/common/network-yarn/pom.xml b/common/network-yarn/pom.xml
index e66a8b4..5d4bde9 100644
--- a/common/network-yarn/pom.xml
+++ b/common/network-yarn/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.11
-2.2.0
+2.2.0-SNAPSHOT
 ../../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/spark/blob/0c422794/common/sketch/pom.xml
--
diff --git a/common/sketch/pom.xml b/common/sketch/pom.xml
index 1a1f652..1356c47 100644
--- a/common/sketch/pom.xml
+++ b/common/sketch/pom.xml
@@ -22,7 +22,7 @@
   
 org.apache.spark
 spark-parent_2.11
-2.2.0
+2.2.0-SNAPSHOT
 ../../pom.xml
   
 

http://git-wip-us.apache.org/repos/asf/spark/blob/0c422794/common/tags/pom.xml
--
diff --git a/common/tags/pom.xml

spark git commit: [SPARK-19236][SQL][BACKPORT-2.2] Added createOrReplaceGlobalTempView method

2017-06-02 Thread lixiao 
Repository: spark
Updated Branches:
  refs/heads/branch-2.2 7f35f5b99 -> 9a4a8e1b0


[SPARK-19236][SQL][BACKPORT-2.2] Added createOrReplaceGlobalTempView method

### What changes were proposed in this pull request?

This PR is to backport two PRs for adding the `createOrReplaceGlobalTempView` 
method
https://github.com/apache/spark/pull/18147
https://github.com/apache/spark/pull/16598

---
Added the createOrReplaceGlobalTempView method for dataset API

### How was this patch tested?
N/A

Author: Xiao Li 

Closes #18167 from gatorsmile/Backport18147.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/9a4a8e1b
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/9a4a8e1b
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/9a4a8e1b

Branch: refs/heads/branch-2.2
Commit: 9a4a8e1b010bcfa187360c8331ef897195732638
Parents: 7f35f5b
Author: Xiao Li 
Authored: Fri Jun 2 11:57:22 2017 -0700
Committer: Xiao Li 
Committed: Fri Jun 2 11:57:22 2017 -0700

--
 python/pyspark/sql/dataframe.py | 17 ++
 .../scala/org/apache/spark/sql/Dataset.scala| 16 ++
 .../sql/execution/GlobalTempViewSuite.scala | 60 +++-
 3 files changed, 67 insertions(+), 26 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/9a4a8e1b/python/pyspark/sql/dataframe.py
--
diff --git a/python/pyspark/sql/dataframe.py b/python/pyspark/sql/dataframe.py
index 8a59fcd..b1eb80e 100644
--- a/python/pyspark/sql/dataframe.py
+++ b/python/pyspark/sql/dataframe.py
@@ -191,6 +191,23 @@ class DataFrame(object):
 """
 self._jdf.createGlobalTempView(name)
 
+@since(2.2)
+def createOrReplaceGlobalTempView(self, name):
+"""Creates or replaces a global temporary view using the given name.
+
+The lifetime of this temporary view is tied to this Spark application.
+
+>>> df.createOrReplaceGlobalTempView("people")
+>>> df2 = df.filter(df.age > 3)
+>>> df2.createOrReplaceGlobalTempView("people")
+>>> df3 = spark.sql("select * from global_temp.people")
+>>> sorted(df3.collect()) == sorted(df2.collect())
+True
+>>> spark.catalog.dropGlobalTempView("people")
+
+"""
+self._jdf.createOrReplaceGlobalTempView(name)
+
 @property
 @since(1.4)
 def write(self):

http://git-wip-us.apache.org/repos/asf/spark/blob/9a4a8e1b/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
--
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala 
b/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
index f491e3c..503b540 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
@@ -2657,6 +2657,22 @@ class Dataset[T] private[sql](
 createTempViewCommand(viewName, replace = false, global = true)
   }
 
+  /**
+   * Creates or replaces a global temporary view using the given name. The 
lifetime of this
+   * temporary view is tied to this Spark application.
+   *
+   * Global temporary view is cross-session. Its lifetime is the lifetime of 
the Spark application,
+   * i.e. it will be automatically dropped when the application terminates. 
It's tied to a system
+   * preserved database `_global_temp`, and we must use the qualified name to 
refer a global temp
+   * view, e.g. `SELECT * FROM _global_temp.view1`.
+   *
+   * @group basic
+   * @since 2.2.0
+   */
+  def createOrReplaceGlobalTempView(viewName: String): Unit = withPlan {
+createTempViewCommand(viewName, replace = true, global = true)
+  }
+
   private def createTempViewCommand(
   viewName: String,
   replace: Boolean,

http://git-wip-us.apache.org/repos/asf/spark/blob/9a4a8e1b/sql/core/src/test/scala/org/apache/spark/sql/execution/GlobalTempViewSuite.scala
--
diff --git 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/GlobalTempViewSuite.scala
 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/GlobalTempViewSuite.scala
index 5c63c6a..a3d75b2 100644
--- 
a/sql/core/src/test/scala/org/apache/spark/sql/execution/GlobalTempViewSuite.scala
+++ 
b/sql/core/src/test/scala/org/apache/spark/sql/execution/GlobalTempViewSuite.scala
@@ -35,39 +35,47 @@ class GlobalTempViewSuite extends QueryTest with 
SharedSQLContext {
   private var globalTempDB: String = _
 
   test("basic semantic") {
-sql("CREATE GLOBAL TEMP VIEW src AS SELECT 1, 'a'")
+try {
+  sql("CREATE GLOBAL TEMP VIEW src AS SELECT 1, 'a'")
+
+  // If there is no database in

spark git commit: [SPARK-20946][SQL] simplify the config setting logic in SparkSession.getOrCreate

2017-06-02 Thread wenchen 
Repository: spark
Updated Branches:
  refs/heads/branch-2.2 ae00d49af -> f36c3ee49


[SPARK-20946][SQL] simplify the config setting logic in SparkSession.getOrCreate

## What changes were proposed in this pull request?

The current conf setting logic is a little complex and has duplication, this PR 
simplifies it.

## How was this patch tested?

existing tests.

Author: Wenchen Fan 

Closes #18172 from cloud-fan/session.

(cherry picked from commit e11d90bf8deb553fd41b8837e3856c11486c2503)
Signed-off-by: Wenchen Fan 


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/f36c3ee4
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/f36c3ee4
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/f36c3ee4

Branch: refs/heads/branch-2.2
Commit: f36c3ee492c6d06e86a93c8e1e4aa1bf922c4e03
Parents: ae00d49
Author: Wenchen Fan 
Authored: Fri Jun 2 10:05:05 2017 -0700
Committer: Wenchen Fan 
Committed: Fri Jun 2 10:05:12 2017 -0700

--
 .../spark/ml/recommendation/ALSSuite.scala  |  4 +---
 .../apache/spark/ml/tree/impl/TreeTests.scala   |  2 --
 .../org/apache/spark/sql/SparkSession.scala | 25 +++-
 3 files changed, 10 insertions(+), 21 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/f36c3ee4/mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala
--
diff --git 
a/mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala 
b/mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala
index 701040f..23f2256 100644
--- a/mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala
+++ b/mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala
@@ -820,15 +820,13 @@ class ALSCleanerSuite extends SparkFunSuite {
   FileUtils.listFiles(localDir, TrueFileFilter.INSTANCE, 
TrueFileFilter.INSTANCE).asScala.toSet
 try {
   conf.set("spark.local.dir", localDir.getAbsolutePath)
-  val sc = new SparkContext("local[2]", "test", conf)
+  val sc = new SparkContext("local[2]", "ALSCleanerSuite", conf)
   try {
 sc.setCheckpointDir(checkpointDir.getAbsolutePath)
 // Generate test data
 val (training, _) = ALSSuite.genImplicitTestData(sc, 20, 5, 1, 0.2, 0)
 // Implicitly test the cleaning of parents during ALS training
 val spark = SparkSession.builder
-  .master("local[2]")
-  .appName("ALSCleanerSuite")
   .sparkContext(sc)
   .getOrCreate()
 import spark.implicits._

http://git-wip-us.apache.org/repos/asf/spark/blob/f36c3ee4/mllib/src/test/scala/org/apache/spark/ml/tree/impl/TreeTests.scala
--
diff --git a/mllib/src/test/scala/org/apache/spark/ml/tree/impl/TreeTests.scala 
b/mllib/src/test/scala/org/apache/spark/ml/tree/impl/TreeTests.scala
index 92a2369..b6894b3 100644
--- a/mllib/src/test/scala/org/apache/spark/ml/tree/impl/TreeTests.scala
+++ b/mllib/src/test/scala/org/apache/spark/ml/tree/impl/TreeTests.scala
@@ -43,8 +43,6 @@ private[ml] object TreeTests extends SparkFunSuite {
   categoricalFeatures: Map[Int, Int],
   numClasses: Int): DataFrame = {
 val spark = SparkSession.builder()
-  .master("local[2]")
-  .appName("TreeTests")
   .sparkContext(data.sparkContext)
   .getOrCreate()
 import spark.implicits._

http://git-wip-us.apache.org/repos/asf/spark/blob/f36c3ee4/sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
--
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala 
b/sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
index d2bf350..bf37b76 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
@@ -757,6 +757,8 @@ object SparkSession {
 
 private[this] var userSuppliedContext: Option[SparkContext] = None
 
+// The `SparkConf` inside the given `SparkContext` may get changed if you 
specify some options
+// for this builder.
 private[spark] def sparkContext(sparkContext: SparkContext): Builder = 
synchronized {
   userSuppliedContext = Option(sparkContext)
   this
@@ -854,7 +856,7 @@ object SparkSession {
  *
  * @since 2.2.0
  */
-def withExtensions(f: SparkSessionExtensions => Unit): Builder = {
+def withExtensions(f: SparkSessionExtensions => Unit): Builder = 
synchronized {
   f(extensions)
   this
 }
@@ -899,22 +901,14 @@ object SparkSession {
 
 // No active nor global

spark git commit: [SPARK-20946][SQL] simplify the config setting logic in SparkSession.getOrCreate

2017-06-02 Thread wenchen 
Repository: spark
Updated Branches:
  refs/heads/master d1b80ab92 -> e11d90bf8


[SPARK-20946][SQL] simplify the config setting logic in SparkSession.getOrCreate

## What changes were proposed in this pull request?

The current conf setting logic is a little complex and has duplication, this PR 
simplifies it.

## How was this patch tested?

existing tests.

Author: Wenchen Fan 

Closes #18172 from cloud-fan/session.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/e11d90bf
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/e11d90bf
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/e11d90bf

Branch: refs/heads/master
Commit: e11d90bf8deb553fd41b8837e3856c11486c2503
Parents: d1b80ab
Author: Wenchen Fan 
Authored: Fri Jun 2 10:05:05 2017 -0700
Committer: Wenchen Fan 
Committed: Fri Jun 2 10:05:05 2017 -0700

--
 .../spark/ml/recommendation/ALSSuite.scala  |  4 +---
 .../apache/spark/ml/tree/impl/TreeTests.scala   |  2 --
 .../org/apache/spark/sql/SparkSession.scala | 25 +++-
 3 files changed, 10 insertions(+), 21 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/e11d90bf/mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala
--
diff --git 
a/mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala 
b/mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala
index 701040f..23f2256 100644
--- a/mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala
+++ b/mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala
@@ -820,15 +820,13 @@ class ALSCleanerSuite extends SparkFunSuite {
   FileUtils.listFiles(localDir, TrueFileFilter.INSTANCE, 
TrueFileFilter.INSTANCE).asScala.toSet
 try {
   conf.set("spark.local.dir", localDir.getAbsolutePath)
-  val sc = new SparkContext("local[2]", "test", conf)
+  val sc = new SparkContext("local[2]", "ALSCleanerSuite", conf)
   try {
 sc.setCheckpointDir(checkpointDir.getAbsolutePath)
 // Generate test data
 val (training, _) = ALSSuite.genImplicitTestData(sc, 20, 5, 1, 0.2, 0)
 // Implicitly test the cleaning of parents during ALS training
 val spark = SparkSession.builder
-  .master("local[2]")
-  .appName("ALSCleanerSuite")
   .sparkContext(sc)
   .getOrCreate()
 import spark.implicits._

http://git-wip-us.apache.org/repos/asf/spark/blob/e11d90bf/mllib/src/test/scala/org/apache/spark/ml/tree/impl/TreeTests.scala
--
diff --git a/mllib/src/test/scala/org/apache/spark/ml/tree/impl/TreeTests.scala 
b/mllib/src/test/scala/org/apache/spark/ml/tree/impl/TreeTests.scala
index 92a2369..b6894b3 100644
--- a/mllib/src/test/scala/org/apache/spark/ml/tree/impl/TreeTests.scala
+++ b/mllib/src/test/scala/org/apache/spark/ml/tree/impl/TreeTests.scala
@@ -43,8 +43,6 @@ private[ml] object TreeTests extends SparkFunSuite {
   categoricalFeatures: Map[Int, Int],
   numClasses: Int): DataFrame = {
 val spark = SparkSession.builder()
-  .master("local[2]")
-  .appName("TreeTests")
   .sparkContext(data.sparkContext)
   .getOrCreate()
 import spark.implicits._

http://git-wip-us.apache.org/repos/asf/spark/blob/e11d90bf/sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
--
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala 
b/sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
index d2bf350..bf37b76 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala
@@ -757,6 +757,8 @@ object SparkSession {
 
 private[this] var userSuppliedContext: Option[SparkContext] = None
 
+// The `SparkConf` inside the given `SparkContext` may get changed if you 
specify some options
+// for this builder.
 private[spark] def sparkContext(sparkContext: SparkContext): Builder = 
synchronized {
   userSuppliedContext = Option(sparkContext)
   this
@@ -854,7 +856,7 @@ object SparkSession {
  *
  * @since 2.2.0
  */
-def withExtensions(f: SparkSessionExtensions => Unit): Builder = {
+def withExtensions(f: SparkSessionExtensions => Unit): Builder = 
synchronized {
   f(extensions)
   this
 }
@@ -899,22 +901,14 @@ object SparkSession {
 
 // No active nor global default session. Create a new one.
 val sparkContext = userSuppliedContext.getOrElse {
-  // set app name if not given
-

spark git commit: [SPARK-20967][SQL] SharedState.externalCatalog is not really lazy

2017-06-02 Thread wenchen 
Repository: spark
Updated Branches:
  refs/heads/master 625cebfde -> d1b80ab92


[SPARK-20967][SQL] SharedState.externalCatalog is not really lazy

## What changes were proposed in this pull request?

`SharedState.externalCatalog` is marked as a `lazy val` but actually it's not 
lazy. We access `externalCatalog` while initializing `SharedState` and thus 
eliminate the effort of `lazy val`. When creating `ExternalCatalog` we will try 
to connect to the metastore and may throw an error, so it makes sense to make 
it a `lazy val` in `SharedState`.

## How was this patch tested?

existing tests.

Author: Wenchen Fan 

Closes #18187 from cloud-fan/minor.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d1b80ab9
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d1b80ab9
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d1b80ab9

Branch: refs/heads/master
Commit: d1b80ab9220d83e5fdaf33c513cc811dd17d0de1
Parents: 625cebf
Author: Wenchen Fan 
Authored: Fri Jun 2 09:58:01 2017 -0700
Committer: Wenchen Fan 
Committed: Fri Jun 2 09:58:01 2017 -0700

--
 .../apache/spark/sql/internal/SharedState.scala | 26 ++--
 1 file changed, 13 insertions(+), 13 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/d1b80ab9/sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala
--
diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala 
b/sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala
index a93b701..7202f12 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala
@@ -90,38 +90,38 @@ private[sql] class SharedState(val sparkContext: 
SparkContext) extends Logging {
   /**
* A catalog that interacts with external systems.
*/
-  lazy val externalCatalog: ExternalCatalog =
-SharedState.reflect[ExternalCatalog, SparkConf, Configuration](
+  lazy val externalCatalog: ExternalCatalog = {
+val externalCatalog = SharedState.reflect[ExternalCatalog, SparkConf, 
Configuration](
   SharedState.externalCatalogClassName(sparkContext.conf),
   sparkContext.conf,
   sparkContext.hadoopConfiguration)
 
-  // Create the default database if it doesn't exist.
-  {
 val defaultDbDefinition = CatalogDatabase(
   SessionCatalog.DEFAULT_DATABASE,
   "default database",
   CatalogUtils.stringToURI(warehousePath),
   Map())
-// Initialize default database if it doesn't exist
+// Create default database if it doesn't exist
 if (!externalCatalog.databaseExists(SessionCatalog.DEFAULT_DATABASE)) {
   // There may be another Spark application creating default database at 
the same time, here we
   // set `ignoreIfExists = true` to avoid `DatabaseAlreadyExists` 
exception.
   externalCatalog.createDatabase(defaultDbDefinition, ignoreIfExists = 
true)
 }
-  }
 
-  // Make sure we propagate external catalog events to the spark listener bus
-  externalCatalog.addListener(new ExternalCatalogEventListener {
-override def onEvent(event: ExternalCatalogEvent): Unit = {
-  sparkContext.listenerBus.post(event)
-}
-  })
+// Make sure we propagate external catalog events to the spark listener bus
+externalCatalog.addListener(new ExternalCatalogEventListener {
+  override def onEvent(event: ExternalCatalogEvent): Unit = {
+sparkContext.listenerBus.post(event)
+  }
+})
+
+externalCatalog
+  }
 
   /**
* A manager for global temporary views.
*/
-  val globalTempViewManager: GlobalTempViewManager = {
+  lazy val globalTempViewManager: GlobalTempViewManager = {
 // System preserved database should not exists in metastore. However it's 
hard to guarantee it
 // for every session, because case-sensitivity differs. Here we always 
lowercase it to make our
 // life easier.


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

spark git commit: [SPARK-20942][WEB-UI] The title style about field is error in the history server web ui.

2017-06-02 Thread srowen 
Repository: spark
Updated Branches:
  refs/heads/branch-2.2 bb3d900b4 -> 25cc80066


[SPARK-20942][WEB-UI] The title style about field is error in the history 
server web ui.

## What changes were proposed in this pull request?

1.The title style about field is error.
fix before:
![before](https://cloud.githubusercontent.com/assets/26266482/26661987/a7bed018-46b3-11e7-8a54-a5152d2df0f4.png)

fix after:
![fix](https://cloud.githubusercontent.com/assets/26266482/26662000/ba6cc814-46b3-11e7-8f33-cfd4cc2c60fe.png)

![fix1](https://cloud.githubusercontent.com/assets/26266482/26662080/3c732e3e-46b4-11e7-8768-20b5a6aeadcb.png)

executor-page style:
![executor_page](https://cloud.githubusercontent.com/assets/26266482/26662384/167cbd10-46b6-11e7-9e07-bf391dbc6e08.png)

2.Title text description, 'the application' should be changed to 'this 
application'.

3.Analysis of code:
 $('#history-summary [data-toggle="tooltip"]').tooltip();
The id of 'history-summary' is not there. We only contain id of 
'history-summary-table'.

## How was this patch tested?

manual tests

Please review http://spark.apache.org/contributing.html before opening a pull 
request.

Author: guoxiaolong 
Author: 郭小龙 10207633 
Author: guoxiaolongzte 

Closes #18170 from guoxiaolongzte/SPARK-20942.

(cherry picked from commit 625cebfde632361122e0db3452c4cc38147f696f)
Signed-off-by: Sean Owen 


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/25cc8006
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/25cc8006
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/25cc8006

Branch: refs/heads/branch-2.2
Commit: 25cc80066d68190c1ced7473dd4fd40f7e8dec3a
Parents: bb3d900
Author: guoxiaolong 
Authored: Fri Jun 2 14:38:00 2017 +0100
Committer: Sean Owen 
Committed: Fri Jun 2 14:38:11 2017 +0100

--
 .../spark/ui/static/historypage-template.html | 18 +-
 .../org/apache/spark/ui/static/historypage.js |  2 +-
 2 files changed, 10 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/25cc8006/core/src/main/resources/org/apache/spark/ui/static/historypage-template.html
--
diff --git 
a/core/src/main/resources/org/apache/spark/ui/static/historypage-template.html 
b/core/src/main/resources/org/apache/spark/ui/static/historypage-template.html
index c2afa99..bfe31aa 100644
--- 
a/core/src/main/resources/org/apache/spark/ui/static/historypage-template.html
+++ 
b/core/src/main/resources/org/apache/spark/ui/static/historypage-template.html
@@ -20,47 +20,47 @@
   
 
   
-
+
   App ID
 
   
   
-
+
   App Name
 
   
   
-
+
   Attempt ID
 
   
   
-
+
   Started
 
   
   
-
+
   Completed
 
   
   
-
+
   Duration
 
   
   
-
+
   Spark User
 
   
   
-
+
   Last Updated
 
   
   
-
+
   Event Log
 
   

http://git-wip-us.apache.org/repos/asf/spark/blob/25cc8006/core/src/main/resources/org/apache/spark/ui/static/historypage.js
--
diff --git a/core/src/main/resources/org/apache/spark/ui/static/historypage.js 
b/core/src/main/resources/org/apache/spark/ui/static/historypage.js
index 7db8c27..5ec1ce1 100644
--- a/core/src/main/resources/org/apache/spark/ui/static/historypage.js
+++ b/core/src/main/resources/org/apache/spark/ui/static/historypage.js
@@ -195,7 +195,7 @@ $(document).ready(function() {
 }
 
 $(selector).DataTable(conf);
-$('#hisotry-summary [data-toggle="tooltip"]').tooltip();
+$('#history-summary [data-toggle="tooltip"]').tooltip();
   });
 });
 });


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

spark git commit: [SPARK-20942][WEB-UI] The title style about field is error in the history server web ui.

2017-06-02 Thread srowen 
Repository: spark
Updated Branches:
  refs/heads/master 0e31e28d4 -> 625cebfde


[SPARK-20942][WEB-UI] The title style about field is error in the history 
server web ui.

## What changes were proposed in this pull request?

1.The title style about field is error.
fix before:
![before](https://cloud.githubusercontent.com/assets/26266482/26661987/a7bed018-46b3-11e7-8a54-a5152d2df0f4.png)

fix after:
![fix](https://cloud.githubusercontent.com/assets/26266482/26662000/ba6cc814-46b3-11e7-8f33-cfd4cc2c60fe.png)

![fix1](https://cloud.githubusercontent.com/assets/26266482/26662080/3c732e3e-46b4-11e7-8768-20b5a6aeadcb.png)

executor-page style:
![executor_page](https://cloud.githubusercontent.com/assets/26266482/26662384/167cbd10-46b6-11e7-9e07-bf391dbc6e08.png)

2.Title text description, 'the application' should be changed to 'this 
application'.

3.Analysis of code:
 $('#history-summary [data-toggle="tooltip"]').tooltip();
The id of 'history-summary' is not there. We only contain id of 
'history-summary-table'.

## How was this patch tested?

manual tests

Please review http://spark.apache.org/contributing.html before opening a pull 
request.

Author: guoxiaolong 
Author: 郭小龙 10207633 
Author: guoxiaolongzte 

Closes #18170 from guoxiaolongzte/SPARK-20942.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/625cebfd
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/625cebfd
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/625cebfd

Branch: refs/heads/master
Commit: 625cebfde632361122e0db3452c4cc38147f696f
Parents: 0e31e28
Author: guoxiaolong 
Authored: Fri Jun 2 14:38:00 2017 +0100
Committer: Sean Owen 
Committed: Fri Jun 2 14:38:00 2017 +0100

--
 .../spark/ui/static/historypage-template.html | 18 +-
 .../org/apache/spark/ui/static/historypage.js |  2 +-
 2 files changed, 10 insertions(+), 10 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/625cebfd/core/src/main/resources/org/apache/spark/ui/static/historypage-template.html
--
diff --git 
a/core/src/main/resources/org/apache/spark/ui/static/historypage-template.html 
b/core/src/main/resources/org/apache/spark/ui/static/historypage-template.html
index c2afa99..bfe31aa 100644
--- 
a/core/src/main/resources/org/apache/spark/ui/static/historypage-template.html
+++ 
b/core/src/main/resources/org/apache/spark/ui/static/historypage-template.html
@@ -20,47 +20,47 @@
   
 
   
-
+
   App ID
 
   
   
-
+
   App Name
 
   
   
-
+
   Attempt ID
 
   
   
-
+
   Started
 
   
   
-
+
   Completed
 
   
   
-
+
   Duration
 
   
   
-
+
   Spark User
 
   
   
-
+
   Last Updated
 
   
   
-
+
   Event Log
 
   

http://git-wip-us.apache.org/repos/asf/spark/blob/625cebfd/core/src/main/resources/org/apache/spark/ui/static/historypage.js
--
diff --git a/core/src/main/resources/org/apache/spark/ui/static/historypage.js 
b/core/src/main/resources/org/apache/spark/ui/static/historypage.js
index 7db8c27..5ec1ce1 100644
--- a/core/src/main/resources/org/apache/spark/ui/static/historypage.js
+++ b/core/src/main/resources/org/apache/spark/ui/static/historypage.js
@@ -195,7 +195,7 @@ $(document).ready(function() {
 }
 
 $(selector).DataTable(conf);
-$('#hisotry-summary [data-toggle="tooltip"]').tooltip();
+$('#history-summary [data-toggle="tooltip"]').tooltip();
   });
 });
 });


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org

spark-website git commit: Fix wrong user list email address

2017-06-02 Thread srowen 
Repository: spark-website
Updated Branches:
  refs/heads/asf-site 80f50ecca -> 004856aaa


Fix wrong user list email address


Project: http://git-wip-us.apache.org/repos/asf/spark-website/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark-website/commit/004856aa
Tree: http://git-wip-us.apache.org/repos/asf/spark-website/tree/004856aa
Diff: http://git-wip-us.apache.org/repos/asf/spark-website/diff/004856aa

Branch: refs/heads/asf-site
Commit: 004856aaa997cf6d70e8ed733a4ef5c5121c8e7d
Parents: 80f50ec
Author: Jens Teglhus Møller 
Authored: Fri Jun 2 11:46:09 2017 +0100
Committer: Jens Teglhus Møller 
Committed: Fri Jun 2 11:46:09 2017 +0100

--
 community.md| 4 ++--
 site/community.html | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark-website/blob/004856aa/community.md
--
diff --git a/community.md b/community.md
index 630bba6..a2ee4cc 100644
--- a/community.md
+++ b/community.md
@@ -25,7 +25,7 @@ Some quick tips when using StackOverflow:
   https://stackoverflow.com/questions/tagged/apache-spark;>`apache-spark`
 tag to see if 
   your question has already been answered
   - Search the nabble archive for
-  http://apache-spark-user-list.1001560.n3.nabble.com/;>us...@spark.apache.org
 
+  http://apache-spark-user-list.1001560.n3.nabble.com/;>u...@spark.apache.org
 
 - Please follow the StackOverflow https://stackoverflow.com/help/how-to-ask;>code of conduct  
 - Always use the `apache-spark` tag when asking questions
 - Please also use a secondary tag to specify components so subject matter 
experts can more easily find them.
@@ -61,7 +61,7 @@ Some quick tips when using email:
   - Search StackOverflow at https://stackoverflow.com/questions/tagged/apache-spark;>`apache-spark`
 
   to see if your question has already been answered
   - Search the nabble archive for
-  http://apache-spark-user-list.1001560.n3.nabble.com/;>us...@spark.apache.org
 
+  http://apache-spark-user-list.1001560.n3.nabble.com/;>u...@spark.apache.org
 
 - Tagging the subject line of your email will help you get a faster response, 
e.g. 
 `[Spark SQL]: Does Spark SQL support LEFT SEMI JOIN?`
 - Tags may help identify a topic by:

http://git-wip-us.apache.org/repos/asf/spark-website/blob/004856aa/site/community.html
--
diff --git a/site/community.html b/site/community.html
index 1feee6a..f5048da 100644
--- a/site/community.html
+++ b/site/community.html
@@ -213,7 +213,7 @@ as it is an active forum for Spark users questions 
and answers.
 https://stackoverflow.com/questions/tagged/apache-spark;>apache-spark
 tag to see if 
 your question has already been answered
   Search the nabble archive for
-http://apache-spark-user-list.1001560.n3.nabble.com/;>us...@spark.apache.org
+http://apache-spark-user-list.1001560.n3.nabble.com/;>u...@spark.apache.org
 
   
   Please follow the StackOverflow https://stackoverflow.com/help/how-to-ask;>code of conduct
@@ -254,7 +254,7 @@ project, and scenarios, it is recommended you use the 
u...@spark.apache.org mail
   Search StackOverflow at https://stackoverflow.com/questions/tagged/apache-spark;>apache-spark
 
 to see if your question has already been answered
   Search the nabble archive for
-http://apache-spark-user-list.1001560.n3.nabble.com/;>us...@spark.apache.org
+http://apache-spark-user-list.1001560.n3.nabble.com/;>u...@spark.apache.org
 
   
   Tagging the subject line of your email will help you get a faster 
response, e.g. 


-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org