spark git commit: [DOC][SQL] update out-of-date code snippets using SQLContext in all documents.

2016-07-06 Thread rxin 
Repository: spark
Updated Branches:
  refs/heads/master 23eff5e51 -> b1310425b


[DOC][SQL] update out-of-date code snippets using SQLContext in all documents.

## What changes were proposed in this pull request?

I search the whole documents directory using SQLContext, and update the 
following places:

- docs/configuration.md, sparkR code snippets.
- docs/streaming-programming-guide.md, several example code.

## How was this patch tested?

N/A

Author: WeichenXu 

Closes #14025 from WeichenXu123/WIP_SQLContext_update.


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/b1310425
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/b1310425
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/b1310425

Branch: refs/heads/master
Commit: b1310425b30cbd711e4834d65a0accb3c5a8403a
Parents: 23eff5e
Author: WeichenXu 
Authored: Wed Jul 6 10:41:48 2016 -0700
Committer: Reynold Xin 
Committed: Wed Jul 6 10:41:48 2016 -0700

--
 docs/configuration.md   |  4 ++--
 docs/streaming-programming-guide.md | 39 +---
 2 files changed, 23 insertions(+), 20 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/b1310425/docs/configuration.md
--
diff --git a/docs/configuration.md b/docs/configuration.md
index cee59cf..1e95b86 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -1564,8 +1564,8 @@ spark.sql("SET -v").show(n=200, truncate=False)
 
 
 {% highlight r %}
-# sqlContext is an existing sqlContext.
-properties <- sql(sqlContext, "SET -v")
+sparkR.session()
+properties <- sql("SET -v")
 showDF(properties, numRows = 200, truncate = FALSE)
 {% endhighlight %}
 

http://git-wip-us.apache.org/repos/asf/spark/blob/b1310425/docs/streaming-programming-guide.md
--
diff --git a/docs/streaming-programming-guide.md 
b/docs/streaming-programming-guide.md
index db06a65..2ee3b80 100644
--- a/docs/streaming-programming-guide.md
+++ b/docs/streaming-programming-guide.md
@@ -1534,7 +1534,7 @@ See the full [source 
code]({{site.SPARK_GITHUB_URL}}/blob/master/examples/src/ma
 ***
 
 ## DataFrame and SQL Operations
-You can easily use [DataFrames and SQL](sql-programming-guide.html) operations 
on streaming data. You have to create a SQLContext using the SparkContext that 
the StreamingContext is using. Furthermore this has to done such that it can be 
restarted on driver failures. This is done by creating a lazily instantiated 
singleton instance of SQLContext. This is shown in the following example. It 
modifies the earlier [word count example](#a-quick-example) to generate word 
counts using DataFrames and SQL. Each RDD is converted to a DataFrame, 
registered as a temporary table and then queried using SQL.
+You can easily use [DataFrames and SQL](sql-programming-guide.html) operations 
on streaming data. You have to create a SparkSession using the SparkContext 
that the StreamingContext is using. Furthermore this has to done such that it 
can be restarted on driver failures. This is done by creating a lazily 
instantiated singleton instance of SparkSession. This is shown in the following 
example. It modifies the earlier [word count example](#a-quick-example) to 
generate word counts using DataFrames and SQL. Each RDD is converted to a 
DataFrame, registered as a temporary table and then queried using SQL.
 
 
 
@@ -1546,9 +1546,9 @@ val words: DStream[String] = ...
 
 words.foreachRDD { rdd =>
 
-  // Get the singleton instance of SQLContext
-  val sqlContext = SQLContext.getOrCreate(rdd.sparkContext)
-  import sqlContext.implicits._
+  // Get the singleton instance of SparkSession
+  val spark = 
SparkSession.builder.config(rdd.sparkContext.getConf).getOrCreate()
+  import spark.implicits._
 
   // Convert RDD[String] to DataFrame
   val wordsDataFrame = rdd.toDF("word")
@@ -1558,7 +1558,7 @@ words.foreachRDD { rdd =>
 
   // Do word count on DataFrame using SQL and print it
   val wordCountsDataFrame = 
-sqlContext.sql("select word, count(*) as total from words group by word")
+spark.sql("select word, count(*) as total from words group by word")
   wordCountsDataFrame.show()
 }
 
@@ -1593,8 +1593,8 @@ words.foreachRDD(
 @Override
 public Void call(JavaRDD rdd, Time time) {
 
-  // Get the singleton instance of SQLContext
-  SQLContext sqlContext = SQLContext.getOrCreate(rdd.context());
+  // Get the singleton instance of SparkSession
+  SparkSession spark = 
SparkSession.builder().config(rdd.sparkContext().getConf()).getOrCreate();
 
   // Convert RDD[String] to RDD[case class] to DataFrame
   JavaRDD rowRDD =

spark git commit: [DOC][SQL] update out-of-date code snippets using SQLContext in all documents.

2016-07-06 Thread rxin 
Repository: spark
Updated Branches:
  refs/heads/branch-2.0 e956bd775 -> 091cd5f26


[DOC][SQL] update out-of-date code snippets using SQLContext in all documents.

## What changes were proposed in this pull request?

I search the whole documents directory using SQLContext, and update the 
following places:

- docs/configuration.md, sparkR code snippets.
- docs/streaming-programming-guide.md, several example code.

## How was this patch tested?

N/A

Author: WeichenXu 

Closes #14025 from WeichenXu123/WIP_SQLContext_update.

(cherry picked from commit b1310425b30cbd711e4834d65a0accb3c5a8403a)
Signed-off-by: Reynold Xin 


Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/091cd5f2
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/091cd5f2
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/091cd5f2

Branch: refs/heads/branch-2.0
Commit: 091cd5f265166512a450333946c62c3eb3440e79
Parents: e956bd7
Author: WeichenXu 
Authored: Wed Jul 6 10:41:48 2016 -0700
Committer: Reynold Xin 
Committed: Wed Jul 6 10:41:54 2016 -0700

--
 docs/configuration.md   |  4 ++--
 docs/streaming-programming-guide.md | 39 +---
 2 files changed, 23 insertions(+), 20 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/spark/blob/091cd5f2/docs/configuration.md
--
diff --git a/docs/configuration.md b/docs/configuration.md
index cee59cf..1e95b86 100644
--- a/docs/configuration.md
+++ b/docs/configuration.md
@@ -1564,8 +1564,8 @@ spark.sql("SET -v").show(n=200, truncate=False)
 
 
 {% highlight r %}
-# sqlContext is an existing sqlContext.
-properties <- sql(sqlContext, "SET -v")
+sparkR.session()
+properties <- sql("SET -v")
 showDF(properties, numRows = 200, truncate = FALSE)
 {% endhighlight %}
 

http://git-wip-us.apache.org/repos/asf/spark/blob/091cd5f2/docs/streaming-programming-guide.md
--
diff --git a/docs/streaming-programming-guide.md 
b/docs/streaming-programming-guide.md
index db06a65..2ee3b80 100644
--- a/docs/streaming-programming-guide.md
+++ b/docs/streaming-programming-guide.md
@@ -1534,7 +1534,7 @@ See the full [source 
code]({{site.SPARK_GITHUB_URL}}/blob/master/examples/src/ma
 ***
 
 ## DataFrame and SQL Operations
-You can easily use [DataFrames and SQL](sql-programming-guide.html) operations 
on streaming data. You have to create a SQLContext using the SparkContext that 
the StreamingContext is using. Furthermore this has to done such that it can be 
restarted on driver failures. This is done by creating a lazily instantiated 
singleton instance of SQLContext. This is shown in the following example. It 
modifies the earlier [word count example](#a-quick-example) to generate word 
counts using DataFrames and SQL. Each RDD is converted to a DataFrame, 
registered as a temporary table and then queried using SQL.
+You can easily use [DataFrames and SQL](sql-programming-guide.html) operations 
on streaming data. You have to create a SparkSession using the SparkContext 
that the StreamingContext is using. Furthermore this has to done such that it 
can be restarted on driver failures. This is done by creating a lazily 
instantiated singleton instance of SparkSession. This is shown in the following 
example. It modifies the earlier [word count example](#a-quick-example) to 
generate word counts using DataFrames and SQL. Each RDD is converted to a 
DataFrame, registered as a temporary table and then queried using SQL.
 
 
 
@@ -1546,9 +1546,9 @@ val words: DStream[String] = ...
 
 words.foreachRDD { rdd =>
 
-  // Get the singleton instance of SQLContext
-  val sqlContext = SQLContext.getOrCreate(rdd.sparkContext)
-  import sqlContext.implicits._
+  // Get the singleton instance of SparkSession
+  val spark = 
SparkSession.builder.config(rdd.sparkContext.getConf).getOrCreate()
+  import spark.implicits._
 
   // Convert RDD[String] to DataFrame
   val wordsDataFrame = rdd.toDF("word")
@@ -1558,7 +1558,7 @@ words.foreachRDD { rdd =>
 
   // Do word count on DataFrame using SQL and print it
   val wordCountsDataFrame = 
-sqlContext.sql("select word, count(*) as total from words group by word")
+spark.sql("select word, count(*) as total from words group by word")
   wordCountsDataFrame.show()
 }
 
@@ -1593,8 +1593,8 @@ words.foreachRDD(
 @Override
 public Void call(JavaRDD rdd, Time time) {
 
-  // Get the singleton instance of SQLContext
-  SQLContext sqlContext = SQLContext.getOrCreate(rdd.context());
+  // Get the singleton instance of SparkSession
+  SparkSession spark =