[GitHub] spark pull request: [SPARK-15052][SQL] Use builder pattern to crea...

[tedyu]

Github user tedyu commented on a diff in the pull request:

https://github.com/apache/spark/pull/12830#discussion_r61987501
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala 
---
@@ -635,6 +642,122 @@ class SparkSession private(
 
 object SparkSession {
 
+  /**
+   * Builder for [[SparkSession]].
+   */
+  class Builder {
--- End diff --

How about adding a clear() method so that Builder instance can be reused ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15052][SQL] Use builder pattern to crea...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12830#issuecomment-216404112
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15052][SQL] Use builder pattern to crea...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12830#issuecomment-216404113
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57566/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15052][SQL] Use builder pattern to crea...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12830#issuecomment-216403979
  
**[Test build #57566 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57566/consoleFull)**
 for PR 12830 at commit 
[`0005a3d`](https://github.com/apache/spark/commit/0005a3deb8292c93b4e699abd922f12ad2606c45).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15052][SQL] Use builder pattern to crea...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/12830#discussion_r61821953
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala 
---
@@ -635,6 +642,122 @@ class SparkSession private(
 
 object SparkSession {
 
+  /**
+   * Builder for [[SparkSession]].
+   */
+  class Builder {
+
+private[this] val options = new 
scala.collection.mutable.HashMap[String, String]
+
+/**
+ * Sets a name for the application, which will be shown in the Spark 
web UI.
+ *
+ * @since 2.0.0
+ */
+def appName(name: String): Builder = config("spark.app.name", name)
+
+/**
+ * Sets a config option. Options set using this method are 
automatically propagated to
+ * both [[SparkConf]] and SparkSession's own configuration.
+ *
+ * @since 2.0.0
+ */
+def config(key: String, value: String): Builder = synchronized {
+  options += key -> value
+  this
+}
+
+/**
+ * Sets a config option. Options set using this method are 
automatically propagated to
+ * both [[SparkConf]] and SparkSession's own configuration.
+ *
+ * @since 2.0.0
+ */
+def config(key: String, value: Long): Builder = synchronized {
--- End diff --

They don't matter as they just map into Long / Double.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15052][SQL] Use builder pattern to crea...

[tedyu]

Github user tedyu commented on a diff in the pull request:

https://github.com/apache/spark/pull/12830#discussion_r61821150
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala 
---
@@ -635,6 +642,122 @@ class SparkSession private(
 
 object SparkSession {
 
+  /**
+   * Builder for [[SparkSession]].
+   */
+  class Builder {
+
+private[this] val options = new 
scala.collection.mutable.HashMap[String, String]
+
+/**
+ * Sets a name for the application, which will be shown in the Spark 
web UI.
+ *
+ * @since 2.0.0
+ */
+def appName(name: String): Builder = config("spark.app.name", name)
+
+/**
+ * Sets a config option. Options set using this method are 
automatically propagated to
+ * both [[SparkConf]] and SparkSession's own configuration.
+ *
+ * @since 2.0.0
+ */
+def config(key: String, value: String): Builder = synchronized {
+  options += key -> value
+  this
+}
+
+/**
+ * Sets a config option. Options set using this method are 
automatically propagated to
+ * both [[SparkConf]] and SparkSession's own configuration.
+ *
+ * @since 2.0.0
+ */
+def config(key: String, value: Long): Builder = synchronized {
--- End diff --

What about other primitive types for the value: Int, Float, Short ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15052][SQL] Use builder pattern to crea...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/12830


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15052][SQL] Use builder pattern to crea...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12830#issuecomment-216383981
  
**[Test build #57566 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57566/consoleFull)**
 for PR 12830 at commit 
[`0005a3d`](https://github.com/apache/spark/commit/0005a3deb8292c93b4e699abd922f12ad2606c45).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15052][SQL] Use builder pattern to crea...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/12830#issuecomment-216383875
  
Thanks - going to merge this. I added removing the existing withHiveSupport 
as a TODO in the pr description.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15052][SQL] Use builder pattern to crea...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/12830#discussion_r61805011
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala 
---
@@ -635,6 +642,122 @@ class SparkSession private(
 
 object SparkSession {
 
+  /**
+   * Builder for [[SparkSession]].
+   */
+  class Builder {
+
+private[this] val options = new 
scala.collection.mutable.HashMap[String, String]
+
+/**
+ * Sets a name for the application, which will be shown in the Spark 
web UI.
+ *
+ * @since 2.0.0
+ */
+def appName(name: String): Builder = config("spark.app.name", name)
+
+/**
+ * Sets a config option. Options set using this method are 
automatically propagated to
+ * both [[SparkConf]] and SparkSession's own configuration.
+ *
+ * @since 2.0.0
+ */
+def config(key: String, value: String): Builder = synchronized {
+  options += key -> value
+  this
+}
+
+/**
+ * Sets a config option. Options set using this method are 
automatically propagated to
+ * both [[SparkConf]] and SparkSession's own configuration.
+ *
+ * @since 2.0.0
+ */
+def config(key: String, value: Long): Builder = synchronized {
+  options += key -> value.toString
+  this
+}
+
+/**
+ * Sets a config option. Options set using this method are 
automatically propagated to
+ * both [[SparkConf]] and SparkSession's own configuration.
+ *
+ * @since 2.0.0
+ */
+def config(key: String, value: Double): Builder = synchronized {
+  options += key -> value.toString
+  this
+}
+
+/**
+ * Sets a config option. Options set using this method are 
automatically propagated to
+ * both [[SparkConf]] and SparkSession's own configuration.
+ *
+ * @since 2.0.0
+ */
+def config(key: String, value: Boolean): Builder = synchronized {
+  options += key -> value.toString
+  this
+}
+
+/**
+ * Sets a list of config options based on the given [[SparkConf]].
+ *
+ * @since 2.0.0
+ */
+def config(conf: SparkConf): Builder = synchronized {
+  conf.getAll.foreach { case (k, v) => options += k -> v }
+  this
+}
+
+/**
+ * Sets the Spark master URL to connect to, such as "local" to run 
locally, "local[4]" to
+ * run locally with 4 cores, or "spark://master:7077" to run on a 
Spark standalone cluster.
+ *
+ * @since 2.0.0
+ */
+def master(master: String): Builder = config("spark.master", master)
+
+/**
+ * Enables Hive support, including connectivity to a persistent Hive 
metastore, support for
+ * Hive serdes, and Hive user-defined functions.
+ *
+ * @return 2.0.0
+ */
+def enableHiveSupport(): Builder = synchronized {
+  if (hiveClassesArePresent) {
+config(CATALOG_IMPLEMENTATION.key, "hive")
+  } else {
+throw new IllegalArgumentException(
+  "Unable to instantiate SparkSession with Hive support because " +
+"Hive classes are not found.")
+  }
--- End diff --

probably as a separate pr


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15052][SQL] Use builder pattern to crea...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/12830#discussion_r61804989
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala 
---
@@ -635,6 +642,122 @@ class SparkSession private(
 
 object SparkSession {
 
+  /**
+   * Builder for [[SparkSession]].
+   */
+  class Builder {
+
+private[this] val options = new 
scala.collection.mutable.HashMap[String, String]
+
+/**
+ * Sets a name for the application, which will be shown in the Spark 
web UI.
+ *
+ * @since 2.0.0
+ */
+def appName(name: String): Builder = config("spark.app.name", name)
+
+/**
+ * Sets a config option. Options set using this method are 
automatically propagated to
+ * both [[SparkConf]] and SparkSession's own configuration.
+ *
+ * @since 2.0.0
+ */
+def config(key: String, value: String): Builder = synchronized {
+  options += key -> value
+  this
+}
+
+/**
+ * Sets a config option. Options set using this method are 
automatically propagated to
+ * both [[SparkConf]] and SparkSession's own configuration.
+ *
+ * @since 2.0.0
+ */
+def config(key: String, value: Long): Builder = synchronized {
+  options += key -> value.toString
+  this
+}
+
+/**
+ * Sets a config option. Options set using this method are 
automatically propagated to
+ * both [[SparkConf]] and SparkSession's own configuration.
+ *
+ * @since 2.0.0
+ */
+def config(key: String, value: Double): Builder = synchronized {
+  options += key -> value.toString
+  this
+}
+
+/**
+ * Sets a config option. Options set using this method are 
automatically propagated to
+ * both [[SparkConf]] and SparkSession's own configuration.
+ *
+ * @since 2.0.0
+ */
+def config(key: String, value: Boolean): Builder = synchronized {
+  options += key -> value.toString
+  this
+}
+
+/**
+ * Sets a list of config options based on the given [[SparkConf]].
+ *
+ * @since 2.0.0
+ */
+def config(conf: SparkConf): Builder = synchronized {
+  conf.getAll.foreach { case (k, v) => options += k -> v }
+  this
+}
+
+/**
+ * Sets the Spark master URL to connect to, such as "local" to run 
locally, "local[4]" to
+ * run locally with 4 cores, or "spark://master:7077" to run on a 
Spark standalone cluster.
+ *
+ * @since 2.0.0
+ */
+def master(master: String): Builder = config("spark.master", master)
+
+/**
+ * Enables Hive support, including connectivity to a persistent Hive 
metastore, support for
+ * Hive serdes, and Hive user-defined functions.
+ *
+ * @return 2.0.0
+ */
+def enableHiveSupport(): Builder = synchronized {
+  if (hiveClassesArePresent) {
+config(CATALOG_IMPLEMENTATION.key, "hive")
+  } else {
+throw new IllegalArgumentException(
+  "Unable to instantiate SparkSession with Hive support because " +
+"Hive classes are not found.")
+  }
--- End diff --

we should probably remove that now


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15052][SQL] Use builder pattern to crea...

[andrewor14]

Github user andrewor14 commented on the pull request:

https://github.com/apache/spark/pull/12830#issuecomment-216352389
  
LGTM. This is beautiful.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15052][SQL] Use builder pattern to crea...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/12830#discussion_r61797568
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala 
---
@@ -635,6 +642,122 @@ class SparkSession private(
 
 object SparkSession {
 
+  /**
+   * Builder for [[SparkSession]].
+   */
+  class Builder {
+
+private[this] val options = new 
scala.collection.mutable.HashMap[String, String]
+
+/**
+ * Sets a name for the application, which will be shown in the Spark 
web UI.
+ *
+ * @since 2.0.0
+ */
+def appName(name: String): Builder = config("spark.app.name", name)
+
+/**
+ * Sets a config option. Options set using this method are 
automatically propagated to
+ * both [[SparkConf]] and SparkSession's own configuration.
+ *
+ * @since 2.0.0
+ */
+def config(key: String, value: String): Builder = synchronized {
+  options += key -> value
+  this
+}
+
+/**
+ * Sets a config option. Options set using this method are 
automatically propagated to
+ * both [[SparkConf]] and SparkSession's own configuration.
+ *
+ * @since 2.0.0
+ */
+def config(key: String, value: Long): Builder = synchronized {
+  options += key -> value.toString
+  this
+}
+
+/**
+ * Sets a config option. Options set using this method are 
automatically propagated to
+ * both [[SparkConf]] and SparkSession's own configuration.
+ *
+ * @since 2.0.0
+ */
+def config(key: String, value: Double): Builder = synchronized {
+  options += key -> value.toString
+  this
+}
+
+/**
+ * Sets a config option. Options set using this method are 
automatically propagated to
+ * both [[SparkConf]] and SparkSession's own configuration.
+ *
+ * @since 2.0.0
+ */
+def config(key: String, value: Boolean): Builder = synchronized {
+  options += key -> value.toString
+  this
+}
+
+/**
+ * Sets a list of config options based on the given [[SparkConf]].
+ *
+ * @since 2.0.0
+ */
+def config(conf: SparkConf): Builder = synchronized {
+  conf.getAll.foreach { case (k, v) => options += k -> v }
+  this
+}
+
+/**
+ * Sets the Spark master URL to connect to, such as "local" to run 
locally, "local[4]" to
+ * run locally with 4 cores, or "spark://master:7077" to run on a 
Spark standalone cluster.
+ *
+ * @since 2.0.0
+ */
+def master(master: String): Builder = config("spark.master", master)
+
+/**
+ * Enables Hive support, including connectivity to a persistent Hive 
metastore, support for
+ * Hive serdes, and Hive user-defined functions.
+ *
+ * @return 2.0.0
+ */
+def enableHiveSupport(): Builder = synchronized {
+  if (hiveClassesArePresent) {
+config(CATALOG_IMPLEMENTATION.key, "hive")
+  } else {
+throw new IllegalArgumentException(
+  "Unable to instantiate SparkSession with Hive support because " +
+"Hive classes are not found.")
+  }
--- End diff --

you can make `withHiveSupport` call this now: 
`SparkSession.enableHiveSupport().getOrCreate()`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15052][SQL] Use builder pattern to crea...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/12830#discussion_r61797056
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala 
---
@@ -635,6 +642,122 @@ class SparkSession private(
 
 object SparkSession {
 
+  /**
+   * Builder for [[SparkSession]].
+   */
+  class Builder {
+
+private[this] val options = new 
scala.collection.mutable.HashMap[String, String]
+
+/**
+ * Sets a name for the application, which will be shown in the Spark 
web UI.
+ *
+ * @since 2.0.0
+ */
+def appName(name: String): Builder = config("spark.app.name", name)
+
+/**
+ * Sets a config option. Options set using this method are 
automatically propagated to
+ * both [[SparkConf]] and SparkSession's own configuration.
+ *
+ * @since 2.0.0
+ */
+def config(key: String, value: String): Builder = synchronized {
+  options += key -> value
+  this
+}
+
+/**
+ * Sets a config option. Options set using this method are 
automatically propagated to
+ * both [[SparkConf]] and SparkSession's own configuration.
+ *
+ * @since 2.0.0
+ */
+def config(key: String, value: Long): Builder = synchronized {
+  options += key -> value.toString
+  this
+}
+
+/**
+ * Sets a config option. Options set using this method are 
automatically propagated to
+ * both [[SparkConf]] and SparkSession's own configuration.
+ *
+ * @since 2.0.0
+ */
+def config(key: String, value: Double): Builder = synchronized {
+  options += key -> value.toString
+  this
+}
+
+/**
+ * Sets a config option. Options set using this method are 
automatically propagated to
+ * both [[SparkConf]] and SparkSession's own configuration.
+ *
+ * @since 2.0.0
+ */
+def config(key: String, value: Boolean): Builder = synchronized {
+  options += key -> value.toString
+  this
+}
+
+/**
+ * Sets a list of config options based on the given [[SparkConf]].
+ *
+ * @since 2.0.0
+ */
+def config(conf: SparkConf): Builder = synchronized {
+  conf.getAll.foreach { case (k, v) => options += k -> v }
+  this
+}
+
+/**
+ * Sets the Spark master URL to connect to, such as "local" to run 
locally, "local[4]" to
+ * run locally with 4 cores, or "spark://master:7077" to run on a 
Spark standalone cluster.
+ *
+ * @since 2.0.0
+ */
+def master(master: String): Builder = config("spark.master", master)
+
+/**
+ * Enables Hive support, including connectivity to a persistent Hive 
metastore, support for
+ * Hive serdes, and Hive user-defined functions.
+ *
+ * @return 2.0.0
--- End diff --

nope!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15052][SQL] Use builder pattern to crea...

[andrewor14]

Github user andrewor14 commented on a diff in the pull request:

https://github.com/apache/spark/pull/12830#discussion_r61797058
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala 
---
@@ -635,6 +642,122 @@ class SparkSession private(
 
 object SparkSession {
 
+  /**
+   * Builder for [[SparkSession]].
+   */
+  class Builder {
+
+private[this] val options = new 
scala.collection.mutable.HashMap[String, String]
+
+/**
+ * Sets a name for the application, which will be shown in the Spark 
web UI.
+ *
+ * @since 2.0.0
+ */
+def appName(name: String): Builder = config("spark.app.name", name)
+
+/**
+ * Sets a config option. Options set using this method are 
automatically propagated to
+ * both [[SparkConf]] and SparkSession's own configuration.
+ *
+ * @since 2.0.0
+ */
+def config(key: String, value: String): Builder = synchronized {
+  options += key -> value
+  this
+}
+
+/**
+ * Sets a config option. Options set using this method are 
automatically propagated to
+ * both [[SparkConf]] and SparkSession's own configuration.
+ *
+ * @since 2.0.0
+ */
+def config(key: String, value: Long): Builder = synchronized {
+  options += key -> value.toString
+  this
+}
+
+/**
+ * Sets a config option. Options set using this method are 
automatically propagated to
+ * both [[SparkConf]] and SparkSession's own configuration.
+ *
+ * @since 2.0.0
+ */
+def config(key: String, value: Double): Builder = synchronized {
+  options += key -> value.toString
+  this
+}
+
+/**
+ * Sets a config option. Options set using this method are 
automatically propagated to
+ * both [[SparkConf]] and SparkSession's own configuration.
+ *
+ * @since 2.0.0
+ */
+def config(key: String, value: Boolean): Builder = synchronized {
+  options += key -> value.toString
+  this
+}
+
+/**
+ * Sets a list of config options based on the given [[SparkConf]].
+ *
+ * @since 2.0.0
+ */
+def config(conf: SparkConf): Builder = synchronized {
+  conf.getAll.foreach { case (k, v) => options += k -> v }
+  this
+}
+
+/**
+ * Sets the Spark master URL to connect to, such as "local" to run 
locally, "local[4]" to
+ * run locally with 4 cores, or "spark://master:7077" to run on a 
Spark standalone cluster.
+ *
+ * @since 2.0.0
+ */
+def master(master: String): Builder = config("spark.master", master)
+
+/**
+ * Enables Hive support, including connectivity to a persistent Hive 
metastore, support for
+ * Hive serdes, and Hive user-defined functions.
+ *
+ * @return 2.0.0
+ */
+def enableHiveSupport(): Builder = synchronized {
+  if (hiveClassesArePresent) {
+config(CATALOG_IMPLEMENTATION.key, "hive")
+  } else {
+throw new IllegalArgumentException(
+  "Unable to instantiate SparkSession with Hive support because " +
+"Hive classes are not found.")
+  }
+}
+
+/**
+ * Gets an existing [[SparkSession]] or, if there is no existing one, 
creates a new one
+ * based on the options set in this builder.
+ *
+ * @since 2.0.0
+ */
+def getOrCreate(): SparkSession = synchronized {
--- End diff --

we should probably also have `SparkSession.getOrCreate`. I think people are 
going to try that before checking out the builder stuff.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15052][SQL] Use builder pattern to crea...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/12830#discussion_r61707542
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala 
---
@@ -635,6 +642,122 @@ class SparkSession private(
 
 object SparkSession {
 
+  /**
+   * Builder for [[SparkSession]].
+   */
+  class Builder {
+
+private[this] val options = new 
scala.collection.mutable.HashMap[String, String]
--- End diff --

But, forget about my comments. `Builder` is so simple and current 
implementation is solid, too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15052][SQL] Use builder pattern to crea...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/12830#discussion_r61706770
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala 
---
@@ -635,6 +642,122 @@ class SparkSession private(
 
 object SparkSession {
 
+  /**
+   * Builder for [[SparkSession]].
+   */
+  class Builder {
+
+private[this] val options = new 
scala.collection.mutable.HashMap[String, String]
--- End diff --

Oh, what I meant was moving locking point from `Building instance` into 
`options`. I thought only `getOrCreate` needs locking on `Builder instance`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15052][SQL] Use builder pattern to crea...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12830#issuecomment-216106479
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15052][SQL] Use builder pattern to crea...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/12830#issuecomment-216106483
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/57501/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15052][SQL] Use builder pattern to crea...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12830#issuecomment-216106407
  
**[Test build #57501 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57501/consoleFull)**
 for PR 12830 at commit 
[`8172d91`](https://github.com/apache/spark/commit/8172d91febbae526378c16eb6c95e5df839e3e73).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15052][SQL] Use builder pattern to crea...

[rxin]

Github user rxin commented on a diff in the pull request:

https://github.com/apache/spark/pull/12830#discussion_r61702820
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala 
---
@@ -635,6 +642,122 @@ class SparkSession private(
 
 object SparkSession {
 
+  /**
+   * Builder for [[SparkSession]].
+   */
+  class Builder {
+
+private[this] val options = new 
scala.collection.mutable.HashMap[String, String]
--- End diff --

It creates a lot of garbage for something that's not expected to be 
concurrent.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15052][SQL] Use builder pattern to crea...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/12830#discussion_r61702208
  
--- Diff: sql/core/src/main/scala/org/apache/spark/sql/SparkSession.scala 
---
@@ -635,6 +642,122 @@ class SparkSession private(
 
 object SparkSession {
 
+  /**
+   * Builder for [[SparkSession]].
+   */
+  class Builder {
+
+private[this] val options = new 
scala.collection.mutable.HashMap[String, String]
--- End diff --

What about using `j.u.c.ConcurrentHashMap`?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15052][SQL] Use builder pattern to crea...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the pull request:

https://github.com/apache/spark/pull/12830#issuecomment-216102730
  
Yes, right. And, this can reduce the `import` statement for `SparkConf` and 
`SparkContext` for those people. It become much simpler. Cool. I will update my 
PR accordingly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15052][SQL] Use builder pattern to crea...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/12830#issuecomment-216102470
  
Yes. Technically we don't really reduce the line length, but definitely 
reduces the number of concepts people need to use if they are just using 
DataFrame/Dataset.



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15052][SQL] Use builder pattern to crea...

[dongjoon-hyun]

Github user dongjoon-hyun commented on the pull request:

https://github.com/apache/spark/pull/12830#issuecomment-216102294
  
Thank you for notifying me. It looks good to me. Then, the three-line 
pattern will be replace into one factory statement, right?

**Spark 1.x**
```
val conf = new SparkConf().setMaster("local[4]").setAppName("App")
val sc = new SparkContext(conf)
val sqlContext = new SQLContext(sc)
```

**Spark 2.0**
```
val spark = 
SparkSession.builder().master("local").config("spark.some.config.option", 
"some-value").getOrCreate()
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15052][SQL] Use builder pattern to crea...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/12830#issuecomment-216101059
  
**[Test build #57501 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/57501/consoleFull)**
 for PR 12830 at commit 
[`8172d91`](https://github.com/apache/spark/commit/8172d91febbae526378c16eb6c95e5df839e3e73).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15052][SQL] Use builder pattern to crea...

[rxin]

Github user rxin commented on the pull request:

https://github.com/apache/spark/pull/12830#issuecomment-216100989
  
cc @yhuai @andrewor14 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-15052][SQL] Use builder pattern to crea...

[rxin]

GitHub user rxin opened a pull request:

https://github.com/apache/spark/pull/12830

[SPARK-15052][SQL] Use builder pattern to create SparkSession

## What changes were proposed in this pull request?
This patch creates a builder pattern for creating SparkSession. The new 
code is unused and mostly deadcode. I'm putting it up here for feedback.

There are a few TODOs that can be done as follow-up pull requests:
- [ ] Update tests to use this
- [ ] Update examples to use this
- [ ] Disable the old constructor so the only way to start a SparkSession 
is through this builder pattern

## How was this patch tested?
Part of the future pull request is to clean this up and switch existing 
tests to use this.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rxin/spark sparksession-builder

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/12830.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #12830


commit 0e969ec3fc88a7a4d84cc1e975364ec89cfb788f
Author: Reynold Xin 
Date:   2016-05-02T03:44:31Z

[SPARK-15052][SQL] Use builder pattern to create SparkSession




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org