[GitHub] spark pull request: [SPARK-11089] [SQL] Adds option for disabling ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9740#issuecomment-157362455
  
**[Test build #46081 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46081/consoleFull)**
 for PR 9740 at commit 
[`8b70477`](https://github.com/apache/spark/commit/8b70477048208b41ba03b524abc510191633e5e0).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11089] [SQL] Adds option for disabling ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9740#issuecomment-157362569
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46081/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11089] [SQL] Adds option for disabling ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9740#issuecomment-157338196
  
Test FAILed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46080/
Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11089] [SQL] Adds option for disabling ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9740#issuecomment-157338193
  
Merged build finished. Test FAILed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11089] [SQL] Adds option for disabling ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9740#issuecomment-157336274
  
**[Test build #46081 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46081/consoleFull)**
 for PR 9740 at commit 
[`8b70477`](https://github.com/apache/spark/commit/8b70477048208b41ba03b524abc510191633e5e0).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11089] [SQL] Adds option for disabling ...

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/9740#issuecomment-157475631
  
Merging to master and 1.6.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11089] [SQL] Adds option for disabling ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/9740


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11089] [SQL] Adds option for disabling ...

[davies]

Github user davies commented on a diff in the pull request:

https://github.com/apache/spark/pull/9740#discussion_r44960873
  
--- Diff: 
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala
 ---
@@ -635,6 +683,7 @@ abstract class HiveThriftServer2Test extends 
SparkFunSuite with BeforeAndAfterAl
|  --driver-class-path $driverClassPath
|  --driver-java-options -Dlog4j.debug
|  --conf spark.ui.enabled=false
+   |  ${extraConf.mkString("\n")}
--- End diff --

Is it possible that we only introduce a command line option here? (Not sure 
how the JDBC server is used).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11089] [SQL] Adds option for disabling ...

[liancheng]

Github user liancheng commented on a diff in the pull request:

https://github.com/apache/spark/pull/9740#discussion_r44964862
  
--- Diff: 
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala
 ---
@@ -635,6 +683,7 @@ abstract class HiveThriftServer2Test extends 
SparkFunSuite with BeforeAndAfterAl
|  --driver-class-path $driverClassPath
|  --driver-java-options -Dlog4j.debug
|  --conf spark.ui.enabled=false
+   |  ${extraConf.mkString("\n")}
--- End diff --

Do you mean something like `--single-session`? The benefit of `SparkConf` 
option is that users can add it in spark-defaults.conf.

Command line options of `start-thriftserver.sh` are parsed by both 
`SparkSubmit` and `HiveServer2.ServerOptionsProcessor`. The latter isn't 
extensible. One option is to add a `--single-session` option in `SparkSubmit` 
and make a synonymous of `--conf 
spark.sql.hive.thriftServer.singleSession=true`. However, this option doesn't 
seem to be general enough to be added in `SparkSubmit`.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11089] [SQL] Adds option for disabling ...

[davies]

Github user davies commented on a diff in the pull request:

https://github.com/apache/spark/pull/9740#discussion_r44966303
  
--- Diff: 
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala
 ---
@@ -635,6 +683,7 @@ abstract class HiveThriftServer2Test extends 
SparkFunSuite with BeforeAndAfterAl
|  --driver-class-path $driverClassPath
|  --driver-java-options -Dlog4j.debug
|  --conf spark.ui.enabled=false
+   |  ${extraConf.mkString("\n")}
--- End diff --

I see, thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11089] [SQL] Adds option for disabling ...

[liancheng]

GitHub user liancheng opened a pull request:

https://github.com/apache/spark/pull/9740

[SPARK-11089] [SQL] Adds option for disabling multi-session in Thrift server

This PR adds a new option `spark.sql.hive.thriftServer.singleSession` for 
disabling multi-session support in the Thrift server.

Note that this option is added as a Spark configuration (retrieved from 
`SparkConf`) rather than Spark SQL configuration (retrieved from `SQLConf`). 
This is because all SQL configurations are session-ized. Since multi-session 
support is by default on, no JDBC connection can modify global configurations 
like the newly added one.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/liancheng/spark 
spark-11089.single-session-option

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/9740.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #9740


commit eeeb194d74ee3bdd4f439bc13886f20512fabc6b
Author: Cheng Lian 
Date:   2015-11-16T17:54:53Z

Adds option for disabling multi-session in Thrift server




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11089] [SQL] Adds option for disabling ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9740#issuecomment-157122865
  
**[Test build #46000 has 
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46000/consoleFull)**
 for PR 9740 at commit 
[`eeeb194`](https://github.com/apache/spark/commit/eeeb194d74ee3bdd4f439bc13886f20512fabc6b).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11089] [SQL] Adds option for disabling ...

[davies]

Github user davies commented on the pull request:

https://github.com/apache/spark/pull/9740#issuecomment-157128229
  
@liancheng This option is only for backward compatibility purpose, could we 
document it somewhere? We should also mention this change (isolated sessions) 
in release note.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11089] [SQL] Adds option for disabling ...

[liancheng]

Github user liancheng commented on the pull request:

https://github.com/apache/spark/pull/9740#issuecomment-157134508
  
Yeah, I'll add it to the programming guide.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11089] [SQL] Adds option for disabling ...

[davies]

Github user davies commented on the pull request:

https://github.com/apache/spark/pull/9740#issuecomment-157127825
  
@liancheng I like the this approach more, all global configurations should 
be placed in SparkConf (could be override in current session)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11089] [SQL] Adds option for disabling ...

[liancheng]

Github user liancheng commented on the pull request:

https://github.com/apache/spark/pull/9740#issuecomment-157121925
  
Another approach for this problem is to have "global" options. For example, 
`SET -g key=value` sets `key` to `value` for all sessions. This adds a new 
concept and requires more work, but seems to be cleaner and in general more 
useful. What need to be done are:

1. Add a new syntax `SET -g`
1. Each `HiveContext` should remember its parent `HiveContext` so that we 
can retrieve the "root" one. 
1. When a global option is set, it's always set in the root `HiveContext`.

Another benefit of this approach is that we don't need to restart the 
Thrift server to switch between single-/multi-session modes.

@marmbrus @davies @yhuai Any suggestions?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11089] [SQL] Adds option for disabling ...

[SparkQA]

Github user SparkQA commented on the pull request:

https://github.com/apache/spark/pull/9740#issuecomment-157157912
  
**[Test build #46000 has 
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/46000/consoleFull)**
 for PR 9740 at commit 
[`eeeb194`](https://github.com/apache/spark/commit/eeeb194d74ee3bdd4f439bc13886f20512fabc6b).
 * This patch passes all tests.
 * This patch merges cleanly.
 * This patch adds no public classes.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11089] [SQL] Adds option for disabling ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9740#issuecomment-157158102
  
Merged build finished. Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11089] [SQL] Adds option for disabling ...

[AmplabJenkins]

Github user AmplabJenkins commented on the pull request:

https://github.com/apache/spark/pull/9740#issuecomment-157158105
  
Test PASSed.
Refer to this link for build results (access rights to CI server needed): 
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/46000/
Test PASSed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11089] [SQL] Adds option for disabling ...

[marmbrus]

Github user marmbrus commented on the pull request:

https://github.com/apache/spark/pull/9740#issuecomment-157157618
  
+1 to this approach instead of adding a new concept.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11089] [SQL] Adds option for disabling ...

[davies]

Github user davies commented on the pull request:

https://github.com/apache/spark/pull/9740#issuecomment-157211822
  
@liancheng Will you update the programming guide in this PR?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request: [SPARK-11089] [SQL] Adds option for disabling ...

[liancheng]

Github user liancheng commented on the pull request:

https://github.com/apache/spark/pull/9740#issuecomment-157225690
  
Yeah, updating this one soon.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org