[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/4758#issuecomment-76503823 Cool, thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/4758#issuecomment-76499219 @andrewor14 I just merged it to master and branch-1.3. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/4758 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4758#issuecomment-76456876 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28077/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4758#issuecomment-76456868 [Test build #28077 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28077/consoleFull) for PR 4758 at commit [`252fa0f`](https://github.com/apache/spark/commit/252fa0f70a93be81cf7b1151d720b5b7703cc011). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/4758#issuecomment-76444830 Updated, you can merge it after Jenkins nods :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4758#issuecomment-76444880 [Test build #28077 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28077/consoleFull) for PR 4758 at commit [`252fa0f`](https://github.com/apache/spark/commit/252fa0f70a93be81cf7b1151d720b5b7703cc011). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/4758#discussion_r25526262
--- Diff:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala
---
@@ -315,7 +315,14 @@ abstract class HiveThriftServer2Test extends FunSuite
with BeforeAndAfterAll wit
logInfo(s"Trying to start HiveThriftServer2: port=$port, mode=$mode,
attempt=$attempt")
-logPath = Process(command, None, "SPARK_TESTING" ->
"0").lines.collectFirst {
+val env = Seq(
+ // Disables SPARK_TESTING to exclude log4j.properties in test
directories.
+ "SPARK_TESTING" -> "0",
+ // Points SPARK_PID_DIR to SPARK_HOME, otherwise only 1 Thrift
server instance can be started
+ // at a time, which is not Jenkins friendly.
+ "SPARK_PID_DIR" -> "../../")
--- End diff --
Updated, you can merge it after Jenkins nods :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4758#discussion_r25485273
--- Diff:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala
---
@@ -315,7 +315,14 @@ abstract class HiveThriftServer2Test extends FunSuite
with BeforeAndAfterAll wit
logInfo(s"Trying to start HiveThriftServer2: port=$port, mode=$mode,
attempt=$attempt")
-logPath = Process(command, None, "SPARK_TESTING" ->
"0").lines.collectFirst {
+val env = Seq(
+ // Disables SPARK_TESTING to exclude log4j.properties in test
directories.
+ "SPARK_TESTING" -> "0",
+ // Points SPARK_PID_DIR to SPARK_HOME, otherwise only 1 Thrift
server instance can be started
+ // at a time, which is not Jenkins friendly.
+ "SPARK_PID_DIR" -> "../../")
--- End diff --
oh you're right. Looks like we currently set it to `/tmp` already, which is
probably what `java.io.tmpdir` is anyway in most cases
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/4758#discussion_r25479512
--- Diff:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala
---
@@ -315,7 +315,14 @@ abstract class HiveThriftServer2Test extends FunSuite
with BeforeAndAfterAll wit
logInfo(s"Trying to start HiveThriftServer2: port=$port, mode=$mode,
attempt=$attempt")
-logPath = Process(command, None, "SPARK_TESTING" ->
"0").lines.collectFirst {
+val env = Seq(
+ // Disables SPARK_TESTING to exclude log4j.properties in test
directories.
+ "SPARK_TESTING" -> "0",
+ // Points SPARK_PID_DIR to SPARK_HOME, otherwise only 1 Thrift
server instance can be started
+ // at a time, which is not Jenkins friendly.
+ "SPARK_PID_DIR" -> "../../")
--- End diff --
We need a different directory for each run, otherwise the PID file will be
overwritten when multiple jenkins unit tests run parallel.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Github user andrewor14 commented on a diff in the pull request:
https://github.com/apache/spark/pull/4758#discussion_r25469585
--- Diff:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala
---
@@ -315,7 +315,14 @@ abstract class HiveThriftServer2Test extends FunSuite
with BeforeAndAfterAll wit
logInfo(s"Trying to start HiveThriftServer2: port=$port, mode=$mode,
attempt=$attempt")
-logPath = Process(command, None, "SPARK_TESTING" ->
"0").lines.collectFirst {
+val env = Seq(
+ // Disables SPARK_TESTING to exclude log4j.properties in test
directories.
+ "SPARK_TESTING" -> "0",
+ // Points SPARK_PID_DIR to SPARK_HOME, otherwise only 1 Thrift
server instance can be started
+ // at a time, which is not Jenkins friendly.
+ "SPARK_PID_DIR" -> "../../")
--- End diff --
actually, can we just use `sys.props(java.io.tmpdir)` instead of creating a
new directory?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/4758#issuecomment-76284574 I said I was going to merge this, but instead I left one more comment inline. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/4758#issuecomment-76284089 I'm going to merge this with @chenghao-intel's suggestion. Thanks everyone. This is going into master 1.3 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Github user liancheng commented on a diff in the pull request:
https://github.com/apache/spark/pull/4758#discussion_r25414445
--- Diff:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala
---
@@ -315,7 +315,14 @@ abstract class HiveThriftServer2Test extends FunSuite
with BeforeAndAfterAll wit
logInfo(s"Trying to start HiveThriftServer2: port=$port, mode=$mode,
attempt=$attempt")
-logPath = Process(command, None, "SPARK_TESTING" ->
"0").lines.collectFirst {
+val env = Seq(
+ // Disables SPARK_TESTING to exclude log4j.properties in test
directories.
+ "SPARK_TESTING" -> "0",
+ // Points SPARK_PID_DIR to SPARK_HOME, otherwise only 1 Thrift
server instance can be started
+ // at a time, which is not Jenkins friendly.
+ "SPARK_PID_DIR" -> "../../")
--- End diff --
Good point.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4758#issuecomment-75909369 [Test build #27935 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27935/consoleFull) for PR 4758 at commit [`1b3d1e3`](https://github.com/apache/spark/commit/1b3d1e3f7215c09aefe28bc0c0b75c3e09de3f8d). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4758#issuecomment-75909373 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27935/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Github user chenghao-intel commented on a diff in the pull request:
https://github.com/apache/spark/pull/4758#discussion_r25318071
--- Diff:
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala
---
@@ -315,7 +315,14 @@ abstract class HiveThriftServer2Test extends FunSuite
with BeforeAndAfterAll wit
logInfo(s"Trying to start HiveThriftServer2: port=$port, mode=$mode,
attempt=$attempt")
-logPath = Process(command, None, "SPARK_TESTING" ->
"0").lines.collectFirst {
+val env = Seq(
+ // Disables SPARK_TESTING to exclude log4j.properties in test
directories.
+ "SPARK_TESTING" -> "0",
+ // Points SPARK_PID_DIR to SPARK_HOME, otherwise only 1 Thrift
server instance can be started
+ // at a time, which is not Jenkins friendly.
+ "SPARK_PID_DIR" -> "../../")
--- End diff --
I am wondering a better path would be `Utils.createTempDir(..)`, just in
case we want to run the test suite simultaneously in the local machine as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4758#issuecomment-75904355 [Test build #27935 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/27935/consoleFull) for PR 4758 at commit [`1b3d1e3`](https://github.com/apache/spark/commit/1b3d1e3f7215c09aefe28bc0c0b75c3e09de3f8d). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PI...
GitHub user liancheng opened a pull request: https://github.com/apache/spark/pull/4758 [SPARK-5751] [SQL] Sets SPARK_HOME as SPARK_PID_DIR when running Thrift server test suites This is a follow-up of #4720. By default, `spark-daemon.sh` writes PID files under `/tmp`, which makes it impossible to start multiple server instances simultaneously. This PR sets `SPARK_PID_DIR` to Spark home directory to workaround this problem. Many thanks to @chenghao-intel for pointing out this issue! You can merge this pull request into a Git repository by running: $ git pull https://github.com/liancheng/spark thriftserver-pid-dir Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4758.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4758 commit 1b3d1e3f7215c09aefe28bc0c0b75c3e09de3f8d Author: Cheng Lian Date: 2015-02-25T04:31:58Z Sets SPARK_HOME as SPARK_PID_DIR when running Thrift server test suites --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
