[GitHub] spark pull request: [SPARK-15565] [SQL] Add the File Scheme to the...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/13348 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15565] [SQL] Add the File Scheme to the...
Github user yhuai commented on the pull request: https://github.com/apache/spark/pull/13348#issuecomment-222197726 Thanks. Merging to master and branch 2.0. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15565] [SQL] Add the File Scheme to the...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13348#issuecomment-222184611 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59495/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15565] [SQL] Add the File Scheme to the...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13348#issuecomment-222184609 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15565] [SQL] Add the File Scheme to the...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13348#issuecomment-222184334 **[Test build #59495 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59495/consoleFull)** for PR 13348 at commit [`1dbfb3b`](https://github.com/apache/spark/commit/1dbfb3b72cac297e335873000bfe66d88343b34c). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15565] [SQL] Add the File Scheme to the...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13348#issuecomment-222163324 **[Test build #59495 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59495/consoleFull)** for PR 13348 at commit [`1dbfb3b`](https://github.com/apache/spark/commit/1dbfb3b72cac297e335873000bfe66d88343b34c). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15565] [SQL] Add the File Scheme to the...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/13348#discussion_r64913387
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -55,7 +56,7 @@ object SQLConf {
val WAREHOUSE_PATH = SQLConfigBuilder("spark.sql.warehouse.dir")
.doc("The default location for managed databases and tables.")
.stringConf
-.createWithDefault("${system:user.dir}/spark-warehouse")
+.createWithDefault("${system:user.dir}" + File.separator +
"spark-warehouse")
--- End diff --
After rethinking it, let me change it by using your suggested way. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15565] [SQL] Add the File Scheme to the...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13348#issuecomment-222072466 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59461/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15565] [SQL] Add the File Scheme to the...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13348#issuecomment-222072464 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15565] [SQL] Add the File Scheme to the...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13348#issuecomment-222072291 **[Test build #59461 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59461/consoleFull)** for PR 13348 at commit [`cfc02af`](https://github.com/apache/spark/commit/cfc02afb1e04218044a594b408252f7f74dcea7b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15565] [SQL] Add the File Scheme to the...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/13348#issuecomment-222065914 ok that makes sense --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15565] [SQL] Add the File Scheme to the...
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/13348#issuecomment-222065198
I created the jira because of
https://issues.apache.org/jira/browse/SPARK-15034?focusedCommentId=15301508&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15301508.
Since `System.getProperty("user.dir")` is pointing to a local dir, seems it
makes to explicitly set the scheme. It may introduce confusion if we add the
default fs scheme to the path generated by `System.getProperty("user.dir")`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15565] [SQL] Add the File Scheme to the...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/13348#discussion_r64859611
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -55,7 +56,7 @@ object SQLConf {
val WAREHOUSE_PATH = SQLConfigBuilder("spark.sql.warehouse.dir")
.doc("The default location for managed databases and tables.")
.stringConf
-.createWithDefault("${system:user.dir}/spark-warehouse")
+.createWithDefault("${system:user.dir}" + File.separator +
"spark-warehouse")
--- End diff --
Yeah, we can do it in that way. I am just afraid it might be not always
right. Below are two related discussions:
https://en.wikipedia.org/wiki/File_URI_scheme#Meaning_of_slash_character
http://stackoverflow.com/questions/1131273/java-file-touri-tourl-on-windows-file?answertab=votes#tab-top
Let me know which way is preferred.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15565] [SQL] Add the File Scheme to the...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/13348#issuecomment-222064155 Should we do this? This means that when there is a HDFS setup, the warehouse directory will go to local? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15565] [SQL] Add the File Scheme to the...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/13348#discussion_r64858995
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -55,7 +56,7 @@ object SQLConf {
val WAREHOUSE_PATH = SQLConfigBuilder("spark.sql.warehouse.dir")
.doc("The default location for managed databases and tables.")
.stringConf
-.createWithDefault("${system:user.dir}/spark-warehouse")
+.createWithDefault("${system:user.dir}" + File.separator +
"spark-warehouse")
--- End diff --
Why not just change the default value to
`file:/${system:user.dir}/spark-warehouse`? I do not really understand why we
have to change `def warehousePath`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15565] [SQL] Add the File Scheme to the...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/13348#discussion_r64858912
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -665,7 +666,8 @@ private[sql] class SQLConf extends Serializable with
CatalystConf with Logging {
def variableSubstituteDepth: Int = getConf(VARIABLE_SUBSTITUTE_DEPTH)
def warehousePath: String = {
-getConf(WAREHOUSE_PATH).replace("${system:user.dir}",
System.getProperty("user.dir"))
+getConf(WAREHOUSE_PATH).replace("${system:user.dir}",
+ new
File(System.getProperty("user.dir")).toURI.toURL.toString).replace("//", "/")
--- End diff --
Very interesting! In the function `normalizePath` of `Path`, it replaces
double forward slashes by a single forward slash.
https://hadoop.apache.org/docs/r2.4.1/api/src-html/org/apache/hadoop/fs/Path.html
We still need to do it here, even if `Path` can normalize it for us.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15565] [SQL] Add the File Scheme to the...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/13348#discussion_r64858638
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -55,7 +56,7 @@ object SQLConf {
val WAREHOUSE_PATH = SQLConfigBuilder("spark.sql.warehouse.dir")
.doc("The default location for managed databases and tables.")
.stringConf
-.createWithDefault("${system:user.dir}/spark-warehouse")
+.createWithDefault("${system:user.dir}" + File.separator +
"spark-warehouse")
--- End diff --
`Path` normalizes `backslashes` to forward slashes on Windows. Maybe it is
safe to use `File.separator`. If anybody hit an issue in Windows, we can make a
change then. : )
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15565] [SQL] Add the File Scheme to the...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/13348#discussion_r64858511
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -665,7 +666,8 @@ private[sql] class SQLConf extends Serializable with
CatalystConf with Logging {
def variableSubstituteDepth: Int = getConf(VARIABLE_SUBSTITUTE_DEPTH)
def warehousePath: String = {
-getConf(WAREHOUSE_PATH).replace("${system:user.dir}",
System.getProperty("user.dir"))
+getConf(WAREHOUSE_PATH).replace("${system:user.dir}",
+ new
File(System.getProperty("user.dir")).toURI.toURL.toString).replace("//", "/")
--- End diff --
In my local mac osx, `new
File(System.getProperty("user.dir")).toURI.toURL.toString)` will generate a
file path with a trailling `/`.
I am afraid this could be OS dependent? Thus, I did not change the default
value.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15565] [SQL] Add the File Scheme to the...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13348#issuecomment-222061692 **[Test build #59461 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59461/consoleFull)** for PR 13348 at commit [`cfc02af`](https://github.com/apache/spark/commit/cfc02afb1e04218044a594b408252f7f74dcea7b). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15565] [SQL] Add the File Scheme to the...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/13348#discussion_r64858379
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -665,7 +666,8 @@ private[sql] class SQLConf extends Serializable with
CatalystConf with Logging {
def variableSubstituteDepth: Int = getConf(VARIABLE_SUBSTITUTE_DEPTH)
def warehousePath: String = {
-getConf(WAREHOUSE_PATH).replace("${system:user.dir}",
System.getProperty("user.dir"))
+getConf(WAREHOUSE_PATH).replace("${system:user.dir}",
+ new
File(System.getProperty("user.dir")).toURI.toURL.toString).replace("//", "/")
--- End diff --
When do we need `.replace("//", "/")`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15565] [SQL] Add the File Scheme to the...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/13348#discussion_r64858019
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -55,7 +56,7 @@ object SQLConf {
val WAREHOUSE_PATH = SQLConfigBuilder("spark.sql.warehouse.dir")
.doc("The default location for managed databases and tables.")
.stringConf
-.createWithDefault("${system:user.dir}/spark-warehouse")
+.createWithDefault("${system:user.dir}" + File.separator +
"spark-warehouse")
--- End diff --
FYI, an interesting JIRA: https://issues.apache.org/jira/browse/HADOOP-1995
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15565] [SQL] Add the File Scheme to the...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13348#issuecomment-222060602 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15565] [SQL] Add the File Scheme to the...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/13348#issuecomment-222060603 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/59455/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15565] [SQL] Add the File Scheme to the...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13348#issuecomment-222060458 **[Test build #59455 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59455/consoleFull)** for PR 13348 at commit [`1c6b6ff`](https://github.com/apache/spark/commit/1c6b6ff7db713b5b02579d0fbdcfc3651e64b950). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15565] [SQL] Add the File Scheme to the...
Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/13348#discussion_r64857643
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -55,7 +56,7 @@ object SQLConf {
val WAREHOUSE_PATH = SQLConfigBuilder("spark.sql.warehouse.dir")
.doc("The default location for managed databases and tables.")
.stringConf
-.createWithDefault("${system:user.dir}/spark-warehouse")
+.createWithDefault("${system:user.dir}" + File.separator +
"spark-warehouse")
--- End diff --
You are right. Let me fix it. Thanks!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15565] [SQL] Add the File Scheme to the...
Github user yhuai commented on a diff in the pull request:
https://github.com/apache/spark/pull/13348#discussion_r64853158
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -55,7 +56,7 @@ object SQLConf {
val WAREHOUSE_PATH = SQLConfigBuilder("spark.sql.warehouse.dir")
.doc("The default location for managed databases and tables.")
.stringConf
-.createWithDefault("${system:user.dir}/spark-warehouse")
+.createWithDefault("${system:user.dir}" + File.separator +
"spark-warehouse")
--- End diff --
Although the path points to a local dir, this path is actually a hadoop
filesystem path. So, I am wondering if we need to use `File.separator` at here?
If we are running on Windows, does a hadoop filesystem path uses `\` as the
separator?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15565] [SQL] Add the File Scheme to the...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/13348#issuecomment-222049923 **[Test build #59455 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/59455/consoleFull)** for PR 13348 at commit [`1c6b6ff`](https://github.com/apache/spark/commit/1c6b6ff7db713b5b02579d0fbdcfc3651e64b950). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-15565] [SQL] Add the File Scheme to the...
GitHub user gatorsmile opened a pull request:
https://github.com/apache/spark/pull/13348
[SPARK-15565] [SQL] Add the File Scheme to the Default Value of
WAREHOUSE_PATH
What changes were proposed in this pull request?
The default value of `spark.sql.warehouse.dir` is
`System.getProperty("user.dir")/spark-warehouse`. Since
`System.getProperty("user.dir")` is a local dir, we should explicitly set the
scheme to local filesystem.
cc @yhuai
How was this patch tested?
Added two test cases
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/gatorsmile/spark
addSchemeToDefaultWarehousePath
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/13348.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #13348
commit 1c6b6ff7db713b5b02579d0fbdcfc3651e64b950
Author: gatorsmile
Date: 2016-05-27T03:01:38Z
add the file scheme to the default path of DB
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
