[jira] [Commented] (SPARK-20516) Spark SQL documentation out of date?
[
https://issues.apache.org/jira/browse/SPARK-20516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989816#comment-15989816
]
Sean Owen commented on SPARK-20516:
---
In general it should be an absolute path (URI). Relative may happen to work in
some contexts. It is not the case that the shell supports it for some reason
and not otherwise.
> Spark SQL documentation out of date?
>
>
> Key: SPARK-20516
> URL: https://issues.apache.org/jira/browse/SPARK-20516
> Project: Spark
> Issue Type: Task
> Components: SQL
>Affects Versions: 2.1.0
>Reporter: Ratandeep Ratti
>Priority: Minor
> Attachments: spark-20516.zip
>
>
> I was trying out the examples on the [Spark Sql
> page|https://spark.apache.org/docs/2.1.0/sql-programming-guide.html]. It
> seems that now we have to specify invoke {{master()}} on the SparkSession
> builder and also warehouseLocation is now a URI.
> I can fix the documentation (sql-programming-guide.html) and send a PR
> request.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20516) Spark SQL documentation out of date?
[
https://issues.apache.org/jira/browse/SPARK-20516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989302#comment-15989302
]
Ratandeep Ratti commented on SPARK-20516:
-
Ah. my bad {{spark-warehouse}} is a valid uri. I have confirmed that setting
{{warehouseLocation}} to a relative path does not work in a standalone program,
whereas setting it to absolute path does work. Spark-shell on the other hand
supports both absolute and relative paths without issue
> Spark SQL documentation out of date?
>
>
> Key: SPARK-20516
> URL: https://issues.apache.org/jira/browse/SPARK-20516
> Project: Spark
> Issue Type: Task
> Components: SQL
>Affects Versions: 2.1.0
>Reporter: Ratandeep Ratti
>Priority: Minor
> Attachments: spark-20516.zip
>
>
> I was trying out the examples on the [Spark Sql
> page|https://spark.apache.org/docs/2.1.0/sql-programming-guide.html]. It
> seems that now we have to specify invoke {{master()}} on the SparkSession
> builder and also warehouseLocation is now a URI.
> I can fix the documentation (sql-programming-guide.html) and send a PR
> request.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20516) Spark SQL documentation out of date?
[
https://issues.apache.org/jira/browse/SPARK-20516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989226#comment-15989226
]
Sean Owen commented on SPARK-20516:
---
{{spark-warehouse}} is a valid URI; it has no scheme, but it will be
interpreted anyway. Have you confirmed it doesn't work? it should, actually.
> Spark SQL documentation out of date?
>
>
> Key: SPARK-20516
> URL: https://issues.apache.org/jira/browse/SPARK-20516
> Project: Spark
> Issue Type: Task
> Components: SQL
>Affects Versions: 2.1.0
>Reporter: Ratandeep Ratti
>Priority: Minor
> Attachments: spark-20516.zip
>
>
> I was trying out the examples on the [Spark Sql
> page|https://spark.apache.org/docs/2.1.0/sql-programming-guide.html]. It
> seems that now we have to specify invoke {{master()}} on the SparkSession
> builder and also warehouseLocation is now a URI.
> I can fix the documentation (sql-programming-guide.html) and send a PR
> request.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20516) Spark SQL documentation out of date?
[
https://issues.apache.org/jira/browse/SPARK-20516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988989#comment-15988989
]
Ratandeep Ratti commented on SPARK-20516:
-
Yes, see this scala example:
https://spark.apache.org/docs/2.1.0/sql-programming-guide.html#tab_scala_15
where the {{warehouse}} loc is not a uri. Note that my point is that the
interpreting the warehouse location should be the same whether in a standalone
program or in Spark shell.
> Spark SQL documentation out of date?
>
>
> Key: SPARK-20516
> URL: https://issues.apache.org/jira/browse/SPARK-20516
> Project: Spark
> Issue Type: Task
> Components: SQL
>Affects Versions: 2.1.0
>Reporter: Ratandeep Ratti
>Priority: Minor
> Attachments: spark-20516.zip
>
>
> I was trying out the examples on the [Spark Sql
> page|https://spark.apache.org/docs/2.1.0/sql-programming-guide.html]. It
> seems that now we have to specify invoke {{master()}} on the SparkSession
> builder and also warehouseLocation is now a URI.
> I can fix the documentation (sql-programming-guide.html) and send a PR
> request.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20516) Spark SQL documentation out of date?
[
https://issues.apache.org/jira/browse/SPARK-20516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988718#comment-15988718
]
Sean Owen commented on SPARK-20516:
---
That's right, you need to specify master one way or the other but you generally
do it via spark-submit. For a stand-alone program you could set it in code. But
the docs are as intended. I don't understand what you're saying about the
exception. It is correct, that's an invalid URI. Do you see it documented
otherwise?
> Spark SQL documentation out of date?
>
>
> Key: SPARK-20516
> URL: https://issues.apache.org/jira/browse/SPARK-20516
> Project: Spark
> Issue Type: Task
> Components: SQL
>Affects Versions: 2.1.0
>Reporter: Ratandeep Ratti
>Priority: Minor
> Attachments: spark-20516.zip
>
>
> I was trying out the examples on the [Spark Sql
> page|https://spark.apache.org/docs/2.1.0/sql-programming-guide.html]. It
> seems that now we have to specify invoke {{master()}} on the SparkSession
> builder and also warehouseLocation is now a URI.
> I can fix the documentation (sql-programming-guide.html) and send a PR
> request.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20516) Spark SQL documentation out of date?
[
https://issues.apache.org/jira/browse/SPARK-20516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988690#comment-15988690
]
Ratandeep Ratti commented on SPARK-20516:
-
Hmm. I needed to specify the master as "local" when running it through my IDE.
I guess we don't need to when running it from a REPL? For the {{warehouseLoc}}
, this is the stacktrace I get.
{noformat}
Caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException:
Relative path in absolute URI: file:./spark-warehouse
at org.apache.hadoop.fs.Path.initialize(Path.java:206)
at org.apache.hadoop.fs.Path.(Path.java:197)
at
org.apache.hadoop.hive.metastore.Warehouse.getDnsPath(Warehouse.java:141)
at
org.apache.hadoop.hive.metastore.Warehouse.getDnsPath(Warehouse.java:146)
at
org.apache.hadoop.hive.metastore.Warehouse.getWhRoot(Warehouse.java:159)
at
org.apache.hadoop.hive.metastore.Warehouse.getDatabasePath(Warehouse.java:170)
at
org.apache.hadoop.hive.metastore.Warehouse.getTablePath(Warehouse.java:184)
at
org.apache.hadoop.hive.metastore.Warehouse.getFileStatusesForUnpartitionedTable(Warehouse.java:520)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.updateUnpartitionedTableStatsFast(MetaStoreUtils.java:179)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.updateUnpartitionedTableStatsFast(MetaStoreUtils.java:174)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1403)
at
org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:1449)
... 49 more
Caused by: java.net.URISyntaxException: Relative path in absolute URI:
file:./spark-warehouse
at java.net.URI.checkPath(URI.java:1823)
at java.net.URI.(URI.java:745)
at org.apache.hadoop.fs.Path.initialize(Path.java:203)
... 60 more
{noformat}
When I run the below code through my IDE
{code}
import org.apache.spark.sql.SparkSession
object SPARK_20516 {
def warehouseLoc: Unit = {
val warehouseLocation = "spark-warehouse"
val spark = SparkSession
.builder()
.master("local")
.appName("Spark Hive Example")
.config("spark.sql.warehouse.dir", warehouseLocation)
.enableHiveSupport()
.getOrCreate()
import spark.sql
sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)")
}
def main(args: Array[String]): Unit = {
warehouseLoc
}
}
{code}
> Spark SQL documentation out of date?
>
>
> Key: SPARK-20516
> URL: https://issues.apache.org/jira/browse/SPARK-20516
> Project: Spark
> Issue Type: Task
> Components: SQL
>Affects Versions: 2.1.0
>Reporter: Ratandeep Ratti
>Priority: Minor
>
> I was trying out the examples on the [Spark Sql
> page|https://spark.apache.org/docs/2.1.0/sql-programming-guide.html]. It
> seems that now we have to specify invoke {{master()}} on the SparkSession
> builder and also warehouseLocation is now a URI.
> I can fix the documentation (sql-programming-guide.html) and send a PR
> request.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20516) Spark SQL documentation out of date?
[
https://issues.apache.org/jira/browse/SPARK-20516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15987933#comment-15987933
]
Ratandeep Ratti commented on SPARK-20516:
-
Not sure why I cannot assign the ticket to myself. :/
> Spark SQL documentation out of date?
>
>
> Key: SPARK-20516
> URL: https://issues.apache.org/jira/browse/SPARK-20516
> Project: Spark
> Issue Type: Task
> Components: SQL
>Affects Versions: 2.1.0
>Reporter: Ratandeep Ratti
>
> I was trying out the examples on the [Spark Sql
> page|https://spark.apache.org/docs/2.1.0/sql-programming-guide.html]. It
> seems that now we have to specify invoke {{master()}} on the SparkSession
> builder and also warehouseLocation is now a URI.
> I can fix the documentation (sql-programming-guide.html) and send a PR
> request.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
