[jira] [Commented] (TOREE-411) Hive tables are not visible from Toree
[
https://issues.apache.org/jira/browse/TOREE-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16999363#comment-16999363
]
sdhalex commented on TOREE-411:
---
Does the second method offered by Ryan Blue effective？ we don`t need to set the
catalog implementation outside of Toree?
> Hive tables are not visible from Toree
> --
>
> Key: TOREE-411
> URL: https://issues.apache.org/jira/browse/TOREE-411
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
> Environment: hadoop-2.7.3 Spark 2.1.0 Hive 2.1
>Reporter: Alex Adria Cuadripani
>Priority: Major
> Labels: hadoop, hive, spark
> Fix For: 0.2.0
>
>
> I am trying to work with Hive tables but is impossible to see it with toree.
> I do:
> from pyspark.sql import SparkSession
> spark =
> SparkSession.builder.master("spark://spark-master-01:7077").enableHiveSupport().getOrCreate()
> spark.sql("show tables").show()
> It works perfectly in pyspark kernel (ipykernel), but in toree kernel it
> returns an empty list.
> I have seen issue 353, but I don't know if it is the same issue, or what I
> have to do exactly.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (TOREE-411) Hive tables are not visible from Toree
[
https://issues.apache.org/jira/browse/TOREE-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16045657#comment-16045657
]
Ryan Blue commented on TOREE-411:
-
The solution to this problem that we use is to set the catalog implementation
to "hive", as suggested by Xin, in our Spark defaults.
Also, TOREE-390 was just committed, which changes the Spark session so that it
is lazily created when you reference {{spark}} or picks up an existing context.
That means you can either use the default session or create your own and call
{{enableHiveSupport}}, among other configuration options. This should address
the problem without needing to set the catalog implementation outside of Toree.
> Hive tables are not visible from Toree
> --
>
> Key: TOREE-411
> URL: https://issues.apache.org/jira/browse/TOREE-411
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
> Environment: hadoop-2.7.3 Spark 2.1.0 Hive 2.1
>Reporter: Alex Adria Cuadripani
> Labels: hadoop, hive, spark
> Fix For: 0.2.0
>
>
> I am trying to work with Hive tables but is impossible to see it with toree.
> I do:
> from pyspark.sql import SparkSession
> spark =
> SparkSession.builder.master("spark://spark-master-01:7077").enableHiveSupport().getOrCreate()
> spark.sql("show tables").show()
> It works perfectly in pyspark kernel (ipykernel), but in toree kernel it
> returns an empty list.
> I have seen issue 353, but I don't know if it is the same issue, or what I
> have to do exactly.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TOREE-411) Hive tables are not visible from Toree
[
https://issues.apache.org/jira/browse/TOREE-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16045089#comment-16045089
]
Xin Wu commented on TOREE-411:
--
Good. Can you close the issue?
> Hive tables are not visible from Toree
> --
>
> Key: TOREE-411
> URL: https://issues.apache.org/jira/browse/TOREE-411
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
> Environment: hadoop-2.7.3 Spark 2.1.0 Hive 2.1
>Reporter: Alex Adria Cuadripani
> Labels: hadoop, hive, spark
>
> I am trying to work with Hive tables but is impossible to see it with toree.
> I do:
> from pyspark.sql import SparkSession
> spark =
> SparkSession.builder.master("spark://spark-master-01:7077").enableHiveSupport().getOrCreate()
> spark.sql("show tables").show()
> It works perfectly in pyspark kernel (ipykernel), but in toree kernel it
> returns an empty list.
> I have seen issue 353, but I don't know if it is the same issue, or what I
> have to do exactly.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TOREE-411) Hive tables are not visible from Toree
[
https://issues.apache.org/jira/browse/TOREE-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16044128#comment-16044128
]
Alex Adria Cuadripani commented on TOREE-411:
-
Tested on server and working perfectly too. The solution works perfectly with
Hive 1.2.1 and 2.1. Thanks again and regards.
> Hive tables are not visible from Toree
> --
>
> Key: TOREE-411
> URL: https://issues.apache.org/jira/browse/TOREE-411
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
> Environment: hadoop-2.7.3 Spark 2.1.0 Hive 2.1
>Reporter: Alex Adria Cuadripani
> Labels: hadoop, hive, spark
>
> I am trying to work with Hive tables but is impossible to see it with toree.
> I do:
> from pyspark.sql import SparkSession
> spark =
> SparkSession.builder.master("spark://spark-master-01:7077").enableHiveSupport().getOrCreate()
> spark.sql("show tables").show()
> It works perfectly in pyspark kernel (ipykernel), but in toree kernel it
> returns an empty list.
> I have seen issue 353, but I don't know if it is the same issue, or what I
> have to do exactly.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TOREE-411) Hive tables are not visible from Toree
[
https://issues.apache.org/jira/browse/TOREE-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16044074#comment-16044074
]
Alex Adria Cuadripani commented on TOREE-411:
-
Thanks a lot. It worked perfectly in my local standalone installation. I will
check it on server and give you feedback.
> Hive tables are not visible from Toree
> --
>
> Key: TOREE-411
> URL: https://issues.apache.org/jira/browse/TOREE-411
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
> Environment: hadoop-2.7.3 Spark 2.1.0 Hive 2.1
>Reporter: Alex Adria Cuadripani
> Labels: hadoop, hive, spark
>
> I am trying to work with Hive tables but is impossible to see it with toree.
> I do:
> from pyspark.sql import SparkSession
> spark =
> SparkSession.builder.master("spark://spark-master-01:7077").enableHiveSupport().getOrCreate()
> spark.sql("show tables").show()
> It works perfectly in pyspark kernel (ipykernel), but in toree kernel it
> returns an empty list.
> I have seen issue 353, but I don't know if it is the same issue, or what I
> have to do exactly.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TOREE-411) Hive tables are not visible from Toree
[
https://issues.apache.org/jira/browse/TOREE-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16044012#comment-16044012
]
Xin Wu commented on TOREE-411:
--
The workaround is to start the three kernel in hive catalog mode.
Add this property into spark-defaults.conf:
{code}spark.sql.catalogImplementation hive{code}
> Hive tables are not visible from Toree
> --
>
> Key: TOREE-411
> URL: https://issues.apache.org/jira/browse/TOREE-411
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
> Environment: hadoop-2.7.3 Spark 2.1.0 Hive 2.1
>Reporter: Alex Adria Cuadripani
> Labels: hadoop, hive, spark
>
> I am trying to work with Hive tables but is impossible to see it with toree.
> I do:
> from pyspark.sql import SparkSession
> spark =
> SparkSession.builder.master("spark://spark-master-01:7077").enableHiveSupport().getOrCreate()
> spark.sql("show tables").show()
> It works perfectly in pyspark kernel (ipykernel), but in toree kernel it
> returns an empty list.
> I have seen issue 353, but I don't know if it is the same issue, or what I
> have to do exactly.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TOREE-411) Hive tables are not visible from Toree
[
https://issues.apache.org/jira/browse/TOREE-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043739#comment-16043739
]
Xin Wu commented on TOREE-411:
--
The root cause is found in Spark.
Toree starts the default SparkSession object without Hive support. so the
catalog is in-memory catalog. Now even though you build a new session with
SparkSession.builder.enableHiveSupport, the underlined catalog object is still
in-memory catalog. So with the new spark session instance, you can still only
create tables under in-memory catalog, not in Hive catalog.
The reason you can see it working in pyspark shell, is that pyspark starts with
hive support by default, such that the underlined catalog object is
HiveExternalCatalog.
This is Spark issue. I will check with [~smilegator] for this.
> Hive tables are not visible from Toree
> --
>
> Key: TOREE-411
> URL: https://issues.apache.org/jira/browse/TOREE-411
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
> Environment: hadoop-2.7.3 Spark 2.1.0 Hive 2.1
>Reporter: Alex Adria Cuadripani
> Labels: hadoop, hive, spark
>
> I am trying to work with Hive tables but is impossible to see it with toree.
> I do:
> from pyspark.sql import SparkSession
> spark =
> SparkSession.builder.master("spark://spark-master-01:7077").enableHiveSupport().getOrCreate()
> spark.sql("show tables").show()
> It works perfectly in pyspark kernel (ipykernel), but in toree kernel it
> returns an empty list.
> I have seen issue 353, but I don't know if it is the same issue, or what I
> have to do exactly.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TOREE-411) Hive tables are not visible from Toree
[
https://issues.apache.org/jira/browse/TOREE-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043350#comment-16043350
]
Xin Wu commented on TOREE-411:
--
do you happen to see a metastore_db directory under the current directory where
toree kernel is launched?
Do you have hive-site.xml under the spark conf directory? What is inside the
hive-site.xml, especially the value of the metastore uris ?
> Hive tables are not visible from Toree
> --
>
> Key: TOREE-411
> URL: https://issues.apache.org/jira/browse/TOREE-411
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
> Environment: hadoop-2.7.3 Spark 2.1.0 Hive 2.1
>Reporter: Alex Adria Cuadripani
> Labels: hadoop, hive, spark
>
> I am trying to work with Hive tables but is impossible to see it with toree.
> I do:
> from pyspark.sql import SparkSession
> spark =
> SparkSession.builder.master("spark://spark-master-01:7077").enableHiveSupport().getOrCreate()
> spark.sql("show tables").show()
> It works perfectly in pyspark kernel (ipykernel), but in toree kernel it
> returns an empty list.
> I have seen issue 353, but I don't know if it is the same issue, or what I
> have to do exactly.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TOREE-411) Hive tables are not visible from Toree
[
https://issues.apache.org/jira/browse/TOREE-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043158#comment-16043158
]
Alex Adria Cuadripani commented on TOREE-411:
-
It returns:
++-+---+
|database|tableName|isTemporary|
++-+---+
|mydb| t1| false|
++-+---+
And the new folder is created in the correct warehouse folder together with the
other tables, that I can't see with toree.
In postgresql metastore database, the new table hasn't been stored into "tbls"
table.
> Hive tables are not visible from Toree
> --
>
> Key: TOREE-411
> URL: https://issues.apache.org/jira/browse/TOREE-411
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
> Environment: hadoop-2.7.3 Spark 2.1.0 Hive 2.1
>Reporter: Alex Adria Cuadripani
> Labels: hadoop, hive, spark
>
> I am trying to work with Hive tables but is impossible to see it with toree.
> I do:
> from pyspark.sql import SparkSession
> spark =
> SparkSession.builder.master("spark://spark-master-01:7077").enableHiveSupport().getOrCreate()
> spark.sql("show tables").show()
> It works perfectly in pyspark kernel (ipykernel), but in toree kernel it
> returns an empty list.
> I have seen issue 353, but I don't know if it is the same issue, or what I
> have to do exactly.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TOREE-411) Hive tables are not visible from Toree
[
https://issues.apache.org/jira/browse/TOREE-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043106#comment-16043106
]
Xin Wu commented on TOREE-411:
--
This is the same issue with 353.
What if you do
{code}
spark.sql("create database mydb")
spark.sql("create table mydb.t1 (c1 int, c2 int) stored as parquet")
spark.sql("use mydb")
spark.sql("show tables").show
{code}
> Hive tables are not visible from Toree
> --
>
> Key: TOREE-411
> URL: https://issues.apache.org/jira/browse/TOREE-411
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
> Environment: hadoop-2.7.3 Spark 2.1.0 Hive 2.1
>Reporter: Alex Adria Cuadripani
> Labels: hadoop, hive, spark
>
> I am trying to work with Hive tables but is impossible to see it with toree.
> I do:
> from pyspark.sql import SparkSession
> spark =
> SparkSession.builder.master("spark://spark-master-01:7077").enableHiveSupport().getOrCreate()
> spark.sql("show tables").show()
> It works perfectly in pyspark kernel (ipykernel), but in toree kernel it
> returns an empty list.
> I have seen issue 353, but I don't know if it is the same issue, or what I
> have to do exactly.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TOREE-411) Hive tables are not visible from Toree
[
https://issues.apache.org/jira/browse/TOREE-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16042991#comment-16042991
]
Alex Adria Cuadripani commented on TOREE-411:
-
Thanks for your help, this command are returning the text "hive"
> Hive tables are not visible from Toree
> --
>
> Key: TOREE-411
> URL: https://issues.apache.org/jira/browse/TOREE-411
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
> Environment: hadoop-2.7.3 Spark 2.1.0 Hive 2.1
>Reporter: Alex Adria Cuadripani
> Labels: hadoop, hive, spark
>
> I am trying to work with Hive tables but is impossible to see it with toree.
> I do:
> from pyspark.sql import SparkSession
> spark =
> SparkSession.builder.master("spark://spark-master-01:7077").enableHiveSupport().getOrCreate()
> spark.sql("show tables").show()
> It works perfectly in pyspark kernel (ipykernel), but in toree kernel it
> returns an empty list.
> I have seen issue 353, but I don't know if it is the same issue, or what I
> have to do exactly.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TOREE-411) Hive tables are not visible from Toree
[
https://issues.apache.org/jira/browse/TOREE-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16042940#comment-16042940
]
Xin Wu commented on TOREE-411:
--
For the Toree kernel, can you do {code}
spark.conf.get("spark.sql.catalogImplementation"){code} ? This is to see if
spark sql is using the in-memory catalog or hive.
> Hive tables are not visible from Toree
> --
>
> Key: TOREE-411
> URL: https://issues.apache.org/jira/browse/TOREE-411
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
> Environment: hadoop-2.7.3 Spark 2.1.0 Hive 2.1
>Reporter: Alex Adria Cuadripani
> Labels: hadoop, hive, spark
>
> I am trying to work with Hive tables but is impossible to see it with toree.
> I do:
> from pyspark.sql import SparkSession
> spark =
> SparkSession.builder.master("spark://spark-master-01:7077").enableHiveSupport().getOrCreate()
> spark.sql("show tables").show()
> It works perfectly in pyspark kernel (ipykernel), but in toree kernel it
> returns an empty list.
> I have seen issue 353, but I don't know if it is the same issue, or what I
> have to do exactly.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (TOREE-411) Hive tables are not visible from Toree
[
https://issues.apache.org/jira/browse/TOREE-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16042787#comment-16042787
]
Alex Adria Cuadripani commented on TOREE-411:
-
Downgraded to Hive 1.2.1 and the problem persists. The database that we use is
postgresql.
> Hive tables are not visible from Toree
> --
>
> Key: TOREE-411
> URL: https://issues.apache.org/jira/browse/TOREE-411
> Project: TOREE
> Issue Type: Bug
> Components: Kernel
> Environment: hadoop-2.7.3 Spark 2.1.0 Hive 2.1
>Reporter: Alex Adria Cuadripani
> Labels: hadoop, hive, spark
>
> I am trying to work with Hive tables but is impossible to see it with toree.
> I do:
> from pyspark.sql import SparkSession
> spark =
> SparkSession.builder.master("spark://spark-master-01:7077").enableHiveSupport().getOrCreate()
> spark.sql("show tables").show()
> It works perfectly in pyspark kernel (ipykernel), but in toree kernel it
> returns an empty list.
> I have seen issue 353, but I don't know if it is the same issue, or what I
> have to do exactly.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
