[
https://issues.apache.org/jira/browse/SPARK-15486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon resolved SPARK-15486.
----------------------------------
Resolution: Incomplete
> dropTempTable does not work with backticks
> ------------------------------------------
>
> Key: SPARK-15486
> URL: https://issues.apache.org/jira/browse/SPARK-15486
> Project: Spark
> Issue Type: Bug
> Affects Versions: 2.0.0
> Reporter: Luca Bruno
> Priority: Major
> Labels: bulk-closed
>
> We're using spark commit db75ccb (not sure if that's unreleased 2.0.0 or
> 2.1.0).
> We don't use Hive as we have a custom filesystem hierarchy and we like to use
> dots in table names. For this reason we use backticks when registering
> temporary tables.
> We have noticed that dropTempTable doesn't work as expected when using
> backticks, as can be reproduced with the code below:
> {code}
> from pyspark import SparkContext
> from pyspark.sql import SQLContext
> sc = SparkContext()
> sqlc = SQLContext(sc)
> data = sc.parallelize([ { "col": "val" } ])
> df = sqlc.createDataFrame(data)
> df.registerTempTable("`a.b.c`")
> print sqlc.sql("select * from `a.b.c`").collect()
> sqlc.dropTempTable("`a.b.c`")
> print sqlc.sql("select * from `a.b.c`").collect()
> {code}
> The above code will print the dataframe twice. We instead expect the second
> collect to fail because the table shouldn't exist. It seems the dropTempTable
> is failing silently.
> Removing backticks from registerTempTable or dropTempTable is not an option
> because we'd get an invalid syntax exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
