[
https://issues.apache.org/jira/browse/SPARK-19012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15784796#comment-15784796
]
Jork Zijlstra commented on SPARK-19012:
---------------------------------------
Good to see that its already being discussed.
MSSQL also has some limitation in tableOrViewNames which is described in the
documentation. Maybe updating the annotation of the method would also be
enough. Having an Exception with a clear reason would definitely already a fix.
[~hvanhovell]
We specify our queries inside a configuration not the code. So we have this in
our config:
dataPath = "hdfs://"
dataQuery: "SELECT column1, column2 FROM \[TABLE] WHERE 1 = 1"
Since we have one SparkSession for the application and the tableOrViewName is
coupled to that and we don't want to specify an extra config option for the
tableOrViewname, I though I'd just use the hashcode from the dataquery as the
tableOrViewName. Use that in the createOrReplaceTempView and replace \[TABLE]
inside the query with that.
{code}
val path = "hdfs://....{path}"
val dataQuery = "SELECT * FROM [TABLE] LIMIT 1"
val tableOrViewName = "_" + Math.abs(path.hashCode).toString +
Math.abs(qry.hashCode).toString
val df = sparkSession.read.orc(path)
df.createOrReplaceTempView(tableOrViewName)
val result = sparkSession.sqlContext.sql(qry.replace("[TABLE]",
tableOrViewName)).collect
{code}
> CreateOrReplaceTempView throws
> org.apache.spark.sql.catalyst.parser.ParseException when viewName first char
> is numerical
> ------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-19012
> URL: https://issues.apache.org/jira/browse/SPARK-19012
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.0.1, 2.0.2
> Reporter: Jork Zijlstra
>
> Using a viewName where the the fist char is a numerical value on
> dataframe.createOrReplaceTempView(viewName: String) causes:
> {code}
> Exception in thread "main"
> org.apache.spark.sql.catalyst.parser.ParseException:
> mismatched input '1468079114' expecting {'SELECT', 'FROM', 'ADD', 'AS',
> 'ALL', 'DISTINCT', 'WHERE', 'GROUP', 'BY', 'GROUPING', 'SETS', 'CUBE',
> 'ROLLUP', 'ORDER', 'HAVING', 'LIMIT', 'AT', 'OR', 'AND', 'IN', NOT, 'NO',
> 'EXISTS', 'BETWEEN', 'LIKE', RLIKE, 'IS', 'NULL', 'TRUE', 'FALSE', 'NULLS',
> 'ASC', 'DESC', 'FOR', 'INTERVAL', 'CASE', 'WHEN', 'THEN', 'ELSE', 'END',
> 'JOIN', 'CROSS', 'OUTER', 'INNER', 'LEFT', 'SEMI', 'RIGHT', 'FULL',
> 'NATURAL', 'ON', 'LATERAL', 'WINDOW', 'OVER', 'PARTITION', 'RANGE', 'ROWS',
> 'UNBOUNDED', 'PRECEDING', 'FOLLOWING', 'CURRENT', 'ROW', 'WITH', 'VALUES',
> 'CREATE', 'TABLE', 'VIEW', 'REPLACE', 'INSERT', 'DELETE', 'INTO', 'DESCRIBE',
> 'EXPLAIN', 'FORMAT', 'LOGICAL', 'CODEGEN', 'CAST', 'SHOW', 'TABLES',
> 'COLUMNS', 'COLUMN', 'USE', 'PARTITIONS', 'FUNCTIONS', 'DROP', 'UNION',
> 'EXCEPT', 'INTERSECT', 'TO', 'TABLESAMPLE', 'STRATIFY', 'ALTER', 'RENAME',
> 'ARRAY', 'MAP', 'STRUCT', 'COMMENT', 'SET', 'RESET', 'DATA', 'START',
> 'TRANSACTION', 'COMMIT', 'ROLLBACK', 'MACRO', 'IF', 'DIV', 'PERCENT',
> 'BUCKET', 'OUT', 'OF', 'SORT', 'CLUSTER', 'DISTRIBUTE', 'OVERWRITE',
> 'TRANSFORM', 'REDUCE', 'USING', 'SERDE', 'SERDEPROPERTIES', 'RECORDREADER',
> 'RECORDWRITER', 'DELIMITED', 'FIELDS', 'TERMINATED', 'COLLECTION', 'ITEMS',
> 'KEYS', 'ESCAPED', 'LINES', 'SEPARATED', 'FUNCTION', 'EXTENDED', 'REFRESH',
> 'CLEAR', 'CACHE', 'UNCACHE', 'LAZY', 'FORMATTED', TEMPORARY, 'OPTIONS',
> 'UNSET', 'TBLPROPERTIES', 'DBPROPERTIES', 'BUCKETS', 'SKEWED', 'STORED',
> 'DIRECTORIES', 'LOCATION', 'EXCHANGE', 'ARCHIVE', 'UNARCHIVE', 'FILEFORMAT',
> 'TOUCH', 'COMPACT', 'CONCATENATE', 'CHANGE', 'CASCADE', 'RESTRICT',
> 'CLUSTERED', 'SORTED', 'PURGE', 'INPUTFORMAT', 'OUTPUTFORMAT', DATABASE,
> DATABASES, 'DFS', 'TRUNCATE', 'ANALYZE', 'COMPUTE', 'LIST', 'STATISTICS',
> 'PARTITIONED', 'EXTERNAL', 'DEFINED', 'REVOKE', 'GRANT', 'LOCK', 'UNLOCK',
> 'MSCK', 'REPAIR', 'RECOVER', 'EXPORT', 'IMPORT', 'LOAD', 'ROLE', 'ROLES',
> 'COMPACTIONS', 'PRINCIPALS', 'TRANSACTIONS', 'INDEX', 'INDEXES', 'LOCKS',
> 'OPTION', 'ANTI', 'LOCAL', 'INPATH', 'CURRENT_DATE', 'CURRENT_TIMESTAMP',
> IDENTIFIER, BACKQUOTED_IDENTIFIER}(line 1, pos 0)
> == SQL ==
> 11111
> {code}
> {code}
> val tableOrViewName = "11111" //fails
> val tableOrViewName = "a1111" //works
> sparkSession.read.orc(path).createOrReplaceTempView(tableOrViewName)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
