[jira] [Commented] (SPARK-4943) Parsing error for query with table name having dot
[
https://issues.apache.org/jira/browse/SPARK-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14278710#comment-14278710
]
Apache Spark commented on SPARK-4943:
-
User 'scwf' has created a pull request for this issue:
https://github.com/apache/spark/pull/4062
> Parsing error for query with table name having dot
> --
>
> Key: SPARK-4943
> URL: https://issues.apache.org/jira/browse/SPARK-4943
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.2.0
>Reporter: Alex Liu
> Fix For: 1.3.0, 1.2.1
>
>
> When integrating Spark 1.2.0 with Cassandra SQL, the following query is
> broken. It was working for Spark 1.1.0 version. Basically we use the table
> name having dot to include database name
> {code}
> [info] java.lang.RuntimeException: [1.29] failure: ``UNION'' expected but
> `.' found
> [info]
> [info] SELECT test1.a FROM sql_test.test1 AS test1 UNION DISTINCT SELECT
> test2.a FROM sql_test.test2 AS test2
> [info] ^
> [info] at scala.sys.package$.error(package.scala:27)
> [info] at
> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:33)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
> [info] at
> org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:174)
> [info] at
> org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:173)
> [info] at
> scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:136)
> [info] at
> scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:135)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Failure.append(Parsers.scala:202)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> [info] at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2.apply(Parsers.scala:890)
> [info] at
> scala.util.parsing.combinator.PackratParsers$$anon$1.apply(PackratParsers.scala:110)
> [info] at
> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:31)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$parseSql$1.apply(SQLContext.scala:83)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$parseSql$1.apply(SQLContext.scala:83)
> [info] at scala.Option.getOrElse(Option.scala:120)
> [info] at org.apache.spark.sql.SQLContext.parseSql(SQLContext.scala:83)
> [info] at
> org.apache.spark.sql.cassandra.CassandraSQLContext.cassandraSql(CassandraSQLContext.scala:53)
> [info] at
> org.apache.spark.sql.cassandra.CassandraSQLContext.sql(CassandraSQLContext.scala:56)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply$mcV$sp(CassandraSQLSpec.scala:169)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply(CassandraSQLSpec.scala:168)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply(CassandraSQLSpec.scala:168)
> [info] at
> org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
> [info] at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
> [info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> [info] at org.scalatest.Transformer.apply(Transformer.scala:22)
> [info] at org.scalatest.Transformer.apply(Transformer.scala:20)
> [info] at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:1647)
> [info] at org.scalatest.Suite$class.withFixture(Suite.scala:1122)
> [info] at org.scalatest.FlatSpec.withFixture(FlatSpec.sc
[jira] [Commented] (SPARK-4943) Parsing error for query with table name having dot
[
https://issues.apache.org/jira/browse/SPARK-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14268830#comment-14268830
]
Apache Spark commented on SPARK-4943:
-
User 'alexliu68' has created a pull request for this issue:
https://github.com/apache/spark/pull/3941
> Parsing error for query with table name having dot
> --
>
> Key: SPARK-4943
> URL: https://issues.apache.org/jira/browse/SPARK-4943
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.2.0
>Reporter: Alex Liu
>
> When integrating Spark 1.2.0 with Cassandra SQL, the following query is
> broken. It was working for Spark 1.1.0 version. Basically we use the table
> name having dot to include database name
> {code}
> [info] java.lang.RuntimeException: [1.29] failure: ``UNION'' expected but
> `.' found
> [info]
> [info] SELECT test1.a FROM sql_test.test1 AS test1 UNION DISTINCT SELECT
> test2.a FROM sql_test.test2 AS test2
> [info] ^
> [info] at scala.sys.package$.error(package.scala:27)
> [info] at
> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:33)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
> [info] at
> org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:174)
> [info] at
> org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:173)
> [info] at
> scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:136)
> [info] at
> scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:135)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Failure.append(Parsers.scala:202)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> [info] at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2.apply(Parsers.scala:890)
> [info] at
> scala.util.parsing.combinator.PackratParsers$$anon$1.apply(PackratParsers.scala:110)
> [info] at
> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:31)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$parseSql$1.apply(SQLContext.scala:83)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$parseSql$1.apply(SQLContext.scala:83)
> [info] at scala.Option.getOrElse(Option.scala:120)
> [info] at org.apache.spark.sql.SQLContext.parseSql(SQLContext.scala:83)
> [info] at
> org.apache.spark.sql.cassandra.CassandraSQLContext.cassandraSql(CassandraSQLContext.scala:53)
> [info] at
> org.apache.spark.sql.cassandra.CassandraSQLContext.sql(CassandraSQLContext.scala:56)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply$mcV$sp(CassandraSQLSpec.scala:169)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply(CassandraSQLSpec.scala:168)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply(CassandraSQLSpec.scala:168)
> [info] at
> org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
> [info] at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
> [info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> [info] at org.scalatest.Transformer.apply(Transformer.scala:22)
> [info] at org.scalatest.Transformer.apply(Transformer.scala:20)
> [info] at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:1647)
> [info] at org.scalatest.Suite$class.withFixture(Suite.scala:1122)
> [info] at org.scalatest.FlatSpec.withFixture(FlatSpec.scala:1683)
> [info] at
> org.sc
[jira] [Commented] (SPARK-4943) Parsing error for query with table name having dot
[
https://issues.apache.org/jira/browse/SPARK-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265351#comment-14265351
]
Alex Liu commented on SPARK-4943:
-
No changes to your approach.
Regarding cluster1.database.table1 and cluster2.database.table passing to
datasources. they are set as tableIdentifier and tableIdentifier is passed to
catalog.lookupRelation method where datasource can use it.
> Parsing error for query with table name having dot
> --
>
> Key: SPARK-4943
> URL: https://issues.apache.org/jira/browse/SPARK-4943
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.2.0
>Reporter: Alex Liu
>
> When integrating Spark 1.2.0 with Cassandra SQL, the following query is
> broken. It was working for Spark 1.1.0 version. Basically we use the table
> name having dot to include database name
> {code}
> [info] java.lang.RuntimeException: [1.29] failure: ``UNION'' expected but
> `.' found
> [info]
> [info] SELECT test1.a FROM sql_test.test1 AS test1 UNION DISTINCT SELECT
> test2.a FROM sql_test.test2 AS test2
> [info] ^
> [info] at scala.sys.package$.error(package.scala:27)
> [info] at
> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:33)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
> [info] at
> org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:174)
> [info] at
> org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:173)
> [info] at
> scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:136)
> [info] at
> scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:135)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Failure.append(Parsers.scala:202)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> [info] at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2.apply(Parsers.scala:890)
> [info] at
> scala.util.parsing.combinator.PackratParsers$$anon$1.apply(PackratParsers.scala:110)
> [info] at
> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:31)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$parseSql$1.apply(SQLContext.scala:83)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$parseSql$1.apply(SQLContext.scala:83)
> [info] at scala.Option.getOrElse(Option.scala:120)
> [info] at org.apache.spark.sql.SQLContext.parseSql(SQLContext.scala:83)
> [info] at
> org.apache.spark.sql.cassandra.CassandraSQLContext.cassandraSql(CassandraSQLContext.scala:53)
> [info] at
> org.apache.spark.sql.cassandra.CassandraSQLContext.sql(CassandraSQLContext.scala:56)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply$mcV$sp(CassandraSQLSpec.scala:169)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply(CassandraSQLSpec.scala:168)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply(CassandraSQLSpec.scala:168)
> [info] at
> org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
> [info] at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
> [info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> [info] at org.scalatest.Transformer.apply(Transformer.scala:22)
> [info] at org.scalatest.Transformer.apply(Transformer.scala:20)
> [info] at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:1647)
> [info] at org.scalatest.S
[jira] [Commented] (SPARK-4943) Parsing error for query with table name having dot
[
https://issues.apache.org/jira/browse/SPARK-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265335#comment-14265335
]
Michael Armbrust commented on SPARK-4943:
-
Thanks for your comments Alex. Are you proposing any changes to what I said?
Another thing I'm confused about is your comment regarding joins. As of now
there is no public API for passing that kind of information down into a
datasource.
Regarding the configuration. We will pass the datasource a SQLContext and you
can do {{.getConf}} using whatever arbitrary string you want. I don't think
Spark SQL needs to have any control here.
> Parsing error for query with table name having dot
> --
>
> Key: SPARK-4943
> URL: https://issues.apache.org/jira/browse/SPARK-4943
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.2.0
>Reporter: Alex Liu
>
> When integrating Spark 1.2.0 with Cassandra SQL, the following query is
> broken. It was working for Spark 1.1.0 version. Basically we use the table
> name having dot to include database name
> {code}
> [info] java.lang.RuntimeException: [1.29] failure: ``UNION'' expected but
> `.' found
> [info]
> [info] SELECT test1.a FROM sql_test.test1 AS test1 UNION DISTINCT SELECT
> test2.a FROM sql_test.test2 AS test2
> [info] ^
> [info] at scala.sys.package$.error(package.scala:27)
> [info] at
> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:33)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
> [info] at
> org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:174)
> [info] at
> org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:173)
> [info] at
> scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:136)
> [info] at
> scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:135)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Failure.append(Parsers.scala:202)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> [info] at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2.apply(Parsers.scala:890)
> [info] at
> scala.util.parsing.combinator.PackratParsers$$anon$1.apply(PackratParsers.scala:110)
> [info] at
> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:31)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$parseSql$1.apply(SQLContext.scala:83)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$parseSql$1.apply(SQLContext.scala:83)
> [info] at scala.Option.getOrElse(Option.scala:120)
> [info] at org.apache.spark.sql.SQLContext.parseSql(SQLContext.scala:83)
> [info] at
> org.apache.spark.sql.cassandra.CassandraSQLContext.cassandraSql(CassandraSQLContext.scala:53)
> [info] at
> org.apache.spark.sql.cassandra.CassandraSQLContext.sql(CassandraSQLContext.scala:56)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply$mcV$sp(CassandraSQLSpec.scala:169)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply(CassandraSQLSpec.scala:168)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply(CassandraSQLSpec.scala:168)
> [info] at
> org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
> [info] at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
> [info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> [info] at org.scalatest.Tr
[jira] [Commented] (SPARK-4943) Parsing error for query with table name having dot
[
https://issues.apache.org/jira/browse/SPARK-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265224#comment-14265224
]
Alex Liu commented on SPARK-4943:
-
For each catalog, the configuration settings should start with catalog name.
e.g.
{noformat}
set cassandra.cluster.database.table.ttl = 1000
set cassandra.database.table.ttl =1000 (default cluster)
set mysql.cluster.database.table.xxx = 200
{noformat}
If there's no catalog in the setting string, use the default catalog.
> Parsing error for query with table name having dot
> --
>
> Key: SPARK-4943
> URL: https://issues.apache.org/jira/browse/SPARK-4943
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.2.0
>Reporter: Alex Liu
>
> When integrating Spark 1.2.0 with Cassandra SQL, the following query is
> broken. It was working for Spark 1.1.0 version. Basically we use the table
> name having dot to include database name
> {code}
> [info] java.lang.RuntimeException: [1.29] failure: ``UNION'' expected but
> `.' found
> [info]
> [info] SELECT test1.a FROM sql_test.test1 AS test1 UNION DISTINCT SELECT
> test2.a FROM sql_test.test2 AS test2
> [info] ^
> [info] at scala.sys.package$.error(package.scala:27)
> [info] at
> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:33)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
> [info] at
> org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:174)
> [info] at
> org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:173)
> [info] at
> scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:136)
> [info] at
> scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:135)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Failure.append(Parsers.scala:202)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> [info] at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2.apply(Parsers.scala:890)
> [info] at
> scala.util.parsing.combinator.PackratParsers$$anon$1.apply(PackratParsers.scala:110)
> [info] at
> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:31)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$parseSql$1.apply(SQLContext.scala:83)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$parseSql$1.apply(SQLContext.scala:83)
> [info] at scala.Option.getOrElse(Option.scala:120)
> [info] at org.apache.spark.sql.SQLContext.parseSql(SQLContext.scala:83)
> [info] at
> org.apache.spark.sql.cassandra.CassandraSQLContext.cassandraSql(CassandraSQLContext.scala:53)
> [info] at
> org.apache.spark.sql.cassandra.CassandraSQLContext.sql(CassandraSQLContext.scala:56)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply$mcV$sp(CassandraSQLSpec.scala:169)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply(CassandraSQLSpec.scala:168)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply(CassandraSQLSpec.scala:168)
> [info] at
> org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
> [info] at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
> [info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> [info] at org.scalatest.Transformer.apply(Transformer.scala:22)
> [info] at org.scalatest.Transformer.apply(Transformer.scala:20)
> [info] at org.scalatest.F
[jira] [Commented] (SPARK-4943) Parsing error for query with table name having dot
[
https://issues.apache.org/jira/browse/SPARK-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265212#comment-14265212
]
Alex Liu commented on SPARK-4943:
-
Catalog part of table identifier should be handled by Spark SQL which calls the
registered catalog Context to connect to the underline datasources.
cluster/database/scheme/table should be handled by datasource(Cassandra Spark
SQL integration can handle cluster, database and table level join).
e.g.
{code}
SELECT test1.a, test1.b, test2.c FROM cassandra.cluster.database.table1 AS test1
LEFT OUTER JOIN mySql.cluster.database.table2 AS test2 ON test1.a = test2.a
{code}
so cluster.database.table1 is passed to cassandra catalog datasource, cassandra
is handled by Spark SQL to call cassandraContext which then call the underline
datasource.
cluster.database.table2 is passed to mySql catalog datasource, mySql is handled
by Spark SQL to call the mySqlContext which then call the underline datasource.
If USE command is used, then all tableIdentifiers are passed to datasource.
e.g.
{code}
USE cassandra
SELECT test1.a, test1.b, test2.c FROM cluster1.database.table AS test1
LEFT OUTER JOIN cluster2.database.table AS test2 ON test1.a = test2.a
{code}
cluster1.database.table1 and cluster2.database.table are passed to cassandra
datasource
> Parsing error for query with table name having dot
> --
>
> Key: SPARK-4943
> URL: https://issues.apache.org/jira/browse/SPARK-4943
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.2.0
>Reporter: Alex Liu
>
> When integrating Spark 1.2.0 with Cassandra SQL, the following query is
> broken. It was working for Spark 1.1.0 version. Basically we use the table
> name having dot to include database name
> {code}
> [info] java.lang.RuntimeException: [1.29] failure: ``UNION'' expected but
> `.' found
> [info]
> [info] SELECT test1.a FROM sql_test.test1 AS test1 UNION DISTINCT SELECT
> test2.a FROM sql_test.test2 AS test2
> [info] ^
> [info] at scala.sys.package$.error(package.scala:27)
> [info] at
> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:33)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
> [info] at
> org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:174)
> [info] at
> org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:173)
> [info] at
> scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:136)
> [info] at
> scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:135)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Failure.append(Parsers.scala:202)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> [info] at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2.apply(Parsers.scala:890)
> [info] at
> scala.util.parsing.combinator.PackratParsers$$anon$1.apply(PackratParsers.scala:110)
> [info] at
> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:31)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$parseSql$1.apply(SQLContext.scala:83)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$parseSql$1.apply(SQLContext.scala:83)
> [info] at scala.Option.getOrElse(Option.scala:120)
> [info] at org.apache.spark.sql.SQLContext.parseSql(SQLContext.scala:83)
> [info] at
> org.apache.spark.sql.cassandra.CassandraSQLContext.cassandraSql(CassandraSQLContext.scala:53)
> [info] a
[jira] [Commented] (SPARK-4943) Parsing error for query with table name having dot
[
https://issues.apache.org/jira/browse/SPARK-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265157#comment-14265157
]
Michael Armbrust commented on SPARK-4943:
-
I wouldn't say that the notion of {{tableIdentifier: Seq\[String\]}} is
unclear. Instead I would say that it is deliberately unspecified in order to
be flexible. Some systems have {{clusters}}, some systems have {{databases}},
some systems have {{schema}}, some have {{tables}}. Thus, this API gives us
one interface for communicating between the parser and the underlying datastore
that makes no assumption about how that datastore is laid out.
If we do make this change then yes, I agree that we should also make it in the
catalog as well. In general our handling of this has always been a little
clunky since there is a whole bunch of code that just ignores the database
field.
One question is: what parts of the table identifier Spark SQL handles and what
parts we pass on to the datasource? A goal here should be to be able to
connect to and join data from multiple sources. Here is what I would propose
as an addition to the current API, which only lets you register individual
tables.
- Users can register external catalogs which are responsible for producing
{{BaseRelation}}s.
- Each external catalog has a user specified name that is given when
registering.
- There is a notion of the current catalog, which can be changed with {{USE}}.
By default, we pass the all the {{tableIdentifiers}} to this default catalog
and its up to it to determine what each part means.
- Users can also specify fully qualified tables when joining multiple data
sources. We detect this case when the first {{tableIdentifier}} matches one of
the registered catalogs. In this case we strip of the catalog name and pass
the remaining {{tableIdentifiers}} to the specified catalog.
What do you think?
> Parsing error for query with table name having dot
> --
>
> Key: SPARK-4943
> URL: https://issues.apache.org/jira/browse/SPARK-4943
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.2.0
>Reporter: Alex Liu
>
> When integrating Spark 1.2.0 with Cassandra SQL, the following query is
> broken. It was working for Spark 1.1.0 version. Basically we use the table
> name having dot to include database name
> {code}
> [info] java.lang.RuntimeException: [1.29] failure: ``UNION'' expected but
> `.' found
> [info]
> [info] SELECT test1.a FROM sql_test.test1 AS test1 UNION DISTINCT SELECT
> test2.a FROM sql_test.test2 AS test2
> [info] ^
> [info] at scala.sys.package$.error(package.scala:27)
> [info] at
> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:33)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
> [info] at
> org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:174)
> [info] at
> org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:173)
> [info] at
> scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:136)
> [info] at
> scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:135)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Failure.append(Parsers.scala:202)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> [info] at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2.apply(Parsers.scala:890)
> [info] at
> scala.util.parsing.combinator.PackratParsers$$anon$1.apply
[jira] [Commented] (SPARK-4943) Parsing error for query with table name having dot
[
https://issues.apache.org/jira/browse/SPARK-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265089#comment-14265089
]
Alex Liu commented on SPARK-4943:
-
The approach of {code}
case class UnresolvedRelation(tableIdentifier: Seq[String], alias:
Option[String])
{code} is a little unclear about what's stored in tableIdentifier by simply
reading the code.
Another approach is storing catalog.cluster.database in databaseName and
tableName in tableName and keep case class no change
{code}
case class UnresolvedRelation(databaseName: Option[String], tableName: String,
alias: Option[String])
{code}
so no API changes.
If we keep clusterName as a separate parameter, then API changes to
{code}
case class UnresolvedRelation(clusterName: Option[String], databaseName:
Option[String], tableName: String, alias: Option[String])
{code}
Catalog API needs change accordingly
> Parsing error for query with table name having dot
> --
>
> Key: SPARK-4943
> URL: https://issues.apache.org/jira/browse/SPARK-4943
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.2.0
>Reporter: Alex Liu
>
> When integrating Spark 1.2.0 with Cassandra SQL, the following query is
> broken. It was working for Spark 1.1.0 version. Basically we use the table
> name having dot to include database name
> {code}
> [info] java.lang.RuntimeException: [1.29] failure: ``UNION'' expected but
> `.' found
> [info]
> [info] SELECT test1.a FROM sql_test.test1 AS test1 UNION DISTINCT SELECT
> test2.a FROM sql_test.test2 AS test2
> [info] ^
> [info] at scala.sys.package$.error(package.scala:27)
> [info] at
> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:33)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
> [info] at
> org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:174)
> [info] at
> org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:173)
> [info] at
> scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:136)
> [info] at
> scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:135)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Failure.append(Parsers.scala:202)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> [info] at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2.apply(Parsers.scala:890)
> [info] at
> scala.util.parsing.combinator.PackratParsers$$anon$1.apply(PackratParsers.scala:110)
> [info] at
> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:31)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$parseSql$1.apply(SQLContext.scala:83)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$parseSql$1.apply(SQLContext.scala:83)
> [info] at scala.Option.getOrElse(Option.scala:120)
> [info] at org.apache.spark.sql.SQLContext.parseSql(SQLContext.scala:83)
> [info] at
> org.apache.spark.sql.cassandra.CassandraSQLContext.cassandraSql(CassandraSQLContext.scala:53)
> [info] at
> org.apache.spark.sql.cassandra.CassandraSQLContext.sql(CassandraSQLContext.scala:56)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply$mcV$sp(CassandraSQLSpec.scala:169)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply(CassandraSQLSpec.scala:168)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply(CassandraSQLSpec.scala:168)
> [i
[jira] [Commented] (SPARK-4943) Parsing error for query with table name having dot
[
https://issues.apache.org/jira/browse/SPARK-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14263230#comment-14263230
]
Alex Liu commented on SPARK-4943:
-
Should we also change the signatures of Catalog methods to use
{code}tableIdentifier: Seq[String] {code} instead of {code}db: Option[String],
tableName: String{code}?
{code}
def tableExists(db: Option[String], tableName: String): Boolean
def lookupRelation(
databaseName: Option[String],
tableName: String,
alias: Option[String] = None): LogicalPlan
def registerTable(databaseName: Option[String], tableName: String, plan:
LogicalPlan): Unit
def unregisterTable(databaseName: Option[String], tableName: String): Unit
def unregisterAllTables(): Unit
protected def processDatabaseAndTableName(
databaseName: Option[String],
tableName: String): (Option[String], String)
{code}
> Parsing error for query with table name having dot
> --
>
> Key: SPARK-4943
> URL: https://issues.apache.org/jira/browse/SPARK-4943
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.2.0
>Reporter: Alex Liu
>
> When integrating Spark 1.2.0 with Cassandra SQL, the following query is
> broken. It was working for Spark 1.1.0 version. Basically we use the table
> name having dot to include database name
> {code}
> [info] java.lang.RuntimeException: [1.29] failure: ``UNION'' expected but
> `.' found
> [info]
> [info] SELECT test1.a FROM sql_test.test1 AS test1 UNION DISTINCT SELECT
> test2.a FROM sql_test.test2 AS test2
> [info] ^
> [info] at scala.sys.package$.error(package.scala:27)
> [info] at
> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:33)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
> [info] at
> org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:174)
> [info] at
> org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:173)
> [info] at
> scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:136)
> [info] at
> scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:135)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Failure.append(Parsers.scala:202)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> [info] at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2.apply(Parsers.scala:890)
> [info] at
> scala.util.parsing.combinator.PackratParsers$$anon$1.apply(PackratParsers.scala:110)
> [info] at
> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:31)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$parseSql$1.apply(SQLContext.scala:83)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$parseSql$1.apply(SQLContext.scala:83)
> [info] at scala.Option.getOrElse(Option.scala:120)
> [info] at org.apache.spark.sql.SQLContext.parseSql(SQLContext.scala:83)
> [info] at
> org.apache.spark.sql.cassandra.CassandraSQLContext.cassandraSql(CassandraSQLContext.scala:53)
> [info] at
> org.apache.spark.sql.cassandra.CassandraSQLContext.sql(CassandraSQLContext.scala:56)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply$mcV$sp(CassandraSQLSpec.scala:169)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply(CassandraSQLSpec.scala:168)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply(CassandraSQLSpec.scala:168)
> [info]
[jira] [Commented] (SPARK-4943) Parsing error for query with table name having dot
[
https://issues.apache.org/jira/browse/SPARK-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14262442#comment-14262442
]
Alex Liu commented on SPARK-4943:
-
The seq can be like
{code}
tableName, databaseName, clusterName, catalog
{code}
First element is tableName, the next one is databaseName, then clusterName and
catalog(option).
SQL query looks like
{code}
SELECT test_table.column1 FROM catalog.cluster.database.table AS test_table
SELECT test_table.column1 FROM cluster.database.table AS test_table
SELECT test_table.column1 FROM database.table AS test_table
SELECT table.column1 FROM table
{code}
Potentially we should not use AS clause here.
Parser only allow maximum four levels in full table name.
[catalog].[cluster].[database].[table]
> Parsing error for query with table name having dot
> --
>
> Key: SPARK-4943
> URL: https://issues.apache.org/jira/browse/SPARK-4943
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.2.0
>Reporter: Alex Liu
>
> When integrating Spark 1.2.0 with Cassandra SQL, the following query is
> broken. It was working for Spark 1.1.0 version. Basically we use the table
> name having dot to include database name
> {code}
> [info] java.lang.RuntimeException: [1.29] failure: ``UNION'' expected but
> `.' found
> [info]
> [info] SELECT test1.a FROM sql_test.test1 AS test1 UNION DISTINCT SELECT
> test2.a FROM sql_test.test2 AS test2
> [info] ^
> [info] at scala.sys.package$.error(package.scala:27)
> [info] at
> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:33)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
> [info] at
> org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:174)
> [info] at
> org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:173)
> [info] at
> scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:136)
> [info] at
> scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:135)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Failure.append(Parsers.scala:202)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> [info] at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2.apply(Parsers.scala:890)
> [info] at
> scala.util.parsing.combinator.PackratParsers$$anon$1.apply(PackratParsers.scala:110)
> [info] at
> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:31)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$parseSql$1.apply(SQLContext.scala:83)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$parseSql$1.apply(SQLContext.scala:83)
> [info] at scala.Option.getOrElse(Option.scala:120)
> [info] at org.apache.spark.sql.SQLContext.parseSql(SQLContext.scala:83)
> [info] at
> org.apache.spark.sql.cassandra.CassandraSQLContext.cassandraSql(CassandraSQLContext.scala:53)
> [info] at
> org.apache.spark.sql.cassandra.CassandraSQLContext.sql(CassandraSQLContext.scala:56)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply$mcV$sp(CassandraSQLSpec.scala:169)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply(CassandraSQLSpec.scala:168)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply(CassandraSQLSpec.scala:168)
> [info] at
> org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
> [info] at org.scalatest.Ou
[jira] [Commented] (SPARK-4943) Parsing error for query with table name having dot
[
https://issues.apache.org/jira/browse/SPARK-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14262412#comment-14262412
]
Michael Armbrust commented on SPARK-4943:
-
One possible approach here is to change the signature of UnresolvedRelation as
follows:
{code}
case class UnresolvedRelation(tableIdentifier: Seq[String], alias:
Option[String])
{code}
This way we can leave parsing and handling of backticks up to the parser and
let the catalogs interpret the identifiers in a system dependent way.
> Parsing error for query with table name having dot
> --
>
> Key: SPARK-4943
> URL: https://issues.apache.org/jira/browse/SPARK-4943
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.2.0
>Reporter: Alex Liu
>
> When integrating Spark 1.2.0 with Cassandra SQL, the following query is
> broken. It was working for Spark 1.1.0 version. Basically we use the table
> name having dot to include database name
> {code}
> [info] java.lang.RuntimeException: [1.29] failure: ``UNION'' expected but
> `.' found
> [info]
> [info] SELECT test1.a FROM sql_test.test1 AS test1 UNION DISTINCT SELECT
> test2.a FROM sql_test.test2 AS test2
> [info] ^
> [info] at scala.sys.package$.error(package.scala:27)
> [info] at
> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:33)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
> [info] at
> org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:174)
> [info] at
> org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:173)
> [info] at
> scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:136)
> [info] at
> scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:135)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Failure.append(Parsers.scala:202)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> [info] at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2.apply(Parsers.scala:890)
> [info] at
> scala.util.parsing.combinator.PackratParsers$$anon$1.apply(PackratParsers.scala:110)
> [info] at
> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:31)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$parseSql$1.apply(SQLContext.scala:83)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$parseSql$1.apply(SQLContext.scala:83)
> [info] at scala.Option.getOrElse(Option.scala:120)
> [info] at org.apache.spark.sql.SQLContext.parseSql(SQLContext.scala:83)
> [info] at
> org.apache.spark.sql.cassandra.CassandraSQLContext.cassandraSql(CassandraSQLContext.scala:53)
> [info] at
> org.apache.spark.sql.cassandra.CassandraSQLContext.sql(CassandraSQLContext.scala:56)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply$mcV$sp(CassandraSQLSpec.scala:169)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply(CassandraSQLSpec.scala:168)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply(CassandraSQLSpec.scala:168)
> [info] at
> org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
> [info] at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
> [info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> [info] at org.scalatest.Transformer.apply(Transformer.scala:22)
> [info] at org.scalatest.Transformer.apply(Transformer.scala:20)
[jira] [Commented] (SPARK-4943) Parsing error for query with table name having dot
[
https://issues.apache.org/jira/browse/SPARK-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14261396#comment-14261396
]
Apache Spark commented on SPARK-4943:
-
User 'alexliu68' has created a pull request for this issue:
https://github.com/apache/spark/pull/3848
> Parsing error for query with table name having dot
> --
>
> Key: SPARK-4943
> URL: https://issues.apache.org/jira/browse/SPARK-4943
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 1.2.0
>Reporter: Alex Liu
>
> When integrating Spark 1.2.0 with Cassandra SQL, the following query is
> broken. It was working for Spark 1.1.0 version. Basically we use the table
> name having dot to include database name
> {code}
> [info] java.lang.RuntimeException: [1.29] failure: ``UNION'' expected but
> `.' found
> [info]
> [info] SELECT test1.a FROM sql_test.test1 AS test1 UNION DISTINCT SELECT
> test2.a FROM sql_test.test2 AS test2
> [info] ^
> [info] at scala.sys.package$.error(package.scala:27)
> [info] at
> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:33)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$1.apply(SQLContext.scala:79)
> [info] at
> org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:174)
> [info] at
> org.apache.spark.sql.catalyst.SparkSQLParser$$anonfun$org$apache$spark$sql$catalyst$SparkSQLParser$$others$1.apply(SparkSQLParser.scala:173)
> [info] at
> scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:136)
> [info] at
> scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:135)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1$$anonfun$apply$2.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Failure.append(Parsers.scala:202)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2$$anonfun$apply$14.apply(Parsers.scala:891)
> [info] at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
> [info] at
> scala.util.parsing.combinator.Parsers$$anon$2.apply(Parsers.scala:890)
> [info] at
> scala.util.parsing.combinator.PackratParsers$$anon$1.apply(PackratParsers.scala:110)
> [info] at
> org.apache.spark.sql.catalyst.AbstractSparkSQLParser.apply(SparkSQLParser.scala:31)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$parseSql$1.apply(SQLContext.scala:83)
> [info] at
> org.apache.spark.sql.SQLContext$$anonfun$parseSql$1.apply(SQLContext.scala:83)
> [info] at scala.Option.getOrElse(Option.scala:120)
> [info] at org.apache.spark.sql.SQLContext.parseSql(SQLContext.scala:83)
> [info] at
> org.apache.spark.sql.cassandra.CassandraSQLContext.cassandraSql(CassandraSQLContext.scala:53)
> [info] at
> org.apache.spark.sql.cassandra.CassandraSQLContext.sql(CassandraSQLContext.scala:56)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply$mcV$sp(CassandraSQLSpec.scala:169)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply(CassandraSQLSpec.scala:168)
> [info] at
> com.datastax.spark.connector.sql.CassandraSQLSpec$$anonfun$20.apply(CassandraSQLSpec.scala:168)
> [info] at
> org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22)
> [info] at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85)
> [info] at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> [info] at org.scalatest.Transformer.apply(Transformer.scala:22)
> [info] at org.scalatest.Transformer.apply(Transformer.scala:20)
> [info] at org.scalatest.FlatSpecLike$$anon$1.apply(FlatSpecLike.scala:1647)
> [info] at org.scalatest.Suite$class.withFixture(Suite.scala:1122)
> [info] at org.scalatest.FlatSpec.withFixture(FlatSpec.scala:1683)
> [info] at
> org.sc
