Re: Supported SQL syntax in Spark SQL
FYI: I've created SPARK-2560 https://issues.apache.org/jira/browse/SPARK-2560 to track creating SQL reference docs for Spark SQL. On Mon, Jul 14, 2014 at 2:06 PM, Michael Armbrust mich...@databricks.com wrote: You can find the parser here: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala In general the hive parser provided by HQL is much more complete at the moment. Long term we will likely stop using parser combinators and either write a more complete parser, or adopt one from an existing project. On Mon, Jul 14, 2014 at 12:25 AM, Martin Gammelsæter martingammelsae...@gmail.com wrote: I am very interested in the original question as well, is there any list (even if it is simply in the code) of all supported syntax for Spark SQL? On Mon, Jul 14, 2014 at 6:41 AM, Nicholas Chammas nicholas.cham...@gmail.com wrote: Are you sure the code running on the cluster has been updated? I launched the cluster using spark-ec2 from the 1.0.1 release, so I’m assuming that’s taken care of, at least in theory. I just spun down the clusters I had up, but I will revisit this tomorrow and provide the information you requested. Nick -- Mvh. Martin Gammelsæter 92209139
Re: Supported SQL syntax in Spark SQL
I am very interested in the original question as well, is there any list (even if it is simply in the code) of all supported syntax for Spark SQL? On Mon, Jul 14, 2014 at 6:41 AM, Nicholas Chammas nicholas.cham...@gmail.com wrote: Are you sure the code running on the cluster has been updated? I launched the cluster using spark-ec2 from the 1.0.1 release, so I’m assuming that’s taken care of, at least in theory. I just spun down the clusters I had up, but I will revisit this tomorrow and provide the information you requested. Nick -- Mvh. Martin Gammelsæter 92209139
Re: Supported SQL syntax in Spark SQL
You can find the parser here: https://github.com/apache/spark/blob/master/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/SqlParser.scala In general the hive parser provided by HQL is much more complete at the moment. Long term we will likely stop using parser combinators and either write a more complete parser, or adopt one from an existing project. On Mon, Jul 14, 2014 at 12:25 AM, Martin Gammelsæter martingammelsae...@gmail.com wrote: I am very interested in the original question as well, is there any list (even if it is simply in the code) of all supported syntax for Spark SQL? On Mon, Jul 14, 2014 at 6:41 AM, Nicholas Chammas nicholas.cham...@gmail.com wrote: Are you sure the code running on the cluster has been updated? I launched the cluster using spark-ec2 from the 1.0.1 release, so I’m assuming that’s taken care of, at least in theory. I just spun down the clusters I had up, but I will revisit this tomorrow and provide the information you requested. Nick -- Mvh. Martin Gammelsæter 92209139
Re: Supported SQL syntax in Spark SQL
For example, are LIKE 'string%' queries supported? Trying one on 1.0.1 yields java.lang.ExceptionInInitializerError. Nick On Sat, Jul 12, 2014 at 10:16 PM, Nick Chammas nicholas.cham...@gmail.com wrote: Is there a place where we can find an up-to-date list of supported SQL syntax in Spark SQL? Nick -- View this message in context: Supported SQL syntax in Spark SQL http://apache-spark-user-list.1001560.n3.nabble.com/Supported-SQL-syntax-in-Spark-SQL-tp9538.html Sent from the Apache Spark User List mailing list archive http://apache-spark-user-list.1001560.n3.nabble.com/ at Nabble.com.
Re: Supported SQL syntax in Spark SQL
Actually, this looks like its some kind of regression in 1.0.1, perhaps related to assembly and packaging with spark-ec2. I don’t see this issue with the same data on a 1.0.0 EC2 cluster. How can I trace this down for a bug report? Nick On Sun, Jul 13, 2014 at 11:18 PM, Nicholas Chammas nicholas.cham...@gmail.com wrote: For example, are LIKE 'string%' queries supported? Trying one on 1.0.1 yields java.lang.ExceptionInInitializerError. Nick On Sat, Jul 12, 2014 at 10:16 PM, Nick Chammas nicholas.cham...@gmail.com wrote: Is there a place where we can find an up-to-date list of supported SQL syntax in Spark SQL? Nick -- View this message in context: Supported SQL syntax in Spark SQL http://apache-spark-user-list.1001560.n3.nabble.com/Supported-SQL-syntax-in-Spark-SQL-tp9538.html Sent from the Apache Spark User List mailing list archive http://apache-spark-user-list.1001560.n3.nabble.com/ at Nabble.com.
Re: Supported SQL syntax in Spark SQL
Are you sure the code running on the cluster has been updated? We recently optimized the execution of LIKE queries that can be evaluated without using full regular expressions. So it's possible this error is due to missing functionality on the executors. How can I trace this down for a bug report? If the above doesn't fix it, the following would be helpful: - The full stack trace - The queryExecution from the SchemaRDD (i.e. println(sql(SELECT ...).queryExecution))
Re: Supported SQL syntax in Spark SQL
Are you sure the code running on the cluster has been updated? I launched the cluster using spark-ec2 from the 1.0.1 release, so I’m assuming that’s taken care of, at least in theory. I just spun down the clusters I had up, but I will revisit this tomorrow and provide the information you requested. Nick
Supported SQL syntax in Spark SQL
Is there a place where we can find an up-to-date list of supported SQL syntax in Spark SQL? Nick -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Supported-SQL-syntax-in-Spark-SQL-tp9538.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
