Re: spark sql registerFunction with 1.2.1
Regarding backticks: Right. You need backticks to quote the column name timestamp because timestamp is a reserved keyword in our parser. On Tue, Feb 10, 2015 at 3:02 PM, Mohnish Kodnani mohnish.kodn...@gmail.com wrote: actually i tried in spark shell , got same error and then for some reason i tried to back tick the timestamp and it worked. val result = sqlContext.sql(select toSeconds(`timestamp`) as t, count(rid) as qps from blah group by toSeconds(`timestamp`),qi.clientName) so, it seems sql context is supporting UDF. On Tue, Feb 10, 2015 at 2:32 PM, Michael Armbrust mich...@databricks.com wrote: The simple SQL parser doesn't yet support UDFs. Try using a HiveContext. On Tue, Feb 10, 2015 at 1:44 PM, Mohnish Kodnani mohnish.kodn...@gmail.com wrote: Hi, I am trying a very simple registerFunction and it is giving me errors. I have a parquet file which I register as temp table. Then I define a UDF. def toSeconds(timestamp: Long): Long = timestamp/10 sqlContext.registerFunction(toSeconds, toSeconds _) val result = sqlContext.sql(select toSeconds(timestamp) from blah); I get the following error. java.lang.RuntimeException: [1.18] failure: ``)'' expected but `timestamp' found select toSeconds(timestamp) from blah My end goal is as follows: We have log file with timestamps in microseconds and I would like to group by entries with second level precision, so eventually I want to run the query select toSeconds(timestamp) as t, count(x) from table group by t,x
Re: spark sql registerFunction with 1.2.1
that explains a lot... Is there a list of reserved keywords ? On Wed, Feb 11, 2015 at 7:56 AM, Yin Huai yh...@databricks.com wrote: Regarding backticks: Right. You need backticks to quote the column name timestamp because timestamp is a reserved keyword in our parser. On Tue, Feb 10, 2015 at 3:02 PM, Mohnish Kodnani mohnish.kodn...@gmail.com wrote: actually i tried in spark shell , got same error and then for some reason i tried to back tick the timestamp and it worked. val result = sqlContext.sql(select toSeconds(`timestamp`) as t, count(rid) as qps from blah group by toSeconds(`timestamp`),qi.clientName) so, it seems sql context is supporting UDF. On Tue, Feb 10, 2015 at 2:32 PM, Michael Armbrust mich...@databricks.com wrote: The simple SQL parser doesn't yet support UDFs. Try using a HiveContext. On Tue, Feb 10, 2015 at 1:44 PM, Mohnish Kodnani mohnish.kodn...@gmail.com wrote: Hi, I am trying a very simple registerFunction and it is giving me errors. I have a parquet file which I register as temp table. Then I define a UDF. def toSeconds(timestamp: Long): Long = timestamp/10 sqlContext.registerFunction(toSeconds, toSeconds _) val result = sqlContext.sql(select toSeconds(timestamp) from blah); I get the following error. java.lang.RuntimeException: [1.18] failure: ``)'' expected but `timestamp' found select toSeconds(timestamp) from blah My end goal is as follows: We have log file with timestamps in microseconds and I would like to group by entries with second level precision, so eventually I want to run the query select toSeconds(timestamp) as t, count(x) from table group by t,x
Re: spark sql registerFunction with 1.2.1
The simple SQL parser doesn't yet support UDFs. Try using a HiveContext. On Tue, Feb 10, 2015 at 1:44 PM, Mohnish Kodnani mohnish.kodn...@gmail.com wrote: Hi, I am trying a very simple registerFunction and it is giving me errors. I have a parquet file which I register as temp table. Then I define a UDF. def toSeconds(timestamp: Long): Long = timestamp/10 sqlContext.registerFunction(toSeconds, toSeconds _) val result = sqlContext.sql(select toSeconds(timestamp) from blah); I get the following error. java.lang.RuntimeException: [1.18] failure: ``)'' expected but `timestamp' found select toSeconds(timestamp) from blah My end goal is as follows: We have log file with timestamps in microseconds and I would like to group by entries with second level precision, so eventually I want to run the query select toSeconds(timestamp) as t, count(x) from table group by t,x
