Hi Akshay,
Thanks for this. I will give it a try. The Java API for .explain returns
void. It doesn't throw any checked exception. so I guess I have to catch
the generic RuntimeException and walk through the stacktrace to see if
there is any ParseException. In short, the code just gets really ugly. I
wish the Spark guys take all of this into account when designing an API
such that it works well for all languages.
Thanks!
On Tue, Mar 5, 2019 at 4:15 AM Akshay Bhardwaj <
akshay.bhardwaj1...@gmail.com> wrote:
> Hi Kant,
>
> You can try "explaining" the sql query.
>
> spark.sql(sqlText).explain(true); //the parameter true is to get more
> verbose query plan and it is optional.
>
>
> This is the safest way to validate sql without actually executing/creating
> a df/view in spark. It validates syntax as well as schema of tables/views
> used.
> If there is an issue with your SQL syntax then the method throws below
> exception that you can catch
>
> org.apache.spark.sql.catalyst.parser.ParseException
>
>
> Hope this helps!
>
>
>
> Akshay Bhardwaj
> +91-97111-33849
>
>
> On Fri, Mar 1, 2019 at 10:23 PM kant kodali <kanth...@gmail.com> wrote:
>
>> Hi All,
>>
>> Is there a way to validate the syntax of raw spark SQL query?
>>
>> for example, I would like to know if there is any isValid API call spark
>> provides?
>>
>> val query = "select * from table"if(isValid(query)) {
>> sparkSession.sql(query) } else {
>> log.error("Invalid Syntax")}
>>
>> I tried the following
>>
>> val query = "select * morf table" // Invalid queryval parser =
>> spark.sessionState.sqlParsertry{
>> parser.parseExpression(query)} catch (ParseException ex) {
>> throw new Exception(ex); //Exception not getting thrown}Dataset<>Row df
>> = sparkSession.sql(query) // Exception gets thrown here
>> df.writeStream.format("console").start()
>>
>> Question: parser.parseExpression is not catching the invalid syntax
>> before I hit the sparkSession.sql. Other words it is not being helpful
>> in the above code. any reason? My whole goal is to catch syntax errors
>> before I pass it on to sparkSession.sql
>>
>>
>>
