Re: to_json not working with selectExpr

2017-07-19 Thread Matthew cao
Another question, how can I find the version in which one function is added into SQL? > On 2017年7月20日, at 10:43, Hyukjin Kwon wrote: > > Yes, I guess it is. > > 2017-07-20 11:31 GMT+09:00 Matthew cao >: > AH, I get it. So

Re: to_json not working with selectExpr

2017-07-19 Thread Matthew cao
Thank you so much! that confuse me quite a time! > On 2017年7月20日, at 10:43, Hyukjin Kwon wrote: > > Yes, I guess it is. > > 2017-07-20 11:31 GMT+09:00 Matthew cao >: > AH, I get it. So that’s why I get the not register

Re: to_json not working with selectExpr

2017-07-19 Thread Hyukjin Kwon
Yes, I guess it is. 2017-07-20 11:31 GMT+09:00 Matthew cao : > AH, I get it. So that’s why I get the not register error? Cuz it not added > into SQL in 2.1.0? > > On 2017年7月19日, at 22:35, Hyukjin Kwon wrote: > > Yea, but it was added into SQL from Spark

Re: to_json not working with selectExpr

2017-07-19 Thread Matthew cao
AH, I get it. So that’s why I get the not register error? Cuz it not added into SQL in 2.1.0? > On 2017年7月19日, at 22:35, Hyukjin Kwon > wrote: > > Yea, but it was added into SQL from Spark 2.2.0 > > 2017-07-19 23:02 GMT+09:00 Matthew cao

Re: to_json not working with selectExpr

2017-07-19 Thread Hyukjin Kwon
Yea, but it was added into SQL from Spark 2.2.0 2017-07-19 23:02 GMT+09:00 Matthew cao : > I am using version 2.1.1 As I could remember, this function was added > since 2.1.0. > > On 2017年7月17日, at 12:05, Burak Yavuz wrote: > > Hi Matthew, > > Which Spark

Re: to_json not working with selectExpr

2017-07-19 Thread Matthew cao
I am using version 2.1.1 As I could remember, this function was added since 2.1.0. > On 2017年7月17日, at 12:05, Burak Yavuz > wrote: > > Hi Matthew, > > Which Spark version are you using? The expression `to_json` was added in 2.2 > with this commit:

Re: to_json not working with selectExpr

2017-07-16 Thread Burak Yavuz
Hi Matthew, Which Spark version are you using? The expression `to_json` was added in 2.2 with this commit: https://github.com/apache/spark/commit/0cdcf9114527a2c359c25e46fd6556b3855bfb28 Best, Burak On Sun, Jul 16, 2017 at 6:24 PM, Matthew cao wrote: > Hi all, > I just

to_json not working with selectExpr

2017-07-16 Thread Matthew cao
Hi all, I just read the databricks blog here: https://docs.databricks.com/_static/notebooks/complex-nested-structured.html When I try to follow the example about the to_json and selectExpr part, it gave error: