Re: to_json not working with selectExpr

Another question, how can I find the version in which one function is added into SQL? > On 2017年7月20日, at 10:43, Hyukjin Kwon <gurwls...@gmail.com> wrote: > > Yes, I guess it is. > > 2017-07-20 11:31 GMT+09:00 Matthew cao <cybea...@gmail.com > <mailto:cybea...@g

Re: to_json not working with selectExpr

Thank you so much! that confuse me quite a time! > On 2017年7月20日, at 10:43, Hyukjin Kwon <gurwls...@gmail.com> wrote: > > Yes, I guess it is. > > 2017-07-20 11:31 GMT+09:00 Matthew cao <cybea...@gmail.com > <mailto:cybea...@gmail.com>>: > AH, I get it. So

Re: to_json not working with selectExpr

AH, I get it. So that’s why I get the not register error? Cuz it not added into SQL in 2.1.0? > On 2017年7月19日, at 22:35, Hyukjin Kwon <gurwls...@gmail.com > <mailto:gurwls...@gmail.com>> wrote: > > Yea, but it was added into SQL from Spark 2.2.0 > > 2017-07-

Re: to_json not working with selectExpr

ded in 2.2 > with this commit: > https://github.com/apache/spark/commit/0cdcf9114527a2c359c25e46fd6556b3855bfb28 > > <https://github.com/apache/spark/commit/0cdcf9114527a2c359c25e46fd6556b3855bfb28> > > Best, > Burak > > On Sun, Jul 16, 2017 at 6:24 PM, Matthe

to_json not working with selectExpr

Hi all, I just read the databricks blog here: https://docs.databricks.com/_static/notebooks/complex-nested-structured.html When I try to follow the example about the to_json and selectExpr part, it gave error:

Re: Why does dataset.union fails but dataset.rdd.union execute correctly?

Hi, I have tried simple test like this: case class A(id: Long) val sample = spark.range(0,10).as[A] sample.createOrReplaceTempView("sample") val df = spark.emptyDataset[A] val df1 = spark.sql("select * from sample").as[A] df.union(df1) It runs ok. And for nullabillity I thought that issue has