Re: question on transforms for spark 2.0 dataset

Subhash, Yea that did the trick thanks! On Wed, Mar 1, 2017 at 12:20 PM, Subhash Sriram wrote: > If I am understanding your problem correctly, I think you can just create > a new DataFrame that is a transformation of sample_data by first > registering sample_data as a

Re: question on transforms for spark 2.0 dataset

If I am understanding your problem correctly, I think you can just create a new DataFrame that is a transformation of sample_data by first registering sample_data as a temp table. //Register temp table sample_data.createOrReplaceTempView("sql_sample_data") //Create new DataSet with transformed

Re: question on transforms for spark 2.0 dataset

Hi I think u need an UDF if u want to transform a column Hth On 1 Mar 2017 4:22 pm, "Bill Schwanitz" wrote: > Hi all, > > I'm fairly new to spark and scala so bear with me. > > I'm working with a dataset containing a set of column / fields. The data > is stored in hdfs as

question on transforms for spark 2.0 dataset

Hi all, I'm fairly new to spark and scala so bear with me. I'm working with a dataset containing a set of column / fields. The data is stored in hdfs as parquet and is sourced from a postgres box so fields and values are reasonably well formed. We are in the process of trying out a switch from