Subhash,
Yea that did the trick thanks!
On Wed, Mar 1, 2017 at 12:20 PM, Subhash Sriram
wrote:
> If I am understanding your problem correctly, I think you can just create
> a new DataFrame that is a transformation of sample_data by first
> registering sample_data as a temp table.
>
> //Register
If I am understanding your problem correctly, I think you can just create a
new DataFrame that is a transformation of sample_data by first registering
sample_data as a temp table.
//Register temp table
sample_data.createOrReplaceTempView("sql_sample_data")
//Create new DataSet with transformed va
Hi I think u need an UDF if u want to transform a column
Hth
On 1 Mar 2017 4:22 pm, "Bill Schwanitz" wrote:
> Hi all,
>
> I'm fairly new to spark and scala so bear with me.
>
> I'm working with a dataset containing a set of column / fields. The data
> is stored in hdfs as parquet and is sour
Hi all,
I'm fairly new to spark and scala so bear with me.
I'm working with a dataset containing a set of column / fields. The data is
stored in hdfs as parquet and is sourced from a postgres box so fields and
values are reasonably well formed. We are in the process of trying out a
switch from pe
