RE: Create column in nested structure?

2015-08-13 Thread Ewan Leith 
Never mind me, I've found an email to this list from Raghavendra Pandey which 
got me what I needed

val nestedCol = struct(df(nested2.column1), df(nested2.column2), 
df(flatcolumn))
val df2 = df.select(df(nested1), nestedCol as nested2)

Thanks,
Ewan

From: Ewan Leith
Sent: 13 August 2015 15:44
To: user@spark.apache.org
Subject: Create column in nested structure?

Has anyone used withColumn (or another method) to add a column to an existing 
nested dataframe?

If I call:

df.withColumn(nested.newcolumn, df(oldcolumn))

then it just creates the new column with a . In it's name, not under the 
nested structure.

Thanks,
Ewan

Create column in nested structure?

2015-08-13 Thread Ewan Leith 
Has anyone used withColumn (or another method) to add a column to an existing 
nested dataframe?

If I call:

df.withColumn(nested.newcolumn, df(oldcolumn))

then it just creates the new column with a . In it's name, not under the 
nested structure.

Thanks,
Ewan