Never mind me, I've found an email to this list from Raghavendra Pandey which got me what I needed
val nestedCol = struct(df("nested2.column1"), df("nested2.column2"), df("flatcolumn")) val df2 = df.select(df("nested1"), nestedCol as "nested2") Thanks, Ewan From: Ewan Leith Sent: 13 August 2015 15:44 To: user@spark.apache.org Subject: Create column in nested structure? Has anyone used withColumn (or another method) to add a column to an existing nested dataframe? If I call: df.withColumn("nested.newcolumn", df("oldcolumn")) then it just creates the new column with a "." In it's name, not under the "nested" structure. Thanks, Ewan