you can use udfs to do it http://stackoverflow.com/questions/31615657/how-to-add-a-new-struct-column-to-a-dataframe
Hope it will help. Thanks, Divya On 9 December 2016 at 00:53, Anton Kravchenko <kravchenko.anto...@gmail.com> wrote: > Hello, > > I wonder if there is a way (preferably efficient) in Spark to reshape hive > table and save it to parquet. > > Here is a minimal example, input hive table: > col1 col2 col3 > 1 2 3 > 4 5 6 > > output parquet: > col1 newcol2 > 1 [2 3] > 4 [5 6] > > p.s. The real input hive table has ~1000 columns. > > Thank you, > Anton >
