Re: Spark DataFrame sum of multiple columns
Easy way of doing it newdf = df.withColumn('total', sum(df[col] for col in df.columns)) On 22 April 2016 at 11:51, Naveen Kumar Pokalawrote: > Hi, > > > > Do we have any way to perform Row level operations in spark dataframes. > > > > > > For example, > > > > I have a dataframe with columns from A,B,C,…Z.. I want to add one more > column New Column with sum of all column values. > > > > A > > B > > C > > D > > . > > . > > . > > Z > > New Column > > 1 > > 2 > > 4 > > 3 > > > > > > > > 26 > > 351 > > > > > > Can somebody help me on this? > > > > > > Thanks, > > Naveen >
Re: Spark DataFrame sum of multiple columns
You can define your own udf, following is one example Thanks Zhan Zhang val foo = udf((a: Int, b: String) => a.toString + b) checkAnswer( // SELECT *, foo(key, value) FROM testData testData.select($"*", foo('key, 'value)).limit(3), On Apr 21, 2016, at 8:51 PM, Naveen Kumar Pokala> wrote: Hi, Do we have any way to perform Row level operations in spark dataframes. For example, I have a dataframe with columns from A,B,C,…Z.. I want to add one more column New Column with sum of all column values. A B C D . . . Z New Column 1 2 4 3 26 351 Can somebody help me on this? Thanks, Naveen
Spark DataFrame sum of multiple columns
Hi, Do we have any way to perform Row level operations in spark dataframes. For example, I have a dataframe with columns from A,B,C,...Z.. I want to add one more column New Column with sum of all column values. A B C D . . . Z New Column 1 2 4 3 26 351 Can somebody help me on this? Thanks, Naveen