Re: Concatenate the columns in dataframe to create new collumns using Java

Thanks Nihed. I was able to do this in exactly the same way. Cheers!! Abhi On Mon, Jul 18, 2016 at 5:56 PM, nihed mbarek wrote: > and if we have this static method > df.show(); > Column c = concatFunction(df, "l1", "firstname,lastname"); >

Re: Concatenate the columns in dataframe to create new collumns using Java

and if we have this static method df.show(); Column c = concatFunction(df, "l1", "firstname,lastname"); df.select(c).show(); with this code : Column concatFunction(DataFrame df, String fieldName, String columns) { String[] array = columns.split(",");

Re: Concatenate the columns in dataframe to create new collumns using Java

Hi Nihed, Thanks for the reply. I am looking for something like this : DataFrame training = orgdf.withColumn("I1", functions.concat(orgdf.col("C0"),orgdf.col("C1"))); Here I have to give C0 and C1 columns, I am looking to write a generic function that concatenates the columns depending on

Re: Concatenate the columns in dataframe to create new collumns using Java

Hi, I just wrote this code to help you. Is it what you need ?? SparkConf conf = new SparkConf().setAppName("hello").setMaster("local"); JavaSparkContext sc = new JavaSparkContext(conf); SQLContext sqlContext = new SQLContext(sc); List persons = new

Concatenate the columns in dataframe to create new collumns using Java

Hi, I have a dataframe say having C0,C1,C2 and so on as columns. I need to create interaction variables to be taken as input for my program. For eg - I need to create I1 as concatenation of C0,C3,C5 Similarly, I2 = concat(C4,C5) and so on .. How can I achieve this in my Java code for