RE: pass one dataframe column value to another dataframe filter expression + Spark 1.5 + scala

2016-02-05 Thread Lohith Samaga M
Hi, If you can also format the condition file as a csv file similar to the main file, then you can join the two dataframes and select only required columns. Best regards / Mit freundlichen Grüßen / Sincères salutations M. Lohith Samaga From: Divya Gehlot

Re: pass one dataframe column value to another dataframe filter expression + Spark 1.5 + scala

2016-02-05 Thread Ali Tajeldin
I think the tricky part here is that the join condition is encoded in the second data frame and not a direct value. Assuming the second data frame (the tags) is small enough, you can collect it (read it into memory) and then construct a "when" expression chain for each of the possible tags ,