答复: 答复: how to generate a column using mapParition and then add it back to the df?

Hi guha, Thanks a lot! This is perfectly what I want and I'll try to implement it. MoTao 发件人: ayan guha 发送时间: 2016年8月8日 18:05:37 收件人: 莫涛 抄送: ndj...@gmail.com; user@spark.apache.org 主题: Re: 答复: how to generate a column using mapParition and

Re: 答复: how to generate a column using mapParition and then add it back to the df?

Hi I think you should modify initModel() function to getOrCreateModel() and create the model as singleton object. You may want to refer this link On Mon, Aug 8, 2016 at 7:44 PM, 莫涛

答复: how to generate a column using mapParition and then add it back to the df?

Hi Ndjido, Thanks for your reply. Yes, it is good idea if the model can be broadcast. I'm working with a built library (on Linux, say classifier.so and classifier.h) and it requires the model file is in the local file system. As I don't have access to the library code, I write JNI to wrap the

Re: how to generate a column using mapParition and then add it back to the df?

Hi MoTao, What about broadcasting the model? Cheers, Ndjido. > On 08 Aug 2016, at 11:00, MoTao wrote: > > Hi all, > > I'm trying to append a column to a df. > I understand that the new column must be created by > 1) using literals, > 2) transforming an existing column in

how to generate a column using mapParition and then add it back to the df?

Hi all, I'm trying to append a column to a df. I understand that the new column must be created by 1) using literals, 2) transforming an existing column in df, or 3) generated from udf over this df In my case, the column to be appended is created by processing each row, like val df =