Hi Wen, AFAIK Spark MLlib implements its machine learning algorithms on top of Spark dataframe API. What did you mean by a grouped dataframe?
On Tue, Aug 23, 2016 at 10:42 AM, Wen Pei Yu <yuw...@cn.ibm.com> wrote: > Hi Nirmal > > I didn't get your point. > Can you tell me more about how to use MLlib to grouped dataframe? > > Regards. > Wenpei. > > [image: Inactive hide details for Nirmal Fernando ---08/23/2016 10:26:36 > AM---You can use Spark MLlib http://spark.apache.org/docs/late]Nirmal > Fernando ---08/23/2016 10:26:36 AM---You can use Spark MLlib > http://spark.apache.org/docs/latest/ml-guide.html# > announcement-dataframe-bas > > From: Nirmal Fernando <nir...@wso2.com> > To: Wen Pei Yu/China/IBM@IBMCN > Cc: User <user@spark.apache.org> > Date: 08/23/2016 10:26 AM > Subject: Re: Apply ML to grouped dataframe > ------------------------------ > > > > You can use Spark MLlib > *http://spark.apache.org/docs/latest/ml-guide.html#announcement-dataframe-based-api-is-primary-api* > <http://spark.apache.org/docs/latest/ml-guide.html#announcement-dataframe-based-api-is-primary-api> > > On Tue, Aug 23, 2016 at 7:34 AM, Wen Pei Yu <*yuw...@cn.ibm.com* > <yuw...@cn.ibm.com>> wrote: > > Hi > > We have a dataframe, then want group it and apply a ML algorithm or > statistics(say t test) to each one. Is there any efficient way for this > situation? > > Currently, we transfer to pyspark, use groupbykey and apply numpy > function to array. But this wasn't an efficient way, right? > > Regards. > Wenpei. > > > > > > -- > > Thanks & regards, > Nirmal > > Team Lead - WSO2 Machine Learner > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: *http://nirmalfdo.blogspot.com/* <http://nirmalfdo.blogspot.com/> > > > > -- Thanks & regards, Nirmal Team Lead - WSO2 Machine Learner Associate Technical Lead - Data Technologies Team, WSO2 Inc. Mobile: +94715779733 Blog: http://nirmalfdo.blogspot.com/
