Re: JavaRDD using Reflection

Hi Rachana I didn't get you r question fully but as the error says you can not perform a rdd transformation or action inside another transformation. In your example you are performing an action "rdd2.values.count()" in side the "map" transformation. It is not allowed and in any case this will be

JavaRDD using Reflection

Hello all, I am working a problem that requires us to create different set of JavaRDD based on different input arguments. We are getting following error when we try to use a factory to create JavaRDD. Error message is clear but I am wondering is there any workaround. Question: How to create

Re: JavaRDD using Reflection

… > > } > > }); > > return featureScoreRDD; > > } > > > > } > > > > Thanks again for all your help and advice. > > > > Regards, > > > > Rach

Re: JavaRDD using Reflection

Hello Rachana, The easiest way would be to start with creating a 'parent' JavaRDD and run different filters (based on different input arguments) to create respective 'child' JavaRDDs dynamically. Notice that the creation of these children RDDs is handled by the application driver. Hope this

Re: Applying transformations on a JavaRDD using reflection

Have you got some code already that demonstrates the problem? > On 9 Sep 2015, at 04:45, Nirmal Fernando wrote: > > Any thoughts? > > On Tue, Sep 8, 2015 at 3:37 PM, Nirmal Fernando > wrote: > Hi All, > > I'd like to apply a chain of

Applying transformations on a JavaRDD using reflection

Hi All, I'd like to apply a chain of Spark transformations (map/filter) on a given JavaRDD. I'll have the set of Spark transformations as Function, and even though I can determine the classes of T and A at the runtime, due to the type erasure, I cannot call JavaRDD's transformations as they

Re: Applying transformations on a JavaRDD using reflection

Any thoughts? On Tue, Sep 8, 2015 at 3:37 PM, Nirmal Fernando wrote: > Hi All, > > I'd like to apply a chain of Spark transformations (map/filter) on a given > JavaRDD. I'll have the set of Spark transformations as Function, and > even though I can determine the classes of