Hello Rachana, The easiest way would be to start with creating a 'parent' JavaRDD and run different filters (based on different input arguments) to create respective 'child' JavaRDDs dynamically.
Notice that the creation of these children RDDs is handled by the application driver. Hope this helps! Ajay On Mon, Sep 14, 2015 at 1:21 PM, Ankur Srivastava < ankur.srivast...@gmail.com> wrote: > Hi Rachana > > I didn't get you r question fully but as the error says you can not > perform a rdd transformation or action inside another transformation. In > your example you are performing an action "rdd2.values.count()" in side > the "map" transformation. It is not allowed and in any case this will be > very inefficient too. > > you should do something like this: > > final long rdd2_count = rdd2.values.count() > rdd1.map(x => rdd2_count * x) > > Hope this helps!! > > - Ankur > > On Mon, Sep 14, 2015 at 9:54 AM, Rachana Srivastava < > rachana.srivast...@markmonitor.com> wrote: > >> Hello all, >> >> >> >> I am working a problem that requires us to create different set of >> JavaRDD based on different input arguments. We are getting following error >> when we try to use a factory to create JavaRDD. Error message is clear but >> I am wondering is there any workaround. >> >> >> >> *Question: * >> >> How to create different set of JavaRDD based on different input arguments >> dynamically. Trying to implement something like factory pattern. >> >> >> >> *Error Message: * >> >> RDD transformations and actions can only be invoked by the driver, not >> inside of other transformations; for example, rdd1.map(x => >> rdd2.values.count() * x) is invalid because the values transformation and >> count action cannot be performed inside of the rdd1.map transformation. For >> more information, see SPARK-5063. >> >> >> >> Thanks, >> >> >> >> Rachana >> > >
