Hi Rachana I didn't get you r question fully but as the error says you can not perform a rdd transformation or action inside another transformation. In your example you are performing an action "rdd2.values.count()" in side the "map" transformation. It is not allowed and in any case this will be very inefficient too.
you should do something like this: final long rdd2_count = rdd2.values.count() rdd1.map(x => rdd2_count * x) Hope this helps!! - Ankur On Mon, Sep 14, 2015 at 9:54 AM, Rachana Srivastava < rachana.srivast...@markmonitor.com> wrote: > Hello all, > > > > I am working a problem that requires us to create different set of JavaRDD > based on different input arguments. We are getting following error when we > try to use a factory to create JavaRDD. Error message is clear but I am > wondering is there any workaround. > > > > *Question: * > > How to create different set of JavaRDD based on different input arguments > dynamically. Trying to implement something like factory pattern. > > > > *Error Message: * > > RDD transformations and actions can only be invoked by the driver, not > inside of other transformations; for example, rdd1.map(x => > rdd2.values.count() * x) is invalid because the values transformation and > count action cannot be performed inside of the rdd1.map transformation. For > more information, see SPARK-5063. > > > > Thanks, > > > > Rachana >
