Yes, this can be done. quick python equivalent: # In Driver fileList=["/file1.txt","/file2.txt"] rdd = [] for f in fileList: rdd = jsc.textFile(f) rdds.append(rdd)
On Sat, Oct 31, 2015 at 11:09 PM, amit tewari <amittewar...@gmail.com> wrote: > Hi > > I need the ability to be able to create RDDs programatically inside my > program (e.g. based on varaible number of input files). > > Can this be done? > > I need this as I want to run the following statement inside an iteration: > > JavaRDD<String> rdd1 = jsc.textFile("/file1.txt"); > > Thanks > Amit > -- Best Regards, Ayan Guha