Saving spark output to multiple files as map

2018-02-24 Thread pooja bhojwani
Hello everyone, I wanted to do something like this: Given a JavaPairRDD(let's say with 10 rows), I want to store each of the rows separately with following requirements: a) Each of them should be a map(Can not use saveAsTextFile) b) The file name should have the key in it(Eg: If the key is 0,1..

Saving each line of RDD as a separate file with key as the file name

2018-01-20 Thread pooja bhojwani
Hi all, So, I have a Java Pair RDD with let’s say n lines, each line has a unique key and a hash map as the value(there are no duplicate keys). I want to save each line as a separate text file and since saveAsTextFile is not serializable, I need to somehow split the RDD into n RDD’s or so and