Hello everyone,
I wanted to do something like this:
Given a JavaPairRDD(let's say with 10 rows), I want to store each of the
rows separately with following requirements:
a) Each of them should be a map(Can not use saveAsTextFile)
b) The file name should have the key in it(Eg: If the key is 0,1.. The file
name should be something like path0_1.
Inorder to do that, I created a class which extends
MultipleTextOutputFormat, and I tweaked it a bit to generate separate files
for each row of rdd(using saveAsHadoopFile to save them).
It works fine on single machine, but for some reason it does not create the
file when I try to run it on cluster:
(works fine with --> .setMaster("local") and doesn't create that individual
file when I change Master to cluster). Any clues.
