Hello all,
I am relatively new to MapReduce and haven't used HBase at all. Is the
following architecture possible?
A distributed key-value store is used (HBase). So along with values, there
would be a timestamp associated with the values. Map & Reduce tasks are
executed iteratively. Map, in each iteration should take in values which
were added in the previous iteration to the store (perhaps the ones with
latest timestamp?). Reduce should take in Map's output as well as the
<key,value> pairs from the store whose key(s) match the key(s) that reduce
has to process in the current iteration. The output of reduce goes to the
store.
If this is possible, which classes (eg: InputFormat, run() of Reduce) should
be extended so that instead of the regular operation the above operation
takes place. If this is not possible, are there any alternatives to achieve
the same?
Thank you.
PS: I have put the same question on mapreduce-user apache mailing list (but
haven't got any replies yet). I found many topics on mapreduce in this
mailing list as well, so thought of posting it here also.
Regards,
Raghava.
