On 05/25/2011 04:27 PM, Giridhar Addepalli wrote:
Hi,
We have MapReduce program which writes data to mysql database using
DBOutputFormat.
Our program has one reducer.
I understand that all the inserts happen during the close() operation
of the reducer.
Is it gauranteed that this operation is atomic ? i.e; what happens if
the writes fail in the middle of the operation.
Does it mean that only partial number of rows get in to mysql database?
What does it take to make the write operation atomic ?
Any suggestions around our situation( alternative solutions ) are welcome.
Thanks,
Giridhar.
Sqoop is done for that kind of tasks.
Definition
========
Sqoop is an open-source tool that allows users to extract data from a
relational database into Hadoop for further processing.
This processing can be done with MapReduce programs or other
higher-level tools such as Hive. When the final results of an analytic
pipeline are available, Sqoop can export
these results back to the database for consumption by other clients.
Source code
==========
http://github.com/cloudera/sqoop
Regards
--
Marcos Luis Ortiz Valmaseda
Software Engineer (Distributed Systems)
http://uncubanitolinuxero.blogspot.com
