Database to use with Hadoop

2009-10-13 Thread Mark Kerzner
Hi, I run Elastic MapReduce. The output of my application is a text file, where each line is essentially a set of fields. It will fit very nicely into a simple database, but which database 1. Is persistent after cluster shutdown; 2. Can be written to by many reducers? Amazon SimpleDB could

Re: Database to use with Hadoop

2009-10-13 Thread Amandeep Khurana
You can put into Hbase. Or you can use the DBOutputFormat and interface with an RDBMS. Amandeep Khurana Computer Science Graduate Student University of California, Santa Cruz On Tue, Oct 13, 2009 at 3:12 PM, Mark Kerzner markkerz...@gmail.com wrote: Hi, I run Elastic MapReduce. The output

Re: Database to use with Hadoop

2009-10-13 Thread Jeff Hammerbacher
Hey Mark, You will probably get some mileage from http://developer.amazonwebservices.com/connect/entry.jspa?externalID=2571. Regards, Jeff On Tue, Oct 13, 2009 at 3:19 PM, Amandeep Khurana ama...@gmail.com wrote: You can put into Hbase. Or you can use the DBOutputFormat and interface with

Re: Database to use with Hadoop

2009-10-13 Thread Mark Kerzner
Thank you, all. It looks like SimpleDB may be good enough for my needs. The forums claim that you can write to it from all reducers at once, being that it is highly optimized for concurrent access. On Tue, Oct 13, 2009 at 5:30 PM, Jeff Hammerbacher ham...@cloudera.comwrote: Hey Mark, You will