So, I thought about that, and I'd considered writing to the HDFS and then copying the file into the DistributedCache so each mapper/reducer doesn't have to reach into the HDFS for these files. Is that the "best" way to handle this?
On Tue, Sep 27, 2011 at 4:01 PM, GOEKE, MATTHEW (AG/1000) < [email protected]> wrote: > The simplest route I can think of is to ingest the data directly into HDFS > using Sqoop if there is a driver currently made for your database. At that > point it would be relatively simple just to read directly from HDFS in your > MR code. > > Matt > > -----Original Message----- > From: lessonz [mailto:[email protected]] > Sent: Tuesday, September 27, 2011 4:48 PM > To: [email protected] > Subject: Temporary Files to be sent to DistributedCache > > I have a need to write information retrieved from a database to a series of > files that need to be made available to my mappers. Because each mapper > needs access to all of these files, I want to put them in the > DistributedCache. Is there a preferred method to writing new information to > the DistributedCache? I can use Java's File.createTempFile(String prefix, > String suffix), but that uses the system default temporary folder. While > that should usually work, I'd rather have a method that doesn't depend on > writing to the local file system before copying files to the > DistributedCache. As I'm extremely new to Hadoop, I hope I'm not missing > something obvious. > > Thank you for your time. > This e-mail message may contain privileged and/or confidential information, > and is intended to be received only by persons entitled > to receive such information. If you have received this e-mail in error, > please notify the sender immediately. Please delete it and > all attachments from any servers, hard drives or any other media. Other use > of this e-mail by you is strictly prohibited. > > All e-mails and attachments sent and received are subject to monitoring, > reading and archival by Monsanto, including its > subsidiaries. The recipient of this e-mail is solely responsible for > checking for the presence of "Viruses" or other "Malware". > Monsanto, along with its subsidiaries, accepts no liability for any damage > caused by any such code transmitted by or accompanying > this e-mail or any attachment. > > > The information contained in this email may be subject to the export > control laws and regulations of the United States, potentially > including but not limited to the Export Administration Regulations (EAR) > and sanctions regulations issued by the U.S. Department of > Treasury, Office of Foreign Asset Controls (OFAC). As a recipient of this > information you are obligated to comply with all > applicable U.S. export laws and regulations. > >
