These are good inputs Sanjay. Thanks for the help. On Wed, Mar 17, 2010 at 11:33 AM, Sanjay Sharma <sanjay.sha...@impetus.co.in > wrote:
> Hi Ninad, > You can always use Java object serialization to store custom objects as > files in Hadoop distributed cache before map/reducer start running. > The thumb rule steps of such usage is- > a. Create the object while configuring your job, serialize it to a file and > put it is distributed cache > b. Initialize the same object in Map or Reducer init/setup > > The known issues with this approach is - > a. additional hit of file based object > ser-de(serialization-de-serialization). So do check whether the cumulative > object creation time is more than the time in object ser-de > b. works only in case of write-once read-only object > > In case the object size is a problem, suggest going for memcache or BDB or > tokyotyrant or similar fast store for such distributed caching > > Regards, > Sanjay Sharma > > -----Original Message----- > From: Ninad Raut [mailto:hbase.user.ni...@gmail.com] > Sent: Wednesday, March 17, 2010 10:32 AM > To: common-user@hadoop.apache.org > Subject: Storing Custom Java Objects in Hadoop Distibuted Cache > > Hi All, > > Currently hadoop cache only allows files to be stored in the distributed > cache. > > Can I store custom objects in hadoop cache so that all my map reduce > programs use those objects? > > > Pleas let me know if any one has a solution to this problem. The Object > creation is a very heavy task and I want to do it only once for all Map - > Reduce Routines. > > > Regards, > Ninad ..... > > Impetus Technologies is participating at the CTIA Wireless 2010 from 23rd > to 25th March 2010. Meet Impetus in Las Vegas to experience our mobile and > wireless domain expertise. Click http://impetus.com/events to know more. > > Follow our updates on www.twitter.com/impetuscalling. > > NOTE: This message may contain information that is confidential, > proprietary, privileged or otherwise protected by law. The message is > intended solely for the named addressee. If received in error, please > destroy and notify the sender. Any use of this email is prohibited when > received in error. Impetus does not represent, warrant and/or guarantee, > that the integrity of this communication has been maintained nor that the > communication is free of errors, virus, interception or interference. >
