Maybe memcahced can help http://www.danga.com/memcached/


-----Original Message-----
From: Arun C Murthy [mailto:[EMAIL PROTECTED]
Sent: Wed 4/30/2008 12:35 PM
To: [email protected]
Subject: Re: DistributedCache on Java Objects?
 

On Apr 30, 2008, at 8:14 AM, ncardoso wrote:

>
> Hello,
>
> I'm using Hadoop for distributed text mining of large collection of
> documents, and in my optimizing process, I want to speed things up  
> a bit,
> and I want to know how can I do this step with Hadoop...
>
> Each Map process takes a group of documents, analyses each  
> sentence, and for
> certain patterns it queries a database and some indexes to provide  
> a proper
> answer. This step can take a while, so I'm caching the results in a
> LinkedHashMap, which works pretty well for standalone jobs, and avoids
> repeated queries for same patterns in a documet.
>
> I think it would be great to share this LinkedHashMap cache object  
> for all
> Map instances, so that if the #2 Map object finds the same pattern  
> as the #1
> Map object previously noticed on other document, it can use the cached
> result that #1 Map placed there for all Map objects, saving some time.
>
> Right now, the DistributedCache just shares files, archives and jar  
> files.
> Is there any way to share such a Java object such as a LinkedHashMap,
> synchronized or not?
>

Not directly. You could serialize your LinkedHashMap into a file via  
something like java.io.DataOutput, store it on HDFS and then  
deserialize it via java.io.DataInput in your map's configure method.  
Does that work?

Arun

> -- 
> View this message in context: http://www.nabble.com/ 
> DistributedCache-on-Java-Objects--tp16985074p16985074.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
>


Reply via email to