Maps and reducers run in separate JVMs so you will have to think of a 
distributed way to share data (data structures or files).
The distributed options can be a distributed cache solution such as memcached 
or Hbase or Tokyotyrant/BDB or a simple file on HDFS. You can serialize your 
java objects including data structures in the cache as values.
With any cache solution, you would have to incur the cost of ser-de as well as 
distribution. Updating value objects in the cache is possible- it will have 
costs/overhead of its own, but remember that map reduce is all about stateless 
programming
In short, data structure sharing might not be a good idea at all- until unless 
it is a read-only cache, else rethink the algorithm.
- Sanjay


-----Original Message-----
From: Denim Live [mailto:[email protected]]
Sent: Monday, July 26, 2010 6:49 PM
To: [email protected]
Subject: sharing data structure among mappers or reducers

Is it possible to share a datastructures such as linked list or array among the
mapper (or reducer) nodes. I am not asking about sharing files, which can be
done by distributed cache. I want to share some data structure.




Meet Impetus at the OSCON 2010 in Portland, Oregon during July 19th to 23rd. 
Listen to our Senior Director of Engineering and expert speaker Vineet Tyagi 
talk about building a Ruby application server.

Click http://www.impetus.com/ to know more. Follow our updates on 
www.twitter.com/impetuscalling .


NOTE: This message may contain information that is confidential, proprietary, 
privileged or otherwise protected by law. The message is intended solely for 
the named addressee. If received in error, please destroy and notify the 
sender. Any use of this email is prohibited when received in error. Impetus 
does not represent, warrant and/or guarantee, that the integrity of this 
communication has been maintained nor that the communication is free of errors, 
virus, interception or interference.

Reply via email to