There should be no need to encode the object (I imagine you're passing
it through the conf). Try (as Tim hints at below) saving the object to a
flat file and using DistributedCache; pass the path to the serialized
object file through the configuration instead.
Although you "shouldn't" put small files in HDFS, for situations like
this where you're distributing serialized objects, data sets, etc. it's
acceptable. Just clean them up after you don't need them anymore.
On 11/30/09 3:04 AM, Tim Robertson wrote:
How do you pass it to every map function?
Are you putting it in a DistributedCache and pulling it during the Map
configuration?
Are you running on a single machine?
Cheers
Tim
On Sun, Nov 29, 2009 at 11:10 PM,<aa...@buffalo.edu> wrote:
Hello Everybody,
I have a question about object serialization in Hadoop. I have
an object A which I want to pass to every map function. Currently the code I am
using for this is as under. The problem is if I run my program, the code crashes
the first time with an error say that Java cannot deserialize the object list(
but no error when java tries to serialize it ) and then when I run the program
for the 2 time, without changing anything, the code works perfectly.
I read on some blog post that the method I have used to serialize is not the
ideal way. But this also does not explain the weird results I am getting.
try
{
ByteArrayOutputStream baos= new ByteArrayOutputStream();
ObjectOutputStream oos= new ObjectOutputStream(baos);
oos.writeObject(list);
stock_list= encode.encode(baos.toByteArray());
}
catch(IOException e)
{
e.printStackTrace();
}
Thank You
Abhishek Agrawal
SUNY- Buffalo
(716-435-7122)
