Hi Doug, Thanks - it is not so much I want to run in a single JVM - I do want a bunch of machines doing the work, it is just I want them all to have this in-memory lookup index, that is configured once per job. Is there some hook somewhere that I can trigger a read from the distributed cache, or is a Mapper.configure() the best place for this? Can it be called multiple times per Job meaning I need to keep some static synchronised indicator flag?
Thanks again, Tim On Tue, Nov 25, 2008 at 8:41 PM, Doug Cutting <[EMAIL PROTECTED]> wrote: > tim robertson wrote: >> >> Thanks Alex - this will allow me to share the shapefile, but I need to >> "one time only per job per jvm" read it, parse it and store the >> objects in the index. >> Is the Mapper.configure() the best place to do this? E.g. will it >> only be called once per job? > > In 0.19, with HADOOP-249, all tasks from a job can be run in a single JVM. > So, yes, you could access a static cache from Mapper.configure(). > > Doug > >
