You can have your item in a separate jar and pin the reference so that it becomes perm-gen, which will pin it. Then you can search the class loader hierarchy for the reference. A quick scan through the Child.java main loop shows no magic with class loaders.
I wrote some code to check this against 0.19.0. Very clearly the JVM is doing nothing special with class loaders. The classes are loaded exactly once. The job counters contain details of how many times the map method was called and the number of times the singleton was taken, including the jvm pid's I wrote a mapper and a singleton, the singleton has 1 method that returns the number of times that the getSingleton The code fragment is from the examples in my book http://www.apress.com/book/view/9781430219422 On Fri, Mar 6, 2009 at 12:55 PM, Scott Carey <sc...@richrelevance.com>wrote: > One further thought on this, the mapper jvm may be loading the jar and > overwriting / throwing away all previous class descriptions from the > previous map job, which will remove the statics and reinitialize. In this > case, the singleton won't work if it is in the job jar. What will work is > putting the singleton in a global classpath (shared library not in the job > jar). > > > On 3/6/09 12:46 PM, "Scott Carey" <sc...@richrelevance.com> wrote: > > The difference is that if the whole mapper class itself is being reloaded > somehow (instantiated by reflection and then de-referenced and gc'd?) the > static won't work the way you expect. Not knowing how that works, and > assuming that statics there don't work, a singleton in another class may > still work. The singleton class is certainly not being instantiated by > reflection so (I believe) only a classloader closing will get rid of it. > > At least, its worth a try, since unlike the mapper class, you control how > it is instantiated. So the two cases are not the same. > > > On 3/6/09 12:13 AM, "Rasit OZDAS" <rasitoz...@gmail.com> wrote: > > Owen, I tried this, it doesn't work. > I doubt if static singleton method will work either, > since it's much or less the same. > > Rasit > > 2009/3/2 Owen O'Malley <omal...@apache.org> > > > > > On Mar 2, 2009, at 3:03 AM, Tom White wrote: > > > > I believe the static singleton approach outlined by Scott will work > >> since the map classes are in a single classloader (but I haven't > >> actually tried this). > >> > > > > Even easier, you should just be able to do it with static initialization > in > > the Mapper class. (I haven't tried it either... ) > > > > -- Owen > > > > > > -- > M. Raşit ÖZDAŞ > > >
The Job is complete and successfull Counter Group: File Systems HDFS bytes read 11539 HDFS bytes written 1116 Counter Group: Input Total 24 Counter Group: Job Counters Launched map tasks 24 Counter Group: Output Total 24 /** Each group here is named for the jvm pid and hostname, and clearly shows the number of tasks run per jvm */ Counter Group: 3...@host1 attempt_200902221346_0096_m_000014_0 1 attempt_200902221346_0096_m_000018_0 1 attempt_200902221346_0096_m_000021_0 1 Counter Group: 16...@host2 attempt_200902221346_0096_m_000013_0 1 attempt_200902221346_0096_m_000009_0 1 attempt_200902221346_0096_m_000006_0 1 attempt_200902221346_0096_m_000017_0 1 attempt_200902221346_0096_m_000023_0 1 Counter Group: 2...@host1 attempt_200902221346_0096_m_000012_0 1 attempt_200902221346_0096_m_000005_0 1 attempt_200902221346_0096_m_000000_0 1 attempt_200902221346_0096_m_000001_0 1 attempt_200902221346_0096_m_000010_0 1 attempt_200902221346_0096_m_000007_0 1 Counter Group: 16...@host2 attempt_200902221346_0096_m_000019_0 1 attempt_200902221346_0096_m_000015_0 1 attempt_200902221346_0096_m_000008_0 1 attempt_200902221346_0096_m_000003_0 1 attempt_200902221346_0096_m_000011_0 1 Counter Group: 2...@host1 attempt_200902221346_0096_m_000022_0 1 attempt_200902221346_0096_m_000002_0 1 attempt_200902221346_0096_m_000020_0 1 attempt_200902221346_0096_m_000016_0 1 attempt_200902221346_0096_m_000004_0 1 /** This shows the number of times the map method was called in each task. */ Counter Group: Tasks attempt_200902221346_0096_m_000014_0 1 attempt_200902221346_0096_m_000012_0 1 attempt_200902221346_0096_m_000015_0 1 attempt_200902221346_0096_m_000009_0 1 attempt_200902221346_0096_m_000006_0 1 attempt_200902221346_0096_m_000022_0 1 attempt_200902221346_0096_m_000021_0 1 attempt_200902221346_0096_m_000020_0 1 attempt_200902221346_0096_m_000001_0 1 attempt_200902221346_0096_m_000003_0 1 attempt_200902221346_0096_m_000017_0 1 attempt_200902221346_0096_m_000004_0 1 attempt_200902221346_0096_m_000019_0 1 attempt_200902221346_0096_m_000013_0 1 attempt_200902221346_0096_m_000005_0 1 attempt_200902221346_0096_m_000000_0 1 attempt_200902221346_0096_m_000018_0 1 attempt_200902221346_0096_m_000002_0 1 attempt_200902221346_0096_m_000008_0 1 attempt_200902221346_0096_m_000010_0 1 attempt_200902221346_0096_m_000016_0 1 attempt_200902221346_0096_m_000007_0 1 attempt_200902221346_0096_m_000023_0 1 attempt_200902221346_0096_m_000011_0 1 /** This shows the number of times the singleton has been requested, in a given task */ Counter Group: Singleton attempt_200902221346_0096_m_000014_0 1 attempt_200902221346_0096_m_000012_0 6 attempt_200902221346_0096_m_000015_0 4 attempt_200902221346_0096_m_000009_0 2 attempt_200902221346_0096_m_000006_0 1 attempt_200902221346_0096_m_000022_0 5 attempt_200902221346_0096_m_000021_0 3 attempt_200902221346_0096_m_000020_0 4 attempt_200902221346_0096_m_000001_0 2 attempt_200902221346_0096_m_000003_0 1 attempt_200902221346_0096_m_000017_0 4 attempt_200902221346_0096_m_000004_0 2 attempt_200902221346_0096_m_000019_0 5 attempt_200902221346_0096_m_000013_0 3 attempt_200902221346_0096_m_000005_0 3 attempt_200902221346_0096_m_000000_0 1 attempt_200902221346_0096_m_000018_0 2 attempt_200902221346_0096_m_000002_0 1 attempt_200902221346_0096_m_000008_0 2 attempt_200902221346_0096_m_000010_0 5 attempt_200902221346_0096_m_000016_0 3 attempt_200902221346_0096_m_000007_0 4 attempt_200902221346_0096_m_000023_0 5 attempt_200902221346_0096_m_000011_0 3 Counter Group: Map-Reduce Framework Map input records 24 Map input bytes 316 Map output records 24
// Code Fragment from the chapter on Advanced Techniques in the book ProHadoop by Apress.com /** This class just keeps a count of the number of times the singleton was gotten through the factory. */ public static class Singleton { /** Our singleton. */ public static Singleton singleton = null; /** counter of the number of times the singleton has been requested. */ static AtomicLong getCounter = new AtomicLong(0); static { /** Construct the singleton here since we know we are going to us it. */ singleton = new Singleton(); } protected Singleton() { } /** Get an instance of the singleton, and increment our static counter. * * * @return the singleton */ public static Singleton getSingleton() { getCounter.incrementAndGet(); return singleton; } /** Return the number of times the singleton has been requested. */ public long getCurrentCounter() { return getCounter.get(); } } /** * @author Jason * */ public static class TestMapper extends MapReduceBase implements Mapper<LongWritable, Text, Text, LongWritable> { Singleton singleton; String taskName; TaskAttemptID taskId; /** grab our singleton and the taskid. */ @Override public void configure(JobConf conf) { taskName = conf.getJobName(); taskId = TaskAttemptID.forName(conf.get("mapred.task.id")); if (taskName == null || taskName.length() == 0) { /** if the job name is essentially unset make something up. */ taskName = taskId.isMap() ? "map." : "reduce." + this.getClass().getName(); } singleton = Singleton.getSingleton(); } /** assume there is only 1 key per map, and report on information about the singleton state. */ @Override public void map(LongWritable key, Text value, OutputCollector<Text, LongWritable> output, Reporter reporter) throws IOException { /** Our key is the line number, which should be 1, and the single line, which should be the file name. */ try { reporter.incrCounter("Input", "Total", 1); long initCount = singleton.getCurrentCounter(); reporter.incrCounter("Singleton", taskId.toString(), initCount); reporter.incrCounter("Tasks", taskId.toString(), 1); key.set(initCount); output.collect(value, key); reporter.incrCounter("Output", "Total", 1); reporter.incrCounter(ManagementFactory.getRuntimeMXBean().getName(), taskId.toString(), 1); } catch (Throwable e) { reporter.incrCounter("Exceptions", "Total", 1); reporter.incrCounter("Exceptions", e.getClass().getName(), 1); if (e instanceof RuntimeException) { throw (RuntimeException) e; } if (e instanceof IOException) { throw (IOException) e; } throw new IOException(e); } } } Logger LOG = Logger.getLogger(JVMReuseAndStaticInitializers.class); /** This is where derived classes define required setup. * @throws IOException */ @Override protected void customSetup(JobConf conf) throws IOException { super.customSetup(conf); /** Setup for NLineInputFormat, which has LongWritable, Text key, values, and only 1 input line per map. */ conf.setInputFormat(NLineInputFormat.class); conf.setInt("mapred.line.input.format.linespermap",1); /** Work out how many map slots there are in the cluster. */ JobClient client = new JobClient(conf); ClusterStatus status = client.getClusterStatus(); int mapSlots = status.getMaxMapTasks(); if (verbose) { LOG.info("There are " + mapSlots + " map execution slots in this cluster, setting up for " + mapSlots * 6 + "inputs"); } /** Setup our input so we have 6 input records per map slot. */ Path root = new Path("JVMReuseAndStaticInitializers"); Utils.makeSampleInputIf(conf, root.toString(), mapSlots*6); FileInputFormat.setInputPaths(conf, root); conf.setMapperClass(TestMapper.class); /** Ensure that each map is run one time only, to avoid biasing the singleton counts per taskid. */ conf.setMapSpeculativeExecution(false); conf.setMaxMapAttempts(1); /** Setup some jvm reuse. */ conf.setNumTasksToExecutePerJvm(6); FileOutputFormat.setOutputPath(conf, root.suffix(".output")); conf.setOutputKeyClass(Text.class); conf.setOutputValueClass(LongWritable.class); conf.setNumReduceTasks(0); conf.set("mapred.child.java.opts", "-Xmx200m -verbose:class"); }