Bug in LocalJobRunner?

[Alex Baranau]

Hi!

In a test I tried to run a job which is in separate jar with
MiniMRYarnCluster and got ClassNotFoundException for the Mapper class that
is in that jar.

I went a bit deeper and found possible bug in LocalJobRunner:

currently:

      this.job = new JobConf(localJobFile);

      // Job (the current object) is a Thread, so we wrap its class loader.
      if (localDistributedCacheManager.hasLocalClasspaths()) {
        setContextClassLoader(localDistributedCacheManager.makeClassLoader(
                getContextClassLoader()));
      }

should be:

      this.job = new JobConf(localJobFile);
      // Job (the current object) is a Thread, so we wrap its class loader.
      if (localDistributedCacheManager.hasLocalClasspaths()) {
        ClassLoader classLoader =
localDistributedCacheManager.makeClassLoader(getContextClassLoader());
        setContextClassLoader(classLoader);
        this.job.setClassLoader(classLoader);
      }

I.e. we need to set classloader for job configuration so that it can load
classes from the jar.

If the above makes sense I will file JIRA with patch, otherwise, what am I
missing?


Thank you,
Alex Baranau