Yes - for the mapper/reduce to run within a tez task on the cluster, the jar has to be localized for it to work.
-- Hitesh On Jul 23, 2013, at 2:24 PM, Achal Soni wrote: > Oh I was wondering why do we need that jar to begin with? Is it because it > needs it for packaging the mapper and reducer classes? > > > On Tue, Jul 23, 2013 at 1:22 PM, Hitesh Shah <[email protected]> wrote: > >> Hi Achal, >> >> I am not sure if that use-case is doable at the moment. It seems like what >> you are asking for is the ability to create a jar on the fly if a jar is >> not found. Feel free to go ahead and open a jira to address this use-case. >> >> In any case, the code does not handle a jar not being found. I filed >> TEZ-315 if you want to take that up. >> >> thanks >> -- Hitesh >> >> >> On Jul 23, 2013, at 12:13 PM, Achal Soni wrote: >> >>> I was referring to this: >>> >>> String jarPath = ClassUtil.findContainingJar(getClass()); >>> >>> Path remoteJarPath = remoteFs.makeQualified( >>> >>> new Path(remoteStagingDir, "dag_job.jar")); >>> >>> remoteFs.copyFromLocalFile(new Path(jarPath), remoteJarPath); >>> >>> FileStatus jarFileStatus = remoteFs.getFileStatus(remoteJarPath); >>> >>> >>> Map<String, LocalResource> commonLocalResources = >>> >>> new HashMap<String, LocalResource>(); >>> >>> LocalResource dagJarLocalRsrc = LocalResource.newInstance( >>> >>> ConverterUtils.getYarnUrlFromPath(remoteJarPath), >>> >>> LocalResourceType.FILE, >>> >>> LocalResourceVisibility.APPLICATION, >>> >>> jarFileStatus.getLen(), >>> >>> jarFileStatus.getModificationTime()); >>> >>> commonLocalResources.put("dag_job.jar", dagJarLocalRsrc); >>> >>> >>> This causes issues in Eclipse because there is no jar per se. >>> >>> >>> On Mon, Jul 22, 2013 at 11:25 PM, Hitesh Shah <[email protected]> wrote: >>> >>>> I am not sure what you mean by the jar containing the submission. All >> the >>>> tez jars ( from the distribution tarball ) are added as local resources >> for >>>> the AM as well as all vertices by default. >>>> >>>> -- Hitesh >>>> >>>> On Jul 22, 2013, at 10:59 PM, Achal Soni wrote: >>>> >>>>> Ah I see. And is it necessary to add the jar containing the submission >>>> as a >>>>> local resource? This is causing issues when running from Eclipse. I >> guess >>>>> it is necessary for the map and reduce classes? >>>>> >>>>> - Achal >>>> >>>> >> >>
