Thanks. some of the confusion comes from DAG offering up commonTaskLocalFiles (which support extraction patterns and magically making the resources classpath aware, -- now obviously not the extracted bits --) but not a 'commonTaskEnvironment', so some naive leaps were made.
ckw On Sep 11, 2014, at 1:15 PM, Hitesh Shah <[email protected]> wrote: > Hi Chris, > > Unlike MR and its support of distributed cache, Tez does not make any > inferences into the structure of the LocalResources specified ( i.e structure > of tarball, jar, etc ) and therefore expects the user to modify the class > path as needed. > > It might be something worth considering as a new feature ( please file a jira > ) but the current implementation expects the user to setup the classpath as > needed to handle tar-balls, fat-jars, etc correctly. > > — Hitesh > > > On Sep 11, 2014, at 12:09 PM, Chris K Wensel <[email protected]> wrote: > >> >> I'm setting my what used to be called a hadoop job jar as a local resource, >> with APPLICATION visibility, of type PATTERN with the pattern >> "(?:classes/|lib/).*" (right from the JobConf) >> >> the good news is when a remote tez client starts, the job jar is downloaded, >> and unpacked using the pattern >> >> proof: >> >> find >> /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/ >> | grep logparser.jar >> /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/.tmp_logparser.jar.crc >> /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar >> /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib >> /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/jgraphx-2.0.0.1.jar >> /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/jgrapht-ext-0.9.0.jar >> /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/jgraph-5.13.0.0.jar >> /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/cascading-xml-3.0.0-wip-dev.jar >> /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/tagsoup-1.2.jar >> /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/riffle-0.1-dev.jar >> /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/jgrapht-core-0.9.0.jar >> /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/cascading-hadoop2-tez-3.0.0-wip-dev.jar >> /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/janino-2.6.1.jar >> /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/cascading-core-3.0.0-wip-dev.jar >> /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/commons-compiler-2.6.1.jar >> /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/logparser.jar >> >> the bad news is that the 'launch_container.sh' is only adding >> ln -sf >> "/tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar" >> "logparser.jar" >> >> so the containers only see >> /tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/container_1410456214816_0001_01_000004/logparser.jar >> >> which isn't terribly helpful as it result in >> >> 2014-09-11 10:57:41,001 INFO [TezChild] >> org.apache.tez.runtime.task.TezTaskRunner: Encounted an error while >> executing task: attempt_1410456214816_0001_1_00_000000_2 >> org.apache.tez.dag.api.TezUncheckedException: Unable to load class: >> cascading.flow.tez.FlowProcessor >> at >> org.apache.tez.common.ReflectionUtils.getClazz(ReflectionUtils.java:45) >> at >> org.apache.tez.common.ReflectionUtils.createClazzInstance(ReflectionUtils.java:96) >> at >> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.createProcessor(LogicalIOProcessorRuntimeTask.java:563) >> at >> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.initialize(LogicalIOProcessorRuntimeTask.java:187) >> >> i'm obviously missing something. or is making classic job jars (with a lib >> folder) isn't really supported transparently anymore (as an option) which >> will cause some grief. >> >> this is hadoop 2.4.1 >> >> ckw >> >> -- >> Chris K Wensel >> [email protected] >> http://concurrentinc.com >> > -- Chris K Wensel [email protected] http://concurrentinc.com
