I'm setting my what used to be called a hadoop job jar as a local resource,
with APPLICATION visibility, of type PATTERN with the pattern
"(?:classes/|lib/).*" (right from the JobConf)
the good news is when a remote tez client starts, the job jar is downloaded,
and unpacked using the pattern
proof:
find
/tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/
| grep logparser.jar
/tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/.tmp_logparser.jar.crc
/tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar
/tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib
/tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/jgraphx-2.0.0.1.jar
/tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/jgrapht-ext-0.9.0.jar
/tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/jgraph-5.13.0.0.jar
/tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/cascading-xml-3.0.0-wip-dev.jar
/tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/tagsoup-1.2.jar
/tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/riffle-0.1-dev.jar
/tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/jgrapht-core-0.9.0.jar
/tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/cascading-hadoop2-tez-3.0.0-wip-dev.jar
/tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/janino-2.6.1.jar
/tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/cascading-core-3.0.0-wip-dev.jar
/tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/lib/commons-compiler-2.6.1.jar
/tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar/logparser.jar
the bad news is that the 'launch_container.sh' is only adding
ln -sf
"/tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/filecache/14/logparser.jar"
"logparser.jar"
so the containers only see
/tmp/hadoop-root/nm-local-dir/usercache/cwensel/appcache/application_1410456214816_0001/container_1410456214816_0001_01_000004/logparser.jar
which isn't terribly helpful as it result in
2014-09-11 10:57:41,001 INFO [TezChild]
org.apache.tez.runtime.task.TezTaskRunner: Encounted an error while executing
task: attempt_1410456214816_0001_1_00_000000_2
org.apache.tez.dag.api.TezUncheckedException: Unable to load class:
cascading.flow.tez.FlowProcessor
at
org.apache.tez.common.ReflectionUtils.getClazz(ReflectionUtils.java:45)
at
org.apache.tez.common.ReflectionUtils.createClazzInstance(ReflectionUtils.java:96)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.createProcessor(LogicalIOProcessorRuntimeTask.java:563)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.initialize(LogicalIOProcessorRuntimeTask.java:187)
i'm obviously missing something. or is making classic job jars (with a lib
folder) isn't really supported transparently anymore (as an option) which will
cause some grief.
this is hadoop 2.4.1
ckw
--
Chris K Wensel
[email protected]
http://concurrentinc.com
