Hello First issue from the stack trace: "org.apache.tez.dag.api.TezUncheckedException: Invalid configuration of tez jars, tez.lib.uris is not defined in the configuration”. It looks like your first run failed due to this. The second run seemed to be configured correctly but failed for a different reason. For this, look for a line along the lines of "org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1426563447689_0001” to find the application ID. Now, invoke “bin/yarn logs -applicationId <appId>” to get all the logs and attach it to your email to this list so that we can take a look to see if there are other issues.
Actually, my guess is that you have 2 different versions of tez in use. Based on the stack trace, you seem to be using tez-0.5.2 on the client machine but your tarball on HDFS is tez-0.6.0. Both of these need to have matching versions. If you go to the ResourceManager and look at the UI for the applicationId, its diagnostics might likely tell you that the AM was incompatible with the client. If you see a different error, please send across the logs from the above command and we can help further. thanks — HItesh On Mar 17, 2015, at 6:27 AM, dagriq <[email protected]> wrote: > hi > > I have downloaded and compiled Tez and I received one error when I tried at > the bash shell: [ pig -x tez myscript.pig ]. > > I executed all steps from the [ http://tez.apache.org/install.html ] and I > have a new folder in /usr/local/apache-tez. > I added one more subfolder called conf where I stored the new file > tez-site.xml. > My first error message is copied below: > > > 2015-03-16 23:39:00,463 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - For vertex > - scope-34: parallelism=1, memory=1024, java > opts=-Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx200m > -Dlog4j.configuration=tez-container-log4j.properties > -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA > 2015-03-16 23:39:00,862 [PigTezLauncher-0] INFO > org.apache.pig.tools.pigstats.tez.TezScriptState - Pig script settings are > added to the job > 2015-03-16 23:39:01,576 [PigTezLauncher-0] INFO > org.apache.tez.client.TezClient - Tez Client Version: [ component=tez-api, > version=0.5.2, revision=5d728786b681e549a6f33375991dcce97cbf79f3, > SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, > buildTIme=20141104-1809 ] > 2015-03-16 23:39:01,757 [PigTezLauncher-0] INFO > org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at > /0.0.0.0:8032 > 2015-03-16 23:39:02,798 [PigTezLauncher-0] INFO > org.apache.tez.client.TezClient - Session mode. Starting session. > > 2015-03-16 23:39:02,800 [PigTezLauncher-0] ERROR > org.apache.pig.backend.hadoop.executionengine.tez.TezJob - Cannot submit DAG > > org.apache.tez.dag.api.TezUncheckedException: Invalid configuration of tez > jars, tez.lib.uris is not defined in the configuration > > at > org.apache.tez.client.TezClientUtils.setupTezJarsLocalResources(TezClientUtils.java:162) > at > org.apache.tez.client.TezClient.getTezJarResources(TezClient.java:711) > at org.apache.tez.client.TezClient.start(TezClient.java:294) > at > org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.createSession(TezSessionManager.java:95) > at > org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.getClient(TezSessionManager.java:195) > at > org.apache.pig.backend.hadoop.executionengine.tez.TezJob.run(TezJob.java:159) > at > org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher$1.run(TezLauncher.java:167) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > 2015-03-16 23:39:03,758 [main] INFO > org.apache.pig.tools.pigstats.tez.TezPigScriptStats - Script Statistics: > > HadoopVersion: 2.6.0 > > PigVersion: 0.14.0 > > TezVersion: 0.5.2 > > UserId: dagriq > > FileName: dev2/email4.pig > > StartedAt: 2015-03-16 23:38:55 > > FinishedAt: 2015-03-16 23:39:03 > > Features: FILTER > > > Failed! > > DAG PigLatin:dummy data-0_scope-0: > ApplicationId: null > > TotalLaunchedTasks: -1 > > FileBytesRead: -1 > > FileBytesWritten: -1 > > HdfsBytesRead: 0 > > HdfsBytesWritten: 0 > > > Input(s): > > Output(s): > > 2015-03-16 23:39:03,781 [main] ERROR org.apache.pig.tools.grunt.GruntParser - > ERROR 2244: Job failed, hadoop does not return any error message > Details at logfile: /usr/local/pig/logs/pig-err.log > 2015-03-16 23:39:03,860 [main] INFO org.apache.pig.Main - Pig script > completed in 19 seconds and 936 milliseconds (19936 ms) > 2015-03-16 23:39:03,860 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher - Shutting down > thread pool > > So I copied the tez-site.xml to the hadoop files and I re-executed the above > mentioned command: pig -x tez myscript.pig. I received another error , I am > copying it below: > > 2015-03-16 23:40:28,679 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.plan.TezCompiler - File > concatenation threshold: 100 optimistic? false > 2015-03-16 23:40:28,805 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - > mapreduce.inputformat.class is deprecated. Instead, use > mapreduce.job.inputformat.class > 2015-03-16 23:40:29,050 [main] INFO > org.apache.tez.mapreduce.hadoop.MRInputHelpers - Generating mapreduce api > input splits > 2015-03-16 23:40:29,276 [main] INFO > org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to > process : 4 > 2015-03-16 23:40:29,280 [main] INFO > org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input > paths to process : 4 > 2015-03-16 23:40:29,449 [main] INFO > org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input > paths (combined) to process : 1 > 2015-03-16 23:40:29,602 [main] INFO > org.apache.tez.mapreduce.hadoop.MRInputHelpers - NumSplits: 1, > SerializedSize: 614 > 2015-03-16 23:40:31,773 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local > resource: joda-time-2.1.jar > 2015-03-16 23:40:31,773 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local > resource: guava-11.0.2.jar > 2015-03-16 23:40:31,774 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local > resource: pig-0.14.0-core-h2.jar > 2015-03-16 23:40:31,774 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local > resource: antlr-runtime-3.4.jar > 2015-03-16 23:40:31,775 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - Local > resource: automaton-1.11-8.jar > 2015-03-16 23:40:31,884 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is > deprecated. Instead, use fs.defaultFS > 2015-03-16 23:40:32,359 [main] INFO > org.apache.hadoop.conf.Configuration.deprecation - mapred.output.compress is > deprecated. Instead, use mapreduce.output.fileoutputformat.compress > 2015-03-16 23:40:32,365 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - > Setting tez.runtime.shuffle.merge.percent to 0.66 from MR setting > mapreduce.reduce.shuffle.merge.percent > 2015-03-16 23:40:32,366 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - > Setting tez.runtime.shuffle.fetch.buffer.percent to 0.70 from MR setting > mapreduce.reduce.shuffle.input.buffer.percent > 2015-03-16 23:40:32,367 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - > Setting tez.runtime.io.sort.mb to 100 from MR setting > mapreduce.task.io.sort.mb > 2015-03-16 23:40:32,367 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - > Setting tez.runtime.shuffle.memory.limit.percent to 0.25 from MR setting > mapreduce.reduce.shuffle.memory.limit.percent > 2015-03-16 23:40:32,368 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - > Setting tez.runtime.io.sort.factor to 10 from MR setting > mapreduce.task.io.sort.factor > 2015-03-16 23:40:32,368 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - > Setting tez.runtime.shuffle.connect.timeout to 180000 from MR setting > mapreduce.reduce.shuffle.connect.timeout > 2015-03-16 23:40:32,369 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - > Setting tez.runtime.internal.sorter.class to org.apache.hadoop.util.QuickSort > from MR setting map.sort.class > 2015-03-16 23:40:32,370 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - > Setting tez.runtime.merge.progress.records to 10000 from MR setting > mapreduce.task.merge.progress.records > 2015-03-16 23:40:32,370 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - > Setting tez.runtime.compress to false from MR setting > mapreduce.map.output.compress > 2015-03-16 23:40:32,371 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - > Setting tez.runtime.sort.spill.percent to 0.80 from MR setting > mapreduce.map.sort.spill.percent > 2015-03-16 23:40:32,372 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - > Setting tez.runtime.shuffle.ssl.enable to false from MR setting > mapreduce.shuffle.ssl.enabled > 2015-03-16 23:40:32,373 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - > Setting tez.runtime.ifile.readahead to true from MR setting > mapreduce.ifile.readahead > 2015-03-16 23:40:32,373 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - > Setting tez.runtime.shuffle.parallel.copies to 5 from MR setting > mapreduce.reduce.shuffle.parallelcopies > 2015-03-16 23:40:32,374 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - > Setting tez.runtime.ifile.readahead.bytes to 4194304 from MR setting > mapreduce.ifile.readahead.bytes > 2015-03-16 23:40:32,374 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - > Setting tez.runtime.task.input.post-merge.buffer.percent to 0.0 from MR > setting mapreduce.reduce.input.buffer.percent > 2015-03-16 23:40:32,375 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - > Setting tez.runtime.shuffle.read.timeout to 180000 from MR setting > mapreduce.reduce.shuffle.read.timeout > 2015-03-16 23:40:32,375 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.util.MRToTezHelper - > Setting tez.runtime.compress.codec to > org.apache.hadoop.io.compress.DefaultCodec from MR setting > mapreduce.map.output.compress.codec > 2015-03-16 23:40:32,707 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.TezJobCompiler - For vertex > - scope-34: parallelism=1, memory=1024, java > opts=-Djava.net.preferIPv4Stack=true -Dhadoop.metrics.log.level=WARN -Xmx200m > -Dlog4j.configuration=tez-container-log4j.properties > -Dyarn.app.container.log.dir=<LOG_DIR> -Dtez.root.logger=INFO,CLA > 2015-03-16 23:40:33,187 [PigTezLauncher-0] INFO > org.apache.pig.tools.pigstats.tez.TezScriptState - Pig script settings are > added to the job > 2015-03-16 23:40:33,899 [PigTezLauncher-0] INFO > org.apache.tez.client.TezClient - Tez Client Version: [ component=tez-api, > version=0.5.2, revision=5d728786b681e549a6f33375991dcce97cbf79f3, > SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, > buildTIme=20141104-1809 ] > 2015-03-16 23:40:34,098 [PigTezLauncher-0] INFO > org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at > /0.0.0.0:8032 > 2015-03-16 23:40:34,923 [PigTezLauncher-0] INFO > org.apache.tez.client.TezClient - Session mode. Starting session. > 2015-03-16 23:40:34,923 [PigTezLauncher-0] INFO > org.apache.tez.client.TezClient - Session mode. Starting session. > 2015-03-16 23:40:34,924 [PigTezLauncher-0] INFO > org.apache.tez.client.TezClientUtils - Using tez.lib.uris value from > configuration: hdfs://localhost:9000/apps/tez-0.6.0-SNAPSHOT/tez-0.6.0.tar.gz > 2015-03-16 23:40:35,129 [PigTezLauncher-0] INFO > org.apache.tez.client.TezClient - Tez system stage directory > hdfs://localhost:9000/tmp/temp-1303134320/.tez/application_1426563447689_0001 > doesn't exist and is created > 2015-03-16 23:40:36,048 [PigTezLauncher-0] INFO > org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application > application_1426563447689_0001 > 2015-03-16 23:40:36,055 [PigTezLauncher-0] INFO > org.apache.tez.client.TezClient - The url to track the Tez Session: > http://optimf:8088/proxy/application_1426563447689_0001/ > > 2015-03-16 23:40:59,709 [PigTezLauncher-0] ERROR > org.apache.pig.backend.hadoop.executionengine.tez.TezJob - Cannot submit DAG > org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown > at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:603) > at > org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.createSession(TezSessionManager.java:100) > at > org.apache.pig.backend.hadoop.executionengine.tez.TezSessionManager.getClient(TezSessionManager.java:195) > at > org.apache.pig.backend.hadoop.executionengine.tez.TezJob.run(TezJob.java:159) > at > org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher$1.run(TezLauncher.java:167) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > 2015-03-16 23:41:00,014 [main] INFO > org.apache.pig.tools.pigstats.tez.TezPigScriptStats - Script Statistics: > > HadoopVersion: 2.6.0 > > PigVersion: 0.14.0 > > TezVersion: 0.5.2 > > UserId: dagriq > > FileName: dev2/email4.pig > > StartedAt: 2015-03-16 23:40:28 > > FinishedAt: 2015-03-16 23:41:00 > > Features: FILTER > > > Failed! > > > DAG PigLatin:dummy data-0_scope-0: > ApplicationId: null > > TotalLaunchedTasks: -1 > > FileBytesRead: -1 > > FileBytesWritten: -1 > > HdfsBytesRead: 0 > > HdfsBytesWritten: 0 > > > Input(s): > > Output(s): > > 2015-03-16 23:41:00,036 [main] ERROR org.apache.pig.tools.grunt.GruntParser - > ERROR 2244: Job failed, hadoop does not return any error message > Details at logfile: /usr/local/pig/logs/pig-err.log > 2015-03-16 23:41:00,146 [main] INFO org.apache.pig.Main - Pig script > completed in 42 seconds and 897 milliseconds (42897 ms) > 2015-03-16 23:41:00,147 [main] INFO > org.apache.pig.backend.hadoop.executionengine.tez.TezLauncher - Shutting down > thread pool > > > What is wrong in my installation ? > Thank you for your help! > >
