@Archit, if the hive shell was shut down, can you obtain all the tez logs using:
$yarn logs -applicationId <appId> Or, for a running query, go to the RM UI, find the application id and use the logs link to view/download the logs. thanks — Hitesh On May 12, 2014, at 11:57 PM, Archit Thakur <[email protected]> wrote: > @Vikram: I have attached the complete hive.log. PLease go in the last to see > that it keeps on going and going. > Applications logs: There are no tez logs generated. > > So, No exception or stack trace. > > @Bikas: Yes, it does in the release notes of > https://issues.apache.org/jira/browse/HIVE-6098. > There should be enough resources left because I am able to run it when I set > set hive.execution.engine=mr; > Thanks and Regards, > Archit Thakur. > > > On Mon, May 12, 2014 at 10:27 PM, Bikas Saha <[email protected]> wrote: > Do the hive on tez instruction not specify that the hive-exec jar needs to be > uploaded first? > > > About the stuck job, you can check the resources in the cluster via the RM > web UI. Please confirm that there are enough resources left in the cluster to > run tasks. It may be that the cluster only has enough resources to run the > App Master (controller process) and not the tasks. > > > From: Archit Thakur [mailto:[email protected]] > Sent: Sunday, May 11, 2014 9:59 PM > To: [email protected] > Cc: [email protected] > Subject: Re: Tez HIve problems. > > > Hi Bikas, > > I was able to resolve this issue, by copying hive-exec.jar to > hdfs://user/admin/, It was not able to localize it while running Tez. > > Though, I am facing some other problem now. My Tez job is not progressing, > > My Tez job doesn't progress, it just keeps printing: > Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 > Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 > Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 > Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 > Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 > Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 > Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 > Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 > Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 > Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 > Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 > Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 > Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 > Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 > Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 > Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 > Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 > Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 > Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 > Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 > Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 > Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 > Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 > Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 > Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 > > In hive logs also, it keeps on printing the same. > > Any idea, where should I look into or how should I debug the problem. Or > pointers on problem itself would help. > > Thanks Archit. > > > On Sun, May 11, 2014 at 7:28 AM, Bikas Saha <[email protected]> wrote: > > Setting execution engine to mr and framework name to yarn = Hive compiles to > MR and runs on MR. > > Setting execution engine to mr and framework name to yarn-tez = Hive compiles > to MR and runs on Tez. > > Setting execution engine to tez = Hive compiles to Tez and runs on Tez. > > > Did you change the execution engine in the same shell that was earlier > running with execution engine set to mr? If yes, then try and set the > execution engine in hive-site.xml and use a new hive shell. > > Have you set all required hive configurations? > > > If you can still reproduce the issue, then you should create a jira in the > Apache hive project to track the bug. > > > Bikas > > > From: Archit Thakur [mailto:[email protected]] > Sent: Thursday, May 08, 2014 4:42 AM > To: [email protected]; [email protected] > Subject: Tez HIve problems. > > > Hi, > > I am facing few problems while using Tez on Hive. > > I am able to run Tez through YARN submitting independent MR procesing job. > > Also, when I run SELECT query on hive shell by setting > > set hive.execution.engine=mr; > > but setting mapreduce.framework.name=yarn-tez > > This launches tez job, I checked by seeing AM's UI of machine. > > But when I set hive.execution.engine=tez; > > It gives me > > Query ID = admin_20140508112525_37cf7b55-75ac-408c-b96f-8e40ba2f8e2b > Total jobs = 1 > Launching Job 1 out of 1 > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.tez.TezTask > > I checked the hive logs: > > exec.Task (TezTask.java:execute(185)) - Failed to execute tez graph. > java.io.FileNotFoundException: File does not exist: hdfs:/user/admin > at > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110) > at > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102) > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.getDefaultDestDir(DagUtils.java:638) > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.getHiveJarDirectory(DagUtils.java:728) > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.createJarLocalResource(TezSessionState.java:314) > at > org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:169) > at > org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:153) > at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153) > at > org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85) > at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504) > at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271) > at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1089) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:912) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902) > at > org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268) > at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220) > at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423) > at > org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792) > at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > > Am I doing something wrong? The Tez I am running by setting > hive.execution.engine=mr; but setting mapreduce.framework.name=yarn-tez. is > it correct? Does it launches Tez job or is it MR? > > Thanks and Regards, > > Archit Thakur. > > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader of > this message is not the intended recipient, you are hereby notified that any > printing, copying, dissemination, distribution, disclosure or forwarding of > this communication is strictly prohibited. If you have received this > communication in error, please contact the sender immediately and delete it > from your system. Thank You. > > > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader of > this message is not the intended recipient, you are hereby notified that any > printing, copying, dissemination, distribution, disclosure or forwarding of > this communication is strictly prohibited. If you have received this > communication in error, please contact the sender immediately and delete it > from your system. Thank You. > > <hive.log>
