unsubscribe
Tez HIve problems.
Hi, I am facing few problems while using Tez on Hive. I am able to run Tez through YARN submitting independent MR procesing job. Also, when I run SELECT query on hive shell by setting *set hive.execution.engine=mr;* but setting *mapreduce.framework.name <http://mapreduce.framework.name>=yarn-tez* This launches tez job, I checked by seeing AM's UI of machine. But when I *set hive.execution.engine=tez;* *It gives me *Query ID = admin_20140508112525_37cf7b55-75ac-408c-b96f-8e40ba2f8e2b Total jobs = 1 Launching Job 1 out of 1 FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.tez.TezTask *I checked the hive logs:* exec.Task (TezTask.java:execute(185)) - Failed to execute tez graph. java.io.FileNotFoundException: File does not exist: hdfs:/user/admin at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110) at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102) at org.apache.hadoop.hive.ql.exec.tez.DagUtils.getDefaultDestDir(DagUtils.java:638) at org.apache.hadoop.hive.ql.exec.tez.DagUtils.getHiveJarDirectory(DagUtils.java:728) at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.createJarLocalResource(TezSessionState.java:314) at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:169) at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:153) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1504) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1271) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1089) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:912) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Am I doing something wrong? The Tez I am running by setting* hive.execution.engine=mr;* but setting *mapreduce.framework.name <http://mapreduce.framework.name>=yarn-tez. is it correct? Does it launches Tez job or is it MR?* Thanks and Regards, Archit Thakur.
Re: Tez HIve problems.
Hitesh: I have attached application logs. On Tue, May 13, 2014 at 8:43 PM, Carter Shanklin wrote: > Archit, > > I have seen this problem on small sandbox environments with very little > memory given to YARN. In those cases, the app master would consume enough > memory that there was no memory left to launch containers. Can you confirm > that you have enough memory to launch a few containers? > > > On Sun, May 11, 2014 at 9:59 PM, Archit Thakur > wrote: > >> Hi Bikas, >> >> I was able to resolve this issue, by copying hive-exec.jar to >> hdfs://user/admin/, It was not able to localize it while running Tez. >> >> Though, I am facing some other problem now. My Tez job is not progressing, >> >> My Tez job doesn't progress, it just keeps printing: >> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer >> 4: 0/1 >> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer >> 4: 0/1 >> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer >> 4: 0/1 >> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer >> 4: 0/1 >> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer >> 4: 0/1 >> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer >> 4: 0/1 >> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer >> 4: 0/1 >> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer >> 4: 0/1 >> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer >> 4: 0/1 >> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer >> 4: 0/1 >> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer >> 4: 0/1 >> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer >> 4: 0/1 >> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer >> 4: 0/1 >> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer >> 4: 0/1 >> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer >> 4: 0/1 >> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer >> 4: 0/1 >> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer >> 4: 0/1 >> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer >> 4: 0/1 >> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer >> 4: 0/1 >> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer >> 4: 0/1 >> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer >> 4: 0/1 >> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer >> 4: 0/1 >> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer >> 4: 0/1 >> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer >> 4: 0/1 >> Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer >> 4: 0/1 >> >> In hive logs also, it keeps on printing the same. >> >> Any idea, where should I look into or how should I debug the problem. Or >> pointers on problem itself would help. >> Thanks Archit. >> >> >> On Sun, May 11, 2014 at 7:28 AM, Bikas Saha wrote: >> >>> Setting execution engine to mr and framework name to yarn = Hive >>> compiles to MR and runs on MR. >>> >>> Setting execution engine to mr and framework name to yarn-tez = Hive >>> compiles to MR and runs on Tez. >>> >>> Setting execution engine to tez = Hive compiles to Tez and runs on Tez. >>> >>> >>> >>> Did you change the execution engine in the same shell that was earlier >>> running with execution engine set to mr? If yes, then try and set the >>> execution engine in hive-site.xml and use a new hive shell. >>> >>> Have you set all required hive configurations? >>> >>> >>> >>> If you can still reproduce the issue, then you should create a jira in >>> the Apache hive project to track the bug. >>> >>> >>> >>> Bikas >>> >>> >>> >>> *From:* Archit Thakur [mailto:archit279tha...@gmail.com] >>> *Sent:* Thursday, May 08, 2014 4:42 AM >>> *To:* user@hive.apache.org; u...@tez.incubator.apache.org >>> *Subject:* Tez HIve problems. >>> >>> >>> >>> Hi, >>> >>> I am facing few problems while using Tez on Hive. >>> >>> I am able to run Tez throu
Re: Tez HIve problems.
Hi Bikas, I was able to resolve this issue, by copying hive-exec.jar to hdfs://user/admin/, It was not able to localize it while running Tez. Though, I am facing some other problem now. My Tez job is not progressing, My Tez job doesn't progress, it just keeps printing: Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 Map 1: 0/2 Map 5: 0/2 Reducer 2: 0/1 Reducer 3: 0/1 Reducer 4: 0/1 In hive logs also, it keeps on printing the same. Any idea, where should I look into or how should I debug the problem. Or pointers on problem itself would help. Thanks Archit. On Sun, May 11, 2014 at 7:28 AM, Bikas Saha wrote: > Setting execution engine to mr and framework name to yarn = Hive compiles > to MR and runs on MR. > > Setting execution engine to mr and framework name to yarn-tez = Hive > compiles to MR and runs on Tez. > > Setting execution engine to tez = Hive compiles to Tez and runs on Tez. > > > > Did you change the execution engine in the same shell that was earlier > running with execution engine set to mr? If yes, then try and set the > execution engine in hive-site.xml and use a new hive shell. > > Have you set all required hive configurations? > > > > If you can still reproduce the issue, then you should create a jira in the > Apache hive project to track the bug. > > > > Bikas > > > > *From:* Archit Thakur [mailto:archit279tha...@gmail.com] > *Sent:* Thursday, May 08, 2014 4:42 AM > *To:* user@hive.apache.org; u...@tez.incubator.apache.org > *Subject:* Tez HIve problems. > > > > Hi, > > I am facing few problems while using Tez on Hive. > > I am able to run Tez through YARN submitting independent MR procesing job. > > Also, when I run SELECT query on hive shell by setting > > *set hive.execution.engine=mr;* > > but setting *mapreduce.framework.name > <http://mapreduce.framework.name>=yarn-tez* > > This launches tez job, I checked by seeing AM's UI of machine. > > But when I *set hive.execution.engine=tez;* > > > > *It gives me *Query ID = > admin_20140508112525_37cf7b55-75ac-408c-b96f-8e40ba2f8e2b > Total jobs = 1 > Launching Job 1 out of 1 > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.tez.TezTask > > *I checked the hive logs:* > > exec.Task (TezTask.java:execute(185)) - Failed to execute tez graph. > java.io.FileNotFoundException: File does not exist: hdfs:/user/admin > at > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1110) > at > org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1102) > at > org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) > at > org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1102) > at > org.apache.hadoop.hive.ql.exec.tez.DagUtils.getDefaultDestDir(DagUti
Re: Issues running Tez & Hadoop 2.2.0
Hi, Issue resolved. There was a need of '*' at the end of the path. Thanks and Regards, Archit Thakur. On Mon, May 5, 2014 at 3:03 PM, Archit Thakur wrote: > Hi, > > I am facing the same issue. Have you resolved it? > > Thanks and Regards, > Archit Thakur. > > > > > > > > On Tue, Apr 29, 2014 at 7:27 AM, Bryan Jeffrey wrote: > >> All, >> >> I attempted to setup Tez per the instructions on the following page: >> http://tez.incubator.apache.org/install.html >> >> After changing my mapreduce framework name to 'yarn-tez', I am getting >> the following error: >> >> [root@viper ~]# /opt/hadoop/latest-hadoop/bin/hadoop job -list >> DEPRECATED: Use of this script to execute mapred command is deprecated. >> Instead use the mapred command for it. >> >> Exception in thread "main" java.io.IOException: Cannot initialize >> Cluster. Please check your configuration for mapreduce.framework.nameand the >> correspond server addresses. >> at >> org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120) >> at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:82) >> at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:75) >> at >> org.apache.hadoop.mapreduce.tools.CLI.createCluster(CLI.java:381) >> at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:247) >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) >> at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1237) >> >> The Tez install guide specifically references using the Hadoop 2.2.0 >> >> Regards, >> >> Bryan Jeffrey >> > >
Re: Issues running Tez & Hadoop 2.2.0
Hi, I am facing the same issue. Have you resolved it? Thanks and Regards, Archit Thakur. On Tue, Apr 29, 2014 at 7:27 AM, Bryan Jeffrey wrote: > All, > > I attempted to setup Tez per the instructions on the following page: > http://tez.incubator.apache.org/install.html > > After changing my mapreduce framework name to 'yarn-tez', I am getting the > following error: > > [root@viper ~]# /opt/hadoop/latest-hadoop/bin/hadoop job -list > DEPRECATED: Use of this script to execute mapred command is deprecated. > Instead use the mapred command for it. > > Exception in thread "main" java.io.IOException: Cannot initialize Cluster. > Please check your configuration for mapreduce.framework.name and the > correspond server addresses. > at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120) > at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:82) > at org.apache.hadoop.mapreduce.Cluster.(Cluster.java:75) > at > org.apache.hadoop.mapreduce.tools.CLI.createCluster(CLI.java:381) > at org.apache.hadoop.mapreduce.tools.CLI.run(CLI.java:247) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at org.apache.hadoop.mapred.JobClient.main(JobClient.java:1237) > > The Tez install guide specifically references using the Hadoop 2.2.0 > > Regards, > > Bryan Jeffrey >
