Re: Getting dot files for DAGs
I don’t believe the binary should need changing at all unless you need enhancements from recent commits. It should just be setting up the UI and configuring Tez for using YARN Timeline. The instructions that you can follow: http://tez.apache.org/tez-ui.html http://tez.apache.org/tez_yarn_timeline.html thanks — Hitesh On Oct 1, 2015, at 11:07 AM, James Pirz wrote: > Thanks for suggesting, I never used Tez UI before, and learned about it > yesterday. > I am trying to find out how I can enable/use it. Apparently it needs some > changes in the binary that I am using (I had built the binary for tez 0.7 > almost 2 months ago). > > > > > On Wed, Sep 30, 2015 at 10:27 PM, Jörn Franke wrote: > Why not use tez ui? > > Le jeu. 1 oct. 2015 à 2:29, James Pirz a écrit : > I am using Tez 0.7.0 on Hadopp 2.6 to run Hive queries. > I am interested in checking DAGs for my queries visually, and I realized that > I can do that by graphviz once I can get "dot" files of my DAGs. My issue is > I can not find those files, they are not in the log directory of Yarn or > Hadoop or under /tmp . > > Any hint as where I can find those files would be great. Do I need to add any > settings to my tez-site.xml in-order to enable generating them ? > > Thanks. >
Re: Getting dot files for DAGs
The .dot file is generated into the Tez Application Master’s container log dir. Firstly, you need to figure out the yarn application in which the query/Tez DAG ran. Once you have the applicationId, you can use one of these 2 approaches: 1) Go to the YARN ResourceManager UI, find the application and click through to the Application Master logs. The .dot file for the dag should be visible there. 2) Using the application Id ( if the application has completed), get the yarn logs using “bin/yarn logs -applicationId ” - once you have the logs, you will be able to find the contents of the .dot file within them. This approach only works if you have YARN log aggregation enabled. thanks — Hitesh On Sep 30, 2015, at 5:29 PM, James Pirz wrote: > I am using Tez 0.7.0 on Hadopp 2.6 to run Hive queries. > I am interested in checking DAGs for my queries visually, and I realized that > I can do that by graphviz once I can get "dot" files of my DAGs. My issue is > I can not find those files, they are not in the log directory of Yarn or > Hadoop or under /tmp . > > Any hint as where I can find those files would be great. Do I need to add any > settings to my tez-site.xml in-order to enable generating them ? > > Thanks.
Re: Hive Tez support matrix
From a Tez perspective, there was a major compatibility change between Tez 0.4 and Tez 0.5. However, Tez-0.7.x and Tez-0.6.x are compatible with Tez-0.5.x. I believe Hive 0.13 is compatible only with Tez 0.4. For Hive 0.14 onwards ( including the Hive-1.x. releases ), they should work with anything in the range of Tez versions: 0.5.2 <= x <= 0.7.x . thanks — Hitesh On Jul 7, 2015, at 10:12 AM, Jim Green wrote: > Hi Team, > > Is there any Hive <-> Tez support matrix? > For example, Hive 1.2 should be on Tez which version? > Tez 0.5.3 only supports which versions of Hive? > etc… > > My understanding is that it does not matter which version of Hive and which > version of Tez. > > -- > Thanks, > www.openkb.info > (Open KnowledgeBase for Hadoop/Database/OS/Network/Tool)
Re: fails to alter table concatenate
Move to user@hive. BCC’ed user@tez. — Hitesh On Jun 30, 2015, at 1:44 AM, patcharee wrote: > Hi, > > I am using hive 0.14 + tez 0.5. It fails to alter table concatenate > occasionally (see the exception below). It is strange that it fails from time > to time not predictable. However, it works on mr. > Is there any suggestion/clue? > > hive> alter table 4dim partition(zone=2,z=15,year=2005,month=4) CONCATENATE; > > >VERTICES STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED > > > File MergeFAILED -1 00 -1 0 0 > > > VERTICES: 00/01 [>>--] 0%ELAPSED TIME: > 1435651968.00 s > > > Status: Failed > Vertex failed, vertexName=File Merge, > vertexId=vertex_1435307579867_0041_1_00, diagnostics=[Vertex > vertex_1435307579867_0041_1_00 [File Merge] killed/failed due > to:ROOT_INPUT_INIT_FAILURE, Vertex Input: > [hdfs://service-10-0.local:8020/apps/hive/warehouse/wrf_tables/4dim/zone=2/z=15/year=2005/month=4] > initializer failed, vertex=vertex_1435307579867_0041_1_00 [File Merge], > java.lang.NullPointerException >at > org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:265) >at > org.apache.hadoop.hive.ql.io.CombineHiveInputFormat.getSplits(CombineHiveInputFormat.java:452) >at > org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:441) >at > org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(MRInputHelpers.java:295) >at > org.apache.tez.mapreduce.common.MRInputAMSplitGenerator.initialize(MRInputAMSplitGenerator.java:124) >at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:245) >at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:239) >at java.security.AccessController.doPrivileged(Native Method) >at javax.security.auth.Subject.doAs(Subject.java:415) >at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) >at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:239) >at > org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:226) >at java.util.concurrent.FutureTask.run(FutureTask.java:262) >at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >at java.lang.Thread.run(Thread.java:745) > ] > DAG failed due to vertex failure. failedVertices:1 killedVertices:0 > FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.DDLTask > > BR, > Patcharee
Re: error on hive insert query
Unless you can pinpoint the problem to something Tez specific, hive-specific questions might be better off being asked on user@hive initially as there is a larger group there that understands Hive as compared to the Tez community. FWIW, TezTask error 1 means “something in the Hive layer using Tez or somewhere within Tez itself” had a failure. Without additional logs/stack trace, this is usually impossible to debug/pinpoint where the problem is. thanks — Hitesh On Jun 16, 2015, at 10:19 AM, Sateesh Karuturi wrote: > iam using hive 1.0.0 and tez 0.5.2. when i set hive.execution.engine value in > hive-site.xml to tezselect query works well... but in case of insert getting > error. the query is : > > insert into table tablename values(intvalue,'string value'); > > and the error is : > > FAILED: Execution Error, return code 1 from > org.apache.hadoop.hive.ql.exec.tez.Tez Task > >
Re: Enabling Tez sessions on HiveServer2
BCC’ed user@tez. This question belongs to either the hive user list or the Hortonworks user forums. thanks — Hitesh On Dec 2, 2014, at 1:28 PM, Pala M Muthaia wrote: > Hi, > > I am trying to get Tez sessions enabled with HS2. I start the HiveServer2 > instance with the flag "-hiveconf hive.execution.engine=tez" and then try to > submit multiple queries one after another, as the same user, to the HS2 > instance. > > When i check the YARN UI, i find that each query of mine is launched as a new > YARN application. While the new Tez application is running, the old Tez > applications are still alive. This is different from Tez session in Hive CLI, > where multiple queries are submitted to the same Tez application (if launched > within the Tez session timeout). > > > I followed the config instructions at > http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.2/bk_installing_manually_book/content/rpm-chap-tez-configure_hive_for_tez.html > so far. > > Is there a separate config flag that i need to turn on for Tez sessions on > HS2? How should i enable Tez sessions with HiveServer2. > > > -pala
Re: Cannot submit Tez application
Re-sending again. > BCC’ing dev@tez. > > Re-directing to hive list. > > @Azury Yu, would you mind providing the query details to the hive folks. > Seems like an issue in the DAG plan generation in Hive. > > thanks > ― Hitesh > > On May 10, 2014, at 8:26 PM, AzuryYu wrote: > >> Hi, >> >> I built Tez-0.5 against hadoop-2.4.0, and I've put all jars to the HDFS, >> configured tez-site.xml correctly. >> >> Hive version is 0.13.0, I've set hive.execution.engine to tez in >> hive-site.xml >> >> then I submit the hive query, got the following exception: >> >> 2014-05-11 11:13:08,533 ERROR [main]: exec.Task (TezTask.java:execute(185)) >> - Failed to execute tez graph. >> java.lang.IllegalStateException: Vertex: Reducer 4 already has group input >> with name:Union 3 >> at org.apache.tez.dag.api.Vertex.addGroupInput(Vertex.java:250) >> at org.apache.tez.dag.api.DAG.processEdgesAndGroups(DAG.java:223) >> at org.apache.tez.dag.api.DAG.verify(DAG.java:284) >> at org.apache.tez.dag.api.DAG.createDag(DAG.java:462) >> at org.apache.tez.client.TezSession.submitDAG(TezSession.java:216) >> at org.apache.tez.client.TezSession.submitDAG(TezSession.java:155) >> at org.apache.hadoop.hive.ql.exec.tez.TezTask.submit(TezTask.java:320) >> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:165) >> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153) >> at >> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:85) >> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1503) >> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1270) >> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1088) >> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:911) >> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:901) >> at >> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268) >> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220) >> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:423) >> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:359) >> at >> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:742) >> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686) >> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:606) >> at org.apache.hadoop.util.RunJar.main(RunJar.java:212) >> 2014-05-11 11:13:08,554 ERROR [main]: ql.Driver >> (SessionState.java:printError(545)) - FAILED: Execution Error, return code 1 >> from org.apache.hadoop.hive.ql.exec.tez.TezTask >> > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
