Re: Getting dot files for DAGs

[James Pirz]

Thanks for suggesting, I never used Tez UI before, and learned about it
yesterday.
I am trying to find out how I can enable/use it. Apparently it needs some
changes in the binary that I am using (I had built the binary for tez 0.7
almost 2 months ago).




On Wed, Sep 30, 2015 at 10:27 PM, Jörn Franke  wrote:

> Why not use tez ui?
>
> Le jeu. 1 oct. 2015 à 2:29, James Pirz  a écrit :
>
>> I am using Tez 0.7.0 on Hadopp 2.6 to run Hive queries.
>> I am interested in checking DAGs for my queries visually, and I realized
>> that I can do that by graphviz once I can get "dot" files of my DAGs. My
>> issue is I can not find those files, they are not in the log directory of
>> Yarn or Hadoop or under /tmp .
>>
>> Any hint as where I can find those files would be great. Do I need to add
>> any settings to my tez-site.xml in-order to enable generating them ?
>>
>> Thanks.
>>
>

Re: Getting dot files for DAGs

[Hitesh Shah]

I don’t believe the binary should need changing at all unless you need 
enhancements from recent commits. It should just be setting up the UI and 
configuring Tez for using YARN Timeline.

The instructions that you can follow:
  http://tez.apache.org/tez-ui.html 
  http://tez.apache.org/tez_yarn_timeline.html

thanks
— Hitesh

On Oct 1, 2015, at 11:07 AM, James Pirz  wrote:

> Thanks for suggesting, I never used Tez UI before, and learned about it 
> yesterday.
> I am trying to find out how I can enable/use it. Apparently it needs some 
> changes in the binary that I am using (I had built the binary for tez 0.7 
> almost 2 months ago).
> 
> 
> 
> 
> On Wed, Sep 30, 2015 at 10:27 PM, Jörn Franke  wrote:
> Why not use tez ui?
> 
> Le jeu. 1 oct. 2015 à 2:29, James Pirz  a écrit :
> I am using Tez 0.7.0 on Hadopp 2.6 to run Hive queries.
> I am interested in checking DAGs for my queries visually, and I realized that 
> I can do that by graphviz once I can get "dot" files of my DAGs. My issue is 
> I can not find those files, they are not in the log directory of Yarn or 
> Hadoop or under /tmp .
> 
> Any hint as where I can find those files would be great. Do I need to add any 
> settings to my tez-site.xml in-order to enable generating them ?
> 
> Thanks. 
> 

Re: Getting dot files for DAGs

[Jianfeng (Jeff) Zhang]

Hi James,

It is under the working directory of the yarn container (should be the first 
container which is AM)


Best Regard,
Jeff Zhang


From: James Pirz >
Reply-To: "u...@tez.apache.org" 
>
Date: Thursday, October 1, 2015 at 8:29 AM
To: "u...@tez.apache.org" 
>
Cc: "user@hive.apache.org" 
>
Subject: Getting dot files for DAGs

I am using Tez 0.7.0 on Hadopp 2.6 to run Hive queries.
I am interested in checking DAGs for my queries visually, and I realized that I 
can do that by graphviz once I can get "dot" files of my DAGs. My issue is I 
can not find those files, they are not in the log directory of Yarn or Hadoop 
or under /tmp .

Any hint as where I can find those files would be great. Do I need to add any 
settings to my tez-site.xml in-order to enable generating them ?

Thanks.

Re: Getting dot files for DAGs

[Hitesh Shah]

The .dot file is generated into the Tez Application Master’s container log dir. 
Firstly, you need to figure out the yarn application in which the query/Tez DAG 
ran. Once you have the applicationId, you can use one of these 2 approaches: 

1) Go to the YARN ResourceManager UI, find the application and click through to 
the Application Master logs. The .dot file for the dag should be visible there.
2) Using the application Id ( if the application has completed), get the yarn 
logs using “bin/yarn logs -applicationId ” - once you have the logs, you 
will be able to find the contents of the .dot file within them. This approach 
only works if you have YARN log aggregation enabled.

thanks
— Hitesh


On Sep 30, 2015, at 5:29 PM, James Pirz  wrote:

> I am using Tez 0.7.0 on Hadopp 2.6 to run Hive queries.
> I am interested in checking DAGs for my queries visually, and I realized that 
> I can do that by graphviz once I can get "dot" files of my DAGs. My issue is 
> I can not find those files, they are not in the log directory of Yarn or 
> Hadoop or under /tmp .
> 
> Any hint as where I can find those files would be great. Do I need to add any 
> settings to my tez-site.xml in-order to enable generating them ?
> 
> Thanks. 

Re: Getting dot files for DAGs

[James Pirz]

Thanks. I could locate them in the proper container's log directory and
visualize them.
I was at the wrong node, assuming that they would be available on any of
the node, but they are really dumped in one of the nodes.



On Wed, Sep 30, 2015 at 7:00 PM, Hitesh Shah  wrote:

> The .dot file is generated into the Tez Application Master’s container log
> dir. Firstly, you need to figure out the yarn application in which the
> query/Tez DAG ran. Once you have the applicationId, you can use one of
> these 2 approaches:
>
> 1) Go to the YARN ResourceManager UI, find the application and click
> through to the Application Master logs. The .dot file for the dag should be
> visible there.
> 2) Using the application Id ( if the application has completed), get the
> yarn logs using “bin/yarn logs -applicationId ” - once you have the
> logs, you will be able to find the contents of the .dot file within them.
> This approach only works if you have YARN log aggregation enabled.
>
> thanks
> — Hitesh
>
>
> On Sep 30, 2015, at 5:29 PM, James Pirz  wrote:
>
> > I am using Tez 0.7.0 on Hadopp 2.6 to run Hive queries.
> > I am interested in checking DAGs for my queries visually, and I realized
> that I can do that by graphviz once I can get "dot" files of my DAGs. My
> issue is I can not find those files, they are not in the log directory of
> Yarn or Hadoop or under /tmp .
> >
> > Any hint as where I can find those files would be great. Do I need to
> add any settings to my tez-site.xml in-order to enable generating them ?
> >
> > Thanks.
>
>

Re: Getting dot files for DAGs

[Russell Jurney]

In tryiegggmbgfzbbvtcsvvedq.

On Thursday, October 1, 2015, Jörn Franke  wrote:

> Why not use tez ui?
>
> Le jeu. 1 oct. 2015 à 2:29, James Pirz  > a écrit :
>
>> I am using Tez 0.7.0 on Hadopp 2.6 to run Hive queries.
>> I am interested in checking DAGs for my queries visually, and I realized
>> that I can do that by graphviz once I can get "dot" files of my DAGs. My
>> issue is I can not find those files, they are not in the log directory of
>> Yarn or Hadoop or under /tmp .
>>
>> Any hint as where I can find those files would be great. Do I need to add
>> any settings to my tez-site.xml in-order to enable generating them ?
>>
>> Thanks.
>>
>

-- 
Russell Jurney twitter.com/rjurney russell.jur...@gmail.com datasyndrome.com

Re: Getting dot files for DAGs

[Jörn Franke]

Why not use tez ui?

Le jeu. 1 oct. 2015 à 2:29, James Pirz  a écrit :

> I am using Tez 0.7.0 on Hadopp 2.6 to run Hive queries.
> I am interested in checking DAGs for my queries visually, and I realized
> that I can do that by graphviz once I can get "dot" files of my DAGs. My
> issue is I can not find those files, they are not in the log directory of
> Yarn or Hadoop or under /tmp .
>
> Any hint as where I can find those files would be great. Do I need to add
> any settings to my tez-site.xml in-order to enable generating them ?
>
> Thanks.
>