Re: relationship between a hive query and a tez dag

[Alan Gates]

Yes, that is correct.

Alan.


Xiaoyong Zhu <mailto:xiaoy...@microsoft.com>
January 21, 2015 at 17:51

Thanks Alan and just for double check: if I have the following script:

1)Create table tbl1, tbl2;

2)Select count(*) from tbl1 group by A;

3)Select count(*) from tbl2 groub by B;

4)Create table tb3;

And this script will be executed exactly in 1 >2 > 3 > 4 order even 2 
and 3 do not have relationship with each other and 4 does not have 
relation with all query 1,2,3?


Xiaoyong

*From:*Alan Gates [mailto:ga...@hortonworks.com]
*Sent:* Thursday, December 18, 2014 8:05 AM
*To:* Xiaoyong Zhu
*Cc:* user@hive.apache.org
*Subject:* Re: relationship between a hive query and a tez dag

DDL operations are done from the client, not as part of a Tez DAG.  
There still is no linkage between the DAGs even if they access the 
same tables or somehow depend on each other.  Hive manages making sure 
queries are executed in the order given.  Tez just executed each query.


Alan.


--

Sent with Postbox <http://www.getpostbox.com>


CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or 
entity to which it is addressed and may contain information that is 
confidential, privileged and exempt from disclosure under applicable 
law. If the reader of this message is not the intended recipient, you 
are hereby notified that any printing, copying, dissemination, 
distribution, disclosure or forwarding of this communication is 
strictly prohibited. If you have received this communication in error, 
please contact the sender immediately and delete it from your system. 
Thank You.


Alan Gates <mailto:ga...@hortonworks.com>
December 17, 2014 at 16:04
DDL operations are done from the client, not as part of a Tez DAG.  
There still is no linkage between the DAGs even if they access the 
same tables or somehow depend on each other.  Hive manages making sure 
queries are executed in the order given.  Tez just executed each query.


Alan.


Xiaoyong Zhu <mailto:xiaoy...@microsoft.com>
December 16, 2014 at 15:41

Thanks Alan! Then what if different hive queries have some links (e.g. 
query 1 creates table abc and query 2 reads data from abc) will this 
be in 1 DAG or 2 DAGs? If 2, then is there any link/relatioinship 
between the 2 DAGs?


Xiaoyong

*From:*Alan Gates [mailto:ga...@hortonworks.com]
*Sent:* Wednesday, December 17, 2014 5:27 AM
*To:* user@hive.apache.org
*Subject:* Re: relationship between a hive query and a tez dag

Several.  Each Hive query becomes one Tez DAG.

Alan.


--

Sent with Postbox <http://www.getpostbox.com>


CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or 
entity to which it is addressed and may contain information that is 
confidential, privileged and exempt from disclosure under applicable 
law. If the reader of this message is not the intended recipient, you 
are hereby notified that any printing, copying, dissemination, 
distribution, disclosure or forwarding of this communication is 
strictly prohibited. If you have received this communication in error, 
please contact the sender immediately and delete it from your system. 
Thank You.


Alan Gates <mailto:ga...@hortonworks.com>
December 16, 2014 at 13:26
Several.  Each Hive query becomes one Tez DAG.

Alan.


Xiaoyong Zhu <mailto:xiaoy...@microsoft.com>
December 15, 2014 at 22:55

Hi experts

I know that Hive could be run on Tez in a DAG "format" -- if I have a 
complex Hive script which is consisted of several Hive queries, will 
it be translated to 1 Tez DAG? Or several Tez DAGs?


Thanks!

Xiaoyong



--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

RE: relationship between a hive query and a tez dag

[Xiaoyong Zhu]

Thanks Alan and just for double check: if I have the following script:

1)  Create table tbl1, tbl2;

2)  Select count(*) from tbl1 group by A;

3)  Select count(*) from tbl2 groub by B;

4)  Create table tb3;

And this script will be executed exactly in 1 >2 > 3 > 4 order even 2 and 3 do 
not have relationship with each other and 4 does not have relation with all 
query 1,2,3?

Xiaoyong

From: Alan Gates [mailto:ga...@hortonworks.com]
Sent: Thursday, December 18, 2014 8:05 AM
To: Xiaoyong Zhu
Cc: user@hive.apache.org
Subject: Re: relationship between a hive query and a tez dag

DDL operations are done from the client, not as part of a Tez DAG.  There still 
is no linkage between the DAGs even if they access the same tables or somehow 
depend on each other.  Hive manages making sure queries are executed in the 
order given.  Tez just executed each query.

Alan.


[cid:image001.jpg@01D03628.5360D0F0]
Xiaoyong Zhu<mailto:xiaoy...@microsoft.com>
December 16, 2014 at 15:41
Thanks Alan! Then what if different hive queries have some links (e.g. query 1 
creates table abc and query 2 reads data from abc) will this be in 1 DAG or 2 
DAGs? If 2, then is there any link/relatioinship between the 2 DAGs?

Xiaoyong

From: Alan Gates [mailto:ga...@hortonworks.com]
Sent: Wednesday, December 17, 2014 5:27 AM
To: user@hive.apache.org<mailto:user@hive.apache.org>
Subject: Re: relationship between a hive query and a tez dag

Several.  Each Hive query becomes one Tez DAG.

Alan.




--
Sent with Postbox<http://www.getpostbox.com>

CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader of 
this message is not the intended recipient, you are hereby notified that any 
printing, copying, dissemination, distribution, disclosure or forwarding of 
this communication is strictly prohibited. If you have received this 
communication in error, please contact the sender immediately and delete it 
from your system. Thank You.
[cid:image002.jpg@01D03628.5360D0F0]
Alan Gates<mailto:ga...@hortonworks.com>
December 16, 2014 at 13:26
Several.  Each Hive query becomes one Tez DAG.

Alan.


--
Sent with Postbox<http://www.getpostbox.com>

CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader of 
this message is not the intended recipient, you are hereby notified that any 
printing, copying, dissemination, distribution, disclosure or forwarding of 
this communication is strictly prohibited. If you have received this 
communication in error, please contact the sender immediately and delete it 
from your system. Thank You.

RE: relationship between a hive query and a tez dag

[Xiaoyong Zhu]

Sorry to update this again - but why don't we do a cross query optimization and 
make the query into 1 DAG (if all the queries in a certain script are linked 
with each other).. this seems a more optimized way..

Xiaoyong

From: Xiaoyong Zhu [mailto:xiaoy...@microsoft.com]
Sent: Thursday, December 18, 2014 8:08 AM
To: Alan Gates
Cc: user@hive.apache.org
Subject: RE: relationship between a hive query and a tez dag

I see, thanks for the explaination!

Xiaoyong

From: Alan Gates [mailto:ga...@hortonworks.com]
Sent: Thursday, December 18, 2014 8:05 AM
To: Xiaoyong Zhu
Cc: user@hive.apache.org<mailto:user@hive.apache.org>
Subject: Re: relationship between a hive query and a tez dag

DDL operations are done from the client, not as part of a Tez DAG.  There still 
is no linkage between the DAGs even if they access the same tables or somehow 
depend on each other.  Hive manages making sure queries are executed in the 
order given.  Tez just executed each query.

Alan.

[cid:image001.jpg@01D01B99.9B4DD560]
Xiaoyong Zhu<mailto:xiaoy...@microsoft.com>
December 16, 2014 at 15:41
Thanks Alan! Then what if different hive queries have some links (e.g. query 1 
creates table abc and query 2 reads data from abc) will this be in 1 DAG or 2 
DAGs? If 2, then is there any link/relatioinship between the 2 DAGs?

Xiaoyong

From: Alan Gates [mailto:ga...@hortonworks.com]
Sent: Wednesday, December 17, 2014 5:27 AM
To: user@hive.apache.org<mailto:user@hive.apache.org>
Subject: Re: relationship between a hive query and a tez dag

Several.  Each Hive query becomes one Tez DAG.

Alan.



--
Sent with Postbox<http://www.getpostbox.com>

CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader of 
this message is not the intended recipient, you are hereby notified that any 
printing, copying, dissemination, distribution, disclosure or forwarding of 
this communication is strictly prohibited. If you have received this 
communication in error, please contact the sender immediately and delete it 
from your system. Thank You.
[cid:image002.jpg@01D01B99.9B4DD560]
Alan Gates<mailto:ga...@hortonworks.com>
December 16, 2014 at 13:26
Several.  Each Hive query becomes one Tez DAG.

Alan.

--
Sent with Postbox<http://www.getpostbox.com>

CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader of 
this message is not the intended recipient, you are hereby notified that any 
printing, copying, dissemination, distribution, disclosure or forwarding of 
this communication is strictly prohibited. If you have received this 
communication in error, please contact the sender immediately and delete it 
from your system. Thank You.

RE: relationship between a hive query and a tez dag

[Xiaoyong Zhu]

I see, thanks for the explaination!

Xiaoyong

From: Alan Gates [mailto:ga...@hortonworks.com]
Sent: Thursday, December 18, 2014 8:05 AM
To: Xiaoyong Zhu
Cc: user@hive.apache.org
Subject: Re: relationship between a hive query and a tez dag

DDL operations are done from the client, not as part of a Tez DAG.  There still 
is no linkage between the DAGs even if they access the same tables or somehow 
depend on each other.  Hive manages making sure queries are executed in the 
order given.  Tez just executed each query.

Alan.


[cid:image001.jpg@01D01A99.C2B8BDF0]
Xiaoyong Zhu<mailto:xiaoy...@microsoft.com>
December 16, 2014 at 15:41
Thanks Alan! Then what if different hive queries have some links (e.g. query 1 
creates table abc and query 2 reads data from abc) will this be in 1 DAG or 2 
DAGs? If 2, then is there any link/relatioinship between the 2 DAGs?

Xiaoyong

From: Alan Gates [mailto:ga...@hortonworks.com]
Sent: Wednesday, December 17, 2014 5:27 AM
To: user@hive.apache.org<mailto:user@hive.apache.org>
Subject: Re: relationship between a hive query and a tez dag

Several.  Each Hive query becomes one Tez DAG.

Alan.




--
Sent with Postbox<http://www.getpostbox.com>

CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader of 
this message is not the intended recipient, you are hereby notified that any 
printing, copying, dissemination, distribution, disclosure or forwarding of 
this communication is strictly prohibited. If you have received this 
communication in error, please contact the sender immediately and delete it 
from your system. Thank You.
[cid:image002.jpg@01D01A99.C2B8BDF0]
Alan Gates<mailto:ga...@hortonworks.com>
December 16, 2014 at 13:26
Several.  Each Hive query becomes one Tez DAG.

Alan.


--
Sent with Postbox<http://www.getpostbox.com>

CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader of 
this message is not the intended recipient, you are hereby notified that any 
printing, copying, dissemination, distribution, disclosure or forwarding of 
this communication is strictly prohibited. If you have received this 
communication in error, please contact the sender immediately and delete it 
from your system. Thank You.

Re: relationship between a hive query and a tez dag

[Alan Gates]

DDL operations are done from the client, not as part of a Tez DAG.  
There still is no linkage between the DAGs even if they access the same 
tables or somehow depend on each other.  Hive manages making sure 
queries are executed in the order given.  Tez just executed each query.


Alan.


Xiaoyong Zhu <mailto:xiaoy...@microsoft.com>
December 16, 2014 at 15:41

Thanks Alan! Then what if different hive queries have some links (e.g. 
query 1 creates table abc and query 2 reads data from abc) will this 
be in 1 DAG or 2 DAGs? If 2, then is there any link/relatioinship 
between the 2 DAGs?


Xiaoyong

*From:*Alan Gates [mailto:ga...@hortonworks.com]
*Sent:* Wednesday, December 17, 2014 5:27 AM
*To:* user@hive.apache.org
*Subject:* Re: relationship between a hive query and a tez dag

Several.  Each Hive query becomes one Tez DAG.

Alan.


--

Sent with Postbox <http://www.getpostbox.com>


CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or 
entity to which it is addressed and may contain information that is 
confidential, privileged and exempt from disclosure under applicable 
law. If the reader of this message is not the intended recipient, you 
are hereby notified that any printing, copying, dissemination, 
distribution, disclosure or forwarding of this communication is 
strictly prohibited. If you have received this communication in error, 
please contact the sender immediately and delete it from your system. 
Thank You.


Alan Gates <mailto:ga...@hortonworks.com>
December 16, 2014 at 13:26
Several.  Each Hive query becomes one Tez DAG.

Alan.




--
Sent with Postbox <http://www.getpostbox.com>

--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

RE: relationship between a hive query and a tez dag

[Xiaoyong Zhu]

Thanks Alan! Then what if different hive queries have some links (e.g. query 1 
creates table abc and query 2 reads data from abc) will this be in 1 DAG or 2 
DAGs? If 2, then is there any link/relatioinship between the 2 DAGs?

Xiaoyong

From: Alan Gates [mailto:ga...@hortonworks.com]
Sent: Wednesday, December 17, 2014 5:27 AM
To: user@hive.apache.org
Subject: Re: relationship between a hive query and a tez dag

Several.  Each Hive query becomes one Tez DAG.

Alan.


[cid:image001.jpg@01D019CC.EE7BFBF0]
Xiaoyong Zhu<mailto:xiaoy...@microsoft.com>
December 15, 2014 at 22:55
Hi experts

I know that Hive could be run on Tez in a DAG "format" - if I have a complex 
Hive script which is consisted of several Hive queries, will it be translated 
to 1 Tez DAG? Or several Tez DAGs?

Thanks!

Xiaoyong


--
Sent with Postbox<http://www.getpostbox.com>

CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader of 
this message is not the intended recipient, you are hereby notified that any 
printing, copying, dissemination, distribution, disclosure or forwarding of 
this communication is strictly prohibited. If you have received this 
communication in error, please contact the sender immediately and delete it 
from your system. Thank You.

Re: relationship between a hive query and a tez dag

[Alan Gates]

Several.  Each Hive query becomes one Tez DAG.

Alan.


Xiaoyong Zhu 
December 15, 2014 at 22:55

Hi experts

I know that Hive could be run on Tez in a DAG "format" -- if I have a 
complex Hive script which is consisted of several Hive queries, will 
it be translated to 1 Tez DAG? Or several Tez DAGs?


Thanks!

Xiaoyong



--
Sent with Postbox 

--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

relationship between a hive query and a tez dag

[Xiaoyong Zhu]

Hi experts

I know that Hive could be run on Tez in a DAG "format" - if I have a complex 
Hive script which is consisted of several Hive queries, will it be translated 
to 1 Tez DAG? Or several Tez DAGs?

Thanks!

Xiaoyong