Re: Running Hive 0.12.0 with Hadoop 2.2

[Thejas Nair]

It is better to first upgrade hive 0.12 to protocol buffer 2.5 first -
https://issues.apache.org/jira/browse/HIVE-5112. This will make it
into hive 0.13 release.
If you want a pre-built hive 0.12 that works with hadoop 2.2.x , you
can download one from hortonworks website -
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.6.0/bk_installing_manually_book/content/rpm-chap13.html
. (that has a few additional patches also, see full list here -
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.6.0/bk_releasenotes_hdp_2.0/content/ch_relnotes-hdp2.0.6.0-hive.html
)

On Thu, Nov 14, 2013 at 8:30 PM, Bill Q  wrote:
> Hi Franks,
> Many thanks for sharing the information.
>
> Many thanks.
>
>
> Bill
>
>
> On Thu, Nov 14, 2013 at 4:31 PM, Frank Davidson 
> wrote:
>>
>> Well, to be honest, it's the first time I've ever set it up, so I'm not
>> entirely sure what constitutes "extra", but I followed the wiki...
>>
>> https://cwiki.apache.org/confluence/display/Hive/GettingStarted
>>
>> ... and didn't seem to have too many problems... There were probably a few
>> settings I had to tinker with, but it seems to be working fine now...
>>
>> Cheers,
>>
>> Frank
>>
>>
>> On Thu, Nov 14, 2013 at 3:57 PM, Bill Q  wrote:
>>>
>>> Hi Frank,
>>> Thanks for your reply. So, you don't have to do any extra work and just
>>> install it as in Hadoop 1.x?
>>>
>>>
>>> On Thursday, November 14, 2013, Frank Davidson wrote:

 Yes, in fact, I just set it up last week...


 On Thu, Nov 14, 2013 at 1:55 PM, Bill Q  wrote:
>
> Anyone knows? Many thanks.
>
>
> On Thursday, November 14, 2013, Bill Q wrote:
>>
>> Hi,
>> Does Hive 0.12 run on Hadoop 2.2? if it does, anything special about
>> installing it or running hive? Will hive start a M/R ApplicationMaster by
>> itself or should I start one for it?
>>
>> Many thanks.
>>
>>
>> Bill
>
>
>
> --
> Many thanks.
>
>
> Bill
>



 --
 Frank Davidson
 Email: ffdavid...@gmail.com
 AIM: davidsonff
>>>
>>>
>>>
>>> --
>>> Many thanks.
>>>
>>>
>>> Bill
>>>
>>
>>
>>
>> --
>> Frank Davidson
>> Email: ffdavid...@gmail.com
>> AIM: davidsonff
>
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

RE: [ANNOUNCE] New Hive PMC Member - Harish Butani

[Eric Hanson (SQL SERVER)]

Yes, Congratulations Harish!

From: Shreepadma Venugopalan [mailto:shreepa...@cloudera.com]
Sent: Friday, November 15, 2013 1:44 PM
To: d...@hive.apache.org
Cc: user@hive.apache.org; rhbut...@apache.org
Subject: Re: [ANNOUNCE] New Hive PMC Member - Harish Butani

Congrats Harish!

On Fri, Nov 15, 2013 at 1:32 PM, Thejas Nair 
mailto:the...@hortonworks.com>> wrote:
Congrats Harish!


On Fri, Nov 15, 2013 at 1:08 AM, Gunther Hagleitner
mailto:ghagleit...@hortonworks.com>> wrote:
> Congratulations Harish! Very cool.
>
>
> On Thu, Nov 14, 2013 at 10:01 PM, Prasad Mujumdar 
> mailto:pras...@cloudera.com>>wrote:
>
>> Congratulations !!
>>
>> thanks
>> Prasad
>>
>>
>>
>> On Thu, Nov 14, 2013 at 5:17 PM, Carl Steinbach 
>> mailto:c...@apache.org>> wrote:
>>
>> > I am pleased to announce that Harish Butani has been elected to the Hive
>> > Project Management Committee. Please join me in congratulating Harish!
>> >
>> > Thanks.
>> >
>> > Carl
>> >
>>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.

Re: [ANNOUNCE] New Hive PMC Member - Harish Butani

[Shreepadma Venugopalan]

Congrats Harish!


On Fri, Nov 15, 2013 at 1:32 PM, Thejas Nair  wrote:

> Congrats Harish!
>
>
> On Fri, Nov 15, 2013 at 1:08 AM, Gunther Hagleitner
>  wrote:
> > Congratulations Harish! Very cool.
> >
> >
> > On Thu, Nov 14, 2013 at 10:01 PM, Prasad Mujumdar  >wrote:
> >
> >> Congratulations !!
> >>
> >> thanks
> >> Prasad
> >>
> >>
> >>
> >> On Thu, Nov 14, 2013 at 5:17 PM, Carl Steinbach  wrote:
> >>
> >> > I am pleased to announce that Harish Butani has been elected to the
> Hive
> >> > Project Management Committee. Please join me in congratulating Harish!
> >> >
> >> > Thanks.
> >> >
> >> > Carl
> >> >
> >>
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>

Re: [ANNOUNCE] New Hive PMC Member - Harish Butani

[Thejas Nair]

Congrats Harish!


On Fri, Nov 15, 2013 at 1:08 AM, Gunther Hagleitner
 wrote:
> Congratulations Harish! Very cool.
>
>
> On Thu, Nov 14, 2013 at 10:01 PM, Prasad Mujumdar wrote:
>
>> Congratulations !!
>>
>> thanks
>> Prasad
>>
>>
>>
>> On Thu, Nov 14, 2013 at 5:17 PM, Carl Steinbach  wrote:
>>
>> > I am pleased to announce that Harish Butani has been elected to the Hive
>> > Project Management Committee. Please join me in congratulating Harish!
>> >
>> > Thanks.
>> >
>> > Carl
>> >
>>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.

handling joins in Hive 0.11.0

[Adrian Popescu]

Hello everyone,

I have two questions on join optimizations in Hive, one of which I 
believe is a bug in the 0.11.0 release.


1. From Hive online documentation I see that multiple map joins can be 
grouped together in one single MapReduce job if the
input tables are joined on the same joining key. From my experiments on 
TPCH I see that multiple map joins can
be grouped inside the same MapReduce job also for the case that all the 
input corresponding to the tables can be fit into
one single mapper. For instance, query 5 joins first nation with region, 
then the result is joined with supplier. From the explain
plan I see that both joins are run in a single MapReduce job as "nested 
map join operators". Explain plan is attached
(please see Stage 21 in "q5_explained_MJ.txt" for the nested MapJoin 
operators).


Is it possible to disable this feature (i.e., to have a MR job for each 
join)? The setting I use to trigger map joins is 
"hive.auto.convert.join=true"



2. In my experiments I also evaluate skewed joins. I enable skew joins 
through "hive.optimize.skewjoin" and I run the same
tpch query 5. The skew join is not actually triggered as the number of 
rows with the same key is less than "hive.skewjoin.key".
Hence, the map join corresponding to the skewed join  is filtered out at 
runtime, but unfortunately all the other stages
are also filtered out. Thus, no result is actually generated. If I 
disable the skew join optimization, the query running only with

common joins returns the result correctly.

I believe this is a bug when the skew join operator is enabled but not 
triggered. Did anyone experienced the same problem with
skew joins on queries of multiple map reduce joins? I attach the explain 
plan. Essentially only stage 6 and 22 are executed.
Everything else is skipped silently with no output result being 
generated, nor error in "hive.log". Similar behaviour is observed

for other TPCH queries.

Many thanks,
Adrian

ABSTRACT SYNTAX TREE:
  (TOK_QUERY (TOK_FROM (TOK_JOIN (TOK_TABREF (TOK_TABNAME customer2) c) 
(TOK_SUBQUERY (TOK_QUERY (TOK_FROM (TOK_JOIN (TOK_TABREF (TOK_TABNAME orders2) 
o) (TOK_SUBQUERY (TOK_QUERY (TOK_FROM (TOK_JOIN (TOK_TABREF (TOK_TABNAME 
lineitem2) l) (TOK_SUBQUERY (TOK_QUERY (TOK_FROM (TOK_JOIN (TOK_TABREF 
(TOK_TABNAME supplier2) s) (TOK_SUBQUERY (TOK_QUERY (TOK_FROM (TOK_JOIN 
(TOK_TABREF (TOK_TABNAME nation2) n) (TOK_TABREF (TOK_TABNAME region2) r) (and 
(= (. (TOK_TABLE_OR_COL n) n_regionkey) (. (TOK_TABLE_OR_COL r) r_regionkey)) 
(= (. (TOK_TABLE_OR_COL r) r_name) 'ASIA' (TOK_INSERT (TOK_DESTINATION 
(TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR (TOK_TABLE_OR_COL n_name)) 
(TOK_SELEXPR (TOK_TABLE_OR_COL n_nationkey) n1) (= (. (TOK_TABLE_OR_COL s) 
s_nationkey) (. (TOK_TABLE_OR_COL n1) n_nationkey (TOK_INSERT 
(TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR 
(TOK_TABLE_OR_COL n_name)) (TOK_SELEXPR (TOK_TABLE_OR_COL s_suppkey)) 
(TOK_SELEXPR (TOK_TABLE_OR_COL s_nationkey) s1) (= (. (TOK_TABLE_OR_COL l) 
l_suppkey) (. (TOK_TABLE_OR_COL s1) s_suppkey (TOK_INSERT (TOK_DESTINATION 
(TOK_DIR TOK_TMP_FILE)) (TOK_SELECT (TOK_SELEXPR (TOK_TABLE_OR_COL n_name)) 
(TOK_SELEXPR (TOK_TABLE_OR_COL l_extendedprice)) (TOK_SELEXPR (TOK_TABLE_OR_COL 
l_discount)) (TOK_SELEXPR (TOK_TABLE_OR_COL l_orderkey)) (TOK_SELEXPR 
(TOK_TABLE_OR_COL s_nationkey) l1) (and (and (= (. (TOK_TABLE_OR_COL l1) 
l_orderkey) (. (TOK_TABLE_OR_COL o) o_orderkey)) (>= (. (TOK_TABLE_OR_COL o) 
o_orderdate) '1994-01-01')) (< (. (TOK_TABLE_OR_COL o) o_orderdate) 
'1995-01-01' (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) 
(TOK_SELECT (TOK_SELEXPR (TOK_TABLE_OR_COL n_name)) (TOK_SELEXPR 
(TOK_TABLE_OR_COL l_extendedprice)) (TOK_SELEXPR (TOK_TABLE_OR_COL l_discount)) 
(TOK_SELEXPR (TOK_TABLE_OR_COL s_nationkey)) (TOK_SELEXPR (TOK_TABLE_OR_COL 
o_custkey) o1) (and (= (. (TOK_TABLE_OR_COL c) c_nationkey) (. 
(TOK_TABLE_OR_COL o1) s_nationkey)) (= (. (TOK_TABLE_OR_COL c) c_custkey) (. 
(TOK_TABLE_OR_COL o1) o_custkey) (TOK_INSERT (TOK_DESTINATION (TOK_TAB 
(TOK_TABNAME q5_local_supplier_volume))) (TOK_SELECT (TOK_SELEXPR 
(TOK_TABLE_OR_COL n_name)) (TOK_SELEXPR (TOK_FUNCTION sum (* (TOK_TABLE_OR_COL 
l_extendedprice) (- 1 (TOK_TABLE_OR_COL l_discount revenue)) (TOK_GROUPBY 
(TOK_TABLE_OR_COL n_name)) (TOK_ORDERBY (TOK_TABSORTCOLNAMEDESC 
(TOK_TABLE_OR_COL revenue)

STAGE DEPENDENCIES:
  Stage-27 is a root stage
  Stage-21 depends on stages: Stage-27
  Stage-19 depends on stages: Stage-21 , consists of Stage-25, Stage-26, Stage-8
  Stage-25 has a backup stage: Stage-8
  Stage-17 depends on stages: Stage-25
  Stage-16 depends on stages: Stage-8, Stage-17, Stage-18 , consists of 
Stage-23, Stage-24, Stage-1
  Stage-23 has a backup stage: Stage-1
  Stage-14 depends on stages: Stage-23
  Stage-22 depends on stages: Stage-1, Stage-14, Stage-15
  Stage-13 depends on stages: Stage-22
  Stage-3 depends on stages: Stage-13
  Stage-4 depends on stages: Sta

Re: Error when issuing select query

[shouvanik.haldar]

Below is the message that I get.


Sent from Samsung Mobile


 Original message 
From: Nitin Pawar
Date:15/11/2013 16:30 (GMT+05:30)
To: "Shouvanik Haldar (Contingent)"
Cc: user@hive.apache.org
Subject: Re: Error when issuing select query

what is the error ?


On Fri, Nov 15, 2013 at 4:21 PM, Shouvanik Haldar (Contingent) 
mailto:shouvanik.hal...@autodesk.com>> wrote:
Hi,


I am facing error while issuing select query on hive tables.

connection-pending 
remote=ec2-xx-xxx-xxx-xxx.us-west-1.compute.amazonaws.com/xx.xxx.xxx.xxx:8020].
 426 millis timeout lef  t.; Host Details : local 
host is: "ip-xxx-xx-xx-xxx/xxx.xx.xx.xxx"; destination host is: 
"ec2-xx-xxx-xxx-xxx.us-west-1.compute.amazonaws.com":8020;

Can you please help.


Thanks,
Shouvanik



--
Nitin Pawar


This message is for the designated recipient only and may contain privileged, 
proprietary, or otherwise confidential information. If you have received it in 
error, please notify the sender immediately and delete the original. Any other 
use of the e-mail by you is prohibited.

Where allowed by local law, electronic communications with Accenture and its 
affiliates, including e-mail and instant messaging (including content), may be 
scanned by our systems for the purposes of information security and assessment 
of internal compliance with Accenture policy.

__

www.accenture.com

Re: Error when issuing select query

[Nitin Pawar]

what is the error ?


On Fri, Nov 15, 2013 at 4:21 PM, Shouvanik Haldar (Contingent) <
shouvanik.hal...@autodesk.com> wrote:

>  Hi,
>
>
>
>
>
> I am facing error while issuing select query on hive tables.
>
>
>
> connection-pending remote=
> ec2-xx-xxx-xxx-xxx.us-west-1.compute.amazonaws.com/xx.xxx.xxx.xxx:8020].
> 426 millis timeout lef  t.; Host Details :
> local host is: "ip-xxx-xx-xx-xxx/xxx.xx.xx.xxx"; destination host is: "
> ec2-xx-xxx-xxx-xxx.us-west-1.compute.amazonaws.com":8020;
>
>
>
> Can you please help.
>
>
>
>
>
> Thanks,
>
> Shouvanik
>



-- 
Nitin Pawar

Re: Create Table in Hive always throwing the error

[Nitin Pawar]

can you share your configuration (hardware wise)?
and whats the hive version are you using?


On Fri, Nov 15, 2013 at 3:11 PM, Valluri, Sathish
wrote:

>  When trying to create table in Hive I am always getting the following
> error always.
>
> Not able to do any hive DDL operations,
>
>
>
> *FAILED: Error in metadata:
> MetaException(message:java.lang.RuntimeException: commitTransaction was
> called but openTransactionCalls = 0. This probably indicates that there are
> unbalanced calls to openTransaction/commitTransaction) *
>
> *FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask *
>
>
>
> Can anyone help or give any idea on the reason for this failure always.
>
>
>
> I have looked into the similar issue reported on Hive
> https://issues.apache.org/jira/browse/HIVE-4996 but this is looks like
> observed intermittently and not always.
>
>
>
>
>
> Regards
>
> Sathish Valluri
>



-- 
Nitin Pawar

Create Table in Hive always throwing the error

[Valluri, Sathish]

When trying to create table in Hive I am always getting the following error 
always.

Not able to do any hive DDL operations,



FAILED: Error in metadata: MetaException(message:java.lang.RuntimeException: 
commitTransaction was called but openTransactionCalls = 0. This probably 
indicates that there are unbalanced calls to openTransaction/commitTransaction)

FAILED: Execution Error, return code 1 from 
org.apache.hadoop.hive.ql.exec.DDLTask



Can anyone help or give any idea on the reason for this failure always.



I have looked into the similar issue reported on Hive 
https://issues.apache.org/jira/browse/HIVE-4996 but this is looks like observed 
intermittently and not always.





Regards

Sathish Valluri

Re: [ANNOUNCE] New Hive PMC Member - Harish Butani

[Gunther Hagleitner]

Congratulations Harish! Very cool.


On Thu, Nov 14, 2013 at 10:01 PM, Prasad Mujumdar wrote:

> Congratulations !!
>
> thanks
> Prasad
>
>
>
> On Thu, Nov 14, 2013 at 5:17 PM, Carl Steinbach  wrote:
>
> > I am pleased to announce that Harish Butani has been elected to the Hive
> > Project Management Committee. Please join me in congratulating Harish!
> >
> > Thanks.
> >
> > Carl
> >
>

-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.