[jira] [Created] (HIVE-15229) 'like any' and 'like all' operators in hive

2016-11-17 Thread Simanchal Das (JIRA) 
Simanchal Das created HIVE-15229:


 Summary: 'like any' and 'like all' operators in hive
 Key: HIVE-15229
 URL: https://issues.apache.org/jira/browse/HIVE-15229
 Project: Hive
  Issue Type: New Feature
  Components: Operators
Reporter: Simanchal Das
Assignee: Simanchal Das
Priority: Minor


In Teradata 'like any' and 'like all' operators are mostly used when we are 
matching a text field with numbers of patterns.

'like any' and 'like all' operator are equivalents of multiple like operator 
like example below.

{noformat}
--like any
select col1 from table1 where col2 like any ('%accountant%', '%accounting%', 
'%retail%', '%bank%', '%insurance%');

--Can be written using multiple like condition 
select col1 from table1 where col2 like '%accountant%' or col2 like 
'%accounting%' or col2 like '%retail%' or col2 like '%bank%' or col2 like 
'%insurance%' ;

--like all
select col1 from table1 where col2 like all ('%accountant%', '%accounting%', 
'%retail%', '%bank%', '%insurance%');

--Can be written using multiple like operator 
select col1 from table1 where col2 like '%accountant%' and col2 like 
'%accounting%' and col2 like '%retail%' and col2 like '%bank%' and col2 like 
'%insurance%' ;
{noformat}

Problem statement:

Now a days so many data warehouse projects are being migrated from Teradata to 
Hive.
Always Data engineer and Business analyst are searching for these two operator.
If we introduce these two operator in hive then so many scripts will be 
migrated smoothly instead of converting these operators to multiple like 
operators.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 53845: 'like any' and 'like all' operators in hive

2016-11-17 Thread Simanchal Das 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53845/
---

(Updated Nov. 17, 2016, 12:49 p.m.)


Review request for hive and Carl Steinbach.


Repository: hive-git


Description
---

https://issues.apache.org/jira/browse/HIVE-15229


In Teradata 'like any' and 'like all' operators are mostly used when we are 
matching a text field with numbers of patterns.
'like any' and 'like all' operator are equivalents of multiple like operator 
like example below.
--like any
select col1 from table1 where col2 like any ('%accountant%', '%accounting%', 
'%retail%', '%bank%', '%insurance%');

--Can be written using multiple like condition 
select col1 from table1 where col2 like '%accountant%' or col2 like 
'%accounting%' or col2 like '%retail%' or col2 like '%bank%' or col2 like 
'%insurance%' ;

--like all
select col1 from table1 where col2 like all ('%accountant%', '%accounting%', 
'%retail%', '%bank%', '%insurance%');

--Can be written using multiple like operator 
select col1 from table1 where col2 like '%accountant%' and col2 like 
'%accounting%' and col2 like '%retail%' and col2 like '%bank%' and col2 like 
'%insurance%' ;

Problem statement:

Now a days so many data warehouse projects are being migrated from Teradata to 
Hive.
Always Data engineer and Business analyst are searching for these two operator.
If we introduce these two operator in hive then so many scripts will be 
migrated smoothly instead of converting these operators to multiple like 
operators.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 0dbbc1d 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g 4357328 
  ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 55915a6 
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g a82083b 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 5e708d3 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLikeAll.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFLikeAny.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFLikeAll.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFLikeAny.java 
PRE-CREATION 
  ql/src/test/queries/clientnegative/udf_likeall_wrong1.q PRE-CREATION 
  ql/src/test/queries/clientnegative/udf_likeany_wrong1.q PRE-CREATION 
  ql/src/test/queries/clientpositive/udf_likeall.q PRE-CREATION 
  ql/src/test/queries/clientpositive/udf_likeany.q PRE-CREATION 
  ql/src/test/results/clientnegative/udf_likeall_wrong1.q.out PRE-CREATION 
  ql/src/test/results/clientnegative/udf_likeany_wrong1.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/udf_likeall.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/udf_likeany.q.out PRE-CREATION 

Diff: https://reviews.apache.org/r/53845/diff/


Testing
---

Junit test cases and query.q files are attached


Thanks,

Simanchal Das

[hive/ql] need your review and suggestions for HIVE-15221

2016-11-17 Thread Hui Fei 
hi, all

here is my report and patches.

https://issues.apache.org/jira/browse/HIVE-15221

any suggestions?

[jira] [Created] (HIVE-15231) query on view results fails with table not found error if view is created with subquery alias (CTE).

2016-11-17 Thread Aihua Xu (JIRA) 
Aihua Xu created HIVE-15231:
---

 Summary: query on view results fails with table not found error if 
view is created with subquery alias (CTE).
 Key: HIVE-15231
 URL: https://issues.apache.org/jira/browse/HIVE-15231
 Project: Hive
  Issue Type: Bug
  Components: Query Planning
Affects Versions: 1.3.0
Reporter: Aihua Xu
Assignee: Aihua Xu


HIVE-10698 fixed one issue of the query on view with CTE, but it seems to break 
another case if a alias is given for the CTE.

use bugtest;
create table basetb(id int, name string);
create view testv1 as
with subtb as (select id, name from bugtest.basetb)
select id from subtb a;
use castest;
explain select * from bugtest.testv1;
hive> explain select * from bugtest.testv1;

FAILED: SemanticException Line 2:21 Table not found 'subtb' in definition of 
VIEW testv1 [
with subtb as (select `basetb`.`id`, `basetb`.`name` from `bugtest`.`basetb`)
select `a`.`id` from `bugtest`.`subtb` `a`
] used as testv1 at Line 1:14




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15232) Add notification events for functions and indexes

2016-11-17 Thread Mohit Sabharwal (JIRA) 
Mohit Sabharwal created HIVE-15232:
--

 Summary: Add notification events for functions and indexes
 Key: HIVE-15232
 URL: https://issues.apache.org/jira/browse/HIVE-15232
 Project: Hive
  Issue Type: Improvement
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal


Create/Drop Function and Create/Drop/Alter Index should also generate metastore 
notification events.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 53838: HIVE-15232: Add notification events for functions and indexes

2016-11-17 Thread Mohit Sabharwal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53838/
---

Review request for hive.


Bugs: HIVE-15232
https://issues.apache.org/jira/browse/HIVE-15232


Repository: hive-git


Description
---

Adds notification events for Create/Drop Function and Create/Drop/Alter Index.


Diffs
-

  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatConstants.java 
72930eb66c2b44fc6ee414c40ead713eb83413c1 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/DbNotificationListener.java
 0b3d89198f1d89c057d2639b89799b53b4a09163 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/AlterIndexMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/CreateFunctionMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/CreateIndexMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/DropFunctionMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/DropIndexMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/HCatEventMessage.java
 538fa68a83081604ca6598d886021a633dfecea7 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/MessageDeserializer.java
 8ea39987f022175f0bcb6e1ecd0945847fd70996 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/MessageFactory.java
 0710dd09dbafb11ce3a4819bc2bd5f50bf70dc84 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONAlterIndexMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONCreateFunctionMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONCreateIndexMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONDropFunctionMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONDropIndexMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONMessageDeserializer.java
 834fdde113a293fcc595c8bfe329a9299ae8aabc 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONMessageFactory.java
 6b74b54b5d8f3ef8735b8bae5535567a7c4dc76a 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 4a7801bbf010dbcfa2044381cfd2e913f83f5445 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java
 1cd32d5859899960337c9525ac033cd1cfb64ab2 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
c0ef25e103d0b7cbb187cea2111efc6a40e01d85 
  
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java 
5e46ae1d83296d11957038c76ee2e4d207db8992 
  
metastore/src/java/org/apache/hadoop/hive/metastore/events/CreateFunctionEvent.java
 PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/events/DropFunctionEvent.java
 PRE-CREATION 
  metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java 
a3b16d049c9c06f08e5ef8809ad386274479d66d 

Diff: https://reviews.apache.org/r/53838/diff/


Testing
---

Verified using TestDbNotificationListener


Thanks,

Mohit Sabharwal

[jira] [Created] (HIVE-15233) UDF UUID() should be non-deterministic

2016-11-17 Thread Chao Sun (JIRA) 
Chao Sun created HIVE-15233:
---

 Summary: UDF UUID() should be non-deterministic
 Key: HIVE-15233
 URL: https://issues.apache.org/jira/browse/HIVE-15233
 Project: Hive
  Issue Type: Bug
  Components: UDF
Affects Versions: 2.2.0
Reporter: Chao Sun
Assignee: Chao Sun


The UUID() function should be non-deterministic.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15234) Semijoin cardinality estimation can be improved

2016-11-17 Thread Ashutosh Chauhan (JIRA) 
Ashutosh Chauhan created HIVE-15234:
---

 Summary: Semijoin cardinality estimation can be improved
 Key: HIVE-15234
 URL: https://issues.apache.org/jira/browse/HIVE-15234
 Project: Hive
  Issue Type: Bug
  Components: CBO, Logical Optimizer
Affects Versions: 2.1.0, 2.0.0
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan


Currently calcite optimization rules rely on (Hive)SemiJoin to represent semi 
join node, whereas Stats estimate use {{leftSemiJoin}} field of Join to 
estimate stats. As a result semi-join specific stats calculation logic is never 
hit since at plan generation time HiveSemiJoin is created and leftSemiJoin 
field of Join is never set.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 53838: HIVE-15232: Add notification events for functions and indexes

2016-11-17 Thread Mohit Sabharwal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53838/
---

(Updated Nov. 17, 2016, 8:48 p.m.)


Review request for hive.


Changes
---

Fixed test failure.


Bugs: HIVE-15232
https://issues.apache.org/jira/browse/HIVE-15232


Repository: hive-git


Description
---

Adds notification events for Create/Drop Function and Create/Drop/Alter Index.


Diffs (updated)
-

  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatConstants.java 
72930eb66c2b44fc6ee414c40ead713eb83413c1 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/DbNotificationListener.java
 0b3d89198f1d89c057d2639b89799b53b4a09163 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/AlterIndexMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/CreateFunctionMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/CreateIndexMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/DropFunctionMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/DropIndexMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/HCatEventMessage.java
 538fa68a83081604ca6598d886021a633dfecea7 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/MessageDeserializer.java
 8ea39987f022175f0bcb6e1ecd0945847fd70996 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/MessageFactory.java
 0710dd09dbafb11ce3a4819bc2bd5f50bf70dc84 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONAlterIndexMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONCreateFunctionMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONCreateIndexMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONDropFunctionMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONDropIndexMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONMessageDeserializer.java
 834fdde113a293fcc595c8bfe329a9299ae8aabc 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONMessageFactory.java
 6b74b54b5d8f3ef8735b8bae5535567a7c4dc76a 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 4a7801bbf010dbcfa2044381cfd2e913f83f5445 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java
 1cd32d5859899960337c9525ac033cd1cfb64ab2 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
c0ef25e103d0b7cbb187cea2111efc6a40e01d85 
  
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java 
5e46ae1d83296d11957038c76ee2e4d207db8992 
  
metastore/src/java/org/apache/hadoop/hive/metastore/events/CreateFunctionEvent.java
 PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/events/DropFunctionEvent.java
 PRE-CREATION 
  metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java 
a3b16d049c9c06f08e5ef8809ad386274479d66d 

Diff: https://reviews.apache.org/r/53838/diff/


Testing
---

Verified using TestDbNotificationListener


Thanks,

Mohit Sabharwal

Re: Review Request 53838: HIVE-15232: Add notification events for functions and indexes

2016-11-17 Thread Chaoyu Tang 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53838/#review156239
---


Fix it, then Ship it!





hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/DbNotificationListener.java
 (line 296)


Nit: remove empty spaces


- Chaoyu Tang


On Nov. 17, 2016, 8:48 p.m., Mohit Sabharwal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/53838/
> ---
> 
> (Updated Nov. 17, 2016, 8:48 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-15232
> https://issues.apache.org/jira/browse/HIVE-15232
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Adds notification events for Create/Drop Function and Create/Drop/Alter Index.
> 
> 
> Diffs
> -
> 
>   
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatConstants.java
>  72930eb66c2b44fc6ee414c40ead713eb83413c1 
>   
> hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/DbNotificationListener.java
>  0b3d89198f1d89c057d2639b89799b53b4a09163 
>   
> hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/AlterIndexMessage.java
>  PRE-CREATION 
>   
> hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/CreateFunctionMessage.java
>  PRE-CREATION 
>   
> hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/CreateIndexMessage.java
>  PRE-CREATION 
>   
> hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/DropFunctionMessage.java
>  PRE-CREATION 
>   
> hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/DropIndexMessage.java
>  PRE-CREATION 
>   
> hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/HCatEventMessage.java
>  538fa68a83081604ca6598d886021a633dfecea7 
>   
> hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/MessageDeserializer.java
>  8ea39987f022175f0bcb6e1ecd0945847fd70996 
>   
> hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/MessageFactory.java
>  0710dd09dbafb11ce3a4819bc2bd5f50bf70dc84 
>   
> hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONAlterIndexMessage.java
>  PRE-CREATION 
>   
> hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONCreateFunctionMessage.java
>  PRE-CREATION 
>   
> hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONCreateIndexMessage.java
>  PRE-CREATION 
>   
> hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONDropFunctionMessage.java
>  PRE-CREATION 
>   
> hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONDropIndexMessage.java
>  PRE-CREATION 
>   
> hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONMessageDeserializer.java
>  834fdde113a293fcc595c8bfe329a9299ae8aabc 
>   
> hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONMessageFactory.java
>  6b74b54b5d8f3ef8735b8bae5535567a7c4dc76a 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
>  4a7801bbf010dbcfa2044381cfd2e913f83f5445 
>   
> itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java
>  1cd32d5859899960337c9525ac033cd1cfb64ab2 
>   metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
> c0ef25e103d0b7cbb187cea2111efc6a40e01d85 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java
>  5e46ae1d83296d11957038c76ee2e4d207db8992 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/events/CreateFunctionEvent.java
>  PRE-CREATION 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/events/DropFunctionEvent.java
>  PRE-CREATION 
>   metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java 
> a3b16d049c9c06f08e5ef8809ad386274479d66d 
> 
> Diff: https://reviews.apache.org/r/53838/diff/
> 
> 
> Testing
> ---
> 
> Verified using TestDbNotificationListener
> 
> 
> Thanks,
> 
> Mohit Sabharwal
> 
>

Re: Review Request 53838: HIVE-15232: Add notification events for functions and indexes

2016-11-17 Thread Mohit Sabharwal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/53838/
---

(Updated Nov. 17, 2016, 9:26 p.m.)


Review request for hive.


Changes
---

Incorp. review feedback.


Bugs: HIVE-15232
https://issues.apache.org/jira/browse/HIVE-15232


Repository: hive-git


Description
---

Adds notification events for Create/Drop Function and Create/Drop/Alter Index.


Diffs (updated)
-

  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/common/HCatConstants.java 
72930eb66c2b44fc6ee414c40ead713eb83413c1 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/listener/DbNotificationListener.java
 0b3d89198f1d89c057d2639b89799b53b4a09163 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/AlterIndexMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/CreateFunctionMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/CreateIndexMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/DropFunctionMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/DropIndexMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/HCatEventMessage.java
 538fa68a83081604ca6598d886021a633dfecea7 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/MessageDeserializer.java
 8ea39987f022175f0bcb6e1ecd0945847fd70996 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/MessageFactory.java
 0710dd09dbafb11ce3a4819bc2bd5f50bf70dc84 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONAlterIndexMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONCreateFunctionMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONCreateIndexMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONDropFunctionMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONDropIndexMessage.java
 PRE-CREATION 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONMessageDeserializer.java
 834fdde113a293fcc595c8bfe329a9299ae8aabc 
  
hcatalog/server-extensions/src/main/java/org/apache/hive/hcatalog/messaging/json/JSONMessageFactory.java
 6b74b54b5d8f3ef8735b8bae5535567a7c4dc76a 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/DummyRawStoreFailEvent.java
 4a7801bbf010dbcfa2044381cfd2e913f83f5445 
  
itests/hcatalog-unit/src/test/java/org/apache/hive/hcatalog/listener/TestDbNotificationListener.java
 1cd32d5859899960337c9525ac033cd1cfb64ab2 
  metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 
c0ef25e103d0b7cbb187cea2111efc6a40e01d85 
  
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java 
5e46ae1d83296d11957038c76ee2e4d207db8992 
  
metastore/src/java/org/apache/hadoop/hive/metastore/events/CreateFunctionEvent.java
 PRE-CREATION 
  
metastore/src/java/org/apache/hadoop/hive/metastore/events/DropFunctionEvent.java
 PRE-CREATION 
  metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java 
a3b16d049c9c06f08e5ef8809ad386274479d66d 

Diff: https://reviews.apache.org/r/53838/diff/


Testing
---

Verified using TestDbNotificationListener


Thanks,

Mohit Sabharwal

[GitHub] hive pull request #115: HIVE-15057: Support other types of operators (other ...

2016-11-17 Thread sunchao 
GitHub user sunchao opened a pull request:

https://github.com/apache/hive/pull/115

HIVE-15057: Support other types of operators (other than SELECT)

Working In Progress. NOT ready for merge yet. Posted here for easier review.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sunchao/hive column-pruning

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/115.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #115


commit 4621f6d15f7fa1274a2f53de8890a3f09210ff95
Author: Chao Sun 
Date:   2016-10-25T22:49:22Z

HIVE-15057: Support other types of operators (other than SELECT)




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (HIVE-15235) sortmergejoin can produce incorrect plan wrt dummy operators

2016-11-17 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-15235:
---

 Summary: sortmergejoin can produce incorrect plan wrt dummy 
operators
 Key: HIVE-15235
 URL: https://issues.apache.org/jira/browse/HIVE-15235
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


{noformat}
Id =13
  
Id =12
  
Id =10
  
Id =11
  
  <\Children>
  Id = 10 null<\Parent>
<\FS>
  <\Children>
  Id = 12 nullId = 9 
Id =9
  null
  <\Children>
  <\Parent>
<\HASHTABLEDUMMY><\Parent>
<\MAPJOIN>
  <\Children>
  Id = 13 null<\Parent>
<\SEL>
  <\Children>
{noformat}

This only happens when sortmergejoin is enabled.
This is on reduce size of a Tez job; during initialization, dummy operators are 
not initialized (presumably, they are not present in the work); that results in 
MapJoin not being initialized, even though its proper parent is.
Manifests as an NPE 
{noformat}
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:350)
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15236) timestamp and date comparison should happen in timestamp

2016-11-17 Thread Ashutosh Chauhan (JIRA) 
Ashutosh Chauhan created HIVE-15236:
---

 Summary: timestamp and date comparison should happen in timestamp
 Key: HIVE-15236
 URL: https://issues.apache.org/jira/browse/HIVE-15236
 Project: Hive
  Issue Type: Bug
  Components: Query Planning
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan


Currently it happens in string, which results in incorrect result.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15237) Propagate Spark job failure to Hive

2016-11-17 Thread Xuefu Zhang (JIRA) 
Xuefu Zhang created HIVE-15237:
--

 Summary: Propagate Spark job failure to Hive
 Key: HIVE-15237
 URL: https://issues.apache.org/jira/browse/HIVE-15237
 Project: Hive
  Issue Type: Bug
  Components: Spark
Affects Versions: 2.1.0
Reporter: Xuefu Zhang


If a Spark job failed for some reason, Hive doesn't get any additional error 
message, which makes it very hard for user to figure out why. Here is an 
example:
{code}
Status: Running (Hive on Spark job[0])
Job Progress Format
CurrentTime StageId_StageAttemptId: 
SucceededTasksCount(+RunningTasksCount-FailedTasksCount)/TotalTasksCount 
[StageCost]
2016-11-17 21:32:53,134 Stage-0_0: 0/23 Stage-1_0: 0/28 
2016-11-17 21:32:55,156 Stage-0_0: 0(+1)/23 Stage-1_0: 0/28 
2016-11-17 21:32:57,167 Stage-0_0: 0(+3)/23 Stage-1_0: 0/28 
2016-11-17 21:33:00,216 Stage-0_0: 0(+3)/23 Stage-1_0: 0/28 
2016-11-17 21:33:03,251 Stage-0_0: 0(+3)/23 Stage-1_0: 0/28 
2016-11-17 21:33:06,286 Stage-0_0: 0(+4)/23 Stage-1_0: 0/28 
2016-11-17 21:33:09,308 Stage-0_0: 0(+2,-3)/23  Stage-1_0: 0/28 
2016-11-17 21:33:12,332 Stage-0_0: 0(+2,-3)/23  Stage-1_0: 0/28 
2016-11-17 21:33:13,338 Stage-0_0: 0(+21,-3)/23 Stage-1_0: 0/28 
2016-11-17 21:33:15,349 Stage-0_0: 0(+21,-5)/23 Stage-1_0: 0/28 
2016-11-17 21:33:16,358 Stage-0_0: 0(+18,-8)/23 Stage-1_0: 0/28 
2016-11-17 21:33:19,373 Stage-0_0: 0(+21,-8)/23 Stage-1_0: 0/28 
2016-11-17 21:33:22,400 Stage-0_0: 0(+18,-14)/23Stage-1_0: 0/28 
2016-11-17 21:33:23,404 Stage-0_0: 0(+15,-20)/23Stage-1_0: 0/28 
2016-11-17 21:33:24,408 Stage-0_0: 0(+12,-23)/23Stage-1_0: 0/28 
2016-11-17 21:33:25,417 Stage-0_0: 0(+9,-26)/23 Stage-1_0: 0/28 
2016-11-17 21:33:26,420 Stage-0_0: 0(+12,-26)/23Stage-1_0: 0/28 
2016-11-17 21:33:28,427 Stage-0_0: 0(+9,-29)/23 Stage-1_0: 0/28 
2016-11-17 21:33:29,432 Stage-0_0: 0(+12,-29)/23Stage-1_0: 0/28 
2016-11-17 21:33:31,444 Stage-0_0: 0(+18,-29)/23Stage-1_0: 0/28 
2016-11-17 21:33:34,464 Stage-0_0: 0(+18,-29)/23Stage-1_0: 0/28 
Status: Failed
FAILED: Execution Error, return code 3 from 
org.apache.hadoop.hive.ql.exec.spark.SparkTask
{code}
It would be better if we can propagate Spark error to Hive.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15238) Make changes to ReplicationSemanticAnalyzer to dump and load events stored in metastore

2016-11-17 Thread Vaibhav Gumashta (JIRA) 
Vaibhav Gumashta created HIVE-15238:
---

 Summary: Make changes to ReplicationSemanticAnalyzer to dump and 
load events stored in metastore
 Key: HIVE-15238
 URL: https://issues.apache.org/jira/browse/HIVE-15238
 Project: Hive
  Issue Type: Sub-task
  Components: repl
Reporter: Vaibhav Gumashta
Assignee: Vaibhav Gumashta






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15239) hive on spark combine equivalentwork get wrong result because of tablescan operaton compare

2016-11-17 Thread wangwenli (JIRA) 
wangwenli created HIVE-15239:


 Summary: hive on spark combine equivalentwork get wrong result 
because of  tablescan operaton compare
 Key: HIVE-15239
 URL: https://issues.apache.org/jira/browse/HIVE-15239
 Project: Hive
  Issue Type: Bug
  Components: Spark
Affects Versions: 2.1.0, 1.2.0
Reporter: wangwenli


env: hive on spark engine
reproduce step:

create table a1(KEHHAO string, START_DT string) partitioned by (END_DT string);
create table a2(KEHHAO string, START_DT string) partitioned by (END_DT string);

alter table a1 add partition(END_DT='20161020');
alter table a1 add partition(END_DT='20161021');

insert into table a1 partition(END_DT='20161020') 
values('2000721360','20161001');


SELECT T1.KEHHAO,COUNT(1) FROM ( 
SELECT KEHHAO FROM a1 T 
WHERE T.KEHHAO = '2000721360' AND '20161018' BETWEEN T.START_DT AND T.END_DT-1 
UNION ALL 
SELECT KEHHAO FROM a2 T
WHERE T.KEHHAO = '2000721360' AND '20161018' BETWEEN T.START_DT AND T.END_DT-1 
) T1 
GROUP BY T1.KEHHAO 
HAVING COUNT(1)>1; 

+-+--+--+
|  t1.kehhao  | _c1  |
+-+--+--+
| 2000721360  | 2|
+-+--+--+


the result should be none record



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15240) Updating/Altering stats in metastore can be expensive in S3

2016-11-17 Thread Rajesh Balamohan (JIRA) 
Rajesh Balamohan created HIVE-15240:
---

 Summary: Updating/Altering stats in metastore can be expensive in 
S3
 Key: HIVE-15240
 URL: https://issues.apache.org/jira/browse/HIVE-15240
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Reporter: Rajesh Balamohan
Priority: Minor


https://github.com/apache/hive/blob/master/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java#L630

https://github.com/apache/hive/blob/master/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java#L367

If there are 100 partitions, it iterates every partition to determine its 
location taking up more than good amount of time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15241) Update/Alter stats in metastore can be expensive in S3

2016-11-17 Thread Rajesh Balamohan (JIRA) 
Rajesh Balamohan created HIVE-15241:
---

 Summary: Update/Alter stats in metastore can be expensive in S3
 Key: HIVE-15241
 URL: https://issues.apache.org/jira/browse/HIVE-15241
 Project: Hive
  Issue Type: Improvement
  Components: Metastore
Reporter: Rajesh Balamohan
Priority: Minor


https://github.com/apache/hive/blob/master/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java#L630

Checks every partition's file stat; Having even 100 partitions ended up taking 
large amount of time. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] hive pull request #104: Implement Parquet vectorization reader for Primitive...

2016-11-17 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/hive/pull/104


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---