[GitHub] hive pull request #513: fix null string check for C++ using thrift api

[xiasiyu]

GitHub user xiasiyu opened a pull request:

https://github.com/apache/hive/pull/513

fix null string check for C++ using thrift api

thrift generate C++ files, using (std::string& principalName) as 
list_privileges parameters. When it only checks null in java code, null string 
can not be passed by C++ api only if generated source code is modified.
api in generated ThriftHiveMetastore.h is:
void list_privileges(std::vector & _return, const std::string& 
principal_name, const PrincipalType::type principal_type, const HiveObjectRef& 
hiveObject);

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xiasiyu/hive patch-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/513.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #513


commit cd8ea1681c9c08fd7557cc4de98183c528a716d4
Author: Summer Xia 
Date:   2019-01-10T04:06:19Z

fix null string check for C++ using thrift api

thrift generate C++ files, using (std::string& principalName) as 
list_privileges parameters. When it only checks null in java code, null string 
can not be passed by C++ api only if generated source code is modified.
api in generated ThriftHiveMetastore.h is:
void list_privileges(std::vector & _return, const std::string& 
principal_name, const PrincipalType::type principal_type, const HiveObjectRef& 
hiveObject);




---

[GitHub] hive pull request #512: fix null string check for C++ using thrift api

[xiasiyu]

Github user xiasiyu closed the pull request at:

https://github.com/apache/hive/pull/512


---

[GitHub] hive pull request #512: fix null string check for C++ using thrift api

[xiasiyu]

GitHub user xiasiyu opened a pull request:

https://github.com/apache/hive/pull/512

fix null string check for C++ using thrift api

thrift generate C++ files, using (std::string& principalName) as 
list_privileges parameters. When it only checks null in java code, null string 
can not be passed by C++ api only if generated source code is modified.
api in generated ThriftHiveMetastore.h is:
void list_privileges(std::vector & _return, const std::string& 
principal_name, const PrincipalType::type principal_type, const HiveObjectRef& 
hiveObject);

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xiasiyu/hive patch-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/hive/pull/512.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #512


commit 54fa2bf3dae195027b61eb78f7cb499bcd19f0ad
Author: Summer Xia 
Date:   2019-01-10T03:57:51Z

fix null string check for C++ using thrift api

thrift generate C++ files, using (std::string& principalName) as 
list_privileges parameters. When it only checks null in java code, null string 
can not be passed by C++ api only if generated source code is modified.
api in generated ThriftHiveMetastore.h is:
void list_privileges(std::vector & _return, const std::string& 
principal_name, const PrincipalType::type principal_type, const HiveObjectRef& 
hiveObject);




---

Re: Review Request 69672: HIVE-21045: Add total API timing stats and connection pool stats to metrics

[Yongzhi Chen via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69672/#review211805
---




standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PersistenceManagerProvider.java
Line 228 (original), 227 (patched)


This is a little bit different from the old impl funtionally : The old impl 
will return null if there is no custom properties, where new impl will still 
return the provider. The old impl has a kind of sanity check. But if the custom 
properties are not required here, it should be fine.



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/BoneCPDataSourceProvider.java
Lines 101 (patched)


If registry is null, should we give a warning in the log?


- Yongzhi Chen


On Jan. 5, 2019, 12:41 a.m., Karthik Manamcheri wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69672/
> ---
> 
> (Updated Jan. 5, 2019, 12:41 a.m.)
> 
> 
> Review request for hive, Adam Holley, Morio Ramdenbourg, Naveen Gangam, and 
> Vihang Karajgaonkar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-21045: Add total API timing stats and connection pool stats to metrics
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PersistenceManagerProvider.java
>  dfd7abff85 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/BoneCPDataSourceProvider.java
>  7e33c519a8 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/DataSourceProvider.java
>  6dc63fb3bc 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/DataSourceProviderFactory.java
>  5a92e104be 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/DbCPDataSourceProvider.java
>  7fe487b184 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/datasource/HikariCPDataSourceProvider.java
>  8f6ae57e36 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/MetricsConstants.java
>  3b188f83af 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/metrics/PerfLogger.java
>  a2def26fc5 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  2a6290315a 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/datasource/TestDataSourceProviderFactory.java
>  6ae7f50471 
> 
> 
> Diff: https://reviews.apache.org/r/69672/diff/1/
> 
> 
> Testing
> ---
> 
> Manual testing to verify that the new metrics show up for hikaricp, bonecp, 
> and also the total stats. Here are samples of
> 1. [HikariCP json metrics 
> sample](https://gist.github.com/kmanamcheri/48ff2a680e85c7e925a6f95a9384dcef)
> 2. [BoneCP json metrics 
> sample](https://gist.github.com/kmanamcheri/b005f68263a1a1be06b25156a159d975)
> 
> In both the reports note that there are pool gauges (for tracking the 
> connection pool info) and also a timer for total api calls.
> 
> 
> Thanks,
> 
> Karthik Manamcheri
> 
>

Re: [DISCUSS] Consistent Timestamps across Hadoop

[Owen O'Malley]

>From an Apache point of view, we really need to move this document and the
discussion to the Apache wiki and mailing lists.

Did you want to take a first pass at moving it to Hive's wiki?

.. Owen

On Tue, Dec 11, 2018 at 10:40 AM Zoltan Ivanfi  wrote:

> Hi Owen,
>
> Thanks, I think your email contains a great summary of the problems
> tackled in the proposal. I would like highlight two particular topics from
> the discussion that we are having in the comments (details can be read in
> the document
> 
> ):
>
>- It seems that we have agreement on the desired semantics of the more
>explicit SQL types. In particular, I was glad to hear that the TIMESTAMP
>WITH LOCAL TIME ZONE type that is already implemented in Hive is supposed
>to have Instant semantics. (In fact, it already does have Instant
>semantics, but it also has additional time zone information that is unused
>at this moment, and I wasn't sure whether that will be utilized, changing
>the semantics, or whether the semantics will remain and the superflous time
>zone data will be removed.)
>- We are still discussing what is the best course of action to take
>with the plain TIMESTAMP type, which behaved differently in different file
>formats in Hive 2 and was made to behave the same way in a
>compatibility-breaking manner in Hive 3. My take on this type is that it
>has already been used to write huge amounts of data and for this reason we
>should restore its Avro- and Parquet-specific incosistent behaviour
>(possibly controlled by a feature flag), so that legacy data remains
>readable and legacy workarounds remain functional. The new, more explicit
>SQL types will provide a clear migration path away from the messy TIMESTAMP
>type.
>
> All in all, I feel that we are converging towards a common goal and I have
> high hopes that the more explicit timestamp types will have much better
> interoperability and consistency across different Hadoop SQL engines.
>
> Thanks,
>
> Zoltan
>
>
> On Mon, Dec 10, 2018 at 7:54 PM Owen O'Malley 
> wrote:
>
>> Thank you for starting this discussion. Clearly the Hive semantics on
>> timestamp are very messed up, but has been moving in the right direction of
>> becoming more SQL standard compliant. I'm pulling this discussion back to
>> the list rather than the personal GoogleDoc, which isn't very
>> collaborative.
>>
>> I like your breakdown of the semantics:
>>
>>- Instant - point in time that will appear different depending on the
>>reader time zone
>>- LocalDateTime - consistent hour and minute regardless of the reader
>>time zone.
>>- OffsetDateTime - consistent hour and minute with the offset of the
>>writer time zone
>>
>> The SQL standard has:
>>
>>- Timestamp & Timestamp without time zone = LocalDateTime
>>- Timestamp with time zone = OffsetDateTime
>>
>> Hive 2 had very confused semantics for timestamp:
>>
>>- When storage was ORC, text, or RCFile with a text serde it was
>>LocalDateTime
>>- When storage was Avro, Parquet, or RCFile with a binary serde it
>>was Instant
>>
>> Hive 3.1 has moved toward the SQL standard extended with Oracles'
>> timestamp with local time zone:
>>
>>- Timestamp = LocalDateTime
>>- Timestamp with local time zone = Instant
>>
>> This leaves us with a few problems:
>>
>>- The Hive bindings to Parquet and Avro don't handle timestamps
>>correctly.
>>- ORC doesn't support timestamps with local time zone. I start
>>working on it in ORC-189.
>>- We don't have timestamp with time zone support.
>>
>> .. Owen
>>
>> On Thu, Dec 6, 2018 at 7:55 AM Marta Kuczora
>>  wrote:
>>
>>> Hi Hive Community,
>>>
>>> I would like to share the following document on our "Consistent Timestamp
>>> types in Hadoop" plans for review.
>>>
>>> https://docs.google.com/document/d/1gNRww9mZJcHvUDCXklzjFEQGpefsuR_akCDfWsdE35Q/edit
>>>
>>> With this plan we would like to get an agreement on consistent timestamp
>>> behavior on Hive, Spark and Impala and in order to achieve this, we are
>>> sharing this document with all three communities.
>>>
>>> Please review and comment, any feedback is much appreciated!
>>>
>>> Regards,
>>> Marta
>>>
>>

Re: Review Request 69664: HIVE-21077 : Database and Catalogs should have creation time

[Vihang Karajgaonkar via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69664/
---

(Updated Jan. 9, 2019, 6:50 p.m.)


Review request for hive, Karthik Manamcheri, Naveen Gangam, and Peter Vary.


Changes
---

Fixed TestCacheStore test failure


Bugs: HIVE-21077
https://issues.apache.org/jira/browse/HIVE-21077


Repository: hive-git


Description
---

HIVE-21077 : Database and Catalogs should have creation time


Diffs (updated)
-

  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Catalog.java
 3eb4dbd51110dd6e5d04c3bdacde2e5bdba09a7c 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java
 994797698a379e0b08604d73d2d6728a2fcee4df 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 2953a8f327eabdee42dbc66e0c65f89d17add59a 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 f8b862862de4dde8dce3d0dc5f70eafb67b02d2c 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 dfc5d7b294c1df8d5c6b0e7d676a77ba1181e076 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
7d09a5c296a8ae924d61b200b4cb9135440fd9a0 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 a9398ae1e79404a15894aa42f451df5d18ed3e4c 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
 58dc6eefcb840d4dd70af7a47811fab1b5e696d9 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 d43c0c1e70cffbebd39b05f89ec396227c58ac77 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/client/builder/DatabaseBuilder.java
 f3d2182a04ab81417a4ba58d9340721513e8 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MCatalog.java
 e82cb4322f6e2ac7afeb5efcec7517a68c8b2dee 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/model/MDatabase.java
 815b39c483b2233660310983d58194fb1ab2d107 
  standalone-metastore/metastore-server/src/main/resources/package.jdo 
caaec457194332a99d5cd57bef746e969dd38161 
  
standalone-metastore/metastore-server/src/main/sql/derby/hive-schema-4.0.0.derby.sql
 a3c4196dbff7e53be5317631b314983d16a99020 
  
standalone-metastore/metastore-server/src/main/sql/derby/upgrade-3.2.0-to-4.0.0.derby.sql
 bcaebd18accf86846ae44a6498046514575fc069 
  
standalone-metastore/metastore-server/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql
 5ea1b4450d8258e841bb4af7381ca6fb0ba1a827 
  
standalone-metastore/metastore-server/src/main/sql/mssql/upgrade-3.2.0-to-4.0.0.mssql.sql
 edde08db9ef7ee01800c7cc3a04c813014abdd18 
  
standalone-metastore/metastore-server/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql
 a59c7d7e933d25d8d5af611e5b6aa0c0c19b 
  
standalone-metastore/metastore-server/src/main/sql/mysql/upgrade-3.2.0-to-4.0.0.mysql.sql
 701acb00984c61f7511dcc48053890b154575d1f 
  
standalone-metastore/metastore-server/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql
 b1980c5b83f16614845063516495188ebdd8c2a3 
  
standalone-metastore/metastore-server/src/main/sql/oracle/upgrade-3.2.0-to-4.0.0.oracle.sql
 b9f63313251ab1fa6278b862ed9e07e62b234c04 
  
standalone-metastore/metastore-server/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql
 9040005aa82b7a8cc5c01f257ecd47a7cc97e9b2 
  
standalone-metastore/metastore-server/src/main/sql/postgres/upgrade-3.2.0-to-4.0.0.postgres.sql
 0c36069d071d4b60cc338ba729da5d22e08ca8ca 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java
 bb20d9f42a855100397140f9e018c04c5f61dde7 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestCatalogs.java
 28eb1fadca80dfd3c962e4163120b83f00410c4a 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/client/TestDatabases.java
 d323ac6c90ed20f092b4e179fdb1bed8602ecf63 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/tools/TestSchemaToolForMetastore.java
 c2eb6c9e22a22f09cc1d2cc6394aa4e0e339b63a 


Diff: https://reviews.apache.org/r/69664/diff/6/

Changes: https://reviews.apache.org/r/69664/diff/5-6/


Testing
---

Ran ITests to check the db install and upgrade scripts are working for mysql, 
postgres, oracle and derby databases. The ITests for mssql is timing out for 
some reason due to container provisioning issues.


Thanks,

Vihang Karajgaonkar

Re: Review Request 69663: HIVE-16976

[Deepak Jaiswal]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69663/
---

(Updated Jan. 9, 2019, 5:50 p.m.)


Review request for hive, Ashutosh Chauhan, Gopal V, Jesús Camacho Rodríguez, 
and Jason Dere.


Changes
---

Updated the patch with review comments.


Bugs: HIVE-16976
https://issues.apache.org/jira/browse/HIVE-16976


Repository: hive-git


Description
---

DPP: SyntheticJoinPredicate transitivity for < > and BETWEEN

The patch supports predicates on non-equi joins and provides an interface for 
storage handler to decide if it can use this optimization.
Work to integrate this with DPP and semijoin will be done in separate JIRA.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d7f069eaa7 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java 
2ebb149354 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
 a1401aac72 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java f8c7e18eb1 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeDynamicListDesc.java 
676dfc9421 
  ql/src/java/org/apache/hadoop/hive/ql/ppd/SyntheticJoinPredicate.java 
e97e44796f 
  ql/src/test/results/clientpositive/llap/cross_prod_1.q.out f900a01be4 
  ql/src/test/results/clientpositive/llap/groupby_groupingset_bug.q.out 
de74af6dff 
  ql/src/test/results/clientpositive/llap/semijoin.q.out 00bc6cec55 
  ql/src/test/results/clientpositive/llap/subquery_in.q.out 07cc4dbabc 
  ql/src/test/results/clientpositive/llap/subquery_notin.q.out 29d8bbfb48 
  ql/src/test/results/clientpositive/llap/subquery_scalar.q.out 1cf281afbd 
  ql/src/test/results/clientpositive/llap/subquery_select.q.out 6255abdd70 


Diff: https://reviews.apache.org/r/69663/diff/2/

Changes: https://reviews.apache.org/r/69663/diff/1-2/


Testing
---


Thanks,

Deepak Jaiswal

Review Request 69700: Push projections and filters to S3 as S3Select queries

[Miklos Gergely]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69700/
---

Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.


Bugs: HIVE-21112
https://issues.apache.org/jira/browse/HIVE-21112


Repository: hive-git


Description
---

S3 Select enables applications to retrieve only a subset of data from an object 
by using simple SQL expressions.

S3A connector for Hadoop adds an API that aims at supporting S3 select queries. 
This issue initiates the work needed on the Hive side to create the S3 queries 
from the projection and filter expressions pushed to TS operators.


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/Constants.java ee954d9 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b213609 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchOperator.java 183fae5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java d7f069e 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 6bac285 
  ql/src/java/org/apache/hadoop/hive/ql/io/S3SelectManager.java PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/io/TestS3SelectManager.java 
PRE-CREATION 


Diff: https://reviews.apache.org/r/69700/diff/1/


Testing
---

Tested on real S3 environment.


Thanks,

Miklos Gergely

[jira] [Created] (HIVE-21112) Push projections and filters to S3 as S3Select queries

[Miklos Gergely (JIRA)]

Miklos Gergely created HIVE-21112:
-

 Summary: Push projections and filters to S3 as S3Select queries
 Key: HIVE-21112
 URL: https://issues.apache.org/jira/browse/HIVE-21112
 Project: Hive
  Issue Type: Improvement
  Components: Hive
Reporter: Miklos Gergely
Assignee: Miklos Gergely


S3 Select enables applications to retrieve only a subset of data from an object 
by using simple SQL expressions.

S3A connector for Hadoop adds an API that aims at supporting S3 select queries. 
This issue initiates the work needed on the Hive side to create the S3 queries 
from the projection and filter expressions pushed to TS operators.
 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)