[jira] [Created] (HIVE-21042) varchar and string will behave differently for non-standard utf8 characters

[yucai (JIRA)]

yucai created HIVE-21042:


 Summary: varchar and string will behave differently for 
non-standard utf8 characters
 Key: HIVE-21042
 URL: https://issues.apache.org/jira/browse/HIVE-21042
 Project: Hive
  Issue Type: Bug
Reporter: yucai


If data contains non-standard utf8 characters, varchar and string will behave 
differently.

The content in /tmp/hex_data is 
"6130373530633166313366306B35*B0A5*46386A*8DAEAB*62B4526F273464613936".

*B0A5 and* *8DAEAB* are non-standard utf8 characters, they are encoded to 
EFBFBD if the data type is varchar, which string will not change it. So:

VARCHAR shows: 
6130373530633166313366306B35EFBFBDEFBFBD46386AEFBFBDEFBFBDEFBFBD62EFBFBD526F273464613936
 STRING shows: 6130373530633166313366306B35B0A546386A8DAEAB62B4526F273464613936

See details:
{code:java}
hive> DROP TABLE TBL_S;
OK
Time taken: 0.562 seconds
hive> CREATE TABLE TBL_S
> (
>   GUID STRING
> )
> row format delimited fields terminated by '\177' stored as textfile
> LOCATION
> '/tmp/hex_data'
> tblproperties('serialization.null.format'='',  'timestamp.formats' = 
'-MM-dd HH:mm:ss')
> ;
OK
Time taken: 3.074 seconds
hive>
> DROP TABLE TBL_V;
OK
Time taken: 0.894 seconds
hive> CREATE TABLE TBL_V
> (
>   GUID VARCHAR(32)
> )
> row format delimited fields terminated by '\177' stored as textfile
> LOCATION
> '/tmp/hex_data'
> tblproperties('serialization.null.format'='',  'timestamp.formats' = 
'-MM-dd HH:mm:ss')
> ;
OK
Time taken: 0.242 seconds
hive> SELECT GUID, hex(GUID) FROM TBL_S;
OK
a0750c1f13f0k5��F8j���b�Ro'4da96
6130373530633166313366306B35B0A546386A8DAEAB62B4526F273464613936
Time taken: 1.581 seconds, Fetched: 1 row(s)
hive> SELECT GUID, hex(GUID) FROM TBL_V;
OK
a0750c1f13f0k5��F8j���b�Ro'4da96
6130373530633166313366306B35EFBFBDEFBFBD46386AEFBFBDEFBFBDEFBFBD62EFBFBD526F273464613936
{code}

Is it expected?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69534: HIVE-20992: Split the property "hive.metastore.dbaccess.ssl.properties" into more coherent and user-friendly properties.

[Karthik Manamcheri via Review Board]

> On Dec. 14, 2018, 2:58 a.m., Karthik Manamcheri wrote:
> > Ship It!

LGTM! Make sure you get someone else to review it as well. Thanks.


- Karthik


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69534/#review211318
---


On Dec. 14, 2018, 1:26 a.m., Morio Ramdenbourg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69534/
> ---
> 
> (Updated Dec. 14, 2018, 1:26 a.m.)
> 
> 
> Review request for hive, Adam Holley, Karthik Manamcheri, Peter Vary, and 
> Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-20992
> https://issues.apache.org/jira/browse/HIVE-20992
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The following new properties were added:
> 
> 1. metastore.dbaccess.use.SSL (hive.metastore.dbaccess.use.SSL)
> 2. javax.net.ssl.trustStore
> 3. javax.net.ssl.trustStorePassword
> 4. javax.net.ssl.trustStoreType
> 
> This was in an effort to guide the user towards an easier SSL
> configuration experience. This is the minimum requirement to set up SSL
> encryption to the HMS backend store.
> 
> This also solves the issue of the truststore password being stored in
> plain text. It can now be encrypted by default and loaded through the
> MetastoreConf.getPassword() method which handles secure password access
> 
> The property "hive.metastore.dbaccess.ssl.properties" is now
> deprecated, but it will still be kept for backwards-compatibility purposes.
> 
> Modified comments to clearly reflect what is / is not deprecated
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  e25a8cf9a19d78c0cc00bb2e5e0abee4d851ad98 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  e598a43e4dc2d2a2c25886ae7cbafd29b47c1f24 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
>  0cf113c927f2274d085e07cd72921fb35227e1f3 
> 
> 
> Diff: https://reviews.apache.org/r/69534/diff/4/
> 
> 
> Testing
> ---
> 
> Tests:
> 1. Unit tests were added to cover the functionality of configuring the Java 
> system properties.
> 2. Performed some manual and sanity tests to ensure that SSL was still 
> configurable to a remote DB. I performed these on MySQL, PostgreSQL, Oracle, 
> and Derby DB by creating generic DB hosts and setting them up with SSL. Once 
> SSL was set up, I triggered the metastore to perform database calls, and 
> captured packets using tcpdump. I then uploaded my packet captures to 
> Wireshark, and ensured that none of the data was human-readable.
> 
> I plan to upload a document to our Wiki explaining the process of enabling 
> TLS to these databases.
> 
> 
> Thanks,
> 
> Morio Ramdenbourg
> 
>

Re: Review Request 69534: HIVE-20992: Split the property "hive.metastore.dbaccess.ssl.properties" into more coherent and user-friendly properties.

[Karthik Manamcheri via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69534/#review211318
---


Ship it!




Ship It!

- Karthik Manamcheri


On Dec. 14, 2018, 1:26 a.m., Morio Ramdenbourg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69534/
> ---
> 
> (Updated Dec. 14, 2018, 1:26 a.m.)
> 
> 
> Review request for hive, Adam Holley, Karthik Manamcheri, Peter Vary, and 
> Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-20992
> https://issues.apache.org/jira/browse/HIVE-20992
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The following new properties were added:
> 
> 1. metastore.dbaccess.use.SSL (hive.metastore.dbaccess.use.SSL)
> 2. javax.net.ssl.trustStore
> 3. javax.net.ssl.trustStorePassword
> 4. javax.net.ssl.trustStoreType
> 
> This was in an effort to guide the user towards an easier SSL
> configuration experience. This is the minimum requirement to set up SSL
> encryption to the HMS backend store.
> 
> This also solves the issue of the truststore password being stored in
> plain text. It can now be encrypted by default and loaded through the
> MetastoreConf.getPassword() method which handles secure password access
> 
> The property "hive.metastore.dbaccess.ssl.properties" is now
> deprecated, but it will still be kept for backwards-compatibility purposes.
> 
> Modified comments to clearly reflect what is / is not deprecated
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  e25a8cf9a19d78c0cc00bb2e5e0abee4d851ad98 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  e598a43e4dc2d2a2c25886ae7cbafd29b47c1f24 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
>  0cf113c927f2274d085e07cd72921fb35227e1f3 
> 
> 
> Diff: https://reviews.apache.org/r/69534/diff/4/
> 
> 
> Testing
> ---
> 
> Tests:
> 1. Unit tests were added to cover the functionality of configuring the Java 
> system properties.
> 2. Performed some manual and sanity tests to ensure that SSL was still 
> configurable to a remote DB. I performed these on MySQL, PostgreSQL, Oracle, 
> and Derby DB by creating generic DB hosts and setting them up with SSL. Once 
> SSL was set up, I triggered the metastore to perform database calls, and 
> captured packets using tcpdump. I then uploaded my packet captures to 
> Wireshark, and ensured that none of the data was human-readable.
> 
> I plan to upload a document to our Wiki explaining the process of enabling 
> TLS to these databases.
> 
> 
> Thanks,
> 
> Morio Ramdenbourg
> 
>

[jira] [Created] (HIVE-21041) NPE, ParseException in getting schema from logical plan

[Teddy Choi (JIRA)]

Teddy Choi created HIVE-21041:
-

 Summary: NPE, ParseException in getting schema from logical plan
 Key: HIVE-21041
 URL: https://issues.apache.org/jira/browse/HIVE-21041
 Project: Hive
  Issue Type: Bug
Reporter: Teddy Choi
Assignee: Teddy Choi


HIVE-20552 makes getting schema from logical plan faster. But it throws 
ParseException when it has column alias, and NullPointerException when it has 
subqueries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69534: HIVE-20992: Split the property "hive.metastore.dbaccess.ssl.properties" into more coherent and user-friendly properties.

[Morio Ramdenbourg via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69534/
---

(Updated Dec. 14, 2018, 1:25 a.m.)


Review request for hive, Adam Holley, Karthik Manamcheri, Peter Vary, and 
Vihang Karajgaonkar.


Bugs: HIVE-20992
https://issues.apache.org/jira/browse/HIVE-20992


Repository: hive-git


Description
---

The following new properties were added:

1. metastore.dbaccess.use.SSL (hive.metastore.dbaccess.use.SSL)
2. javax.net.ssl.trustStore
3. javax.net.ssl.trustStorePassword
4. javax.net.ssl.trustStoreType

This was in an effort to guide the user towards an easier SSL
configuration experience. This is the minimum requirement to set up SSL
encryption to the HMS backend store.

This also solves the issue of the truststore password being stored in
plain text. It can now be encrypted by default and loaded through the
MetastoreConf.getPassword() method which handles secure password access

The property "hive.metastore.dbaccess.ssl.properties" is now
deprecated, but it will still be kept for backwards-compatibility purposes.

Modified comments to clearly reflect what is / is not deprecated


Diffs
-

  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 e25a8cf9a19d78c0cc00bb2e5e0abee4d851ad98 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 e598a43e4dc2d2a2c25886ae7cbafd29b47c1f24 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
 0cf113c927f2274d085e07cd72921fb35227e1f3 


Diff: https://reviews.apache.org/r/69534/diff/4/


Testing (updated)
---

Tests:
1. Unit tests were added to cover the functionality of configuring the Java 
system properties.
2. Performed some manual and sanity tests to ensure that SSL was still 
configurable to a remote DB. I performed these on MySQL, PostgreSQL, Oracle, 
and Derby DB by creating generic DB hosts and setting them up with SSL. Once 
SSL was set up, I triggered the metastore to perform database calls, and 
captured packets using tcpdump. I then uploaded my packet captures to 
Wireshark, and ensured that none of the data was human-readable.


Thanks,

Morio Ramdenbourg

Re: Review Request 69534: HIVE-20992: Split the property "hive.metastore.dbaccess.ssl.properties" into more coherent and user-friendly properties.

[Morio Ramdenbourg via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69534/
---

(Updated Dec. 14, 2018, 1:26 a.m.)


Review request for hive, Adam Holley, Karthik Manamcheri, Peter Vary, and 
Vihang Karajgaonkar.


Bugs: HIVE-20992
https://issues.apache.org/jira/browse/HIVE-20992


Repository: hive-git


Description
---

The following new properties were added:

1. metastore.dbaccess.use.SSL (hive.metastore.dbaccess.use.SSL)
2. javax.net.ssl.trustStore
3. javax.net.ssl.trustStorePassword
4. javax.net.ssl.trustStoreType

This was in an effort to guide the user towards an easier SSL
configuration experience. This is the minimum requirement to set up SSL
encryption to the HMS backend store.

This also solves the issue of the truststore password being stored in
plain text. It can now be encrypted by default and loaded through the
MetastoreConf.getPassword() method which handles secure password access

The property "hive.metastore.dbaccess.ssl.properties" is now
deprecated, but it will still be kept for backwards-compatibility purposes.

Modified comments to clearly reflect what is / is not deprecated


Diffs
-

  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 e25a8cf9a19d78c0cc00bb2e5e0abee4d851ad98 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 e598a43e4dc2d2a2c25886ae7cbafd29b47c1f24 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
 0cf113c927f2274d085e07cd72921fb35227e1f3 


Diff: https://reviews.apache.org/r/69534/diff/4/


Testing (updated)
---

Tests:
1. Unit tests were added to cover the functionality of configuring the Java 
system properties.
2. Performed some manual and sanity tests to ensure that SSL was still 
configurable to a remote DB. I performed these on MySQL, PostgreSQL, Oracle, 
and Derby DB by creating generic DB hosts and setting them up with SSL. Once 
SSL was set up, I triggered the metastore to perform database calls, and 
captured packets using tcpdump. I then uploaded my packet captures to 
Wireshark, and ensured that none of the data was human-readable.

I plan to upload a document to our Wiki explaining the process of enabling TLS 
to these databases.


Thanks,

Morio Ramdenbourg

Re: Review Request 69534: HIVE-20992: Split the property "hive.metastore.dbaccess.ssl.properties" into more coherent and user-friendly properties.

[Morio Ramdenbourg via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69534/
---

(Updated Dec. 14, 2018, 1:23 a.m.)


Review request for hive, Adam Holley, Karthik Manamcheri, Peter Vary, and 
Vihang Karajgaonkar.


Changes
---

Made modifications to the comments and deprecation to clearly state what is / 
is not supported


Bugs: HIVE-20992
https://issues.apache.org/jira/browse/HIVE-20992


Repository: hive-git


Description (updated)
---

The following new properties were added:

1. metastore.dbaccess.use.SSL (hive.metastore.dbaccess.use.SSL)
2. javax.net.ssl.trustStore
3. javax.net.ssl.trustStorePassword
4. javax.net.ssl.trustStoreType

This was in an effort to guide the user towards an easier SSL
configuration experience. This is the minimum requirement to set up SSL
encryption to the HMS backend store.

This also solves the issue of the truststore password being stored in
plain text. It can now be encrypted by default and loaded through the
MetastoreConf.getPassword() method which handles secure password access

The property "hive.metastore.dbaccess.ssl.properties" is now
deprecated, but it will still be kept for backwards-compatibility purposes.

Modified comments to clearly reflect what is / is not deprecated


Diffs (updated)
-

  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 e25a8cf9a19d78c0cc00bb2e5e0abee4d851ad98 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
 e598a43e4dc2d2a2c25886ae7cbafd29b47c1f24 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
 0cf113c927f2274d085e07cd72921fb35227e1f3 


Diff: https://reviews.apache.org/r/69534/diff/4/

Changes: https://reviews.apache.org/r/69534/diff/3-4/


Testing
---

Tests:
1. Unit tests were added to cover the functionality of configuring the Java 
system properties.
2. Performed some manual and sanity tests to ensure that SSL was still 
configurable to a remote DB.


Thanks,

Morio Ramdenbourg

Re: Review Request 69534: HIVE-20992: Split the property "hive.metastore.dbaccess.ssl.properties" into more coherent and user-friendly properties.

[Morio Ramdenbourg via Review Board]

> On Dec. 12, 2018, 10:54 p.m., Karthik Manamcheri wrote:
> > standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
> > Lines 472 (patched)
> > 
> >
> > Is this specific to a Java version?

It looks like Java 8 and beyond can actually read more types than this - I 
found this resource here: 
https://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html#KeyStore
I'll update this to use only the types specific to Java 8.


> On Dec. 12, 2018, 10:54 p.m., Karthik Manamcheri wrote:
> > standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
> > Lines 473-474 (patched)
> > 
> >
> > Move this to the top and state that if this is set to false, we'll 
> > ignore the other SSL properties.

Wanted to keep alphabetical order :) Will put a comment on top though


> On Dec. 12, 2018, 10:54 p.m., Karthik Manamcheri wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> > Line 332 (original), 398 (patched)
> > 
> >
> > State that this is deprecated here and maybe increase the LOG to warn 
> > and note that this might be changed in next version of hive.

Done


- Morio


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69534/#review211260
---


On Dec. 14, 2018, 1:23 a.m., Morio Ramdenbourg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69534/
> ---
> 
> (Updated Dec. 14, 2018, 1:23 a.m.)
> 
> 
> Review request for hive, Adam Holley, Karthik Manamcheri, Peter Vary, and 
> Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-20992
> https://issues.apache.org/jira/browse/HIVE-20992
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The following new properties were added:
> 
> 1. metastore.dbaccess.use.SSL (hive.metastore.dbaccess.use.SSL)
> 2. javax.net.ssl.trustStore
> 3. javax.net.ssl.trustStorePassword
> 4. javax.net.ssl.trustStoreType
> 
> This was in an effort to guide the user towards an easier SSL
> configuration experience. This is the minimum requirement to set up SSL
> encryption to the HMS backend store.
> 
> This also solves the issue of the truststore password being stored in
> plain text. It can now be encrypted by default and loaded through the
> MetastoreConf.getPassword() method which handles secure password access
> 
> The property "hive.metastore.dbaccess.ssl.properties" is now
> deprecated, but it will still be kept for backwards-compatibility purposes.
> 
> Modified comments to clearly reflect what is / is not deprecated
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  e25a8cf9a19d78c0cc00bb2e5e0abee4d851ad98 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  e598a43e4dc2d2a2c25886ae7cbafd29b47c1f24 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
>  0cf113c927f2274d085e07cd72921fb35227e1f3 
> 
> 
> Diff: https://reviews.apache.org/r/69534/diff/4/
> 
> 
> Testing
> ---
> 
> Tests:
> 1. Unit tests were added to cover the functionality of configuring the Java 
> system properties.
> 2. Performed some manual and sanity tests to ensure that SSL was still 
> configurable to a remote DB.
> 
> 
> Thanks,
> 
> Morio Ramdenbourg
> 
>

[jira] [Created] (HIVE-21040) msck does unnecessary file listing at last level of partitions

[Vihang Karajgaonkar (JIRA)]

Vihang Karajgaonkar created HIVE-21040:
--

 Summary: msck does unnecessary file listing at last level of 
partitions
 Key: HIVE-21040
 URL: https://issues.apache.org/jira/browse/HIVE-21040
 Project: Hive
  Issue Type: Improvement
Reporter: Vihang Karajgaonkar
Assignee: Vihang Karajgaonkar


Here is the code snippet which is run by {{msck}} to list directories

{noformat}
final Path currentPath = pd.p;
  final int currentDepth = pd.depth;
  FileStatus[] fileStatuses = fs.listStatus(currentPath, 
FileUtils.HIDDEN_FILES_PATH_FILTER);
  // found no files under a sub-directory under table base path; it is 
possible that the table
  // is empty and hence there are no partition sub-directories created 
under base path
  if (fileStatuses.length == 0 && currentDepth > 0 && currentDepth < 
partColNames.size()) {
// since maxDepth is not yet reached, we are missing partition
// columns in currentPath
logOrThrowExceptionWithMsg(
"MSCK is missing partition columns under " + 
currentPath.toString());
  } else {
// found files under currentPath add them to the queue if it is a 
directory
for (FileStatus fileStatus : fileStatuses) {
  if (!fileStatus.isDirectory() && currentDepth < partColNames.size()) {
// found a file at depth which is less than number of partition keys
logOrThrowExceptionWithMsg(
"MSCK finds a file rather than a directory when it searches for 
"
+ fileStatus.getPath().toString());
  } else if (fileStatus.isDirectory() && currentDepth < 
partColNames.size()) {
// found a sub-directory at a depth less than number of partition 
keys
// validate if the partition directory name matches with the 
corresponding
// partition colName at currentDepth
Path nextPath = fileStatus.getPath();
String[] parts = nextPath.getName().split("=");
if (parts.length != 2) {
  logOrThrowExceptionWithMsg("Invalid partition name " + nextPath);
} else if 
(!parts[0].equalsIgnoreCase(partColNames.get(currentDepth))) {
  logOrThrowExceptionWithMsg(
  "Unexpected partition key " + parts[0] + " found at " + 
nextPath);
} else {
  // add sub-directory to the work queue if maxDepth is not yet 
reached
  pendingPaths.add(new PathDepthInfo(nextPath, currentDepth + 1));
}
  }
}
if (currentDepth == partColNames.size()) {
  return currentPath;
}
  }
{noformat}

You can see that when the {{currentDepth}} at the {{maxDepth}} it still does a 
unnecessary listing of the files. We can improve this call by checking the 
currentDepth and bailing out early.

This can improve the performance of msck command significantly especially when 
there are lot of files in each partitions on remote filesystems like S3 or ADLS



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (HIVE-21039) CURRENT_TIMESTAMP returns value in UTC time zone

[Jesus Camacho Rodriguez (JIRA)]

Jesus Camacho Rodriguez created HIVE-21039:
--

 Summary: CURRENT_TIMESTAMP returns value in UTC time zone
 Key: HIVE-21039
 URL: https://issues.apache.org/jira/browse/HIVE-21039
 Project: Hive
  Issue Type: Bug
Affects Versions: 3.1.1
Reporter: Andrey Zinovyev
Assignee: Jesus Camacho Rodriguez


We're upgrading from hive 1.2 to 3.1 and it seems like new hive returns 
CURRENT_TIMESTAMP in UTC timezone. But before it was in local (system's 
default) timezone.

According to HIVE-5472 current_timestamp should use user's local timezone. This 
behaviour was changed in HIVE-12192  (if I got it right). 
GenericUDFCurrentTimestamp now explicitly uses UTC as timezone to initialise 
org.apache.hadoop.hive.common.type.Timestamp .

For example
Old hive:
{code}
hive> select current_timestamp;
OK
2018-12-12 22:43:39.024
{code}

New hive:
{code}
> select current_timestamp;
+--+
|   _c0|
+--+
| 2018-12-12 19:43:57.024  |
+--+
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69562: HIVE-16957

[Jesús Camacho Rodríguez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69562/
---

(Updated Dec. 13, 2018, 4:50 p.m.)


Review request for hive and Ashutosh Chauhan.


Bugs: HIVE-16957
https://issues.apache.org/jira/browse/HIVE-16957


Repository: hive-git


Description
---

HIVE-16957


Diffs (updated)
-

  itests/hive-unit/src/test/java/org/apache/hadoop/hive/hooks/TestHs2Hooks.java 
d26af3b08130ce26006cc57c53e68efca1d01166 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/LimitPushdownOptimizer.java 
859c18f3c2059a8e0d4e3fd7f62a521a72e691fd 
  ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
3a51d9795b0384356daa0a8ab576374fb05c3378 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsAutoGatherContext.java 
11ccff44588e20d6acc47af31bfa05e3beba7e2e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 
9aff0069fd0170cfec877caf481e8b6653435b81 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
b330d710a185aa44c4a89088bb025ecb28ba8856 
  ql/src/java/org/apache/hadoop/hive/ql/plan/CreateViewDesc.java 
f0f7b18d192f85b489ccde4e8a80e92dc11a0494 
  ql/src/test/queries/clientpositive/cbo_rp_cross_product_check_2.q 
00c19c74ad45fc13e0a2cf74af3f0fb33b73a1a3 
  ql/src/test/queries/clientpositive/materialized_view_create_rewrite.q 
9735e61598520469f176719bc51b4437204fd522 
  ql/src/test/queries/clientpositive/materialized_view_create_rewrite_2.q 
3f695d1ee212902a0415ac2912a4f15d521cd380 
  ql/src/test/queries/clientpositive/materialized_view_create_rewrite_3.q 
eb668a90acb546504cffb994ce25a1ab03c5b0c0 
  ql/src/test/queries/clientpositive/materialized_view_create_rewrite_4.q 
f21db8a8d87fe47eb22a3c43d0856dd5463a1671 
  ql/src/test/queries/clientpositive/materialized_view_create_rewrite_5.q 
3026d9093eddbf53611a79df5fbe1ee55884273a 
  ql/src/test/queries/clientpositive/materialized_view_create_rewrite_dummy.q 
8c9da8ae69967d1e11a333a46fddc51957cd5f31 
  
ql/src/test/queries/clientpositive/materialized_view_create_rewrite_multi_db.q 
85d926f9eb8c40d01bee6dab87baf5bf29790278 
  
ql/src/test/queries/clientpositive/materialized_view_create_rewrite_rebuild_dummy.q
 72e3d65117c0712929bb217e3a5e769101b27ebd 
  
ql/src/test/queries/clientpositive/materialized_view_create_rewrite_time_window.q
 4cdb715d2873b726561979a5b6d93c086467cd3d 
  
ql/src/test/queries/clientpositive/materialized_view_create_rewrite_time_window_2.q
 6873673a55580b3d94f2af4b7c7c0b20f191d879 
  ql/src/test/queries/clientpositive/materialized_view_rewrite_1.q 
18b9f7d418eff200d551ce4f95a399741dfca3f8 
  ql/src/test/queries/clientpositive/materialized_view_rewrite_10.q 
95427923164a28b1b0ef73f03fa69544daa5ff1c 
  ql/src/test/queries/clientpositive/materialized_view_rewrite_2.q 
3a447fc1873bf98d748fb9fd09278d60b7c9ac55 
  ql/src/test/queries/clientpositive/materialized_view_rewrite_3.q 
0823f59394dd00d9d02b0ae517454c035b21baed 
  ql/src/test/queries/clientpositive/materialized_view_rewrite_4.q 
6724cec7710f981d094576f6befccd2491ebe936 
  ql/src/test/queries/clientpositive/materialized_view_rewrite_5.q 
d87928c07363f6bd0ba4ae8f1986f5f53f513731 
  ql/src/test/queries/clientpositive/materialized_view_rewrite_6.q 
23fc3c14ce5a0c43483824de9ac0835453f74c44 
  ql/src/test/queries/clientpositive/materialized_view_rewrite_7.q 
3d1cedc4f56a1bceecba390292fcd26ad6ce1863 
  ql/src/test/queries/clientpositive/materialized_view_rewrite_8.q 
cfcfddce506d80ad55ac4d61dc0c8069b354c5cd 
  ql/src/test/queries/clientpositive/materialized_view_rewrite_9.q 
18d5cec8f98fdc7c44d3a358c765ab111019941d 
  ql/src/test/queries/clientpositive/materialized_view_rewrite_empty.q 
9ae1d4e81b033a07cf38bb4900cb819d34bf3d3f 
  ql/src/test/queries/clientpositive/materialized_view_rewrite_no_join_opt.q 
8de9c7087a80b0bd5a5a68a46f336390b02a33ff 
  ql/src/test/queries/clientpositive/materialized_view_rewrite_no_join_opt_2.q 
a1372301feb27e0845a6dd8e260ba18bd91f03fd 
  ql/src/test/queries/clientpositive/materialized_view_rewrite_part_1.q 
e6980c07f130a44ae4ade0a36e5591d75999eb6f 
  ql/src/test/queries/clientpositive/materialized_view_rewrite_part_2.q 
b2e6ebd6956c945b454a8646e32688374522326c 
  ql/src/test/queries/clientpositive/materialized_view_rewrite_ssb.q 
aed5bdbffdd225ee3d462b407d289205ff9775c8 
  ql/src/test/queries/clientpositive/materialized_view_rewrite_ssb_2.q 
0982b66ad7841272f234608a88cbe8e255cb3bfd 
  ql/src/test/results/clientnegative/masking_mv.q.out 
54e984321209b4893f0178d49bdc66aebc38ff44 
  ql/src/test/results/clientpositive/alter_table_update_status.q.out 
ec8a64cd65c77592fae39c1b251d4d65837043a8 
  
ql/src/test/results/clientpositive/alter_table_update_status_disable_bitvector.q.out
 1b787af0a884ff4f27415b8fc5cc1593c29e7190 
  ql/src/test/results/clientpositive/autoColumnStats_4.q.out 
83ee0f76da409f525c00aa932e40cf8d5d5630be 
  

[jira] [Created] (HIVE-21038) Fix checkstyle for standalone-metastore

[Miklos Gergely (JIRA)]

Miklos Gergely created HIVE-21038:
-

 Summary: Fix checkstyle for standalone-metastore
 Key: HIVE-21038
 URL: https://issues.apache.org/jira/browse/HIVE-21038
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.1.0
Reporter: Miklos Gergely
 Fix For: 3.2.0


Since HIVE-17506 checkstyle is not working.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 69560: HIVE-21035: Race condition in SparkUtilities#getSparkSession

[Adam Szita via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/69560/#review211278
---


Ship it!




Ship It!

- Adam Szita


On Dec. 12, 2018, 3:13 p.m., Antal Sinkovits wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/69560/
> ---
> 
> (Updated Dec. 12, 2018, 3:13 p.m.)
> 
> 
> Review request for hive, Denys Kuzmenko, Peter Vary, and Adam Szita.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> It can happen, that when in one given session, multiple queries are executed, 
> that due to a race condition, multiple spark application master gets kicked 
> off.
> In this case, the one that started earlier, will not be killed, when the hive 
> session closes, consuming resources.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java 
> d384ed6db6f4630ee2ef309854236e8f12926688 
>   ql/src/test/org/apache/hadoop/hive/ql/exec/spark/TestSparkUtilities.java 
> PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/69560/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Antal Sinkovits
> 
>

[jira] [Created] (HIVE-21037) Replicate column statistics for Hive tables

[Ashutosh Bapat (JIRA)]

Ashutosh Bapat created HIVE-21037:
-

 Summary: Replicate column statistics for Hive tables
 Key: HIVE-21037
 URL: https://issues.apache.org/jira/browse/HIVE-21037
 Project: Hive
  Issue Type: Improvement
  Components: HiveServer2
Reporter: Ashutosh Bapat
Assignee: Ashutosh Bapat


Statistics is important for query optimizations and thus keeping those 
up-to-date on replica is important from query performance perspective. The 
statistics are collected by scanning a table entirely. Thus when the data is 
replicated a. we could update the statistics by scanning it on replica or b. we 
could just replicate the statistics also. For following reasons we desire to go 
by the second approach instead of the first.
 # Scanning the data on replica isn’t a good option since it wastes CPU cycles 
and puts load during replication, which can be significant.
 # Storages like S3 may not have compute capabilities and thus when we are 
replicating from on-prem to cloud, we can not rely on the target to gather 
statistics.
 # For ACID tables, the statistics should be associated with the snapshot. This 
means the statistics collection on target should sync with the write-id on the 
source since target doesn't generate target ids of its own.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)