[jira] [Assigned] (HIVE-22434) Hive MetaStore CachedStore listTableNamesByFilter Method fix
[
https://issues.apache.org/jira/browse/HIVE-22434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
fanshilun reassigned HIVE-22434:
Assignee: fanshilun
> Hive MetaStore CachedStore listTableNamesByFilter Method fix
>
>
> Key: HIVE-22434
> URL: https://issues.apache.org/jira/browse/HIVE-22434
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Minor
>
> @Override
> // TODO: implement using SharedCache
> public List listTableNamesByFilter(String catName, String dbName,
> String filter, short maxTables)
> throws MetaException, UnknownDBException {
> return rawStore.listTableNamesByFilter(catName, dbName, filter,
> maxTables);
> }
> do this
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22423) Improve Logging In HadoopThriftAuthBridge
[
https://issues.apache.org/jira/browse/HIVE-22423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962721#comment-16962721
]
Hive QA commented on HIVE-22423:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12984286/HIVE-22423.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17546 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestMetastoreHousekeepingLeaderEmptyConfig.testHouseKeepingThreadExistence
(batchId=243)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19204/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19204/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19204/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12984286 - PreCommit-HIVE-Build
> Improve Logging In HadoopThriftAuthBridge
> -
>
> Key: HIVE-22423
> URL: https://issues.apache.org/jira/browse/HIVE-22423
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22423.1.patch
>
>
> # Remove superfluous debug log guards
> # Improve messages
> # Improve message format
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22431) Hive JDBC Storage Handler: java.lang.ClassCastException on accessing TINYINT, SMALLINT Data Type From JDBC Data Source
[
https://issues.apache.org/jira/browse/HIVE-22431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Syed Shameerur Rahman updated HIVE-22431:
-
Description:
Steps to Reproduce:
{code:java}
//MySQL table:
create table testtbl(a TINYINT, b SMALLINT);
// Insert to table via mysql connector
//Hive table:
CREATE EXTERNAL TABLE `hive_table`(
a TINYINT, b SMALLINT
)
STORED BY
'org.apache.hive.storage.jdbc.JdbcStorageHandler'
TBLPROPERTIES (
'hive.sql.database.type'='MYSQL',
'hive.sql.dbcp.password'='',
'hive.sql.dbcp.username'='',
'hive.sql.jdbc.driver'='com.mysql.jdbc.Driver',
'hive.sql.jdbc.url'='',
'hive.sql.table'='testtbl');
//Hive query:
select * from hive_table;
{code}
*Failed with exception
java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.ClassCastException: java.lang.Integer cannot be cast to
java.lang.Byte*
*Failed with exception
java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.ClassCastException: java.lang.Integer cannot be cast to
java.lang.Short*
{code:java}
{code}
was:
Steps to Reproduce:
{code:java}
//MySQL table:
create table testtbl(a TINYINT, b SMALLINT);
// Insert to table
//Hive table:
CREATE EXTERNAL TABLE `hive_table`(
a TINYINT, b SMALLINT
)
STORED BY
'org.apache.hive.storage.jdbc.JdbcStorageHandler'
TBLPROPERTIES (
'hive.sql.database.type'='MYSQL',
'hive.sql.dbcp.password'='',
'hive.sql.dbcp.username'='',
'hive.sql.jdbc.driver'='com.mysql.jdbc.Driver',
'hive.sql.jdbc.url'='',
'hive.sql.table'='testtbl');
//Hive query:
select * from hive_table;
{code}
*Failed with exception
java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.ClassCastException: java.lang.Integer cannot be cast to
java.lang.Byte*
*Failed with exception
java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.ClassCastException: java.lang.Integer cannot be cast to
java.lang.Short*
{code:java}
{code}
> Hive JDBC Storage Handler: java.lang.ClassCastException on accessing TINYINT,
> SMALLINT Data Type From JDBC Data Source
> --
>
> Key: HIVE-22431
> URL: https://issues.apache.org/jira/browse/HIVE-22431
> Project: Hive
> Issue Type: Bug
> Components: StorageHandler
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22431.01.patch
>
>
> Steps to Reproduce:
> {code:java}
> //MySQL table:
> create table testtbl(a TINYINT, b SMALLINT);
> // Insert to table via mysql connector
> //Hive table:
> CREATE EXTERNAL TABLE `hive_table`(
> a TINYINT, b SMALLINT
> )
> STORED BY
> 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
>
> 'hive.sql.database.type'='MYSQL',
> 'hive.sql.dbcp.password'='',
> 'hive.sql.dbcp.username'='',
> 'hive.sql.jdbc.driver'='com.mysql.jdbc.Driver',
> 'hive.sql.jdbc.url'='',
> 'hive.sql.table'='testtbl');
> //Hive query:
> select * from hive_table;
> {code}
> *Failed with exception
> java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.lang.Integer cannot be cast to
> java.lang.Byte*
> *Failed with exception
> java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.lang.Integer cannot be cast to
> java.lang.Short*
> {code:java}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22433) Hive JDBC Storage Handler: Incorrect results fetched from BOOLEAN and TIMESTAMP DataType From JDBC Data Source
[
https://issues.apache.org/jira/browse/HIVE-22433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Syed Shameerur Rahman updated HIVE-22433:
-
Labels: pull-request-available (was: )
> Hive JDBC Storage Handler: Incorrect results fetched from BOOLEAN and
> TIMESTAMP DataType From JDBC Data Source
> --
>
> Key: HIVE-22433
> URL: https://issues.apache.org/jira/browse/HIVE-22433
> Project: Hive
> Issue Type: Bug
> Components: StorageHandler
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22433.01.patch
>
>
> Steps to Reproduce:
> {code:java}
> //Derby table:
> create table testtbl(a BOOLEAN, b TIMESTAMP);
> // Insert to table via mysql connector
> // data in db
> true 2019-11-11 12:00:00
> //Hive table:
> CREATE EXTERNAL TABLE `hive_table`(
> a BOOLEAN, b TIMESTAMP
> )
> STORED BY
> 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
>
> 'hive.sql.database.type'='DERBY',
> 'hive.sql.dbcp.password'='',
> 'hive.sql.dbcp.username'='',
> 'hive.sql.jdbc.driver'='',
> 'hive.sql.jdbc.url'='',
> 'hive.sql.table'='testtbl');
> //Hive query:
> select * from hive_table;
> // result from select query
> false 2019-11-11 20:00:00
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22433) Hive JDBC Storage Handler: Incorrect results fetched from BOOLEAN and TIMESTAMP DataType From JDBC Data Source
[
https://issues.apache.org/jira/browse/HIVE-22433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Syed Shameerur Rahman updated HIVE-22433:
-
Attachment: HIVE-22433.01.patch
Status: Patch Available (was: Open)
> Hive JDBC Storage Handler: Incorrect results fetched from BOOLEAN and
> TIMESTAMP DataType From JDBC Data Source
> --
>
> Key: HIVE-22433
> URL: https://issues.apache.org/jira/browse/HIVE-22433
> Project: Hive
> Issue Type: Bug
> Components: StorageHandler
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22433.01.patch
>
>
> Steps to Reproduce:
> {code:java}
> //Derby table:
> create table testtbl(a BOOLEAN, b TIMESTAMP);
> // Insert to table via mysql connector
> // data in db
> true 2019-11-11 12:00:00
> //Hive table:
> CREATE EXTERNAL TABLE `hive_table`(
> a BOOLEAN, b TIMESTAMP
> )
> STORED BY
> 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
>
> 'hive.sql.database.type'='DERBY',
> 'hive.sql.dbcp.password'='',
> 'hive.sql.dbcp.username'='',
> 'hive.sql.jdbc.driver'='',
> 'hive.sql.jdbc.url'='',
> 'hive.sql.table'='testtbl');
> //Hive query:
> select * from hive_table;
> // result from select query
> false 2019-11-11 20:00:00
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22433) Hive JDBC Storage Handler: Incorrect results fetched from BOOLEAN and TIMESTAMP DataType From JDBC Data Source
[
https://issues.apache.org/jira/browse/HIVE-22433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Syed Shameerur Rahman updated HIVE-22433:
-
Component/s: StorageHandler
> Hive JDBC Storage Handler: Incorrect results fetched from BOOLEAN and
> TIMESTAMP DataType From JDBC Data Source
> --
>
> Key: HIVE-22433
> URL: https://issues.apache.org/jira/browse/HIVE-22433
> Project: Hive
> Issue Type: Bug
> Components: StorageHandler
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Fix For: 4.0.0
>
>
> Steps to Reproduce:
> {code:java}
> //Derby table:
> create table testtbl(a BOOLEAN, b TIMESTAMP);
> // Insert to table via mysql connector
> // data in db
> true 2019-11-11 12:00:00
> //Hive table:
> CREATE EXTERNAL TABLE `hive_table`(
> a BOOLEAN, b TIMESTAMP
> )
> STORED BY
> 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
>
> 'hive.sql.database.type'='DERBY',
> 'hive.sql.dbcp.password'='',
> 'hive.sql.dbcp.username'='',
> 'hive.sql.jdbc.driver'='',
> 'hive.sql.jdbc.url'='',
> 'hive.sql.table'='testtbl');
> //Hive query:
> select * from hive_table;
> // result from select query
> false 2019-11-11 20:00:00
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22433) Hive JDBC Storage Handler: Incorrect results fetched from BOOLEAN and TIMESTAMP DataType From JDBC Data Source
[
https://issues.apache.org/jira/browse/HIVE-22433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Syed Shameerur Rahman reassigned HIVE-22433:
> Hive JDBC Storage Handler: Incorrect results fetched from BOOLEAN and
> TIMESTAMP DataType From JDBC Data Source
> --
>
> Key: HIVE-22433
> URL: https://issues.apache.org/jira/browse/HIVE-22433
> Project: Hive
> Issue Type: Bug
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Fix For: 4.0.0
>
>
> Steps to Reproduce:
> {code:java}
> //Derby table:
> create table testtbl(a BOOLEAN, b TIMESTAMP);
> // Insert to table via mysql connector
> // data in db
> true 2019-11-11 12:00:00
> //Hive table:
> CREATE EXTERNAL TABLE `hive_table`(
> a BOOLEAN, b TIMESTAMP
> )
> STORED BY
> 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
>
> 'hive.sql.database.type'='DERBY',
> 'hive.sql.dbcp.password'='',
> 'hive.sql.dbcp.username'='',
> 'hive.sql.jdbc.driver'='',
> 'hive.sql.jdbc.url'='',
> 'hive.sql.table'='testtbl');
> //Hive query:
> select * from hive_table;
> // result from select query
> false 2019-11-11 20:00:00
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22431) Hive JDBC Storage Handler: java.lang.ClassCastException on accessing TINYINT, SMALLINT Data Type From JDBC Data Source
[
https://issues.apache.org/jira/browse/HIVE-22431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Syed Shameerur Rahman updated HIVE-22431:
-
Description:
Steps to Reproduce:
{code:java}
//MySQL table:
create table testtbl(a TINYINT, b SMALLINT);
// Insert to table
//Hive table:
CREATE EXTERNAL TABLE `hive_table`(
a TINYINT, b SMALLINT
)
STORED BY
'org.apache.hive.storage.jdbc.JdbcStorageHandler'
TBLPROPERTIES (
'hive.sql.database.type'='MYSQL',
'hive.sql.dbcp.password'='',
'hive.sql.dbcp.username'='',
'hive.sql.jdbc.driver'='com.mysql.jdbc.Driver',
'hive.sql.jdbc.url'='',
'hive.sql.table'='testtbl');
//Hive query:
select * from hive_table;
{code}
*Failed with exception
java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.ClassCastException: java.lang.Integer cannot be cast to
java.lang.Byte*
*Failed with exception
java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.ClassCastException: java.lang.Integer cannot be cast to
java.lang.Short*
{code:java}
{code}
was:
Steps to Reproduce:
{code:java}
//MySQL table:
create table testtbl(a TINYINT, b SMALLINT);
// Insert to table
//Hive table:
CREATE EXTERNAL TABLE `hive_table`(
a TINYINT, b SMALLINT
)
STORED BY
'org.apache.hive.storage.jdbc.JdbcStorageHandler'
TBLPROPERTIES (
'hive.sql.database.type'='MYSQL',
'hive.sql.dbcp.password'='hive',
'hive.sql.dbcp.username'='hive',
'hive.sql.jdbc.driver'='com.mysql.jdbc.Driver',
'hive.sql.jdbc.url'='jdbc:mysql://hadoop/test',
'hive.sql.table'='testtbl');
//Hive query:
select * from hive_table;
{code}
*Failed with exception
java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.ClassCastException: java.lang.Integer cannot be cast to
java.lang.Byte*
*Failed with exception
java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.ClassCastException: java.lang.Integer cannot be cast to
java.lang.Short*
{code:java}
{code}
> Hive JDBC Storage Handler: java.lang.ClassCastException on accessing TINYINT,
> SMALLINT Data Type From JDBC Data Source
> --
>
> Key: HIVE-22431
> URL: https://issues.apache.org/jira/browse/HIVE-22431
> Project: Hive
> Issue Type: Bug
> Components: StorageHandler
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22431.01.patch
>
>
> Steps to Reproduce:
> {code:java}
> //MySQL table:
> create table testtbl(a TINYINT, b SMALLINT);
> // Insert to table
> //Hive table:
> CREATE EXTERNAL TABLE `hive_table`(
> a TINYINT, b SMALLINT
> )
> STORED BY
> 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
>
> 'hive.sql.database.type'='MYSQL',
> 'hive.sql.dbcp.password'='',
> 'hive.sql.dbcp.username'='',
> 'hive.sql.jdbc.driver'='com.mysql.jdbc.Driver',
> 'hive.sql.jdbc.url'='',
> 'hive.sql.table'='testtbl');
> //Hive query:
> select * from hive_table;
> {code}
> *Failed with exception
> java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.lang.Integer cannot be cast to
> java.lang.Byte*
> *Failed with exception
> java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.lang.Integer cannot be cast to
> java.lang.Short*
> {code:java}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22431) Hive JDBC Storage Handler: java.lang.ClassCastException on accessing TINYINT, SMALLINT Data Type From JDBC Data Source
[
https://issues.apache.org/jira/browse/HIVE-22431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Syed Shameerur Rahman updated HIVE-22431:
-
Description:
Steps to Reproduce:
{code:java}
//MySQL table:
create table testtbl(a TINYINT, b SMALLINT);
// Insert to table
//Hive table:
CREATE EXTERNAL TABLE `hive_table`(
a TINYINT, b SMALLINT
)
STORED BY
'org.apache.hive.storage.jdbc.JdbcStorageHandler'
TBLPROPERTIES (
'hive.sql.database.type'='MYSQL',
'hive.sql.dbcp.password'='hive',
'hive.sql.dbcp.username'='hive',
'hive.sql.jdbc.driver'='com.mysql.jdbc.Driver',
'hive.sql.jdbc.url'='jdbc:mysql://hadoop/test',
'hive.sql.table'='testtbl');
//Hive query:
select * from hive_table;
{code}
*Failed with exception
java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.ClassCastException: java.lang.Integer cannot be cast to
java.lang.Byte*
*Failed with exception
java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.ClassCastException: java.lang.Integer cannot be cast to
java.lang.Short*
{code:java}
{code}
was:
Steps to Reproduce:
{code:java}
//MySQL table:
create table testtbl(a TINYINT, b SMALLINT);
// Insert to table
//Hive table:
CREATE EXTERNAL TABLE `hive_table`(
a TINYINT, b SMALLINT
)
ROW FORMAT SERDE
'org.apache.hive.storage.jdbc.JdbcSerDe'
STORED BY
'org.apache.hive.storage.jdbc.JdbcStorageHandler'
TBLPROPERTIES (
'hive.sql.database.type'='MYSQL',
'hive.sql.dbcp.password'='hive',
'hive.sql.dbcp.username'='hive',
'hive.sql.jdbc.driver'='com.mysql.jdbc.Driver',
'hive.sql.jdbc.url'='jdbc:mysql://hadoop/test',
'hive.sql.table'='testtbl');
//Hive query:
select * from hive_table;
{code}
*Failed with exception
java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.ClassCastException: java.lang.Integer cannot be cast to
java.lang.Byte*
*Failed with exception
java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.ClassCastException: java.lang.Integer cannot be cast to
java.lang.Short*
{code:java}
{code}
> Hive JDBC Storage Handler: java.lang.ClassCastException on accessing TINYINT,
> SMALLINT Data Type From JDBC Data Source
> --
>
> Key: HIVE-22431
> URL: https://issues.apache.org/jira/browse/HIVE-22431
> Project: Hive
> Issue Type: Bug
> Components: StorageHandler
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22431.01.patch
>
>
> Steps to Reproduce:
> {code:java}
> //MySQL table:
> create table testtbl(a TINYINT, b SMALLINT);
> // Insert to table
> //Hive table:
> CREATE EXTERNAL TABLE `hive_table`(
> a TINYINT, b SMALLINT
> )
> STORED BY
> 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
>
> 'hive.sql.database.type'='MYSQL',
> 'hive.sql.dbcp.password'='hive',
> 'hive.sql.dbcp.username'='hive',
> 'hive.sql.jdbc.driver'='com.mysql.jdbc.Driver',
> 'hive.sql.jdbc.url'='jdbc:mysql://hadoop/test',
> 'hive.sql.table'='testtbl');
> //Hive query:
> select * from hive_table;
> {code}
> *Failed with exception
> java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.lang.Integer cannot be cast to
> java.lang.Byte*
> *Failed with exception
> java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.lang.Integer cannot be cast to
> java.lang.Short*
> {code:java}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22431) Hive JDBC Storage Handler: java.lang.ClassCastException on accessing TINYINT, SMALLINT Data Type From JDBC Data Source
[
https://issues.apache.org/jira/browse/HIVE-22431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Syed Shameerur Rahman updated HIVE-22431:
-
Labels: pull-request-available (was: )
> Hive JDBC Storage Handler: java.lang.ClassCastException on accessing TINYINT,
> SMALLINT Data Type From JDBC Data Source
> --
>
> Key: HIVE-22431
> URL: https://issues.apache.org/jira/browse/HIVE-22431
> Project: Hive
> Issue Type: Bug
> Components: StorageHandler
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22431.01.patch
>
>
> Steps to Reproduce:
> {code:java}
> //MySQL table:
> create table testtbl(a TINYINT, b SMALLINT);
> // Insert to table
> //Hive table:
> CREATE EXTERNAL TABLE `hive_table`(
> a TINYINT, b SMALLINT
> )
> ROW FORMAT SERDE
> 'org.apache.hive.storage.jdbc.JdbcSerDe'
> STORED BY
> 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
>
> 'hive.sql.database.type'='MYSQL',
> 'hive.sql.dbcp.password'='hive',
> 'hive.sql.dbcp.username'='hive',
> 'hive.sql.jdbc.driver'='com.mysql.jdbc.Driver',
> 'hive.sql.jdbc.url'='jdbc:mysql://hadoop/test',
> 'hive.sql.table'='testtbl');
> //Hive query:
> select * from hive_table;
> {code}
> *Failed with exception
> java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.lang.Integer cannot be cast to
> java.lang.Byte*
> *Failed with exception
> java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.lang.Integer cannot be cast to
> java.lang.Short*
> {code:java}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22432) CachedStore Log Level
[
https://issues.apache.org/jira/browse/HIVE-22432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
fanshilun updated HIVE-22432:
-
Attachment: (was: HIVE-22432.1.patch)
> CachedStore Log Level
> -
>
> Key: HIVE-22432
> URL: https://issues.apache.org/jira/browse/HIVE-22432
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Trivial
> Attachments: HIVE-22432.1.patch
>
>
> CachedStore Log Level
> debug -> info
> LOG.info("CachedStore: updated cached objects. Shared cache update count is:
> {}",
> sharedCache.getUpdateCount());
> show update state
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22432) CachedStore Log Level
[
https://issues.apache.org/jira/browse/HIVE-22432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
fanshilun reassigned HIVE-22432:
Assignee: fanshilun
> CachedStore Log Level
> -
>
> Key: HIVE-22432
> URL: https://issues.apache.org/jira/browse/HIVE-22432
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Trivial
> Attachments: HIVE-22432.1.patch
>
>
> CachedStore Log Level
> debug -> info
> LOG.info("CachedStore: updated cached objects. Shared cache update count is:
> {}",
> sharedCache.getUpdateCount());
> show update state
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22432) CachedStore Log Level
[
https://issues.apache.org/jira/browse/HIVE-22432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
fanshilun updated HIVE-22432:
-
Attachment: HIVE-22432.1.patch
Status: Patch Available (was: Open)
> CachedStore Log Level
> -
>
> Key: HIVE-22432
> URL: https://issues.apache.org/jira/browse/HIVE-22432
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Trivial
> Attachments: HIVE-22432.1.patch
>
>
> CachedStore Log Level
> debug -> info
> LOG.info("CachedStore: updated cached objects. Shared cache update count is:
> {}",
> sharedCache.getUpdateCount());
> show update state
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22432) CachedStore Log Level
[
https://issues.apache.org/jira/browse/HIVE-22432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
fanshilun updated HIVE-22432:
-
Attachment: HIVE-22432.1.patch
> CachedStore Log Level
> -
>
> Key: HIVE-22432
> URL: https://issues.apache.org/jira/browse/HIVE-22432
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: fanshilun
>Priority: Trivial
> Attachments: HIVE-22432.1.patch
>
>
> CachedStore Log Level
> debug -> info
> LOG.info("CachedStore: updated cached objects. Shared cache update count is:
> {}",
> sharedCache.getUpdateCount());
> show update state
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22431) Hive JDBC Storage Handler: java.lang.ClassCastException on accessing TINYINT, SMALLINT Data Type From JDBC Data Source
[
https://issues.apache.org/jira/browse/HIVE-22431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Syed Shameerur Rahman updated HIVE-22431:
-
Attachment: HIVE-22431.01.patch
Status: Patch Available (was: Open)
> Hive JDBC Storage Handler: java.lang.ClassCastException on accessing TINYINT,
> SMALLINT Data Type From JDBC Data Source
> --
>
> Key: HIVE-22431
> URL: https://issues.apache.org/jira/browse/HIVE-22431
> Project: Hive
> Issue Type: Bug
> Components: StorageHandler
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Attachments: HIVE-22431.01.patch
>
>
> Steps to Reproduce:
> {code:java}
> //MySQL table:
> create table testtbl(a TINYINT, b SMALLINT);
> // Insert to table
> //Hive table:
> CREATE EXTERNAL TABLE `hive_table`(
> a TINYINT, b SMALLINT
> )
> ROW FORMAT SERDE
> 'org.apache.hive.storage.jdbc.JdbcSerDe'
> STORED BY
> 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
>
> 'hive.sql.database.type'='MYSQL',
> 'hive.sql.dbcp.password'='hive',
> 'hive.sql.dbcp.username'='hive',
> 'hive.sql.jdbc.driver'='com.mysql.jdbc.Driver',
> 'hive.sql.jdbc.url'='jdbc:mysql://hadoop/test',
> 'hive.sql.table'='testtbl');
> //Hive query:
> select * from hive_table;
> {code}
> *Failed with exception
> java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.lang.Integer cannot be cast to
> java.lang.Byte*
> *Failed with exception
> java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.lang.Integer cannot be cast to
> java.lang.Short*
> {code:java}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22431) Hive JDBC Storage Handler: java.lang.ClassCastException on accessing TINYINT, SMALLINT Data Type From JDBC Data Source
[
https://issues.apache.org/jira/browse/HIVE-22431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Syed Shameerur Rahman reassigned HIVE-22431:
> Hive JDBC Storage Handler: java.lang.ClassCastException on accessing TINYINT,
> SMALLINT Data Type From JDBC Data Source
> --
>
> Key: HIVE-22431
> URL: https://issues.apache.org/jira/browse/HIVE-22431
> Project: Hive
> Issue Type: Bug
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
>
> Steps to Reproduce:
> {code:java}
> //MySQL table:
> create table testtbl(a TINYINT, b SMALLINT);
> // Insert to table
> //Hive table:
> CREATE EXTERNAL TABLE `hive_table`(
> a TINYINT, b SMALLINT
> )
> ROW FORMAT SERDE
> 'org.apache.hive.storage.jdbc.JdbcSerDe'
> STORED BY
> 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
>
> 'hive.sql.database.type'='MYSQL',
> 'hive.sql.dbcp.password'='hive',
> 'hive.sql.dbcp.username'='hive',
> 'hive.sql.jdbc.driver'='com.mysql.jdbc.Driver',
> 'hive.sql.jdbc.url'='jdbc:mysql://hadoop/test',
> 'hive.sql.table'='testtbl');
> //Hive query:
> select * from hive_table;
> {code}
> *Failed with exception
> java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.lang.Integer cannot be cast to
> java.lang.Byte*
> *Failed with exception
> java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.lang.Integer cannot be cast to
> java.lang.Short*
> {code:java}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22431) Hive JDBC Storage Handler: java.lang.ClassCastException on accessing TINYINT, SMALLINT Data Type From JDBC Data Source
[
https://issues.apache.org/jira/browse/HIVE-22431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Syed Shameerur Rahman updated HIVE-22431:
-
Component/s: StorageHandler
> Hive JDBC Storage Handler: java.lang.ClassCastException on accessing TINYINT,
> SMALLINT Data Type From JDBC Data Source
> --
>
> Key: HIVE-22431
> URL: https://issues.apache.org/jira/browse/HIVE-22431
> Project: Hive
> Issue Type: Bug
> Components: StorageHandler
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
>
> Steps to Reproduce:
> {code:java}
> //MySQL table:
> create table testtbl(a TINYINT, b SMALLINT);
> // Insert to table
> //Hive table:
> CREATE EXTERNAL TABLE `hive_table`(
> a TINYINT, b SMALLINT
> )
> ROW FORMAT SERDE
> 'org.apache.hive.storage.jdbc.JdbcSerDe'
> STORED BY
> 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
>
> 'hive.sql.database.type'='MYSQL',
> 'hive.sql.dbcp.password'='hive',
> 'hive.sql.dbcp.username'='hive',
> 'hive.sql.jdbc.driver'='com.mysql.jdbc.Driver',
> 'hive.sql.jdbc.url'='jdbc:mysql://hadoop/test',
> 'hive.sql.table'='testtbl');
> //Hive query:
> select * from hive_table;
> {code}
> *Failed with exception
> java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.lang.Integer cannot be cast to
> java.lang.Byte*
> *Failed with exception
> java.io.IOException:org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.lang.Integer cannot be cast to
> java.lang.Short*
> {code:java}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22423) Improve Logging In HadoopThriftAuthBridge
[
https://issues.apache.org/jira/browse/HIVE-22423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962697#comment-16962697
]
Hive QA commented on HIVE-22423:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
22s{color} | {color:blue} standalone-metastore/metastore-common in master has
37 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} standalone-metastore/metastore-common: The patch
generated 0 new + 32 unchanged - 3 fixed = 32 total (was 35) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 0s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19204/dev-support/hive-personality.sh
|
| git revision | master / 305e710 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common U:
standalone-metastore/metastore-common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19204/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improve Logging In HadoopThriftAuthBridge
> -
>
> Key: HIVE-22423
> URL: https://issues.apache.org/jira/browse/HIVE-22423
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22423.1.patch
>
>
> # Remove superfluous debug log guards
> # Improve messages
> # Improve message format
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22421) Improve Logging If Configuration File Not Found
[
https://issues.apache.org/jira/browse/HIVE-22421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962674#comment-16962674
]
Hive QA commented on HIVE-22421:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12984284/HIVE-22421.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17546 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19203/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19203/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19203/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12984284 - PreCommit-HIVE-Build
> Improve Logging If Configuration File Not Found
> ---
>
> Key: HIVE-22421
> URL: https://issues.apache.org/jira/browse/HIVE-22421
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22421.1.patch
>
>
> {code:none}
> 2019-10-28T21:07:27,599 INFO [main] conf.MetastoreConf: Unable to find
> config file metastore-site.xml
> 2019-10-28T21:07:27,599 INFO [main] conf.MetastoreConf: Found configuration
> file null
> {code}
> Prints 'unable to find' followed by 'null'. Just print one or the other.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22421) Improve Logging If Configuration File Not Found
[
https://issues.apache.org/jira/browse/HIVE-22421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962656#comment-16962656
]
Hive QA commented on HIVE-22421:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
32s{color} | {color:blue} standalone-metastore/metastore-common in master has
37 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19203/dev-support/hive-personality.sh
|
| git revision | master / 305e710 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common U:
standalone-metastore/metastore-common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19203/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improve Logging If Configuration File Not Found
> ---
>
> Key: HIVE-22421
> URL: https://issues.apache.org/jira/browse/HIVE-22421
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22421.1.patch
>
>
> {code:none}
> 2019-10-28T21:07:27,599 INFO [main] conf.MetastoreConf: Unable to find
> config file metastore-site.xml
> 2019-10-28T21:07:27,599 INFO [main] conf.MetastoreConf: Found configuration
> file null
> {code}
> Prints 'unable to find' followed by 'null'. Just print one or the other.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22410) CachedStore Prewarm Failure NPE
[ https://issues.apache.org/jira/browse/HIVE-22410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961777#comment-16961777 ] fanshilun edited comment on HIVE-22410 at 10/30/19 3:19 AM: [~kkasa] [~kuczoram] can you please take a look? Thanks! was (Author: slfan1989): [~kkasa] can you please take a look? Thanks! > CachedStore Prewarm Failure NPE > --- > > Key: HIVE-22410 > URL: https://issues.apache.org/jira/browse/HIVE-22410 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 3.1.2, 2.3.7 > Environment: JDK 1.7.0_40 >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: cachedstore > Fix For: 3.1.2, 2.3.7 > > Attachments: HIVE-22410.1.patch > > > java.lang.NullPointerException > at > org.apache.hadoop.hive.metastore.utils.IncrementalObjectSizeEstimator$ObjectEstimator.estimateArrayElements(IncrementalObjectSizeEstimator.java:501) > ~[hive-metastore-2.3.5.jar:2.3.5] > at > org.apache.hadoop.hive.metastore.utils.IncrementalObjectSizeEstimator$ObjectEstimator.estimate(IncrementalObjectSizeEstimator.java:443) > ~[hive-metastore-2.3.5.jar:2.3.5] > at > org.apache.hadoop.hive.metastore.utils.IncrementalObjectSizeEstimator$ObjectEstimator.estimate(IncrementalObjectSizeEstimator.java:433) > ~[hive-metastore-2.3.5.jar:2.3.5] > at > org.apache.hadoop.hive.metastore.utils.IncrementalObjectSizeEstimator$ObjectEstimator.estimate(IncrementalObjectSizeEstimator.java:369) > ~[hive-metastore-2.3.5.jar:2.3.5] > at > org.apache.hadoop.hive.metastore.cached.SharedCache.populateTableInCache(SharedCache.java:1028) > ~[hive-metastore-2.3.5.jar:2.3.5] > at > org.apache.hadoop.hive.metastore.cached.CachedStore.prewarm(CachedStore.java:314) > ~[hive-metastore-2.3.5.jar:2.3.5] > at > org.apache.hadoop.hive.metastore.cached.CachedStore$CacheUpdateMasterWork.run(CachedStore.java:492) > [hive-metastore-2.3.5.jar:2.3.5] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > [?:1.7.0_40] > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > [?:1.7.0_40] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > [?:1.7.0_40] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > [?:1.7.0_40] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > [?:1.7.0_40] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > [?:1.7.0_40] > at java.lang.Thread.run(Thread.java:724) [?:1.7.0_40] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-22384) Hive MetaStore CachedStore Funcion Cache Dev
[ https://issues.apache.org/jira/browse/HIVE-22384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961757#comment-16961757 ] fanshilun edited comment on HIVE-22384 at 10/30/19 3:12 AM: [~pvary] [~daijy] [~kuczoram] can you please review code ? Thanks! was (Author: slfan1989): [~pvary] [~daijy] can you please review code ? Thanks! > Hive MetaStore CachedStore Funcion Cache Dev > > > Key: HIVE-22384 > URL: https://issues.apache.org/jira/browse/HIVE-22384 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Affects Versions: 2.3.7 >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: cachedstore > Fix For: 2.3.7 > > Attachments: HIVE-22384.1.patch, HIVE-22384.2.patch, > HIVE-22384.3.patch, HIVE-22384.4.patch, HIVE-22384.5.patch, > HIVE-22384.6.patch, HIVE-22384.7.patch, HIVE-22384.8.patch, HIVE-22384.9.patch > > > Hive MetaStore CachedStore Funcion Cache Dev -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22419) Improve Messages Emitted From HiveMetaStoreClient
[
https://issues.apache.org/jira/browse/HIVE-22419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962650#comment-16962650
]
Hive QA commented on HIVE-22419:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12984282/HIVE-22419.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17546 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19201/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19201/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19201/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12984282 - PreCommit-HIVE-Build
> Improve Messages Emitted From HiveMetaStoreClient
> -
>
> Key: HIVE-22419
> URL: https://issues.apache.org/jira/browse/HIVE-22419
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22419.1.patch
>
>
> After reviewing some logs and errors emitted during a QTest run, I would like
> to propose some improvements to logging in {{HiveMetaStoreClient}}.
> * Remove duplicate logging
> * Remove superfluous class {{StackTraceLogger}}
> * Do not use contractions in public-facing error messages and logs
> * Make all logging side-effect free (see {{connCount}})
> * Code simplification
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22430) Avoid creation of additional RS for limit if it is equal to zero
[ https://issues.apache.org/jira/browse/HIVE-22430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22430: --- Priority: Minor (was: Major) > Avoid creation of additional RS for limit if it is equal to zero > > > Key: HIVE-22430 > URL: https://issues.apache.org/jira/browse/HIVE-22430 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Minor > Attachments: HIVE-22430.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22430) Avoid creation of additional RS for limit if it is equal to zero
[ https://issues.apache.org/jira/browse/HIVE-22430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22430: --- Attachment: HIVE-22430.patch > Avoid creation of additional RS for limit if it is equal to zero > > > Key: HIVE-22430 > URL: https://issues.apache.org/jira/browse/HIVE-22430 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-22430.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22430) Avoid creation of additional RS for limit if it is equal to zero
[ https://issues.apache.org/jira/browse/HIVE-22430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22430: --- Status: Patch Available (was: In Progress) > Avoid creation of additional RS for limit if it is equal to zero > > > Key: HIVE-22430 > URL: https://issues.apache.org/jira/browse/HIVE-22430 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-22430) Avoid creation of additional RS for limit if it is equal to zero
[ https://issues.apache.org/jira/browse/HIVE-22430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-22430 started by Jesus Camacho Rodriguez. -- > Avoid creation of additional RS for limit if it is equal to zero > > > Key: HIVE-22430 > URL: https://issues.apache.org/jira/browse/HIVE-22430 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22430) Avoid creation of additional RS for limit if it is equal to zero
[ https://issues.apache.org/jira/browse/HIVE-22430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-22430: -- > Avoid creation of additional RS for limit if it is equal to zero > > > Key: HIVE-22430 > URL: https://issues.apache.org/jira/browse/HIVE-22430 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962639#comment-16962639
]
Hive QA commented on HIVE-21884:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
51s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 6m
19s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
21s{color} | {color:blue} standalone-metastore/metastore-common in master has
37 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
32s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
10s{color} | {color:blue} standalone-metastore/metastore-server in master has
171 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
4s{color} | {color:blue} ql in master has 1545 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} service in master has 49 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
48s{color} | {color:blue} itests/util in master has 53 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 11m
41s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} standalone-metastore: The patch generated 2 new + 1679
unchanged - 16 fixed = 1681 total (was 1695) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} The patch metastore-common passed checkstyle {color}
|
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} common: The patch generated 1 new + 367 unchanged - 0
fixed = 368 total (was 367) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} The patch metastore passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
17s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 2 new + 1185 unchanged - 16 fixed = 1187 total (was 1201) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
22s{color} | {color:red} ql: The patch generated 30 new + 698 unchanged - 6
fixed = 728 total (was 704) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} service: The patch generated 0 new + 38 unchanged -
1 fixed = 38 total (was 39) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 4m
5s{color} | {color:red} root: The patch generated 43 new + 2867 unchanged - 26
fixed = 2910 total (was 2893) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} The patch hcatalog-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} The patch hive-minikdc passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
[jira] [Commented] (HIVE-22419) Improve Messages Emitted From HiveMetaStoreClient
[
https://issues.apache.org/jira/browse/HIVE-22419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962634#comment-16962634
]
Hive QA commented on HIVE-22419:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
34s{color} | {color:blue} standalone-metastore/metastore-common in master has
37 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} standalone-metastore/metastore-common: The patch
generated 0 new + 215 unchanged - 2 fixed = 215 total (was 217) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19201/dev-support/hive-personality.sh
|
| git revision | master / 305e710 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common U:
standalone-metastore/metastore-common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19201/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improve Messages Emitted From HiveMetaStoreClient
> -
>
> Key: HIVE-22419
> URL: https://issues.apache.org/jira/browse/HIVE-22419
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22419.1.patch
>
>
> After reviewing some logs and errors emitted during a QTest run, I would like
> to propose some improvements to logging in {{HiveMetaStoreClient}}.
> * Remove duplicate logging
> * Remove superfluous class {{StackTraceLogger}}
> * Do not use contractions in public-facing error messages and logs
> * Make all logging side-effect free (see {{connCount}})
> * Code simplification
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22390) Remove Dependency on JODA Time Library
[
https://issues.apache.org/jira/browse/HIVE-22390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962618#comment-16962618
]
David Mollitor commented on HIVE-22390:
---
[~Absolutesantaja] Hive team informed me 4.0 is slated for JDK 11. :)
> Remove Dependency on JODA Time Library
> --
>
> Key: HIVE-22390
> URL: https://issues.apache.org/jira/browse/HIVE-22390
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
>
> Hive uses Joda time library.
> {quote}
> Joda-Time is the de facto standard date and time library for Java prior to
> Java SE 8. Users are now asked to migrate to java.time (JSR-310).
> https://www.joda.org/joda-time/
> {quote}
> Remove this dependency from classes, POM files, and licence files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22403) Print ENV Variables In Command Line Help Debug Mode
[
https://issues.apache.org/jira/browse/HIVE-22403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962615#comment-16962615
]
David Mollitor commented on HIVE-22403:
---
[~pvary] Review please? :)
> Print ENV Variables In Command Line Help Debug Mode
> ---
>
> Key: HIVE-22403
> URL: https://issues.apache.org/jira/browse/HIVE-22403
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Affects Versions: 2.4.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22403.1.patch
>
>
> Beeline should print the various CONF and HOME directories it is utilizing
> when it starts up.
>
>
> {code:none}
> > export DEBUG=1; beeline --help
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22417) Remove stringifyException from MetaStore
[ https://issues.apache.org/jira/browse/HIVE-22417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962612#comment-16962612 ] David Mollitor commented on HIVE-22417: --- [~pvary] [~gopalv] > Remove stringifyException from MetaStore > > > Key: HIVE-22417 > URL: https://issues.apache.org/jira/browse/HIVE-22417 > Project: Hive > Issue Type: Sub-task > Components: Metastore, Standalone Metastore >Affects Versions: 3.2.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-22417.1.patch, HIVE-22417.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22411) Performance degradation on single row inserts
[ https://issues.apache.org/jira/browse/HIVE-22411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962611#comment-16962611 ] Rajesh Balamohan commented on HIVE-22411: - [~amagyar]: Is this with non-partitioned table? > Performance degradation on single row inserts > - > > Key: HIVE-22411 > URL: https://issues.apache.org/jira/browse/HIVE-22411 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: Screen Shot 2019-10-17 at 8.40.50 PM.png > > > Executing single insert statements on a transactional table effects write > performance on a s3 file system. Each insert creates a new delta directory. > After each insert hive calculates statistics like number of file in the table > and total size of the table. In order to calculate these, it traverses the > directory recursively. During the recursion for each path a separate > listStatus call is executed. In the end the more delta directory you have the > more time it takes to calculate the statistics. > Therefore insertion time goes up linearly: > !Screen Shot 2019-10-17 at 8.40.50 PM.png|width=601,height=436! > The fix is to use fs.listFiles(path, /**recursive**/ true) instead the > handcrafter recursive method/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22402) Deprecate and Replace Hive PerfLogger
[
https://issues.apache.org/jira/browse/HIVE-22402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-22402:
--
Attachment: HIVE-22402.4.patch
> Deprecate and Replace Hive PerfLogger
> -
>
> Key: HIVE-22402
> URL: https://issues.apache.org/jira/browse/HIVE-22402
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-22402.1.patch, HIVE-22402.2.patch,
> HIVE-22402.3.patch, HIVE-22402.4.patch, HIVE-22402.4.patch, HIVE-22402.4.patch
>
>
> Recently I wanted to add some additional capability, and add more,
> performance logging to support my troubleshooting efforts. I started looking
> at PerfLogger and started to examine its usage. I discovered a few things:
> # Since 'loggers' must be open and closed manually, I found a couple of
> places where loggers were opened, but not closed, rendering them useless
> # Since 'loggers' must be closed manually, I found a few places where an
> early-return or Exception thrown would cause a logger to not be closed,
> thereby rendering it useless
> # Session information is not logged, so it can be difficult to precisely
> pinpoint which session is taking lots of time
> # PerfLogger overloaded. Most of the time, it's being used as a simple timer
> mechanism with automatic logging in SLF4J debug. However, it is also a facade
> over the Hive Metrics subsystem and timing results are automatically
> published to Metrics and then there becomes this dependency on a 'logger' to
> be able to access metric data as well.
> The last bullet is the most challenging part and why I propose to deprecate
> the Hive {{PerfLogger}} and not simply remove it. I am proposing a new
> system... a {{PerfTimer}} that is allows for Java 8's try-with-resources
> feature to protect against the developer having to care about manually close
> measurements and not having to carefully consider all early-exits. The base
> implementation logs to SLF4J. An extended version automatically publishes to
> the Hive Metric subsystem as well.
> The current Hive {{PerfLogger}} has a bit of a clunky system for allowing
> plugable implementations. However, the current default implementation has a
> side-effect of also publishing timing information to the Hive Metrics
> subsystem. There are code sections that look up various timers in the Metrics
> Subsytem and publish the results back to the client. Since, in theory, the
> implementation is plugable, any other implementation that does not also have
> this side-effect of also publishing to the Metrics Subsystem will break these
> non-optional code paths. Also, these code paths create and interact with
> {{PerfLoggers in a static way, and then the publishing code pulls the data
> from the {{PerfLogger (as a facade to the Metrics subsystem) in a static
> way. Therefore, when I tried to replace the entire {{PerfLogger}} code, I
> came across an issue because there is not (and should not) be a way to just
> statically pull this information down from any point in the code. Information
> that is required for publishing should be passed around within some sort of
> context object, separate from the Metrics subsystem. There was no obvious way
> to string a new {{PerfTimer}} to all the required locations. I propose
> marking the {{PerfLogger}} as deprecated and leaving these complex section
> alone. Instead, replace only the simple "I want a timer" use cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22402) Deprecate and Replace Hive PerfLogger
[
https://issues.apache.org/jira/browse/HIVE-22402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-22402:
--
Status: Open (was: Patch Available)
> Deprecate and Replace Hive PerfLogger
> -
>
> Key: HIVE-22402
> URL: https://issues.apache.org/jira/browse/HIVE-22402
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-22402.1.patch, HIVE-22402.2.patch,
> HIVE-22402.3.patch, HIVE-22402.4.patch, HIVE-22402.4.patch, HIVE-22402.4.patch
>
>
> Recently I wanted to add some additional capability, and add more,
> performance logging to support my troubleshooting efforts. I started looking
> at PerfLogger and started to examine its usage. I discovered a few things:
> # Since 'loggers' must be open and closed manually, I found a couple of
> places where loggers were opened, but not closed, rendering them useless
> # Since 'loggers' must be closed manually, I found a few places where an
> early-return or Exception thrown would cause a logger to not be closed,
> thereby rendering it useless
> # Session information is not logged, so it can be difficult to precisely
> pinpoint which session is taking lots of time
> # PerfLogger overloaded. Most of the time, it's being used as a simple timer
> mechanism with automatic logging in SLF4J debug. However, it is also a facade
> over the Hive Metrics subsystem and timing results are automatically
> published to Metrics and then there becomes this dependency on a 'logger' to
> be able to access metric data as well.
> The last bullet is the most challenging part and why I propose to deprecate
> the Hive {{PerfLogger}} and not simply remove it. I am proposing a new
> system... a {{PerfTimer}} that is allows for Java 8's try-with-resources
> feature to protect against the developer having to care about manually close
> measurements and not having to carefully consider all early-exits. The base
> implementation logs to SLF4J. An extended version automatically publishes to
> the Hive Metric subsystem as well.
> The current Hive {{PerfLogger}} has a bit of a clunky system for allowing
> plugable implementations. However, the current default implementation has a
> side-effect of also publishing timing information to the Hive Metrics
> subsystem. There are code sections that look up various timers in the Metrics
> Subsytem and publish the results back to the client. Since, in theory, the
> implementation is plugable, any other implementation that does not also have
> this side-effect of also publishing to the Metrics Subsystem will break these
> non-optional code paths. Also, these code paths create and interact with
> {{PerfLoggers in a static way, and then the publishing code pulls the data
> from the {{PerfLogger (as a facade to the Metrics subsystem) in a static
> way. Therefore, when I tried to replace the entire {{PerfLogger}} code, I
> came across an issue because there is not (and should not) be a way to just
> statically pull this information down from any point in the code. Information
> that is required for publishing should be passed around within some sort of
> context object, separate from the Metrics subsystem. There was no obvious way
> to string a new {{PerfTimer}} to all the required locations. I propose
> marking the {{PerfLogger}} as deprecated and leaving these complex section
> alone. Instead, replace only the simple "I want a timer" use cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22402) Deprecate and Replace Hive PerfLogger
[
https://issues.apache.org/jira/browse/HIVE-22402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-22402:
--
Status: Patch Available (was: Open)
> Deprecate and Replace Hive PerfLogger
> -
>
> Key: HIVE-22402
> URL: https://issues.apache.org/jira/browse/HIVE-22402
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-22402.1.patch, HIVE-22402.2.patch,
> HIVE-22402.3.patch, HIVE-22402.4.patch, HIVE-22402.4.patch, HIVE-22402.4.patch
>
>
> Recently I wanted to add some additional capability, and add more,
> performance logging to support my troubleshooting efforts. I started looking
> at PerfLogger and started to examine its usage. I discovered a few things:
> # Since 'loggers' must be open and closed manually, I found a couple of
> places where loggers were opened, but not closed, rendering them useless
> # Since 'loggers' must be closed manually, I found a few places where an
> early-return or Exception thrown would cause a logger to not be closed,
> thereby rendering it useless
> # Session information is not logged, so it can be difficult to precisely
> pinpoint which session is taking lots of time
> # PerfLogger overloaded. Most of the time, it's being used as a simple timer
> mechanism with automatic logging in SLF4J debug. However, it is also a facade
> over the Hive Metrics subsystem and timing results are automatically
> published to Metrics and then there becomes this dependency on a 'logger' to
> be able to access metric data as well.
> The last bullet is the most challenging part and why I propose to deprecate
> the Hive {{PerfLogger}} and not simply remove it. I am proposing a new
> system... a {{PerfTimer}} that is allows for Java 8's try-with-resources
> feature to protect against the developer having to care about manually close
> measurements and not having to carefully consider all early-exits. The base
> implementation logs to SLF4J. An extended version automatically publishes to
> the Hive Metric subsystem as well.
> The current Hive {{PerfLogger}} has a bit of a clunky system for allowing
> plugable implementations. However, the current default implementation has a
> side-effect of also publishing timing information to the Hive Metrics
> subsystem. There are code sections that look up various timers in the Metrics
> Subsytem and publish the results back to the client. Since, in theory, the
> implementation is plugable, any other implementation that does not also have
> this side-effect of also publishing to the Metrics Subsystem will break these
> non-optional code paths. Also, these code paths create and interact with
> {{PerfLoggers in a static way, and then the publishing code pulls the data
> from the {{PerfLogger (as a facade to the Metrics subsystem) in a static
> way. Therefore, when I tried to replace the entire {{PerfLogger}} code, I
> came across an issue because there is not (and should not) be a way to just
> statically pull this information down from any point in the code. Information
> that is required for publishing should be passed around within some sort of
> context object, separate from the Metrics subsystem. There was no obvious way
> to string a new {{PerfTimer}} to all the required locations. I propose
> marking the {{PerfLogger}} as deprecated and leaving these complex section
> alone. Instead, replace only the simple "I want a timer" use cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21884) Scheduled query support
[
https://issues.apache.org/jira/browse/HIVE-21884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962584#comment-16962584
]
Hive QA commented on HIVE-21884:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12984279/HIVE-21884.25.patch
{color:green}SUCCESS:{color} +1 due to 20 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 17585 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.client.TestMetastoreScheduledQueries.testCleanup[Embedded]
(batchId=225)
org.apache.hadoop.hive.metastore.client.TestMetastoreScheduledQueries.testCleanup[Remote]
(batchId=225)
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.org.apache.hadoop.hive.ql.TestWarehouseExternalDir
(batchId=275)
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.testExternalDefaultPaths
(batchId=275)
org.apache.hadoop.hive.ql.TestWarehouseExternalDir.testManagedPaths
(batchId=275)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19200/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19200/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19200/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12984279 - PreCommit-HIVE-Build
> Scheduled query support
> ---
>
> Key: HIVE-21884
> URL: https://issues.apache.org/jira/browse/HIVE-21884
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21844.04.patch, HIVE-21844.05.patch,
> HIVE-21844.06.patch, HIVE-21844.07.patch, HIVE-21844.08.patch,
> HIVE-21844.09.patch, HIVE-21844.15.patch, HIVE-21844.19.patch,
> HIVE-21844.21.patch, HIVE-21844.22.patch, HIVE-21884.01.patch,
> HIVE-21884.02.patch, HIVE-21884.03.patch, HIVE-21884.09.patch,
> HIVE-21884.10.patch, HIVE-21884.10.patch, HIVE-21884.11.patch,
> HIVE-21884.12.patch, HIVE-21884.13.patch, HIVE-21884.14.patch,
> HIVE-21884.14.patch, HIVE-21884.14.patch, HIVE-21884.16.patch,
> HIVE-21884.17.patch, HIVE-21884.18.patch, HIVE-21884.20.patch,
> HIVE-21884.23.patch, HIVE-21884.24.patch, HIVE-21884.25.patch, Scheduled
> queries2.pdf
>
> Time Spent: 7h
> Remaining Estimate: 0h
>
> design document:
> https://docs.google.com/document/d/1mJSFdJi_1cbxJTXC9QvGw2rQ3zzJkNfxOO6b5esmyCE/edit#
> in case the google doc is not reachable: [^Scheduled queries2.pdf]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22403) Print ENV Variables In Command Line Help Debug Mode
[
https://issues.apache.org/jira/browse/HIVE-22403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962546#comment-16962546
]
Hive QA commented on HIVE-22403:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12984273/HIVE-22403.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17546 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19199/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19199/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19199/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12984273 - PreCommit-HIVE-Build
> Print ENV Variables In Command Line Help Debug Mode
> ---
>
> Key: HIVE-22403
> URL: https://issues.apache.org/jira/browse/HIVE-22403
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Affects Versions: 2.4.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22403.1.patch
>
>
> Beeline should print the various CONF and HOME directories it is utilizing
> when it starts up.
>
>
> {code:none}
> > export DEBUG=1; beeline --help
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22429) Migrated clustered tables using bucketing_version 1 on hive 3 uses bucketing_version 2 for inserts
[ https://issues.apache.org/jira/browse/HIVE-22429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22429: Description: Migrated clustered tables using bucketing_version 1 on hive 3 uses bucketing_version 2 for inserts (was: Migrated clustered tables using bucketing_version 1 on hive 3 uses bucketing_version 2) Summary: Migrated clustered tables using bucketing_version 1 on hive 3 uses bucketing_version 2 for inserts (was: Migrated clustered tables using bucketing_version 1 on hive 3 uses bucketing_version 2) > Migrated clustered tables using bucketing_version 1 on hive 3 uses > bucketing_version 2 for inserts > -- > > Key: HIVE-22429 > URL: https://issues.apache.org/jira/browse/HIVE-22429 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22429.1.patch > > > Migrated clustered tables using bucketing_version 1 on hive 3 uses > bucketing_version 2 for inserts -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22429) Migrated clustered tables using bucketing_version 1 on hive 3 uses bucketing_version 2
[ https://issues.apache.org/jira/browse/HIVE-22429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22429: Attachment: HIVE-22429.1.patch Status: Patch Available (was: Open) > Migrated clustered tables using bucketing_version 1 on hive 3 uses > bucketing_version 2 > -- > > Key: HIVE-22429 > URL: https://issues.apache.org/jira/browse/HIVE-22429 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22429.1.patch > > > Migrated clustered tables using bucketing_version 1 on hive 3 uses > bucketing_version 2 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22429) Migrated clustered tables using bucketing_version 1 on hive 3 uses bucketing_version 2
[ https://issues.apache.org/jira/browse/HIVE-22429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan reassigned HIVE-22429: --- Assignee: Ramesh Kumar Thangarajan > Migrated clustered tables using bucketing_version 1 on hive 3 uses > bucketing_version 2 > -- > > Key: HIVE-22429 > URL: https://issues.apache.org/jira/browse/HIVE-22429 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > > Migrated clustered tables using bucketing_version 1 on hive 3 uses > bucketing_version 2 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22357) Schema mismatch between the Hive table definition and the "hive.sql.query" Parameter
[ https://issues.apache.org/jira/browse/HIVE-22357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22357: --- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master, thanks [~Rajkumar Singh]! > Schema mismatch between the Hive table definition and the "hive.sql.query" > Parameter > > > Key: HIVE-22357 > URL: https://issues.apache.org/jira/browse/HIVE-22357 > Project: Hive > Issue Type: Improvement > Components: Hive, HiveServer2 >Affects Versions: 3.1.0 >Reporter: Krishnadevan Purushothaman >Assignee: Rajkumar Singh >Priority: Critical > Fix For: 4.0.0 > > Attachments: HIVE-22357.patch, Hive-JDBC schema matching > sensitivity.txt > > > The problem is that, for certain types of schema mismatch between the Hive > table definition and the "hive.sql.query" parameter, Hive simply returns no > rows rather than throwing an error when queried. > Ideally Hive would check for schema compatibility during table definition and > throw an error if there's a problem - the earlier the better. But even if > that cannot be made a requirement, I'd definitely expect an error rather than > a silent failure (i.e. zero rows returned) at runtime. > I'm attaching investigation of this issue in "Hive-JDBC schema matching > sensitivity.txt". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22311) Propagate min/max column values from statistics to the optimizer for timestamp type
[ https://issues.apache.org/jira/browse/HIVE-22311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962513#comment-16962513 ] Jesus Camacho Rodriguez commented on HIVE-22311: [~kgyrtkirk], [~mgergely], could you review? https://github.com/apache/hive/pull/821 Thanks > Propagate min/max column values from statistics to the optimizer for > timestamp type > --- > > Key: HIVE-22311 > URL: https://issues.apache.org/jira/browse/HIVE-22311 > Project: Hive > Issue Type: Bug > Components: Statistics >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22311.01.patch, HIVE-22311.01.patch, > HIVE-22311.02.patch, HIVE-22311.03.patch, HIVE-22311.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Currently stats annotation does not consider timestamp type e.g. for > estimates with range predicates. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22357) Schema mismatch between the Hive table definition and the "hive.sql.query" Parameter
[ https://issues.apache.org/jira/browse/HIVE-22357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962511#comment-16962511 ] Jesus Camacho Rodriguez commented on HIVE-22357: +1 > Schema mismatch between the Hive table definition and the "hive.sql.query" > Parameter > > > Key: HIVE-22357 > URL: https://issues.apache.org/jira/browse/HIVE-22357 > Project: Hive > Issue Type: Improvement > Components: Hive, HiveServer2 >Affects Versions: 3.1.0 >Reporter: Krishnadevan Purushothaman >Assignee: Rajkumar Singh >Priority: Critical > Attachments: HIVE-22357.patch, Hive-JDBC schema matching > sensitivity.txt > > > The problem is that, for certain types of schema mismatch between the Hive > table definition and the "hive.sql.query" parameter, Hive simply returns no > rows rather than throwing an error when queried. > Ideally Hive would check for schema compatibility during table definition and > throw an error if there's a problem - the earlier the better. But even if > that cannot be made a requirement, I'd definitely expect an error rather than > a silent failure (i.e. zero rows returned) at runtime. > I'm attaching investigation of this issue in "Hive-JDBC schema matching > sensitivity.txt". -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22394) Duplicate Jars in druid classpath causing issues
[
https://issues.apache.org/jira/browse/HIVE-22394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-22394:
---
Fix Version/s: 4.0.0
> Duplicate Jars in druid classpath causing issues
>
>
> Key: HIVE-22394
> URL: https://issues.apache.org/jira/browse/HIVE-22394
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22394.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> hive-druid-handler jar has shaded version of druid classes,
> druid-hdfs-storage also has non-shaded classes.
> {code}
> [hive@hiveserver2-1 lib]$ ls |grep druid
> calcite-druid-1.19.0.7.0.2.0-163.jar
> druid-bloom-filter-0.15.1.7.0.2.0-163.jar
> druid-hdfs-storage-0.15.1.7.0.2.0-163.jar
> hive-druid-handler-3.1.2000.7.0.2.0-163.jar
> hive-druid-handler.jar
> {code}
> Exception below -
> {code}
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.fs.HadoopFsWrapper
> at
> org.apache.hive.druid.com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> at
> org.apache.hive.druid.com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> at
> org.apache.hive.druid.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> at
> org.apache.hadoop.hive.druid.io.DruidRecordWriter.pushSegments(DruidRecordWriter.java:177)
> ... 22 more
> Caused by: java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.fs.HadoopFsWrapper
> at
> org.apache.hive.druid.org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:765)
> at
> org.apache.hive.druid.org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:630)
> at
> org.apache.hive.druid.com.google.common.util.concurrent.Futures$1.apply(Futures.java:713)
> at
> org.apache.hive.druid.com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861)
> ... 3 more
> Caused by: java.lang.RuntimeException: java.lang.NoClassDefFoundError: Could
> not initialize class org.apache.hadoop.fs.HadoopFsWrapper
> at
> org.apache.hive.druid.org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:96)
> at
> org.apache.hive.druid.org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:114)
> at
> org.apache.hive.druid.org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:104)
> at
> org.apache.hive.druid.org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:743)
> ... 6 more
> Caused by: java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.fs.HadoopFsWrapper
> at
> org.apache.hive.druid.org.apache.druid.storage.hdfs.HdfsDataSegmentPusher.copyFilesWithChecks(HdfsDataSegmentPusher.java:163)
> at
> org.apache.hive.druid.org.apache.druid.storage.hdfs.HdfsDataSegmentPusher.push(HdfsDataSegmentPusher.java:145)
> at
> org.apache.hive.druid.org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$mergeAndPush$4(AppenderatorImpl.java:747)
> at
> org.apache.hive.druid.org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22403) Print ENV Variables In Command Line Help Debug Mode
[
https://issues.apache.org/jira/browse/HIVE-22403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962508#comment-16962508
]
Hive QA commented on HIVE-22403:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
58s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19199/dev-support/hive-personality.sh
|
| git revision | master / 1f5a206 |
| modules | C: . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19199/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Print ENV Variables In Command Line Help Debug Mode
> ---
>
> Key: HIVE-22403
> URL: https://issues.apache.org/jira/browse/HIVE-22403
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Affects Versions: 2.4.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22403.1.patch
>
>
> Beeline should print the various CONF and HOME directories it is utilizing
> when it starts up.
>
>
> {code:none}
> > export DEBUG=1; beeline --help
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22414) Make LLAP CacheTags more memory efficient
[
https://issues.apache.org/jira/browse/HIVE-22414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962505#comment-16962505
]
Hive QA commented on HIVE-22414:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12984242/HIVE-22414.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19198/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19198/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19198/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12984242/HIVE-22414.1.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12984242 - PreCommit-HIVE-Build
> Make LLAP CacheTags more memory efficient
> -
>
> Key: HIVE-22414
> URL: https://issues.apache.org/jira/browse/HIVE-22414
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Attachments: HIVE-22414.0.patch, HIVE-22414.1.patch
>
>
> MultiPartitionCacheTag relies on LinkedLists.
> A LinkedList object that holds e.g. 2 nodes consumes 112 bytes roughly in
> this composition:
> * 16 bytes for LinkedList object header
> * 8 bytes for referring head
> * 8 bytes for referring tail
> * 80 = 2 x (16 bytes for LinkedList$Node header, 3 x 8 bytes for referring
> prev, next, item)
> This is a lot, so I propose to replace LinkedList in MultiPartitionCacheTag
> with a simple String array. (For a similar scenario an array would take 16 +
> 8 + 2 x 8 = 40 bytes, as per header, count, and 2 references for our actual
> objects).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22402) Deprecate and Replace Hive PerfLogger
[
https://issues.apache.org/jira/browse/HIVE-22402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962504#comment-16962504
]
Hive QA commented on HIVE-22402:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12984271/HIVE-22402.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17490 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat.org.apache.hadoop.hive.ql.parse.TestReplWithJsonMessageFormat
(batchId=253)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19197/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19197/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19197/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12984271 - PreCommit-HIVE-Build
> Deprecate and Replace Hive PerfLogger
> -
>
> Key: HIVE-22402
> URL: https://issues.apache.org/jira/browse/HIVE-22402
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 4.0.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-22402.1.patch, HIVE-22402.2.patch,
> HIVE-22402.3.patch, HIVE-22402.4.patch, HIVE-22402.4.patch
>
>
> Recently I wanted to add some additional capability, and add more,
> performance logging to support my troubleshooting efforts. I started looking
> at PerfLogger and started to examine its usage. I discovered a few things:
> # Since 'loggers' must be open and closed manually, I found a couple of
> places where loggers were opened, but not closed, rendering them useless
> # Since 'loggers' must be closed manually, I found a few places where an
> early-return or Exception thrown would cause a logger to not be closed,
> thereby rendering it useless
> # Session information is not logged, so it can be difficult to precisely
> pinpoint which session is taking lots of time
> # PerfLogger overloaded. Most of the time, it's being used as a simple timer
> mechanism with automatic logging in SLF4J debug. However, it is also a facade
> over the Hive Metrics subsystem and timing results are automatically
> published to Metrics and then there becomes this dependency on a 'logger' to
> be able to access metric data as well.
> The last bullet is the most challenging part and why I propose to deprecate
> the Hive {{PerfLogger}} and not simply remove it. I am proposing a new
> system... a {{PerfTimer}} that is allows for Java 8's try-with-resources
> feature to protect against the developer having to care about manually close
> measurements and not having to carefully consider all early-exits. The base
> implementation logs to SLF4J. An extended version automatically publishes to
> the Hive Metric subsystem as well.
> The current Hive {{PerfLogger}} has a bit of a clunky system for allowing
> plugable implementations. However, the current default implementation has a
> side-effect of also publishing timing information to the Hive Metrics
> subsystem. There are code sections that look up various timers in the Metrics
> Subsytem and publish the results back to the client. Since, in theory, the
> implementation is plugable, any other implementation that does not also have
> this side-effect of also publishing to the Metrics Subsystem will break these
> non-optional code paths. Also, these code paths create and interact with
> {{PerfLoggers in a static way, and then the publishing code pulls the data
> from the {{PerfLogger (as a facade to the Metrics subsystem) in a static
> way. Therefore, when I tried to replace the entire {{PerfLogger}} code, I
> came across an issue because there is not (and should not) be a way to just
> statically pull this information down from any point in the code. Information
> that is required for publishing should be passed around within some sort of
> context object, separate from the Metrics subsystem. There was no obvious way
> to string a new {{PerfTimer}} to all the required locations. I propose
> marking the {{PerfLogger}} as deprecated and leaving these complex section
> alone. Instead, replace only the simple "I want a timer" use cases.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22394) Duplicate Jars in druid classpath causing issues
[
https://issues.apache.org/jira/browse/HIVE-22394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962490#comment-16962490
]
Nishant Bangarwa commented on HIVE-22394:
-
merged to master via 1f5a2064aa1e2466901548991ab8720c0f8c6ddb
https://github.com/apache/hive/commit/1f5a2064aa1e2466901548991ab8720c0f8c6ddb
> Duplicate Jars in druid classpath causing issues
>
>
> Key: HIVE-22394
> URL: https://issues.apache.org/jira/browse/HIVE-22394
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22394.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> hive-druid-handler jar has shaded version of druid classes,
> druid-hdfs-storage also has non-shaded classes.
> {code}
> [hive@hiveserver2-1 lib]$ ls |grep druid
> calcite-druid-1.19.0.7.0.2.0-163.jar
> druid-bloom-filter-0.15.1.7.0.2.0-163.jar
> druid-hdfs-storage-0.15.1.7.0.2.0-163.jar
> hive-druid-handler-3.1.2000.7.0.2.0-163.jar
> hive-druid-handler.jar
> {code}
> Exception below -
> {code}
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.fs.HadoopFsWrapper
> at
> org.apache.hive.druid.com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> at
> org.apache.hive.druid.com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> at
> org.apache.hive.druid.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> at
> org.apache.hadoop.hive.druid.io.DruidRecordWriter.pushSegments(DruidRecordWriter.java:177)
> ... 22 more
> Caused by: java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.fs.HadoopFsWrapper
> at
> org.apache.hive.druid.org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:765)
> at
> org.apache.hive.druid.org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:630)
> at
> org.apache.hive.druid.com.google.common.util.concurrent.Futures$1.apply(Futures.java:713)
> at
> org.apache.hive.druid.com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861)
> ... 3 more
> Caused by: java.lang.RuntimeException: java.lang.NoClassDefFoundError: Could
> not initialize class org.apache.hadoop.fs.HadoopFsWrapper
> at
> org.apache.hive.druid.org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:96)
> at
> org.apache.hive.druid.org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:114)
> at
> org.apache.hive.druid.org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:104)
> at
> org.apache.hive.druid.org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:743)
> ... 6 more
> Caused by: java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.fs.HadoopFsWrapper
> at
> org.apache.hive.druid.org.apache.druid.storage.hdfs.HdfsDataSegmentPusher.copyFilesWithChecks(HdfsDataSegmentPusher.java:163)
> at
> org.apache.hive.druid.org.apache.druid.storage.hdfs.HdfsDataSegmentPusher.push(HdfsDataSegmentPusher.java:145)
> at
> org.apache.hive.druid.org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$mergeAndPush$4(AppenderatorImpl.java:747)
> at
> org.apache.hive.druid.org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22394) Duplicate Jars in druid classpath causing issues
[
https://issues.apache.org/jira/browse/HIVE-22394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa updated HIVE-22394:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Duplicate Jars in druid classpath causing issues
>
>
> Key: HIVE-22394
> URL: https://issues.apache.org/jira/browse/HIVE-22394
> Project: Hive
> Issue Type: Bug
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22394.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> hive-druid-handler jar has shaded version of druid classes,
> druid-hdfs-storage also has non-shaded classes.
> {code}
> [hive@hiveserver2-1 lib]$ ls |grep druid
> calcite-druid-1.19.0.7.0.2.0-163.jar
> druid-bloom-filter-0.15.1.7.0.2.0-163.jar
> druid-hdfs-storage-0.15.1.7.0.2.0-163.jar
> hive-druid-handler-3.1.2000.7.0.2.0-163.jar
> hive-druid-handler.jar
> {code}
> Exception below -
> {code}
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.fs.HadoopFsWrapper
> at
> org.apache.hive.druid.com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> at
> org.apache.hive.druid.com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> at
> org.apache.hive.druid.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> at
> org.apache.hadoop.hive.druid.io.DruidRecordWriter.pushSegments(DruidRecordWriter.java:177)
> ... 22 more
> Caused by: java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.fs.HadoopFsWrapper
> at
> org.apache.hive.druid.org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:765)
> at
> org.apache.hive.druid.org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$push$1(AppenderatorImpl.java:630)
> at
> org.apache.hive.druid.com.google.common.util.concurrent.Futures$1.apply(Futures.java:713)
> at
> org.apache.hive.druid.com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861)
> ... 3 more
> Caused by: java.lang.RuntimeException: java.lang.NoClassDefFoundError: Could
> not initialize class org.apache.hadoop.fs.HadoopFsWrapper
> at
> org.apache.hive.druid.org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:96)
> at
> org.apache.hive.druid.org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:114)
> at
> org.apache.hive.druid.org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:104)
> at
> org.apache.hive.druid.org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:743)
> ... 6 more
> Caused by: java.lang.NoClassDefFoundError: Could not initialize class
> org.apache.hadoop.fs.HadoopFsWrapper
> at
> org.apache.hive.druid.org.apache.druid.storage.hdfs.HdfsDataSegmentPusher.copyFilesWithChecks(HdfsDataSegmentPusher.java:163)
> at
> org.apache.hive.druid.org.apache.druid.storage.hdfs.HdfsDataSegmentPusher.push(HdfsDataSegmentPusher.java:145)
> at
> org.apache.hive.druid.org.apache.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$mergeAndPush$4(AppenderatorImpl.java:747)
> at
> org.apache.hive.druid.org.apache.druid.java.util.common.RetryUtils.retry(RetryUtils.java:86)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22402) Deprecate and Replace Hive PerfLogger
[
https://issues.apache.org/jira/browse/HIVE-22402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962481#comment-16962481
]
Hive QA commented on HIVE-22402:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
52s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
58s{color} | {color:blue} ql in master has 1545 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} common: The patch generated 1 new + 372 unchanged - 2
fixed = 373 total (was 374) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
48s{color} | {color:red} ql: The patch generated 5 new + 810 unchanged - 40
fixed = 815 total (was 850) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
22s{color} | {color:red} ql generated 2 new + 1540 unchanged - 5 fixed = 1542
total (was 1545) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Exception is caught when Exception is not thrown in
org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.doInit(JobConf,
OutputCollector, Reporter) At SparkReduceRecordHandler.java:is not thrown in
org.apache.hadoop.hive.ql.exec.spark.SparkReduceRecordHandler.doInit(JobConf,
OutputCollector, Reporter) At SparkReduceRecordHandler.java:[line 230] |
| |
org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.doInit(MRTaskReporter,
Map, Map) makes inefficient use of keySet iterator instead of entrySet iterator
At ReduceRecordProcessor.java:use of keySet iterator instead of entrySet
iterator At ReduceRecordProcessor.java:[line 189] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19197/dev-support/hive-personality.sh
|
| git revision | master / 6d0f461 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19197/yetus/diff-checkstyle-common.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19197/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19197/yetus/new-findbugs-ql.html
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19197/yetus.txt |
| Powered by | Apache Yetus
[jira] [Updated] (HIVE-21954) QTest: support for running qtests on various metastore DBs
[
https://issues.apache.org/jira/browse/HIVE-21954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-21954:
Attachment: HIVE-21954.05.patch
> QTest: support for running qtests on various metastore DBs
> --
>
> Key: HIVE-21954
> URL: https://issues.apache.org/jira/browse/HIVE-21954
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore, Testing Infrastructure
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21954.01.patch, HIVE-21954.02.patch,
> HIVE-21954.03.patch, HIVE-21954.03.patch, HIVE-21954.03.patch,
> HIVE-21954.04.patch, HIVE-21954.05.patch
>
>
> In HIVE-21940, a postgres metastore related issue has been fixed, and a local
> reproduction has been provided.
> {code}
> export QTEST_LEAVE_FILES=true
> docker kill metastore-test-postgres-install
> docker rm metastore-test-postgres-install
> cd standalone-metastore
> mvn verify -DskipITests=false -Dit.test=ITestPostgres#install -Dtest=nosuch
> -Dmetastore.itest.no.stop.container=true
> cd ..
> mvn test -Dtest.output.overwrite=true -Pitests,hadoop-2 -pl itests/qtest
> -Dtest=TestCliDriver -Dqfile=partition_params_postgres.q
> -Dhive.metastore.rawstore.impl=org.apache.hadoop.hive.metastore.ObjectStore
> {code}
> The problem with this solution is that data/conf/hive-site.xml has to be
> edited manually. My proposal is to introduce a property
> (-Dmetastore.db=postgres), which can take care of the parameters on the fly.
> 2 supported solutions could be:
> 1. simple parameters: -Dmetastore.db=postgres
> In this case, tests depend on settings from ITestPostgres class (password,
> db, etc.)
> 2. verbose but flexible parameters: [see hive-site.xml HIVE-21940's repro
> patch|https://issues.apache.org/jira/secure/attachment/12973534/HIVE-21940.repro.patch]
>
> In the first implementation, I would not start metastore db automatically
> (which is done be 'mvn verify ...'), but it's still under planning.
> In the long term, we should consider running this kind of tests in precommit
> phase, so maybe -Dmetastore.db=postgres could start metastore db
> automatically. Also we should consider running some qtests on various
> metastores. I would not pick randomly, but choose some "metastore-heavy" ones
> instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22417) Remove stringifyException from MetaStore
[
https://issues.apache.org/jira/browse/HIVE-22417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962466#comment-16962466
]
Hive QA commented on HIVE-22417:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12984269/HIVE-22417.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17546 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19196/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19196/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19196/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12984269 - PreCommit-HIVE-Build
> Remove stringifyException from MetaStore
>
>
> Key: HIVE-22417
> URL: https://issues.apache.org/jira/browse/HIVE-22417
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore, Standalone Metastore
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-22417.1.patch, HIVE-22417.2.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22417) Remove stringifyException from MetaStore
[
https://issues.apache.org/jira/browse/HIVE-22417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962436#comment-16962436
]
Hive QA commented on HIVE-22417:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
27s{color} | {color:blue} standalone-metastore/metastore-common in master has
37 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
9s{color} | {color:blue} standalone-metastore/metastore-server in master has
171 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 1545 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
36s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 8m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19196/dev-support/hive-personality.sh
|
| git revision | master / 6d0f461 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: standalone-metastore/metastore-common metastore
standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19196/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Remove stringifyException from MetaStore
>
>
> Key: HIVE-22417
> URL: https://issues.apache.org/jira/browse/HIVE-22417
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore, Standalone Metastore
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-22417.1.patch, HIVE-22417.2.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22363) ReduceDeduplication may leave an invalid GroupByOperator behind in some cases
[
https://issues.apache.org/jira/browse/HIVE-22363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962340#comment-16962340
]
Hive QA commented on HIVE-22363:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12984263/HIVE-22363.04.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 17546 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_7]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby10] (batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby11] (batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby8] (batchId=83)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[groupby2]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[ptf]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_windowing]
(batchId=187)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_ptf]
(batchId=179)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby10]
(batchId=141)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby11]
(batchId=146)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby2]
(batchId=135)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[groupby8]
(batchId=147)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ptf] (batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_ptf]
(batchId=142)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19195/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19195/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19195/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12984263 - PreCommit-HIVE-Build
> ReduceDeduplication may leave an invalid GroupByOperator behind in some cases
> -
>
> Key: HIVE-22363
> URL: https://issues.apache.org/jira/browse/HIVE-22363
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 3.1.2
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22363.01.patch, HIVE-22363.02.patch,
> HIVE-22363.03.patch, HIVE-22363.04.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> since HIVE-11387 reducededup may traverse {{GroupByOperators}} [as
> well|https://github.com/apache/hive/blob/c6626edb65c2cd00576647e54db1995628fe64da/ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/CorrelationUtilities.java#L244]
> But the removal logic only removes the first parent; so if there is some
> other operator (a FIL in this case) between the sink and the gby - the
> removal may not happen
> [here|https://github.com/apache/hive/blob/c6626edb65c2cd00576647e54db1995628fe64da/ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/CorrelationUtilities.java#L458]
> {code}
> set hive.cbo.enable=false;
> drop table if exists xl1;
> create table xl1 as
> select '1' as mdl_yr_desc, 2 as seq_no,'3' as opt_desc1,4 as opt_desc,1 as
> row_num;
> explain
> select trim(base.mdl_yr_desc) mdl_yr_desc, trim(base.opt_desc) opt_desc
> from
> (
> SELECT trim(mdl_yr_desc) mdl_yr_desc, concat_ws(' ',
> collect_set(trim(opt_desc1))) AS opt_desc
> from
> (
> select t14304.*
> from
> (
> select * from xl1
> ) t14304
> where row_num = 1
> order by trim(mdl_yr_desc), cast(seq_no as int) asc
> ) x
> group by trim(mdl_yr_desc)
> ) base
> inner join
> (
> select 1 as v
> ) dedup
> on trim(base.mdl_yr_desc) != dedup.v
> group by trim(base.mdl_yr_desc), trim(base.opt_desc) ;
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22382) Support Decimal64 column division with decimal64 Column
[ https://issues.apache.org/jira/browse/HIVE-22382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22382: Attachment: HIVE-22382.4.patch Status: Patch Available (was: Open) > Support Decimal64 column division with decimal64 Column > --- > > Key: HIVE-22382 > URL: https://issues.apache.org/jira/browse/HIVE-22382 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22382.2.patch, HIVE-22382.3.patch, > HIVE-22382.4.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Support Decimal64 column division with decimal64 Column -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22382) Support Decimal64 column division with decimal64 Column
[ https://issues.apache.org/jira/browse/HIVE-22382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22382: Status: Open (was: Patch Available) > Support Decimal64 column division with decimal64 Column > --- > > Key: HIVE-22382 > URL: https://issues.apache.org/jira/browse/HIVE-22382 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22382.2.patch, HIVE-22382.3.patch, > HIVE-22382.4.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Support Decimal64 column division with decimal64 Column -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22428) Superfluous "Failed to get database" WARN Logging in ObjectStore
[
https://issues.apache.org/jira/browse/HIVE-22428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-22428:
--
Status: Patch Available (was: Open)
> Superfluous "Failed to get database" WARN Logging in ObjectStore
>
>
> Key: HIVE-22428
> URL: https://issues.apache.org/jira/browse/HIVE-22428
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-22428.1.patch
>
>
> In my testing, I get lots of logs like this:
> {code:none}
> Line 26319: 2019-10-28T21:09:52,134 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.compdb, returning
> NoSuchObjectException
> Line 26327: 2019-10-28T21:09:52,135 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.compdb, returning
> NoSuchObjectException
> Line 26504: 2019-10-28T21:09:52,600 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.tstatsfast, returning
> NoSuchObjectException
> Line 26519: 2019-10-28T21:09:52,606 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.tstatsfast, returning
> NoSuchObjectException
> Line 26695: 2019-10-28T21:09:52,922 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.createDb, returning
> NoSuchObjectException
> Line 26703: 2019-10-28T21:09:52,923 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.createDb, returning
> NoSuchObjectException
> Line 26763: 2019-10-28T21:09:52,936 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.compdb, returning
> NoSuchObjectException
> Line 26778: 2019-10-28T21:09:52,939 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.compdb, returning
> NoSuchObjectException
> Line 26963: 2019-10-28T21:09:53,273 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.db1, returning
> NoSuchObjectException
> Line 26978: 2019-10-28T21:09:53,276 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.db2, returning
> NoSuchObjectException
> Line 26986: 2019-10-28T21:09:53,277 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.db1, returning
> NoSuchObjectException
> Line 27018: 2019-10-28T21:09:53,300 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.db2, returning
> NoSuchObjectException
> {code}
> This is a superfluous log message. It might be pretty common for a database
> to not exists if, for example, a user fat-fingers the name of the database.
> The code also has the bad habit of log-and-throw. Just log or throw, not
> both.
> Since I'm looking at this class, touch up some of the other logging as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22428) Superfluous "Failed to get database" WARN Logging in ObjectStore
[
https://issues.apache.org/jira/browse/HIVE-22428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor reassigned HIVE-22428:
-
> Superfluous "Failed to get database" WARN Logging in ObjectStore
>
>
> Key: HIVE-22428
> URL: https://issues.apache.org/jira/browse/HIVE-22428
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-22428.1.patch
>
>
> In my testing, I get lots of logs like this:
> {code:none}
> Line 26319: 2019-10-28T21:09:52,134 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.compdb, returning
> NoSuchObjectException
> Line 26327: 2019-10-28T21:09:52,135 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.compdb, returning
> NoSuchObjectException
> Line 26504: 2019-10-28T21:09:52,600 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.tstatsfast, returning
> NoSuchObjectException
> Line 26519: 2019-10-28T21:09:52,606 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.tstatsfast, returning
> NoSuchObjectException
> Line 26695: 2019-10-28T21:09:52,922 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.createDb, returning
> NoSuchObjectException
> Line 26703: 2019-10-28T21:09:52,923 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.createDb, returning
> NoSuchObjectException
> Line 26763: 2019-10-28T21:09:52,936 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.compdb, returning
> NoSuchObjectException
> Line 26778: 2019-10-28T21:09:52,939 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.compdb, returning
> NoSuchObjectException
> Line 26963: 2019-10-28T21:09:53,273 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.db1, returning
> NoSuchObjectException
> Line 26978: 2019-10-28T21:09:53,276 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.db2, returning
> NoSuchObjectException
> Line 26986: 2019-10-28T21:09:53,277 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.db1, returning
> NoSuchObjectException
> Line 27018: 2019-10-28T21:09:53,300 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.db2, returning
> NoSuchObjectException
> {code}
> This is a superfluous log message. It might be pretty common for a database
> to not exists if, for example, a user fat-fingers the name of the database.
> The code also has the bad habit of log-and-throw. Just log or throw, not
> both.
> Since I'm looking at this class, touch up some of the other logging as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22428) Superfluous "Failed to get database" WARN Logging in ObjectStore
[
https://issues.apache.org/jira/browse/HIVE-22428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-22428:
--
Attachment: HIVE-22428.1.patch
> Superfluous "Failed to get database" WARN Logging in ObjectStore
>
>
> Key: HIVE-22428
> URL: https://issues.apache.org/jira/browse/HIVE-22428
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Attachments: HIVE-22428.1.patch
>
>
> In my testing, I get lots of logs like this:
> {code:none}
> Line 26319: 2019-10-28T21:09:52,134 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.compdb, returning
> NoSuchObjectException
> Line 26327: 2019-10-28T21:09:52,135 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.compdb, returning
> NoSuchObjectException
> Line 26504: 2019-10-28T21:09:52,600 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.tstatsfast, returning
> NoSuchObjectException
> Line 26519: 2019-10-28T21:09:52,606 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.tstatsfast, returning
> NoSuchObjectException
> Line 26695: 2019-10-28T21:09:52,922 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.createDb, returning
> NoSuchObjectException
> Line 26703: 2019-10-28T21:09:52,923 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.createDb, returning
> NoSuchObjectException
> Line 26763: 2019-10-28T21:09:52,936 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.compdb, returning
> NoSuchObjectException
> Line 26778: 2019-10-28T21:09:52,939 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.compdb, returning
> NoSuchObjectException
> Line 26963: 2019-10-28T21:09:53,273 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.db1, returning
> NoSuchObjectException
> Line 26978: 2019-10-28T21:09:53,276 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.db2, returning
> NoSuchObjectException
> Line 26986: 2019-10-28T21:09:53,277 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.db1, returning
> NoSuchObjectException
> Line 27018: 2019-10-28T21:09:53,300 WARN [pool-6-thread-5]
> metastore.ObjectStore: Failed to get database hive.db2, returning
> NoSuchObjectException
> {code}
> This is a superfluous log message. It might be pretty common for a database
> to not exists if, for example, a user fat-fingers the name of the database.
> The code also has the bad habit of log-and-throw. Just log or throw, not
> both.
> Since I'm looking at this class, touch up some of the other logging as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22363) ReduceDeduplication may leave an invalid GroupByOperator behind in some cases
[
https://issues.apache.org/jira/browse/HIVE-22363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962311#comment-16962311
]
Hive QA commented on HIVE-22363:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
5s{color} | {color:blue} ql in master has 1545 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19195/dev-support/hive-personality.sh
|
| git revision | master / 6d0f461 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19195/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> ReduceDeduplication may leave an invalid GroupByOperator behind in some cases
> -
>
> Key: HIVE-22363
> URL: https://issues.apache.org/jira/browse/HIVE-22363
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
>Affects Versions: 3.1.2
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22363.01.patch, HIVE-22363.02.patch,
> HIVE-22363.03.patch, HIVE-22363.04.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> since HIVE-11387 reducededup may traverse {{GroupByOperators}} [as
> well|https://github.com/apache/hive/blob/c6626edb65c2cd00576647e54db1995628fe64da/ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/CorrelationUtilities.java#L244]
> But the removal logic only removes the first parent; so if there is some
> other operator (a FIL in this case) between the sink and the gby - the
> removal may not happen
> [here|https://github.com/apache/hive/blob/c6626edb65c2cd00576647e54db1995628fe64da/ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/CorrelationUtilities.java#L458]
> {code}
> set hive.cbo.enable=false;
> drop table if exists xl1;
> create table xl1 as
> select '1' as mdl_yr_desc, 2 as seq_no,'3' as opt_desc1,4 as opt_desc,1 as
> row_num;
> explain
> select trim(base.mdl_yr_desc) mdl_yr_desc, trim(base.opt_desc) opt_desc
> from
> (
> SELECT trim(mdl_yr_desc) mdl_yr_desc, concat_ws(' ',
> collect_set(trim(opt_desc1))) AS opt_desc
> from
> (
>
[jira] [Updated] (HIVE-22427) PersistenceManagerProvider Logs a Warning About datanucleus.autoStartMechanismMode
[
https://issues.apache.org/jira/browse/HIVE-22427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-22427:
--
Attachment: HIVE-22427.1.patch
> PersistenceManagerProvider Logs a Warning About
> datanucleus.autoStartMechanismMode
> --
>
> Key: HIVE-22427
> URL: https://issues.apache.org/jira/browse/HIVE-22427
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22427.1.patch
>
>
> {code:none}
> WARN [pool-6-thread-2] metastore.PersistenceManagerProvider:
> datanucleus.autoStartMechanismMode is set to unsupported value null . Setting
> it to value: ignored
> {code}
> This does not need to be a WARN level logging for this scenario. Perhaps if
> user configures the value to some non-null value, then emit a warning,
> otherwise, simply emit an INFO level stating that the configuration is not
> set and that a reasonable default value will be used.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22427) PersistenceManagerProvider Logs a Warning About datanucleus.autoStartMechanismMode
[
https://issues.apache.org/jira/browse/HIVE-22427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-22427:
--
Status: Patch Available (was: Open)
> PersistenceManagerProvider Logs a Warning About
> datanucleus.autoStartMechanismMode
> --
>
> Key: HIVE-22427
> URL: https://issues.apache.org/jira/browse/HIVE-22427
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22427.1.patch
>
>
> {code:none}
> WARN [pool-6-thread-2] metastore.PersistenceManagerProvider:
> datanucleus.autoStartMechanismMode is set to unsupported value null . Setting
> it to value: ignored
> {code}
> This does not need to be a WARN level logging for this scenario. Perhaps if
> user configures the value to some non-null value, then emit a warning,
> otherwise, simply emit an INFO level stating that the configuration is not
> set and that a reasonable default value will be used.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22427) PersistenceManagerProvider Logs a Warning About datanucleus.autoStartMechanismMode
[
https://issues.apache.org/jira/browse/HIVE-22427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor reassigned HIVE-22427:
-
> PersistenceManagerProvider Logs a Warning About
> datanucleus.autoStartMechanismMode
> --
>
> Key: HIVE-22427
> URL: https://issues.apache.org/jira/browse/HIVE-22427
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
>
> {code:none}
> WARN [pool-6-thread-2] metastore.PersistenceManagerProvider:
> datanucleus.autoStartMechanismMode is set to unsupported value null . Setting
> it to value: ignored
> {code}
> This does not need to be a WARN level logging for this scenario. Perhaps if
> user configures the value to some non-null value, then emit a warning,
> otherwise, simply emit an INFO level stating that the configuration is not
> set and that a reasonable default value will be used.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21954) QTest: support for running qtests on various metastore DBs
[
https://issues.apache.org/jira/browse/HIVE-21954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962292#comment-16962292
]
Hive QA commented on HIVE-21954:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12984251/HIVE-21954.04.patch
{color:green}SUCCESS:{color} +1 due to 11 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 17106 tests
executed
*Failed tests:*
{noformat}
TestBeeLineDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=299)
TestDummy - did not produce a TEST-*.xml file (likely timed out) (batchId=299)
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=299)
TestTezPerfCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=299)
TestTezPerfConstraintsCliDriver - did not produce a TEST-*.xml file (likely
timed out) (batchId=299)
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver
(batchId=301)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances
(batchId=265)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19194/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19194/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19194/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12984251 - PreCommit-HIVE-Build
> QTest: support for running qtests on various metastore DBs
> --
>
> Key: HIVE-21954
> URL: https://issues.apache.org/jira/browse/HIVE-21954
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore, Testing Infrastructure
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21954.01.patch, HIVE-21954.02.patch,
> HIVE-21954.03.patch, HIVE-21954.03.patch, HIVE-21954.03.patch,
> HIVE-21954.04.patch
>
>
> In HIVE-21940, a postgres metastore related issue has been fixed, and a local
> reproduction has been provided.
> {code}
> export QTEST_LEAVE_FILES=true
> docker kill metastore-test-postgres-install
> docker rm metastore-test-postgres-install
> cd standalone-metastore
> mvn verify -DskipITests=false -Dit.test=ITestPostgres#install -Dtest=nosuch
> -Dmetastore.itest.no.stop.container=true
> cd ..
> mvn test -Dtest.output.overwrite=true -Pitests,hadoop-2 -pl itests/qtest
> -Dtest=TestCliDriver -Dqfile=partition_params_postgres.q
> -Dhive.metastore.rawstore.impl=org.apache.hadoop.hive.metastore.ObjectStore
> {code}
> The problem with this solution is that data/conf/hive-site.xml has to be
> edited manually. My proposal is to introduce a property
> (-Dmetastore.db=postgres), which can take care of the parameters on the fly.
> 2 supported solutions could be:
> 1. simple parameters: -Dmetastore.db=postgres
> In this case, tests depend on settings from ITestPostgres class (password,
> db, etc.)
> 2. verbose but flexible parameters: [see hive-site.xml HIVE-21940's repro
> patch|https://issues.apache.org/jira/secure/attachment/12973534/HIVE-21940.repro.patch]
>
> In the first implementation, I would not start metastore db automatically
> (which is done be 'mvn verify ...'), but it's still under planning.
> In the long term, we should consider running this kind of tests in precommit
> phase, so maybe -Dmetastore.db=postgres could start metastore db
> automatically. Also we should consider running some qtests on various
> metastores. I would not pick randomly, but choose some "metastore-heavy" ones
> instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-21954) QTest: support for running qtests on various metastore DBs
[
https://issues.apache.org/jira/browse/HIVE-21954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962290#comment-16962290
]
Hive QA commented on HIVE-21954:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
41s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
54s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
10s{color} | {color:blue} standalone-metastore/metastore-server in master has
171 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 1545 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
52s{color} | {color:blue} itests/util in master has 53 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 10m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m
22s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
21s{color} | {color:red} standalone-metastore: The patch generated 7 new + 50
unchanged - 6 fixed = 57 total (was 56) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
16s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 7 new + 50 unchanged - 6 fixed = 57 total (was 56) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
51s{color} | {color:red} root: The patch generated 7 new + 104 unchanged - 6
fixed = 111 total (was 110) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
5s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 10m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 86m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense xml javac javadoc findbugs checkstyle
compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19194/dev-support/hive-personality.sh
|
| git revision | master / aceb8b6 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19194/yetus/diff-checkstyle-standalone-metastore.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19194/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19194/yetus/diff-checkstyle-root.txt
|
| modules | C: standalone-metastore standalone-metastore/metastore-server ql .
itests/hive-unit itests/qtest
[jira] [Updated] (HIVE-22426) Use DependencyManagement in Root POM for itests
[ https://issues.apache.org/jira/browse/HIVE-22426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22426: -- Description: Improve reliance on Maven inheritance for itest modules. > Use DependencyManagement in Root POM for itests > --- > > Key: HIVE-22426 > URL: https://issues.apache.org/jira/browse/HIVE-22426 > Project: Hive > Issue Type: Improvement > Components: Test, Tests >Affects Versions: 3.2.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-22426.1.patch > > > Improve reliance on Maven inheritance for itest modules. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22426) Use DependencyManagement in Root POM for itests
[ https://issues.apache.org/jira/browse/HIVE-22426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22426: -- Attachment: HIVE-22426.1.patch > Use DependencyManagement in Root POM for itests > --- > > Key: HIVE-22426 > URL: https://issues.apache.org/jira/browse/HIVE-22426 > Project: Hive > Issue Type: Improvement > Components: Test, Tests >Affects Versions: 3.2.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-22426.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22426) Use DependencyManagement in Root POM for itests
[ https://issues.apache.org/jira/browse/HIVE-22426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22426: -- Status: Patch Available (was: Open) > Use DependencyManagement in Root POM for itests > --- > > Key: HIVE-22426 > URL: https://issues.apache.org/jira/browse/HIVE-22426 > Project: Hive > Issue Type: Improvement > Components: Test, Tests >Affects Versions: 3.2.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-22426.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22426) Use DependencyManagement in Root POM for itests
[ https://issues.apache.org/jira/browse/HIVE-22426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor reassigned HIVE-22426: - > Use DependencyManagement in Root POM for itests > --- > > Key: HIVE-22426 > URL: https://issues.apache.org/jira/browse/HIVE-22426 > Project: Hive > Issue Type: Improvement > Components: Test, Tests >Affects Versions: 3.2.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22411) Performance degradation on single row inserts
[ https://issues.apache.org/jira/browse/HIVE-22411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962280#comment-16962280 ] Steve Loughran commented on HIVE-22411: --- FYI [~gabor.bota][~rajesh.balamohan] > Performance degradation on single row inserts > - > > Key: HIVE-22411 > URL: https://issues.apache.org/jira/browse/HIVE-22411 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: Screen Shot 2019-10-17 at 8.40.50 PM.png > > > Executing single insert statements on a transactional table effects write > performance on a s3 file system. Each insert creates a new delta directory. > After each insert hive calculates statistics like number of file in the table > and total size of the table. In order to calculate these, it traverses the > directory recursively. During the recursion for each path a separate > listStatus call is executed. In the end the more delta directory you have the > more time it takes to calculate the statistics. > Therefore insertion time goes up linearly: > !Screen Shot 2019-10-17 at 8.40.50 PM.png|width=601,height=436! > The fix is to use fs.listFiles(path, /**recursive**/ true) instead the > handcrafter recursive method/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22411) Performance degradation on single row inserts
[ https://issues.apache.org/jira/browse/HIVE-22411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962271#comment-16962271 ] Steve Loughran commented on HIVE-22411: --- Why do you need to list every single file under a directory tree just to update the counter? That is a very expensive operation. On S3 it is O(files/5000) and you are billed for it; With S3Guard it is slightly faster and you are billed more for it. In both cases it lines you up for throttling by the service. Can't Hive count the amount of data during job commit? > Performance degradation on single row inserts > - > > Key: HIVE-22411 > URL: https://issues.apache.org/jira/browse/HIVE-22411 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: Screen Shot 2019-10-17 at 8.40.50 PM.png > > > Executing single insert statements on a transactional table effects write > performance on a s3 file system. Each insert creates a new delta directory. > After each insert hive calculates statistics like number of file in the table > and total size of the table. In order to calculate these, it traverses the > directory recursively. During the recursion for each path a separate > listStatus call is executed. In the end the more delta directory you have the > more time it takes to calculate the statistics. > Therefore insertion time goes up linearly: > !Screen Shot 2019-10-17 at 8.40.50 PM.png|width=601,height=436! > The fix is to use fs.listFiles(path, /**recursive**/ true) instead the > handcrafter recursive method/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22425) ReplChangeManager Not Logging Database Name
[
https://issues.apache.org/jira/browse/HIVE-22425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962264#comment-16962264
]
Gopal Vijayaraghavan commented on HIVE-22425:
-
LGTM - +1
> ReplChangeManager Not Logging Database Name
> ---
>
> Key: HIVE-22425
> URL: https://issues.apache.org/jira/browse/HIVE-22425
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Trivial
> Attachments: HIVE-22425.1.patch
>
>
> {code:java|title=ReplChangeManager.java}
> LOG.debug("Repl policy is not set for database ", db.getName());
> {code}
> The log statement is missing the placeholder '{}' so the DB name is not
> getting logged.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22031) HiveRelDecorrelator fails with IndexOutOfBoundsException if the query contains several "constant" columns
[
https://issues.apache.org/jira/browse/HIVE-22031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Artem Velykorodnyi updated HIVE-22031:
--
Attachment: HIVE-22031.7.branch-2.patch
Status: Patch Available (was: Open)
> HiveRelDecorrelator fails with IndexOutOfBoundsException if the query
> contains several "constant" columns
> -
>
> Key: HIVE-22031
> URL: https://issues.apache.org/jira/browse/HIVE-22031
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 2.3.5
>Reporter: Artem Velykorodnyi
>Assignee: Artem Velykorodnyi
>Priority: Major
> Attachments: HIVE-22031.02.patch, HIVE-22031.1.patch,
> HIVE-22031.3.patch, HIVE-22031.4.patch, HIVE-22031.6.branch-2.patch,
> HIVE-22031.7.branch-2.patch, HIVE-22031.patch
>
>
> Steps for reproducing:
> {code}
> 1. Create table orders
> create table orders (ORD_NUM INT, CUST_CODE STRING);
> 2. Create table customers
> create table customers (CUST_CODE STRING);
> 3. Make select with constants and with a subquery:
> select DISTINCT(CUST_CODE), '777' as ANY, ORD_NUM, '888' as CONSTANT
> from orders
> WHERE not exists
> (select 1
> from customers
> WHERE CUST_CODE=orders.CUST_CODE
> );
> {code}
> Query fails with IndexOutOfBoundsException
> {code}
> Exception in thread "main" java.lang.AssertionError: Internal error: While
> invoking method 'public
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator$Frame
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.decorrelateRel(org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject)
> throws org.apache.hadoop.hive.ql.parse.SemanticException'
> at org.apache.calcite.util.Util.newInternal(Util.java:792)
> at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:534)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.getInvoke(HiveRelDecorrelator.java:660)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.decorrelate(HiveRelDecorrelator.java:252)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.decorrelateQuery(HiveRelDecorrelator.java:218)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1347)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1261)
> at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:113)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:997)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:149)
> at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:106)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1069)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:1085)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:364)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11138)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:512)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> Caused by: java.lang.reflect.InvocationTargetException
> at
[jira] [Updated] (HIVE-22031) HiveRelDecorrelator fails with IndexOutOfBoundsException if the query contains several "constant" columns
[
https://issues.apache.org/jira/browse/HIVE-22031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Artem Velykorodnyi updated HIVE-22031:
--
Status: Open (was: Patch Available)
> HiveRelDecorrelator fails with IndexOutOfBoundsException if the query
> contains several "constant" columns
> -
>
> Key: HIVE-22031
> URL: https://issues.apache.org/jira/browse/HIVE-22031
> Project: Hive
> Issue Type: Bug
> Components: CBO
>Affects Versions: 2.3.5
>Reporter: Artem Velykorodnyi
>Assignee: Artem Velykorodnyi
>Priority: Major
> Attachments: HIVE-22031.02.patch, HIVE-22031.1.patch,
> HIVE-22031.3.patch, HIVE-22031.4.patch, HIVE-22031.6.branch-2.patch,
> HIVE-22031.patch
>
>
> Steps for reproducing:
> {code}
> 1. Create table orders
> create table orders (ORD_NUM INT, CUST_CODE STRING);
> 2. Create table customers
> create table customers (CUST_CODE STRING);
> 3. Make select with constants and with a subquery:
> select DISTINCT(CUST_CODE), '777' as ANY, ORD_NUM, '888' as CONSTANT
> from orders
> WHERE not exists
> (select 1
> from customers
> WHERE CUST_CODE=orders.CUST_CODE
> );
> {code}
> Query fails with IndexOutOfBoundsException
> {code}
> Exception in thread "main" java.lang.AssertionError: Internal error: While
> invoking method 'public
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator$Frame
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.decorrelateRel(org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject)
> throws org.apache.hadoop.hive.ql.parse.SemanticException'
> at org.apache.calcite.util.Util.newInternal(Util.java:792)
> at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:534)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.getInvoke(HiveRelDecorrelator.java:660)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.decorrelate(HiveRelDecorrelator.java:252)
> at
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveRelDecorrelator.decorrelateQuery(HiveRelDecorrelator.java:218)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1347)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:1261)
> at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:113)
> at
> org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:997)
> at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:149)
> at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:106)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.logicalPlan(CalcitePlanner.java:1069)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:1085)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:364)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11138)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286)
> at
> org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:512)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:821)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
[jira] [Commented] (HIVE-22413) Avoid dirty read when reading the ACID table while compaction is running
[ https://issues.apache.org/jira/browse/HIVE-22413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962236#comment-16962236 ] Peter Vary commented on HIVE-22413: --- This kind of problem supposed to be solved by HIVE-20823 Running compaction in transaction will prevent dirty reads of the new folders > Avoid dirty read when reading the ACID table while compaction is running > > > Key: HIVE-22413 > URL: https://issues.apache.org/jira/browse/HIVE-22413 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Hocheol Park >Priority: Major > Attachments: HIVE-22413.1.patch > > > There is a problem that dirty read occurs when reading the ACID table while > base or delta directories are being created by the compactor. Especially it > is highly likely to occur in the S3 storage because the “move” logic of S3 is > “copy and delete”, and it takes a long time to copy if the size of files are > large or bucketing count is large. > So here’s the logic to avoid this problem. If “_tmp” prefixed directories are > existed in the partition directory on the process of listing the child > directories when reading the ACID table, compare the names of the directory > in the “_tmp” one and skip it in case of the same. Then it will read the > files before merging, no difference on the results. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22406) TRUNCATE TABLE fails due MySQL limitations on limit value
[
https://issues.apache.org/jira/browse/HIVE-22406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962232#comment-16962232
]
Sam An commented on HIVE-22406:
---
+1
> TRUNCATE TABLE fails due MySQL limitations on limit value
> -
>
> Key: HIVE-22406
> URL: https://issues.apache.org/jira/browse/HIVE-22406
> Project: Hive
> Issue Type: Bug
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>Priority: Major
> Attachments: HIVE-22406.patch, HIVE-22406.patch
>
>
> HMS currently has some APIs that accepts an integer limit value. Prior to the
> change in HIVE-21734, HMS was silently converting this int to short and thus
> we havent seen this issue. But semantically, its incorrect to do so quietly.
> {noformat}
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
> Caused by: java.sql.SQLException: setMaxRows() out of range. 2147483647 >
> 5000.
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:996) ~[mysql-
> connector-java.jar:5.1.33]
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:935) ~[mysql-
> connector-java.jar:5.1.33]
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:924) ~[mysql-
> connector-java.jar:5.1.33]
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:870) ~[mysql-
> connector-java.jar:5.1.33]
> at com.mysql.jdbc.StatementImpl.setMaxRows(StatementImpl.java:2525) ~[mysql-
> connector-java.jar:5.1.33]
> at
> com.zaxxer.hikari.pool.HikariProxyPreparedStatement.setMaxRows(HikariProxyPreparedS
> tatement.java) ~[HikariCP-2.6.1.jar:?]
> {noformat}
> We cannot change the RawStore api to accept shorts instead of ints.
> So we have to fix the caller to use a lower limit instead of
> Integer.MAX_VALUE.
> {noformat}
> Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Exception
> thrown when executing query : SELECT DISTINCT
> 'org.apache.hadoop.hive.metastore.model.MPartition' AS
> `NUCLEUS_TYPE`,`A0`.`CREATE_TIME`,`A0`.`LAST_ACCESS_TIME`,`A0`.`PART_NAME`,`A0`.`WRITE_ID`,`A0`.`PART_ID`,`A0`.`PART_NAME`
> AS `NUCORDER0` FROM `PARTITIONS` `A0` LEFT OUTER JOIN `TBLS` `B0` ON
> `A0`.`TBL_ID` = `B0`.`TBL_ID` LEFT OUTER JOIN `DBS` `C0` ON `B0`.`DB_ID` =
> `C0`.`DB_ID` WHERE `B0`.`TBL_NAME` = ? AND `C0`.`NAME` = ? AND
> `C0`.`CTLG_NAME` = ? ORDER BY `NUCORDER0` LIMIT 0,2147483647
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$truncate_table_req_result$truncate_table_req_resultStandardScheme.read(ThriftHiveMetastore.java)
> ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$truncate_table_req_result$truncate_table_req_resultStandardScheme.read(ThriftHiveMetastore.java)
> ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$truncate_table_req_result.read(ThriftHiveMetastore.java)
> ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
> ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_truncate_table_req(ThriftHiveMetastore.java:1999)
> ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.truncate_table_req(ThriftHiveMetastore.java:1986)
> ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.truncateTableInternal(HiveMetaStoreClient.java:1450)
> ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.truncateTable(HiveMetaStoreClient.java:1427)
> ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.truncateTable(SessionHiveMetaStoreClient.java:171)
> ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
> at sun.reflect.GeneratedMethodAccessor112.invoke(Unknown Source)
> ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_191]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_191]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:212)
> ~[hive-exec-3.1.0.3.1.5.0-17.jar:3.1.0.3.1.5.0-17]
> at com.sun.proxy.$Proxy59.truncateTable(Unknown Source) ~[?:?]
> at sun.reflect.GeneratedMethodAccessor112.invoke(Unknown Source)
> ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_191]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_191]
> at
>
[jira] [Commented] (HIVE-22422) Missing documentation from HiveSqlDateTimeFormatter: list of date-based patterns
[ https://issues.apache.org/jira/browse/HIVE-22422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962224#comment-16962224 ] Marta Kuczora commented on HIVE-22422: -- Pushed to master. Thanks [~klcopp] for the update. > Missing documentation from HiveSqlDateTimeFormatter: list of date-based > patterns > > > Key: HIVE-22422 > URL: https://issues.apache.org/jira/browse/HIVE-22422 > Project: Hive > Issue Type: Task > Components: Documentation >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Trivial > Attachments: HIVE-22422.01.patch > > > Documentation referenced a "List of Date-Based Patterns" but didn't include > the list anywhere. > NO PRECOMMIT TESTS -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22422) Missing documentation from HiveSqlDateTimeFormatter: list of date-based patterns
[ https://issues.apache.org/jira/browse/HIVE-22422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marta Kuczora updated HIVE-22422: - Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Missing documentation from HiveSqlDateTimeFormatter: list of date-based > patterns > > > Key: HIVE-22422 > URL: https://issues.apache.org/jira/browse/HIVE-22422 > Project: Hive > Issue Type: Task > Components: Documentation >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Trivial > Fix For: 4.0.0 > > Attachments: HIVE-22422.01.patch > > > Documentation referenced a "List of Date-Based Patterns" but didn't include > the list anywhere. > NO PRECOMMIT TESTS -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22238) PK/FK selectivity estimation underscales estimations
[ https://issues.apache.org/jira/browse/HIVE-22238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-22238: Attachment: HIVE-22238.05.patch > PK/FK selectivity estimation underscales estimations > > > Key: HIVE-22238 > URL: https://issues.apache.org/jira/browse/HIVE-22238 > Project: Hive > Issue Type: Bug > Components: Statistics >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22238.01.patch, HIVE-22238.02.patch, > HIVE-22238.03.patch, HIVE-22238.04.patch, HIVE-22238.05.patch, > HIVE-22238.05.patch, HIVE-22238.05.patch > > Time Spent: 10m > Remaining Estimate: 0h > > at [this > point|https://github.com/apache/hive/blob/5098d155a1e6a164253f5fa98755273bc34085df/ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java#L2182] > the parent operators rownum is scaled according to pkfkselectivity > however [pkfkselectivity is > computed|https://github.com/apache/hive/blob/5098d155a1e6a164253f5fa98755273bc34085df/ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java#L2157] > on a whole subtree. > Scaling it by that amount will count in estimation already used when > parentstats was calculated...so depending on the number of upstream joins - > this may lead to severe underestimations > what happened was: > * optimization was able to push the filter to the other side of the join > * as a result the incoming data was already filtered > * scaling down by the PK selectiviy - was actually already there...but a new > "scaling" happened -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22425) ReplChangeManager Not Logging Database Name
[
https://issues.apache.org/jira/browse/HIVE-22425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-22425:
--
Attachment: HIVE-22425.1.patch
> ReplChangeManager Not Logging Database Name
> ---
>
> Key: HIVE-22425
> URL: https://issues.apache.org/jira/browse/HIVE-22425
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Trivial
> Attachments: HIVE-22425.1.patch
>
>
> {code:java|title=ReplChangeManager.java}
> LOG.debug("Repl policy is not set for database ", db.getName());
> {code}
> The log statement is missing the placeholder '{}' so the DB name is not
> getting logged.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22425) ReplChangeManager Not Logging Database Name
[
https://issues.apache.org/jira/browse/HIVE-22425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-22425:
--
Status: Patch Available (was: Open)
> ReplChangeManager Not Logging Database Name
> ---
>
> Key: HIVE-22425
> URL: https://issues.apache.org/jira/browse/HIVE-22425
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Trivial
> Attachments: HIVE-22425.1.patch
>
>
> {code:java|title=ReplChangeManager.java}
> LOG.debug("Repl policy is not set for database ", db.getName());
> {code}
> The log statement is missing the placeholder '{}' so the DB name is not
> getting logged.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22425) ReplChangeManager Not Logging Database Name
[
https://issues.apache.org/jira/browse/HIVE-22425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor reassigned HIVE-22425:
-
> ReplChangeManager Not Logging Database Name
> ---
>
> Key: HIVE-22425
> URL: https://issues.apache.org/jira/browse/HIVE-22425
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Trivial
> Attachments: HIVE-22425.1.patch
>
>
> {code:java|title=ReplChangeManager.java}
> LOG.debug("Repl policy is not set for database ", db.getName());
> {code}
> The log statement is missing the placeholder '{}' so the DB name is not
> getting logged.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22424) User PerfLogger in MetastoreDirectSqlUtils.java
[ https://issues.apache.org/jira/browse/HIVE-22424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22424: -- Description: Currently has its own way of doing it: https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java#L100 > User PerfLogger in MetastoreDirectSqlUtils.java > --- > > Key: HIVE-22424 > URL: https://issues.apache.org/jira/browse/HIVE-22424 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Affects Versions: 3.2.0 >Reporter: David Mollitor >Priority: Minor > Fix For: 4.0.0 > > > Currently has its own way of doing it: > https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java#L100 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22238) PK/FK selectivity estimation underscales estimations
[
https://issues.apache.org/jira/browse/HIVE-22238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962191#comment-16962191
]
Hive QA commented on HIVE-22238:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12984247/HIVE-22238.05.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19193/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19193/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19193/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12984247/HIVE-22238.05.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12984247 - PreCommit-HIVE-Build
> PK/FK selectivity estimation underscales estimations
>
>
> Key: HIVE-22238
> URL: https://issues.apache.org/jira/browse/HIVE-22238
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22238.01.patch, HIVE-22238.02.patch,
> HIVE-22238.03.patch, HIVE-22238.04.patch, HIVE-22238.05.patch,
> HIVE-22238.05.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> at [this
> point|https://github.com/apache/hive/blob/5098d155a1e6a164253f5fa98755273bc34085df/ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java#L2182]
> the parent operators rownum is scaled according to pkfkselectivity
> however [pkfkselectivity is
> computed|https://github.com/apache/hive/blob/5098d155a1e6a164253f5fa98755273bc34085df/ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java#L2157]
> on a whole subtree.
> Scaling it by that amount will count in estimation already used when
> parentstats was calculated...so depending on the number of upstream joins -
> this may lead to severe underestimations
> what happened was:
> * optimization was able to push the filter to the other side of the join
> * as a result the incoming data was already filtered
> * scaling down by the PK selectiviy - was actually already there...but a new
> "scaling" happened
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22414) Make LLAP CacheTags more memory efficient
[
https://issues.apache.org/jira/browse/HIVE-22414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962188#comment-16962188
]
Hive QA commented on HIVE-22414:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12984242/HIVE-22414.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17546 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_select_read_only_unencrypted_tbl]
(batchId=190)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19192/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19192/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19192/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12984242 - PreCommit-HIVE-Build
> Make LLAP CacheTags more memory efficient
> -
>
> Key: HIVE-22414
> URL: https://issues.apache.org/jira/browse/HIVE-22414
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Attachments: HIVE-22414.0.patch, HIVE-22414.1.patch
>
>
> MultiPartitionCacheTag relies on LinkedLists.
> A LinkedList object that holds e.g. 2 nodes consumes 112 bytes roughly in
> this composition:
> * 16 bytes for LinkedList object header
> * 8 bytes for referring head
> * 8 bytes for referring tail
> * 80 = 2 x (16 bytes for LinkedList$Node header, 3 x 8 bytes for referring
> prev, next, item)
> This is a lot, so I propose to replace LinkedList in MultiPartitionCacheTag
> with a simple String array. (For a similar scenario an array would take 16 +
> 8 + 2 x 8 = 40 bytes, as per header, count, and 2 references for our actual
> objects).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22422) Missing documentation from HiveSqlDateTimeFormatter: list of date-based patterns
[ https://issues.apache.org/jira/browse/HIVE-22422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962168#comment-16962168 ] Marta Kuczora commented on HIVE-22422: -- +1 > Missing documentation from HiveSqlDateTimeFormatter: list of date-based > patterns > > > Key: HIVE-22422 > URL: https://issues.apache.org/jira/browse/HIVE-22422 > Project: Hive > Issue Type: Task > Components: Documentation >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Trivial > Attachments: HIVE-22422.01.patch > > > Documentation referenced a "List of Date-Based Patterns" but didn't include > the list anywhere. > NO PRECOMMIT TESTS -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22423) Improve Logging In HadoopThriftAuthBridge
[ https://issues.apache.org/jira/browse/HIVE-22423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22423: -- Status: Patch Available (was: Open) > Improve Logging In HadoopThriftAuthBridge > - > > Key: HIVE-22423 > URL: https://issues.apache.org/jira/browse/HIVE-22423 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22423.1.patch > > > # Remove superfluous debug log guards > # Improve messages > # Improve message format -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22423) Improve Logging In HadoopThriftAuthBridge
[ https://issues.apache.org/jira/browse/HIVE-22423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22423: -- Attachment: HIVE-22423.1.patch > Improve Logging In HadoopThriftAuthBridge > - > > Key: HIVE-22423 > URL: https://issues.apache.org/jira/browse/HIVE-22423 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22423.1.patch > > > # Remove superfluous debug log guards > # Improve messages > # Improve message format -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22423) Improve Logging In HadoopThriftAuthBridge
[ https://issues.apache.org/jira/browse/HIVE-22423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor reassigned HIVE-22423: - > Improve Logging In HadoopThriftAuthBridge > - > > Key: HIVE-22423 > URL: https://issues.apache.org/jira/browse/HIVE-22423 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > > # Remove superfluous debug log guards > # Improve messages > # Improve message format -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22422) Missing documentation from HiveSqlDateTimeFormatter: list of date-based patterns
[ https://issues.apache.org/jira/browse/HIVE-22422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-22422: - Description: Documentation referenced a "List of Date-Based Patterns" but didn't include the list anywhere. NO PRECOMMIT TESTS was:Documentation referenced a "List of Date-Based Patterns" but didn't include the list anywhere. > Missing documentation from HiveSqlDateTimeFormatter: list of date-based > patterns > > > Key: HIVE-22422 > URL: https://issues.apache.org/jira/browse/HIVE-22422 > Project: Hive > Issue Type: Task > Components: Documentation >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Trivial > Attachments: HIVE-22422.01.patch > > > Documentation referenced a "List of Date-Based Patterns" but didn't include > the list anywhere. > NO PRECOMMIT TESTS -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22422) Missing documentation from HiveSqlDateTimeFormatter: list of date-based patterns
[ https://issues.apache.org/jira/browse/HIVE-22422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-22422: - Attachment: HIVE-22422.01.patch Status: Patch Available (was: Open) > Missing documentation from HiveSqlDateTimeFormatter: list of date-based > patterns > > > Key: HIVE-22422 > URL: https://issues.apache.org/jira/browse/HIVE-22422 > Project: Hive > Issue Type: Task > Components: Documentation >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Trivial > Attachments: HIVE-22422.01.patch > > > Documentation referenced a "List of Date-Based Patterns" but didn't include > the list anywhere. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22422) Missing documentation from HiveSqlDateTimeFormatter: list of date-based patterns
[ https://issues.apache.org/jira/browse/HIVE-22422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage reassigned HIVE-22422: > Missing documentation from HiveSqlDateTimeFormatter: list of date-based > patterns > > > Key: HIVE-22422 > URL: https://issues.apache.org/jira/browse/HIVE-22422 > Project: Hive > Issue Type: Task > Components: Documentation >Affects Versions: 4.0.0 >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Trivial > Attachments: HIVE-22422.01.patch > > > Documentation referenced a "List of Date-Based Patterns" but didn't include > the list anywhere. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22421) Improve Logging If Configuration File Not Found
[
https://issues.apache.org/jira/browse/HIVE-22421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-22421:
--
Status: Patch Available (was: Open)
> Improve Logging If Configuration File Not Found
> ---
>
> Key: HIVE-22421
> URL: https://issues.apache.org/jira/browse/HIVE-22421
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22421.1.patch
>
>
> {code:none}
> 2019-10-28T21:07:27,599 INFO [main] conf.MetastoreConf: Unable to find
> config file metastore-site.xml
> 2019-10-28T21:07:27,599 INFO [main] conf.MetastoreConf: Found configuration
> file null
> {code}
> Prints 'unable to find' followed by 'null'. Just print one or the other.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22421) Improve Logging If Configuration File Not Found
[
https://issues.apache.org/jira/browse/HIVE-22421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-22421:
--
Attachment: HIVE-22421.1.patch
> Improve Logging If Configuration File Not Found
> ---
>
> Key: HIVE-22421
> URL: https://issues.apache.org/jira/browse/HIVE-22421
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22421.1.patch
>
>
> {code:none}
> 2019-10-28T21:07:27,599 INFO [main] conf.MetastoreConf: Unable to find
> config file metastore-site.xml
> 2019-10-28T21:07:27,599 INFO [main] conf.MetastoreConf: Found configuration
> file null
> {code}
> Prints 'unable to find' followed by 'null'. Just print one or the other.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22421) Improve Logging If Configuration File Not Found
[
https://issues.apache.org/jira/browse/HIVE-22421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor reassigned HIVE-22421:
-
> Improve Logging If Configuration File Not Found
> ---
>
> Key: HIVE-22421
> URL: https://issues.apache.org/jira/browse/HIVE-22421
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
>
> {code:none}
> 2019-10-28T21:07:27,599 INFO [main] conf.MetastoreConf: Unable to find
> config file metastore-site.xml
> 2019-10-28T21:07:27,599 INFO [main] conf.MetastoreConf: Found configuration
> file null
> {code}
> Prints 'unable to find' followed by 'null'. Just print one or the other.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22419) Improve Messages Emitted From HiveMetaStoreClient
[
https://issues.apache.org/jira/browse/HIVE-22419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-22419:
--
Status: Patch Available (was: Open)
> Improve Messages Emitted From HiveMetaStoreClient
> -
>
> Key: HIVE-22419
> URL: https://issues.apache.org/jira/browse/HIVE-22419
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22419.1.patch
>
>
> After reviewing some logs and errors emitted during a QTest run, I would like
> to propose some improvements to logging in {{HiveMetaStoreClient}}.
> * Remove duplicate logging
> * Remove superfluous class {{StackTraceLogger}}
> * Do not use contractions in public-facing error messages and logs
> * Make all logging side-effect free (see {{connCount}})
> * Code simplification
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22419) Improve Messages Emitted From HiveMetaStoreClient
[
https://issues.apache.org/jira/browse/HIVE-22419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-22419:
--
Attachment: HIVE-22419.1.patch
> Improve Messages Emitted From HiveMetaStoreClient
> -
>
> Key: HIVE-22419
> URL: https://issues.apache.org/jira/browse/HIVE-22419
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 4.0.0, 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-22419.1.patch
>
>
> After reviewing some logs and errors emitted during a QTest run, I would like
> to propose some improvements to logging in {{HiveMetaStoreClient}}.
> * Remove duplicate logging
> * Remove superfluous class {{StackTraceLogger}}
> * Do not use contractions in public-facing error messages and logs
> * Make all logging side-effect free (see {{connCount}})
> * Code simplification
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22420) Driver.releaseLocksAndCommitOrRollback is not thread safe
[
https://issues.apache.org/jira/browse/HIVE-22420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aron Hamvas reassigned HIVE-22420:
--
> Driver.releaseLocksAndCommitOrRollback is not thread safe
> -
>
> Key: HIVE-22420
> URL: https://issues.apache.org/jira/browse/HIVE-22420
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Aron Hamvas
>Assignee: Aron Hamvas
>Priority: Major
>
> When a transactional query is being executed and interrupted via HS2 close
> operation request, both the background pool thread executing the query and
> the HttpHandler thread running the close operation logic will eventually call
> the below method:
> {noformat}
> Driver.releaseLocksAndCommitOrRollback(commit boolean)
> {noformat}
> Since this method is invoked several times in both threads, it can happen
> that the two threads invoke it at the same time, and due to a race
> condition, the txnId field of the DbTxnManager used by both threads could be
> set to 0 without actually successfully aborting the transaction.
> E.g. if the two threads reach the stopHeartbeat() call at the same time, one
> will set the heartbeat task to null, the other will run into a
> NullPointerException and due to the unsuccessful call, set the value of txnId
> to 0 before the other thread (which successfully ran the stopHeartbeat()
> call) could invoke rollback on HMS with the proper txnId.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
