[jira] [Commented] (HIVE-23452) Exception occur when a SQL query across data stored in two relational DB by JDBCStorageHandler with Tez
[ https://issues.apache.org/jira/browse/HIVE-23452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106026#comment-17106026 ] De Li commented on HIVE-23452: -- Yes, table 'test_mysql.test_gp' doesn't exist because it is a wrong combination produced by Hive, so it lead to the bug. > Exception occur when a SQL query across data stored in two relational DB by > JDBCStorageHandler with Tez > --- > > Key: HIVE-23452 > URL: https://issues.apache.org/jira/browse/HIVE-23452 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 3.1.0 >Reporter: De Li >Priority: Major > > Exception occur when a SQL query across data stored in two relational DB by > JDBCStorageHandler with Tez. It seems there is an incorrect JDBC driver by > Tez and it works when query with MR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23458) Introduce unified thread pool for scheduled jobs
[ https://issues.apache.org/jira/browse/HIVE-23458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Chung updated HIVE-23458: Attachment: HIVE-23458.01.patch Status: Patch Available (was: Open) > Introduce unified thread pool for scheduled jobs > > > Key: HIVE-23458 > URL: https://issues.apache.org/jira/browse/HIVE-23458 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Eugene Chung >Assignee: Eugene Chung >Priority: Major > Attachments: HIVE-23458.01.patch > > > As I mentioned in [the comment of > HIVE-23164|https://issues.apache.org/jira/browse/HIVE-23164?focusedCommentId=17089506&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17089506], > I've made the unified scheduled executor service like > org.apache.hadoop.hive.metastore.ThreadPool. > I think it could help > 1. to minimize the possibility of making non-daemon threads when developers > need ScheduledExecutorService > 2. to achieve the utilization of server resources because the current > situation is all of the modules make its own ScheduledExecutorService and all > of the threads are just using for one job. > 3. an administrator of Hive server by providing > hive.exec.scheduler.num.threads configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (HIVE-23452) Exception occur when a SQL query across data stored in two relational DB by JDBCStorageHandler with Tez
[
https://issues.apache.org/jira/browse/HIVE-23452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105975#comment-17105975
]
De Li edited comment on HIVE-23452 at 5/13/20, 7:02 AM:
{code:java}
// mysql table
CREATE EXTERNAL TABLE IF NOT EXISTS test_mysqltable_ex (
id int,
name String
)
STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
TBLPROPERTIES (
"hive.sql.database.type" = "MYSQL",
"hive.sql.jdbc.driver" = "com.mysql.jdbc.Driver",
"hive.sql.jdbc.url" = "jdbc:mysql://xxx/test_mysql",
"hive.sql.dbcp.username" = "xxx",
"hive.sql.dbcp.password" = "",
"hive.sql.table" = "test_mysqltable"
);
{code}
{code:java}
// postgresql
CREATE EXTERNAL TABLE IF NOT EXISTS gp_test (
id int
)
STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
TBLPROPERTIES (
"hive.sql.database.type" = "POSTGRES",
"hive.sql.jdbc.driver" = "org.postgresql.Driver",
"hive.sql.jdbc.url" = "jdbc:postgresql://:5432/pg",
"hive.sql.dbcp.username" = "",
"hive.sql.dbcp.password" = "",
"hive.sql.table" = "test_gp"
);
{code}
{code:java}
// code placeholder
SELECT aa.id,bb.name FROM default.gp_test aa left join
default.test_mysqltable_ex bb on aa.id=bb.id
where bb.id >1;
{code}
was (Author: lide):
{code:java}
// mysql table
CREATE EXTERNAL TABLE IF NOT EXISTS test_mysqltable_ex (
id int,
name String
)
STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
TBLPROPERTIES (
"hive.sql.database.type" = "MYSQL",
"hive.sql.jdbc.driver" = "com.mysql.jdbc.Driver",
"hive.sql.jdbc.url" = "jdbc:mysql://xxx/test_mysql",
"hive.sql.dbcp.username" = "xxx",
"hive.sql.dbcp.password" = "",
"hive.sql.table" = "test_mysqltable"
);
{code}
{code:java}
// postgresql
CREATE EXTERNAL TABLE IF NOT EXISTS gp_test (
id int
)
STORED BY 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
TBLPROPERTIES (
"hive.sql.database.type" = "POSTGRES",
"hive.sql.jdbc.driver" = "org.postgresql.Driver",
"hive.sql.jdbc.url" = "jdbc:postgresql://:5432/cdn",
"hive.sql.dbcp.username" = "",
"hive.sql.dbcp.password" = "",
"hive.sql.table" = "test_gp"
);
{code}
{code:java}
// code placeholder
SELECT aa.id,bb.name FROM default.gp_test aa left join
default.test_mysqltable_ex bb on aa.id=bb.id
where bb.id >1;
{code}
> Exception occur when a SQL query across data stored in two relational DB by
> JDBCStorageHandler with Tez
> ---
>
> Key: HIVE-23452
> URL: https://issues.apache.org/jira/browse/HIVE-23452
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 3.1.0
>Reporter: De Li
>Priority: Major
>
> Exception occur when a SQL query across data stored in two relational DB by
> JDBCStorageHandler with Tez. It seems there is an incorrect JDBC driver by
> Tez and it works when query with MR.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23458) Introduce unified thread pool for scheduled jobs
[ https://issues.apache.org/jira/browse/HIVE-23458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Chung updated HIVE-23458: Fix Version/s: 4.0.0 > Introduce unified thread pool for scheduled jobs > > > Key: HIVE-23458 > URL: https://issues.apache.org/jira/browse/HIVE-23458 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Eugene Chung >Assignee: Eugene Chung >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23458.01.patch > > > As I mentioned in [the comment of > HIVE-23164|https://issues.apache.org/jira/browse/HIVE-23164?focusedCommentId=17089506&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17089506], > I've made the unified scheduled executor service like > org.apache.hadoop.hive.metastore.ThreadPool. > I think it could help > 1. to minimize the possibility of making non-daemon threads when developers > need ScheduledExecutorService > 2. to achieve the utilization of server resources because the current > situation is all of the modules make its own ScheduledExecutorService and all > of the threads are just using for one job. > 3. an administrator of Hive server by providing > hive.exec.scheduler.num.threads configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23458) Introduce unified thread pool for scheduled jobs
[ https://issues.apache.org/jira/browse/HIVE-23458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Chung updated HIVE-23458: Description: As I mentioned in [the comment of HIVE-23164|https://issues.apache.org/jira/browse/HIVE-23164?focusedCommentId=17089506&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17089506], I've made the unified scheduled executor service like org.apache.hadoop.hive.metastore.ThreadPool. I think it could help 1. to minimize the possibility of making non-daemon threads when developers need ScheduledExecutorService 2. to achieve the utilization of server resources because the current situation is all of the modules make its own ScheduledExecutorService and all of the threads are just using for one job. 3. administrators of Hive servers by providing hive.exec.scheduler.num.threads configuration so that they can predict and set how many threads are used and needed. was: As I mentioned in [the comment of HIVE-23164|https://issues.apache.org/jira/browse/HIVE-23164?focusedCommentId=17089506&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17089506], I've made the unified scheduled executor service like org.apache.hadoop.hive.metastore.ThreadPool. I think it could help 1. to minimize the possibility of making non-daemon threads when developers need ScheduledExecutorService 2. to achieve the utilization of server resources because the current situation is all of the modules make its own ScheduledExecutorService and all of the threads are just using for one job. 3. an administrator of Hive server by providing hive.exec.scheduler.num.threads configuration. > Introduce unified thread pool for scheduled jobs > > > Key: HIVE-23458 > URL: https://issues.apache.org/jira/browse/HIVE-23458 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Eugene Chung >Assignee: Eugene Chung >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23458.01.patch > > > As I mentioned in [the comment of > HIVE-23164|https://issues.apache.org/jira/browse/HIVE-23164?focusedCommentId=17089506&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17089506], > I've made the unified scheduled executor service like > org.apache.hadoop.hive.metastore.ThreadPool. > I think it could help > 1. to minimize the possibility of making non-daemon threads when developers > need ScheduledExecutorService > 2. to achieve the utilization of server resources because the current > situation is all of the modules make its own ScheduledExecutorService and all > of the threads are just using for one job. > 3. administrators of Hive servers by providing > hive.exec.scheduler.num.threads configuration so that they can predict and > set how many threads are used and needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23452) Exception occur when a SQL query across data stored in two relational DB by JDBCStorageHandler with Tez
[ https://issues.apache.org/jira/browse/HIVE-23452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106028#comment-17106028 ] De Li commented on HIVE-23452: -- Both the table 'pg.test_gp' and 'test_mysql.test_mysqltable' are exist. > Exception occur when a SQL query across data stored in two relational DB by > JDBCStorageHandler with Tez > --- > > Key: HIVE-23452 > URL: https://issues.apache.org/jira/browse/HIVE-23452 > Project: Hive > Issue Type: Bug > Components: Tez >Affects Versions: 3.1.0 >Reporter: De Li >Priority: Major > > Exception occur when a SQL query across data stored in two relational DB by > JDBCStorageHandler with Tez. It seems there is an incorrect JDBC driver by > Tez and it works when query with MR. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23133) Numeric operations can have different result across hardware archs
[ https://issues.apache.org/jira/browse/HIVE-23133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106042#comment-17106042 ] Zhenyu Zheng commented on HIVE-23133: - [~gopalv] Thanks for the review, in previous patches (1-3) we didn't add config, if that is the better way, we can keep it that way. I will add the funcs that we changed to the ticket description. > Numeric operations can have different result across hardware archs > -- > > Key: HIVE-23133 > URL: https://issues.apache.org/jira/browse/HIVE-23133 > Project: Hive > Issue Type: Sub-task >Reporter: Zhenyu Zheng >Assignee: Zhenyu Zheng >Priority: Major > Attachments: HIVE-23133.1.patch, HIVE-23133.2.patch, > HIVE-23133.3.patch, HIVE-23133.4.patch, HIVE-23133.5.patch > > > Currently, we have set up an ARM CI to test out how Hive works on ARM > platform: > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/] > Among the failures, we have observed that some numeric operations can have > different result across hardware archs, such as: > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/] > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/] > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_vectorized_math_funcs_/] > we can see that the calculation results of log, exp, cos, toRadians etc is > slitly different than the .out file results that we are > comparing(they are tested and wrote on X86 machines), this is because of we > use [Math > Library|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html] for > these kind of calculations. > and according to the > [illustration|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html]: > _Unlike some of the numeric methods of class StrictMath, all implementations > of the equivalent functions of class Math are not_ > _defined to return the bit-for-bit same results. This relaxation permits > better-performing implementations where strict reproducibility_ > _is not required._ > _By default many of the Math methods simply call the equivalent method in > StrictMath for their implementation._ > _Code generators are encouraged to use platform-specific native libraries or > microprocessor instructions, where available,_ > _to provide higher-performance implementations of Math methods._ > so the result will have difference across hardware archs. > On the other hand, JAVA provided another library > [StrictMath|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html] > that will not have this kind of problem as according to its' > [reference|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html]: > _To help ensure portability of Java programs, the definitions of some of the > numeric functions in this package require that they produce_ > _the same results as certain published algorithms._ > So in order to fix the above mentioned problem, we have to consider switch to > use StrictMath instead of Math. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23133) Numeric operations can have different result across hardware archs
[ https://issues.apache.org/jira/browse/HIVE-23133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenyu Zheng updated HIVE-23133: Description: Use java.lang.StrictMath rather than java.lang.Math in UDFs: UDF Currently, we have set up an ARM CI to test out how Hive works on ARM platform: [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/] Among the failures, we have observed that some numeric operations can have different result across hardware archs, such as: [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/] [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/] [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_vectorized_math_funcs_/] we can see that the calculation results of log, exp, cos, toRadians etc is slitly different than the .out file results that we are comparing(they are tested and wrote on X86 machines), this is because of we use [Math Library|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html] for these kind of calculations. and according to the [illustration|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html]: _Unlike some of the numeric methods of class StrictMath, all implementations of the equivalent functions of class Math are not_ _defined to return the bit-for-bit same results. This relaxation permits better-performing implementations where strict reproducibility_ _is not required._ _By default many of the Math methods simply call the equivalent method in StrictMath for their implementation._ _Code generators are encouraged to use platform-specific native libraries or microprocessor instructions, where available,_ _to provide higher-performance implementations of Math methods._ so the result will have difference across hardware archs. On the other hand, JAVA provided another library [StrictMath|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html] that will not have this kind of problem as according to its' [reference|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html]: _To help ensure portability of Java programs, the definitions of some of the numeric functions in this package require that they produce_ _the same results as certain published algorithms._ So in order to fix the above mentioned problem, we have to consider switch to use StrictMath instead of Math. was: Currently, we have set up an ARM CI to test out how Hive works on ARM platform: [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/] Among the failures, we have observed that some numeric operations can have different result across hardware archs, such as: [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/] [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/] [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_vectorized_math_funcs_/] we can see that the calculation results of log, exp, cos, toRadians etc is slitly different than the .out file results that we are comparing(they are tested and wrote on X86 machines), this is because of we use [Math Library|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html] for these kind of calculations. and according to the [illustration|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html]: _Unlike some of the numeric methods of class StrictMath, all implementations of the equivalent functions of class Math are not_ _defined to return the bit-for-bit same results. This relaxation permits better-performing implementations where strict reproducibility_ _is not required._ _By default many of the Math methods simply call the equivalent method in StrictMath for their implementation._ _Code generators are encouraged to use platform-specific native libraries or microprocessor instructions, where available,_ _to provide higher-performance implementations of Math methods._ so the result will have difference across hardware archs. On the other hand, JAVA provided another library [StrictMath|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html] that will not have this kind of problem as according to its' [reference|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html]: _To help ensure portability of Java programs, the definitions of some of the numeric functions in this package require that
[jira] [Updated] (HIVE-23133) Numeric operations can have different result across hardware archs
[ https://issues.apache.org/jira/browse/HIVE-23133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenyu Zheng updated HIVE-23133: Description: Use java.lang.StrictMath rather than java.lang.Math in UDFs: UDFCos.java UDFExp.java UDFLn.java UDFLog.java Currently, we have set up an ARM CI to test out how Hive works on ARM platform: [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/] Among the failures, we have observed that some numeric operations can have different result across hardware archs, such as: [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/] [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/] [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_vectorized_math_funcs_/] we can see that the calculation results of log, exp, cos, toRadians etc is slitly different than the .out file results that we are comparing(they are tested and wrote on X86 machines), this is because of we use [Math Library|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html] for these kind of calculations. and according to the [illustration|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html]: _Unlike some of the numeric methods of class StrictMath, all implementations of the equivalent functions of class Math are not_ _defined to return the bit-for-bit same results. This relaxation permits better-performing implementations where strict reproducibility_ _is not required._ _By default many of the Math methods simply call the equivalent method in StrictMath for their implementation._ _Code generators are encouraged to use platform-specific native libraries or microprocessor instructions, where available,_ _to provide higher-performance implementations of Math methods._ so the result will have difference across hardware archs. On the other hand, JAVA provided another library [StrictMath|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html] that will not have this kind of problem as according to its' [reference|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html]: _To help ensure portability of Java programs, the definitions of some of the numeric functions in this package require that they produce_ _the same results as certain published algorithms._ So in order to fix the above mentioned problem, we have to consider switch to use StrictMath instead of Math. was: Use java.lang.StrictMath rather than java.lang.Math in UDFs: UDF Currently, we have set up an ARM CI to test out how Hive works on ARM platform: [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/] Among the failures, we have observed that some numeric operations can have different result across hardware archs, such as: [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/] [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/] [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_vectorized_math_funcs_/] we can see that the calculation results of log, exp, cos, toRadians etc is slitly different than the .out file results that we are comparing(they are tested and wrote on X86 machines), this is because of we use [Math Library|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html] for these kind of calculations. and according to the [illustration|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html]: _Unlike some of the numeric methods of class StrictMath, all implementations of the equivalent functions of class Math are not_ _defined to return the bit-for-bit same results. This relaxation permits better-performing implementations where strict reproducibility_ _is not required._ _By default many of the Math methods simply call the equivalent method in StrictMath for their implementation._ _Code generators are encouraged to use platform-specific native libraries or microprocessor instructions, where available,_ _to provide higher-performance implementations of Math methods._ so the result will have difference across hardware archs. On the other hand, JAVA provided another library [StrictMath|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html] that will not have this kind of problem as according to its' [reference|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html]: _To h
[jira] [Commented] (HIVE-23449) LLAP: Reduce mkdir and config creations in submitWork hotpath
[
https://issues.apache.org/jira/browse/HIVE-23449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106059#comment-17106059
]
Hive QA commented on HIVE-23449:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002779/HIVE-23449.3.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17268 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[special_character_in_tabnames_quotes_1]
(batchId=48)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22303/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22303/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22303/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002779 - PreCommit-HIVE-Build
> LLAP: Reduce mkdir and config creations in submitWork hotpath
> -
>
> Key: HIVE-23449
> URL: https://issues.apache.org/jira/browse/HIVE-23449
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Major
> Attachments: HIVE-23449.1.patch, HIVE-23449.2.patch,
> HIVE-23449.3.patch, Screenshot 2020-05-12 at 1.09.35 PM.png
>
>
> !Screenshot 2020-05-12 at 1.09.35 PM.png|width=885,height=558!
>
> For short jobs, submitWork gets into hotpath. This can lazy load conf and can
> get rid of dir creations (which needs to be enabled only when DirWatcher is
> enabled)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23458) Introduce unified thread pool for scheduled jobs
[ https://issues.apache.org/jira/browse/HIVE-23458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Chung updated HIVE-23458: Labels: todoc4.0 (was: ) > Introduce unified thread pool for scheduled jobs > > > Key: HIVE-23458 > URL: https://issues.apache.org/jira/browse/HIVE-23458 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Eugene Chung >Assignee: Eugene Chung >Priority: Major > Labels: todoc4.0 > Fix For: 4.0.0 > > Attachments: HIVE-23458.01.patch > > > As I mentioned in [the comment of > HIVE-23164|https://issues.apache.org/jira/browse/HIVE-23164?focusedCommentId=17089506&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17089506], > I've made the unified scheduled executor service like > org.apache.hadoop.hive.metastore.ThreadPool. > I think it could help > 1. to minimize the possibility of making non-daemon threads when developers > need ScheduledExecutorService > 2. to achieve the utilization of server resources because the current > situation is all of the modules make its own ScheduledExecutorService and all > of the threads are just using for one job. > 3. administrators of Hive servers by providing > hive.exec.scheduler.num.threads configuration so that they can predict and > set how many threads are used and needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23444) Concurrent ACID direct inserts may fail with FileNotFoundException
[
https://issues.apache.org/jira/browse/HIVE-23444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marta Kuczora updated HIVE-23444:
-
Attachment: HIVE-23444.1.patch
> Concurrent ACID direct inserts may fail with FileNotFoundException
> --
>
> Key: HIVE-23444
> URL: https://issues.apache.org/jira/browse/HIVE-23444
> Project: Hive
> Issue Type: Bug
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23444.1.patch, HIVE-23444.1.patch
>
>
> The following exception may occur when concurrently inserting into an ACID
> table with static partitions and the 'hive.acid.direct.insert.enabled'
> parameter is true. This issue occurs intermittently.
> {noformat}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: java.io.FileNotFoundException: File
> hdfs://ns1/warehouse/tablespace/managed/hive/tpch_unbucketed.db/concurrent_insert_partitioned/l_tax=0.0/_tmp.delta_001_001_
> does not exist.
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:2465)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:2228)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.exec.MoveTask.handleStaticParts(MoveTask.java:522)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:442)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:359)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:721)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:488)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:482)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:225)
> ~[hive-service-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> ... 13 more
> Caused by: java.io.IOException: java.io.FileNotFoundException: File
> hdfs://ns1/warehouse/tablespace/managed/hive/tpch_unbucketed.db/concurrent_insert_partitioned/l_tax=0.0/_tmp.delta_001_001_
> does not exist.
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getHdfsDirSnapshots(AcidUtils.java:1472)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getAcidState(AcidUtils.java:1297)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getAcidFilesForStats(AcidUtils.java:2695)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:2448)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:2228)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.exec.MoveTask.handleStaticParts(MoveTask.java:522)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:442)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Execu
[jira] [Commented] (HIVE-23451) FileSinkOperator calls deleteOnExit (hdfs call) twice for the same file
[
https://issues.apache.org/jira/browse/HIVE-23451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106094#comment-17106094
]
Hive QA commented on HIVE-23451:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
31s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 34s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22304/dev-support/hive-personality.sh
|
| git revision | master / 9ffbbdc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22304/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> FileSinkOperator calls deleteOnExit (hdfs call) twice for the same file
> ---
>
> Key: HIVE-23451
> URL: https://issues.apache.org/jira/browse/HIVE-23451
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-23451.1.patch, HIVE-23451.2.patch
>
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L826]
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L797]
> Can avoid a NN call here (i.e, mainly for small queries).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23133) Numeric operations can have different result across hardware archs
[ https://issues.apache.org/jira/browse/HIVE-23133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenyu Zheng updated HIVE-23133: Status: Open (was: Patch Available) > Numeric operations can have different result across hardware archs > -- > > Key: HIVE-23133 > URL: https://issues.apache.org/jira/browse/HIVE-23133 > Project: Hive > Issue Type: Sub-task >Reporter: Zhenyu Zheng >Assignee: Zhenyu Zheng >Priority: Major > Attachments: HIVE-23133.1.patch, HIVE-23133.2.patch, > HIVE-23133.3.patch, HIVE-23133.4.patch, HIVE-23133.5.patch, HIVE-23133.6.patch > > > Use java.lang.StrictMath rather than java.lang.Math in UDFs: > UDFCos.java > UDFExp.java > UDFLn.java > UDFLog.java > > Currently, we have set up an ARM CI to test out how Hive works on ARM > platform: > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/] > Among the failures, we have observed that some numeric operations can have > different result across hardware archs, such as: > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/] > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/] > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_vectorized_math_funcs_/] > we can see that the calculation results of log, exp, cos, toRadians etc is > slitly different than the .out file results that we are > comparing(they are tested and wrote on X86 machines), this is because of we > use [Math > Library|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html] for > these kind of calculations. > and according to the > [illustration|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html]: > _Unlike some of the numeric methods of class StrictMath, all implementations > of the equivalent functions of class Math are not_ > _defined to return the bit-for-bit same results. This relaxation permits > better-performing implementations where strict reproducibility_ > _is not required._ > _By default many of the Math methods simply call the equivalent method in > StrictMath for their implementation._ > _Code generators are encouraged to use platform-specific native libraries or > microprocessor instructions, where available,_ > _to provide higher-performance implementations of Math methods._ > so the result will have difference across hardware archs. > On the other hand, JAVA provided another library > [StrictMath|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html] > that will not have this kind of problem as according to its' > [reference|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html]: > _To help ensure portability of Java programs, the definitions of some of the > numeric functions in this package require that they produce_ > _the same results as certain published algorithms._ > So in order to fix the above mentioned problem, we have to consider switch to > use StrictMath instead of Math. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23133) Numeric operations can have different result across hardware archs
[ https://issues.apache.org/jira/browse/HIVE-23133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenyu Zheng updated HIVE-23133: Attachment: HIVE-23133.6.patch Status: Patch Available (was: Open) > Numeric operations can have different result across hardware archs > -- > > Key: HIVE-23133 > URL: https://issues.apache.org/jira/browse/HIVE-23133 > Project: Hive > Issue Type: Sub-task >Reporter: Zhenyu Zheng >Assignee: Zhenyu Zheng >Priority: Major > Attachments: HIVE-23133.1.patch, HIVE-23133.2.patch, > HIVE-23133.3.patch, HIVE-23133.4.patch, HIVE-23133.5.patch, HIVE-23133.6.patch > > > Use java.lang.StrictMath rather than java.lang.Math in UDFs: > UDFCos.java > UDFExp.java > UDFLn.java > UDFLog.java > > Currently, we have set up an ARM CI to test out how Hive works on ARM > platform: > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/] > Among the failures, we have observed that some numeric operations can have > different result across hardware archs, such as: > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/] > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/] > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_vectorized_math_funcs_/] > we can see that the calculation results of log, exp, cos, toRadians etc is > slitly different than the .out file results that we are > comparing(they are tested and wrote on X86 machines), this is because of we > use [Math > Library|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html] for > these kind of calculations. > and according to the > [illustration|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html]: > _Unlike some of the numeric methods of class StrictMath, all implementations > of the equivalent functions of class Math are not_ > _defined to return the bit-for-bit same results. This relaxation permits > better-performing implementations where strict reproducibility_ > _is not required._ > _By default many of the Math methods simply call the equivalent method in > StrictMath for their implementation._ > _Code generators are encouraged to use platform-specific native libraries or > microprocessor instructions, where available,_ > _to provide higher-performance implementations of Math methods._ > so the result will have difference across hardware archs. > On the other hand, JAVA provided another library > [StrictMath|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html] > that will not have this kind of problem as according to its' > [reference|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html]: > _To help ensure portability of Java programs, the definitions of some of the > numeric functions in this package require that they produce_ > _the same results as certain published algorithms._ > So in order to fix the above mentioned problem, we have to consider switch to > use StrictMath instead of Math. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23133) Numeric operations can have different result across hardware archs
[ https://issues.apache.org/jira/browse/HIVE-23133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenyu Zheng updated HIVE-23133: Description: Use java.lang.StrictMath rather than java.lang.Math in UDFs: UDFCos.java UDFExp.java UDFLn.java UDFLog.java UDFLog2.java UDFLog10.java Currently, we have set up an ARM CI to test out how Hive works on ARM platform: [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/] Among the failures, we have observed that some numeric operations can have different result across hardware archs, such as: [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/] [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/] [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_vectorized_math_funcs_/] we can see that the calculation results of log, exp, cos, toRadians etc is slitly different than the .out file results that we are comparing(they are tested and wrote on X86 machines), this is because of we use [Math Library|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html] for these kind of calculations. and according to the [illustration|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html]: _Unlike some of the numeric methods of class StrictMath, all implementations of the equivalent functions of class Math are not_ _defined to return the bit-for-bit same results. This relaxation permits better-performing implementations where strict reproducibility_ _is not required._ _By default many of the Math methods simply call the equivalent method in StrictMath for their implementation._ _Code generators are encouraged to use platform-specific native libraries or microprocessor instructions, where available,_ _to provide higher-performance implementations of Math methods._ so the result will have difference across hardware archs. On the other hand, JAVA provided another library [StrictMath|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html] that will not have this kind of problem as according to its' [reference|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html]: _To help ensure portability of Java programs, the definitions of some of the numeric functions in this package require that they produce_ _the same results as certain published algorithms._ So in order to fix the above mentioned problem, we have to consider switch to use StrictMath instead of Math. was: Use java.lang.StrictMath rather than java.lang.Math in UDFs: UDFCos.java UDFExp.java UDFLn.java UDFLog.java Currently, we have set up an ARM CI to test out how Hive works on ARM platform: [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/] Among the failures, we have observed that some numeric operations can have different result across hardware archs, such as: [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/] [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/] [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_vectorized_math_funcs_/] we can see that the calculation results of log, exp, cos, toRadians etc is slitly different than the .out file results that we are comparing(they are tested and wrote on X86 machines), this is because of we use [Math Library|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html] for these kind of calculations. and according to the [illustration|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html]: _Unlike some of the numeric methods of class StrictMath, all implementations of the equivalent functions of class Math are not_ _defined to return the bit-for-bit same results. This relaxation permits better-performing implementations where strict reproducibility_ _is not required._ _By default many of the Math methods simply call the equivalent method in StrictMath for their implementation._ _Code generators are encouraged to use platform-specific native libraries or microprocessor instructions, where available,_ _to provide higher-performance implementations of Math methods._ so the result will have difference across hardware archs. On the other hand, JAVA provided another library [StrictMath|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html] that will not have this kind of problem as according to its' [reference|h
[jira] [Commented] (HIVE-23451) FileSinkOperator calls deleteOnExit (hdfs call) twice for the same file
[
https://issues.apache.org/jira/browse/HIVE-23451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106121#comment-17106121
]
Hive QA commented on HIVE-23451:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002780/HIVE-23451.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 17268 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters1]
(batchId=31)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters]
(batchId=34)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_basic]
(batchId=29)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=32)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22304/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22304/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22304/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002780 - PreCommit-HIVE-Build
> FileSinkOperator calls deleteOnExit (hdfs call) twice for the same file
> ---
>
> Key: HIVE-23451
> URL: https://issues.apache.org/jira/browse/HIVE-23451
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-23451.1.patch, HIVE-23451.2.patch
>
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L826]
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L797]
> Can avoid a NN call here (i.e, mainly for small queries).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23458) Introduce unified thread pool for scheduled jobs
[
https://issues.apache.org/jira/browse/HIVE-23458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106123#comment-17106123
]
Hive QA commented on HIVE-23458:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002787/HIVE-23458.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22305/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22305/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22305/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-05-13 08:53:12.303
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-22305/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-05-13 08:53:12.307
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 9ffbbdc HIVE-23409 : If TezSession application reopen fails for
Timeline service down, default TezSession from SessionPool is closed after a
retry ( Naresh PR via Ashutosh Chauhan)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 9ffbbdc HIVE-23409 : If TezSession application reopen fails for
Timeline service down, default TezSession from SessionPool is closed after a
retry ( Naresh PR via Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-05-13 08:53:13.294
+ rm -rf ../yetus_PreCommit-HIVE-Build-22305
+ mkdir ../yetus_PreCommit-HIVE-Build-22305
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-22305
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-22305/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/PerPoolTriggerValidatorRunnable.java:18
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/exec/tez/PerPoolTriggerValidatorRunnable.java'
with conflicts.
error: patch failed:
service/src/java/org/apache/hive/service/server/HiveServer2.java:41
Falling back to three-way merge...
Applied patch to
'service/src/java/org/apache/hive/service/server/HiveServer2.java' cleanly.
Going to apply patch with: git apply -p0
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/PerPoolTriggerValidatorRunnable.java:18
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/exec/tez/PerPoolTriggerValidatorRunnable.java'
with conflicts.
error: patch failed:
service/src/java/org/apache/hive/service/server/HiveServer2.java:41
Falling back to three-way merge...
Applied patch to
'service/src/java/org/apache/hive/service/server/HiveServer2.java' cleanly.
U
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/PerPoolTriggerValidatorRunnable.java
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-22305
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002787 - PreCommit-HIVE-Build
> Introduce unified thread pool for scheduled jobs
>
>
> Key: HIVE-23458
> URL: https://issues.apache.org/jira/browse/HIVE-23458
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Eugene Chung
>Assignee: Eugene Chung
>Priority: Major
> Labels: todoc4.0
> Fix For: 4.
[jira] [Updated] (HIVE-23458) Introduce unified thread pool for scheduled jobs
[ https://issues.apache.org/jira/browse/HIVE-23458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Chung updated HIVE-23458: Status: Open (was: Patch Available) > Introduce unified thread pool for scheduled jobs > > > Key: HIVE-23458 > URL: https://issues.apache.org/jira/browse/HIVE-23458 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Eugene Chung >Assignee: Eugene Chung >Priority: Major > Labels: todoc4.0 > Fix For: 4.0.0 > > Attachments: HIVE-23458.01.patch > > > As I mentioned in [the comment of > HIVE-23164|https://issues.apache.org/jira/browse/HIVE-23164?focusedCommentId=17089506&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17089506], > I've made the unified scheduled executor service like > org.apache.hadoop.hive.metastore.ThreadPool. > I think it could help > 1. to minimize the possibility of making non-daemon threads when developers > need ScheduledExecutorService > 2. to achieve the utilization of server resources because the current > situation is all of the modules make its own ScheduledExecutorService and all > of the threads are just using for one job. > 3. administrators of Hive servers by providing > hive.exec.scheduler.num.threads configuration so that they can predict and > set how many threads are used and needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23458) Introduce unified thread pool for scheduled jobs
[ https://issues.apache.org/jira/browse/HIVE-23458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Chung updated HIVE-23458: Attachment: HIVE-23458.01.patch Status: Patch Available (was: Open) By mistake, I used my old forked repo. and it surely made the patch applying error. > Introduce unified thread pool for scheduled jobs > > > Key: HIVE-23458 > URL: https://issues.apache.org/jira/browse/HIVE-23458 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Eugene Chung >Assignee: Eugene Chung >Priority: Major > Labels: todoc4.0 > Fix For: 4.0.0 > > Attachments: HIVE-23458.01.patch, HIVE-23458.01.patch > > > As I mentioned in [the comment of > HIVE-23164|https://issues.apache.org/jira/browse/HIVE-23164?focusedCommentId=17089506&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17089506], > I've made the unified scheduled executor service like > org.apache.hadoop.hive.metastore.ThreadPool. > I think it could help > 1. to minimize the possibility of making non-daemon threads when developers > need ScheduledExecutorService > 2. to achieve the utilization of server resources because the current > situation is all of the modules make its own ScheduledExecutorService and all > of the threads are just using for one job. > 3. administrators of Hive servers by providing > hive.exec.scheduler.num.threads configuration so that they can predict and > set how many threads are used and needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23365) Put RS deduplication optimization under cost based decision
[ https://issues.apache.org/jira/browse/HIVE-23365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis reassigned HIVE-23365: -- Assignee: Stamatis Zampetakis > Put RS deduplication optimization under cost based decision > --- > > Key: HIVE-23365 > URL: https://issues.apache.org/jira/browse/HIVE-23365 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Stamatis Zampetakis >Priority: Major > > Currently, RS deduplication is always executed whenever it is semantically > correct. However, it could be beneficial to leave both RS operators in the > plan, e.g., if the NDV of the second RS is very low. Thus, we would like this > decision to be cost-based. We could use a simple heuristic that would work > fine for most of the cases without introducing regressions for existing > cases, e.g., if NDV for partition column is less than estimated parallelism > in the second RS, do not execute deduplication. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23444) Concurrent ACID direct inserts may fail with FileNotFoundException
[
https://issues.apache.org/jira/browse/HIVE-23444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106156#comment-17106156
]
Hive QA commented on HIVE-23444:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
26s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
30s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
49s{color} | {color:red} ql: The patch generated 1 new + 167 unchanged - 0
fixed = 168 total (was 167) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22306/dev-support/hive-personality.sh
|
| git revision | master / 9ffbbdc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22306/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22306/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Concurrent ACID direct inserts may fail with FileNotFoundException
> --
>
> Key: HIVE-23444
> URL: https://issues.apache.org/jira/browse/HIVE-23444
> Project: Hive
> Issue Type: Bug
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23444.1.patch, HIVE-23444.1.patch
>
>
> The following exception may occur when concurrently inserting into an ACID
> table with static partitions and the 'hive.acid.direct.insert.enabled'
> parameter is true. This issue occurs intermittently.
> {noformat}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: java.io.FileNotFoundException: File
> hdfs://ns1/warehouse/tablespace/managed/hive/tpch_unbucketed.db/concurrent_insert_partitioned/l_tax=0.0/_tmp.delta_001_001_
> does not exist.
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:2465)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:2228)
> ~[hive-exec-
[jira] [Updated] (HIVE-23133) Numeric operations can have different result across hardware archs
[ https://issues.apache.org/jira/browse/HIVE-23133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenyu Zheng updated HIVE-23133: Description: Use java.lang.StrictMath rather than java.lang.Math in UDFs for compatibility: UDFCos.java UDFExp.java UDFLn.java UDFLog.java UDFLog2.java UDFLog10.java Currently, we have set up an ARM CI to test out how Hive works on ARM platform: [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/] Among the failures, we have observed that some numeric operations can have different result across hardware archs, such as: [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/] [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/] [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_vectorized_math_funcs_/] we can see that the calculation results of log, exp, cos, toRadians etc is slitly different than the .out file results that we are comparing(they are tested and wrote on X86 machines), this is because of we use [Math Library|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html] for these kind of calculations. and according to the [illustration|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html]: _Unlike some of the numeric methods of class StrictMath, all implementations of the equivalent functions of class Math are not_ _defined to return the bit-for-bit same results. This relaxation permits better-performing implementations where strict reproducibility_ _is not required._ _By default many of the Math methods simply call the equivalent method in StrictMath for their implementation._ _Code generators are encouraged to use platform-specific native libraries or microprocessor instructions, where available,_ _to provide higher-performance implementations of Math methods._ so the result will have difference across hardware archs. On the other hand, JAVA provided another library [StrictMath|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html] that will not have this kind of problem as according to its' [reference|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html]: _To help ensure portability of Java programs, the definitions of some of the numeric functions in this package require that they produce_ _the same results as certain published algorithms._ So in order to fix the above mentioned problem, we have to consider switch to use StrictMath instead of Math. was: Use java.lang.StrictMath rather than java.lang.Math in UDFs: UDFCos.java UDFExp.java UDFLn.java UDFLog.java UDFLog2.java UDFLog10.java Currently, we have set up an ARM CI to test out how Hive works on ARM platform: [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/] Among the failures, we have observed that some numeric operations can have different result across hardware archs, such as: [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/] [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/] [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_vectorized_math_funcs_/] we can see that the calculation results of log, exp, cos, toRadians etc is slitly different than the .out file results that we are comparing(they are tested and wrote on X86 machines), this is because of we use [Math Library|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html] for these kind of calculations. and according to the [illustration|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html]: _Unlike some of the numeric methods of class StrictMath, all implementations of the equivalent functions of class Math are not_ _defined to return the bit-for-bit same results. This relaxation permits better-performing implementations where strict reproducibility_ _is not required._ _By default many of the Math methods simply call the equivalent method in StrictMath for their implementation._ _Code generators are encouraged to use platform-specific native libraries or microprocessor instructions, where available,_ _to provide higher-performance implementations of Math methods._ so the result will have difference across hardware archs. On the other hand, JAVA provided another library [StrictMath|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html] that will not have this kin
[jira] [Updated] (HIVE-23133) Numeric operations can have different result across hardware archs
[ https://issues.apache.org/jira/browse/HIVE-23133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenyu Zheng updated HIVE-23133: Description: Use java.lang.StrictMath rather than java.lang.Math for compatibility in the following UDFs: UDFCos.java UDFExp.java UDFLn.java UDFLog.java UDFLog2.java UDFLog10.java Currently, we have set up an ARM CI to test out how Hive works on ARM platform: [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/] Among the failures, we have observed that some numeric operations can have different result across hardware archs, such as: [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/] [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/] [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_vectorized_math_funcs_/] we can see that the calculation results of log, exp, cos, toRadians etc is slitly different than the .out file results that we are comparing(they are tested and wrote on X86 machines), this is because of we use [Math Library|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html] for these kind of calculations. and according to the [illustration|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html]: _Unlike some of the numeric methods of class StrictMath, all implementations of the equivalent functions of class Math are not_ _defined to return the bit-for-bit same results. This relaxation permits better-performing implementations where strict reproducibility_ _is not required._ _By default many of the Math methods simply call the equivalent method in StrictMath for their implementation._ _Code generators are encouraged to use platform-specific native libraries or microprocessor instructions, where available,_ _to provide higher-performance implementations of Math methods._ so the result will have difference across hardware archs. On the other hand, JAVA provided another library [StrictMath|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html] that will not have this kind of problem as according to its' [reference|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html]: _To help ensure portability of Java programs, the definitions of some of the numeric functions in this package require that they produce_ _the same results as certain published algorithms._ So in order to fix the above mentioned problem, we have to consider switch to use StrictMath instead of Math. was: Use java.lang.StrictMath rather than java.lang.Math in UDFs for compatibility: UDFCos.java UDFExp.java UDFLn.java UDFLog.java UDFLog2.java UDFLog10.java Currently, we have set up an ARM CI to test out how Hive works on ARM platform: [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/] Among the failures, we have observed that some numeric operations can have different result across hardware archs, such as: [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/] [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/] [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_vectorized_math_funcs_/] we can see that the calculation results of log, exp, cos, toRadians etc is slitly different than the .out file results that we are comparing(they are tested and wrote on X86 machines), this is because of we use [Math Library|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html] for these kind of calculations. and according to the [illustration|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html]: _Unlike some of the numeric methods of class StrictMath, all implementations of the equivalent functions of class Math are not_ _defined to return the bit-for-bit same results. This relaxation permits better-performing implementations where strict reproducibility_ _is not required._ _By default many of the Math methods simply call the equivalent method in StrictMath for their implementation._ _Code generators are encouraged to use platform-specific native libraries or microprocessor instructions, where available,_ _to provide higher-performance implementations of Math methods._ so the result will have difference across hardware archs. On the other hand, JAVA provided another library [StrictMath|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.h
[jira] [Comment Edited] (HIVE-23133) Numeric operations can have different result across hardware archs
[ https://issues.apache.org/jira/browse/HIVE-23133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106042#comment-17106042 ] Zhenyu Zheng edited comment on HIVE-23133 at 5/13/20, 10:03 AM: [~gopalv] Thanks for the review, in previous patches (1-3) we didn't add config, if that is the better way, we can keep it that way. The modified UDFs includes: UDFCos.java UDFExp.java UDFLn.java UDFLog.java UDFLog2.java UDFLog10.java I will also add the funcs that we changed to the ticket description. was (Author: kevin_zheng): [~gopalv] Thanks for the review, in previous patches (1-3) we didn't add config, if that is the better way, we can keep it that way. The modified UDFs includes: I will also add the funcs that we changed to the ticket description. > Numeric operations can have different result across hardware archs > -- > > Key: HIVE-23133 > URL: https://issues.apache.org/jira/browse/HIVE-23133 > Project: Hive > Issue Type: Sub-task >Reporter: Zhenyu Zheng >Assignee: Zhenyu Zheng >Priority: Major > Attachments: HIVE-23133.1.patch, HIVE-23133.2.patch, > HIVE-23133.3.patch, HIVE-23133.4.patch, HIVE-23133.5.patch, HIVE-23133.6.patch > > > Use java.lang.StrictMath rather than java.lang.Math for compatibility in the > following UDFs: > UDFCos.java > UDFExp.java > UDFLn.java > UDFLog.java > UDFLog2.java > UDFLog10.java > > Currently, we have set up an ARM CI to test out how Hive works on ARM > platform: > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/] > Among the failures, we have observed that some numeric operations can have > different result across hardware archs, such as: > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/] > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/] > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_vectorized_math_funcs_/] > we can see that the calculation results of log, exp, cos, toRadians etc is > slitly different than the .out file results that we are > comparing(they are tested and wrote on X86 machines), this is because of we > use [Math > Library|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html] for > these kind of calculations. > and according to the > [illustration|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html]: > _Unlike some of the numeric methods of class StrictMath, all implementations > of the equivalent functions of class Math are not_ > _defined to return the bit-for-bit same results. This relaxation permits > better-performing implementations where strict reproducibility_ > _is not required._ > _By default many of the Math methods simply call the equivalent method in > StrictMath for their implementation._ > _Code generators are encouraged to use platform-specific native libraries or > microprocessor instructions, where available,_ > _to provide higher-performance implementations of Math methods._ > so the result will have difference across hardware archs. > On the other hand, JAVA provided another library > [StrictMath|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html] > that will not have this kind of problem as according to its' > [reference|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html]: > _To help ensure portability of Java programs, the definitions of some of the > numeric functions in this package require that they produce_ > _the same results as certain published algorithms._ > So in order to fix the above mentioned problem, we have to consider switch to > use StrictMath instead of Math. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23444) Concurrent ACID direct inserts may fail with FileNotFoundException
[
https://issues.apache.org/jira/browse/HIVE-23444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106176#comment-17106176
]
Hive QA commented on HIVE-23444:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002794/HIVE-23444.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17269 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_delimited]
(batchId=225)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[special_character_in_tabnames_1]
(batchId=78)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22306/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22306/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22306/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002794 - PreCommit-HIVE-Build
> Concurrent ACID direct inserts may fail with FileNotFoundException
> --
>
> Key: HIVE-23444
> URL: https://issues.apache.org/jira/browse/HIVE-23444
> Project: Hive
> Issue Type: Bug
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23444.1.patch, HIVE-23444.1.patch
>
>
> The following exception may occur when concurrently inserting into an ACID
> table with static partitions and the 'hive.acid.direct.insert.enabled'
> parameter is true. This issue occurs intermittently.
> {noformat}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: java.io.FileNotFoundException: File
> hdfs://ns1/warehouse/tablespace/managed/hive/tpch_unbucketed.db/concurrent_insert_partitioned/l_tax=0.0/_tmp.delta_001_001_
> does not exist.
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:2465)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:2228)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.exec.MoveTask.handleStaticParts(MoveTask.java:522)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:442)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:359)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:721)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:488)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:482)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:225)
> ~[hive-service-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> ... 13 more
> Caused by: java.io.IOException: java.io.FileNotFoundException: File
> hdfs://ns1/warehouse/tablespace/managed/hive/tpch_unbucketed.db/concurrent_insert_partitioned/l_tax=0.0/_tmp.delta_001_001_
> does not exist.
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getHdfsDirSnapshots(AcidUtils.java:1472)
> ~[hive-exec-3.1.3000.7.1.1.0-493.ja
[jira] [Comment Edited] (HIVE-23133) Numeric operations can have different result across hardware archs
[ https://issues.apache.org/jira/browse/HIVE-23133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106042#comment-17106042 ] Zhenyu Zheng edited comment on HIVE-23133 at 5/13/20, 10:03 AM: [~gopalv] Thanks for the review, in previous patches (1-3) we didn't add config, if that is the better way, we can keep it that way. The modified UDFs includes: I will also add the funcs that we changed to the ticket description. was (Author: kevin_zheng): [~gopalv] Thanks for the review, in previous patches (1-3) we didn't add config, if that is the better way, we can keep it that way. I will add the funcs that we changed to the ticket description. > Numeric operations can have different result across hardware archs > -- > > Key: HIVE-23133 > URL: https://issues.apache.org/jira/browse/HIVE-23133 > Project: Hive > Issue Type: Sub-task >Reporter: Zhenyu Zheng >Assignee: Zhenyu Zheng >Priority: Major > Attachments: HIVE-23133.1.patch, HIVE-23133.2.patch, > HIVE-23133.3.patch, HIVE-23133.4.patch, HIVE-23133.5.patch, HIVE-23133.6.patch > > > Use java.lang.StrictMath rather than java.lang.Math for compatibility in the > following UDFs: > UDFCos.java > UDFExp.java > UDFLn.java > UDFLog.java > UDFLog2.java > UDFLog10.java > > Currently, we have set up an ARM CI to test out how Hive works on ARM > platform: > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/] > Among the failures, we have observed that some numeric operations can have > different result across hardware archs, such as: > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/] > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/] > [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_vectorized_math_funcs_/] > we can see that the calculation results of log, exp, cos, toRadians etc is > slitly different than the .out file results that we are > comparing(they are tested and wrote on X86 machines), this is because of we > use [Math > Library|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html] for > these kind of calculations. > and according to the > [illustration|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html]: > _Unlike some of the numeric methods of class StrictMath, all implementations > of the equivalent functions of class Math are not_ > _defined to return the bit-for-bit same results. This relaxation permits > better-performing implementations where strict reproducibility_ > _is not required._ > _By default many of the Math methods simply call the equivalent method in > StrictMath for their implementation._ > _Code generators are encouraged to use platform-specific native libraries or > microprocessor instructions, where available,_ > _to provide higher-performance implementations of Math methods._ > so the result will have difference across hardware archs. > On the other hand, JAVA provided another library > [StrictMath|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html] > that will not have this kind of problem as according to its' > [reference|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html]: > _To help ensure portability of Java programs, the definitions of some of the > numeric functions in this package require that they produce_ > _the same results as certain published algorithms._ > So in order to fix the above mentioned problem, we have to consider switch to > use StrictMath instead of Math. > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-23460) Add qoption to disable qtests
[
https://issues.apache.org/jira/browse/HIVE-23460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich reassigned HIVE-23460:
---
> Add qoption to disable qtests
> -
>
> Key: HIVE-23460
> URL: https://issues.apache.org/jira/browse/HIVE-23460
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-23460.01.patch
>
>
> instead other ways to exclude them... (testconfiguration.properties;
> CliConfig#excludeQuery)
> {code}
> --! qt:disabled:reason
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23460) Add qoption to disable qtests
[
https://issues.apache.org/jira/browse/HIVE-23460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-23460:
Attachment: HIVE-23460.01.patch
> Add qoption to disable qtests
> -
>
> Key: HIVE-23460
> URL: https://issues.apache.org/jira/browse/HIVE-23460
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-23460.01.patch
>
>
> instead other ways to exclude them... (testconfiguration.properties;
> CliConfig#excludeQuery)
> {code}
> --! qt:disabled:reason
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23460) Add qoption to disable qtests
[
https://issues.apache.org/jira/browse/HIVE-23460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-23460:
Status: Patch Available (was: Open)
> Add qoption to disable qtests
> -
>
> Key: HIVE-23460
> URL: https://issues.apache.org/jira/browse/HIVE-23460
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-23460.01.patch
>
>
> instead other ways to exclude them... (testconfiguration.properties;
> CliConfig#excludeQuery)
> {code}
> --! qt:disabled:reason
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23133) Numeric operations can have different result across hardware archs
[
https://issues.apache.org/jira/browse/HIVE-23133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106197#comment-17106197
]
Hive QA commented on HIVE-23133:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
35s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22307/dev-support/hive-personality.sh
|
| git revision | master / 9ffbbdc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: vector-code-gen ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22307/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Numeric operations can have different result across hardware archs
> --
>
> Key: HIVE-23133
> URL: https://issues.apache.org/jira/browse/HIVE-23133
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zhenyu Zheng
>Assignee: Zhenyu Zheng
>Priority: Major
> Attachments: HIVE-23133.1.patch, HIVE-23133.2.patch,
> HIVE-23133.3.patch, HIVE-23133.4.patch, HIVE-23133.5.patch, HIVE-23133.6.patch
>
>
> Use java.lang.StrictMath rather than java.lang.Math for compatibility in the
> following UDFs:
> UDFCos.java
> UDFExp.java
> UDFLn.java
> UDFLog.java
> UDFLog2.java
> UDFLog10.java
>
> Currently, we have set up an ARM CI to test out how Hive works on ARM
> platform:
> [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/]
> Among the failures, we have observed that some numeric operations can have
> different result across hardware archs, such as:
> [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/]
> [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/]
> [https://builds.apache.org/view/H-L/
[jira] [Commented] (HIVE-23133) Numeric operations can have different result across hardware archs
[
https://issues.apache.org/jira/browse/HIVE-23133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106213#comment-17106213
]
Hive QA commented on HIVE-23133:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002798/HIVE-23133.6.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17268 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22307/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22307/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22307/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002798 - PreCommit-HIVE-Build
> Numeric operations can have different result across hardware archs
> --
>
> Key: HIVE-23133
> URL: https://issues.apache.org/jira/browse/HIVE-23133
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zhenyu Zheng
>Assignee: Zhenyu Zheng
>Priority: Major
> Attachments: HIVE-23133.1.patch, HIVE-23133.2.patch,
> HIVE-23133.3.patch, HIVE-23133.4.patch, HIVE-23133.5.patch, HIVE-23133.6.patch
>
>
> Use java.lang.StrictMath rather than java.lang.Math for compatibility in the
> following UDFs:
> UDFCos.java
> UDFExp.java
> UDFLn.java
> UDFLog.java
> UDFLog2.java
> UDFLog10.java
>
> Currently, we have set up an ARM CI to test out how Hive works on ARM
> platform:
> [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/]
> Among the failures, we have observed that some numeric operations can have
> different result across hardware archs, such as:
> [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestCliDriver/testCliDriver_vector_decimal_udf2_/]
> [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_subquery_select_/]
> [https://builds.apache.org/view/H-L/view/Hive/job/Hive-linux-ARM-trunk/25/testReport/org.apache.hadoop.hive.cli/TestSparkCliDriver/testCliDriver_vectorized_math_funcs_/]
> we can see that the calculation results of log, exp, cos, toRadians etc is
> slitly different than the .out file results that we are
> comparing(they are tested and wrote on X86 machines), this is because of we
> use [Math
> Library|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html] for
> these kind of calculations.
> and according to the
> [illustration|https://docs.oracle.com/javase/6/docs/api/java/lang/Math.html]:
> _Unlike some of the numeric methods of class StrictMath, all implementations
> of the equivalent functions of class Math are not_
> _defined to return the bit-for-bit same results. This relaxation permits
> better-performing implementations where strict reproducibility_
> _is not required._
> _By default many of the Math methods simply call the equivalent method in
> StrictMath for their implementation._
> _Code generators are encouraged to use platform-specific native libraries or
> microprocessor instructions, where available,_
> _to provide higher-performance implementations of Math methods._
> so the result will have difference across hardware archs.
> On the other hand, JAVA provided another library
> [StrictMath|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html]
> that will not have this kind of problem as according to its'
> [reference|https://docs.oracle.com/javase/6/docs/api/java/lang/StrictMath.html]:
> _To help ensure portability of Java programs, the definitions of some of the
> numeric functions in this package require that they produce_
> _the same results as certain published algorithms._
> So in order to fix the above mentioned problem, we have to consider switch to
> use StrictMath instead of Math.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23458) Introduce unified thread pool for scheduled jobs
[
https://issues.apache.org/jira/browse/HIVE-23458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106216#comment-17106216
]
Hive QA commented on HIVE-23458:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002800/HIVE-23458.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22308/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22308/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22308/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-05-13 11:15:13.255
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-22308/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-05-13 11:15:13.258
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 9ffbbdc HIVE-23409 : If TezSession application reopen fails for
Timeline service down, default TezSession from SessionPool is closed after a
retry ( Naresh PR via Ashutosh Chauhan)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 9ffbbdc HIVE-23409 : If TezSession application reopen fails for
Timeline service down, default TezSession from SessionPool is closed after a
retry ( Naresh PR via Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-05-13 11:15:14.191
+ rm -rf ../yetus_PreCommit-HIVE-Build-22308
+ mkdir ../yetus_PreCommit-HIVE-Build-22308
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-22308
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-22308/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
Going to apply patch with: git apply -p0
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc6207579259697332681.exe, --version]
libprotoc 2.6.1
protoc-jar: executing: [/tmp/protoc6207579259697332681.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
protoc-jar: executing: [/tmp/protoc186711488808264886.exe, --version]
libprotoc 2.6.1
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/parser/target/generated-sources/antlr3/org/apache/hadoop/hive/ql/parse/HiveLexer.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/parser/src/java/org/apache/hadoop/hive/ql/parse/HiveLexer.g
org/apache/hadoop/hive/ql/parse/HiveLex
[jira] [Updated] (HIVE-23444) Concurrent ACID direct inserts may fail with FileNotFoundException
[
https://issues.apache.org/jira/browse/HIVE-23444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marta Kuczora updated HIVE-23444:
-
Attachment: HIVE-23444.1.patch
> Concurrent ACID direct inserts may fail with FileNotFoundException
> --
>
> Key: HIVE-23444
> URL: https://issues.apache.org/jira/browse/HIVE-23444
> Project: Hive
> Issue Type: Bug
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23444.1.patch, HIVE-23444.1.patch,
> HIVE-23444.1.patch
>
>
> The following exception may occur when concurrently inserting into an ACID
> table with static partitions and the 'hive.acid.direct.insert.enabled'
> parameter is true. This issue occurs intermittently.
> {noformat}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: java.io.FileNotFoundException: File
> hdfs://ns1/warehouse/tablespace/managed/hive/tpch_unbucketed.db/concurrent_insert_partitioned/l_tax=0.0/_tmp.delta_001_001_
> does not exist.
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:2465)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:2228)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.exec.MoveTask.handleStaticParts(MoveTask.java:522)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:442)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:359)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:721)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:488)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:482)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:225)
> ~[hive-service-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> ... 13 more
> Caused by: java.io.IOException: java.io.FileNotFoundException: File
> hdfs://ns1/warehouse/tablespace/managed/hive/tpch_unbucketed.db/concurrent_insert_partitioned/l_tax=0.0/_tmp.delta_001_001_
> does not exist.
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getHdfsDirSnapshots(AcidUtils.java:1472)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getAcidState(AcidUtils.java:1297)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getAcidFilesForStats(AcidUtils.java:2695)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:2448)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:2228)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.exec.MoveTask.handleStaticParts(MoveTask.java:522)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:442)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apac
[jira] [Updated] (HIVE-23434) Add option to rewrite PERCENTILE_CONT to sketch functions
[ https://issues.apache.org/jira/browse/HIVE-23434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-23434: Status: Patch Available (was: Open) > Add option to rewrite PERCENTILE_CONT to sketch functions > - > > Key: HIVE-23434 > URL: https://issues.apache.org/jira/browse/HIVE-23434 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-23434.01.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23434) Add option to rewrite PERCENTILE_CONT to sketch functions
[ https://issues.apache.org/jira/browse/HIVE-23434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-23434: Attachment: HIVE-23434.01.patch > Add option to rewrite PERCENTILE_CONT to sketch functions > - > > Key: HIVE-23434 > URL: https://issues.apache.org/jira/browse/HIVE-23434 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-23434.01.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23460) Add qoption to disable qtests
[
https://issues.apache.org/jira/browse/HIVE-23460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106241#comment-17106241
]
Hive QA commented on HIVE-23460:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
27s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
38s{color} | {color:blue} accumulo-handler in master has 20 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
0s{color} | {color:blue} itests/util in master has 54 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
47s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
33s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} The patch ql passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} The patch accumulo-handler passed checkstyle {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} itests/util: The patch generated 0 new + 59
unchanged - 6 fixed = 59 total (was 65) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22309/dev-support/hive-personality.sh
|
| git revision | master / 9ffbbdc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql accumulo-handler itests itests/util U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22309/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add qoption to disable qtests
> -
>
> Key: HIVE-23460
> URL: https://issues.apache.org/jira/browse/HIVE-23460
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-23460.01.patch
>
>
> instead other ways to exclude them... (testconfiguration.properties;
> CliConfig#excludeQuery)
> {code}
> --! qt:disabled:reason
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23460) Add qoption to disable qtests
[
https://issues.apache.org/jira/browse/HIVE-23460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106259#comment-17106259
]
Hive QA commented on HIVE-23460:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002807/HIVE-23460.01.patch
{color:green}SUCCESS:{color} +1 due to 38 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 17274 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_joins]
(batchId=222)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_subq_in]
(batchId=3)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_uri_import]
(batchId=17)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[cbo_query44]
(batchId=228)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[cbo_query45]
(batchId=228)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[cbo_query67]
(batchId=228)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[cbo_query70]
(batchId=228)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[cbo_query86]
(batchId=228)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query44]
(batchId=227)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query45]
(batchId=227)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query67]
(batchId=227)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query70]
(batchId=227)
org.apache.hadoop.hive.cli.TestTezPerfConstraintsCliDriver.testCliDriver[cbo_query86]
(batchId=227)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage
(batchId=182)
org.apache.hive.beeline.TestBeeLineWithArgs.testRowsAffected (batchId=208)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22309/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22309/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22309/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002807 - PreCommit-HIVE-Build
> Add qoption to disable qtests
> -
>
> Key: HIVE-23460
> URL: https://issues.apache.org/jira/browse/HIVE-23460
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-23460.01.patch
>
>
> instead other ways to exclude them... (testconfiguration.properties;
> CliConfig#excludeQuery)
> {code}
> --! qt:disabled:reason
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23434) Add option to rewrite PERCENTILE_CONT to sketch functions
[ https://issues.apache.org/jira/browse/HIVE-23434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106266#comment-17106266 ] Zoltan Haindrich commented on HIVE-23434: - opened an [rb|https://reviews.apache.org/r/72506/] in the hope that it will be able to track the file rename - apparently it's not... there is also a [PR|https://github.com/apache/hive/pull/1017] > Add option to rewrite PERCENTILE_CONT to sketch functions > - > > Key: HIVE-23434 > URL: https://issues.apache.org/jira/browse/HIVE-23434 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-23434.01.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23444) Concurrent ACID direct inserts may fail with FileNotFoundException
[
https://issues.apache.org/jira/browse/HIVE-23444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106294#comment-17106294
]
Hive QA commented on HIVE-23444:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
27s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
33s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
51s{color} | {color:red} ql: The patch generated 1 new + 167 unchanged - 0
fixed = 168 total (was 167) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 27s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22310/dev-support/hive-personality.sh
|
| git revision | master / 9ffbbdc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22310/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22310/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Concurrent ACID direct inserts may fail with FileNotFoundException
> --
>
> Key: HIVE-23444
> URL: https://issues.apache.org/jira/browse/HIVE-23444
> Project: Hive
> Issue Type: Bug
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23444.1.patch, HIVE-23444.1.patch,
> HIVE-23444.1.patch
>
>
> The following exception may occur when concurrently inserting into an ACID
> table with static partitions and the 'hive.acid.direct.insert.enabled'
> parameter is true. This issue occurs intermittently.
> {noformat}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: java.io.FileNotFoundException: File
> hdfs://ns1/warehouse/tablespace/managed/hive/tpch_unbucketed.db/concurrent_insert_partitioned/l_tax=0.0/_tmp.delta_001_001_
> does not exist.
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:2465)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.jav
[jira] [Work started] (HIVE-23375) Track MJ HashTable Load time
[ https://issues.apache.org/jira/browse/HIVE-23375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-23375 started by Panagiotis Garefalakis. - > Track MJ HashTable Load time > > > Key: HIVE-23375 > URL: https://issues.apache.org/jira/browse/HIVE-23375 > Project: Hive > Issue Type: Improvement >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23375.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Introduce TezCounter to track MJ HashTable Load time -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23461) Needs to capture input/output entities in explainRewrite
[
https://issues.apache.org/jira/browse/HIVE-23461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchao Li updated HIVE-23461:
--
Description:
HIVE-18778(CVE-2018-1314) capture input/output entitles in explain semantic
analyzer so when a query is disallowed by Ranger, Sentry or Sqlstd
authorizizer, the corresponding explain statement will be disallowed either.
However, ExplainSQRewriteSemanticAnalyzer also uses an instance of
DDLSemanticAnalyzer to analyze the explain rewrite query.
{code:java}
SemanticAnalyzer sem = (SemanticAnalyzer)
SemanticAnalyzerFactory.get(queryState, input);
sem.analyze(input, ctx);
sem.validate();{code}
The inputs/outputs entities for this query are never set on the instance of
ExplainSQRewriteSemanticAnalyzer itself and thus is not propagated into the
HookContext in the calling Driver code. It is a similar issue to HIVE-18778.
was:
HIVE-18778(CVE-2018-1314) capture input/output entitles in explain semantic
analyzer so when a query is disallowed by Ranger, Sentry or Sqlstd
authorizizer, the corresponding explain statement will be disallowed either.
However, ExplainSQRewriteSemanticAnalyzer also uses an instance of
DDLSemanticAnalyzer to analyze the explain rewrite query.
{code:java}
SemanticAnalyzer sem = (SemanticAnalyzer)
SemanticAnalyzerFactory.get(queryState, input);
sem.analyze(input, ctx);
sem.validate();
The inputs/outputs entities for this query are never set on the instance of
ExplainSQRewriteSemanticAnalyzer itself and thus is not propagated into the
HookContext in the calling Driver code. It is a similar issue to
HIVE-18778.{code}
> Needs to capture input/output entities in explainRewrite
>
>
> Key: HIVE-23461
> URL: https://issues.apache.org/jira/browse/HIVE-23461
> Project: Hive
> Issue Type: Improvement
>Reporter: Wenchao Li
>Priority: Major
>
> HIVE-18778(CVE-2018-1314) capture input/output entitles in explain semantic
> analyzer so when a query is disallowed by Ranger, Sentry or Sqlstd
> authorizizer, the corresponding explain statement will be disallowed either.
> However, ExplainSQRewriteSemanticAnalyzer also uses an instance of
> DDLSemanticAnalyzer to analyze the explain rewrite query.
> {code:java}
> SemanticAnalyzer sem = (SemanticAnalyzer)
> SemanticAnalyzerFactory.get(queryState, input);
> sem.analyze(input, ctx);
> sem.validate();{code}
>
> The inputs/outputs entities for this query are never set on the instance of
> ExplainSQRewriteSemanticAnalyzer itself and thus is not propagated into the
> HookContext in the calling Driver code. It is a similar issue to HIVE-18778.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23461) Needs to capture input/output entities in explainRewrite
[
https://issues.apache.org/jira/browse/HIVE-23461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchao Li updated HIVE-23461:
--
Issue Type: Bug (was: Improvement)
> Needs to capture input/output entities in explainRewrite
>
>
> Key: HIVE-23461
> URL: https://issues.apache.org/jira/browse/HIVE-23461
> Project: Hive
> Issue Type: Bug
>Reporter: Wenchao Li
>Priority: Major
>
> HIVE-18778(CVE-2018-1314) capture input/output entitles in explain semantic
> analyzer so when a query is disallowed by Ranger, Sentry or Sqlstd
> authorizizer, the corresponding explain statement will be disallowed either.
> However, ExplainSQRewriteSemanticAnalyzer also uses an instance of
> DDLSemanticAnalyzer to analyze the explain rewrite query.
> {code:java}
> SemanticAnalyzer sem = (SemanticAnalyzer)
> SemanticAnalyzerFactory.get(queryState, input);
> sem.analyze(input, ctx);
> sem.validate();{code}
>
> The inputs/outputs entities for this query are never set on the instance of
> ExplainSQRewriteSemanticAnalyzer itself and thus is not propagated into the
> HookContext in the calling Driver code. It is a similar issue to HIVE-18778.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23461) Needs to capture input/output entities in explainRewrite
[
https://issues.apache.org/jira/browse/HIVE-23461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchao Li reassigned HIVE-23461:
-
Assignee: Wenchao Li
> Needs to capture input/output entities in explainRewrite
>
>
> Key: HIVE-23461
> URL: https://issues.apache.org/jira/browse/HIVE-23461
> Project: Hive
> Issue Type: Bug
>Reporter: Wenchao Li
>Assignee: Wenchao Li
>Priority: Major
>
> HIVE-18778(CVE-2018-1314) capture input/output entitles in explain semantic
> analyzer so when a query is disallowed by Ranger, Sentry or Sqlstd
> authorizizer, the corresponding explain statement will be disallowed either.
> However, ExplainSQRewriteSemanticAnalyzer also uses an instance of
> DDLSemanticAnalyzer to analyze the explain rewrite query.
> {code:java}
> SemanticAnalyzer sem = (SemanticAnalyzer)
> SemanticAnalyzerFactory.get(queryState, input);
> sem.analyze(input, ctx);
> sem.validate();{code}
>
> The inputs/outputs entities for this query are never set on the instance of
> ExplainSQRewriteSemanticAnalyzer itself and thus is not propagated into the
> HookContext in the calling Driver code. It is a similar issue to HIVE-18778.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-23462) Add option to rewrite NTILE to sketch functions
[ https://issues.apache.org/jira/browse/HIVE-23462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich reassigned HIVE-23462: --- Assignee: Zoltan Haindrich > Add option to rewrite NTILE to sketch functions > --- > > Key: HIVE-23462 > URL: https://issues.apache.org/jira/browse/HIVE-23462 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23444) Concurrent ACID direct inserts may fail with FileNotFoundException
[
https://issues.apache.org/jira/browse/HIVE-23444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106312#comment-17106312
]
Hive QA commented on HIVE-23444:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002813/HIVE-23444.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17269 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniDruidKafkaCliDriver.testCliDriver[druidkafkamini_delimited]
(batchId=225)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22310/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22310/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22310/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002813 - PreCommit-HIVE-Build
> Concurrent ACID direct inserts may fail with FileNotFoundException
> --
>
> Key: HIVE-23444
> URL: https://issues.apache.org/jira/browse/HIVE-23444
> Project: Hive
> Issue Type: Bug
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-23444.1.patch, HIVE-23444.1.patch,
> HIVE-23444.1.patch
>
>
> The following exception may occur when concurrently inserting into an ACID
> table with static partitions and the 'hive.acid.direct.insert.enabled'
> parameter is true. This issue occurs intermittently.
> {noformat}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.io.IOException: java.io.FileNotFoundException: File
> hdfs://ns1/warehouse/tablespace/managed/hive/tpch_unbucketed.db/concurrent_insert_partitioned/l_tax=0.0/_tmp.delta_001_001_
> does not exist.
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:2465)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.metadata.Hive.loadPartition(Hive.java:2228)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.exec.MoveTask.handleStaticParts(MoveTask.java:522)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:442)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:359)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:721)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:488)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:482)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:225)
> ~[hive-service-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> ... 13 more
> Caused by: java.io.IOException: java.io.FileNotFoundException: File
> hdfs://ns1/warehouse/tablespace/managed/hive/tpch_unbucketed.db/concurrent_insert_partitioned/l_tax=0.0/_tmp.delta_001_001_
> does not exist.
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getHdfsDirSnapshots(AcidUtils.java:1472)
> ~[hive-exec-3.1.3000.7.1.1.0-493.jar:3.1.3000.7.1.1.0-493]
> at
> org.apache.hadoop.hive.ql.io.AcidUtils.getAcidState(Ac
[jira] [Commented] (HIVE-23434) Add option to rewrite PERCENTILE_CONT to sketch functions
[
https://issues.apache.org/jira/browse/HIVE-23434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106313#comment-17106313
]
Hive QA commented on HIVE-23434:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002816/HIVE-23434.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22311/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22311/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22311/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2020-05-13 13:39:14.344
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-22311/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2020-05-13 13:39:14.347
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 9ffbbdc HIVE-23409 : If TezSession application reopen fails for
Timeline service down, default TezSession from SessionPool is closed after a
retry ( Naresh PR via Ashutosh Chauhan)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 9ffbbdc HIVE-23409 : If TezSession application reopen fails for
Timeline service down, default TezSession from SessionPool is closed after a
retry ( Naresh PR via Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2020-05-13 13:39:15.422
+ rm -rf ../yetus_PreCommit-HIVE-Build-22311
+ mkdir ../yetus_PreCommit-HIVE-Build-22311
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-22311
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-22311/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Trying to apply the patch with -p0
Going to apply patch with: git apply -p0
/data/hiveptest/working/scratch/build.patch:682: trailing whitespace.
;
/data/hiveptest/working/scratch/build.patch:706: trailing whitespace.
;
/data/hiveptest/working/scratch/build.patch:886: trailing whitespace.
Map 1
/data/hiveptest/working/scratch/build.patch:911: trailing whitespace.
Reducer 2
/data/hiveptest/working/scratch/build.patch:1002: trailing whitespace.
Map 1
warning: squelched 5 whitespace errors
warning: 10 lines add whitespace errors.
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc3435626319438392730.exe, --version]
libprotoc 2.6.1
protoc-jar: executing: [/tmp/protoc3435626319438392730.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-common/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
protoc-jar: executing: [/tmp/protoc8254914134498978338.exe, --version]
libprotoc 2.6.1
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-server/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/standalone-metastore/metastore-s
[jira] [Updated] (HIVE-23458) Introduce unified thread pool for scheduled jobs
[ https://issues.apache.org/jira/browse/HIVE-23458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Chung updated HIVE-23458: Attachment: (was: HIVE-23458.01.patch) > Introduce unified thread pool for scheduled jobs > > > Key: HIVE-23458 > URL: https://issues.apache.org/jira/browse/HIVE-23458 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Eugene Chung >Assignee: Eugene Chung >Priority: Major > Labels: todoc4.0 > Fix For: 4.0.0 > > > As I mentioned in [the comment of > HIVE-23164|https://issues.apache.org/jira/browse/HIVE-23164?focusedCommentId=17089506&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17089506], > I've made the unified scheduled executor service like > org.apache.hadoop.hive.metastore.ThreadPool. > I think it could help > 1. to minimize the possibility of making non-daemon threads when developers > need ScheduledExecutorService > 2. to achieve the utilization of server resources because the current > situation is all of the modules make its own ScheduledExecutorService and all > of the threads are just using for one job. > 3. administrators of Hive servers by providing > hive.exec.scheduler.num.threads configuration so that they can predict and > set how many threads are used and needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23458) Introduce unified thread pool for scheduled jobs
[ https://issues.apache.org/jira/browse/HIVE-23458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Chung updated HIVE-23458: Attachment: (was: HIVE-23458.01.patch) > Introduce unified thread pool for scheduled jobs > > > Key: HIVE-23458 > URL: https://issues.apache.org/jira/browse/HIVE-23458 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Eugene Chung >Assignee: Eugene Chung >Priority: Major > Labels: todoc4.0 > Fix For: 4.0.0 > > > As I mentioned in [the comment of > HIVE-23164|https://issues.apache.org/jira/browse/HIVE-23164?focusedCommentId=17089506&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17089506], > I've made the unified scheduled executor service like > org.apache.hadoop.hive.metastore.ThreadPool. > I think it could help > 1. to minimize the possibility of making non-daemon threads when developers > need ScheduledExecutorService > 2. to achieve the utilization of server resources because the current > situation is all of the modules make its own ScheduledExecutorService and all > of the threads are just using for one job. > 3. administrators of Hive servers by providing > hive.exec.scheduler.num.threads configuration so that they can predict and > set how many threads are used and needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23458) Introduce unified thread pool for scheduled jobs
[ https://issues.apache.org/jira/browse/HIVE-23458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Chung updated HIVE-23458: Status: Open (was: Patch Available) > Introduce unified thread pool for scheduled jobs > > > Key: HIVE-23458 > URL: https://issues.apache.org/jira/browse/HIVE-23458 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Eugene Chung >Assignee: Eugene Chung >Priority: Major > Labels: todoc4.0 > Fix For: 4.0.0 > > > As I mentioned in [the comment of > HIVE-23164|https://issues.apache.org/jira/browse/HIVE-23164?focusedCommentId=17089506&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17089506], > I've made the unified scheduled executor service like > org.apache.hadoop.hive.metastore.ThreadPool. > I think it could help > 1. to minimize the possibility of making non-daemon threads when developers > need ScheduledExecutorService > 2. to achieve the utilization of server resources because the current > situation is all of the modules make its own ScheduledExecutorService and all > of the threads are just using for one job. > 3. administrators of Hive servers by providing > hive.exec.scheduler.num.threads configuration so that they can predict and > set how many threads are used and needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23375) Track MJ HashTable Load time
[ https://issues.apache.org/jira/browse/HIVE-23375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106318#comment-17106318 ] Ashutosh Chauhan commented on HIVE-23375: - Aggregated counter at vertex level is a good start. Along with min,max and total load time we will already get useful info here. +1 [~pgaref] can you reattach your patch to trigger Hive QA run. > Track MJ HashTable Load time > > > Key: HIVE-23375 > URL: https://issues.apache.org/jira/browse/HIVE-23375 > Project: Hive > Issue Type: Improvement >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23375.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Introduce TezCounter to track MJ HashTable Load time -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23458) Introduce unified thread pool for scheduled jobs
[ https://issues.apache.org/jira/browse/HIVE-23458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Chung updated HIVE-23458: Attachment: HIVE-23458.01.patch Status: Patch Available (was: Open) > Introduce unified thread pool for scheduled jobs > > > Key: HIVE-23458 > URL: https://issues.apache.org/jira/browse/HIVE-23458 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: Eugene Chung >Assignee: Eugene Chung >Priority: Major > Labels: todoc4.0 > Fix For: 4.0.0 > > Attachments: HIVE-23458.01.patch > > > As I mentioned in [the comment of > HIVE-23164|https://issues.apache.org/jira/browse/HIVE-23164?focusedCommentId=17089506&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17089506], > I've made the unified scheduled executor service like > org.apache.hadoop.hive.metastore.ThreadPool. > I think it could help > 1. to minimize the possibility of making non-daemon threads when developers > need ScheduledExecutorService > 2. to achieve the utilization of server resources because the current > situation is all of the modules make its own ScheduledExecutorService and all > of the threads are just using for one job. > 3. administrators of Hive servers by providing > hive.exec.scheduler.num.threads configuration so that they can predict and > set how many threads are used and needed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23375) Track MJ HashTable Load time
[ https://issues.apache.org/jira/browse/HIVE-23375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106321#comment-17106321 ] Panagiotis Garefalakis commented on HIVE-23375: --- Sure, latest patch should trigger precommit tests. > Track MJ HashTable Load time > > > Key: HIVE-23375 > URL: https://issues.apache.org/jira/browse/HIVE-23375 > Project: Hive > Issue Type: Improvement >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23375.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Introduce TezCounter to track MJ HashTable Load time -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23375) Track MJ HashTable Load time
[ https://issues.apache.org/jira/browse/HIVE-23375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-23375: -- Status: Patch Available (was: In Progress) > Track MJ HashTable Load time > > > Key: HIVE-23375 > URL: https://issues.apache.org/jira/browse/HIVE-23375 > Project: Hive > Issue Type: Improvement >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23375.01.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Introduce TezCounter to track MJ HashTable Load time -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23432) Add Ranger Replication Metrics
[ https://issues.apache.org/jira/browse/HIVE-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23432: --- Status: In Progress (was: Patch Available) > Add Ranger Replication Metrics > --- > > Key: HIVE-23432 > URL: https://issues.apache.org/jira/browse/HIVE-23432 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-23432.01.patch, HIVE-23432.02.patch, > HIVE-23432.03.patch, HIVE-23432.04.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23432) Add Ranger Replication Metrics
[ https://issues.apache.org/jira/browse/HIVE-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23432: --- Attachment: HIVE-23432.04.patch Status: Patch Available (was: In Progress) > Add Ranger Replication Metrics > --- > > Key: HIVE-23432 > URL: https://issues.apache.org/jira/browse/HIVE-23432 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-23432.01.patch, HIVE-23432.02.patch, > HIVE-23432.03.patch, HIVE-23432.04.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23434) Add option to rewrite PERCENTILE_CONT to sketch functions
[ https://issues.apache.org/jira/browse/HIVE-23434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-23434: Attachment: HIVE-23434.02.patch > Add option to rewrite PERCENTILE_CONT to sketch functions > - > > Key: HIVE-23434 > URL: https://issues.apache.org/jira/browse/HIVE-23434 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-23434.01.patch, HIVE-23434.02.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23460) Add qoption to disable qtests
[
https://issues.apache.org/jira/browse/HIVE-23460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-23460:
Attachment: HIVE-23460.02.patch
> Add qoption to disable qtests
> -
>
> Key: HIVE-23460
> URL: https://issues.apache.org/jira/browse/HIVE-23460
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-23460.01.patch, HIVE-23460.02.patch
>
>
> instead other ways to exclude them... (testconfiguration.properties;
> CliConfig#excludeQuery)
> {code}
> --! qt:disabled:reason
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23374) QueryDisplay must be threadsafe
[
https://issues.apache.org/jira/browse/HIVE-23374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106346#comment-17106346
]
Zoltan Haindrich commented on HIVE-23374:
-
[~abstractdog] Could you please take a look?
> QueryDisplay must be threadsafe
> ---
>
> Key: HIVE-23374
> URL: https://issues.apache.org/jira/browse/HIVE-23374
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-23374.01.patch, HIVE-23374.02.patch
>
>
> apparently right now it might bump into
> java.util.ConcurrentModificationException -s
> http://34.66.156.144:8080//job/hive-precommit/job/PR-7/41//testReport/junit/org.apache.hive.jdbc/TestJdbcWithMiniLlapRow/Testing___split_23___Archive___testInvalidReferenceCountScenario/
> {code}
> org.apache.hive.service.cli.HiveSQLException
> at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:360)
> at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:346)
> at
> org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:352)
> at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:239)
> at
> org.apache.hive.jdbc.BaseJdbcWithMiniLlap.testInvalidReferenceCountScenario(BaseJdbcWithMiniLlap.java:396)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> Caused by: java.util.ConcurrentModificationException: null
> at
> java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719)
> at
> java.util.LinkedHashMap$LinkedValueIterator.next(LinkedHashMap.java:747)
> at java.util.AbstractCollection.toArray(AbstractCollection.java:141)
> at java.util.ArrayList.addAll(ArrayList.java:581)
> at
> org.apache.hadoop.hive.ql.QueryDisplay.getTaskDisplays(QueryDisplay.java:273)
> at
> org.apache.hive.service.cli.operation.SQLOperation.getTaskStatus(SQLOperation.java:494)
> at
> org.apache.hive.service.cli.operation.Operation.getStatus(Operation.java:141)
> at
> org.apache.hive.service.cli.CLIService.getOperationStatus(CLIService.java:478)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.GetOperationStatus(ThriftCLIService.java:697)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetOperationStatus.getResult(TCLIService.java:1757)
> at
> org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetOperationStatus.getResult(TCLIService.java:1742)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23434) Add option to rewrite PERCENTILE_CONT to sketch functions
[ https://issues.apache.org/jira/browse/HIVE-23434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106350#comment-17106350 ] Zoltan Haindrich commented on HIVE-23434: - [~jcamachorodriguez] could you please take a look? > Add option to rewrite PERCENTILE_CONT to sketch functions > - > > Key: HIVE-23434 > URL: https://issues.apache.org/jira/browse/HIVE-23434 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-23434.01.patch, HIVE-23434.02.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23432) Add Ranger Replication Metrics
[ https://issues.apache.org/jira/browse/HIVE-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23432: --- Status: In Progress (was: Patch Available) > Add Ranger Replication Metrics > --- > > Key: HIVE-23432 > URL: https://issues.apache.org/jira/browse/HIVE-23432 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-23432.01.patch, HIVE-23432.02.patch, > HIVE-23432.03.patch, HIVE-23432.04.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23432) Add Ranger Replication Metrics
[ https://issues.apache.org/jira/browse/HIVE-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23432: --- Attachment: HIVE-23432.05.patch Status: Patch Available (was: In Progress) > Add Ranger Replication Metrics > --- > > Key: HIVE-23432 > URL: https://issues.apache.org/jira/browse/HIVE-23432 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-23432.01.patch, HIVE-23432.02.patch, > HIVE-23432.03.patch, HIVE-23432.04.patch, HIVE-23432.05.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23451) FileSinkOperator calls deleteOnExit (hdfs call) twice for the same file
[ https://issues.apache.org/jira/browse/HIVE-23451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23451: Status: Open (was: Patch Available) > FileSinkOperator calls deleteOnExit (hdfs call) twice for the same file > --- > > Key: HIVE-23451 > URL: https://issues.apache.org/jira/browse/HIVE-23451 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-23451.1.patch, HIVE-23451.2.patch > > > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L826] > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L797] > Can avoid a NN call here (i.e, mainly for small queries). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23451) FileSinkOperator calls deleteOnExit (hdfs call) twice for the same file
[ https://issues.apache.org/jira/browse/HIVE-23451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23451: Status: Patch Available (was: Open) +1 > FileSinkOperator calls deleteOnExit (hdfs call) twice for the same file > --- > > Key: HIVE-23451 > URL: https://issues.apache.org/jira/browse/HIVE-23451 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-23451.1.patch, HIVE-23451.2.patch, > HIVE-23451.3.patch > > > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L826] > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L797] > Can avoid a NN call here (i.e, mainly for small queries). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23451) FileSinkOperator calls deleteOnExit (hdfs call) twice for the same file
[ https://issues.apache.org/jira/browse/HIVE-23451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23451: Attachment: HIVE-23451.3.patch > FileSinkOperator calls deleteOnExit (hdfs call) twice for the same file > --- > > Key: HIVE-23451 > URL: https://issues.apache.org/jira/browse/HIVE-23451 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-23451.1.patch, HIVE-23451.2.patch, > HIVE-23451.3.patch > > > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L826] > [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java#L797] > Can avoid a NN call here (i.e, mainly for small queries). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23432) Add Ranger Replication Metrics
[
https://issues.apache.org/jira/browse/HIVE-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106364#comment-17106364
]
Hive QA commented on HIVE-23432:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
3s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
29s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
44s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} ql: The patch generated 0 new + 7 unchanged - 2
fixed = 7 total (was 9) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22312/dev-support/hive-personality.sh
|
| git revision | master / 9ffbbdc |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22312/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add Ranger Replication Metrics
> ---
>
> Key: HIVE-23432
> URL: https://issues.apache.org/jira/browse/HIVE-23432
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Attachments: HIVE-23432.01.patch, HIVE-23432.02.patch,
> HIVE-23432.03.patch, HIVE-23432.04.patch, HIVE-23432.05.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22675) Replace Base64 in hive-standalone-metastore Package
[ https://issues.apache.org/jira/browse/HIVE-22675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22675: -- Parent: HIVE-22611 Issue Type: Sub-task (was: Improvement) > Replace Base64 in hive-standalone-metastore Package > --- > > Key: HIVE-22675 > URL: https://issues.apache.org/jira/browse/HIVE-22675 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22675.1.patch, HIVE-22675.2.patch, > HIVE-22675.3.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22675) Replace Base64 in hive-standalone-metastore Package
[ https://issues.apache.org/jira/browse/HIVE-22675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22675: -- Attachment: HIVE-22675.3.patch > Replace Base64 in hive-standalone-metastore Package > --- > > Key: HIVE-22675 > URL: https://issues.apache.org/jira/browse/HIVE-22675 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22675.1.patch, HIVE-22675.2.patch, > HIVE-22675.3.patch, HIVE-22675.3.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23344) Bump scala version to 2.12.4, spark to 2.4.5
[ https://issues.apache.org/jira/browse/HIVE-23344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-23344: - Summary: Bump scala version to 2.12.4, spark to 2.4.5 (was: Bump scala version to 2.12.4, ) > Bump scala version to 2.12.4, spark to 2.4.5 > > > Key: HIVE-23344 > URL: https://issues.apache.org/jira/browse/HIVE-23344 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-23344.01.patch, HIVE-23344.01.patch, > HIVE-23344.02.patch, HIVE-23344.03.patch, HIVE-23344.03.patch, > HIVE-23344.04-dep-tree, HIVE-23344.04.patch > > > And bump up spark version, as 2.3.3 is not compatible with scala 2.12 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23344) Bump scala version to 2.12.4,
[ https://issues.apache.org/jira/browse/HIVE-23344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-23344: - Summary: Bump scala version to 2.12.4, (was: Bump org.scala-lang:scala-compiler version to 2.12.4) > Bump scala version to 2.12.4, > -- > > Key: HIVE-23344 > URL: https://issues.apache.org/jira/browse/HIVE-23344 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-23344.01.patch, HIVE-23344.01.patch, > HIVE-23344.02.patch, HIVE-23344.03.patch, HIVE-23344.03.patch, > HIVE-23344.04-dep-tree, HIVE-23344.04.patch > > > And bump up spark version, as 2.3.3 is not compatible with scala 2.12 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23338) Bump jackson version to 2.10.0
[ https://issues.apache.org/jira/browse/HIVE-23338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karen Coppage updated HIVE-23338: - Summary: Bump jackson version to 2.10.0 (was: Bump jackson-databind version up to 2.9.10.4) > Bump jackson version to 2.10.0 > -- > > Key: HIVE-23338 > URL: https://issues.apache.org/jira/browse/HIVE-23338 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Attachments: HIVE-23338-patch-3.txt, HIVE-23338.01.patch, > HIVE-23338.01.patch, HIVE-23338.01.patch, HIVE-23338.01.patch, > HIVE-23338.02.patch, HIVE-23338.03.patch, HIVE-23338.03.patch, > HIVE-23338.03.patch, HIVE-23338.04-dep-tree.txt, HIVE-23338.04.patch > > > And exclude a transitive dependency on > com.fasterxml.jackson.core:jackson-databind:2.6.5 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23432) Add Ranger Replication Metrics
[
https://issues.apache.org/jira/browse/HIVE-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106382#comment-17106382
]
Hive QA commented on HIVE-23432:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002825/HIVE-23432.04.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17270 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22312/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22312/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22312/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002825 - PreCommit-HIVE-Build
> Add Ranger Replication Metrics
> ---
>
> Key: HIVE-23432
> URL: https://issues.apache.org/jira/browse/HIVE-23432
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Attachments: HIVE-23432.01.patch, HIVE-23432.02.patch,
> HIVE-23432.03.patch, HIVE-23432.04.patch, HIVE-23432.05.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23338) Bump jackson version to 2.10.0
[ https://issues.apache.org/jira/browse/HIVE-23338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-23338: -- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks for the patch [~klcopp]! > Bump jackson version to 2.10.0 > -- > > Key: HIVE-23338 > URL: https://issues.apache.org/jira/browse/HIVE-23338 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23338-patch-3.txt, HIVE-23338.01.patch, > HIVE-23338.01.patch, HIVE-23338.01.patch, HIVE-23338.01.patch, > HIVE-23338.02.patch, HIVE-23338.03.patch, HIVE-23338.03.patch, > HIVE-23338.03.patch, HIVE-23338.04-dep-tree.txt, HIVE-23338.04.patch > > > And exclude a transitive dependency on > com.fasterxml.jackson.core:jackson-databind:2.6.5 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23344) Bump scala version to 2.12.4, spark to 2.4.5
[ https://issues.apache.org/jira/browse/HIVE-23344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-23344: -- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks for the patch [~klcopp]! > Bump scala version to 2.12.4, spark to 2.4.5 > > > Key: HIVE-23344 > URL: https://issues.apache.org/jira/browse/HIVE-23344 > Project: Hive > Issue Type: Improvement >Reporter: Karen Coppage >Assignee: Karen Coppage >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-23344.01.patch, HIVE-23344.01.patch, > HIVE-23344.02.patch, HIVE-23344.03.patch, HIVE-23344.03.patch, > HIVE-23344.04-dep-tree, HIVE-23344.04.patch > > > And bump up spark version, as 2.3.3 is not compatible with scala 2.12 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23099) Improve Logger for Operation Child Classes
[
https://issues.apache.org/jira/browse/HIVE-23099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23099:
--
Attachment: HIVE-23099.5.patch
> Improve Logger for Operation Child Classes
> --
>
> Key: HIVE-23099
> URL: https://issues.apache.org/jira/browse/HIVE-23099
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23099.1.patch, HIVE-23099.2.patch,
> HIVE-23099.3.patch, HIVE-23099.4.patch, HIVE-23099.4.patch,
> HIVE-23099.5.patch, HIVE-23099.5.patch
>
>
> The {{Operation}} class declares its logger this way:
> {code:java|title=Operation.java}
> public abstract class Operation {
> public static final Logger LOG =
> LoggerFactory.getLogger(Operation.class.getName());
> ...
> }
> {code}
> Notice that this is an {{abstract}} class, but the {{Logger}} is tied to the
> {{Operation.class.getName()}}. This means that logging cannot be controlled
> for each subclass of {{Operation}} independently since they all use the same
> static {{Logger}} instance.
> Make the LOG a {{protected}} instance variable that inherits the name of the
> child class.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23017) Use Inherited Logger in Tasks
[
https://issues.apache.org/jira/browse/HIVE-23017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23017:
--
Attachment: HIVE-23017.5.patch
> Use Inherited Logger in Tasks
> -
>
> Key: HIVE-23017
> URL: https://issues.apache.org/jira/browse/HIVE-23017
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23017.1.patch, HIVE-23017.2.patch,
> HIVE-23017.3.patch, HIVE-23017.4.patch, HIVE-23017.5.patch
>
>
> {code:java|title=FetchTask}
> public class FetchTask extends Task implements Serializable {
> private static final long serialVersionUID = 1L;
> private int maxRows = 100;
> private FetchOperator fetch;
> private ListSinkOperator sink;
> private int totalRows;
> private static transient final Logger LOG =
> LoggerFactory.getLogger(FetchTask.class);
> JobConf job = null;
> {code}
> The Tasks class is base class for many other. Cleaner to simply make the
> {{Logger}} {{protected}} and usable by other classes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22674) Replace Base64 in serde Package
[ https://issues.apache.org/jira/browse/HIVE-22674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22674: -- Attachment: HIVE-22674.2.patch > Replace Base64 in serde Package > --- > > Key: HIVE-22674 > URL: https://issues.apache.org/jira/browse/HIVE-22674 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22674.1.patch, HIVE-22674.2.patch, > HIVE-22674.2.patch, HIVE-22674.2.patch, HIVE-22674.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22678) Run Eclipse Cleanup Against hive-accumulo-handler Module
[ https://issues.apache.org/jira/browse/HIVE-22678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22678: -- Attachment: HIVE-22678.2.patch > Run Eclipse Cleanup Against hive-accumulo-handler Module > > > Key: HIVE-22678 > URL: https://issues.apache.org/jira/browse/HIVE-22678 > Project: Hive > Issue Type: Sub-task >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22678.1.patch, HIVE-22678.2.patch, > HIVE-22678.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22570) Review of ExprNodeDesc.java
[ https://issues.apache.org/jira/browse/HIVE-22570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22570: -- Attachment: HIVE-22570.1.patch > Review of ExprNodeDesc.java > --- > > Key: HIVE-22570 > URL: https://issues.apache.org/jira/browse/HIVE-22570 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Minor > Attachments: HIVE-22570.1.patch, HIVE-22570.1.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23375) Track MJ HashTable Load time
[
https://issues.apache.org/jira/browse/HIVE-23375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106413#comment-17106413
]
Hive QA commented on HIVE-23375:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
26s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
52s{color} | {color:red} ql: The patch generated 3 new + 29 unchanged - 1 fixed
= 32 total (was 30) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22313/dev-support/hive-personality.sh
|
| git revision | master / 472aca8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22313/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22313/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Track MJ HashTable Load time
>
>
> Key: HIVE-23375
> URL: https://issues.apache.org/jira/browse/HIVE-23375
> Project: Hive
> Issue Type: Improvement
>Reporter: Panagiotis Garefalakis
>Assignee: Panagiotis Garefalakis
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-23375.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Introduce TezCounter to track MJ HashTable Load time
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23449) LLAP: Reduce mkdir and config creations in submitWork hotpath
[ https://issues.apache.org/jira/browse/HIVE-23449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23449: Status: Patch Available (was: Open) > LLAP: Reduce mkdir and config creations in submitWork hotpath > - > > Key: HIVE-23449 > URL: https://issues.apache.org/jira/browse/HIVE-23449 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Major > Attachments: HIVE-23449.1.patch, HIVE-23449.2.patch, > HIVE-23449.3.patch, HIVE-23449.4.patch, Screenshot 2020-05-12 at 1.09.35 > PM.png > > > !Screenshot 2020-05-12 at 1.09.35 PM.png|width=885,height=558! > > For short jobs, submitWork gets into hotpath. This can lazy load conf and can > get rid of dir creations (which needs to be enabled only when DirWatcher is > enabled) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23449) LLAP: Reduce mkdir and config creations in submitWork hotpath
[ https://issues.apache.org/jira/browse/HIVE-23449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23449: Attachment: HIVE-23449.4.patch > LLAP: Reduce mkdir and config creations in submitWork hotpath > - > > Key: HIVE-23449 > URL: https://issues.apache.org/jira/browse/HIVE-23449 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Major > Attachments: HIVE-23449.1.patch, HIVE-23449.2.patch, > HIVE-23449.3.patch, HIVE-23449.4.patch, Screenshot 2020-05-12 at 1.09.35 > PM.png > > > !Screenshot 2020-05-12 at 1.09.35 PM.png|width=885,height=558! > > For short jobs, submitWork gets into hotpath. This can lazy load conf and can > get rid of dir creations (which needs to be enabled only when DirWatcher is > enabled) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23449) LLAP: Reduce mkdir and config creations in submitWork hotpath
[ https://issues.apache.org/jira/browse/HIVE-23449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-23449: Status: Open (was: Patch Available) > LLAP: Reduce mkdir and config creations in submitWork hotpath > - > > Key: HIVE-23449 > URL: https://issues.apache.org/jira/browse/HIVE-23449 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Major > Attachments: HIVE-23449.1.patch, HIVE-23449.2.patch, > HIVE-23449.3.patch, HIVE-23449.4.patch, Screenshot 2020-05-12 at 1.09.35 > PM.png > > > !Screenshot 2020-05-12 at 1.09.35 PM.png|width=885,height=558! > > For short jobs, submitWork gets into hotpath. This can lazy load conf and can > get rid of dir creations (which needs to be enabled only when DirWatcher is > enabled) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-18882) Do Not Hide Exception in Hive Metastore Client Connection
[
https://issues.apache.org/jira/browse/HIVE-18882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-18882:
--
Attachment: HIVE-18882.5.patch
> Do Not Hide Exception in Hive Metastore Client Connection
> -
>
> Key: HIVE-18882
> URL: https://issues.apache.org/jira/browse/HIVE-18882
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: noob
> Attachments: HIVE-18882.1.patch, HIVE-18882.2.patch,
> HIVE-18882.3.patch, HIVE-18882.4.patch, HIVE-18882.5.patch, HIVE-18882.patch
>
>
> [https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java#L526-L531]
>
> {code:java}
> if (LOG.isDebugEnabled()) {
> LOG.warn("Failed to connect to the MetaStore Server...", e);
> } else {
> // Don't print full exception trace if DEBUG is not on.
> LOG.warn("Failed to connect to the MetaStore Server...");
> }
> {code}
> I do not understand the logic here. I always want to see the reason for the
> failure. Otherwise, I do not know why it is failing unless I restart the
> server with debug logging enabled. By that point, the error may have
> cleared. Please just use the Exception in the WARN output without adding
> confusing logging for debugging. This is never an expected behavior... that
> enabling debug would change a _warn_ level log message.
> Also... please remove the ellipsis, they add no value.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-18882) Do Not Hide Exception in Hive Metastore Client Connection
[
https://issues.apache.org/jira/browse/HIVE-18882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106418#comment-17106418
]
David Mollitor commented on HIVE-18882:
---
A lot of this work was already covered in [HIVE-22419], but there are a couple
of little nits from this patch to submit.
> Do Not Hide Exception in Hive Metastore Client Connection
> -
>
> Key: HIVE-18882
> URL: https://issues.apache.org/jira/browse/HIVE-18882
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.0.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Labels: noob
> Attachments: HIVE-18882.1.patch, HIVE-18882.2.patch,
> HIVE-18882.3.patch, HIVE-18882.4.patch, HIVE-18882.5.patch, HIVE-18882.patch
>
>
> [https://github.com/apache/hive/blob/4047befe48c8f762c58d8854e058385c1df151c6/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java#L526-L531]
>
> {code:java}
> if (LOG.isDebugEnabled()) {
> LOG.warn("Failed to connect to the MetaStore Server...", e);
> } else {
> // Don't print full exception trace if DEBUG is not on.
> LOG.warn("Failed to connect to the MetaStore Server...");
> }
> {code}
> I do not understand the logic here. I always want to see the reason for the
> failure. Otherwise, I do not know why it is failing unless I restart the
> server with debug logging enabled. By that point, the error may have
> cleared. Please just use the Exception in the WARN output without adding
> confusing logging for debugging. This is never an expected behavior... that
> enabling debug would change a _warn_ level log message.
> Also... please remove the ellipsis, they add no value.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23099) Improve Logger for Operation Child Classes
[
https://issues.apache.org/jira/browse/HIVE-23099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-23099:
--
Attachment: HIVE-23099.5.patch
> Improve Logger for Operation Child Classes
> --
>
> Key: HIVE-23099
> URL: https://issues.apache.org/jira/browse/HIVE-23099
> Project: Hive
> Issue Type: Improvement
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Attachments: HIVE-23099.1.patch, HIVE-23099.2.patch,
> HIVE-23099.3.patch, HIVE-23099.4.patch, HIVE-23099.4.patch,
> HIVE-23099.5.patch, HIVE-23099.5.patch, HIVE-23099.5.patch
>
>
> The {{Operation}} class declares its logger this way:
> {code:java|title=Operation.java}
> public abstract class Operation {
> public static final Logger LOG =
> LoggerFactory.getLogger(Operation.class.getName());
> ...
> }
> {code}
> Notice that this is an {{abstract}} class, but the {{Logger}} is tied to the
> {{Operation.class.getName()}}. This means that logging cannot be controlled
> for each subclass of {{Operation}} independently since they all use the same
> static {{Logger}} instance.
> Make the LOG a {{protected}} instance variable that inherits the name of the
> child class.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23375) Track MJ HashTable Load time
[
https://issues.apache.org/jira/browse/HIVE-23375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106434#comment-17106434
]
Hive QA commented on HIVE-23375:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002167/HIVE-23375.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17268 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestPartitionManagement.testPartitionDiscoveryTransactionalTable
(batchId=152)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22313/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22313/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22313/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002167 - PreCommit-HIVE-Build
> Track MJ HashTable Load time
>
>
> Key: HIVE-23375
> URL: https://issues.apache.org/jira/browse/HIVE-23375
> Project: Hive
> Issue Type: Improvement
>Reporter: Panagiotis Garefalakis
>Assignee: Panagiotis Garefalakis
>Priority: Minor
> Labels: pull-request-available
> Attachments: HIVE-23375.01.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Introduce TezCounter to track MJ HashTable Load time
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-23375) Track MJ HashTable Load time
[ https://issues.apache.org/jira/browse/HIVE-23375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated HIVE-23375: -- Attachment: HIVE-23375.02.patch > Track MJ HashTable Load time > > > Key: HIVE-23375 > URL: https://issues.apache.org/jira/browse/HIVE-23375 > Project: Hive > Issue Type: Improvement >Reporter: Panagiotis Garefalakis >Assignee: Panagiotis Garefalakis >Priority: Minor > Labels: pull-request-available > Attachments: HIVE-23375.01.patch, HIVE-23375.02.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Introduce TezCounter to track MJ HashTable Load time -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23463) Investigate why the results have changed at udf_mask_show_last_n.q, udf_mask_show_first_n.q, udf_mask_last_n.q, udf_mask_first_n.q, udf_mask.q
[ https://issues.apache.org/jira/browse/HIVE-23463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-23463: -- Description: Check [https://reviews.apache.org/r/72491/] for details. > Investigate why the results have changed at udf_mask_show_last_n.q, > udf_mask_show_first_n.q, udf_mask_last_n.q, udf_mask_first_n.q, udf_mask.q > -- > > Key: HIVE-23463 > URL: https://issues.apache.org/jira/browse/HIVE-23463 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Priority: Major > > Check [https://reviews.apache.org/r/72491/] for details. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23432) Add Ranger Replication Metrics
[ https://issues.apache.org/jira/browse/HIVE-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106479#comment-17106479 ] PRAVIN KUMAR SINHA commented on HIVE-23432: --- +1 > Add Ranger Replication Metrics > --- > > Key: HIVE-23432 > URL: https://issues.apache.org/jira/browse/HIVE-23432 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-23432.01.patch, HIVE-23432.02.patch, > HIVE-23432.03.patch, HIVE-23432.04.patch, HIVE-23432.05.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23458) Introduce unified thread pool for scheduled jobs
[
https://issues.apache.org/jira/browse/HIVE-23458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106492#comment-17106492
]
Hive QA commented on HIVE-23458:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
42s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
31s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
50s{color} | {color:blue} service in master has 51 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
44s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
50s{color} | {color:red} ql: The patch generated 4 new + 32 unchanged - 2 fixed
= 36 total (was 34) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
18s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22314/dev-support/hive-personality.sh
|
| git revision | master / 472aca8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22314/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22314/yetus/patch-asflicense-problems.txt
|
| modules | C: common ql service U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22314/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Introduce unified thread pool for scheduled jobs
>
>
> Key: HIVE-23458
> URL: https://issues.apache.org/jira/browse/HIVE-23458
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Eugene Chung
>Assignee: Eugene Chung
>Priority: Major
> Labels: todoc4.0
> Fix For: 4.0.0
>
> Attachments: HIVE-23458.01.patch
>
>
> As I mentioned in [the comment of
> HIVE-23164|https://issues.apache.org/jira/browse/HIVE-23164?focusedCommentId=17089506&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17089506],
> I've made the unified scheduled executor service like
> org.apache.hadoop.hive.metastore.T
[jira] [Assigned] (HIVE-23466) ZK registry base should remove only specific instance instead of host
[ https://issues.apache.org/jira/browse/HIVE-23466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran reassigned HIVE-23466: > ZK registry base should remove only specific instance instead of host > - > > Key: HIVE-23466 > URL: https://issues.apache.org/jira/browse/HIVE-23466 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > > When ZKRegistryBase detects new ZK nodes it maintains path based cache and > host based cache. The host based cached already handles multiple instances > running in same host. But even if single instance is removed all instances > belonging to the host are removed. > Another issue is that, if single host has multiple instances it returns a Set > with no ordering. Ideally, we want the newest instance to be top of the set > (use TreeSet maybe?). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23458) Introduce unified thread pool for scheduled jobs
[
https://issues.apache.org/jira/browse/HIVE-23458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106509#comment-17106509
]
Hive QA commented on HIVE-23458:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002823/HIVE-23458.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17268 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22314/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22314/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22314/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002823 - PreCommit-HIVE-Build
> Introduce unified thread pool for scheduled jobs
>
>
> Key: HIVE-23458
> URL: https://issues.apache.org/jira/browse/HIVE-23458
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: Eugene Chung
>Assignee: Eugene Chung
>Priority: Major
> Labels: todoc4.0
> Fix For: 4.0.0
>
> Attachments: HIVE-23458.01.patch
>
>
> As I mentioned in [the comment of
> HIVE-23164|https://issues.apache.org/jira/browse/HIVE-23164?focusedCommentId=17089506&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17089506],
> I've made the unified scheduled executor service like
> org.apache.hadoop.hive.metastore.ThreadPool.
> I think it could help
> 1. to minimize the possibility of making non-daemon threads when developers
> need ScheduledExecutorService
> 2. to achieve the utilization of server resources because the current
> situation is all of the modules make its own ScheduledExecutorService and all
> of the threads are just using for one job.
> 3. administrators of Hive servers by providing
> hive.exec.scheduler.num.threads configuration so that they can predict and
> set how many threads are used and needed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23434) Add option to rewrite PERCENTILE_CONT to sketch functions
[
https://issues.apache.org/jira/browse/HIVE-23434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106570#comment-17106570
]
Hive QA commented on HIVE-23434:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002826/HIVE-23434.02.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 17265 tests
executed
*Failed tests:*
{noformat}
TestStatsReplicationScenariosACID - did not produce a TEST-*.xml file (likely
timed out) (batchId=184)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sketches_materialized_view_rollup2]
(batchId=45)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sketches_materialized_view_safety]
(batchId=81)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22315/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22315/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22315/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002826 - PreCommit-HIVE-Build
> Add option to rewrite PERCENTILE_CONT to sketch functions
> -
>
> Key: HIVE-23434
> URL: https://issues.apache.org/jira/browse/HIVE-23434
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Attachments: HIVE-23434.01.patch, HIVE-23434.02.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-23434) Add option to rewrite PERCENTILE_CONT to sketch functions
[
https://issues.apache.org/jira/browse/HIVE-23434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106590#comment-17106590
]
Hive QA commented on HIVE-23434:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
30s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} common in master has 63 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
27s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 9m
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
29s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
49s{color} | {color:red} ql: The patch generated 7 new + 107 unchanged - 0
fixed = 114 total (was 107) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
43s{color} | {color:red} root: The patch generated 7 new + 484 unchanged - 0
fixed = 491 total (was 484) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
49s{color} | {color:red} patch/common cannot run setBugDatabaseInfo from
findbugs {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 8m
13s{color} | {color:red} patch/ql cannot run setBugDatabaseInfo from findbugs
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 6m
34s{color} | {color:red} root in the patch failed. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 84m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22315/dev-support/hive-personality.sh
|
| git revision | master / 472aca8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22315/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22315/yetus/diff-checkstyle-root.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22315/yetus/whitespace-eol.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22315/yetus/patch-findbugs-common.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22315/yetus/patch-findbugs-ql.txt
|
| javadoc |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22315/yetus/patch-javadoc-root.txt
|
| modules | C: common ql . itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22315/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add option to rewrite PERCENTILE_CONT to sketch functions
>
[jira] [Commented] (HIVE-23432) Add Ranger Replication Metrics
[
https://issues.apache.org/jira/browse/HIVE-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106615#comment-17106615
]
Hive QA commented on HIVE-23432:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
28s{color} | {color:blue} ql in master has 1527 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
44s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
30s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
34s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
49s{color} | {color:red} ql: The patch generated 4 new + 3 unchanged - 6 fixed
= 7 total (was 9) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 37m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-22316/dev-support/hive-personality.sh
|
| git revision | master / 472aca8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22316/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-22316/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add Ranger Replication Metrics
> ---
>
> Key: HIVE-23432
> URL: https://issues.apache.org/jira/browse/HIVE-23432
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Attachments: HIVE-23432.01.patch, HIVE-23432.02.patch,
> HIVE-23432.03.patch, HIVE-23432.04.patch, HIVE-23432.05.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22066) Upgrade Apache parent POM to version 21
[ https://issues.apache.org/jira/browse/HIVE-22066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106616#comment-17106616 ] Ashutosh Chauhan commented on HIVE-22066: - [~belugabehr] you want to refresh the patch and create RB for it? > Upgrade Apache parent POM to version 21 > --- > > Key: HIVE-22066 > URL: https://issues.apache.org/jira/browse/HIVE-22066 > Project: Hive > Issue Type: Improvement >Reporter: Jesus Camacho Rodriguez >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-22066.2.patch, HIVE-22066.3.patch, > HIVE-22066.4.patch, HIVE-22066.5.patch, HIVE-22066.6.patch, > HIVE-22066.6.patch, HIVE-22066.7.patch, HIVE-22066.8.patch, > HIVE-22066.8.patch, HIVE-22066.9.patch, HIVE-22066.999.patch, > HIVE-22066.patch, HIVE-22066.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23432) Add Ranger Replication Metrics
[ https://issues.apache.org/jira/browse/HIVE-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23432: --- Status: In Progress (was: Patch Available) > Add Ranger Replication Metrics > --- > > Key: HIVE-23432 > URL: https://issues.apache.org/jira/browse/HIVE-23432 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-23432.01.patch, HIVE-23432.02.patch, > HIVE-23432.03.patch, HIVE-23432.04.patch, HIVE-23432.05.patch, > HIVE-23432.06.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-23432) Add Ranger Replication Metrics
[ https://issues.apache.org/jira/browse/HIVE-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated HIVE-23432: --- Attachment: HIVE-23432.06.patch Status: Patch Available (was: In Progress) > Add Ranger Replication Metrics > --- > > Key: HIVE-23432 > URL: https://issues.apache.org/jira/browse/HIVE-23432 > Project: Hive > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: HIVE-23432.01.patch, HIVE-23432.02.patch, > HIVE-23432.03.patch, HIVE-23432.04.patch, HIVE-23432.05.patch, > HIVE-23432.06.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-23432) Add Ranger Replication Metrics
[
https://issues.apache.org/jira/browse/HIVE-23432?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17106635#comment-17106635
]
Hive QA commented on HIVE-23432:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/13002831/HIVE-23432.05.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 17270 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.exec.repl.TestRangerDumpTask.testSuccessNonEmptyRangerPolicies
(batchId=291)
org.apache.hadoop.hive.ql.exec.repl.TestRangerDumpTask.testSuccessRangerDumpMetrics
(batchId=291)
org.apache.hadoop.hive.ql.exec.repl.TestRangerDumpTask.testSuccessValidAuthProviderEndpoint
(batchId=291)
org.apache.hadoop.hive.ql.exec.repl.TestRangerLoadTask.testSuccessNonEmptyRangerPolicies
(batchId=292)
org.apache.hadoop.hive.ql.exec.repl.TestRangerLoadTask.testSuccessRangerDumpMetrics
(batchId=292)
org.apache.hadoop.hive.ql.parse.TestReplAcrossInstancesWithJsonMessageFormat.testRangerReplication
(batchId=191)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcrossInstances.testRangerReplication
(batchId=195)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/22316/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/22316/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-22316/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 13002831 - PreCommit-HIVE-Build
> Add Ranger Replication Metrics
> ---
>
> Key: HIVE-23432
> URL: https://issues.apache.org/jira/browse/HIVE-23432
> Project: Hive
> Issue Type: Task
>Reporter: Aasha Medhi
>Assignee: Aasha Medhi
>Priority: Major
> Attachments: HIVE-23432.01.patch, HIVE-23432.02.patch,
> HIVE-23432.03.patch, HIVE-23432.04.patch, HIVE-23432.05.patch,
> HIVE-23432.06.patch
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
