[jira] [Commented] (HIVE-22315) Support Decimal64 column division with decimal64 scalar
[
https://issues.apache.org/jira/browse/HIVE-22315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951658#comment-16951658
]
Hive QA commented on HIVE-22315:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12983000/HIVE-22315.3.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17539 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18992/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18992/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18992/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12983000 - PreCommit-HIVE-Build
> Support Decimal64 column division with decimal64 scalar
> ---
>
> Key: HIVE-22315
> URL: https://issues.apache.org/jira/browse/HIVE-22315
> Project: Hive
> Issue Type: Bug
>Reporter: Ramesh Kumar Thangarajan
>Assignee: Ramesh Kumar Thangarajan
>Priority: Major
> Attachments: HIVE-22315.1.patch, HIVE-22315.2.patch,
> HIVE-22315.3.patch
>
>
> Currently division operation is not supported for Decimal64 column. This Jira
> will take care of supporting decimal64 column division with a decimal64
> scalar.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HIVE-21761) Support table level replication in Hive
[
https://issues.apache.org/jira/browse/HIVE-21761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan resolved HIVE-21761.
-
Resolution: Fixed
All patches committed to master.
> Support table level replication in Hive
> ---
>
> Key: HIVE-21761
> URL: https://issues.apache.org/jira/browse/HIVE-21761
> Project: Hive
> Issue Type: New Feature
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, Replication
>
> *Requirements:*
> {code:java}
> - User needs to define replication policy to replicate any specific table.
> This enables user to replicate only the business critical tables instead of
> replicating all tables which may throttle the network bandwidth, storage and
> also slow-down Hive replication.
> - User needs to define replication policy using regular expressions (such as
> db.sales_*) and needs to include additional tables which are non-matching
> given pattern and exclude some tables which are matching given pattern.
> - User needs to dynamically add/remove tables to the list either by manually
> changing the replication policy during run time.
> {code}
> *Design:*
> {code:java}
> 1. Hive continue to support DB level replication policy of format
> but logically, we support the policy as .'t1|t3| …'.'t*'.
> 2. Regular expression can also be supported as replication policy. For
> example,
> a. .''
> b. .'<*suffix>'
> c. .''
> d. .''
> 3. User can provide include and exclude list to specify the tables to be
> included in the replication policy.
> a. Include list specifies the tables to be included.
> b. Exclude list specifies the tables to be excluded even if it satisfies
> the expression in include list.
> c. So the tables included in the policy is a-b.
> d. For backward compatibility, if no include or exclude list is given, then
> all the tables will be included in
> the policy.
> 4. New format for the Replication policy have 3 parts all separated with Dot
> (.).
> a. First part is DB name.
> b. Second part is included list. Valid java regex within single quote.
> c. Third part is excluded list. Valid java regex within single quote.
> - -- Full DB replication which is currently supported
> - .'.*?' -- Full DB replication
> - .'t1|t3' -- DB replication with static list of tables t1 and
> t3 included.
> - .'(t1*)|t2'.'t100' -- DB replication with all tables having
> prefix t1 and also include table t2 which doesn’t have prefix t1 and exclude
> t100 which has the prefix t1.
> 5. If the DB property “repl.source.for” is set, then by default all the
> tables in the DB will be enabled for replication and will continue to archive
> deleted data to CM path.
> 6. REPL DUMP takes 2 inputs along with existing FROM and WITH clause.
> a. REPL DUMP [REPLACE FROM
> WITH ;
> current_repl_policy and previous_repl_policy can be any format mentioned in
> Point-4.
> b. REPLACE clause to be supported to take previous repl policy as input.
> c. Rest of the format remains same.
> 7. Now, REPL DUMP on this DB will replicate the tables based on
> current_repl_policy.
> 8. Single table replication of format .t1 is not supported. User can
> provide the same with .'t1' format.
> 9. If any table is added dynamically either due to change in regular
> expression or added to include list should be bootstrapped.
> a. Hive will automatically figure out the list of tables newly included in
> the list by comparing the current_repl_policy & previous_repl_policy inputs
> and combine bootstrap dump for added tables as part of incremental dump. As
> we can combine first incremental with bootstrap dump, it removes the current
> limitation of target DB being inconsistent after bootstrap unless we run
> first incremental replication.
> b. If any table is renamed, then it may gets dynamically added/removed for
> replication based on defined replication policy + include/exclude list. So,
> Hive will perform bootstrap for the table which is just included after
> rename.
> c. Also, if renamed table is excluded from replication policy, then need to
> drop the old table at target as well.
> 10. Only the initial bootstrap load expects the target DB to be empty but the
> intermediate bootstrap on tables due to regex or inclusion/exclusion list
> change or renames doesn’t expect the target DB or table to be empty. If any
> table with same name exist during such bootstrap, the table will be
> overwritten including data.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21761) Support table level replication in Hive
[
https://issues.apache.org/jira/browse/HIVE-21761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sankar Hariappan updated HIVE-21761:
Fix Version/s: 4.0.0
> Support table level replication in Hive
> ---
>
> Key: HIVE-21761
> URL: https://issues.apache.org/jira/browse/HIVE-21761
> Project: Hive
> Issue Type: New Feature
> Components: repl
>Reporter: Sankar Hariappan
>Assignee: Sankar Hariappan
>Priority: Major
> Labels: DR, Replication
> Fix For: 4.0.0
>
>
> *Requirements:*
> {code:java}
> - User needs to define replication policy to replicate any specific table.
> This enables user to replicate only the business critical tables instead of
> replicating all tables which may throttle the network bandwidth, storage and
> also slow-down Hive replication.
> - User needs to define replication policy using regular expressions (such as
> db.sales_*) and needs to include additional tables which are non-matching
> given pattern and exclude some tables which are matching given pattern.
> - User needs to dynamically add/remove tables to the list either by manually
> changing the replication policy during run time.
> {code}
> *Design:*
> {code:java}
> 1. Hive continue to support DB level replication policy of format
> but logically, we support the policy as .'t1|t3| …'.'t*'.
> 2. Regular expression can also be supported as replication policy. For
> example,
> a. .''
> b. .'<*suffix>'
> c. .''
> d. .''
> 3. User can provide include and exclude list to specify the tables to be
> included in the replication policy.
> a. Include list specifies the tables to be included.
> b. Exclude list specifies the tables to be excluded even if it satisfies
> the expression in include list.
> c. So the tables included in the policy is a-b.
> d. For backward compatibility, if no include or exclude list is given, then
> all the tables will be included in
> the policy.
> 4. New format for the Replication policy have 3 parts all separated with Dot
> (.).
> a. First part is DB name.
> b. Second part is included list. Valid java regex within single quote.
> c. Third part is excluded list. Valid java regex within single quote.
> - -- Full DB replication which is currently supported
> - .'.*?' -- Full DB replication
> - .'t1|t3' -- DB replication with static list of tables t1 and
> t3 included.
> - .'(t1*)|t2'.'t100' -- DB replication with all tables having
> prefix t1 and also include table t2 which doesn’t have prefix t1 and exclude
> t100 which has the prefix t1.
> 5. If the DB property “repl.source.for” is set, then by default all the
> tables in the DB will be enabled for replication and will continue to archive
> deleted data to CM path.
> 6. REPL DUMP takes 2 inputs along with existing FROM and WITH clause.
> a. REPL DUMP [REPLACE FROM
> WITH ;
> current_repl_policy and previous_repl_policy can be any format mentioned in
> Point-4.
> b. REPLACE clause to be supported to take previous repl policy as input.
> c. Rest of the format remains same.
> 7. Now, REPL DUMP on this DB will replicate the tables based on
> current_repl_policy.
> 8. Single table replication of format .t1 is not supported. User can
> provide the same with .'t1' format.
> 9. If any table is added dynamically either due to change in regular
> expression or added to include list should be bootstrapped.
> a. Hive will automatically figure out the list of tables newly included in
> the list by comparing the current_repl_policy & previous_repl_policy inputs
> and combine bootstrap dump for added tables as part of incremental dump. As
> we can combine first incremental with bootstrap dump, it removes the current
> limitation of target DB being inconsistent after bootstrap unless we run
> first incremental replication.
> b. If any table is renamed, then it may gets dynamically added/removed for
> replication based on defined replication policy + include/exclude list. So,
> Hive will perform bootstrap for the table which is just included after
> rename.
> c. Also, if renamed table is excluded from replication policy, then need to
> drop the old table at target as well.
> 10. Only the initial bootstrap load expects the target DB to be empty but the
> intermediate bootstrap on tables due to regex or inclusion/exclusion list
> change or renames doesn’t expect the target DB or table to be empty. If any
> table with same name exist during such bootstrap, the table will be
> overwritten including data.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21926) REPL - With table list - "TO" and "FROM" clause should not be allowed along with table filter list
[ https://issues.apache.org/jira/browse/HIVE-21926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-21926: Parent: (was: HIVE-21761) Issue Type: Bug (was: Sub-task) > REPL - With table list - "TO" and "FROM" clause should not be allowed along > with table filter list > -- > > Key: HIVE-21926 > URL: https://issues.apache.org/jira/browse/HIVE-21926 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: DR, Replication, pull-request-available > > If table filter list is provided during dump command, there is a chance that > tables will be bootstrapped during rename and during policy replace. If all > existing events are not replayed completely, then it may happen that the > remaining events in the next replication cycle causes data corruption. So all > the existing events should be replayed and "TO" and "FROM" should not be > allowed if table filter list is provided. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22175) TestBudyAllocator#testMTT test is flaky
[
https://issues.apache.org/jira/browse/HIVE-22175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951618#comment-16951618
]
Hive QA commented on HIVE-22175:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982998/HIVE-22175.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 17540 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestObjectStore.catalogs (batchId=234)
org.apache.hadoop.hive.metastore.TestObjectStore.testDatabaseOps (batchId=234)
org.apache.hadoop.hive.metastore.TestObjectStore.testDeprecatedConfigIsOverwritten
(batchId=234)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropParitionsCleanup
(batchId=234)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSQLDropPartitionsCacheCrossSession
(batchId=234)
org.apache.hadoop.hive.metastore.TestObjectStore.testDirectSqlErrorMetrics
(batchId=234)
org.apache.hadoop.hive.metastore.TestObjectStore.testEmptyTrustStoreProps
(batchId=234)
org.apache.hadoop.hive.metastore.TestObjectStore.testMasterKeyOps (batchId=234)
org.apache.hadoop.hive.metastore.TestObjectStore.testMaxEventResponse
(batchId=234)
org.apache.hadoop.hive.metastore.TestObjectStore.testPartitionOps (batchId=234)
org.apache.hadoop.hive.metastore.TestObjectStore.testQueryCloseOnError
(batchId=234)
org.apache.hadoop.hive.metastore.TestObjectStore.testRoleOps (batchId=234)
org.apache.hadoop.hive.metastore.TestObjectStore.testTableOps (batchId=234)
org.apache.hadoop.hive.metastore.TestObjectStore.testUseSSLProperty
(batchId=234)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18991/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18991/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18991/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982998 - PreCommit-HIVE-Build
> TestBudyAllocator#testMTT test is flaky
> ---
>
> Key: HIVE-22175
> URL: https://issues.apache.org/jira/browse/HIVE-22175
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Ádám Szita
>Assignee: John Sherman
>Priority: Major
> Attachments: HIVE-22175.1.patch, HIVE-22175.2.patch
>
>
> This test has a fail rate of about 20%-25%
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22240) Function percentile_cont fails when array parameter passed
[
https://issues.apache.org/jira/browse/HIVE-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22240:
--
Status: Open (was: Patch Available)
> Function percentile_cont fails when array parameter passed
> --
>
> Key: HIVE-22240
> URL: https://issues.apache.org/jira/browse/HIVE-22240
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22240.1.patch, HIVE-22240.2.patch,
> HIVE-22240.3.patch, HIVE-22240.3.patch, HIVE-22240.4.patch
>
>
> {code}
> SELECT
> percentile_cont(array(0.2, 0.5, 0.9)) WITHIN GROUP (ORDER BY value)
> FROM t_test;
> {code}
> hive.log:
> {code}
> 2019-09-24T21:00:43,203 ERROR [LocalJobRunner Map Task Executor #0]
> mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:573)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:148)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.util.ArrayList cannot be cast to
> org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:793)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:128)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:152)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
> ... 11 more
> Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast
> to org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileCont$PercentileContEvaluator.iterate(GenericUDAFPercentileCont.java:259)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:214)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:639)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:814)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:720)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:788)
> ... 17 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22240) Function percentile_cont fails when array parameter passed
[
https://issues.apache.org/jira/browse/HIVE-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22240:
--
Attachment: HIVE-22240.4.patch
> Function percentile_cont fails when array parameter passed
> --
>
> Key: HIVE-22240
> URL: https://issues.apache.org/jira/browse/HIVE-22240
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22240.1.patch, HIVE-22240.2.patch,
> HIVE-22240.3.patch, HIVE-22240.3.patch, HIVE-22240.4.patch
>
>
> {code}
> SELECT
> percentile_cont(array(0.2, 0.5, 0.9)) WITHIN GROUP (ORDER BY value)
> FROM t_test;
> {code}
> hive.log:
> {code}
> 2019-09-24T21:00:43,203 ERROR [LocalJobRunner Map Task Executor #0]
> mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:573)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:148)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.util.ArrayList cannot be cast to
> org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:793)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:128)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:152)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
> ... 11 more
> Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast
> to org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileCont$PercentileContEvaluator.iterate(GenericUDAFPercentileCont.java:259)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:214)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:639)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:814)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:720)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:788)
> ... 17 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22240) Function percentile_cont fails when array parameter passed
[
https://issues.apache.org/jira/browse/HIVE-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22240:
--
Status: Patch Available (was: Open)
> Function percentile_cont fails when array parameter passed
> --
>
> Key: HIVE-22240
> URL: https://issues.apache.org/jira/browse/HIVE-22240
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22240.1.patch, HIVE-22240.2.patch,
> HIVE-22240.3.patch, HIVE-22240.3.patch, HIVE-22240.4.patch
>
>
> {code}
> SELECT
> percentile_cont(array(0.2, 0.5, 0.9)) WITHIN GROUP (ORDER BY value)
> FROM t_test;
> {code}
> hive.log:
> {code}
> 2019-09-24T21:00:43,203 ERROR [LocalJobRunner Map Task Executor #0]
> mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:573)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:148)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.util.ArrayList cannot be cast to
> org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:793)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:128)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:152)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
> ... 11 more
> Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast
> to org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileCont$PercentileContEvaluator.iterate(GenericUDAFPercentileCont.java:259)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:214)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:639)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:814)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:720)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:788)
> ... 17 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22175) TestBudyAllocator#testMTT test is flaky
[
https://issues.apache.org/jira/browse/HIVE-22175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951566#comment-16951566
]
Hive QA commented on HIVE-22175:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
46s{color} | {color:blue} llap-server in master has 90 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 55s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18991/dev-support/hive-personality.sh
|
| git revision | master / ce855d8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18991/yetus/patch-asflicense-problems.txt
|
| modules | C: llap-server U: llap-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18991/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TestBudyAllocator#testMTT test is flaky
> ---
>
> Key: HIVE-22175
> URL: https://issues.apache.org/jira/browse/HIVE-22175
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Ádám Szita
>Assignee: John Sherman
>Priority: Major
> Attachments: HIVE-22175.1.patch, HIVE-22175.2.patch
>
>
> This test has a fail rate of about 20%-25%
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22312) MapJoinCounterHook doesnot work for tez
[ https://issues.apache.org/jira/browse/HIVE-22312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951553#comment-16951553 ] Pulkit Sharma commented on HIVE-22312: -- [~jcamachorodriguez] Can you please take a look? > MapJoinCounterHook doesnot work for tez > --- > > Key: HIVE-22312 > URL: https://issues.apache.org/jira/browse/HIVE-22312 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: All Versions >Reporter: Pulkit Sharma >Assignee: Pulkit Sharma >Priority: Major > Attachments: HIVE-22312.01.patch, HIVE-22312.02.patch, > HIVE-22312.03.patch > > > In [HIVE-1792|https://issues.apache.org/jira/browse/HIVE-1792], > MapJoinCounterHook hook was added to track joins that get converted to map > join. This hook gets List of Tasks from hookContext and check Tag associated > with each task. For mr, we create Conditional tasks in case of joins and add > tags for respective join conversions. This does not work in case of tez as we > only create TezTask(no Conditional Task is created) which can handle multiple > joins in contrast to one Conditional Task per join in mr. > The current approach will fail even if we add tag to TezTask as it can have > multiple joins of the same type which will require counter. > One possible solution for tez, is to parse query-plan after query completion > which we get from hookContext to get workGraph. Using workGraph, we can walk > through Operator Tree to find join conversions. > If this approach looks good, I can raise Pull Request > cc [~ashutoshc] [~jcamachorodriguez] [~pxiong] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22339) Change default time for MVs refresh in registry
[
https://issues.apache.org/jira/browse/HIVE-22339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951547#comment-16951547
]
Hive QA commented on HIVE-22339:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982996/HIVE-22339.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17537 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18990/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18990/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18990/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982996 - PreCommit-HIVE-Build
> Change default time for MVs refresh in registry
> ---
>
> Key: HIVE-22339
> URL: https://issues.apache.org/jira/browse/HIVE-22339
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Minor
> Attachments: HIVE-22339.patch
>
>
> Default was set to 60secs in HIVE-21344. It seems it may be too aggressive;
> suggestion is to change default to 1500secs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22105) Update ORC to 1.5.6.
[ https://issues.apache.org/jira/browse/HIVE-22105?focusedWorklogId=328283&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328283 ] ASF GitHub Bot logged work on HIVE-22105: - Author: ASF GitHub Bot Created on: 15/Oct/19 03:08 Start Date: 15/Oct/19 03:08 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #751: HIVE-22105 Update ORC to 1.5.6. URL: https://github.com/apache/hive/pull/751 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 328283) Time Spent: 20m (was: 10m) > Update ORC to 1.5.6. > > > Key: HIVE-22105 > URL: https://issues.apache.org/jira/browse/HIVE-22105 > Project: Hive > Issue Type: Bug >Reporter: Owen O'Malley >Assignee: Owen O'Malley >Priority: Major > Labels: pull-request-available > Fix For: 3.2.0, 3.1.2 > > Time Spent: 20m > Remaining Estimate: 0h > > ORC has had some important fixes in the 1.5 branch and they should be picked > up by Hive. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22074) Slow compilation due to IN to OR transformation
[ https://issues.apache.org/jira/browse/HIVE-22074?focusedWorklogId=328282&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328282 ] ASF GitHub Bot logged work on HIVE-22074: - Author: ASF GitHub Bot Created on: 15/Oct/19 03:07 Start Date: 15/Oct/19 03:07 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #746: HIVE-22074: Slow compilation due to IN to OR transformation URL: https://github.com/apache/hive/pull/746 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 328282) Time Spent: 1.5h (was: 1h 20m) > Slow compilation due to IN to OR transformation > --- > > Key: HIVE-22074 > URL: https://issues.apache.org/jira/browse/HIVE-22074 > Project: Hive > Issue Type: Improvement > Components: Logical Optimizer >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22074.1.patch, HIVE-22074.2.patch, > HIVE-22074.3.patch, HIVE-22074.4.patch, HIVE-22074.5.patch, HIVE-22074.6.patch > > Time Spent: 1.5h > Remaining Estimate: 0h > > Currently Hive transform IN expressions to OR to apply various CBO rules. > This incur significant performance hit if IN consist of large number of > expressions. > It is better to not transform IN expressions to OR in such cases because > overall benefit of various optimizations/transformations is unrealized due to > the compilation overhead -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22107) Correlated subquery producing wrong schema
[
https://issues.apache.org/jira/browse/HIVE-22107?focusedWorklogId=328280&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328280
]
ASF GitHub Bot logged work on HIVE-22107:
-
Author: ASF GitHub Bot
Created on: 15/Oct/19 03:05
Start Date: 15/Oct/19 03:05
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #755: HIVE-22107
Correlated subquery producing wrong schema
URL: https://github.com/apache/hive/pull/755
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 328280)
Time Spent: 20m (was: 10m)
> Correlated subquery producing wrong schema
> --
>
> Key: HIVE-22107
> URL: https://issues.apache.org/jira/browse/HIVE-22107
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 4.0.0
>Reporter: Vineet Garg
>Assignee: Vineet Garg
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22107.1.patch, HIVE-22107.2.patch,
> HIVE-22107.3.patch, HIVE-22107.4.patch, HIVE-22107.5.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> *Repro*
> {code:sql}
> create table test(id int, name string,dept string);
> insert into test values(1,'a','it'),(2,'b','eee'),(NULL, 'c', 'cse');
> select distinct 'empno' as eid, a.id from test a where NOT EXISTS (select
> c.id from test c where a.id=c.id);
> {code}
> {code}
> +---++
> | eid | a.id |
> +---++
> | NULL | empno |
> +---++
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-21924) Split text files even if header/footer exists
[
https://issues.apache.org/jira/browse/HIVE-21924?focusedWorklogId=328279&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328279
]
ASF GitHub Bot logged work on HIVE-21924:
-
Author: ASF GitHub Bot
Created on: 15/Oct/19 03:04
Start Date: 15/Oct/19 03:04
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #791: HIVE-21924
URL: https://github.com/apache/hive/pull/791
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 328279)
Time Spent: 4h 40m (was: 4.5h)
> Split text files even if header/footer exists
> -
>
> Key: HIVE-21924
> URL: https://issues.apache.org/jira/browse/HIVE-21924
> Project: Hive
> Issue Type: Improvement
> Components: File Formats
>Affects Versions: 2.4.0, 4.0.0, 3.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Mustafa Iman
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21924.2.patch, HIVE-21924.3.patch,
> HIVE-21924.4.patch, HIVE-21924.5.patch, HIVE-21924.6.patch, HIVE-21924.patch
>
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> https://github.com/apache/hive/blob/967a1cc98beede8e6568ce750ebeb6e0d048b8ea/ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java#L494-L503
>
> {code}
> int headerCount = 0;
> int footerCount = 0;
> if (table != null) {
> headerCount = Utilities.getHeaderCount(table);
> footerCount = Utilities.getFooterCount(table, conf);
> if (headerCount != 0 || footerCount != 0) {
> // Input file has header or footer, cannot be splitted.
> HiveConf.setLongVar(conf, ConfVars.MAPREDMINSPLITSIZE,
> Long.MAX_VALUE);
> }
> }
> {code}
> this piece of code makes the CSV (or any text files with header/footer) files
> not splittable if header or footer is present.
> If only header is present, we can find the offset after first line break and
> use that to split. Similarly for footer, may be read few KB's of data at the
> end and find the last line break offset. Use that to determine the data range
> which can be used for splitting. Few reads during split generation are
> cheaper than not splitting the file at all.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-20983) Vectorization: Scale up small hashtables, when collisions are detected
[ https://issues.apache.org/jira/browse/HIVE-20983?focusedWorklogId=328278&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328278 ] ASF GitHub Bot logged work on HIVE-20983: - Author: ASF GitHub Bot Created on: 15/Oct/19 03:03 Start Date: 15/Oct/19 03:03 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #771: HIVE-20983 URL: https://github.com/apache/hive/pull/771 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 328278) Time Spent: 20m (was: 10m) > Vectorization: Scale up small hashtables, when collisions are detected > -- > > Key: HIVE-20983 > URL: https://issues.apache.org/jira/browse/HIVE-20983 > Project: Hive > Issue Type: Bug >Reporter: Gopal Vijayaraghavan >Assignee: Mustafa Iman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-20983.1.patch, HIVE-20983.2.patch, > HIVE-20983.3.patch, HIVE-20983.4.patch, HIVE-20983.5.patch > > Time Spent: 20m > Remaining Estimate: 0h > > Hive's hashtable estimates are getting better with HyperLogLog stats in > place, but an accurate estimate does not always result in a low number of > collisions. > The hashtables which contain a very small number of items tend to lose their > O(1) lookup performance where there are collisions. Since collisions are easy > to detect within the fast hashtable implementation, a rehashing to a higher > size will help these small hashtables avoid collisions and go back to O(1) > perf. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22338) Shade dependent jars into the kudu-handler
[ https://issues.apache.org/jira/browse/HIVE-22338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22338: --- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master, thanks [~granthenke]. > Shade dependent jars into the kudu-handler > -- > > Key: HIVE-22338 > URL: https://issues.apache.org/jira/browse/HIVE-22338 > Project: Hive > Issue Type: Sub-task >Reporter: Grant Henke >Assignee: Grant Henke >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22338.0.patch > > Time Spent: 20m > Remaining Estimate: 0h > > We should shade the Kudu client and it’s async dependency into the > kudu-handler jar to simplify add jar style usage and ensure the > addDependencyJars method adds all the required classes. This is a common > pattern the other storage handlers use. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22338) Shade dependent jars into the kudu-handler
[ https://issues.apache.org/jira/browse/HIVE-22338?focusedWorklogId=328274&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328274 ] ASF GitHub Bot logged work on HIVE-22338: - Author: ASF GitHub Bot Created on: 15/Oct/19 03:00 Start Date: 15/Oct/19 03:00 Worklog Time Spent: 10m Work Description: asfgit commented on pull request #816: HIVE-22338: Shade dependent jars into the kudu-handler URL: https://github.com/apache/hive/pull/816 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 328274) Time Spent: 20m (was: 10m) > Shade dependent jars into the kudu-handler > -- > > Key: HIVE-22338 > URL: https://issues.apache.org/jira/browse/HIVE-22338 > Project: Hive > Issue Type: Sub-task >Reporter: Grant Henke >Assignee: Grant Henke >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22338.0.patch > > Time Spent: 20m > Remaining Estimate: 0h > > We should shade the Kudu client and it’s async dependency into the > kudu-handler jar to simplify add jar style usage and ensure the > addDependencyJars method adds all the required classes. This is a common > pattern the other storage handlers use. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22339) Change default time for MVs refresh in registry
[
https://issues.apache.org/jira/browse/HIVE-22339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951510#comment-16951510
]
Hive QA commented on HIVE-22339:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
17s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
35s{color} | {color:blue} common in master has 65 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18990/dev-support/hive-personality.sh
|
| git revision | master / 9044fee |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18990/yetus/patch-asflicense-problems.txt
|
| modules | C: common U: common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18990/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Change default time for MVs refresh in registry
> ---
>
> Key: HIVE-22339
> URL: https://issues.apache.org/jira/browse/HIVE-22339
> Project: Hive
> Issue Type: Improvement
> Components: Materialized views
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Minor
> Attachments: HIVE-22339.patch
>
>
> Default was set to 60secs in HIVE-21344. It seems it may be too aggressive;
> suggestion is to change default to 1500secs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22338) Shade dependent jars into the kudu-handler
[
https://issues.apache.org/jira/browse/HIVE-22338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951491#comment-16951491
]
Hive QA commented on HIVE-22338:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982984/HIVE-22338.0.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17537 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18989/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18989/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18989/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982984 - PreCommit-HIVE-Build
> Shade dependent jars into the kudu-handler
> --
>
> Key: HIVE-22338
> URL: https://issues.apache.org/jira/browse/HIVE-22338
> Project: Hive
> Issue Type: Sub-task
>Reporter: Grant Henke
>Assignee: Grant Henke
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22338.0.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We should shade the Kudu client and it’s async dependency into the
> kudu-handler jar to simplify add jar style usage and ensure the
> addDependencyJars method adds all the required classes. This is a common
> pattern the other storage handlers use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22342) HMS Translation: HIVE-22189 too strict with location for EXTERNAL tables
[ https://issues.apache.org/jira/browse/HIVE-22342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-22342: - Status: Patch Available (was: In Progress) > HMS Translation: HIVE-22189 too strict with location for EXTERNAL tables > > > Key: HIVE-22342 > URL: https://issues.apache.org/jira/browse/HIVE-22342 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-22342.patch > > > HIVE-22189 restricts EXTERNAL tables being created to be restricted to the > EXTERNAL_WAREHOUSE_DIR. This might be too strict as any other location should > be allowed as long as the location is outside the MANAGED warehouse directory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-22342) HMS Translation: HIVE-22189 too strict with location for EXTERNAL tables
[ https://issues.apache.org/jira/browse/HIVE-22342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-22342 started by Naveen Gangam. > HMS Translation: HIVE-22189 too strict with location for EXTERNAL tables > > > Key: HIVE-22342 > URL: https://issues.apache.org/jira/browse/HIVE-22342 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-22342.patch > > > HIVE-22189 restricts EXTERNAL tables being created to be restricted to the > EXTERNAL_WAREHOUSE_DIR. This might be too strict as any other location should > be allowed as long as the location is outside the MANAGED warehouse directory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22342) HMS Translation: HIVE-22189 too strict with location for EXTERNAL tables
[ https://issues.apache.org/jira/browse/HIVE-22342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam updated HIVE-22342: - Attachment: HIVE-22342.patch > HMS Translation: HIVE-22189 too strict with location for EXTERNAL tables > > > Key: HIVE-22342 > URL: https://issues.apache.org/jira/browse/HIVE-22342 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > Attachments: HIVE-22342.patch > > > HIVE-22189 restricts EXTERNAL tables being created to be restricted to the > EXTERNAL_WAREHOUSE_DIR. This might be too strict as any other location should > be allowed as long as the location is outside the MANAGED warehouse directory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22342) HMS Translation: HIVE-22189 too strict with location for EXTERNAL tables
[ https://issues.apache.org/jira/browse/HIVE-22342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Naveen Gangam reassigned HIVE-22342: > HMS Translation: HIVE-22189 too strict with location for EXTERNAL tables > > > Key: HIVE-22342 > URL: https://issues.apache.org/jira/browse/HIVE-22342 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Naveen Gangam >Assignee: Naveen Gangam >Priority: Major > > HIVE-22189 restricts EXTERNAL tables being created to be restricted to the > EXTERNAL_WAREHOUSE_DIR. This might be too strict as any other location should > be allowed as long as the location is outside the MANAGED warehouse directory. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22338) Shade dependent jars into the kudu-handler
[
https://issues.apache.org/jira/browse/HIVE-22338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951460#comment-16951460
]
Hive QA commented on HIVE-22338:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} kudu-handler in master has 1 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} kudu-handler: The patch generated 1 new + 0 unchanged
- 0 fixed = 1 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 25s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18989/dev-support/hive-personality.sh
|
| git revision | master / 9044fee |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18989/yetus/diff-checkstyle-kudu-handler.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18989/yetus/patch-asflicense-problems.txt
|
| modules | C: kudu-handler U: kudu-handler |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18989/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Shade dependent jars into the kudu-handler
> --
>
> Key: HIVE-22338
> URL: https://issues.apache.org/jira/browse/HIVE-22338
> Project: Hive
> Issue Type: Sub-task
>Reporter: Grant Henke
>Assignee: Grant Henke
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22338.0.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We should shade the Kudu client and it’s async dependency into the
> kudu-handler jar to simplify add jar style usage and ensure the
> addDependencyJars method adds all the required classes. This is a common
> pattern the other storage handlers use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22333) HiveServer2 Handler Pool blocked
[
https://issues.apache.org/jira/browse/HIVE-22333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kevin updated HIVE-22333:
-
Description:
Query thousands of fields and 100 partitions of a table, the thread that
hiveserver2 parses SQL will die, and HiveServer2-Handler-Pool will block.
Too many partitions and fields cause the hiveserver2 thread to die , but hive
cli runs normally。
was:Query thousands of fields and 100 partitions of a table, the thread that
hiveserver2 parses SQL will die, and HiveServer2-Handler-Pool will block.
> HiveServer2 Handler Pool blocked
> -
>
> Key: HIVE-22333
> URL: https://issues.apache.org/jira/browse/HIVE-22333
> Project: Hive
> Issue Type: Bug
> Components: Beeline, HiveServer2
>Affects Versions: 1.1.0
> Environment: cdh5.14.4-hive1.1.0
>Reporter: kevin
>Priority: Major
> Attachments: %XM@S]F7]U7B{(2U3SE~A)D.png
>
>
> Query thousands of fields and 100 partitions of a table, the thread that
> hiveserver2 parses SQL will die, and HiveServer2-Handler-Pool will block.
> Too many partitions and fields cause the hiveserver2 thread to die , but
> hive cli runs normally。
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22337) Improve and Expand Text-Based SerDes
[
https://issues.apache.org/jira/browse/HIVE-22337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951453#comment-16951453
]
Hive QA commented on HIVE-22337:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982976/HIVE-22337.1.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17571 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_part2] (batchId=22)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18988/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18988/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18988/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982976 - PreCommit-HIVE-Build
> Improve and Expand Text-Based SerDes
>
>
> Key: HIVE-22337
> URL: https://issues.apache.org/jira/browse/HIVE-22337
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 4.0.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22337.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> * Add new SerDe package just for text-based formats:
> org.apache.hadoop.hive.serde2.text.*
> * Add new SerDe package just for text-based log formats:
> org.apache.hadoop.hive.serde2.text.log.*
> * Create a coherent hierarchy for processing delimited data: AbstractSerDe ->
> TextSerDe -> EncodingAwareTextSerde -> DelimitedSerDe -> CsvTestSerDe
> * Create a coherent hierarchy for processing regex'ed data: AbstractSerDe ->
> TextSerDe -> EncodingAwareTextSerde -> RegexSerDe -> CommonFormatLogSerDe
> * Create some standard text processors for super-quick out-of-the-box
> processing: TSV SerDe and CSV SerDe
> * Create some standard log processors for super-quick out-of-the-box
> processing: Apache Common Log Format and Apache Combined Log Format (Apache
> HTTP Server Log Parsers)
> * Better default behaviors for processing text
> The default behavior should allow users to quick query data without any
> failures.
> # When a blank line is encountered, insert a 'null' value for each column
> # When there are fewer fields in the data than defined in the table schema,
> shift all available fields left, and fill in 'null' values for all remaining
> fields
> # When there are too many fields in the data, the last field in the results
> will contain all remaining values. Currently, the data is silently swallows
> and a warning is issued in the YARN logs. A normal user will never see this
> warning, especially if the job completes successfully. Better to (by
> default) provide them all the data than to hide anything.
> {code:none|title=CSV SerDe}
> "1,2,3"= ["1","2","3"]
> "1,2," = ["1","2",null]
> "" = [null,null,null]
> "1,2,3,4" = ["1","2","3,4"]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22340) Prevent shaded imports
[ https://issues.apache.org/jira/browse/HIVE-22340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22340: -- Attachment: HIVE-22340.01.patch > Prevent shaded imports > -- > > Key: HIVE-22340 > URL: https://issues.apache.org/jira/browse/HIVE-22340 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22340.01.patch > > > Make sure that hive developers don't import the shaded version of some class > by accident. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22340) Prevent shaded imports
[ https://issues.apache.org/jira/browse/HIVE-22340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22340: -- Status: Patch Available (was: Open) > Prevent shaded imports > -- > > Key: HIVE-22340 > URL: https://issues.apache.org/jira/browse/HIVE-22340 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22340.01.patch > > > Make sure that hive developers don't import the shaded version of some class > by accident. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22340) Prevent shaded imports
[ https://issues.apache.org/jira/browse/HIVE-22340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely reassigned HIVE-22340: - > Prevent shaded imports > -- > > Key: HIVE-22340 > URL: https://issues.apache.org/jira/browse/HIVE-22340 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Fix For: 4.0.0 > > > Make sure that hive developers don't import the shaded version of some class > by accident. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22276) Break up DDLSemanticAnalyzer - extract View related analyzers
[ https://issues.apache.org/jira/browse/HIVE-22276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22276: -- Attachment: HIVE-22276.03.patch > Break up DDLSemanticAnalyzer - extract View related analyzers > - > > Key: HIVE-22276 > URL: https://issues.apache.org/jira/browse/HIVE-22276 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Attachments: HIVE-22276.01.patch, HIVE-22276.02.patch, > HIVE-22276.03.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is > to refactor it in order to have everything cut into more handleable classes > under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each analyzers > * have a package for each operation, containing an analyzer, a description, > and an operation, so the amount of classes under a package is more manageable > Step #6: extract all the vire related analyzers from DDLSemanticAnalyzer, and > move them under the new package. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22276) Break up DDLSemanticAnalyzer - extract View related analyzers
[ https://issues.apache.org/jira/browse/HIVE-22276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22276: -- Attachment: (was: HIVE-22276.03.patch) > Break up DDLSemanticAnalyzer - extract View related analyzers > - > > Key: HIVE-22276 > URL: https://issues.apache.org/jira/browse/HIVE-22276 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Attachments: HIVE-22276.01.patch, HIVE-22276.02.patch, > HIVE-22276.03.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is > to refactor it in order to have everything cut into more handleable classes > under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each analyzers > * have a package for each operation, containing an analyzer, a description, > and an operation, so the amount of classes under a package is more manageable > Step #6: extract all the vire related analyzers from DDLSemanticAnalyzer, and > move them under the new package. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22337) Improve and Expand Text-Based SerDes
[
https://issues.apache.org/jira/browse/HIVE-22337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951433#comment-16951433
]
Hive QA commented on HIVE-22337:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
55s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
44s{color} | {color:blue} standalone-metastore/metastore-common in master has
37 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} serde in master has 199 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
22s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
43s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} serde: The patch generated 12 new + 0 unchanged - 0
fixed = 12 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
31s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m
17s{color} | {color:red} serde generated 15 new + 20 unchanged - 0 fixed = 35
total (was 20) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
16s{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18988/dev-support/hive-personality.sh
|
| git revision | master / 9044fee |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18988/yetus/diff-checkstyle-serde.txt
|
| javadoc |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18988/yetus/diff-javadoc-javadoc-serde.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18988/yetus/patch-asflicense-problems.txt
|
| modules | C: standalone-metastore/metastore-common serde U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18988/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improve and Expand Text-Based SerDes
>
>
> Key: HIVE-22337
> URL: https://issues.apache.org/jira/browse/HIVE-22337
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 4.0.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22337.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> * Add new SerDe package just for text-based formats:
> org.apache.hadoop.hive.serde2.text.*
> * Add new SerDe package just for text-based log formats:
> org.apache
[jira] [Commented] (HIVE-22284) Improve LLAP CacheContentsTracker to collect and display correct statistics
[
https://issues.apache.org/jira/browse/HIVE-22284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951425#comment-16951425
]
Hive QA commented on HIVE-22284:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982970/HIVE-22284.7.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18987/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18987/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18987/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Tests exited with: Exception: Patch URL
https://issues.apache.org/jira/secure/attachment/12982970/HIVE-22284.7.patch
was found in seen patch url's cache and a test was probably run already on it.
Aborting...
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982970 - PreCommit-HIVE-Build
> Improve LLAP CacheContentsTracker to collect and display correct statistics
> ---
>
> Key: HIVE-22284
> URL: https://issues.apache.org/jira/browse/HIVE-22284
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Attachments: HIVE-22284.0.patch, HIVE-22284.1.patch,
> HIVE-22284.2.patch, HIVE-22284.3.patch, HIVE-22284.4.patch,
> HIVE-22284.5.patch, HIVE-22284.6.patch, HIVE-22284.7.patch
>
>
> When keeping track of which buffers correspond to what Hive objects,
> CacheContentsTracker relies on cache tags.
> Currently a tag is a simple String that ideally holds DB and table name, and
> a partition spec concatenated by . and / . The information here is derived
> from the Path of the file that is getting cached. Needless to say sometimes
> this produces a wrong tag especially for external tables.
> Also there's a bug when calculating aggregated stats for a 'parent' tag
> (corresponding to the table of the partition) because the overall maxCount
> and maxSize do not add up to the sum of those in the partitions. This happens
> when buffers get removed from the cache.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22276) Break up DDLSemanticAnalyzer - extract View related analyzers
[
https://issues.apache.org/jira/browse/HIVE-22276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951422#comment-16951422
]
Hive QA commented on HIVE-22276:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982965/HIVE-22276.03.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17537 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[strict_pruning_2]
(batchId=102)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=274)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18986/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18986/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18986/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982965 - PreCommit-HIVE-Build
> Break up DDLSemanticAnalyzer - extract View related analyzers
> -
>
> Key: HIVE-22276
> URL: https://issues.apache.org/jira/browse/HIVE-22276
> Project: Hive
> Issue Type: Sub-task
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available, refactor-ddl
> Attachments: HIVE-22276.01.patch, HIVE-22276.02.patch,
> HIVE-22276.03.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is
> to refactor it in order to have everything cut into more handleable classes
> under the package org.apache.hadoop.hive.ql.exec.ddl:
> * have a separate class for each analyzers
> * have a package for each operation, containing an analyzer, a description,
> and an operation, so the amount of classes under a package is more manageable
> Step #6: extract all the vire related analyzers from DDLSemanticAnalyzer, and
> move them under the new package.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22315) Support Decimal64 column division with decimal64 scalar
[ https://issues.apache.org/jira/browse/HIVE-22315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22315: Attachment: HIVE-22315.3.patch Status: Patch Available (was: Open) > Support Decimal64 column division with decimal64 scalar > --- > > Key: HIVE-22315 > URL: https://issues.apache.org/jira/browse/HIVE-22315 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22315.1.patch, HIVE-22315.2.patch, > HIVE-22315.3.patch > > > Currently division operation is not supported for Decimal64 column. This Jira > will take care of supporting decimal64 column division with a decimal64 > scalar. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22315) Support Decimal64 column division with decimal64 scalar
[ https://issues.apache.org/jira/browse/HIVE-22315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ramesh Kumar Thangarajan updated HIVE-22315: Status: Open (was: Patch Available) > Support Decimal64 column division with decimal64 scalar > --- > > Key: HIVE-22315 > URL: https://issues.apache.org/jira/browse/HIVE-22315 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Attachments: HIVE-22315.1.patch, HIVE-22315.2.patch, > HIVE-22315.3.patch > > > Currently division operation is not supported for Decimal64 column. This Jira > will take care of supporting decimal64 column division with a decimal64 > scalar. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22323) Fix Desc Table bugs
[ https://issues.apache.org/jira/browse/HIVE-22323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951412#comment-16951412 ] Jesus Camacho Rodriguez commented on HIVE-22323: +1 > Fix Desc Table bugs > --- > > Key: HIVE-22323 > URL: https://issues.apache.org/jira/browse/HIVE-22323 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22323.01.patch, HIVE-22323.02.patch, > HIVE-22323.03.patch, HIVE-22323.04.patch, HIVE-22323.05.patch > > Time Spent: 1h > Remaining Estimate: 0h > > DESC TABLE operation is having the following bugs: > # Whole table descs have two headers. > # Table column desc has incorrect long header, while the table is transposed > having the headers in the first column. > # Json formatted data also has the headers. > # Json formatted data doesn't have the column statistics in it. > # There is no TestBeeLineDriver test for desc table, thus the actual output > is not tested, just some intermediary. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22175) TestBudyAllocator#testMTT test is flaky
[ https://issues.apache.org/jira/browse/HIVE-22175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sherman updated HIVE-22175: Attachment: HIVE-22175.2.patch > TestBudyAllocator#testMTT test is flaky > --- > > Key: HIVE-22175 > URL: https://issues.apache.org/jira/browse/HIVE-22175 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Ádám Szita >Assignee: John Sherman >Priority: Major > Attachments: HIVE-22175.1.patch, HIVE-22175.2.patch > > > This test has a fail rate of about 20%-25% -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22175) TestBudyAllocator#testMTT test is flaky
[ https://issues.apache.org/jira/browse/HIVE-22175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951404#comment-16951404 ] John Sherman commented on HIVE-22175: - [~bslim] - Done. I also re-enabled the test in the patch. > TestBudyAllocator#testMTT test is flaky > --- > > Key: HIVE-22175 > URL: https://issues.apache.org/jira/browse/HIVE-22175 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Ádám Szita >Assignee: John Sherman >Priority: Major > Attachments: HIVE-22175.1.patch, HIVE-22175.2.patch > > > This test has a fail rate of about 20%-25% -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22276) Break up DDLSemanticAnalyzer - extract View related analyzers
[
https://issues.apache.org/jira/browse/HIVE-22276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951397#comment-16951397
]
Hive QA commented on HIVE-22276:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
16s{color} | {color:blue} ql in master has 1549 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} ql: The patch generated 0 new + 727 unchanged - 13
fixed = 727 total (was 740) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18986/dev-support/hive-personality.sh
|
| git revision | master / 9044fee |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18986/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18986/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Break up DDLSemanticAnalyzer - extract View related analyzers
> -
>
> Key: HIVE-22276
> URL: https://issues.apache.org/jira/browse/HIVE-22276
> Project: Hive
> Issue Type: Sub-task
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available, refactor-ddl
> Attachments: HIVE-22276.01.patch, HIVE-22276.02.patch,
> HIVE-22276.03.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is
> to refactor it in order to have everything cut into more handleable classes
> under the package org.apache.hadoop.hive.ql.exec.ddl:
> * have a separate class for each analyzers
> * have a package for each operation, containing an analyzer, a description,
> and an operation, so the amount of classes under a package is more manageable
> Step #6: extract all the vire related analyzers from DDLSemanticAnalyzer, and
> move them under the new package.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22240) Function percentile_cont fails when array parameter passed
[
https://issues.apache.org/jira/browse/HIVE-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951384#comment-16951384
]
Hive QA commented on HIVE-22240:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982963/HIVE-22240.3.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17537 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18985/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18985/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18985/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982963 - PreCommit-HIVE-Build
> Function percentile_cont fails when array parameter passed
> --
>
> Key: HIVE-22240
> URL: https://issues.apache.org/jira/browse/HIVE-22240
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22240.1.patch, HIVE-22240.2.patch,
> HIVE-22240.3.patch, HIVE-22240.3.patch
>
>
> {code}
> SELECT
> percentile_cont(array(0.2, 0.5, 0.9)) WITHIN GROUP (ORDER BY value)
> FROM t_test;
> {code}
> hive.log:
> {code}
> 2019-09-24T21:00:43,203 ERROR [LocalJobRunner Map Task Executor #0]
> mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:573)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:148)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.util.ArrayList cannot be cast to
> org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:793)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:128)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:152)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
> ... 11 more
> Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast
> to org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileCont$PercentileContEvaluator.iterate(GenericUDAFPercentileCont.java:259)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:214)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:639)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:814)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:720)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:788)
> ... 17 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22175) TestBudyAllocator#testMTT test is flaky
[ https://issues.apache.org/jira/browse/HIVE-22175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951377#comment-16951377 ] Slim Bouguerra commented on HIVE-22175: --- good catch looks like we need to update the count before [this block|https://github.com/apache/hive/blob/c507156811a11c075544a0338b9c0c825367f482/llap-server/src/java/org/apache/hadoop/hive/llap/cache/BuddyAllocator.java#L355] as well. Can you please add that to the patch ? > TestBudyAllocator#testMTT test is flaky > --- > > Key: HIVE-22175 > URL: https://issues.apache.org/jira/browse/HIVE-22175 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Ádám Szita >Assignee: John Sherman >Priority: Major > Attachments: HIVE-22175.1.patch > > > This test has a fail rate of about 20%-25% -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22323) Fix Desc Table bugs
[ https://issues.apache.org/jira/browse/HIVE-22323?focusedWorklogId=328164&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328164 ] ASF GitHub Bot logged work on HIVE-22323: - Author: ASF GitHub Bot Created on: 14/Oct/19 21:26 Start Date: 14/Oct/19 21:26 Worklog Time Spent: 10m Work Description: miklosgergely commented on pull request #813: HIVE-22323 Fix Desc Table bugs URL: https://github.com/apache/hive/pull/813#discussion_r334367697 ## File path: ql/src/test/results/clientpositive/alterColumnStats.q.out ## @@ -144,15 +144,35 @@ PREHOOK: Input: default@p_n0 POSTHOOK: query: desc formatted p_n0 c1 POSTHOOK: type: DESCTABLE POSTHOOK: Input: default@p_n0 -# col_name data_type comment Review comment: Yes, it was intentional. I think it is much better to have a consistent output format for `desc formatted ` like commands, showing the same table all the time, explicitly showing that we don't have statistics for the column, instead of showing the smaller table as if a `desc ` command would have been executed. The previous form is only implicitly showing that there are no statistics, which can be misleading. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 328164) Time Spent: 1h (was: 50m) > Fix Desc Table bugs > --- > > Key: HIVE-22323 > URL: https://issues.apache.org/jira/browse/HIVE-22323 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22323.01.patch, HIVE-22323.02.patch, > HIVE-22323.03.patch, HIVE-22323.04.patch, HIVE-22323.05.patch > > Time Spent: 1h > Remaining Estimate: 0h > > DESC TABLE operation is having the following bugs: > # Whole table descs have two headers. > # Table column desc has incorrect long header, while the table is transposed > having the headers in the first column. > # Json formatted data also has the headers. > # Json formatted data doesn't have the column statistics in it. > # There is no TestBeeLineDriver test for desc table, thus the actual output > is not tested, just some intermediary. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22323) Fix Desc Table bugs
[ https://issues.apache.org/jira/browse/HIVE-22323?focusedWorklogId=328163&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328163 ] ASF GitHub Bot logged work on HIVE-22323: - Author: ASF GitHub Bot Created on: 14/Oct/19 21:26 Start Date: 14/Oct/19 21:26 Worklog Time Spent: 10m Work Description: miklosgergely commented on pull request #813: HIVE-22323 Fix Desc Table bugs URL: https://github.com/apache/hive/pull/813#discussion_r334401590 ## File path: ql/src/test/results/clientpositive/autoColumnStats_5.q.out ## @@ -202,19 +202,19 @@ PREHOOK: Input: default@partitioned1_n1 POSTHOOK: query: desc formatted partitioned1_n1 partition(part=1) a POSTHOOK: type: DESCTABLE POSTHOOK: Input: default@partitioned1_n1 -col_name data_type min max num_nulls distinct_count avg_col_len max_col_len num_trues num_falses bitvector comment Review comment: If there are no statistics available then there is the regular header added by TextMetaDataFormatter during the table creation, and still the fetch task adds the header with the statistics to the table. If there are statistics then TextMetaDataFormatter doesn't add any header, it transposes the table, and then the fetch task adds the long header - which is still bad. Both of these are fixed in the patch. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 328163) Time Spent: 1h (was: 50m) > Fix Desc Table bugs > --- > > Key: HIVE-22323 > URL: https://issues.apache.org/jira/browse/HIVE-22323 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22323.01.patch, HIVE-22323.02.patch, > HIVE-22323.03.patch, HIVE-22323.04.patch, HIVE-22323.05.patch > > Time Spent: 1h > Remaining Estimate: 0h > > DESC TABLE operation is having the following bugs: > # Whole table descs have two headers. > # Table column desc has incorrect long header, while the table is transposed > having the headers in the first column. > # Json formatted data also has the headers. > # Json formatted data doesn't have the column statistics in it. > # There is no TestBeeLineDriver test for desc table, thus the actual output > is not tested, just some intermediary. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22240) Function percentile_cont fails when array parameter passed
[
https://issues.apache.org/jira/browse/HIVE-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951370#comment-16951370
]
Hive QA commented on HIVE-22240:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
23s{color} | {color:blue} ql in master has 1549 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 6s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18985/dev-support/hive-personality.sh
|
| git revision | master / 9044fee |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18985/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18985/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Function percentile_cont fails when array parameter passed
> --
>
> Key: HIVE-22240
> URL: https://issues.apache.org/jira/browse/HIVE-22240
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22240.1.patch, HIVE-22240.2.patch,
> HIVE-22240.3.patch, HIVE-22240.3.patch
>
>
> {code}
> SELECT
> percentile_cont(array(0.2, 0.5, 0.9)) WITHIN GROUP (ORDER BY value)
> FROM t_test;
> {code}
> hive.log:
> {code}
> 2019-09-24T21:00:43,203 ERROR [LocalJobRunner Map Task Executor #0]
> mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:573)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:148)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.
[jira] [Updated] (HIVE-22339) Change default time for MVs refresh in registry
[ https://issues.apache.org/jira/browse/HIVE-22339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22339: --- Attachment: HIVE-22339.patch > Change default time for MVs refresh in registry > --- > > Key: HIVE-22339 > URL: https://issues.apache.org/jira/browse/HIVE-22339 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Minor > Attachments: HIVE-22339.patch > > > Default was set to 60secs in HIVE-21344. It seems it may be too aggressive; > suggestion is to change default to 1500secs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22339) Change default time for MVs refresh in registry
[ https://issues.apache.org/jira/browse/HIVE-22339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reassigned HIVE-22339: -- > Change default time for MVs refresh in registry > --- > > Key: HIVE-22339 > URL: https://issues.apache.org/jira/browse/HIVE-22339 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Minor > > Default was set to 60secs in HIVE-21344. It seems it may be too aggressive; > suggestion is to change default to 1500secs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-22339) Change default time for MVs refresh in registry
[ https://issues.apache.org/jira/browse/HIVE-22339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-22339 started by Jesus Camacho Rodriguez. -- > Change default time for MVs refresh in registry > --- > > Key: HIVE-22339 > URL: https://issues.apache.org/jira/browse/HIVE-22339 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Minor > > Default was set to 60secs in HIVE-21344. It seems it may be too aggressive; > suggestion is to change default to 1500secs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22339) Change default time for MVs refresh in registry
[ https://issues.apache.org/jira/browse/HIVE-22339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22339: --- Status: Patch Available (was: In Progress) > Change default time for MVs refresh in registry > --- > > Key: HIVE-22339 > URL: https://issues.apache.org/jira/browse/HIVE-22339 > Project: Hive > Issue Type: Improvement > Components: Materialized views >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Minor > > Default was set to 60secs in HIVE-21344. It seems it may be too aggressive; > suggestion is to change default to 1500secs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22298) Allow Llap IO cache for reading tables without delete delta
[
https://issues.apache.org/jira/browse/HIVE-22298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951341#comment-16951341
]
Hive QA commented on HIVE-22298:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982957/HIVE-22298.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17537 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original]
(batchId=184)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18984/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18984/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18984/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982957 - PreCommit-HIVE-Build
> Allow Llap IO cache for reading tables without delete delta
> ---
>
> Key: HIVE-22298
> URL: https://issues.apache.org/jira/browse/HIVE-22298
> Project: Hive
> Issue Type: Improvement
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-22298.2.patch, HIVE-22298.3.patch, HIVE-22298.patch
>
>
> Since HIVE-19985 we can use Llap IO cache for queries which are not
> explicitly reading ROW_ID even for original files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22239) Scale data size using column value ranges
[ https://issues.apache.org/jira/browse/HIVE-22239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951338#comment-16951338 ] Jesus Camacho Rodriguez commented on HIVE-22239: [~kgyrtkirk], [~mgergely], please let me know if there is anything else that should be addressed within the scope of this patch. Thanks > Scale data size using column value ranges > - > > Key: HIVE-22239 > URL: https://issues.apache.org/jira/browse/HIVE-22239 > Project: Hive > Issue Type: Improvement > Components: Physical Optimizer >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22239.01.patch, HIVE-22239.02.patch, > HIVE-22239.03.patch, HIVE-22239.04.patch, HIVE-22239.04.patch, > HIVE-22239.05.patch, HIVE-22239.05.patch, HIVE-22239.06.patch, > HIVE-22239.patch > > Time Spent: 5h 10m > Remaining Estimate: 0h > > Currently, min/max values for columns are only used to determine whether a > certain range filter falls out of range and thus filters all rows or none at > all. If it does not, we just use a heuristic that the condition will filter > 1/3 of the input rows. Instead of using that heuristic, we can use another > one that assumes that data will be uniformly distributed across that range, > and calculate the selectivity for the condition accordingly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22274) Upgrade Calcite version to 1.21.0
[ https://issues.apache.org/jira/browse/HIVE-22274?focusedWorklogId=328124&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328124 ] ASF GitHub Bot logged work on HIVE-22274: - Author: ASF GitHub Bot Created on: 14/Oct/19 20:35 Start Date: 14/Oct/19 20:35 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #809: HIVE-22274: upgrade Calcite to 1.21 URL: https://github.com/apache/hive/pull/809#discussion_r334651939 ## File path: ql/src/test/results/clientpositive/llap/limit_pushdown.q.out ## @@ -816,12 +816,54 @@ POSTHOOK: type: QUERY POSTHOOK: Input: default@src A masked pattern was here STAGE DEPENDENCIES: - Stage-0 is a root stage + Stage-1 is a root stage + Stage-0 depends on stages: Stage-1 STAGE PLANS: + Stage: Stage-1 +Tez + A masked pattern was here + Edges: +Reducer 2 <- Map 1 (SIMPLE_EDGE) + A masked pattern was here + Vertices: +Map 1 +Map Operator Tree: +TableScan + alias: src + Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE + Select Operator +expressions: key (type: string), value (type: string) +outputColumnNames: _col0, _col1 +Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE +Reduce Output Operator + key expressions: _col0 (type: string) + sort order: + + Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE + value expressions: _col1 (type: string) +Execution mode: vectorized, llap +LLAP IO: no inputs +Reducer 2 +Execution mode: vectorized, llap +Reduce Operator Tree: + Select Operator +expressions: KEY.reducesinkkey0 (type: string), VALUE._col0 (type: string) +outputColumnNames: _col0, _col1 +Statistics: Num rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats: COMPLETE +Filter Operator + predicate: false (type: boolean) + Statistics: Num rows: 1 Data size: 178 Basic stats: COMPLETE Column stats: COMPLETE + File Output Operator +compressed: false +Statistics: Num rows: 1 Data size: 178 Basic stats: COMPLETE Column stats: COMPLETE +table: +input format: org.apache.hadoop.mapred.SequenceFileInputFormat +output format: org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat +serde: org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe + Stage: Stage-0 Fetch Operator - limit: 0 + limit: -1 Review comment: It seems the initial assessment that filter with false condition was going to be less disruptive does not hold... Which ones were the regressions with limit 0? It seems it may be worth to move to original logic again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 328124) Time Spent: 50m (was: 40m) > Upgrade Calcite version to 1.21.0 > - > > Key: HIVE-22274 > URL: https://issues.apache.org/jira/browse/HIVE-22274 > Project: Hive > Issue Type: Task >Affects Versions: 3.1.2 >Reporter: Steve Carlin >Assignee: Steve Carlin >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22274.1.patch, HIVE-22274.2.patch, > HIVE-22274.3.patch, HIVE-22274.4.patch, HIVE-22274.5.patch, > HIVE-22274.6.patch, HIVE-22274.7.patch, HIVE-22274.patch > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22274) Upgrade Calcite version to 1.21.0
[
https://issues.apache.org/jira/browse/HIVE-22274?focusedWorklogId=328123&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328123
]
ASF GitHub Bot logged work on HIVE-22274:
-
Author: ASF GitHub Bot
Created on: 14/Oct/19 20:33
Start Date: 14/Oct/19 20:33
Worklog Time Spent: 10m
Work Description: jcamachor commented on pull request #809: HIVE-22274:
upgrade Calcite to 1.21
URL: https://github.com/apache/hive/pull/809#discussion_r334651216
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/HiveJdbcImplementor.java
##
@@ -109,7 +109,7 @@ public HiveJdbcImplementor(SqlDialect dialect,
JavaTypeFactory typeFactory) {
SqlNode sqlCondition = null;
SqlLiteral condType = JoinConditionType.ON.symbol(POS);
JoinType joinType = joinType(e.getJoinType());
-if (e.getJoinType() == JoinRelType.INNER &&
e.getCondition().isAlwaysTrue()) {
+if ((e.getJoinType() == JoinRelType.INNER || e.isSemiJoin()) &&
e.getCondition().isAlwaysTrue()) {
Review comment:
`ql/src/test/queries/clientpositive/external_jdbc_table*` files contain
tests for jdbc storage handler. We should add a test over there.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 328123)
Time Spent: 40m (was: 0.5h)
> Upgrade Calcite version to 1.21.0
> -
>
> Key: HIVE-22274
> URL: https://issues.apache.org/jira/browse/HIVE-22274
> Project: Hive
> Issue Type: Task
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Assignee: Steve Carlin
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22274.1.patch, HIVE-22274.2.patch,
> HIVE-22274.3.patch, HIVE-22274.4.patch, HIVE-22274.5.patch,
> HIVE-22274.6.patch, HIVE-22274.7.patch, HIVE-22274.patch
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22274) Upgrade Calcite version to 1.21.0
[
https://issues.apache.org/jira/browse/HIVE-22274?focusedWorklogId=328113&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328113
]
ASF GitHub Bot logged work on HIVE-22274:
-
Author: ASF GitHub Bot
Created on: 14/Oct/19 20:23
Start Date: 14/Oct/19 20:23
Worklog Time Spent: 10m
Work Description: vineetgarg02 commented on pull request #809:
HIVE-22274: upgrade Calcite to 1.21
URL: https://github.com/apache/hive/pull/809#discussion_r334609909
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/jdbc/HiveJdbcImplementor.java
##
@@ -109,7 +109,7 @@ public HiveJdbcImplementor(SqlDialect dialect,
JavaTypeFactory typeFactory) {
SqlNode sqlCondition = null;
SqlLiteral condType = JoinConditionType.ON.symbol(POS);
JoinType joinType = joinType(e.getJoinType());
-if (e.getJoinType() == JoinRelType.INNER &&
e.getCondition().isAlwaysTrue()) {
+if ((e.getJoinType() == JoinRelType.INNER || e.isSemiJoin()) &&
e.getCondition().isAlwaysTrue()) {
Review comment:
Existing logic looks wrong. Even for SEMI JOIN `JoinType.COMMA` with
condition `NONE` is created (in case condition is always true).
@jcamachor Are there JDBC tests where I can test this logic for SEMI JOIN
type?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 328113)
Time Spent: 0.5h (was: 20m)
> Upgrade Calcite version to 1.21.0
> -
>
> Key: HIVE-22274
> URL: https://issues.apache.org/jira/browse/HIVE-22274
> Project: Hive
> Issue Type: Task
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Assignee: Steve Carlin
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22274.1.patch, HIVE-22274.2.patch,
> HIVE-22274.3.patch, HIVE-22274.4.patch, HIVE-22274.5.patch,
> HIVE-22274.6.patch, HIVE-22274.7.patch, HIVE-22274.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22274) Upgrade Calcite version to 1.21.0
[
https://issues.apache.org/jira/browse/HIVE-22274?focusedWorklogId=328111&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328111
]
ASF GitHub Bot logged work on HIVE-22274:
-
Author: ASF GitHub Bot
Created on: 14/Oct/19 20:23
Start Date: 14/Oct/19 20:23
Worklog Time Spent: 10m
Work Description: vineetgarg02 commented on pull request #809:
HIVE-22274: upgrade Calcite to 1.21
URL: https://github.com/apache/hive/pull/809#discussion_r334647026
##
File path: ql/src/test/results/clientpositive/concat_op.q.out
##
@@ -311,7 +311,7 @@ POSTHOOK: query: explain formatted select key || value
from src
POSTHOOK: type: QUERY
POSTHOOK: Input: default@src
A masked pattern was here
-{"CBOPlan":"{\n \"rels\": [\n{\n \"id\": \"0\",\n \"relOp\":
\"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan\",\n
\"table\": [\n\"default\",\n\"src\"\n ],\n
\"table:alias\": \"src\",\n \"inputs\": [],\n \"rowCount\": 500.0,\n
\"avgRowSize\": 9.624,\n \"rowType\": [\n{\n
\"type\": \"VARCHAR\",\n \"nullable\": true,\n \"precision\":
2147483647,\n \"name\": \"key\"\n},\n{\n
\"type\": \"VARCHAR\",\n \"nullable\": true,\n \"precision\":
2147483647,\n \"name\": \"value\"\n},\n{\n
\"type\": \"BIGINT\",\n \"nullable\": true,\n \"name\":
\"BLOCK__OFFSET__INSIDE__FILE\"\n},\n{\n \"type\":
\"VARCHAR\",\n \"nullable\": true,\n \"precision\":
2147483647,\n \"name\": \"INPUT__FILE__NAME\"\n},\n{\n
\"fields\": [\n{\n \"type\": \"BIGINT\",\n
\"nullable\": true,\n \"name\": \"writeid\"\n
},\n{\n \"type\": \"INTEGER\",\n
\"nullable\": true,\n \"name\": \"bucketid\"\n},\n
{\n \"type\": \"BIGINT\",\n \"nullable\":
true,\n \"name\": \"rowid\"\n}\n ],\n
\"name\": \"ROW__ID\"\n}\n ],\n \"colStats\": [\n{\n
\"name\": \"key\",\n \"ndv\": 316\n},\n{\n
\"name\": \"value\",\n \"ndv\": 307\n}\n ]\n
},\n{\n \"id\": \"1\",\n \"relOp\":
\"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveProject\",\n
\"fields\": [\n\"_o__c0\"\n ],\n \"exprs\": [\n{\n
\"op\": \"||\",\n \"operands\": [\n{\n
\"input\": 0,\n \"name\": \"$0\"\n},\n{\n
\"input\": 1,\n \"name\": \"$1\"\n}\n
]\n}\n ],\n \"rowCount\": 500.0\n}\n
]\n}","optimizedSQL":"SELECT `key` || `value` AS `_o__c0`\nFROM
`default`.`src`","cboInfo":"Plan optimized by CBO.","STAGE
DEPENDENCIES":{"Stage-1":{"ROOT STAGE":"TRUE"},"Stage-0":{"DEPENDENT
STAGES":"Stage-1"}},"STAGE PLANS":{"Stage-1":{"Map Reduce":{"Map Operator
Tree:":[{"TableScan":{"alias:":"src","columns:":["key","value"],"database:":"default","Statistics:":"Num
rows: 500 Data size: 89000 Basic stats: COMPLETE Column stats:
COMPLETE","table:":"src","isTempTable:":"false","OperatorId:":"TS_0","children":{"Select
Operator":{"expressions:":"concat(key, value) (type:
string)","columnExprMap:":{"_col0":"concat(key,
value)"},"outputColumnNames:":["_col0"],"Statistics:":"Num rows: 500 Data size:
92000 Basic stats: COMPLETE Column stats:
COMPLETE","OperatorId:":"SEL_3","children":{"File Output
Operator":{"compressed:":"false","Statistics:":"Num rows: 500 Data size: 92000
Basic stats: COMPLETE Column stats: COMPLETE","table:":{"input
format:":"org.apache.hadoop.mapred.SequenceFileInputFormat","output
format:":"org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat","serde:":"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"},"OperatorId:":"FS_4"}}],"Execution
mode:":"vectorized"}},"Stage-0":{"Fetch Operator":{"limit:":"-1","Processor
Tree:":{"ListSink":{"OperatorId:":"LIST_SINK_5"}}
+{"CBOPlan":"{\n \"rels\": [\n{\n \"id\": \"0\",\n \"relOp\":
\"org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan\",\n
\"table\": [\n\"default\",\n\"src\"\n ],\n
\"table:alias\": \"src\",\n \"inputs\": [],\n \"rowCount\": 500.0,\n
\"avgRowSize\": 9.624,\n \"rowType\": [\n{\n
\"type\": \"VARCHAR\",\n \"nullable\": true,\n \"precision\":
2147483647,\n \"name\": \"key\"\n},\n{\n
\"type\": \"VARCHAR\",\n \"nullabl
[jira] [Work logged] (HIVE-22274) Upgrade Calcite version to 1.21.0
[
https://issues.apache.org/jira/browse/HIVE-22274?focusedWorklogId=328112&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328112
]
ASF GitHub Bot logged work on HIVE-22274:
-
Author: ASF GitHub Bot
Created on: 14/Oct/19 20:23
Start Date: 14/Oct/19 20:23
Worklog Time Spent: 10m
Work Description: vineetgarg02 commented on pull request #809:
HIVE-22274: upgrade Calcite to 1.21
URL: https://github.com/apache/hive/pull/809#discussion_r334602095
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/reloperators/HiveMultiJoin.java
##
@@ -225,7 +225,7 @@ public JoinPredicateInfo getJoinPredicateInfo() {
private boolean containsOuter() {
for (JoinRelType joinType : joinTypes) {
- if (joinType != JoinRelType.INNER) {
+ if (joinType != JoinRelType.INNER && joinType != JoinRelType.SEMI) {
Review comment:
It would be better to whitelist this for outer types (FULL, LEFT, RIGHT).
@scarlin-cloudera Either we can create separate jira or we can make the
change with this patch..upto you.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 328112)
Time Spent: 0.5h (was: 20m)
> Upgrade Calcite version to 1.21.0
> -
>
> Key: HIVE-22274
> URL: https://issues.apache.org/jira/browse/HIVE-22274
> Project: Hive
> Issue Type: Task
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Assignee: Steve Carlin
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22274.1.patch, HIVE-22274.2.patch,
> HIVE-22274.3.patch, HIVE-22274.4.patch, HIVE-22274.5.patch,
> HIVE-22274.6.patch, HIVE-22274.7.patch, HIVE-22274.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22274) Upgrade Calcite version to 1.21.0
[
https://issues.apache.org/jira/browse/HIVE-22274?focusedWorklogId=328110&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328110
]
ASF GitHub Bot logged work on HIVE-22274:
-
Author: ASF GitHub Bot
Created on: 14/Oct/19 20:23
Start Date: 14/Oct/19 20:23
Worklog Time Spent: 10m
Work Description: vineetgarg02 commented on pull request #809:
HIVE-22274: upgrade Calcite to 1.21
URL: https://github.com/apache/hive/pull/809#discussion_r333775973
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HiveRelBuilder.java
##
@@ -113,9 +113,8 @@ public RelBuilder filter(Iterable
predicates) {
@Override
public RelBuilder empty() {
final RelNode input = build();
-final RelNode sort = HiveRelFactories.HIVE_SORT_FACTORY.createSort(
-input, RelCollations.of(), null, literal(0));
-return this.push(sort);
+final RelNode filter =
HiveRelFactories.HIVE_FILTER_FACTORY.createFilter(input, literal(false));
Review comment:
Method's comment is outdated (referring to Limit 0)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 328110)
Time Spent: 20m (was: 10m)
> Upgrade Calcite version to 1.21.0
> -
>
> Key: HIVE-22274
> URL: https://issues.apache.org/jira/browse/HIVE-22274
> Project: Hive
> Issue Type: Task
>Affects Versions: 3.1.2
>Reporter: Steve Carlin
>Assignee: Steve Carlin
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22274.1.patch, HIVE-22274.2.patch,
> HIVE-22274.3.patch, HIVE-22274.4.patch, HIVE-22274.5.patch,
> HIVE-22274.6.patch, HIVE-22274.7.patch, HIVE-22274.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22298) Allow Llap IO cache for reading tables without delete delta
[
https://issues.apache.org/jira/browse/HIVE-22298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951288#comment-16951288
]
Hive QA commented on HIVE-22298:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
40s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
24s{color} | {color:blue} ql in master has 1549 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
46s{color} | {color:blue} llap-server in master has 90 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} ql: The patch generated 0 new + 54 unchanged - 3
fixed = 54 total (was 57) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} The patch llap-server passed checkstyle {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
16s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 30m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18984/dev-support/hive-personality.sh
|
| git revision | master / 9044fee |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18984/yetus/whitespace-eol.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18984/yetus/patch-asflicense-problems.txt
|
| modules | C: ql llap-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18984/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Allow Llap IO cache for reading tables without delete delta
> ---
>
> Key: HIVE-22298
> URL: https://issues.apache.org/jira/browse/HIVE-22298
> Project: Hive
> Issue Type: Improvement
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Attachments: HIVE-22298.2.patch, HIVE-22298.3.patch, HIVE-22298.patch
>
>
> Since HIVE-19985 we can use Llap IO cache for queries which are not
> explicitly reading ROW_ID even for original files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22284) Improve LLAP CacheContentsTracker to collect and display correct statistics
[
https://issues.apache.org/jira/browse/HIVE-22284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951258#comment-16951258
]
Hive QA commented on HIVE-22284:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982970/HIVE-22284.7.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17539 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18983/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18983/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18983/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982970 - PreCommit-HIVE-Build
> Improve LLAP CacheContentsTracker to collect and display correct statistics
> ---
>
> Key: HIVE-22284
> URL: https://issues.apache.org/jira/browse/HIVE-22284
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Attachments: HIVE-22284.0.patch, HIVE-22284.1.patch,
> HIVE-22284.2.patch, HIVE-22284.3.patch, HIVE-22284.4.patch,
> HIVE-22284.5.patch, HIVE-22284.6.patch, HIVE-22284.7.patch
>
>
> When keeping track of which buffers correspond to what Hive objects,
> CacheContentsTracker relies on cache tags.
> Currently a tag is a simple String that ideally holds DB and table name, and
> a partition spec concatenated by . and / . The information here is derived
> from the Path of the file that is getting cached. Needless to say sometimes
> this produces a wrong tag especially for external tables.
> Also there's a bug when calculating aggregated stats for a 'parent' tag
> (corresponding to the table of the partition) because the overall maxCount
> and maxSize do not add up to the sum of those in the partitions. This happens
> when buffers get removed from the cache.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22323) Fix Desc Table bugs
[ https://issues.apache.org/jira/browse/HIVE-22323?focusedWorklogId=328044&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328044 ] ASF GitHub Bot logged work on HIVE-22323: - Author: ASF GitHub Bot Created on: 14/Oct/19 18:33 Start Date: 14/Oct/19 18:33 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #813: HIVE-22323 Fix Desc Table bugs URL: https://github.com/apache/hive/pull/813#discussion_r334611107 ## File path: ql/src/test/results/clientpositive/beeline/escape_comments.q.out ## @@ -78,7 +78,6 @@ PREHOOK: Input: escape_comments_db@escape_comments_tbl1 POSTHOOK: query: describe formatted escape_comments_tbl1 POSTHOOK: type: DESCTABLE POSTHOOK: Input: escape_comments_db@escape_comments_tbl1 -# col_name data_type comment Review comment: Yeah, I wonder what may be different between the test env of the driver and the real one... Anyway, good to know, thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 328044) Time Spent: 50m (was: 40m) > Fix Desc Table bugs > --- > > Key: HIVE-22323 > URL: https://issues.apache.org/jira/browse/HIVE-22323 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22323.01.patch, HIVE-22323.02.patch, > HIVE-22323.03.patch, HIVE-22323.04.patch, HIVE-22323.05.patch > > Time Spent: 50m > Remaining Estimate: 0h > > DESC TABLE operation is having the following bugs: > # Whole table descs have two headers. > # Table column desc has incorrect long header, while the table is transposed > having the headers in the first column. > # Json formatted data also has the headers. > # Json formatted data doesn't have the column statistics in it. > # There is no TestBeeLineDriver test for desc table, thus the actual output > is not tested, just some intermediary. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22284) Improve LLAP CacheContentsTracker to collect and display correct statistics
[
https://issues.apache.org/jira/browse/HIVE-22284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951235#comment-16951235
]
Hive QA commented on HIVE-22284:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
4s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
41s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
21s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
26s{color} | {color:blue} storage-api in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} llap-common in master has 90 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
8s{color} | {color:blue} ql in master has 1549 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
45s{color} | {color:blue} llap-server in master has 90 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
44s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} storage-api: The patch generated 2 new + 4 unchanged -
0 fixed = 6 total (was 4) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 2 new + 165 unchanged - 2
fixed = 167 total (was 167) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
17s{color} | {color:red} llap-server: The patch generated 1 new + 252 unchanged
- 13 fixed = 253 total (was 265) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 36m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18983/dev-support/hive-personality.sh
|
| git revision | master / 9044fee |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18983/yetus/diff-checkstyle-storage-api.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18983/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18983/yetus/diff-checkstyle-llap-server.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18983/yetus/patch-asflicense-problems.txt
|
| modules | C: storage-api llap-common ql llap-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18983/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improve LL
[jira] [Commented] (HIVE-22276) Break up DDLSemanticAnalyzer - extract View related analyzers
[ https://issues.apache.org/jira/browse/HIVE-22276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951213#comment-16951213 ] Jesus Camacho Rodriguez commented on HIVE-22276: Left a minor comment in latest PR. Once that is fix, +1 (pending tests) > Break up DDLSemanticAnalyzer - extract View related analyzers > - > > Key: HIVE-22276 > URL: https://issues.apache.org/jira/browse/HIVE-22276 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Attachments: HIVE-22276.01.patch, HIVE-22276.02.patch, > HIVE-22276.03.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is > to refactor it in order to have everything cut into more handleable classes > under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each analyzers > * have a package for each operation, containing an analyzer, a description, > and an operation, so the amount of classes under a package is more manageable > Step #6: extract all the vire related analyzers from DDLSemanticAnalyzer, and > move them under the new package. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22276) Break up DDLSemanticAnalyzer - extract View related analyzers
[ https://issues.apache.org/jira/browse/HIVE-22276?focusedWorklogId=328032&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328032 ] ASF GitHub Bot logged work on HIVE-22276: - Author: ASF GitHub Bot Created on: 14/Oct/19 18:06 Start Date: 14/Oct/19 18:06 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #800: HIVE-22276 Break up DDLSemanticAnalyzer - extract View related analyzers URL: https://github.com/apache/hive/pull/800#discussion_r334600636 ## File path: ql/src/java/org/apache/hadoop/hive/ql/ddl/DDLSemanticAnalyzerFactory.java ## @@ -29,10 +29,13 @@ import org.apache.hadoop.hive.ql.metadata.Hive; import org.apache.hadoop.hive.ql.parse.ASTNode; import org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer; +import org.apache.hadoop.hive.ql.parse.CalcitePlanner; import org.reflections.Reflections; import com.google.common.annotations.VisibleForTesting; +import avro.shaded.com.google.common.collect.Sets; Review comment: We should not use the shaded version. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 328032) Time Spent: 1h 10m (was: 1h) > Break up DDLSemanticAnalyzer - extract View related analyzers > - > > Key: HIVE-22276 > URL: https://issues.apache.org/jira/browse/HIVE-22276 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Attachments: HIVE-22276.01.patch, HIVE-22276.02.patch, > HIVE-22276.03.patch > > Time Spent: 1h 10m > Remaining Estimate: 0h > > DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is > to refactor it in order to have everything cut into more handleable classes > under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each analyzers > * have a package for each operation, containing an analyzer, a description, > and an operation, so the amount of classes under a package is more manageable > Step #6: extract all the vire related analyzers from DDLSemanticAnalyzer, and > move them under the new package. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22338) Shade dependent jars into the kudu-handler
[ https://issues.apache.org/jira/browse/HIVE-22338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951206#comment-16951206 ] Jesus Camacho Rodriguez commented on HIVE-22338: +1 (pending tests) > Shade dependent jars into the kudu-handler > -- > > Key: HIVE-22338 > URL: https://issues.apache.org/jira/browse/HIVE-22338 > Project: Hive > Issue Type: Sub-task >Reporter: Grant Henke >Assignee: Grant Henke >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22338.0.patch > > Time Spent: 10m > Remaining Estimate: 0h > > We should shade the Kudu client and it’s async dependency into the > kudu-handler jar to simplify add jar style usage and ensure the > addDependencyJars method adds all the required classes. This is a common > pattern the other storage handlers use. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22338) Shade dependent jars into the kudu-handler
[ https://issues.apache.org/jira/browse/HIVE-22338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated HIVE-22338: --- Status: Patch Available (was: Open) > Shade dependent jars into the kudu-handler > -- > > Key: HIVE-22338 > URL: https://issues.apache.org/jira/browse/HIVE-22338 > Project: Hive > Issue Type: Sub-task >Reporter: Grant Henke >Assignee: Grant Henke >Priority: Major > Attachments: HIVE-22338.0.patch > > > We should shade the Kudu client and it’s async dependency into the > kudu-handler jar to simplify add jar style usage and ensure the > addDependencyJars method adds all the required classes. This is a common > pattern the other storage handlers use. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22338) Shade dependent jars into the kudu-handler
[ https://issues.apache.org/jira/browse/HIVE-22338?focusedWorklogId=328029&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-328029 ] ASF GitHub Bot logged work on HIVE-22338: - Author: ASF GitHub Bot Created on: 14/Oct/19 18:02 Start Date: 14/Oct/19 18:02 Worklog Time Spent: 10m Work Description: granthenke commented on pull request #816: HIVE-22338: Shade dependent jars into the kudu-handler URL: https://github.com/apache/hive/pull/816 This patch shades the Kudu client and it’s async dependency into the kudu-handler jar to simplify add jar style usage and ensure the addDependencyJars method adds all the required classes. It also adjusts the master address property to fallback to the default instead of looking it up first. This allows the default to be unset when the table property exists. I ran into this when using the handler via `add jar`. Change-Id: Ie900abe21b685c9591e6147efe5c3b4b96791f5e This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 328029) Remaining Estimate: 0h Time Spent: 10m > Shade dependent jars into the kudu-handler > -- > > Key: HIVE-22338 > URL: https://issues.apache.org/jira/browse/HIVE-22338 > Project: Hive > Issue Type: Sub-task >Reporter: Grant Henke >Assignee: Grant Henke >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22338.0.patch > > Time Spent: 10m > Remaining Estimate: 0h > > We should shade the Kudu client and it’s async dependency into the > kudu-handler jar to simplify add jar style usage and ensure the > addDependencyJars method adds all the required classes. This is a common > pattern the other storage handlers use. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22338) Shade dependent jars into the kudu-handler
[ https://issues.apache.org/jira/browse/HIVE-22338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke updated HIVE-22338: --- Attachment: HIVE-22338.0.patch > Shade dependent jars into the kudu-handler > -- > > Key: HIVE-22338 > URL: https://issues.apache.org/jira/browse/HIVE-22338 > Project: Hive > Issue Type: Sub-task >Reporter: Grant Henke >Assignee: Grant Henke >Priority: Major > Attachments: HIVE-22338.0.patch > > > We should shade the Kudu client and it’s async dependency into the > kudu-handler jar to simplify add jar style usage and ensure the > addDependencyJars method adds all the required classes. This is a common > pattern the other storage handlers use. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22338) Shade dependent jars into the kudu-handler
[ https://issues.apache.org/jira/browse/HIVE-22338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-22338: -- Labels: pull-request-available (was: ) > Shade dependent jars into the kudu-handler > -- > > Key: HIVE-22338 > URL: https://issues.apache.org/jira/browse/HIVE-22338 > Project: Hive > Issue Type: Sub-task >Reporter: Grant Henke >Assignee: Grant Henke >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22338.0.patch > > > We should shade the Kudu client and it’s async dependency into the > kudu-handler jar to simplify add jar style usage and ensure the > addDependencyJars method adds all the required classes. This is a common > pattern the other storage handlers use. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22338) Shade dependent jars into the kudu-handler
[ https://issues.apache.org/jira/browse/HIVE-22338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Grant Henke reassigned HIVE-22338: -- > Shade dependent jars into the kudu-handler > -- > > Key: HIVE-22338 > URL: https://issues.apache.org/jira/browse/HIVE-22338 > Project: Hive > Issue Type: Sub-task >Reporter: Grant Henke >Assignee: Grant Henke >Priority: Major > > We should shade the Kudu client and it’s async dependency into the > kudu-handler jar to simplify add jar style usage and ensure the > addDependencyJars method adds all the required classes. This is a common > pattern the other storage handlers use. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22331) unix_timestamp without argument returns timestamp in millisecond instead of second.
[ https://issues.apache.org/jira/browse/HIVE-22331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22331: --- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master, thanks [~nareshpr]! > unix_timestamp without argument returns timestamp in millisecond instead of > second. > --- > > Key: HIVE-22331 > URL: https://issues.apache.org/jira/browse/HIVE-22331 > Project: Hive > Issue Type: Bug >Reporter: Naresh P R >Assignee: Naresh P R >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22331.1.patch > > > After HIVE-22170, select unix_timestamp(); is returning milliseconds, but > expected output is in seconds. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22332) Hive should ensure valid schema evolution settings since ORC-540
[
https://issues.apache.org/jira/browse/HIVE-22332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951194#comment-16951194
]
Hive QA commented on HIVE-22332:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982938/HIVE-22332.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17537 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18982/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18982/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18982/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982938 - PreCommit-HIVE-Build
> Hive should ensure valid schema evolution settings since ORC-540
>
>
> Key: HIVE-22332
> URL: https://issues.apache.org/jira/browse/HIVE-22332
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Critical
> Fix For: 4.0.0
>
> Attachments: HIVE-22332.01.patch, HIVE-22332.02.patch
>
>
> For details please see: https://issues.apache.org/jira/browse/ORC-558
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22337) Improve and Expand Text-Based SerDes
[
https://issues.apache.org/jira/browse/HIVE-22337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-22337:
--
Description:
* Add new SerDe package just for text-based formats:
org.apache.hadoop.hive.serde2.text.*
* Add new SerDe package just for text-based log formats:
org.apache.hadoop.hive.serde2.text.log.*
* Create a coherent hierarchy for processing delimited data: AbstractSerDe ->
TextSerDe -> EncodingAwareTextSerde -> DelimitedSerDe -> CsvTestSerDe
* Create a coherent hierarchy for processing regex'ed data: AbstractSerDe ->
TextSerDe -> EncodingAwareTextSerde -> RegexSerDe -> CommonFormatLogSerDe
* Create some standard text processors for super-quick out-of-the-box
processing: TSV SerDe and CSV SerDe
* Create some standard log processors for super-quick out-of-the-box
processing: Apache Common Log Format and Apache Combined Log Format (Apache
HTTP Server Log Parsers)
* Better default behaviors for processing text
The default behavior should allow users to quick query data without any
failures.
# When a blank line is encountered, insert a 'null' value for each column
# When there are fewer fields in the data than defined in the table schema,
shift all available fields left, and fill in 'null' values for all remaining
fields
# When there are too many fields in the data, the last field in the results
will contain all remaining values. Currently, the data is silently swallows
and a warning is issued in the YARN logs. A normal user will never see this
warning, especially if the job completes successfully. Better to (by default)
provide them all the data than to hide anything.
{code:none|title=CSV SerDe}
"1,2,3"= ["1","2","3"]
"1,2," = ["1","2",null]
"" = [null,null,null]
"1,2,3,4" = ["1","2","3,4"]
{code}
was:
* Add new SerDe package just for text-based formats:
org.apache.hadoop.hive.serde2.text.*
* Add new SerDe package just for text-based log formats:
org.apache.hadoop.hive.serde2.text.log.*
* Create a coherent hierarchy for processing delimited data: AbstractSerDe ->
TextSerDe -> EncodingAwareTextSerde -> DelimitedSerDe
* Create a coherent hierarchy for processing regex'ed data: AbstractSerDe ->
TextSerDe -> EncodingAwareTextSerde -> RegexSerDe
* Create some standard text processors for super-quick out-of-the-box
processing: TSV SerDe and CSV SerDe
* Create some standard log processors for super-quick out-of-the-box
processing: Apache Common Log Format and Apache Combined Log Format (Apache
HTTP Server Log Parsers)
* Better default behaviors for processing text
The default behavior should allow users to quick query data without any
failures.
# When a blank line is encountered, insert a 'null' value for each column
# When there are fewer fields in the data than defined in the table schema,
shift all available fields left, and fill in 'null' values for all remaining
fields
# When there are too many fields in the data, the last field in the results
will contain all remaining values. Currently, the data is silently swallows
and a warning is issued in the YARN logs. A normal user will never see this
warning, especially if the job completes successfully. Better to (by default)
provide them all the data than to hide anything.
{code:none|title=CSV SerDe}
"1,2,3"= ["1","2","3"]
"1,2," = ["1","2",null]
"" = [null,null,null]
"1,2,3,4" = ["1","2","3,4"]
{code}
> Improve and Expand Text-Based SerDes
>
>
> Key: HIVE-22337
> URL: https://issues.apache.org/jira/browse/HIVE-22337
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 4.0.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22337.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> * Add new SerDe package just for text-based formats:
> org.apache.hadoop.hive.serde2.text.*
> * Add new SerDe package just for text-based log formats:
> org.apache.hadoop.hive.serde2.text.log.*
> * Create a coherent hierarchy for processing delimited data: AbstractSerDe ->
> TextSerDe -> EncodingAwareTextSerde -> DelimitedSerDe -> CsvTestSerDe
> * Create a coherent hierarchy for processing regex'ed data: AbstractSerDe ->
> TextSerDe -> EncodingAwareTextSerde -> RegexSerDe -> CommonFormatLogSerDe
> * Create some standard text processors for super-quick out-of-the-box
> processing: TSV SerDe and CSV SerDe
> * Create some standard log processors for super-quick out-of-the-box
> processing: Apache Common Log Format and Apache Combined Log Format (Apache
> HTTP Server Log Parsers)
> * Better default behaviors for processing text
> The default behavior should allow users to quick query da
[jira] [Updated] (HIVE-22337) Improve and Expand Text-Based SerDes
[
https://issues.apache.org/jira/browse/HIVE-22337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-22337:
--
Description:
* Add new SerDe package just for text-based formats:
org.apache.hadoop.hive.serde2.text.*
* Add new SerDe package just for text-based log formats:
org.apache.hadoop.hive.serde2.text.log.*
* Create a coherent hierarchy for processing delimited data: AbstractSerDe ->
TextSerDe -> EncodingAwareTextSerde -> DelimitedSerDe
* Create a coherent hierarchy for processing regex'ed data: AbstractSerDe ->
TextSerDe -> EncodingAwareTextSerde -> RegexSerDe
* Create some standard text processors for super-quick out-of-the-box
processing: TSV SerDe and CSV SerDe
* Create some standard log processors for super-quick out-of-the-box
processing: Apache Common Log Format and Apache Combined Log Format (Apache
HTTP Server Log Parsers)
* Better default behaviors for processing text
The default behavior should allow users to quick query data without any
failures.
# When a blank line is encountered, insert a 'null' value for each column
# When there are fewer fields in the data than defined in the table schema,
shift all available fields left, and fill in 'null' values for all remaining
fields
# When there are too many fields in the data, the last field in the results
will contain all remaining values. Currently, the data is silently swallows
and a warning is issued in the YARN logs. A normal user will never see this
warning, especially if the job completes successfully. Better to (by default)
provide them all the data than to hide anything.
{code:none|title=CSV SerDe}
"1,2,3"= ["1","2","3"]
"1,2," = ["1","2",null]
"" = [null,null,null]
"1,2,3,4" = ["1","2","3,4"]
{code}
> Improve and Expand Text-Based SerDes
>
>
> Key: HIVE-22337
> URL: https://issues.apache.org/jira/browse/HIVE-22337
> Project: Hive
> Issue Type: Improvement
> Components: Serializers/Deserializers
>Affects Versions: 4.0.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22337.1.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> * Add new SerDe package just for text-based formats:
> org.apache.hadoop.hive.serde2.text.*
> * Add new SerDe package just for text-based log formats:
> org.apache.hadoop.hive.serde2.text.log.*
> * Create a coherent hierarchy for processing delimited data: AbstractSerDe ->
> TextSerDe -> EncodingAwareTextSerde -> DelimitedSerDe
> * Create a coherent hierarchy for processing regex'ed data: AbstractSerDe ->
> TextSerDe -> EncodingAwareTextSerde -> RegexSerDe
> * Create some standard text processors for super-quick out-of-the-box
> processing: TSV SerDe and CSV SerDe
> * Create some standard log processors for super-quick out-of-the-box
> processing: Apache Common Log Format and Apache Combined Log Format (Apache
> HTTP Server Log Parsers)
> * Better default behaviors for processing text
> The default behavior should allow users to quick query data without any
> failures.
> # When a blank line is encountered, insert a 'null' value for each column
> # When there are fewer fields in the data than defined in the table schema,
> shift all available fields left, and fill in 'null' values for all remaining
> fields
> # When there are too many fields in the data, the last field in the results
> will contain all remaining values. Currently, the data is silently swallows
> and a warning is issued in the YARN logs. A normal user will never see this
> warning, especially if the job completes successfully. Better to (by
> default) provide them all the data than to hide anything.
> {code:none|title=CSV SerDe}
> "1,2,3"= ["1","2","3"]
> "1,2," = ["1","2",null]
> "" = [null,null,null]
> "1,2,3,4" = ["1","2","3,4"]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22331) unix_timestamp without argument returns timestamp in millisecond instead of second.
[ https://issues.apache.org/jira/browse/HIVE-22331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951181#comment-16951181 ] Jesus Camacho Rodriguez commented on HIVE-22331: +1 > unix_timestamp without argument returns timestamp in millisecond instead of > second. > --- > > Key: HIVE-22331 > URL: https://issues.apache.org/jira/browse/HIVE-22331 > Project: Hive > Issue Type: Bug >Reporter: Naresh P R >Assignee: Naresh P R >Priority: Major > Attachments: HIVE-22331.1.patch > > > After HIVE-22170, select unix_timestamp(); is returning milliseconds, but > expected output is in seconds. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22332) Hive should ensure valid schema evolution settings since ORC-540
[
https://issues.apache.org/jira/browse/HIVE-22332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951155#comment-16951155
]
Hive QA commented on HIVE-22332:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
17s{color} | {color:blue} ql in master has 1549 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
16s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 10s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18982/dev-support/hive-personality.sh
|
| git revision | master / 74fee23 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18982/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18982/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive should ensure valid schema evolution settings since ORC-540
>
>
> Key: HIVE-22332
> URL: https://issues.apache.org/jira/browse/HIVE-22332
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Critical
> Fix For: 4.0.0
>
> Attachments: HIVE-22332.01.patch, HIVE-22332.02.patch
>
>
> For details please see: https://issues.apache.org/jira/browse/ORC-558
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22337) Improve and Expand Text-Based SerDes
[ https://issues.apache.org/jira/browse/HIVE-22337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22337: -- Attachment: HIVE-22337.1.patch > Improve and Expand Text-Based SerDes > > > Key: HIVE-22337 > URL: https://issues.apache.org/jira/browse/HIVE-22337 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 4.0.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22337.1.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22337) Improve and Expand Text-Based SerDes
[ https://issues.apache.org/jira/browse/HIVE-22337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22337: -- Status: Patch Available (was: Open) > Improve and Expand Text-Based SerDes > > > Key: HIVE-22337 > URL: https://issues.apache.org/jira/browse/HIVE-22337 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 4.0.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22337.1.patch > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22284) Improve LLAP CacheContentsTracker to collect and display correct statistics
[
https://issues.apache.org/jira/browse/HIVE-22284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951116#comment-16951116
]
Hive QA commented on HIVE-22284:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982943/HIVE-22284.6.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17509 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=184)
[vector_windowing_expressions.q,tez_union_group_by.q,materialized_view_rewrite_no_join_opt_2.q,vector_like_2.q,llap_acid.q,sqlmerge.q,schema_evol_orc_acid_part_update_llap_io.q,vector_windowing_gby.q,rfc5424_parser.q,lateral_view.q,vector_decimal_join.q,optimize_nullscan.q,vectorization_decimal_date.q,schema_evol_orc_nonvec_table_llap_io.q,udaf_all_keyword.q,acid_vectorization_original.q,tez_fsstat.q,vector_fullouter_mapjoin_1_optimized_passthru.q,vector_mapjoin_reduce.q,join_acid_non_acid.q,empty_join.q,auto_join21.q,materialized_view_partition_cluster.q,tez_input_counters.q,skiphf_aggr2.q,vector_groupby_sort_11.q,schema_evol_orc_nonvec_part_all_complex_llap_io.q,orc_ppd_timestamp.q,vector_decimal_1.q,tez_fixed_bucket_pruning.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18981/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18981/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18981/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982943 - PreCommit-HIVE-Build
> Improve LLAP CacheContentsTracker to collect and display correct statistics
> ---
>
> Key: HIVE-22284
> URL: https://issues.apache.org/jira/browse/HIVE-22284
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Attachments: HIVE-22284.0.patch, HIVE-22284.1.patch,
> HIVE-22284.2.patch, HIVE-22284.3.patch, HIVE-22284.4.patch,
> HIVE-22284.5.patch, HIVE-22284.6.patch, HIVE-22284.7.patch
>
>
> When keeping track of which buffers correspond to what Hive objects,
> CacheContentsTracker relies on cache tags.
> Currently a tag is a simple String that ideally holds DB and table name, and
> a partition spec concatenated by . and / . The information here is derived
> from the Path of the file that is getting cached. Needless to say sometimes
> this produces a wrong tag especially for external tables.
> Also there's a bug when calculating aggregated stats for a 'parent' tag
> (corresponding to the table of the partition) because the overall maxCount
> and maxSize do not add up to the sum of those in the partitions. This happens
> when buffers get removed from the cache.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22337) Improve and Expand Text-Based SerDes
[ https://issues.apache.org/jira/browse/HIVE-22337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-22337: -- Labels: pull-request-available (was: ) > Improve and Expand Text-Based SerDes > > > Key: HIVE-22337 > URL: https://issues.apache.org/jira/browse/HIVE-22337 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 4.0.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22337) Improve and Expand Text-Based SerDes
[ https://issues.apache.org/jira/browse/HIVE-22337?focusedWorklogId=327940&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327940 ] ASF GitHub Bot logged work on HIVE-22337: - Author: ASF GitHub Bot Created on: 14/Oct/19 16:19 Start Date: 14/Oct/19 16:19 Worklog Time Spent: 10m Work Description: belugabehr commented on pull request #815: HIVE-22337: Improve and Expand Text-Based SerDes URL: https://github.com/apache/hive/pull/815 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 327940) Remaining Estimate: 0h Time Spent: 10m > Improve and Expand Text-Based SerDes > > > Key: HIVE-22337 > URL: https://issues.apache.org/jira/browse/HIVE-22337 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 4.0.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22337) Improve and Expand Text-Based SerDes
[ https://issues.apache.org/jira/browse/HIVE-22337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor reassigned HIVE-22337: - > Improve and Expand Text-Based SerDes > > > Key: HIVE-22337 > URL: https://issues.apache.org/jira/browse/HIVE-22337 > Project: Hive > Issue Type: Improvement > Components: Serializers/Deserializers >Affects Versions: 4.0.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Fix For: 4.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22284) Improve LLAP CacheContentsTracker to collect and display correct statistics
[
https://issues.apache.org/jira/browse/HIVE-22284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951080#comment-16951080
]
Hive QA commented on HIVE-22284:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
23s{color} | {color:blue} storage-api in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} llap-common in master has 90 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
58s{color} | {color:blue} ql in master has 1549 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
40s{color} | {color:blue} llap-server in master has 90 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} storage-api: The patch generated 2 new + 4 unchanged -
0 fixed = 6 total (was 4) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 2 new + 165 unchanged - 2
fixed = 167 total (was 167) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} llap-server: The patch generated 1 new + 252 unchanged
- 13 fixed = 253 total (was 265) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
15s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18981/dev-support/hive-personality.sh
|
| git revision | master / 74fee23 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18981/yetus/diff-checkstyle-storage-api.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18981/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18981/yetus/diff-checkstyle-llap-server.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18981/yetus/patch-asflicense-problems.txt
|
| modules | C: storage-api llap-common ql llap-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18981/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Improve LL
[jira] [Commented] (HIVE-22332) Hive should ensure valid schema evolution settings since ORC-540
[ https://issues.apache.org/jira/browse/HIVE-22332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951073#comment-16951073 ] Ashutosh Chauhan commented on HIVE-22332: - +1 > Hive should ensure valid schema evolution settings since ORC-540 > > > Key: HIVE-22332 > URL: https://issues.apache.org/jira/browse/HIVE-22332 > Project: Hive > Issue Type: Bug >Reporter: László Bodor >Assignee: László Bodor >Priority: Critical > Fix For: 4.0.0 > > Attachments: HIVE-22332.01.patch, HIVE-22332.02.patch > > > For details please see: https://issues.apache.org/jira/browse/ORC-558 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-21098) DPP: SyntheticJoinPredicate transitivity for < > and BETWEEN needs extension
[ https://issues.apache.org/jira/browse/HIVE-21098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Zhang reassigned HIVE-21098: Assignee: Attila Magyar (was: Deepak Jaiswal) > DPP: SyntheticJoinPredicate transitivity for < > and BETWEEN needs extension > > > Key: HIVE-21098 > URL: https://issues.apache.org/jira/browse/HIVE-21098 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Deepak Jaiswal >Assignee: Attila Magyar >Priority: Major > > SyntheticJoinPredicates are supported for equality. Both in regular and > extended format. > Similar extended format is needed for non-equi joins too. > > See HIVE-16976 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-20155) Semijoin Reduction : Put all the min-max filters before all the bloom filters
[ https://issues.apache.org/jira/browse/HIVE-20155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Richard Zhang reassigned HIVE-20155: Assignee: Attila Magyar (was: Deepak Jaiswal) > Semijoin Reduction : Put all the min-max filters before all the bloom filters > - > > Key: HIVE-20155 > URL: https://issues.apache.org/jira/browse/HIVE-20155 > Project: Hive > Issue Type: Task >Reporter: Deepak Jaiswal >Assignee: Attila Magyar >Priority: Major > > If there are more than 1 semijoin reduction filters, apply all min-max > filters before any of the bloom filters are applied as bloom filter lookup is > expensive. > > cc [~gopalv] [~jdere] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-21196) Support semijoin reduction on multiple column join
[
https://issues.apache.org/jira/browse/HIVE-21196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Richard Zhang reassigned HIVE-21196:
Assignee: Mustafa Iman (was: Deepak Jaiswal)
> Support semijoin reduction on multiple column join
> --
>
> Key: HIVE-21196
> URL: https://issues.apache.org/jira/browse/HIVE-21196
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Mustafa Iman
>Priority: Major
>
> Currently for a query involving join on multiple columns creates separate
> semi join edges for each key which in turn create a bloom filter for each of
> them, like below,
> EXPLAIN select count(*) from srcpart_date_n7 join srcpart_small_n3 on
> (srcpart_date_n7.key = srcpart_small_n3.key1 and srcpart_date_n7.value =
> srcpart_small_n3.value1)
> {code:java}
> Map 1 <- Reducer 5 (BROADCAST_EDGE)
> Reducer 2 <- Map 1 (SIMPLE_EDGE), Map 4 (SIMPLE_EDGE)
> Reducer 3 <- Reducer 2 (CUSTOM_SIMPLE_EDGE)
> Reducer 5 <- Map 4 (CUSTOM_SIMPLE_EDGE)
> A masked pattern was here
> Vertices:
> Map 1
> Map Operator Tree:
> TableScan
> alias: srcpart_date_n7
> filterExpr: (key is not null and value is not null and (key
> BETWEEN DynamicValue(RS_7_srcpart_small_n3_key1_min) AND
> DynamicValue(RS_7_srcpart_small_n3_key1_max) and in_bloom_filter(key,
> DynamicValue(RS_7_srcpart_small_n3_key1_bloom_filter (type: boolean)
> Statistics: Num rows: 2000 Data size: 356000 Basic stats:
> COMPLETE Column stats: COMPLETE
> Filter Operator
> predicate: ((key BETWEEN
> DynamicValue(RS_7_srcpart_small_n3_key1_min) AND
> DynamicValue(RS_7_srcpart_small_n3_key1_max) and in_bloom_filter(key,
> DynamicValue(RS_7_srcpart_small_n3_key1_bloom_filter))) and key is not null
> and value is not null) (type: boolean)
> Statistics: Num rows: 2000 Data size: 356000 Basic stats:
> COMPLETE Column stats: COMPLETE
> Select Operator
> expressions: key (type: string), value (type: string)
> outputColumnNames: _col0, _col1
> Statistics: Num rows: 2000 Data size: 356000 Basic
> stats: COMPLETE Column stats: COMPLETE
> Reduce Output Operator
> key expressions: _col0 (type: string), _col1 (type:
> string)
> sort order: ++
> Map-reduce partition columns: _col0 (type: string),
> _col1 (type: string)
> Statistics: Num rows: 2000 Data size: 356000 Basic
> stats: COMPLETE Column stats: COMPLETE
> Execution mode: vectorized, llap
> LLAP IO: all inputs
> Map 4
> Map Operator Tree:
> TableScan
> alias: srcpart_small_n3
> filterExpr: (key1 is not null and value1 is not null)
> (type: boolean)
> Statistics: Num rows: 20 Data size: 3560 Basic stats:
> PARTIAL Column stats: PARTIAL
> Filter Operator
> predicate: (key1 is not null and value1 is not null)
> (type: boolean)
> Statistics: Num rows: 20 Data size: 3560 Basic stats:
> PARTIAL Column stats: PARTIAL
> Select Operator
> expressions: key1 (type: string), value1 (type: string)
> outputColumnNames: _col0, _col1
> Statistics: Num rows: 20 Data size: 3560 Basic stats:
> PARTIAL Column stats: PARTIAL
> Reduce Output Operator
> key expressions: _col0 (type: string), _col1 (type:
> string)
> sort order: ++
> Map-reduce partition columns: _col0 (type: string),
> _col1 (type: string)
> Statistics: Num rows: 20 Data size: 3560 Basic stats:
> PARTIAL Column stats: PARTIAL
> Select Operator
> expressions: _col0 (type: string)
> outputColumnNames: _col0
> Statistics: Num rows: 20 Data size: 3560 Basic stats:
> PARTIAL Column stats: PARTIAL
> Group By Operator
> aggregations: min(_col0), max(_col0),
> bloom_filter(_col0, expectedEntries=20)
> mode: hash
> outputColumnNames: _col0, _col1, _col2
> Statistics: Num rows: 1 Data size: 730 Basic stats:
> PARTIAL Column stats: PARTIAL
> Reduce Output Operator
> sort order:
> Statist
[jira] [Updated] (HIVE-22284) Improve LLAP CacheContentsTracker to collect and display correct statistics
[ https://issues.apache.org/jira/browse/HIVE-22284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ádám Szita updated HIVE-22284: -- Attachment: HIVE-22284.7.patch > Improve LLAP CacheContentsTracker to collect and display correct statistics > --- > > Key: HIVE-22284 > URL: https://issues.apache.org/jira/browse/HIVE-22284 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Attachments: HIVE-22284.0.patch, HIVE-22284.1.patch, > HIVE-22284.2.patch, HIVE-22284.3.patch, HIVE-22284.4.patch, > HIVE-22284.5.patch, HIVE-22284.6.patch, HIVE-22284.7.patch > > > When keeping track of which buffers correspond to what Hive objects, > CacheContentsTracker relies on cache tags. > Currently a tag is a simple String that ideally holds DB and table name, and > a partition spec concatenated by . and / . The information here is derived > from the Path of the file that is getting cached. Needless to say sometimes > this produces a wrong tag especially for external tables. > Also there's a bug when calculating aggregated stats for a 'parent' tag > (corresponding to the table of the partition) because the overall maxCount > and maxSize do not add up to the sum of those in the partitions. This happens > when buffers get removed from the cache. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21327) Predicate is not pushed to Parquet if hive.parquet.timestamp.skip.conversion=true
[
https://issues.apache.org/jira/browse/HIVE-21327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951049#comment-16951049
]
Hive QA commented on HIVE-21327:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982936/HIVE-21327.2.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17537 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18980/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18980/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18980/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982936 - PreCommit-HIVE-Build
> Predicate is not pushed to Parquet if
> hive.parquet.timestamp.skip.conversion=true
> -
>
> Key: HIVE-21327
> URL: https://issues.apache.org/jira/browse/HIVE-21327
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Attachments: HIVE-21327.1.patch, HIVE-21327.2.patch
>
>
> The Parquet FilterPredicate is created and set to the configuration in the
> ParquetRecordReaderBase.setFilter method. This method is used from the
> ParquetRecordReaderWrapper constructor through the
> ParquetRecordReaderBase.getSplit method and expects a JobConf as parameter
> where it sets the created filter predicate. In the ParquetRecordReaderWrapper
> constructor, multiple JobConf object is used:
> {noformat}
> jobConf = oldJobConf;
> final ParquetInputSplit split = getSplit(oldSplit, jobConf);
> TaskAttemptID taskAttemptID =
> TaskAttemptID.forName(jobConf.get(IOConstants.MAPRED_TASK_ID));
> if (taskAttemptID == null) {
> taskAttemptID = new TaskAttemptID();
> }
> // create a TaskInputOutputContext
> Configuration conf = jobConf;
> if (skipTimestampConversion ^ HiveConf.getBoolVar(
> conf, HiveConf.ConfVars.HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION)) {
> conf = new JobConf(oldJobConf);
> HiveConf.setBoolVar(conf,
> HiveConf.ConfVars.HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION,
> skipTimestampConversion);
> }
> final TaskAttemptContext taskContext =
> ContextUtil.newTaskAttemptContext(conf, taskAttemptID);
> {noformat}
> So we have the jobConf, oldJobConf and conf objects and the getSplit is
> called with the jobConf object, so the filter predicate will be set into this
> config object. Based on this code part, the jobConf and oldJobConf should be
> the same reference inside the if statement, so the newly created conf should
> also contain the filter predicate. However in the getSplit method the value
> of the jobConf is changed by the projectionPusher.pushProjectionsAndFilters
> method, so inside the if statement, the jobConf and the oldJobConf are
> actually different references. The filter predicate is set in the jobConf,
> but if the if condition is true, the conf will be created from the oldJobConf
> so it won't contain the filter predicate.
> Just for reference, this behavior was introduced in
> [HIVE-9873|https://issues.apache.org/jira/browse/HIVE-9873].
> Since the goal of the if statement is only to update the
> HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION property in the configuration, it
> should be using the jobConf where the filter predicate is correctly set.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22276) Break up DDLSemanticAnalyzer - extract View related analyzers
[ https://issues.apache.org/jira/browse/HIVE-22276?focusedWorklogId=327822&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327822 ] ASF GitHub Bot logged work on HIVE-22276: - Author: ASF GitHub Bot Created on: 14/Oct/19 14:04 Start Date: 14/Oct/19 14:04 Worklog Time Spent: 10m Work Description: miklosgergely commented on pull request #800: HIVE-22276 Break up DDLSemanticAnalyzer - extract View related analyzers URL: https://github.com/apache/hive/pull/800#discussion_r334498447 ## File path: ql/src/test/results/clientnegative/materialized_view_no_transactional_rewrite_2.q.out ## @@ -27,4 +27,4 @@ POSTHOOK: type: CREATE_MATERIALIZED_VIEW POSTHOOK: Input: default@cmv_basetable POSTHOOK: Output: database:default POSTHOOK: Output: default@cmv_mat_view -FAILED: SemanticException Automatic rewriting for materialized view cannot be enabled if the materialized view uses non-transactional tables +FAILED: SemanticException Automatic rewriting for materialized view cannot be enabled if the materialized view uses non-transactional tables Review comment: Fixed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 327822) Time Spent: 40m (was: 0.5h) > Break up DDLSemanticAnalyzer - extract View related analyzers > - > > Key: HIVE-22276 > URL: https://issues.apache.org/jira/browse/HIVE-22276 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Attachments: HIVE-22276.01.patch, HIVE-22276.02.patch, > HIVE-22276.03.patch > > Time Spent: 40m > Remaining Estimate: 0h > > DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is > to refactor it in order to have everything cut into more handleable classes > under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each analyzers > * have a package for each operation, containing an analyzer, a description, > and an operation, so the amount of classes under a package is more manageable > Step #6: extract all the vire related analyzers from DDLSemanticAnalyzer, and > move them under the new package. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22276) Break up DDLSemanticAnalyzer - extract View related analyzers
[
https://issues.apache.org/jira/browse/HIVE-22276?focusedWorklogId=327825&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327825
]
ASF GitHub Bot logged work on HIVE-22276:
-
Author: ASF GitHub Bot
Created on: 14/Oct/19 14:04
Start Date: 14/Oct/19 14:04
Worklog Time Spent: 10m
Work Description: miklosgergely commented on pull request #800:
HIVE-22276 Break up DDLSemanticAnalyzer - extract View related analyzers
URL: https://github.com/apache/hive/pull/800#discussion_r334498615
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/DDLSemanticAnalyzerFactory.java
##
@@ -52,8 +55,11 @@ private DDLSemanticAnalyzerFactory() {
private static final Map>
TYPE_TO_ANALYZER = new HashMap<>();
static {
-Set> analyzerClasses =
+Set> analyzerClasses1 =
new
Reflections("org.apache.hadoop.hive.ql.ddl").getSubTypesOf(BaseSemanticAnalyzer.class);
Review comment:
Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 327825)
Time Spent: 1h (was: 50m)
> Break up DDLSemanticAnalyzer - extract View related analyzers
> -
>
> Key: HIVE-22276
> URL: https://issues.apache.org/jira/browse/HIVE-22276
> Project: Hive
> Issue Type: Sub-task
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available, refactor-ddl
> Attachments: HIVE-22276.01.patch, HIVE-22276.02.patch,
> HIVE-22276.03.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is
> to refactor it in order to have everything cut into more handleable classes
> under the package org.apache.hadoop.hive.ql.exec.ddl:
> * have a separate class for each analyzers
> * have a package for each operation, containing an analyzer, a description,
> and an operation, so the amount of classes under a package is more manageable
> Step #6: extract all the vire related analyzers from DDLSemanticAnalyzer, and
> move them under the new package.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22276) Break up DDLSemanticAnalyzer - extract View related analyzers
[
https://issues.apache.org/jira/browse/HIVE-22276?focusedWorklogId=327823&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327823
]
ASF GitHub Bot logged work on HIVE-22276:
-
Author: ASF GitHub Bot
Created on: 14/Oct/19 14:04
Start Date: 14/Oct/19 14:04
Worklog Time Spent: 10m
Work Description: miklosgergely commented on pull request #800:
HIVE-22276 Break up DDLSemanticAnalyzer - extract View related analyzers
URL: https://github.com/apache/hive/pull/800#discussion_r334498535
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/alter/rewrite/AlterMaterializedViewRewriteDesc.java
##
@@ -27,17 +27,17 @@
*/
@Explain(displayName = "Alter Materialized View Rewrite", explainLevels = {
Level.USER, Level.DEFAULT, Level.EXTENDED })
public class AlterMaterializedViewRewriteDesc implements DDLDescWithWriteId {
- private final String fqMaterializedViewName;
+ private final String materializedViewName;
private final boolean rewriteEnable;
- public AlterMaterializedViewRewriteDesc(String fqMaterializedViewName,
boolean rewriteEnable) {
-this.fqMaterializedViewName = fqMaterializedViewName;
+ public AlterMaterializedViewRewriteDesc(String materializedViewName, boolean
rewriteEnable) {
+this.materializedViewName = materializedViewName;
Review comment:
Put back the fq prefix.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 327823)
Time Spent: 50m (was: 40m)
> Break up DDLSemanticAnalyzer - extract View related analyzers
> -
>
> Key: HIVE-22276
> URL: https://issues.apache.org/jira/browse/HIVE-22276
> Project: Hive
> Issue Type: Sub-task
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Labels: pull-request-available, refactor-ddl
> Attachments: HIVE-22276.01.patch, HIVE-22276.02.patch,
> HIVE-22276.03.patch
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is
> to refactor it in order to have everything cut into more handleable classes
> under the package org.apache.hadoop.hive.ql.exec.ddl:
> * have a separate class for each analyzers
> * have a package for each operation, containing an analyzer, a description,
> and an operation, so the amount of classes under a package is more manageable
> Step #6: extract all the vire related analyzers from DDLSemanticAnalyzer, and
> move them under the new package.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22276) Break up DDLSemanticAnalyzer - extract View related analyzers
[ https://issues.apache.org/jira/browse/HIVE-22276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Gergely updated HIVE-22276: -- Attachment: HIVE-22276.03.patch > Break up DDLSemanticAnalyzer - extract View related analyzers > - > > Key: HIVE-22276 > URL: https://issues.apache.org/jira/browse/HIVE-22276 > Project: Hive > Issue Type: Sub-task >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available, refactor-ddl > Attachments: HIVE-22276.01.patch, HIVE-22276.02.patch, > HIVE-22276.03.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > DDLSemanticAnalyzer is a huge class, more than 4000 lines long. The goal is > to refactor it in order to have everything cut into more handleable classes > under the package org.apache.hadoop.hive.ql.exec.ddl: > * have a separate class for each analyzers > * have a package for each operation, containing an analyzer, a description, > and an operation, so the amount of classes under a package is more manageable > Step #6: extract all the vire related analyzers from DDLSemanticAnalyzer, and > move them under the new package. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21327) Predicate is not pushed to Parquet if hive.parquet.timestamp.skip.conversion=true
[
https://issues.apache.org/jira/browse/HIVE-21327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16951021#comment-16951021
]
Hive QA commented on HIVE-21327:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
53s{color} | {color:blue} ql in master has 1549 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 3 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-18980/dev-support/hive-personality.sh
|
| git revision | master / 74fee23 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18980/yetus/patch-asflicense-problems.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-18980/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Predicate is not pushed to Parquet if
> hive.parquet.timestamp.skip.conversion=true
> -
>
> Key: HIVE-21327
> URL: https://issues.apache.org/jira/browse/HIVE-21327
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Marta Kuczora
>Assignee: Marta Kuczora
>Priority: Major
> Attachments: HIVE-21327.1.patch, HIVE-21327.2.patch
>
>
> The Parquet FilterPredicate is created and set to the configuration in the
> ParquetRecordReaderBase.setFilter method. This method is used from the
> ParquetRecordReaderWrapper constructor through the
> ParquetRecordReaderBase.getSplit method and expects a JobConf as parameter
> where it sets the created filter predicate. In the ParquetRecordReaderWrapper
> constructor, multiple JobConf object is used:
> {noformat}
> jobConf = oldJobConf;
> final ParquetInputSplit split = getSplit(oldSplit, jobConf);
> TaskAttemptID taskAttemptID =
> TaskAttemptID.forName(jobConf.get(IOConstants.MAPRED_TASK_ID));
> if (taskAttemptID == null) {
> taskAttemptID = new TaskAttemptID();
> }
> // create a TaskInputOutputContext
> Configuration conf = jobConf;
> if (skipTimestampConversion ^ HiveConf.getBoolVar(
> conf, HiveConf.ConfVars.HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION)) {
> conf = new JobConf(oldJobConf);
> HiveConf.setBoolVar(conf,
> HiveConf.ConfVars.HIVE_PARQUET_TIMESTAMP_SKIP_CONVERSION,
> skipTimestampC
[jira] [Updated] (HIVE-22240) Function percentile_cont fails when array parameter passed
[
https://issues.apache.org/jira/browse/HIVE-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22240:
--
Status: Patch Available (was: Open)
> Function percentile_cont fails when array parameter passed
> --
>
> Key: HIVE-22240
> URL: https://issues.apache.org/jira/browse/HIVE-22240
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22240.1.patch, HIVE-22240.2.patch,
> HIVE-22240.3.patch, HIVE-22240.3.patch
>
>
> {code}
> SELECT
> percentile_cont(array(0.2, 0.5, 0.9)) WITHIN GROUP (ORDER BY value)
> FROM t_test;
> {code}
> hive.log:
> {code}
> 2019-09-24T21:00:43,203 ERROR [LocalJobRunner Map Task Executor #0]
> mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:573)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:148)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.util.ArrayList cannot be cast to
> org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:793)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:128)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:152)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
> ... 11 more
> Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast
> to org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileCont$PercentileContEvaluator.iterate(GenericUDAFPercentileCont.java:259)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:214)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:639)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:814)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:720)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:788)
> ... 17 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22240) Function percentile_cont fails when array parameter passed
[
https://issues.apache.org/jira/browse/HIVE-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22240:
--
Attachment: HIVE-22240.3.patch
> Function percentile_cont fails when array parameter passed
> --
>
> Key: HIVE-22240
> URL: https://issues.apache.org/jira/browse/HIVE-22240
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22240.1.patch, HIVE-22240.2.patch,
> HIVE-22240.3.patch, HIVE-22240.3.patch
>
>
> {code}
> SELECT
> percentile_cont(array(0.2, 0.5, 0.9)) WITHIN GROUP (ORDER BY value)
> FROM t_test;
> {code}
> hive.log:
> {code}
> 2019-09-24T21:00:43,203 ERROR [LocalJobRunner Map Task Executor #0]
> mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:573)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:148)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.util.ArrayList cannot be cast to
> org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:793)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:128)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:152)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
> ... 11 more
> Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast
> to org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileCont$PercentileContEvaluator.iterate(GenericUDAFPercentileCont.java:259)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:214)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:639)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:814)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:720)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:788)
> ... 17 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22240) Function percentile_cont fails when array parameter passed
[
https://issues.apache.org/jira/browse/HIVE-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22240:
--
Status: Open (was: Patch Available)
> Function percentile_cont fails when array parameter passed
> --
>
> Key: HIVE-22240
> URL: https://issues.apache.org/jira/browse/HIVE-22240
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22240.1.patch, HIVE-22240.2.patch,
> HIVE-22240.3.patch, HIVE-22240.3.patch
>
>
> {code}
> SELECT
> percentile_cont(array(0.2, 0.5, 0.9)) WITHIN GROUP (ORDER BY value)
> FROM t_test;
> {code}
> hive.log:
> {code}
> 2019-09-24T21:00:43,203 ERROR [LocalJobRunner Map Task Executor #0]
> mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:573)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:148)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.util.ArrayList cannot be cast to
> org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:793)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:128)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:152)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
> ... 11 more
> Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast
> to org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileCont$PercentileContEvaluator.iterate(GenericUDAFPercentileCont.java:259)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:214)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:639)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:814)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:720)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:788)
> ... 17 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22240) Function percentile_cont fails when array parameter passed
[
https://issues.apache.org/jira/browse/HIVE-22240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16950998#comment-16950998
]
Hive QA commented on HIVE-22240:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12982935/HIVE-22240.3.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17537 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.llap.daemon.impl.TestTaskExecutorService.testSetCapacity
(batchId=364)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/18979/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/18979/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-18979/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12982935 - PreCommit-HIVE-Build
> Function percentile_cont fails when array parameter passed
> --
>
> Key: HIVE-22240
> URL: https://issues.apache.org/jira/browse/HIVE-22240
> Project: Hive
> Issue Type: Bug
> Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22240.1.patch, HIVE-22240.2.patch,
> HIVE-22240.3.patch
>
>
> {code}
> SELECT
> percentile_cont(array(0.2, 0.5, 0.9)) WITHIN GROUP (ORDER BY value)
> FROM t_test;
> {code}
> hive.log:
> {code}
> 2019-09-24T21:00:43,203 ERROR [LocalJobRunner Map Task Executor #0]
> mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:573)
> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:148)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapRunner.run(ExecMapRunner.java:37)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:465)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ClassCastException: java.util.ArrayList cannot be cast to
> org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:793)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:888)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:128)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:152)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:552)
> ... 11 more
> Caused by: java.lang.ClassCastException: java.util.ArrayList cannot be cast
> to org.apache.hadoop.hive.serde2.io.HiveDecimalWritable
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileCont$PercentileContEvaluator.iterate(GenericUDAFPercentileCont.java:259)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:214)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.updateAggregations(GroupByOperator.java:639)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processHashAggr(GroupByOperator.java:814)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.processKey(GroupByOperator.java:720)
> at
> org.apache.hadoop.hive.ql.exec.GroupByOperator.process(GroupByOperator.java:788)
> ... 17 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22298) Allow Llap IO cache for reading tables without delete delta
[ https://issues.apache.org/jira/browse/HIVE-22298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-22298: -- Attachment: HIVE-22298.3.patch > Allow Llap IO cache for reading tables without delete delta > --- > > Key: HIVE-22298 > URL: https://issues.apache.org/jira/browse/HIVE-22298 > Project: Hive > Issue Type: Improvement >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-22298.2.patch, HIVE-22298.3.patch, HIVE-22298.patch > > > Since HIVE-19985 we can use Llap IO cache for queries which are not > explicitly reading ROW_ID even for original files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22328) Min value for column in stats is not set correctly for some data types in partitioned tables
[ https://issues.apache.org/jira/browse/HIVE-22328?focusedWorklogId=327797&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-327797 ] ASF GitHub Bot logged work on HIVE-22328: - Author: ASF GitHub Bot Created on: 14/Oct/19 12:47 Start Date: 14/Oct/19 12:47 Worklog Time Spent: 10m Work Description: miklosgergely commented on pull request #811: HIVE-22328 Min value for column in stats is not set correctly for some data types in partitioned tables URL: https://github.com/apache/hive/pull/811 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 327797) Time Spent: 20m (was: 10m) > Min value for column in stats is not set correctly for some data types in > partitioned tables > > > Key: HIVE-22328 > URL: https://issues.apache.org/jira/browse/HIVE-22328 > Project: Hive > Issue Type: Bug >Reporter: Miklos Gergely >Assignee: Miklos Gergely >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22328.01.patch, HIVE-22328.02.patch > > Time Spent: 20m > Remaining Estimate: 0h > > This is a follow up Jira for HIVE-22248. For partitioned tables the > statistics aggregation happens at in the *ColumnStatsAggregator classes > instead of the *ColumnStatsMerger classes, and they still fail to handle the > unset low values correctly. Beside that they need to be fixed the two classes > should use the same codes for merging statistics. -- This message was sent by Atlassian Jira (v8.3.4#803005)
