[jira] [Resolved] (HIVE-28001) Fix the flaky test TestLeaderElection
[
https://issues.apache.org/jira/browse/HIVE-28001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihua Deng resolved HIVE-28001.
Fix Version/s: 4.0.0
Resolution: Fixed
Fix has been merged. Thank you for the review [~hemanth619]!
> Fix the flaky test TestLeaderElection
> -
>
> Key: HIVE-28001
> URL: https://issues.apache.org/jira/browse/HIVE-28001
> Project: Hive
> Issue Type: Test
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
>
> The TestLeaderElection is failing sometimes, example:
> [http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/master/2032/tests]
> [http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4995/3/tests]
> [http://ci.hive.apache.org/job/hive-flaky-check/796/]
> They are failed because of:
> java.lang.AssertionError
> at
> org.apache.hadoop.hive.metastore.leader.TestLeaderElection.testLeaseLeaderElection(TestLeaderElection.java:121)
> {code:java}
> assertFalse(lockId1 == instance1.getLockId()); {code}
> Lock remains the same, however it's supposed to be changed after a new leader
> is elected.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-28001) Fix the flaky test TestLeaderElection
[
https://issues.apache.org/jira/browse/HIVE-28001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihua Deng updated HIVE-28001:
---
Description:
The TestLeaderElection is failing sometimes, example:
[http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/master/2032/tests]
[http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4995/3/tests]
[http://ci.hive.apache.org/job/hive-flaky-check/796/]
They are failed because of:
java.lang.AssertionError
at
org.apache.hadoop.hive.metastore.leader.TestLeaderElection.testLeaseLeaderElection(TestLeaderElection.java:121)
{code:java}
assertFalse(lockId1 == instance1.getLockId()); {code}
Lock remains the same, however it's supposed to be changed after a new leader
is elected.
was:
The TestLeaderElection is failing sometimes, example:
[http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/master/2032/tests]
[http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4995/3/tests]
[http://ci.hive.apache.org/job/hive-flaky-check/796/]
> Fix the flaky test TestLeaderElection
> -
>
> Key: HIVE-28001
> URL: https://issues.apache.org/jira/browse/HIVE-28001
> Project: Hive
> Issue Type: Test
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
>
> The TestLeaderElection is failing sometimes, example:
> [http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/master/2032/tests]
> [http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4995/3/tests]
> [http://ci.hive.apache.org/job/hive-flaky-check/796/]
> They are failed because of:
> java.lang.AssertionError
> at
> org.apache.hadoop.hive.metastore.leader.TestLeaderElection.testLeaseLeaderElection(TestLeaderElection.java:121)
> {code:java}
> assertFalse(lockId1 == instance1.getLockId()); {code}
> Lock remains the same, however it's supposed to be changed after a new leader
> is elected.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-26463) Filter scheduled queries from the scheduler interface
[ https://issues.apache.org/jira/browse/HIVE-26463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17807925#comment-17807925 ] Attila Turoczy commented on HIVE-26463: --- [~shreenidhi] Don't you wanna open a PR with this patch? > Filter scheduled queries from the scheduler interface > - > > Key: HIVE-26463 > URL: https://issues.apache.org/jira/browse/HIVE-26463 > Project: Hive > Issue Type: Task > Components: Hive >Affects Versions: 4.0.0 >Reporter: Shreenidhi >Assignee: Shreenidhi >Priority: Minor > Attachments: get_repl_queries.patch > > > Scheduler will return all queries. But we should have a way to return only > the replication scheduled queries -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27512) CalciteSemanticException.UnsupportedFeature enum to capital
[
https://issues.apache.org/jira/browse/HIVE-27512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17807917#comment-17807917
]
Attila Turoczy commented on HIVE-27512:
---
Move ahead with this. This PR is simple. [~abstractdog] [~scarlin] Could you
pls check?
> CalciteSemanticException.UnsupportedFeature enum to capital
> ---
>
> Key: HIVE-27512
> URL: https://issues.apache.org/jira/browse/HIVE-27512
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: Mahesh Raju Somalaraju
>Priority: Major
> Labels: newbie, pull-request-available
>
> https://github.com/apache/hive/blob/3bc62cbc2d42c22dfd55f78ad7b41ec84a71380f/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSemanticException.java#L32-L39
> {code}
> public enum UnsupportedFeature {
> Distinct_without_an_aggreggation, Duplicates_in_RR,
> Filter_expression_with_non_boolean_return_type,
> Having_clause_without_any_groupby, Invalid_column_reference,
> Invalid_decimal,
> Less_than_equal_greater_than, Others, Same_name_in_multiple_expressions,
> Schema_less_table, Select_alias_in_having_clause, Select_transform,
> Subquery,
> Table_sample_clauses, UDTF, Union_type, Unique_join,
> HighPrecissionTimestamp // CALCITE-1690
> };
> {code}
> this just hurts my eyes, I expect it as DISTINCT_WITHOUT_AN_AGGREGATION ...
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27833) Hive Acid Replication Support for Dell Powerscale
[ https://issues.apache.org/jira/browse/HIVE-27833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Turoczy updated HIVE-27833: -- Fix Version/s: 4.0.0-beta-1 Resolution: Fixed Status: Resolved (was: Patch Available) Move to resolved state as this ticket is merged > Hive Acid Replication Support for Dell Powerscale > - > > Key: HIVE-27833 > URL: https://issues.apache.org/jira/browse/HIVE-27833 > Project: Hive > Issue Type: Bug > Components: repl >Reporter: Harshal Patel >Assignee: Harshal Patel >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-beta-1 > > > Hive Acid replication on Dell Powerscale Isilon fails because Isilon only > supports getXAttrs on /.reserved/raw paths -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-27950) STACK UDTF returns wrong results when # of argument is not a multiple of N
[
https://issues.apache.org/jira/browse/HIVE-27950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Turoczy updated HIVE-27950:
--
Affects Version/s: 4.0.0-beta-1
> STACK UDTF returns wrong results when # of argument is not a multiple of N
> --
>
> Key: HIVE-27950
> URL: https://issues.apache.org/jira/browse/HIVE-27950
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0-beta-1
>Reporter: okumin
>Assignee: okumin
>Priority: Major
> Labels: pull-request-available
>
> GenericUDTFStack nullifies a wrong cell when the number of values is
> indivisible. In the following case, the `col2` column of the last row should
> be `NULL`. But, `col1` is NULL somehow.
> {code:java}
> 0: jdbc:hive2://hive-hiveserver2:1/defaul> select stack(2, 'a', 'b', 'c',
> 'd', 'e');
> +---+---+---+
> | col0 | col1 | col2 |
> +---+---+---+
> | a | b | c |
> | d | NULL | c |
> +---+---+---+{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-28002) Make hive.metastore.batch.retrieve.max property to be read from hiveconf when it is requested from HS2
[ https://issues.apache.org/jira/browse/HIVE-28002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Turoczy updated HIVE-28002: -- Affects Version/s: 4.0.0-beta-1 > Make hive.metastore.batch.retrieve.max property to be read from hiveconf when > it is requested from HS2 > -- > > Key: HIVE-28002 > URL: https://issues.apache.org/jira/browse/HIVE-28002 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0-beta-1 >Reporter: Vikram Ahuja >Priority: Minor > Labels: pull-request-available > > The property hive.metastore.batch.retrieve.max is used by both the > client(Hs2) and the HMS when fetching objects(tables/partitions) in batches. > This property is present in both hiveConf as well as metastoreConf. For some > use cases in the client(Hs2), this value is referred from the metastoreConf. > To change these use cases to HiveConf.java to have a set demarkation between > HS2 and HMS use cases. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28005) Remove upgrade-acid module
[ https://issues.apache.org/jira/browse/HIVE-28005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Turoczy updated HIVE-28005: -- Affects Version/s: 4.0.0-beta-1 > Remove upgrade-acid module > -- > > Key: HIVE-28005 > URL: https://issues.apache.org/jira/browse/HIVE-28005 > Project: Hive > Issue Type: Task >Affects Versions: 4.0.0-beta-1 >Reporter: Ayush Saxena >Priority: Major > > Remove upgrade acid module, since it is relevant only for version 2 & 3 > Dev List: > https://lists.apache.org/thread/39m8ccz2y8rhlz5pygowb3rk0sjk929c -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28006) Materialized view with aggregate function incorrectly shows it allows incremental rebuild
[
https://issues.apache.org/jira/browse/HIVE-28006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Turoczy updated HIVE-28006:
--
Affects Version/s: 4.0.0-beta-1
4.0.0
4.1.0
> Materialized view with aggregate function incorrectly shows it allows
> incremental rebuild
> -
>
> Key: HIVE-28006
> URL: https://issues.apache.org/jira/browse/HIVE-28006
> Project: Hive
> Issue Type: Bug
> Components: Materialized views
>Affects Versions: 4.0.0, 4.0.0-beta-1, 4.1.0
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
>
> {code}
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> create table store_sales (
> ss_sold_date_sk int,
> ss_ext_sales_price int,
> ss_customer_sk int
> ) stored as orc TBLPROPERTIES ('transactional'='true');
> insert into store_sales (ss_sold_date_sk, ss_ext_sales_price, ss_customer_sk)
> values (2, 2, 2);
> create materialized view mat1 stored as orc tblproperties
> ('format-version'='2') as
> select ss_customer_sk
> ,min(ss_ext_sales_price)
> ,count(*)
> from store_sales
> group by ss_customer_sk;
> delete from store_sales where ss_sold_date_sk = 1;
> show materialized views;
> explain cbo
> alter materialized view mat1 rebuild;
> {code}
> Incremental rebuild is available
> {code}
> # MV Name Rewriting Enabled Mode
> Incremental rebuild
> mat1 Yes Manual refresh
> Available
> {code}
> vs full rebuild plan
> {code}
> CBO PLAN:
> HiveAggregate(group=[{2}], agg#0=[min($1)], agg#1=[count()])
> HiveTableScan(table=[[default, store_sales]], table:alias=[store_sales])
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-28008) ParquetFileReader is not closed in ParquetHiveSerDe.readSchema
[ https://issues.apache.org/jira/browse/HIVE-28008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Turoczy updated HIVE-28008: -- Affects Version/s: 4.1.0 > ParquetFileReader is not closed in ParquetHiveSerDe.readSchema > -- > > Key: HIVE-28008 > URL: https://issues.apache.org/jira/browse/HIVE-28008 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0, 4.1.0 >Reporter: Michal Lorek >Assignee: Michal Lorek >Priority: Minor > Labels: pull-request-available > > The readSchema method inside ParquetHiveSerDe does not close > ParquetFileReader. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-28002) Make hive.metastore.batch.retrieve.max property to be read from hiveconf when it is requested from HS2
[ https://issues.apache.org/jira/browse/HIVE-28002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17807909#comment-17807909 ] Attila Turoczy commented on HIVE-28002: --- Pls move to in-progress state > Make hive.metastore.batch.retrieve.max property to be read from hiveconf when > it is requested from HS2 > -- > > Key: HIVE-28002 > URL: https://issues.apache.org/jira/browse/HIVE-28002 > Project: Hive > Issue Type: Bug >Reporter: Vikram Ahuja >Priority: Minor > Labels: pull-request-available > > The property hive.metastore.batch.retrieve.max is used by both the > client(Hs2) and the HMS when fetching objects(tables/partitions) in batches. > This property is present in both hiveConf as well as metastoreConf. For some > use cases in the client(Hs2), this value is referred from the metastoreConf. > To change these use cases to HiveConf.java to have a set demarkation between > HS2 and HMS use cases. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28007) Java 11: Failure of TimeStamp tests
[ https://issues.apache.org/jira/browse/HIVE-28007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Turoczy updated HIVE-28007: -- Affects Version/s: 4.1.0 > Java 11: Failure of TimeStamp tests > > > Key: HIVE-28007 > URL: https://issues.apache.org/jira/browse/HIVE-28007 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 4.1.0 >Reporter: Kokila N >Assignee: Kokila N >Priority: Major > > Test cases are failing with value changes in timestamp. Check if its jdk > behavioural change and solve. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (HIVE-28004) DELETE on ACID table failed with NoClassDefFoundError: com/sun/tools/javac/util/List
[
https://issues.apache.org/jira/browse/HIVE-28004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17807898#comment-17807898
]
Attila Turoczy edited comment on HIVE-28004 at 1/17/24 9:54 PM:
[~lvegh] you still think this is a blocker for 4.0? Also pls move to in-progress
was (Author: JIRAUSER300479):
[~lvegh] you still think this is a blocker for 4.0?
> DELETE on ACID table failed with NoClassDefFoundError:
> com/sun/tools/javac/util/List
>
>
> Key: HIVE-28004
> URL: https://issues.apache.org/jira/browse/HIVE-28004
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0-beta-1
>Reporter: Butao Zhang
>Assignee: László Végh
>Priority: Blocker
>
> I am not sure if it is a bug or usage question.
> Test on Hive master branch:
>
> {code:java}
> set hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> set hive.support.concurrency = true;
> create table testacid4(id int) stored as orc TBLPROPERTIES
> ('transactional'='true');
> delete from testacid4 where id=110;
> {code}
>
> *beeline console shows error：*
> {code:java}
> 0: jdbc:hive2://127.0.0.1:1/default> delete from testacid4 where id=110;
> INFO : Compiling
> command(queryId=hive_20240116180628_ec5ac4d8-473b-4b42-b0dd-eecebec71268):
> delete from testacid4 where id=110
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:row__id,
> type:struct, comment:null)],
> properties:null)
> INFO : Completed compiling
> command(queryId=hive_20240116180628_ec5ac4d8-473b-4b42-b0dd-eecebec71268);
> Time taken: 3.554 seconds
> INFO : Operation QUERY obtained 1 locks
> ERROR : FAILED: Hive Internal Error:
> org.apache.hadoop.hive.ql.lockmgr.LockException(org.apache.thrift.TApplicationException:
> Internal error processing get_latest_txnid_in_conflict)
> org.apache.hadoop.hive.ql.lockmgr.LockException:
> org.apache.thrift.TApplicationException: Internal error processing
> get_latest_txnid_in_conflict
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.getLatestTxnIdInConflict(DbTxnManager.java:1055)
> at
> org.apache.hadoop.hive.ql.DriverTxnHandler.isValidTxnListState(DriverTxnHandler.java:435)
> at org.apache.hadoop.hive.ql.Driver.validateTxnList(Driver.java:250)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:199)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:154)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:149)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:185)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$500(SQLOperation.java:90)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:336)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:356)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.thrift.TApplicationException: Internal error processing
> get_latest_txnid_in_conflict
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_latest_txnid_in_conflict(ThriftHiveMetastore.java:6404)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_latest_txnid_in_conflict(ThriftHiveMetastore.java:6391)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getLatestTxnIdInConflict(HiveMetaStoreClient.java:4421)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreCl
[jira] [Commented] (HIVE-28003) Fix Batch execution issues
[ https://issues.apache.org/jira/browse/HIVE-28003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17807908#comment-17807908 ] Attila Turoczy commented on HIVE-28003: --- What is the root cause of the issue? > Fix Batch execution issues > -- > > Key: HIVE-28003 > URL: https://issues.apache.org/jira/browse/HIVE-28003 > Project: Hive > Issue Type: Task >Reporter: László Végh >Assignee: László Végh >Priority: Major > > Postgresql and Mysql JDBC drivers are returning -2 instead of the number of > affected rows in batch execution mode. Therefore the result policy check > needs to be removed for batch executions. > Fix Batch execution issues. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-28005) Remove upgrade-acid module
[ https://issues.apache.org/jira/browse/HIVE-28005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17807907#comment-17807907 ] Attila Turoczy commented on HIVE-28005: --- As I see it is accepted by the community > Remove upgrade-acid module > -- > > Key: HIVE-28005 > URL: https://issues.apache.org/jira/browse/HIVE-28005 > Project: Hive > Issue Type: Task >Reporter: Ayush Saxena >Priority: Major > > Remove upgrade acid module, since it is relevant only for version 2 & 3 > Dev List: > https://lists.apache.org/thread/39m8ccz2y8rhlz5pygowb3rk0sjk929c -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-27996) Revert HIVE-27406 & HIVE-27481
[
https://issues.apache.org/jira/browse/HIVE-27996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17807901#comment-17807901
]
Attila Turoczy commented on HIVE-27996:
---
Is this a blocker for Hive4?
> Revert HIVE-27406 & HIVE-27481
> --
>
> Key: HIVE-27996
> URL: https://issues.apache.org/jira/browse/HIVE-27996
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0-beta-1
>Reporter: László Végh
>Assignee: Laszlo Vegh
>Priority: Blocker
> Labels: pull-request-available
>
> Revert HIVE-27406 & HIVE-27481
>
> The introduced changes were causing DB incompatibility issues.
> {code}
> create table if not exists tab_acid (a int) partitioned by (p string) stored
> as orc TBLPROPERTIES ('transactional'='true');
> insert into tab_acid values(1,'foo'),(3,'bar');
> Caused by: MetaException(message:The update count was rejected in at least
> one of the result array. Rolling back.)
> at
> org.apache.hadoop.hive.metastore.txn.jdbc.MultiDataSourceJdbcResource.execute(MultiDataSourceJdbcResource.java:217)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.addDynamicPartitions(TxnHandler.java:876)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-27749) SchemaTool initSchema fails on Mariadb 10.2
[
https://issues.apache.org/jira/browse/HIVE-27749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17807900#comment-17807900
]
Attila Turoczy commented on HIVE-27749:
---
Guys do we want to fix it for 10.2? Is it not working with Maria 11? To be
honest 6 years old MariaDB version does not so excite me :)
Even if we decide to fix it, at least should not be a blocker for Hive4. Those
who wants to use 6 years old RDBMS has an own place in the purgatory :)
cc: [~ngangam] [~sbadhya] [~ayushtkn] [~dkuzmenko]
> SchemaTool initSchema fails on Mariadb 10.2
> ---
>
> Key: HIVE-27749
> URL: https://issues.apache.org/jira/browse/HIVE-27749
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 4.0.0-alpha-2, 4.0.0-beta-1
>Reporter: Stamatis Zampetakis
>Assignee: Sourabh Badhya
>Priority: Critical
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: mariadb-metastore-schema-tests.patch
>
>
> Schema initialization for 4.0.0-beta-1 fails when run on Mariadb 10.2.
> The problem is reproducible on current
> (e5a7ce2f091da1f8a324da6e489cda59b9e4bfc6) master by applying the
> [^mariadb-metastore-schema-tests.patch] and then running:
> {noformat}
> mvn test -Dtest=TestMariadb#install -Dtest.groups=""{noformat}
> The error is shown below:
> {noformat}
> 315/409 ALTER TABLE `NOTIFICATION_SEQUENCE` MODIFY COLUMN `NNI_ID`
> BIGINT(20) GENERATED ALWAYS AS (1) STORED NOT NULL;
> Error: (conn=11) You have an error in your SQL syntax; check the manual that
> corresponds to your MariaDB server version for the right syntax to use near
> 'NOT NULL' at line 1 (state=42000,code=1064)
> Aborting command set because "force" is false and command failed: "ALTER
> TABLE `NOTIFICATION_SEQUENCE` MODIFY COLUMN `NNI_ID` BIGINT(20) GENERATED
> ALWAYS AS (1) STORED NOT NULL;"
> [ERROR] 2023-09-27 21:36:30.317 [main] MetastoreSchemaTool - Schema
> initialization FAILED! Metastore state would be inconsistent!
> Schema initialization FAILED! Metastore state would be inconsistent!
> [ERROR] 2023-09-27 21:36:30.317 [main] MetastoreSchemaTool - Underlying
> cause: java.io.IOException : Schema script failed, errorcode OTHER
> Underlying cause: java.io.IOException : Schema script failed, errorcode OTHER
> org.apache.hadoop.hive.metastore.HiveMetaException: Schema initialization
> FAILED! Metastore state would be inconsistent!
> at
> org.apache.hadoop.hive.metastore.tools.schematool.SchemaToolTaskInit.execute(SchemaToolTaskInit.java:66)
> at
> org.apache.hadoop.hive.metastore.tools.schematool.MetastoreSchemaTool.run(MetastoreSchemaTool.java:480)
> at
> org.apache.hadoop.hive.metastore.tools.schematool.MetastoreSchemaTool.run(MetastoreSchemaTool.java:425)
> at
> org.apache.hadoop.hive.metastore.dbinstall.rules.DatabaseRule.installLatest(DatabaseRule.java:269)
> at
> org.apache.hadoop.hive.metastore.dbinstall.DbInstallBase.install(DbInstallBase.java:34)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner
[jira] [Assigned] (HIVE-27996) Revert HIVE-27406 & HIVE-27481
[
https://issues.apache.org/jira/browse/HIVE-27996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Turoczy reassigned HIVE-27996:
-
Assignee: Laszlo Vegh
> Revert HIVE-27406 & HIVE-27481
> --
>
> Key: HIVE-27996
> URL: https://issues.apache.org/jira/browse/HIVE-27996
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0-beta-1
>Reporter: László Végh
>Assignee: Laszlo Vegh
>Priority: Blocker
> Labels: pull-request-available
>
> Revert HIVE-27406 & HIVE-27481
>
> The introduced changes were causing DB incompatibility issues.
> {code}
> create table if not exists tab_acid (a int) partitioned by (p string) stored
> as orc TBLPROPERTIES ('transactional'='true');
> insert into tab_acid values(1,'foo'),(3,'bar');
> Caused by: MetaException(message:The update count was rejected in at least
> one of the result array. Rolling back.)
> at
> org.apache.hadoop.hive.metastore.txn.jdbc.MultiDataSourceJdbcResource.execute(MultiDataSourceJdbcResource.java:217)
> at
> org.apache.hadoop.hive.metastore.txn.TxnHandler.addDynamicPartitions(TxnHandler.java:876)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HIVE-28004) DELETE on ACID table failed with NoClassDefFoundError: com/sun/tools/javac/util/List
[
https://issues.apache.org/jira/browse/HIVE-28004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17807898#comment-17807898
]
Attila Turoczy commented on HIVE-28004:
---
[~lvegh] you still think this is a blocker for 4.0?
> DELETE on ACID table failed with NoClassDefFoundError:
> com/sun/tools/javac/util/List
>
>
> Key: HIVE-28004
> URL: https://issues.apache.org/jira/browse/HIVE-28004
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0-beta-1
>Reporter: Butao Zhang
>Assignee: László Végh
>Priority: Blocker
>
> I am not sure if it is a bug or usage question.
> Test on Hive master branch:
>
> {code:java}
> set hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> set hive.support.concurrency = true;
> create table testacid4(id int) stored as orc TBLPROPERTIES
> ('transactional'='true');
> delete from testacid4 where id=110;
> {code}
>
> *beeline console shows error：*
> {code:java}
> 0: jdbc:hive2://127.0.0.1:1/default> delete from testacid4 where id=110;
> INFO : Compiling
> command(queryId=hive_20240116180628_ec5ac4d8-473b-4b42-b0dd-eecebec71268):
> delete from testacid4 where id=110
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:row__id,
> type:struct, comment:null)],
> properties:null)
> INFO : Completed compiling
> command(queryId=hive_20240116180628_ec5ac4d8-473b-4b42-b0dd-eecebec71268);
> Time taken: 3.554 seconds
> INFO : Operation QUERY obtained 1 locks
> ERROR : FAILED: Hive Internal Error:
> org.apache.hadoop.hive.ql.lockmgr.LockException(org.apache.thrift.TApplicationException:
> Internal error processing get_latest_txnid_in_conflict)
> org.apache.hadoop.hive.ql.lockmgr.LockException:
> org.apache.thrift.TApplicationException: Internal error processing
> get_latest_txnid_in_conflict
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.getLatestTxnIdInConflict(DbTxnManager.java:1055)
> at
> org.apache.hadoop.hive.ql.DriverTxnHandler.isValidTxnListState(DriverTxnHandler.java:435)
> at org.apache.hadoop.hive.ql.Driver.validateTxnList(Driver.java:250)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:199)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:154)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:149)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:185)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$500(SQLOperation.java:90)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:336)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:356)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.thrift.TApplicationException: Internal error processing
> get_latest_txnid_in_conflict
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_latest_txnid_in_conflict(ThriftHiveMetastore.java:6404)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_latest_txnid_in_conflict(ThriftHiveMetastore.java:6391)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getLatestTxnIdInConflict(HiveMetaStoreClient.java:4421)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:213)
> at com.sun.proxy.$Proxy32.getLatestTxnIdInConflict(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Metho
[jira] [Commented] (HIVE-28001) Fix the flaky test TestLeaderElection
[ https://issues.apache.org/jira/browse/HIVE-28001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17807876#comment-17807876 ] Sai Hemanth Gantasala commented on HIVE-28001: -- [~dengzh] - I think it would be good to post the actual failure of the test for future reference in the Jira details. The links posted in the Jira can expire at any time. > Fix the flaky test TestLeaderElection > - > > Key: HIVE-28001 > URL: https://issues.apache.org/jira/browse/HIVE-28001 > Project: Hive > Issue Type: Test >Reporter: Zhihua Deng >Assignee: Zhihua Deng >Priority: Major > Labels: pull-request-available > > The TestLeaderElection is failing sometimes, example: > [http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/master/2032/tests] > [http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4995/3/tests] > [http://ci.hive.apache.org/job/hive-flaky-check/796/] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28008) ParquetFileReader is not closed in ParquetHiveSerDe.readSchema
[ https://issues.apache.org/jira/browse/HIVE-28008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-28008: -- Labels: pull-request-available (was: ) > ParquetFileReader is not closed in ParquetHiveSerDe.readSchema > -- > > Key: HIVE-28008 > URL: https://issues.apache.org/jira/browse/HIVE-28008 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0 >Reporter: Michal Lorek >Assignee: Michal Lorek >Priority: Minor > Labels: pull-request-available > > The readSchema method inside ParquetHiveSerDe does not close > ParquetFileReader. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28008) ParquetFileReader is not closed in ParquetHiveSerDe.readSchema
Michal Lorek created HIVE-28008: --- Summary: ParquetFileReader is not closed in ParquetHiveSerDe.readSchema Key: HIVE-28008 URL: https://issues.apache.org/jira/browse/HIVE-28008 Project: Hive Issue Type: Improvement Components: Hive Affects Versions: 4.0.0 Reporter: Michal Lorek Assignee: Michal Lorek The readSchema method inside ParquetHiveSerDe does not close ParquetFileReader. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28007) Java 11: Failure of TimeStamp tests
[ https://issues.apache.org/jira/browse/HIVE-28007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kokila N updated HIVE-28007: Description: Test cases are failing with value changes in timestamp. Check if its jdk behavioural change and solve. > Java 11: Failure of TimeStamp tests > > > Key: HIVE-28007 > URL: https://issues.apache.org/jira/browse/HIVE-28007 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Kokila N >Assignee: Kokila N >Priority: Major > > Test cases are failing with value changes in timestamp. Check if its jdk > behavioural change and solve. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (HIVE-28007) Java 11: Failure of TimeStamp tests
[ https://issues.apache.org/jira/browse/HIVE-28007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kokila N reassigned HIVE-28007: --- Assignee: Kokila N > Java 11: Failure of TimeStamp tests > > > Key: HIVE-28007 > URL: https://issues.apache.org/jira/browse/HIVE-28007 > Project: Hive > Issue Type: Sub-task > Components: Hive >Reporter: Kokila N >Assignee: Kokila N >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HIVE-28007) Java 11: Failure of TimeStamp tests
Kokila N created HIVE-28007: --- Summary: Java 11: Failure of TimeStamp tests Key: HIVE-28007 URL: https://issues.apache.org/jira/browse/HIVE-28007 Project: Hive Issue Type: Sub-task Components: Hive Reporter: Kokila N -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-28004) DELETE on ACID table failed with NoClassDefFoundError: com/sun/tools/javac/util/List
[
https://issues.apache.org/jira/browse/HIVE-28004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17807664#comment-17807664
]
László Végh commented on HIVE-28004:
[~zhangbutao] This will be addressed as a part of this PR:
[https://github.com/apache/hive/pull/5010]
There are multiple issues found with after merging
https://issues.apache.org/jira/browse/HIVE-27481
Unfortunately the unit and qtets tests are only running on derby HMS backend,
which did not reveal them. The addendum will try to cover all of these, by
running the qtests manually on a PostgreSQL backend and fix the upcoming issues.
> DELETE on ACID table failed with NoClassDefFoundError:
> com/sun/tools/javac/util/List
>
>
> Key: HIVE-28004
> URL: https://issues.apache.org/jira/browse/HIVE-28004
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0-beta-1
>Reporter: Butao Zhang
>Assignee: László Végh
>Priority: Blocker
>
> I am not sure if it is a bug or usage question.
> Test on Hive master branch:
>
> {code:java}
> set hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> set hive.support.concurrency = true;
> create table testacid4(id int) stored as orc TBLPROPERTIES
> ('transactional'='true');
> delete from testacid4 where id=110;
> {code}
>
> *beeline console shows error：*
> {code:java}
> 0: jdbc:hive2://127.0.0.1:1/default> delete from testacid4 where id=110;
> INFO : Compiling
> command(queryId=hive_20240116180628_ec5ac4d8-473b-4b42-b0dd-eecebec71268):
> delete from testacid4 where id=110
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:row__id,
> type:struct, comment:null)],
> properties:null)
> INFO : Completed compiling
> command(queryId=hive_20240116180628_ec5ac4d8-473b-4b42-b0dd-eecebec71268);
> Time taken: 3.554 seconds
> INFO : Operation QUERY obtained 1 locks
> ERROR : FAILED: Hive Internal Error:
> org.apache.hadoop.hive.ql.lockmgr.LockException(org.apache.thrift.TApplicationException:
> Internal error processing get_latest_txnid_in_conflict)
> org.apache.hadoop.hive.ql.lockmgr.LockException:
> org.apache.thrift.TApplicationException: Internal error processing
> get_latest_txnid_in_conflict
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.getLatestTxnIdInConflict(DbTxnManager.java:1055)
> at
> org.apache.hadoop.hive.ql.DriverTxnHandler.isValidTxnListState(DriverTxnHandler.java:435)
> at org.apache.hadoop.hive.ql.Driver.validateTxnList(Driver.java:250)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:199)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:154)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:149)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:185)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$500(SQLOperation.java:90)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:336)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:356)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.thrift.TApplicationException: Internal error processing
> get_latest_txnid_in_conflict
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_latest_txnid_in_conflict(ThriftHiveMetastore.java:6404)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_latest_txnid_in_conflict(ThriftHiveMetastore.java:6391)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getLatestTxnIdInConflict(HiveMetaStoreClient.java:4421)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMet
[jira] [Updated] (HIVE-28004) DELETE on ACID table failed with NoClassDefFoundError: com/sun/tools/javac/util/List
[
https://issues.apache.org/jira/browse/HIVE-28004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-28004:
--
Affects Version/s: 4.0.0-beta-1
> DELETE on ACID table failed with NoClassDefFoundError:
> com/sun/tools/javac/util/List
>
>
> Key: HIVE-28004
> URL: https://issues.apache.org/jira/browse/HIVE-28004
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0-beta-1
>Reporter: Butao Zhang
>Priority: Blocker
>
> I am not sure if it is a bug or usage question.
> Test on Hive master branch:
>
> {code:java}
> set hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> set hive.support.concurrency = true;
> create table testacid4(id int) stored as orc TBLPROPERTIES
> ('transactional'='true');
> delete from testacid4 where id=110;
> {code}
>
> *beeline console shows error：*
> {code:java}
> 0: jdbc:hive2://127.0.0.1:1/default> delete from testacid4 where id=110;
> INFO : Compiling
> command(queryId=hive_20240116180628_ec5ac4d8-473b-4b42-b0dd-eecebec71268):
> delete from testacid4 where id=110
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:row__id,
> type:struct, comment:null)],
> properties:null)
> INFO : Completed compiling
> command(queryId=hive_20240116180628_ec5ac4d8-473b-4b42-b0dd-eecebec71268);
> Time taken: 3.554 seconds
> INFO : Operation QUERY obtained 1 locks
> ERROR : FAILED: Hive Internal Error:
> org.apache.hadoop.hive.ql.lockmgr.LockException(org.apache.thrift.TApplicationException:
> Internal error processing get_latest_txnid_in_conflict)
> org.apache.hadoop.hive.ql.lockmgr.LockException:
> org.apache.thrift.TApplicationException: Internal error processing
> get_latest_txnid_in_conflict
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.getLatestTxnIdInConflict(DbTxnManager.java:1055)
> at
> org.apache.hadoop.hive.ql.DriverTxnHandler.isValidTxnListState(DriverTxnHandler.java:435)
> at org.apache.hadoop.hive.ql.Driver.validateTxnList(Driver.java:250)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:199)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:154)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:149)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:185)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$500(SQLOperation.java:90)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:336)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:356)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.thrift.TApplicationException: Internal error processing
> get_latest_txnid_in_conflict
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_latest_txnid_in_conflict(ThriftHiveMetastore.java:6404)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_latest_txnid_in_conflict(ThriftHiveMetastore.java:6391)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getLatestTxnIdInConflict(HiveMetaStoreClient.java:4421)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:213)
> at com.sun.proxy.$Proxy32.getLatestTxnIdInConflict(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAcces
[jira] [Updated] (HIVE-28004) DELETE on ACID table failed with NoClassDefFoundError: com/sun/tools/javac/util/List
[
https://issues.apache.org/jira/browse/HIVE-28004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-28004:
--
Component/s: Transactions
> DELETE on ACID table failed with NoClassDefFoundError:
> com/sun/tools/javac/util/List
>
>
> Key: HIVE-28004
> URL: https://issues.apache.org/jira/browse/HIVE-28004
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0-beta-1
>Reporter: Butao Zhang
>Priority: Blocker
>
> I am not sure if it is a bug or usage question.
> Test on Hive master branch:
>
> {code:java}
> set hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> set hive.support.concurrency = true;
> create table testacid4(id int) stored as orc TBLPROPERTIES
> ('transactional'='true');
> delete from testacid4 where id=110;
> {code}
>
> *beeline console shows error：*
> {code:java}
> 0: jdbc:hive2://127.0.0.1:1/default> delete from testacid4 where id=110;
> INFO : Compiling
> command(queryId=hive_20240116180628_ec5ac4d8-473b-4b42-b0dd-eecebec71268):
> delete from testacid4 where id=110
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:row__id,
> type:struct, comment:null)],
> properties:null)
> INFO : Completed compiling
> command(queryId=hive_20240116180628_ec5ac4d8-473b-4b42-b0dd-eecebec71268);
> Time taken: 3.554 seconds
> INFO : Operation QUERY obtained 1 locks
> ERROR : FAILED: Hive Internal Error:
> org.apache.hadoop.hive.ql.lockmgr.LockException(org.apache.thrift.TApplicationException:
> Internal error processing get_latest_txnid_in_conflict)
> org.apache.hadoop.hive.ql.lockmgr.LockException:
> org.apache.thrift.TApplicationException: Internal error processing
> get_latest_txnid_in_conflict
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.getLatestTxnIdInConflict(DbTxnManager.java:1055)
> at
> org.apache.hadoop.hive.ql.DriverTxnHandler.isValidTxnListState(DriverTxnHandler.java:435)
> at org.apache.hadoop.hive.ql.Driver.validateTxnList(Driver.java:250)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:199)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:154)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:149)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:185)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$500(SQLOperation.java:90)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:336)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:356)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.thrift.TApplicationException: Internal error processing
> get_latest_txnid_in_conflict
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_latest_txnid_in_conflict(ThriftHiveMetastore.java:6404)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_latest_txnid_in_conflict(ThriftHiveMetastore.java:6391)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getLatestTxnIdInConflict(HiveMetaStoreClient.java:4421)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:213)
> at com.sun.proxy.$Proxy32.getLatestTxnIdInConflict(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun
[jira] [Assigned] (HIVE-28004) DELETE on ACID table failed with NoClassDefFoundError: com/sun/tools/javac/util/List
[
https://issues.apache.org/jira/browse/HIVE-28004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko reassigned HIVE-28004:
-
Assignee: László Végh
> DELETE on ACID table failed with NoClassDefFoundError:
> com/sun/tools/javac/util/List
>
>
> Key: HIVE-28004
> URL: https://issues.apache.org/jira/browse/HIVE-28004
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0-beta-1
>Reporter: Butao Zhang
>Assignee: László Végh
>Priority: Blocker
>
> I am not sure if it is a bug or usage question.
> Test on Hive master branch:
>
> {code:java}
> set hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> set hive.support.concurrency = true;
> create table testacid4(id int) stored as orc TBLPROPERTIES
> ('transactional'='true');
> delete from testacid4 where id=110;
> {code}
>
> *beeline console shows error：*
> {code:java}
> 0: jdbc:hive2://127.0.0.1:1/default> delete from testacid4 where id=110;
> INFO : Compiling
> command(queryId=hive_20240116180628_ec5ac4d8-473b-4b42-b0dd-eecebec71268):
> delete from testacid4 where id=110
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:row__id,
> type:struct, comment:null)],
> properties:null)
> INFO : Completed compiling
> command(queryId=hive_20240116180628_ec5ac4d8-473b-4b42-b0dd-eecebec71268);
> Time taken: 3.554 seconds
> INFO : Operation QUERY obtained 1 locks
> ERROR : FAILED: Hive Internal Error:
> org.apache.hadoop.hive.ql.lockmgr.LockException(org.apache.thrift.TApplicationException:
> Internal error processing get_latest_txnid_in_conflict)
> org.apache.hadoop.hive.ql.lockmgr.LockException:
> org.apache.thrift.TApplicationException: Internal error processing
> get_latest_txnid_in_conflict
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.getLatestTxnIdInConflict(DbTxnManager.java:1055)
> at
> org.apache.hadoop.hive.ql.DriverTxnHandler.isValidTxnListState(DriverTxnHandler.java:435)
> at org.apache.hadoop.hive.ql.Driver.validateTxnList(Driver.java:250)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:199)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:154)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:149)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:185)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$500(SQLOperation.java:90)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:336)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:356)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.thrift.TApplicationException: Internal error processing
> get_latest_txnid_in_conflict
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_latest_txnid_in_conflict(ThriftHiveMetastore.java:6404)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_latest_txnid_in_conflict(ThriftHiveMetastore.java:6391)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getLatestTxnIdInConflict(HiveMetaStoreClient.java:4421)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:213)
> at com.sun.proxy.$Proxy32.getLatestTxnIdInConflict(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcc
[jira] [Updated] (HIVE-28004) DELETE on ACID table failed with NoClassDefFoundError: com/sun/tools/javac/util/List
[
https://issues.apache.org/jira/browse/HIVE-28004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-28004:
--
Priority: Blocker (was: Major)
> DELETE on ACID table failed with NoClassDefFoundError:
> com/sun/tools/javac/util/List
>
>
> Key: HIVE-28004
> URL: https://issues.apache.org/jira/browse/HIVE-28004
> Project: Hive
> Issue Type: Bug
>Reporter: Butao Zhang
>Priority: Blocker
>
> I am not sure if it is a bug or usage question.
> Test on Hive master branch:
>
> {code:java}
> set hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> set hive.support.concurrency = true;
> create table testacid4(id int) stored as orc TBLPROPERTIES
> ('transactional'='true');
> delete from testacid4 where id=110;
> {code}
>
> *beeline console shows error：*
> {code:java}
> 0: jdbc:hive2://127.0.0.1:1/default> delete from testacid4 where id=110;
> INFO : Compiling
> command(queryId=hive_20240116180628_ec5ac4d8-473b-4b42-b0dd-eecebec71268):
> delete from testacid4 where id=110
> INFO : Semantic Analysis Completed (retrial = false)
> INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:row__id,
> type:struct, comment:null)],
> properties:null)
> INFO : Completed compiling
> command(queryId=hive_20240116180628_ec5ac4d8-473b-4b42-b0dd-eecebec71268);
> Time taken: 3.554 seconds
> INFO : Operation QUERY obtained 1 locks
> ERROR : FAILED: Hive Internal Error:
> org.apache.hadoop.hive.ql.lockmgr.LockException(org.apache.thrift.TApplicationException:
> Internal error processing get_latest_txnid_in_conflict)
> org.apache.hadoop.hive.ql.lockmgr.LockException:
> org.apache.thrift.TApplicationException: Internal error processing
> get_latest_txnid_in_conflict
> at
> org.apache.hadoop.hive.ql.lockmgr.DbTxnManager.getLatestTxnIdInConflict(DbTxnManager.java:1055)
> at
> org.apache.hadoop.hive.ql.DriverTxnHandler.isValidTxnListState(DriverTxnHandler.java:435)
> at org.apache.hadoop.hive.ql.Driver.validateTxnList(Driver.java:250)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:199)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:154)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:149)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:185)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$500(SQLOperation.java:90)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:336)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
> at
> org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:356)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.thrift.TApplicationException: Internal error processing
> get_latest_txnid_in_conflict
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_latest_txnid_in_conflict(ThriftHiveMetastore.java:6404)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_latest_txnid_in_conflict(ThriftHiveMetastore.java:6391)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getLatestTxnIdInConflict(HiveMetaStoreClient.java:4421)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:213)
> at com.sun.proxy.$Proxy32.getLatestTxnIdInConflict(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
[jira] [Commented] (HIVE-21520) Query "Submit plan" time reported is incorrect
[
https://issues.apache.org/jira/browse/HIVE-21520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17807655#comment-17807655
]
László Bodor commented on HIVE-21520:
-
merged to master, thanks [~zhangbutao] for fixing this and [~rajesh.balamohan]
for the initial patch!
> Query "Submit plan" time reported is incorrect
> --
>
> Key: HIVE-21520
> URL: https://issues.apache.org/jira/browse/HIVE-21520
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Butao Zhang
>Priority: Trivial
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21520.1.patch
>
>
> Hive master branch + LLAP
> {noformat}
> Query Execution Summary
> --
> OPERATION DURATION
> --
> Compile Query 0.00s
> Prepare Plan 0.00s
> Get Query Coordinator (AM) 0.00s
> Submit Plan 1553658149.89s
> Start DAG 0.53s
> Run DAG 0.43s
> --
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-21520) Query "Submit plan" time reported is incorrect
[
https://issues.apache.org/jira/browse/HIVE-21520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-21520:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Query "Submit plan" time reported is incorrect
> --
>
> Key: HIVE-21520
> URL: https://issues.apache.org/jira/browse/HIVE-21520
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Butao Zhang
>Priority: Trivial
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21520.1.patch
>
>
> Hive master branch + LLAP
> {noformat}
> Query Execution Summary
> --
> OPERATION DURATION
> --
> Compile Query 0.00s
> Prepare Plan 0.00s
> Get Query Coordinator (AM) 0.00s
> Submit Plan 1553658149.89s
> Start DAG 0.53s
> Run DAG 0.43s
> --
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-21520) Query "Submit plan" time reported is incorrect
[
https://issues.apache.org/jira/browse/HIVE-21520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor updated HIVE-21520:
Fix Version/s: 4.0.0
> Query "Submit plan" time reported is incorrect
> --
>
> Key: HIVE-21520
> URL: https://issues.apache.org/jira/browse/HIVE-21520
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Butao Zhang
>Priority: Trivial
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-21520.1.patch
>
>
> Hive master branch + LLAP
> {noformat}
> Query Execution Summary
> --
> OPERATION DURATION
> --
> Compile Query 0.00s
> Prepare Plan 0.00s
> Get Query Coordinator (AM) 0.00s
> Submit Plan 1553658149.89s
> Start DAG 0.53s
> Run DAG 0.43s
> --
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-21520) Query "Submit plan" time reported is incorrect
[
https://issues.apache.org/jira/browse/HIVE-21520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
László Bodor reassigned HIVE-21520:
---
Assignee: Butao Zhang
> Query "Submit plan" time reported is incorrect
> --
>
> Key: HIVE-21520
> URL: https://issues.apache.org/jira/browse/HIVE-21520
> Project: Hive
> Issue Type: Bug
>Reporter: Rajesh Balamohan
>Assignee: Butao Zhang
>Priority: Trivial
> Labels: pull-request-available
> Attachments: HIVE-21520.1.patch
>
>
> Hive master branch + LLAP
> {noformat}
> Query Execution Summary
> --
> OPERATION DURATION
> --
> Compile Query 0.00s
> Prepare Plan 0.00s
> Get Query Coordinator (AM) 0.00s
> Submit Plan 1553658149.89s
> Start DAG 0.53s
> Run DAG 0.43s
> --
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HIVE-28006) Materialized view with aggregate function incorrectly shows it allows incremental rebuild
Krisztian Kasa created HIVE-28006:
-
Summary: Materialized view with aggregate function incorrectly
shows it allows incremental rebuild
Key: HIVE-28006
URL: https://issues.apache.org/jira/browse/HIVE-28006
Project: Hive
Issue Type: Bug
Components: Materialized views
Reporter: Krisztian Kasa
Assignee: Krisztian Kasa
{code}
set hive.support.concurrency=true;
set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
create table store_sales (
ss_sold_date_sk int,
ss_ext_sales_price int,
ss_customer_sk int
) stored as orc TBLPROPERTIES ('transactional'='true');
insert into store_sales (ss_sold_date_sk, ss_ext_sales_price, ss_customer_sk)
values (2, 2, 2);
create materialized view mat1 stored as orc tblproperties
('format-version'='2') as
select ss_customer_sk
,min(ss_ext_sales_price)
,count(*)
from store_sales
group by ss_customer_sk;
delete from store_sales where ss_sold_date_sk = 1;
show materialized views;
explain cbo
alter materialized view mat1 rebuild;
{code}
Incremental rebuild is available
{code}
# MV Name Rewriting Enabled Mode
Incremental rebuild
mat1Yes Manual refresh
Available
{code}
vs full rebuild plan
{code}
CBO PLAN:
HiveAggregate(group=[{2}], agg#0=[min($1)], agg#1=[count()])
HiveTableScan(table=[[default, store_sales]], table:alias=[store_sales])
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HIVE-27999) Run Sonar analysis using Java 17
[
https://issues.apache.org/jira/browse/HIVE-27999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis resolved HIVE-27999.
Fix Version/s: 4.1.0
Resolution: Fixed
Fixed in
https://github.com/apache/hive/commit/1760304401ff65d2f1fd7276d8a2bc241babb18a.
Thanks for the PR [~wechar] and [~aturoczy] for the review!
> Run Sonar analysis using Java 17
>
>
> Key: HIVE-27999
> URL: https://issues.apache.org/jira/browse/HIVE-27999
> Project: Hive
> Issue Type: Bug
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.1.0
>
>
> http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4905/4/pipeline/596/
> CI failed by sonar plugin:
> {code:bash}
> [2024-01-15T03:58:10.100Z] [INFO] BUILD FAILURE
> [2024-01-15T03:58:10.100Z] [INFO]
>
> [2024-01-15T03:58:10.100Z] [INFO] Total time: 01:07 min
> [2024-01-15T03:58:10.100Z] [INFO] Finished at: 2024-01-15T03:58:09Z
> [2024-01-15T03:58:10.100Z] [INFO]
>
> [2024-01-15T03:58:10.100Z] [ERROR] Failed to execute goal
> org.sonarsource.scanner.maven:sonar-maven-plugin:3.9.1.2184:sonar
> (default-cli) on project hive:
> [2024-01-15T03:58:10.100Z] [ERROR]
> [2024-01-15T03:58:10.100Z] [ERROR] The version of Java (11.0.8) used to run
> this analysis is deprecated, and SonarCloud no longer supports it. Please
> upgrade to Java 17 or later.
> [2024-01-15T03:58:10.100Z] [ERROR] As a temporary measure, you can set the
> property 'sonar.scanner.force-deprecated-java-version' to 'true' to continue
> using Java 11.0.8
> [2024-01-15T03:58:10.100Z] [ERROR] This workaround will only be effective
> until January 28, 2024. After this date, all scans using the deprecated Java
> 11 will fail.
> [2024-01-15T03:58:10.100Z] [ERROR] -> [Help 1]
> [2024-01-15T03:58:10.100Z] [ERROR]
> [2024-01-15T03:58:10.100Z] [ERROR] To see the full stack trace of the errors,
> re-run Maven with the -e switch.
> [2024-01-15T03:58:10.100Z] [ERROR] Re-run Maven using the -X switch to enable
> full debug logging.
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27999) Run Sonar analysis using Java 17
[
https://issues.apache.org/jira/browse/HIVE-27999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stamatis Zampetakis updated HIVE-27999:
---
Summary: Run Sonar analysis using Java 17 (was: Upgrade to Java 17 to run
sonar analysis)
> Run Sonar analysis using Java 17
>
>
> Key: HIVE-27999
> URL: https://issues.apache.org/jira/browse/HIVE-27999
> Project: Hive
> Issue Type: Bug
>Reporter: Wechar
>Assignee: Wechar
>Priority: Major
> Labels: pull-request-available
>
> http://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-4905/4/pipeline/596/
> CI failed by sonar plugin:
> {code:bash}
> [2024-01-15T03:58:10.100Z] [INFO] BUILD FAILURE
> [2024-01-15T03:58:10.100Z] [INFO]
>
> [2024-01-15T03:58:10.100Z] [INFO] Total time: 01:07 min
> [2024-01-15T03:58:10.100Z] [INFO] Finished at: 2024-01-15T03:58:09Z
> [2024-01-15T03:58:10.100Z] [INFO]
>
> [2024-01-15T03:58:10.100Z] [ERROR] Failed to execute goal
> org.sonarsource.scanner.maven:sonar-maven-plugin:3.9.1.2184:sonar
> (default-cli) on project hive:
> [2024-01-15T03:58:10.100Z] [ERROR]
> [2024-01-15T03:58:10.100Z] [ERROR] The version of Java (11.0.8) used to run
> this analysis is deprecated, and SonarCloud no longer supports it. Please
> upgrade to Java 17 or later.
> [2024-01-15T03:58:10.100Z] [ERROR] As a temporary measure, you can set the
> property 'sonar.scanner.force-deprecated-java-version' to 'true' to continue
> using Java 11.0.8
> [2024-01-15T03:58:10.100Z] [ERROR] This workaround will only be effective
> until January 28, 2024. After this date, all scans using the deprecated Java
> 11 will fail.
> [2024-01-15T03:58:10.100Z] [ERROR] -> [Help 1]
> [2024-01-15T03:58:10.100Z] [ERROR]
> [2024-01-15T03:58:10.100Z] [ERROR] To see the full stack trace of the errors,
> re-run Maven with the -e switch.
> [2024-01-15T03:58:10.100Z] [ERROR] Re-run Maven using the -X switch to enable
> full debug logging.
> {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-27960) Invalid function error when using custom udaf
[
https://issues.apache.org/jira/browse/HIVE-27960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
gaoxiong updated HIVE-27960:
Description:
When a permanent udaf used before over() function, hive will throw invalid
function error.
-In HIVE-12719 , it fix this issue for hive 3, but it can't work in hive 2.-
This issue reproduce on master.
In hive 2, it should get FunctionInfo from FunctionRegistry before get
WindowFunctionInfo same to hive 3. Because it will register window function to
session. Then hive can get WindowFunctionInfo correctly.
err detail
register a permanent udaf:
{code:java}
create function row_number2 as
'org.apache.hadoop.hive.ql.udf.generic.GenericUDAFRowNumber'; {code}
execute query in a new cli session:
{code:java}
select row_number2() over();{code}
blew is error log:
{code:java}
FAILED: SemanticException Failed to breakup Windowing invocations into Groups.
At least 1 group must only depend on input columns. Also check for circular
dependencies.Underlying error: Invalid function
row_number22023-12-06T10:17:30,348 ERROR [0b7764ce-cde3-49c5-9d32-f96d61b20773
main] ql.Driver: FAILED: SemanticException Failed to breakup Windowing
invocations into Groups. At least 1 group must only depend on input columns.
Also check for circular dependencies.Underlying error: Invalid function
row_number2org.apache.hadoop.hive.ql.parse.SemanticException: Failed to breakup
Windowing invocations into Groups. At least 1 group must only depend on input
columns. Also check for circular dependencies.Underlying error: Invalid
function row_number2 at
org.apache.hadoop.hive.ql.parse.WindowingComponentizer.next(WindowingComponentizer.java:97)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genWindowingPlan(SemanticAnalyzer.java:13270)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:9685)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:9644)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10549)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:10427)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:11125)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:481)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11138)
at
org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:512) at
org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317) at
org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1457) at
org.apache.hadoop.hive.ql.Driver.run(Driver.java:1237) at
org.apache.hadoop.hive.ql.Driver.run(Driver.java:1227) at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:233) at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:184) at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:403) at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:336) at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:787) at
org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:759) at
org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:686) at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498) at
org.apache.hadoop.util.RunJar.run(RunJar.java:239) at
org.apache.hadoop.util.RunJar.main(RunJar.java:153) {code}
was:
When a permanent udaf used before over() function, hive will throw invalid
function error.
In HIVE-12719 , it fix this issue for hive 3, but it can't work in hive 2.
In hive 2, it should get FunctionInfo from FunctionRegistry before get
WindowFunctionInfo same to hive 3. Because it will register window function to
session. Then hive can get WindowFunctionInfo correctly.
err detail
register a permanent udaf:
{code:java}
create function row_number2 as
'org.apache.hadoop.hive.ql.udf.generic.GenericUDAFRowNumber'; {code}
execute query in a new cli session:
{code:java}
select row_number2() over();{code}
blew is error log:
{code:java}
FAILED: SemanticException Failed to breakup Windowing invocations into Groups.
At least 1 group must only depend on input columns. Also check for circular
dependencies.Underlying error: Invalid f
