Re: Review Request 72470: ACID: Concurrent MERGE INSERT operations produce duplicates

[Peter Varga via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72470/#review220620
---



Looks good, just one question.


ql/src/java/org/apache/hadoop/hive/ql/Context.java
Lines 1198 (patched)


Do you need a separate field for that, you could just check the operation 
type?


- Peter Varga


On May 4, 2020, 4:33 p.m., Denys Kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72470/
> ---
> 
> (Updated May 4, 2020, 4:33 p.m.)
> 
> 
> Review request for hive, Marton Bod, Peter Varga, and Peter Vary.
> 
> 
> Bugs: HIVE-23349
> https://issues.apache.org/jira/browse/HIVE-23349
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> 2 concurrent MERGE INSERT operations generate duplicates due to lack of 
> locking.
> MERGE INSERT is treated as regular INSERT, it acquires SHARED_READ lock that 
> doesn't prevent other INSERTs. We should use EXCLUSIVE lock here or 
> EXCL_WRITE if hive.txn.write.xlock=false;
> 
> create table target (a int, b int) stored as orc TBLPROPERTIES 
> ('transactional'='true')");
> insert into target values (1,2), (3,4)
> create table source (a int, b int)
> execute in parallel:
> 
> insert into source values (5,6), (7,8)
> 
> PS:
> 
> Current patch doesn cover following scenario:
> 1) T1 (merge-insert) opens txns & records snapshot;
> 2) T2 (insert/merge-insert) opens txns, records snapshot & locks it;
> 3) T2 commits it's changes and unlocks T1; 
> 4) T1 locks snapshot and validates txn list, however only txns with txnId 
> lower than T1's is beeing considered, T2 changes are ignored -> duplicates 
> are generated.
> 
> 
> merge-insert/merge-insert scenario could be fixed by leveraging write-write 
> conflict check mechanism. We just need to set operation type for merge-insert 
> to update.
> However it won't solve issue with merge-insert/insert. 
> 
> We should consider moving locking before snapshot generation, current 
> snapshot re-check mechanism doesn't handle described scenarios.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/Context.java 9f59d4cea3 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java c1f94d165b 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 998c05e37d 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java deaab89c1f 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/MergeSemanticAnalyzer.java 
> 3ffdcec528 
>   
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/DbTxnManagerEndToEndTestBase.java
>  b435e79c3c 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> 1687425bcb 
>   ql/src/test/results/clientpositive/llap/explain_locks.q.out d62f6ccafd 
> 
> 
> Diff: https://reviews.apache.org/r/72470/diff/2/
> 
> 
> Testing
> ---
> 
> Manual, added number of merge related test scenarios into TestDbTxnManager2, 
> modified explain_locks.q
> 
> 
> Thanks,
> 
> Denys Kuzmenko
> 
>

Re: Review Request 72465: HIVE-23340 TxnHandler cleanup

[Peter Varga via Review Board]

> On May 4, 2020, 3 p.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
> > Lines 400 (patched)
> > 
> >
> > Is is a generic txn object (i.e TxnInfo)? I don't think TxnUtils is a 
> > good place for domain objects.

I didn't really know which way to go. The TxnInfo is missing the type field and 
I did not want to change the api.
The tables used by txnhandler don't have domain objects, should I create one in 
the model package for txns? I think that would be confusing too, since it would 
seem like the txns table is managed by jdo.
What do you suggest?


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72465/#review220593
---


On May 4, 2020, 1:22 p.m., Peter Varga wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72465/
> ---
> 
> (Updated May 4, 2020, 1:22 p.m.)
> 
> 
> Review request for hive and Denys Kuzmenko.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> * Merge getOpenTxns and getOpenTxnInfo to avoid code duplication
>   * Remove TxnStatus character constants and use the enum values
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
>  a1bc10955a 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  8fded608d0 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
>  4ee1a45aae 
> 
> 
> Diff: https://reviews.apache.org/r/72465/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Peter Varga
> 
>

Re: Review Request 72466: Move q tests to TestMiniLlapLocal from TestCliDriver where the output is different, batch 2

[Zoltan Haindrich]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72466/#review220619
---




ql/src/test/results/clientpositive/llap/join_map_ppr.q.out
Lines 277 (patched)


the hints suggest that table x and y should be using mapjoin - however an 
n-way mergejoin is happening



ql/src/test/results/clientpositive/llap/join_map_ppr.q.out
Line 1044 (original), 861 (patched)


same: mapjoin(x,y) but mergejoin



ql/src/test/results/clientpositive/llap/join_star.q.out
Line 491 (original)


seems like the order of the resultset have changed



ql/src/test/results/clientpositive/llap/keyword_1.q.out
Line 33 (original)


statistics disappeared



ql/src/test/results/clientpositive/llap/lateral_view_noalias.q.out
Line 19 (original)


stats missing



ql/src/test/results/clientpositive/llap/lateral_view_onview.q.out
Line 379 (original)


I wonder if stats are missing because of fetch.task.conversion was able to 
convert it


- Zoltan Haindrich


On May 4, 2020, 12:40 p.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72466/
> ---
> 
> (Updated May 4, 2020, 12:40 p.m.)
> 
> 
> Review request for hive, Jesús Camacho Rodríguez, John Sherman, Zoltan 
> Haindrich, Krisztian Kasa, Steve Carlin, and Vineet Garg.
> 
> 
> Bugs: HIVE-23337
> https://issues.apache.org/jira/browse/HIVE-23337
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Move q tests to TestMiniLlapLocal from TestCliDriver where the output is 
> different, batch 2
> 
> 
> Diffs
> -
> 
>   ql/src/test/results/clientpositive/llap/groupby9.q.out 8eaa2e9d1f 
>   ql/src/test/results/clientpositive/llap/groupby_complex_types.q.out 
> e784a5e04a 
>   
> ql/src/test/results/clientpositive/llap/groupby_complex_types_multi_single_reducer.q.out
>  dd2ea4a357 
>   ql/src/test/results/clientpositive/llap/groupby_cube1.q.out 0ac1490e34 
>   ql/src/test/results/clientpositive/llap/groupby_cube_multi_gby.q.out 
> af37eaca1a 
>   ql/src/test/results/clientpositive/llap/groupby_distinct_samekey.q.out 
> 901d6378ff 
>   ql/src/test/results/clientpositive/llap/groupby_duplicate_key.q.out 
> 44e8ef6952 
>   ql/src/test/results/clientpositive/llap/groupby_grouping_id3.q.out 
> cdc063b370 
>   ql/src/test/results/clientpositive/llap/groupby_grouping_sets1.q.out 
> 43ab99b9f1 
>   ql/src/test/results/clientpositive/llap/groupby_grouping_sets2.q.out 
> 7831a49e95 
>   ql/src/test/results/clientpositive/llap/groupby_grouping_sets3.q.out 
> a08dd02490 
>   ql/src/test/results/clientpositive/llap/groupby_grouping_sets4.q.out 
> b61aba926d 
>   ql/src/test/results/clientpositive/llap/groupby_grouping_sets5.q.out 
> b6b4dcb339 
>   ql/src/test/results/clientpositive/llap/groupby_grouping_sets6.q.out 
> f6571b4645 
>   
> ql/src/test/results/clientpositive/llap/groupby_grouping_sets_grouping.q.out 
> 93e081b729 
>   ql/src/test/results/clientpositive/llap/groupby_grouping_sets_limit.q.out 
> b4aa6d1dd0 
>   ql/src/test/results/clientpositive/llap/groupby_grouping_sets_view.q.out 
> 582b78092c 
>   ql/src/test/results/clientpositive/llap/groupby_grouping_window.q.out 
> 21d92567d5 
>   ql/src/test/results/clientpositive/llap/groupby_join_pushdown.q.out 
> 2138eae171 
>   ql/src/test/results/clientpositive/llap/groupby_map_ppr.q.out 952f310071 
>   
> ql/src/test/results/clientpositive/llap/groupby_map_ppr_multi_distinct.q.out 
> bd43f546dd 
>   
> ql/src/test/results/clientpositive/llap/groupby_multi_insert_common_distinct.q.out
>  991f343394 
>   ql/src/test/results/clientpositive/llap/groupby_multi_single_reducer.q.out 
> 756c179e8b 
>   ql/src/test/results/clientpositive/llap/groupby_multi_single_reducer2.q.out 
> d151470d6c 
>   ql/src/test/results/clientpositive/llap/groupby_multi_single_reducer3.q.out 
> 4b4d57f2a0 
>   ql/src/test/results/clientpositive/llap/groupby_multialias.q.out 1a42ff23a7 
>   ql/src/test/results/clientpositive/llap/groupby_nocolumnalign.q.out 
> 3a92e71a75 
>   ql/src/test/results/clientpositive/llap/groupby_position.q.out 17f02c9089 
>   ql/src/test/results/clientpositive/llap/groupby_ppd.q.out 5731e9d5c2 
>   ql/src/test/results/clientpositive/llap/groupby_ppr.q.out d7549d9536 
>   ql/src/test/results/clientpositive/llap/groupby_ppr_multi_distinct.q.out 
> 95f95b0613 
>   ql/src/test/results/clientpositive/llap/groupby_rollup1.q.out e7b61b4a33 
>   ql/src/test/results/clientpositive/ll

Re: Review Request 72469: HIVE-23325: Clean up cleanup tasks for TxnHandler/CompactionTxnHandler

[Marton Bod]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72469/
---

(Updated May 5, 2020, 7:42 a.m.)


Review request for hive, Denys Kuzmenko and Peter Vary.


Repository: hive-git


Description
---

Consolidate different metastore housekeeper threads into one, move cleaner 
methods out of compactor initiator. 
Create separate txn cleaner service which will need to run more frequently.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 829791e0a9 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 23512e252e 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 48bf8529fa 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands3.java 5b8c6701e1 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForMmTable.java 
eac2c6307f 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
1687425bcb 
  ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestInitiator.java 
e4ff14a140 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
 842b7fe53d 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 7bba8d6ee6 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MaterializationsRebuildLockCleanerTask.java
 d35c9602a6 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/AcidCompactionHistoryService.java
 e96a7ba289 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/AcidHouseKeeperService.java
 c4a488bac0 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/AcidOpenTxnsCounterService.java
 2ad5a89f03 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/AcidWriteSetService.java
 5ec513dfd4 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 8fded608d0 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java
 e8ac71ae55 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/conf/TestMetastoreConf.java
 9905a14983 
  
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/txn/TestAcidTxnCleanerService.java
 PRE-CREATION 


Diff: https://reviews.apache.org/r/72469/diff/3/

Changes: https://reviews.apache.org/r/72469/diff/2-3/


Testing
---


Thanks,

Marton Bod

Re: Review Request 72469: HIVE-23325: Clean up cleanup tasks for TxnHandler/CompactionTxnHandler

[Marton Bod]

> On May 4, 2020, 3:21 p.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/AcidCompactionHistoryService.java
> > Line 31 (original), 32 (patched)
> > 
> >
> > What does empty txn mean? Is it obsolete/abandoned/timed-out txns?
> 
> Marton Bod wrote:
> It refers to aborted/committed TXNs that do not have any entries in 
> TXN_COMPONENTS (hence the word empty).
> 
> Denys Kuzmenko wrote:
> could we name it TxnCleanerServic? EmptyTxn is a bit confusing.

Yeah, I was also a bit on the fence about the naming. Changed it.


- Marton


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72469/#review220594
---


On May 5, 2020, 7:42 a.m., Marton Bod wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72469/
> ---
> 
> (Updated May 5, 2020, 7:42 a.m.)
> 
> 
> Review request for hive, Denys Kuzmenko and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Consolidate different metastore housekeeper threads into one, move cleaner 
> methods out of compactor initiator. 
> Create separate txn cleaner service which will need to run more frequently.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 829791e0a9 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java 
> 23512e252e 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 48bf8529fa 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands3.java 5b8c6701e1 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommandsForMmTable.java 
> eac2c6307f 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> 1687425bcb 
>   ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/TestInitiator.java 
> e4ff14a140 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
>  842b7fe53d 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  7bba8d6ee6 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MaterializationsRebuildLockCleanerTask.java
>  d35c9602a6 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/AcidCompactionHistoryService.java
>  e96a7ba289 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/AcidHouseKeeperService.java
>  c4a488bac0 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/AcidOpenTxnsCounterService.java
>  2ad5a89f03 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/AcidWriteSetService.java
>  5ec513dfd4 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  8fded608d0 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java
>  e8ac71ae55 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/conf/TestMetastoreConf.java
>  9905a14983 
>   
> standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/txn/TestAcidTxnCleanerService.java
>  PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/72469/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Marton Bod
> 
>

Re: Review Request 72437: Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal

[Attila Magyar]

> On May 5, 2020, 5:04 a.m., Ashutosh Chauhan wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
> > Line 832 (original), 830 (patched)
> > 
> >
> > isView only used for this check here, which can be eliminated.

Not sure what you mean by eliminating it? Removing it altogeather?


- Attila


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72437/#review220616
---


On April 27, 2020, 9:15 a.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72437/
> ---
> 
> (Updated April 27, 2020, 9:15 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Rajesh Balamohan, and Vineet Garg.
> 
> 
> Bugs: HIVE-23282
> https://issues.apache.org/jira/browse/HIVE-23282
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> ObjectStore::getPartitionsByExprInternal internally uses Table information 
> for getting partitionKeys, table, catalog name.
> 
>  
> 
> For this, it ends up populating entire table data from DB (including skew 
> column, parameters, sort, bucket cols etc). This makes it a lot more 
> expensive call. It would be good to check if MTable itself can be used 
> instead of Table.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  4f58cd91efc 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  d1558876f14 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  53b7a67a429 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java
>  9834883f00f 
> 
> 
> Diff: https://reviews.apache.org/r/72437/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

[jira] [Created] (HIVE-23368) MV rebuild should produce the same view as the one configured at creation time

[Zoltan Haindrich (Jira)]

Zoltan Haindrich created HIVE-23368:
---

 Summary: MV rebuild should produce the same view as the one 
configured at creation time
 Key: HIVE-23368
 URL: https://issues.apache.org/jira/browse/HIVE-23368
 Project: Hive
  Issue Type: Sub-task
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


There might be some configrations which might affect the rel-tree of the 
materialized views.

In case rewrites to use datasketches for count(distinct) is enabled; the view 
should store sketches instead of distinct values



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23369) schq_ingest may run twice during a test execution

[Zoltan Haindrich (Jira)]

Zoltan Haindrich created HIVE-23369:
---

 Summary: schq_ingest may run twice during a test execution
 Key: HIVE-23369
 URL: https://issues.apache.org/jira/browse/HIVE-23369
 Project: Hive
  Issue Type: Sub-task
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: What's master key in hive metastore?

[Xi Chen]

Many thanks, Alan

On Tue, May 5, 2020 at 1:01 AM Alan Gates  wrote:

> It is used in storing delegation tokens, which HS2 and the megastore need
> to act on the user's behalf when reading files, launching jobs, etc.  See
> DelegationTokenStore.
>
> Alan.
>
> On Mon, May 4, 2020 at 2:00 AM Xi Chen  wrote:
>
> > Dear all,
> >
> > I find several APIs about master key in hive metastore and
> > IMetastoreClient. I searched around but didn't find any usage in hive DDL
> > tasks. Any idea where this master key is designed for?
> >
> > Thanks in advance,
> > bargitta
> >
> > int addMasterKey(String key) throws MetaException, TException;
> >
> > void updateMasterKey(Integer seqNo, String key)
> > throws NoSuchObjectException, MetaException, TException;
> >
> > boolean removeMasterKey(Integer keySeq) throws TException;
> >
> > String[] getMasterKeys() throws TException;
> >
>

Re: Review Request 72470: ACID: Concurrent MERGE INSERT operations produce duplicates

[Denys Kuzmenko via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72470/
---

(Updated May 5, 2020, 8:33 a.m.)


Review request for hive, Marton Bod, Peter Varga, and Peter Vary.


Changes
---

Removed isMerge property


Bugs: HIVE-23349
https://issues.apache.org/jira/browse/HIVE-23349


Repository: hive-git


Description
---

2 concurrent MERGE INSERT operations generate duplicates due to lack of locking.
MERGE INSERT is treated as regular INSERT, it acquires SHARED_READ lock that 
doesn't prevent other INSERTs. We should use EXCLUSIVE lock here or EXCL_WRITE 
if hive.txn.write.xlock=false;

create table target (a int, b int) stored as orc TBLPROPERTIES 
('transactional'='true')");
insert into target values (1,2), (3,4)
create table source (a int, b int)
execute in parallel:

insert into source values (5,6), (7,8)

PS:

Current patch doesn cover following scenario:
1) T1 (merge-insert) opens txns & records snapshot;
2) T2 (insert/merge-insert) opens txns, records snapshot & locks it;
3) T2 commits it's changes and unlocks T1; 
4) T1 locks snapshot and validates txn list, however only txns with txnId lower 
than T1's is beeing considered, T2 changes are ignored -> duplicates are 
generated.


merge-insert/merge-insert scenario could be fixed by leveraging write-write 
conflict check mechanism. We just need to set operation type for merge-insert 
to update.
However it won't solve issue with merge-insert/insert. 

We should consider moving locking before snapshot generation, current snapshot 
re-check mechanism doesn't handle described scenarios.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/Context.java 9f59d4cea3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java c1f94d165b 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 998c05e37d 
  ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java deaab89c1f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/MergeSemanticAnalyzer.java 
3ffdcec528 
  
ql/src/test/org/apache/hadoop/hive/ql/lockmgr/DbTxnManagerEndToEndTestBase.java 
b435e79c3c 
  ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
1687425bcb 
  ql/src/test/results/clientpositive/llap/explain_locks.q.out d62f6ccafd 


Diff: https://reviews.apache.org/r/72470/diff/3/

Changes: https://reviews.apache.org/r/72470/diff/2-3/


Testing
---

Manual, added number of merge related test scenarios into TestDbTxnManager2, 
modified explain_locks.q


Thanks,

Denys Kuzmenko

Re: Review Request 72470: ACID: Concurrent MERGE INSERT operations produce duplicates

[Denys Kuzmenko via Review Board]

> On May 5, 2020, 7:20 a.m., Peter Varga wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/Context.java
> > Lines 1198 (patched)
> > 
> >
> > Do you need a separate field for that, you could just check the 
> > operation type?

I actually started with checking operation type and ended up with the new 
field. Reverted back to the original approach.


- Denys


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72470/#review220620
---


On May 5, 2020, 8:33 a.m., Denys Kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72470/
> ---
> 
> (Updated May 5, 2020, 8:33 a.m.)
> 
> 
> Review request for hive, Marton Bod, Peter Varga, and Peter Vary.
> 
> 
> Bugs: HIVE-23349
> https://issues.apache.org/jira/browse/HIVE-23349
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> 2 concurrent MERGE INSERT operations generate duplicates due to lack of 
> locking.
> MERGE INSERT is treated as regular INSERT, it acquires SHARED_READ lock that 
> doesn't prevent other INSERTs. We should use EXCLUSIVE lock here or 
> EXCL_WRITE if hive.txn.write.xlock=false;
> 
> create table target (a int, b int) stored as orc TBLPROPERTIES 
> ('transactional'='true')");
> insert into target values (1,2), (3,4)
> create table source (a int, b int)
> execute in parallel:
> 
> insert into source values (5,6), (7,8)
> 
> PS:
> 
> Current patch doesn cover following scenario:
> 1) T1 (merge-insert) opens txns & records snapshot;
> 2) T2 (insert/merge-insert) opens txns, records snapshot & locks it;
> 3) T2 commits it's changes and unlocks T1; 
> 4) T1 locks snapshot and validates txn list, however only txns with txnId 
> lower than T1's is beeing considered, T2 changes are ignored -> duplicates 
> are generated.
> 
> 
> merge-insert/merge-insert scenario could be fixed by leveraging write-write 
> conflict check mechanism. We just need to set operation type for merge-insert 
> to update.
> However it won't solve issue with merge-insert/insert. 
> 
> We should consider moving locking before snapshot generation, current 
> snapshot re-check mechanism doesn't handle described scenarios.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/Context.java 9f59d4cea3 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java c1f94d165b 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 998c05e37d 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java deaab89c1f 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/MergeSemanticAnalyzer.java 
> 3ffdcec528 
>   
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/DbTxnManagerEndToEndTestBase.java
>  b435e79c3c 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> 1687425bcb 
>   ql/src/test/results/clientpositive/llap/explain_locks.q.out d62f6ccafd 
> 
> 
> Diff: https://reviews.apache.org/r/72470/diff/3/
> 
> 
> Testing
> ---
> 
> Manual, added number of merge related test scenarios into TestDbTxnManager2, 
> modified explain_locks.q
> 
> 
> Thanks,
> 
> Denys Kuzmenko
> 
>

[jira] [Created] (HIVE-23370) stats_list_bucketing breaks sysdb output

[Zoltan Haindrich (Jira)]

Zoltan Haindrich created HIVE-23370:
---

 Summary: stats_list_bucketing breaks sysdb output
 Key: HIVE-23370
 URL: https://issues.apache.org/jira/browse/HIVE-23370
 Project: Hive
  Issue Type: Sub-task
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich


it seems like it leaves behind an SDS and probably a table

http://34.66.156.144:8080/job/hive-precommit/job/PR-7/35/testReport/org.apache.hadoop.hive.cli.split12/TestMiniLlapLocalCliDriver/Testing___split_10___Archive___testCliDriver_sysdb_/



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23371) StatsUtils.getConstValue may log misleading exception

[Zoltan Haindrich (Jira)]

Zoltan Haindrich created HIVE-23371:
---

 Summary: StatsUtils.getConstValue may log misleading exception
 Key: HIVE-23371
 URL: https://issues.apache.org/jira/browse/HIVE-23371
 Project: Hive
  Issue Type: Bug
Reporter: Zoltan Haindrich
Assignee: Zoltan Haindrich



it logs exceptions like:
{code}
2020-05-05T01:51:59,609 DEBUG [cba65ded-3bb5-4295-9b3f-cdf844a8faec main] 
stats.StatsUtils: Interpreting constant (2008-04-08)  resulted in exception
java.lang.NumberFormatException: For input string: "2008-04-08"
at 
sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043) 
~[?:1.8.0_252]
at sun.misc.FloatingDecimal.parseDouble(FloatingDecimal.java:110) 
~[?:1.8.0_252]
at java.lang.Double.parseDouble(Double.java:538) ~[?:1.8.0_252]
at 
org.apache.hadoop.hive.ql.stats.StatsUtils.getConstValue(StatsUtils.java:1653) 
[hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.stats.StatsUtils.buildColStatForConstant(StatsUtils.java:1632)
 [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
at 
org.apache.hadoop.hive.ql.stats.StatsUtils.getColStatisticsFromExpression(StatsUtils.java:1538)
 [hive-exec-4.0.0-SNAPSHOT.jar:4.0.0-SNAPSHOT]
{code}

however...these exceptions are not that important; might just draw attention 
away from important ones



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Review Request 72472: HIVE-23318: TxnHandler need not delete from MATERIALIZATION_REBUILD_LOCKS on need basis

[Marton Bod]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72472/
---

Review request for hive, Denys Kuzmenko and Peter Vary.


Repository: hive-git


Description
---

Introduces a new TxnType for materialization view rebuilds, and only issues 
rebuild lock delete calls during commit for that specific type


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 998c05e37d 
  ql/src/test/org/apache/hadoop/hive/ql/parse/TestParseUtils.java e91a7ed760 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TxnType.java
 349f37f914 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
 e4b0bc726c 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
 1a0fee319b 
  
standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
 e6224ec16f 
  standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
8462b3d7a3 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 8fded608d0 


Diff: https://reviews.apache.org/r/72472/diff/1/


Testing
---


Thanks,

Marton Bod

Re: Review Request 72470: ACID: Concurrent MERGE INSERT operations produce duplicates

[Peter Varga via Review Board]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72470/#review220631
---


Ship it!




Ship It!

- Peter Varga


On May 5, 2020, 8:33 a.m., Denys Kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72470/
> ---
> 
> (Updated May 5, 2020, 8:33 a.m.)
> 
> 
> Review request for hive, Marton Bod, Peter Varga, and Peter Vary.
> 
> 
> Bugs: HIVE-23349
> https://issues.apache.org/jira/browse/HIVE-23349
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> 2 concurrent MERGE INSERT operations generate duplicates due to lack of 
> locking.
> MERGE INSERT is treated as regular INSERT, it acquires SHARED_READ lock that 
> doesn't prevent other INSERTs. We should use EXCLUSIVE lock here or 
> EXCL_WRITE if hive.txn.write.xlock=false;
> 
> create table target (a int, b int) stored as orc TBLPROPERTIES 
> ('transactional'='true')");
> insert into target values (1,2), (3,4)
> create table source (a int, b int)
> execute in parallel:
> 
> insert into source values (5,6), (7,8)
> 
> PS:
> 
> Current patch doesn cover following scenario:
> 1) T1 (merge-insert) opens txns & records snapshot;
> 2) T2 (insert/merge-insert) opens txns, records snapshot & locks it;
> 3) T2 commits it's changes and unlocks T1; 
> 4) T1 locks snapshot and validates txn list, however only txns with txnId 
> lower than T1's is beeing considered, T2 changes are ignored -> duplicates 
> are generated.
> 
> 
> merge-insert/merge-insert scenario could be fixed by leveraging write-write 
> conflict check mechanism. We just need to set operation type for merge-insert 
> to update.
> However it won't solve issue with merge-insert/insert. 
> 
> We should consider moving locking before snapshot generation, current 
> snapshot re-check mechanism doesn't handle described scenarios.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/Context.java 9f59d4cea3 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java c1f94d165b 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 998c05e37d 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java deaab89c1f 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/MergeSemanticAnalyzer.java 
> 3ffdcec528 
>   
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/DbTxnManagerEndToEndTestBase.java
>  b435e79c3c 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> 1687425bcb 
>   ql/src/test/results/clientpositive/llap/explain_locks.q.out d62f6ccafd 
> 
> 
> Diff: https://reviews.apache.org/r/72470/diff/3/
> 
> 
> Testing
> ---
> 
> Manual, added number of merge related test scenarios into TestDbTxnManager2, 
> modified explain_locks.q
> 
> 
> Thanks,
> 
> Denys Kuzmenko
> 
>

Re: Review Request 72470: ACID: Concurrent MERGE INSERT operations produce duplicates

[Marton Bod]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72470/#review220632
---


Ship it!




Ship It!

- Marton Bod


On May 5, 2020, 8:33 a.m., Denys Kuzmenko wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72470/
> ---
> 
> (Updated May 5, 2020, 8:33 a.m.)
> 
> 
> Review request for hive, Marton Bod, Peter Varga, and Peter Vary.
> 
> 
> Bugs: HIVE-23349
> https://issues.apache.org/jira/browse/HIVE-23349
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> 2 concurrent MERGE INSERT operations generate duplicates due to lack of 
> locking.
> MERGE INSERT is treated as regular INSERT, it acquires SHARED_READ lock that 
> doesn't prevent other INSERTs. We should use EXCLUSIVE lock here or 
> EXCL_WRITE if hive.txn.write.xlock=false;
> 
> create table target (a int, b int) stored as orc TBLPROPERTIES 
> ('transactional'='true')");
> insert into target values (1,2), (3,4)
> create table source (a int, b int)
> execute in parallel:
> 
> insert into source values (5,6), (7,8)
> 
> PS:
> 
> Current patch doesn cover following scenario:
> 1) T1 (merge-insert) opens txns & records snapshot;
> 2) T2 (insert/merge-insert) opens txns, records snapshot & locks it;
> 3) T2 commits it's changes and unlocks T1; 
> 4) T1 locks snapshot and validates txn list, however only txns with txnId 
> lower than T1's is beeing considered, T2 changes are ignored -> duplicates 
> are generated.
> 
> 
> merge-insert/merge-insert scenario could be fixed by leveraging write-write 
> conflict check mechanism. We just need to set operation type for merge-insert 
> to update.
> However it won't solve issue with merge-insert/insert. 
> 
> We should consider moving locking before snapshot generation, current 
> snapshot re-check mechanism doesn't handle described scenarios.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/Context.java 9f59d4cea3 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExplainTask.java c1f94d165b 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 998c05e37d 
>   ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java deaab89c1f 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/MergeSemanticAnalyzer.java 
> 3ffdcec528 
>   
> ql/src/test/org/apache/hadoop/hive/ql/lockmgr/DbTxnManagerEndToEndTestBase.java
>  b435e79c3c 
>   ql/src/test/org/apache/hadoop/hive/ql/lockmgr/TestDbTxnManager2.java 
> 1687425bcb 
>   ql/src/test/results/clientpositive/llap/explain_locks.q.out d62f6ccafd 
> 
> 
> Diff: https://reviews.apache.org/r/72470/diff/3/
> 
> 
> Testing
> ---
> 
> Manual, added number of merge related test scenarios into TestDbTxnManager2, 
> modified explain_locks.q
> 
> 
> Thanks,
> 
> Denys Kuzmenko
> 
>

Re: Review Request 72466: Move q tests to TestMiniLlapLocal from TestCliDriver where the output is different, batch 2

[Miklos Gergely]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72466/
---

(Updated May 5, 2020, 10:09 a.m.)


Review request for hive, Jesús Camacho Rodríguez, John Sherman, Zoltan 
Haindrich, Krisztian Kasa, Steve Carlin, and Vineet Garg.


Bugs: HIVE-23337
https://issues.apache.org/jira/browse/HIVE-23337


Repository: hive-git


Description
---

Move q tests to TestMiniLlapLocal from TestCliDriver where the output is 
different, batch 2


Diffs (updated)
-

  ql/src/test/results/clientpositive/llap/groupby9.q.out 8eaa2e9d1f 
  ql/src/test/results/clientpositive/llap/groupby_complex_types.q.out 
e784a5e04a 
  
ql/src/test/results/clientpositive/llap/groupby_complex_types_multi_single_reducer.q.out
 dd2ea4a357 
  ql/src/test/results/clientpositive/llap/groupby_cube1.q.out 0ac1490e34 
  ql/src/test/results/clientpositive/llap/groupby_cube_multi_gby.q.out 
af37eaca1a 
  ql/src/test/results/clientpositive/llap/groupby_distinct_samekey.q.out 
901d6378ff 
  ql/src/test/results/clientpositive/llap/groupby_duplicate_key.q.out 
44e8ef6952 
  ql/src/test/results/clientpositive/llap/groupby_grouping_id3.q.out cdc063b370 
  ql/src/test/results/clientpositive/llap/groupby_grouping_sets1.q.out 
43ab99b9f1 
  ql/src/test/results/clientpositive/llap/groupby_grouping_sets2.q.out 
7831a49e95 
  ql/src/test/results/clientpositive/llap/groupby_grouping_sets3.q.out 
a08dd02490 
  ql/src/test/results/clientpositive/llap/groupby_grouping_sets4.q.out 
b61aba926d 
  ql/src/test/results/clientpositive/llap/groupby_grouping_sets5.q.out 
b6b4dcb339 
  ql/src/test/results/clientpositive/llap/groupby_grouping_sets6.q.out 
f6571b4645 
  ql/src/test/results/clientpositive/llap/groupby_grouping_sets_grouping.q.out 
93e081b729 
  ql/src/test/results/clientpositive/llap/groupby_grouping_sets_limit.q.out 
b4aa6d1dd0 
  ql/src/test/results/clientpositive/llap/groupby_grouping_sets_view.q.out 
582b78092c 
  ql/src/test/results/clientpositive/llap/groupby_grouping_window.q.out 
21d92567d5 
  ql/src/test/results/clientpositive/llap/groupby_join_pushdown.q.out 
2138eae171 
  ql/src/test/results/clientpositive/llap/groupby_map_ppr.q.out 621a80aae8 
  ql/src/test/results/clientpositive/llap/groupby_map_ppr_multi_distinct.q.out 
4db2458fb0 
  
ql/src/test/results/clientpositive/llap/groupby_multi_insert_common_distinct.q.out
 991f343394 
  ql/src/test/results/clientpositive/llap/groupby_multi_single_reducer.q.out 
756c179e8b 
  ql/src/test/results/clientpositive/llap/groupby_multi_single_reducer2.q.out 
d151470d6c 
  ql/src/test/results/clientpositive/llap/groupby_multi_single_reducer3.q.out 
4b4d57f2a0 
  ql/src/test/results/clientpositive/llap/groupby_multialias.q.out 1a42ff23a7 
  ql/src/test/results/clientpositive/llap/groupby_nocolumnalign.q.out 
3a92e71a75 
  ql/src/test/results/clientpositive/llap/groupby_position.q.out 17f02c9089 
  ql/src/test/results/clientpositive/llap/groupby_ppd.q.out 5731e9d5c2 
  ql/src/test/results/clientpositive/llap/groupby_ppr.q.out bb5e7e65fa 
  ql/src/test/results/clientpositive/llap/groupby_ppr_multi_distinct.q.out 
43f1ac82cd 
  ql/src/test/results/clientpositive/llap/groupby_rollup1.q.out e7b61b4a33 
  ql/src/test/results/clientpositive/llap/groupby_sort_10.q.out 570d3eeeaf 
  ql/src/test/results/clientpositive/llap/groupby_sort_11.q.out 76d3c7c51a 
  ql/src/test/results/clientpositive/llap/groupby_sort_1_23.q.out 8061ffad5e 
  ql/src/test/results/clientpositive/llap/groupby_sort_2.q.out a6b2403f47 
  ql/src/test/results/clientpositive/llap/groupby_sort_3.q.out e657a28396 
  ql/src/test/results/clientpositive/llap/groupby_sort_4.q.out cadc717f68 
  ql/src/test/results/clientpositive/llap/groupby_sort_5.q.out 90312062f9 
  ql/src/test/results/clientpositive/llap/groupby_sort_6.q.out 6bf1057799 
  ql/src/test/results/clientpositive/llap/groupby_sort_7.q.out a0a193d720 
  ql/src/test/results/clientpositive/llap/groupby_sort_8.q.out b5f581e6e6 
  ql/src/test/results/clientpositive/llap/groupby_sort_9.q.out 33e21a3e08 
  ql/src/test/results/clientpositive/llap/groupby_sort_skew_1_23.q.out 
0588f5fbe0 
  ql/src/test/results/clientpositive/llap/groupby_sort_test_1.q.out 405374af77 
  ql/src/test/results/clientpositive/llap/groupingset_high_columns.q.out 
a7c47cd453 
  ql/src/test/results/clientpositive/llap/hashjoin.q.out 27194bf841 
  ql/src/test/results/clientpositive/llap/having2.q.out 74bb312940 
  ql/src/test/results/clientpositive/llap/hll.q.out 9cb85bb61c 
  ql/src/test/results/clientpositive/llap/implicit_cast1.q.out 1b2d13ce34 
  ql/src/test/results/clientpositive/llap/implicit_cast_during_insert.q.out 
255823c08b 
  ql/src/test/results/clientpositive/llap/implicit_decimal.q.out 6d7b28a730 
  ql/src/test/results/clientpositive/llap/in_typecheck_char.q.out d2fcdf48f0 
  ql/src/test/results/clientpositive/llap/in_typecheck_mixed.q.out a7f89d1b65 
  ql/src

[jira] [Created] (HIVE-23372) Project not defined correctly after reordering a join ADDENDUM - fix sharedwork

[Krisztian Kasa (Jira)]

Krisztian Kasa created HIVE-23372:
-

 Summary: Project not defined correctly after reordering a join 
ADDENDUM - fix sharedwork
 Key: HIVE-23372
 URL: https://issues.apache.org/jira/browse/HIVE-23372
 Project: Hive
  Issue Type: Improvement
  Components: CBO
Reporter: Krisztian Kasa
Assignee: Krisztian Kasa
 Fix For: 4.0.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[VOTE] Should we release Hive Storage API 2.7.2-rc1?

[Jesus Camacho Rodriguez]

All,

I'd like to make a storage-api release with HIVE-22959
 and HIVE-23215 <
https://issues.apache.org/jira/browse/HIVE-23215> in it.

Should we release the following artifacts as Hive Storage API 2.7.2?

tar: http://home.apache.org/~jcamacho/hive-storage-2.7.2/
tag: https://github.com/apache/hive/releases/tag/storage-release-2.7.2-rc1
jiras: https://issues.apache.org/jira/projects/HIVE/versions/12347828

Thanks!

-Jesús

Re: Review Request 72437: Reduce number of DB calls in ObjectStore::getPartitionsByExprInternal

[Ashutosh Chauhan]

> On May 5, 2020, 5:04 a.m., Ashutosh Chauhan wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
> > Line 832 (original), 830 (patched)
> > 
> >
> > isView only used for this check here, which can be eliminated.
> 
> Attila Magyar wrote:
> Not sure what you mean by eliminating it? Removing it altogeather?

yes remove this if(), its only purpose is to throw exception. We can remove 
this to save RDBMS query for isViewTable()


- Ashutosh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72437/#review220616
---


On April 27, 2020, 9:15 a.m., Attila Magyar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72437/
> ---
> 
> (Updated April 27, 2020, 9:15 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan, Rajesh Balamohan, and Vineet Garg.
> 
> 
> Bugs: HIVE-23282
> https://issues.apache.org/jira/browse/HIVE-23282
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> ObjectStore::getPartitionsByExprInternal internally uses Table information 
> for getting partitionKeys, table, catalog name.
> 
>  
> 
> For this, it ends up populating entire table data from DB (including skew 
> column, parameters, sort, bucket cols etc). This makes it a lot more 
> expensive call. It would be good to check if MTable itself can be used 
> instead of Table.
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  4f58cd91efc 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
>  d1558876f14 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
>  53b7a67a429 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/parser/ExpressionTree.java
>  9834883f00f 
> 
> 
> Diff: https://reviews.apache.org/r/72437/diff/4/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Attila Magyar
> 
>

[jira] [Created] (HIVE-23373) Reusing of nonulls in the column vector is giving incorrect results in CastLongToDate

[Roohi Syeda (Jira)]

Roohi Syeda created HIVE-23373:
--

 Summary: Reusing of nonulls in the column vector is giving 
incorrect results in CastLongToDate
 Key: HIVE-23373
 URL: https://issues.apache.org/jira/browse/HIVE-23373
 Project: Hive
  Issue Type: Bug
Reporter: Roohi Syeda
Assignee: Roohi Syeda






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72472: HIVE-23318: TxnHandler need not delete from MATERIALIZATION_REBUILD_LOCKS on need basis

[Jesús Camacho Rodríguez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72472/#review220647
---




ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java
Line 3153 (original), 3153 (patched)


Unrelated to this patch but related to compilation latency. I am not sure 
if it is actually possible but would be great to find a way to avoid the AST 
traversal (assuming this method is executed for every incoming query), since it 
adds to compilation latency and can become noticible for large queries.



ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java
Lines 3165 (patched)


LGTM. Have you confirmed that you need the traversal and that 
TOK_ALTER_MATERIALIZED_VIEW_REBUILD is not the root (I am not sure about this)?

Also wanted to mention. MV rebuild is translated internally into INSERT 
OVERWRITE, INSERT, or MERGE statements, depending on the MV semantics. Setting 
this TxnType to MATER_VIEW_REBUILD will not have any kind of side effect on 
those operations. Is that correct?


- Jesús Camacho Rodríguez


On May 5, 2020, 8:58 a.m., Marton Bod wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72472/
> ---
> 
> (Updated May 5, 2020, 8:58 a.m.)
> 
> 
> Review request for hive, Denys Kuzmenko, Jesús Camacho Rodríguez, and Peter 
> Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Introduces a new TxnType for materialization view rebuilds, and only issues 
> rebuild lock delete calls during commit for that specific type
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java 998c05e37d 
>   ql/src/test/org/apache/hadoop/hive/ql/parse/TestParseUtils.java e91a7ed760 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/TxnType.java
>  349f37f914 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-php/metastore/Types.php
>  e4b0bc726c 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-py/hive_metastore/ttypes.py
>  1a0fee319b 
>   
> standalone-metastore/metastore-common/src/gen/thrift/gen-rb/hive_metastore_types.rb
>  e6224ec16f 
>   standalone-metastore/metastore-common/src/main/thrift/hive_metastore.thrift 
> 8462b3d7a3 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  8fded608d0 
> 
> 
> Diff: https://reviews.apache.org/r/72472/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Marton Bod
> 
>

Re: Request write access to the Hive wiki

[Jiajie Zhong]

Hi,

Did I sent to wrong mailing list?
Should I also sent to dev mailing list?


Best Wish
— Jiajie



> On Apr 28, 2020, at 12:40, Jiajie Zhong  wrote:
> 
> Ping again.
> Did someone could grant permission to me to change the HIVE wiki?
> 
> Best Wish
> — Jiajie
> 
> 
> 
>> On Apr 25, 2020, at 18:29, Jiajie Zhong > > wrote:
>> 
>> Hi folks,
>> 
>> I reading HIVE doc and find some death link to it,
>> I want to update it so request write access to the Hive wiki
>> My confluence name is
>> 
>> Jiajie Zhong
>> 
>> Could someone grant permission to me, Thx
>> 
>> 
>> Best Wish
>> — Jiajie
>> 
> 

[jira] [Created] (HIVE-23374) QueryDisplay must be threadsafe

[Zoltan Haindrich (Jira)]

Zoltan Haindrich created HIVE-23374:
---

 Summary: QueryDisplay must be threadsafe
 Key: HIVE-23374
 URL: https://issues.apache.org/jira/browse/HIVE-23374
 Project: Hive
  Issue Type: Sub-task
Reporter: Zoltan Haindrich


apparently right now it might bump into 
java.util.ConcurrentModificationException -s

http://34.66.156.144:8080//job/hive-precommit/job/PR-7/41//testReport/junit/org.apache.hive.jdbc/TestJdbcWithMiniLlapRow/Testing___split_23___Archive___testInvalidReferenceCountScenario/

{code}
org.apache.hive.service.cli.HiveSQLException
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:360)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:346)
at 
org.apache.hive.jdbc.HiveStatement.waitForOperationToComplete(HiveStatement.java:352)
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:239)
at 
org.apache.hive.jdbc.BaseJdbcWithMiniLlap.testInvalidReferenceCountScenario(BaseJdbcWithMiniLlap.java:396)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
Caused by: java.util.ConcurrentModificationException: null
at 
java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719)
at 
java.util.LinkedHashMap$LinkedValueIterator.next(LinkedHashMap.java:747)
at java.util.AbstractCollection.toArray(AbstractCollection.java:141)
at java.util.ArrayList.addAll(ArrayList.java:581)
at 
org.apache.hadoop.hive.ql.QueryDisplay.getTaskDisplays(QueryDisplay.java:273)
at 
org.apache.hive.service.cli.operation.SQLOperation.getTaskStatus(SQLOperation.java:494)
at 
org.apache.hive.service.cli.operation.Operation.getStatus(Operation.java:141)
at 
org.apache.hive.service.cli.CLIService.getOperationStatus(CLIService.java:478)
at 
org.apache.hive.service.cli.thrift.ThriftCLIService.GetOperationStatus(ThriftCLIService.java:697)
at 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetOperationStatus.getResult(TCLIService.java:1757)
at 
org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetOperationStatus.getResult(TCLIService.java:1742)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at 
org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at 
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)