date:20200514

Re: Review Request 72499: HIVE-23446:LLAP: Reduce IPC connection misses to AM for short queries

2020-05-14 Thread Rajesh Balamohan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72499/
---

(Updated May 14, 2020, 7:09 a.m.)


Review request for hive, Ashutosh Chauhan and Gopal V.


Changes
---

Addressing review comments. For some reason, earlier upload wasn't successful.


Bugs: HIVE-23446
https://issues.apache.org/jira/browse/HIVE-23446


Repository: hive-git


Description
---

Currently UGI pool is maintained at QueryInfo level. However, when short 
queries and lots of AMs are there, it ends missing IPC connection cache. Too 
many connections are are also established. Patch tries to avoid that by 
maintaining this at ContainerRunner level. It retains the current behaviour of 
having multiple connection to same AM (otherwise can get bottlenecked on single 
connection)


Diffs (updated)
-

  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 6a13b55e69 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryInfo.java 
00fed15d2b 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
eae8e08540 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
 50dec4759e 


Diff: https://reviews.apache.org/r/72499/diff/2/

Changes: https://reviews.apache.org/r/72499/diff/1-2/


Testing
---


Thanks,

Rajesh Balamohan

Re: Review Request 72499: HIVE-23446:LLAP: Reduce IPC connection misses to AM for short queries

2020-05-14 Thread Rajesh Balamohan



> On May 14, 2020, 4:31 a.m., Ashutosh Chauhan wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
> > Lines 646-651 (patched)
> > 
> >
> > Is this logic needed? You already have valueloader in get() which must 
> > return a ugi, so it cant be null.
> 
> Rajesh Balamohan wrote:
> Yes, value loader is for initial miss. This is to avoid single connection 
> becoming a contention for AM communication. 
> https://issues.apache.org/jira/browse/HIVE-16634
> 
> Ashutosh Chauhan wrote:
> Not sure I follow. Can you add comments in code to explain the need for 
> this?

Addded comment in recent upload of the patch. Earlier patch wasn't uploaded 
correctly.

Value loader would be returning the queue (not ugi directly). Queue can 
maintain set of connections to same AM. Depending on query pattern, we need 
multiple connections to AM (addressed in HIVE-16634). Actually we are retaining 
the same code here as it was in QueryInfo earlier.


> On May 14, 2020, 4:31 a.m., Ashutosh Chauhan wrote:
> > llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
> > Lines 663 (patched)
> > 
> >
> > if its null, then its programming error. Better to not do this null 
> > check and offer without checking for null.
> 
> Ashutosh Chauhan wrote:
> better to throw NPE then to leak ugi failing to return to pool.

This is not a leak. It is gc-able ugi in case someone returns the ugi after 
expiry.


- Rajesh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72499/#review220748
---


On May 14, 2020, 7:14 a.m., Rajesh Balamohan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72499/
> ---
> 
> (Updated May 14, 2020, 7:14 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Gopal V.
> 
> 
> Bugs: HIVE-23446
> https://issues.apache.org/jira/browse/HIVE-23446
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Currently UGI pool is maintained at QueryInfo level. However, when short 
> queries and lots of AMs are there, it ends missing IPC connection cache. Too 
> many connections are are also established. Patch tries to avoid that by 
> maintaining this at ContainerRunner level. It retains the current behaviour 
> of having multiple connection to same AM (otherwise can get bottlenecked on 
> single connection)
> 
> 
> Diffs
> -
> 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
>  6a13b55e69 
>   llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryInfo.java 
> 00fed15d2b 
>   
> llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java
>  eae8e08540 
>   
> llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
>  50dec4759e 
> 
> 
> Diff: https://reviews.apache.org/r/72499/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Rajesh Balamohan
> 
>

Re: Review Request 72499: HIVE-23446:LLAP: Reduce IPC connection misses to AM for short queries

2020-05-14 Thread Rajesh Balamohan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72499/
---

(Updated May 14, 2020, 7:14 a.m.)


Review request for hive, Ashutosh Chauhan and Gopal V.


Bugs: HIVE-23446
https://issues.apache.org/jira/browse/HIVE-23446


Repository: hive-git


Description
---

Currently UGI pool is maintained at QueryInfo level. However, when short 
queries and lots of AMs are there, it ends missing IPC connection cache. Too 
many connections are are also established. Patch tries to avoid that by 
maintaining this at ContainerRunner level. It retains the current behaviour of 
having multiple connection to same AM (otherwise can get bottlenecked on single 
connection)


Diffs (updated)
-

  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java
 6a13b55e69 
  llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryInfo.java 
00fed15d2b 
  
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/QueryTracker.java 
eae8e08540 
  
llap-server/src/test/org/apache/hadoop/hive/llap/daemon/impl/TaskExecutorTestHelpers.java
 50dec4759e 


Diff: https://reviews.apache.org/r/72499/diff/3/

Changes: https://reviews.apache.org/r/72499/diff/2-3/


Testing
---


Thanks,

Rajesh Balamohan

[jira] [Created] (HIVE-23468) LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN

2020-05-14 Thread Rajesh Balamohan (Jira)

Rajesh Balamohan created HIVE-23468:
---

 Summary: LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN
 Key: HIVE-23468
 URL: https://issues.apache.org/jira/browse/HIVE-23468
 Project: Hive
  Issue Type: Improvement
Reporter: Rajesh Balamohan


OrcEncodedDataReader materializes the supplier to check if it is a HDFS 
system or not. This causes unwanted call to NN even in cases when cache is 
completely warmed up.

[https://github.com/apache/hive/blob/master/llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java#L540]

[https://github.com/apache/hive/blob/9f40d7cc1d889aa3079f3f494cf810fabe326e44/ql/src/java/org/apache/hadoop/hive/ql/io/HdfsUtils.java#L107]

Workaround is to set "hive.llap.io.use.fileid.path=false" to avoid this case.

IO elevator could get 100% cache hit from FileSystem impl in warmed up scenario.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23469) Use hostname + pod UID for shuffle manager caching

2020-05-14 Thread Attila Magyar (Jira)

Attila Magyar created HIVE-23469:


 Summary: Use hostname + pod UID for shuffle manager caching
 Key: HIVE-23469
 URL: https://issues.apache.org/jira/browse/HIVE-23469
 Project: Hive
  Issue Type: Bug
  Components: Tez
Reporter: Attila Magyar
Assignee: Attila Magyar


When a pod restarts, it uses the same hostname and shuffle port. Now when 
fetcher threads connects to download the shuffle data it will use the cached 
connection info and since the pod has died it's shuffle data will also get 
cleaned up. When the pod restarts, it receives connection from clients to 
download specific shuffle data but the daemon will not have it because of the 
restart.

In ShuffleManager.java's knownSrcHosts the key should be updated to HostInfo 
which is a combination of host+port and the host's unique ID. The host host Id 
changes when a node is killed or restarted.

 

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23470) Move TestCliDriver tests to TestMiniTezCliDriver if they are failing with TestMiniLlapLocalCliDriver

2020-05-14 Thread Miklos Gergely (Jira)

Miklos Gergely created HIVE-23470:
-

 Summary: Move TestCliDriver tests to TestMiniTezCliDriver if they 
are failing with TestMiniLlapLocalCliDriver
 Key: HIVE-23470
 URL: https://issues.apache.org/jira/browse/HIVE-23470
 Project: Hive
  Issue Type: Sub-task
Reporter: Miklos Gergely
Assignee: Miklos Gergely






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HIVE-23471) Statement.executeUpdate() does not return correct affected records causing "No such lock"

2020-05-14 Thread Riju Trivedi (Jira)

Riju Trivedi created HIVE-23471:
---

 Summary: Statement.executeUpdate() does not return correct 
affected records causing "No such lock"
 Key: HIVE-23471
 URL: https://issues.apache.org/jira/browse/HIVE-23471
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 3.1.0
Reporter: Riju Trivedi
Assignee: Denys Kuzmenko


In TxnHandler.acquire() call , Statement.executeUpdate() does not return 
correct  number of records updated in HIVE_LOCKS table as requested number of 
locks.

This results in error "*Couldn't find a lock we just created! No such lock(s)*" 
as acquire is rolled back.
{code:java}
int rc = stmt.executeUpdate(s);
  if (rc < locksBeingChecked.size()) {
LOG.debug("Going to rollback acquire(Connection dbConn, Statement stmt, 
List locksBeingChecked)");
dbConn.rollback();
  /*select all locks for this ext ID and see which ones are missing*/
  StringBuilder sb = new StringBuilder("No such lock(s): (" + 
JavaUtils.lockIdToString(extLockId) + ":");
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72481: HIVE-23234: Optimize TxnHandler::allocateTableWriteIds

2020-05-14 Thread Denys Kuzmenko via Review Board


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72481/#review220758
---



LGTM, some minor comments


standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Line 2114 (original), 2105 (patched)


you can use txnToWriteIds.size() instead of counter



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
Line 4120 (original), 4114 (patched)


could we try not to place every method argument on a new line



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java
Line 49 (original), 49 (patched)


Could you please. remove checkLock here as well.


- Denys Kuzmenko


On May 7, 2020, 3:55 p.m., Marton Bod wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72481/
> ---
> 
> (Updated May 7, 2020, 3:55 p.m.)
> 
> 
> Review request for hive, Denys Kuzmenko and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Removed global mutex on writeId allocation, which means write ids can now be 
> allocated concurrently for different tables without blocking each other, 
> speeding up execution (perf test results below). Concurrent 
> allocateTableWriteIds() operations targeting the same table are still mutexed 
> by an S4U if the table is already present in next_write_id, otherwise a race 
> condition to insert the table into next_write_id is solved by retrying after 
> catching the duplicate key exception (the thread which commits later will be 
> the one to retry).
> 
> The situation is similar when allocateTableWriteIds() and 
> replTableWriteIdState() are running concurrently - if they target different 
> tables, they won't block each other anymore. If they target the same table, 
> and the table is already inserted into next_write_id, replTableWriteIdState() 
> returns early and allocateTableWriteIds() updates the next id. If the table 
> is not yet in next_write_id, they might attempt to insert the same row 
> concurrently, in which case who commits later will get a duplicate key 
> exception and retry the operation, just as above.
> 
> 
> Diffs
> -
> 
>   ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
> 868da0c7a0 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
>  d59f863b11 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  cf41ef8aaf 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java
>  1e177f4a7b 
> 
> 
> Diff: https://reviews.apache.org/r/72481/diff/1/
> 
> 
> Testing
> ---
> 
> Unit test in TestTxnHandler
> + Perf tests:
> dbTypesameTable variant  ms/op  error
> MYSQL FALSE original 46.93  3.041
> MYSQL FALSE patched  19.283 1.311
> MYSQL TRUE  original 50.185 3.595
> MYSQL TRUE  patched  32.254 2.164
> ORACLEFALSE original 57.609 4.461
> ORACLEFALSE patched  25.721 2.551
> ORACLETRUE  original 59.668 3.172
> ORACLETRUE  patched  39.061 2.548
> POSTGRES  FALSE original 39.364 2.94 
> POSTGRES  FALSE patched  18.518 1.038
> POSTGRES  TRUE  original 39.868 2.679
> POSTGRES  TRUE  patched  28.874 1.768
> SQLSERVER FALSE original 45.252 1.643
> SQLSERVER FALSE patched  24.583 1.529
> SQLSERVER TRUE  original 49.149 3.45 
> SQLSERVER TRUE  patched  32.918 1.654
> (sameTable=true means that all threads were trying to allocate ids for the 
> same db.table,
> false means they all targeted different tables)
> 
> 
> Thanks,
> 
> Marton Bod
> 
>

Re: Review Request 72481: HIVE-23234: Optimize TxnHandler::allocateTableWriteIds

2020-05-14 Thread Denys Kuzmenko via Review Board



> On May 8, 2020, 10:23 a.m., Peter Vary wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
> > Line 2067 (original), 2057 (patched)
> > 
> >
> > Why is this change?
> 
> Marton Bod wrote:
> this was causing a checkstyle issue (line lenght too long)

it doesn't look long, maybe you can remove some leading spaces


- Denys


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72481/#review220689
---


On May 7, 2020, 3:55 p.m., Marton Bod wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72481/
> ---
> 
> (Updated May 7, 2020, 3:55 p.m.)
> 
> 
> Review request for hive, Denys Kuzmenko and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Removed global mutex on writeId allocation, which means write ids can now be 
> allocated concurrently for different tables without blocking each other, 
> speeding up execution (perf test results below). Concurrent 
> allocateTableWriteIds() operations targeting the same table are still mutexed 
> by an S4U if the table is already present in next_write_id, otherwise a race 
> condition to insert the table into next_write_id is solved by retrying after 
> catching the duplicate key exception (the thread which commits later will be 
> the one to retry).
> 
> The situation is similar when allocateTableWriteIds() and 
> replTableWriteIdState() are running concurrently - if they target different 
> tables, they won't block each other anymore. If they target the same table, 
> and the table is already inserted into next_write_id, replTableWriteIdState() 
> returns early and allocateTableWriteIds() updates the next id. If the table 
> is not yet in next_write_id, they might attempt to insert the same row 
> concurrently, in which case who commits later will get a duplicate key 
> exception and retry the operation, just as above.
> 
> 
> Diffs
> -
> 
>   ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
> 868da0c7a0 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
>  d59f863b11 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  cf41ef8aaf 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java
>  1e177f4a7b 
> 
> 
> Diff: https://reviews.apache.org/r/72481/diff/1/
> 
> 
> Testing
> ---
> 
> Unit test in TestTxnHandler
> + Perf tests:
> dbTypesameTable variant  ms/op  error
> MYSQL FALSE original 46.93  3.041
> MYSQL FALSE patched  19.283 1.311
> MYSQL TRUE  original 50.185 3.595
> MYSQL TRUE  patched  32.254 2.164
> ORACLEFALSE original 57.609 4.461
> ORACLEFALSE patched  25.721 2.551
> ORACLETRUE  original 59.668 3.172
> ORACLETRUE  patched  39.061 2.548
> POSTGRES  FALSE original 39.364 2.94 
> POSTGRES  FALSE patched  18.518 1.038
> POSTGRES  TRUE  original 39.868 2.679
> POSTGRES  TRUE  patched  28.874 1.768
> SQLSERVER FALSE original 45.252 1.643
> SQLSERVER FALSE patched  24.583 1.529
> SQLSERVER TRUE  original 49.149 3.45 
> SQLSERVER TRUE  patched  32.918 1.654
> (sameTable=true means that all threads were trying to allocate ids for the 
> same db.table,
> false means they all targeted different tables)
> 
> 
> Thanks,
> 
> Marton Bod
> 
>

Re: Review Request 72481: HIVE-23234: Optimize TxnHandler::allocateTableWriteIds

2020-05-14 Thread Marton Bod


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72481/
---

(Updated May 14, 2020, 3:38 p.m.)


Review request for hive, Denys Kuzmenko and Peter Vary.


Repository: hive-git


Description
---

Removed global mutex on writeId allocation, which means write ids can now be 
allocated concurrently for different tables without blocking each other, 
speeding up execution (perf test results below). Concurrent 
allocateTableWriteIds() operations targeting the same table are still mutexed 
by an S4U if the table is already present in next_write_id, otherwise a race 
condition to insert the table into next_write_id is solved by retrying after 
catching the duplicate key exception (the thread which commits later will be 
the one to retry).

The situation is similar when allocateTableWriteIds() and 
replTableWriteIdState() are running concurrently - if they target different 
tables, they won't block each other anymore. If they target the same table, and 
the table is already inserted into next_write_id, replTableWriteIdState() 
returns early and allocateTableWriteIds() updates the next id. If the table is 
not yet in next_write_id, they might attempt to insert the same row 
concurrently, in which case who commits later will get a duplicate key 
exception and retry the operation, just as above.


Diffs (updated)
-

  ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
868da0c7a0 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
 d59f863b11 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
 cf41ef8aaf 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java
 1e177f4a7b 


Diff: https://reviews.apache.org/r/72481/diff/2/

Changes: https://reviews.apache.org/r/72481/diff/1-2/


Testing
---

Unit test in TestTxnHandler
+ Perf tests:
dbTypesameTable variant  ms/op  error
MYSQL FALSE original 46.93  3.041
MYSQL FALSE patched  19.283 1.311
MYSQL TRUE  original 50.185 3.595
MYSQL TRUE  patched  32.254 2.164
ORACLEFALSE original 57.609 4.461
ORACLEFALSE patched  25.721 2.551
ORACLETRUE  original 59.668 3.172
ORACLETRUE  patched  39.061 2.548
POSTGRES  FALSE original 39.364 2.94 
POSTGRES  FALSE patched  18.518 1.038
POSTGRES  TRUE  original 39.868 2.679
POSTGRES  TRUE  patched  28.874 1.768
SQLSERVER FALSE original 45.252 1.643
SQLSERVER FALSE patched  24.583 1.529
SQLSERVER TRUE  original 49.149 3.45 
SQLSERVER TRUE  patched  32.918 1.654
(sameTable=true means that all threads were trying to allocate ids for the same 
db.table,
false means they all targeted different tables)


Thanks,

Marton Bod

Re: Review Request 72481: HIVE-23234: Optimize TxnHandler::allocateTableWriteIds

2020-05-14 Thread Marton Bod



> On May 14, 2020, 3:07 p.m., Denys Kuzmenko wrote:
> > LGTM, some minor comments

Thanks Denys, I've address your comments


- Marton


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72481/#review220758
---


On May 14, 2020, 3:38 p.m., Marton Bod wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72481/
> ---
> 
> (Updated May 14, 2020, 3:38 p.m.)
> 
> 
> Review request for hive, Denys Kuzmenko and Peter Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Removed global mutex on writeId allocation, which means write ids can now be 
> allocated concurrently for different tables without blocking each other, 
> speeding up execution (perf test results below). Concurrent 
> allocateTableWriteIds() operations targeting the same table are still mutexed 
> by an S4U if the table is already present in next_write_id, otherwise a race 
> condition to insert the table into next_write_id is solved by retrying after 
> catching the duplicate key exception (the thread which commits later will be 
> the one to retry).
> 
> The situation is similar when allocateTableWriteIds() and 
> replTableWriteIdState() are running concurrently - if they target different 
> tables, they won't block each other anymore. If they target the same table, 
> and the table is already inserted into next_write_id, replTableWriteIdState() 
> returns early and allocateTableWriteIds() updates the next id. If the table 
> is not yet in next_write_id, they might attempt to insert the same row 
> concurrently, in which case who commits later will get a duplicate key 
> exception and retry the operation, just as above.
> 
> 
> Diffs
> -
> 
>   ql/src/test/org/apache/hadoop/hive/metastore/txn/TestTxnHandler.java 
> 868da0c7a0 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
>  d59f863b11 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
>  cf41ef8aaf 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnStore.java
>  1e177f4a7b 
> 
> 
> Diff: https://reviews.apache.org/r/72481/diff/2/
> 
> 
> Testing
> ---
> 
> Unit test in TestTxnHandler
> + Perf tests:
> dbTypesameTable variant  ms/op  error
> MYSQL FALSE original 46.93  3.041
> MYSQL FALSE patched  19.283 1.311
> MYSQL TRUE  original 50.185 3.595
> MYSQL TRUE  patched  32.254 2.164
> ORACLEFALSE original 57.609 4.461
> ORACLEFALSE patched  25.721 2.551
> ORACLETRUE  original 59.668 3.172
> ORACLETRUE  patched  39.061 2.548
> POSTGRES  FALSE original 39.364 2.94 
> POSTGRES  FALSE patched  18.518 1.038
> POSTGRES  TRUE  original 39.868 2.679
> POSTGRES  TRUE  patched  28.874 1.768
> SQLSERVER FALSE original 45.252 1.643
> SQLSERVER FALSE patched  24.583 1.529
> SQLSERVER TRUE  original 49.149 3.45 
> SQLSERVER TRUE  patched  32.918 1.654
> (sameTable=true means that all threads were trying to allocate ids for the 
> same db.table,
> false means they all targeted different tables)
> 
> 
> Thanks,
> 
> Marton Bod
> 
>

Review Request 72510: Move TestCliDriver tests to TestMiniTezCliDriver if they are failing with TestMiniLlapLocalCliDriver

2020-05-14 Thread Miklos Gergely


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72510/
---

Review request for hive.


Bugs: HIVE-23470
https://issues.apache.org/jira/browse/HIVE-23470


Repository: hive-git


Description
---

Move TestCliDriver tests to TestMiniTezCliDriver if they are failing with 
TestMiniLlapLocalCliDriver


Diffs
-

  ql/src/test/results/clientpositive/tez/autoColumnStats_6.q.out ff708cb6b0 
  ql/src/test/results/clientpositive/tez/binary_output_format.q.out b414360855 
  ql/src/test/results/clientpositive/tez/create_genericudaf.q.out 85d7850888 
  ql/src/test/results/clientpositive/tez/create_udaf.q.out 7bfce125f0 
  ql/src/test/results/clientpositive/tez/create_view.q.out 9a251fcd2f 
  ql/src/test/results/clientpositive/tez/f_is_null.q.out e6862180b6 
  ql/src/test/results/clientpositive/tez/gen_udf_example_add10.q.out bfe313967b 
  ql/src/test/results/clientpositive/tez/groupby_bigdata.q.out 90ccc8cdfb 
  ql/src/test/results/clientpositive/tez/infer_bucket_sort.q.out bfdc84e24e 
  ql/src/test/results/clientpositive/tez/input14.q.out 0e61434791 
  ql/src/test/results/clientpositive/tez/input14_limit.q.out fe9d907663 
  ql/src/test/results/clientpositive/tez/input17.q.out 9c03f5b0af 
  ql/src/test/results/clientpositive/tez/input18.q.out ce731e6b2b 
  ql/src/test/results/clientpositive/tez/input20.q.out d90b9083c3 
  ql/src/test/results/clientpositive/tez/input33.q.out c8df2efede 
  ql/src/test/results/clientpositive/tez/input34.q.out 00dd35d803 
  ql/src/test/results/clientpositive/tez/input35.q.out cee491fc82 
  ql/src/test/results/clientpositive/tez/input36.q.out 45289b2143 
  ql/src/test/results/clientpositive/tez/input38.q.out d46ddf03ca 
  ql/src/test/results/clientpositive/tez/input5.q.out becfc1876a 
  ql/src/test/results/clientpositive/tez/insert_into3.q.out 60fd42d6fe 
  ql/src/test/results/clientpositive/tez/insert_into4.q.out 031d562a43 
  ql/src/test/results/clientpositive/tez/insert_into5.q.out 8ca94ee136 
  ql/src/test/results/clientpositive/tez/insert_into6.q.out 2c6cab53e6 
  ql/src/test/results/clientpositive/tez/load_binary_data.q.out b0d5c634b5 
  ql/src/test/results/clientpositive/tez/localtimezone.q.out 6f85d87c18 
  ql/src/test/results/clientpositive/tez/macro_1.q.out 28230f90e5 
  ql/src/test/results/clientpositive/tez/macro_duplicate.q.out 9598126c92 
  ql/src/test/results/clientpositive/tez/mapreduce3.q.out 9c0157c923 
  ql/src/test/results/clientpositive/tez/mapreduce4.q.out a606df0894 
  ql/src/test/results/clientpositive/tez/mapreduce7.q.out ab369e667b 
  ql/src/test/results/clientpositive/tez/mapreduce8.q.out d00ede826b 
  ql/src/test/results/clientpositive/tez/merge_test_dummy_operator.q.out 
31d4ae16f7 
  ql/src/test/results/clientpositive/tez/newline.q.out bea4e6ce1c 
  
ql/src/test/results/clientpositive/tez/nonreserved_keywords_insert_into1.q.out 
6435e8b5a3 
  ql/src/test/results/clientpositive/tez/nullscript.q.out cd926aa170 
  ql/src/test/results/clientpositive/tez/orc_createas1.q.out 6884e8654e 
  ql/src/test/results/clientpositive/tez/partcols1.q.out edd7db2357 
  ql/src/test/results/clientpositive/tez/partition_vs_table_metadata.q.out 
1b576ee10a 
  ql/src/test/results/clientpositive/tez/ppd_transform.q.out a38042c6fe 
  ql/src/test/results/clientpositive/tez/query_with_semi.q.out 93da006251 
  ql/src/test/results/clientpositive/tez/rcfile_bigdata.q.out c1ada45ad0 
  ql/src/test/results/clientpositive/tez/regexp_extract.q.out 95f7c22bc9 
  ql/src/test/results/clientpositive/tez/script_env_var1.q.out c1181b2635 
  ql/src/test/results/clientpositive/tez/script_env_var2.q.out 58a0936858 
  ql/src/test/results/clientpositive/tez/script_pipe.q.out f56107ebb1 
  ql/src/test/results/clientpositive/tez/scriptfile1.q.out 29ee2df4b3 
  ql/src/test/results/clientpositive/tez/select_transform_hint.q.out e0c1cea301 
  ql/src/test/results/clientpositive/tez/str_to_map.q.out f1aa4c14a0 
  ql/src/test/results/clientpositive/tez/temp_table_partcols1.q.out f2da1bf431 
  ql/src/test/results/clientpositive/tez/timestamptz_2.q.out 7f614c02f4 
  ql/src/test/results/clientpositive/tez/transform1.q.out d8d22443b8 
  ql/src/test/results/clientpositive/tez/transform2.q.out aeeaebf06b 
  ql/src/test/results/clientpositive/tez/transform3.q.out e3c55b8466 
  ql/src/test/results/clientpositive/tez/transform_ppr1.q.out 25468bcd9c 
  ql/src/test/results/clientpositive/tez/transform_ppr2.q.out 8aeb688513 
  
ql/src/test/results/clientpositive/tez/type_change_test_fraction_vectorized.q.out
 955d268fa5 
  ql/src/test/results/clientpositive/tez/type_change_test_int_vectorized.q.out 
87a5b3dd7f 
  ql/src/test/results/clientpositive/tez/typechangetest.q.out 4a617d70b4 
  ql/src/test/results/clientpositive/tez/udaf_sum_list.q.out 8afb053261 
  ql/src/test/results/clientpositive/tez/udf_printf.q.out 9809f9567d 
  ql/src/test/results/clientpositive

Re: Review Request 72510: Move TestCliDriver tests to TestMiniTezCliDriver if they are failing with TestMiniLlapLocalCliDriver

2020-05-14 Thread Miklos Gergely


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72510/#review220763
---




ql/src/test/results/clientpositive/tez/infer_bucket_sort.q.out
Line 53 (original), 53 (patched)


Bucket number and bucket column change



ql/src/test/results/clientpositive/tez/select_transform_hint.q.out
Line 102 (original), 65 (patched)


Result set change, most likely just the order.


- Miklos Gergely


On May 14, 2020, 4:18 p.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72510/
> ---
> 
> (Updated May 14, 2020, 4:18 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-23470
> https://issues.apache.org/jira/browse/HIVE-23470
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Move TestCliDriver tests to TestMiniTezCliDriver if they are failing with 
> TestMiniLlapLocalCliDriver
> 
> 
> Diffs
> -
> 
>   ql/src/test/results/clientpositive/tez/autoColumnStats_6.q.out ff708cb6b0 
>   ql/src/test/results/clientpositive/tez/binary_output_format.q.out 
> b414360855 
>   ql/src/test/results/clientpositive/tez/create_genericudaf.q.out 85d7850888 
>   ql/src/test/results/clientpositive/tez/create_udaf.q.out 7bfce125f0 
>   ql/src/test/results/clientpositive/tez/create_view.q.out 9a251fcd2f 
>   ql/src/test/results/clientpositive/tez/f_is_null.q.out e6862180b6 
>   ql/src/test/results/clientpositive/tez/gen_udf_example_add10.q.out 
> bfe313967b 
>   ql/src/test/results/clientpositive/tez/groupby_bigdata.q.out 90ccc8cdfb 
>   ql/src/test/results/clientpositive/tez/infer_bucket_sort.q.out bfdc84e24e 
>   ql/src/test/results/clientpositive/tez/input14.q.out 0e61434791 
>   ql/src/test/results/clientpositive/tez/input14_limit.q.out fe9d907663 
>   ql/src/test/results/clientpositive/tez/input17.q.out 9c03f5b0af 
>   ql/src/test/results/clientpositive/tez/input18.q.out ce731e6b2b 
>   ql/src/test/results/clientpositive/tez/input20.q.out d90b9083c3 
>   ql/src/test/results/clientpositive/tez/input33.q.out c8df2efede 
>   ql/src/test/results/clientpositive/tez/input34.q.out 00dd35d803 
>   ql/src/test/results/clientpositive/tez/input35.q.out cee491fc82 
>   ql/src/test/results/clientpositive/tez/input36.q.out 45289b2143 
>   ql/src/test/results/clientpositive/tez/input38.q.out d46ddf03ca 
>   ql/src/test/results/clientpositive/tez/input5.q.out becfc1876a 
>   ql/src/test/results/clientpositive/tez/insert_into3.q.out 60fd42d6fe 
>   ql/src/test/results/clientpositive/tez/insert_into4.q.out 031d562a43 
>   ql/src/test/results/clientpositive/tez/insert_into5.q.out 8ca94ee136 
>   ql/src/test/results/clientpositive/tez/insert_into6.q.out 2c6cab53e6 
>   ql/src/test/results/clientpositive/tez/load_binary_data.q.out b0d5c634b5 
>   ql/src/test/results/clientpositive/tez/localtimezone.q.out 6f85d87c18 
>   ql/src/test/results/clientpositive/tez/macro_1.q.out 28230f90e5 
>   ql/src/test/results/clientpositive/tez/macro_duplicate.q.out 9598126c92 
>   ql/src/test/results/clientpositive/tez/mapreduce3.q.out 9c0157c923 
>   ql/src/test/results/clientpositive/tez/mapreduce4.q.out a606df0894 
>   ql/src/test/results/clientpositive/tez/mapreduce7.q.out ab369e667b 
>   ql/src/test/results/clientpositive/tez/mapreduce8.q.out d00ede826b 
>   ql/src/test/results/clientpositive/tez/merge_test_dummy_operator.q.out 
> 31d4ae16f7 
>   ql/src/test/results/clientpositive/tez/newline.q.out bea4e6ce1c 
>   
> ql/src/test/results/clientpositive/tez/nonreserved_keywords_insert_into1.q.out
>  6435e8b5a3 
>   ql/src/test/results/clientpositive/tez/nullscript.q.out cd926aa170 
>   ql/src/test/results/clientpositive/tez/orc_createas1.q.out 6884e8654e 
>   ql/src/test/results/clientpositive/tez/partcols1.q.out edd7db2357 
>   ql/src/test/results/clientpositive/tez/partition_vs_table_metadata.q.out 
> 1b576ee10a 
>   ql/src/test/results/clientpositive/tez/ppd_transform.q.out a38042c6fe 
>   ql/src/test/results/clientpositive/tez/query_with_semi.q.out 93da006251 
>   ql/src/test/results/clientpositive/tez/rcfile_bigdata.q.out c1ada45ad0 
>   ql/src/test/results/clientpositive/tez/regexp_extract.q.out 95f7c22bc9 
>   ql/src/test/results/clientpositive/tez/script_env_var1.q.out c1181b2635 
>   ql/src/test/results/clientpositive/tez/script_env_var2.q.out 58a0936858 
>   ql/src/test/results/clientpositive/tez/script_pipe.q.out f56107ebb1 
>   ql/src/test/results/clientpositive/tez/scriptfile1.q.out 29ee2df4b3 
>   ql/src/test/results/clientpositive/tez/select_transform_hint.q.out 
> e0c1cea301 
>   ql/src/test/results/clientpositive/tez/str_to_map.q.out f1aa4c14a0 
>   ql/src/test/results/clientpositive/t

[jira] [Created] (HIVE-23472) LLAP Guaranteed state update should trigger queue re-ordering

2020-05-14 Thread Prasanth Jayachandran (Jira)

Prasanth Jayachandran created HIVE-23472:


 Summary: LLAP Guaranteed state update should trigger queue 
re-ordering
 Key: HIVE-23472
 URL: https://issues.apache.org/jira/browse/HIVE-23472
 Project: Hive
  Issue Type: Bug
Reporter: Prasanth Jayachandran


This is follow up to HIVE-23443 to handle the guaranteed state update case.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72510: Move TestCliDriver tests to TestMiniTezCliDriver if they are failing with TestMiniLlapLocalCliDriver

2020-05-14 Thread Jesús Camacho Rodríguez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72510/#review220766
---




ql/src/test/results/clientpositive/tez/create_genericudaf.q.out
Line 10 (original), 10 (patched)


Why is `EXPLAIN CREATE TEMPORARY FUNCTION` producing JSON in this case?



ql/src/test/results/clientpositive/tez/create_udaf.q.out
Line 10 (original), 10 (patched)


Same as above.



ql/src/test/results/clientpositive/tez/create_view.q.out
Line 174 (original), 173 (patched)


Why is `EXPLAIN CREATE VIEW` producing JSON in this case?



ql/src/test/results/clientpositive/tez/infer_bucket_sort.q.out
Line 53 (original), 53 (patched)


Yeah... @Zoltan, what is the expectation here? Is this happening because 
there was a single bucket? Is behavior anyhow different in Tez vs MR after your 
patch went in?



ql/src/test/results/clientpositive/tez/input14_limit.q.out
Line 31 (original), 31 (patched)


You may create a follow-up for these too so we can keep track easily of any 
remaining CBO support issue.



ql/src/test/results/clientpositive/tez/insert_into3.q.out
Line 37 (original), 37 (patched)


Not sure if that was the case before too. It may be worth exploring (there 
are a few in this file).



ql/src/test/results/clientpositive/tez/str_to_map.q.out
Line 143 (original), 116 (patched)


Not optimized by CBO.


- Jesús Camacho Rodríguez


On May 14, 2020, 4:18 p.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72510/
> ---
> 
> (Updated May 14, 2020, 4:18 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-23470
> https://issues.apache.org/jira/browse/HIVE-23470
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Move TestCliDriver tests to TestMiniTezCliDriver if they are failing with 
> TestMiniLlapLocalCliDriver
> 
> 
> Diffs
> -
> 
>   ql/src/test/results/clientpositive/tez/autoColumnStats_6.q.out ff708cb6b0 
>   ql/src/test/results/clientpositive/tez/binary_output_format.q.out 
> b414360855 
>   ql/src/test/results/clientpositive/tez/create_genericudaf.q.out 85d7850888 
>   ql/src/test/results/clientpositive/tez/create_udaf.q.out 7bfce125f0 
>   ql/src/test/results/clientpositive/tez/create_view.q.out 9a251fcd2f 
>   ql/src/test/results/clientpositive/tez/f_is_null.q.out e6862180b6 
>   ql/src/test/results/clientpositive/tez/gen_udf_example_add10.q.out 
> bfe313967b 
>   ql/src/test/results/clientpositive/tez/groupby_bigdata.q.out 90ccc8cdfb 
>   ql/src/test/results/clientpositive/tez/infer_bucket_sort.q.out bfdc84e24e 
>   ql/src/test/results/clientpositive/tez/input14.q.out 0e61434791 
>   ql/src/test/results/clientpositive/tez/input14_limit.q.out fe9d907663 
>   ql/src/test/results/clientpositive/tez/input17.q.out 9c03f5b0af 
>   ql/src/test/results/clientpositive/tez/input18.q.out ce731e6b2b 
>   ql/src/test/results/clientpositive/tez/input20.q.out d90b9083c3 
>   ql/src/test/results/clientpositive/tez/input33.q.out c8df2efede 
>   ql/src/test/results/clientpositive/tez/input34.q.out 00dd35d803 
>   ql/src/test/results/clientpositive/tez/input35.q.out cee491fc82 
>   ql/src/test/results/clientpositive/tez/input36.q.out 45289b2143 
>   ql/src/test/results/clientpositive/tez/input38.q.out d46ddf03ca 
>   ql/src/test/results/clientpositive/tez/input5.q.out becfc1876a 
>   ql/src/test/results/clientpositive/tez/insert_into3.q.out 60fd42d6fe 
>   ql/src/test/results/clientpositive/tez/insert_into4.q.out 031d562a43 
>   ql/src/test/results/clientpositive/tez/insert_into5.q.out 8ca94ee136 
>   ql/src/test/results/clientpositive/tez/insert_into6.q.out 2c6cab53e6 
>   ql/src/test/results/clientpositive/tez/load_binary_data.q.out b0d5c634b5 
>   ql/src/test/results/clientpositive/tez/localtimezone.q.out 6f85d87c18 
>   ql/src/test/results/clientpositive/tez/macro_1.q.out 28230f90e5 
>   ql/src/test/results/clientpositive/tez/macro_duplicate.q.out 9598126c92 
>   ql/src/test/results/clientpositive/tez/mapreduce3.q.out 9c0157c923 
>   ql/src/test/results/clientpositive/tez/mapreduce4.q.out a606df0894 
>   ql/src/test/results/clientpositive/tez/mapreduce7.q.out ab369e667b 
>   ql/src/test/results/clientpositive/tez/mapreduce8.q.out d00ede826b 
>   ql/src/test/results/clientpositive/tez/merge_test_dummy_operator.q.out 
> 31d4ae16f7 
>   ql/src/test/results/clientpositive/tez/newline.q.out bea4e6

Re: Review Request 72462: MSCK REPAIR cannot discover partitions with upper case directory names

2020-05-14 Thread Sankar Hariappan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72462/#review220771
---




ql/src/test/queries/clientpositive/msck_repair_4.q
Lines 8 (patched)


Add a testcase with table path repairtable_n4 having upper case. You can 
achieve it by setting location in create table.



standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java
Lines 116 (patched)


Why do we need pathSet? Can we check if path == null instead?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
Line 1441 (original), 1441 (patched)


Even table name in directory can be any case. Should we use 
equalsIgnoresCase here?



standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
Line 1454 (original), 1456 (patched)


Need to check how Hive treats ptn='A' and ptn='a' as ptn keys are lowercase 
but values can be any case.


- Sankar Hariappan


On May 3, 2020, 5:26 a.m., Adesh Rao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72462/
> ---
> 
> (Updated May 3, 2020, 5:26 a.m.)
> 
> 
> Review request for hive and Sankar Hariappan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The fix converts partition keys to lowercase present in hdfs directory, but 
> store the hdfs directory as is for partition path.
> 
> 
> Diffs
> -
> 
>   ql/src/test/queries/clientpositive/msck_repair_4.q PRE-CREATION 
>   ql/src/test/results/clientpositive/msck_repair_4.q.out PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java
>  5287f47e21 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java
>  6f4400a8ef 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
>  f4e109d1b0 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
>  7c4e129738 
> 
> 
> Diff: https://reviews.apache.org/r/72462/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Adesh Rao
> 
>

Review Request 72514: Move TestCliDriver tests to TestMiniLlapCliDriver if they are failing with TestMiniLlapLocalCliDriver

2020-05-14 Thread Miklos Gergely


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72514/
---

Review request for hive.


Bugs: HIVE-23470
https://issues.apache.org/jira/browse/HIVE-23470


Repository: hive-git


Description
---

Move TestCliDriver tests to TestMiniLlapCliDriver if they are failing with 
TestMiniLlapLocalCliDriver


Diffs
-

  ql/src/test/results/clientpositive/llap/autoColumnStats_6.q.out ff708cb6b0 
  ql/src/test/results/clientpositive/llap/binary_output_format.q.out b414360855 
  ql/src/test/results/clientpositive/llap/create_genericudaf.q.out 85d7850888 
  ql/src/test/results/clientpositive/llap/create_udaf.q.out 7bfce125f0 
  ql/src/test/results/clientpositive/llap/create_view.q.out 9a251fcd2f 
  ql/src/test/results/clientpositive/llap/gen_udf_example_add10.q.out 
bfe313967b 
  ql/src/test/results/clientpositive/llap/groupby_bigdata.q.out 90ccc8cdfb 
  ql/src/test/results/clientpositive/llap/infer_bucket_sort.q.out bfdc84e24e 
  ql/src/test/results/clientpositive/llap/input14.q.out 0e61434791 
  ql/src/test/results/clientpositive/llap/input14_limit.q.out fe9d907663 
  ql/src/test/results/clientpositive/llap/input17.q.out 9c03f5b0af 
  ql/src/test/results/clientpositive/llap/input18.q.out ce731e6b2b 
  ql/src/test/results/clientpositive/llap/input20.q.out d90b9083c3 
  ql/src/test/results/clientpositive/llap/input33.q.out c8df2efede 
  ql/src/test/results/clientpositive/llap/input34.q.out 00dd35d803 
  ql/src/test/results/clientpositive/llap/input35.q.out cee491fc82 
  ql/src/test/results/clientpositive/llap/input36.q.out 45289b2143 
  ql/src/test/results/clientpositive/llap/input38.q.out d46ddf03ca 
  ql/src/test/results/clientpositive/llap/input5.q.out becfc1876a 
  ql/src/test/results/clientpositive/llap/insert_into3.q.out 60fd42d6fe 
  ql/src/test/results/clientpositive/llap/insert_into4.q.out 031d562a43 
  ql/src/test/results/clientpositive/llap/insert_into5.q.out 8ca94ee136 
  ql/src/test/results/clientpositive/llap/insert_into6.q.out 2c6cab53e6 
  ql/src/test/results/clientpositive/llap/load_binary_data.q.out b0d5c634b5 
  ql/src/test/results/clientpositive/llap/macro_1.q.out 28230f90e5 
  ql/src/test/results/clientpositive/llap/macro_duplicate.q.out 9598126c92 
  ql/src/test/results/clientpositive/llap/mapreduce3.q.out 9c0157c923 
  ql/src/test/results/clientpositive/llap/mapreduce4.q.out a606df0894 
  ql/src/test/results/clientpositive/llap/mapreduce7.q.out ab369e667b 
  ql/src/test/results/clientpositive/llap/mapreduce8.q.out d00ede826b 
  ql/src/test/results/clientpositive/llap/merge_test_dummy_operator.q.out 
31d4ae16f7 
  ql/src/test/results/clientpositive/llap/newline.q.out bea4e6ce1c 
  
ql/src/test/results/clientpositive/llap/nonreserved_keywords_insert_into1.q.out 
6435e8b5a3 
  ql/src/test/results/clientpositive/llap/nullscript.q.out cd926aa170 
  ql/src/test/results/clientpositive/llap/orc_createas1.q.out 6884e8654e 
  ql/src/test/results/clientpositive/llap/partcols1.q.out edd7db2357 
  ql/src/test/results/clientpositive/llap/ppd_transform.q.out a38042c6fe 
  ql/src/test/results/clientpositive/llap/query_with_semi.q.out 93da006251 
  ql/src/test/results/clientpositive/llap/rcfile_bigdata.q.out c1ada45ad0 
  ql/src/test/results/clientpositive/llap/regexp_extract.q.out 95f7c22bc9 
  ql/src/test/results/clientpositive/llap/script_env_var1.q.out c1181b2635 
  ql/src/test/results/clientpositive/llap/script_env_var2.q.out 58a0936858 
  ql/src/test/results/clientpositive/llap/script_pipe.q.out f56107ebb1 
  ql/src/test/results/clientpositive/llap/scriptfile1.q.out 29ee2df4b3 
  ql/src/test/results/clientpositive/llap/select_transform_hint.q.out 
e0c1cea301 
  ql/src/test/results/clientpositive/llap/str_to_map.q.out f1aa4c14a0 
  ql/src/test/results/clientpositive/llap/temp_table_partcols1.q.out f2da1bf431 
  ql/src/test/results/clientpositive/llap/transform1.q.out d8d22443b8 
  ql/src/test/results/clientpositive/llap/transform2.q.out aeeaebf06b 
  ql/src/test/results/clientpositive/llap/transform3.q.out e3c55b8466 
  ql/src/test/results/clientpositive/llap/transform_acid.q.out bfa9aee0b0 
  ql/src/test/results/clientpositive/llap/transform_ppr1.q.out 25468bcd9c 
  ql/src/test/results/clientpositive/llap/transform_ppr2.q.out 8aeb688513 
  ql/src/test/results/clientpositive/llap/udaf_sum_list.q.out 8afb053261 
  ql/src/test/results/clientpositive/llap/udf_printf.q.out 9809f9567d 
  ql/src/test/results/clientpositive/llap/union23.q.out 742b282b62 
  ql/src/test/results/clientpositive/llap/union_script.q.out b8dfeb1319 
  ql/src/test/results/clientpositive/llap/vector_custom_udf_configure.q.out 
738fa32d78 
  ql/src/test/results/clientpositive/llap/vector_udf3.q.out 16ce5f3142 


Diff: https://reviews.apache.org/r/72514/diff/1/


Testing
---


Thanks,

Miklos Gergely

Re: Review Request 72514: Move TestCliDriver tests to TestMiniLlapCliDriver if they are failing with TestMiniLlapLocalCliDriver

2020-05-14 Thread Jesús Camacho Rodríguez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72514/#review220772
---




ql/src/test/results/clientpositive/llap/infer_bucket_sort.q.out
Line 53 (original), 53 (patched)


This has changed in this patch too.



ql/src/test/results/clientpositive/llap/input5.q.out
Line 164 (original)


Same results, different order.



ql/src/test/results/clientpositive/llap/regexp_extract.q.out
Line 179 (original)


Same results, different order?



ql/src/test/results/clientpositive/llap/regexp_extract.q.out
Line 441 (original)


Same results, different order?



ql/src/test/results/clientpositive/llap/select_transform_hint.q.out
Line 102 (original), 113 (patched)


Same results, different order?


- Jesús Camacho Rodríguez


On May 14, 2020, 9:15 p.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72514/
> ---
> 
> (Updated May 14, 2020, 9:15 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-23470
> https://issues.apache.org/jira/browse/HIVE-23470
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Move TestCliDriver tests to TestMiniLlapCliDriver if they are failing with 
> TestMiniLlapLocalCliDriver
> 
> 
> Diffs
> -
> 
>   ql/src/test/results/clientpositive/llap/autoColumnStats_6.q.out ff708cb6b0 
>   ql/src/test/results/clientpositive/llap/binary_output_format.q.out 
> b414360855 
>   ql/src/test/results/clientpositive/llap/create_genericudaf.q.out 85d7850888 
>   ql/src/test/results/clientpositive/llap/create_udaf.q.out 7bfce125f0 
>   ql/src/test/results/clientpositive/llap/create_view.q.out 9a251fcd2f 
>   ql/src/test/results/clientpositive/llap/gen_udf_example_add10.q.out 
> bfe313967b 
>   ql/src/test/results/clientpositive/llap/groupby_bigdata.q.out 90ccc8cdfb 
>   ql/src/test/results/clientpositive/llap/infer_bucket_sort.q.out bfdc84e24e 
>   ql/src/test/results/clientpositive/llap/input14.q.out 0e61434791 
>   ql/src/test/results/clientpositive/llap/input14_limit.q.out fe9d907663 
>   ql/src/test/results/clientpositive/llap/input17.q.out 9c03f5b0af 
>   ql/src/test/results/clientpositive/llap/input18.q.out ce731e6b2b 
>   ql/src/test/results/clientpositive/llap/input20.q.out d90b9083c3 
>   ql/src/test/results/clientpositive/llap/input33.q.out c8df2efede 
>   ql/src/test/results/clientpositive/llap/input34.q.out 00dd35d803 
>   ql/src/test/results/clientpositive/llap/input35.q.out cee491fc82 
>   ql/src/test/results/clientpositive/llap/input36.q.out 45289b2143 
>   ql/src/test/results/clientpositive/llap/input38.q.out d46ddf03ca 
>   ql/src/test/results/clientpositive/llap/input5.q.out becfc1876a 
>   ql/src/test/results/clientpositive/llap/insert_into3.q.out 60fd42d6fe 
>   ql/src/test/results/clientpositive/llap/insert_into4.q.out 031d562a43 
>   ql/src/test/results/clientpositive/llap/insert_into5.q.out 8ca94ee136 
>   ql/src/test/results/clientpositive/llap/insert_into6.q.out 2c6cab53e6 
>   ql/src/test/results/clientpositive/llap/load_binary_data.q.out b0d5c634b5 
>   ql/src/test/results/clientpositive/llap/macro_1.q.out 28230f90e5 
>   ql/src/test/results/clientpositive/llap/macro_duplicate.q.out 9598126c92 
>   ql/src/test/results/clientpositive/llap/mapreduce3.q.out 9c0157c923 
>   ql/src/test/results/clientpositive/llap/mapreduce4.q.out a606df0894 
>   ql/src/test/results/clientpositive/llap/mapreduce7.q.out ab369e667b 
>   ql/src/test/results/clientpositive/llap/mapreduce8.q.out d00ede826b 
>   ql/src/test/results/clientpositive/llap/merge_test_dummy_operator.q.out 
> 31d4ae16f7 
>   ql/src/test/results/clientpositive/llap/newline.q.out bea4e6ce1c 
>   
> ql/src/test/results/clientpositive/llap/nonreserved_keywords_insert_into1.q.out
>  6435e8b5a3 
>   ql/src/test/results/clientpositive/llap/nullscript.q.out cd926aa170 
>   ql/src/test/results/clientpositive/llap/orc_createas1.q.out 6884e8654e 
>   ql/src/test/results/clientpositive/llap/partcols1.q.out edd7db2357 
>   ql/src/test/results/clientpositive/llap/ppd_transform.q.out a38042c6fe 
>   ql/src/test/results/clientpositive/llap/query_with_semi.q.out 93da006251 
>   ql/src/test/results/clientpositive/llap/rcfile_bigdata.q.out c1ada45ad0 
>   ql/src/test/results/clientpositive/llap/regexp_extract.q.out 95f7c22bc9 
>   ql/src/test/results/clientpositive/llap/script_env_var1.q.out c1181b2635 
>   ql/src/test/results/clientpositive/llap/script_env_var2.q.out 58a0936858 
>   ql/src/test/results/clientpositive

Re: Review Request 72514: Move TestCliDriver tests to TestMiniLlapCliDriver if they are failing with TestMiniLlapLocalCliDriver

2020-05-14 Thread Miklos Gergely


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72514/#review220773
---




ql/src/test/results/clientpositive/llap/autoColumnStats_6.q.out
Line 57 (original), 64 (patched)


Statistics have changed!



ql/src/test/results/clientpositive/llap/infer_bucket_sort.q.out
Line 53 (original), 53 (patched)


Number of buckets modified



ql/src/test/results/clientpositive/llap/input5.q.out
Line 164 (original)


Order change



ql/src/test/results/clientpositive/llap/regexp_extract.q.out
Line 179 (original)


Results changed, most likely just the order



ql/src/test/results/clientpositive/llap/regexp_extract.q.out
Line 441 (original)


Results have changed, most likely just the order.


- Miklos Gergely


On May 14, 2020, 9:15 p.m., Miklos Gergely wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72514/
> ---
> 
> (Updated May 14, 2020, 9:15 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-23470
> https://issues.apache.org/jira/browse/HIVE-23470
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Move TestCliDriver tests to TestMiniLlapCliDriver if they are failing with 
> TestMiniLlapLocalCliDriver
> 
> 
> Diffs
> -
> 
>   ql/src/test/results/clientpositive/llap/autoColumnStats_6.q.out ff708cb6b0 
>   ql/src/test/results/clientpositive/llap/binary_output_format.q.out 
> b414360855 
>   ql/src/test/results/clientpositive/llap/create_genericudaf.q.out 85d7850888 
>   ql/src/test/results/clientpositive/llap/create_udaf.q.out 7bfce125f0 
>   ql/src/test/results/clientpositive/llap/create_view.q.out 9a251fcd2f 
>   ql/src/test/results/clientpositive/llap/gen_udf_example_add10.q.out 
> bfe313967b 
>   ql/src/test/results/clientpositive/llap/groupby_bigdata.q.out 90ccc8cdfb 
>   ql/src/test/results/clientpositive/llap/infer_bucket_sort.q.out bfdc84e24e 
>   ql/src/test/results/clientpositive/llap/input14.q.out 0e61434791 
>   ql/src/test/results/clientpositive/llap/input14_limit.q.out fe9d907663 
>   ql/src/test/results/clientpositive/llap/input17.q.out 9c03f5b0af 
>   ql/src/test/results/clientpositive/llap/input18.q.out ce731e6b2b 
>   ql/src/test/results/clientpositive/llap/input20.q.out d90b9083c3 
>   ql/src/test/results/clientpositive/llap/input33.q.out c8df2efede 
>   ql/src/test/results/clientpositive/llap/input34.q.out 00dd35d803 
>   ql/src/test/results/clientpositive/llap/input35.q.out cee491fc82 
>   ql/src/test/results/clientpositive/llap/input36.q.out 45289b2143 
>   ql/src/test/results/clientpositive/llap/input38.q.out d46ddf03ca 
>   ql/src/test/results/clientpositive/llap/input5.q.out becfc1876a 
>   ql/src/test/results/clientpositive/llap/insert_into3.q.out 60fd42d6fe 
>   ql/src/test/results/clientpositive/llap/insert_into4.q.out 031d562a43 
>   ql/src/test/results/clientpositive/llap/insert_into5.q.out 8ca94ee136 
>   ql/src/test/results/clientpositive/llap/insert_into6.q.out 2c6cab53e6 
>   ql/src/test/results/clientpositive/llap/load_binary_data.q.out b0d5c634b5 
>   ql/src/test/results/clientpositive/llap/macro_1.q.out 28230f90e5 
>   ql/src/test/results/clientpositive/llap/macro_duplicate.q.out 9598126c92 
>   ql/src/test/results/clientpositive/llap/mapreduce3.q.out 9c0157c923 
>   ql/src/test/results/clientpositive/llap/mapreduce4.q.out a606df0894 
>   ql/src/test/results/clientpositive/llap/mapreduce7.q.out ab369e667b 
>   ql/src/test/results/clientpositive/llap/mapreduce8.q.out d00ede826b 
>   ql/src/test/results/clientpositive/llap/merge_test_dummy_operator.q.out 
> 31d4ae16f7 
>   ql/src/test/results/clientpositive/llap/newline.q.out bea4e6ce1c 
>   
> ql/src/test/results/clientpositive/llap/nonreserved_keywords_insert_into1.q.out
>  6435e8b5a3 
>   ql/src/test/results/clientpositive/llap/nullscript.q.out cd926aa170 
>   ql/src/test/results/clientpositive/llap/orc_createas1.q.out 6884e8654e 
>   ql/src/test/results/clientpositive/llap/partcols1.q.out edd7db2357 
>   ql/src/test/results/clientpositive/llap/ppd_transform.q.out a38042c6fe 
>   ql/src/test/results/clientpositive/llap/query_with_semi.q.out 93da006251 
>   ql/src/test/results/clientpositive/llap/rcfile_bigdata.q.out c1ada45ad0 
>   ql/src/test/results/clientpositive/llap/regexp_extract.q.out 95f7c22bc9 
>   ql/src/test/results/clientpositive/llap/script_env_var1.q.out c1181b2635 
>   ql/src/test/results/clientpositive/llap/script_env_var2.q.out 58a0936858 
>   ql/src/test/results/clientpositive/llap/script_pip

Re: Review Request 72462: MSCK REPAIR cannot discover partitions with upper case directory names

2020-05-14 Thread Adesh Rao


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72462/
---

(Updated May 15, 2020, 5:42 a.m.)


Review request for hive and Sankar Hariappan.


Repository: hive-git


Description
---

The fix converts partition keys to lowercase present in hdfs directory, but 
store the hdfs directory as is for partition path.


Diffs (updated)
-

  ql/src/test/queries/clientpositive/msck_repair_4.q PRE-CREATION 
  ql/src/test/queries/clientpositive/msck_repair_5.q PRE-CREATION 
  ql/src/test/queries/clientpositive/msck_repair_6.q PRE-CREATION 
  ql/src/test/results/clientpositive/msck_repair_4.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/msck_repair_5.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/msck_repair_6.q.out PRE-CREATION 
  
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java
 5287f47e21 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java
 6f4400a8ef 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
 f4e109d1b0 
  
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
 92d10cd0e1 


Diff: https://reviews.apache.org/r/72462/diff/2/

Changes: https://reviews.apache.org/r/72462/diff/1-2/


Testing
---


Thanks,

Adesh Rao

Re: Review Request 72462: MSCK REPAIR cannot discover partitions with upper case directory names

2020-05-14 Thread Adesh Rao



> On May 14, 2020, 8:12 p.m., Sankar Hariappan wrote:
> > ql/src/test/queries/clientpositive/msck_repair_4.q
> > Lines 8 (patched)
> > 
> >
> > Add a testcase with table path repairtable_n4 having upper case. You 
> > can achieve it by setting location in create table.

Done.


> On May 14, 2020, 8:12 p.m., Sankar Hariappan wrote:
> > standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java
> > Lines 116 (patched)
> > 
> >
> > Why do we need pathSet? Can we check if path == null instead?

Removed.


> On May 14, 2020, 8:12 p.m., Sankar Hariappan wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
> > Line 1441 (original), 1441 (patched)
> > 
> >
> > Even table name in directory can be any case. Should we use 
> > equalsIgnoresCase here?

Msck list all the directories under the actual table path, and then checks if 
the new child directories are valid partitions. Since all the partition 
directories are child of the actual table path, it won't matter if we are using 
equals/equalsIgnoreCase.

The second question which comes is, should we check for partitions under all 
paths that matches the actual table path ignoring the case. I don't think we 
should do that, because, then msck will have to check in too many combinations 
of base directory in hdfs (which will be ~2^length of table name).


> On May 14, 2020, 8:12 p.m., Sankar Hariappan wrote:
> > standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
> > Line 1454 (original), 1456 (patched)
> > 
> >
> > Need to check how Hive treats ptn='A' and ptn='a' as ptn keys are 
> > lowercase but values can be any case.

Added a test case. These two values should be treated as different as they both 
are different strings.


- Adesh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72462/#review220771
---


On May 15, 2020, 5:42 a.m., Adesh Rao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72462/
> ---
> 
> (Updated May 15, 2020, 5:42 a.m.)
> 
> 
> Review request for hive and Sankar Hariappan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> The fix converts partition keys to lowercase present in hdfs directory, but 
> store the hdfs directory as is for partition path.
> 
> 
> Diffs
> -
> 
>   ql/src/test/queries/clientpositive/msck_repair_4.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/msck_repair_5.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/msck_repair_6.q PRE-CREATION 
>   ql/src/test/results/clientpositive/msck_repair_4.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/msck_repair_5.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/msck_repair_6.q.out PRE-CREATION 
>   
> standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/CheckResult.java
>  5287f47e21 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreChecker.java
>  6f4400a8ef 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/Msck.java
>  f4e109d1b0 
>   
> standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreServerUtils.java
>  92d10cd0e1 
> 
> 
> Diff: https://reviews.apache.org/r/72462/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Adesh Rao
> 
>

Re: Review Request 72499: HIVE-23446:LLAP: Reduce IPC connection misses to AM for short queries

Re: Review Request 72499: HIVE-23446:LLAP: Reduce IPC connection misses to AM for short queries

Re: Review Request 72499: HIVE-23446:LLAP: Reduce IPC connection misses to AM for short queries

[jira] [Created] (HIVE-23468) LLAP: Optimise OrcEncodedDataReader to avoid FS init to NN

[jira] [Created] (HIVE-23469) Use hostname + pod UID for shuffle manager caching

[jira] [Created] (HIVE-23470) Move TestCliDriver tests to TestMiniTezCliDriver if they are failing with TestMiniLlapLocalCliDriver

[jira] [Created] (HIVE-23471) Statement.executeUpdate() does not return correct affected records causing "No such lock"

Re: Review Request 72481: HIVE-23234: Optimize TxnHandler::allocateTableWriteIds

Re: Review Request 72481: HIVE-23234: Optimize TxnHandler::allocateTableWriteIds

Re: Review Request 72481: HIVE-23234: Optimize TxnHandler::allocateTableWriteIds

Re: Review Request 72481: HIVE-23234: Optimize TxnHandler::allocateTableWriteIds

Review Request 72510: Move TestCliDriver tests to TestMiniTezCliDriver if they are failing with TestMiniLlapLocalCliDriver

Re: Review Request 72510: Move TestCliDriver tests to TestMiniTezCliDriver if they are failing with TestMiniLlapLocalCliDriver

[jira] [Created] (HIVE-23472) LLAP Guaranteed state update should trigger queue re-ordering

Re: Review Request 72510: Move TestCliDriver tests to TestMiniTezCliDriver if they are failing with TestMiniLlapLocalCliDriver

Re: Review Request 72462: MSCK REPAIR cannot discover partitions with upper case directory names

Review Request 72514: Move TestCliDriver tests to TestMiniLlapCliDriver if they are failing with TestMiniLlapLocalCliDriver

Re: Review Request 72514: Move TestCliDriver tests to TestMiniLlapCliDriver if they are failing with TestMiniLlapLocalCliDriver

Re: Review Request 72514: Move TestCliDriver tests to TestMiniLlapCliDriver if they are failing with TestMiniLlapLocalCliDriver

Re: Review Request 72462: MSCK REPAIR cannot discover partitions with upper case directory names

Re: Review Request 72462: MSCK REPAIR cannot discover partitions with upper case directory names

21 matches

Site Navigation

Mail list logo

Footer information