Re: Review Request 72112: HIVE-22869 - Add locking benchmark to metastore-tools/metastore-benchmarks

2020-05-18 Thread Zoltan Chovan via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/
---

(Updated May 18, 2020, 4:59 p.m.)


Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter 
Vary.


Repository: hive-git


Description
---

Add the possibility to run benchmarks on opening lock in the HMS. Currently 
this change only introduces single-threaded/single client testing. I'm planning 
to add multi-client support in a separate change.

Example parametrisation is as follows:
hbench -M ".*Lock.*" -N 10 -d hive_test --params 10 --params 100 -d hive_test

This will create N number (10) of locks for first --params number of tables 
(10) with second --params number of partitions (100) on T (8) threads where 
each thread will strart an HMS client and it'll use -d (hive_test) database;


Diffs
-

  
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java
 2ab9388301 
  
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
 d80c290b60 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkSuite.java
 5211082a7d 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
 4e75edeae6 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java
 101d6759c5 


Diff: https://reviews.apache.org/r/72112/diff/2/


Testing
---


File Attachments (updated)


HIVE-22869.2.patch
  
https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
HIVE-22869.3.patch
  
https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
HIVE-22869.4.patch
  
https://reviews.apache.org/media/uploaded/files/2020/04/23/423c45d7-911e-4dd2-80b8-c6d3ad90633c__HIVE-22869.4.patch
HIVE-22869.5.patch
  
https://reviews.apache.org/media/uploaded/files/2020/05/12/a06f3b8c-f4ca-4067-a079-e0b6185266d4__HIVE-22869.5.patch
HIVE-22869.6.patch
  
https://reviews.apache.org/media/uploaded/files/2020/05/15/01254e94-1a8d-496d-ab31-628bd5584193__HIVE-22869.6.patch
HIVE-22869.7.patch
  
https://reviews.apache.org/media/uploaded/files/2020/05/18/343e8c4e-1f7c-4638-849a-15b448bc2515__HIVE-22869.7.patch


Thanks,

Zoltan Chovan

Re: Review Request 72112: HIVE-22869 - Add locking benchmark to metastore-tools/metastore-benchmarks

2020-05-18 Thread Zoltan Chovan via Review Board 


> On May 18, 2020, 7:58 a.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java
> > Lines 308 (patched)
> > 
> >
> > Could we have null here? summarisedBenchmarkSuite.getResult().get(name)

there's a null check for that 4 lines above :)


> On May 18, 2020, 7:58 a.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
> > Lines 362 (patched)
> > 
> >
> > I think howMany is always 1. Isn't it?

no, this will be determined by the -N parameter (also called instances) see 
BenchmarkTool line 269


- Zoltan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220804
---


On May 15, 2020, 9:12 a.m., Zoltan Chovan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> ---
> 
> (Updated May 15, 2020, 9:12 a.m.)
> 
> 
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter 
> Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Add the possibility to run benchmarks on opening lock in the HMS. Currently 
> this change only introduces single-threaded/single client testing. I'm 
> planning to add multi-client support in a separate change.
> 
> Example parametrisation is as follows:
> hbench -M ".*Lock.*" -N 10 -d hive_test --params 10 --params 100 -d hive_test
> 
> This will create N number (10) of locks for first --params number of tables 
> (10) with second --params number of partitions (100) on T (8) threads where 
> each thread will strart an HMS client and it'll use -d (hive_test) database;
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java
>  2ab9388301 
>   
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
>  d80c290b60 
>   
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkSuite.java
>  5211082a7d 
>   
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
>  4e75edeae6 
>   
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java
>  101d6759c5 
> 
> 
> Diff: https://reviews.apache.org/r/72112/diff/2/
> 
> 
> Testing
> ---
> 
> 
> File Attachments
> 
> 
> HIVE-22869.2.patch
>   
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
> HIVE-22869.3.patch
>   
> https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
> HIVE-22869.4.patch
>   
> https://reviews.apache.org/media/uploaded/files/2020/04/23/423c45d7-911e-4dd2-80b8-c6d3ad90633c__HIVE-22869.4.patch
> HIVE-22869.5.patch
>   
> https://reviews.apache.org/media/uploaded/files/2020/05/12/a06f3b8c-f4ca-4067-a079-e0b6185266d4__HIVE-22869.5.patch
> HIVE-22869.6.patch
>   
> https://reviews.apache.org/media/uploaded/files/2020/05/15/01254e94-1a8d-496d-ab31-628bd5584193__HIVE-22869.6.patch
> 
> 
> Thanks,
> 
> Zoltan Chovan
> 
>

Re: Review Request 72112: HIVE-22869 - Add locking benchmark to metastore-tools/metastore-benchmarks

2020-05-15 Thread Zoltan Chovan via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/
---

(Updated May 15, 2020, 9:12 a.m.)


Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter 
Vary.


Repository: hive-git


Description
---

Add the possibility to run benchmarks on opening lock in the HMS. Currently 
this change only introduces single-threaded/single client testing. I'm planning 
to add multi-client support in a separate change.

Example parametrisation is as follows:
hbench -M ".*Lock.*" -N 10 -d hive_test --params 10 --params 100 -d hive_test

This will create N number (10) of locks for first --params number of tables 
(10) with second --params number of partitions (100) on T (8) threads where 
each thread will strart an HMS client and it'll use -d (hive_test) database;


Diffs
-

  
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java
 2ab9388301 
  
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
 d80c290b60 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkSuite.java
 5211082a7d 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
 4e75edeae6 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java
 101d6759c5 


Diff: https://reviews.apache.org/r/72112/diff/2/


Testing
---


File Attachments (updated)


HIVE-22869.2.patch
  
https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
HIVE-22869.3.patch
  
https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
HIVE-22869.4.patch
  
https://reviews.apache.org/media/uploaded/files/2020/04/23/423c45d7-911e-4dd2-80b8-c6d3ad90633c__HIVE-22869.4.patch
HIVE-22869.5.patch
  
https://reviews.apache.org/media/uploaded/files/2020/05/12/a06f3b8c-f4ca-4067-a079-e0b6185266d4__HIVE-22869.5.patch
HIVE-22869.6.patch
  
https://reviews.apache.org/media/uploaded/files/2020/05/15/01254e94-1a8d-496d-ab31-628bd5584193__HIVE-22869.6.patch


Thanks,

Zoltan Chovan

Re: Review Request 72112: HIVE-22869 - Add locking benchmark to metastore-tools/metastore-benchmarks

2020-05-12 Thread Zoltan Chovan via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/
---

(Updated May 12, 2020, 11:38 a.m.)


Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter 
Vary.


Repository: hive-git


Description
---

Add the possibility to run benchmarks on opening lock in the HMS. Currently 
this change only introduces single-threaded/single client testing. I'm planning 
to add multi-client support in a separate change.

Example parametrisation is as follows:
hbench -M "lock" -N 10 -d hive_test -W 0 -L 100
hbench -M ".*Lock.*" -N 10 -d hive_test -W 0 -L 100 -T 8 --params 100

This will create N number (10) of tables to lock and it'll execute the lock() 
for L number (100) of times on T (8) threads where each thread will strart an 
HMS client


Diffs (updated)
-

  
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java
 2ab9388301 
  
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
 d80c290b60 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkSuite.java
 5211082a7d 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
 4e75edeae6 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java
 101d6759c5 


Diff: https://reviews.apache.org/r/72112/diff/2/

Changes: https://reviews.apache.org/r/72112/diff/1-2/


Testing
---


File Attachments (updated)


HIVE-22869.2.patch
  
https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
HIVE-22869.3.patch
  
https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
HIVE-22869.4.patch
  
https://reviews.apache.org/media/uploaded/files/2020/04/23/423c45d7-911e-4dd2-80b8-c6d3ad90633c__HIVE-22869.4.patch
HIVE-22869.5.patch
  
https://reviews.apache.org/media/uploaded/files/2020/05/12/a06f3b8c-f4ca-4067-a079-e0b6185266d4__HIVE-22869.5.patch


Thanks,

Zoltan Chovan

Re: Review Request 72112: HIVE-22869 - Add locking benchmark to metastore-tools/metastore-benchmarks

2020-04-23 Thread Zoltan Chovan via Review Board 


> On April 3, 2020, 11:28 a.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
> > Lines 422 (patched)
> > 
> >
> > I think, for more realistic scenario we should move commit inside of 
> > bench + add some sleep to simulate query execution. Currently locks number 
> > is just keep increasing and could go very high that shouldn't happen in 
> > real system.

If we start going down the query simulation road then this will be an e2e test, 
not an api benchmark :) 
Let's say we add sleep to simulate queries. If every query takes the same time 
(which is again not realistic) then we just add a wait and every client would 
try the calls at the same time again. If we add variable lenghts to quereis 
then this will not be a benchmark that is reproducable at will.


- Zoltan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220213
---


On April 9, 2020, 2:58 p.m., Zoltan Chovan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> ---
> 
> (Updated April 9, 2020, 2:58 p.m.)
> 
> 
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter 
> Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Add the possibility to run benchmarks on opening lock in the HMS. Currently 
> this change only introduces single-threaded/single client testing. I'm 
> planning to add multi-client support in a separate change.
> 
> Example parametrisation is as follows:
> hbench -M "lock" -N 10 -d hive_test -W 0 -L 100
> hbench -M ".*Lock.*" -N 10 -d hive_test -W 0 -L 100 -T 8 --params 100
> 
> This will create N number (10) of tables to lock and it'll execute the lock() 
> for L number (100) of times on T (8) threads where each thread will strart an 
> HMS client
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java
>  041cd76234 
>   
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
>  f53f2ef43b 
>   
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
>  7cc1e42a8b 
>   
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java
>  101d6759c5 
> 
> 
> Diff: https://reviews.apache.org/r/72112/diff/1/
> 
> 
> Testing
> ---
> 
> 
> File Attachments
> 
> 
> HIVE-22869.2.patch
>   
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
> HIVE-22869.3.patch
>   
> https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
> 
> 
> Thanks,
> 
> Zoltan Chovan
> 
>

Re: Review Request 72112: HIVE-22869 - Add locking benchmark to metastore-tools/metastore-benchmarks

2020-04-23 Thread Zoltan Chovan via Review Board 


> On April 3, 2020, 11:28 a.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
> > Lines 414 (patched)
> > 
> >
> > I think more realistic scenario is having up to 10 tables with 10-1000 
> > (configurable) partitions and locking not only on table level but partition.

I agree with that, however from a strictly api-testing viewpoint I'm not sure 
if there's any practical difference between creating locks for 10-1000 tables 
or 10-1000 partitions. The lock objects would just point to a different object, 
the number of HMS calls should be the same.


- Zoltan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220213
---


On April 9, 2020, 2:58 p.m., Zoltan Chovan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> ---
> 
> (Updated April 9, 2020, 2:58 p.m.)
> 
> 
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter 
> Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Add the possibility to run benchmarks on opening lock in the HMS. Currently 
> this change only introduces single-threaded/single client testing. I'm 
> planning to add multi-client support in a separate change.
> 
> Example parametrisation is as follows:
> hbench -M "lock" -N 10 -d hive_test -W 0 -L 100
> hbench -M ".*Lock.*" -N 10 -d hive_test -W 0 -L 100 -T 8 --params 100
> 
> This will create N number (10) of tables to lock and it'll execute the lock() 
> for L number (100) of times on T (8) threads where each thread will strart an 
> HMS client
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java
>  041cd76234 
>   
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
>  f53f2ef43b 
>   
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
>  7cc1e42a8b 
>   
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java
>  101d6759c5 
> 
> 
> Diff: https://reviews.apache.org/r/72112/diff/1/
> 
> 
> Testing
> ---
> 
> 
> File Attachments
> 
> 
> HIVE-22869.2.patch
>   
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
> HIVE-22869.3.patch
>   
> https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch
> 
> 
> Thanks,
> 
> Zoltan Chovan
> 
>

Re: Review Request 72112: HIVE-22869 - Add locking benchmark to metastore-tools/metastore-benchmarks

2020-04-09 Thread Zoltan Chovan via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/
---

(Updated April 9, 2020, 2:58 p.m.)


Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter 
Vary.


Repository: hive-git


Description
---

Add the possibility to run benchmarks on opening lock in the HMS. Currently 
this change only introduces single-threaded/single client testing. I'm planning 
to add multi-client support in a separate change.

Example parametrisation is as follows:
hbench -M "lock" -N 10 -d hive_test -W 0 -L 100
hbench -M ".*Lock.*" -N 10 -d hive_test -W 0 -L 100 -T 8 --params 100

This will create N number (10) of tables to lock and it'll execute the lock() 
for L number (100) of times on T (8) threads where each thread will strart an 
HMS client


Diffs
-

  
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java
 041cd76234 
  
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
 f53f2ef43b 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
 7cc1e42a8b 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java
 101d6759c5 


Diff: https://reviews.apache.org/r/72112/diff/1/


Testing
---


File Attachments (updated)


HIVE-22869.2.patch
  
https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
HIVE-22869.3.patch
  
https://reviews.apache.org/media/uploaded/files/2020/04/09/458beaa7-4743-40fb-a213-1ae4527be823__HIVE-22869.3.patch


Thanks,

Zoltan Chovan

Re: Review Request 72112: HIVE-22869 - Add locking benchmark to metastore-tools/metastore-benchmarks

2020-04-09 Thread Zoltan Chovan via Review Board 


> On April 3, 2020, 9:59 a.m., Peter Vary wrote:
> > standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
> > Lines 430 (patched)
> > 
> >
> > Do we need this? Shouldn't the lock be unlocked by commitTnx?

you're right


- Zoltan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220212
---


On April 2, 2020, 2:13 p.m., Zoltan Chovan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> ---
> 
> (Updated April 2, 2020, 2:13 p.m.)
> 
> 
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter 
> Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Add the possibility to run benchmarks on opening lock in the HMS. Currently 
> this change only introduces single-threaded/single client testing. I'm 
> planning to add multi-client support in a separate change.
> 
> Example parametrisation is as follows:
> hbench -M "lock" -N 10 -d hive_test -W 0 -L 100
> hbench -M ".*Lock.*" -N 10 -d hive_test -W 0 -L 100 -T 8 --params 100
> 
> This will create N number (10) of tables to lock and it'll execute the lock() 
> for L number (100) of times on T (8) threads where each thread will strart an 
> HMS client
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java
>  041cd76234 
>   
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
>  f53f2ef43b 
>   
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
>  7cc1e42a8b 
>   
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java
>  101d6759c5 
> 
> 
> Diff: https://reviews.apache.org/r/72112/diff/1/
> 
> 
> Testing
> ---
> 
> 
> File Attachments
> 
> 
> HIVE-22869.2.patch
>   
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
> 
> 
> Thanks,
> 
> Zoltan Chovan
> 
>

Re: Review Request 72112: HIVE-22869 - Add locking benchmark to metastore-tools/metastore-benchmarks

2020-04-09 Thread Zoltan Chovan via Review Board 


> On April 3, 2020, 9:59 a.m., Peter Vary wrote:
> > standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
> > Lines 341 (patched)
> > 
> >
> > I put this in the code with HIVE-23042:
> >   boolean openTxn(int numTxns) throws TException {
> > client.open_txns(new OpenTxnRequest(numTxns, "Test", "Host"));
> > return true;
> >   }
> >   
> > Maybe merge those?

The main difference between our two implementations of openTxn is that mine 
automatically returns the opened txn's id, in your version there has to be an 
additional getOpenTxns() call made to get the Id. 
Not sure if getOpenTxns would return some other ids that belong to an other 
client when multiple threads are used, o sI might be misunderstanding the 
getOpenTxns() call.
What do you think?


- Zoltan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220212
---


On April 2, 2020, 2:13 p.m., Zoltan Chovan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> ---
> 
> (Updated April 2, 2020, 2:13 p.m.)
> 
> 
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter 
> Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Add the possibility to run benchmarks on opening lock in the HMS. Currently 
> this change only introduces single-threaded/single client testing. I'm 
> planning to add multi-client support in a separate change.
> 
> Example parametrisation is as follows:
> hbench -M "lock" -N 10 -d hive_test -W 0 -L 100
> hbench -M ".*Lock.*" -N 10 -d hive_test -W 0 -L 100 -T 8 --params 100
> 
> This will create N number (10) of tables to lock and it'll execute the lock() 
> for L number (100) of times on T (8) threads where each thread will strart an 
> HMS client
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java
>  041cd76234 
>   
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
>  f53f2ef43b 
>   
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
>  7cc1e42a8b 
>   
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java
>  101d6759c5 
> 
> 
> Diff: https://reviews.apache.org/r/72112/diff/1/
> 
> 
> Testing
> ---
> 
> 
> File Attachments
> 
> 
> HIVE-22869.2.patch
>   
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
> 
> 
> Thanks,
> 
> Zoltan Chovan
> 
>

Re: Review Request 72112: HIVE-22869 - Add locking benchmark to metastore-tools/metastore-benchmarks

2020-04-09 Thread Zoltan Chovan via Review Board 


> On April 3, 2020, 11:28 a.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
> > Lines 410 (patched)
> > 
> >
> > Do you really need to have physical tables?

no we don't :) removed this part


> On April 3, 2020, 11:28 a.m., Denys Kuzmenko wrote:
> > standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
> > Lines 455 (patched)
> > 
> >
> > Should we care about replPolicy here?

probably not, removed it


- Zoltan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/#review220213
---


On April 2, 2020, 2:13 p.m., Zoltan Chovan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72112/
> ---
> 
> (Updated April 2, 2020, 2:13 p.m.)
> 
> 
> Review request for hive, Denys Kuzmenko, Aron Hamvas, Marton Bod, and Peter 
> Vary.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Add the possibility to run benchmarks on opening lock in the HMS. Currently 
> this change only introduces single-threaded/single client testing. I'm 
> planning to add multi-client support in a separate change.
> 
> Example parametrisation is as follows:
> hbench -M "lock" -N 10 -d hive_test -W 0 -L 100
> hbench -M ".*Lock.*" -N 10 -d hive_test -W 0 -L 100 -T 8 --params 100
> 
> This will create N number (10) of tables to lock and it'll execute the lock() 
> for L number (100) of times on T (8) threads where each thread will strart an 
> HMS client
> 
> 
> Diffs
> -
> 
>   
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java
>  041cd76234 
>   
> standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
>  f53f2ef43b 
>   
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
>  7cc1e42a8b 
>   
> standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java
>  101d6759c5 
> 
> 
> Diff: https://reviews.apache.org/r/72112/diff/1/
> 
> 
> Testing
> ---
> 
> 
> File Attachments
> 
> 
> HIVE-22869.2.patch
>   
> https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch
> 
> 
> Thanks,
> 
> Zoltan Chovan
> 
>

Re: Review Request 72112: HIVE-22869 - Add locking benchmark to metastore-tools/metastore-benchmarks

2020-04-02 Thread Zoltan Chovan via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/
---

(Updated April 2, 2020, 2:13 p.m.)


Review request for hive, Denys Kuzmenko and Aron Hamvas.


Repository: hive-git


Description (updated)
---

Add the possibility to run benchmarks on opening lock in the HMS. Currently 
this change only introduces single-threaded/single client testing. I'm planning 
to add multi-client support in a separate change.

Example parametrisation is as follows:
hbench -M "lock" -N 10 -d hive_test -W 0 -L 100
hbench -M ".*Lock.*" -N 10 -d hive_test -W 0 -L 100 -T 8 --params 100

This will create N number (10) of tables to lock and it'll execute the lock() 
for L number (100) of times on T (8) threads where each thread will strart an 
HMS client


Diffs
-

  
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java
 041cd76234 
  
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
 f53f2ef43b 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
 7cc1e42a8b 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java
 101d6759c5 


Diff: https://reviews.apache.org/r/72112/diff/1/


Testing
---


File Attachments (updated)


HIVE-22869.2.patch
  
https://reviews.apache.org/media/uploaded/files/2020/04/02/5e35e835-f383-495f-9964-e66773fd6a90__HIVE-22869.2.patch


Thanks,

Zoltan Chovan

Review Request 72112: HIVE-22869 - Add locking benchmark to metastore-tools/metastore-benchmarks

2020-02-11 Thread Zoltan Chovan via Review Board 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72112/
---

Review request for hive, Denys Kuzmenko and Aron Hamvas.


Repository: hive-git


Description
---

Add the possibility to run benchmarks on opening lock in the HMS. Currently 
this change only introduces single-threaded/single client testing. I'm planning 
to add multi-client support in a separate change.

Example parametrisation is as follows:
hbench -M "lock" -N 10 -d hive_test -W 0 -L 100

This will create N number (10) of tables to lock and it'll execute the lock() 
for L number (100) of times.


Diffs
-

  
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/BenchmarkTool.java
 041cd76234 
  
standalone-metastore/metastore-tools/metastore-benchmarks/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSBenchmarks.java
 f53f2ef43b 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/HMSClient.java
 7cc1e42a8b 
  
standalone-metastore/metastore-tools/tools-common/src/main/java/org/apache/hadoop/hive/metastore/tools/Util.java
 101d6759c5 


Diff: https://reviews.apache.org/r/72112/diff/1/


Testing
---


Thanks,

Zoltan Chovan