Re: Review Request 67468: HIVE-18118: provide supportability support for Erasure Coding Update number of Erasure Coded Files in a directory as part of Basic (aka Quick) Stats This information is then

2018-06-21 Thread Andrew Sherman via Review Board

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67468/
---

(Updated June 21, 2018, 4:32 p.m.)


Review request for hive and Sahil Takiar.


Summary (updated)
-

HIVE-18118: provide supportability support for Erasure Coding Update number of 
Erasure Coded Files in a directory as part of Basic (aka Quick) Stats This 
information is then (mostly) available through 'EXPLAIN EXTENDED' and 'DESCRIBE 
EXTENDED' Extend the MiniHS2 Builder to allow configuring the numb


Repository: hive-git


Description
---

HIVE-18118: provide supportability support for Erasure Coding Update number of 
Erasure Coded Files in a directory as part of Basic (aka Quick) Stats This 
information is then (mostly) available through 'EXPLAIN EXTENDED' and 'DESCRIBE 
EXTENDED' Extend the MiniHS2 Builder to allow configuring the number of 
datanodes. Add a jdbc MiniHS2/Spark test that uses Erasure Coding. There are 
some change to StatsSetupConst to make checkstyle happy.


Diffs (updated)
-

  common/src/java/org/apache/hive/common/util/HiveStringUtils.java 
cfe9b2208a60586a05d293f222aa90b37e9a06ac 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
d7d7097336fc6be4c2f7a35cd6897e0375486e81 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2ErasureCoding.java
 PRE-CREATION 
  itests/src/test/resources/testconfiguration.properties 
aeb6211f5a11f6b9466d731cccb3e55cb03281cb 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
2106fec7af75958644eb831498b725b771ddf47a 
  itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 
1700c08d3f37285de43b5d4fe5c77ef55c170235 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
8e32b02b59c4e36e0dd610beb6aacf80c3ac555d 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
58c8960c096f8885086be4f46dc1e33edd26249a 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java
 44687ef471f76bb6c8baee3c9081a191e2d0e74d 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/TextMetaDataFormatter.java
 326cbedcf0194bfa42b66557fc88f6285df1c619 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkMapJoinOptimizer.java
 8e75db9e08d575eb8ba7123251eaca9e2097a7af 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
 d0be33bd0fd83c829584f069a12e36b278e4d6b2 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java 
61458b4e256f7bf63f781a3135059257a2b8ddd4 
  ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 
2c5b6557ce6462151e764c17064354f448ee708d 
  ql/src/java/org/apache/hadoop/hive/ql/plan/Statistics.java 
fd461ae52930de54d993f0df74c0635f82fcc799 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TableDesc.java 
4068e5670fe8a65593228d2e3df9d809e836a696 
  ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsNoJobTask.java 
d4d46a3671efdaaed32f63b7262b963cce00b94e 
  ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsTask.java 
8c238871765b0d5312a459a0e7f68c81f3837c13 
  ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 
e00098529f78aa4950c8cec301b966999ae9bf96 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
60447192b11261727a63219fb2e69f09fd425aa0 
  ql/src/test/queries/clientpositive/erasure_explain.q PRE-CREATION 
  ql/src/test/queries/clientpositive/erasure_simple.q 
c08409c17787417b986d90a43104f5ddd456e600 
  ql/src/test/results/clientpositive/erasurecoding/erasure_explain.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out 
01f6015a346c1e4283fd6a8cf1eaa3b670450e20 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/common/StatsSetupConst.java
 78ea01d9687fe043d63441430c46b30c25cd9756 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 e88f9a5fee4b2cbd99ec7c5c5350f8c2b8015384 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 16a8c758010c1f81e07296362157bb24260bcf3f 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/utils/TestMetaStoreUtils.java
 55ff1502d415dea52095cfdd523d01f1e49ce084 


Diff: https://reviews.apache.org/r/67468/diff/3/

Changes: https://reviews.apache.org/r/67468/diff/2-3/


Testing
---

Ran driver tests and new jdbc test


Thanks,

Andrew Sherman



Re: Review Request 67468: HIVE-18118: provide supportability support for Erasure Coding Update number of Erasure Coded Files in a directory as part of Basic (aka Quick) Stats This information is then

2018-06-15 Thread Andrew Sherman via Review Board


> On June 15, 2018, 1:37 p.m., Sahil Takiar wrote:
> > Why not show the #of EC files for regular explain plans too? To decrease 
> > the # of q file updates, it can be omitted if the # of EC files = 0

I saw that regular explain did not report numFiles so I did not report 
numEcFiles there.
I think you are saying that IF EC files are in a dir then instead of 
"Statistics: Num rows: 1 Data size: 15812" I would print  "Statistics: Num 
rows: 1 Data size: 15812 Erasure files: 2". Is that right?
Are you also suggesting that in extended explain I should also not report 
erasure coded file count unless it is positive? (This would require some small 
code changes as we currently I think just rely on dumping the properties )


> On June 15, 2018, 1:37 p.m., Sahil Takiar wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/common/StatsSetupConst.java
> > Line 116 (original), 122 (patched)
> > 
> >
> > why change this from an array to a list?

When I changed some of the arrays to have new members, checkstyle didn't like 
it and suggested Lists. It was ugly to have some Lists and some arrays so I 
changed all the similar ones to be Lists.


- Andrew


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67468/#review204836
---


On June 6, 2018, 12:46 a.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67468/
> ---
> 
> (Updated June 6, 2018, 12:46 a.m.)
> 
> 
> Review request for hive and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18118: provide supportability support for Erasure Coding 
> [NOTE THIS REVIEW INITIALLY OMITS 200+ .q.out changes]
> Update number of Erasure Coded Files in a directory as part of Basic (aka 
> Quick) Stats 
> This information is then (mostly) available through 'EXPLAIN EXTENDED' and 
> 'DESCRIBE EXTENDED' 
> Extend the MiniHS2 Builder to allow configuring the number of datanodes. 
> Add a jdbc MiniHS2/Spark test that uses Erasure Coding. 
> There are some change to StatsSetupConst to make checkstyle happy.
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> d7d7097336fc6be4c2f7a35cd6897e0375486e81 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2ErasureCoding.java
>  PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 
> 463fda1913f6d5b928fcee038f19e124b0239e96 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 2365fb76bd08f3a310e81ac3a19ca64971aeec8e 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 
> 1700c08d3f37285de43b5d4fe5c77ef55c170235 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
> e06949928d179cfd9a4dcb7176203b885509 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  209fdfb287cabc5bb7cab2117d771f7907deb2b9 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsNoJobTask.java 
> d4d46a3671efdaaed32f63b7262b963cce00b94e 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsTask.java 
> 8c238871765b0d5312a459a0e7f68c81f3837c13 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 982b18076180ba300094f30a7f87f025f993b265 
>   ql/src/test/queries/clientpositive/erasure_explain.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/erasure_simple.q 
> c08409c17787417b986d90a43104f5ddd456e600 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_explain.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out 
> 01f6015a346c1e4283fd6a8cf1eaa3b670450e20 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/common/StatsSetupConst.java
>  78ea01d9687fe043d63441430c46b30c25cd9756 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  77ed2b4de4569fa8aca23b16f2b362b187c7c4fc 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
>  9b36d09eb9fb332e913d442bb476628eca334b6e 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/utils/TestMetaStoreUtils.java
>  55ff1502d415dea52095cfdd523d01f1e49ce084 
> 
> 
> Diff: https://reviews.apache.org/r/67468/diff/1/
> 
> 
> Testing
> ---
> 
> Ran driver tests and new jdbc test
> 
> 
> Thanks,
> 
> Andrew Sherman
> 
>



Re: Review Request 67468: HIVE-18118: provide supportability support for Erasure Coding Update number of Erasure Coded Files in a directory as part of Basic (aka Quick) Stats This information is then

2018-06-15 Thread Sahil Takiar

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67468/#review204836
---



Why not show the #of EC files for regular explain plans too? To decrease the # 
of q file updates, it can be omitted if the # of EC files = 0


standalone-metastore/src/main/java/org/apache/hadoop/hive/common/StatsSetupConst.java
Line 116 (original), 122 (patched)


why change this from an array to a list?


- Sahil Takiar


On June 6, 2018, 12:46 a.m., Andrew Sherman wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67468/
> ---
> 
> (Updated June 6, 2018, 12:46 a.m.)
> 
> 
> Review request for hive and Sahil Takiar.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-18118: provide supportability support for Erasure Coding 
> [NOTE THIS REVIEW INITIALLY OMITS 200+ .q.out changes]
> Update number of Erasure Coded Files in a directory as part of Basic (aka 
> Quick) Stats 
> This information is then (mostly) available through 'EXPLAIN EXTENDED' and 
> 'DESCRIBE EXTENDED' 
> Extend the MiniHS2 Builder to allow configuring the number of datanodes. 
> Add a jdbc MiniHS2/Spark test that uses Erasure Coding. 
> There are some change to StatsSetupConst to make checkstyle happy.
> 
> 
> Diffs
> -
> 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> d7d7097336fc6be4c2f7a35cd6897e0375486e81 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2ErasureCoding.java
>  PRE-CREATION 
>   itests/src/test/resources/testconfiguration.properties 
> 463fda1913f6d5b928fcee038f19e124b0239e96 
>   itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
> 2365fb76bd08f3a310e81ac3a19ca64971aeec8e 
>   itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 
> 1700c08d3f37285de43b5d4fe5c77ef55c170235 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
> e06949928d179cfd9a4dcb7176203b885509 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  209fdfb287cabc5bb7cab2117d771f7907deb2b9 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsNoJobTask.java 
> d4d46a3671efdaaed32f63b7262b963cce00b94e 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsTask.java 
> 8c238871765b0d5312a459a0e7f68c81f3837c13 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 982b18076180ba300094f30a7f87f025f993b265 
>   ql/src/test/queries/clientpositive/erasure_explain.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/erasure_simple.q 
> c08409c17787417b986d90a43104f5ddd456e600 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_explain.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out 
> 01f6015a346c1e4283fd6a8cf1eaa3b670450e20 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/common/StatsSetupConst.java
>  78ea01d9687fe043d63441430c46b30c25cd9756 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
>  77ed2b4de4569fa8aca23b16f2b362b187c7c4fc 
>   
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
>  9b36d09eb9fb332e913d442bb476628eca334b6e 
>   
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/utils/TestMetaStoreUtils.java
>  55ff1502d415dea52095cfdd523d01f1e49ce084 
> 
> 
> Diff: https://reviews.apache.org/r/67468/diff/1/
> 
> 
> Testing
> ---
> 
> Ran driver tests and new jdbc test
> 
> 
> Thanks,
> 
> Andrew Sherman
> 
>



Review Request 67468: HIVE-18118: provide supportability support for Erasure Coding Update number of Erasure Coded Files in a directory as part of Basic (aka Quick) Stats This information is then (mos

2018-06-05 Thread Andrew Sherman via Review Board

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67468/
---

Review request for hive and Sahil Takiar.


Repository: hive-git


Description
---

HIVE-18118: provide supportability support for Erasure Coding 
[NOTE THIS REVIEW INITIALLY OMITS 200+ .q.out changes]
Update number of Erasure Coded Files in a directory as part of Basic (aka 
Quick) Stats 
This information is then (mostly) available through 'EXPLAIN EXTENDED' and 
'DESCRIBE EXTENDED' 
Extend the MiniHS2 Builder to allow configuring the number of datanodes. 
Add a jdbc MiniHS2/Spark test that uses Erasure Coding. 
There are some change to StatsSetupConst to make checkstyle happy.


Diffs
-

  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
d7d7097336fc6be4c2f7a35cd6897e0375486e81 
  
itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2ErasureCoding.java
 PRE-CREATION 
  itests/src/test/resources/testconfiguration.properties 
463fda1913f6d5b928fcee038f19e124b0239e96 
  itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 
2365fb76bd08f3a310e81ac3a19ca64971aeec8e 
  itests/util/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java 
1700c08d3f37285de43b5d4fe5c77ef55c170235 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 
e06949928d179cfd9a4dcb7176203b885509 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
209fdfb287cabc5bb7cab2117d771f7907deb2b9 
  ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsNoJobTask.java 
d4d46a3671efdaaed32f63b7262b963cce00b94e 
  ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsTask.java 
8c238871765b0d5312a459a0e7f68c81f3837c13 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
982b18076180ba300094f30a7f87f025f993b265 
  ql/src/test/queries/clientpositive/erasure_explain.q PRE-CREATION 
  ql/src/test/queries/clientpositive/erasure_simple.q 
c08409c17787417b986d90a43104f5ddd456e600 
  ql/src/test/results/clientpositive/erasurecoding/erasure_explain.q.out 
PRE-CREATION 
  ql/src/test/results/clientpositive/erasurecoding/erasure_simple.q.out 
01f6015a346c1e4283fd6a8cf1eaa3b670450e20 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/common/StatsSetupConst.java
 78ea01d9687fe043d63441430c46b30c25cd9756 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
 77ed2b4de4569fa8aca23b16f2b362b187c7c4fc 
  
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java
 9b36d09eb9fb332e913d442bb476628eca334b6e 
  
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/utils/TestMetaStoreUtils.java
 55ff1502d415dea52095cfdd523d01f1e49ce084 


Diff: https://reviews.apache.org/r/67468/diff/1/


Testing
---

Ran driver tests and new jdbc test


Thanks,

Andrew Sherman