[jira] [Created] (HIVE-17175) Improve desc formatted for bitvectors

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-17175:
--

 Summary: Improve desc formatted for bitvectors
 Key: HIVE-17175
 URL: https://issues.apache.org/jira/browse/HIVE-17175
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 61009: Extend object store to store bit vectors

[pengcheng xiong]

/varchar_udf1.q.out 2e9d88e343 
  ql/src/test/results/clientpositive/llap/vector_udf1.q.out 9a164fe130 
  ql/src/test/results/clientpositive/partial_column_stats.q.out 87d47dae22 
  ql/src/test/results/clientpositive/partition_coltype_literals.q.out 
d459b36ff0 
  
ql/src/test/results/clientpositive/reduceSinkDeDuplication_pRS_key_empty.q.out 
4bddd3bef8 
  ql/src/test/results/clientpositive/rename_external_partition_location.q.out 
19546c38bc 
  ql/src/test/results/clientpositive/rename_table_update_column_stats.q.out 
16b3a38c46 
  ql/src/test/results/clientpositive/spark/avro_decimal_native.q.out b73b5f5679 
  ql/src/test/results/clientpositive/spark/stats_only_null.q.out 359eea3acb 
  ql/src/test/results/clientpositive/stats_only_null.q.out 88c2114356 
  ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out 
ad92058cab 
  ql/src/test/results/clientpositive/tez/explainanalyze_5.q.out 626e1fd4d0 
  ql/src/test/results/clientpositive/tunable_ndv.q.out 437beafc0d 


Diff: https://reviews.apache.org/r/61009/diff/5/

Changes: https://reviews.apache.org/r/61009/diff/4-5/


Testing
---


Thanks,

pengcheng xiong

Re: Review Request 61009: Extend object store to store bit vectors

[pengcheng xiong]

/clientpositive/partial_column_stats.q.out 87d47dae22 
  ql/src/test/results/clientpositive/partition_coltype_literals.q.out 
d459b36ff0 
  
ql/src/test/results/clientpositive/reduceSinkDeDuplication_pRS_key_empty.q.out 
4bddd3bef8 
  ql/src/test/results/clientpositive/rename_external_partition_location.q.out 
19546c38bc 
  ql/src/test/results/clientpositive/rename_table_update_column_stats.q.out 
16b3a38c46 
  ql/src/test/results/clientpositive/spark/avro_decimal_native.q.out b73b5f5679 
  ql/src/test/results/clientpositive/spark/stats_only_null.q.out 359eea3acb 
  ql/src/test/results/clientpositive/stats_only_null.q.out 88c2114356 
  ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out 
ad92058cab 
  ql/src/test/results/clientpositive/tez/explainanalyze_5.q.out 626e1fd4d0 
  ql/src/test/results/clientpositive/tunable_ndv.q.out 437beafc0d 


Diff: https://reviews.apache.org/r/61009/diff/4/

Changes: https://reviews.apache.org/r/61009/diff/3-4/


Testing
---


Thanks,

pengcheng xiong

Re: Review Request 61009: Extend object store to store bit vectors

[pengcheng xiong]

> On July 21, 2017, 10:45 p.m., Ashutosh Chauhan wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
> > Lines 1507-1508 (original)
> > <https://reviews.apache.org/r/61009/diff/1/?file=1780388#file1780388line1510>
> >
> > Seems like we are going to loose this extrapolation logic.

it is back in the new version.


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61009/#review181146
---


On July 24, 2017, 6:08 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61009/
> ---
> 
> (Updated July 24, 2017, 6:08 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16997
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/ndv/FMSketch.java e20d29954a 
>   
> common/src/java/org/apache/hadoop/hive/common/ndv/NumDistinctValueEstimatorFactory.java
>  e810ac5487 
>   common/src/java/org/apache/hadoop/hive/common/ndv/fm/FMSketchUtils.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/ndv/hll/HyperLogLog.java 
> d1955468a6 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java df45f2cc32 
>   
> common/src/test/org/apache/hadoop/hive/common/ndv/fm/TestFMSketchSerialization.java
>  PRE-CREATION 
>   metastore/scripts/upgrade/derby/044-HIVE-16997.derby.sql PRE-CREATION 
>   metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql a9a532906f 
>   metastore/scripts/upgrade/derby/upgrade-2.3.0-to-3.0.0.derby.sql 30513dc882 
>   metastore/scripts/upgrade/mssql/029-HIVE-16997.mssql.sql PRE-CREATION 
>   metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql 1cfe2d1b2d 
>   metastore/scripts/upgrade/mssql/upgrade-2.3.0-to-3.0.0.mssql.sql 5683254b04 
>   metastore/scripts/upgrade/mysql/044-HIVE-16997.mysql.sql PRE-CREATION 
>   metastore/scripts/upgrade/mysql/hive-schema-3.0.0.mysql.sql 97d881f263 
>   metastore/scripts/upgrade/mysql/upgrade-2.3.0-to-3.0.0.mysql.sql ba62939809 
>   metastore/scripts/upgrade/oracle/044-HIVE-16997.oracle.sql PRE-CREATION 
>   metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql 8fdb552367 
>   metastore/scripts/upgrade/oracle/upgrade-2.3.0-to-3.0.0.oracle.sql 
> 0a70d47cca 
>   metastore/scripts/upgrade/postgres/043-HIVE-16997.postgres.sql PRE-CREATION 
>   metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql 
> 1cdeb6b45a 
>   metastore/scripts/upgrade/postgres/upgrade-2.3.0-to-3.0.0.postgres.sql 
> c44dd067fc 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
> a960b2d26b 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> b52c94c9fb 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> db4ec91cdb 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java 
> 2dc2804343 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 
> 3ac4fe1604 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/DateColumnStatsAggregator.java
>  PRE-CREATION 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/StringColumnStatsAggregator.java
>  PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/StatsCache.java 
> 0e119896a5 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BinaryColumnStatsAggregator.java
>  d81d612e92 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BooleanColumnStatsAggregator.java
>  e796df2422 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregator.java
>  29a05390bf 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregatorFactory.java
>  568bf0609b 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DecimalColumnStatsAggregator.java
>  8eb64e0143 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DoubleColumnStatsAggregator.java
>  b6b86123b2 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/IExtrapolatePartStatus.java
>  af75bced72 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/LongColumnStatsAggregator.java
>  2da6f60167 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/StringColumnS

Re: Review Request 61009: Extend object store to store bit vectors

[pengcheng xiong]

 
d459b36ff0 
  
ql/src/test/results/clientpositive/reduceSinkDeDuplication_pRS_key_empty.q.out 
4bddd3bef8 
  ql/src/test/results/clientpositive/rename_external_partition_location.q.out 
19546c38bc 
  ql/src/test/results/clientpositive/rename_table_update_column_stats.q.out 
16b3a38c46 
  ql/src/test/results/clientpositive/spark/avro_decimal_native.q.out b73b5f5679 
  ql/src/test/results/clientpositive/spark/stats_only_null.q.out 359eea3acb 
  ql/src/test/results/clientpositive/stats_only_null.q.out 88c2114356 
  ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out 
ad92058cab 
  ql/src/test/results/clientpositive/tez/explainanalyze_5.q.out 626e1fd4d0 
  ql/src/test/results/clientpositive/tunable_ndv.q.out 437beafc0d 


Diff: https://reviews.apache.org/r/61009/diff/3/

Changes: https://reviews.apache.org/r/61009/diff/2-3/


Testing
---


Thanks,

pengcheng xiong

Re: Review Request 61009: Extend object store to store bit vectors

[pengcheng xiong]

> On July 23, 2017, 4:58 p.m., Ashutosh Chauhan wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java
> > Lines 1846 (patched)
> > <https://reviews.apache.org/r/61009/diff/2/?file=1780981#file1780981line1846>
> >
> > This cast of null is not necessary.

it will throw exception if i just use null for derby. It seems that derby 
expects a type for null.


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61009/#review181182
---


On July 22, 2017, 9:19 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61009/
> ---
> 
> (Updated July 22, 2017, 9:19 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16997
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/ndv/FMSketch.java e20d29954a 
>   
> common/src/java/org/apache/hadoop/hive/common/ndv/NumDistinctValueEstimatorFactory.java
>  e810ac5487 
>   common/src/java/org/apache/hadoop/hive/common/ndv/fm/FMSketchUtils.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/ndv/hll/HyperLogLog.java 
> d1955468a6 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java df45f2cc32 
>   
> common/src/test/org/apache/hadoop/hive/common/ndv/fm/TestFMSketchSerialization.java
>  PRE-CREATION 
>   metastore/scripts/upgrade/derby/044-HIVE-16997.derby.sql PRE-CREATION 
>   metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql a9a532906f 
>   metastore/scripts/upgrade/derby/upgrade-2.3.0-to-3.0.0.derby.sql 30513dc882 
>   metastore/scripts/upgrade/mssql/029-HIVE-16997.mssql.sql PRE-CREATION 
>   metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql 1cfe2d1b2d 
>   metastore/scripts/upgrade/mssql/upgrade-2.3.0-to-3.0.0.mssql.sql 5683254b04 
>   metastore/scripts/upgrade/mysql/044-HIVE-16997.mysql.sql PRE-CREATION 
>   metastore/scripts/upgrade/mysql/hive-schema-3.0.0.mysql.sql 97d881f263 
>   metastore/scripts/upgrade/mysql/upgrade-2.3.0-to-3.0.0.mysql.sql ba62939809 
>   metastore/scripts/upgrade/oracle/044-HIVE-16997.oracle.sql PRE-CREATION 
>   metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql 8fdb552367 
>   metastore/scripts/upgrade/oracle/upgrade-2.3.0-to-3.0.0.oracle.sql 
> 0a70d47cca 
>   metastore/scripts/upgrade/postgres/043-HIVE-16997.postgres.sql PRE-CREATION 
>   metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql 
> 1cdeb6b45a 
>   metastore/scripts/upgrade/postgres/upgrade-2.3.0-to-3.0.0.postgres.sql 
> c44dd067fc 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
> a960b2d26b 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> b52c94c9fb 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> db4ec91cdb 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java 
> 2dc2804343 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 
> 3ac4fe1604 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/DateColumnStatsAggregator.java
>  PRE-CREATION 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/StringColumnStatsAggregator.java
>  PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/StatsCache.java 
> 0e119896a5 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BinaryColumnStatsAggregator.java
>  d81d612e92 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BooleanColumnStatsAggregator.java
>  e796df2422 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregator.java
>  29a05390bf 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregatorFactory.java
>  568bf0609b 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DecimalColumnStatsAggregator.java
>  8eb64e0143 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DoubleColumnStatsAggregator.java
>  b6b86123b2 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/IExtrapolatePartStatus.java
>  af75bced72 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/LongColumnStatsAggregator.java
>  2da6f60167 
>   
> metastore/src/java/org/apache/hadoop/hi

Re: Review Request 61009: Extend object store to store bit vectors

[pengcheng xiong]

/src/test/results/clientpositive/rename_table_update_column_stats.q.out 
16b3a38c46 
  ql/src/test/results/clientpositive/spark/avro_decimal_native.q.out b73b5f5679 
  ql/src/test/results/clientpositive/spark/stats_only_null.q.out 359eea3acb 
  ql/src/test/results/clientpositive/stats_only_null.q.out 88c2114356 
  ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out 
ad92058cab 
  ql/src/test/results/clientpositive/tez/explainanalyze_5.q.out 626e1fd4d0 
  ql/src/test/results/clientpositive/tunable_ndv.q.out 437beafc0d 


Diff: https://reviews.apache.org/r/61009/diff/2/

Changes: https://reviews.apache.org/r/61009/diff/1-2/


Testing
---


Thanks,

pengcheng xiong

Re: Review Request 61009: Extend object store to store bit vectors

[pengcheng xiong]

> On July 21, 2017, 10:45 p.m., Ashutosh Chauhan wrote:
> > common/src/java/org/apache/hadoop/hive/common/ndv/NumDistinctValueEstimatorFactory.java
> > Lines 38 (patched)
> > <https://reviews.apache.org/r/61009/diff/1/?file=1780367#file1780367line38>
> >
> > Whats the reason for Base64 encoding. All databases we use support 
> > binary type where we can store bit vector as is. No need to use varchar.

because UDAF compute stats is using text and now the merge and store of 
bitvector are all using text rather than bytes.


> On July 21, 2017, 10:45 p.m., Ashutosh Chauhan wrote:
> > common/src/java/org/apache/hadoop/hive/common/ndv/fm/FMSketchUtils.java
> > Lines 89 (patched)
> > <https://reviews.apache.org/r/61009/diff/1/?file=1780368#file1780368line89>
> >
> > Caller made this check. No advantage of redoing it.

Just in case it will be called separately, like HLL.


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/61009/#review181146
---


On July 20, 2017, 10:41 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/61009/
> ---
> 
> (Updated July 20, 2017, 10:41 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16997
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/common/ndv/FMSketch.java e20d29954a 
>   
> common/src/java/org/apache/hadoop/hive/common/ndv/NumDistinctValueEstimatorFactory.java
>  e810ac5487 
>   common/src/java/org/apache/hadoop/hive/common/ndv/fm/FMSketchUtils.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/ndv/hll/HyperLogLog.java 
> d1955468a6 
>   
> common/src/test/org/apache/hadoop/hive/common/ndv/fm/TestFMSketchSerialization.java
>  PRE-CREATION 
>   metastore/scripts/upgrade/derby/044-HIVE-16997.derby.sql PRE-CREATION 
>   metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql a9a532906f 
>   metastore/scripts/upgrade/derby/upgrade-2.3.0-to-3.0.0.derby.sql 30513dc882 
>   metastore/scripts/upgrade/mssql/029-HIVE-16997.mssql.sql PRE-CREATION 
>   metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql 1cfe2d1b2d 
>   metastore/scripts/upgrade/mssql/upgrade-2.3.0-to-3.0.0.mssql.sql 5683254b04 
>   metastore/scripts/upgrade/mysql/044-HIVE-16997.mysql.sql PRE-CREATION 
>   metastore/scripts/upgrade/mysql/hive-schema-3.0.0.mysql.sql 97d881f263 
>   metastore/scripts/upgrade/mysql/upgrade-2.3.0-to-3.0.0.mysql.sql ba62939809 
>   metastore/scripts/upgrade/oracle/044-HIVE-16997.oracle.sql PRE-CREATION 
>   metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql 8fdb552367 
>   metastore/scripts/upgrade/oracle/upgrade-2.3.0-to-3.0.0.oracle.sql 
> 0a70d47cca 
>   metastore/scripts/upgrade/postgres/043-HIVE-16997.postgres.sql PRE-CREATION 
>   metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql 
> 1cdeb6b45a 
>   metastore/scripts/upgrade/postgres/upgrade-2.3.0-to-3.0.0.postgres.sql 
> c44dd067fc 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/IExtrapolatePartStatus.java
>  d0569fb8d8 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/LinearExtrapolatePartStatus.java
>  f4e5ef7045 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 
> a960b2d26b 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> b52c94c9fb 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/StatObjectConverter.java 
> 2dc2804343 
>   metastore/src/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java 
> 3ac4fe1604 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/DateColumnStatsAggregator.java
>  PRE-CREATION 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/columnstats/aggr/StringColumnStatsAggregator.java
>  PRE-CREATION 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/StatsCache.java 
> 0e119896a5 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BinaryColumnStatsAggregator.java
>  d81d612e92 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/BooleanColumnStatsAggregator.java
>  e796df2422 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregator.java
>  29a05390bf 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAg

Review Request 61009: Extend object store to store bit vectors

[pengcheng xiong]

/clientpositive/spark/stats_only_null.q.out 359eea3acb 
  ql/src/test/results/clientpositive/stats_only_null.q.out 88c2114356 
  ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out 
ad92058cab 
  ql/src/test/results/clientpositive/tez/explainanalyze_5.q.out 626e1fd4d0 
  ql/src/test/results/clientpositive/tunable_ndv.q.out 437beafc0d 


Diff: https://reviews.apache.org/r/61009/diff/1/


Testing
---


Thanks,

pengcheng xiong

[jira] [Created] (HIVE-17137) Fix javolution conflict

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-17137:
--

 Summary: Fix javolution conflict
 Key: HIVE-17137
 URL: https://issues.apache.org/jira/browse/HIVE-17137
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong


as reported by [~jcamachorodriguez]
{code}
[WARNING] Some problems were encountered while building the effective model for 
org.apache.hive:hive-exec:jar:3.0.0-SNAPSHOT
[WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must 
be unique: javolution:javolution:jar -> duplicate declaration of version 
${javolution.version} @ org.apache.hive:hive-exec:[unknown-version], 
/grid/5/dev/jcamachorodriguez/dist/tez-autobuild/hive/ql/pom.xml, line 366, 
column 17
{code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[ANNOUNCE] Apache Hive 2.3.0 Released

[Pengcheng Xiong]

The Apache Hive team is proud to announce the release of Apache Hive
version 2.3.0.

The Apache Hive (TM) data warehouse software facilitates querying and
managing large datasets residing in distributed storage. Built on top
of Apache Hadoop (TM), it provides, among others:

* Tools to enable easy data extract/transform/load (ETL)

* A mechanism to impose structure on a variety of data formats

* Access to files stored either directly in Apache HDFS (TM) or in other
  data storage systems such as Apache HBase (TM)

* Query execution via Apache Hadoop MapReduce and Apache Tez frameworks.

For Hive release details and downloads, please
visit:https://hive.apache.org/downloads.html

Hive 2.3.0 Release Notes are available here:


https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12340269&styleName=Text&projectId=12310843

We would like to thank the many contributors who made this release
possible.

Regards,

The Apache Hive Team

Re: [VOTE] Apache Hive 2.3.0 Release Candidate 1

[Pengcheng Xiong]

I believe Owen and I checked every one of them and set it to correct fix
version. Thanks.

On Sat, Jul 15, 2017 at 8:28 PM, Lefty Leverenz 
wrote:

> Is the list of jiras compiled from github or from the "Fix Version/s" field
> in JIRA?
>
> If it's a Fix Version/s list, it could be incomplete because many jiras
> were originally set to fix version 2.2.0 and ended up in branch-2.3 but not
> branch-2.2.  Some of them have been updated, but probably not all.
>
> -- Lefty
>
>
> On Fri, Jul 14, 2017 at 5:47 PM, Gunther Hagleitner <
> ghagleit...@hortonworks.com> wrote:
>
> > I'm +1. Checked signature and checksum, verified binaries, compiled, ran
> > some tests and queries - didn't find any issues.
> >
> > The concern seems to be that we don't release often enough - rebasing the
> > branch and re-stabilizing would just delay having a release further. We
> can
> > always put a new one together, rather than scrapping this one.
> >
> > Thanks,
> > Gunther.
> > 
> > From: Eugene Koifman 
> > Sent: Friday, July 14, 2017 10:18 AM
> > To: dev@hive.apache.org
> > Subject: Re: [VOTE] Apache Hive 2.3.0 Release Candidate 1
> >
> > Another issue is that the branch for 2.3 was cut 2 or 3 month ago.  Given
> > how infrequently releases are made, would it not make more sense to recut
> > it at this point?
> >
> > On 7/14/17, 8:20 AM, "Sergio Pena"  wrote:
> >
> > What happened with the 2.2.0 release? is it confusing to jump from
> 2.1
> > ->
> > 2.3 without having a 2.2 release previously?
> >
> > On Fri, Jul 14, 2017 at 2:02 AM, Pengcheng Xiong 
> > wrote:
> >
> > > Apache Hive 2.3.0 Release Candidate 1 is available here:
> > >
> > > Artifacts:
> > > tag: *https://github.com/apache/hive/releases/tag/release-2.3.
> 0-rc1
> > > <https://github.com/apache/hive/releases/tag/release-2.3.0-rc1>*
> > > tar ball: http://home.apache.org/~pxiong/apache-hive-2.3.0
> > > <http://home.apache.org/~pxiong/hive-storage-2.4.0/>
> > >
> > > Voting will conclude in 72 hours.
> > >
> > > Hive PMC Members: Please test and vote.
> > >
> > > Here is my +1 after running rat check, md5 check and simple
> queries.
> > >
> > > Thanks.
> > >
> > > Best
> > > Pengcheng
> > >
> >
> >
> >
> >
> >
>

Re: Review Request 60753: Add HLL as an alternative to FM sketch to compute stats

[pengcheng xiong]

/clientpositive/llap/vectorization_short_regress.q.out 
00577620d8 
  ql/src/test/results/clientpositive/llap/vectorized_distinct_gby.q.out 
c3e5f7c90d 
  
ql/src/test/results/clientpositive/llap/vectorized_dynamic_semijoin_reduction.q.out
 9a1c44c3e6 
  
ql/src/test/results/clientpositive/llap/vectorized_dynamic_semijoin_reduction2.q.out
 a03466f859 
  ql/src/test/results/clientpositive/llap/vectorized_mapjoin.q.out 8160bc7c44 
  ql/src/test/results/clientpositive/llap/vectorized_multi_output_select.q.out 
f744eb6513 
  ql/src/test/results/clientpositive/llap/vectorized_nested_mapjoin.q.out 
28a8340a9c 
  ql/src/test/results/clientpositive/llap/vectorized_shufflejoin.q.out 
73ab9fca82 
  ql/src/test/results/clientpositive/llap/windowing_gby.q.out 2c47b8b2a6 
  ql/src/test/results/clientpositive/parallel_colstats.q.out c85113137b 
  ql/src/test/results/clientpositive/partial_column_stats.q.out 5876efacf3 
  ql/src/test/results/clientpositive/partition_coltype_literals.q.out 
3505556029 
  
ql/src/test/results/clientpositive/reduceSinkDeDuplication_pRS_key_empty.q.out 
3ad09e815c 
  ql/src/test/results/clientpositive/remove_exprs_stats.q.out 33cf90ae9d 
  ql/src/test/results/clientpositive/rename_external_partition_location.q.out 
ec4076f908 
  ql/src/test/results/clientpositive/rename_table_update_column_stats.q.out 
b3d6f039ac 
  ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out 
bd024a7ab1 
  ql/src/test/results/clientpositive/tez/explainanalyze_1.q.out 660ed7 
  ql/src/test/results/clientpositive/tez/explainanalyze_2.q.out 0916565f0f 
  ql/src/test/results/clientpositive/tez/explainanalyze_3.q.out e5c8d6c51e 
  ql/src/test/results/clientpositive/tez/explainanalyze_4.q.out 9fbe8c5263 
  ql/src/test/results/clientpositive/tez/explainanalyze_5.q.out b35e294813 
  ql/src/test/results/clientpositive/tez/explainuser_3.q.out 65c9114b20 
  ql/src/test/results/clientpositive/tez/hybridgrace_hashjoin_1.q.out 
0a1e039cf1 
  ql/src/test/results/clientpositive/tez/hybridgrace_hashjoin_2.q.out 
6f5a3a96ca 
  ql/src/test/results/clientpositive/tez/vectorization_limit.q.out afcae8c34c 
  ql/src/test/results/clientpositive/tunable_ndv.q.out 6ae54b4927 


Diff: https://reviews.apache.org/r/60753/diff/4/

Changes: https://reviews.apache.org/r/60753/diff/3-4/


Testing
---


Thanks,

pengcheng xiong

Re: Review Request 60753: Add HLL as an alternative to FM sketch to compute stats

[pengcheng xiong]

> On July 14, 2017, 12:55 a.m., Ashutosh Chauhan wrote:
> > common/src/java/org/apache/hadoop/hive/common/ndv/NumDistinctValueEstimatorFactory.java
> > Lines 30-34 (patched)
> > <https://reviews.apache.org/r/60753/diff/2/?file=1775584#file1775584line30>
> >
> > As discussed this determination should happen using first byte of 
> > serialized string.

That is true. But I would prefer to do it in the next following up jira as this 
will totally change the serde of hll as well as fm. And, as I analyzed, I may 
need to do bitpacking for fm sketch in order to fit 64k limit for varchar in 
the worst case.


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60753/#review180494
-------


On July 14, 2017, 12:08 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60753/
> ---
> 
> (Updated July 14, 2017, 12:08 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16966
> 
> 
> Diffs
> -
> 
>   common/pom.xml 023f084511 
>   
> common/src/java/org/apache/hadoop/hive/common/ndv/NumDistinctValueEstimatorFactory.java
>  PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/ndv/hll/HLLConstants.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/ndv/hll/HLLDenseRegister.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/ndv/hll/HLLRegister.java 
> PRE-CREATION 
>   
> common/src/java/org/apache/hadoop/hive/common/ndv/hll/HLLSparseRegister.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/ndv/hll/HyperLogLog.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/ndv/hll/HyperLogLogUtils.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java c7afe2bc4a 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/NumDistinctValueEstimator.java
>  92f9a845e3 
>   metastore/src/java/org/apache/hadoop/hive/metastore/hbase/StatsCache.java 
> 18f8afc9ad 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregator.java
>  31955b4363 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/ColumnStatsAggregatorFactory.java
>  daf85692eb 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DecimalColumnStatsAggregator.java
>  36b2c9c56b 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DoubleColumnStatsAggregator.java
>  a88ef84e5c 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/LongColumnStatsAggregator.java
>  8ac6561aec 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/StringColumnStatsAggregator.java
>  2aa4046a46 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMerger.java
>  33c7e3e52c 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java
>  fe890e4e27 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java
>  3179b23438 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DecimalColumnStatsMerger.java
>  c13add9d9c 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DoubleColumnStatsMerger.java
>  fbdba24b0a 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/LongColumnStatsMerger.java
>  ac65590505 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/StringColumnStatsMerger.java
>  41587477d3 
>   
> metastore/src/test/org/apache/hadoop/hive/metastore/hbase/TestHBaseAggregateStatsNDVUniformDist.java
>  87b1ac870d 
>   pom.xml 32f5fd1493 
>   
> ql/src/java/org/apache/hadoop/hive/ql/parse/ColumnStatsSemanticAnalyzer.java 
> 0a5cf00c44 
>   ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUtils.java 76f7daeb1b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/DecimalNumDistinctValueEstimator.java
>  a05906edfa 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/DoubleNumDistinctValueEstimator.java
>  e76fc74dbc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java
>  2ebfcb2360 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/LongNumDistinctValueEstimator.java
>  1c197a028a 

[VOTE] Apache Hive 2.3.0 Release Candidate 1

[Pengcheng Xiong]

Apache Hive 2.3.0 Release Candidate 1 is available here:

Artifacts:
tag: *https://github.com/apache/hive/releases/tag/release-2.3.0-rc1
*
tar ball: http://home.apache.org/~pxiong/apache-hive-2.3.0


Voting will conclude in 72 hours.

Hive PMC Members: Please test and vote.

Here is my +1 after running rat check, md5 check and simple queries.

Thanks.

Best
Pengcheng

Re: [VOTE] storage-release-2.4.0rc1

[Pengcheng Xiong]

With three +1's the vote passes after 72 hours. Thanks Alan and Ashutosh!

Pengcheng


On Wed, Jul 12, 2017 at 12:56 PM, Pengcheng Xiong  wrote:

> Oh, thanks for the reminder! OK, let's continue the voting process until
> July 13 noon. Thanks!
>
> On Wed, Jul 12, 2017 at 11:56 AM, Alan Gates  wrote:
>
>> I believe we have to wait at least 72 hours from the start of the vote so
>> that everyone has a chance to vote.
>>
>> Alan.
>>
>> On Wed, Jul 12, 2017 at 11:11 AM, Pengcheng Xiong 
>> wrote:
>>
>> > With three +1's the vote passes. Thanks Alan and Ashutosh!
>> >
>> > Pengcheng
>> >
>> >
>> > On Wed, Jul 12, 2017 at 11:08 AM, Ashutosh Chauhan <
>> hashut...@apache.org>
>> > wrote:
>> >
>> > > +1
>> > > Did a build and some basic testing.
>> > >
>> > > On Tue, Jul 11, 2017 at 1:36 PM, Alan Gates 
>> > wrote:
>> > >
>> > > > +1.  Checked the signatures, did a build with a fresh maven repo,
>> ran
>> > > rat.
>> > > >
>> > > > On Mon, Jul 10, 2017 at 11:58 PM, Pengcheng Xiong <
>> pxi...@apache.org>
>> > > > wrote:
>> > > >
>> > > > > Hi all,
>> > > > >
>> > > > >Here is a new release candidate for hive-storage-2.4.0 which
>> Hive
>> > > 2.3
>> > > > > will depend on. Please take a look and vote. Thanks.
>> > > > >
>> > > > > Here is my +1 after testing it with Hive 2.3 branch.
>> > > > >
>> > > > > Artifacts:
>> > > > > tag: https://github.com/apache/hive/releases/tag/storage-
>> > > > release-2.4.0rc1
>> > > > > tar ball: http://home.apache.org/~pxiong/hive-storage-2.4.0/
>> > > > >
>> > > > > Thanks!
>> > > > >
>> > > > > Best
>> > > > > Pengcheng
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

[jira] [Created] (HIVE-17096) Fix test failures in 2.3 branch

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-17096:
--

 Summary: Fix test failures in 2.3 branch
 Key: HIVE-17096
 URL: https://issues.apache.org/jira/browse/HIVE-17096
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 60753: Add HLL as an alternative to FM sketch to compute stats

[pengcheng xiong]

/vectorized_nested_mapjoin.q.out 
28a8340a9c 
  ql/src/test/results/clientpositive/llap/vectorized_shufflejoin.q.out 
73ab9fca82 
  ql/src/test/results/clientpositive/llap/windowing_gby.q.out 2c47b8b2a6 
  ql/src/test/results/clientpositive/parallel_colstats.q.out c85113137b 
  ql/src/test/results/clientpositive/partial_column_stats.q.out 5876efacf3 
  ql/src/test/results/clientpositive/partition_coltype_literals.q.out 
3505556029 
  
ql/src/test/results/clientpositive/reduceSinkDeDuplication_pRS_key_empty.q.out 
3ad09e815c 
  ql/src/test/results/clientpositive/remove_exprs_stats.q.out 33cf90ae9d 
  ql/src/test/results/clientpositive/rename_external_partition_location.q.out 
ec4076f908 
  ql/src/test/results/clientpositive/rename_table_update_column_stats.q.out 
b3d6f039ac 
  ql/src/test/results/clientpositive/temp_table_display_colstats_tbllvl.q.out 
bd024a7ab1 
  ql/src/test/results/clientpositive/tez/explainanalyze_1.q.out 660ed7 
  ql/src/test/results/clientpositive/tez/explainanalyze_2.q.out 0916565f0f 
  ql/src/test/results/clientpositive/tez/explainanalyze_3.q.out e5c8d6c51e 
  ql/src/test/results/clientpositive/tez/explainanalyze_4.q.out 9fbe8c5263 
  ql/src/test/results/clientpositive/tez/explainanalyze_5.q.out b35e294813 
  ql/src/test/results/clientpositive/tez/explainuser_3.q.out 65c9114b20 
  ql/src/test/results/clientpositive/tez/hybridgrace_hashjoin_1.q.out 
0a1e039cf1 
  ql/src/test/results/clientpositive/tez/hybridgrace_hashjoin_2.q.out 
6f5a3a96ca 
  ql/src/test/results/clientpositive/tez/vectorization_limit.q.out afcae8c34c 
  ql/src/test/results/clientpositive/tunable_ndv.q.out 6ae54b4927 


Diff: https://reviews.apache.org/r/60753/diff/3/

Changes: https://reviews.apache.org/r/60753/diff/2-3/


Testing
---


Thanks,

pengcheng xiong

Re: Review Request 60753: Add HLL as an alternative to FM sketch to compute stats

[pengcheng xiong]

/src/test/results/clientpositive/tez/explainuser_3.q.out 65c9114b20 
  ql/src/test/results/clientpositive/tez/hybridgrace_hashjoin_1.q.out 
0a1e039cf1 
  ql/src/test/results/clientpositive/tez/hybridgrace_hashjoin_2.q.out 
6f5a3a96ca 
  ql/src/test/results/clientpositive/tez/vectorization_limit.q.out afcae8c34c 
  ql/src/test/results/clientpositive/tunable_ndv.q.out 6ae54b4927 


Diff: https://reviews.apache.org/r/60753/diff/2/

Changes: https://reviews.apache.org/r/60753/diff/1-2/


Testing
---


Thanks,

pengcheng xiong

Re: [VOTE] storage-release-2.4.0rc1

[Pengcheng Xiong]

Oh, thanks for the reminder! OK, let's continue the voting process until
July 13 noon. Thanks!

On Wed, Jul 12, 2017 at 11:56 AM, Alan Gates  wrote:

> I believe we have to wait at least 72 hours from the start of the vote so
> that everyone has a chance to vote.
>
> Alan.
>
> On Wed, Jul 12, 2017 at 11:11 AM, Pengcheng Xiong 
> wrote:
>
> > With three +1's the vote passes. Thanks Alan and Ashutosh!
> >
> > Pengcheng
> >
> >
> > On Wed, Jul 12, 2017 at 11:08 AM, Ashutosh Chauhan  >
> > wrote:
> >
> > > +1
> > > Did a build and some basic testing.
> > >
> > > On Tue, Jul 11, 2017 at 1:36 PM, Alan Gates 
> > wrote:
> > >
> > > > +1.  Checked the signatures, did a build with a fresh maven repo, ran
> > > rat.
> > > >
> > > > On Mon, Jul 10, 2017 at 11:58 PM, Pengcheng Xiong  >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > >Here is a new release candidate for hive-storage-2.4.0 which
> Hive
> > > 2.3
> > > > > will depend on. Please take a look and vote. Thanks.
> > > > >
> > > > > Here is my +1 after testing it with Hive 2.3 branch.
> > > > >
> > > > > Artifacts:
> > > > > tag: https://github.com/apache/hive/releases/tag/storage-
> > > > release-2.4.0rc1
> > > > > tar ball: http://home.apache.org/~pxiong/hive-storage-2.4.0/
> > > > >
> > > > > Thanks!
> > > > >
> > > > > Best
> > > > > Pengcheng
> > > > >
> > > >
> > >
> >
>

Re: [VOTE] storage-release-2.4.0rc1

[Pengcheng Xiong]

With three +1's the vote passes. Thanks Alan and Ashutosh!

Pengcheng


On Wed, Jul 12, 2017 at 11:08 AM, Ashutosh Chauhan 
wrote:

> +1
> Did a build and some basic testing.
>
> On Tue, Jul 11, 2017 at 1:36 PM, Alan Gates  wrote:
>
> > +1.  Checked the signatures, did a build with a fresh maven repo, ran
> rat.
> >
> > On Mon, Jul 10, 2017 at 11:58 PM, Pengcheng Xiong 
> > wrote:
> >
> > > Hi all,
> > >
> > >Here is a new release candidate for hive-storage-2.4.0 which Hive
> 2.3
> > > will depend on. Please take a look and vote. Thanks.
> > >
> > > Here is my +1 after testing it with Hive 2.3 branch.
> > >
> > > Artifacts:
> > > tag: https://github.com/apache/hive/releases/tag/storage-
> > release-2.4.0rc1
> > > tar ball: http://home.apache.org/~pxiong/hive-storage-2.4.0/
> > >
> > > Thanks!
> > >
> > > Best
> > > Pengcheng
> > >
> >
>

Re: Review Request 60753: Add HLL as an alternative to FM sketch to compute stats

[pengcheng xiong]

> On July 10, 2017, 10:02 p.m., Prasanth_J wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> > Line 1723 (original), 1723 (patched)
> > <https://reviews.apache.org/r/60753/diff/1/?file=1773281#file1773281line1723>
> >
> > I am not sure if we need this config. 
> > Any reason to support storing FM sketch's bitvectors?
> > 
> > If this is for 3.0.0 release only, we could even remove this config.
> > 
> > If we need to support FM sketch based NDV then having a separate field 
> > in metastore will be better as Ashutosh suggested.
> 
> pengcheng xiong wrote:
> This config is just to give user an option to choose whether to use FM or 
> HLL to COMPUTE ndv. This patch does not change the fact that we do not store 
> bit vectors for both FM and NDV in object store.
> 
> Prasanth_J wrote:
> I don't think it will be useful for user to tune this.
> IMHO, most users won't care a lot about this. This can be used as 
> fallback but this will only add more checks.
> 
> Ashutosh Chauhan wrote:
> I agree with Prasanth. Overloading this config seems hacky. It is leading 
> to confusing code, e.g, NDVEstimator uses number of bit vectors to determine 
> ndv algo, thats confusing. Its confusing for end user too that +ve value from 
> [0,100] is useful for one algo, but -ve value is only to switch over to diff 
> algo. I suggest that we leave this config as is and introduce a new config 
> which dictates which algo is used for ndv computation. This config value is 
> than passed to compute_stats() udaf by ColumnStatsSemanticAnalyzer. 
> GenericUDAFComputeStats than uses this config to determine which algo its 
> using. 
> This config we can store in metastore (either as is or mapped as int) so 
> that we can deserialize the bit vectors correctly when we retrieve them.

I agree it is a bit confusing but then we need to store two extra information 
in a metastore: (1) the algorithm that we use, FMSketch or HLL and (2) the 
number of bitvectors for FMSketch.


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60753/#review180119
---


On July 10, 2017, 9:29 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60753/
> ---
> 
> (Updated July 10, 2017, 9:29 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16966
> 
> 
> Diffs
> -
> 
>   common/pom.xml e6722babd8 
>   common/src/java/org/apache/hadoop/hive/common/HiveStatsUtils.java 
> 7c9d72fbd2 
>   
> common/src/java/org/apache/hadoop/hive/common/ndv/NumDistinctValueEstimator.java
>  PRE-CREATION 
>   
> common/src/java/org/apache/hadoop/hive/common/ndv/NumDistinctValueEstimatorFactory.java
>  PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/ndv/hll/HLLConstants.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/ndv/hll/HLLDenseRegister.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/ndv/hll/HLLRegister.java 
> PRE-CREATION 
>   
> common/src/java/org/apache/hadoop/hive/common/ndv/hll/HLLSparseRegister.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/ndv/hll/HyperLogLog.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/ndv/hll/HyperLogLogUtils.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5700fb9325 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java.orig da48a7ccbd 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/NumDistinctValueEstimator.java
>  92f9a845e3 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DecimalColumnStatsAggregator.java
>  36b2c9c56b 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DoubleColumnStatsAggregator.java
>  a88ef84e5c 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/LongColumnStatsAggregator.java
>  8ac6561aec 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/StringColumnStatsAggregator.java
>  2aa4046a46 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMerger.java
>  33c7e3e52c 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.jav

Re: Review Request 60776: HIVE-16907: "INSERT INTO" overwrite old data when destination table encapsulated by backquote

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60776/#review180224
---




ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java
Lines 346 (patched)
<https://reviews.apache.org/r/60776/#comment255291>

thanks Nemon Lou for discovering this and thanks Bing Li for the patch. 
However, it seems to me that although hive parse "`tdb.t1`" as a whole table 
name in AST, when it really processes it, it treats it as tdb.t1. Can u check 
other db's behavior, e.g., oracle and postgres, mysql for this? I doubt that 
there is a bug for table name when it contains "dot" in current hive.


- pengcheng xiong


On July 11, 2017, 3:51 p.m., Bing Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60776/
> ---
> 
> (Updated July 11, 2017, 3:51 p.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-16907
> https://issues.apache.org/jira/browse/HIVE-16907
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Wrong behavior when the target table is written as `db.table`
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 
> 136e951 
> 
> 
> Diff: https://reviews.apache.org/r/60776/diff/1/
> 
> 
> Testing
> ---
> 
> explain insert into default.tbl3 select * from tbl2;
> explain insert into `default.tbl3` select * from tbl2;
> 
> create table `default.xxx`(col int);
> create table default.yyy (col int);
> 
> 
> Thanks,
> 
> Bing Li
> 
>

[VOTE] storage-release-2.4.0rc1

[Pengcheng Xiong]

Hi all,

   Here is a new release candidate for hive-storage-2.4.0 which Hive 2.3
will depend on. Please take a look and vote. Thanks.

Here is my +1 after testing it with Hive 2.3 branch.

Artifacts:
tag: https://github.com/apache/hive/releases/tag/storage-release-2.4.0rc1
tar ball: http://home.apache.org/~pxiong/hive-storage-2.4.0/

Thanks!

Best
Pengcheng

[jira] [Created] (HIVE-17071) Make hive 2.3 depend on storage-api-2.3

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-17071:
--

 Summary: Make hive 2.3 depend on storage-api-2.3
 Key: HIVE-17071
 URL: https://issues.apache.org/jira/browse/HIVE-17071
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 60753: Add HLL as an alternative to FM sketch to compute stats

[pengcheng xiong]

> On July 10, 2017, 10:02 p.m., Prasanth_J wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> > Line 1723 (original), 1723 (patched)
> > <https://reviews.apache.org/r/60753/diff/1/?file=1773281#file1773281line1723>
> >
> > I am not sure if we need this config. 
> > Any reason to support storing FM sketch's bitvectors?
> > 
> > If this is for 3.0.0 release only, we could even remove this config.
> > 
> > If we need to support FM sketch based NDV then having a separate field 
> > in metastore will be better as Ashutosh suggested.

This config is just to give user an option to choose whether to use FM or HLL 
to COMPUTE ndv. This patch does not change the fact that we do not store bit 
vectors for both FM and NDV in object store.


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60753/#review180119
-------


On July 10, 2017, 9:29 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60753/
> ---
> 
> (Updated July 10, 2017, 9:29 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16966
> 
> 
> Diffs
> -
> 
>   common/pom.xml e6722babd8 
>   common/src/java/org/apache/hadoop/hive/common/HiveStatsUtils.java 
> 7c9d72fbd2 
>   
> common/src/java/org/apache/hadoop/hive/common/ndv/NumDistinctValueEstimator.java
>  PRE-CREATION 
>   
> common/src/java/org/apache/hadoop/hive/common/ndv/NumDistinctValueEstimatorFactory.java
>  PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/ndv/hll/HLLConstants.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/ndv/hll/HLLDenseRegister.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/ndv/hll/HLLRegister.java 
> PRE-CREATION 
>   
> common/src/java/org/apache/hadoop/hive/common/ndv/hll/HLLSparseRegister.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/ndv/hll/HyperLogLog.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/ndv/hll/HyperLogLogUtils.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5700fb9325 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java.orig da48a7ccbd 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/NumDistinctValueEstimator.java
>  92f9a845e3 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DecimalColumnStatsAggregator.java
>  36b2c9c56b 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DoubleColumnStatsAggregator.java
>  a88ef84e5c 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/LongColumnStatsAggregator.java
>  8ac6561aec 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/StringColumnStatsAggregator.java
>  2aa4046a46 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMerger.java
>  33c7e3e52c 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java
>  fe890e4e27 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java
>  3179b23438 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DecimalColumnStatsMerger.java
>  c13add9d9c 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DoubleColumnStatsMerger.java
>  fbdba24b0a 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/LongColumnStatsMerger.java
>  ac65590505 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/StringColumnStatsMerger.java
>  41587477d3 
>   pom.xml f9fae59a5d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/DecimalNumDistinctValueEstimator.java
>  a05906edfa 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/DoubleNumDistinctValueEstimator.java
>  e76fc74dbc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java
>  2ebfcb2360 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/LongNumDistinctValueEstimator.java
>  1c197a028a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NumDistinctValueEstimator.java
>  fa70f49857 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/StringNumDistinctValueEstimator.java
>  601901c163 
>   ql/src/test/queries/clientpositive/hll.q PRE-CREATION 
>   ql/src/test/results/clientpositive/hll.q.out PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/60753/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: Review Request 60753: Add HLL as an alternative to FM sketch to compute stats

[pengcheng xiong]

> On July 10, 2017, 9:42 p.m., Prasanth_J wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
> > Line 1723 (original), 1723 (patched)
> > <https://reviews.apache.org/r/60753/diff/1/?file=1773281#file1773281line1723>
> >
> > Does this mean any positive value will use FM Sketch?
> > If so, how will storing bitvectors in metastore identify if the 
> > serialized representation is HLL or FM sketch?

yes. The following question is a very good question. Now, I am using a try 
catch block. First try to deser with HLL, then deser with FMSketch. Ashutosh 
suggested adding another field in metastore to distinguish it.


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60753/#review180111
---


On July 10, 2017, 9:29 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60753/
> ---
> 
> (Updated July 10, 2017, 9:29 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Prasanth_J.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16966
> 
> 
> Diffs
> -
> 
>   common/pom.xml e6722babd8 
>   common/src/java/org/apache/hadoop/hive/common/HiveStatsUtils.java 
> 7c9d72fbd2 
>   
> common/src/java/org/apache/hadoop/hive/common/ndv/NumDistinctValueEstimator.java
>  PRE-CREATION 
>   
> common/src/java/org/apache/hadoop/hive/common/ndv/NumDistinctValueEstimatorFactory.java
>  PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/ndv/hll/HLLConstants.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/ndv/hll/HLLDenseRegister.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/ndv/hll/HLLRegister.java 
> PRE-CREATION 
>   
> common/src/java/org/apache/hadoop/hive/common/ndv/hll/HLLSparseRegister.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/ndv/hll/HyperLogLog.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/common/ndv/hll/HyperLogLogUtils.java 
> PRE-CREATION 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5700fb9325 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java.orig da48a7ccbd 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/NumDistinctValueEstimator.java
>  92f9a845e3 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DecimalColumnStatsAggregator.java
>  36b2c9c56b 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DoubleColumnStatsAggregator.java
>  a88ef84e5c 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/LongColumnStatsAggregator.java
>  8ac6561aec 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/StringColumnStatsAggregator.java
>  2aa4046a46 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMerger.java
>  33c7e3e52c 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java
>  fe890e4e27 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java
>  3179b23438 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DecimalColumnStatsMerger.java
>  c13add9d9c 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DoubleColumnStatsMerger.java
>  fbdba24b0a 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/LongColumnStatsMerger.java
>  ac65590505 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/StringColumnStatsMerger.java
>  41587477d3 
>   pom.xml f9fae59a5d 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/DecimalNumDistinctValueEstimator.java
>  a05906edfa 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/DoubleNumDistinctValueEstimator.java
>  e76fc74dbc 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java
>  2ebfcb2360 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/LongNumDistinctValueEstimator.java
>  1c197a028a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NumDistinctValueEstimator.java
>  fa70f49857 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/StringNumDistinctValueEstimator.java
>  601901c163 
>   ql/src/test/queries/clientpositive/hll.q PRE-CREATION 
>   ql/src/test/results/clientpositive/hll.q.out PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/60753/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Review Request 60753: Add HLL as an alternative to FM sketch to compute stats

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60753/
---

Review request for hive, Ashutosh Chauhan and Prasanth_J.


Repository: hive-git


Description
---

HIVE-16966


Diffs
-

  common/pom.xml e6722babd8 
  common/src/java/org/apache/hadoop/hive/common/HiveStatsUtils.java 7c9d72fbd2 
  
common/src/java/org/apache/hadoop/hive/common/ndv/NumDistinctValueEstimator.java
 PRE-CREATION 
  
common/src/java/org/apache/hadoop/hive/common/ndv/NumDistinctValueEstimatorFactory.java
 PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/common/ndv/hll/HLLConstants.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/common/ndv/hll/HLLDenseRegister.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/common/ndv/hll/HLLRegister.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/common/ndv/hll/HLLSparseRegister.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/common/ndv/hll/HyperLogLog.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/common/ndv/hll/HyperLogLogUtils.java 
PRE-CREATION 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 5700fb9325 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java.orig da48a7ccbd 
  
metastore/src/java/org/apache/hadoop/hive/metastore/NumDistinctValueEstimator.java
 92f9a845e3 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DecimalColumnStatsAggregator.java
 36b2c9c56b 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/DoubleColumnStatsAggregator.java
 a88ef84e5c 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/LongColumnStatsAggregator.java
 8ac6561aec 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/StringColumnStatsAggregator.java
 2aa4046a46 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMerger.java
 33c7e3e52c 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java
 fe890e4e27 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DateColumnStatsMerger.java
 3179b23438 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DecimalColumnStatsMerger.java
 c13add9d9c 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/DoubleColumnStatsMerger.java
 fbdba24b0a 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/LongColumnStatsMerger.java
 ac65590505 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/StringColumnStatsMerger.java
 41587477d3 
  pom.xml f9fae59a5d 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/DecimalNumDistinctValueEstimator.java
 a05906edfa 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/DoubleNumDistinctValueEstimator.java
 e76fc74dbc 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFComputeStats.java 
2ebfcb2360 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/LongNumDistinctValueEstimator.java
 1c197a028a 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/NumDistinctValueEstimator.java
 fa70f49857 
  
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/StringNumDistinctValueEstimator.java
 601901c163 
  ql/src/test/queries/clientpositive/hll.q PRE-CREATION 
  ql/src/test/results/clientpositive/hll.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/60753/diff/1/


Testing
---


Thanks,

pengcheng xiong

Re: [Announce] New committer: Deepesh Khandelwal

[Pengcheng Xiong]

Congrats!

Best
Pengcheng

On Fri, Jul 7, 2017 at 8:43 PM, Rajesh Balamohan 
wrote:

> Congrats Deepesh!!
>
> On Sat, Jul 8, 2017 at 7:09 AM, Lefty Leverenz 
> wrote:
>
> > Congratulations!
> >
> > -- Lefty
> >
> > On Fri, Jul 7, 2017 at 5:04 PM, Chaoyu Tang  wrote:
> >
> > > Congratulations Deepesh!
> > >
> > > On Fri, Jul 7, 2017 at 4:07 PM, Peter Vary  wrote:
> > >
> > > > Congratulations Deepesh!
> > > >
> > > > > On Jul 7, 2017, at 8:38 PM, Sergio Pena 
> > > > wrote:
> > > > >
> > > > > Congratulations Deepesh !!!
> > > > >
> > > > > On Wed, Jul 5, 2017 at 7:48 PM, Ashutosh Chauhan <
> > hashut...@apache.org
> > > >
> > > > > wrote:
> > > > >
> > > > >> The Project Management Committee (PMC) for Apache Hive has invited
> > > > Deepesh
> > > > >> Khandelwal to become a committer and we are pleased to announce
> that
> > > he
> > > > has
> > > > >> accepted.
> > > > >>
> > > > >> Welcome, Deepesh!
> > > > >>
> > > > >> Thanks,
> > > > >> Ashutosh
> > > > >>
> > > >
> > > >
> > >
> >
>

Re: [Announce] New committer: Sahil Takiar

[Pengcheng Xiong]

Congrats!

Best
Pengcheng

On Fri, Jul 7, 2017 at 8:42 PM, Rajesh Balamohan  wrote:

> Congrats Sahil!!
>
> On Sat, Jul 8, 2017 at 7:07 AM, Lefty Leverenz 
> wrote:
>
> > Congratulations!
> >
> > -- Lefty
> >
> > On Fri, Jul 7, 2017 at 5:05 PM, Chaoyu Tang  wrote:
> >
> > > Congratulations Sahil !!!
> > >
> > > On Fri, Jul 7, 2017 at 2:38 PM, Sergio Pena 
> > > wrote:
> > >
> > > > Congratulations Sahil !!!
> > > >
> > > > On Wed, Jul 5, 2017 at 7:49 PM, Ashutosh Chauhan <
> hashut...@apache.org
> > >
> > > > wrote:
> > > >
> > > > > The Project Management Committee (PMC) for Apache Hive has invited
> > > Sahil
> > > > > Takiar to become a committer and we are pleased to announce that he
> > has
> > > > > accepted.
> > > > >
> > > > > Welcome, Sahil!
> > > > >
> > > > > Thanks,
> > > > > Ashutosh
> > > > >
> > > >
> > >
> >
>
>
>
> --
> ~Rajesh.B
>

Re: [Announce] New committer: Peter Vary

[Pengcheng Xiong]

Congrats!

Best
Pengcheng

On Fri, Jul 7, 2017 at 8:41 PM, Rajesh Balamohan 
wrote:

> Congrats Peter!
>
> On Sat, Jul 8, 2017 at 7:09 AM, Lefty Leverenz 
> wrote:
>
> > Congrats!
> >
> > -- Lefty
> >
> > On Fri, Jul 7, 2017 at 5:04 PM, Chaoyu Tang  wrote:
> >
> > > Congratulations Peter!
> > >
> > > On Fri, Jul 7, 2017 at 3:29 PM, Jimmy Xiang  wrote:
> > >
> > > > Congrats!!
> > > >
> > > > On Fri, Jul 7, 2017 at 11:38 AM, Sergio Pena <
> sergio.p...@cloudera.com
> > >
> > > > wrote:
> > > > > Congrats Peter !!
> > > > >
> > > > > On Fri, Jul 7, 2017 at 12:51 PM, Vaibhav Gumashta <
> > > > vgumas...@hortonworks.com
> > > > >> wrote:
> > > > >
> > > > >> Congratulations Peter!
> > > > >>
> > > > >> On 7/7/17, 10:46 AM, "Vineet Garg"  wrote:
> > > > >>
> > > > >> >Congrats Peter!
> > > > >> >
> > > > >> >> On Jul 7, 2017, at 9:27 AM, Xuefu Zhang 
> wrote:
> > > > >> >>
> > > > >> >> Congratulations!
> > > > >> >>
> > > > >> >> On Fri, Jul 7, 2017 at 4:17 AM, Adam Szita  >
> > > > wrote:
> > > > >> >>
> > > > >> >>> Congrats all!
> > > > >> >>>
> > > > >> >>> On 7 July 2017 at 10:03, Zoltan Haindrich <
> > > > zhaindr...@hortonworks.com>
> > > > >> >>> wrote:
> > > > >> >>>
> > > > >>  Congratulations Peter, Teddy, Deepesh, Vihang and Sahil!
> > > > >>  It's great to see that the Hive community is growing!
> > > > >> 
> > > > >>  On 6 Jul 2017 02:52, Ashutosh Chauhan 
> > > > wrote:
> > > > >>  The Project Management Committee (PMC) for Apache Hive has
> > > invited
> > > > >> Peter
> > > > >>  Vary to become a committer and we are pleased to announce
> that
> > he
> > > > has
> > > > >>  accepted.
> > > > >> 
> > > > >>  Welcome, Peter!
> > > > >> 
> > > > >>  Thanks,
> > > > >>  Ashutosh
> > > > >> 
> > > > >> 
> > > > >> >>>
> > > > >> >
> > > > >>
> > > > >>
> > > >
> > >
> >
>

Re: [Announce] New committer: Teddy Choi

[Pengcheng Xiong]

Congrats!

Best
Pengcheng

On Fri, Jul 7, 2017 at 8:42 PM, Rajesh Balamohan 
wrote:

> Congrats Teddy!
>
> On Sat, Jul 8, 2017 at 7:09 AM, Lefty Leverenz 
> wrote:
>
> > Congrats!
> >
> > -- Lefty
> >
> > On Fri, Jul 7, 2017 at 5:05 PM, Chaoyu Tang  wrote:
> >
> > > Congratulations Teddy!
> > >
> > > On Fri, Jul 7, 2017 at 4:07 PM, Peter Vary  wrote:
> > >
> > > > Congratulatins Teddy!
> > > >
> > > > > On Jul 7, 2017, at 9:28 PM, Jimmy Xiang  wrote:
> > > > >
> > > > > Congrats!!
> > > > >
> > > > > On Fri, Jul 7, 2017 at 11:37 AM, Sergio Pena <
> > sergio.p...@cloudera.com
> > > >
> > > > wrote:
> > > > >> Congratulations Teddy!
> > > > >>
> > > > >> On Fri, Jul 7, 2017 at 12:51 PM, Vaibhav Gumashta <
> > > > vgumas...@hortonworks.com
> > > > >>> wrote:
> > > > >>
> > > > >>> Congratulations Teddy!
> > > > >>>
> > > > >>> On 7/7/17, 10:46 AM, "Vineet Garg" 
> wrote:
> > > > >>>
> > > >  Congratulations!
> > > > 
> > > > > On Jul 7, 2017, at 10:25 AM, Gunther Hagleitner
> > > > >  wrote:
> > > > >
> > > > > Congrats Teddy!
> > > > > 
> > > > > From: Xuefu Zhang 
> > > > > Sent: Friday, July 07, 2017 9:26 AM
> > > > > To: dev@hive.apache.org
> > > > > Subject: Re: [Announce] New committer: Teddy Choi
> > > > >
> > > > > Congratulations!
> > > > >
> > > > > On Fri, Jul 7, 2017 at 12:09 AM, Matthew McCline
> > > > > 
> > > > > wrote:
> > > > >
> > > > >> Congratulations Teddy!
> > > > >>
> > > > >> Get Outlook for iOS
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Wed, Jul 5, 2017 at 5:53 PM -0700, "Ashutosh Chauhan" <
> > > > >> hashut...@apache.org> wrote:
> > > > >>
> > > > >>
> > > > >> The Project Management Committee (PMC) for Apache Hive has
> > invited
> > > > >> Teddy
> > > > >> Choi to become a committer and we are pleased to announce that
> > he
> > > > has
> > > > >> accepted.
> > > > >>
> > > > >> Welcome, Teddy!
> > > > >>
> > > > >> Thanks,
> > > > >> Ashutosh
> > > > >>
> > > > >>
> > > > >
> > > > >
> > > > 
> > > > >>>
> > > > >>>
> > > >
> > > >
> > >
> >
>

Re: [Announce] New committer: Vihang Karajgaonkar

[Pengcheng Xiong]

Congrats!

Best
Pengcheng

On Fri, Jul 7, 2017 at 8:43 PM, Rajesh Balamohan 
wrote:

> Congrats Vihang!!
>
> On Sat, Jul 8, 2017 at 7:07 AM, Lefty Leverenz 
> wrote:
>
> > Congratulations!
> >
> > -- Lefty
> >
> > On Fri, Jul 7, 2017 at 5:06 PM, Chaoyu Tang  wrote:
> >
> > > Congratulations Vihang !!!.
> > >
> > > On Fri, Jul 7, 2017 at 2:38 PM, Sergio Pena 
> > > wrote:
> > >
> > > > Congratulations Vihang !!!.
> > > >
> > > > - Sergio
> > > >
> > > > On Fri, Jul 7, 2017 at 12:50 PM, Vaibhav Gumashta <
> > > > vgumas...@hortonworks.com
> > > > > wrote:
> > > >
> > > > > Congratulations Vihang!
> > > > >
> > > > > On 7/7/17, 10:47 AM, "Vineet Garg"  wrote:
> > > > >
> > > > > >Congratulations Vihang!
> > > > > >
> > > > > >> On Jul 5, 2017, at 5:51 PM, Ashutosh Chauhan <
> > hashut...@apache.org>
> > > > > >>wrote:
> > > > > >>
> > > > > >> The Project Management Committee (PMC) for Apache Hive has
> invited
> > > > > >>Vihang
> > > > > >> Karajgaonkar to become a committer and we are pleased to
> announce
> > > that
> > > > > >>he
> > > > > >> has accepted.
> > > > > >>
> > > > > >> Welcome, Vihang!
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Ashutosh
> > > > > >
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: [ANNOUNCE] New PMC Member : Matt McCline

[Pengcheng Xiong]

Congrats Matt! :)

Best
Pengcheng

On Fri, Jul 7, 2017 at 8:41 PM, Rajesh Balamohan 
wrote:

> Congrats Matt!
>
> ~Rajesh.B
>
> On Sat, Jul 8, 2017 at 2:35 AM, Chaoyu Tang  wrote:
>
> > Congratulations Matt!
> >
> > On Fri, Jul 7, 2017 at 12:27 PM, Xuefu Zhang  wrote:
> >
> > > Congratulations!
> > >
> > > On Fri, Jul 7, 2017 at 8:27 AM, Eugene Koifman <
> ekoif...@hortonworks.com
> > >
> > > wrote:
> > >
> > > > Congratulations!
> > > >
> > > > On 7/7/17, 1:04 AM, "Zoltan Haindrich" 
> > > wrote:
> > > >
> > > > Congrats Matt!
> > > >
> > > > On 7 Jul 2017 09:46, Peter Vary  wrote:
> > > > Congratulations Matt! :)
> > > >
> > > > 2017. júl. 7. 0:34 ezt írta ("Jesus Camacho Rodriguez" <
> > > > jcama...@apache.org
> > > > >):
> > > >
> > > > > Congrats Matt!
> > > > >
> > > > > -Jesús
> > > > >
> > > > >
> > > > >
> > > > > On 7/6/17, 11:13 PM, "Lefty Leverenz"  >
> > > > wrote:
> > > > >
> > > > > >Congratulations Matt!  Well deserved.
> > > > > >
> > > > > >-- Lefty
> > > > > >
> > > > > >On Thu, Jul 6, 2017 at 11:31 AM, Ashutosh Chauhan <
> > > > hashut...@apache.org>
> > > > > >wrote:
> > > > > >
> > > > > >> On behalf of the Hive PMC I am delighted to announce Matt
> > > McCline
> > > > is
> > > > > >> joining Hive PMC.
> > > > > >> Matt is a long time contributor in Hive and is focusing on
> > > > vectorization
> > > > > >> these days.
> > > > > >>
> > > > > >> Welcome, Matt!
> > > > > >>
> > > > > >> Thanks,
> > > > > >> Ashutosh
> > > > > >>
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > >
> >
>

[jira] [Created] (HIVE-17062) make hive.optimize.bucketingsorting work for smb_mapjoin_20.q

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-17062:
--

 Summary: make hive.optimize.bucketingsorting work for 
smb_mapjoin_20.q
 Key: HIVE-17062
 URL: https://issues.apache.org/jira/browse/HIVE-17062
 Project: Hive
  Issue Type: Bug
Reporter: Pengcheng Xiong


follow-up of HIVE-16981



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-17045) Add HyperLogLog as an UDAF

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-17045:
--

 Summary: Add HyperLogLog as an UDAF
 Key: HIVE-17045
 URL: https://issues.apache.org/jira/browse/HIVE-17045
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-16997) Extend object store to store bit vectors

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-16997:
--

 Summary: Extend object store to store bit vectors
 Key: HIVE-16997
 URL: https://issues.apache.org/jira/browse/HIVE-16997
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-16995) Merge NDV across partitions using bit vectors

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-16995:
--

 Summary: Merge NDV across partitions using bit vectors
 Key: HIVE-16995
 URL: https://issues.apache.org/jira/browse/HIVE-16995
 Project: Hive
  Issue Type: Improvement
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-16996) Add HLL as an alternative to FM sketch to compute stats

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-16996:
--

 Summary: Add HLL as an alternative to FM sketch to compute stats
 Key: HIVE-16996
 URL: https://issues.apache.org/jira/browse/HIVE-16996
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Review Request 60552: hive.optimize.bucketingsorting should compare the schema before removing RS

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60552/
---

Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

HIVE-16981


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java
 ac1c803b32 
  ql/src/test/queries/clientpositive/smb_mapjoin_20.q aa1e9fa9d8 
  ql/src/test/results/clientpositive/beeline/smb_mapjoin_12.q.out 9928a60095 
  ql/src/test/results/clientpositive/bucketsortoptimize_insert_8.q.out 
f3d30068ad 
  ql/src/test/results/clientpositive/smb_mapjoin_12.q.out 9928a60095 
  ql/src/test/results/clientpositive/smb_mapjoin_20.q.out 6c411716e7 
  ql/src/test/results/clientpositive/spark/smb_mapjoin_20.q.out f35a33d8dc 


Diff: https://reviews.apache.org/r/60552/diff/1/


Testing
---


Thanks,

pengcheng xiong

Re: Review Request 57614: Auto-gather column stats - phase 2

[pengcheng xiong]

> On June 27, 2017, 6:37 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/llap/auto_sortmerge_join_13.q.out
> > Line 313 (original), 393 (patched)
> > <https://reviews.apache.org/r/57614/diff/2/?file=1755048#file1755048line393>
> >
> > Auto convert to map join failed.

plan in new patch looks good.


> On June 27, 2017, 6:37 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/vector_multi_insert.q.out
> > Line 165 (original), 198 (patched)
> > <https://reviews.apache.org/r/57614/diff/2/?file=1755711#file1755711line207>
> >
> > vectorization turned off.

notVectorizedReason: Aggregation Function expression for GROUPBY operator: UDF 
compute_stats not supported


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57614/#review178954
-------


On June 20, 2017, 10 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57614/
> ---
> 
> (Updated June 20, 2017, 10 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-13567
> 
> 
> Diffs
> -
> 
>   accumulo-handler/src/test/results/positive/accumulo_queries.q.out 
> de82857c25 
>   
> accumulo-handler/src/test/results/positive/accumulo_single_sourced_multi_insert.q.out
>  6621a4e204 
>   common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 
> 7c27d07024 
>   common/src/java/org/apache/hadoop/hive/common/jsonexplain/Vertex.java 
> 799355a971 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a8bdefdad6 
>   contrib/src/test/results/clientpositive/serde_typedbytes.q.out 6876ca8775 
>   contrib/src/test/results/clientpositive/serde_typedbytes2.q.out 79cf8fe1e5 
>   contrib/src/test/results/clientpositive/serde_typedbytes3.q.out fec58ef026 
>   contrib/src/test/results/clientpositive/serde_typedbytes4.q.out 1131478a7b 
>   contrib/src/test/results/clientpositive/serde_typedbytes5.q.out 8d3b95ece8 
>   data/conf/hive-site.xml 62364fe4ea 
>   
> hbase-handler/src/test/results/positive/hbase_single_sourced_multi_insert.q.out
>  68a417d0c1 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out
>  e55b1c257e 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 
> 663a572748 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_directory.q.out
>  6e95fd123c 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out
>  660cebba5f 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
>  8052fd86ee 
>   
> itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out
>  2ababb1eec 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestMTQueries.java 
> ad2baa2e26 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> 4a9af80fdc 
>   itests/src/test/resources/testconfiguration.properties 07fd5bfe48 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> 1aaba4ca01 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> e13612ee97 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java
>  fe890e4e27 
>   ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java f43992c85d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java d96f432fee 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java f329b5111b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/StatsNoJobTask.java 3807f434a7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java c22d69bb19 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java d61a4607ea 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 88c73f090b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  4642ec2faa 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 
> 9297a0b874 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 88bf82 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MemoryDecider.java 
> 3a20cfe7ac 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SerializeFilter.java 
> dc433fed22 
>

[jira] [Created] (HIVE-16986) Support vectorization for UDAF compute_stats

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-16986:
--

 Summary: Support vectorization for UDAF compute_stats
 Key: HIVE-16986
 URL: https://issues.apache.org/jira/browse/HIVE-16986
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-16981) hive.optimize.bucketingsorting should compare the schema before removing RS

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-16981:
--

 Summary: hive.optimize.bucketingsorting should compare the schema 
before removing RS
 Key: HIVE-16981
 URL: https://issues.apache.org/jira/browse/HIVE-16981
 Project: Hive
  Issue Type: Bug
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong


on master, smb_mapjoin_20.q, run
{code}
select * from test_table3;
{code}
you will get
{code}
val_0  0   NULL1
...
{code}

The correct result is
{code}
val_0  0   val_01
...
{code}





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 57614: Auto-gather column stats - phase 2

[pengcheng xiong]

> On June 27, 2017, 2:20 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/llap_acid.q.out
> > Line 94 (original), 94 (patched)
> > <https://reviews.apache.org/r/57614/diff/2/?file=1755264#file1755264line94>
> >
> > Column stats and basic stats should be complete.

why it is complete? orc_llap is an acid table.


> On June 27, 2017, 2:20 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/merge3.q.out
> > Line 181 (original), 181-182 (patched)
> > <https://reviews.apache.org/r/57614/diff/2/?file=1755285#file1755285line181>
> >
> > Plan is modified to collect stats. But no column stats desc in explain.

No, column stats is not collect for CTAS.


> On June 27, 2017, 2:20 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/stats3.q.out
> > Lines 58 (patched)
> > <https://reviews.apache.org/r/57614/diff/2/?file=1755651#file1755651line58>
> >
> > This should say invalidating stats.

filed a jira for this.


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57614/#review178941
---


On June 20, 2017, 10 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57614/
> ---
> 
> (Updated June 20, 2017, 10 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-13567
> 
> 
> Diffs
> -
> 
>   accumulo-handler/src/test/results/positive/accumulo_queries.q.out 
> de82857c25 
>   
> accumulo-handler/src/test/results/positive/accumulo_single_sourced_multi_insert.q.out
>  6621a4e204 
>   common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 
> 7c27d07024 
>   common/src/java/org/apache/hadoop/hive/common/jsonexplain/Vertex.java 
> 799355a971 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a8bdefdad6 
>   contrib/src/test/results/clientpositive/serde_typedbytes.q.out 6876ca8775 
>   contrib/src/test/results/clientpositive/serde_typedbytes2.q.out 79cf8fe1e5 
>   contrib/src/test/results/clientpositive/serde_typedbytes3.q.out fec58ef026 
>   contrib/src/test/results/clientpositive/serde_typedbytes4.q.out 1131478a7b 
>   contrib/src/test/results/clientpositive/serde_typedbytes5.q.out 8d3b95ece8 
>   data/conf/hive-site.xml 62364fe4ea 
>   
> hbase-handler/src/test/results/positive/hbase_single_sourced_multi_insert.q.out
>  68a417d0c1 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out
>  e55b1c257e 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 
> 663a572748 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_directory.q.out
>  6e95fd123c 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out
>  660cebba5f 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
>  8052fd86ee 
>   
> itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out
>  2ababb1eec 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestMTQueries.java 
> ad2baa2e26 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> 4a9af80fdc 
>   itests/src/test/resources/testconfiguration.properties 07fd5bfe48 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> 1aaba4ca01 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> e13612ee97 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java
>  fe890e4e27 
>   ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java f43992c85d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java d96f432fee 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java f329b5111b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/StatsNoJobTask.java 3807f434a7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java c22d69bb19 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java d61a4607ea 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 88c73f090b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  4642ec2faa 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 
> 9297a0b874 
&

[jira] [Created] (HIVE-16971) improve explain when invalidate stats

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-16971:
--

 Summary: improve explain when invalidate stats
 Key: HIVE-16971
 URL: https://issues.apache.org/jira/browse/HIVE-16971
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong


for example, in a load statement, we use statsTask to invalidate stats.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 57614: Auto-gather column stats - phase 2

[pengcheng xiong]

> On June 24, 2017, 7:13 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/correlationoptimizer5.q.out
> > Line 386 (original)
> > <https://reviews.apache.org/r/57614/diff/2/?file=1754853#file1754853line443>
> >
> > No Mux or Demux operator in plan anymore? Seems like correlation 
> > optimizer is turned off. Expected?

-- Currently, a query with multiple FileSinkOperators are not supported.


> On June 24, 2017, 7:13 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/groupby_multi_single_reducer.q.out
> > Line 479 (original)
> > <https://reviews.apache.org/r/57614/diff/2/?file=1754923#file1754923line756>
> >
> > TopN optimization disabled.

due to L137 in LimitPushDownOptimization "// Not safe to continue for 
RS-GBY-GBY-LIM kind of pipelines. See HIVE-10607 for more." I think we may 
disable the autogather for this q test.


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57614/#review178843
---


On June 20, 2017, 10 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57614/
> ---
> 
> (Updated June 20, 2017, 10 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-13567
> 
> 
> Diffs
> -
> 
>   accumulo-handler/src/test/results/positive/accumulo_queries.q.out 
> de82857c25 
>   
> accumulo-handler/src/test/results/positive/accumulo_single_sourced_multi_insert.q.out
>  6621a4e204 
>   common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 
> 7c27d07024 
>   common/src/java/org/apache/hadoop/hive/common/jsonexplain/Vertex.java 
> 799355a971 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a8bdefdad6 
>   contrib/src/test/results/clientpositive/serde_typedbytes.q.out 6876ca8775 
>   contrib/src/test/results/clientpositive/serde_typedbytes2.q.out 79cf8fe1e5 
>   contrib/src/test/results/clientpositive/serde_typedbytes3.q.out fec58ef026 
>   contrib/src/test/results/clientpositive/serde_typedbytes4.q.out 1131478a7b 
>   contrib/src/test/results/clientpositive/serde_typedbytes5.q.out 8d3b95ece8 
>   data/conf/hive-site.xml 62364fe4ea 
>   
> hbase-handler/src/test/results/positive/hbase_single_sourced_multi_insert.q.out
>  68a417d0c1 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out
>  e55b1c257e 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 
> 663a572748 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_directory.q.out
>  6e95fd123c 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out
>  660cebba5f 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
>  8052fd86ee 
>   
> itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out
>  2ababb1eec 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestMTQueries.java 
> ad2baa2e26 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> 4a9af80fdc 
>   itests/src/test/resources/testconfiguration.properties 07fd5bfe48 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> 1aaba4ca01 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> e13612ee97 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java
>  fe890e4e27 
>   ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java f43992c85d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java d96f432fee 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java f329b5111b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/StatsNoJobTask.java 3807f434a7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java c22d69bb19 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java d61a4607ea 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 88c73f090b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  4642ec2faa 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 
> 9297a0b874 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 88bf82 
>   ql/src/java/org/apac

Re: Review Request 57614: Auto-gather column stats - phase 2

[pengcheng xiong]

> On June 24, 2017, 7:13 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/constprog_type.q.out
> > Line 70 (original), 70 (patched)
> > <https://reviews.apache.org/r/57614/diff/2/?file=1754851#file1754851line70>
> >
> > No info about Column Stats desc (Column name, type and table name)?

We do not support the stats merging for date type yet. Thus no auto column 
stats gather for date type. Open a new jira to track this.


> On June 24, 2017, 7:13 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/ctas.q.out
> > Line 101 (original), 101 (patched)
> > <https://reviews.apache.org/r/57614/diff/2/?file=1754855#file1754855line101>
> >
> > No Column Stats Desc?

We do not support auto gather for CTAS yet.


> On June 24, 2017, 7:13 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/groupby6.q.out
> > Lines 22-23 (patched)
> > <https://reviews.apache.org/r/57614/diff/2/?file=1754901#file1754901line22>
> >
> > Why do we need two jobs in this case to compute column stats?

because of the setting "set hive.map.aggr=false;
set hive.groupby.skewindata=true;"


> On June 24, 2017, 7:13 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/input4.q.out
> > Line 31 (original), 31 (patched)
> > <https://reviews.apache.org/r/57614/diff/2/?file=1754970#file1754970line31>
> >
> > No ColumnStatsDesc in explain.

This is a load command, which should not trigger column stats autogather.


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57614/#review178843
---


On June 20, 2017, 10 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57614/
> ---
> 
> (Updated June 20, 2017, 10 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-13567
> 
> 
> Diffs
> -
> 
>   accumulo-handler/src/test/results/positive/accumulo_queries.q.out 
> de82857c25 
>   
> accumulo-handler/src/test/results/positive/accumulo_single_sourced_multi_insert.q.out
>  6621a4e204 
>   common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 
> 7c27d07024 
>   common/src/java/org/apache/hadoop/hive/common/jsonexplain/Vertex.java 
> 799355a971 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a8bdefdad6 
>   contrib/src/test/results/clientpositive/serde_typedbytes.q.out 6876ca8775 
>   contrib/src/test/results/clientpositive/serde_typedbytes2.q.out 79cf8fe1e5 
>   contrib/src/test/results/clientpositive/serde_typedbytes3.q.out fec58ef026 
>   contrib/src/test/results/clientpositive/serde_typedbytes4.q.out 1131478a7b 
>   contrib/src/test/results/clientpositive/serde_typedbytes5.q.out 8d3b95ece8 
>   data/conf/hive-site.xml 62364fe4ea 
>   
> hbase-handler/src/test/results/positive/hbase_single_sourced_multi_insert.q.out
>  68a417d0c1 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out
>  e55b1c257e 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 
> 663a572748 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_directory.q.out
>  6e95fd123c 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out
>  660cebba5f 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
>  8052fd86ee 
>   
> itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out
>  2ababb1eec 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestMTQueries.java 
> ad2baa2e26 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> 4a9af80fdc 
>   itests/src/test/resources/testconfiguration.properties 07fd5bfe48 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> 1aaba4ca01 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> e13612ee97 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java
>  fe890e4e27 
>   ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java f43992c85d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java d96f432fee 
>   ql/src/java/org/apache/hado

[jira] [Created] (HIVE-16957) Support CTAS for auto gather column stats

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-16957:
--

 Summary: Support CTAS for auto gather column stats
 Key: HIVE-16957
 URL: https://issues.apache.org/jira/browse/HIVE-16957
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (HIVE-16956) Support date type for merging column stats

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-16956:
--

 Summary: Support date type for merging column stats
 Key: HIVE-16956
 URL: https://issues.apache.org/jira/browse/HIVE-16956
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 57614: Auto-gather column stats - phase 2

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57614/#review178849
---




ql/src/test/results/clientpositive/autoColumnStats_4.q.out
Line 200 (original)
<https://reviews.apache.org/r/57614/#comment253112>

yes, no auto stats for acid tables. Right now we do not even merge stats 
for normal tables when the old stats is inaccurate. Last time when we 
discussed, we assumed that the old stats is enough (i.e., we do not wipe it 
clean).


- pengcheng xiong


On June 20, 2017, 10 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57614/
> ---
> 
> (Updated June 20, 2017, 10 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-13567
> 
> 
> Diffs
> -
> 
>   accumulo-handler/src/test/results/positive/accumulo_queries.q.out 
> de82857c25 
>   
> accumulo-handler/src/test/results/positive/accumulo_single_sourced_multi_insert.q.out
>  6621a4e204 
>   common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 
> 7c27d07024 
>   common/src/java/org/apache/hadoop/hive/common/jsonexplain/Vertex.java 
> 799355a971 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a8bdefdad6 
>   contrib/src/test/results/clientpositive/serde_typedbytes.q.out 6876ca8775 
>   contrib/src/test/results/clientpositive/serde_typedbytes2.q.out 79cf8fe1e5 
>   contrib/src/test/results/clientpositive/serde_typedbytes3.q.out fec58ef026 
>   contrib/src/test/results/clientpositive/serde_typedbytes4.q.out 1131478a7b 
>   contrib/src/test/results/clientpositive/serde_typedbytes5.q.out 8d3b95ece8 
>   data/conf/hive-site.xml 62364fe4ea 
>   
> hbase-handler/src/test/results/positive/hbase_single_sourced_multi_insert.q.out
>  68a417d0c1 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out
>  e55b1c257e 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 
> 663a572748 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_directory.q.out
>  6e95fd123c 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out
>  660cebba5f 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
>  8052fd86ee 
>   
> itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out
>  2ababb1eec 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestMTQueries.java 
> ad2baa2e26 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> 4a9af80fdc 
>   itests/src/test/resources/testconfiguration.properties 07fd5bfe48 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> 1aaba4ca01 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> e13612ee97 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java
>  fe890e4e27 
>   ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java f43992c85d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java d96f432fee 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java f329b5111b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/StatsNoJobTask.java 3807f434a7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java c22d69bb19 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java d61a4607ea 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 88c73f090b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  4642ec2faa 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 
> 9297a0b874 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 88bf82 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MemoryDecider.java 
> 3a20cfe7ac 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SerializeFilter.java 
> dc433fed22 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> e9a4ff0748 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java 
> 7a0d4a752e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java ca544b4549 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
> bda94ff765 
>   ql/src/java/org/apache/hadoop/hive/ql/

Re: Review Request 57614: Auto-gather column stats - phase 2

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57614/#review178848
---




ql/src/test/queries/clientpositive/smb_join_partition_key.q
Lines 1 (patched)
<https://reviews.apache.org/r/57614/#comment253111>

I think i showed you this issue long time ago. In derby, when it retrieves 
partition with decimal, it will use partval = 100.0, rather than 100. As a 
result, the partition will not be found and it throws exception. If you use 
mysql, we do not have this problem.


- pengcheng xiong


On June 20, 2017, 10 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57614/
> ---
> 
> (Updated June 20, 2017, 10 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-13567
> 
> 
> Diffs
> -
> 
>   accumulo-handler/src/test/results/positive/accumulo_queries.q.out 
> de82857c25 
>   
> accumulo-handler/src/test/results/positive/accumulo_single_sourced_multi_insert.q.out
>  6621a4e204 
>   common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 
> 7c27d07024 
>   common/src/java/org/apache/hadoop/hive/common/jsonexplain/Vertex.java 
> 799355a971 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a8bdefdad6 
>   contrib/src/test/results/clientpositive/serde_typedbytes.q.out 6876ca8775 
>   contrib/src/test/results/clientpositive/serde_typedbytes2.q.out 79cf8fe1e5 
>   contrib/src/test/results/clientpositive/serde_typedbytes3.q.out fec58ef026 
>   contrib/src/test/results/clientpositive/serde_typedbytes4.q.out 1131478a7b 
>   contrib/src/test/results/clientpositive/serde_typedbytes5.q.out 8d3b95ece8 
>   data/conf/hive-site.xml 62364fe4ea 
>   
> hbase-handler/src/test/results/positive/hbase_single_sourced_multi_insert.q.out
>  68a417d0c1 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out
>  e55b1c257e 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 
> 663a572748 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_directory.q.out
>  6e95fd123c 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out
>  660cebba5f 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
>  8052fd86ee 
>   
> itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out
>  2ababb1eec 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestMTQueries.java 
> ad2baa2e26 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> 4a9af80fdc 
>   itests/src/test/resources/testconfiguration.properties 07fd5bfe48 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> 1aaba4ca01 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> e13612ee97 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java
>  fe890e4e27 
>   ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java f43992c85d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java d96f432fee 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java f329b5111b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/StatsNoJobTask.java 3807f434a7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java c22d69bb19 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java d61a4607ea 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 88c73f090b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  4642ec2faa 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 
> 9297a0b874 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 88bf82 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MemoryDecider.java 
> 3a20cfe7ac 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SerializeFilter.java 
> dc433fed22 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> e9a4ff0748 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java 
> 7a0d4a752e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java ca544b4549 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
> bda94ff765 
&

Re: Review Request 57614: Auto-gather column stats - phase 2

[pengcheng xiong]

> On June 22, 2017, 7:08 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/columnstats_partlvl_dp.q.out
> > Line 252 (original), 264 (patched)
> > <https://reviews.apache.org/r/57614/diff/2/?file=1754842#file1754842line264>
> >
> > Is this change expected. State of basic state changed from Complete to 
> > Partial.

It should be complete. please see the new patch.


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57614/#review178583
---


On June 20, 2017, 10 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57614/
> ---
> 
> (Updated June 20, 2017, 10 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-13567
> 
> 
> Diffs
> -
> 
>   accumulo-handler/src/test/results/positive/accumulo_queries.q.out 
> de82857c25 
>   
> accumulo-handler/src/test/results/positive/accumulo_single_sourced_multi_insert.q.out
>  6621a4e204 
>   common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 
> 7c27d07024 
>   common/src/java/org/apache/hadoop/hive/common/jsonexplain/Vertex.java 
> 799355a971 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a8bdefdad6 
>   contrib/src/test/results/clientpositive/serde_typedbytes.q.out 6876ca8775 
>   contrib/src/test/results/clientpositive/serde_typedbytes2.q.out 79cf8fe1e5 
>   contrib/src/test/results/clientpositive/serde_typedbytes3.q.out fec58ef026 
>   contrib/src/test/results/clientpositive/serde_typedbytes4.q.out 1131478a7b 
>   contrib/src/test/results/clientpositive/serde_typedbytes5.q.out 8d3b95ece8 
>   data/conf/hive-site.xml 62364fe4ea 
>   
> hbase-handler/src/test/results/positive/hbase_single_sourced_multi_insert.q.out
>  68a417d0c1 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out
>  e55b1c257e 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 
> 663a572748 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_directory.q.out
>  6e95fd123c 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out
>  660cebba5f 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
>  8052fd86ee 
>   
> itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out
>  2ababb1eec 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestMTQueries.java 
> ad2baa2e26 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> 4a9af80fdc 
>   itests/src/test/resources/testconfiguration.properties 07fd5bfe48 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> 1aaba4ca01 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> e13612ee97 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java
>  fe890e4e27 
>   ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java f43992c85d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java d96f432fee 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java f329b5111b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/StatsNoJobTask.java 3807f434a7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java c22d69bb19 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java d61a4607ea 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 88c73f090b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  4642ec2faa 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 
> 9297a0b874 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 88bf82 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MemoryDecider.java 
> 3a20cfe7ac 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SerializeFilter.java 
> dc433fed22 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
> e9a4ff0748 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java 
> 7a0d4a752e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java ca544b4549 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
> bda9

Re: Review Request 57614: Auto-gather column stats - phase 2

[pengcheng xiong]

> On June 22, 2017, 7:08 p.m., Ashutosh Chauhan wrote:
> > accumulo-handler/src/test/results/positive/accumulo_queries.q.out
> > Lines 63 (patched)
> > <https://reviews.apache.org/r/57614/diff/2/?file=1754690#file1754690line63>
> >
> > No basic stats work?

true. should be no stats task and no column stats task for non-native tables.


> On June 22, 2017, 7:08 p.m., Ashutosh Chauhan wrote:
> > accumulo-handler/src/test/results/positive/accumulo_queries.q.out
> > Lines 559 (patched)
> > <https://reviews.apache.org/r/57614/diff/2/?file=1754690#file1754690line559>
> >
> > There should be a basic stats work also, no? Since column stats task 
> > also collects basic stats.

we do not do auto gather stats for non-native tables, e.g., accumulo or hbase. 
Please see new patch.


> On June 22, 2017, 7:08 p.m., Ashutosh Chauhan wrote:
> > metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java
> > Lines 1939-1946 (patched)
> > <https://reviews.apache.org/r/57614/diff/2/?file=1754711#file1754711line1939>
> >
> > Change this to assert, instead?

sure.


> On June 22, 2017, 7:08 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java
> > Lines 399-401 (patched)
> > <https://reviews.apache.org/r/57614/diff/2/?file=1754715#file1754715line400>
> >
> > This doesn't look efficient. Retrieving all partition objects on client 
> > just to determine whether stats merging is needed. 
> > This logic should execute on metastore side.

it will only retrieve only a specific partition and do it one by one. however 
,i think your comment is valid.


> On June 22, 2017, 7:08 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java
> > Lines 480-492 (patched)
> > <https://reviews.apache.org/r/57614/diff/2/?file=1754715#file1754715line484>
> >
> > Goal of merging two tasks was to minimize metastore calls which won't 
> > happen as its done right now.
> > 
> > Further, this is confusing. Creating and executing a task within 
> > another task.

we can do refactoring later.


> On June 22, 2017, 7:08 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/plan/StatsNoJobWork.java
> > Line 32 (original), 31 (patched)
> > <https://reviews.apache.org/r/57614/diff/2/?file=1754736#file1754736line32>
> >
> > Any reason to remove @Explain annotation?

because it will show duplicate "Stats-Aggr Operator"


> On June 22, 2017, 7:08 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/plan/StatsWork.java
> > Line 32 (original), 32 (patched)
> > <https://reviews.apache.org/r/57614/diff/2/?file=1754737#file1754737line32>
> >
> > Any reason to remove @Explain annotation?

yes, otherwise it will show duplicate Stats-Aggr Operator


> On June 22, 2017, 7:08 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientnegative/stats_aggregator_error_1.q
> > Lines 13 (patched)
> > <https://reviews.apache.org/r/57614/diff/2/?file=1754740#file1754740line13>
> >
> > Any reason for this?

there is a bug and i fixed it.


> On June 22, 2017, 7:08 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/combine1.q
> > Lines 10 (patched)
> > <https://reviews.apache.org/r/57614/diff/2/?file=1754743#file1754743line10>
> >
> > Any reason for this?

due to the compression. this is a corner case


> On June 22, 2017, 7:08 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/exec_parallel_column_stats.q
> > Lines 3-5 (original), 3-5 (patched)
> > <https://reviews.apache.org/r/57614/diff/2/?file=1754747#file1754747line3>
> >
> > Any reason for this?

we can not compute basic stats for src in q tests.


> On June 22, 2017, 7:08 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/orc_wide_table.q
> > Lines 2 (patched)
> > <https://reviews.apache.org/r/57614/diff/2/?file=1754751#file1754751line2>
> >
> > Any reason for this?

limitation of HMS for too many columns


> On June 22, 2017, 7:08 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/smb_join_partition_key.q
> > Lines 1 (patched)
> > <https://reviews.apache.org/r/57614/diff/2/?file=1754754#file1754754line1>
> >
> > decimals should be supported.

we do not support retrieve of partitions in decimal.


> On June 22, 2017, 7:08 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/udf_round_2.q
> > Lines 2 (patc

Re: Review Request 57614: Auto-gather column stats - phase 2

[pengcheng xiong]

> On June 22, 2017, 7:08 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/udf_round_2.q
> > Lines 2 (patched)
> > <https://reviews.apache.org/r/57614/diff/2/?file=1754755#file1754755line2>
> >
> > Any reason for this?

We can not store NaN for column stats in metastore.


> On June 22, 2017, 7:08 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/columnstats_partlvl.q.out
> > Line 311 (original), 320-321 (patched)
> > <https://reviews.apache.org/r/57614/diff/2/?file=1754841#file1754841line320>
> >
> > Surprised this didn't happen as part of HIVE-15903 but is happening 
> > now. Expected?

Simply because we do not support basic stats collection in MR. 15903 only 
supports Tez.


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57614/#review178583
---


On June 20, 2017, 10 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57614/
> ---
> 
> (Updated June 20, 2017, 10 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-13567
> 
> 
> Diffs
> -
> 
>   accumulo-handler/src/test/results/positive/accumulo_queries.q.out 
> de82857c25 
>   
> accumulo-handler/src/test/results/positive/accumulo_single_sourced_multi_insert.q.out
>  6621a4e204 
>   common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 
> 7c27d07024 
>   common/src/java/org/apache/hadoop/hive/common/jsonexplain/Vertex.java 
> 799355a971 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a8bdefdad6 
>   contrib/src/test/results/clientpositive/serde_typedbytes.q.out 6876ca8775 
>   contrib/src/test/results/clientpositive/serde_typedbytes2.q.out 79cf8fe1e5 
>   contrib/src/test/results/clientpositive/serde_typedbytes3.q.out fec58ef026 
>   contrib/src/test/results/clientpositive/serde_typedbytes4.q.out 1131478a7b 
>   contrib/src/test/results/clientpositive/serde_typedbytes5.q.out 8d3b95ece8 
>   data/conf/hive-site.xml 62364fe4ea 
>   
> hbase-handler/src/test/results/positive/hbase_single_sourced_multi_insert.q.out
>  68a417d0c1 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out
>  e55b1c257e 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 
> 663a572748 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_directory.q.out
>  6e95fd123c 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out
>  660cebba5f 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
>  8052fd86ee 
>   
> itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out
>  2ababb1eec 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestMTQueries.java 
> ad2baa2e26 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> 4a9af80fdc 
>   itests/src/test/resources/testconfiguration.properties 07fd5bfe48 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> 1aaba4ca01 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> e13612ee97 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java
>  fe890e4e27 
>   ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java f43992c85d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java d96f432fee 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java f329b5111b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/StatsNoJobTask.java 3807f434a7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java c22d69bb19 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java d61a4607ea 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 88c73f090b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  4642ec2faa 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 
> 9297a0b874 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 88bf82 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MemoryDecider.java 
> 3a20cfe7ac 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SerializeFilter

Re: Review Request 57614: Auto-gather column stats - phase 2

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57614/#review178430
---




ql/src/test/results/clientpositive/autoColumnStats_3.q.out
Line 39 (original), 41 (patched)
<https://reviews.apache.org/r/57614/#comment252387>

Note that we only compute stats for key but both key and value are accurate 
although value is empty (inherited from create table.)



ql/src/test/results/clientpositive/autoColumnStats_3.q.out
Line 210 (original), 212 (patched)
<https://reviews.apache.org/r/57614/#comment252388>

Due to newly created partition.



ql/src/test/results/clientpositive/autoColumnStats_5.q.out
Line 414 (original), 408 (patched)
<https://reviews.apache.org/r/57614/#comment252391>

part=2 is an empty partition



ql/src/test/results/clientpositive/autoColumnStats_5.q.out
Line 606 (original), 597 (patched)
<https://reviews.apache.org/r/57614/#comment252392>

part=1 already contains data. the new columns c and d should not be merged 
as their stats is inaccurate.


- pengcheng xiong


On June 20, 2017, 10 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57614/
> ---
> 
> (Updated June 20, 2017, 10 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-13567
> 
> 
> Diffs
> -
> 
>   accumulo-handler/src/test/results/positive/accumulo_queries.q.out 
> de82857c25 
>   
> accumulo-handler/src/test/results/positive/accumulo_single_sourced_multi_insert.q.out
>  6621a4e204 
>   common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 
> 7c27d07024 
>   common/src/java/org/apache/hadoop/hive/common/jsonexplain/Vertex.java 
> 799355a971 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a8bdefdad6 
>   contrib/src/test/results/clientpositive/serde_typedbytes.q.out 6876ca8775 
>   contrib/src/test/results/clientpositive/serde_typedbytes2.q.out 79cf8fe1e5 
>   contrib/src/test/results/clientpositive/serde_typedbytes3.q.out fec58ef026 
>   contrib/src/test/results/clientpositive/serde_typedbytes4.q.out 1131478a7b 
>   contrib/src/test/results/clientpositive/serde_typedbytes5.q.out 8d3b95ece8 
>   data/conf/hive-site.xml 62364fe4ea 
>   
> hbase-handler/src/test/results/positive/hbase_single_sourced_multi_insert.q.out
>  68a417d0c1 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out
>  e55b1c257e 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 
> 663a572748 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_directory.q.out
>  6e95fd123c 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out
>  660cebba5f 
>   
> itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
>  8052fd86ee 
>   
> itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out
>  2ababb1eec 
>   itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestMTQueries.java 
> ad2baa2e26 
>   
> itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
> 4a9af80fdc 
>   itests/src/test/resources/testconfiguration.properties 07fd5bfe48 
>   metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
> 1aaba4ca01 
>   metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
> e13612ee97 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java
>  fe890e4e27 
>   ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java f43992c85d 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java d96f432fee 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java f329b5111b 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/StatsNoJobTask.java 3807f434a7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java c22d69bb19 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java d61a4607ea 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 88c73f090b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java
>  4642ec2faa 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 
> 9297a0b874 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
> 88bf82 
>   ql/src/java/org/apache/hadoop/hive/

Re: Review Request 57614: Auto-gather column stats - phase 2

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57614/
---

(Updated June 20, 2017, 10 p.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

This patch also includes (1) HIVE-16495 ColumnStats merge should consider the 
accuracy of the current stats (2) HIVE-16827 Merge stats task and column stats 
task into a single task. After the change, for all the execution engines, if we 
collect column stats, it will automatically collect basic stats as well.


Repository: hive-git


Description
---

HIVE-13567


Diffs (updated)
-

  accumulo-handler/src/test/results/positive/accumulo_queries.q.out de82857c25 
  
accumulo-handler/src/test/results/positive/accumulo_single_sourced_multi_insert.q.out
 6621a4e204 
  common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 7c27d07024 
  common/src/java/org/apache/hadoop/hive/common/jsonexplain/Vertex.java 
799355a971 
  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java a8bdefdad6 
  contrib/src/test/results/clientpositive/serde_typedbytes.q.out 6876ca8775 
  contrib/src/test/results/clientpositive/serde_typedbytes2.q.out 79cf8fe1e5 
  contrib/src/test/results/clientpositive/serde_typedbytes3.q.out fec58ef026 
  contrib/src/test/results/clientpositive/serde_typedbytes4.q.out 1131478a7b 
  contrib/src/test/results/clientpositive/serde_typedbytes5.q.out 8d3b95ece8 
  data/conf/hive-site.xml 62364fe4ea 
  
hbase-handler/src/test/results/positive/hbase_single_sourced_multi_insert.q.out 
68a417d0c1 
  
itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out
 e55b1c257e 
  itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 
663a572748 
  
itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_directory.q.out
 6e95fd123c 
  
itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out
 660cebba5f 
  
itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
 8052fd86ee 
  
itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out
 2ababb1eec 
  itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestMTQueries.java 
ad2baa2e26 
  itests/hive-unit/src/test/java/org/apache/hive/jdbc/TestJdbcWithMiniHS2.java 
4a9af80fdc 
  itests/src/test/resources/testconfiguration.properties 07fd5bfe48 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
1aaba4ca01 
  metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 
e13612ee97 
  
metastore/src/java/org/apache/hadoop/hive/metastore/hbase/stats/merge/ColumnStatsMergerFactory.java
 fe890e4e27 
  ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java f43992c85d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java d96f432fee 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java f329b5111b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/StatsNoJobTask.java 3807f434a7 
  ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java c22d69bb19 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java d61a4607ea 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 88c73f090b 
  
ql/src/java/org/apache/hadoop/hive/ql/metadata/SessionHiveMetaStoreClient.java 
4642ec2faa 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRTableScan1.java 
9297a0b874 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java 
88bf82 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MemoryDecider.java 
3a20cfe7ac 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SerializeFilter.java 
dc433fed22 
  ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 
e9a4ff0748 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ExplainSemanticAnalyzer.java 
7a0d4a752e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java ca544b4549 
  ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 
bda94ff765 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ProcessAnalyzeTable.java 
b6d7ee8a92 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 9e84a29470 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java 08a8f00e06 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java 
52af3af2ea 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsDesc.java 97f323f4b7 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ColumnStatsWork.java 76811b1a93 
  ql/src/java/org/apache/hadoop/hive/ql/plan/StatsNoJobWork.java 77c04f6c6e 
  ql/src/java/org/apache/hadoop/hive/ql/plan/StatsWork.java a5050c5368 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java 7c66955e14 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java 5786c4f659 
  ql/src/test/queries/clientnegative/stats_aggregator_error_1.q 1b2872d3d7 
  ql/src/test/queries/clientp

Re: Review Request 59808: Enhance HiveFilterSetOpTransposeRule to remove union branches

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59808/
---

(Updated June 19, 2017, 6:11 p.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

HIVE-16797


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties 1f6939bc91 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSetOpTransposeRule.java
 3ee29e0482 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveUnionMergeRule.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 348331e052 
  ql/src/test/queries/clientpositive/filter_union.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query11.q PRE-CREATION 
  ql/src/test/results/clientpositive/filter_aggr.q.out db7dcaed3f 
  ql/src/test/results/clientpositive/filter_union.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 8b04bc9261 
  ql/src/test/results/clientpositive/llap/explainuser_2.q.out e3f70b097f 
  ql/src/test/results/clientpositive/llap/filter_union.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 5382c42412 
  ql/src/test/results/clientpositive/llap/tez_union_multiinsert.q.out 
14e8e4389f 
  ql/src/test/results/clientpositive/perf/query11.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query14.q.out 048a17f92f 
  ql/src/test/results/clientpositive/perf/query23.q.out 1fd8cb4f25 
  ql/src/test/results/clientpositive/perf/query33.q.out c1a5fa28ed 
  ql/src/test/results/clientpositive/perf/query4.q.out 1b2048649a 
  ql/src/test/results/clientpositive/perf/query5.q.out a3f2d58fec 
  ql/src/test/results/clientpositive/perf/query56.q.out 4ec7201fa7 
  ql/src/test/results/clientpositive/perf/query60.q.out 12d8cdd9b4 
  ql/src/test/results/clientpositive/perf/query71.q.out 44658081b5 
  ql/src/test/results/clientpositive/perf/query74.q.out bb4a71e6ce 
  ql/src/test/results/clientpositive/perf/query76.q.out dcd5004166 
  ql/src/test/results/clientpositive/perf/query77.q.out d46ba6b13c 
  ql/src/test/results/clientpositive/perf/query80.q.out 3cf41f3fed 
  ql/src/test/results/clientpositive/spark/union30.q.out 12eda1d3b6 
  ql/src/test/results/clientpositive/tez/explainanalyze_2.q.out f6844c4a38 
  ql/src/test/results/clientpositive/union24.q.out d6b1a79b20 
  ql/src/test/results/clientpositive/union30.q.out 26a27c8e15 
  ql/src/test/results/clientpositive/union34.q.out 9d593315af 
  ql/src/test/results/clientpositive/unionall_unbalancedppd.q.out b3e128a3d6 


Diff: https://reviews.apache.org/r/59808/diff/3/

Changes: https://reviews.apache.org/r/59808/diff/2-3/


Testing
---


Thanks,

pengcheng xiong

Re: Review Request 59808: Enhance HiveFilterSetOpTransposeRule to remove union branches

[pengcheng xiong]

> On June 18, 2017, 8:30 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/filter_union.q.out
> > Lines 39-43 (patched)
> > <https://reviews.apache.org/r/59808/diff/2/?file=1747655#file1747655line39>
> >
> > Metadataonly optimizer should have kicked in and turned this to null 
> > scan. Seems like it didn't. Can you confirm with explain extended?

Yes, it is kicked in. I have updated the q file to show explain extended. As 
you will see, for 2 branches out of 4 branches, nullscan is kicking in.


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59808/#review178162
-------


On June 10, 2017, 9:57 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59808/
> ---
> 
> (Updated June 10, 2017, 9:57 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16797
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSetOpTransposeRule.java
>  3ee29e0482 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveUnionMergeRule.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 348331e052 
>   ql/src/test/queries/clientpositive/filter_union.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query11.q PRE-CREATION 
>   ql/src/test/results/clientpositive/filter_aggr.q.out db7dcaed3f 
>   ql/src/test/results/clientpositive/filter_union.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 8b04bc9261 
>   ql/src/test/results/clientpositive/llap/explainuser_2.q.out e3f70b097f 
>   ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 5382c42412 
>   ql/src/test/results/clientpositive/llap/tez_union_multiinsert.q.out 
> 14e8e4389f 
>   ql/src/test/results/clientpositive/perf/query11.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query14.q.out 048a17f92f 
>   ql/src/test/results/clientpositive/perf/query23.q.out 1fd8cb4f25 
>   ql/src/test/results/clientpositive/perf/query33.q.out c1a5fa28ed 
>   ql/src/test/results/clientpositive/perf/query4.q.out 1b2048649a 
>   ql/src/test/results/clientpositive/perf/query5.q.out a3f2d58fec 
>   ql/src/test/results/clientpositive/perf/query56.q.out 4ec7201fa7 
>   ql/src/test/results/clientpositive/perf/query60.q.out 12d8cdd9b4 
>   ql/src/test/results/clientpositive/perf/query71.q.out 44658081b5 
>   ql/src/test/results/clientpositive/perf/query74.q.out bb4a71e6ce 
>   ql/src/test/results/clientpositive/perf/query76.q.out dcd5004166 
>   ql/src/test/results/clientpositive/perf/query77.q.out d46ba6b13c 
>   ql/src/test/results/clientpositive/perf/query80.q.out 3cf41f3fed 
>   ql/src/test/results/clientpositive/spark/union30.q.out 12eda1d3b6 
>   ql/src/test/results/clientpositive/tez/explainanalyze_2.q.out f6844c4a38 
>   ql/src/test/results/clientpositive/union24.q.out d6b1a79b20 
>   ql/src/test/results/clientpositive/union30.q.out 26a27c8e15 
>   ql/src/test/results/clientpositive/union34.q.out 9d593315af 
>   ql/src/test/results/clientpositive/unionall_unbalancedppd.q.out b3e128a3d6 
> 
> 
> Diff: https://reviews.apache.org/r/59808/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: Review Request 59808: Enhance HiveFilterSetOpTransposeRule to remove union branches

[pengcheng xiong]

> On June 12, 2017, 10:41 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSetOpTransposeRule.java
> > Lines 102 (patched)
> > <https://reviews.apache.org/r/59808/diff/2/?file=1747649#file1747649line102>
> >
> > It might be better to call simplify(RexNode) so as not to miss 
> > simplificaiton on operands other than And.

IMHO, the pulled constant and the condition from the filter are composed as 
"AND" relationship and there is no other options.


> On June 12, 2017, 10:41 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveUnionMergeRule.java
> > Lines 42 (patched)
> > <https://reviews.apache.org/r/59808/diff/2/?file=1747650#file1747650line42>
> >
> > Should it extend UnionMergeRule instead and pass on HiveRelBuilder? If 
> > UnionMergeRule doesnt accept RelBuilder, please create a calcite jira.

HiveRelBuilder is not enough. Calcite UnionMergeRule has a bug. It will fail on 
union30.q. I will file a Calcite bug for it.


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59808/#review177678
---


On June 10, 2017, 9:57 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59808/
> ---
> 
> (Updated June 10, 2017, 9:57 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16797
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSetOpTransposeRule.java
>  3ee29e0482 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveUnionMergeRule.java
>  PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 348331e052 
>   ql/src/test/queries/clientpositive/filter_union.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query11.q PRE-CREATION 
>   ql/src/test/results/clientpositive/filter_aggr.q.out db7dcaed3f 
>   ql/src/test/results/clientpositive/filter_union.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 8b04bc9261 
>   ql/src/test/results/clientpositive/llap/explainuser_2.q.out e3f70b097f 
>   ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 5382c42412 
>   ql/src/test/results/clientpositive/llap/tez_union_multiinsert.q.out 
> 14e8e4389f 
>   ql/src/test/results/clientpositive/perf/query11.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query14.q.out 048a17f92f 
>   ql/src/test/results/clientpositive/perf/query23.q.out 1fd8cb4f25 
>   ql/src/test/results/clientpositive/perf/query33.q.out c1a5fa28ed 
>   ql/src/test/results/clientpositive/perf/query4.q.out 1b2048649a 
>   ql/src/test/results/clientpositive/perf/query5.q.out a3f2d58fec 
>   ql/src/test/results/clientpositive/perf/query56.q.out 4ec7201fa7 
>   ql/src/test/results/clientpositive/perf/query60.q.out 12d8cdd9b4 
>   ql/src/test/results/clientpositive/perf/query71.q.out 44658081b5 
>   ql/src/test/results/clientpositive/perf/query74.q.out bb4a71e6ce 
>   ql/src/test/results/clientpositive/perf/query76.q.out dcd5004166 
>   ql/src/test/results/clientpositive/perf/query77.q.out d46ba6b13c 
>   ql/src/test/results/clientpositive/perf/query80.q.out 3cf41f3fed 
>   ql/src/test/results/clientpositive/spark/union30.q.out 12eda1d3b6 
>   ql/src/test/results/clientpositive/tez/explainanalyze_2.q.out f6844c4a38 
>   ql/src/test/results/clientpositive/union24.q.out d6b1a79b20 
>   ql/src/test/results/clientpositive/union30.q.out 26a27c8e15 
>   ql/src/test/results/clientpositive/union34.q.out 9d593315af 
>   ql/src/test/results/clientpositive/unionall_unbalancedppd.q.out b3e128a3d6 
> 
> 
> Diff: https://reviews.apache.org/r/59808/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

[jira] [Created] (HIVE-16916) UpdateColumnStatsTask should set column stats as inaccurate

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-16916:
--

 Summary: UpdateColumnStatsTask should set column stats as 
inaccurate
 Key: HIVE-16916
 URL: https://issues.apache.org/jira/browse/HIVE-16916
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong


It seems that it is now set default as accurate.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Re: Review Request 59808: Enhance HiveFilterSetOpTransposeRule to remove union branches

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59808/
---

(Updated June 10, 2017, 9:57 p.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

HIVE-16797


Diffs (updated)
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSetOpTransposeRule.java
 3ee29e0482 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveUnionMergeRule.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 348331e052 
  ql/src/test/queries/clientpositive/filter_union.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query11.q PRE-CREATION 
  ql/src/test/results/clientpositive/filter_aggr.q.out db7dcaed3f 
  ql/src/test/results/clientpositive/filter_union.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out 8b04bc9261 
  ql/src/test/results/clientpositive/llap/explainuser_2.q.out e3f70b097f 
  ql/src/test/results/clientpositive/llap/orc_ppd_basic.q.out 5382c42412 
  ql/src/test/results/clientpositive/llap/tez_union_multiinsert.q.out 
14e8e4389f 
  ql/src/test/results/clientpositive/perf/query11.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query14.q.out 048a17f92f 
  ql/src/test/results/clientpositive/perf/query23.q.out 1fd8cb4f25 
  ql/src/test/results/clientpositive/perf/query33.q.out c1a5fa28ed 
  ql/src/test/results/clientpositive/perf/query4.q.out 1b2048649a 
  ql/src/test/results/clientpositive/perf/query5.q.out a3f2d58fec 
  ql/src/test/results/clientpositive/perf/query56.q.out 4ec7201fa7 
  ql/src/test/results/clientpositive/perf/query60.q.out 12d8cdd9b4 
  ql/src/test/results/clientpositive/perf/query71.q.out 44658081b5 
  ql/src/test/results/clientpositive/perf/query74.q.out bb4a71e6ce 
  ql/src/test/results/clientpositive/perf/query76.q.out dcd5004166 
  ql/src/test/results/clientpositive/perf/query77.q.out d46ba6b13c 
  ql/src/test/results/clientpositive/perf/query80.q.out 3cf41f3fed 
  ql/src/test/results/clientpositive/spark/union30.q.out 12eda1d3b6 
  ql/src/test/results/clientpositive/tez/explainanalyze_2.q.out f6844c4a38 
  ql/src/test/results/clientpositive/union24.q.out d6b1a79b20 
  ql/src/test/results/clientpositive/union30.q.out 26a27c8e15 
  ql/src/test/results/clientpositive/union34.q.out 9d593315af 
  ql/src/test/results/clientpositive/unionall_unbalancedppd.q.out b3e128a3d6 


Diff: https://reviews.apache.org/r/59808/diff/2/

Changes: https://reviews.apache.org/r/59808/diff/1-2/


Testing
---


Thanks,

pengcheng xiong

[jira] [Created] (HIVE-16836) improve query28 for count distinct rewrite

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-16836:
--

 Summary: improve query28 for count distinct rewrite
 Key: HIVE-16836
 URL: https://issues.apache.org/jira/browse/HIVE-16836
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16837) improve query28 for count distinct rewrite

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-16837:
--

 Summary: improve query28 for count distinct rewrite
 Key: HIVE-16837
 URL: https://issues.apache.org/jira/browse/HIVE-16837
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16827) Merge stats task and column stats task into a single task

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-16827:
--

 Summary: Merge stats task and column stats task into a single task
 Key: HIVE-16827
 URL: https://issues.apache.org/jira/browse/HIVE-16827
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong


Within the task, we can specify whether to compute basic stats only or column 
stats only or both.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Review Request 59808: Enhance HiveFilterSetOpTransposeRule to remove union branches

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59808/
---

Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

HIVE-16797


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterSetOpTransposeRule.java
 3ee29e0482 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveUnionMergeRule.java
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 348331e052 
  ql/src/test/queries/clientpositive/filter_union.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query11.q PRE-CREATION 
  ql/src/test/results/clientpositive/filter_union.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query11.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query4.q.out 1b2048649a 
  ql/src/test/results/clientpositive/perf/query74.q.out bb4a71e6ce 


Diff: https://reviews.apache.org/r/59808/diff/1/


Testing
---


Thanks,

pengcheng xiong

Re: Review Request 59697: Fix HiveFilterAggregateTransposeRule when filter is always false

[pengcheng xiong]

> On June 1, 2017, 9:08 p.m., pengcheng xiong wrote:
> > ql/src/test/results/clientpositive/perf/query4.q.out
> > Lines 436 (patched)
> > <https://reviews.apache.org/r/59697/diff/2/?file=1738886#file1738886line436>
> >
> > The pattern TS-FIL(false) is here. I will take another look to see why 
> > it does not fire.
> 
> pengcheng xiong wrote:
> The reason is because of SharedScan. As all the branches in the union 
> share the same 3 table scan, although only one of them will participate in 
> the join, we still need to scan the table and then branch out. You will see 
> lots of filters out of the same scan in the explain extended. If you turn off 
> the shared scan, you will see the null scan optimization. This is 
> interesting
> 
> Jesús Camacho Rodríguez wrote:
> When SharedScan optimizer is executed, it seems we still do not have the 
> information about the branches that will be removed. Is there a chance to 
> extend NullScanOptimizer to remove those branches that will not contribute to 
> result? Or is it even possible to move NullScanOptimizer to be executed 
> before SharedScanOptimizer?
> 
> Ashutosh Chauhan wrote:
> Ideal thing to do here is to remove branches of tree corresponding to 
> TS-FIL(false) But that removal is non-trivial in general case since it may 
> make tree and result schema invalid e.g., in case of joins. Thats why 
> NullScan instead of removing branch just replaces TS with ZeroRowInputFormat. 
> We can remove them for union though which doesnt alter result schema and 
> which is what HIVE-16797 is doing. Since that will happen early in logical 
> planning, tree will be optimal by the time Shared Scan or NullScan sees it. 
> So, I think with the rules we have, plan currently generated is optimal one 
> we can generate for non-union cases like join like here. There is one minor 
> extension which can be done is to enhance Shared Scan to simplify generated 
> OR expression tree to drop false filter (and potentially other expression 
> simplifications) after generating combining OR filter but that as I said is 
> minor.

Actually, did you guys think in this way? Shared scan is a special case of 
reusing of materialized view as this "materialized" view is truly materialized? 
:) I just would like to connect shared scan with jesus' work on materialized 
view. :)


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59697/#review176665
---


On June 1, 2017, 7:46 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59697/
> ---
> 
> (Updated June 1, 2017, 7:46 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> hive-16775
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterAggregateTransposeRule.java
>  0e5c7313b6 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 7f583ed075 
>   ql/src/test/queries/clientpositive/filter_aggr.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query4.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query74.q PRE-CREATION 
>   ql/src/test/results/clientpositive/filter_aggr.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query4.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query74.q.out PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/59697/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: Review Request 59580: Mergejoin should give itself a correct tag

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59580/
---

(Updated June 2, 2017, 1:16 a.m.)


Review request for hive, Ashutosh Chauhan and Gunther Hagleitner.


Repository: hive-git


Description
---

HIVE-16653


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties e613374b4f 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/MergeJoinProc.java 5b73866a10 
  ql/src/test/queries/clientpositive/tez-tag.q PRE-CREATION 
  ql/src/test/results/clientpositive/tez/tez-tag.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/59580/diff/2/

Changes: https://reviews.apache.org/r/59580/diff/1-2/


Testing
---


Thanks,

pengcheng xiong

Re: Review Request 59697: Fix HiveFilterAggregateTransposeRule when filter is always false

[pengcheng xiong]

> On June 1, 2017, 9:08 p.m., pengcheng xiong wrote:
> > ql/src/test/results/clientpositive/perf/query4.q.out
> > Lines 436 (patched)
> > <https://reviews.apache.org/r/59697/diff/2/?file=1738886#file1738886line436>
> >
> > The pattern TS-FIL(false) is here. I will take another look to see why 
> > it does not fire.

The reason is because of SharedScan. As all the branches in the union share the 
same 3 table scan, although only one of them will participate in the join, we 
still need to scan the table and then branch out. You will see lots of filters 
out of the same scan in the explain extended. If you turn off the shared scan, 
you will see the null scan optimization. This is interesting


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59697/#review176665
---


On June 1, 2017, 7:46 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59697/
> ---
> 
> (Updated June 1, 2017, 7:46 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> hive-16775
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterAggregateTransposeRule.java
>  0e5c7313b6 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 7f583ed075 
>   ql/src/test/queries/clientpositive/filter_aggr.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query4.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query74.q PRE-CREATION 
>   ql/src/test/results/clientpositive/filter_aggr.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query4.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query74.q.out PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/59697/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: Review Request 59697: Fix HiveFilterAggregateTransposeRule when filter is always false

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59697/#review176665
---




ql/src/test/results/clientpositive/perf/query4.q.out
Lines 436 (patched)
<https://reviews.apache.org/r/59697/#comment250080>

The pattern TS-FIL(false) is here. I will take another look to see why it 
does not fire.


- pengcheng xiong


On June 1, 2017, 7:46 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59697/
> ---
> 
> (Updated June 1, 2017, 7:46 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> hive-16775
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterAggregateTransposeRule.java
>  0e5c7313b6 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 7f583ed075 
>   ql/src/test/queries/clientpositive/filter_aggr.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query4.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query74.q PRE-CREATION 
>   ql/src/test/results/clientpositive/filter_aggr.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query4.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query74.q.out PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/59697/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: Review Request 59697: Fix HiveFilterAggregateTransposeRule when filter is always false

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59697/#review176661
---




ql/src/test/results/clientpositive/filter_aggr.q.out
Lines 266 (patched)
<https://reviews.apache.org/r/59697/#comment250076>

This shows null scan optimization is in.


- pengcheng xiong


On June 1, 2017, 7:46 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59697/
> ---
> 
> (Updated June 1, 2017, 7:46 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> hive-16775
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterAggregateTransposeRule.java
>  0e5c7313b6 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 7f583ed075 
>   ql/src/test/queries/clientpositive/filter_aggr.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query4.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/perf/query74.q PRE-CREATION 
>   ql/src/test/results/clientpositive/filter_aggr.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query4.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/perf/query74.q.out PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/59697/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: Review Request 59697: Fix HiveFilterAggregateTransposeRule when filter is always false

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59697/
---

(Updated June 1, 2017, 7:46 p.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

hive-16775


Diffs (updated)
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterAggregateTransposeRule.java
 0e5c7313b6 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 7f583ed075 
  ql/src/test/queries/clientpositive/filter_aggr.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query4.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query74.q PRE-CREATION 
  ql/src/test/results/clientpositive/filter_aggr.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query4.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query74.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/59697/diff/2/

Changes: https://reviews.apache.org/r/59697/diff/1-2/


Testing
---


Thanks,

pengcheng xiong

[jira] [Created] (HIVE-16798) Flaky test query14.q

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-16798:
--

 Summary: Flaky test query14.q
 Key: HIVE-16798
 URL: https://issues.apache.org/jira/browse/HIVE-16798
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Review Request 59697: Fix HiveFilterAggregateTransposeRule when filter is always false

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59697/
---

Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

hive-16775


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveFilterAggregateTransposeRule.java
 0e5c7313b6 
  ql/src/test/queries/clientpositive/filter_aggr.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query4.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query74.q PRE-CREATION 
  ql/src/test/results/clientpositive/filter_aggr.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query4.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query74.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/59697/diff/1/


Testing
---


Thanks,

pengcheng xiong

[jira] [Created] (HIVE-16797) Support a new rule RemoveUnionBranchRule

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-16797:
--

 Summary: Support a new rule RemoveUnionBranchRule
 Key: HIVE-16797
 URL: https://issues.apache.org/jira/browse/HIVE-16797
 Project: Hive
  Issue Type: Bug
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong


in query4.q, we can see that it creates a CTE with union all of 3 branches. 
Then it is going to do a 3 way self-join of the CTE with predicates. The 
predicates actually specifies only one of the branch in CTE to participate in 
the join. Thus, in some cases, e.g.,
{code}
   /- filter(false) -TS0 
union all  - filter(false) -TS1
   \-TS2
{code}
we can cut the branches of TS0 and TS1. The union becomes only TS2.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Review Request 59674: Support position in ORDER BY when using SELECT *

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59674/
---

Review request for hive, Ashutosh Chauhan and Sergey Shelukhin.


Repository: hive-git


Description
---

HIVE-16774


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 721dac8542 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 7f5051cd5b 
  ql/src/test/queries/clientnegative/orderby_position_unsupported.q a490c2306e 
  ql/src/test/queries/clientpositive/order_by_pos.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query47.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query57.q PRE-CREATION 
  ql/src/test/results/clientnegative/orderby_position_unsupported.q.out 
56d32407f3 
  ql/src/test/results/clientpositive/order_by_pos.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query47.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query57.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/59674/diff/1/


Testing
---


Thanks,

pengcheng xiong

Re: Review Request 59468: Optimize a combination of avg(), sum(), count(distinct) etc

[pengcheng xiong]

> On May 27, 2017, 4:41 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/count_dist_rewrite.q
> > Lines 63-65 (patched)
> > <https://reviews.apache.org/r/59468/diff/3/?file=1733999#file1733999line63>
> >
> > As mentioned previously, lets delete these tests.

I assume that previoulsy you want some negative test for this? no?


> On May 27, 2017, 4:41 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/perf/query16.q.out
> > Lines 3-5 (original), 3-5 (patched)
> > <https://reviews.apache.org/r/59468/diff/3/?file=1734004#file1734004line3>
> >
> > Optimization shouldn't have fired in this case. Aggregations are on 
> > different columns.

IMHO, i think it should fire in this case. In this case and the following ones, 
there will be a single reducer producing a single row with constant group by 
key, i.e., everything should go to the same group. After the patch, in the 
first stage, we just introduce the partial result with group by the distinct 
column. Then in the second stage, we aggregate all of the partial results 
together. I think this is exactly what you want previously, i.e., use extra 
stage to reduce the result step by step. Please correct me if my understanding 
is wrong... FYI, we also have test cases in count_dist_rewrite.q to cover this.


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59468/#review176243
-------


On May 27, 2017, 2:20 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59468/
> ---
> 
> (Updated May 27, 2017, 2:20 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Gopal V.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16654
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2dfc8b6f89 
>   itests/src/test/resources/testconfiguration.properties 47a13c93b9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 8b04cd44fa 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/CountDistinctRewriteProc.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 7dace9076f 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 38a9ef2af1 
>   ql/src/test/queries/clientpositive/count_dist_rewrite.q PRE-CREATION 
>   ql/src/test/results/clientpositive/count_dist_rewrite.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/groupby_sort_11.q.out 2b3bf4a07a 
>   ql/src/test/results/clientpositive/llap/count_dist_rewrite.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/nullgroup4.q.out e5a814 
>   ql/src/test/results/clientpositive/perf/query16.q.out cf90c0c162 
>   ql/src/test/results/clientpositive/perf/query28.q.out 78129cf68b 
>   ql/src/test/results/clientpositive/perf/query94.q.out 836b16bf9f 
>   ql/src/test/results/clientpositive/perf/query95.q.out fa94d0842b 
>   ql/src/test/results/clientpositive/udf_count.q.out f60ad0485e 
> 
> 
> Diff: https://reviews.apache.org/r/59468/diff/3/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: Review Request 59468: Optimize a combination of avg(), sum(), count(distinct) etc

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59468/
---

(Updated May 27, 2017, 2:20 a.m.)


Review request for hive, Ashutosh Chauhan and Gopal V.


Repository: hive-git


Description
---

HIVE-16654


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2dfc8b6f89 
  itests/src/test/resources/testconfiguration.properties 47a13c93b9 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 8b04cd44fa 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/CountDistinctRewriteProc.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 7dace9076f 
  ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 38a9ef2af1 
  ql/src/test/queries/clientpositive/count_dist_rewrite.q PRE-CREATION 
  ql/src/test/results/clientpositive/count_dist_rewrite.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/groupby_sort_11.q.out 2b3bf4a07a 
  ql/src/test/results/clientpositive/llap/count_dist_rewrite.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/nullgroup4.q.out e5a814 
  ql/src/test/results/clientpositive/perf/query16.q.out cf90c0c162 
  ql/src/test/results/clientpositive/perf/query28.q.out 78129cf68b 
  ql/src/test/results/clientpositive/perf/query94.q.out 836b16bf9f 
  ql/src/test/results/clientpositive/perf/query95.q.out fa94d0842b 
  ql/src/test/results/clientpositive/udf_count.q.out f60ad0485e 


Diff: https://reviews.apache.org/r/59468/diff/3/

Changes: https://reviews.apache.org/r/59468/diff/2-3/


Testing
---


Thanks,

pengcheng xiong

Re: Review Request 59468: Optimize a combination of avg(), sum(), count(distinct) etc

[pengcheng xiong]

> On May 23, 2017, 5 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/CountDistinctRewriteProc.java
> > Lines 61 (patched)
> > <https://reviews.apache.org/r/59468/diff/1/?file=1727326#file1727326line61>
> >
> > Comment: Queries of form : select max(c), count(distinct c) from T; 
> > generates a plan of form TS->mGBy->RS->rGBy->FS 
> > This plan suffers from a problem that vertex containing rGBy->FS 
> > necessarily need to have 1 task. This limitation results in slow execution 
> > because that task gets all the data. 
> > This optimization if successful will rewrite above plan to 
> > TS->mGby->RS->mGby2->RS->rGBy->FS This introduces extra vertex of mGby2->RS 
> > Note this vertex can have multiple tasks and since we are doing 
> > aggregation, output of this must necessarily be smaller than its input, 
> > which results in much less data going in to rGby->FS vertex, which 
> > continues to have single task.
> > Also note on calcite tree we have HiveExpandDistinctAggregatesRule rule 
> > which does similiar plan transformation but has different conditions which 
> > needs to be satisified.
> > Additionally, we don't do any costing here but this is possibly that 
> > this transformation may slow down query a bit since if data is small enough 
> > to fit in a single task of last reducer, injecting additional vertex in 
> > pipeline may make query slower.

Thanks for the detailed comments.


> On May 23, 2017, 5 p.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/CountDistinctRewriteProc.java
> > Lines 313 (patched)
> > <https://reviews.apache.org/r/59468/diff/1/?file=1727326#file1727326line313>
> >
> > This should be PARTIAL2 mode as well, since GBy operator is running in 
> > Partial2 mode.

partial2 is expecting integer as input. However, here we are counting key_col0, 
which is a string. Thus, hash is more appropriate.


- pengcheng


-------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59468/#review175801
---


On May 25, 2017, 4:03 a.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59468/
> ---
> 
> (Updated May 25, 2017, 4:03 a.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Gopal V.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16654
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2dfc8b6f89 
>   itests/src/test/resources/testconfiguration.properties 47a13c93b9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 8b04cd44fa 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/CountDistinctRewriteProc.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 7dace9076f 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 38a9ef2af1 
>   ql/src/test/queries/clientpositive/count_dist_rewrite.q PRE-CREATION 
>   ql/src/test/results/clientpositive/groupby_sort_11.q.out 2b3bf4a07a 
>   ql/src/test/results/clientpositive/groupby_sort_8.q.out 4faa0757cc 
>   ql/src/test/results/clientpositive/llap/count_dist_rewrite.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/nullgroup4.q.out e5a814 
>   ql/src/test/results/clientpositive/perf/query16.q.out cf90c0c162 
>   ql/src/test/results/clientpositive/perf/query28.q.out 78129cf68b 
>   ql/src/test/results/clientpositive/perf/query94.q.out 836b16bf9f 
>   ql/src/test/results/clientpositive/perf/query95.q.out fa94d0842b 
>   ql/src/test/results/clientpositive/udf_count.q.out f60ad0485e 
>   ql/src/test/results/clientpositive/vector_empty_where.q.out b2dec6d7f6 
> 
> 
> Diff: https://reviews.apache.org/r/59468/diff/2/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Review Request 59610: Support numeric as same as decimal

[pengcheng xiong]

 af77d9a6d9 
  ql/src/test/results/clientpositive/perf/query70.q.out d4b7731bfe 
  ql/src/test/results/clientpositive/perf/query71.q.out 6d8562917a 
  ql/src/test/results/clientpositive/perf/query73.q.out 9e1a39a375 
  ql/src/test/results/clientpositive/perf/query75.q.out 55da3a2d9b 
  ql/src/test/results/clientpositive/perf/query76.q.out 19243db563 
  ql/src/test/results/clientpositive/perf/query77.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query78.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query79.q.out 49557453b8 
  ql/src/test/results/clientpositive/perf/query81.q.out 3acee4d5a5 
  ql/src/test/results/clientpositive/perf/query83.q.out f96306b1bd 
  ql/src/test/results/clientpositive/perf/query84.q.out cf687292a4 
  ql/src/test/results/clientpositive/perf/query85.q.out 82df27bef3 
  ql/src/test/results/clientpositive/perf/query86.q.out bc862917cb 
  ql/src/test/results/clientpositive/perf/query88.q.out f7af4ef855 
  ql/src/test/results/clientpositive/perf/query89.q.out 27467bd3a7 
  ql/src/test/results/clientpositive/perf/query9.q.out 4dded91452 
  ql/src/test/results/clientpositive/perf/query90.q.out aae0aec657 
  ql/src/test/results/clientpositive/perf/query91.q.out 0ee7e43f1b 
  ql/src/test/results/clientpositive/perf/query92.q.out d3abc231ee 
  ql/src/test/results/clientpositive/perf/query93.q.out f28ba412b4 
  ql/src/test/results/clientpositive/perf/query94.q.out 836b16bf9f 
  ql/src/test/results/clientpositive/perf/query95.q.out fa94d0842b 
  ql/src/test/results/clientpositive/perf/query96.q.out a6c1f46287 
  ql/src/test/results/clientpositive/perf/query97.q.out ef9a791e67 
  ql/src/test/results/clientpositive/perf/query98.q.out 2894f7e607 
  ql/src/test/results/clientpositive/perf/query99.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/59610/diff/1/


Testing
---


Thanks,

pengcheng xiong

Re: [Announce] New PMC members

[Pengcheng Xiong]

Yongzhi, Daniel, Vaibhav, Sergio, Aihua, and Chaoyu!

Congratulations to all of you!

Best
Pengcheng

On Thu, May 25, 2017 at 10:26 PM, Peter Vary  wrote:

> Wow!
> That's a spring shower of PMCs. :)
> Well deserved Yongzhi, Daniel, Vaibhav, Sergio, Aihua, Chaoyu!
>
> Congratulations to all of you!
>
> Peter
>
> 2017. máj. 26. 6:42 ezt írta ("Ashutosh Chauhan" ):
>
> The Project Management Committee (PMC) for Apache Hive has invited Yongzhi
> Chen to become a PMC member and we are pleased to announce that he has
> accepted.
>
> Please join me in congratulating Yongzhi!
>
> Thanks,
> Ashutosh on behalf of Hive PMC
>

[jira] [Created] (HIVE-16775) Augment ASTConverter for TPCDS queries

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-16775:
--

 Summary: Augment ASTConverter for TPCDS queries
 Key: HIVE-16775
 URL: https://issues.apache.org/jira/browse/HIVE-16775
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong


query4.q,query74.q
{code}
[7e490527-156a-48c7-aa87-8c80093cdfa8 main] ql.Driver: FAILED: 
NullPointerException null
java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter$QBVisitor.visit(ASTConverter.java:457)
at org.apache.calcite.rel.RelVisitor.go(RelVisitor.java:61)
at 
org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter.convert(ASTConverter.java:110)
at 
org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter.convertSource(ASTConverter.java:393)
at 
org.apache.hadoop.hive.ql.optimizer.calcite.translator.ASTConverter.convert(ASTConverter.java:115)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16774) Support position in ORDER BY when using SELECT *

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-16774:
--

 Summary: Support position in ORDER BY when using SELECT *
 Key: HIVE-16774
 URL: https://issues.apache.org/jira/browse/HIVE-16774
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong


query47.q query57.q



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16773) Support non-equi join predicate in scalar subqueries with aggregate

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-16773:
--

 Summary: Support non-equi join predicate in scalar subqueries with 
aggregate 
 Key: HIVE-16773
 URL: https://issues.apache.org/jira/browse/HIVE-16773
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong


query41.q
{code}
[5e84b202-205a-4fea-a457-94f28e63f0b4 main] ql.Driver: FAILED: 
SemanticException [Error 10250]: 
org.apache.hadoop.hive.ql.optimizer.calcite.CalciteSubquerySemanticException: 
Line 8:13 Invalid SubQuery expression ''medium'': Scalar subqueries with 
aggregate cannot have non-equi join predicate
org.apache.hadoop.hive.ql.parse.SemanticException: 
org.apache.hadoop.hive.ql.optimizer.calcite.CalciteSubquerySemanticException: 
Line 8:13 Invalid SubQuery expression ''medium'': Scalar subqueries with 
aggregate cannot have non-equi join predicate
at 
org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:466)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16772) Support TPCDS query11.q in PerfCliDriver

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-16772:
--

 Summary: Support TPCDS query11.q in PerfCliDriver
 Key: HIVE-16772
 URL: https://issues.apache.org/jira/browse/HIVE-16772
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong


{code}
org.apache.hadoop.hive.ql.parse.SemanticException: Line 54:22 Invalid column 
reference 'customer_preferred_cust_flag'
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:11744)
at 
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:11692)

{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16764) Support numeric as same as decimal

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-16764:
--

 Summary: Support numeric as same as decimal
 Key: HIVE-16764
 URL: https://issues.apache.org/jira/browse/HIVE-16764
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong


for example numeric(12,2) -> decimal(12,2) 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16763) Support space in quoted column alias

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-16763:
--

 Summary: Support space in quoted column alias
 Key: HIVE-16763
 URL: https://issues.apache.org/jira/browse/HIVE-16763
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong


{code}
select key as 'k y' from src;
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (HIVE-16762) Support unmodified TPCDS queries in Hive

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-16762:
--

 Summary: Support unmodified TPCDS queries in Hive
 Key: HIVE-16762
 URL: https://issues.apache.org/jira/browse/HIVE-16762
 Project: Hive
  Issue Type: Bug
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Review Request 59580: Mergejoin should give itself a correct tag

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59580/
---

Review request for hive, Ashutosh Chauhan and Gunther Hagleitner.


Repository: hive-git


Description
---

HIVE-16653


Diffs
-

  itests/src/test/resources/testconfiguration.properties 7510ddcb38 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/MergeJoinProc.java 5b73866a10 
  ql/src/test/queries/clientpositive/tez-tag.q PRE-CREATION 
  ql/src/test/results/clientpositive/tez/tez-tag.q.out PRE-CREATION 


Diff: https://reviews.apache.org/r/59580/diff/1/


Testing
---


Thanks,

pengcheng xiong

Re: Welcome Rui Li to Hive PMC

[Pengcheng Xiong]

Congrats, Rui!

Best
Pengcheng

On Wed, May 24, 2017 at 9:19 PM, Xuefu Zhang  wrote:

> Hi all,
>
> It's an honer to announce that Apache Hive PMC has recently voted to invite
> Rui Li as a new Hive PMC member. Rui is a long time Hive contributor and
> committer, and has made significant contribution in Hive especially in Hive
> on Spark. Please join me in congratulating him and looking forward to a
> bigger role that he will play in Apache Hive project.
>
> Thanks,
> Xuefu
>

Re: Jimmy Xiang now a Hive PMC member

[Pengcheng Xiong]

Congrats Jimmy!

Best
Pengcheng

On Wed, May 24, 2017 at 9:16 PM, Xuefu Zhang  wrote:

> Hi all,
>
> It's an honer to announce that Apache Hive PMC has recently voted to invite
> Jimmy Xiang as a new Hive PMC member. Please join me in congratulating him
> and looking forward to a bigger role that he will play in Apache Hive
> project.
>
> Thanks,
> Xuefu
>

Re: Review Request 59468: Optimize a combination of avg(), sum(), count(distinct) etc

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59468/
---

(Updated May 25, 2017, 4:03 a.m.)


Review request for hive, Ashutosh Chauhan and Gopal V.


Repository: hive-git


Description
---

HIVE-16654


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 2dfc8b6f89 
  itests/src/test/resources/testconfiguration.properties 47a13c93b9 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 8b04cd44fa 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/CountDistinctRewriteProc.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 7dace9076f 
  ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 38a9ef2af1 
  ql/src/test/queries/clientpositive/count_dist_rewrite.q PRE-CREATION 
  ql/src/test/results/clientpositive/groupby_sort_11.q.out 2b3bf4a07a 
  ql/src/test/results/clientpositive/groupby_sort_8.q.out 4faa0757cc 
  ql/src/test/results/clientpositive/llap/count_dist_rewrite.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/nullgroup4.q.out e5a814 
  ql/src/test/results/clientpositive/perf/query16.q.out cf90c0c162 
  ql/src/test/results/clientpositive/perf/query28.q.out 78129cf68b 
  ql/src/test/results/clientpositive/perf/query94.q.out 836b16bf9f 
  ql/src/test/results/clientpositive/perf/query95.q.out fa94d0842b 
  ql/src/test/results/clientpositive/udf_count.q.out f60ad0485e 
  ql/src/test/results/clientpositive/vector_empty_where.q.out b2dec6d7f6 


Diff: https://reviews.apache.org/r/59468/diff/2/

Changes: https://reviews.apache.org/r/59468/diff/1-2/


Testing
---


Thanks,

pengcheng xiong

Re: Review Request 59468: Optimize a combination of avg(), sum(), count(distinct) etc

[pengcheng xiong]

> On May 23, 2017, 12:36 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java
> > Line 206 (original), 210 (patched)
> > <https://reviews.apache.org/r/59468/diff/1/?file=1727328#file1727328line210>
> >
> > I think we should call this optimizer here. This way a) its called 
> > after ReduceSinkDedup so that there is no chance that extra RS introduced 
> > by this optimization does not get removed and b) It won't violate any 
> > assumptions made by GroupbyOptimizer.

ColumnPruner will prune some of the columns, which violates the assumption that 
we make in the rule. Thus, we need to put it before Column Pruner.


> On May 23, 2017, 12:36 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/results/clientpositive/llap/metadataonly1.q.out
> > Line 256 (original), 256-257 (patched)
> > <https://reviews.apache.org/r/59468/diff/1/?file=1727335#file1727335line256>
> >
> > Earlier plan was better. When metadataonly optimizer can kick in that 
> > should.

Yes, but i think the performance gain is minor as the table is empty. In 
NullScanTaskDispatcher, it expects that the GBY is distinct like. However, 
after our patch, we remove the distinct.


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59468/#review175736
---


On May 22, 2017, 10:31 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59468/
> ---
> 
> (Updated May 22, 2017, 10:31 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Gopal V.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-16654
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7dedd23591 
>   itests/src/test/resources/testconfiguration.properties e23ef6317f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 8b04cd44fa 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/CountDistinctRewriteProc.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/GroupByOptimizer.java 
> 3233157d8d 
>   ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 7dace9076f 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 38a9ef2af1 
>   ql/src/test/queries/clientpositive/count_dist_rewrite.q PRE-CREATION 
>   ql/src/test/results/clientpositive/count_dist_rewrite.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/groupby_sort_11.q.out 2b3bf4a07a 
>   ql/src/test/results/clientpositive/groupby_sort_8.q.out 4faa0757cc 
>   ql/src/test/results/clientpositive/llap/count_dist_rewrite.q.out 
> PRE-CREATION 
>   ql/src/test/results/clientpositive/llap/metadataonly1.q.out 27218cf599 
>   ql/src/test/results/clientpositive/nullgroup4.q.out e5a814 
>   ql/src/test/results/clientpositive/perf/query16.q.out cf90c0c162 
>   ql/src/test/results/clientpositive/perf/query28.q.out 78129cf68b 
>   ql/src/test/results/clientpositive/perf/query94.q.out 836b16bf9f 
>   ql/src/test/results/clientpositive/perf/query95.q.out fa94d0842b 
>   ql/src/test/results/clientpositive/spark/nullgroup4.q.out 24f0291dec 
>   ql/src/test/results/clientpositive/udf_count.q.out f60ad0485e 
>   ql/src/test/results/clientpositive/vector_empty_where.q.out b2dec6d7f6 
> 
> 
> Diff: https://reviews.apache.org/r/59468/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> pengcheng xiong
> 
>

Re: Review Request 56140: Can't order by an unselected column

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56140/
---

(Updated May 23, 2017, 9:38 p.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

HIVE-15160


Diffs (updated)
-

  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java
 1487ed4f8e 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java fa96e94f64 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 35fc68a555 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
f678d0b0a0 
  ql/src/test/queries/clientpositive/order_by_expr_1.q PRE-CREATION 
  ql/src/test/queries/clientpositive/order_by_expr_2.q PRE-CREATION 
  ql/src/test/results/clientpositive/annotate_stats_select.q.out 873f1abb25 
  ql/src/test/results/clientpositive/cp_sel.q.out 1778ccd6a6 
  ql/src/test/results/clientpositive/druid_basic2.q.out 6177d56987 
  ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out fc0030965a 
  ql/src/test/results/clientpositive/groupby_grouping_sets_grouping.q.out 
473d17a1bd 
  ql/src/test/results/clientpositive/llap/bucket_groupby.q.out d724131fca 
  ql/src/test/results/clientpositive/llap/explainuser_1.q.out f701cabffe 
  ql/src/test/results/clientpositive/llap/limit_pushdown.q.out 0a8df615fd 
  ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out 24645b6426 
  ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
77062c737e 
  ql/src/test/results/clientpositive/llap/subquery_in.q.out d7fd29e194 
  ql/src/test/results/clientpositive/llap/vector_coalesce.q.out 840210476b 
  ql/src/test/results/clientpositive/llap/vector_date_1.q.out a4f1050c89 
  ql/src/test/results/clientpositive/llap/vector_decimal_2.q.out 144356c108 
  ql/src/test/results/clientpositive/llap/vector_decimal_round.q.out 00bb50a5a5 
  
ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_grouping.q.out
 5af9e61b0a 
  
ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out
 f731ceecdc 
  ql/src/test/results/clientpositive/llap/vector_interval_1.q.out 8d4f12e203 
  ql/src/test/results/clientpositive/llap/vector_interval_arithmetic.q.out 
1d14092408 
  ql/src/test/results/clientpositive/order3.q.out 898f7a8853 
  ql/src/test/results/clientpositive/order_by_expr_1.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/order_by_expr_2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/pcr.q.out a1301fdf79 
  ql/src/test/results/clientpositive/perf/query31.q.out 9e3dad472a 
  ql/src/test/results/clientpositive/perf/query36.q.out 57ab26acc6 
  ql/src/test/results/clientpositive/perf/query39.q.out dcf3cb264e 
  ql/src/test/results/clientpositive/perf/query42.q.out 3bebac3321 
  ql/src/test/results/clientpositive/perf/query52.q.out 74ecaf28ba 
  ql/src/test/results/clientpositive/perf/query64.q.out 7f97e392e1 
  ql/src/test/results/clientpositive/perf/query66.q.out ec7b6af471 
  ql/src/test/results/clientpositive/perf/query70.q.out 55c1461da8 
  ql/src/test/results/clientpositive/perf/query75.q.out 0ecc9852ed 
  ql/src/test/results/clientpositive/perf/query81.q.out dfd46396b5 
  ql/src/test/results/clientpositive/perf/query85.q.out ba8659e8f2 
  ql/src/test/results/clientpositive/perf/query86.q.out 734e6a480b 
  ql/src/test/results/clientpositive/perf/query89.q.out 66481f710b 
  ql/src/test/results/clientpositive/perf/query91.q.out e592bba8d9 
  ql/src/test/results/clientpositive/pointlookup2.q.out 3438c74608 
  ql/src/test/results/clientpositive/pointlookup3.q.out 2c3e39fd15 
  ql/src/test/results/clientpositive/ppd_udf_case.q.out 7678d03415 
  ql/src/test/results/clientpositive/spark/dynamic_rdd_cache.q.out bcb50cfadc 
  ql/src/test/results/clientpositive/spark/limit_pushdown.q.out ede0096c73 
  ql/src/test/results/clientpositive/spark/pcr.q.out 77ac020d07 
  ql/src/test/results/clientpositive/spark/subquery_in.q.out 5e38938ad6 
  ql/src/test/results/clientpositive/vector_coalesce.q.out 87ab937abb 
  ql/src/test/results/clientpositive/vector_date_1.q.out c2389e6b1e 
  ql/src/test/results/clientpositive/vector_decimal_round.q.out d92b6c241e 
  ql/src/test/results/clientpositive/vector_interval_1.q.out 2a398ae5d3 
  ql/src/test/results/clientpositive/vector_interval_arithmetic.q.out 
b67231c8c4 
  ql/src/test/results/clientpositive/view_alias.q.out 90bf28dd9b 


Diff: https://reviews.apache.org/r/56140/diff/9/

Changes: https://reviews.apache.org/r/56140/diff/8-9/


Testing
---


Thanks,

pengcheng xiong

Re: Review Request 56140: Can't order by an unselected column

[pengcheng xiong]

> On May 18, 2017, 12:29 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java
> > Lines 70-71 (original), 81-83 (patched)
> > <https://reviews.apache.org/r/56140/diff/8/?file=1703996#file1703996line81>
> >
> > This change looks correct. But don't understand why it was needed. Can 
> > you describe the need for it?

This is a bug exposed by this patch. The query is 
{code}
create table s as select * from src limit 10;
set hive.optimize.limittranspose=true;

explain
select key from s a
union all
select key from s b
order by key
limit 5;
{code}

HiveProjectSortTransposeRule is triggered for

{code}
HiveProject(key=[$0])
  HiveSortLimit(sort0=[$1], dir0=[ASC-nulls-first], offset=[0], fetch=[5])
{code}

 when 
{code}
if (map.getTarget(fc.getFieldIndex()) < 0) {
return;
  }
{code}
is called, fc.getFieldIndex() is 1, but map is 0->0. Then it throws

org.apache.calcite.util.mapping.Mappings$NoElementException: source #1 has no 
target in mapping [size=1, sourceCount=2, targetCount=1, elements=[0:0]]


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56140/#review175299
-------


On May 1, 2017, 5:30 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56140/
> ---
> 
> (Updated May 1, 2017, 5:30 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15160
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java
>  1487ed4f8e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 1b054a7e24 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/RowResolver.java 262dafb487 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 654f3b1772 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
> 8f8eab0d9c 
>   ql/src/test/queries/clientpositive/order_by_expr_1.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/order_by_expr_2.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out 873f1abb25 
>   ql/src/test/results/clientpositive/cp_sel.q.out 1778ccd6a6 
>   ql/src/test/results/clientpositive/druid_basic2.q.out 6177d56987 
>   ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out 2abb819558 
>   ql/src/test/results/clientpositive/groupby_grouping_sets_grouping.q.out 
> 473d17a1bd 
>   ql/src/test/results/clientpositive/llap/bucket_groupby.q.out d724131fca 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 584c3b5520 
>   ql/src/test/results/clientpositive/llap/limit_pushdown.q.out dd54dd22a6 
>   ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out 24645b6426 
>   ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
> 83de1fbea1 
>   ql/src/test/results/clientpositive/llap/vector_coalesce.q.out 578f849bdb 
>   ql/src/test/results/clientpositive/llap/vector_date_1.q.out a4f1050c89 
>   ql/src/test/results/clientpositive/llap/vector_decimal_2.q.out 144356c108 
>   ql/src/test/results/clientpositive/llap/vector_decimal_round.q.out 
> 8bd80cf860 
>   
> ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_grouping.q.out
>  5af9e61b0a 
>   
> ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out
>  f731ceecdc 
>   ql/src/test/results/clientpositive/llap/vector_interval_1.q.out debf5ab39e 
>   ql/src/test/results/clientpositive/llap/vector_interval_arithmetic.q.out 
> aadb6e72cd 
>   ql/src/test/results/clientpositive/order3.q.out 898f7a8853 
>   ql/src/test/results/clientpositive/order_by_expr_1.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/order_by_expr_2.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/pcr.q.out a1301fdf79 
>   ql/src/test/results/clientpositive/perf/query31.q.out 3ed312d3e3 
>   ql/src/test/results/clientpositive/perf/query36.q.out 57ab26acc6 
>   ql/src/test/results/clientpositive/perf/query39.q.out 19472c4d5e 
>   ql/src/test/results/clientpositive/perf/query42.q.out 3bebac3321 
>   ql/src/test/results/clientpositive/perf/query52.q.out 74ecaf28ba 
>   ql/src/test/results/clientpositive/perf/query64.q.out 6b42393aad 
>   ql/src/test/results/clientpositive/perf/query66.q.out 072bfee92b 
>   ql/src/test/results/clientpositive/perf/query70.q.out 8e42

Re: Review Request 56140: Can't order by an unselected column

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56140/#review175753
---




ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
Lines 3093 (patched)
<https://reviews.apache.org/r/56140/#comment249106>

We can see 
Stage: Stage-0
Fetch Operator
  limit: 10
  Processor Tree:
ListSink

if we run explain rather than explain vectorization. This means, we may 
need to modify explain vectorization to show more info. for fetch work.

And, if there is no limit 10, it will return 43 rows. 
This means the result is correct.


- pengcheng xiong


On May 1, 2017, 5:30 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56140/
> ---
> 
> (Updated May 1, 2017, 5:30 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15160
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java
>  1487ed4f8e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 1b054a7e24 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/RowResolver.java 262dafb487 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 654f3b1772 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
> 8f8eab0d9c 
>   ql/src/test/queries/clientpositive/order_by_expr_1.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/order_by_expr_2.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out 873f1abb25 
>   ql/src/test/results/clientpositive/cp_sel.q.out 1778ccd6a6 
>   ql/src/test/results/clientpositive/druid_basic2.q.out 6177d56987 
>   ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out 2abb819558 
>   ql/src/test/results/clientpositive/groupby_grouping_sets_grouping.q.out 
> 473d17a1bd 
>   ql/src/test/results/clientpositive/llap/bucket_groupby.q.out d724131fca 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 584c3b5520 
>   ql/src/test/results/clientpositive/llap/limit_pushdown.q.out dd54dd22a6 
>   ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out 24645b6426 
>   ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
> 83de1fbea1 
>   ql/src/test/results/clientpositive/llap/vector_coalesce.q.out 578f849bdb 
>   ql/src/test/results/clientpositive/llap/vector_date_1.q.out a4f1050c89 
>   ql/src/test/results/clientpositive/llap/vector_decimal_2.q.out 144356c108 
>   ql/src/test/results/clientpositive/llap/vector_decimal_round.q.out 
> 8bd80cf860 
>   
> ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_grouping.q.out
>  5af9e61b0a 
>   
> ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out
>  f731ceecdc 
>   ql/src/test/results/clientpositive/llap/vector_interval_1.q.out debf5ab39e 
>   ql/src/test/results/clientpositive/llap/vector_interval_arithmetic.q.out 
> aadb6e72cd 
>   ql/src/test/results/clientpositive/order3.q.out 898f7a8853 
>   ql/src/test/results/clientpositive/order_by_expr_1.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/order_by_expr_2.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/pcr.q.out a1301fdf79 
>   ql/src/test/results/clientpositive/perf/query31.q.out 3ed312d3e3 
>   ql/src/test/results/clientpositive/perf/query36.q.out 57ab26acc6 
>   ql/src/test/results/clientpositive/perf/query39.q.out 19472c4d5e 
>   ql/src/test/results/clientpositive/perf/query42.q.out 3bebac3321 
>   ql/src/test/results/clientpositive/perf/query52.q.out 74ecaf28ba 
>   ql/src/test/results/clientpositive/perf/query64.q.out 6b42393aad 
>   ql/src/test/results/clientpositive/perf/query66.q.out 072bfee92b 
>   ql/src/test/results/clientpositive/perf/query70.q.out 8e42fac9c5 
>   ql/src/test/results/clientpositive/perf/query75.q.out b1e236d325 
>   ql/src/test/results/clientpositive/perf/query81.q.out a09d5c99b5 
>   ql/src/test/results/clientpositive/perf/query85.q.out 168bcd2a4a 
>   ql/src/test/results/clientpositive/perf/query86.q.out 734e6a480b 
>   ql/src/test/results/clientpositive/perf/query89.q.out 66481f710b 
>   ql/src/test/results/clientpositive/perf/query91.q.out e592bba8d9 
>   ql/src/test/results/clientpositive/pointlookup2.q.out 3438c74608 
>   ql/src/test/results/clientpositive/pointlookup3.q.out 2c3e39fd15 
>   ql/src/test/results/clientp

[jira] [Created] (HIVE-16734) Support original tpcds queries in perfclidriver after order by unselect column feature is done

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-16734:
--

 Summary: Support original tpcds queries in perfclidriver after 
order by unselect column feature is done
 Key: HIVE-16734
 URL: https://issues.apache.org/jira/browse/HIVE-16734
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong






--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Review Request 56140: Can't order by an unselected column

[pengcheng xiong]

> On May 18, 2017, 12:29 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
> > Lines 4049-4059 (original)
> > <https://reviews.apache.org/r/56140/diff/8/?file=1703997#file1703997line4137>
> >
> > Aren't all these restrictions still valid?

They are not valid any more as we generate a select before OB and a select 
after OB for all the cases. There is no need to fix the schema after OB anymore.


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56140/#review175299
---


On May 1, 2017, 5:30 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56140/
> ---
> 
> (Updated May 1, 2017, 5:30 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15160
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java
>  1487ed4f8e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 1b054a7e24 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/RowResolver.java 262dafb487 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 654f3b1772 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
> 8f8eab0d9c 
>   ql/src/test/queries/clientpositive/order_by_expr_1.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/order_by_expr_2.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out 873f1abb25 
>   ql/src/test/results/clientpositive/cp_sel.q.out 1778ccd6a6 
>   ql/src/test/results/clientpositive/druid_basic2.q.out 6177d56987 
>   ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out 2abb819558 
>   ql/src/test/results/clientpositive/groupby_grouping_sets_grouping.q.out 
> 473d17a1bd 
>   ql/src/test/results/clientpositive/llap/bucket_groupby.q.out d724131fca 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 584c3b5520 
>   ql/src/test/results/clientpositive/llap/limit_pushdown.q.out dd54dd22a6 
>   ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out 24645b6426 
>   ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
> 83de1fbea1 
>   ql/src/test/results/clientpositive/llap/vector_coalesce.q.out 578f849bdb 
>   ql/src/test/results/clientpositive/llap/vector_date_1.q.out a4f1050c89 
>   ql/src/test/results/clientpositive/llap/vector_decimal_2.q.out 144356c108 
>   ql/src/test/results/clientpositive/llap/vector_decimal_round.q.out 
> 8bd80cf860 
>   
> ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_grouping.q.out
>  5af9e61b0a 
>   
> ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out
>  f731ceecdc 
>   ql/src/test/results/clientpositive/llap/vector_interval_1.q.out debf5ab39e 
>   ql/src/test/results/clientpositive/llap/vector_interval_arithmetic.q.out 
> aadb6e72cd 
>   ql/src/test/results/clientpositive/order3.q.out 898f7a8853 
>   ql/src/test/results/clientpositive/order_by_expr_1.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/order_by_expr_2.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/pcr.q.out a1301fdf79 
>   ql/src/test/results/clientpositive/perf/query31.q.out 3ed312d3e3 
>   ql/src/test/results/clientpositive/perf/query36.q.out 57ab26acc6 
>   ql/src/test/results/clientpositive/perf/query39.q.out 19472c4d5e 
>   ql/src/test/results/clientpositive/perf/query42.q.out 3bebac3321 
>   ql/src/test/results/clientpositive/perf/query52.q.out 74ecaf28ba 
>   ql/src/test/results/clientpositive/perf/query64.q.out 6b42393aad 
>   ql/src/test/results/clientpositive/perf/query66.q.out 072bfee92b 
>   ql/src/test/results/clientpositive/perf/query70.q.out 8e42fac9c5 
>   ql/src/test/results/clientpositive/perf/query75.q.out b1e236d325 
>   ql/src/test/results/clientpositive/perf/query81.q.out a09d5c99b5 
>   ql/src/test/results/clientpositive/perf/query85.q.out 168bcd2a4a 
>   ql/src/test/results/clientpositive/perf/query86.q.out 734e6a480b 
>   ql/src/test/results/clientpositive/perf/query89.q.out 66481f710b 
>   ql/src/test/results/clientpositive/perf/query91.q.out e592bba8d9 
>   ql/src/test/results/clientpositive/pointlookup2.q.out 3438c74608 
>   ql/src/test/results/clientpositive/pointlookup3.q.out 2c3e39fd15 
>   ql/src/test/results/clientpositive/ppd_udf_case.q.out 7678d034

Re: Review Request 56140: Can't order by an unselected column

[pengcheng xiong]

> On May 18, 2017, 6:59 p.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/order_by_expr_1.q
> > Lines 42 (patched)
> > <https://reviews.apache.org/r/56140/diff/8/?file=1704001#file1704001line42>
> >
> > Can you also modify perfclidriver queries to their original form which 
> > contained such examples? They are currently rewritten to avoid this missing 
> > feature.

Open a new jira to track this.


- pengcheng


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/56140/#review175406
-------


On May 1, 2017, 5:30 p.m., pengcheng xiong wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/56140/
> ---
> 
> (Updated May 1, 2017, 5:30 p.m.)
> 
> 
> Review request for hive and Ashutosh Chauhan.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15160
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveProjectSortTransposeRule.java
>  1487ed4f8e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 1b054a7e24 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/RowResolver.java 262dafb487 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
> 654f3b1772 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
> 8f8eab0d9c 
>   ql/src/test/queries/clientpositive/order_by_expr_1.q PRE-CREATION 
>   ql/src/test/queries/clientpositive/order_by_expr_2.q PRE-CREATION 
>   ql/src/test/results/clientpositive/annotate_stats_select.q.out 873f1abb25 
>   ql/src/test/results/clientpositive/cp_sel.q.out 1778ccd6a6 
>   ql/src/test/results/clientpositive/druid_basic2.q.out 6177d56987 
>   ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out 2abb819558 
>   ql/src/test/results/clientpositive/groupby_grouping_sets_grouping.q.out 
> 473d17a1bd 
>   ql/src/test/results/clientpositive/llap/bucket_groupby.q.out d724131fca 
>   ql/src/test/results/clientpositive/llap/explainuser_1.q.out 584c3b5520 
>   ql/src/test/results/clientpositive/llap/limit_pushdown.q.out dd54dd22a6 
>   ql/src/test/results/clientpositive/llap/limit_pushdown3.q.out 24645b6426 
>   ql/src/test/results/clientpositive/llap/offset_limit_ppd_optimizer.q.out 
> 83de1fbea1 
>   ql/src/test/results/clientpositive/llap/vector_coalesce.q.out 578f849bdb 
>   ql/src/test/results/clientpositive/llap/vector_date_1.q.out a4f1050c89 
>   ql/src/test/results/clientpositive/llap/vector_decimal_2.q.out 144356c108 
>   ql/src/test/results/clientpositive/llap/vector_decimal_round.q.out 
> 8bd80cf860 
>   
> ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_grouping.q.out
>  5af9e61b0a 
>   
> ql/src/test/results/clientpositive/llap/vector_groupby_grouping_sets_limit.q.out
>  f731ceecdc 
>   ql/src/test/results/clientpositive/llap/vector_interval_1.q.out debf5ab39e 
>   ql/src/test/results/clientpositive/llap/vector_interval_arithmetic.q.out 
> aadb6e72cd 
>   ql/src/test/results/clientpositive/order3.q.out 898f7a8853 
>   ql/src/test/results/clientpositive/order_by_expr_1.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/order_by_expr_2.q.out PRE-CREATION 
>   ql/src/test/results/clientpositive/pcr.q.out a1301fdf79 
>   ql/src/test/results/clientpositive/perf/query31.q.out 3ed312d3e3 
>   ql/src/test/results/clientpositive/perf/query36.q.out 57ab26acc6 
>   ql/src/test/results/clientpositive/perf/query39.q.out 19472c4d5e 
>   ql/src/test/results/clientpositive/perf/query42.q.out 3bebac3321 
>   ql/src/test/results/clientpositive/perf/query52.q.out 74ecaf28ba 
>   ql/src/test/results/clientpositive/perf/query64.q.out 6b42393aad 
>   ql/src/test/results/clientpositive/perf/query66.q.out 072bfee92b 
>   ql/src/test/results/clientpositive/perf/query70.q.out 8e42fac9c5 
>   ql/src/test/results/clientpositive/perf/query75.q.out b1e236d325 
>   ql/src/test/results/clientpositive/perf/query81.q.out a09d5c99b5 
>   ql/src/test/results/clientpositive/perf/query85.q.out 168bcd2a4a 
>   ql/src/test/results/clientpositive/perf/query86.q.out 734e6a480b 
>   ql/src/test/results/clientpositive/perf/query89.q.out 66481f710b 
>   ql/src/test/results/clientpositive/perf/query91.q.out e592bba8d9 
>   ql/src/test/results/clientpositive/pointlookup2.q.out 3438c74608 
>   ql/src/test/results/clientpositive/pointlookup3.q.out 2c3e39fd15 
>   ql/src/test/results/clientpositive/ppd_udf_case.q.out 7678d03415 
>   ql/src/tes

[jira] [Created] (HIVE-16733) Support conflict column name in order by

[Pengcheng Xiong (JIRA)]

Pengcheng Xiong created HIVE-16733:
--

 Summary: Support conflict column name in order by
 Key: HIVE-16733
 URL: https://issues.apache.org/jira/browse/HIVE-16733
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong


There is a bug in RR which is exposed in HIVE-15160. After resolving the bug, 
we can support both:

select key as value from src order by src.value
select key as value from src order by value





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Review Request 59468: Optimize a combination of avg(), sum(), count(distinct) etc

[pengcheng xiong]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59468/
---

Review request for hive, Ashutosh Chauhan and Gopal V.


Repository: hive-git


Description
---

HIVE-16654


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7dedd23591 
  itests/src/test/resources/testconfiguration.properties e23ef6317f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 8b04cd44fa 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/CountDistinctRewriteProc.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/GroupByOptimizer.java 
3233157d8d 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 7dace9076f 
  ql/src/java/org/apache/hadoop/hive/ql/plan/GroupByDesc.java 38a9ef2af1 
  ql/src/test/queries/clientpositive/count_dist_rewrite.q PRE-CREATION 
  ql/src/test/results/clientpositive/count_dist_rewrite.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/groupby_sort_11.q.out 2b3bf4a07a 
  ql/src/test/results/clientpositive/groupby_sort_8.q.out 4faa0757cc 
  ql/src/test/results/clientpositive/llap/count_dist_rewrite.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/llap/metadataonly1.q.out 27218cf599 
  ql/src/test/results/clientpositive/nullgroup4.q.out e5a814 
  ql/src/test/results/clientpositive/perf/query16.q.out cf90c0c162 
  ql/src/test/results/clientpositive/perf/query28.q.out 78129cf68b 
  ql/src/test/results/clientpositive/perf/query94.q.out 836b16bf9f 
  ql/src/test/results/clientpositive/perf/query95.q.out fa94d0842b 
  ql/src/test/results/clientpositive/spark/nullgroup4.q.out 24f0291dec 
  ql/src/test/results/clientpositive/udf_count.q.out f60ad0485e 
  ql/src/test/results/clientpositive/vector_empty_where.q.out b2dec6d7f6 


Diff: https://reviews.apache.org/r/59468/diff/1/


Testing
---


Thanks,

pengcheng xiong