[jira] [Created] (HIVE-15666) Select query with view adds base table partition as direct input in spark engine

2017-01-19 Thread Niklaus Xiao (JIRA) 
Niklaus Xiao created HIVE-15666:
---

 Summary: Select query with view adds base table partition as 
direct input in spark engine
 Key: HIVE-15666
 URL: https://issues.apache.org/jira/browse/HIVE-15666
 Project: Hive
  Issue Type: Bug
  Components: Authorization
Affects Versions: 1.3.0
Reporter: Niklaus Xiao


repo steps:
{code}
set hive.execution.engine=spark;
create table base(id int) partitioned by (dt string);
alter table base add partition(dt='2017');

create view view1 as select * from base where id < 10;
select * from view1;
{code}

 it requires the access not only for view1 but also for base@dt=2017 partition, 
which should not be required.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15667) hive-blobstore tests are failing due to output differences

2017-01-19 Thread JIRA 
Sergio Peña created HIVE-15667:
--

 Summary: hive-blobstore tests are failing due to output differences
 Key: HIVE-15667
 URL: https://issues.apache.org/jira/browse/HIVE-15667
 Project: Hive
  Issue Type: Bug
  Components: Tests
Affects Versions: 2.2.0
Reporter: Sergio Peña
Assignee: Sergio Peña
 Attachments: HIVE-15667.1.patch

All itests/hive-blobstore are failing and their .q.out files need to be updated.

CC: [~poeppt]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 55723: HIVE-15667: TestBlobstoreCliDriver tests are failing due to output differences

2017-01-19 Thread Sergio Pena 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55723/
---

Review request for hive and Thomas Poepping.


Bugs: HIVE-15667
https://issues.apache.org/jira/browse/HIVE-15667


Repository: hive-git


Description
---

Updated q.out files.


Diffs
-

  itests/hive-blobstore/src/test/results/clientpositive/ctas.q.out 
9f25b26213674c73c05639cda9f108511289bafc 
  
itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out
 d7613f3b02f47fc16fac23baa74992888f477248 
  itests/hive-blobstore/src/test/results/clientpositive/insert_into_table.q.out 
fe7fdb03122a583b9eaae5365759a104c7b1815e 
  
itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_directory.q.out
 82fb95d6f7fa7d1183c97602168eb7f1d83b1565 
  
itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out
 4d0c1535eee8b0485728d3654e9c47eefdefc242 
  
itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_table.q.out
 82ae25f3156077201ad0a0fbc4a2065d99c17b5d 
  
itests/hive-blobstore/src/test/results/clientpositive/write_final_output_blobstore.q.out
 1b1ea971d0bb9fcca41e029bc7d748638c3aa01e 

Diff: https://reviews.apache.org/r/55723/diff/


Testing
---

All blobstore tests worked locally.
Waiting for Jenkins tests.


Thanks,

Sergio Pena

Re: Review Request 55156: Min-max runtime filtering

2017-01-19 Thread Jason Dere 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55156/#review162321
---




ql/src/test/queries/clientpositive/dynamic_semijoin_reduction.q (line 33)


This still looks off in the latest version of the patch - line 28 sets 
hive.tez.dynamic.semijoin.reduction=false, a statement is explained/run, and 
then line 33 sets hive.tez.dynamic.semijoin.reduction to false again, and 
explains/runs the same statement.

And then it is set to true after running the 2nd set of statements.


- Jason Dere


On Jan. 18, 2017, 8:42 p.m., Deepak Jaiswal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55156/
> ---
> 
> (Updated Jan. 18, 2017, 8:42 p.m.)
> 
> 
> Review request for hive, Gopal V, Gunther Hagleitner, Jason Dere, Prasanth_J, 
> and Rajesh Balamohan.
> 
> 
> Bugs: HIVE-15269
> https://issues.apache.org/jira/browse/HIVE-15269
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-15269 min-max runtime filtering.
> The patch also contains the patch for HIVE-15270.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 96928db 
>   itests/src/test/resources/testconfiguration.properties be5a747 
>   orc/src/test/org/apache/orc/impl/TestRecordReaderImpl.java cdd62ac 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java 
> 69ba4a2 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java 940f2dd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DynamicValueRegistry.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeColumnEvaluator.java 
> 24c8281 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeConstantDefaultEvaluator.java
>  89a75eb 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeConstantEvaluator.java 
> 4fe72a0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeDynamicValueEvaluator.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluator.java b8d6ab7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorFactory.java 
> 0d03d8f 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorHead.java 
> 42685fb 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorRef.java 
> 0a6b66a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeFieldEvaluator.java 
> ff32626 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java 
> 221abd9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java bd0d28c 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 6f01da0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java d98cea9 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 
> ac5331e 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java 9718c48 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCache.java 440e0a1 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/ObjectCacheWrapper.java 9768efa 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java 9049ddd 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ObjectCache.java 008f8a4 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/DynamicValueRegistryTez.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/LlapObjectCache.java 0141230 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 
> 955fa80 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ObjectCache.java 06dca00 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ReduceRecordProcessor.java 
> d80f201 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorMapJoinOperator.java 
> 0cb6c8a 
>   
> ql/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorSMBMapJoinOperator.java
>  80b0a14 
>   ql/src/java/org/apache/hadoop/hive/ql/io/sarg/ConvertAstToSearchArg.java 
> 9d900e4 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/DynamicPartitionPruningOptimization.java
>  26fcc45 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/FixedBucketPruningOptimizer.java
>  9e9beb0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/RedundantDynamicPruningConditionsRemoval.java
>  d9ce017 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/stats/annotation/StatsRulesProcFactory.java
>  aa1e509 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/GenTezUtils.java e2363eb 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 35f34da 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/RuntimeValuesInfo.java 
> PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TaskCompiler.java e8b003e 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java cdb9e

[jira] [Created] (HIVE-15668) change REPL DUMP syntax to use "LIMIT" instead of "BATCH" keyword

2017-01-19 Thread Sushanth Sowmyan (JIRA) 
Sushanth Sowmyan created HIVE-15668:
---

 Summary: change REPL DUMP syntax to use "LIMIT" instead of "BATCH" 
keyword
 Key: HIVE-15668
 URL: https://issues.apache.org/jira/browse/HIVE-15668
 Project: Hive
  Issue Type: Sub-task
Reporter: Sushanth Sowmyan
Assignee: Sushanth Sowmyan


Currently, REPL DUMP syntax goes:

{noformat}
REPL DUMP [[.]] [FROM  [BATCH ]]
{noformat}

The BATCH directive says that when doing an event dump, to not dump out more 
than _batchSize_ number of events. However, there is a clearer keyword for the 
same effect, and that is LIMIT. Thus, rephrasing the syntax as follows makes it 
clearer:

{noformat}
REPL DUMP [[.]] [FROM  [LIMIT ]]
{noformat}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 55731: HIVE-15653: Some ALTER TABLE commands drop table stats

2017-01-19 Thread Chaoyu Tang 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55731/
---

Review request for hive and pengcheng xiong.


Bugs: HIVE-15653
https://issues.apache.org/jira/browse/HIVE-15653


Repository: hive-git


Description
---

For most of alter table operations like table rename, add columns, change 
column type etc (besides the set table properties), the table stats status 
should not change. But for some other operations like update statistics, change 
location, the basic stats status should change.


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a1fb874 
  ql/src/test/queries/clientpositive/alter_table_stats_status.q PRE-CREATION 
  ql/src/test/results/clientpositive/alter_table_stats_status.q.out 
PRE-CREATION 

Diff: https://reviews.apache.org/r/55731/diff/


Testing
---

1. Manual tests
2. new unit tests


Thanks,

Chaoyu Tang

Re: Review Request 55731: HIVE-15653: Some ALTER TABLE commands drop table stats

2017-01-19 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55731/#review162357
---




ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java (line 3645)


If it is for user update stats, you will have 
StatsSetupConst.STATS_GENERATED = StatsSetupConst.USER automatically. Thus it 
is not necessary to have/call hasStatsInParameters function.



ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java (line 3655)


The same as above



ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java (line 3657)


And, could u just remove the property rather than set it to false? We treat 
the missing of the property the same as set it to false.



ql/src/test/queries/clientpositive/alter_table_stats_status.q (line 1)


Could u add some more test cases for partition stats? Thanks.


- pengcheng xiong


On Jan. 19, 2017, 10:29 p.m., Chaoyu Tang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/55731/
> ---
> 
> (Updated Jan. 19, 2017, 10:29 p.m.)
> 
> 
> Review request for hive and pengcheng xiong.
> 
> 
> Bugs: HIVE-15653
> https://issues.apache.org/jira/browse/HIVE-15653
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> For most of alter table operations like table rename, add columns, change 
> column type etc (besides the set table properties), the table stats status 
> should not change. But for some other operations like update statistics, 
> change location, the basic stats status should change.
> 
> 
> Diffs
> -
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java a1fb874 
>   ql/src/test/queries/clientpositive/alter_table_stats_status.q PRE-CREATION 
>   ql/src/test/results/clientpositive/alter_table_stats_status.q.out 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/55731/diff/
> 
> 
> Testing
> ---
> 
> 1. Manual tests
> 2. new unit tests
> 
> 
> Thanks,
> 
> Chaoyu Tang
> 
>

Re: Review Request 55553: HIVE-15544 Support for scalar subqueries in filter (WHERE/HAVING)

2017-01-19 Thread Vineet Garg 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3/
---

(Updated Jan. 19, 2017, 11:58 p.m.)


Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.


Changes
---

Addressed review comments


Bugs: HIVE-15544
https://issues.apache.org/jira/browse/HIVE-15544


Repository: hive-git


Description
---

This patch provides support for scalar subqueries in filter clause. Note that 
this patch disables the following:
* IN/NOT IN correlated subqueries containing aggregates (HIVE checks for such 
queries and throw an exception)
* SCALAR correlated subqueries containing aggregates with non-equi join 
predicates on correlated columns (HIVE throws an exception for such queries).


Diffs (updated)
-

  itests/src/test/resources/testconfiguration.properties be5a747 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 6f01da0 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSubquerySemanticException.java
 PRE-CREATION 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HivePlannerContext.java 
8beb0dd 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
 a373cdd 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java
 f1e8ebd 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
 8d2e535 
  ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 9f1b9d5 
  ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g cd9adfc 
  ql/src/java/org/apache/hadoop/hive/ql/parse/QBSubQuery.java 24381b9 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java f275f6a 
  ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 6c30efd 
  ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeSubQueryDesc.java aec331b 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSQCountCheck.java 
PRE-CREATION 
  
ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/TestCBORuleFiredOnlyOnce.java
 44e157b 
  ql/src/test/queries/clientnegative/subquery_exists_implicit_gby.q 9013df6 
  ql/src/test/queries/clientnegative/subquery_in_implicit_gby.q PRE-CREATION 
  ql/src/test/queries/clientnegative/subquery_notexists_implicit_gby.q 852b295 
  ql/src/test/queries/clientnegative/subquery_scalar_implicit_gby.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/subquery_scalar_multi_columns.q 
PRE-CREATION 
  ql/src/test/queries/clientnegative/subquery_scalar_multi_rows.q PRE-CREATION 
  ql/src/test/queries/clientnegative/subquery_with_or_cond.q c2c3221 
  ql/src/test/queries/clientpositive/perf/query1.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query23.q e8ebd86 
  ql/src/test/queries/clientpositive/perf/query30.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query6.q PRE-CREATION 
  ql/src/test/queries/clientpositive/perf/query81.q PRE-CREATION 
  ql/src/test/queries/clientpositive/subquery_exists.q cac8e1b 
  ql/src/test/queries/clientpositive/subquery_in.q fe0c9c8 
  ql/src/test/queries/clientpositive/subquery_in_having.q 40b7e32 
  ql/src/test/queries/clientpositive/subquery_notin.q f9b5405 
  ql/src/test/queries/clientpositive/subquery_scalar.q PRE-CREATION 
  ql/src/test/queries/clientpositive/subquery_unqualcolumnrefs.q bdfa648 
  ql/src/test/results/clientnegative/subquery_exists_implicit_gby.q.out f7251e3 
  ql/src/test/results/clientnegative/subquery_in_implicit_gby.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/subquery_multiple_cols_in_select.q.out 
7a16bae 
  ql/src/test/results/clientnegative/subquery_notexists_implicit_gby.q.out 
da38f5f 
  ql/src/test/results/clientnegative/subquery_scalar_implicit_gby.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/subquery_scalar_multi_columns.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/subquery_scalar_multi_rows.q.out 
PRE-CREATION 
  ql/src/test/results/clientnegative/subquery_windowing_corr.q.out dcd3026 
  ql/src/test/results/clientnegative/subquery_with_or_cond.q.out d2d743d 
  ql/src/test/results/clientpositive/llap/subquery_exists.q.out 35c64e7 
  ql/src/test/results/clientpositive/llap/subquery_in.q.out 95c13e6 
  ql/src/test/results/clientpositive/llap/subquery_multi.q.out 536bfdb 
  ql/src/test/results/clientpositive/llap/subquery_notin.q.out 21dea74 
  ql/src/test/results/clientpositive/llap/subquery_scalar.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query1.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query23.q.out 6d4cfca 
  ql/src/test/results/clientpositive/perf/query30.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query6.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/perf/query81.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/subquery_exists.q.out 9c02d6a 
  ql/src/te

Re: Review Request 55553: HIVE-15544 Support for scalar subqueries in filter (WHERE/HAVING)

2017-01-19 Thread Vineet Garg 


> On Jan. 19, 2017, 7:53 a.m., Ashutosh Chauhan wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java,
> >  line 234
> > 
> >
> > Any reason for disabling this rule?

For nested scalar subqueries this rule was, for whatever reason, pushing filter 
on top of LogicalCorrelate to within left input of LogicalCorrelate, which is 
wrong and causes an exception during decorrelation. I'll add a comment and plan 
to open a JIRA to investigate why it is doing so.


> On Jan. 19, 2017, 7:53 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/subquery_in.q, line 63
> > 
> >
> > Instead of removing you may add group by in subquery, unless this query 
> > pattern is covered elsewhere in tests.

This was removed since now we disable corr IN/NOT IN queries with aggregate


> On Jan. 19, 2017, 7:53 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/subquery_in_having.q, line 43
> > 
> >
> > This satisifies all conditions: correlated and no implied gby. Will 
> > this still wont work?

That criteria works for Scalar queries only for now. For IN/NOT IN we disable 
corr queries with aggregate for now.


> On Jan. 19, 2017, 7:53 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/subquery_in_having.q, line 128
> > 
> >
> > Dont we allow correlated predicates here?

same reason as above.


> On Jan. 19, 2017, 7:53 a.m., Ashutosh Chauhan wrote:
> > ql/src/test/queries/clientpositive/subquery_notin.q, line 65
> > 
> >
> > add gby ?

this was remove because of same reason as above


- Vineet


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/3/#review162239
---


On Jan. 19, 2017, 11:58 p.m., Vineet Garg wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/3/
> ---
> 
> (Updated Jan. 19, 2017, 11:58 p.m.)
> 
> 
> Review request for hive, Ashutosh Chauhan and Jesús Camacho Rodríguez.
> 
> 
> Bugs: HIVE-15544
> https://issues.apache.org/jira/browse/HIVE-15544
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> This patch provides support for scalar subqueries in filter clause. Note that 
> this patch disables the following:
> * IN/NOT IN correlated subqueries containing aggregates (HIVE checks for such 
> queries and throw an exception)
> * SCALAR correlated subqueries containing aggregates with non-equi join 
> predicates on correlated columns (HIVE throws an exception for such queries).
> 
> 
> Diffs
> -
> 
>   itests/src/test/resources/testconfiguration.properties be5a747 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 6f01da0 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/CalciteSubquerySemanticException.java
>  PRE-CREATION 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/HivePlannerContext.java
>  8beb0dd 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveRelDecorrelator.java
>  a373cdd 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java
>  f1e8ebd 
>   
> ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/RexNodeConverter.java
>  8d2e535 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 9f1b9d5 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/IdentifiersParser.g cd9adfc 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/QBSubQuery.java 24381b9 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java f275f6a 
>   ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 
> 6c30efd 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/ExprNodeSubQueryDesc.java 
> aec331b 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFSQCountCheck.java 
> PRE-CREATION 
>   
> ql/src/test/org/apache/hadoop/hive/ql/optimizer/calcite/TestCBORuleFiredOnlyOnce.java
>  44e157b 
>   ql/src/test/queries/clientnegative/subquery_exists_implicit_gby.q 9013df6 
>   ql/src/test/queries/clientnegative/subquery_in_implicit_gby.q PRE-CREATION 
>   ql/src/test/queries/clientnegative/subquery_notexists_implicit_gby.q 
> 852b295 
>   ql/src/test/queries/clientnegative/subquery_scalar_implicit_gby.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/subquery_scalar_multi_columns.q 
> PRE-CREATION 
>   ql/src/test/queries/clientnegative/

[jira] [Created] (HIVE-15669) LLAP: Improve aging in shortest job first scheduler

2017-01-19 Thread Prasanth Jayachandran (JIRA) 
Prasanth Jayachandran created HIVE-15669:


 Summary: LLAP: Improve aging in shortest job first scheduler
 Key: HIVE-15669
 URL: https://issues.apache.org/jira/browse/HIVE-15669
 Project: Hive
  Issue Type: Bug
  Components: llap
Affects Versions: 2.2.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


Under high concurrency, some jobs can gets starved for longer time when 
hive.llap.task.scheduler.locality.delay is set to -1 (infinitely wait for 
locality).





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15670) column_stats_accurate may not fit in PARTITION_PARAMS.VALUE

2017-01-19 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-15670:
---

 Summary: column_stats_accurate may not fit in 
PARTITION_PARAMS.VALUE
 Key: HIVE-15670
 URL: https://issues.apache.org/jira/browse/HIVE-15670
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin


The JSON can be too big with many columns.
We can make JSON more compact by only storing the list of columns with true 
values. Or we can even store a bitmask in a dedicated column, and adjust it 
when altering table (rare enough). Or we can just change the VALUE column to 
text blob (might be a painful change wrt upgrade scripts, and supporting all 
the DBs' varied blob implementations, esp. in directsql).
Storing denormalized flags in a separate table will probably be slow, 
comparatively.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15671) RPCServer.registerClient() erroneously uses server/client handshake timeout for connection timeout

2017-01-19 Thread Xuefu Zhang (JIRA) 
Xuefu Zhang created HIVE-15671:
--

 Summary: RPCServer.registerClient() erroneously uses server/client 
handshake timeout for connection timeout
 Key: HIVE-15671
 URL: https://issues.apache.org/jira/browse/HIVE-15671
 Project: Hive
  Issue Type: Bug
  Components: Spark
Affects Versions: 1.1.0
Reporter: Xuefu Zhang
Assignee: Xuefu Zhang


{code}
  /**
   * Tells the RPC server to expect a connection from a new client.
   * ...
   */
  public Future registerClient(final String clientId, String secret,
  RpcDispatcher serverDispatcher) {
return registerClient(clientId, secret, serverDispatcher, 
config.getServerConnectTimeoutMs());
  }
{code}

config.getServerConnectTimeoutMs() returns value for 
hive.spark.client.server.connect.timeout, which is meant for timeout for 
handshake between Hive client and remote Spark driver. Instead, the timeout 
should be hive.spark.client.connect.timeout, which is for timeout for remote 
Spark driver in connecting back to Hive client.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15672) LLAP text cache: improve first query perf II

2017-01-19 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-15672:
---

 Summary: LLAP text cache: improve first query perf II
 Key: HIVE-15672
 URL: https://issues.apache.org/jira/browse/HIVE-15672
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin


1) Don't use ORC dictionary.
2) Use VectorDeserialize.
3) Don't parse the columns that are not included (cannot avoid reading them).
4) Send VRB to the pipeline and write ORC in parallel (in background).

Also add an option to disable the encoding pipeline server-side.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 55746: HIVE-15664 LLAP text cache: improve first query perf I

2017-01-19 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/55746/
---

Review request for hive and Gopal V.


Repository: hive-git


Description
---

see jira


Diffs
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7f1f91a 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/EvictionDispatcher.java 
2d3197c 
  
llap-server/src/java/org/apache/hadoop/hive/llap/cache/SerDeLowLevelCacheImpl.java
 caa60e5 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapIoImpl.java 
7cfd133 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapRecordReader.java
 1a76eae 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ColumnVectorProducer.java
 2e4f2ba 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/GenericColumnVectorProducer.java
 5cddae5 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java
 565e3d2 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/SerDeEncodedDataReader.java
 dd189f1 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/VertorDeserializeOrcWriter.java
 PRE-CREATION 
  orc/src/java/org/apache/orc/OrcConf.java 357318d 
  orc/src/java/org/apache/orc/impl/WriterImpl.java 988d9d8 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 3ee8fdc 
  ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java e3158f6 
  
storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/VectorizedRowBatch.java
 9c066e0 

Diff: https://reviews.apache.org/r/55746/diff/


Testing
---


Thanks,

Sergey Shelukhin

[jira] [Created] (HIVE-15673) Allow multiple queries with disjunction

2017-01-19 Thread Vineet Garg (JIRA) 
Vineet Garg created HIVE-15673:
--

 Summary: Allow multiple queries with disjunction
 Key: HIVE-15673
 URL: https://issues.apache.org/jira/browse/HIVE-15673
 Project: Hive
  Issue Type: Sub-task
  Components: Query Planning
Reporter: Vineet Garg
Assignee: Vineet Garg


HIVE currently doesn't allow multiple subqueries with {{OR}} since calcite has 
a bug in determining logic for OR expression. See [CALCITE-1546 
|https://issues.apache.org/jira/browse/CALCITE-1546].

Once calcite is released containing fix for the bug HIVE will need to lift the 
restriction and add tests cases



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15674) Add more setOp tests to HivePerfCliDriver

2017-01-19 Thread Pengcheng Xiong (JIRA) 
Pengcheng Xiong created HIVE-15674:
--

 Summary: Add more setOp tests to HivePerfCliDriver
 Key: HIVE-15674
 URL: https://issues.apache.org/jira/browse/HIVE-15674
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15675) ql.hooks.TestQueryHooks failure

2017-01-19 Thread Jun He (JIRA) 
Jun He created HIVE-15675:
-

 Summary: ql.hooks.TestQueryHooks failure
 Key: HIVE-15675
 URL: https://issues.apache.org/jira/browse/HIVE-15675
 Project: Hive
  Issue Type: Bug
  Components: Tests
Reporter: Jun He






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15676) Remove Bloom Filters from semi join reduction if it is too big.

2017-01-19 Thread Deepak Jaiswal (JIRA) 
Deepak Jaiswal created HIVE-15676:
-

 Summary: Remove Bloom Filters from semi join reduction if it is 
too big.
 Key: HIVE-15676
 URL: https://issues.apache.org/jira/browse/HIVE-15676
 Project: Hive
  Issue Type: Improvement
Reporter: Deepak Jaiswal
Assignee: Deepak Jaiswal


Bloom filters themselves could become really big if the row count is high. 
Aggregating such bloom filters in reducers could be even more expensive. For 
e.g., a bloom filter for 100M rows can be as big as 170MB. Aggregating 100 such 
filters in reducer could end up taking 17GB of memory.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-15677) Remove semijoin optimization if the tables involved have PK-FK relationship

2017-01-19 Thread Deepak Jaiswal (JIRA) 
Deepak Jaiswal created HIVE-15677:
-

 Summary: Remove semijoin optimization if the tables involved have 
PK-FK relationship
 Key: HIVE-15677
 URL: https://issues.apache.org/jira/browse/HIVE-15677
 Project: Hive
  Issue Type: Improvement
Reporter: Deepak Jaiswal
Assignee: Deepak Jaiswal


If a semijoin optimization is created on tables which have PK-FK relationship 
on join keys(e.g. customer-order ) then the reduction will not take place.
Remove the optimization in such cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)