[jira] [Created] (HIVE-12788) Setting hive.optimize.union.remove to TRUE will break UNION ALL with aggregate functions

2016-01-06 Thread Eric Lin (JIRA) 
Eric Lin created HIVE-12788:
---

 Summary: Setting hive.optimize.union.remove to TRUE will break 
UNION ALL with aggregate functions
 Key: HIVE-12788
 URL: https://issues.apache.org/jira/browse/HIVE-12788
 Project: Hive
  Issue Type: Bug
  Components: Hive
Affects Versions: 1.1.1
Reporter: Eric Lin


See the test case below:

{code}
0: jdbc:hive2://localhost:1/default> create table test (a int);

0: jdbc:hive2://localhost:1/default> set hive.optimize.union.remove=true;
No rows affected (0.01 seconds)

0: jdbc:hive2://localhost:1/default> set 
hive.mapred.supports.subdirectories=true;
No rows affected (0.007 seconds)

0: jdbc:hive2://localhost:1/default> SELECT COUNT(1) FROM test UNION ALL 
SELECT COUNT(1) FROM test;
+--+--+
| _u1._c0  |
+--+--+
+--+--+
{code}

Run the same query without setting hive.mapred.supports.subdirectories and 
hive.optimize.union.remove to true will give correct result:

{code}
0: jdbc:hive2://localhost:1/default> SELECT COUNT(1) FROM test UNION ALL 
SELECT COUNT(1) FROM test;
+--+--+
| _u1._c0  |
+--+--+
| 1|
| 1|
+--+--+
{code}

UNION ALL without COUNT function will work as expected:

{code}
0: jdbc:hive2://localhost:1/default> select * from test UNION ALL SELECT * 
FROM test;
++--+
| _u1.a  |
++--+
| 1  |
| 1  |
++--+
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12789) Fix output twice in the history command of Beeline

2016-01-06 Thread Shinichi Yamashita (JIRA) 
Shinichi Yamashita created HIVE-12789:
-

 Summary: Fix output twice in the history command of Beeline
 Key: HIVE-12789
 URL: https://issues.apache.org/jira/browse/HIVE-12789
 Project: Hive
  Issue Type: Bug
  Components: Beeline
Reporter: Shinichi Yamashita
Assignee: Shinichi Yamashita


When I revised HIVE-12780, I confirmed that a result of history output it twice.

{quote}
[root@hadoop ~]# cat ~/.beeline/history
select 1;
select 2;
select 3;
[root@hadoop ~]# beeline
which: no hbase in 
(/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/hadoop/bin:/usr/local/hive/bin:/usr/pgsql-9.4/bin:/root/bin)
Beeline version 2.1.0-SNAPSHOT by Apache Hive
beeline> !history
1.0: select 1;
1.1: select 2;
1.2: select 3;
1.3: select 1;
1.4: select 2;
1.5: select 3;
{quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 41984: HIVE-12762: Common join on parquet tables returns incorrect result when hive.optimize.index.filter set to true

2016-01-06 Thread Aihua Xu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41984/
---

Review request for hive, Sergio Pena and Xuefu Zhang.


Repository: hive-git


Description
---

HIVE-12762: Common join on parquet tables returns incorrect result when 
hive.optimize.index.filter set to true


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 
9a7d990baaabfde8e564f00bb1fcfe30cd16dc90 
  ql/src/java/org/apache/hadoop/hive/ql/io/parquet/ProjectionPusher.java 
017676bec2163f04fb95f43224a4f8743fa49f55 
  ql/src/test/queries/clientpositive/parquet_join2.q PRE-CREATION 
  ql/src/test/results/clientpositive/parquet_join2.q.out PRE-CREATION 
  storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/ExpressionTree.java 
577d95d1a15a54c2804349b3e5e68d83b72df664 
  
storage-api/src/java/org/apache/hadoop/hive/ql/io/sarg/SearchArgumentImpl.java 
eeff131cbc14d7ef554517109612ae7d891f8003 

Diff: https://reviews.apache.org/r/41984/diff/


Testing
---

We have two issues: 1. We are filtering the parquet columns based on the last 
filter condition in the query. So if the query contains multiple instances of 
the same table, e.g., join on the same table with different filter conditions, 
then we could get incorrect result; 2. rewriteLeaves implementation in 
SearchArgumentImpl is not accurate since the different leaves could be sharing 
the same object. The current implementation could change the leave index 
multiple times to an incorrect value.

The patch will merge all the filter conditions (create OR expression on all the 
filters) so that the columns which will be used during operator won't be 
filtered during earlier splitting stage. rewriteLeaves is reimplemented to get 
all the unique leaves first and replace in place.


Thanks,

Aihua Xu

[jira] [Created] (HIVE-12790) Metastore connection leaks in HiveServer2

2016-01-06 Thread Naveen Gangam (JIRA) 
Naveen Gangam created HIVE-12790:


 Summary: Metastore connection leaks in HiveServer2
 Key: HIVE-12790
 URL: https://issues.apache.org/jira/browse/HIVE-12790
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 1.1.0
Reporter: Naveen Gangam
Assignee: Naveen Gangam


HiveServer2 keeps opening new connections to HMS each time it launches a task. 
These connections do not appear to be closed when the task completes thus 
causing a HMS connection leak. "lsof" for the HS2 process shows connections to 
port 9083.

{code}
2015-12-03 04:20:56,352 INFO  [HiveServer2-Background-Pool: Thread-424756()]: 
ql.Driver (SessionState.java:printInfo(558)) - Launching Job 11 out of 41
2015-12-03 04:20:56,354 INFO  [Thread-405728()]: hive.metastore 
(HiveMetaStoreClient.java:open(311)) - Trying to connect to metastore with URI 
thrift://:9083
2015-12-03 04:20:56,360 INFO  [Thread-405728()]: hive.metastore 
(HiveMetaStoreClient.java:open(351)) - Opened a connection to metastore, 
current connections: 14824
2015-12-03 04:20:56,360 INFO  [Thread-405728()]: hive.metastore 
(HiveMetaStoreClient.java:open(400)) - Connected to metastore.

2015-12-03 04:21:06,355 INFO  [HiveServer2-Background-Pool: Thread-424756()]: 
ql.Driver (SessionState.java:printInfo(558)) - Launching Job 12 out of 41
2015-12-03 04:21:06,357 INFO  [Thread-405756()]: hive.metastore 
(HiveMetaStoreClient.java:open(311)) - Trying to connect to metastore with URI 
thrift://:9083
2015-12-03 04:21:06,362 INFO  [Thread-405756()]: hive.metastore 
(HiveMetaStoreClient.java:open(351)) - Opened a connection to metastore, 
current connections: 14825
2015-12-03 04:21:06,362 INFO  [Thread-405756()]: hive.metastore 
(HiveMetaStoreClient.java:open(400)) - Connected to metastore.
...
2015-12-03 04:21:08,357 INFO  [HiveServer2-Background-Pool: Thread-424756()]: 
ql.Driver (SessionState.java:printInfo(558)) - Launching Job 13 out of 41
2015-12-03 04:21:08,360 INFO  [Thread-405782()]: hive.metastore 
(HiveMetaStoreClient.java:open(311)) - Trying to connect to metastore with URI 
thrift://:9083
2015-12-03 04:21:08,364 INFO  [Thread-405782()]: hive.metastore 
(HiveMetaStoreClient.java:open(351)) - Opened a connection to metastore, 
current connections: 14826
2015-12-03 04:21:08,365 INFO  [Thread-405782()]: hive.metastore 
(HiveMetaStoreClient.java:open(400)) - Connected to metastore.
... 
{code}

The TaskRunner thread starts a new SessionState each time, which creates a new 
connection to the HMS (via Hive.get(conf).getMSC()) that is never closed.

Even SessionState.close(), currently not being called by the TaskRunner thread, 
does not close this connection.

Attaching a anonymized log snippet where the number of HMS connections reaches 
north of 25000+ connections.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12791) Truncated table stats should return 0 as datasize

2016-01-06 Thread Pengcheng Xiong (JIRA) 
Pengcheng Xiong created HIVE-12791:
--

 Summary: Truncated table stats should return 0 as datasize
 Key: HIVE-12791
 URL: https://issues.apache.org/jira/browse/HIVE-12791
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong


{code}
create table s as select * from src;
truncate table s;
hive> explain select * from s;
OK
STAGE DEPENDENCIES:
  Stage-0 is a root stage

STAGE PLANS:
  Stage: Stage-0
Fetch Operator
  limit: -1
  Processor Tree:
TableScan
  alias: s
  Statistics: Num rows: 29 Data size: 5812 Basic stats: COMPLETE Column 
stats: NONE
  Select Operator
expressions: key (type: string), value (type: string)
outputColumnNames: _col0, _col1
Statistics: Num rows: 29 Data size: 5812 Basic stats: COMPLETE 
Column stats: NONE
ListSink

Time taken: 0.048 seconds, Fetched: 17 row(s)
{code}

should be 
{code}
Num rows: 1 Data size: 0
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12792) HIVE-12075 didn't update operation type for plugins

2016-01-06 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-12792:
---

 Summary: HIVE-12075 didn't update operation type for plugins
 Key: HIVE-12792
 URL: https://issues.apache.org/jira/browse/HIVE-12792
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.1.0
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 41482: StatsSetupConst.COLUMN_STATS_ACCURATE is not used correctly

2016-01-06 Thread pengcheng xiong 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41482/
---

(Updated Jan. 6, 2016, 9:30 p.m.)


Review request for hive and Ashutosh Chauhan.


Repository: hive-git


Description
---

HIVE-12661


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/StatsSetupConst.java 2ff76ee 
  metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 
432f7d0 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java f6fbe74 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java ac0ecd9 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java dca1198 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java f151871 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/StatsOptimizer.java 03c1c3f 
  ql/src/test/queries/clientpositive/columnStatsUpdateForStatsOptimizer.q 
PRE-CREATION 
  ql/src/test/results/clientnegative/stats_partialscan_autogether.q.out d03c249 
  ql/src/test/results/clientnegative/unset_table_property.q.out 158ed38 
  ql/src/test/results/clientpositive/alter_file_format.q.out c9e88f8 
  ql/src/test/results/clientpositive/alter_merge_stats_orc.q.out cefe069 
  
ql/src/test/results/clientpositive/alter_numbuckets_partitioned_table2_h23.q.out
 29a4c4b 
  
ql/src/test/results/clientpositive/alter_numbuckets_partitioned_table_h23.q.out 
4d0f841 
  ql/src/test/results/clientpositive/alter_partition_clusterby_sortby.q.out 
f5e8d1f 
  ql/src/test/results/clientpositive/alter_partition_coltype.q.out 08bd2b8 
  ql/src/test/results/clientpositive/alter_skewed_table.q.out 03904e6 
  ql/src/test/results/clientpositive/alter_table_not_sorted.q.out 40974e4 
  ql/src/test/results/clientpositive/alter_table_serde2.q.out 3b63e7d 
  ql/src/test/results/clientpositive/analyze_table_null_partition.q.out c7254d1 
  ql/src/test/results/clientpositive/auto_join_reordering_values.q.out 91b89d7 
  ql/src/test/results/clientpositive/auto_sortmerge_join_1.q.out a48c148 
  ql/src/test/results/clientpositive/auto_sortmerge_join_11.q.out 206a619 
  ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out 69ae0bf 
  ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out fbc500b 
  ql/src/test/results/clientpositive/auto_sortmerge_join_3.q.out 9ae6861 
  ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out 357bc4c 
  ql/src/test/results/clientpositive/auto_sortmerge_join_5.q.out b763e52 
  ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out cef3bd3 
  ql/src/test/results/clientpositive/auto_sortmerge_join_8.q.out 567e3cf 
  ql/src/test/results/clientpositive/binary_output_format.q.out 30d0037 
  ql/src/test/results/clientpositive/bucket1.q.out 8009514 
  ql/src/test/results/clientpositive/bucket2.q.out 75de8a9 
  ql/src/test/results/clientpositive/bucket3.q.out c459870 
  ql/src/test/results/clientpositive/bucket4.q.out e0e5fa6 
  ql/src/test/results/clientpositive/bucket5.q.out c1dd90d 
  ql/src/test/results/clientpositive/bucket_many.q.out 04ec255 
  ql/src/test/results/clientpositive/bucket_map_join_1.q.out c7a8a20 
  ql/src/test/results/clientpositive/bucket_map_join_2.q.out 3c3793f 
  ql/src/test/results/clientpositive/bucket_map_join_spark1.q.out 8411a9b 
  ql/src/test/results/clientpositive/bucket_map_join_spark2.q.out 008d57a 
  ql/src/test/results/clientpositive/bucket_map_join_spark3.q.out d5ca1ad 
  ql/src/test/results/clientpositive/bucket_map_join_spark4.q.out 42abaa3 
  ql/src/test/results/clientpositive/bucketcontext_1.q.out d28e582 
  ql/src/test/results/clientpositive/bucketcontext_2.q.out 9f92030 
  ql/src/test/results/clientpositive/bucketcontext_3.q.out 5a847f0 
  ql/src/test/results/clientpositive/bucketcontext_4.q.out 19ed7f9 
  ql/src/test/results/clientpositive/bucketcontext_5.q.out e1a911b 
  ql/src/test/results/clientpositive/bucketcontext_6.q.out 0d3bf6e 
  ql/src/test/results/clientpositive/bucketcontext_7.q.out 15f1335 
  ql/src/test/results/clientpositive/bucketcontext_8.q.out 25d0c3c 
  ql/src/test/results/clientpositive/bucketmapjoin1.q.out 8999fe6 
  ql/src/test/results/clientpositive/bucketmapjoin10.q.out ec6069f 
  ql/src/test/results/clientpositive/bucketmapjoin11.q.out 15be0af 
  ql/src/test/results/clientpositive/bucketmapjoin12.q.out b87685c 
  ql/src/test/results/clientpositive/bucketmapjoin13.q.out cdb7637 
  ql/src/test/results/clientpositive/bucketmapjoin2.q.out fb0e601 
  ql/src/test/results/clientpositive/bucketmapjoin3.q.out e71c5e1 
  ql/src/test/results/clientpositive/bucketmapjoin4.q.out 7dd2202 
  ql/src/test/results/clientpositive/bucketmapjoin5.q.out 15969db 
  ql/src/test/results/clientpositive/bucketmapjoin7.q.out e845006 
  ql/src/test/results/clientpositive/bucketmapjoin8.q.out 0ef69ea 
  ql/src/test/results/clientpositive/bucketmapjoin9.q.out 02c5153 
  ql/src/test/results/clientpositive/bucketmapjoin_negative.q.out 124ccb3 
  ql/src/te

[jira] [Created] (HIVE-12793) Address TestSparkCliDriver.testCliDriver_order2 failure due to HIVE-12782

2016-01-06 Thread Pengcheng Xiong (JIRA) 
Pengcheng Xiong created HIVE-12793:
--

 Summary: Address TestSparkCliDriver.testCliDriver_order2 failure 
due to HIVE-12782
 Key: HIVE-12793
 URL: https://issues.apache.org/jira/browse/HIVE-12793
 Project: Hive
  Issue Type: Sub-task
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12794) LLAP cannot run queries against HBase due to missing HBase jars

2016-01-06 Thread Sergey Shelukhin (JIRA) 
Sergey Shelukhin created HIVE-12794:
---

 Summary: LLAP cannot run queries against HBase due to missing 
HBase jars
 Key: HIVE-12794
 URL: https://issues.apache.org/jira/browse/HIVE-12794
 Project: Hive
  Issue Type: Bug
Reporter: Sergey Shelukhin
Assignee: Sergey Shelukhin






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 41955: HIVE-12758 Parallel compilation: Operator::resetId() is not thread-safe

2016-01-06 Thread Sergey Shelukhin 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41955/
---

(Updated Jan. 6, 2016, 10:24 p.m.)


Review request for hive, Gopal V and Vikram Dixit Kumaraswamy.


Repository: hive-git


Description
---

see jira


Diffs (updated)
-

  
hcatalog/core/src/test/java/org/apache/hive/hcatalog/mapreduce/TestHCatMultiOutputFormat.java
 8148faa 
  ql/src/java/org/apache/hadoop/hive/ql/CompilationOpContext.java PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/Context.java affaec8 
  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 29e6315 
  ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractFileMergeOperator.java 
a3ec0e1 
  ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java 
7302688 
  ql/src/java/org/apache/hadoop/hive/ql/exec/AppMasterEventOperator.java 
7114177 
  ql/src/java/org/apache/hadoop/hive/ql/exec/CollectOperator.java e2f4f58 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsTask.java f6fbe74 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnStatsUpdateTask.java dcbbe2e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java b0170f5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/CommonMergeJoinOperator.java 
1cbd13d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ConditionalTask.java 031331e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java ac0ecd9 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DemuxOperator.java 0888c7b 
  ql/src/java/org/apache/hadoop/hive/ql/exec/DummyStoreOperator.java 0c12570 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 1634143 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 32bfcf5 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java 0e7e79d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ForwardOperator.java 7a4c58a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java ec755a8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java 1693ec3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableDummyOperator.java 
1de8c76 
  ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java 76308f6 
  ql/src/java/org/apache/hadoop/hive/ql/exec/JoinOperator.java 3453fc9 
  ql/src/java/org/apache/hadoop/hive/ql/exec/LateralViewForwardOperator.java 
e866eed 
  ql/src/java/org/apache/hadoop/hive/ql/exec/LateralViewJoinOperator.java 
55bb08f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/LimitOperator.java fc85bea 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ListSinkOperator.java 919e72f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java dc0b85e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java 99724c1 
  ql/src/java/org/apache/hadoop/hive/ql/exec/MuxOperator.java 4f4abd3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 85ab6b2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/OperatorFactory.java f619a56 
  ql/src/java/org/apache/hadoop/hive/ql/exec/OrcFileMergeOperator.java 2c9deac 
  ql/src/java/org/apache/hadoop/hive/ql/exec/PTFOperator.java 113ac21 
  ql/src/java/org/apache/hadoop/hive/ql/exec/RCFileMergeOperator.java c34454c 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 4b65952 
  ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 62ae630 
  ql/src/java/org/apache/hadoop/hive/ql/exec/ScriptOperator.java 63837ce 
  ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java b1b8459 
  ql/src/java/org/apache/hadoop/hive/ql/exec/SerializationUtilities.java 
45f50c3 
  ql/src/java/org/apache/hadoop/hive/ql/exec/SparkHashTableSinkOperator.java 
7a8de2d 
  ql/src/java/org/apache/hadoop/hive/ql/exec/StatsNoJobTask.java bb4bde9 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 32806dc 
  ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 0eab63e 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TemporaryHashSinkOperator.java 
9e3a84f 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TerminalOperator.java 069eab2 
  ql/src/java/org/apache/hadoop/hive/ql/exec/TezDummyStoreOperator.java e9f65be 
  ql/src/java/org/apache/hadoop/hive/ql/exec/UDTFOperator.java b3c6d91 
  ql/src/java/org/apache/hadoop/hive/ql/exec/UnionOperator.java ddb23ee 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java ab7fd93 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapper.java 23497a9 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HashTableLoader.java 1070384 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java ee99269 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HashTableLoader.java 7ada611 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkMapRecordHandler.java 
62be3f8 
  ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkTask.java 31eee45 
  ql/src/java/org/apache/hadoop/hive/ql/exec/tez/MapRecordProcessor.java 
ee62ab3 
  
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/Vector

[jira] [Created] (HIVE-12795) Vectorized execution causes ClassCastException

2016-01-06 Thread Yongzhi Chen (JIRA) 
Yongzhi Chen created HIVE-12795:
---

 Summary: Vectorized execution causes ClassCastException
 Key: HIVE-12795
 URL: https://issues.apache.org/jira/browse/HIVE-12795
 Project: Hive
  Issue Type: Bug
  Components: Query Processor
Affects Versions: 1.1.0
Reporter: Yongzhi Chen
Assignee: Yongzhi Chen


In some hive versions, when
set hive.auto.convert.join=false;
set hive.vectorized.execution.enabled = true;

Some join queries fail with ClassCastException:
The stack:
{noformat}
Caused by: java.lang.ClassCastException: 
org.apache.hadoop.hive.serde2.lazy.objectinspector.primitive.LazyStringObjectInspector
 cannot be cast to 
org.apache.hadoop.hive.serde2.objectinspector.primitive.SettableStringObjectInspector
at 
org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory.genVectorExpressionWritable(VectorExpressionWriterFactory.java:419)
at 
org.apache.hadoop.hive.ql.exec.vector.expressions.VectorExpressionWriterFactory.processVectorInspector(VectorExpressionWriterFactory.java:1102)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorReduceSinkOperator.initializeOp(VectorReduceSinkOperator.java:55)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:469)
at org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:425)
at 
org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:193)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
at org.apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java:431)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:126)
... 22 more

{noformat}
It can not be reproduced in hive 2.0 and 1.3 because of different code path. 
Reproduce:
{noformat}

CREATE TABLE test1
 (
   id string)
   PARTITIONED BY (
  cr_year bigint,
  cr_month bigint)
 ROW FORMAT SERDE
  'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe'
STORED AS INPUTFORMAT
  'org.apache.hadoop.hive.ql.io.RCFileInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.RCFileOutputFormat'
TBLPROPERTIES (
  'serialization.null.format'='' );
  
  CREATE TABLE test2(
id string
  )
   PARTITIONED BY (
  cr_year bigint,
  cr_month bigint)
ROW FORMAT SERDE
  'org.apache.hadoop.hive.serde2.columnar.ColumnarSerDe'
STORED AS INPUTFORMAT
  'org.apache.hadoop.hive.ql.io.RCFileInputFormat'
OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.RCFileOutputFormat'
TBLPROPERTIES (
  'serialization.null.format'=''
 );
set hive.auto.convert.join=false;
set hive.vectorized.execution.enabled = true;
 SELECT cr.id1 ,
cr.id2 
FROM
(SELECT t1.id id1,
 t2.id id2
 from
 (select * from test1 ) t1
 left outer join test2  t2
 on t1.id=t2.id) cr;

{noformat}





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12796) Switch to 32-bits containers for HMS upgrade tests

2016-01-06 Thread JIRA 
Sergio Peña created HIVE-12796:
--

 Summary: Switch to 32-bits containers for HMS upgrade tests
 Key: HIVE-12796
 URL: https://issues.apache.org/jira/browse/HIVE-12796
 Project: Hive
  Issue Type: Task
  Components: Testing Infrastructure
Affects Versions: 1.2.1
Reporter: Sergio Peña
Assignee: Sergio Peña


The Hive metastore upgrade tests create LXC containers for each of the 
databases server supported by HMS. These containers are default to Ubuntu 
64-bits. 

The Oracle database libraries are correctly executed on 32-bits only. We should 
switch to 32-bits containers for all the database servers to allow tests being 
executed for Oracle as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-12797) Synchronization issues with tez/llap session pool in hs2

2016-01-06 Thread Vikram Dixit K (JIRA) 
Vikram Dixit K created HIVE-12797:
-

 Summary: Synchronization issues with tez/llap session pool in hs2
 Key: HIVE-12797
 URL: https://issues.apache.org/jira/browse/HIVE-12797
 Project: Hive
  Issue Type: Bug
  Components: HiveServer2
Affects Versions: 2.0.0
Reporter: Vikram Dixit K
Assignee: Vikram Dixit K


The changes introduced as part of HIVE-12674 causes issues while shutting down 
hs2 when session pools are used.

{code}
java.util.ConcurrentModificationException
at 
java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) 
~[?:1.8.0_45]
at java.util.LinkedList$ListItr.remove(LinkedList.java:921) 
~[?:1.8.0_45]
at 
org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.stop(TezSessionPoolManager.java:288)
 ~[hive-exec-2.0.0.2.3.5.0-79.jar:2.0.0.2.3.5.0-79]
at 
org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:479) 
[hive-jdbc-2.0.0.2.3.5.0-79-standalone.jar:2.0.0.2.3.5.0-79]
at 
org.apache.hive.service.server.HiveServer2$2.run(HiveServer2.java:183) 
[hive-jdbc-2.0.0.2.3.5.0-79-standalone.jar:2.0.0.2.3.5.0-79]
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 42000: HIVE-12645 : ConstantPropagateProcCtx.resolve() should verify internal names in addition to alias to match 2 columns from different row schemas

2016-01-06 Thread Hari Sankar Sivarama Subramaniyan 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42000/
---

Review request for hive, Ashutosh Chauhan and John Pullokkaran.


Repository: hive-git


Description
---

ConstantPropagateProcCtx.resolve() should verify internal names in addition to 
alias to match 2 columns from different row schemas


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConstantPropagateProcCtx.java 
0a61f12 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/correlation/ReduceSinkDeDuplication.java
 8da9ad5 
  ql/src/test/results/clientpositive/bucket_groupby.q.out 0e30801 
  ql/src/test/results/clientpositive/dynamic_rdd_cache.q.out 66d9a78 
  ql/src/test/results/clientpositive/dynpart_sort_optimization2.q.out 141bcd8 
  ql/src/test/results/clientpositive/groupby_ppd.q.out e3e4a50 
  ql/src/test/results/clientpositive/groupby_sort_1_23.q.out ceecbb9 
  ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out 009ab2e 
  ql/src/test/results/clientpositive/list_bucket_query_oneskew_2.q.out 48cc6ea 
  ql/src/test/results/clientpositive/partition_multilevels.q.out 0fe7f82 
  ql/src/test/results/clientpositive/perf/query31.q.out 8c370c4 
  ql/src/test/results/clientpositive/perf/query39.q.out f4ad98c 
  ql/src/test/results/clientpositive/perf/query42.q.out 94f7e32 
  ql/src/test/results/clientpositive/perf/query52.q.out b4f46cc 
  ql/src/test/results/clientpositive/perf/query64.q.out 68730e7 
  ql/src/test/results/clientpositive/perf/query66.q.out b2e6bf7 
  ql/src/test/results/clientpositive/quotedid_basic.q.out 29736af 
  ql/src/test/results/clientpositive/quotedid_partition.q.out e40d0d0 
  ql/src/test/results/clientpositive/spark/dynamic_rdd_cache.q.out 1f26724 
  ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out fb75a98 
  ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out b9fb6b1 
  ql/src/test/results/clientpositive/tez/dynpart_sort_optimization2.q.out 
8ef2b06 
  ql/src/test/results/clientpositive/tez/vector_null_projection.q.out 88587e9 
  ql/src/test/results/clientpositive/vector_null_projection.q.out c787fd4 

Diff: https://reviews.apache.org/r/42000/diff/


Testing
---


Thanks,

Hari Sankar Sivarama Subramaniyan

[jira] [Created] (HIVE-12798) MiniTezCliDriver failures

2016-01-06 Thread Hari Sankar Sivarama Subramaniyan (JIRA) 
Hari Sankar Sivarama Subramaniyan created HIVE-12798:


 Summary: MiniTezCliDriver failures
 Key: HIVE-12798
 URL: https://issues.apache.org/jira/browse/HIVE-12798
 Project: Hive
  Issue Type: Sub-task
Reporter: Hari Sankar Sivarama Subramaniyan
Assignee: Hari Sankar Sivarama Subramaniyan


As of 01/04/2016, the following tests fail in the MiniTezCliDriver mode when 
the cbo return path is enabled. We need to fix them :
{code}
 vector_leftsemi_mapjoin
 vector_join_filters
 vector_interval_mapjoin
 vector_left_outer_join
 vectorized_mapjoin
 vector_inner_join
 vectorized_context
 tez_vector_dynpart_hashjoin_1
 count
 auto_sortmerge_join_6
 skewjoin
 vector_auto_smb_mapjoin_14
 auto_join_filters
 vector_outer_join0
 vector_outer_join1
 vector_outer_join2
 vector_outer_join3
 vector_outer_join4
 vector_outer_join5
 hybridgrace_hashjoin_1
 vector_mapjoin_reduce
 vectorized_nested_mapjoin
 vector_left_outer_join2
 vector_char_mapjoin1
 vector_decimal_mapjoin
 vectorized_dynamic_partition_pruning
 vector_varchar_mapjoin1
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 41928: HIVE-12777: Add capability to restore session

2016-01-06 Thread Amareshwari Sriramadasu 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/41928/#review113180
---



service/src/java/org/apache/hive/service/cli/session/SessionManager.java (line 
297)


Dont we need changes here?



service/src/test/org/apache/hive/service/cli/CLIServiceTest.java (line 91)


Restoring a closed session? I dont think this should be allowed.

The test should wrt service restart and restore session.


- Amareshwari Sriramadasu


On Jan. 6, 2016, 6:59 a.m., Rajat Khandelwal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/41928/
> ---
> 
> (Updated Jan. 6, 2016, 6:59 a.m.)
> 
> 
> Review request for hive.
> 
> 
> Bugs: HIVE-12777
> https://issues.apache.org/jira/browse/HIVE-12777
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Extensions using Hive session handles should be able to restore the hive 
> session from the handle. 
> 
> Apache Lens depends on a fork of hive and that fork has such a capability. 
> 
> Relevant commit: 
> https://github.com/InMobi/hive/commit/931fe9116161a18952c082c14223ad6745fefe00#diff-0acb35f7cab7492f522b0c40ce3ce1be
> 
> 
> Diffs
> -
> 
>   service/if/TCLIService.thrift baf583fb3ecd4a0ab248540ffb5db57100028968 
>   service/src/java/org/apache/hive/service/cli/CLIService.java 
> adc98098437b2d0429dd2af40d8eaae323fbf3f0 
>   service/src/java/org/apache/hive/service/cli/CLIServiceClient.java 
> 3155c238ff688bfea16b0aaeea950599bb659b5b 
>   service/src/java/org/apache/hive/service/cli/EmbeddedCLIServiceClient.java 
> 9cad5be198c063115a8e90c67b1c2fd910ca8bc6 
>   service/src/java/org/apache/hive/service/cli/ICLIService.java 
> c9cc1f4da56f1cd10f6348ea2b9e17e203b87664 
>   service/src/java/org/apache/hive/service/cli/SessionHandle.java 
> 52e0ad4834d8b44d5038bb5c1a851eb702cd38ca 
>   service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
> 50c912ee606bce73f447fba69ce0f57d8c7fc6c9 
>   service/src/java/org/apache/hive/service/cli/session/SessionManager.java 
> 637cdca3bcea9d320f397de032fd78d59ce6afef 
>   
> service/src/java/org/apache/hive/service/cli/thrift/RetryingThriftCLIServiceClient.java
>  529eaa45d29d22e0e80ea2c08a27b84f2909eda3 
>   service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIService.java 
> 8434965e87366e86d27685c5ca70ee904446bae9 
>   
> service/src/java/org/apache/hive/service/cli/thrift/ThriftCLIServiceClient.java
>  1af45398b895cd7616c5627d318422e14b81e734 
>   service/src/test/org/apache/hive/service/cli/CLIServiceTest.java 
> e78181a15993d99f1cab5a061c08bb21823d2171 
> 
> Diff: https://reviews.apache.org/r/41928/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Rajat Khandelwal
> 
>