date:20160727

Re: Review Request 50510: HIVE-14360 Starting BeeLine after using !save, there is an error logged: "Error setting configuration: conf"

2016-07-27 Thread Peter Vary



> On July 28, 2016, 4:37 a.m., Vihang Karajgaonkar wrote:
> > beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java, line 627
> > 
> >
> > Adding @Ignore may not be the right solution here. This will skip 
> > saving the conf variable which can potentially lead to loss of user's 
> > custom settings for HiveConf.
> > 
> > The issue is caused because the method name is setHiveConf while the 
> > Reflector is looking for setConf. Did you try renaming the method from 
> > setHiveConf to setConf?

You are correct, but this variable is only used in BeeLine compatibility mode 
(in BeeLine mode it is not possible to initialize the HiveConf object), and 
then it contains every hiveconf variable set on the server, above the ones set 
by the user. I do not think it is a good idea to store this in a configuration 
file, and will not work, since when initialized, these are overwritten by the 
server when the connection is estabilished (beeLine.updateOptsForCli()).

What do you think?


- Peter


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50510/#review143886
---


On July 27, 2016, 4:26 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50510/
> ---
> 
> (Updated July 27, 2016, 4:26 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Sergio Pena, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-14360
> https://issues.apache.org/jira/browse/HIVE-14360
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Adding the @Ignore annotation to the getConf method, so the saving should 
> ignore the attribute
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java e6e24b1 
>   beeline/src/test/org/apache/hive/beeline/TestBeeLineOpts.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/50510/diff/
> 
> 
> Testing
> ---
> 
> Manually + unit test
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

[jira] [Created] (HIVE-14368) ThriftCLIService.GetOperationStatus should include exception's stack trace to the error message.

2016-07-27 Thread zhihai xu (JIRA)

zhihai xu created HIVE-14368:


 Summary: ThriftCLIService.GetOperationStatus should include 
exception's stack trace to the error message.
 Key: HIVE-14368
 URL: https://issues.apache.org/jira/browse/HIVE-14368
 Project: Hive
  Issue Type: Improvement
  Components: Thrift API
Reporter: zhihai xu
Assignee: zhihai xu
Priority: Minor


ThriftCLIService.GetOperationStatus should include exception's stack trace to 
the error message. The stack trace will be really helpful for client to debug 
failed queries.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 50510: HIVE-14360 Starting BeeLine after using !save, there is an error logged: "Error setting configuration: conf"

2016-07-27 Thread Vihang Karajgaonkar


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50510/#review143886
---




beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java (line 627)


Adding @Ignore may not be the right solution here. This will skip saving 
the conf variable which can potentially lead to loss of user's custom settings 
for HiveConf.

The issue is caused because the method name is setHiveConf while the 
Reflector is looking for setConf. Did you try renaming the method from 
setHiveConf to setConf?


- Vihang Karajgaonkar


On July 27, 2016, 4:26 p.m., Peter Vary wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50510/
> ---
> 
> (Updated July 27, 2016, 4:26 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Sergio Pena, and Vihang Karajgaonkar.
> 
> 
> Bugs: HIVE-14360
> https://issues.apache.org/jira/browse/HIVE-14360
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Adding the @Ignore annotation to the getConf method, so the saving should 
> ignore the attribute
> 
> 
> Diffs
> -
> 
>   beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java e6e24b1 
>   beeline/src/test/org/apache/hive/beeline/TestBeeLineOpts.java PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/50510/diff/
> 
> 
> Testing
> ---
> 
> Manually + unit test
> 
> 
> Thanks,
> 
> Peter Vary
> 
>

Re: Review Request 50548: HIVE-14355 - Schema evolution for ORC in llap is broken for int to string conversion

2016-07-27 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50548/
---

(Updated July 28, 2016, 3:38 a.m.)


Review request for hive and Sergey Shelukhin.


Changes
---

Fixed wrong jira description


Summary (updated)
-

HIVE-14355 - Schema evolution for ORC in llap is broken for int to string 
conversion


Bugs: HIVE-14355
https://issues.apache.org/jira/browse/HIVE-14355


Repository: hive-git


Description
---

see jira


Diffs
-

  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
 9d16889 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ColumnVectorProducer.java
 abd4533 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java
 7db519c 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
 3dfab63 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ReadPipeline.java 
21b1772 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 1dcd2cd 

Diff: https://reviews.apache.org/r/50548/diff/


Testing
---


Thanks,

Prasanth_J

Review Request 50548: HIVE-14333 - ORC schema evolution from float to double changes precision and breaks filters

2016-07-27 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50548/
---

Review request for hive and Sergey Shelukhin.


Repository: hive-git


Description
---

see jira


Diffs
-

  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
 9d16889 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ColumnVectorProducer.java
 abd4533 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java
 7db519c 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
 3dfab63 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ReadPipeline.java 
21b1772 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 1dcd2cd 

Diff: https://reviews.apache.org/r/50548/diff/


Testing
---


Thanks,

Prasanth_J

Re: Review Request 50548: HIVE-14333 - ORC schema evolution from float to double changes precision and breaks filters

2016-07-27 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50548/
---

(Updated July 28, 2016, 3:37 a.m.)


Review request for hive and Sergey Shelukhin.


Bugs: HIVE-14333
https://issues.apache.org/jira/browse/HIVE-14333


Repository: hive-git


Description
---

see jira


Diffs
-

  
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
 9d16889 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ColumnVectorProducer.java
 abd4533 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcColumnVectorProducer.java
 7db519c 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/OrcEncodedDataConsumer.java
 3dfab63 
  llap-server/src/java/org/apache/hadoop/hive/llap/io/decode/ReadPipeline.java 
21b1772 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 1dcd2cd 

Diff: https://reviews.apache.org/r/50548/diff/


Testing
---


Thanks,

Prasanth_J

[jira] [Created] (HIVE-14367) Estimated size for constant nulls is 0

2016-07-27 Thread Ashutosh Chauhan (JIRA)

Ashutosh Chauhan created HIVE-14367:
---

 Summary: Estimated size for constant nulls is 0
 Key: HIVE-14367
 URL: https://issues.apache.org/jira/browse/HIVE-14367
 Project: Hive
  Issue Type: Bug
  Components: Physical Optimizer
Affects Versions: 2.1.0, 2.0.0
Reporter: Ashutosh Chauhan
Assignee: Ashutosh Chauhan


since type is incorrectly assumed as void.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14366) Conversion of a Non-ACID table to an ACID table produces non-unique primary keys

2016-07-27 Thread Saket Saurabh (JIRA)

Saket Saurabh created HIVE-14366:


 Summary: Conversion of a Non-ACID table to an ACID table produces 
non-unique primary keys
 Key: HIVE-14366
 URL: https://issues.apache.org/jira/browse/HIVE-14366
 Project: Hive
  Issue Type: Bug
  Components: Transactions
Reporter: Saket Saurabh


When a Non-ACID table is converted to an ACID table, the primary key consisting 
of (original transaction id, bucket_id, row_id) is not generated uniquely. 
Currently, the row_id is always set to 0 for most rows. This leads to 
correctness issue for such tables.

Quickest way to reproduce is to add the following unit test to 
ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java

{code:title=ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java|borderStyle=solid}
  @Test
  public void testOriginalReader() throws Exception {
FileSystem fs = FileSystem.get(hiveConf);
FileStatus[] status;

// 1. Insert five rows to Non-ACID table.
runStatementOnDriver("insert into " + Table.NONACIDORCTBL + "(a,b) 
values(1,2),(3,4),(5,6),(7,8),(9,10)");

// 2. Convert NONACIDORCTBL to ACID table.
runStatementOnDriver("alter table " + Table.NONACIDORCTBL + " SET 
TBLPROPERTIES ('transactional'='true')");

// 3. Perform a major compaction.
runStatementOnDriver("alter table "+ Table.NONACIDORCTBL + " compact 
'MAJOR'");
runWorker(hiveConf);

// 3. Perform a delete.
runStatementOnDriver("delete from " + Table.NONACIDORCTBL + " where a = 1");

// Now do a projection should have (3,4) (5,6),(7,8),(9,10) only since 
(1,2) has been deleted.
List rs = runStatementOnDriver("select a,b from " + 
Table.NONACIDORCTBL + " order by a,b");
int[][] resultData = new int[][] {{3,4}, {5,6}, {7,8}, {9,10}};
Assert.assertEquals(stringifyValues(resultData), rs);
  }
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 49766: HIVE-14035 Enable predicate pushdown to delta files created by ACID Transactions

2016-07-27 Thread Lefty Leverenz



> On July 21, 2016, 7:42 a.m., Lefty Leverenz wrote:
> >
> 
> Saket Saurabh wrote:
> @Lefty, does the updated description for this config variable seem to 
> explain better now?

Great, just what it needed!  Thanks Saket.

I recommend adding "the" or "a" in a few places and a final period, but these 
edits aren't really necessary so don't make a new patch just for them:

"versions of Hive ACID subsystem" --> "versions of the Hive ACID subsystem" 
(line 1768 of revision 4)

"enable split-update feature found in newer version" --> "enable a split-update 
feature found in the newer version" (line 1769)

"of Hive ACID subsystem. (See HIVE-14035 for details)" --> "of the Hive ACID 
subsystem. (See HIVE-14035 for details.)" (line 1770)


- Lefty


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49766/#review143059
---


On July 27, 2016, 9:54 p.m., Saket Saurabh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49766/
> ---
> 
> (Updated July 27, 2016, 9:54 p.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-14035
> 
> In current Hive version, delta files created by ACID transactions do not 
> allow predicate pushdown if they contain any update/delete events. This is 
> done to preserve correctness when following a multi-version approach during 
> event collapsing, where an update event overwrites an existing insert event. 
> This JIRA proposes to split an update event into a combination of a delete 
> event followed by a new insert event, that can enable predicate push down to 
> all delta files without breaking correctness. To support backward 
> compatibility for this feature, this JIRA also proposes to add some sort of 
> versioning to ACID that can allow different versions of ACID transactions to 
> co-exist together.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e92466f 
>   
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java
>  14f7316 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/AbstractRecordWriter.java
>  974c6b8 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  ca2a912 
>   metastore/if/hive_metastore.thrift 4d92b73 
>   metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.h ae14bd1 
>   metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.cpp f982bf2 
>   
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/hive_metastoreConstants.java
>  5a666f2 
>   metastore/src/gen/thrift/gen-php/metastore/Types.php f505208 
>   metastore/src/gen/thrift/gen-py/hive_metastore/constants.py d1c07a5 
>   metastore/src/gen/thrift/gen-rb/hive_metastore_constants.rb eeccc84 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/TransactionalValidationListener.java
>  3e74675 
>   orc/src/java/org/apache/orc/impl/TreeReaderFactory.java c4a2093 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java db6848a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 57b6c67 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 23a13d6 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java dd90a95 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java c150ec5 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 945b828 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 63d02fb 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 1a1af28 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 9d927bd 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 8cf261d 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 6caca98 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java d48e441 
>   ql/src/test/org/apache/hadoop/hive/ql/io/TestAcidUtils.java b83cea4 
> 
> Diff: https://reviews.apache.org/r/49766/diff/
> 
> 
> Testing
> ---
> 
> Tests for the feature are in 
> ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java. These are mostly 
> integration tests that test end-to-end insert/update/delete scenarios 
> followed by compaction and cleaning.
> 
> 
> Thanks,
> 
> Saket Saurabh
> 
>

[jira] [Created] (HIVE-14365) Simplify logic for check introduced in HIVE-10022

2016-07-27 Thread Sushanth Sowmyan (JIRA)

Sushanth Sowmyan created HIVE-14365:
---

 Summary: Simplify logic for check introduced in HIVE-10022
 Key: HIVE-14365
 URL: https://issues.apache.org/jira/browse/HIVE-14365
 Project: Hive
  Issue Type: Bug
Reporter: Sushanth Sowmyan


We introduced a parent-check/glob-check/file-check in SQLAuthorizationUtils in 
HIVE-10022, but the logic for that is more convoluted than it needs to be. 
Taking a cue off RANGER-1126 , we should simplify this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 50359: HIVE-14270: Write temporary data to HDFS when doing inserts on tables located on S3

2016-07-27 Thread Sergio Pena


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50359/
---

(Updated July 27, 2016, 10:56 p.m.)


Review request for hive.


Bugs: HIVE-14270
https://issues.apache.org/jira/browse/HIVE-14270


Repository: hive-git


Description
---

This patch will create a temporary directory for Hive intermediate data on HDFS 
when S3 tables are used.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/common/BlobStorageUtils.java 
PRE-CREATION 
  common/src/test/org/apache/hadoop/hive/common/TestBlobStorageUtils.java 
PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/Context.java 
ec5d693d28a40925c44f844a05ebf3f5c10173c9 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
9d927bd1a519f79bc7fa88c3b7e5c6cc2ef0637f 
  ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 
2671cb1cf2ef74f9d6628f8cdf3f5ac99283dbd8 
  ql/src/test/org/apache/hadoop/hive/ql/exec/TestContext.java PRE-CREATION 

Diff: https://reviews.apache.org/r/50359/diff/


Testing
---

NO PATCH
** NON-PARTITIONED TABLE

- create table dummy (id int);  
 3.651s
- insert into table s3dummy values (1); 
39.231s
- insert overwrite table s3dummy values (1);
42.569s
- insert overwrite directory 's3a://spena-bucket/dirs/s3dummy' select * from 
dummy; 30.136s

EXTERNAL TABLE

- create table s3dummy_ext like s3dummy location 
's3a://spena-bucket/user/hive/warehouse/s3dummy';   9.297s
- insert into table s3dummy_ext values (1); 
45.855s

WITH PATCH

** NON-PARTITIONED TABLE
- create table s3dummy (id int) location 
's3a://spena-bucket/user/hive/warehouse/s3dummy';   3.945s
- insert into table s3dummy values (1); 
15.025s
- insert overwrite table s3dummy values (1);
25.149s 
- insert overwrite directory 's3a://spena-bucket/dirs/s3dummy' select * from 
dummy; 19.158s  
- from dummy insert overwrite table s3dummy select *;   
25.469s  
- from dummy insert into table s3dummy select *;
14.501s

** EXTERNAL TABLE
- create table s3dummy_ext like s3dummy location 
's3a://spena-bucket/user/hive/warehouse/s3dummy';   4.827s
- insert into table s3dummy_ext values (1); 
16.070s

** PARTITIONED TABLE
- create table s3dummypart (id int) partitioned by (part int)
  location 's3a://spena-bucket/user/hive/warehouse/s3dummypart';
 3.176s
- alter table s3dummypart add partition (part=1);   
 3.229s
- alter table s3dummypart add partition (part=2);   
 3.124s
- insert into table s3dummypart partition (part=1) values (1);  
14.876s
- insert overwrite table s3dummypart partition (part=1) values (1); 
27.594s 
- insert overwrite directory 's3a://spena-bucket/dirs/s3dummypart' select * 
from dummypart; 22.298s  
- from dummypart insert overwrite table s3dummypart partition (part=1) select 
id;   29.001s  
- from dummypart insert into table s3dummypart partition (part=1) select id;
14.869s

** DYNAMIC PARTITIONS
- insert into table s3dummypart partition (part) select id, 1 from dummypart;   
15.185s
- insert into table s3dummypart partition (part) select id, 1 from dummypart;   
18.820s


Thanks,

Sergio Pena

Re: Review Request 49766: HIVE-14035 Enable predicate pushdown to delta files created by ACID Transactions

2016-07-27 Thread Saket Saurabh



> On July 21, 2016, 12:42 a.m., Lefty Leverenz wrote:
> >

@Lefty, does the updated description for this config variable seem to explain 
better now?


- Saket


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49766/#review143059
---


On July 27, 2016, 2:54 p.m., Saket Saurabh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/49766/
> ---
> 
> (Updated July 27, 2016, 2:54 p.m.)
> 
> 
> Review request for hive and Eugene Koifman.
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> https://issues.apache.org/jira/browse/HIVE-14035
> 
> In current Hive version, delta files created by ACID transactions do not 
> allow predicate pushdown if they contain any update/delete events. This is 
> done to preserve correctness when following a multi-version approach during 
> event collapsing, where an update event overwrites an existing insert event. 
> This JIRA proposes to split an update event into a combination of a delete 
> event followed by a new insert event, that can enable predicate push down to 
> all delta files without breaking correctness. To support backward 
> compatibility for this feature, this JIRA also proposes to add some sort of 
> versioning to ACID that can allow different versions of ACID transactions to 
> co-exist together.
> 
> 
> Diffs
> -
> 
>   common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e92466f 
>   
> hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java
>  14f7316 
>   
> hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/AbstractRecordWriter.java
>  974c6b8 
>   
> itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
>  ca2a912 
>   metastore/if/hive_metastore.thrift 4d92b73 
>   metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.h ae14bd1 
>   metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.cpp f982bf2 
>   
> metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/hive_metastoreConstants.java
>  5a666f2 
>   metastore/src/gen/thrift/gen-php/metastore/Types.php f505208 
>   metastore/src/gen/thrift/gen-py/hive_metastore/constants.py d1c07a5 
>   metastore/src/gen/thrift/gen-rb/hive_metastore_constants.rb eeccc84 
>   
> metastore/src/java/org/apache/hadoop/hive/metastore/TransactionalValidationListener.java
>  3e74675 
>   orc/src/java/org/apache/orc/impl/TreeReaderFactory.java c4a2093 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java db6848a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 57b6c67 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 23a13d6 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java dd90a95 
>   ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java c150ec5 
>   ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 945b828 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 63d02fb 
>   ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 1a1af28 
>   ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 9d927bd 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 8cf261d 
>   ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 
> 6caca98 
>   ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java d48e441 
>   ql/src/test/org/apache/hadoop/hive/ql/io/TestAcidUtils.java b83cea4 
> 
> Diff: https://reviews.apache.org/r/49766/diff/
> 
> 
> Testing
> ---
> 
> Tests for the feature are in 
> ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java. These are mostly 
> integration tests that test end-to-end insert/update/delete scenarios 
> followed by compaction and cleaning.
> 
> 
> Thanks,
> 
> Saket Saurabh
> 
>

[jira] [Created] (HIVE-14364) Update timeouts for llap comparator tests

2016-07-27 Thread Siddharth Seth (JIRA)

Siddharth Seth created HIVE-14364:
-

 Summary: Update timeouts for llap comparator tests
 Key: HIVE-14364
 URL: https://issues.apache.org/jira/browse/HIVE-14364
 Project: Hive
  Issue Type: Task
Reporter: Siddharth Seth
Assignee: Siddharth Seth


The tests timeout occasionally. Increasing to 60 seconds from 5 seconds. 

NO_PRECOMMIT_TESTS



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 49766: HIVE-14035 Enable predicate pushdown to delta files created by ACID Transactions

2016-07-27 Thread Saket Saurabh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/49766/
---

(Updated July 27, 2016, 2:54 p.m.)


Review request for hive and Eugene Koifman.


Changes
---

Refactor the way delete event writers are created for compaction case in favor 
of a better abstraction.


Repository: hive-git


Description
---

https://issues.apache.org/jira/browse/HIVE-14035

In current Hive version, delta files created by ACID transactions do not allow 
predicate pushdown if they contain any update/delete events. This is done to 
preserve correctness when following a multi-version approach during event 
collapsing, where an update event overwrites an existing insert event. 
This JIRA proposes to split an update event into a combination of a delete 
event followed by a new insert event, that can enable predicate push down to 
all delta files without breaking correctness. To support backward compatibility 
for this feature, this JIRA also proposes to add some sort of versioning to 
ACID that can allow different versions of ACID transactions to co-exist 
together.


Diffs (updated)
-

  common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e92466f 
  
hcatalog/core/src/main/java/org/apache/hive/hcatalog/mapreduce/FosterStorageHandler.java
 14f7316 
  
hcatalog/streaming/src/java/org/apache/hive/hcatalog/streaming/AbstractRecordWriter.java
 974c6b8 
  
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/txn/compactor/TestCompactor.java
 ca2a912 
  metastore/if/hive_metastore.thrift 4d92b73 
  metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.h ae14bd1 
  metastore/src/gen/thrift/gen-cpp/hive_metastore_constants.cpp f982bf2 
  
metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/hive_metastoreConstants.java
 5a666f2 
  metastore/src/gen/thrift/gen-php/metastore/Types.php f505208 
  metastore/src/gen/thrift/gen-py/hive_metastore/constants.py d1c07a5 
  metastore/src/gen/thrift/gen-rb/hive_metastore_constants.rb eeccc84 
  
metastore/src/java/org/apache/hadoop/hive/metastore/TransactionalValidationListener.java
 3e74675 
  orc/src/java/org/apache/orc/impl/TreeReaderFactory.java c4a2093 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java db6848a 
  ql/src/java/org/apache/hadoop/hive/ql/exec/SMBMapJoinOperator.java 57b6c67 
  ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 23a13d6 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidOutputFormat.java dd90a95 
  ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java c150ec5 
  ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java 945b828 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 63d02fb 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcRecordUpdater.java 1a1af28 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 9d927bd 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 8cf261d 
  ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java 6caca98 
  ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java d48e441 
  ql/src/test/org/apache/hadoop/hive/ql/io/TestAcidUtils.java b83cea4 

Diff: https://reviews.apache.org/r/49766/diff/


Testing
---

Tests for the feature are in 
ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java. These are mostly 
integration tests that test end-to-end insert/update/delete scenarios followed 
by compaction and cleaning.


Thanks,

Saket Saurabh

[jira] [Created] (HIVE-14363) bucketmap inner join query fails due to NullPointerException in some cases

2016-07-27 Thread Jagruti Varia (JIRA)

Jagruti Varia created HIVE-14363:


 Summary: bucketmap inner join query fails due to 
NullPointerException in some cases
 Key: HIVE-14363
 URL: https://issues.apache.org/jira/browse/HIVE-14363
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Jagruti Varia
Assignee: Hari Sankar Sivarama Subramaniyan


Bucketmap inner join query between bucketed tables throws following exception 
when one table contains all the empty buckets while other has all the non-empty 
buckets.
{noformat}
Vertex failed, vertexName=Map 2, vertexId=vertex_1466710232033_0432_4_01, 
diagnostics=[Task failed, taskId=task_1466710232033_0432_4_01_00, 
diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task ( 
failure ) : 
attempt_1466710232033_0432_4_01_00_0:java.lang.RuntimeException: 
java.lang.RuntimeException: Map operator initialization failed
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Map operator initialization failed
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:330)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:184)
... 14 more
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getKeyValueReader(MapRecordProcessor.java:372)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.initializeMapRecordSources(MapRecordProcessor.java:344)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:292)
... 15 more
], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) : 
attempt_1466710232033_0432_4_01_00_1:java.lang.RuntimeException: 
java.lang.RuntimeException: Map operator initialization failed
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Map operator initialization failed
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:330)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:184)
... 14 more
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getKeyValueReader(MapRecordProcessor.java:372)
at 
org.a

Review Request 50536: HIVE-14333 - ORC schema evolution from float to double changes precision and breaks filters

2016-07-27 Thread j . prasanth . j


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50536/
---

Review request for hive, Matt McCline and Owen O'Malley.


Repository: hive-git


Description
---

HIVE-14310 - ORC schema evolution should not completely disable PPD


Diffs
-

  itests/src/test/resources/testconfiguration.properties f356663 
  
llap-server/src/java/org/apache/hadoop/hive/llap/io/encoded/OrcEncodedDataReader.java
 1dcd2cd 
  orc/src/java/org/apache/orc/impl/ConvertTreeReaderFactory.java c0c6964 
  orc/src/java/org/apache/orc/impl/RecordReaderImpl.java eb43ed6 
  orc/src/java/org/apache/orc/impl/SchemaEvolution.java ce3af7a 
  orc/src/test/org/apache/orc/impl/TestSchemaEvolution.java 3cd0390 
  ql/src/java/org/apache/hadoop/hive/ql/io/orc/OrcInputFormat.java 63d02fb 
  ql/src/test/queries/clientpositive/orc_ppd_schema_evol_3a.q PRE-CREATION 
  ql/src/test/results/clientpositive/orc_ppd_schema_evol_3a.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/tez/orc_ppd_schema_evol_3a.q.out 
PRE-CREATION 

Diff: https://reviews.apache.org/r/50536/diff/


Testing
---


Thanks,

Prasanth_J

[jira] [Created] (HIVE-14362) Support explain analyze in Hive

2016-07-27 Thread Pengcheng Xiong (JIRA)

Pengcheng Xiong created HIVE-14362:
--

 Summary: Support explain analyze in Hive
 Key: HIVE-14362
 URL: https://issues.apache.org/jira/browse/HIVE-14362
 Project: Hive
  Issue Type: New Feature
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong


Right now all the explain levels only support stats before query runs. We would 
like to have an explain analyze similar to Postgres for real stats after query 
runs. This will help to identify the major gap between estimated/real stats and 
make not only query optimization better but also query performance debugging 
easier.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14361) Empty method in TestClientCommandHookFactory

2016-07-27 Thread Peter Vary (JIRA)

Peter Vary created HIVE-14361:
-

 Summary: Empty method in TestClientCommandHookFactory
 Key: HIVE-14361
 URL: https://issues.apache.org/jira/browse/HIVE-14361
 Project: Hive
  Issue Type: Bug
Reporter: Peter Vary
Assignee: Peter Vary
Priority: Trivial


Remove the empty method left in TestClientCommandHookFactory



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 50525: HIVE-14341: Altered skewed location is not respected for list bucketing

2016-07-27 Thread Aihua Xu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50525/
---

Review request for hive.


Repository: hive-git


Description
---

HIVE-14341: Altered skewed location is not respected for list bucketing


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 
b0c3d3f16232cb8579e11a64e095fd45993cf044 
  ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 
9d927bd1a519f79bc7fa88c3b7e5c6cc2ef0637f 
  ql/src/test/queries/clientpositive/create_alter_list_bucketing_table1.q 
bf89e8feb7ccfcf4f6d0924dad4e3d29c50cfab4 
  ql/src/test/results/clientpositive/create_alter_list_bucketing_table1.q.out 
df4a75b95e193c3ab07c095251901d85e0a02cb3 

Diff: https://reviews.apache.org/r/50525/diff/


Testing
---


Thanks,

Aihua Xu

Re: Review Request 47040: Monitor changes to FairScheduler.xml file and automatically update / validate jobs submitted to fair-scheduler

2016-07-27 Thread Reuben Kuhnert


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/47040/
---

(Updated 七月 27, 2016, 5:35 p.m.)


Review request for hive, Lenni Kuff, Mohit Sabharwal, and Sergio Pena.


Changes
---

Diff Rebase


Bugs: HIVE-13696
https://issues.apache.org/jira/browse/HIVE-13696


Repository: hive-git


Description
---

Ensure that jobs sent to YARN with impersonation off are correctly routed to 
the proper queue based on fair-scheduler.xml. Monitor this file for changes and 
validate that jobs can only be sent to queues authorized for the user.


Diffs (updated)
-

  ql/src/java/org/apache/hadoop/hive/ql/Driver.java 14f221a 
  ql/src/java/org/apache/hadoop/hive/ql/session/YarnFairScheduling.java 
PRE-CREATION 
  service/src/java/org/apache/hive/service/cli/session/HiveSessionImpl.java 
72ad86c 
  shims/common/src/main/java/org/apache/hadoop/hive/shims/SchedulerShim.java 
63803b8 
  shims/scheduler/pom.xml 9141c1e 
  
shims/scheduler/src/main/java/org/apache/hadoop/hive/schshim/FairSchedulerQueueAllocator.java
 PRE-CREATION 
  
shims/scheduler/src/main/java/org/apache/hadoop/hive/schshim/FairSchedulerShim.java
 372244d 
  
shims/scheduler/src/main/java/org/apache/hadoop/hive/schshim/QueueAllocator.java
 PRE-CREATION 
  
shims/scheduler/src/test/java/org/apache/hadoop/hive/schshim/TestFairSchedulerQueueAllocator.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/47040/diff/


Testing
---


Thanks,

Reuben Kuhnert

Re: Review Request 50502: HIVE-14294: HiveSchemaConverter for Parquet doesn't translate TINYINT and SMALLINT into proper Parquet types

2016-07-27 Thread Cheng Lian


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50502/#review143754
---


Ship it!




Ship It!

- Cheng Lian


On July 27, 2016, 1:05 p.m., Gabor Szadovszky wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50502/
> ---
> 
> (Updated July 27, 2016, 1:05 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Cheng Lian, and Sergio Pena.
> 
> 
> Bugs: HIVE-14294
> https://issues.apache.org/jira/browse/HIVE-14294
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-14294: HiveSchemaConverter for Parquet doesn't translate TINYINT and 
> SMALLINT into proper Parquet types
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/HiveSchemaConverter.java
>  40f6256cc3d9e11e0f9760bfeab364316ebc638c 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestHiveSchemaConverter.java 
> 589b5b59a4757986c22355fd1557265efbd0c537 
> 
> Diff: https://reviews.apache.org/r/50502/diff/
> 
> 
> Testing
> ---
> 
> Unit tests updated; see patch for details. + manual tests
> 
> 
> Thanks,
> 
> Gabor Szadovszky
> 
>

Review Request 50510: HIVE-14360 Starting BeeLine after using !save, there is an error logged: "Error setting configuration: conf"

2016-07-27 Thread Peter Vary


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50510/
---

Review request for hive, Aihua Xu, Sergio Pena, and Vihang Karajgaonkar.


Bugs: HIVE-14360
https://issues.apache.org/jira/browse/HIVE-14360


Repository: hive-git


Description
---

Adding the @Ignore annotation to the getConf method, so the saving should 
ignore the attribute


Diffs
-

  beeline/src/java/org/apache/hive/beeline/BeeLineOpts.java e6e24b1 
  beeline/src/test/org/apache/hive/beeline/TestBeeLineOpts.java PRE-CREATION 

Diff: https://reviews.apache.org/r/50510/diff/


Testing
---

Manually + unit test


Thanks,

Peter Vary

[jira] [Created] (HIVE-14360) Starting BeeLine after using !save, there is an error logged: "Error setting configuration: conf"

2016-07-27 Thread Peter Vary (JIRA)

Peter Vary created HIVE-14360:
-

 Summary: Starting BeeLine after using !save, there is an error 
logged: "Error setting configuration: conf"
 Key: HIVE-14360
 URL: https://issues.apache.org/jira/browse/HIVE-14360
 Project: Hive
  Issue Type: Bug
  Components: Beeline
Affects Versions: 2.2.0
Reporter: Peter Vary
Assignee: Peter Vary
Priority: Minor


When saving the configuration in BeeLine the conf attribute is persisted, which 
should not. When loading the configuration this causes an error message to be 
printed:
{noformat}
Error setting configuration: conf: java.lang.IllegalArgumentException: No 
method matching "setconf" was found in org.apache.hive.beeline.BeeLineOpts.
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: contributor role request

2016-07-27 Thread Ashutosh Chauhan

Hi Gabor,

I have added you to contributor role in Hive Jira. Welcome to Hive!

Thanks,
Ashutosh

On Wed, Jul 27, 2016 at 6:51 AM, Gabor Szadovszky <
gabor.szadovs...@cloudera.com> wrote:

> Hi,
>
> Could you please grant me contributor access to the JIRA for Hive so that
> I can resolve issues? My username is gszadovszky.
> Thanks a lot in advance.
>
> Regards,
> Gabor

contributor role request

2016-07-27 Thread Gabor Szadovszky

Hi,

Could you please grant me contributor access to the JIRA for Hive so that I can 
resolve issues? My username is gszadovszky.
Thanks a lot in advance.

Regards,
Gabor

Re: Review Request 50503: HIVE-12954: NPE with str_to_map on null strings

2016-07-27 Thread Sergio Pena


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50503/#review143742
---



There are some extra spaces or tabs on the files. Could you remove them?


ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFStringToMap.java 
(line 102)


Remove extra spaces here.



ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFStringToMap.java
 (line 53)


Remove extra spaces



ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFStringToMap.java
 (line 78)


Remove extra spaces


- Sergio Pena


On July 27, 2016, 2:51 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50503/
> ---
> 
> (Updated July 27, 2016, 2:51 p.m.)
> 
> 
> Review request for hive, Aihua Xu and Sergio Pena.
> 
> 
> Bugs: HIVE-12954
> https://issues.apache.org/jira/browse/HIVE-12954
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-12954: NPE with str_to_map on null strings
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFStringToMap.java 
> ed60fbf 
>   
> ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFStringToMap.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/50503/diff/
> 
> 
> Testing
> ---
> 
> Added new unit test for GenericUDFStringToMap, please see the patch for 
> details.
> Also did manual testing.
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

Re: Review Request 50503: HIVE-12954: NPE with str_to_map on null strings

2016-07-27 Thread Aihua Xu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50503/#review143738
---




ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFStringToMap.java 
(line 95)


Are line 90- 93 already dealing with null delimiters ? Seems unnessary to 
check here?



ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFStringToMap.java
 (line 34)


Please add @Test annotation for unit tests.



ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFStringToMap.java
 (line 53)


Please remove all those empty spaces in this file. :)


- Aihua Xu


On July 27, 2016, 2:51 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50503/
> ---
> 
> (Updated July 27, 2016, 2:51 p.m.)
> 
> 
> Review request for hive, Aihua Xu and Sergio Pena.
> 
> 
> Bugs: HIVE-12954
> https://issues.apache.org/jira/browse/HIVE-12954
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-12954: NPE with str_to_map on null strings
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFStringToMap.java 
> ed60fbf 
>   
> ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFStringToMap.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/50503/diff/
> 
> 
> Testing
> ---
> 
> Added new unit test for GenericUDFStringToMap, please see the patch for 
> details.
> Also did manual testing.
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

Re: Review Request 50503: HIVE-12954: NPE with str_to_map on null strings

2016-07-27 Thread Chaoyu Tang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50503/#review143736
---




ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFStringToMap.java 
(line 102)


Could you remove the trailing space shown as the red blocks.


Patch looks good.

- Chaoyu Tang


On July 27, 2016, 2:51 p.m., Marta Kuczora wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50503/
> ---
> 
> (Updated July 27, 2016, 2:51 p.m.)
> 
> 
> Review request for hive, Aihua Xu and Sergio Pena.
> 
> 
> Bugs: HIVE-12954
> https://issues.apache.org/jira/browse/HIVE-12954
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-12954: NPE with str_to_map on null strings
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFStringToMap.java 
> ed60fbf 
>   
> ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFStringToMap.java
>  PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/50503/diff/
> 
> 
> Testing
> ---
> 
> Added new unit test for GenericUDFStringToMap, please see the patch for 
> details.
> Also did manual testing.
> 
> 
> Thanks,
> 
> Marta Kuczora
> 
>

Review Request 50503: HIVE-12954: NPE with str_to_map on null strings

2016-07-27 Thread Marta Kuczora


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50503/
---

Review request for hive, Aihua Xu and Sergio Pena.


Bugs: HIVE-12954
https://issues.apache.org/jira/browse/HIVE-12954


Repository: hive-git


Description
---

HIVE-12954: NPE with str_to_map on null strings


Diffs
-

  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFStringToMap.java 
ed60fbf 
  
ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFStringToMap.java
 PRE-CREATION 

Diff: https://reviews.apache.org/r/50503/diff/


Testing
---

Added new unit test for GenericUDFStringToMap, please see the patch for details.
Also did manual testing.


Thanks,

Marta Kuczora

[jira] [Created] (HIVE-14359) Spark fails might fail in LDAP authentication in kerberized cluster

2016-07-27 Thread Chaoyu Tang (JIRA)

Chaoyu Tang created HIVE-14359:
--

 Summary: Spark fails might fail in LDAP authentication in 
kerberized cluster
 Key: HIVE-14359
 URL: https://issues.apache.org/jira/browse/HIVE-14359
 Project: Hive
  Issue Type: Bug
Reporter: Chaoyu Tang
Assignee: Chaoyu Tang


When HS2 is used as a gateway for the LDAP users to access and run the queries 
in kerborized cluster, it's authentication mode is configured as LDAP and at 
this time, HoS might fail by the same reason as HIVE-10594. 
hive.server2.authentication is not a proper property to determine if a cluster 
is kerberized, instead hadoop.security.authentication should be used.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 50502: HIVE-14294: HiveSchemaConverter for Parquet doesn't translate TINYINT and SMALLINT into proper Parquet types

2016-07-27 Thread Aihua Xu


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50502/#review143729
---


Ship it!




Ship It!

- Aihua Xu


On July 27, 2016, 1:05 p.m., Gabor Szadovszky wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/50502/
> ---
> 
> (Updated July 27, 2016, 1:05 p.m.)
> 
> 
> Review request for hive, Aihua Xu, Cheng Lian, and Sergio Pena.
> 
> 
> Bugs: HIVE-14294
> https://issues.apache.org/jira/browse/HIVE-14294
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> HIVE-14294: HiveSchemaConverter for Parquet doesn't translate TINYINT and 
> SMALLINT into proper Parquet types
> 
> 
> Diffs
> -
> 
>   
> ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/HiveSchemaConverter.java
>  40f6256cc3d9e11e0f9760bfeab364316ebc638c 
>   
> ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestHiveSchemaConverter.java 
> 589b5b59a4757986c22355fd1557265efbd0c537 
> 
> Diff: https://reviews.apache.org/r/50502/diff/
> 
> 
> Testing
> ---
> 
> Unit tests updated; see patch for details. + manual tests
> 
> 
> Thanks,
> 
> Gabor Szadovszky
> 
>

[jira] [Created] (HIVE-14358) Add metrics for number of queries executed for each execution engine (mr, spark, tez)

2016-07-27 Thread Lenni Kuff (JIRA)

Lenni Kuff created HIVE-14358:
-

 Summary: Add metrics for number of queries executed for each 
execution engine (mr, spark, tez)
 Key: HIVE-14358
 URL: https://issues.apache.org/jira/browse/HIVE-14358
 Project: Hive
  Issue Type: Task
  Components: HiveServer2
Affects Versions: 2.1.0
Reporter: Lenni Kuff


HiveServer2 currently has a metric for the total number of queries ran since 
last restart, but it would be useful to also have metrics for number of queries 
ran for each execution engine. This would improve supportability by allowing 
users to get a high-level understanding of what workloads had been running on 
the server. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 50502: HIVE-14294: HiveSchemaConverter for Parquet doesn't translate TINYINT and SMALLINT into proper Parquet types

2016-07-27 Thread Gabor Szadovszky


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50502/
---

Review request for hive, Aihua Xu, Cheng Lian, and Sergio Pena.


Bugs: HIVE-14294
https://issues.apache.org/jira/browse/HIVE-14294


Repository: hive-git


Description
---

HIVE-14294: HiveSchemaConverter for Parquet doesn't translate TINYINT and 
SMALLINT into proper Parquet types


Diffs
-

  
ql/src/java/org/apache/hadoop/hive/ql/io/parquet/convert/HiveSchemaConverter.java
 40f6256cc3d9e11e0f9760bfeab364316ebc638c 
  ql/src/test/org/apache/hadoop/hive/ql/io/parquet/TestHiveSchemaConverter.java 
589b5b59a4757986c22355fd1557265efbd0c537 

Diff: https://reviews.apache.org/r/50502/diff/


Testing
---

Unit tests updated; see patch for details. + manual tests


Thanks,

Gabor Szadovszky

[jira] [Created] (HIVE-14357) TestDbTxnManager2#testLocksInSubquery failing in branch-2.1

2016-07-27 Thread Rajat Khandelwal (JIRA)

Rajat Khandelwal created HIVE-14357:
---

 Summary: TestDbTxnManager2#testLocksInSubquery failing in 
branch-2.1
 Key: HIVE-14357
 URL: https://issues.apache.org/jira/browse/HIVE-14357
 Project: Hive
  Issue Type: Bug
Reporter: Rajat Khandelwal


{noformat}
checkCmdOnDriver(driver.compileAndRespond("insert into R select * from S where 
a in (select a from T where b = 1)"));
txnMgr.openTxn("three");
txnMgr.acquireLocks(driver.getPlan(), ctx, "three");
locks = getLocks();
Assert.assertEquals("Unexpected lock count", 3, locks.size());
checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "T", null, 
locks.get(0));
checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "S", null, 
locks.get(1));
checkLock(LockType.SHARED_READ, LockState.ACQUIRED, "default", "R", null, 
locks.get(2));
{noformat}

This test case is failing. The expected order of locks is supposed to be T, S, 
R. But upon closer inspection, it seems to be R,S,T. 

I'm not much familiar with what these locks are and why the order is important. 
Raising this jira so while I try to understand it all. Meanwhile, if somebody 
can explain here, would be helpful. 





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 39768: HIVE-12181

2016-07-27 Thread Jesús Camacho Rodríguez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39768/#review143714
---




ql/src/test/results/clientpositive/tez/bucket_map_join_tez1.q.out (line 1715)


Data size is increasing in the Filter operator.


- Jesús Camacho Rodríguez


On July 27, 2016, 5:22 a.m., Ashutosh Chauhan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39768/
> ---
> 
> (Updated July 27, 2016, 5:22 a.m.)
> 
> 
> Review request for hive and Jesús Camacho Rodríguez.
> 
> 
> Bugs: HIVE-12181
> https://issues.apache.org/jira/browse/HIVE-12181
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Change hive.stats.fetch.column.stats value to true for MiniTezCliDriver
> 
> 
> Diffs
> -
> 
>   data/conf/tez/hive-site.xml 85f82b5 
>   ql/src/test/queries/clientpositive/bucket_map_join_tez1.q 95585db 
>   ql/src/test/queries/clientpositive/dynamic_partition_pruning.q 0c2ba83 
>   ql/src/test/queries/clientpositive/dynamic_partition_pruning_2.q 4a9532d 
>   ql/src/test/queries/clientpositive/explainuser_2.q da107dc 
>   ql/src/test/queries/clientpositive/explainuser_4.q 4ee5b4e 
>   ql/src/test/queries/clientpositive/mapjoin_mapjoin.q f4ea69d 
>   ql/src/test/queries/clientpositive/mrr.q 324f2b1 
>   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_1.q fc0edec 
>   ql/src/test/queries/clientpositive/tez_smb_main.q 1890aa7 
>   ql/src/test/queries/clientpositive/tez_vector_dynpart_hashjoin_1.q 22c5f9d 
>   ql/src/test/queries/clientpositive/unionDistinct_1.q fd7f075 
>   ql/src/test/queries/clientpositive/vectorized_dynamic_partition_pruning.q 
> 09847b1 
>   ql/src/test/results/clientpositive/tez/auto_join0.q.out 6d3f944 
>   ql/src/test/results/clientpositive/tez/auto_join1.q.out b3d4acc 
>   ql/src/test/results/clientpositive/tez/auto_join21.q.out 9c9083e 
>   ql/src/test/results/clientpositive/tez/auto_join29.q.out 2ff99a5 
>   ql/src/test/results/clientpositive/tez/auto_join30.q.out d045623 
>   ql/src/test/results/clientpositive/tez/auto_sortmerge_join_12.q.out aefe1a7 
>   ql/src/test/results/clientpositive/tez/auto_sortmerge_join_6.q.out cf260a7 
>   ql/src/test/results/clientpositive/tez/bucket2.q.out e78a1a7 
>   ql/src/test/results/clientpositive/tez/bucket3.q.out 250d03d 
>   ql/src/test/results/clientpositive/tez/bucket4.q.out b14c672 
>   ql/src/test/results/clientpositive/tez/bucket_map_join_tez1.q.out 30c4107 
>   ql/src/test/results/clientpositive/tez/bucket_map_join_tez2.q.out 3d65afc 
>   ql/src/test/results/clientpositive/tez/bucketpruning1.q.out bb9d3bb 
>   ql/src/test/results/clientpositive/tez/correlationoptimizer1.q.out 31dcd1f 
>   ql/src/test/results/clientpositive/tez/cross_join.q.out 83672fb 
>   ql/src/test/results/clientpositive/tez/ctas.q.out 9dffc0b 
>   ql/src/test/results/clientpositive/tez/cte_3.q.out fedbb7d 
>   ql/src/test/results/clientpositive/tez/cte_5.q.out 579b4f3 
>   ql/src/test/results/clientpositive/tez/cte_mat_1.q.out 7101f91 
>   ql/src/test/results/clientpositive/tez/cte_mat_2.q.out 7101f91 
>   ql/src/test/results/clientpositive/tez/cte_mat_3.q.out b700d44 
>   ql/src/test/results/clientpositive/tez/cte_mat_4.q.out bf13958 
>   ql/src/test/results/clientpositive/tez/cte_mat_5.q.out 52342c5 
>   ql/src/test/results/clientpositive/tez/disable_merge_for_bucketing.q.out 
> c1717e3 
>   ql/src/test/results/clientpositive/tez/dynamic_partition_pruning.q.out 
> 89987aa 
>   ql/src/test/results/clientpositive/tez/dynamic_partition_pruning_2.q.out 
> 21cd9c8 
>   ql/src/test/results/clientpositive/tez/dynpart_sort_opt_vectorization.q.out 
> 789dd5e 
>   ql/src/test/results/clientpositive/tez/dynpart_sort_optimization2.q.out 
> 6931398 
>   ql/src/test/results/clientpositive/tez/explainuser_1.q.out b8383fd 
>   ql/src/test/results/clientpositive/tez/explainuser_3.q.out 345d5f7 
>   ql/src/test/results/clientpositive/tez/explainuser_4.q.out 0b07a29 
>   ql/src/test/results/clientpositive/tez/groupby1.q.out b70f00a 
>   ql/src/test/results/clientpositive/tez/groupby2.q.out 7e6a5c1 
>   ql/src/test/results/clientpositive/tez/groupby3.q.out c46d1df 
>   ql/src/test/results/clientpositive/tez/having.q.out 65a54d0 
>   ql/src/test/results/clientpositive/tez/hybridgrace_hashjoin_1.q.out 575d3da 
>   ql/src/test/results/clientpositive/tez/hybridgrace_hashjoin_2.q.out e233052 
>   ql/src/test/results/clientpositive/tez/insert_into1.q.out b03bcc3 
>   ql/src/test/results/clientpositive/tez/insert_into2.q.out 8b1fdac 
>   ql/src/test/results/clientpositive/tez/join0.q.out 7d3122a 
>   ql/src/test/results/clientpositive/tez/join1.q.out 3f4c59b 
>   ql/src/test/results/clientpositive/tez/limit_pushdown.q.out 828

Re: Review Request 39768: HIVE-12181

2016-07-27 Thread Jesús Camacho Rodríguez


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/39768/#review143712
---




ql/src/test/results/clientpositive/tez/schema_evol_orc_vec_mapwork_part_all_primitive.q.out
 (line 150)


Follow-up: It seems here we are only taking into account a single column to 
calculate the data size (also see PARTIAL columns stats below). I guess we 
should use default size for the columns for which we do not have stats? 
Otherwise, we might end up underestimating the data size badly.



ql/src/test/results/clientpositive/tez/vector_coalesce.q.out (line 403)


0 size.



ql/src/test/results/clientpositive/tez/vector_left_outer_join.q.out (line 59)


0 size.



ql/src/test/results/clientpositive/tez/vector_outer_join3.q.out (line 288)


0 size.



ql/src/test/results/clientpositive/tez/vector_outer_join4.q.out (line 957)


0 size.


- Jesús Camacho Rodríguez


On July 27, 2016, 5:22 a.m., Ashutosh Chauhan wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/39768/
> ---
> 
> (Updated July 27, 2016, 5:22 a.m.)
> 
> 
> Review request for hive and Jesús Camacho Rodríguez.
> 
> 
> Bugs: HIVE-12181
> https://issues.apache.org/jira/browse/HIVE-12181
> 
> 
> Repository: hive-git
> 
> 
> Description
> ---
> 
> Change hive.stats.fetch.column.stats value to true for MiniTezCliDriver
> 
> 
> Diffs
> -
> 
>   data/conf/tez/hive-site.xml 85f82b5 
>   ql/src/test/queries/clientpositive/bucket_map_join_tez1.q 95585db 
>   ql/src/test/queries/clientpositive/dynamic_partition_pruning.q 0c2ba83 
>   ql/src/test/queries/clientpositive/dynamic_partition_pruning_2.q 4a9532d 
>   ql/src/test/queries/clientpositive/explainuser_2.q da107dc 
>   ql/src/test/queries/clientpositive/explainuser_4.q 4ee5b4e 
>   ql/src/test/queries/clientpositive/mapjoin_mapjoin.q f4ea69d 
>   ql/src/test/queries/clientpositive/mrr.q 324f2b1 
>   ql/src/test/queries/clientpositive/tez_dynpart_hashjoin_1.q fc0edec 
>   ql/src/test/queries/clientpositive/tez_smb_main.q 1890aa7 
>   ql/src/test/queries/clientpositive/tez_vector_dynpart_hashjoin_1.q 22c5f9d 
>   ql/src/test/queries/clientpositive/unionDistinct_1.q fd7f075 
>   ql/src/test/queries/clientpositive/vectorized_dynamic_partition_pruning.q 
> 09847b1 
>   ql/src/test/results/clientpositive/tez/auto_join0.q.out 6d3f944 
>   ql/src/test/results/clientpositive/tez/auto_join1.q.out b3d4acc 
>   ql/src/test/results/clientpositive/tez/auto_join21.q.out 9c9083e 
>   ql/src/test/results/clientpositive/tez/auto_join29.q.out 2ff99a5 
>   ql/src/test/results/clientpositive/tez/auto_join30.q.out d045623 
>   ql/src/test/results/clientpositive/tez/auto_sortmerge_join_12.q.out aefe1a7 
>   ql/src/test/results/clientpositive/tez/auto_sortmerge_join_6.q.out cf260a7 
>   ql/src/test/results/clientpositive/tez/bucket2.q.out e78a1a7 
>   ql/src/test/results/clientpositive/tez/bucket3.q.out 250d03d 
>   ql/src/test/results/clientpositive/tez/bucket4.q.out b14c672 
>   ql/src/test/results/clientpositive/tez/bucket_map_join_tez1.q.out 30c4107 
>   ql/src/test/results/clientpositive/tez/bucket_map_join_tez2.q.out 3d65afc 
>   ql/src/test/results/clientpositive/tez/bucketpruning1.q.out bb9d3bb 
>   ql/src/test/results/clientpositive/tez/correlationoptimizer1.q.out 31dcd1f 
>   ql/src/test/results/clientpositive/tez/cross_join.q.out 83672fb 
>   ql/src/test/results/clientpositive/tez/ctas.q.out 9dffc0b 
>   ql/src/test/results/clientpositive/tez/cte_3.q.out fedbb7d 
>   ql/src/test/results/clientpositive/tez/cte_5.q.out 579b4f3 
>   ql/src/test/results/clientpositive/tez/cte_mat_1.q.out 7101f91 
>   ql/src/test/results/clientpositive/tez/cte_mat_2.q.out 7101f91 
>   ql/src/test/results/clientpositive/tez/cte_mat_3.q.out b700d44 
>   ql/src/test/results/clientpositive/tez/cte_mat_4.q.out bf13958 
>   ql/src/test/results/clientpositive/tez/cte_mat_5.q.out 52342c5 
>   ql/src/test/results/clientpositive/tez/disable_merge_for_bucketing.q.out 
> c1717e3 
>   ql/src/test/results/clientpositive/tez/dynamic_partition_pruning.q.out 
> 89987aa 
>   ql/src/test/results/clientpositive/tez/dynamic_partition_pruning_2.q.out 
> 21cd9c8 
>   ql/src/test/results/clientpositive/tez/dynpart_sort_opt_vectorization.q.out 
> 789dd5e 
>   ql/src/test/results/clientpositive/tez/dynpart_sort_optimization2.q.out 
> 6931398 
>   ql/src/test/results/clientpositive/tez/explainuser_1.q.out b8383fd 
>   ql/src/test/results/clientpositive/tez/explainuser_3.q.out 345d5f

[jira] [Created] (HIVE-14356) ERROR [ExecutionCompletionThread #0[]]: impl.TaskRunnerCallable (:())

2016-07-27 Thread dancer (JIRA)

dancer created HIVE-14356:
-

 Summary: ERROR [ExecutionCompletionThread #0[]]: 
impl.TaskRunnerCallable (:())
 Key: HIVE-14356
 URL: https://issues.apache.org/jira/browse/HIVE-14356
 Project: Hive
  Issue Type: Bug
Reporter: dancer


2016-07-27T17:08:56,112 ERROR [ExecutionCompletionThread #0[]]: 
impl.TaskRunnerCallable (:()) - TezTaskRunner execution failed for : 
AppId=applicatio
n_1468307232009_0315, containerId=container_1_0315_01_03, 
Dag=select name,age from llaptest order by age(Stage-1), Vertex=Map 1, Fragment
Num=0, Attempt=2
 
java.lang.NoSuchMethodError: 
org.apache.tez.runtime.task.TezTaskRunner2.(Lorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/security/User
GroupInformation;[Ljava/lang/String;Lorg/apache/tez/runtime/api/impl/TaskSpec;ILjava/util/Map;Ljava/util/Map;Lcom/google/common/collect/Multimap;Lorg
/apache/tez/runtime/internals/api/TaskReporterInterface;Lcom/google/common/util/concurrent/ListeningExecutorService;Lorg/apache/tez/runtime/api/Objec
tRegistry;Ljava/lang/String;Lorg/apache/tez/runtime/api/ExecutionContext;JZLorg/apache/tez/hadoop/shim/HadoopShim;)V
at 
org.apache.hadoop.hive.llap.daemon.impl.TaskRunnerCallable.callInternal(TaskRunnerCallable.java:233)
  
at 
org.apache.hadoop.hive.llap.daemon.impl.TaskRunnerCallable.callInternal(TaskRunnerCallable.java:83)
   
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-07-27T17:08:56,113 ERROR [ExecutionCompletionThread #0[]]: 
impl.TaskExecutorService (:()) - Failed notification received: Stacktrace: 
java.lang.
NoSuchMethodError: 
org.apache.tez.runtime.task.TezTaskRunner2.(Lorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/security/UserGroupInfor
mation;[Ljava/lang/String;Lorg/apache/tez/runtime/api/impl/TaskSpec;ILjava/util/Map;Ljava/util/Map;Lcom/google/common/collect/Multimap;Lorg/apache/te
z/runtime/internals/api/TaskReporterInterface;Lcom/google/common/util/concurrent/ListeningExecutorService;Lorg/apache/tez/runtime/api/ObjectRegistry;
Ljava/lang/String;Lorg/apache/tez/runtime/api/ExecutionContext;JZLorg/apache/tez/hadoop/shim/HadoopShim;)V
at 
org.apache.hadoop.hive.llap.daemon.impl.TaskRunnerCallable.callInternal(TaskRunnerCallable.java:233)
  
at 
org.apache.hadoop.hive.llap.daemon.impl.TaskRunnerCallable.callInternal(TaskRunnerCallable.java:83)
   
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)  
 
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
  
at java.lang.Thread.run(Thread.java:745)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14355) Schema evolution for ORC in llap is broken for int to string conversion

2016-07-27 Thread Prasanth Jayachandran (JIRA)

Prasanth Jayachandran created HIVE-14355:


 Summary: Schema evolution for ORC in llap is broken for int to 
string conversion
 Key: HIVE-14355
 URL: https://issues.apache.org/jira/browse/HIVE-14355
 Project: Hive
  Issue Type: Bug
Affects Versions: 2.2.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran
Priority: Critical


When schema is evolved from any integer type to string then following 
exceptions are thrown in LLAP (Works fine in Tez). I guess this should happen 
even for other conversions.

{code}
hive> create table orc_integer(b bigint) stored as orc;
hive> insert into orc_integer values(100);
hive> select count(*) from orc_integer where b=100;
OK
1
hive> alter table orc_integer change column b b string;
hive> select count(*) from orc_integer where b=100;
// FAIL with following exception
{code}

{code:title=When vectorization is enabled}
2016-07-27T01:48:05,611  INFO [TezTaskRunner ()] 
vector.VectorReduceSinkOperator: RECORDS_OUT_INTERMEDIATE_Map_1:0,
2016-07-27T01:48:05,611 ERROR [TezTaskRunner ()] tez.TezProcessor: 
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: 
Hive Runtime Error while processing row
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:95)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:70)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:393)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
at 
org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
at 
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
at 
org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
at 
org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at 
org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error 
while processing row
at 
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:866)
at 
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
... 18 more
Caused by: java.lang.ClassCastException: 
org.apache.hadoop.hive.ql.exec.vector.LongColumnVector cannot be cast to 
org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector
at 
org.apache.hadoop.hive.ql.exec.vector.expressions.gen.FilterStringGroupColEqualStringGroupScalarBase.evaluate(FilterStringGroupColEqualStringGroupScalarBase.java:42)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:110)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
at 
org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
at 
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:774)
... 19 more
{code}

{code:title=When vectorization is disabled}
2016-07-27T01:52:43,328  INFO [TezTaskRunner 
(1469608604787_0002_26_00_00_0)] exec.ReduceSinkOperator: Using tag = -1
2016-07-27T01:52:43,328  INFO [TezTaskRunner 
(1469608604787_0002_26_00_00_0)] exec.OperatorUtils: Setting output 
collector: RS[4] --> Reducer 2
2016-07-27T01:52:43,329 ERROR [TezTaskRunner 
(1469608604787_0002_26_00_00_0)] io.BatchToRowReader: Error at row 0/1, 
column 0/1 org.apache.hadoop.hive.ql.exec.vector.LongColumnVector@7630e56a
java.lang.ClassCastException: 
org.apache.hadoop.hive.ql.exec.vector.LongColumnVector cannot be cast to 
org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector
at 
org.apache.hadoop.hive.ql.io.BatchToRowReader.nextString(BatchToRowReader.java:334)
 ~[hive-exec-2.2.0-SNAPSHOT.jar:2.2.0-SNAPSHOT]
at 
org.apache

[jira] [Created] (HIVE-14354) Cleanup ORC reader interfaces and redundant metadata objects

2016-07-27 Thread Prasanth Jayachandran (JIRA)

Prasanth Jayachandran created HIVE-14354:


 Summary: Cleanup ORC reader interfaces and redundant metadata 
objects
 Key: HIVE-14354
 URL: https://issues.apache.org/jira/browse/HIVE-14354
 Project: Hive
  Issue Type: Bug
  Components: ORC
Affects Versions: 2.2.0
Reporter: Prasanth Jayachandran
Assignee: Prasanth Jayachandran


ORC reader interface has to separated out to a new interface that only returns 
metadata. 
OrcTail should implement the new interface that only provides metadata 
information about the file.
OrcFileMetadata object contains many redundant information that can be obtained 
from OrcTail. This class should be cleaned up and OrcTail should be reused.
Remove all references to List and replace them with 
TypeDescription.
SchemaEvolution objects are created in multiple places. But the information 
required for creating SchemaEvolution is already present within the Reader 
(file types, projection and schema on read).
Schema on read can be specified only to RecordReader options. Reader options 
should also accept schema on read type description.
Consolidate Reader options and RecordReader options.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (HIVE-14353) Performance degradation after Projection Pruning in CBO

2016-07-27 Thread Nemon Lou (JIRA)

Nemon Lou created HIVE-14353:


 Summary: Performance degradation  after Projection Pruning in CBO
 Key: HIVE-14353
 URL: https://issues.apache.org/jira/browse/HIVE-14353
 Project: Hive
  Issue Type: Bug
  Components: CBO, Logical Optimizer
Affects Versions: 1.2.1
Reporter: Nemon Lou


TPC-DS with factor 1024.
Hive on Spark. 
With and without projection prunning,time spent are quite different.
The way to disable projection prunning : disable HiveRelFieldTrimmer in code 
and compile a new jar.
||queries||CBO_no_projection_prune||CBO||
|q27|   160|251 | 
|q7 |   200|312 |
|q88|   701|1092|
|q68|   234|345 |
|q39|53|78  |
|q73|   160|228 |
|q31|   463|659 |
|q79|   242|343 |
|q46|   256|363 |
|q60|   271|382 |
|q66|   198|278 |
|q34|   155|217 |
|q19|   184|256 |
|q26|   154|214 |
|q56|   262|364 |
|q75|   942|1303|
|q71|   288|388 |
|q25|   329|442 |
|q52|   142|190 |
|q42|   142|189 |
|q3 |   139|185 |
|q98|   153|203 |
|q89|   187|248 |
|q58|   264|340 |
|q43|   127|162 |
|q32|   174|221 |
|q96|   156|197 |
|q70|   320|404 |
|q29|   499|629 |
|q18|   266|329 |
|q21|   76 |92  |
|q90|   139|165 |




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

40 matches

Mail list logo